noamaanMulla-03/transformers-issues huggingface dataset:
support new marian models
Author: patil-suraj
Pull request: yes

# What does this PR do?

This PR updates the Marian model:
1. To allow not sharing embeddings between encoder and decoder.
2. Allow tying only decoder embeddings with `lm_head`.
3. Separate two `vocabs` in `tokenizer` for `src` and `tgt` language

To support this, the PR introduces the following new methods:
- `get_decoder_input_embeddings` and `set_decoder_input_embeddings`
To get and set the decoder embeddings when the embeddings are not shared. These methods will raise an error if the embeddings are shared.
- `resize_decoder_token_embeddings`
To only resize the decoder embeddings. Will raise an error if the embeddings are shared.

This PR also adds two new config attributes to `MarianConfig`:
- `share_encoder_decoder_embeddings`: to indicate if emb should be shared or not
- `decoder_vocab_size`: to specify the vocab size for decoder when emb are not shared.

And the following methods from `PreTrainedModel` class are overridden to support these changes:
- `tie_weights`
- `_resize_token_embeddings`

Fixes #15109 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15831). All of your documentation changes will be reflected on that endpoint.
Fix duplicate arguments passed to dummy inputs in ONNX export
Author: lewtun
Pull request: yes

# What does this PR do?

This PR fixes: 

- a bug that was introduced in #15658 where a preprocessor and tokenizer were being passed together to the `generate_dummy_inputs()` function during the ONNX export.
- an oversight in the refactoring of the ONNX config for M2M-100

It also removes problematic TensorFlow integration tests, where the model implementation doesn't have parity with the PyTorch one (e.g. `camembert-base` is missing the causal LM head in TensorFlow). I'll address those issues in separate PRs as it involves touching the TensorFlow modeling files.

With these fixes, all slow ONNX tests now pass in all environments (only `torch`, only `tensorflow`, `torch` and `tensorflow`):

```bash
RUN_SLOW=1 python -m pytest tests/onnx/test_onnx_v2.py
```

cc @michaelbenayoun 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update ArrowWriter path in examples
Author: mariosasko
Pull request: yes

`ArrowWriter` is no longer in the top-level namespace in `datasets` v2.0 (see https://github.com/huggingface/datasets/pull/3875#discussion_r822924405), so I'm replacing the`datasets.ArrowWriter` occurances with `datasets.arrow_writer.ArrowWriter`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Both are research projects that have `dataets` pinned to version 1.0.0 in their requirements, so I think they should be left as is.
FIX: updating doc/example for fine-tune for downstream Token Classification
Author: davidsbatista
Pull request: yes

# What does this PR do?

The documentation for fine tuning for downstream tasks, specifically for, Token Classification has an error, in the command AutoModelForTokenClassification command 

`model = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=2)`

it should be instead 14

`model = AutoModelForTokenClassification.from_pretrained(model_checkpoint, num_labels=14)`

check the following command:

`len({x for x in sample for sample in tokenized_wnut["train"]['labels']})`

14 labels:
- The `WNUT’17 Emerging and Rare Entities task` dataset has 13 labels 
- `-100` to the special tokens `[CLS]` and `[SEP]`


@sgugger I think you are the one reviewing this types of PR, i.e. doc related.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
MarianMTModel is not trained after finished training and loading
Author: marekninja
Pull request: no

## Environment info

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.4.0-92-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.12.0.dev20220208+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten

## Information

I am training MarianMTModel for translation from scratch on EuroParl dataset. 
* validation BLEU during training is cca 30, checkpointing every 5000 steps, with load_best_model_at_end argument
* At the end of the training I save the model with `trainer.save_model('./trained')`.
* I then load this model with `MarianMTModel.from_pretrained("./trained")`. Loaded model does not appear to be trained, BLEU score on validation set is around 4
* Came around this: 
https://github.com/huggingface/transformers/blob/e66743e6c9601a4b12ffc2335a9f60a41d1ca60c/src/transformers/models/marian/modeling_marian.py#L1211-L1224
* Finally, solved the issue by setting `model._keys_to_ignore_on_save = None`

I believe the problem is exactly the same as in issue #9525

I do not know about `_keys_to_ignore_on_load_missing`, but I think the `_keys_to_ignore_on_save` should be None.

My config also has `"static_position_embeddings": true,`, but still it seems like, embedings are trained too...


## To reproduce

Steps to reproduce the behavior:
1. Train MarianMTModel from scratch
2. Save checkpoint/best model with `trainer.save_model('./trained')`
3. Load checkpoint with from_pretrained
4. BLEU score is not the same (it is much lower)

## Expected behavior

Model should also save weights of positional  #embeddings.



Comments:
Found out the real issue. I was resetting the model incorrectly... I did reset_parameters on all layers (including positional embeddings), but in combination with static positional embeddings it made the model unusable after loading. My bad.
🚀 Add early stopping to the trainer
Author: BramVanroy
Pull request: no
Labels: name: High-Level feature

name: Good First Issue

# 🚀 Feature request

The trainer ([pt](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py), [tf](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py)) is an easy access point for users who rather not spend too much time building their own trainer class but prefer an out-of-the-box solution. Even though `transformers` was never meant to be a fully fletched training library, it might please users to add an additional feature: early stopping. 

## Motivation

Early stopping ensures that the trainer does not needlessly keep training when the loss does not improve. This saves time, money, and let's not forget the trees. 😉 Performance-wise this should not lead to different results.

## Your contribution

At the moment I cannot work on this, but here are my thoughts:

- a training argument should be added ([pt](https://github.com/huggingface/transformers/blob/master/src/transformers/training_args.py), [tf](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py)). This would only work when `evaluate_during_training` is enabled.
- for PyTorch: at every evaluation step, an early stopper (can be a separate class even) checks if the loss has improved in the last n steps. Potentially with a minimal threshold that the loss should have improved. If not, the trainer should stop
- for Tensorflow: I don't have experience with TF myself, but I assume one could use [`tf.keras.callbacks.EarlyStopping`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/EarlyStopping).

Comments:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Looking at the interest this topic has, I am bumping it to re-open it.

Hi,

So when #4186 is closed, this will close as well? Or is there any more changes expected. on this issue, apart from what #4186 adds?

Thanks


If I've understood things correctly, I think #4186 only addresses the Pytorch implementation of the trainer. @BramVanroy if that's the case I'm happy to work on implementing this feature in Tensorflow (trainer_tf.py).

@san7988 @KMFODA This issue should not directly be closed when that PR is merged because as @KMFODA mentions, it only seems to address PyTorch. A PR for Tensorflow is also welcome!

Thanks for clarifying @BramVanroy. Apologies I was out for the past month due to a personal issue. I'll submit a PR for Tensorflow early stopping now.

An early stopping callback has now been introduced in the PyTorch trainer by @cbrochtrup! 👏 

AFAIK the implementation the TF Trainer is still under way (https://github.com/huggingface/transformers/pull/7533) so I'll keep this topic open for now.

I gather from the conversation on #7533 that this issue should now be closed; is that correct, @BramVanroy ?
Fix a TF test name (LayoutLMModelTest)
Author: ydshieh
Pull request: yes

# What does this PR do?

Change `LayoutLMModelTest` to `TFLayoutLMModelTest` in `test_modeling_tf_layoutlm.py`.



Comments:
_The documentation is not available anymore as the PR was closed or merged._

merge now (think it's not necessary to wait Matt's approval for this tiny and obvious fix :-) )
Move QDQBert in just PyTorch block
Author: sgugger
Pull request: yes

# What does this PR do?

This PR moves `QDQBert` in the main init to make it accessible as soon as PyTorch is installed. This is because otherwise, building the documentation requires to have the quantization modules installed, which is not available in Python 3.9 and makes it harder for a contributor to build the doc. The code of the model file is clean and properly isolates those modules, so this is safe to do. The user will have an explicit error message when trying to use the models, since the base model contains the `requires_backends(self, "pytorch_quantization")` line.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Remove assertion over possible activation functions in DistilBERT
Author: mfuntowicz
Pull request: yes

Remove old code remaining in DistilBERT preventing the use of alternative activation functions.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Failures look unrelated to this PR, let me know if there is anything to be done on my side 👍🏻 

Good point, updated both codebase.
Documentation error 
Author: Moumeneb1
Pull request: no

## 

Hi, I think there is a tiny issue in the documentation of using pipelines

The current [doc](https://github.com/huggingface/transformers/blob/4a6a35bc6506dd5deb528580184c39e3e778d53b/docs/source/main_classes/pipelines.mdx) says that the KeyDataset class is in the transformers.pipelines.base, however the class is actually in the [pt_utils](https://github.com/huggingface/transformers/blob/19d37c2dd36d73537a6855c56808e524fb584459/src/transformers/pipelines/pt_utils.py)


```python
#The current doc (not working)
from transformers.pipelines.base import KeyDataset
```

```python
#The actual class is in 
from transformers.pipelines.pt_utils import KeyDataset
```

Running the current doc's examples won't work : 
```
ImportError: cannot import name 'KeyDataset' from 'transformers.pipelines.base' (/usr/local/lib/python3.7/dist-packages/transformers/pipelines/base.py)
```

Comments:
Hi,

Thanks for spotting. Feel free to open a PR to fix this!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Update troubleshoot guide
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR adds more content to the troubleshooting guide. I filtered the repo issues for errors and browsed the most viewed topics on the forums to find some of the most common issues users encounter and how to resolve them. Updates include:

- CUDA out of memory errors
- Unable to load a saved TF model (thanks @gante)
- ImportError
- CUDA error: device-side assert triggered

@ydshieh also mentioned users have trouble with [`resize_token_embeddings`](https://github.com/huggingface/transformers/issues?q=resize_token_embeddings), but I'm not sure what exactly users are having issues with (seems like it could be more than one thing?). Let me know if you have more context on this, or if it should be included! :) cc @patrickvonplaten @patil-suraj 

Feel free to let me know if there are any other common issues that can be documented!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Another `resize_token_embeddings()` issue 👉  https://github.com/huggingface/transformers/issues/16008
Add missing type hints for all flavors of RoBERTa PyTorch models.
Author: chainyo
Pull request: yes

# What does this PR do?

I added type hints for all `RoBERTa` PyTorch classes as described in #16059 .
As I commented on the linked issue, `CamemBERT` inherits from `RoBERTa`, so it should be also good for PyTorch version of `CamemBERT` models.

@Rocketknight1 

I was not sure about `encoder_hidden_states` and `encoder_attention_mask` which have `torch.FloatTensor` as type in the docstring, but I don't know if it needs to be specified or `torch.Tensor` is enough. Tell me if I need to change that.


**EDIT:** Oh it seems that I have put some `torch.LongTensor` like `inputs_ids` as `torch.Tensor`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ChainYo it's totally okay to just use `torch.Tensor`, specifying `torch.FloatTensor` or `torch.LongTensor` is a bonus but not at all necessary!

Also, this PR looks great, but note that the methods can return a tuple if `return_dict` is False, so the real return type is usually `Union[Tuple, OutputClass]` and not just `OutputClass`!

> @ChainYo it's totally okay to just use `torch.Tensor`, specifying `torch.FloatTensor` or `torch.LongTensor` is a bonus but not at all necessary!
> 
> Also, this PR looks great, but note that the methods can return a tuple if `return_dict` is False, so the real return type is usually `Union[Tuple, OutputClass]` and not just `OutputClass`!

Fine, I will change that.
[Fix doc example] FSMT
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix doc example. 

Without this fix, I get the following error


```
            batch_size = inputs_tensor.shape[0]
          File "/home/yih_dar_huggingface_co/transformers/src/transformers/tokenization_utils_base.py", line 253, in __getattr__
            raise AttributeError
        AttributeError
```

where `inputs_tensor` has type `BatchEncoding`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Audio/vision task guides
Author: stevhliu
Pull request: yes
Labels: name: Documentation

This PR adds fine-tuning guides for image classification, audio classification, and ASR since the APIs for these tasks are stable. Some general notes:

- The audio guides don't have TensorFlow sections yet.
- For image classification, there is a difference between the PyTorch and TensorFlow preprocessing methods. The TensorFlow example doesn't really add any transforms because I couldn't figure out how to integrate the Keras preprocessing/augmentation layers. ☹️ If we want to show some more substantial transforms, let me know and I can share my notebook or create an issue.
- Would be cool to add a widget to each guide so users can see the end result and what they can create!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15808). All of your documentation changes will be reflected on that endpoint.
[ZeRO] Fixes issue with embedding resize
Author: jeffra
Pull request: yes
Labels: name: DeepSpeed

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #16008.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@stas00 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[Deepspeed] add support for bf16 mode
Author: stas00
Pull request: yes
Labels: name: WIP

This PR:

- adds support for bfloat16 for ZeRO-1, ZeRO-2 and ZeRO-3 to HF/DS integration.
- most functional tests are now run for bf16 as well - so this PR almost doubles the number of tests. model zoo tests are left with fp16 for now, as both should work the same.
- docs are updated to document the new feature

Requirements:

1. [x] merged https://github.com/huggingface/transformers/pull/13207
2. [x] ZeRO-3 support merged https://github.com/microsoft/DeepSpeed/pull/1453
3. [x] Need new deepspeed release after the above is merged 
4. [x] update version dependency table

---------------------------

for users who want to try this early:

1. add `--bf16` to your previous fp16 or fp32 deepspeed command line 

done ;)

@sgugger 

Comments:
Thanks a lot for catching those few development process leftovers, Sylvain! Much appreciating. All cleaned up now.
Change unpacking of TF Bart inputs to use decorator
Author: osanseviero
Pull request: yes

# What does this PR do?

As part of #16051  this PR changes the unpacking of the inputs of bart to use the decorator instead.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@Rocketknight1 

Am I a TF engineer now? :sweat_smile: 


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Just straight-up assigning @gante to review the decorator PRs 👼
Add type annotations for segformer pytorch
Author: p-mishra1
Pull request: yes

This PR adds type annotation for segformer pytorch as mentioned in #16059.
@Rocketknight1 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I believe there are some minor mistakes in the types. `Optional[...]` is a shorthand notation for `Union[..., None]`, see [python doc](https://docs.python.org/3/library/typing.html#typing.Optional). Thus it must be used when `parameter : ... = None`. 

This PR makes `Optional` the following parameters in this [`forward method`](https://github.com/p-mishra1/transformers/blob/6ed256d4b04e99f24f62268c1f81efc47a5870f4/src/transformers/models/segformer/modeling_segformer.py#L396)

```
output_attentions: Optional[bool] = False,
output_hidden_states: Optional[bool] = False,
return_dict: Optional[bool] = True,
 ```
 
 While to be correct with the `Optional` definition given in python. 
 
 ```
output_attentions: bool = False,
output_hidden_states: bool = False,
return_dict: bool = True,
 ```
 
Since the code in the `forward` method doesn't check `output_attentions is not None`, to my understanding they should not be `Optional`.

cc @Rocketknight1
Add class re-weighting mechanism for classification tasks
Author: francisco-perez-sorrosal
Pull request: yes

In supervised ML classification problems, specially multiclass/multilabel, the classes in the datasets used to train a model may have very different distribution of discrete values, which is usually called class imbalance. The class imbalance may cause biases in the training of the models. DL frameworks like PyTorch or TF, allow in their loss functions (e.g. CrossEntropyLoss, BCEWithLogitsLoss...) certain parameters to adjust the possible imbalance of data in the classes. Currently, the HF framework does not allow to take advantage of these parameters, so it would be nice to have support of a class re-weighting mechanism at the HF level to improve the possible biases in the models when doing fine-tunings.

# What does this PR do?

Tentative implementation for a class re-weighting mechanism for classification tasks. WIP


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15449). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.