[trainer] add Trainer methods for metrics logging and saving

This PR introduces:
* [x] `trainer.log_metrics` - to perform consistent formatting for logged metrics
* [x] `trainer.save_metrics` - to save the metrics

This removes a lot of pointless noise from the example scripts and makes them much easier to read and understand. It doesn't take away from a user understanding the example, since these helper methods are just removing formatting and file saving.

If accepted it should be easy to replicate to other example scripts so that they all produce a consistent output and are all easier to read.

@sgugger


Tapas Tokenizer  makes DataFrame iterrows() iterator crazy ...

## Environment info

- `transformers` version: 4.3.2
- Platform: Colab Pro
- Python version: Python 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101    torch-scatter  2.0.5
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: Tesla P100
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
@n1t0, @LysandreJik
 -->

## Information

Model I am using : Tapas

Very very very strange (at least for me, a Computer Science newbie) with the function when the table ingested has been resampled with the pd.DataFrame.sample() method.

In the following block of code, the rows iterator returns corrupted rows with my table.
I have check the iterrows() outside the Tapas Tokenizer and the rows returned are correct.
But inside the Tokenizer, the rows are sometimes ok but sometimes Cell objects and corresponding to wrong rows !!

```
# Second, replace cell values by Cell objects
for row_index, row in table.iterrows():
    for col_index, cell in enumerate(row):
            table.iloc[row_index, col_index] = Cell(text=cell)
```
The direct result in my case is a crash in the normalize_for_match() method :
AttributeError: 'Cell' object has no attribute 'lower'
which is normal since several rows in the table now are of Cell type and not str.

I cannot see why the rows iterator suddenly returns corrupted data, for both Type and Values.

Thanks

Best regards

Jerome

The problem arises when using:
* [ X] my own modified scripts: I am using the Tapas Tokenizer with shuffled Pandas DataFrame for table.

The tasks I am working on is:
* [ X] my own task or dataset: Total R&D

## To reproduce

Steps to reproduce the behavior:

1. Use a standard Pandas DataFrame read from csv
2. Shuffle this DataFrame by using sample with frac=1
3. Tokenize the DataFrame as table using the Tapas Tokenizer

# Second, replace cell values by Cell objects
```
for row_index, row in table.iterrows():
    for col_index, cell in enumerate(row):
            table.iloc[row_index, col_index] = Cell(text=cell)
```

## Expected behavior
The iterrows() is returning unconsistent row information for both type and content.

<!-- A clear and concise description of what you would expect to happen. -->
The iterrows() should return consistent row values.


Making TF TransfoXL model compliant with AMP

# What does this PR do?

This PR makes the TF TransfoXL model compliant with AMP. All the slow tests are passing as well for these models.

These two models cannot be XLA compliant for now, as it seems that tf.where cannot be used in XLA if the x and y parameters are None. See the _get_global_attn_indices method which has this case. I have opened [an issue](https://github.com/tensorflow/tensorflow/issues/47211) on the TF repo in order to ask if it is an expected behavior or a bug.


NER label re-alignment always expects B labelled first sub-words

Good Second Issue

Issues that are more difficult to do than "Good First" issues - give it a try if you want!

## Environment info

- `transformers` version: 4.3.1
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

- bert, tokenizers, pipelines: @LysandreJik
- trainer, maintained examples: @sgugger

## Information

Model I am using (Bert, XLNet ...): [DistilBERT fine-tuned for conll03](https://huggingface.co/elastic/distilbert-base-cased-finetuned-conll03-english)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fine-tune a BERT model for NER/conll03 using the `run_ner.py` example script, all default values
2. Correct the label alignments, see [config.json](https://huggingface.co/elastic/distilbert-base-cased-finetuned-conll03-english/blob/main/config.json)
3. Infer using entities that have not been seen at training time, and are composed of multiple word-parts as defined by WordPiece (my assumption as to the cause).
4. Sub-words are labelled but pipeline re-grouping/label alignment relies on perfect sub-word labelling:

E.g. Accenture → A ##cc ##ent ##ure → B-ORG O O O → A (ORG)
E.g. Max Mustermann → Max Must ##erman ##n → B-PER I-PER I-PER O → Max Musterman (PER)
E.g. Elasticsearch → El ##astic ##sea #rch → O O I-MISC O → ##sea (MISC)

## Expected behavior

I would expect that the realignment takes the label from the first word part or the best scoring sub-word part and propogates that label to the entire word, never returning sub-words. The default in `run_ner.py` is to use a padded sub-word label at training as per the BERT paper, but I've not tried setting that to `False` yet as that's not the typical/standard practice.

E.g. Accenture → A ##cc ##ent ##ure → B-ORG O O O → Accenture (ORG)
E.g. Max Mustermann → Max Must ##erman ##n → B-PER I-PER I-PER O → Max Mustermann (PER)
E.g. Elasticsearch → El ##astic ##sea #rch → O O I-MISC O → Elasticsearch (MISC)

I'll add that it seems odd that this business logic is in the `pipeline`. When evaluating on conll03, I assume we are using the sub-words/first word, but this realignment should be considered during evaluation. As-is, I suspect the recall is lower than it should be.


Making TF T5 model compliant with AMP and XLA

# What does this PR do?

This PR makes the TF T5 model compliant with AMP and XLA. All the slow tests are passing as well for the model.


Making TF OpenAI GPT model compliant with AMP and XLA

# What does this PR do?

This PR makes the TF OpenAI GPT model compliant with AMP and XLA. All the slow tests are passing as well for the model.


Making TF MPNet model compliant with XLA

# What does this PR do?

This PR makes the TF MPNet model compliant with XLA. All the slow tests are passing as well for the model.


Making TF MobileBert model compliant with AMP

# What does this PR do?

This PR makes the TF MobileBert model compliant with AMP. All the slow tests are passing as well for the model.


Deberta Tokenizer convert_ids_to_tokens() is not giving expected results

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0
- Platform: Colab
- Python version: 3.9
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information
I am using Deberta Tokenizer. `convert_ids_to_tokens()` of the tokenizer is not working fine.

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset

## To reproduce

Steps to reproduce the behavior:

1. Get Debrta Tokenizer
```python
from transformers import DebertaTokenizer
deberta_tokenizer = DebertaTokenizer.from_pretrained('microsoft/deberta-base')
```
2. Encode Some Example Using Tokenizer
```python
example = "Hi I am Bhadresh. I found an issue in Deberta Tokenizer"
encoded_example = distilbert_tokenizer.encode(example)
```
3. Convert Ids to tokens:
```python
distilbert_tokenizer.convert_ids_to_tokens(encoded_example )
"""
Output: ['[CLS]', '17250', '314', '716', '16581', '324', '3447', '13', '314', '1043', '281', '2071', '287', '1024', '4835', '64', '29130', '7509', '[SEP]']
"""
```
[Colab Link For Reproducing](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/DebertaTokenizerIssue.ipynb)
## Expected behavior

It should return some tokens like this
```
['[CLS]', 'hi', 'i', 'am', 'b', '##had', '##resh', '.', 'i', 'found', 'an', 'issue', 'in', 'de', '##bert', '##a', 'token', '##izer', '[SEP]']
```
Not just convert an integer to string like the current behavior

#### Tagging SMEs for help:
@n1t0, @LysandreJik


Making TF Lxmert model compliant with AMP

# What does this PR do?

This PR makes the TF Lxmert model compliant with AMP. All the slow tests are passing as well for the model.


[Question]: Register new Tokenizer

Hi there,

I'm in the process of creating a new Transformer model. I have my own codebase and I'm using Transformers as an external library. If I implement a new Tokenizer that inherits from an existing one (say the BERT one) is there any way to "register" my new tokenizer so that Huggingface automatically instantiate it? I would like to support the `AutoTokenizer` API:

```python
tokenizer = AutoTokenizer.from_pretrained("heriot-watt/my_model_name")
```
And I would like that `AutoTokenizer` looks in my PYTHONPATH and automatically resolves the `Tokenizer` class with the name `my_model_name`. I've seen that currently, Transformers uses a hardcoded resolution strategy defined in `configuration_auto.py` or `tokenization_auto.py`. For instance, AllenNLP uses a nice register annotation to automatically resolve models, dataset reader and so on. What would be the best solution here?

Thanks for your answer,
Alessandro


Addition of on-the-fly loading for MLM training and fix for default pad_to_max_length value for TPU

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #10204, #10024

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
@sgugger @lhoestq @patil-suraj

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Linux-4.19.121-linuxkit-x86_64-with-debian-10.1
- Python version: 3.7.4
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: <NO>
- Using distributed or parallel set-up in script?: <NO>

### Who can help

Model:
https://huggingface.co/facebook/mbart-large-50-one-to-many-mmt
@patrickvonplaten, @patil-suraj

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (translation)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
import os
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_en = "The head of the United Nations says there is no military solution in Syria"
model = MBartForConditionalGeneration.from_pretrained(
    "facebook/mbart-large-50-one-to-many-mmt", cache_dir=os.getenv("cache_dir", "model"))
tokenizer = MBart50TokenizerFast.from_pretrained(
    "facebook/mbart-large-50-one-to-many-mmt", src_lang="en_XX")

model_inputs = tokenizer(article_en, return_tensors="pt")

# translate from English to Hindi
generated_tokens = model.generate(
    **model_inputs,
    forced_bos_token_id=tokenizer.lang_code_to_id["hi_IN"]
)
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => 'संयुक्त राष्ट्र के नेता कहते हैं कि सीरिया में कोई सैन्य समाधान नहीं है'

# translate from English to Chinese
generated_tokens = model.generate(
    **model_inputs,
    forced_bos_token_id=tokenizer.lang_code_to_id["zh_CN"]
)
decoded = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => '联合国首脑说,叙利亚没有军事解决办法'
print(decoded)
````

ERROR:

```
Traceback (most recent call last):
  File "src/translation/run.py", line 7, in <module>
    from transformers import MBartForConditionalGeneration, MBart50TokenizerFast
ImportError: cannot import name 'MBart50TokenizerFast' from 'transformers' (unknown location)
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
no error
<!-- A clear and concise description of what you would expect to happen. -->


Load custom models

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: RHEL 7
- Python version: 3.7
- PyTorch version (GPU?): 1.7.0 (GPU)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger @LysandreJik

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): A custom model

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [* ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

I created a custom model by extending the SqueezeBertPreTrainedModel and added another classification head for multi-task learning. Trained with Trainer and TrainingArguments successfully, and saved the model by calling trainer.save_model(TRAINED_MODEL_PATH). Everything worked fine.

However, when I tried to load the model by calling MyCustomModelClass.from_pretrained(TRAINED_MODEL_PATH, local_files_only=True), an error was thrown:

```
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/configuration_utils.py", line 424, in get_config_dict
    use_auth_token=use_auth_token,
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/file_utils.py", line 1086, in cached_path
    local_files_only=local_files_only,
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/file_utils.py", line 1259, in get_from_cache
    "Cannot find the requested files in the cached path and outgoing traffic has been"
FileNotFoundError: Cannot find the requested files in the cached path and outgoing traffic has been disabled. To enable model look-ups and downloads online, set 'local_files_only' to False.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ec2-user/workspaces/compressed_transformers/src/Compressed_transformers/compression/evaluate.py", line 135, in <module>
    local_files_only=True,
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/modeling_utils.py", line 962, in from_pretrained
    **kwargs,
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/configuration_utils.py", line 376, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/ec2-user/anaconda3/envs/pytorch_latest_p37/lib/python3.7/site-packages/transformers/configuration_utils.py", line 436, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for './SqueezeBert/results/best_checkpoint/config.json'. Make sure that:

- './SqueezeBert/results/best_checkpoint/config.json' is a correct model identifier listed on 'https://huggingface.co/models'

- or './SqueezeBert/results/best_checkpoint/config.json' is the correct path to a directory containing a config.json file

```

## To reproduce

Steps to reproduce the behavior:

1. Extend the SqueezeBertPreTrainedModel (maybe other PreTrainedModel classes as well) class and create a model with a dataset
2. Train the model with the dataset and save the model using trainer.save_model(MODEL_DIR)
3. Load the model by calling MyCustomModelClass.from_pretrained(MODEL_DIR)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
It shouldn't look for models from the internet or model classes available in the library when AutoModel or AutoConfig is not used. When MyCustomModelClass.from_pretrained(MODEL_DIR) is called, it should be able to look up config.json and load the checkpoint correctly.

<!-- A clear and concise description of what you would expect to happen. -->


microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract not available for tensorflow

@jplu
The above said model is available for pytorch but not for tensorflow. How to convert a pytorch checkpoint to tensorflow for this one? Is it possible for doing a contribution for the same?


[ci] scheduled job test

please ignore

this time a branch on huggingface and not a fork


[CI] force scheduled action hub re-run

please ignore

testing only SLOW pt job


[CI] force scheduled action hub re-run

please ignore


[CI] 2 fixes

This PR:
- fixes invalid port
- adds missing requirements install which lead to multiple test failures

@LysandreJik


[BUG] [Ray-Tune] ValueError: checkpoint not in list

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes

### Who can help

Models:
- tensorflow: @jplu

Library:

- ray/raytune: @richardliaw, @amogkam

 -->

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I want to do this as a text classification task where I have a sequence and I want to classify it into either one of the 20 labels (all of them numeric).

Whenever I start tuning/Hyperparameter search, it starts running the first trial and logs a bit, then goes to the second trial showing the first one to be "running". This is how the logs look like:-
```

You are using PopulationBasedTraining but you haven't enabled checkpointing. This means your trials will train from scratch everytime they are exploiting new configurations. Consider enabling checkpointing by passing `keep_checkpoints_num=1` as an additional argument to `Trainer.hyperparameter_search`.

== Status ==
Memory usage on this node: 4.3/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 1/100 (1 RUNNING)
+-----------------+----------+-------+-----------+-------+----------------+--------------+
| Trial name      | status   | loc   |   w_decay |    lr |   train_bs/gpu |   num_epochs |
|-----------------+----------+-------+-----------+-------+----------------+--------------|
| _inner_0755e982 | RUNNING  |       |  0.366291 | 4e-05 |              8 |           15 |
+-----------------+----------+-------+-----------+-------+----------------+--------------+

Result for _inner_0755e982:
  date: 2021-02-17_15-52-04
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.17737865447998
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8976
  eval_samples_per_second: 56.252
  experiment_id: 5d2db84f7e9745a997bfcadedbd7d440
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39957
  time_since_restore: 21.89615297317505
  time_this_iter_s: 21.89615297317505
  time_total_s: 21.89615297317505
  timestamp: 1613577124
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0755e982

== Status ==
Memory usage on this node: 6.3/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 2/100 (1 PENDING, 1 RUNNING)
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
| Trial name      | status   | loc              |   w_decay |    lr |   train_bs/gpu |   num_epochs |   eval_loss |   training_iteration |
|-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------|
| _inner_0755e982 | RUNNING  | 172.28.0.2:39957 |  0.366291 | 4e-05 |              8 |           15 |     8.17738 |                    1 |
| _inner_07580d98 | PENDING  |                  |  0.376876 | 6e-05 |              8 |           10 |             |                      |
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+

Result for _inner_0755e982:
  date: 2021-02-17_15-52-04
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.17737865447998
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8976
  eval_samples_per_second: 56.252
  experiment_id: 5d2db84f7e9745a997bfcadedbd7d440
  experiment_tag: 1_num_train_epochs=15,per_device_eval_batch_size=16,per_device_train_batch_size=8
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39957
  time_since_restore: 21.89615297317505
  time_this_iter_s: 21.89615297317505
  time_total_s: 21.89615297317505
  timestamp: 1613577124
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 0755e982

Result for _inner_07580d98:
  date: 2021-02-17_15-52-29
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.1666898727417
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8883
  eval_samples_per_second: 56.327
  experiment_id: e5cb4d5b00524454b7f673f971318b30
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39986
  time_since_restore: 21.889320135116577
  time_this_iter_s: 21.889320135116577
  time_total_s: 21.889320135116577
  timestamp: 1613577149
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 07580d98

== Status ==
Memory usage on this node: 6.3/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 3/100 (1 ERROR, 1 PENDING, 1 RUNNING)
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
| Trial name      | status   | loc              |   w_decay |    lr |   train_bs/gpu |   num_epochs |   eval_loss |   training_iteration |
|-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------|
| _inner_07580d98 | RUNNING  | 172.28.0.2:39986 |  0.376876 | 6e-05 |              8 |           10 |     8.16669 |                    1 |
| _inner_15e144f6 | PENDING  |                  |  0.196785 | 2e-08 |              8 |           10 |             |                      |
| _inner_0755e982 | ERROR    |                  |  0.366291 | 4e-05 |              8 |           15 |     8.17738 |                    1 |
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
Number of errored trials: 1
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name      |   # failures | error file                                                                                |
|-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _inner_0755e982 |            1 | /root/ray_results/tune_transformer_pbt/_inner_0755e982_1_num_train_epochs=15,per_device_eval_batch_size=16,per_device_train_batch_size=8_2021-02-17_15-51-42/error.txt |
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Result for _inner_07580d98:
  date: 2021-02-17_15-52-29
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.1666898727417
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.8883
  eval_samples_per_second: 56.327
  experiment_id: e5cb4d5b00524454b7f673f971318b30
  experiment_tag: 2_num_train_epochs=10,per_device_eval_batch_size=16,per_device_train_batch_size=8
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 39986
  time_since_restore: 21.889320135116577
  time_this_iter_s: 21.889320135116577
  time_total_s: 21.889320135116577
  timestamp: 1613577149
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 07580d98

Result for _inner_15e144f6:
  date: 2021-02-17_15-52-53
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.2146635055542
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.917
  eval_samples_per_second: 56.094
  experiment_id: 2dc02378bb5d4a20a7c6d0228ad81076
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 40016
  time_since_restore: 21.961735486984253
  time_this_iter_s: 21.961735486984253
  time_total_s: 21.961735486984253
  timestamp: 1613577173
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 15e144f6

== Status ==
Memory usage on this node: 6.2/25.5 GiB
PopulationBasedTraining: 0 checkpoints, 0 perturbs
Resources requested: 3/4 CPUs, 1/1 GPUs, 0.0/14.99 GiB heap, 0.0/5.18 GiB objects (0/1.0 accelerator_type:P100)
Result logdir: /root/ray_results/tune_transformer_pbt
Number of trials: 4/100 (2 ERROR, 1 PENDING, 1 RUNNING)
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
| Trial name      | status   | loc              |   w_decay |    lr |   train_bs/gpu |   num_epochs |   eval_loss |   training_iteration |
|-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------|
| _inner_15e144f6 | RUNNING  | 172.28.0.2:40016 |  0.196785 | 2e-08 |              8 |           10 |     8.21466 |                    1 |
| _inner_247daa04 | PENDING  |                  |  0.49907  | 5e-05 |              8 |           15 |             |                      |
| _inner_0755e982 | ERROR    |                  |  0.366291 | 4e-05 |              8 |           15 |     8.17738 |                    1 |
| _inner_07580d98 | ERROR    |                  |  0.376876 | 6e-05 |              8 |           10 |     8.16669 |                    1 |
+-----------------+----------+------------------+-----------+-------+----------------+--------------+-------------+----------------------+
Number of errored trials: 2
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Trial name      |   # failures | error file                                                                                |
|-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| _inner_0755e982 |            1 | /root/ray_results/tune_transformer_pbt/_inner_0755e982_1_num_train_epochs=15,per_device_eval_batch_size=16,per_device_train_batch_size=8_2021-02-17_15-51-42/error.txt |
| _inner_07580d98 |            1 | /root/ray_results/tune_transformer_pbt/_inner_07580d98_2_num_train_epochs=10,per_device_eval_batch_size=16,per_device_train_batch_size=8_2021-02-17_15-52-06/error.txt |
+-----------------+--------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Result for _inner_15e144f6:
  date: 2021-02-17_15-52-53
  done: false
  eval_accuracy: 0.14948453608247422
  eval_f1: 0.14948453608247422
  eval_loss: 8.2146635055542
  eval_precision: 0.14948453608247422
  eval_recall: 0.14948453608247422
  eval_runtime: 6.917
  eval_samples_per_second: 56.094
  experiment_id: 2dc02378bb5d4a20a7c6d0228ad81076
  experiment_tag: 3_num_train_epochs=10,per_device_eval_batch_size=16,per_device_train_batch_size=8
  hostname: 0df2f30fd76b
  iterations_since_restore: 1
  node_ip: 172.28.0.2
  objective: 0.14948453608247422
  pid: 40016
  time_since_restore: 21.961735486984253
  time_this_iter_s: 21.961735486984253
  time_total_s: 21.961735486984253
  timestamp: 1613577173
  timesteps_since_restore: 0
  training_iteration: 1
  trial_id: 15e144f6
```
And this is the error that usually pops up:-

```
2021-02-17 16:22:17,040	ERROR trial_runner.py:616 -- Trial _inner_34e77498: Error processing event.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trial_runner.py", line 586, in _process_trial
    results = self.trial_executor.fetch_result(trial)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/ray_trial_executor.py", line 609, in fetch_result
    result = ray.get(trial_future[0], timeout=DEFAULT_GET_TIMEOUT)
  File "/usr/local/lib/python3.6/dist-packages/ray/_private/client_mode_hook.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ray/worker.py", line 1456, in get
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TuneError): ray::ImplicitFunc.train_buffered() (pid=473, ip=172.28.0.2)
  File "python/ray/_raylet.pyx", line 480, in ray._raylet.execute_task
  File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 167, in train_buffered
    result = self.train()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/trainable.py", line 226, in train
    result = self.step()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 366, in step
    self._report_thread_runner_error(block=True)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 513, in _report_thread_runner_error
    ("Trial raised an exception. Traceback:\n{}".format(err_tb_str)
ray.tune.error.TuneError: Trial raised an exception. Traceback:
ray::ImplicitFunc.train_buffered() (pid=473, ip=172.28.0.2)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 248, in run
    self._entrypoint()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 316, in entrypoint
    self._status_reporter.get_checkpoint())
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 576, in _trainable_func
    output = fn()
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 651, in _inner
    inner(config, checkpoint_dir=None)
  File "/usr/local/lib/python3.6/dist-packages/ray/tune/function_runner.py", line 645, in inner
    fn(config, **fn_kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/integrations.py", line 160, in _objective
    local_trainer.train(resume_from_checkpoint=checkpoint, trial=trial)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 983, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1062, in _maybe_log_save_evaluate
    self._save_checkpoint(model, trial, metrics=metrics)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1130, in _save_checkpoint
    self._rotate_checkpoints(use_mtime=True)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1460, in _rotate_checkpoints
    checkpoints_sorted = self._sorted_checkpoints(use_mtime=use_mtime)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1448, in _sorted_checkpoints
    best_model_index = checkpoints_sorted.index(str(Path(self.state.best_model_checkpoint)))
ValueError: 'results/run-34e77498/checkpoint-10' is not in list

```
Seems like it is trying to retrieve the best model but there is some sort of bug there (note that I am not checkpointing due to lack of storage capacity).

This is a part of the code that may have a clue as to the reason for this bug:-

```

from ray.tune.suggest.hyperopt import HyperOptSearch
from ray.tune.schedulers import PopulationBasedTraining
from ray.tune import CLIReporter, JupyterNotebookReporter
from ray import tune
import random

pbt = PopulationBasedTraining(
    time_attr="training_iteration",
    metric="eval_accuracy",
    mode="max",
    perturbation_interval=10,  # every 10 `time_attr` units
                               # (training_iterations in this case)
    hyperparam_mutations={

        "weight_decay": tune.uniform(1, 0.0001),
        "seed": tune.uniform(1,20000),
        "learning_rate": tune.choice([1e-5, 2e-5, 3e-5, 4e-5, 5e-5, 6e-5, 2e-7, 1e-7, 3e-7, 2e-8]),
        "adafactor": tune.choice(['True','False']),
        "adam_beta1": tune.uniform(1.0, 0.0),
        "adam_beta2": tune.uniform(1.0, 0),
        "adam_epsilon": tune.choice([1e-8, 2e-8, 3e-8, 1e-9, 2e-9, 3e-10]),
        "max_grad_norm": tune.uniform(1.0, 0),

    })

reporter = JupyterNotebookReporter(
    overwrite = True,
    metric = 'eval_accuracy',

    parameter_columns={
        "weight_decay": "w_decay",
        "learning_rate": "lr",
        "per_device_train_batch_size": "train_bs/gpu",
        "num_train_epochs": "num_epochs"},

    metric_columns=["eval_acc", "eval_loss", "epoch", "training_iteration"])

tune_config = {
        "per_device_train_batch_size": 8,
        "per_device_eval_batch_size": 16,
        "num_train_epochs": tune.choice([10,15])
        }

def compute_objective(metrics):
  return metrics["eval_accuracy"]

best = trainer.hyperparameter_search(hp_space = lambda _: tune_config,
                                     n_trials=100, compute_objective=compute_objective, direction="maximize", backend='ray',    #search_alg=HyperOptSearch(metric='accuracy', mode='max', use_early_stopped_trials=True)
                                     scheduler=pbt, resources_per_trial={"cpu": 3, "gpu": 1}, keep_checkpoints_num=1,
                                     name = "tune_transformer_pbt", progress_reporter=reporter,
                                     search_alg=HyperOptSearch(metric='eval_accuracy', mode='max', use_early_stopped_trials=True),
                                     reuse_actors=True, checkpoint_at_end=True)
```
I can supply more code if requested, but it is more or less tweaked from official examples.
So basically, the first trial keeps running, the second is put to pending while for the continuation of the rest of the trials, it results in an error (strangely not with the first one).

I tried waiting till most trials have errored out to see whether it would continue to train the 1st trial but it just terminated giving the list of all trials that couldn't be completed.

> I am not sure where this bug is to be filed - rayproject or Huggingface, so I apologize in advance if I have posted in the wrong place.


TensorFlow Question-Answering example fails to run (cardinality error)

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.15.0-111-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased or roberta-base

The problem arises when using:
* [ ] the official example scripts: question-answering (run_tf_squad.py)

Error message:
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
87599it [01:00, 1437.03it/s]
10570it [00:11, 958.83it/s]
convert squad examples to features:   2%|_         | 1697/87599 [00:13<10:43, 133.40it/s][WARNING|squad.py:118] 2021-02-17 22:20:03,736 >> Could not find answer: 'municipal building and' vs. 'a municipal building'
convert squad examples to features:  50%|_____     | 43393/87599 [05:04<05:04, 145.24it/s][WARNING|squad.py:118] 2021-02-17 22:24:55,103 >> Could not find answer: 'message stick,' vs. 'a message stick'
convert squad examples to features: 100%|__________| 87599/87599 [10:10<00:00, 143.59it/s]
add example index and unique id: 100%|__________| 87599/87599 [00:00<00:00, 784165.53it/s]
convert squad examples to features: 100%|__________| 10570/10570 [01:14<00:00, 140.99it/s]
add example index and unique id: 100%|__________| 10570/10570 [00:00<00:00, 510000.04it/s]
[WARNING|integrations.py:60] 2021-02-17 22:31:16,214 >> Using the `WAND_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).
[INFO|trainer_tf.py:125] 2021-02-17 22:31:16,214 >> To use comet_ml logging, run `pip/conda install comet_ml` see https://www.comet.ml/docs/python-sdk/huggingface/
Traceback (most recent call last):
  File "run_tf_squad.py", line 256, in <module>
    main()
  File "run_tf_squad.py", line 250, in main
    trainer.train()
  File "/home/transformers/src/transformers/trainer_tf.py", line 457, in train
    train_ds = self.get_train_tfdataset()
  File "/home/transformers/src/transformers/trainer_tf.py", line 141, in get_train_tfdataset
    self.num_train_examples = self.train_dataset.cardinality().numpy()
AttributeError: '_AssertCardinalityDataset' object has no attribute 'cardinality'

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQUaD v1

## To reproduce

1. Use the latest master from huggingface/transformers
2. Go to examples/question-answering
3. Run WANDB_DISABLED=true python run_tf_squad.py --model_name_or_path roberta-base --output_dir model --max_seq_length 384 --num_train_epochs 2 --per_device_train_batch_size 8 --per_device_eval_batch_size 16 --do_train --do_eval --logging_dir logs --logging_steps 10 --learning_rate 3e-5 --no_cuda=True --doc_stride 128

Could you take a look @sgugger?


`compute_metrics` show better results than `generate` because target data leaks

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Linux-5.8.18-050818-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using:
- t5

The problem arises when using:
- my own modified scripts: (give details below)

The tasks I am working on is:
- my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. train model with `compute_metrics` function to monitor metrics
2. use `generate` to predict text with trained model

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I expect the metrics of `compute_metrics` to be equal to my generated text.

<!-- A clear and concise description of what you would expect to happen. -->

## More information

While training, I used `compute_metrics` to calculate the metric on my validation set every X steps. I was surprised to see that after training my model did not perform as expected using the `generate` function provided by huggingface.

After some digging through the code I think I understand what the problem is. [`compute_metrics`](https://github.com/huggingface/transformers/blob/cd48078ce59a195473729759c76d88ae612b0f7a/src/transformers/trainer.py#L1665) takes as input `preds`, which is [a collection of `logits`](https://github.com/huggingface/transformers/blob/cd48078ce59a195473729759c76d88ae612b0f7a/src/transformers/trainer.py#L1635) from [`prediction_step`](https://github.com/huggingface/transformers/blob/cd48078ce59a195473729759c76d88ae612b0f7a/src/transformers/trainer.py#L1630) which internally [calls `model` with the inputs and targets of the model](https://github.com/huggingface/transformers/blob/cd48078ce59a195473729759c76d88ae612b0f7a/src/transformers/trainer.py#L1733).
This means that the target text leaks into `preds.predictions` because `mode.forward` used the targets as [input for the decoder](https://github.com/huggingface/transformers/blob/1cd16512dc8060aa8c2419664f9cb83813ade4d5/src/transformers/models/t5/modeling_t5.py#L1331). This makes the metrics of `compute_metrics` seem much better than they really are.

In my opinion the target data should not be used to create `preds.predictions`. Maybe the `generate` function is a better fit.


Script for distilling zero-shot classifier to more efficient student

Distillation

Related to model distillation

Examples

Which is related to examples in general

This PR introduces a script that provides a way to improve the speed and memory performance of a zero-shot classifier by training a more efficient student model from the zero-shot teacher's predictions over an unlabeled dataset.

For a given sequence, the zero-shot classification pipeline requires each possible label to be fed through the large NLI model separately. This requirement slows results considerably, particularly for tasks with a large number of classes `K`.

Given (1) an unlabeled corpus and (2) a set of candidate class names, this script allows a user to train a standard classification head with `K` output dimensions. The script generates a softmax distribution for the provided data & class names, and a student classifier is then fine-tuned on these proxy labels. The resulting student model can be used for classifying novel text instances over these `K` classes with an order-of-magnitude boost in inference speed in addition to decreased memory usage.

A teacher NLI model can be distilled to a student model by running `distill_classifier.py` like so:

```
python distill_classifier.py \
--data_file unlabeled_data.txt \
--class_names_file class_names.txt \
--output_dir ./distilled_model
```

A number of other args are provided as well, such as `--teacher_name_or_path` and `--student_name_or_path` for specifying the pre-trained student & teacher models to be used (by default `roberta-large-mnli` and `distillbert-base-uncased`) and `--hypothesis_template` for customizing the [hypothesis template](https://huggingface.co/transformers/main_classes/pipelines.html#transformers.ZeroShotClassificationPipeline.__call__) used by the teacher zero-shot model. The training is implemented via `Trainer`, so any [`TrainingArguments`](https://huggingface.co/transformers/main_classes/trainer.html#trainingarguments) can be specified as well.

The resulting model can then be used trivially in a text classification pipeline or in any other way:

```python
model = AutoModelForSequenceClassification.from_pretrained("./distilled_model")
tokenizer = AutoTokenizer.from_pretrained("./distilled_model")
distilled_classifier = TextClassificationPipeline(model=model, tokenizer=tokenizer)
```

See the included [README.md](https://github.com/joeddav/transformers/blob/zero-shot-distillation/examples/research_projects/zero-shot-distillation/README.md) for more details and examples.

Soon I'll introduce a similar script for self-training an NLI model, boosting the model's performance after training on only unlabeled data, which model can then be subsequently distilled with this script like any NLI model.

**Update**: I also just added a link to a working [colab notebook demo](https://colab.research.google.com/drive/1mjBjd0cR8G57ZpsnFCS3ngGyo5nCa9ya?usp=sharing).


[trainer] refactor place_model_on_device logic, add deepspeed

This PR:

* refactors 3 places of `place_model_on_device` logic - into one public attribute with the same name as the `TrainingArguments.place_model_on_device` attribute
* adds deepspeed to that logic (it was missing in 2 places)

@sgugger


Upgrade transformers from 3.5.0 to 4.3.2 instance error

Hi guys. I tried to update the transformers module from the version 3.5.0 to the version 4.3.2.
After this upgrade, the code that previously was working now has some problems.

This is my code:
```
from transformers import AutoConfig, AutoTokenizer, TFAutoModel

config = AutoConfig.from_pretrained("amazon/bort")
model = TFAutoModel.from_pretrained("amazon/bort", config=config)
bert_main_layer = model.bert

encoder, pooler = bert_main_layer(input_ids_in, attention_mask=input_masks_in)

X = tf.keras.layers.Dropout(config.hidden_dropout_prob)(pooler)
X = tf.keras.layers.Dense(
    constants.CLASSES,
    kernel_initializer=tf.keras.initializers.TruncatedNormal(stddev=config.initializer_range),
    activation="softmax"
)(X)

model = tf.keras.Model(
    inputs=[input_ids_in, input_masks_in],
    outputs=[X]
)
```

In particular, when I instantiate the class `bert_model` (`bert_main_layer(input_ids_in, attention_mask=input_masks_in)`), with the 3.5.0 version it returns two tensors:
```
<tf.Tensor 'bert/encoder/layer_._3/output/LayerNorm/batchnorm/add_1:0' shape=(None, 333, 1024) dtype=float32>
<tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(None, 1024) dtype=float32>
```

With the version 4.3.2 it returns two strings:
```
last_hidden_state
pooler_output
```

The consequence is that now I have this exception on the Dense layer:
```
ValueError: Input 0 of layer dense is incompatible with the layer: : expected min_ndim=2, found ndim=0. Full shape received: []
```

Since I haven't found any documentation or guidance, can you please help me? What I'm doing wrong?
Thank you


[Trainer] doc update

Trainer doc update:

* [x] port the instructions to use the new `run_seq2seq.py` script
* [x] add clarifications to using DeepSpeed in the notebook

@sgugger


CUDA memory error on increasing the number of generations

I am getting CUDA memory error when generating text with num_return_sequences set to more than 100 for a self-trained gpt2 model. This is not expected since after every generation there should be nothing left in the GPU.


Question about (no_decay = ['bias', 'LayerNorm.weight']) in BERT(Transformer-based)

Hi, I have a Question about the BERT model code.
I saw "no_decay = ['bias', 'LayerNorm.weight']" in BERT code(especially, in Optimizer part). It seemed reasonable, however, did this prove to be better performance? Or is it just to speed up calculations?


ConvBert not compatible with torch v1.6

ConvBERT uses statements like `torch.multiply` which did not exist in pytorch v1.6 => ConvBERT is not compatible with v1.6 (cc @abhishekkrthakur).

This can easily be checked when running:

`pytest tests/test_modeling_convbert.py`

@LysandreJik, @sgugger - It would be great to test all the different pytorch versions in a slow test I think.


TransCoder

# What does this PR do?

Adds TransCoder https://github.com/facebookresearch/TransCoder


Add m2m100

# What does this PR do?

Adds the M2M100 model
https://github.com/pytorch/fairseq/tree/master/examples/m2m_100

Fixes #8054


[file_utils] do not gobble certain kinds of requests.ConnectionError

Backport from https://github.com/huggingface/huggingface_hub/pull/14/commits/34b7b70d07ab1c9fc2f7da603d47cb344e256af6

might close (or at the very least provide more transparency into) #8690, #10067, and others


Request to add Switch Transformer

Google has come up with yet another transformer: https://arxiv.org/pdf/2101.03961.pdf


Making TF Longformer-like models compliant with AMP

# What does this PR do?

This PR makes the TF Longformer-like models compliant with AMP. All the slow tests are passing as well for these models.

These two models cannot be XLA compliant for now, as it seems that `tf.where` cannot be used in XLA if the `x` and `y` parameters are `None`. See the `_get_global_attn_indices` method which has this case. I have opened [an issue](https://github.com/tensorflow/tensorflow/issues/47211) on the TF repo in order to ask if it is an expected behavior or a bug.


Multilabel Sequence Classification in trainer

Feature request

Request for a new feature

# 🚀 Feature request

We need to be able to use the trainer for multilabel classification problems.

## Motivation

Right now we create our models in the old fashioned way, with a sigmoid layer at the end so we can do multilabel. However if we could use the trainer directly, we wouldn't need to maintain different training scripts. Are there any plans for adding this to the trainer?

## Your contribution

I could try to help but I don't even know where to start with.

Thanks you very much for reading


Wav2Vec2 finetune

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Can you please share code to how to finetune transformers.Wav2Vec2ForCTC, or maybe on how to give labels to the model in order to get loss?

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Making TF GPT2 compliant with XLA and AMP

# What does this PR do?

This PR makes the TF GPT2 model compliant with XLA and AMP. All the slow tests are passing as well.


Introduce warmup_ratio training argument in both
TrainingArguments and TFTrainingArguments classes (#6673)

# What does this PR do?
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->
This PR will add a new argument `warmup_ratio` to both `TrainingArguments` and `TFTrainingArguments` classes. This can be used to specify the ratio of total training steps for which linear warmup will happen.
This is especially convenient when the user wants to play around with the `num_train_epochs` or `max_steps` arguments while keeping the ratio of warmup steps a constant.
Fixes #6673

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. [Link to the issue raised](https://github.com/huggingface/transformers/issues/6673).
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
Since modifications in trainer: @sgugger


Converting original T5 to be used in Transformers

I want to use original T5 checkpoint in Transformers library. I found multiple answers referring to `convert_t5_original_tf_checkpoint_to_pytorch.py` which does not seem to exist. Any other way? Or where can I find a (currently working) version of that file?


Showing individual token and corresponding score during beam search

## Who can help
@patrickvonplaten

## Information

Hello,

I am using beam search with a pre-trained T5 model for summarization. I would like to visualize the beam search process by showing the tokens with the highest scores, and eventually the chosen beam like this diagram:
![image](https://user-images.githubusercontent.com/35655790/108174867-fc4eda00-713a-11eb-8565-dc3b0c87cf14.png)
(Taken from https://huggingface.co/blog/how-to-generate)

**I am unsure how I can show the tokens and their corresponding scores.**

I followed the discussion https://discuss.huggingface.co/t/announcement-generationoutputs-scores-attentions-and-hidden-states-now-available-as-outputs-to-generate/3094 and https://github.com/huggingface/transformers/pull/9150.

Following the docs, upon calling `generate`, I have set `return_dict_in_generate=True`, `output_scores=True`

```
generated_outputs = model_t5summary.generate(
  input_ids=input_ids.to(device),
  attention_mask=features['attention_mask'].to(device),
  max_length=input_ids.shape[-1] + 2,
  return_dict_in_generate=True,
  output_scores=True,
  output_hidden_states=True,
  output_attentions=True,
  no_repeat_ngram_size=2,
  early_stopping=True,
  num_return_sequences=3,
  num_beams=5,
)
```

Now I have an instance of `BeamSearchEncoderDecoderOutput`.

If I understand the docs (https://huggingface.co/transformers/master/internal/generation_utils.html#generate-outputs) correctly, `scores` will provide me with what I want but I am unsure on how to use the `scores`.

Any help/pointers from the community would be greatly appreciated, thank you 🙏


Trainer.train() is stuck

Hi,
I'm training roberta-base using HF Trainer, but it's stuck at the starting itself. Here's my code -

```
train_dataset[0]
{'input_ids': tensor([  0, 100, 657,  ...,   1,   1,   1]),
 'attention_mask': tensor([1, 1, 1,  ..., 0, 0, 0]),
 'labels': tensor(0)}

val_dataset[0]
{'input_ids': tensor([    0, 11094,    14,  ...,     1,     1,     1]),
 'attention_mask': tensor([1, 1, 1,  ..., 0, 0, 0]),
 'labels': tensor(0)}

## simple test
model(train_dataset[:2]['input_ids'], attention_mask = train_dataset[:2]['attention_mask'], labels=train_dataset[:2]['labels'])
SequenceClassifierOutput(loss=tensor(0.6995, grad_fn=<NllLossBackward>), logits=tensor([[ 0.0438, -0.1893],
        [ 0.0530, -0.1786]], grad_fn=<AddmmBackward>), hidden_states=None, attentions=None)

train_args = transformers.TrainingArguments(
             output_dir='test_1',
             overwrite_output_dir=True,
             evaluation_strategy="epoch",
             per_device_train_batch_size=8,
             per_device_eval_batch_size=8,
             learning_rate=3e-5,
             weight_decay=0.01,
             num_train_epochs=2,
             load_best_model_at_end=True,
             )

trainer = transformers.Trainer(
             model=model,
             args=train_args,
             train_dataset=train_dataset,
             eval_dataset=val_dataset,
             tokenizer=tok,
             )

trainer.train()

```
I saw memory consumption and it is stuck at -

```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.51.06    Driver Version: 450.51.06    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:62:00.0 Off |                    0 |
| N/A   49C    P0    60W / 300W |   1756MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:8A:00.0 Off |                    0 |
| N/A   50C    P0    61W / 300W |   1376MiB / 32510MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
```

Plz let me know how to proceed further..


[Trainer] memory tracker metrics

This PR introduced memory usage metrics in Trainer:

* [x] adds `TrainerMemoryTracker` (pytorch only, no-op for tf), which records deltas of the first gpu and cpu of the main process - and records them for `init|train|eval|test` stages - if there is no gpu it reports cpu only.
* [x] adds `--skip_memory_metrics` to disable this new behavior - i.e. by default it'll print the memory metrics
* [x] adds `trainer.metrics_format` which will intelligently reformat the metrics to do the right thing - this is only for logger - moves manual rounding from the scripts into that helper method.
* [x] formats GFlops as GF number, so ` 2285698228224.0`, which is very unreadable and now it will be a nice `2128GF` (similar to `100MB`)
* [x] as a sample changes `run_seq2seq.py` to use  `trainer.metrics_format` - can replicate to other scripts in another PR.
* [x] changes the metrics logger in `run_seq2seq.py` to align data, so that it's easy to read the relative numbers e.g. allocated plus peak memory should be in the same column to make a quick read of the situation.
* [x] adds a new file_utils helper function `is_torch_cuda_available` to detect no gpu setups in one call.
* [x] adds a test
* [x] consistently use the strange `train/eval/test` trio - it's very confusing - but at least it's consistent - I proposed to fix this `examples`-wide in https://github.com/huggingface/transformers/issues/10165

Request: I beg you to allow me to restore the original refactored metrics dump logic in `run_seq2seq.py` - the current repetition doesn't help the readability and it's just dumping a dict - nothing ML/NLP specific here, there is nothing to understand there IMHO. and then it'd be easy to replicate this to other examples. Thanks. This is the original (and will need to add to it a few formatting entries I added in this PR):

https://github.com/huggingface/transformers/blob/e94d63f6cbf5efe288e41d9840a96a5857090617/examples/legacy/seq2seq/finetune_trainer.py#L132-L145

A picture is worth a thousand words:
```
export BS=16; rm -r output_dir; PYTHONPATH=src USE_TF=0 CUDA_VISIBLE_DEVICES=0 python examples/seq2seq/run_seq2seq.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --do_eval --do_train --do_predict --evaluation_strategy=steps  --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro  --val_max_target_length 128 --warmup_steps 500 --max_train_samples 100 --max_val_samples 100 --max_test_samples 100  --dataset_name wmt16-en-ro-pre-processed --source_prefix "translate English to Romanian: "
```
gives:
```
02/16/2021 17:06:39 - INFO - __main__ -   ***** train metrics *****
02/16/2021 17:06:39 - INFO - __main__ -     epoch                      =    1.0
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_cpu_alloc_delta   =    2MB
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_cpu_peaked_delta  =    0MB
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_gpu_alloc_delta   =  230MB
02/16/2021 17:06:39 - INFO - __main__ -     init_mem_gpu_peaked_delta  =    0MB
02/16/2021 17:06:39 - INFO - __main__ -     total_flos                 = 2128GF
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_cpu_alloc_delta  =   55MB
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_cpu_peaked_delta =    0MB
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_gpu_alloc_delta  =  692MB
02/16/2021 17:06:39 - INFO - __main__ -     train_mem_gpu_peaked_delta =  661MB
02/16/2021 17:06:39 - INFO - __main__ -     train_runtime              = 2.3114
02/16/2021 17:06:39 - INFO - __main__ -     train_samples              =    100
02/16/2021 17:06:39 - INFO - __main__ -     train_samples_per_second   =  3.028

02/16/2021 17:06:43 - INFO - __main__ -   ***** val metrics *****
02/16/2021 17:13:05 - INFO - __main__ -     epoch                     =     1.0
02/16/2021 17:13:05 - INFO - __main__ -     eval_bleu                 = 24.6502
02/16/2021 17:13:05 - INFO - __main__ -     eval_gen_len              =    32.9
02/16/2021 17:13:05 - INFO - __main__ -     eval_loss                 =  3.7533
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_cpu_alloc_delta  =     0MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_cpu_peaked_delta =     0MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_gpu_alloc_delta  =     0MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_mem_gpu_peaked_delta =   510MB
02/16/2021 17:13:05 - INFO - __main__ -     eval_runtime              =  3.9266
02/16/2021 17:13:05 - INFO - __main__ -     eval_samples              =     100
02/16/2021 17:13:05 - INFO - __main__ -     eval_samples_per_second   =  25.467

02/16/2021 17:06:48 - INFO - __main__ -     ***** test metrics *****
02/16/2021 17:06:48 - INFO - __main__ -     test_bleu                 = 27.146
02/16/2021 17:06:48 - INFO - __main__ -     test_gen_len              =  41.37
02/16/2021 17:06:48 - INFO - __main__ -     test_loss                 = 3.6682
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_cpu_alloc_delta  =    0MB
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_cpu_peaked_delta =    0MB
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_gpu_alloc_delta  =    0MB
02/16/2021 17:06:48 - INFO - __main__ -     test_mem_gpu_peaked_delta =  645MB
02/16/2021 17:06:48 - INFO - __main__ -     test_runtime              = 5.1136
02/16/2021 17:06:48 - INFO - __main__ -     test_samples              =    100
02/16/2021 17:06:48 - INFO - __main__ -     test_samples_per_second   = 19.556
```

To understand the memory reports:
- `alloc_delta` - is the difference in the used/allocated memory counter between the end and the start of the stage - it can be negative if a function released more memory than it allocated
- `peaked_delta` - is any extra memory that was consumed and then freed - relative to the current allocated memory counter - it is never negative - this is the mysterious cause of OOM, since normally it doesn't register when everything fits into the memory.
- so when you look at the metrics of any stage you add up `alloc_delta` + `peaked_delta` and you know how much memory was needed to complete that stage. But the two numbers need to be separate.

We can change the names if you'd like, but if we do, let's make sure that allocated/used shows up before peaked  when alphabetically sorted - as they should be read in that order.

Also it would be useful to have them of the same length so it's less noisy vertically. I was thinking perhaps to add `m` to `alloc`? Then it becomes perfect:
```
test_mem_cpu_malloc_delta =    0MB
test_mem_cpu_peaked_delta =    0MB
```
Logic behind `init`:
- since Trainer's `__init__` can consume a lot of memory, it's important that we trace it too, but since any of the stages can be skipped, I basically push it into the metrics of whichever stage gets to update metrics first, so it gets tacked on to that group of metrics. In the above example it happens to be `train`.

Logic behind nested calls:
- since eval calls may be intermixed with train calls, we can't handle nested invocations because `torch.cuda.max_memory_allocated` is a single counter, so if it gets reset by a nested eval call, train will report incorrect info. One day pytorch will fix this issue: https://github.com/pytorch/pytorch/issues/16266 and then it will be possible to be re-entrant, for now we will only track the outer level `train` / `evaluation` / `predict` functions.

After this addition we can already profile/detect regressions for specific training stages. But this doesn't give us the full picture as there other allocations outside of the trainer - i.e. in user's code. It's a start.

Down the road I may code a different version, based on pynvml, which gives somewhat different numbers, and has its own complications. But it gives you the exact gpu memory usage, so you know exactly how much memory is used or left. PyTorch only reports its internal allocations on the other hand.

@patrickvonplaten, this feature should give us already a partial way to track memory regression. So this could be the low hanging fruit you and I were discussing.

It also should be possible to extend the tracker to use TF, but I don't know anything about TF.

@sgugger, @patil-suraj, @LysandreJik, @patrickvonplaten


No module named 'tasks'

## Environment info

- `transformers` version: 4.3.2
- Platform: 5.10.8-200.fc33.x86_64
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (GPU)
- Tensorflow version (GPU?): 2.4.1 (GPU)
- Using GPU in script?: No.
- Using distributed or parallel set-up in script?: No.

@sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): allenai/scibert_scivocab_uncased

The problem arises when using:
* [X ] the official example scripts: (give details below)

I'm using the old NER script, since the model I'm using doesn't support Fast Tokenizers.
https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py
However, I get an error on trying to import "tasks.py". I did find a previous git issue for this, which recommended downloading said file.... but it no longer exists.

The tasks I am working on is:
* [X ] my own task or dataset: (give details below)
I'm using the bc2gm-corpus dataset.

## To reproduce

Steps to reproduce the behaviour:

1. Download the script from the provided github link.
2. Run it with any real model name, a real directory for data (doesn't need to include data), and an output directory.
I used the following command: python3 run_ner.py --model_name_or_path allenai/scibert_scivocab_uncased --data_dir bc2gm-corpus/conll --output_dir ./output

`  File "run_ner.py", line 323, in <module>
    main()
  File "run_ner.py", line 122, in main
    module = import_module("tasks")
  File "/home/dkaplan/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tasks' `
## Expected behavior

I expect the NER script to run, finetuning a model on the provided dataset.


Slow Multi-GPU DDP training with run_clm.py and GPT2

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0
- Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes (using official `run_clm.py`)
- Using distributed or parallel set-up in script?: using DDP with `run_clm.py`

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
- gpt2: @patrickvonplaten, @LysandreJik
- trainer, maintained examples: @sgugger

## Information

Model I am using (Bert, XLNet ...): gpt2-medium

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Generate some dummy text data (3000 examples) and save a csv:
```python
import pandas as pd
text = " ".join(100*["Here is some very long text."])
text = 3000*[text]
pd.Series(text).to_frame("text").to_csv("data_temp.csv",index=False)
```
2. Run official [`run_clm.py`](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) from examples using 3 gpus and DDP:
```shell
CUDA_VISIBLE_DEVICES="1,2,3" python -m torch.distributed.launch --nproc_per_node 3 run_clm.py \
--model_name_or_path gpt2-medium \
--do_train \
--output_dir /proj/semafor/kirill/tmp \
--per_device_train_batch_size 4 \
--block_size 128 \
--train_file data_temp.csv \
--fp16
```
3. Using 3 GeForce RTX 2080Ti with 11Gbs, tqdm says it should approximately take 1 hour: `48/4101 [00:38<53:51,  1.25it/s`. The memory in each GPU is maxed out: ` 10782MiB / 11019MiB `

4. Now, if I just run the same script on a single GPU:
```shell
CUDA_VISIBLE_DEVICES="3" python run_clm.py \
--model_name_or_path gpt2-medium \
--do_train \
--output_dir /proj/semafor/kirill/tmp \
--per_device_train_batch_size 4 \
--block_size 128 \
--train_file data_temp.csv \
--fp16
```

It's actually a little faster: `260/12303 [00:57<44:02,  4.56it/s`  and the GPU memory is not maxed out: `9448MiB / 11019MiB`

I can actually double the `--per_device_train_batch_size` from `4 -> 8` and get it down to under 30 mins per epoch: `138/6153 [00:36<26:30,  3.78it/s`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

So I expected that:
-  DDP training on 3 GPUs to be faster than a single GPU (It's actually a little slower).
-  If I can load a batch of 8 on a device in a single GPU mode then it should work in multi-GPU mode as well (it doesn't, I get OOM error).


the change from single mask to multi mask support for pytorch

# What does this PR do?

A draft PR for the Feature request to change from single mask to multi mask support for the fill mask pipeline.

As discussed this is one a draft PR to discuss the changes that need to be made to the output format to jointly support multiple and single mask in one pipeline call. The PR implements the change for Pytorch and code has not been pushed in yet for  when the keyword argument is called.

The pipeline tests are expected to fail since the output format changed.
#10158

Example code that tests this feature is below.
```
import json
from transformers import pipeline
unmasker = pipeline('fill-mask', model='bert-base-uncased')
t = unmasker("hi [MASK] morning I'm a [MASK] model.")
print(json.dumps(t, indent=4))
```
@LysandreJik


T5 relative attention bias: Discrepancy to original implementation

### Who can help
@patrickvonplaten

## Information

Model I am using: T5

In the huggingface TF T5 implementation, the relative attention bias only seems to be applied to the first layer of the stack. If I understand the original implementation correctly, though, it is applied to all layers there.

HF: https://github.com/huggingface/transformers/blob/master/src/transformers/models/t5/modeling_tf_t5.py#L570
Mesh: https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/transformer_layers.py#L263


fix deprecated reference `tokenizer.max_len` in glue.py

This is to fix deprecated reference to `tokenizer.max_len` with `tokenizer.model_max_length` - similar to [issue 8739](https://github.com/huggingface/transformers/issues/8739) and [PR 8604](https://github.com/huggingface/transformers/pull/8604).
See error example [in Colab here](https://colab.research.google.com/gist/poedator/f8776349e5c625ce287fc6fcd312fa1e/tokenizer-max_len-error-in-transformers_glue.ipynb).  it causes `AttributeError: 'BertTokenizer' object has no attribute 'max_len'`
The error happens when `glue_convert_examples_to_features()` is called without `max_length` parameter specified. In that case [line 119](https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/glue.py#L119) with wrong reference gets called. This simple fix should  do it.


[trainer] fix ignored columns logger

This PR fixes a confusing log entry that says:
```
The following columns in the evaluation set don't have a corresponding argument in `T5ForConditionalGeneration.forward` and have been ignored: .
```
when everything is in order.

@sgugger


discrepancy between the Huggingface T5Tokenizer and the original T5tokenizer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `Transformers` version: 4.3.2
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?): -
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

- tokenizers: @n1t0, @LysandreJik

## Information

Model I am using T5Tokenizer, and I adapted the code of  run_mlm.py [1] to use it with T5 tokenizer, when I run the code I am getting

```
This tokenizer does not have a mask token which is necessary for masked language modeling. "
ValueError: This tokenizer does not have a mask token which is necessary for masked language modeling. You should pass `mlm=False` to train on causal language modeling instead.
```

I checked the error and this is because:  tokenizer.mask_token  is None for T5Tokenizer, checking T5 paper, they use a masked language modeling with their seq2seq objective as the pretraining objective, so they must have trained a masked token as their paper says, could you give me some insight why masked token does not exist in huggingface implementation of T5Tokenizer and how I can correct this to be able to run run_mlm codes ? thank you

[1] https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py

## To reproduce

```
tokenizer = AutoTokenizer.from_pretrained("t5-small")
print(tokenizer.mask_token) => this is None
```

## Expected behavior

The masked token as per T5 paper should exist in T5Tokenizer.


Fix add_token_positions in custom datasets tutorial

Discussed in #10210. The example `add_token_positions` function incorrectly converts `answers[i]['answer_end']` to its corresponding tokenized index rather than `answers[i]['answer_end'] - 1`.


Making TF Funnel compliant with AMP

# What does this PR do?

This PR makes the TF Funnel model compliant with AMP. All the slow tests are passing as well.


Factor out methods

With PyTorch's DataParallel, it is not possible to simply iterate over parameters in order to find the `nn.Module`'s dtype or device.

Some efforts were made to catch the error (`StopIteration`) in most cases, but the some were forgotten. This PR factors the try/except in a method, which is applied everywhere instead.

Closes #10214


StopIteration Error when running beam search for squad 2.0

I'm using  `huggingface/transformers-pytorch-gpu:4.3.0` on Ubuntu DGX1 server with 8 V100 GPUs.
`NVIDIA-SMI 418.126.02   Driver Version: 418.126.02   CUDA Version: 10.1`

When running the step in `examples/question_answering/README.md` for beam search for squad 2.0

```
python run_qa_beam_search.py \
    --model_name_or_path xlnet-large-cased \
    --dataset_name squad_v2 \
    --do_train \
    --do_eval \
    --version_2_with_negative \
    --learning_rate 3e-5 \
    --num_train_epochs 4 \
    --max_seq_length 384 \
    --doc_stride 128 \
    --output_dir ./wwm_cased_finetuned_squad/ \
    --per_device_eval_batch_size=2  \
    --per_device_train_batch_size=2   \
    --save_steps 5000
```
Error
```
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/xlnet/modeling_xlnet.py", line 1978, in forward
    start_logits = self.start_logits(hidden_states, p_mask=p_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py", line 1241, in forward
    if next(self.parameters()).dtype == torch.float16:
StopIteration
```


Store FLOS as floats to avoid overflow.

# What does this PR do?

As pointed out in #10212, the `total_flos` stored as ints can result in overflowing errors: when in Python ints there is no risk, but when in distributed training, we use torch.int64 to gather all FLOS on all processes which can trigger that error.

Fixes #10212