Adding a new `return_full_text` parameter to TextGenerationPipeline.

# What does this PR do?

For text-generation, it's sometimes used as prompting text.
In that context, prefixing `generated_text` with the actual input
forces the caller to take an extra step to remove it.

The proposed change adds a new parameter (for backward compatibility).
`return_full_text` that enables the caller to prevent adding the prefix.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@LysandreJik
@patrickvonplaten

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[GA forks] Test on every push

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Some model use serve previous version can not do inference in web api.

the model serve in
https://huggingface.co/wptoux/albert-chinese-large-qa
can not do inference by click “compute” button.
because it use transformers 3.0.2 but can not
properly load in current version.
I think model online server should consider its implement transformer version.


Labeled pull requests


Add XLA test

# What does this PR do?

In the same spirit than for the mixed precision test, this PR adds one for XLA compliancy.


TFBart lables consider both pad token and -100

For #9770,

1. ```TFBartModels``` use -100 as a masking token for ```decoder_input_ids``` and ```compute_loss``` like other models(```T5```).
2. For legacy, all the  ```padding token``` in ```labels``` are replace by ```-100`` token.

Below examples show the same result for ```labels``` with ```-100 token``` or ```padding token```, where previously ```Nan``` are shown in the latter case(#9770).

```
import tensorflow as tf
from transformers import BartTokenizer, TFBartForConditionalGeneration

tokenizer = BartTokenizer.from_pretrained("facebook/bart-base")
model = TFBartForConditionalGeneration.from_pretrained("facebook/bart-base")

inputs = tokenizer("My dog is <mask>", return_tensors='tf', truncation=True, max_length=16, padding="max_length")
labels_ids = tokenizer("My dog is cute", return_tensors='tf', truncation=True, max_length=16, padding="max_length").input_ids

## labels padding_token = 1
loss = model(inputs, labels=labels_ids)[0]
print(labels_ids)
print(loss)

## labels padding_token = -100
labels_ids = tf.where(
    labels_ids == 1, tf.fill(tf.shape(labels_ids), tf.constant(-100, dtype='int32')), labels_ids
)

loss = model(inputs, labels=labels_ids)[0]
print(labels_ids)
print(loss)
```

```
tf.Tensor(
[[    0  2387  2335    16 11962     2     1     1     1     1     1     1
      1     1     1     1]], shape=(1, 16), dtype=int32)
tf.Tensor(
[2.2291888e-05 4.8874615e-05 3.7192607e-05 7.9230859e-04 6.1941862e+00
 1.1058818e+00], shape=(6,), dtype=float32)
tf.Tensor(
[[    0  2387  2335    16 11962     2  -100  -100  -100  -100  -100  -100
   -100  -100  -100  -100]], shape=(1, 16), dtype=int32)
tf.Tensor(
[2.2291888e-05 4.8874615e-05 3.7192607e-05 7.9230859e-04 6.1941862e+00
 1.1058818e+00], shape=(6,), dtype=float32)
```

TFBart gives the same result with both -100 and padding token.
However, ```Bart(pytorch) with -100 token in labels```, ```Bart with padding token in labels``` and ```TFBart(tensorflow with -100 or padding token)``` gives three different results. This is noticed but not treated in this PR.

@patrickvonplaten
@jplu
@patil-suraj


Adding new parameter to `generate`:  `max_time`.

Generation by tokens number is sometimes a bit clunky because we don't
know how many tokens are good enough or even how many tokens are in
the payload (for pipelines users for instance). This leads to hard
to understand behavior.

This PR proposes a new argument `max_time` which is a float of seconds
for the allowed time for `generate` to run on.
Ideally combinations of `max_tokens=None`, `max_time=2` could be used to
generate as many tokens as possible within time budget.

NB: Another possible approach consists of passing a callback to `generate`
  putting the caller in charge of the actual decision of when to stop
  generating tokens. It opens the door to 'which args should we pass'
  to this callback. It's hard to imagine other use-cases for this
  early stopping behavior than time (that are not already covered by
  parameters of generate)

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten @LysandreJik
@jplu

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[WIP/ don't merge] T5 gradient checkpointing


[examples/seq2seq] support label smoothing

# What does this PR do?

Add support for label smoothing by adding `prepare_decoder_input_ids_from_labels` method to all seq2seq models which will let us prepare `decoder_input_ids` outside the model.

For context, we need to pass  `decoder_input_ids`  for label smoothing because we don't pass `labels`  to avoid calculating loss twice, which leads to speeds degradation, see #9713.

@sgugger , @patrickvonplaten  what do we think about adding `prepare_decoder_input_ids_from_labels` to every seq2seq model, there are already `shift_tokens_right/_shift_right` methods, but the name is a bit confusing IMO to use outside the model.


SQUAD Question Answering example:: RuntimeError: Could not infer dtype of NoneType

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.1
- Platform: Linux
- Python version: 3.6.12
- PyTorch version (GPU?): 1.7.1
- Tensorflow version (GPU?): 2.4.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
- @sgugger, @patil-suraj
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library: @sgugger

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples: @patil-suraj

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [X] the official example scripts: (give details below)

        mkdir squad
        wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -O squad/train-v2.0.json
        wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O squad/dev-v2.0.json

        import json
        from pathlib import Path

        def read_squad(path):
            path = Path(path)
            with open(path, 'rb') as f:
                squad_dict = json.load(f)

            contexts = []
            questions = []
            answers = []
            for group in squad_dict['data']:
                for passage in group['paragraphs']:
                    context = passage['context']
                    for qa in passage['qas']:
                        question = qa['question']
                        for answer in qa['answers']:
                            contexts.append(context)
                            questions.append(question)
                            answers.append(answer)

            return contexts, questions, answers

        train_contexts, train_questions, train_answers = read_squad('squad/train-v2.0.json')
        val_contexts, val_questions, val_answers = read_squad('squad/dev-v2.0.json')

        def add_end_idx(answers, contexts):
            for answer, context in zip(answers, contexts):
                gold_text = answer['text']
                start_idx = answer['answer_start']
                end_idx = start_idx + len(gold_text)

                # sometimes squad answers are off by a character or two – fix this
                if context[start_idx:end_idx] == gold_text:
                    answer['answer_end'] = end_idx
                elif context[start_idx-1:end_idx-1] == gold_text:
                    answer['answer_start'] = start_idx - 1
                    answer['answer_end'] = end_idx - 1     # When the gold label is off by one character
                elif context[start_idx-2:end_idx-2] == gold_text:
                    answer['answer_start'] = start_idx - 2
                    answer['answer_end'] = end_idx - 2     # When the gold label is off by two characters

        add_end_idx(train_answers, train_contexts)
        add_end_idx(val_answers, val_contexts)

        from transformers import DistilBertTokenizerFast
        tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')

        train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True)
        val_encodings = tokenizer(val_contexts, val_questions, truncation=True, padding=True)

        def add_token_positions(encodings, answers):
            start_positions = []
            end_positions = []
            for i in range(len(answers)):
                start_positions.append(encodings.char_to_token(i, answers[i]['answer_start']))
                end_positions.append(encodings.char_to_token(i, answers[i]['answer_end']))

                # if start position is None, the answer passage has been truncated
                if start_positions[-1] is None:
                    start_positions[-1] = tokenizer.model_max_length

                # if end position is None, the 'char_to_token' function points to the space before the correct token - > add + 1
                if end_positions[-1] is None:
                    end_positions[-1] = encodings.char_to_token(i, answers[i]['answer_end'] + 1)
            encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

        add_token_positions(train_encodings, train_answers)
        add_token_positions(val_encodings, val_answers)

        import torch

        class SquadDataset(torch.utils.data.Dataset):
            def __init__(self, encodings):
                self.encodings = encodings

            def __getitem__(self, idx):
                return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}

            def __len__(self):
                return len(self.encodings.input_ids)

        train_dataset = SquadDataset(train_encodings)
        val_dataset = SquadDataset(val_encodings)

        from transformers import DistilBertForQuestionAnswering, Trainer, TrainingArguments

        training_args = TrainingArguments(
            output_dir='./results',          # output directory
            num_train_epochs=3,              # total number of training epochs
            per_device_train_batch_size=16,  # batch size per device during training
            per_device_eval_batch_size=64,   # batch size for evaluation
            warmup_steps=500,                # number of warmup steps for learning rate scheduler
            weight_decay=0.01,               # strength of weight decay
            logging_dir='./logs',            # directory for storing logs
            logging_steps=10,
        )

        model = DistilBertForQuestionAnswering.from_pretrained("distilbert-base-uncased")

        trainer = Trainer(
            model=model,                         # the instantiated 🤗 Transformers model to be trained
            args=training_args,                  # training arguments, defined above
            train_dataset=train_dataset,         # training dataset
            eval_dataset=val_dataset             # evaluation dataset
        )

        trainer.train()

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name) SQUaD
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Ran the example from [squad question answering](https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0)
2. getting the following

        ---------------------------------------------------------------------------
        RuntimeError                              Traceback (most recent call last)
        <ipython-input-9-fe1badbb2679> in <module>
             21 )
             22
        ---> 23 trainer.train()

        /media/data2/anaconda/envs/hr/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path, trial)
            871             self.control = self.callback_handler.on_epoch_begin(self.args, self.state, self.control)
            872
        --> 873             for step, inputs in enumerate(epoch_iterator):
            874
            875                 # Skip past any already trained steps if resuming training

        /media/data2/anaconda/envs/hr/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __next__(self)
            433         if self._sampler_iter is None:
            434             self._reset()
        --> 435         data = self._next_data()
            436         self._num_yielded += 1
            437         if self._dataset_kind == _DatasetKind.Iterable and \

        /media/data2/anaconda/envs/hr/lib/python3.6/site-packages/torch/utils/data/dataloader.py in _next_data(self)
            473     def _next_data(self):
            474         index = self._next_index()  # may raise StopIteration
        --> 475         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
            476         if self._pin_memory:
            477             data = _utils.pin_memory.pin_memory(data)

        /media/data2/anaconda/envs/hr/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
             42     def fetch(self, possibly_batched_index):
             43         if self.auto_collation:
        ---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
             45         else:
             46             data = self.dataset[possibly_batched_index]

        /media/data2/anaconda/envs/hr/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
             42     def fetch(self, possibly_batched_index):
             43         if self.auto_collation:
        ---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
             45         else:
             46             data = self.dataset[possibly_batched_index]

        <ipython-input-8-a9d5c9a06902> in __getitem__(self, idx)
              6
              7     def __getitem__(self, idx):
        ----> 8         return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
              9
             10     def __len__(self):

        <ipython-input-8-a9d5c9a06902> in <dictcomp>(.0)
              6
              7     def __getitem__(self, idx):
        ----> 8         return {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
              9
             10     def __len__(self):

        RuntimeError: Could not infer dtype of NoneType
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior: should have run without the error.

<!-- A clear and concise description of what you would expect to happen. -->


Fix model templates

Fixes the style issue with model templates


Multi-TPU training uses just 1 out of 8 cores.

## Environment info
- `transformers` version: 4.2.2
- Platform: n1-standard-64 Google Cloud
- Python version: 3.7
- PyTorch version (GPU?): 1.7 XLA
- Tensorflow version (GPU?):
- Using GPU in script?: NO, using TPU
- Using distributed or parallel set-up in script?: YES; I try to run it in parallel using all 8 cores with xla_spawn.py setting num_cores to 8 in a V3-8.

### Who can help
@patrickvonplaten, @LysandreJik @sgugger

## Information

Model I am using (Bert, XLNet ...): ALBERT base

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The problem occurs when I try to train with run_mlm_wwm.py through xla_spawn.py. I've checked that when xla_spawn calls run_mlm_ww.py, xm.xrt_world_size() is 8, which should be. However, when the Trainer starts to train, its batch size is only 64, but should be 64 * num_cores = 512. I've printed out the parameters sent by xla_spawn and those received by run_mlm_wwm.py, and they coincide, thus I don't understand why in line 690 of trainer: ```{python}total_train_batch_size = self.args.train_batch_size * xm.xrt_world_size()``` the total_train_batch_size is not converted to 512...

This is the full call:
```{bash}
XRT_TPU_CONFIG="tpu_worker;0;10.44.99.146:8470" python -u transformers/examples/xla_spawn.py --num_cores 8 \
    transformers/examples/language-modeling/run_mlm_wwm.py \
    --model_type albert \
    --config_name ./config/albert-base-v2.json \
    --tokenizer_name ./tokenizer_2912 \
    --train_file ./train_texts_1_percent.txt \
    --validation_file ./validation_data/good_texts.csv \
    --output_dir ./models/model_1_percent \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --evaluation_strategy steps \
    --per_device_train_batch_size 64 \
    --per_device_eval_batch_size 128 \
    --gradient_accumulation_steps 8 \
    --learning_rate 0.00176 \
    --save_steps 1000 \
    --logging_steps 1000 \
    --overwrite_cache \
    --max_seq_length 512 \
    --eval_accumulation_steps 10 \
    --load_best_model_at_end \
    --run_name model_1_percent \
    --save_total_limit 20 --tpu_metrics_debug
```

The model starts to train, but it doesn't take into account that it has 8 tpu cores:

```
[INFO|trainer.py:662] 2021-01-27 12:22:50,282 >> ***** Running training *****
[INFO|trainer.py:663] 2021-01-27 12:22:50,282 >>   Num examples = 5835032
[INFO|trainer.py:664] 2021-01-27 12:22:50,282 >>   Num Epochs = 3
[INFO|trainer.py:665] 2021-01-27 12:22:50,282 >>   Instantaneous batch size per device = 64
[INFO|trainer.py:666] 2021-01-27 12:22:50,282 >>   Total train batch size (w. parallel, distributed & accumulation) = 512
[INFO|trainer.py:667] 2021-01-27 12:22:50,282 >>   Gradient Accumulation steps = 8
[INFO|trainer.py:668] 2021-01-27 12:22:50,282 >>   Total optimization steps = 4272
  0%|                                                              | 3/4272 [04:18<113:20:52, 95.58s/it]
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Whole Word Masked Language Modelling
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Instantiate a Google Cloud V3-8 TPU and a n1-standard-64 Google Cloud instance.
2. Use any toy text dataset and any tokenizer and model name from the ones available in Transformers (these won't change the problem, so it's not necessary to have your own pretrained tokenizer or own dataset).
3. Try to execute the command I posted above but setting XRT_TPU_CONFIG to the IP address of your TPU.

## Expected behavior

It's expected that xla_spawn.py runs the python file passed to it in a multiprocessing fashion, distributing the batches and model over the TPU cores; however, at some point the xrt_world_size() turns to 1 and it doesn't see all the devices available anymore, but only one.


Fix TF template

# What does this PR do?

This PR fixes the template and a cast issue.


Run GA on forks

Tests

Related to tests


logging_epochs argument for TrainingArguments

Feature request

Request for a new feature

# 🚀 Feature request

There is no `logging_epochs` argument in `TrainingArguments`. When someone wants to train with `EvaluationStrategy.EPOCH`, he/she wants to see the logs after each epoch. Currently it is not possible.

## Motivation

Better logging for training

## Your contribution

If I have time, I would like to add it. However, I am not available in a coming couple of weeks.


Fixing flaky conversational test + flag it as a pipeline test.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[docs] use `versionadded`, `versionchanged` and `deprecated` directive

# 🚀 Feature request

## Documentation
Use `.. versionadded::`, `.. versionchanged::` and `.. deprecated::` directive, so that user knows which features are added / changed / deprecated in which version and they can navigate the docs easily without changing the docs from version to version.

Ref: https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html#directive-versionadded

## Motivation
To be able to know (without going to github) which features are introduced / changed / deprecated / improved in which version just from the docs.

Since `transfomers` is widely used in production, this slight change to the docs can help users see bird eye view for the features of the library.

Let me know what do you think


support Mixed Precision and avoid 'dtype=float32' in implementation

# 🚀 Feature request
When using Longformer model, I found the dtype of many tensors are assigned directly by `dtype=tf.dtypes.float32` or equations, which makes mixed precision raining impossible. And I found that other models also have this problem.

Of course It is not a bug to not support mixed precision training. But because Transformer models are usually very large, it will be appreciated to support mix precision training when implementing new models.

So I suggest if possible please use dtype inference instead of direct dtype assignment.


Improved TF inputs

# What does this PR do?

This PR aims to improve the input processing of the TF models. Currently the inputs of each model are processed at least twice (once in the model, once in the main layer) and at most four times for the Seq2Seq models (once in the model, once in the main layer, once in the encoder layer and once in the decoder layer). This is a bit overkill and slows down the performance of a forward pass.

To fix this issue, we introduce a flag in order to know if the incoming inputs are already processed or not, if yes we keep them as they are otherwise we run the input processing.


Mixed Precision support and avoid ‘分咯啊太’

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


from transformers.trainer_utils import get_last_checkpoint, is_main_process
ImportError: cannot import name 'get_last_checkpoint'


[Setup.py] update jaxlib

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes failing Ciricle CI because of version mismatch: https://app.circleci.com/pipelines/github/huggingface/transformers/19040/workflows/75599a81-f58c-40c6-8feb-f824d57a1d65/jobs/157385

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[MT5 Import init] Fix typo

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes a typo

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Update run_xnli.py to use Datasets library

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #9754

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[LedFastTokenizer] Correct missing None statement

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

LEDTokenizerFast was not set to None when not being imported, which broke this script, e.g.:
https://colab.research.google.com/github/pytorch/pytorch.github.io/blob/master/assets/hub/huggingface_pytorch-transformers.ipynb.
This PR should fix it.

Ci-Failure is unrelated.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

cc @sgugger @LysandreJik


I am trying to Fine tune on BartForConditionalGeneration but I end up getting all <pad_tokens>. Can you please help resolve it?


Delete a needless duplicate condition

# What does this PR do?

Delete a needless duplicate condition in the class `PrefixConstrainedLogitsProcessor` (`src/transformers/generation_logits_process.py`).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Add tpu_zone and gcp_project in training_args_tf.py

# What does this PR do?
Add ```tpu_zone``` and ```gcp_project``` in ```training_args_tf.py```.
For using TPUs created in a zone different from the VM zone, ```tpu_zone``` must be allocated.

See official Bert repo,
https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/run_pretraining.py#L426

- trainer: @sgugger


[wip] [doc] Performance and Scalability notes

Let's start another doc. I think it works the best to work on these as an issue and not a PR since anybody can read these easily, rather than reading a markdown.

As in the other similar [work-in-progress-doc](https://github.com/huggingface/transformers/issues/9766), let me write the bulk of it out and then you can ask questions / make requests and clarifications.

---------------------------------------------

# Performance and Scalability: How To Fit a Bigger Model and Train It Faster

Quick notes:

This section gives brief ideas on how to make training faster and support bigger models. Later sections will expand, demonstrate and elucidate each of these.

### Faster Training

HW:

- fast connectivity between GPUs
    * same node: NVLink
    * multiple nodes:  ???

SW:

- Data Parallel / Distributed Data Parallel
- fp16 (autocast caching)

### Bigger Models

HW:

- bigger GPUs

SW:

- ZeRO-Offload
- ZeRO-DP
- Pipeline Parallelism
- fp16 (smaller data)

## Hardware

### Multi-GPU Connectivity

If you use multiple GPUs the way cards are inter-connected can have a huge impact on the total training time.

If the GPUs are on the same physical node, you can run:

```
nvidia-smi topo -m
```

and it will tell you how the GPUs are inter-connected.

On a machine with dual-GPU and which are connected with NVLink, you will most likely see something like:

```
        GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      NV2     0-23            N/A
GPU1    NV2      X      0-23            N/A
```

on a different machine w/o NVLink we may see:
```
        GPU0    GPU1    CPU Affinity    NUMA Affinity
GPU0     X      PHB     0-11            N/A
GPU1    PHB      X      0-11            N/A
```

The report includes this Legend:

```
  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
```

So the first report `NV2` tells us the GPUs are interconnected with 2 NVLinks, and the second report `PHB` we have a typical consumer-level PCIe+Bridge setup.

Check what type of connectivity you have on your setup. Some of these will make the communication between cards faster (e.g. NVLink), others slower (e.g. PHB).

Depending on the type of scalability solution used, the connectivity speed could have a major or a minor impact. If the GPUs need to sync rarely, as in DDP, the impact of a slower connection will be less significant. If the GPUs need to send messages to each other often, as in ZeRO-DP, then faster connectivity becomes super important to achieve faster training.

### NVlink

[NVLink](https://en.wikipedia.org/wiki/NVLink) is a wire-based serial multi-lane near-range communications link developed by Nvidia.

Each new generation provides a faster bandwidth, e.g. here is a quote from [Nvidia Ampere GA102 GPU Architecture](https://www.nvidia.com/content/dam/en-zz/Solutions/geforce/ampere/pdf/NVIDIA-ampere-GA102-GPU-Architecture-Whitepaper-V1.pdf):

> Third-Generation NVLink®
> GA102 GPUs utilize NVIDIA’s third-generation NVLink interface, which includes four x4 links,
> with each link providing 14.0625 GB/sec bandwidth in each direction between two GPUs. Four
> links provide 56.25 GB/sec bandwidth in each direction, and 112.5 GB/sec total bandwidth
> between two GPUs. Two RTX 3090 GPUs can be connected together for SLI using NVLink.
> (Note that 3-Way and 4-Way SLI configurations are not supported.)

So the higher `X` you get in the report of `NVX` in the output of `nvidia-smi topo -m` the better. The generation will depend on your GPU architecture.

Let's compare the execution of a gpt2 language model training over a small sample of wikitext.

The results are:

|type| time secs |
|----|-----|
| w/ NVlink| 101 |
| w/o NVlink | 131 |

You can see that NVLink completes the training ~23% faster.

In the second benchmark we use `NCCL_P2P_DISABLE=1` to tell the GPUs not to use NVLink.

Here is the full benchmark code and outputs:

```
# DDP w/ NVLink

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node 2 \
examples/language-modeling/run_clm.py --model_name_or_path gpt2 --dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 --do_train --output_dir /tmp/test-clm
--per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}

# DDP w/o NVLink

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 NCCL_P2P_DISABLE=1 python -m torch.distributed.launch \
--nproc_per_node 2 examples/language-modeling/run_clm.py --model_name_or_path gpt2 --dataset_name wikitext \
--dataset_config_name wikitext-2-raw-v1 --do_train --output_dir /tmp/test-clm \
--per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}
```

Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`)
Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`

## Software

### Anatomy of Model's Memory

The components on GPU memory are the following:
- the model weights
- the forward activations saved for gradient computation
- the gradients
- the optimizer state

### `forward` vs `backward` Execution Speed

For convolutions and linear layers there are 2x flops in the backward compared to the forward, which generally translates into ~2x slower (sometimes more, because sizes in the backward tend to be more awkward). Activations are usually bandwidth-limited, and it’s typical for an activation to have to read more data in the backward than in the forward (e.g. activation forward reads once, writes once, activation backward reads twice, gradOutput and output of the forward, and writes once, gradInput).

### fp16

AMP = Automatic Mixed Precision

If we look at what's happening with FP16 training (mixed precision) we have:
- the model in full precision so no memory saved there
- the forward activations saved for gradient computation are in mixed precision
- the gradients are computed in mixed precision *but* converted to full precision for the update, so no saving there
- the optimizer state is in full precision as all the updates are done in full precision

So the saving only happen for the forward activations saved for the backward computation, and there is a slight overhead because the gradients are properly stored both in half and full precision. (This is probably over-simplified but I think it's enough to explain what follows.)

Now let's look at a simple text-classification fine-tuning on 2 GPUs (I'm giving the command for reference):
```
export BS=16
python -m torch.distributed.launch \
    --nproc_per_node 2 examples/text-classification/run_glue.py \
    --model_name_or_path bert-base-cased \
    --task_name mrpc \
    --do_train \
    --do_eval \
    --max_seq_length 128 \
    --per_device_train_batch_size $BS \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir /tmp/mrpc \
    --overwrite_output_dir \
    --fp16
```
Since the only savings we get are in the model activations saved for the backward passed, it's logical that the bigger those activations are, the bigger the saving will be. If we try different batch sizes, I indeed get (this is with nvidia-smi so not completely reliable as said above but it will be a fair comparison):
| batch size | without --fp16 | with --fp16 | FP16 savings |
|:-:|:-:|:-:|:-:|
| 8 | 4247 | 4163 | 84 |
| 16 | 4971 | 4793 | 178 |
| 32 | 6827 | 6207 | 620 |
| 64 | 10037 | 8061 | 1976 |

So there is only a real memory saving if we train at a high batch size (and it's not half) and at batch sizes lower than 8, you actually get a bigger memory footprint (because of the overhead mentioned above). The gain for FP16 training is that in each of those cases, the training with the flag `--fp16` is twice as fast, which does require every tensor to have every dimension be a multiple of 8 (so if your batch size is not a multiple of 8, you won't get that speed-up, and the script `finetune_trainer.py` does not pad the tensors to a sequence length that is a multiple of 8).

TL;DR: FP16 with apex or AMP will only give you some memory savings with a reasonably high batch size.

Some amazing tutorials to read on mixed precision:
- @sgugger wrote a great explanation of mixed precision [here](https://docs.fast.ai/callback.fp16.html#A-little-bit-of-theory)
- Aleksey Bilogur's [A developer-friendly guide to mixed precision training with PyTorch](https://spell.ml/blog/mixed-precision-training-with-pytorch-Xuk7YBEAACAASJam)

### fp16 caching

pytorch `autocast` which performs AMP include a caching feature, which speed things up by caching fp16-converted values. Here is the full description from this [comment](https://discuss.pytorch.org/t/autocast-and-torch-no-grad-unexpected-behaviour/93475/3):

Autocast maintains a cache of the FP16 casts of model params (leaves). This helps streamline parameter reuse: if the same FP32 param is used in several different FP16list ops, like several matmuls, instead of re-casting the param to FP16 on entering each matmul, the cast will occur on the first matmul, the casted FP16 copy will be cached, and for all later matmuls the FP16 copy will be reused. The cache is maintained only within a particular outermost autocast context. When you exit the autocast context the cache is dropped. For recommended usage, in which autocast wraps the forward pass, and then you exit the context before calling backward(), this means the cache only lasts the duration of the forward pass each iteration, and will be rebuilt next iteration. (The cache of FP16-casted copies MUST be rebuilt each iteration. The FP32 params get updated by the optimizer, so the FP16 copies must be recreated, otherwise the FP16 values will be stale.)

### DP vs DDP

`DistributedDataParallel` (DDP) is typically faster than `DataParallel` (DP), but it is not always the case:
* while DP is python threads-based, DDP is multiprocess-based - and as such it has no python threads limitations, such as GIL
* on the other hand a slow inter-connectivity between the GPU cards could lead to an actual slower outcome with DDP

Here are the main differences in the inter-GPU communication overhead between the two modes:

[DDP](https://pytorch.org/docs/master/notes/ddp.html):

- At the start time the main process replicates the model once from gpu 0 to the rest of gpus
- Then for each batch:
   1. each gpu consumes each own mini-batch of data directly
   2. during `backward`, once the local gradients are ready, they are then averaged across all processes

[DP](https://pytorch.org/docs/master/generated/torch.nn.DataParallel.html):

For each batch:
   1. gpu 0 reads the batch of data and then sends a mini-batch to each gpu
   2. replicates the up-to-date model from gpu 0 to each gpu
   3. runs `forward` and sends output from each gpu to gpu 0, computes loss
   4. scatters loss from gpu 0 to all gpus, runs `backward`
   5. sends gradients from each gpu to gpu 0 and averages those

The only communication DDP performs per batch is sending gradients, whereas DP does 5 different data exchanges per batch.

DP copies data within the process via python threads, whereas DDP copies data via [torch.distributed](https://pytorch.org/docs/master/distributed.html).

Under DP gpu 0 performs a lot more work than the rest of the gpus, thus resulting in under-utilization of gpus.

You can use DDP across multiple machines, but this is not the case with DP.

There are other differences between DP and DDP but they aren't relevant to this discussion.

If you want to go really deep into understanding these 2 modes, this [article](https://www.telesens.co/2019/04/04/distributed-data-parallel-training-using-pytorch-on-aws/) is highly recommended, as it has great diagrams, includes multiple benchmarks and profiler outputs on various hardware, explains all the nuances that you may need to know.

Let's look at an actual benchmark:

|type| time secs |
|----|-----|
| 2:DP w/ NVlink| 110 |
| 2:DDP w/ NVlink| 101 |
| 2:DDP w/o NVlink | 131 |

Analysis:

Here DP is ~10% slower than DDP w/ NVlink, but ~15% faster than DDP w/o NVlink

The real difference will depend on how much data each GPU needs to sync with the others - the more there is to sync, the more a slow link will slow down the total runtime.

Here is the full benchmark code and outputs:

`NCCL_P2P_DISABLE=1` was used to disable the NVLink feature on the corresponding benchmark.

```

# DP

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python examples/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 110.5948, 'train_samples_per_second': 1.808, 'epoch': 0.69}

# DDP w/ NVlink

rm -r /tmp/test-clm; CUDA_VISIBLE_DEVICES=0,1 \
python -m torch.distributed.launch --nproc_per_node 2 examples/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 101.9003, 'train_samples_per_second': 1.963, 'epoch': 0.69}

# DDP w/o NVlink

rm -r /tmp/test-clm; NCCL_P2P_DISABLE=1 CUDA_VISIBLE_DEVICES=0,1 \
python -m torch.distributed.launch --nproc_per_node 2 examples/language-modeling/run_clm.py \
--model_name_or_path gpt2 --dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \
--do_train --output_dir /tmp/test-clm --per_device_train_batch_size 4 --max_steps 200

{'train_runtime': 131.4367, 'train_samples_per_second': 1.522, 'epoch': 0.69}
```

Hardware: 2x TITAN RTX 24GB each + NVlink with 2 NVLinks (`NV2` in `nvidia-smi topo -m`)
Software: `pytorch-1.8-to-be` + `cuda-11.0` / `transformers==4.3.0.dev0`

### Batch Sizes

The best performance is achieved when the tensor's batch size dimension is a multiple of 8. It's the final batch size of the tensor that gets passed to the GPU to calculate something that's important.

Examples:
- if you use a DP or DDP on 2 GPUs you want to have a total batch size of at least 16 (2x8), or a higher multiple. If your total batch size is 8, then each GPU will get a mini-batch of 4.
- if you use a Pipeline you want to make sure that after chunking you end up with micro-batches that are multiples of 8. For example if `chunks=3` is used, you want the batch size to be 24 (or a higher multiple of 8). Because if you use a batch size of 16, you will end up with 3 micro-batches of size 6,5,5.

There is no harm in using smaller batch sizes and at times one can hardly squeeze a batch size of 1 before getting OOM, it just won't be as fast as it can be.

### DataLoader

One of the important requirements to reach great training speed is the ability to feed the GPU at the maximum speed it can handle. By default everything happens in the main process and it might not be able to read the data from disk fast enough, and thus create a bottleneck, leading to GPU under-utilization.

- `DataLoader(`pin_memory=True`, ...)` which ensures that the data gets preloaded into the pinned memory on CPU and typically leads to much faster transfers from CPU to GPU memory.
-  `DataLoader(`num_workers=4`, ...)` - spawn several workers to pre-load data faster - during training watch the GPU utilization stats and if it's far from 100% experiment with raising the number of workers. Of course, the problem could be elsewhere so a very big number of workers won't necessarily lead to a better performance.

### Faster optimizer

pytorch-nightly introduced `torch.optim._multi_tensor` which should significantly speed up the optimizers for situations with lots of small feature tensors. It should eventually become the default, but if you want to experiment with it sooner and don't mind using the bleed-edge, see: https://github.com/huggingface/transformers/issues/9965
-----------------

## Credits

It'd be difficult to track and record every contribution, so in order to keep things practical I will try to keep track of major contributors. And I have a huge gratitude to everybody who has ever asked or answered a question on forums/issues/slacks/SO/etc., parts or summaries of which were integrated into this article. Thank you!

The major contributors:
- @sgugger: fp16 section from [here](https://github.com/huggingface/transformers/issues/9742#issuecomment-765488087)
- @moyix: ideas on NVLink testing https://github.com/huggingface/transformers/issues/9371
- @ngimel: multiple insights on pytorch slack/issues
- @mcarilli: pytorch autocast


Allow --arg Value for booleans in HfArgumentParser

# What does this PR do?

The primary reason I dived into this PR is because when launching a training with sagemaker, bool arguments need to be passed along with something (e.g. we can't do `--do_train`, we have to do `--do_train True` because the arguments are passed as a dict). This is refused by the current argparser.

This PR changes a little bit the way `HfArgumentParser` handles bool fields in dataclasses. Up until now:
- a bool arg `foo` with `True` as a default gives a flag `--no_foo` that stores `False` in foo
- a bool arg `bar` with no default value or `False` as a default value gives a flag `bar` that stores `True` in bar
- an optional bool arg `opt` gives the same as a bool if its default is True or False. If the default is None, it gives a flag `--opt` that requires an argument that accepts anything and stores the value as a string (which is obviously a bug)

After this PR, the following happens:
- a bool arg `foo` with `True` as a default gives a flag `--no_foo` that stores `False` in foo, it also gives a flag `--foo` that can be used as is (will store `True` in foo), or by using any truthy/falsy value (`--foo yes`, `--foo True`, `--foo no`...) that will store the result as a proper bool.
- a bool arg  `bar` with no default value or `False` gives a flag `bar` that can be used as is (will store `True` in bar), or by using any truthy/falsy value (`--bar yes`, `--bar True`, `--bar no`...) that will store the result as a proper bool.
- an optional bool arg `opt` gives the same as a bool if its default is True or False. If the default is None, it gives a flag `--opt` that requires an argument that accepts a truthy value and stores the value as a proper bool.

In all cases above, when a truthy value is expected but something else is passed (that is not `true`, `false`, `yes`, `no`, `1`, `0`, `t`, `f`, `y`, `n`), an error is raised.

So no breaking changes at all and all bool values can be used with an argument so that sagemaker is happy. Tests are updated and improved to check the behaviors summarized above are all correct.


Fix auto-resume training from checkpoint

# What does this PR do?

This fixes a few minor issues with training auto-resume, as discussed [here](https://github.com/huggingface/transformers/pull/9776#issuecomment-767841895)

1. `checkpoints = [path for path in content if _re_checkpoint.search(path) is not None and os.path.isdir(path)]` was returning empty. I changed `os.path.isdir(path)` to `os.path.isdir(os.path.join(folder, path))` and now it returns a list of the checkpoint folders as expected.
2. Similarly, the `get_last_checkpoint` function was returning the basename of the checkpoint folder, not the full path, which seems to be expected based on the updates to the example scripts. I changed the last line of the function to `return os.path.join(folder, max(checkpoints, key=lambda x: int(_re_checkpoint.search(x).groups()[0])))`
3. After I made those update, it was resuming from the oldest checkpoint, not the newest. I noticed the checkpoint regex was only capturing the final digit in the directory name. I changed it to `_re_checkpoint = re.compile(r"^" + PREFIX_CHECKPOINT_DIR + r"\-(\d+)$")` with the `+` inside the capture group, and now `get_last_checkpoint` is giving me the newest checkpoint as expected.

## Who can review?

- trainer: @sgugger


[trainer] renaming cl args/ trainer attributes to be clear per-gpu vs total

As we started discussing here https://github.com/huggingface/transformers/issues/9801#issuecomment-767825869 perhaps we could have a design session where we look at all of the trainer cl args (and their class attribute counterparts) and see which of them contain ambiguity wrt per-gpu vs total (and perhaps other important renames where we find things are confusing).

The intention is to make the API more intuitive and minimize the number of time we introduce breaking changes, but to attempt to do that in one go as much as possible.

One such item we started to discuss is `--max_steps`, then @sgugger mentioned `--num_train_epochs` and there are probably others.

I also proposed to potentially entertain creating a back-compat module to minimize the breaking changes pain where it's possible - renames fall perfectly into this category. I wrote:

> In some previous projects for such things we also had a back-compat mode, which ones enabled supported a whole bunch of old ways until the user was ready to make the shift to the new code. Surely a rename of a cl arg could be easily supported by such feature. So here, instead of a deprecation cycle per item the approach is to keep anything old around but only if it's loaded from a helper module. So that the main code remains clean of deprecated things. This was in a different programming environment where it was developer, so I will have to think how to do the same here.

@LysandreJik, @patrickvonplaten, @sgugger


Add a flag for find_unused_parameters

# What does this PR do?

This PR adds a flag to control whether `find_unused_parameters` is set to `True` or not in DDP training, while keeping the current behavior as default to avoid any breaking change.
Fixes #9802


Add head_mask and decoder_head_mask to FSMT

This PR implements `head_mask` and `decoder_head_mask` for FSMT and it is the follow-up to the open issue #9814.

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9569).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewer: @stas00


When resuming training from checkpoint, Trainer loads model

# What does this PR do?

Trainer was not reloading model when resuming training from a checkpoint, which was confusing for users (see #9099) and also was preventing the recent auto-reload from checkpoint to fully work.

This isn't a breaking change (if users were passing a model with the checkpoint already loaded, it is just loaded twice).


[docs] expand install instructions

This PR: expands the "install from source" section in the instruction file to:

  - clarify that the user is not installing the release version but bleed edge
  - expand how to update it
  - gives the shortcut at doing it all in one command and not needing to keep the checkout folder around

@LysandreJik, @sgugger


Setup logging with a stdout handler

# What does this PR do?

Explicitly add stdout as a handler for the logging configuration in the example scripts, otherwise no logs are reported when training on sagemaker. Also consistently sets the level of the logging outside of the config method, as otherwise it does not work (probably a bug in the logging module).


Fix a bug in run_glue.py (#9812)

# What does this PR do?

It seems the `if` statement for `label_to_id` seems to be strange.
There should be `not` before `is_regression`.

Fixes #9812

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj


Missing head_mask and decoder_head_mask arguments in encoder-decoder models

# 🚀 Feature request

Following the PRs #9569, #9634 and #9639, there are other encoder-decoder models, which either do not support `head_mask` and `decoder_head_mask` input arguments at all or can be only provided with a single `head_mask` argument used for head masking both in encoder and decoder. It would be, therefore, nice to make this feature uniform over all the decoder-models.

**Models:**

| Model | Pytorch | TensorFlow | PR | Copy dependency |
| ------ | :------: | :---------: | :--: | :-----: |
| BERTGeneration | ☑  | ✖  | - | - |
| EncoderDecoderModel | ☑  | ✖  | - | - |
| FSMT | ✅ | ✖  | #9819 | - |
| LED | ✅ | ☑ | PT - #9856 ; TF - #9988 | - |
| ProphetNet | ☑  | ✖  | #9964 | - |
| Longformer | ✅  | ☑  | PT - #9856; TF - #9988  | LED |

## Your contribution

I'm happy to add this feature in the following days, both for PyTorch and TensorFlow models. (Likely in shorter PRs in order not to create large, overwhelming PRs)

Reviewers: @patrickvonplaten, @jplu, @sgugger, @LysandreJik, @stas00 .


ADD BORT

Hi,

this is a "clean" follow-up PR to the first attempt of adding Bort to Transformers (see #9112).

As Bort is based on the BERT architecture, there's no need to define own model classes, such as `BortModel`. This is done in the main Bort configuration via:

```json
"model_type": "bert"
```

Bort uses the same vocab as RoBERTa, so the tokenizer instance is also configured in the model configuration:

```json
"tokenizer_class": "RobertaTokenizer"
```

Basic integration tests and a (hopefully verbose) conversion script are also included in this PR.


`label_to_id` in `run_glue.py` seems to have a wrong `if` statement

## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.4.0-179-generic-x86_64-with-glibc2.10
- Python version: 3.8.0
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): Bert, xlm-roberta-large

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

It seems the `if` statement for `label_to_id` seems to be strange.

https://github.com/huggingface/transformers/blob/eba418ac5df71d08927efb7e3b738833998162ff/examples/text-classification/run_glue.py#L316-L333

About `and is_regression` in L320, should not it be `and not is_regression`?

I inserted `logging.info` to check the True/False as below:

```python
    label_to_id = None
    logger.info("--- label_to_id if statement check ---")
    logger.info(f"{model.config.label2id != PretrainedConfig(num_labels=num_labels).label2id}")
    logger.info(f"{data_args.task_name is not None}")
    logger.info(f"{is_regression}")

    if (
        model.config.label2id != PretrainedConfig(num_labels=num_labels).label2id
        and data_args.task_name is not None
        and is_regression
    ):
        logger.info("loading model.config.label2id")
        # Some have all caps in their config, some don't.
        label_name_to_id = {k.lower(): v for k, v in model.config.label2id.items()}
```

Then I got:

```python
01/27/2021 03:23:24 - INFO - __main__ -   --- label_to_id if statement check ---
01/27/2021 03:23:24 - INFO - __main__ -   False
01/27/2021 03:23:24 - INFO - __main__ -   True
01/27/2021 03:23:24 - INFO - __main__ -   False
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00,  6.02ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 20.86ba/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 12.80ba/s]
01/27/2021 03:23:25 - INFO - __main__ -   Sample 2619 of the training set: {'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'idx': 2916, 'input_ids': [0, 581, 172337, 5180, 3542, 39958, 1257, 678, 117303, 1010, 22230, 1810, 150, 592, 2363, 7225, 26548, 2022, 112478, 6, 4, 16454, 3912, 37967, 111, 60525, 1810, 150, 592, 747, 125682, 7, 26548, 4049, 6, 5, 2, 2, 84607, 26420, 5180, 3542, 39958, 1257, 678, 117303, 1010, 22230, 1810, 150, 592, 2363, 7225, 26548, 2022, 112478, 6, 4, 16454, 10, 3912, 9, 22469, 94309, 1363, 31330, 47, 70, 29685, 6, 5, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'label': 1, 'sentence1': 'The proceedings were taken up with prosecutors outlining their case against Amrozi , reading 33 pages of documents outlining allegations against him .', 'sentence2': 'Proceedings were taken up with prosecutors outlining their case against Amrozi , reading a 33-page accusation letter to the court .'}.
```

When `is_regression` is False (when the task is `classification`), `model.config.label2id` is never used for `label_to_id`.
If I'm not mistaken, would not this behave differently than what is intended?

I am sorry that I could not find an appropriate task/model combination to show when all other conditions would be true.

Thank you in advance.


adapt mbart and generate for Mbart50

# What does this PR do?
This PR adapts `MBartForConditionalGeneration` and `generate` for mbart-50 models.

There are two main differences between mbart-50 and existing mbart-cc25 models
1. for mbart-50 both source and target language text begin with the `<language token>`,  in mbart-cc25 `<language_token>` is used as suffix token.
2. Also the `decoder_input_ids` begin with `[eos] [tgt_lang_token] ...`, so for generation we need to use `eos` as the `decoder_start_token_id` and force the `tgt_lang_token` as the first generated token.

This PR
1. adds `MBart50Tokenizer` which encodes the text as described above, IMO adding a new tokenizer makes sense as it'll make it explicit that mbart-50 encodes the text differently.
2.  introduces two new `generate` arguments  and `LogitsProcessor`
    - `forced_bos_token_id` and `forced_eos_token_id`,  to force a specific start and end token. This is particularly useful for
       many to many and one to many translation models, so we can pass different language tokens as `forced_bos_token_id` to
       `generate`,
    - `ForcedBosTokenLogitsProcessor` and `ForcedEosTokenLogitsProcessor`
3. Remove `adjust_logits_during_generation` method from all models (except from `Marian`) and handle that use case using the newly introduced logits processors.
4. remove the `force_bos_token_to_be_generated` argument from `BartConfig`

For `Marian` we still need to keep the `adjust_logits_during_generation` method to force the model to not generate pad token. Adding the pad token to `bad_words_ids` does not resolve this issue, the score of `pad_token_id` needs to be set to `-inf` before calling `log_softmax`.

Below is an example of mbart-50 model using `forced_bos_token_id`

```python
from transformers import MBartForConditionalGeneration, MBart50Tokenizer

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"
article_ar = "الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-50-large-many-to-many")
tokenizer = MBart50Tokenizer.from_pretrained("facebook/mbart-50-large-many-to-many")

# translate Hindi to French
encoded_hi = tokenizer.prepare_seq2seq_batch(src_texts=article_hi, src_lang="hi_IN", return_tensors="pt")
generated_tokens = model.generate(**encoded_hi, forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire dans la Syrie."

# translate Arabic to English
encoded_ar = tokenizer.prepare_seq2seq_batch(src_texts=article_ar, src_lang="ar_AR", return_tensors="pt")
generated_tokens = model.generate(**encoded_ar, forced_bos_token_id=tokenizer.lang_code_to_id["en_XX"])
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
# => "The Secretary-General of the United Nations says there is no military solution in Syria."
```

TODOs:
- [x] Make sure all generation related slow integration tests pass for affected models
  - [x] BART
  - [x] mBART (one test is failing, but it's failing on master as well, so not related to this PR)
  - [x] Blender
  - [x] FSMT
  - [x] Marian
  - [x] Pegasus
  - [x] Generation integration test
- [x] add tests for `ForcedBosTokenLogitsProcessor` and `ForcedEosTokenLogitsProcessor`
- [x] document mBART-50
- [x] Add model cards ([all mbart-50 models](https://huggingface.co/models?filter=mbart-50))
- [x] add the forced params to `facebook/bart-large-cnn`'s config on the hub
- [ ] notebook explaining  how to use the one-to-many and many-to-many translation models

Fixes #7060


Can I use a smaller base model than allenai/led-base-16384 for LED?

Hello, I'm trying to fine tune my own Longformer Encoder Decoder following this [notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing#scrollTo=jpUr9QeebZ-n). However, I was wondering if there was a way to consider a base model like

`allenai/longformer-base-4096`

instead of

`led-base-16384`?

When I try doing
```python
led = AutoModelForSeq2SeqLM.from_pretrained(
    "allenai/longformer-base-4096",
    config="roberta-base",
    gradient_checkpointing=True,
    use_cache=False,
)
```
but that gives me
```
   led = AutoModelForSeq2SeqLM.from_pretrained(
  File "/usr/local/lib/python3.8/dist-packages/transformers/models/auto/modeling_auto.py", line 1221, in from_pretrained
    raise ValueError(
ValueError: Unrecognized configuration class <class 'transformers.models.longformer.configuration_longformer.LongformerConfig'> for this kind of AutoModel: AutoModelForSeq2SeqLM.
Model type should be one of LEDConfig, BlenderbotSmallConfig, MT5Config, T5Config, PegasusConfig, MarianConfig, MBartConfig, BlenderbotConfig, BartConfig, FSMTConfig, EncoderDecoderConfig, XLMProphetNetConfig, ProphetNetConfig.
```

my `encoder_max_length` is only 2048 since I'm not planning on feeding transcripts of 8k word to summarize but rather transcripts of 2k words. So using the smaller base model would work great. I'm basically trying to replicate the fine tuning for this
https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16
^but I would want to use that model more as a checkpoint to fine tune further on domain specific data. Hence why I'm trying to create:
1.) a model fine tuned on cnn dailymail data(basically replicate the [longformer2roberta](https://huggingface.co/patrickvonplaten/longformer2roberta-cnn_dailymail-fp16) but a version that we can fine tune further)
2.) using part 1 as a checkpoint, fine tune that on domain specific transcript+summary data

@patrickvonplaten or others in the community, I'd greatly appreciate any advice on this


Fix fine-tuning translation scripts

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

In the [seq2seq README.md](https://github.com/huggingface/transformers/tree/master/examples/seq2seq#new-script) there are some errors. This PR fixes typos that cause the following problem:
```
Traceback (most recent call last):
  File "transformers/examples/seq2seq/run_seq2seq.py", line 536, in <module>
    main()
  File "transformers/examples/seq2seq/run_seq2seq.py", line 419, in main
    load_from_cache_file=not data_args.overwrite_cache,
  File ".../lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1240, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File ".../lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1211, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "transformers/examples/seq2seq/run_seq2seq.py", line 388, in preprocess_function
    inputs = [ex[source_lang] for ex in examples["translation"]]
  File "transformers/examples/seq2seq/run_seq2seq.py", line 388, in <listcomp>
    inputs = [ex[source_lang] for ex in examples["translation"]]
KeyError: 'en-XX'
```

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
<!--
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? -->

## Who can review?

@sgugger, @patil-suraj


Adding a test to prevent late failure in the Table question answering pipeline.

# What does this PR do?

- If table is empty then the line that contain `answer[0]` will fail.
- This PR add a check to prevent `answer[0]`.
- Also adds an early check for presence of `table` and `query` to
prevent late failure and give better error message.
- Adds a few tests to make sure these errors are correctly raised.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@LysandreJik
@patrickvonplaten

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Partial local tokenizer load

This PR aims to allow partial loading of a cached tokenizer.

Fixes #9147 which explains the issue in a lot of detail.

Currently, if we download a tokenizer from the hub using the `from_pretrained` method:

```py
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("google/bert_uncased_L-2_H-128_A-2", local_files_only=True)
```

It caches the files to be reused later. Reloading the tokenizer while specifying `local_files_only=True`

```py
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("google/bert_uncased_L-2_H-128_A-2", local_files_only=True)
```

results in a failure as it tries to fetch all of the tokenizer files, even those that are necessary. It currently fails with a hard error.

This PR changes that error to an info log, and prints a single log containing all the files that were not loaded. I put it as an `info` and not as a `warning` or an `error`, because the situation where this is an actual issue is imo very rare; it is a real issue only when the initial `from_pretrained` managed to obtain only some of the necessary files, i.e., when the download was interrupted.

Running the last snippet results in the following warning:

```
Can't load following files from cache: ['added_tokens_file', 'special_tokens_map_file', 'tokenizer_config_file', 'tokenizer_file'] and cannot check if these files are necessary for the tokenizer to operate.
```


Add a test for TF mixed precision

# What does this PR do?

This PR adds a test to check if our TF models are float16 compliant or not. It also helps me to detect what are those that have to be fixed.


Commit the last step on world_process_zero in WandbCallback

# What does this PR do?

Only commit the last step on the first process (`is_world_process_zero == True`), to avoid calling `wandb.log()` without a prior call to `wandb.init()` (the latter being only called on the first process) with DDP.

Fixes (at least partially) #9623

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@borisdayma @sgugger