[Seq2Seq Templates] Add forgotten imports to templates

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

I accidentally forgot to add this import in #9342.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Training of BART slow on TPU - aten ops investigation

Referencing: https://github.com/huggingface/transformers/issues/8339

Versions:
* `transformers==4.0.1`
* `pytorch==1.7.0`
* `pytorch_xla==1.7.0`

### problem

I've been trying to find out why training of `BartForConditionalGeneration` with `Trainer` is so **slow on TPU**. With slow I mean >30 min/batch (8000 samples) on 8 cores of TPUv3. I can very likely exclude slowdowns on behalf of the host machine.

Following the [xla troubleshooting guide](https://github.com/pytorch/xla/blob/master/TROUBLESHOOTING.md), I made sure that the **input tensors have fixed shape**. Furthermore, I created a `metrics` report which reveals that there occur many context switches between the XLA device and CPU due to:
* aten::_local_scalar_dense
* aten::isnan

I tried to localize the culprits via **debugging** on 1 TPU core and printing the metrics report at each breakpoint.

`aten::isnan` is obviously caused by `torch.isnan` in [L362 of BertEncoderLayer](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L362). I don't know any fix but I'd just turn the condition off for my training.

### questions
1. According to this [issue](https://github.com/pytorch/xla/issues/909), the counter for `aten::_local_scalar_dense` increases "every time the Python code switches from tensor context" to Python scalar context". I couldn't really pinpoint, however, which line causes this. I'm aware that the printing of the metrics report at each breakpoint does so but there are others. Any idea?
2. Just to be on the safe side: `pytorch_xla`'s wrapper `ParallelLoader` for the `DataLoader` doesn't have to do anything with TPU-related slowdown or context-switch in e.g. the `Dataset` instance because the loading still takes place on the host machine and the tensors are put into TPU-queues _afterwards_ as far as I read the code, correct?
3. Is training of `BART` (and probably similar models) running fast on TPU for anyone?

AFAIU @patil-suraj knows more.


MBart prepare_seq2seq_batch

- `transformers` version: 4.1.1
- mBART: @patrickvonplaten
## Information
Model I am using (Bert, XLNet ...): mBART

The problem arises when using:
* [x] the official example scripts: (give details below)
`example_english_phrase = "UN Chief Says There Is No Military Solution in Syria"
expected_translation_romanian = "Şeful ONU declară că nu există o soluţie militară în Siria"
batch = tokenizer.prepare_seq2seq_batch(example_english_phrase, src_lang="en_XX", tgt_lang="ro_RO", tgt_texts=expected_translation_romanian, return_tensors="pt")
model(input_ids=batch['input_ids'], labels=batch['labels']) # forward pass`

The encoded target sequence from "prepare_seq2seq_batch" is inconsistent with its description "[tgt_lang_code] X [eos]".
Moreover, "labels" doesn't seem to be the appropriate param for the model input.


[PyTorch Bart] Split Bart into different models

# What does this PR do?

This PR splits all Bart-like models into their own respective classes for PyTorch models only. This is more in line with the general philosophy of the library to have self-contained model files.

As discussed with @jplu, the TF models will be separated in a future PR as there are still some issues and improvements (TF serving) blocking the separation - see https://github.com/huggingface/transformers/issues/9313.

In short, after this PR all those "model-specific" config parameters are removed from all Bart-like configs:

- `extra_pos_embeddings`
- `normalize_embedding`
- `add_final_layer_norm`
- `normalize_before`
- `do_blenderbot_90_layernorm`
- `static_position_embeddings`
- `add_bias_logits`
- `force_bos_token_to_be_generated` (this one has to be kept for Bart though)

and each "bart" model (Pegasus, Bart, MBart, Marian, Blenderbot, BlenderbotSmall) will get its own `modeling_....py` file.

At the moment the models have the following configurations:

| | `extra_pos_embeddings` |  `normalize_before`   | `add_final_layer_norm`  | `do_blenderbot_90_layernorm` | `normalize_embedding` | `static_position_embeddings` | `add_bias_logits` | `force_bos_token_to_be_generated` |
|:---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
| `bart` |  2 | ❌  | ❌   | ❌  |  ✔  | ❌  | ❌ | ✔ |
| `mbart` | 2  | ✔ |  ✔   | ❌  |  ✔  | ❌  |  ❌  | ❌  |
| `marian` | ❌ | ❌   |  ❌  | ❌  | ❌   | ✔  | ❌   | ❌  |
| `pegasus` | ❌  |  ✔  |  ✔  | ❌  |  ❌  | ✔  |  ❌  | ❌  |
| `blenderbot90M (BlenderbotSmall)` | 0  | ❌  |  ❌  | ✔   | ✔   | ❌  |  ❌  |  ❌ |
| `blenderbot3B + rest (Blenderbot)` | 0  |  ✔  | ✔   | ✔  | ❌   | ❌  |  ❌  |  ❌ |

We can see that `add_bias_logits` is actually never used, so I think the best option is to just delete the functionality. Also, one can see that no two models have the exact same usage of the above params, so we'll make 6 different modeling_....py files.

## Resulting Improvements:

- The model files are much more readable and should be much easier to navigate for the user. No difficult config parameters anymore where the user doesn't know what to set anyways, such as  `normalize_before`.

- All irrelevant Bart-like features for other models are removed. Those features are a) never mentioned in the paper, b) don't make any sense since the model wasn't trained with those features, so that the usage of those features leads to non-sense outputs. *E.g.* Marian was never supposed to be a "mask-filling" model, yet it has "mask-filling" functionality, when doing:

```python
marian = MarianMTModel.from_pretrained(...)
marian(input_ids)  # no decoder_input_ids for mask filling like tasks such as in Bart
# => output makes 0 sense
```

The big gain here is that users are better guided on how to use the model and wonder less about whether the model is used correctly & whether there is a bug in the model.

- Docstrings are improved with more Model-specific examples and fewer comparisons to Bart. *E.g.* Pegasus, Marian, and Blenderbot never really mention BART in their paper and have no direct relation to BART IMO => these models should not be compared to BART in the docs -> it's confusing for the user

- Some small improvements, memory is slightly improved for beam search and gradient checkpointing is added.
- All previous tests are copied + some additional tests are added for each model

## Possible drawback

- The drawback as expected is code duplication. This is remedied to some extent by using the # Copied from ... safety features
- Some breaking changes as explained further below
- Models might now diverge easier in the future which could make it harder to have the same API for training. This is however also prevented by some function signature tests that are already implemented.

## Breaking changes

🚨🚨 **Important: We cannot keep 100% backward compatibility here or the PR won't make much sense** 🚨🚨
- Since all models were packed into a single model file a lot of different model design are at the moment possible. E.g.
Pegasus was only ever used with Sinusoidal position embeddings (as mentioned in the paper) but since it's merged into `modeling_bart.py`, one could theoretically use Pegasus with Learned position embeddings. This is not done in any config on the model hub however and will not be possible anymore after the PR. Also, Marian's model design has never normalized the word embeddings, but it could be possible with the current design. But again no config in the model hub does that, so this will also not be possible anymore after the PR. **In short: All model designs that were never foreseen in the original model and that are never used on the model hub at the moment won't be allowed anymore after the PR**.
If we would not make this change, it would mean that we would have to keep all those `normalize_before` configs, which in return would mean that the modeling code of all Bart-like models would be the same again.

- Blenderbot needs to be divided into two models IMO. Blenderbot 90M not only has a very different architecture (see table above), but also uses a different tokenizer. I created a new `BlenderbotSmallModel` class. Thus I need to update one Blenderbot config online, changing it's class. This means that from this PR onward the following is not supported anymore:

```python
from transformers import BlenderbotForConditionalGeneration

model = BlenderbotForConditionalGeneration.from_pretrained("facebook/blenderbot-90M")
# => this is a wrong model. It should be
model = BlenderbotSmallForConditionalGeneration.from_pretrained("facebook/blenderbot-90M")
```

That's a big breaking change, but I don't see another way. If we keep the small blenderbot in the "normal" blenderbot, we have to keep the config params `normalize_before` which I really don't want to do.... I think the best option here is to add a warning (or even an error) by overwriting `from_pretrained(...)` in `BlenderbotForConditionalGeneration` so that
```python
model = BlenderbotForConditionalGeneration.from_pretrained("facebook/blenderbot-90M")
```
will throw an error or give a warning. There are no fine-tuning blenderbot models on the hub and this is the only config. I think it's the right approach to separate the model here

- Barthez has essentially a `mbart` architecture, but has `bart` defined as its `model_type` in the configs. Here I'd also like to change the configs online to make sure the correct model is loaded when using `AutoModelForSeq2SeqLM`. I should also contact the author here.

- Bart allowed to automatically create `decoder_input_ids` by shifting the `input_ids` to the right. Thus, in Bart one can do the following:

```python
bart = BartForConditionalGeneration(...)
bart(input_ids) # not that no decoder_input_ids are passed here
```

This is a very special case and should only be used for Bart-like denoising pre-training or mask-filling. The only models that were trained in this fashion and thus can do mask-filling are Bart and MBart. All other models cannot do mask-filling so that `decoder_input_ids` should never be created from shifting `input_ids`. => this feature is removed therefore from Pegasus, Marian, Blenderbot, and BlenderbotSmall

Those are all breaking changes. Blenderbot is the big one, the other one should be fine. To be sure, I wrote some scripts that verify that no model on the model hub that contains one of the keywords `bart`, `mbart`, `pegasus`, `blenderbot`, `opus-mt`, `barthez` has incorrect/unexpected parameter settings after the PR.

## TODO:

- [x] Create Bart model file & pass all tests
- [x] Create MBart model file & pass all tests
- [x] Greate Pegasus model file & pass all tests
- [x] Create Marian model file & pass all tests
- [x] Create Blenderbot model file & pass all tests
- [x] Create BlenderbotSmall model file & pass all tests
- [x] Clean PR (delete all helper files)
- [x] Clean docs
- [x] Add #Copied From statements
- [x] To a very in-detail review of own PR to make sure no hidden bugs were introduced.
- [x] Correct configs of barthez online to be of type `mbart` instead of `bart`.
- [x] Correct config of https://huggingface.co/facebook/blenderbot-90M online.

## Future TODO:
- [ ] Communitace about this PR on the forum
- [ ] Add Copied From statements to seq2seq bart model templates
- [ ] Add Copied From statements to LED


[Seq2Seq Templates] Add embedding scale to templates

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

The config.embed_scale parameter is too heavily used in Bart-like models to delete it in future leaner bart versions.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


[PyTorch Bart] Split Bart

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Possible bug in `train_batch_size`

## Environment info
- `transformers` version: 4.1.1
- Platform: Linux-4.4.0-62-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
 Trainer: @sgugger

## Information

Model I am using (Bert, XLNet ...):

BERT

The problem arises when using:
* [X ] my own modified scripts: (give details below)

I'm running a model on a toy dataset with only 2 examples and a batch size of 2. In trainer, `num_examples` is 2, but `total_train_batch_size` is 12 even though I do not have the `model_parallel` flag set to `True` (Note I do have 6 GPUs available on the machine). This doesn't seem to impact my code because `train_dataset_is_sized=True`, but it seems strange.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ X] my own task or dataset: (give details below)

toy classification jsonl dataset with 2 examples

## To reproduce

I think that [this line](https://github.com/huggingface/transformers/blob/64103fb6beac8cc865945d3956266fd80b44f18f/src/transformers/training_args.py#L454) has an unnecessary `not`. Should this be `if self.model_parallel` instead of `if not self.model_parallel`? Thanks!


Arrow file is too large when saving vector data

I computed the sentence embedding of each sentence of bookcorpus data using bert base and saved them to disk. I used 20M sentences and the obtained arrow file is about 59GB while the original text file is only about 1.3GB. Are there any ways to reduce the size of the arrow file?


Multiprocessing CUDA issues when importing transformers

## Environment info
- `transformers` version: 4.2.0dev0
- Platform: Linux-4.15.0-128-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Partly

### Who can help
@stas00

## Information

When using multiprocessing, importing the transformers package causes
`RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method`

The problem arises when using:
* `import transformers` is used in the main process.
This is due to the [following line](https://github.com/huggingface/transformers/blob/master/src/transformers/models/fsmt/modeling_fsmt.py#L268) in `modeling_fsmt.py`,  removing `torch.cuda.is_available()` call resolves the issue.

## To reproduce

```
import multiprocessing
import transformers  # NOQA
# You can also call torch.cuda instead of import transformers to get the same error
import torch.nn as nn

class Net(nn.Module):

    def __init__(self):
        super().__init__()
        self.linear = nn.Linear(10, 10)

    def forward(self, x):
        return self.linear(x)

def to_cuda(i):
    net = Net().cuda()
    print(f'Called {i} process')

try:
    cpus = multiprocessing.cpu_count()
except NotImplementedError:
    cpus = 2  # arbitrary default

pool = multiprocessing.Pool(processes=cpus)
pool.map(to_cuda, range(10))
```

## Expected behavior
The code snippet above runs without issues.


[WIP][Research projects] Add folder for Music AI / Music Transformers

A new dir specifically for Music AI/Music Transformers.

Created as suggested by Patrick von Platen.

I am still figuring out PRs so please correct this PR if I've done something wrong.

Thank you for your help/guidance and for the welcome to the Huggingface community :)

Looking forward to contributing what I can :)

 GPT2: @LysandreJik, @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten


"RuntimeError: Input, output and indices must be on the current device" when trying to finetune MBart

### Environment info
- Platform: Linux-4.15.0-123-generic-x86_64-with-glibc2.10
- Tried `transformers` versions 4.1.1 (installed with pip) and 4.2.2 (installed from master branch of the repository)
- Python version: 3.7
- PyTorch version: 1.7
- Tensorflow version: 2.4
- Number of available GPU: 2 (GeForce RTX 2080 Ti, with ~11GB of memory each)

### Information
Model I am using (Bert, XLNet ...): MBart -> facebook/mbart-large-cc25

The problem arises when using: the official example scripts: (details below)

The tasks I am working on is: my own task or dataset: (details below)

I am fine-tuning MBart using my own dataset, using the `examples/seq2seq/finetune.sh` script. When I run it on a single GPU, I get a memory error, as one GPU has not enough memory to load the MBart model. When I try to distribute the model on two GPUs, I get a RuntimeError:
`RuntimeError: Input, output and indices must be on the current device`

### To reproduce
I am running the script in the following way:
`CUDA_VISIBLE_DEVICES=0,1 transformers/examples/seq2seq/finetune.sh --model_name_or_path "facebook/mbart-large-cc25" --output_dir output --data_dir data --overwrite_output_dir --model_parallel --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --freeze_encoder --freeze_embeds --tgt_lang "en"`

I have also tried:
`CUDA_VISIBLE_DEVICES=0,1 transformers/examples/seq2seq/finetune.sh --model_name_or_path "facebook/mbart-large-cc25" --output_dir output --data_dir data --overwrite_output_dir --model_parallel --tgt_lang "en"`

I also tried limiting the length of source and target sentences by trying several values for `--max_target_length` and `--max_source_length'`. In addition, I tried using more GPUs (up to 4).

If I run `wc -l` on my `data` directory, I get:
```
3004 data/test.source
3004 data/test.target
686623 data/train.source
686623 data/train.target
2999 data/val.source
2999 data/val.target
```


Data Loading as a Service

# 🚀 Not a Feature request, what am I here for then?

Well, mainly I'd just like to get some feedback from fellow software engineers. Some of the frustrations I've experienced might not have been big issues at all or there could have been easy ways to get around them which I've failed to notice. Getting roasted can be a good way for us to identify obvious flaws in our thought processes that aren't so obvious from our own often tunneled point of view.

## Motivation

Data loading in PyTorch requires the user to define the Dataset, collator, as well as a sampling strategy.
I found it rather hard to stick to the framework when I have to deal with
- Extremely large datasets that do not fit in system memory
- The previous point + training with multiple processes and nodes

For Language Modeling and Machine Translation,
- We often have to deal with large text/CSV files that are multiple times the size of system memory.
- Also running training on a single GPU is often slow to the point of frustration.
- Then we have to duplicate the extremely large dataset across each machine.
- To format the data we have into a PyTorch Dataset, we simply just do a hack where we wrap an IO stream into a PyTorch Dataset
- Or preprocess the entire dataset beforehand to speed up data loading.
- Sometimes we would keep an index table to "lookup" the position of a data entry in a file which can be slow if not using SSD to perform random access.

Instead of trying to write code that fits into the format expected by PyTorch. I simply threw everything out the window and just made data loading a standalone service instead...

## Your contribution

Well, I've uploaded my work on https://github.com/mingruimingrui/data-loader-as-a-service-demo.

If you have taken your time to read up to this point, I would like to give you my gratitude as it had made me quite happy (^o^)
I would also like to ask for you to leave some comments for me which can be any of the following.
1. Can you relate to the problems I've faced?
2. Which part of the Data Loading as a Service do you like?
3. Which part of the Data Loading as a Service do you not like or have problems agreeing with?
5. Any other comments would also be appreciated.


[Seq2Seq Templates] Correct some TF-serving errors and add gradient checkpointing to PT by default.

# What does this PR do?

This PR improves the Seq2Seq model templates.
Notably:

- a too model-specific test is removed from PyTorch
- gradient checkpointing is added to PyTorch
- some tf-serving incompatible statements are removed


TF Longformer has some graph compilation/execution issue

TF longformer has the following issues to make it 100% graph compilation/execution compliant. I succeed to fix most of the issues but two still remains:

1.  The first issue starts at line [1762](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_tf_longformer.py#L1762). The test to know if the inputs needs to be padded prevent the graph to be compiled because `input_ids`, `position_ids` and `input_embeds` can be `None` at the end of the main branch.

As a solution I propose to export the padding process (from line 1769 to 1786) outside the `if` as if `padding_len == 0` the calls to `tf.pad(...)` and `tf.concat(...)` will have no effect on the different inputs.

2. The second issue is at line [1527](https://github.com/huggingface/transformers/blob/master/src/transformers/models/longformer/modeling_tf_longformer.py#L1527). Here `all_global_attentions` can be either a tuple or `None` in a same execution because `is_global_attn` is not defined globally but during the execution.

I don't know how to solve this one.

As a first test you can run:
```
from transformers import TFLongformerModel
model = TFLongformerModel.from_pretrained("lysandre/tiny-longformer-random", output_attentions=True, output_hidden_states=True)
model.save("path")
```

Ping @patrickvonplaten the Longformer expert :)


block sparse bert

I got following error while running example usage from https://huggingface.co/madlag/bert-base-uncased-squad1.1-block-sparse-0.09-ampere-v1

Do I need specific torch or transformers setup? Thanks in advance!

Some weights of BertModel were not initialized from the model checkpoint at madlag/bert-base-uncased-squad1.1-block-sparse-0.09-ampere-v1 and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "test.py", line 6, in <module>
    tokenizer="madlag/bert-base-uncased-squad1.1-block-sparse-0.09-ampere-v1"
  File "/workspace/transformers/src/transformers/pipelines.py", line 3231, in pipeline
    framework = framework or get_framework(model)
  File "/workspace/transformers/src/transformers/pipelines.py", line 107, in get_framework
    model = AutoModel.from_pretrained(model, revision=revision)
  File "/workspace/transformers/src/transformers/models/auto/modeling_auto.py", line 698, in from_pretrained
    pretrained_model_name_or_path, *model_args, config=config, **kwargs
  File "/workspace/transformers/src/transformers/modeling_utils.py", line 1156, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for BertModel:
	size mismatch for bert.encoder.layer.0.attention.self.query.weight: copying a param with shape torch.Size([768, 768]) from checkpoint, the shape in current model is torch.Size([256, 768]).
	size mismatch for bert.encoder.layer.0.attention.self.query.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for bert.encoder.layer.0.attention.self.key.weight: copying a param with shape torch.Size([768, 768]) from checkpoint, the shape in current model is torch.Size([256, 768]).
	size mismatch for bert.encoder.layer.0.attention.self.key.bias: copying a param with shape torch.Size([768]) from checkpoint, the shape in current model is torch.Size([256]).
	size mismatch for bert.encoder.layer.0.attention.self.value.weight: copying a param with shape torch.Size([768, 768]) from checkpoint, the shape in current model is torch.Size([256, 768]).


[WIP] Temp work on pipelines.


Fail to reload tokenizer from save_pretrained method

Hi,
to reproduce:
```
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
tokenizer.save_pretrained(".")
tokenizer = AutoTokenizer.from_pretrained(".")
```
with error msg:
```
file ./config.json not found
Traceback (most recent call last):
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/configuration_utils.py", line 389, in get_config_dict
    local_files_only=local_files_only,
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/file_utils.py", line 1015, in cached_path
    raise EnvironmentError("file {} not found".format(url_or_filename))
OSError: file ./config.json not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/models/auto/tokenization_auto.py", line 337, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/models/auto/configuration_auto.py", line 341, in from_pretrained
    config_dict, _ = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/data/stars/user/jhou/Test/pytorch_test/huggingface/jc-hou_fork/transformers/src/transformers/configuration_utils.py", line 401, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for '.'. Make sure that:

- '.' is a correct model identifier listed on 'https://huggingface.co/models'

- or '.' is the correct path to a directory containing a config.json file

```
Thanks.
transformers:4.1.0
 tokenizers: @mfuntowicz


how to checkpoint all the validation scores in huggingface trainer

Hi
I want to find the best model per evaluation score. Could you please give me more info, how I can checkpoint all evaluation scores in each step of training to find the best model? thanks


## Environment info
- `transformers` version: 4.1.1
- Platform: Darwin-18.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): I don't know

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below) I was trying to use this for further transfer learning.

## To reproduce

Steps to reproduce the behavior(the snippet I used):
```
import deepchem as dc
import tensorflow as tf
from transformers import AutoTokenizer, AutoModelForMaskedLM

tokenizer = AutoTokenizer.from_pretrained("seyonec/SMILES_tokenized_PubChem_shard00_160k")

model = AutoModelForMaskedLM.from_pretrained("seyonec/SMILES_tokenized_PubChem_shard00_160k")
```
Then I got this error message: "TypeError: expected str, bytes or os.PathLike object, not NoneType".

I appreciate any help/suggestions! Thanks very much.


No module named 'transformers.modeling_albert'

- Platform: Colab
- Python version:
- PyTorch version (GPU?):GPU
- Tensorflow version (GPU?):GPU
- Using GPU in script?:Yes

 examples/token-classification: @stefan-it
 -->

The problem arises when using:
* [ ] the official example scripts: I'm following the tutorial "23_Transfer_Learning_With_ChemBERTa_Transformers_Pt_2.ipynb" to reproduce the results. However I got this error message " No module named 'transformers.modeling_albert". I cannot figure out the reason. The following is the snippet I used:

```
!curl -Lo conda_installer.py https://raw.githubusercontent.com/deepchem/deepchem/master/scripts/colab_install.py
import conda_installer
conda_installer.install()
!/root/miniconda/bin/conda info -e
!pip install --pre deepchem
import deepchem
deepchem.__version__
from rdkit import Chem

!git clone https://github.com/NVIDIA/apex
!cd /content/apex
!pip install -v --no-cache-dir /content/apex
!pip install transformers
!pip install git+https://github.com/seyonechithrananda/simpletransformers.git@pip
!pip install wandb
!cd ..

!git clone https://github.com/seyonechithrananda/bert-loves-chemistry.git

%cd /content/bert-loves-chemistry

import os
import numpy as np
import pandas as pd
from typing import List
# import molnet loaders from deepchem
from deepchem.molnet import load_bbbp, load_clearance, load_clintox, load_delaney, load_hiv, load_qm7, load_tox21
from rdkit import Chem
# import MolNet dataloder from bert-loves-chemistry fork
from utils.molnet_dataloader import load_molnet_dataset, write_molnet_dataset_for_chemprop

tasks, (train_df, valid_df, test_df), transformers = load_molnet_dataset("clintox", tasks_wanted=None)

from simpletransformers.classification import ClassificationModel
import logging
logging.basicConfig(level=logging.INFO)
transformers_logger = logging.getLogger("transformers")
transformers_logger.setLevel(logging.WARNING)
```
Any suggestions and help are appreciated! Thank you!


Issue with 'char_to_token()' function of DistilBertTokenizerFast

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1
- Platform: Google Colab
- Python version: 3.8
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: NA

### Who can help:   **tokenizers: @mfuntowicz**

## Information

Model I am using DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased') to tokenize Squad 2.0 train and validate dataset.

The problem arises when using below code snippet to add_token_positions (start and end position) as below from https://huggingface.co/transformers/custom_datasets.html:

_def add_token_positions(encodings, answers):
    start_positions = []
    end_positions = []
    for i in range(len(answers)):
        start_positions.append(**encodings.char_to_token(i, answers[i]['answer_start'])**)
        end_positions.append(**encodings.char_to_token(i, answers[i]['answer_end'] - 1**))
        # if None, the answer passage has been truncated
        if start_positions[-1] is None:
            start_positions[-1] = tokenizer.model_max_length
        if end_positions[-1] is None:
            end_positions[-1] = tokenizer.model_max_length
    encodings.update({'start_positions': start_positions, 'end_positions': end_positions})

add_token_positions(train_encodings, train_answers)
add_token_positions(val_encodings, val_answers)_

The tasks I am working on is:
*Training model on SQUaD 2.0 using code given on https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0

## To reproduce

Steps to reproduce the behavior:

1. Follow the steps given on https://huggingface.co/transformers/custom_datasets.html#question-answering-with-squad-2-0 and then verify start and end position outcome using below code snippet in Expected behavior

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior:
- Start and End position are being defined using above code snippet which will be provided as training/validation data to model but end position is not derived as correct value due to some issue with char_to_token() function which is used to find out end position.
- Please find below snippet for verification that answer using start and end position after tokenization is not matching with actual answer.
- So the training data which is being fed to model after tokenization is incorrect

idx=8
print(f'Actual context: {train_contexts[idx]}')
print(f'Actual question: {train_questions[idx]}')
print(f"Actual answer: {train_answers[idx]['text']}")

start_position=train_encodings.char_to_token(idx,train_answers[idx]['answer_start'])
end_position =train_encodings.char_to_token(idx,train_answers[idx]['answer_end'])
print(f"Answer after tokenization: {tokenizer.convert_ids_to_tokens(train_encodings['input_ids'][idx][start_position:end_position])}")

OUTPUT:
**Actual context:** Beyoncé Giselle Knowles-Carter (/biːˈjɒnseɪ/ bee-YON-say) (born September 4, 1981) is an American singer, songwriter, record producer and actress. Born and raised in Houston, Texas, she performed in various singing and dancing competitions as a child, and rose to fame in the late 1990s as lead singer of R&B girl-group Destiny's Child. Managed by her father, Mathew Knowles, the group became one of the world's best-selling girl groups of all time. Their hiatus saw the release of Beyoncé's debut album, Dangerously in Love (2003), which established her as a solo artist worldwide, earned five Grammy Awards and featured the Billboard Hot 100 number-one singles "Crazy in Love" and "Baby Boy".
**Actual question:** When did Beyoncé rise to fame?
**Actual answer:** late 1990s
**Answer after tokenization:** ['late', '1990s', 'as', 'lead', 'singer', 'of', 'r', '&', 'b', 'girl', '-', 'group', 'destiny', "'", 's', 'child', '.', 'managed', 'by', 'her', 'father', ',', 'mathew', 'knowles', ',', 'the', 'group', 'became', 'one', 'of', 'the', 'world', "'", 's', 'best', '-', 'selling', 'girl', 'groups', 'of', 'all', 'time', '.', 'their', 'hiatus', 'saw', 'the', 'release', 'of', 'beyonce', "'", 's', 'debut', 'album', ',', 'dangerously', 'in', 'love', '(', '2003', ')', ',', 'which', 'established', 'her', 'as', 'a', 'solo', 'artist', 'worldwide', ',', 'earned', 'five', 'grammy', 'awards', 'and', 'featured', 'the', 'billboard', 'hot', '100', 'number', '-', 'one', 'singles', '"', 'crazy', 'in', 'love', '"', 'and', '"', 'baby', 'boy', '"', '.', '[SEP]', 'when', 'did', 'beyonce', 'rise', 'to', 'fame', '?', '[SEP]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]', '[PAD]']


Add FAVOR+ / Performer attention

## What does this PR do?

Adds support for the Performer / FAVOR+ attention mechanism, as described in the paper "Rethinking Attention with Performers" by Choromanski et al., 2020. Fixes #7675.

## How is it implemented?

Since Performer attention can be an unbiased estimator of traditional softmax attention, and pretrained models can be finetuned to work with it, the general consensus in the discussion on #7675 was that it should not be implemented as a single separate Transformer model. Ideally, we want all or most models in the transformers library to be able to use Performer attention.

In view of this, I've implemented the feature by creating three new classes: `PerformerAttention`, `TFPerformerAttention`, and `PerformerAttentionConfig`. These are implemented in the files `modeling_performer_attention.py`, `modeling_tf_performer_attention.py`, and `configuration_performer_attention.py` respectively in `src/transformers`.

Models are marked as supporting Performer attention by adding the `@supports_performer_attention` class decorator to the corresponding config class. This decorator adds the `attention_type: str` and `performer_attention_config: Optional[Union[dict, PerformerAttentionConfig]]` attributes to the config class, and also adds some boilerplate code to the class's `to_dict()` method to make sure JSON serialization works properly. It also registers the class so that the user can get a full list of Performer attention-supporting models with the function `performer_supporting_models_and_configs()`.

This isn't quite enough to get Performer attention to work for a new model, though. Adding Performer support to a model is inherently a somewhat tedious process, but I've tried to make it less tedious by implementing a `@init_performer_attention()` function decorator which can be added to the `__init__` method on the immediate parent of an attention module within a model— this will initialize either the model's own softmax attention module, or a `PerformerAttention` module, depending on how `attention_type` is set. You can see how this is implemented in `performer_attention_utils.py`.

This is all that you need to do for some models, although others will need a bit of extra work due to idiosyncracies in their implementation. I've already added Performer support to the following models: DistilBERT, BERT, RoBERTa, ELECTRA, LayoutLM, and TAPAS (in both PyTorch and TensorFlow). My hope is that other contributors will add support to other models relatively quickly.

Unit tests can be found in `test_performer_attention.py`. They do an exhaustive grid search of the enum and boolean config options and make sure that none of the 4.6k+ combinations causes a crash or a shape mismatch, and also make sure that the PyTorch and TensorFlow implementations have the same output, within numerical error, under all configurations.

## Rough edges

While I added extensive docstrings to `PerformerAttention` and `PerformerAttentionConfig`, which can be used to generate documentation, I haven't actually made the documentation files themselves. That will have to be left to another contributor, or to my future self— although honestly I've put quite a lot of time into this PR and would like to get on to other projects, so I would really appreciate it if someone else did it.

`PerformerAttention` supports using a custom CUDA kernel from the `fast_transformers` library to implement causally masked attention, although I have never actually been able to test this functionality because I don't have root access to the GPU server I use and therefore can't install NVCC. I'm hoping a reviewer could do that— it's a relatively straightforward feature so if there are any bugs in it it should be pretty easy to fix.

Also, the current code throws some odd linter errors which I haven't been able to figure out how to resolve and which don't seem to be consequential. Something about the code in RoBERTA, LayoutLM, etc. that is marked as being copied from BERT not matching the BERT code exactly. If a reviewer could figure out how to silence that error that would be greatly appreciated.

## Who can review?

@patrickvonplaten commented on #7675 and seemed excited about the PR, so I think he would be a good reviewer for this.


Music Transformers

# 🚀 Feature request

Hello guys! Thanks for your amazing work on the transformers! This is very needed and appreciated :)

I wanted to ask if it is possible to add a section/transformers dedicated specifically to Music. I searched GitHub and your model's repo but I could not find even a single model/solution that would be suitable for music.

NLP models are most capable when it comes to Music AI and I think it would be a great feature/section/branch to investigate/cover.

## Motivation

OpenAI MuseNet and Google Music Transformer. Enough said I think. If you never tried either, you have been really missing out. AFAIK MuseNet is built on a custom GPT2-like model/architecture. And Google used XLNet I think.

## Your contribution

I was able to create a decent model/code/implementation of Music AI based on GPT2 model/architecture. You are welcome to check it out here: https://github.com/asigalov61/Intelligent-VIRTUOSO

I used minGPT implementation to do it and it turned out quite capable and nice :)

However, I do want to ask the following:

1) What is the best Hugginface model/architecture you can recommend for Music AI applications? Please be specific and please give me a simple example of how to try it. This will be very much appreciated. I want to use Hugginface Transformers, so whatever works, please let me know. I am attaching a sample text file so that you can see my encoding, but I can adjust easily to any specs/needs of Huggnface Transformers. I have heard that BERT would be best at something like this but I can be mistaken...

2) Is there a nice Google Colab to try? I would prefer a simple working example to Python repos...

3) What would be the most optimal settings/hyperparameters you can recommend for GPT2 (right now I follow minGPT guidelines) and also what can you recommend to try for the most suitable Huggnface Transformer?

I really hope to hear constructive suggestions/advice because I want to learn and improve my skills and knowledge. Plus I love music almost as much as I love computers so I am quite passionate about both and would love to connect with others who are into Music and AI, if you guys exist...

Thank you very much in advance for your time and responses.
[TMIDI-TXT-Composition (13).txt](https://github.com/huggingface/transformers/files/5745869/TMIDI-TXT-Composition.13.txt)


[T5 model parallel] implement input auto-relocation + lots of refactoring/cleanup

Model Parallel

Model Parallelilsm Implementations

As I commented on in another incarnation of generalizing t5 model parallelism https://github.com/huggingface/transformers/pull/9316 so that it could be easily ported to other models I realized that it's quite unnecessary to try and remap inputs to specific devices where they will be needed in the future ahead of time. Since we have `forward` where we have access to the device of the parameters of that layer - we can completely automate the relocation of inputs to the correct devices just before `forward` is called. So this PR builds upon https://github.com/huggingface/transformers/pull/9316 and:

* [x] creates `@model_parallel_inputs_to_device` decorator used for `forward`, which automatically takes any inputs and puts them on the same device as the parameters of that layer. This allowed a complete removal of most of the `.to()` juggling logic for inputs, which was quite complex and noisy.
* [x] a lot of refactoring to make the MP as little invasive and noisy as possible, and fixing some small issues on the way.

I have tested this with:
```
pyt -sv tests/test_modeling_t5.py -k parallel
```
Which I'm not sure covers all bases, but the above tests pass.

@alexorona, please let me know what you think. And if you have real applications besides the great tests you wrote please see if it still works correctly. (It was so awesome having those tests in place! Thank you!) If it looks good and others support this proposal we can then look at doing the same for gpt2 and meanwhile I will look at bart.

@patrickvonplaten, @LysandreJik


Conda dependencies conflict with pip dependencies

## Environment info

- `transformers` version: 3.5.1, 3.3.1
- Platform: Windows 10, Anaconda
- Python version: 3.8

## Information

I'm installing a package built on top of `transformers` v3 in an Anaconda environment. The package is not available on Anaconda Cloud so I have to install it via `pip`. According to [the best practice](https://www.anaconda.com/blog/using-pip-in-a-conda-environment), I try to install as many requirements as possible with `conda`, including `transformers`. However, it turns out that the conda dependencies conflict with pip dependencies for `transformers` so that pip would try to downgrade the conda-installed `tokenizers` package, which `transformers` depends on.

The dependency information is as follows:

    `transformers` version
    conda dependency
    pip dependency

    3.5.1
    tokenizers==0.9.4
    tokenizers==0.9.3

    3.3.1
    tokenizers==0.9.3
    tokenizers==0.8.1rc2

I can't seem to find any resolution other than leaving the `transformers` installation to pip completely. Is there any other possible resolution?

### Who can help
Maybe @mfuntowicz can help

## To reproduce

Steps to reproduce the behavior:

1. Run "conda install transformers=3.5.1" or "conda install transformers=3.3.1"
2. Run "pip check"

## Expected behavior

Make conda dependencies compatible with pip dependencies.


Splitting texts longer that `tokenizer.max_length` into blocks of same size

## Environment info
`transformers-cli env` raises an ModuleNotFoundError, though I don't think it is relevant for my problem.

- `transformers` version: 4.0.0
- Platform: Arch Linux x86_64
- Python version: 3.9.1
- CPU only

### Who can help
It's a probably trivial tokenizer problem: @mfuntowicz
using a pretrained bert: @LysandreJik

## Information

I'm running successfully (exemplary for several models):
`
tokenizer = AutoTokenizer.from_pretrained("oliverguhr/german-sentiment-bert")
model = AutoModelForSequenceClassification.from_pretrained("oliverguhr/german-sentiment-bert")
inputs = tokenizer(text, return_tensors = "pt")
proOrCon = self.model(**inputs)
`

Now I have several `text`s that produce more than 512 tokens. I tried to split the `inputs` manually by copying and modifying as well as creating a dict in the same format, but apparently, the class object stores additional information that are required and not easily accessible.
I Also tried built-in functions from the Tokenizer:
`inputs = tokenizer(text, return_tensors = "pt", max_length=512, stride=0, return_overflowing_tokens=True, truncation=True, padding=True)
mapping = inputs.pop('overflow_to_sample_mapping')
`
But I don't get how to use the mapping for the next iteration, It's just a tensor with as many entries as tokens, counting up from 0.

I've looked at the documentation (@sgugger) here https://huggingface.co/transformers/internal/tokenization_utils.html and here https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer but the output format does not exactly match my results, since I don't get the overflowing tokens, just the mapping. I tried to look at the flair library as well, since it already implements something similar for transformers but their approach seems to be for another data format too. ( https://github.com/flairNLP/flair/blob/4d1bfec296ae8000268f8bbf62d71042e3714ace/flair/embeddings/token.py#L949 )

Can someone tell me what I am doing wrong? I just want to split the tokens in sizes that a bert model (512) can handle (blocks or sliding-window, I will have to test what works best). I didn't think it would be that hard, but I spent already a whole day on this.


[Seq2SeqTrainer] Fix Typo

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes a bug when one does not want to use `generate()` to evaluate in Seq2SeqTrainer. This PR probably deserves a test, but leaving this for a future PR when @sgugger is back.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Albert leverage sentence coherence predict loss(SOP) to optimizer its parameters, and I wanna employ it to score the coherence between two sentences. But when I use AlbertForPreTraining to load albert-xxlarge-v2 checkpoint, it reminds me that:
_Some weights of AlbertForPreTraining were not initialized from the model checkpoint at albert-base-v2 and are newly initialized: ['sop_classifier.classifier.weight', 'sop_classifier.classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference._

I try many times and find the output of each time is different, this means that the final linear layer for classification has not been loaded and just initial by random. I wanna know how to leverage pretrain SOP classification without fine-tuning?

Hope to get response rapidly.


Fail when running the multimodal example

Hi,
I tried to run the [multimodal example](https://github.com/huggingface/transformers/tree/master/examples/research_projects/mm-imdb).
By running:
```
python run_mmimdb.py \
    --data_dir ../dataset/ \
    --model_name_or_path bert-base-uncased \
    --output_dir ../output \
    --do_train \
    --do_eval \
    --max_seq_len 512 \
    --gradient_accumulation_steps 20 \
    --num_image_embeds 3 \
    --num_train_epochs 100 \
    --patience 5 \
    --overwrite_output_dir
```
I met the following error message:

```
12/27/2020 16:01:33 - INFO - __main__ -   ***** Running training *****
12/27/2020 16:01:33 - INFO - __main__ -     Num examples = 15513
12/27/2020 16:01:33 - INFO - __main__ -     Num Epochs = 100
12/27/2020 16:01:33 - INFO - __main__ -     Instantaneous batch size per GPU = 8
12/27/2020 16:01:33 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 160
12/27/2020 16:01:33 - INFO - __main__ -     Gradient Accumulation steps = 20
12/27/2020 16:01:33 - INFO - __main__ -     Total optimization steps = 9700
Epoch:   0%|                                                                                | 0/100 [00:00<?, ?it/s/data/stars/user/jhou/collection-stars/anaconda3/envs/pytorch_stable_171_pip_cu102/lib/python3.6/site-packages/PIL/Image.py:2837: DecompressionBombWarning: Image size (96592500 pixels) exceeds limit of 89478485 pixels, could be decompression bomb DOS attack.
  DecompressionBombWarning,
Iteration:   0%|                                                                                | 0/1940 [00:02<?, ?it/s]
Epoch:   0%|                                                                                | 0/100 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "run_mmimdb.py", line 572, in <module>
    main()
  File "run_mmimdb.py", line 525, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer, criterion)
  File "run_mmimdb.py", line 151, in train
    outputs = model(**inputs)
  File "/data/stars/user/jhou/collection-stars/anaconda3/envs/pytorch_stable_171_pip_cu102/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/stars/user/jhou/collection-stars/anaconda3/envs/pytorch_stable_171_pip_cu102/lib/python3.6/site-packages/transformers/models/mmbt/modeling_mmbt.py", line 366, in forward
    return_dict = return_dict if return_dict is not None else self.config.use_return_dict
  File "/data/stars/user/jhou/collection-stars/anaconda3/envs/pytorch_stable_171_pip_cu102/lib/python3.6/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'MMBTForClassification' object has no attribute 'config'

```
torch:1.7.1
transformers:4.0.1

I tried with torch:1.7.0, transformers:4.1.0, also failed with the same error.
Any adivce?
Thanks.


Bug: metrics inside on_evalute callback is passed wrongly

Hi
this is very helpful to save all metrics every eval_step with evaluation_stategy = steps, for this I wrote a callback as follows to access metrics inside this callback:

```
class EvaluationCallback(TrainerCallback):
    def  on_evaluate(self, args, state, control, **kwargs):
        print("### kwargs ", kwargs['metrics']) #
```

{'eval_loss': 972.89990234375, 'eval_acc': 0.0}

I pass this callback to the trainer.py

from what I see this metrics does not match the output of evaluate function, for instance in my case this is
`{'boolq_eval_loss': 525.3097534179688, 'boolq_eval_acc': 60.6, 'rte_eval_loss': 972.89990234375, 'rte_eval_acc': 0.0}
`
could you tell me how I can access output of evaluate() inside this callback as this is? thanks


[t5 model parallel] misc fixes

Model Parallelilsm Implementations

This PR:

* in 2 places fixes an assumption that devices on the device map are always ` (0, 1, 2, 3)` and:
   1. are ordered by their cuda device id and not `(2, 3, 0, 1)`
   2. have a stride of 1 and not `(0, 2, 3, 5) `
* adds a missing `to()`, removes a redundant `to()`
* removes obvious comments
* removes code that gets run twice
* this PR continues at #9323 -  I branched off from this PR and implemented an automatic remap of inputs and a lot refactoring.

I will comment on the reasons for changes in the code.

There is one gotcha wrt py36 w/o cython not having its dict ordered. Please see https://github.com/huggingface/transformers/pull/9316#discussion_r549068073

I think sorting out the logic first device/last device/is_this_the_last_layer_of_this_device and such logic should be abstracted away for readability, and not needing to replicate the same logic in each model. Perhaps `self.device_map` should be a smart class that can provide all the answers via its methods.

@alexorona, I'm studying your t5-mp implementation to do the same for bart. Thank you for doing the hard work of putting the foundation in place and porting 2 models!!!

Please have a look and let me know if my tweaks make sense. Your original code is excellent - I'm just trying to think how to make it easier to replicate it in other models and improve readability, hence a gazillion of questions/suggestions.

Also, if you don't mind I have a few design questions:

1. Could you please comment on why you are splitting the encoder between all devices on the device map and the same for the decoder? Won't it be more efficient performance-wise to put the encoder on the first group of devices and decoder on the second?

2. I also find it confusing that the device map doesn't map out the whole model, but just the encoder and assumes that the decoder has the same config. I'm not familiar with t5 but other models definitely can have encoder and decoder that don't at all match number of layers-wise. And while this is perhaps not the case for t5, I think the device map should be intuitively similar for all models as we slowly progress with porting other models to MP. That is I think it should include all layers of the model and not half of them.

@patrickvonplaten, @LysandreJik


[model site] search UI: language: tags, directionality and filtering

I tried to use the models site to find which models I can use for translation of specific languages, and here are the issues I have encountered while doing that:

1. many HF created models aren't tagged for the languages they are trained for - e.g. t5: https://github.com/huggingface/transformers/issues/9314 - would it be possible to go over the HF-created main models and ensure they are clearly tagged with languages they were trained for? These get downloaded a lot, so putting a bit of metadata will go a long way of making user's life a bit easier.

2. the language tags presume bi-direction, but many models have been trained in one direction only - e.g. most wmt and most t5 models. Would it be helpful to support not just the language tags but also the directional language tags if they are one-way only?

   The t5 models are one direction only, so probably need one direction tags.  Not sure how that would work with Language tags in search UI. Perhaps it's enough to indicate that in README, but as the number of models grows being able to quickly filter what's needed will save a lot of user's time, so perhaps planning ahead would be useful. i.e. if I need to perform a FR to EN translation, the user may benefit from getting hits for only models that can do that.

3. wrt search UI - I don't understand how a handful of special language tags is selected - there are about a dozen of language tags in the search API dropdown, Malay is there when there are hardly any models trained on that language, but Russian which is 5th on the list of number of models is not there.

   And those languages that are "favorite" aren't sorted... very strange.

4. search UI 2: And to get to the language one wants which is not on the favorite list - one has to solve the puzzle:

   * select "See All languages", which goes to https://huggingface.co/languages
   * hit on the list of models for that language,
   * then filter by the model type by typing it in and then one has arrived.

   Surely, there must be an easier way to select a language filter that doesn't take 3 steps which aren't obvious at all

5.  Moreover if I want to select 2 languages that aren't on the favorite list, I'm out of luck, since it's not possible with the current API. It only works for the favorite list.

   And even if some of us know how to hack the URL and manually insert: https://huggingface.co/models?filter=ru,en - this is an OR operation, how do I do AND operation or I guess this is related to item 2 of this Issue - how do I filter by the to/from language.

All of these requests/questions are nice to have and none a showstopper.

Thank you!

@julien-c


[model site] missing language tags for t5 models

Would it be possible to update core t5-* models' cards to include what languages they were trained on? Currently it says "en", which is very lacking.

e.g., see:
* https://huggingface.co/t5-base
* https://huggingface.co/t5-small
* etc.

The core t5 models should somehow have hits with https://huggingface.co/models?search=t5&filter=de, but they don't. Probably because they aren't tagged with the language tags. So they aren't found.

From: https://s3.amazonaws.com/models.huggingface.co/bert/t5-small-config.json
it looks like: French/German/Romanian. But also it looks to only support one direction, so probably adding the following would be clear enough to the end user:

* en_to_fr
* en_to_ge
* en_to_ro

Thank you.

@patrickvonplaten


[TFBart-like models] Problem with tf saving

## Context

Usually, encoder-decoder models require both `input_ids` and `decoder_input_ids` in order to do one forward pass. If one *e.g.* only passes the `input_ids` to TFT5 -> the model will complain:

```python
from transformers import TFT5ForConditionalGeneration
import tensorflow as tf
model = TFT5ForConditionalGeneration.from_pretrained("t5-small")

model(input_ids=tf.convert_to_tensor([10 * [2]])) # => will result in error saying `decoder_input_ids` have to be provided which is expected and correct
```

Now TFBart is a bit special in that it automatically generates the `decoder_input_ids` if they are not passed -> so that the above example would not throw an error for TFBartForConditionalGeneration.

The reason for this is this line: https://github.com/huggingface/transformers/blob/61443cd7d917ef323a799ee27bb4abc4344f0d11/src/transformers/models/bart/modeling_tf_bart.py#L1053

-> it automatically creates the `decoder_input_ids` from the `input_ids` if they are not provided. This is however more a hack than a good solution IMO. Soon we want to decouple the Bart-like models from each other and it would be good to delete this line from at least new Bart-like models. Now the problem.

## Problem:

The problem is now that if we delete these lines from Bart, then the `tf.saved_model.save(model, tmpdirname)` function does not work anymore. To reproduce:
Go into master and comment out this if statement in TFBart: https://github.com/huggingface/transformers/blob/61443cd7d917ef323a799ee27bb4abc4344f0d11/src/transformers/models/bart/modeling_tf_bart.py#L1053.

Then run the following code:

```python
from transformers import TFBartForConditionalGeneration
import tempfile
import tensorflow as tf

model = TFBartForConditionalGeneration.from_pretrained("sshleifer/bart-tiny-random")

input_ids = tf.convert_to_tensor([10 * [1]])
decoder_input_ids = tf.convert_to_tensor([10 * [8]])
inputs_dict = {"input_ids": input_ids, "decoder_input_ids": decoder_input_ids}

logits = model(inputs_dict).logits

model._saved_model_inputs_spec = None
model._set_save_spec(inputs_dict)

with tempfile.TemporaryDirectory() as tmpdirname:
    tf.saved_model.save(model, tmpdirname)
    model = tf.keras.models.load_model(tmpdirname)
    logits_2 = model(inputs_dict)["logits"]
```

=> the code will throw an error, but it should not! It seems like there is a weird naming mismatch between `input_ids` of `TFBartDecoder` and the `decoder_input_ids` in `TFBartModel`...@jplu I'd be thrilled if you could take a look at this and see how it can be solved.


RAG model implementation seems different from the paper

Hi folks,

Thanks for open-sourcing RAG! After reading the model description in the paper and the actual code, I noticed a few discrepancies:

1. The marginalization for `RagSequenceForGeneration` seems a bit strange. From line 998 to line 1001 ([link](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/modeling_rag.py#L998)), only the second tokens in `seq_logprobs` are getting scored by `doc_logprobs`:

```
# RAG-sequence marginalization
first_token_scores = seq_logprobs[:, :, :1, :]
second_token_scores = seq_logprobs[:, :, 1:2, :]
remainder = seq_logprobs[:, :, 2:, :]
rag_logprobs = torch.cat([first_token_scores, second_token_scores + doc_logprobs, remainder], dim=2)
```

I wonder if this is intended? I couldn't find this mentioned in the paper.

2. The marginalization for `RagTokenForGeneration` seems more like the Rag-sequence model in the paper, because the doc_scores are the same for all tokens in the same sequence. Is this correct?

```
# RAG-token marginalization
seq_logprobs = torch.nn.functional.log_softmax(seq_logits, dim=-1).view(
            seq_logits.shape[0] // n_docs, n_docs, -1, seq_logits.size(-1)
        )
doc_logprobs = torch.log_softmax(doc_scores, dim=1)
log_prob_sum = seq_logprobs + doc_logprobs.unsqueeze(-1).unsqueeze(-1)
return torch.logsumexp(log_prob_sum, dim=1)
```

Thanks in advance!


T5-base goes out of memory on 4 GPUs with as small batch size as 4

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.1
- Platform: LINUX
- Python version: 3.7
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?): -
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

 Trainer: @sgugger
 T5: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information

Model I am using T5-base with batch size of 8 and on 4 GPUs, I am always getting out of memory even with small batch sizes, This looks like a bug as this model is not really big. I am under time pressure. Is there anyone who could help me with this bug? thanks

The tasks I am working on is:
* GLUE benchmark

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

<!-- A clear and concise description of what you would expect to happen. -->

## Error Stack
```
  0%|                                                                                | 0/148395 [00:00<?, ?it/s]Traceback (most recent call last):
  File "finetune_trainer.py", line 303, in <module>
    main()
  File "finetune_trainer.py", line 239, in main
    training_args.optimize_from_scratch) else None
  File "/julia/codes/trainers/trainer.py", line 804, in train
    self.optimizer.step()
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/torch-1.7.1-py3.7-linux-x86_64.egg/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/transformers-3.5.1-py3.7.egg/transformers/optimization.py", line 285, in step
    state["exp_avg_sq"] = torch.zeros_like(p.data)
RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 2; 15.78 GiB total capacity; 14.10 GiB already allocated; 20.25 MiB free; 14.42 GiB reserved in total by PyTorch)
Traceback (most recent call last):
  File "finetune_trainer.py", line 303, in <module>
    main()
  File "finetune_trainer.py", line 239, in main
    training_args.optimize_from_scratch) else None
  File "/julia/codes/trainers/trainer.py", line 804, in train
    self.optimizer.step()
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/torch-1.7.1-py3.7-linux-x86_64.egg/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/transformers-3.5.1-py3.7.egg/transformers/optimization.py", line 296, in step
    denom = exp_avg_sq.sqrt().add_(group["eps"])
RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 15.78 GiB total capacity; 14.06 GiB already allocated; 4.25 MiB free; 14.44 GiB reserved in total by PyTorch)
Traceback (most recent call last):
  File "finetune_trainer.py", line 303, in <module>
    main()
  File "finetune_trainer.py", line 239, in main
Traceback (most recent call last):
  File "finetune_trainer.py", line 303, in <module>
    training_args.optimize_from_scratch) else None
  File "/julia/codes/trainers/trainer.py", line 804, in train
    main()
  File "finetune_trainer.py", line 239, in main
    self.optimizer.step()
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/torch-1.7.1-py3.7-linux-x86_64.egg/torch/optim/lr_scheduler.py", line 67, in wrapper
        training_args.optimize_from_scratch) else Nonereturn wrapped(*args, **kwargs)

  File "/julia/codes/trainers/trainer.py", line 804, in train
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/transformers-3.5.1-py3.7.egg/transformers/optimization.py", line 296, in step
    denom = exp_avg_sq.sqrt().add_(group["eps"])
RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 1; 15.78 GiB total capacity; 14.13 GiB already allocated; 10.25 MiB free; 14.46 GiB reserved in total by PyTorch)
    self.optimizer.step()
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/torch-1.7.1-py3.7-linux-x86_64.egg/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/transformers-3.5.1-py3.7.egg/transformers/optimization.py", line 285, in step
    state["exp_avg_sq"] = torch.zeros_like(p.data)
RuntimeError: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 3; 15.78 GiB total capacity; 14.10 GiB already allocated; 26.25 MiB free; 14.44 GiB reserved in total by PyTorch)
  0%|                                                                                | 0/148395 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/opt/conda/envs/t5/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/t5/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/torch-1.7.1-py3.7-linux-x86_64.egg/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/t5/lib/python3.7/site-packages/torch-1.7.1-py3.7-linux-x86_64.egg/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/t5/bin/python', '-u', 'finetune_trainer.py', '--local_rank=3', 'configs/glue.json']' returned non-zero exit status 1.
```


```
    nlp = en_trf_bertbaseuncased_lg.load()
  File "/usr/lib/python3.9/site-packages/en_trf_bertbaseuncased_lg/__init__.py", line 12, in load
    return load_model_from_init_py(__file__, **overrides)
  File "/usr/lib/python3.9/site-packages/spacy/util.py", line 239, in load_model_from_init_py
    return load_model_from_path(data_path, meta, **overrides)
  File "/usr/lib/python3.9/site-packages/spacy/util.py", line 202, in load_model_from_path
    cls = get_lang_class(lang)
  File "/usr/lib/python3.9/site-packages/spacy/util.py", line 74, in get_lang_class
    if lang in registry.languages:
  File "/usr/lib/python3.9/site-packages/catalogue.py", line 56, in __contains__
    has_entry_point = self.entry_points and self.get_entry_point(name)
  File "/usr/lib/python3.9/site-packages/catalogue.py", line 140, in get_entry_point
    return entry_point.load()
  File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/lib/python3.9/site-packages/spacy_transformers/__init__.py", line 2, in <module>
    from .pipeline.tok2vec import TransformersTok2Vec  # noqa
  File "/usr/lib/python3.9/site-packages/spacy_transformers/pipeline/__init__.py", line 3, in <module>
    from .wordpiecer import TransformersWordPiecer  # noqa
  File "/usr/lib/python3.9/site-packages/spacy_transformers/pipeline/wordpiecer.py", line 3, in <module>
    from tokenizations import get_alignments
  File "/usr/lib/python3.9/site-packages/tokenizations/__init__.py", line 2, in <module>
    from .tokenizations import (
ModuleNotFoundError: No module named 'tokenizations.tokenizations'
```


Entry-level demo of visual question answering

## Environment info
Is there any entry-level demo of visual question answering？(I am also interested in adding title for each image later on)
Better with `Trainer` added @sgugger. I follow the example from [Lxmert](https://huggingface.co/transformers/model_doc/lxmert.html).
```python
from transformers import LxmertTokenizer, LxmertModel
import torch

tokenizer = LxmertTokenizer.from_pretrained('unc-nlp/lxmert-base-uncased')
model = LxmertModel.from_pretrained('unc-nlp/lxmert-base-uncased')

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```
comes up
```
File "/home/yezli/miniconda3/lib/python3.8/site-packages/transformers/models/lxmert/modeling_lxmert.py", line 933, in forward
    assert visual_feats is not None, "`visual_feats` cannot be `None`"
AssertionError: `visual_feats` cannot be `None`
```
- `transformers` version:
- Platform: `Ubuntu 16.04.7 LTS`
- Python version: `Python 3.7.0`
- PyTorch version (GPU?): `No. But I am using PyTorch`
- Tensorflow version (GPU?): `No`
- Using GPU in script?: `No`
- Using distributed or parallel set-up in script?: `No`

### Who can help
@airsplay @bryant1410 Trainer @sgugger

## Information

Model I am using (Bert, XLNet ...): `[Lxmert](https://huggingface.co/transformers/model_doc/lxmert.html)`

The problem arises when using:
* [v] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
* [ ]
Following example from [Lxmert](https://huggingface.co/transformers/model_doc/lxmert.html).
```python
from transformers import LxmertTokenizer, LxmertModel
import torch

tokenizer = LxmertTokenizer.from_pretrained('unc-nlp/lxmert-base-uncased')
model = LxmertModel.from_pretrained('unc-nlp/lxmert-base-uncased')

inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

last_hidden_states = outputs.last_hidden_state
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
`Visual question answering` following example from [Lxmert](https://huggingface.co/transformers/model_doc/lxmert.html).

-- If you have code snippets, error messages, stack traces please provide them here as well.

```
File "/home/yezli/miniconda3/lib/python3.8/site-packages/transformers/models/lxmert/modeling_lxmert.py", line 933, in forward
    assert visual_feats is not None, "`visual_feats` cannot be `None`"
AssertionError: `visual_feats` cannot be `None`
```


[GPT2] Correct gradient checkpointing

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Previously, it was not possible to train GPT2 with gradient_checkpointing and `use_cache=False`. However `use_cache` should not be set to `True` when training. This PR corrects the behavior so that
gradient checkpointing does not require `use_cache=True`.

In addition, this PR changes lists to tuples in GPT2 for consistency with other models.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


from_pretrained does not load the modified part of model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): 1.7
- Tensorflow version (GPU?): -
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 Model Cards: @julien-c
 -->
 Model Cards: @julien-c
## Information

Hi
1) I am observing that if one modifies a model lest say T5ForConditionalGeneration and then use  T5ForConditionalGeneration.from_pretrained(...) then not all components of the models are loaded, meaning that the
parts of the model the user has modified are initialized to random!

2) I observe this from accuracy, could you tell me how I can check which weights from_pretrained is loading, I am a bit lost in the repository. thanks
## Expected behavior

If the model has been changed to have more layers, .... all the trained weights need to be loaded.


comment correction in test_retrieval_rag.py?

HI, in
https://github.com/huggingface/transformers/blob/master/tests/test_retrieval_rag.py#L223

comments of L223 and L224 are the same, maybe one of it should be "min inner product is reached with ...:
But I am not sure which one.
Pardon me if it is already correct.


[Don't merge] New design proposition for MAPPINGS in "auto" files

This PR would solve the issue: https://github.com/huggingface/transformers/issues/9250 but should not be used as a solution.

The PR should rather just show how the current design of all `OrderedDicts`, called `MAPPINGS_...` is suboptimal. It's impossible to add two values if both values have the same key. We need to be able to add a tokenizer class to `AutoTokenizers` even if the tokenizer does not have its own unique configuration class. We had a similar problem for Rag, since there is `RagForSequenceGeneration` and `RagForTokenGeneration` which both should be in the same mapping. IMO, the only 100% where we prevent "key" conflicts is if we use "multi-key" to "value" mappings as follows:

Tokenizer:
(PretrainedConfig (the corresponding config class, we're using now), str (the tokenizer class as a string, sometimes saved under `config.tokenizer_class`) -> TokenizerClass

Model:
(PretrainedConfig (the corresponding config class, we're using now), str (the model type as a string, sometimes saved under `config.model_type`) -> ModelClass

Some other "less" important shortcomings of this design:
- Because we often check with `isinstance` whether a config class is in an OrderedDict, we need to be very careful about the position of the key in the ordered dict and even wrote a test for this: https://github.com/huggingface/transformers/blob/21fc676645b1cae7cb9b5835435d57d90f9bc714/tests/test_modeling_auto.py#L221. This added complexity for such a simple feature is quite unnecessary IMO.
- These functions: https://github.com/huggingface/transformers/blob/21fc676645b1cae7cb9b5835435d57d90f9bc714/src/transformers/models/auto/tokenization_auto.py#L249 are more of a hack than a permanent solution IMO.
- We currently don't document those classes. I guess we could but it's just a mapping.

=> I would propose that we change all "MAPPING_FOR_..." to a class `MAPPING_FOR_` where we make sure that 100% backward compatibility is kept (except for that now it's not anymore a `OrderedDict` anymore, but a class.)

We can implement a `__getitem__` that could take inputs of different types (config for backward comp, but maybe also a "str" corresponding to the `"tokenizer_class"` or `"model_type"`). In general, it would give us more flexibility and prevent errors such as the one linked to this PR.

A possible design could look like this:

```python
class MappingGenerator:

    def __init__(self, keys_to_values: List[Tuple[Union[PretrainedConfig, str, Any]]]):
         self.tuple_to_class = OrderedDict(keys_to_values)
         all_configs = [keys_to_value[0] for keys_to_value in keys_to_values]
         self.duplicated_configs = set([x for x in all_configs if all_configs.count(x) > 1])
         self.config_to_class = OrderedDict([(keys_to_value[0], keys_to_value[2]) for keys_to_value in keys_to_values])
         # not possible to have key conflicts here
         self.str_to_class = OrderedDict([(keys_to_value[1], keys_to_value[2]) for keys_to_value in keys_to_values])

    def __getattr__(key: Union[PretrainedConfig, str, Tuple[PretrainedConfig, str]]):
         if isintance(key, str):
             return self.str_to_class[key]
         elif isinstance(key, PretrainedConfig):
             if key in self.duplicade_configs:
                 raise ...
             return self.config_to_class[key]
        elif isinstance(key, Tuple):
             return self.tuple_to_class[key]
        raise ...

TOKENIZER_MAPPING = MappingGenerator([
    (BertConfig, "BertTokenizer", BertTokenizer),
    (GPT2Config, "GPT2Tokenizer", GPT2Tokenizer),
    ...,
])
```

Keen to hear your thoughts on this @LysandreJik, @sgugger, @julien-c before opening a PR.


【 run_mlm.py 】attention_mask will be set to [1,1,...1] with DataCollatorForLanguageModeling

tokenized_datasets has been padded when padding="max_length"
when get a dataloader, we will use DataCollatorForLanguageModeling with tokenizer.pad at first
tokenizer.pad will set attention_mask to all 1 because input_ids have already been padded
so i want to know whether the attention mask meet expectations.


add translation example

# What does this PR do?

This PR will add translation example to the repo as per discussion with @thomwolf.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed.

 @patil-suraj, @sgugger
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Fix TF TransfoXL

# What does this PR do?

This PR fixes TransfoXL for graph compliancy.


Fix TF T5

# What does this PR do?

This PR fix a couple of bug in T5, one for graph compliancy and another one for the `past` output.


Fix  TF Funnel

# What does this PR do?

This PR fixes Funnel to make it full graph compliant. Even though all the slow/quick tests are passing and got similar results with few experiements, @sgugger I would appreciate that you thoroughly look at the changes in order to be sure no bugs have been introduced.