return_dict scores are inconsistent between sampling and beam search

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.4.0-1035-aws-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@SBrandeis @patrickvonplaten

## Information

When generating text using `model.generate` and `return_dict_in_generate` and beam search, beam search [sets the score for the first token to be 1e-9 across all beams other than the first](https://github.com/huggingface/transformers/blob/a449ffcbd2887b936e6b70a89e533a0bb713743a/src/transformers/generation_utils.py#L1576). This is not consistent with the sampling score which maintains the same score for the same token across different generations, and results in rather confusing behavior for score when using beam search.

## To reproduce

Steps to reproduce the behavior:

```python
import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
text = "How are"
input_ids = tokenizer.encode(text, return_tensors='pt')

# Beam Search
generated_outputs = model.generate(input_ids, return_dict_in_generate=True, output_scores=True, num_return_sequences=4, num_beams=4, max_length=input_ids.shape[-1] + 2)

gen_sequences = generated_outputs.sequences[:, input_ids.shape[-1]:]
# tensor([[ 345, 1804],
#        [ 345, 1016],
#        [ 356, 1016],
#        [ 345, 4203]])
probs = torch.stack(generated_outputs.scores, dim=1).softmax(-1)
gen_probs = torch.gather(probs, 2, gen_sequences[:, :, None]).squeeze(-1)
# tensor([[3.7034e-01, 1.4759e-01],
#        [1.9898e-05, 2.7981e-01],
#        [1.9898e-05, 3.2767e-05],
#        [1.9898e-05, 2.9494e-03]])

np.random.seed(42)
torch.manual_seed(42)

# Sampling
generated_outputs = model.generate(input_ids, return_dict_in_generate=True, output_scores=True, num_return_sequences=3, do_sample=True, top_p=.9, max_length=input_ids.shape[-1] + 2)

gen_sequences = generated_outputs.sequences[:, input_ids.shape[-1]:]
probs = torch.stack(generated_outputs.scores, dim=1).softmax(-1)
gen_probs = torch.gather(probs, 2, gen_sequences[:, :, None]).squeeze(-1)

gen_sequences
# tensor([[ 262, 1180],
#         [ 345, 1016],
#         [ 345, 4203]])

gen_probs[:, 0]
# tensor([0.1121, 0.5147, 0.5147])
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

When performing beam search the score from the model's logits should be output as the score rather than 1e-9.


OOM when trying to fine tune patrickvonplaten/led-large-16384-pubmed

I'm currently following this [notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing#scrollTo=tLM3niQqhEzP) but instead I'm using `patrickvonplaten/led-large-16384-pubmed`
```python

tokenizer = AutoTokenizer.from_pretrained("patrickvonplaten/led-large-16384-pubmed",)

led = AutoModelForSeq2SeqLM.from_pretrained(
   "patrickvonplaten/led-large-16384-pubmed",
    gradient_checkpointing=True,
    use_cache=False,
)
```

 instead of `allenai/led-large-16384` as the base model and tokenizer. I'm also using my own train/test data. With the exception of that, I kept everything else the same/consistent to that notebook as far as fine tuning. However, I'm running into OOM errors

```
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 15.78 GiB total capacity; 13.96 GiB already allocated; 20.00 MiB free; 14.56 GiB reserved in total by PyTorch)

  0%|          | 0/3 [00:10<?, ?it/s]
```
on a  couple of`Tesla V100-SXM2-16GB` and I'm not sure why that might be. The `batch_size=2` seems pretty small and I also set `gradient_checkpoint=True`. @patrickvonplaten and/or the surrounding community, I'd greatly appreciate any help with this


Problem fine-tuning BERTweet

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Linux-3.10.0-957.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no, single gpu

Maybe  @LysandreJik could help?

## Information

Model I am using BERTweet

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I get the following error:
```
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [27,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [28,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [29,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [30,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:658: indexSelectLargeIndex: block: [549,0,0], thread: [31,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "../models/jigsaw/tr-3.4//run_puppets.py", line 284, in <module>
    main()
  File "../models/jigsaw/tr-3.4//run_puppets.py", line 195, in main
    trainer.train(
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/trainer.py", line 888, in train
    tr_loss += self.training_step(model, inputs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/trainer.py", line 1250, in training_step
    loss = self.compute_loss(model, inputs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/trainer.py", line 1277, in compute_loss
    outputs = model(**inputs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1137, in forward
    outputs = self.roberta(
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 791, in forward
    embedding_output = self.embeddings(
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/dccstor/redrug_ier/envs/attack/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 121, in forward
    embeddings += position_embeddings
```

I just upgraded to the latest pytorch and transformers, I had same issue with different versions (transformers 3.4, torch 1.5.1).

Some more info on how I got here: https://github.com/VinAIResearch/BERTweet/issues/26

I've used the same code with 10+ other models (e.g., bert, roberta, distillbert) with no issues. One difference that I noticed in the config files for these models compared to BERTweet: max_position_embeddings is 512 for the models I'm using with no issues, while it is set to 130 in the config file for BERTweet.

One (related?) clarification question: what's the relation between `max_position_embeddings` and `max_seq_length`?

Any insights, more than welcome. Thanks!


Why two separators?

I want to fine-tune the model in Keras with my own dataset and I'm trying to figure out the format of the input sentences. When I take a look at `input_ids` I can see two sentence separators (`</s>` which has id 2) between each sentence after tokenization. Is this the expected behaviour? In that case, why are two separators needed? Will I get the same performance if I use one?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: `4.2.2`
- Platform: Ubuntu 18.04
- Python version: `3.7.5`
- PyTorch version (GPU?):
- Tensorflow version (GPU): `2.3.1`
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Nope

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (TFRoberta, TFXLMRoberta...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: https://huggingface.co/transformers/training.html#fine-tuning-in-native-tensorflow-2
* [x] my own task or dataset:

## To reproduce

```python
from transformers import RobertaTokenizer, glue_convert_examples_to_features
import tensorflow_datasets as tfds
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
data = tfds.load('glue/mrpc')
train_dataset = glue_convert_examples_to_features(data['train'].take(4), tokenizer, max_length=128, task='mrpc')
list(train_dataset.as_numpy_iterator())
```
```
Out[48]:
[({'input_ids': array([    0,   133, 14085,  4533,  3697,    40,  1760,    25, 20701,
           5473, 10974,  2156,  6062,    13,  1283,     9,   375,   514,
            479,     2,     2,   133,  4533,  3697,  1760,    25, 20701,
           5473, 10974,  2156,  1375,    15,   411, 10562,   479,     2,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1,     1,     1,     1,     1,     1,     1,     1,
              1,     1], dtype=int32),
   'attention_mask': array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)},
  0),
 ...
  ```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


[models] why aren't .bin files compressed for faster download?

Why are not the pretrained model files compressed?

It took ~25minutes yesterday to download 45GB t5-11b on a slow connection.

I did a quick test on a random `pytorch_model.bin` with default gzip options and it's 1/3rd less in size. And surely there must be a better compressor that can be used - but will require to be on the client's side so gzip might be good enough. This is not much of a difference for under 1GB files, but for large models this starts to add up.

It's not like you can diff a .bin file, so there is little value in having it stored as is from the RCS point of view. But perhaps I'm missing other aspects.

Perhaps 2 versions can be stored and the retriever could favor the compressed version for large files?

Cost-wise this change would introduce some 60% increase in storage if both versions are to be stored, but will have a huge saving in downloads.

The compression process can be a simple cronjob, so that it won't need to make users do anything special.

@julien-c, @LysandreJik, @patrickvonplaten, @sgugger


Fix TF LED/Longformer attentions computation

# What does this PR do?

This PR fixes the test `test_saved_model_with_attentions_output` for TF Longformer and LED that was failing due to an issue in computing some shapes in the attentions.

All the slow tests are now passing 🎉


run_ner.py raised error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: MacOS
- Python version: 3.6
- PyTorch version (GPU?):   1.7.1
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## To reproduce

Steps to reproduce the behavior:

1. bash [run.sh](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run.sh) to [run_ner.py](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)

## Error

```
ReferenceError: {'help': 'The name of the task (ner, pos...).'} does not reference a class __dict__
```


[License info] Longformer SQuAD finetuned model

Hello @patil-suraj ,

would it be possible to provide licensing information for the pretrained model weights shared at:
https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1

I would be interested in offering a Rust implementation for this model, but would like to know under which license this model was shared so that I can document my codebase accordingly.

Thank you!


Converting wav2vec2-base-960h to ONNX report an error while converting

First of all, I want to say thanks to @patrickvonplaten for the work done in adding the model. Great job!

I tried to convert the model to ONNX but got an error, do you have any ideas how to fix it?

What I did:
`python -m transformers.convert_graph_to_onnx --framework pt  --model facebook/wav2vec2-base-960h wav2vec2-base-960h.onnx`

But got an error:
```====== Converting model to ONNX ======
ONNX opset version set to: 11
Loading pipeline (model: facebook/wav2vec2-base-960h, tokenizer: facebook/wav2vec2-base-960h)
Using framework PyTorch: 1.7.0
Error while converting the model: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length.
```


Hotfixing tests

# What does this PR do?

Blenderbot decoderonly tests, also need to remove `encoder_no_repeat_ngram_size` from their config.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger
@LysandreJik

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Cleaning up `ConversationalPipeline` to support more than DialoGPT.

# What does this PR do?

Currently ConversationalPipeline was heavily biased towards DialoGPT
,which is the default model for this pipeline.

This PR proposes changes to put back the modifications specific to
DialoGPT into tokenizer-specific behavior wherever possible, by
creating `_build_conversation_input_ids` function that takes
conversation as input, and returns a list of ints corresponding
to the tokens. It feels natural to put here because all models
have probably different strategies to build input_ids from the
full conversation and it's the tokenizer's job to transform strings
into tokens (and vice-versa)

If `_build_conversation_input_ids` is missing, previous behavior is
used so we don't break anything so far (except for blenderbot where it's a fix).

This PR also contains a fix for too long inputs. There used
to be dead code for trying to limit the size of incoming input.
The introduced fixed is that we limit
within `_build_conversation_input_ids` to `tokenizer.model_max_length`.
It corresponds to the intent of the removed dead code and is actually
better because it corresponds to `model_max_length` which is different
from `max_length` (which is a default parameter for `generate`).

- Removed `history` logic from the Conversation as it's not relevant
anymore because tokenization logic has been moved to tokenizer.
And tokenizer cannot save any cache, and conversation cannot know
what is relevant or not.
Also it's not usable from `blenderbot` because the input_ids are
not append only (EOS tokens is always at the end).

- Added `iter_texts` method on `Conversation` because all
the code was literred with some form of this iteration of
past/generated_responses.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@LysandreJik
@patrickvonplaten

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


BART CausalLM example


German DistilBertModel raises an issue

## Environment info
- `transformers` version: 4.2.2
- Platform: Linux-5.4.0-65-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.8.0.dev20201202 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@julien-c @stefan-it  @LysandreJik

## Information

Model I am using: DistilBert

The problem arises when using:
* [ ] the official example scripts: (give details below)
```
from transformers import DistilBertTokenizer, DistilBertModel
import torch

tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-german-cased')
model = DistilBertModel.from_pretrained('distilbert-base-german-cased')
```

The tasks I am working on is:
* [ ] my own task or dataset:
Word2Vec encoding

## To reproduce

Steps to reproduce the behavior:

1. Simply run the code above
2. See the error message:
```
Traceback (most recent call last):
  File "/home/tarask/Desktop/Work/Code/Git/probabilistic-gesticulator/my_code/data_processing/annotations/encode_text.py", line 5, in <module>
    model = DistilBertModel.from_pretrained('distilbert-base-german-cased')
  File "/home/tarask/anaconda3/envs/gesture_flow/lib/python3.7/site-packages/transformers/modeling_utils.py", line 1034, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
  File "/home/tarask/anaconda3/envs/gesture_flow/lib/python3.7/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 419, in __init__
    self.embeddings = Embeddings(config)  # Embeddings
  File "/home/tarask/anaconda3/envs/gesture_flow/lib/python3.7/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 89, in __init__
    n_pos=config.max_position_embeddings, dim=config.dim, out=self.position_embeddings.weight
  File "/home/tarask/anaconda3/envs/gesture_flow/lib/python3.7/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 76, in create_sinusoidal_embeddings
    out[:, 0::2] = torch.FloatTensor(np.sin(position_enc[:, 0::2]))
RuntimeError: a view of a leaf Variable that requires grad is being used in an in-place operation.
```

## Expected behavior
No errors


Fix model templates

Some things were forgotten in the model templates after merging #9128


Add DETR

# What does this PR do?

It adds the first vision-only Transformer to the library! Namely [DETR](https://arxiv.org/abs/2005.12872), End-to-End Object Detection with Transformers, by Facebook AI. The main contribution of DETR is its simplicity: it replaces a lot of hand-engineered features (which models like Faster-R-CNN and Mask-R-CNN include) such as non-maximum suppression and anchor generation by just an end-to-end model and a clever loss function, while matching the performance of these heavily complex models.

For a really good explanation (which helped me a lot), see Yannic Kilcher's video [here](https://youtu.be/T35ba_VXkMYr). I'll provide a TLDR here:

The main thing to know is that an image of shape (batch_size, num_channels, height, width), so in case of a single image, a tensor of shape `(1, 3, height, width)` is first sent through a CNN backbone, outputting a lower-resolution feature map, typically of shape `(1, 2048, height/32, width/32)`. This is then projected to match the hidden dimension of the Transformer, which is `256` by default, using `nn.Conv2D`. So now we have a tensor of shape `(1, 256, height/32, width/32)`. Next, the image is flattened and transposed to obtain a tensor of shape `(batch_size, seq_len, d_model)` = `(1, width/32*height/32, 256)`. So a difference with NLP models is that the sequence length is actually longer than usual, but with a smaller `hidden_size` (which in NLP is typically 768 or higher).

This is sent through the encoder, outputting `encoder_hidden_states` of the same shape. Next, so-called **object queries** are sent through the decoder. This is just a tensor of shape `(batch_size, num_queries, d_lidek)`, with `num_queries` typically set to 100 and is initialized with zeros. Each object query looks for a particular object in the image. Next, the decoder updates these object queries through multiple self-attention and encoder-decoder attention layers to output `decoder_hidden_states` of the same shape: `(batch_size, num_queries, d_model)`. Next, two heads are added on top for object detection: a linear layer for classifying each object query into one of the objects or "no object", and a MLP to predict bounding boxes for each query. So the number of queries actually determines the maximum number of objects the model can detect in an image.

The model is trained using a **"bipartite matching loss"**: so what we actually do is compare the predicted classes + bounding boxes of each of the N = 100 object queries to the ground truth annotations, padded up to the same length N (so if an image only contains 4 objects, 96 annotations will just have a "no object" as class and "no bounding box" as bounding box). The [Hungarian matching algorithm](https://en.wikipedia.org/wiki/Hungarian_algorithm) is used to create an optimal one-to-one mapping between each of the N queries and each of the N annotations. Next, standard cross-entropy for the classes and L1 regression loss for the bounding boxes are used to optimize the parameters of the model.

Paper: https://arxiv.org/abs/2005.12872
Original repo: https://github.com/facebookresearch/detr

# Usage
Quick demo of my current implementation (with some cool attention visualizations): https://colab.research.google.com/drive/1aJ00yPxT4-PCMhSx2BipbTKqMSBQ80vJ?usp=sharing

(Old demo: https://colab.research.google.com/drive/1G4oWTOg_Jotp_2jJhdYkYVfkcT9ucX4P?usp=sharing)

Note that the authors did release 7 model variants (4 for object detection, 3 for panoptic segmentation). Currenty I've defined two models: the base `DetrModel` (which outputs the raw hidden states of the decoder) and `DetrForObjectDetection`, which adds object detection heads (classes + bounding boxes) on top. I've currently only converted and tested the base model for object detection (DETR-resnet-50). Adding the other models for object detection seems quite easy (as these only use a different backbone and I copied the code of the backbone from the original repo). Adding the models for panoptic segmentation (`DetrForPanopticSegmentation`) is on the to-do list as can be seen below.

# Done

- [x] load pretrained weights into the model
- [x] make sure forward pass yields equal outputs on the same input data
- [x] successful transcription
- [ ] add tokenizer (not sure if DETR needs one, see discussion below)
- [ ] add model tests: currently added 2 integration tests which pass, more tests to follow
- [ ] add tokenizer tests (not sure if DETR needs one, see discussion below)
- [ ] add docstrings
- [ ] fill in rst file

# Discussion

Writing DETR in `modeling_detr.py` went quite fast thanks to the CookieCutter template (seriously, the person who added this, thank you!!). The main thing to write was the conversion script (basically translating PyTorch's default [`nn.MultiHeadAttention`](https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html) to the self-attention mechanism defined in this library). DETR is an encoder-decoder Transformer, with only some minor differences, namely:

- it uses parallel decoding instead of autoregressive. So I assume I can delete all the `past_key_values` and `causal_mask` mechanisms? cc @patrickvonplaten
- it adds positional embeddings to the hidden states (in both the encoder and decoder) in each self-attention and encoder-decoder attention before projecting to queries and keys
- it uses the "relu" activation function instead of the default "gelu" one.
- during training, it helps to train on the outputs of each decoder layer. So what the authors do is predict classes + bounding boxes based on the output of each decoder layer, and also train these. This is a hyperparameter of `DetrConfig` called `auxiliary_loss`. This is also why I defined an additional `ModelOutput` called `BaseModelOutputWithCrossAttentionsAndIntermediateHiddenStates`, which adds intermediate activations of the decoder layers as output.

I wonder whether DETR needs a tokenizer. Currently, it accepts a `NestedTensor` as input to the encoder, not the usual `input_ids`, `attention_mask` and `token_type_ids`. The authors of DETR really like this data type because of its flexibility. It basically allows to batch images of different sizes and pad them up to the biggest image in the batch, also providing a mask indicating which pixels are real and which are padding. See [here](https://github.com/facebookresearch/detr/issues/116#issuecomment-651047468) for a motivation on why they chose this data type (the authors of PyTorch are also experimenting with this, see their project [here](https://github.com/pytorch/nestedtensor)). So maybe NestedTensor is something we could use as well, since it automatically batches different images and adds a mask, which Transformer models require?

Also, no special tokens are used, as the input of the encoder are just flattened images. The decoder on the other hand accepts object queries as input (which are created in `DetrModel`), instead of regular `input_ids`, `attention_mask` and `token_type_ids`. So I wonder whether these can also be removed.

# Future to-do

- [ ] Add `DetrForPanopticSegmentation`
- [ ] Let DETR support any backbone, perhaps those of the timm library as well as any model in the torchvision package

## Who can review?
@LysandreJik @patrickvonplaten @sgugger

Fixes #4663

Unfortunately, self-attention and MultiHeadAttention seem to be easier to understand than git.. I'm having some issues with line endings on Windows. Any help is greatly appreciated. I'm mainly opening this for discussing how to finish DETR.


Remove unintentional "double" assignment  in TF-BART like models

This PR fixes unintentionally used "double" assignment during reshaping of `attn_wegihts` in the TF BART-like models.

**Description:** Replace `attn_weights = attn_wegihts = tf.reshape(...)` with `attn_weights = tf.reshape(...)` and thus remove unintentionally used "double" assignment.

<hr>

Reviewer: @jplu


[DeepSpeed] [success] trained t5-11b on 1x 40GB gpu

Managed to train t5-11b on 1x 40GB gpu w/ Deepspeed (A100-SXM4-40GB)

Thank you, @PeterAJansen for letting me use your hardware!

Thank you, @jeffra and @samyam, for not believing that it is not possible to train t5-11b on 1x 40GB gpu w/ Deepspeed and supporting me that lead me to find a few bugs in the integration.

Sharing details for those who need.

**If you want to try this at home please make sure you use transformers master as some bug fixes were just merged in**

Well, it's similar to the t5-3b on 24GB success reported [here](https://huggingface.co/blog/zero-deepspeed-fairscale) and [here](https://github.com/huggingface/transformers/issues/8771#issuecomment-759176685).
But this time t5-11b on 1x 40GB gpu (or 4x if you wanted things faster)

As someone asked me before you need a huge amount of general RAM to use ZeRO-Offload for a huge model:
- for t5-3b on 1x 24GB gpu: ~71GB RAM
- for t5-11b on 1x 40GB gpu: ~234GB RAM

I was using `/usr/bin/time -v program` to get the peak memory measurement - it's the `Maximum resident set size` entry in the final report.

Question: I don't think  `/usr/bin/time` does the right thing for multi-process - I think it only measures the parent process. e.g. with 4x gpus it reported only 102GB RAM, but I clearly saw in top that it was around 240GB. If you have an easy way to measure peak memory that takes into an account forked processes I'm all ears.

Batch sizes on one gpu:
- with buffers of 5e8 I was able to run BS=2, which might be too small for training,
- but with 2e8 I managed to squeeze in BS=10 for training, but OOMed on prediction

I'm referring to these batch sizes in `ds_config.json`:
```
        "allgather_bucket_size": 2e8,
        "reduce_bucket_size": 2e8,
```
And I tested for 2x and 4x DDP as well, BS=16 OOMed, BS=8 was good so I used that - but could probably squeeze some more.

**edit1:** later tests show that my test was too short and wasn't getting the CPU Adam optimizer kick in, as it skips the first 20 or so tests because of the overflow. So once it kicks in it takes more GPU memory, so the practical BS is much smaller - I think around 2 on this setup. So most likely you will need to use `BS=2` for real work, until things get optimized even more.

**edit2:** things are getting re-shuffling in the tests, so the default `ds_config.json` file has moved in master to a new, hopefully permanent home. It's now at `examples/tests/deepspeed/ds_config.json` so you will need to adjust the command line to reflect this new location or simply copy it over to where the old one used to be.

here is the full benchmark:

```
# 1 gpu:
# only training fits with this BS, eval needs a smaller BS

export BS=8; rm -rf output_dir; PYTHONPATH=../../src USE_TF=0 /usr/bin/time -v deepspeed --num_gpus=1 ./finetune_trainer.py --model_name_or_path t5-11b --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16

{'train_runtime': 31.0897, 'train_samples_per_second': 0.257, 'epoch': 1.0}

# 2 gpus:

export BS=8; rm -rf output_dir; PYTHONPATH=../../src USE_TF=0 /usr/bin/time -v deepspeed --num_gpus=2 ./finetune_trainer.py --model_name_or_path t5-11b --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16

{'train_runtime': 17.9026, 'train_samples_per_second': 0.223, 'epoch': 1.0}

# 4 gpus

export BS=8; rm -rf output_dir; PYTHONPATH=../../src USE_TF=0 /usr/bin/time -v deepspeed --num_gpus=4 ./finetune_trainer.py --model_name_or_path t5-11b --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16

{'train_runtime': 10.4404, 'train_samples_per_second': 0.192, 'epoch': 1.0}
```

Checkpointing should allow making even bigger batch sizes.


Added Integration testing for Pytorch implementation of DistilBert model from issue #9948'

# Adds Integration testing  for Pytorch implementation of DistilBert from issue #9948

*Redid pull request
*My environment wasn't set up right.

I implemented the test as described in the issue linked. I ran the test and it passed. I can extend the tests after confirmation of this current PR. Please let me know what you think. Thank you

Fixes #9948

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


# 🚀 Faster batch translation with FSMT model

Currently, generating translations for multiple inputs at once is very slow using Transformers' `FSMTForConditionalGeneration` implementation. In fact it's about 10x slower than using the original FairSeq library. Can we speed this up by improving the implementation, potentially leaning on the original FairSeq approach?

## Motivation

I'm using FairSeq models for back translation as a way to augment text data. I've implemented this using the original FairSeq model (from PyTorch Hub) and Transformers.

### FairSeq implementation

```python
import torch

en2de = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.en-de.single_model', tokenizer='moses', bpe='fastbpe').cuda()
de2en = torch.hub.load('pytorch/fairseq', 'transformer.wmt19.de-en.single_model', tokenizer='moses', bpe='fastbpe').cuda()

def back_translate_fairseq(texts: List[str]) -> List[List[str]]:
    tokenized_texts = [en2de.encode(text) for text in texts]
    back_translations = [set() for _ in range(len(texts))]

    # Translate texts to German
    tokenized_de_texts = [
        [output['tokens'].cpu() for output in batch_output]
        for batch_output in en2de.generate(tokenized_texts, beam=2, sampling=True, sampling_topp=0.7)
    ]
    tokenized_de_texts_flat = [t for tt in tokenized_de_texts for t in tt]

    # Translate back to English
    tokenized_en_texts = [
        [output['tokens'].cpu() for output in batch_output]
        for batch_output in de2en.generate(tokenized_de_texts_flat, beam=2, sampling=True, sampling_topp=0.8)
    ]
    tokenized_en_texts_flat = [t for tt in tokenized_en_texts for t in tt]

    # Decode and deduplicate back-translations and assign to original text indices
    for i, t in enumerate(tokenized_en_texts_flat):
        back_translations[i // 4].add(de2en.decode(t).lower())

    # Remove back translations that are equal to the original text
    return [[bt for bt in s if bt != t] for s, t in zip(back_translations, map(str.lower, texts))]
```

### Transformers implementation

```python
from transformers import FSMTForConditionalGeneration, FSMTTokenizer

en2de_model_name = "facebook/wmt19-en-de"
en2de_tokenizer = FSMTTokenizer.from_pretrained(en2de_model_name)
en2de_model = FSMTForConditionalGeneration.from_pretrained(en2de_model_name)

de2en_model_name = "facebook/wmt19-de-en"
de2en_tokenizer = FSMTTokenizer.from_pretrained(de2en_model_name)
de2en_model = FSMTForConditionalGeneration.from_pretrained(de2en_model_name)

def back_translate_transformers(texts: List[str]) -> List[List[str]]:
    tokenized_texts = en2de_tokenizer.prepare_seq2seq_batch(texts, return_tensors="pt")
    back_translations = [set() for _ in range(len(texts))]

    # Translate texts to German and back to English
    generate_kwargs = {"num_beams": 1, "do_sample": True, "num_return_sequences": 2}
    tokenized_de_texts = en2de_model.generate(tokenized_texts["input_ids"], attention_mask=tokenized_texts["attention_mask"], top_p=0.7, **generate_kwargs)
    tokenized_en_texts = de2en_model.generate(tokenized_de_texts, top_p=0.8, **generate_kwargs)

    # Decode and deduplicate back-translations and assign to original text indices
    for i, t in enumerate(tokenized_en_texts):
        back_translations[i // 4].add(de2en_tokenizer.decode(t, skip_special_tokens=True).lower())

    # Remove back translations that are empty or equal to the original text
    return [[bt for bt in s if bt and bt != t] for s, t in zip(back_translations, map(str.lower, texts))]
```

Both of these functions generate comparable results, but using Transformers it takes **about 10x longer**.

In my use case I need back translations for hundreds of thousands of text snippets, which unfortunately makes the Transformers implementation unfeasible. I'd love to use Transformers though, as it is much easier to install and deploy (as we use Transformers for text classification anyway).


[trainer] a few fixes

This PR:
- removes `model.to(device)` - it's not needed for DeepSpeed.  but primarily this allows loading models that otherwise won't load - e.g. loading 45GB (fp32) to a 40GB GPU when using Deepspeed with fp16 - as it loads only 22GB of it. But currently we load all 45GB right away and well nothing works
- decouples 2 unrelated logical things related to model parallel, which was very confusing in the previous if/else incarnation
- fixes a bug that left a deepspeed model to be  wrapped in DDP, but it shouldn't, like a few other bugs of the same kind I created as things just happened to work until they didn't.

This PR enables t5-11b training on 1x 40GB gpu w/ Deepspeed  https://github.com/huggingface/transformers/issues/9996

@sgugger


Adversarial/amnesic heads

Feature request

Request for a new feature

# 🚀 Feature request

Task heads that backpropagate deliberately reversed gradients to the encoder. A flag requesting this behavior when constructing a task head.

## Motivation

Transfer learning experiments lend themselves to questions about the extent to which two tasks rely on the same information about a word/sentence, and to experiments probing whether and how word encodings contain/correspond to syntax trees, lemmas, frequencies, and other objects of linguistic/psycholinguistic study.

A difficulty is that a pretrained model, without fine-tuning, may already encode certain information too thoroughly and accessibly for intermediate training to make much of a difference. For example, BERT's masked language modeling objective produces word encodings in which syntax information is readily accessible. Intermediate training on a syntax task requires training a task head to extract this information, of course, but it will result in very little reorganization of the encoder itself.

Adversarial training, such as the amnesic probing of Elazar et al. 2020, can avoid this pitfall. Intermediate training can aim to burn particular information *out* of the encodings, and measure how much this impairs trainability of the target task. Strictly reversing the sense of the training data won't do it though; getting all the answers exactly wrong requires just as much domain knowledge as getting them all right does. And randomizing the labels on training data may just result in a feckless task head, one that discards useful information passed to it from the encoder, rather than affecting the encoder itself.

Ideally, then, the task head would be trained toward correctly reproducing gold-standard labels, but would flip all its gradients before backpropagating them to the shared encoder, thus training it not to produce precisely the signals that the task head found most informative. The following work by Cory Shain illustrates flipping gradients in this way (although it's not applied to shared-encoder transfer learning, but rather to development of encoders that disentangle semantics from syntax).

	https://docs.google.com/presentation/d/1E89yZ8jXXeSARDLmlksOCJo83QZdNbd7phBrR_dRogg/edit#slide=id.g79452223cd_0_19
	https://github.com/coryshain/synsemnet

## Your contribution

I am deeply unfamiliar with pytorch, unfortunately, and utterly ignorant of tensorflow. I can't offer much.


[documentation] non-PR doc editing

Is there a way we could have some of the docs that can be edited other than through PRs?

For example I've been working on these 2 docs:
- https://github.com/huggingface/transformers/issues/9766
- https://github.com/huggingface/transformers/issues/9824

1. So I do a lot of incremental edits and doing that via PRs would be very difficult to do as it's a big work in progress - that's why I started with just an Issue comment
2. it's important that the work in progress is readable, PRs aren't great for that
2. I'd be great if others could collaborate on editing
3. Yet, as these shape up, we want these in the documentation and not a random page somewhere
4. I already run into a problem with git where somehow it switched to an old edition of the comment and won't let me revert to the newer version of the comment.

Perhaps we could have some wiki pages that can be linked into the main menu? Then many can collaborate and there is no need to do frequent PR cycles. Not sure if it's great, since it'd take the user away from the main website?

Or perhaps the source could be wiki but when the docs are built it could pull the .md from the wiki and build it as if it were a normal .md page in the git repo?

I'm totally open to other ideas.

Thank you!

@sgugger


Implementing the test integration of BertGeneration

# What does this PR do?

this PR aims to fix issue #9947 by implementing an integration test


create LxmertModelIntegrationTest

# What does this PR do?

this pr fix issue #9951 as it implements an integration test for  LXMERT

@LysandreJik


Add head_mask and decoder_head_mask to TF LED

This PR implements `head_mask` and `decoder_head_mask` for TF LED (and Longformer as there's a copy dependency) and it is the follow-up to the open issue #9814.

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9639).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewers: @jplu @patrickvonplaten @LysandreJik @sgugger


Add `from_slow` in fast tokenizers build and fixes some bugs

# What does this PR do?

This PR adds an argument to the initialization of the `PreTrainedTokenizerFast` to force the conversion from a slow tokenizer. This will be useful to help users re-build the `tokenizer.json` file for some models where we can't update faulty ones right now without breaking backward compatibility (see #9637).

In passing it fixes a few bugs:
- wrong formatting for the documentation
- the fast sentencepiece tokenziers don't have an `sp_model` attribute so remove the documentation for that
- BarthezTokenizerFast was not registered properly in the autotokenizers, so `AutoTokenizer` was not finding it


How to train on shards of bookcorpus + wikipedia + openwebtext on 1 TB disk.

# 🚀 Feature request

Hello, I am trying to pretrain from scratch a custom model on bookcorpus + wikipedia + openwebtext but I only have a 1TB disk. I tried to merge 20% of each one and then reload the training on other 20% of each, but I am having issues with the learning rate scheduler. So if I hardcode the max_steps to the total size of the dataset (100% of all concatenated) it do various passes to the 20%. The same that putting 5 epochs. But I have to deal with lots of points like LambdaLR that is in pure pytorch to set the epoch, current step and all the states. It's a little pain!

Any suggestion?

## Motivation

I wan to train from scratch a linear attention model with some modifications

## Your contribution

The idea on how to train medium models with big datasets and regular hardware.


Loss function inputs for DistilBertForTokenClassification-like model using DistilBertModel

I want to fine tune my `DistilBertModel` just like `DistilBertForTokenClassification` for NER task by using nn.Module and building classifier on top myself.

But the problem is - I do not understand how to calculate loss function. In [official tutorial](https://huggingface.co/transformers/custom_datasets.html) it is explained only for seq classification which has multiple labels for input. But token classification is different!

I am trying to do something like

```
        # For each batch of training data...
        for step, batch in enumerate(train_dataloader):
            batch_counts +=1
            # Load batch to GPU
            # b_input_ids, b_attn_mask, b_labels = tuple(t.to(device) for t in batch)
            b_input_ids, b_attn_mask, b_labels = \
              batch['input_ids'].to(device), batch['attention_mask'].to(device), batch['labels'].to(device)

            # Zero out any previously calculated gradients
            model.zero_grad()

            # Perform a forward pass. This will return logits.
            logits = model(b_input_ids, b_attn_mask)

            # Compute loss and accumulate the loss values
            print('[DEBUG]', logits.shape, b_labels.shape)
            loss = loss_fn(logits, b_labels)
```

The last line:

> loss = loss_fn(logits, b_labels)

will definitely raise error.

I don't know how the expected label should look like and even the labels have `-100` extra instead of instead of indices.

Full code(fairly straightforward with comments) : https://colab.research.google.com/drive/1FWPEV_5eOhveiT2AQyuSYm1Ka1pgeY2f?usp=sharing


[Proposal] Adding new `encoder_no_repeat_ngram_size` to `generate`.

# What does this PR do?

Blenderbot results seemed off compared to original ParlAI script:
`https://parl.ai/projects/recipes/`. Notably the model seems
to repeat a lot what was said during the conversation.

The actual problem was that `no_repeat_ngram_size` actually applies
to the `encoder_input_ids` but HF's `no_repeat_ngram_size` applies
to the previously generated ids (within the decoder). The history
conversation of blenderbot is within the `encoder` part so that
explains why HF's implementation had the repetitions.

This fix was focused on blenderbot *not* small and added tests
for those because they are quite different in configuration.

This change includes:

- Adding a new EncoderNoRepeatLogitProcessor.
- Adding 1 new arg to `generate` (`encoder_no_repeat_ngram_size`)
- Adding 1 new config parameter `encoder_no_repeat_ngram_size`.
- Adding 2 tests, one for the pipeline (high level, inputs exhibited
repeat behavior, one low level for EncoderNoRepeatLogitProcessor)
- Factored NoRepeatLogitProcessor so that logic could be reused.

Further work:

- Blenderbot conversational pipeline still does not behave correctly
 as they way input is prepared within the pipeline is still incorrect
(follow up PR)
- Blenderbot allows the bot to have personas, which is done by
prepending "your personna: XXXX" to the input, this could be explored
too in a follow up PR.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@patrickvonplaten
@LysandreJik

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Added integration tests for Pytorch implementation of the FlauBert model

Fixes #9950

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik


Added integration tests for Pytorch implementation of the ELECTRA model

Fixes #9949

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik


Can't make sense of encoding for a downloadable AutoTokenizer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2

### Who can help

Probably @n1t0 or @LysandreJik (AutoTokenizer)

## To reproduce

Steps to reproduce the behavior:

1. Boot up an AutoTokenizer using `AutoTokenizer.from_pretrained("sberbank-ai/rugpt3small_based_on_gpt2")`
2. Execute `tokenizer.get_vocab()`

The vocabulary contains gibberish instead of Russian tokens (yet the model works fine):

![image](https://user-images.githubusercontent.com/31738272/106778421-03fc8200-6657-11eb-8f24-48daad35ff16.png)

How do I decode and read the actual tokens?


Added integration tests for Pytorch implementation of the ALBERT model

Fixes #9945

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik


Added integration tests for TensorFlow implementation of the MPNet model

Added integration tests for TensorFlow implementation of the ALBERT model

Fixes #9956

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik


Added integration tests for TensorFlow implementation of the mobileBERT

Added integration tests for TensorFlow implementation of the ALBERT model

Fixes #9955

Before submitting

  This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  Did you read the contributor guideline,
Pull Request section?
  Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
  Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
  Did you write any new necessary tests?

@LysandreJik


[run_clm.py] fix getting extention

# What does this PR do?

Fixes  #9927


Added integration tests for TensorFlow implementation of the ALBERT model

Fixes #9946

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

@LysandreJik


TF DistilBERT integration tests

Added integration tests for TensorFlow implementation of the DistilBERT model

Fixes #9953

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Who can review?
@LysandreJik


Make use of attention_mask in Trainer's compute_metrics

# 🚀 Feature request
In Trainer's training loop, the `compute_metrics` function takes a `EvalPrediction(predictions=preds, label_ids=label_ids)` object as input.
It should also be able to use `inputs['attention_mask']` to mask the irrelevant predictions (those for which attention_mask is 0).

## Motivation
I am working on a NER task and find myself having no way to filter out irrelevant predictions.
In the following example, the `raw_pred` leads to an accuracy of 43%, while the `masked_pred` gives me almost 77% (value 0 in attention_mask has been casted into `nan` before being applied to `raw_pred`)

**ground_truth**
```python
tensor([[0, 5, 5, 5, 0, 6, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 1, 1, 1, 1, 2, 2, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 0],
        [0, 5, 6, 6, 6, 6, 6, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 3, 3, 4, 0, 3, 3, 0, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [0, 0, 0, 0, 0, 0, 0, 0, 5, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
```
**raw_pred**
```python
tensor([[0, 2, 5, 5, 6, 4, 1, 2, 3, 1, 3, 3, 2, 1, 2, 1, 2, 6, 3],
        [2, 1, 2, 2, 2, 2, 2, 0, 1, 2, 2, 2, 0, 0, 0, 0, 0, 0, 2],
        [4, 4, 4, 0, 0, 0, 0, 4, 0, 0, 0, 4, 4, 4, 4, 4, 4, 4, 0],
        [0, 6, 6, 4, 4, 5, 6, 6, 4, 0, 3, 3, 3, 2, 3, 3, 1, 2, 2],
        [0, 0, 0, 0, 5, 4, 5, 5, 5, 6, 4, 0, 2, 1, 2, 1, 1, 3, 3]])
```
**attention_mask**
```python
tensor([[1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0]])
```
**masked_pred**
```python
tensor([[0., 2., 5., 5., 6., 4., 1., 2., nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,nan],
        [2., 1., 2., 2., 2., 2., 2., 0., 1., 2., 2., 2., 0., 0., 0., 0., 0., 0.,nan],
        [4., 4., 4., 0., 0., 0., 0., 4., 0., 0., 0., 4., 4., 4., 4., 4., 4., 4., 0.],
        [0., 6., 6., 4., 4., 5., 6., 6., 4., 0., nan, nan, nan, nan, nan, nan, nan, nan,nan],
        [0., 0., 0., 0., 5., 4., 5., 5., 5., 6., 4., 0., nan, nan, nan, nan, nan, nan,nan]])
```


attention_mask -> encoder_attention_mask in cross attn of BERT-like models


Fix GroupedLinearLayer in TF ConvBERT

Fixing an issue with `call` function in `GroupedLinearLayer` of ConvBERT


DebertaForSequenceClassification documents  examples  report RuntimeError: Index tensor must have the same number of dimensions as input tensor

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: macos
- Python version: 3.8.3
- PyTorch version (GPU?): no
- Tensorflow version (GPU?): no
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):  Deberta

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

    from transformers import DebertaTokenizer, DebertaForSequenceClassification
    import torch
    tokenizer = DebertaTokenizer.from_pretrained('microsoft/deberta-base')
    model = DebertaForSequenceClassification.from_pretrained('microsoft/deberta-base')
    inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
    labels = torch.tensor([1]).unsqueeze(0)
    outputs = model(**inputs, labels=labels)
    loss = outputs.loss
    logits = outputs.logits
    print(loss)
    print(logits)

Exception like below:

Traceback (most recent call last):
  File "/Users/admin/git/transformers/myexample4/deberta_MLM.py", line 65, in <module>
    sequence_classify()
  File "/Users/admin/git/transformers/myexample4/deberta_MLM.py", line 45, in sequence_classify
    outputs = model(**inputs, labels=labels)
  File "/Users/admin/virtulenv/py38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/admin/git/transformers/src/transformers/models/deberta/modeling_deberta.py", line 1169, in forward
    labels = torch.gather(labels, 0, label_index.view(-1))
RuntimeError: Index tensor must have the same number of dimensions as input tensor

Process finished with exit code 1

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


[research proj] [lxmert] remove bleach dependency

github reports `bleach==3.1.5` to have a vulnerability and it's not really used anywhere in the code, and because it has a fixed version set that is vulnerable, so just as well remove it completely from deps.
https://github.com/huggingface/transformers/security/dependabot/examples/research_projects/lxmert/requirements.txt/bleach/open

@LysandreJik, @sgugger, @patrickvonplaten


fix steps_in_epoch variable in trainer when using max_steps

# What does this PR do?

This PR fix the calculation of `steps_in_epoch` in `trainer.py`

The 'step' in `steps_in_epoch` means one backward
The 'step' in `max_steps` means one parameter updating (taking gradient accumulation into account)

This bug does not affect training process, just make logging info weired

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Maybe @sgugger will be more interested in this

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Disk memory management

I am wondering if you guys could add support for disk memory management when running large transformer models. At least when running on my laptop with limited DRAM, it is not feasible to fully materialize some of the larger models (T5-3b e.g. or even T5-large) in DRAM, especially if there are other memory intensive tasks running (like the IDE). I'm wondering if it's possible for the Huggingface library to not materialize the entire model in DRAM as a Python object for these larger models and instead re-materialize them layer by layer from the disk.


Added an integration test for the Pytorch implementation of the DistilBERT model from issue #9948

# Adds Integration testing  for pytorch implementation of DistilBert from issue #9948

I implemented the test as described in the issue linked. I ran the test and it passed. I can extend the tests after confirmation of this current PR. Please let me know what you think. Thank you

Fixes #9948

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


Bump bleach from 3.1.5 to 3.3.0 in /examples/research_projects/lxmert

dependencies

Pull requests that update a dependency file

Bumps [bleach](https://github.com/mozilla/bleach) from 3.1.5 to 3.3.0.

Changelog

Sourced from bleach's changelog.

Version 3.3.0 (February 1st, 2021)

Backwards incompatible changes

clean escapes HTML comments even when strip_comments=False

Security fixes

Fix bug 1621692 / GHSA-m6xf-fq7q-8743. See the advisory for details.

Features

None

Bug fixes

None

Version 3.2.3 (January 26th, 2021)

Security fixes

None

Features

None

Bug fixes

fix clean and linkify raising ValueErrors for certain inputs. Thank you @Google-Autofuzz.

Version 3.2.2 (January 20th, 2021)

Security fixes

None

Features

Migrate CI to Github Actions. Thank you @hugovk.

Bug fixes

fix linkify raising an IndexError on certain inputs. Thank you @Google-Autofuzz.

Version 3.2.1 (September 18th, 2020)

... (truncated)

Commits

79b7a3c Merge pull request from GHSA-vv2x-vrpj-qqpq

842fcb4 Update for v3.3.0 release

1334134 sanitizer: escape HTML comments

c045a8b Merge pull request #581 from mozilla/nit-fixes

491abb0 fix typo s/vnedoring/vendoring/

10b1c5d vendor: add html5lib-1.1.dist-info/REQUESTED

cd838c3 Merge pull request #579 from mozilla/validate-convert-entity-code-points

612b808 Update for v3.2.3 release

6879f6a html5lib_shim: validate unicode points for convert_entity

90cb80b Update for v3.2.2 release

Additional commits viewable in compare view

[![Dependabot compatibility score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=bleach&package-manager=pip&previous-version=3.1.5&new-version=3.3.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
- `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
- `@dependabot use these labels` will set the current labels as the default for future PRs for this repo and language
- `@dependabot use these reviewers` will set the current reviewers as the default for future PRs for this repo and language
- `@dependabot use these assignees` will set the current assignees as the default for future PRs for this repo and language
- `@dependabot use this milestone` will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/huggingface/transformers/network/alerts).


[trainer] new in pytorch: `torch.optim._multi_tensor` faster optimizers

WIP

Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

Back in September pytorch introduced `torch.optim._multi_tensor` https://github.com/pytorch/pytorch/pull/43507 which should be much more efficient for situations with lots of small feature tensors (`transformers`) and thus should show an appreciable speed up in training. If someone is interested in the progress of this project here is the stack to track: https://github.com/pytorch/pytorch/pull/48223

This feature is currently an alpha stage, so users can try to use it by simply replacing `torch.optim` with `torch.optim._multi_tensor` in HF Trainer or their own trainer.

Eventually it'll replace `torch.optim` so there is nothing that we need to do otherwise.

@blefaudeux who alerted me to this improvement suggested it should have good speed ups for the DDP/Sharded DDP training.

If resources allow it'd be good to run some benchmarks. Please feel free to beat me to it.

Thanks to @blefaudeux for the heads up, and @izdeby for working on this enhancement and clarifying where things are at.

heads up to: @sgugger, @patrickvonplaten - nothing else that needs to be done.


Add head_mask, decoder_head_mask, cross_head_mask to ProphetNet

This PR implements `head_mask`, `decoder_head_mask` and `cross_head_mask` for ProphetNet (and Longformer as there's a copy dependency) and it is the follow-up to the open issue #9814.

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9569).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewers: @patrickvonplaten


Model Save/Load Fails for Hadoop File Server

## Environment info
- `transformers` version: 4.2.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@n1t0, @LysandreJik @sgugger

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [X] the official example scripts:
```python
config = pipeline.get_common_model_file(model_name, Constants.CONFIG)
model = AutoModelForSequenceClassification.from_pretrained(config=config, pretrained_model_name_or_path='http://192.168.0.61:50070/webhdfs/v1/user/root/NLPEngine/models/bert-base-uncased/pytorch_model.bin?op=OPEN')
```
* [ ] my own modified scripts:

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset:
A Conversational (Multi-Turn Dialog) Dataset for Task of Knowledge Selection. Dataset is raising no issues.

## To reproduce

Steps to reproduce the behavior:

1. Install Hadoop File Server
2. Setup WebHDFS to use cURL commands to load/save files on HDFS or directly use hadoop for that. Just make sure that all your servers are listed in your system's hosts configurations.
3. Place bert-base-uncased model on Hadoop (anywhere).
4. Try and access it from the code mentioned above.

## Expected behavior

The model should be loaded from the file, as it would be loaded locally or from a server that returns an E-Tag, but Hadoop is not configured / built for returning E-Tags. It first returns a temporary-redirect URL, and then from that the actual object retrieved off of one of the servers in its cluster(s).
If I turn off the E-Tag validation in the source code, then it starts working perfectly, but as of now, its part of the source-code, and thats causing this code to crash.
Here is the change I made to get it to work (in my copy of the library's code):

```python
File "C:\Users\rsiddiqui\Anaconda3\Lib\site-packages\transformers\file_utils.py", line 1182, in get_from_cache
etag = ''
File "C:\Users\rsiddiqui\Anaconda3\Lib\site-packages\transformers\file_utils.py", line 1187, in get_from_cache
etag = r.headers.get("X-Linked-Etag", '') or r.headers.get("ETag", '')
```


Deepseep configs keys probelm

https://github.com/huggingface/transformers/blob/24881008a6743e958cc619133b8ee6994ed1cb8c/src/transformers/integrations.py#L288

I guess it should be `if len([x for x in bs_keys if x in config.keys()]) <= 0: ` or `if not len([x for x in bs_keys if x in config.keys()]) <= 0: `


What is the correct way to use Adafactor?

Hi, from the papers I've seen that Adafactor is typically used with no learning rate (as in Pegasus paper), however, when I try to execute run_seq2seq.py or seq2seq/finetune_trainer.py from your examples, and set --adafactor parameter, without specifying learning rate (for no learning rate), it uses the default 3e-05. Is there a way to use Adafactor without learning rate?


How to resize RobertaLMHead with pretrained weights?

Hi, I'm trying to train my model with a new token 'name', but it keeps throwing size mismatch error.
I don't know how to **resize RobertaLMHead** while loading pretrained weights from 'roberta-base'

Setting Tokenizer
```
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
tokenizer.add_tokens('<name>')
```

Setting Model
```
model = MaskedLM.from_pretrained('roberta-base')
```

Model Structure
```
class MaskedLM(RobertaPreTrainedModel):
    def __init__(self, config):
        super().__init__(config=config)
        self.roberta = RobertaModel(config)
        self.lm_head = RobertaLMHead(config)
        self.refinement_num = 5
        self.mask_id = 50264
        self.init_weights()

    def forward(...):
         self.roberta.resize_token_embeddings(50266)
         ## HOW TO RESIZE LM HEAD?! ##
         # self.lm_head.resize_token_embeddings(50266)
         outputs = self.roberta(input_ids, attention_mask)
         prediction_scores = self.lm_head(outputs[0])
         ...

```

I tried ```_get_resized_lm_head``` from [here](https://huggingface.co/transformers/_modules/transformers/modeling_utils.html#PreTrainedModel._get_resized_lm_head)
But it doesn't work as RobertaLMHead has no ```weight``` attribute.

```
    def _get_resized_lm_head(
        self, old_lm_head: torch.nn.Linear, new_num_tokens: Optional[int] = None, transposed: Optional[bool] = False
    ) -> torch.nn.Linear:
        """
        Build a resized Linear Module from a provided old Linear Module. Increasing the size will add newly initialized
        vectors at the end. Reducing the size will remove vectors from the end

        Args:
            old_lm_head (:obj:`torch.nn.Linear`):
                Old lm head liner layer to be resized.
            new_num_tokens (:obj:`int`, `optional`):
                New number of tokens in the linear matrix.

                Increasing the size will add newly initialized vectors at the end. Reducing the size will remove
                vectors from the end. If not provided or :obj:`None`, just returns a pointer to the input tokens
                :obj:`torch.nn.Linear`` module of the model without doing anything.
            transposed (:obj:`bool`, `optional`, defaults to :obj:`False`):
                Whether ``old_lm_head`` is transposed or not. If True ``old_lm_head.size()`` is ``lm_head_dim,
                vocab_size`` else ``vocab_size, lm_head_dim``.

        Return:
            :obj:`torch.nn.Linear`: Pointer to the resized Linear Module or the old Linear Module if
            :obj:`new_num_tokens` is :obj:`None`
        """
        if new_num_tokens is None:
            return old_lm_head

        old_num_tokens, old_lm_head_dim = (
            old_lm_head.weight.size() if not transposed else old_lm_head.weight.t().size()
        )

        if old_num_tokens == new_num_tokens:
            return old_lm_head

        if not isinstance(old_lm_head, nn.Linear):
            raise TypeError(
                f"Old language model head is of type {type(old_lm_head)}, which is not an instance of {nn.Linear}."
                f"You should either use a different resize function or make sure that `old_embeddings` are an instance of {nn.Linear}."
            )

        # Build new lm head
        new_lm_head_shape = (old_lm_head_dim, new_num_tokens) if not transposed else (new_num_tokens, old_lm_head_dim)
        has_new_lm_head_bias = old_lm_head.bias is not None
        new_lm_head = nn.Linear(*new_lm_head_shape, bias=has_new_lm_head_bias).to(self.device)

        # initialize new lm head (in particular added tokens)
        self._init_weights(new_lm_head)

        num_tokens_to_copy = min(old_num_tokens, new_num_tokens)

        # Copy old lm head weights to new lm head
        if not transposed:
            new_lm_head.weight.data[:num_tokens_to_copy, :] = old_lm_head.weight.data[:num_tokens_to_copy, :]
        else:
            new_lm_head.weight.data[:, :num_tokens_to_copy] = old_lm_head.weight.data[:, :num_tokens_to_copy]

        # Copy bias weights to new lm head
        if has_new_lm_head_bias:
            new_lm_head.bias.data[:num_tokens_to_copy] = old_lm_head.bias.data[:num_tokens_to_copy]

        return new_lm_head
```


Problem while initializing custom model with added tokens

Hi, I'm trying to train my model with new special token 'name', but it keeps throwing size mismatch error.
I think the problem is that my model has pretrained models inside initialization.

Model Structure
```
class MaskedLM(RobertaPreTrainedModel):
    def __init__(self, config):
        super().__init__(config=config)
        self.roberta = RobertaModel(config)
        self.lm_head = RobertaLMHead(config)
        self.refinement_num = 5
        self.mask_id = 50264
        self.init_weights()

    def forward(...):

```

After resizing my model, embedding size changed from 50265 to 50266.

```
MaskedLM(
  (roberta): RobertaModel(
    (embeddings): RobertaEmbeddings(
      (word_embeddings): Embedding(50266, 768)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (token_type_embeddings): Embedding(1, 768)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)

```

But the problem is out_features size in lm_head remains the same. (50265)

```
  (lm_head): RobertaLMHead(
    (dense): Linear(in_features=768, out_features=768, bias=True)
    (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
    (decoder): Linear(in_features=768, out_features=50265, bias=True)
  )

```

Is there any way that I can both load the pretrained weights and add one new token?

-------------

Setting Tokenizer
```
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
tokenizer.add_tokens('<name>')
```

Setting Model
```
model = MaskedLM.from_pretrained('roberta-base')
model.resize_token_embeddings(len(tokenizer))
```


Hi,
The tokenizer is slow when adding new tokens even with the Fast class:

```
from transformers import GPT2Config, TFGPT2LMHeadModel, GPT2TokenizerFast, GPT2Tokenizer

# Maybe this url for the files:
# https://huggingface.co/transformers/v3.1.0/_modules/transformers/tokenization_gpt2.html
paths = dict()
paths["tokenizer"] = "whatever/is/the/path/to/pretrained/vocab.json/merges.txt"

# They have to be sorted in reverse by length, otherwise the tokens arent
newtokens = range(0, 20000)
newtokens = list(newtokens)
newtokens.sort(reverse=True)
newtokens = ["new_" + str(x) for x in newtokens]

# loading tokenizer from the saved model path
tokenizers = dict()
tokenizers["fast"] = GPT2TokenizerFast.from_pretrained(paths["tokenizer"])
tokenizers["fast_custom"] = GPT2TokenizerFast.from_pretrained(paths["tokenizer"])
tokenizers["slow_custom"] = GPT2Tokenizer.from_pretrained(paths["tokenizer"])
tokenizers["slow"] = GPT2Tokenizer.from_pretrained(paths["tokenizer"])

tokenizer.add_special_tokens({
  "eos_token": "</s>",
  "bos_token": "<s>",
  "unk_token": "<unk>",
  "pad_token": "<pad>",
  "mask_token": "<mask>"
})

# Add new vocab
# https://huggingface.co/transformers/v2.11.0/main_classes/tokenizer.html
# https://github.com/deepset-ai/FARM/issues/157
for k in tokenizers:
    if "custom" in k:
        print(k)
        print("Vocab length before:", len(tokenizers[k].get_vocab()))
        tokenizers[k].add_tokens(newtokens)
        print("Vocab length after:", len(tokenizers[k].get_vocab()))

# creating the configurations from which the model can be made
config = GPT2Config(
  vocab_size=len(tokenizer),
  bos_token_id=tokenizer.bos_token_id,
  eos_token_id=tokenizer.eos_token_id
)

# creating the model
# https://huggingface.co/transformers/_modules/transformers/configuration_gpt2.html
model = TFGPT2LMHeadModel(config)

# Differences when tokenising the text...
text = "this is a sentence containing new_200"
for k,v in tokenizers.items():
    print(k, v.tokenize(text))
```

and then profiling the speed in jupyter:
```
for k in tokenizers:
    print(k)
    %timeit tokenizers[k].tokenize(text)
```

any ideas why this may be happening? I understand that the vocab size could increase by ~20% and that may slow things down but in this code there's a performance difference of 1000 fold in the speed. That doesn't seem right?


[mBART] one slow integration test is failing on master

The `test_enro_generate_batch` slow test is now failing on master

command
```bash
RUN_SLOW=1 pytest tests/test_modeling_mbart.py::MBartEnroIntegrationTest::test_enro_generate_batch
```

Traceback
```
tests/test_modeling_mbart.py F                                           [100%]

=================================== FAILURES ===================================
______________ MBartEnroIntegrationTest.test_enro_generate_batch _______________

self = <tests.test_modeling_mbart.MBartEnroIntegrationTest testMethod=test_enro_generate_batch>

    @slow
    def test_enro_generate_batch(self):
        batch: BatchEncoding = self.tokenizer.prepare_seq2seq_batch(self.src_text, return_tensors="pt").to(
            torch_device
        )
        translated_tokens = self.model.generate(**batch)
        decoded = self.tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)
>       assert self.tgt_text == decoded
E       AssertionError: assert ['Şeful ONU d...e de oameni.'] == ['Şeful ONU d...e de oameni.']
E         At index 1 diff: 'Secretarul General Ban Ki-moon declară că răspunsul său la intensificarea sprijinului militar al Rusiei pentru Siria este că "nu există o soluţie militară" la conflictul de aproape cinci ani şi că noi arme nu vor face decât să înrăutăţească violenţa şi mizeria pentru milioane de oameni.' != 'Secretarul General Ban Ki-moon declară că răspunsul său la intensificarea sprijinului militar al Rusiei pentru Siria este că "nu există o soluţie militară" la conflictul de aproape cinci ani şi că noi arme nu vor face decât să înrăutăţească violenţa şi mizeria a mi...
E
E         ...Full output truncated (2 lines hidden), use '-vv' to show

tests/test_modeling_mbart.py:366: AssertionError
```

cc @patrickvonplaten


[Good first issue] MPNet TensorFlow Integration tests

TensorFlow

Anything TensorFlow

Tests

Related to tests

The TensorFlow implementation of the MPNet model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_mpnet.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_mpnet.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] MobileBERT TensorFlow Integration tests

TensorFlow

Anything TensorFlow

Tests

Related to tests

The TensorFlow implementation of the MobileBERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_mobilebert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_mobilebert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.


[Good first issue] LXMERT TensorFlow Integration tests

TensorFlow

Anything TensorFlow

Tests

Related to tests

The TensorFlow implementation of the LXMERT model currently has no integration tests. This is problematic as the behavior can diverge without being noticed.

The [test_modeling_tf_lxmert.py](https://github.com/huggingface/transformers/blob/master/tests/test_modeling_tf_lxmert.py) file should be updated to include integration testing.

An example of a good modeling integration test is visible in the [test_modeling_tf_bert.py#L365-L387](https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387) file:

https://github.com/huggingface/transformers/blob/1809de5165804666ba6c6a02a9d177f6683869cc/tests/test_modeling_tf_bert.py#L365-L387

Some additional tips:
- The test must be marked as slow using the `@slow` decorator, so as to be run *daily*, and not on every commit of every branch/pull request of this repository.
- The test must be decorated with the `@require_tf` decorator so as to only be run in environments using PyTorch.
- A single test is necessary. If you feel like implementing multiple of these, then sharing the same checkpoint would be ideal so as to reduce download time.