Error "Expected input batch_size (16) to match target batch_size (1440)" in the WNUT NER example

## Environment info
- `transformers` version: 4.2.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@joeddav
@sgugger

## Information

Model I am using: DistilBert

The problem arises when using:
* [ x] the official example script

Steps to reproduce the behavior:

reproducing the NER example from

https://huggingface.co/transformers/master/custom_datasets.html

verbatim (in colab) I get
```
"Expected input batch_size (16) to match target batch_size (1440)."
```

when running trainer.train

Stack trace:
```
ValueError                                Traceback (most recent call last)
<ipython-input-12-aa1378d94d0f> in <module>()
     21 )
     22
---> 23 trainer.train()

8 frames
/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in train(self, model_path, trial)
    886                         tr_loss += self.training_step(model, inputs)
    887                 else:
--> 888                     tr_loss += self.training_step(model, inputs)
    889                 self._total_flos += self.floating_point_ops(inputs)
    890

/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   1248                 loss = self.compute_loss(model, inputs)
   1249         else:
-> 1250             loss = self.compute_loss(model, inputs)
   1251
   1252         if self.args.n_gpu > 1:

/usr/local/lib/python3.6/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs)
   1275         Subclass and override for custom behavior.
   1276         """
-> 1277         outputs = model(**inputs)
   1278         # Save past state if it exists
   1279         # TODO: this needs to be fixed and made cleaner later.

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/models/distilbert/modeling_distilbert.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
    637             else:
    638                 loss_fct = nn.CrossEntropyLoss()
--> 639                 loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
    640
    641         if not return_dict:

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    960     def forward(self, input: Tensor, target: Tensor) -> Tensor:
    961         return F.cross_entropy(input, target, weight=self.weight,
--> 962                                ignore_index=self.ignore_index, reduction=self.reduction)
    963
    964

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2466     if size_average is not None or reduce is not None:
   2467         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2468     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2469
   2470

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2260     if input.size(0) != target.size(0):
   2261         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2262                          .format(input.size(0), target.size(0)))
   2263     if dim == 2:
   2264         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (16) to match target batch_size (1440).
```


exe executable file

I want to use pyinstaller to convert my test.py into an exe executable file, but unfortunately, I failed. I checked the reason, it may be that the transformer library did not successfully convert. Could it be that my method is wrong? The command I used is pyinstaller -D test.py, however, there are no transformers in the generated library.
pyinstaller -D test.py


Tokenizer return offsets

# 🚀 Feature request

Request for the feature raised in [Issue #1263](https://github.com/huggingface/transformers/issues/1263).

Previous PRs have attempted to address this but none of them were merged - https://github.com/huggingface/transformers/pull/1274 and https://github.com/huggingface/transformers/pull/2178.

## Motivation

Refer [Issue #1263](https://github.com/huggingface/transformers/issues/1263)

## Your contribution

Please guide me on how to submit a PR.


Clarify definition of seed argument in TrainingArguments

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Clarifies the definition of the `seed` argument in `TrainingArguments` to:

* Explain what "initialisation" refers to
* How to ensure reproducibility across runs

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Link to discussion on the HF forum: https://discuss.huggingface.co/t/fixing-the-random-seed-in-the-trainer-does-not-produce-the-same-results-across-runs/3442?u=lewtun

## Who can review?

@sgugger


PPLM example - AttributeError issue

Hi all,
Thank you for great library.
Now I try to understand PPLM model (see reference https://eng.uber.com/pplm/ ), and when I try to start example from HuggingFace repository https://github.com/huggingface/transformers/tree/master/examples/research_projects/pplm (run_pplm.py) - I faced with the next issue:

```
Traceback (most recent call last):
  File "run_pplm.py", line 820, in <module>
    run_pplm_example(**vars(args))
  File "run_pplm.py", line 678, in run_pplm_example
    repetition_penalty=repetition_penalty,
  File "run_pplm.py", line 405, in full_text_generation
    repetition_penalty=repetition_penalty,
  File "run_pplm.py", line 511, in generate_text_pplm
    device=device,
  File "run_pplm.py", line 115, in perturb_past
    grad_accumulator = [(np.zeros(p.shape).astype("float32")) for p in past]
  File "run_pplm.py", line 115, in <listcomp>
    grad_accumulator = [(np.zeros(p.shape).astype("float32")) for p in past]
AttributeError: 'tuple' object has no attribute 'shape'

```
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: 4.3.0.dev0
- Python version: 3.8
- PyTorch version (GPU?): 1.7.1

I started this example on my own laptop and in Google Colab environment

## To reproduce

Steps to reproduce the behavior:

1. Follow to https://github.com/huggingface/transformers/tree/master/examples/research_projects/pplm and do Setup steps
2. Do command: `python run_pplm.py -B military --cond_text "The potato" --length 50 --gamma 1.5 --num_iterations 3 --num_samples 10 --stepsize 0.03 --window_length 5 --kl_scale 0.01 --gm_scale 0.99 --colorama --sample
`

## Expected behavior
This script should work without an error :)


Missing model license information

Hello,

A significant number of models uploaded to the model hub do not contain any license information. I wanted to check if your conditions set a default license under which models are uploaded when not specified?

In order to have a missing license added when it is missing, could you please advise on the standard way to proceed?
- Should an issue be created and tagging the author of the model asking for the license?
- Should I contact the author directly without raising an issue?

The community now has contributed a large number of very useful models, but more transparency regarding licensing (or default license) would be great.

Below are a few models I would be very interested in getting license information for, but a more general approach would be very beneficial:
- https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1 (@patil-suraj )
- https://huggingface.co/mrm8488/longformer-base-4096-finetuned-squadv2 (@mrm8488 )
- https://huggingface.co/mrm8488/mobilebert-uncased-finetuned-squadv2 (@mrm8488 )
- https://huggingface.co/mrm8488/mobilebert-finetuned-ner (@mrm8488 )
- https://huggingface.co/mrm8488/mobilebert-finetuned-pos (@mrm8488 )

Thank you,


run_seq2seq.py doesn't work after enabling sortish sampler

It gives an error saying **AttributeError: 'Dataset' object has no attribute 'make_sortish_sampler'**. Seems like something is wrong with the pipeline or versions. I install both transformers and datasets from the sources.

[Exact line](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_seq2seq.py#L46)
```

    self.train_dataset.make_sortish_sampler(
AttributeError: 'Dataset' object has no attribute 'make_sortish_sampler'

```

## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Linux-5.8.18-050818-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@LysandreJik  @patrickvonplaten


Does Sortish Sampler work with multiple GPUs in seq2seq?

I am referring to the training script of run_seq2seq.py.  I am exactly referring to  [this line in seq2seq_trainer.py](https://github.com/huggingface/transformers/blob/1420b5ff675ccdc3296c6776b339a08a22d2e941/src/transformers/trainer_seq2seq.py#L48). So when should I enable **distributed** parameters and how should I do it?


[doc] nested markup is invalid in rst

Apparently nested markup in RST is invalid: https://docutils.sourceforge.io/FAQ.html#is-nested-inline-markup-possible

So currently this line doesn't get rendered properly, leaving inner markdown unrendered, resulting in:
```
You can create a model repo directly from `the /new page on the website <https://huggingface.co/new>`__.
```

This PR removes the bold markdown which fixes the link.

@sgugger


[t5 tokenizer] add info logs

This PR (was modified from the original):
- adds info logs that correlated to saved tokenizer files on  `tokenizer.save_pretrained()`

--------------------------
original PR note

This PR
- adds code to save t5 fast tokenizer `tokenizer.json` file on `tokenizer.save_pretrained()`
- adds info logs that correlated to saved tokenizer files on  `tokenizer.save_pretrained()`

Context:
- I needed to create a new t5 smallish model and the created model won't work w/o `tokenizer.json`.
- Also as I was debugging why I was missing that file, I enabled logging and saw that we were getting logs for every saved file, but tokenizer files, so this PR fixes that, so it's consistent and helps one to see if something is missing.

Here is an example:
```
TRANSFORMERS_VERBOSITY=info PYTHONPATH=/hf/transformers-master/src python t5-make-very-small-model.py
[....]
Configuration saved in t5-very-small-random/config.json
Model weights saved in t5-very-small-random/pytorch_model.bin
Configuration saved in t5-very-small-random/config.json
tokenizer config file saved in t5-very-small-random/tokenizer_config.json
Special tokens file saved in t5-very-small-random/special_tokens_map.json
Copy vocab file to t5-very-small-random/spiece.model
tokenizer config file saved in t5-very-small-random/tokenizer_config.json
Special tokens file saved in t5-very-small-random/special_tokens_map.json
Copy vocab file to t5-very-small-random/spiece.model
Copy tokenizer file to t5-very-small-random/tokenizer.json
```

I'm not sure why I needed to save both:
```
tokenizer.save_pretrained(mname_very_small)
tokenizer_fast.save_pretrained(mname_very_small)
```
note `tokenization_t5.py` doesn't have it! both t5 tokenizer files:
```
VOCAB_FILES_NAMES = {"vocab_file": "spiece.model"}
VOCAB_FILES_NAMES = {"vocab_file": "spiece.model", "tokenizer_file": "tokenizer.json"}
```

As I flagged on slack `https://huggingface.co/sshleifer/t5-tinier-random` fails to be used since it's missing this fast `tokenizer.json` file from the s3 set of files,
```
Traceback (most recent call last):
 File "./finetune_trainer.py", line 373, in <module>
   main()
 File "./finetune_trainer.py", line 205, in main
   tokenizer = AutoTokenizer.from_pretrained(
 File "/home/stas/hf/transformers/src/transformers/models/auto/tokenization_auto.py", line 385, in from_pretrained
   return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
 File "/home/stas/hf/transformers/src/transformers/tokenization_utils_base.py", line 1768, in from_pretrained
   return cls._from_pretrained(
 File "/home/stas/hf/transformers/src/transformers/tokenization_utils_base.py", line 1841, in _from_pretrained
   tokenizer = cls(*init_inputs, **init_kwargs)
 File "/home/stas/hf/transformers/src/transformers/models/t5/tokenization_t5_fast.py", line 139, in __init__
   super().__init__(
 File "/home/stas/hf/transformers/src/transformers/tokenization_utils_fast.py", line 86, in __init__
   fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
```

it could be a symptom for another problem in our code.

@LysandreJik, @sgugger


[wandb] restore WANDB_DISABLED=true to disable wandb

This PR
* extends `ENV_VARS_TRUE_VALUES` with "true"
* restores `WANDB_DISABLED=true` to disable wandb
* documents this exact setting
* syncs trainer_tf with the same solution.

Context: we are still dealing with https://github.com/huggingface/transformers/issues/9623 where wandb fails no matter if you have it installed or not.

It looks like due to https://github.com/huggingface/transformers/issues/9699 a few days ago this behavior was changed to be one of `ENV_VARS_TRUE_VALUES = {"1", "ON", "YES"}`. And it's not documented anywhere.

This PR tries to restore the original behavior where any value of `WANDB_DISABLED` should disable wandb.

And wandb integration is broken, that's why we need a way disable it - it's so annoying when trying to develop and wandb keeps on breaking things whether it's installed or not. See: https://github.com/huggingface/transformers/issues/9623

Alternatively, instead of the proposed change in this PR, let's document this API that it has to be on of `{"1", "ON", "YES"}``, so that it doesn't change from day to day.

@sgugger


TFGPT2LMHeadModel unknown location

I have been playing around with tensorflow (CPU), and some language model'ing - and it have been a blast so far - everything working great.
But after watching my old CPU slowly getting killed from all the model-training - i decided it was time to  finally get some use out of my RTX 2080. I have been following the guide from [washinton university](https://github.com/jeffheaton/t81_558_deep_learning/blob/master/install/tensorflow-install-jul-2020.ipynb):.  Pretty quickly i got tensorflow-gpu running, ran it on some light grade-prediction and stuff like that.

But when i got to running GPT2 language model, i ran into some minor problems. I start by tokenizing the data:

    from tokenizers.models import BPE
    from tokenizers import Tokenizer
    from tokenizers.decoders import ByteLevel as ByteLevelDecoder
    from tokenizers.normalizers import NFKC, Sequence
    from tokenizers.pre_tokenizers import ByteLevel
    from tokenizers.trainers import BpeTrainer

    class BPE_token(object):
    def __init__(self):
        self.tokenizer = Tokenizer(BPE())
        self.tokenizer.normalizer = Sequence([
            NFKC()
        ])
        self.tokenizer.pre_tokenizer = ByteLevel()
        self.tokenizer.decoder = ByteLevelDecoder()

    def bpe_train(self, paths):
        trainer = BpeTrainer(vocab_size=50000, show_progress=True, inital_alphabet=ByteLevel.alphabet(),         special_tokens=[
            "<s>",
            "<pad>",
            "</s>",
            "<unk>",
            "<mask>"
        ])
        self.tokenizer.train(trainer, paths)

    def save_tokenizer(self, location, prefix=None):
        if not os.path.exists(location):
            os.makedirs(location)
        self.tokenizer.model.save(location, prefix)

    # ////////// TOKENIZE DATA ////////////
    from pathlib import Pa th
    import os# the folder 'text' contains all the files
    paths = [str(x) for x in Path("./da_corpus/").glob("**/*.txt")]
    tokenizer = BPE_token()# train the tokenizer model
    tokenizer.bpe_train(paths)# saving the tokenized data in our specified folder
    save_path = 'tokenized_data'
    tokenizer.save_tokenizer(save_path)

Code above works perfectly and tokenizes the data - just like with tensorflow (CPU). After having my data tokenized i start to train my model - but before it even gets start, i get the following ImportError:

    from transformers import GPT2Config, TFGPT2LMHeadModel, GPT2Tokenizer # loading tokenizer from the   saved model path
    ImportError: cannot import name 'TFGPT2LMHeadModel' from 'transformers' (unknown location)

Transformers package seems to be installed correctly in the site-packages lib, and i seem to be able to use the other transformers - but not **TFGPT2LMHeadModel**
I have read everything on google and [hugging.co](https://huggingface.co/transformers/) - tried different versions of tensorflow-gpu, transformers, tokenizers and alot of other packages -  -  sadly nothing helps.

**Packages:**
 - Python, 3.7.1
 - Tensorflow 2.1.0
 - Tensorflow-gpu 2.1.0
 - Tensorflow-base 2.1.0
 - Tensorflow-estimator 2.1.0
 - Transformers 4.2.2
 - Tokenizers 0.9.4
 - cudnn 7.6.5
 - cudatoolkit 10.1.243

 @LysandreJik


Hi,

I am using the library to pretrain my model of choice. I am now interested in setting up an encoder-decoder architecture with my pretrained models, and  the "combiners" seems quite a straightforward way to do that.
Unfortunately, I am getting an import error on both "PreTrainedEncoderDecoder" and "Model2Model"
What am I missing ?
Thanks
Gianfilippo

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0.dev0
- Platform: Linux-3.10.0-1062.33.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: NA
- Using distributed or parallel set-up in script?: NA

## To reproduce
Steps to reproduce the behavior:
1.python -c "from transformers import PreTrainedEncoderDecoder"

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
ImportError: cannot import name 'PreTrainedEncoderDecoder' from 'transformers' (unknown location)

## Expected behavior
<!-- A clear and concise description of what you would expect to happen. -->
no error


rfc: new benchmark tool

Benchmarks

Issues related to Memory regressions in tests and scripts

WIP

Label your PR/Issue with WIP for some long outstanding Issues/PRs that are work in progress

This issue is to collect notes and ideas on creating a new benchmarking tool.

This is not about the other speed/memory regression project we have been discussing elsewhere.

This is about integration and various comparisons that we need to run in order to give users the best advice on how to deploy transformers in the most efficient way.

Please share the comments ideas/suggestions/concerns/needs, and I will compile them here.

- important: not part of examples - the goal is performance and integration tooling and not user-facing - totally different needs and priorities
- the cmd line has to continue working the same months later - so that old benchmarks could be re-run - ok to change interface with back-compat option so that the old benchmarks can be still re-validated and compared to
- ideally work with any transformers model - a single tool to rule them all
- minimal amount of arguments - just the important ones
- ability to generate markdown table entries directly and json files that contain not just the outcome but also the key variables that are being tested -
- the report to include critical hardware/software params as well in a compact form and allow these to be merged from multiple recordings - i.e. if the hw/sw are the same - they can be merged into a single report. will need to figure out how to record hardware nuances
    *  e.g. the same DDP test with 2 gpus connected w/ NVLink gives dramatically different results than the same 2 gpus w/o NVLink.
    * not sure how to record CPU-capacity/ free RAM, etc., since all these impact the outcome
- crucial to be able to truncate the dataset


Seeking clarification on T5  prefix for summarization

In the paper, I see the the prefix for summarization is "TL;DR:" . If I look into the model [config.json](https://huggingface.co/t5-base/blob/main/config.json)  of T5-Base, I see it is "summarization:".

"task_specific_params": {
    "summarization": {
      "early_stopping": true,
      "length_penalty": 2.0,
      "max_length": 200,
      "min_length": 30,
      "no_repeat_ngram_size": 3,
      "num_beams": 4,
      "prefix": "summarize: "
    },
  .....
 If I want to finetune Huggingface T5 for summarization, which prefix should I use?

Thank you


Remove Token from Vocab?

Is there a way I can remove a token from vocab.json?


Restore TF embeddings and attention layers to their previous version

# What does this PR do?

This PR restores the attention layers and the embeddings as it was in v4.1, even though the embeddings got few improvements compared to their original version to keep XLA compliancy. The reason is because we realized that some used operators were not compatible with some NN SDKs such as the one from Qualcomm or ONNX.


m2m_100

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[Quick poll] Give your opinion on the future of 🤗 transformers: 40k edition!

Discussion

Discussion on a topic (keep it focused or open a new issue though)

Thanks to all of you, Transformers just passed 40k :star2:  this week!

Our libraries have always been about the community and we need your input to define the direction of the next 40k stars.

If you have a couple of minutes and want to participate in shaping the future of the library, please share your thoughts: https://forms.gle/FackvXzWJBWQz2WY8

(please reply in the above feedback form rather than to this thread)

Thank you all on behalf of the HuggingFace team! 🤗


Fit chinese wwm to new datasets

Sorry for my later update.
I make my code(especially in chinese mlm_wwm) fit the newest code.
Here are the changes:
1. add `chinese_ref` key to avoid miss ref inf.
2. fix the type bug in `data_collator.py`
3. re-add `run_chinese_ref.py` cause it could run with the newest version code (4.2.2).
4. update readme


Conversion of BPE tokenizer for Marian models

Hello,

I was searching for the pt-en model of Marian and noticed that it has not been converted for the huggingface library apparently because it uses a BPE tokenizer. Is it possible to convert BPE-based models to be used in huggingface somehow?

Thank you


Finetune_Trainer Question

Hi, I'm new to HuggingFace. I want to fine-tune a BARTForConditionalGeneration model by finetunr_trainer.py for the translation task on google colab, but I didn't figure out how to use the script to fine-tune the model. Could anyone help me to show a quick example? Thanks


Exporting model to onnx increases the model size

Hi, I'm trying to convert some models as mentioned below to onnx as follows:
ktrapeznikov/albert-xlarge-v2-squad-v2
albert-xlarge-v1
albert-xlarge-v2

The common issue with exporting all these models is that I get an exception that the protobuf size increases to more than 2gb while all these are less than 800mb. When I use the use_external_data_format=True flag, the exported model files(network layers as I searched in other issues) sum up to in gbs of size. For eg, the model ktrapeznikov/albert-xlarge-v2-squad-v2 is sized to 210mb but when I convert it to onnx using use_external_data_format flag, the model size sums up to 4gb.

## Code example
```
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
import torch

model_name = "ktrapeznikov/albert-xlarge-v2-squad-v2"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForQuestionAnswering.from_pretrained(model_name)
model.eval()
question = "what is google specialization"
text = "Google LLC is an American multinational technology company that specializes in Internet-related services and products, which include online advertising technologies, a search engine, cloud computing, software, and hardware."
encoding = tokenizer.encode_plus(question, text)
input_ids, attention_mask, token_type_ids = encoding["input_ids"],encoding["attention_mask"], encoding["token_type_ids"]

input_ids = torch.tensor([input_ids])
attention_mask = torch.tensor([attention_mask])
token_type_ids = torch.tensor([token_type_ids])

torch.onnx.export(
    model,
    (input_ids,attention_mask, token_type_ids),
    f"{model_name}.onnx",
    input_names = ['input_ids','attention_mask', 'token_type_ids'],
    output_names = ['qa_outputs'],
    opset_version=12, ##opset has to be set to 12
    do_constant_folding=True,
    use_external_data_format=True,
    dynamic_axes = {
        'input_ids' : {0: 'batch', 1: 'sequence'},
        'attention_mask' : {0: 'batch', 1: 'sequence'},
        'token_type_ids' : {0: 'batch', 1: 'sequence'},
        'qa_outputs': {0: 'batch'}
    }
)
```
![image](https://user-images.githubusercontent.com/55797177/106251673-4549f780-623b-11eb-8735-9bf90f18bf4e.png)
![image](https://user-images.githubusercontent.com/55797177/106251758-63aff300-623b-11eb-81e7-defe45429eaf.png)

## System Info
PyTorch version: 1.7.0+cu101
CUDA used to build PyTorch: 10.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.12.0

Python version: 3.6 (64-bit runtime)
Is CUDA available: False
CUDA runtime version: 10.1.243
GPU models and configuration: Could not collect
Nvidia driver version: Could not collect
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.19.5
[pip3] torch==1.7.0+cu101
[pip3] torchsummary==1.5.1
[pip3] torchtext==0.3.1
[pip3] torchvision==0.8.1+cu101

Any help would be much appreciated.
Thanks in advance!


examples/seq2seq , where can I find the  definition for the sortish_sampler argument?


Some weights of {} were not initialized from the model checkpoint

I keep failing to load model checkpoint.
I built a model inheriting PreTrainedModel and have roberta inside initialization.
Training this model with trainer works fine, but when I try to load the checkpoint using ```from_pretrained```, it keeps failing to load the checkpoint. Can someone help me out? Thanks

Structure of my model
```
class MaskClassifier(PreTrainedModel):
    def __init__(self, config, path):
        super().__init__(config=config)
        self.roberta = RobertaModel.from_pretrained(path)
        self.max_mask = 10
        self.hidden_size = RobertaConfig().hidden_size
        self.linear1 = torch.nn.Linear(2 * self.hidden_size, self.hidden_size)
        self.linear2 = torch.nn.Linear(self.hidden_size, self.max_mask + 1)
        self.softmax = torch.nn.Softmax(dim=1)

    def forward(self, input_ids, attention_mask, token_type_ids, labels=None):
       ...
        # Feed input to RoBERTa
```

Initialize before training
```
config = RobertaConfig()
config.max_position_embeddings = 514
config.type_vocab_size = 1
config.vocab_size = 50265

model = MaskClassifier(config=config, path='roberta-base')
```

Saving after training
```trainer.save_model('./slogan_pretrained')```

Loading the checkpoint
```
config = RobertaConfig()
config.max_position_embeddings = 514
config.type_vocab_size = 1
config.vocab_size = 50265

 model = MaskClassifier.from_pretrained(path, config=config, path='roberta-base')
```

I found similair issue(https://github.com/huggingface/transformers/issues/2886), but I don't know exactly how I should override the function ```from_pretrained``` and even I tried overriding this functions, it still can't load the checkpoint.

Error Message

> Some weights of MaskClassifier were not initialized from the model checkpoint at /home/yeoun/slogans/slogan_pretrained and are newly initialized: ['.roberta.embeddings.position_ids', '.roberta.embeddings.word_embeddings.weight', '.roberta.embeddings.position_embeddings.weight', '.roberta.embeddings.token_type_embeddings.weight', '.roberta.embeddings.LayerNorm.weight', '.roberta.embeddings.LayerNorm.bias', '.roberta.encoder.layer.0.attention.self.query.weight', '.roberta.encoder.layer.0.attention.self.query.bias', '.roberta.encoder.layer.0.attention.self.key.weight', '.roberta.encoder.layer.0.attention.self.key.bias', '.roberta.encoder.layer.0.attention.self.value.weight', ...


DeBERTa pretraining using MLM: model gradients become NAN

## Environment info

- `transformers` version: 4.3.0.dev0
- Platform: Ubuntu
- Python version: 3.6.12
- PyTorch version : 1.7.1
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: Y, using 8 GPU machine.

### Who can help
@BigBird01 @NielsRogge

Models:
DeBERTa Base

## Information

I am using DeBERTa base model and training it with Masked Language Modeling task using single file from wikipedia text
dataset. For the first step the loss is around 11 and after backward pass, gradients become nan and gradient norm goes to infinity.
I reduced learning rate from 1e-4 to 5e-10, still the issue persists. Batch size per GPU is 32 and with 8 GPUs, total batchsize becomes 256. Configured hyperparameters according to paper are as below.

* Number of Layers: 12
* Hidden size: 768
* FNN inner hidden size: 3072
* Attention Heads: 12
* Attention Head size: 64
* Dropout: 0.1
* Warmup Steps: 10k
* Learning Rates: 1e-4
* Batch Size: 256
* Weight Decay: 0.01
* Max Steps: 1M
* Learning Rate Decay: Linear
* Adam ε: 1e-6
* Adam β1: 0.9
* Adam β2: 0.999
* Gradient Clipping: 1.0

## To reproduce

Steps to reproduce the behavior:

```python

from transformers import (
    DebertaConfig,
    DebertaTokenizer,
    DebertaForMaskedLM,
    LineByLineTextDataset,
    DataCollatorForLanguageModeling,
    Trainer,
    TrainingArguments
)

tokenizer = DebertaTokenizer.from_pretrained('microsoft/deberta-base')

train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="/data/wikidemo/wiki_01",
    block_size=128,
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)
config = DebertaConfig()

model = DebertaForMaskedLM(config=config)

training_args = TrainingArguments(
    output_dir="./deberta",
    overwrite_output_dir=True,

    num_train_epochs=1000,
    per_gpu_train_batch_size=2,
    learning_rate=5e-10,
    weight_decay=0.01,
    adam_beta1=0.9,
    adam_beta2=0.999,
    adam_epsilon=1e06,
    max_grad_norm=1.0,
    save_steps=10_000,
    save_total_limit=2,
    logging_first_step=False,
    logging_steps=1,
    max_steps=10000,
    gradient_accumulation_steps=10,

)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
)

print("Starting training")
trainer.train()
```


[trainer] [deepspeed] refactor deepspeed setup devices

Following the discussion at https://github.com/huggingface/transformers/pull/9798#pullrequestreview-578822445 as we now have multiple integrations with complex unique setups, @sgugger and I agreed that it's better to have a small duplication of a few lines of code but to make it much easier to understand what goes on for a specific integration, so rather than further refactoring the recently added sage branch, this PR creates a dedicated branch for DeepSpeed and thus simplifies the general case when straight DDP is used.

There is no functionality change - just a small code reshuffle.

@sgugger


[seq2seq] correctly handle mt5

This PR fixes `seq2seq/utils.py` to handle `mt5` like it does `t5`.

Ideally there should be a test, which would require creating a tiny model for mt5, but I'm being told this code is going away anyway, so there is no point investing energy into it.

Fixes: https://github.com/huggingface/transformers/issues/9865

@patil-suraj, @sgugger


[DOCS] curl links go to 404 not found in NER tutorial

Hey, in [NER tutorial](https://huggingface.co/transformers/v2.2.0/examples.html#named-entity-recognition) the curl commands seem outdated

```
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-train.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-dev.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp
curl -L 'https://sites.google.com/site/germeval2014ner/data/NER-de-test.tsv?attredirects=0&d=1' \
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp
```

Can you please share new curl requests?


Fix head masking for TFT5 models

* This PR fixes head masking in TFT5 models (#9859)
* This PR further fixes the name of an error message variable from `__HEAD_MASK_WARNING_MSG` to `_HEAD_MASK_WARNING_MSG` as the former one was not working properly and raised error (double underscore made troubles)

<hr>

Fixes: #9859

Reviewers: @jplu


When on sagemaker use their env variables for saves

# What does this PR do?

When on SageMaker, the content of the env variable "SM_OUTPUT_DATA_DIR" should be used to save training artifacts (such as our checkpoint) so make it overwrite the `output_dir` (and make that argument optional so it doesn't need to be passed for sagemaker training).
Then the final model will be easy to deploy if it's also saved to the content of the env variable "SM_MODEL_DIR" so adding that as well.


Clarify use of unk_token in slow tokenizers' docstrings

# What does this PR do?

Currently, the docstrings for slow tokenizers' `tokenize()` method claim that unknown tokens will be left in place, in contrast to the fast tokenizers' behavior.  In reality, both convert unknown tokens to `unk_token`.

Fixes #9714

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@LysandreJik


pin_memory -> dataloader_pin_memory

Ref: https://github.com/huggingface/transformers/pull/9857#issuecomment-769256215

This PR adds a new argument `dataloader_pin_memory` to `TrainingArguments`. You can use this to pin memory in `DataLoader`.


Strange hyperparameter warning

## Environment info

- `transformers` version: Master Branch
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.7 (YES)
- Tensorflow version (GPU?):
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO

### Who can help
@amogkam

## Information

Model I am using (Bert, XLNet ...):

BART

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Go to https://docs.ray.io/en/master/tune/examples/pbt_transformers.html and copy that code.
2. Execute that code.
3. Wait and when some iterations have been made, you'll see that a constant warning persists:

```{python}
2021-01-28 17:33:56,863	WARNING trial_runner.py:420 -- Trial Runner checkpointing failed: Checkpoint must not be in-memory.
```
Although it seems a ray-related problem, from reading https://github.com/huggingface/transformers/pull/6747 I have arrived to the conclusion that maybe as the checkpointing integration has been removed from Transformers, PBT is no longer working.
When I look into the logs, I see that effectively no perturbation is being made, and it should because perturbation_interval is set to

## Expected behavior

It's expected that if I set perturbation_interval to 1, perturbations are made every 1 training iteration, but PBT is not doing any perturbation at all and I think it's because of some problem in the integration for checkpointing between Transformers and Ray Tune.


on_log event should occur *after* the current log is written


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Linux-3.10.107-1-tlinux2_kvm_guest-0049-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [1 ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. open https://github.com/agemagician/ProtTrans/blob/master/Embedding/PyTorch/Basic/ProtAlbert.ipynb
2. when run the code 'tokenizer = AutoTokenizer.from_pretrained("Rostlab/prot_albert", do_lower_case=False )'
3. report errors as the follow:
Downloading: 100%|█████████████████████████████████████████████████████████████████| 505/505 [00:00<00:00, 516kB/s]
Downloading: 100%|██████████████████████████████████████████████████████████████| 238k/238k [00:03<00:00, 77.0kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/models/auto/tokenization_auto.py", line 385, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1768, in from_pretrained
    return cls._from_pretrained(
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 1841, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/models/albert/tokenization_albert_fast.py", line 136, in __init__
    super().__init__(
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 89, in __init__
    fast_tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/convert_slow_tokenizer.py", line 659, in convert_slow_tokenizer
    return converter_class(transformer_tokenizer).converted()
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/convert_slow_tokenizer.py", line 349, in converted
    tokenizer = self.tokenizer(self.proto)
  File "/home/anaconda3/envs/prottrans/lib/python3.8/site-packages/transformers/convert_slow_tokenizer.py", line 335, in tokenizer
    raise Exception(
Exception: You're trying to run a `Unigram` model but you're file was trained with a different algorithm

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


IndexError when finetuning barthez on summarization

## Environment info
- `transformers` version: 4.3.0.dev0
- Platform: Linux-4.4.0-197-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik  @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): BARThez

## To reproduce

Steps to reproduce the behavior:

```
python finetune_trainer.py --learning_rate 3e-5 --fp16 --evaluation_strategy steps --predict_with_generate --model_name_or_path moussaKam/barthez --data_dir xsum --do_train --do_eval --output_dir welcome_back --per_device_train_batch_size 4 --task summarization --max_target_length 50 --overwrite_output_dir --eval_steps 50 --n_val 20
```

```
Traceback (most recent call last):
  File "finetune_trainer.py", line 373, in <module>
    main()
  File "finetune_trainer.py", line 303, in main
    train_result = trainer.train(
  File "/datadisks/datadisk1/transformers/src/transformers/trainer.py", line 942, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch)
  File "/datadisks/datadisk1/transformers/src/transformers/trainer.py", line 1017, in _maybe_log_save_evaluate
    metrics = self.evaluate()
  File "/datadisks/datadisk1/transformers/src/transformers/trainer_seq2seq.py", line 96, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/datadisks/datadisk1/transformers/src/transformers/trainer.py", line 1458, in evaluate
    output = self.prediction_loop(
  File "/datadisks/datadisk1/transformers/src/transformers/trainer.py", line 1617, in prediction_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=preds, label_ids=label_ids))
  File "/datadisks/datadisk1/transformers/examples/seq2seq/utils.py", line 92, in summarization_metrics
    pred_str, label_str = decode_pred(pred)
  File "/datadisks/datadisk1/transformers/examples/seq2seq/utils.py", line 86, in decode_pred
    label_str = tokenizer.batch_decode(pred.label_ids, skip_special_tokens=True)
  File "/datadisks/datadisk1/transformers/src/transformers/tokenization_utils_base.py", line 3070, in batch_decode
    return [
  File "/datadisks/datadisk1/transformers/src/transformers/tokenization_utils_base.py", line 3071, in <listcomp>
    self.decode(
  File "/datadisks/datadisk1/transformers/src/transformers/tokenization_utils_base.py", line 3109, in decode
    return self._decode(
  File "/datadisks/datadisk1/transformers/src/transformers/tokenization_utils.py", line 711, in _decode
    filtered_tokens = self.convert_ids_to_tokens(token_ids, skip_special_tokens=skip_special_tokens)
  File "/datadisks/datadisk1/transformers/src/transformers/tokenization_utils.py", line 695, in convert_ids_to_tokens
    tokens.append(self._convert_id_to_token(index))
  File "/datadisks/datadisk1/transformers/src/transformers/models/barthez/tokenization_barthez.py", line 237, in _convert_id_to_token
    return self.sp_model.IdToPiece(index)
  File "/home/dascim/anaconda3/envs/transformers/lib/python3.8/site-packages/sentencepiece/__init__.py", line 501, in _batched_func
    return _func(self, arg)
  File "/home/dascim/anaconda3/envs/transformers/lib/python3.8/site-packages/sentencepiece/__init__.py", line 494, in _func
    raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.
```

## Expected behavior

For some reason the tokenizer is trying to decode some -100 id.


Added do_lower_case parameters for tokenizer in mlm training.

Added do_lower_case while training mlm. Useful for training cased BER, for instance.


Remove submodule

Removes the `datasets` submodule that was introduced in https://github.com/huggingface/transformers/pull/9825.


where is position_embedding_type used

When I was using pytorch Electra Model, I read its source code but I didn't find where position_embedding_type is used.
So did I miss something?


Whole word mask in run_mlm_wwm.py

I find that `run_mlm_wwm.py` uses the whole word mask class `DataCollatorForWholeWordMask`.
But in this class `_whole_word_mask` function, we recognize if a token is the beginning of a word by this way:

```
cand_indexes = []
for (i, token) in enumerate(input_tokens):
    if token == "[CLS]" or token == "[SEP]":
        continue

    if len(cand_indexes) >= 1 and token.startswith("##"):
        cand_indexes[-1].append(i)
    else:
        cand_indexes.append([i])
```

I also notice that `run_mlm_wwm.py` is also used for Roberta per-training in the examples. However, the tokenizer for Roberta doesn't contain tokens like `[CLS]` and `[SEP]`. Also, the subwords do not start with `##`.

How can this code handle language models use Roberta-like tokenizer?

Thanks!


[trainer] seq2seq doesn't handle mt5 correctly

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.2.2
- Platform: Linux-5.4.0-58-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <yes>
- Using distributed or parallel set-up in script?: <yes>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@stas00,@patrickvonplaten, @patil-suraj

## Information

Model I am using (MT5-xl,MT5-large):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (official example scripts task)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. The script I used is `exmaples/seq2seq/finetune_trainer.py`, which was originally used to reproduce the training of T5-3b on single 3090. All processes are the same as [#8771](https://github.com/huggingface/transformers/issues/8771#issuecomment-759176685) and it can reproduce the training of T5-3b(whether single card or 2/4 cards).
2. Here is the problem, when I try to train MT5-xl, `--freeze_embeds` seems to bring bugs. I used 4*3090, My script is
```
export BS=1; PYTHONPATH=../../src; USE_TF=0;
/usr/bin/time -v deepspeed --num_gpus=4 ./finetune_trainer.py --model_name_or_path /<my_model_dir>/models/mt5/xl/v0 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size 1 --per_device_train_batch_size 1 --predict_with_generate --eval_steps 25000 --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16
```
Here is my report:

```
[2021-01-27 14:59:52,982] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-01-27 14:59:57,024] [INFO] [runner.py:358:main] cmd = /<my_dir>/miniconda3/envs/nlp/bin/python -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMSwgMiwgM119 --master_addr=127.0.0.1 --master_port=29500 ./finetune_trainer.py --model_name_or_path /<my_model_dir>/models/mt5/xl/v0 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size 1 --per_device_train_batch_size 1 --predict_with_generate --eval_steps 25000 --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16
[2021-01-27 14:59:57,793] [INFO] [launch.py:78:main] WORLD INFO DICT: {'localhost': [0, 1, 2, 3]}
[2021-01-27 14:59:57,793] [INFO] [launch.py:87:main] nnodes=1, num_local_procs=4, node_rank=0
[2021-01-27 14:59:57,793] [INFO] [launch.py:99:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1, 2, 3]})
[2021-01-27 14:59:57,793] [INFO] [launch.py:100:main] dist_world_size=4
[2021-01-27 14:59:57,793] [INFO] [launch.py:103:main] Setting CUDA_VISIBLE_DEVICES=0,1,2,3
[2021-01-27 15:00:01,106] [INFO] [distributed.py:40:init_distributed] Initializing torch distributed with backend: nccl
[2021-01-27 15:00:01,340] [INFO] [distributed.py:40:init_distributed] Initializing torch distributed with backend: nccl
[2021-01-27 15:00:01,672] [INFO] [distributed.py:40:init_distributed] Initializing torch distributed with backend: nccl
[2021-01-27 15:00:01,870] [INFO] [distributed.py:40:init_distributed] Initializing torch distributed with backend: nccl
01/27/2021 15:00:05 - WARNING - __main__ -   Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, 16-bits training: True
01/27/2021 15:00:05 - WARNING - __main__ -   Process rank: 2, device: cuda:2, n_gpu: 1, distributed training: True, 16-bits training: True
01/27/2021 15:00:05 - WARNING - __main__ -   Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, 16-bits training: True
01/27/2021 15:00:05 - INFO - __main__ -   Training/evaluation parameters Seq2SeqTrainingArguments(output_dir='output_dir', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=1, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-06, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, lr_scheduler_type=<SchedulerType.LINEAR: 'linear'>, warmup_steps=5, logging_dir='runs/Jan27_15-00-01_user-SYS-4029GP-TRT', logging_first_step=True, logging_steps=1000, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=True, fp16_opt_level='O1', fp16_backend='auto', local_rank=0, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=25000, dataloader_num_workers=0, past_index=-1, run_name='output_dir', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None, ignore_data_skip=False, sharded_ddp=False, deepspeed='ds_config.json', label_smoothing_factor=0.1, adafactor=False, sortish_sampler=True, predict_with_generate=True)
01/27/2021 15:00:05 - WARNING - __main__ -   Process rank: 3, device: cuda:3, n_gpu: 1, distributed training: True, 16-bits training: True
[INFO|configuration_utils.py:443] 2021-01-27 15:00:05,352 >> loading configuration file /<my_model_dir>/models/mt5/xl/v0/config.json
[INFO|configuration_utils.py:481] 2021-01-27 15:00:05,353 >> Model config MT5Config {
  "_name_or_path": "/home/patrick/t5/mt5-xl",
  "architectures": [
    "T5ForConditionalGeneration"
  ],
  "d_ff": 5120,
  "d_kv": 64,
  "d_model": 2048,
  "decoder_start_token_id": 0,
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "gated-gelu",
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "layer_norm_epsilon": 1e-06,
  "model_type": "mt5",
  "num_decoder_layers": 24,
  "num_heads": 32,
  "num_layers": 24,
  "output_past": true,
  "pad_token_id": 0,
  "relative_attention_num_buckets": 32,
  "tie_word_embeddings": false,
  "tokenizer_class": "T5Tokenizer",
  "transformers_version": "4.2.1",
  "use_cache": true,
  "vocab_size": 250112
}

[INFO|configuration_utils.py:443] 2021-01-27 15:00:05,353 >> loading configuration file /<my_model_dir>/models/mt5/xl/v0/config.json
[INFO|configuration_utils.py:481] 2021-01-27 15:00:05,354 >> Model config MT5Config {
  "_name_or_path": "/home/patrick/t5/mt5-xl",
  "architectures": [
    "T5ForConditionalGeneration"
  ],
  "d_ff": 5120,
  "d_kv": 64,
  "d_model": 2048,
  "decoder_start_token_id": 0,
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "gated-gelu",
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "layer_norm_epsilon": 1e-06,
  "model_type": "mt5",
  "num_decoder_layers": 24,
  "num_heads": 32,
  "num_layers": 24,
  "output_past": true,
  "pad_token_id": 0,
  "relative_attention_num_buckets": 32,
  "tie_word_embeddings": false,
  "tokenizer_class": "T5Tokenizer",
  "transformers_version": "4.2.1",
  "use_cache": true,
  "vocab_size": 250112
}

[INFO|tokenization_utils_base.py:1685] 2021-01-27 15:00:05,354 >> Model name '/<my_model_dir>/models/mt5/xl/v0' not found in model shortcut name list (t5-small, t5-base, t5-large, t5-3b, t5-11b). Assuming '/<my_model_dir>/models/mt5/xl/v0' is a path, a model identifier, or url to a directory containing tokenizer files.
[INFO|tokenization_utils_base.py:1718] 2021-01-27 15:00:05,354 >> Didn't find file /<my_model_dir>/models/mt5/xl/v0/tokenizer.json. We won't load it.
[INFO|tokenization_utils_base.py:1718] 2021-01-27 15:00:05,355 >> Didn't find file /<my_model_dir>/models/mt5/xl/v0/added_tokens.json. We won't load it.
[INFO|tokenization_utils_base.py:1718] 2021-01-27 15:00:05,355 >> Didn't find file /<my_model_dir>/models/mt5/xl/v0/special_tokens_map.json. We won't load it.
[INFO|tokenization_utils_base.py:1718] 2021-01-27 15:00:05,355 >> Didn't find file /<my_model_dir>/models/mt5/xl/v0/tokenizer_config.json. We won't load it.
[INFO|tokenization_utils_base.py:1764] 2021-01-27 15:00:05,355 >> loading file /<my_model_dir>/models/mt5/xl/v0/spiece.model
[INFO|tokenization_utils_base.py:1764] 2021-01-27 15:00:05,355 >> loading file None
[INFO|tokenization_utils_base.py:1764] 2021-01-27 15:00:05,355 >> loading file None
[INFO|tokenization_utils_base.py:1764] 2021-01-27 15:00:05,355 >> loading file None
[INFO|tokenization_utils_base.py:1764] 2021-01-27 15:00:05,355 >> loading file None
[INFO|modeling_utils.py:1025] 2021-01-27 15:00:06,472 >> loading weights file /<my_model_dir>/models/mt5/xl/v0/pytorch_model.bin
Traceback (most recent call last):
  File "./finetune_trainer.py", line 367, in <module>
    main()
  File "./finetune_trainer.py", line 230, in main
    freeze_embeds(model)
  File "/<my_dir>/transformers/examples/seq2seq/utils.py", line 567, in freeze_embeds
[INFO|modeling_utils.py:1143] 2021-01-27 15:05:03,683 >> All model checkpoint weights were used when initializing MT5ForConditionalGeneration.

[INFO|modeling_utils.py:1152] 2021-01-27 15:05:03,683 >> All the weights of MT5ForConditionalGeneration were initialized from the model checkpoint at /<my_model_dir>/models/mt5/xl/v0.
If your task is similar to the task the model of the checkpoint was trained on, you can already use MT5ForConditionalGeneration for predictions without further training.
Traceback (most recent call last):
  File "./finetune_trainer.py", line 367, in <module>
    main()
  File "./finetune_trainer.py", line 230, in main
    freeze_embeds(model)
  File "/<my_dir>/transformers/examples/seq2seq/utils.py", line 567, in freeze_embeds
    freeze_params(model.model.shared)
  File "/<my_dir>/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    freeze_params(model.model.shared)
  File "/<my_dir>/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'MT5ForConditionalGeneration' object has no attribute 'model'
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'MT5ForConditionalGeneration' object has no attribute 'model'
Traceback (most recent call last):
  File "./finetune_trainer.py", line 367, in <module>
    main()
  File "./finetune_trainer.py", line 230, in main
    freeze_embeds(model)
  File "/<my_dir>/transformers/examples/seq2seq/utils.py", line 567, in freeze_embeds
    freeze_params(model.model.shared)
  File "/<my_dir>/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'MT5ForConditionalGeneration' object has no attribute 'model'
Traceback (most recent call last):
  File "./finetune_trainer.py", line 367, in <module>
    main()
  File "./finetune_trainer.py", line 230, in main
    freeze_embeds(model)
  File "/<my_dir>/transformers/examples/seq2seq/utils.py", line 567, in freeze_embeds
    freeze_params(model.model.shared)
  File "/<my_dir>/miniconda3/envs/nlp/lib/python3.7/site-packages/torch/nn/modules/module.py", line 779, in __getattr__
    type(self).__name__, name))
torch.nn.modules.module.ModuleAttributeError: 'MT5ForConditionalGeneration' object has no attribute 'model'
	Command being timed: "deepspeed --num_gpus=4 ./finetune_trainer.py --model_name_or_path /<my_model_dir>/models/mt5/xl/v0 --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size 1 --per_device_train_batch_size 1 --predict_with_generate --eval_steps 25000 --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 60 --n_val 10 --n_test 10 --deepspeed ds_config.json --fp16"
	User time (seconds): 348.34
	System time (seconds): 177.55
	Percent of CPU this job got: 166%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 5:15.88
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 33558800
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 1
	Minor (reclaiming a frame) page faults: 67111048
	Voluntary context switches: 132337
	Involuntary context switches: 6635761
	Swaps: 0
	File system inputs: 29248712
	File system outputs: 32
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0
```
3. So I removed `--freeze_embeds` and tried to train MT5-xl again, but I got CUDA out of memory. My device is 4*24G 3090, with BS=1, ZeRO stage=2, and CPU_offload=true. I assume that T5-3b and MT5-xl should be in the same order of magnitude, and I can do it on t5-3b, so I think this should not happen.
4. I also tried training MT5-large. Just replace mt5-xl to mt5-large, under the same conditions in 3. And I got the overflow problem. This is not surprising me because MT5-large seems not fixed FP16 yet. In short, I want to know if there is any problem with my operation or if this is the case. If it is because the MT5-large has not been repaired, does huggingface have any plans to repair it?

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
1. Why can't mt5-xl train on 4*3090? Or what should I do?
2. Can mt5-large FP16 (mainly DeepSpeed) be used? If not, is there any plan to fix it?
<!-- A clear and concise description of what you would expect to happen. -->


Longformer: raise TypeError("pred must not be a Python bool", pred)

## Environment info

- `transformers` version: 4.2.2
- Platform: Ubuntu 18.04
- Python version: 3.7.6
- PyTorch version (GPU?): None
- Tensorflow version (GPU?): 2.3.1, 2.3.2 (with or without GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help@patrick-s-h-lewis

Models:
- longformer @patrickvonplaten

## Information
Errors occur when I use the TFLongformerMainLayer as a layer of my model. I will give a simple example below to reproduce this bug.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
Errors occurs in 'transformers/models/longformer/modeling_tf_longformer.py:1799 _pad_to_window_size  *
        inputs_embeds = tf.cond(padding_len > 0, pad_embeddings, lambda: inputs_embeds)'
It looks like `padding_len >0` is a python bool which caused this error.
According to the [official guide of tf.cond](https://www.tensorflow.org/api_docs/python/tf/cond) example: `result = tf.cond(x < y, lambda: tf.add(x, z), lambda: tf.square(y))`, I think this is because both 'padding_len' and '0' are not tensors, so `padding_len >0` just returns a python bool.
```
TypeError: in user code:

    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    test.py:44 call  *
        x = self.longformer(input_ids=None, inputs_embeds=inputs)[0]
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1680 call  *
        (
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/transformers/models/longformer/modeling_tf_longformer.py:1799 _pad_to_window_size  *
        inputs_embeds = tf.cond(padding_len > 0, pad_embeddings, lambda: inputs_embeds)
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper  **
        return target(*args, **kwargs)
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py:1396 cond_for_tf_v2
        return cond(pred, true_fn=true_fn, false_fn=false_fn, strict=True, name=name)
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201 wrapper
        return target(*args, **kwargs)
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py:507 new_func
        return func(*args, **kwargs)
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py:1180 cond
        return cond_v2.cond_v2(pred, true_fn, false_fn, name)
    /home/xingya/.conda/envs/tf2/lib/python3.7/site-packages/tensorflow/python/ops/cond_v2.py:62 cond_v2
        raise TypeError("pred must not be a Python bool", pred)

    TypeError: ('pred must not be a Python bool', True)
```
Here is a snipt to reproduce this bug:
```
from transformers.models.longformer.modeling_tf_longformer import TFLongformerMainLayer
from tensorflow.keras.layers import Input, Embedding, Dense
from tensorflow.keras.models import Model
from transformers import LongformerConfig
import tensorflow as tf
import numpy as np

tf.random.set_seed(200)

class LongFormerMain(tf.keras.layers.Layer):
    def __init__(self, name='longformer', **kwargs):
        super(LongFormerMain, self).__init__(name=name, **kwargs)
        config = LongformerConfig(attention_window=4, num_hidden_layers=1, vocab_size=10)
        self.longformer = TFLongformerMainLayer(config)

    def call(self, inputs):
        x = self.longformer(input_ids=None, inputs_embeds=inputs)[0]
        return x

inputs = Input(shape=(None,), dtype='int32')
output = Embedding(100, 768)(inputs)
longformer = LongFormerMain()
output = longformer(output)
output = Dense(9, activation='softmax')(output)
model = Model(inputs, output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

x = np.array([[5, 2, 3] * 3] * 100)
y = np.array([[1, 2, 3] * 3] * 100)
model.fit(x=x, y=y, epochs=10, batch_size=4, validation_split=0.1)
print(model.predict([[5, 2, 3] * 3]))
```


Add support for tf2 encoder_decoder

# 🌟 New model addition

I would like to add `TensorFlow-2` support for `encoder_decoder` model. I will soon create a PR, if this is approved.


AttributeError with T5Tokenizer

I am trying to use **T5Tokenizer** and **t5-base** model to fine-tune on **SQuAD** dataset. But each time, when I run the tokenizer code I get errors (e.g, `'NoneType' object has no attribute 'encode'/'batch_encode_plus'/'encode_plus'`).

Example code
```
tokenizer = T5Tokenizer.from_pretrained('t5-base')
ids_neg = tokenizer.encode('negative </s>')
ids_pos = tokenizer.encode('positive </s>')
```
I get the following error:
> AttributeError                            Traceback (most recent call last)
> <ipython-input-19-f34cd55ac673> in <module>()
> ----> 1 ids_neg = tokenizer.encode('negative </s>')
>       2 ids_pos = tokenizer.encode('positive </s>')
>       3 len(ids_neg), len(ids_pos)
>
> AttributeError: 'NoneType' object has no attribute 'encode'


Rag modification

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Padding tokens affect MobileBert output

## Environment info

- `transformers` version: 4.2.2
- Platform: Windows-10-10.0.17134-SP0
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

Model I am using (Bert, XLNet ...): MobileBert

Adding padding tokens to the end of a sequence affects MobileBert output even when masked. I've tried this on a few other models (`bert-base-uncased`, `roberta-base`, `xlm-roberta-base`) and was only able to replicate this with `google/mobilebert-uncased`.

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_string = 'google/mobilebert-uncased'
tokenizer = AutoTokenizer.from_pretrained(model_string)
model = AutoModelForSequenceClassification.from_pretrained(model_string)

example_text = 'Hello, world!'

input_with_pad = tokenizer.encode_plus(
    example_text,
    padding='max_length',
    max_length=32,
    return_tensors='pt'
)
print(input_with_pad)
# {'input_ids': tensor([[ 101, 7592, 1010, 2088,  999,  102,    0,    0,    0,    0,    0,    0,
#             0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
#             0,    0,    0,    0,    0,    0,    0,    0]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
#          0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
#          0, 0, 0, 0, 0, 0, 0, 0]])}

input_without_pad = tokenizer.encode_plus(
    example_text,
    padding='longest',
    max_length=32,
    return_tensors='pt'
)

print(input_without_pad)
# {'input_ids': tensor([[ 101, 7592, 1010, 2088,  999,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1]])}

with torch.no_grad():
    model.eval()

    out_with_pad = model(**input_with_pad)
    print(out_with_pad.logits)
    # tensor([[12693366., -5310913.]])

    out_without_pad = model(**input_without_pad)
    print(out_without_pad.logits)
    # tensor([[12741167., -5327575.]])
```

## Expected behavior

Padding tokens should not affect the output of the model as long as they are masked. As far as I can tell, this only occurs with mobilebert.


Head masking and test_head_masking not working properly for TFT5 models.

When removing `test_head_masking` flags during #9858, I found out `test_headmasking` was actually never run for `TFT5Model` and it seems there must be a bug, please see below:
```
_______________________________________________________________________________________________________ TFT5ModelTest.test_headmasking _______________________________________________________________________________________________________

self = <tests.test_modeling_tf_t5.TFT5ModelTest testMethod=test_headmasking>

    def test_headmasking(self):
        if not self.test_head_masking:
            return

        random.Random().seed(42)
        config, inputs_dict = self.model_tester.prepare_config_and_inputs_for_common()
        random.Random().seed()

        inputs_dict["output_attentions"] = True
        config.output_hidden_states = True
        configs_no_init = _config_zero_init(config)  # To be sure we have no Nan
        for model_class in self.all_model_classes:
            model = model_class(config=configs_no_init)

            # Prepare head_mask
            def prepare_layer_head_mask(i, attention_heads, num_hidden_layers):
                if i == 0:
                    return tf.concat(
                        (tf.zeros(1, dtype=tf.float32), tf.ones(attention_heads - 1, dtype=tf.float32)), 0
                    )
                elif i == num_hidden_layers - 1:
                    return tf.concat(
                        (tf.zeros(attention_heads - 1, dtype=tf.float32), tf.ones(1, dtype=tf.float32)), 0
                    )
                else:
                    return tf.ones(attention_heads, dtype=tf.float32)

            head_mask = tf.stack(
                [
                    prepare_layer_head_mask(i, config.num_attention_heads, config.num_hidden_layers)
                    for i in range(config.num_hidden_layers)
                ],
                0,
            )

            inputs = self._prepare_for_class(inputs_dict, model_class).copy()
            inputs["head_mask"] = head_mask
            if model.config.is_encoder_decoder:
                signature = inspect.signature(model.call)
                arg_names = [*signature.parameters.keys()]
                if "decoder_head_mask" in arg_names:  # necessary diferentiation because of T5 model
                    inputs["decoder_head_mask"] = head_mask

>           outputs = model(**inputs, return_dict=True)

test_modeling_tf_common.py:686:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../../../../miniconda3/envs/bart/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:1012: in __call__
    outputs = call_fn(inputs, *args, **kwargs)
../src/transformers/models/t5/modeling_tf_t5.py:1160: in call
    inputs["encoder_outputs"] = self.encoder(
../../../../../../miniconda3/envs/bart/lib/python3.8/site-packages/tensorflow/python/keras/engine/base_layer.py:1012: in __call__
    outputs = call_fn(inputs, *args, **kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <transformers.models.t5.modeling_tf_t5.TFT5MainLayer object at 0x7f8c38206a30>
input_ids = <tf.Tensor: shape=(13, 7), dtype=int32, numpy=
array([[63, 79, 60,  1, 57, 50, 42],
       [27,  6, 27, 88, 79, 14,  3... [95, 95, 79, 95, 63, 32, 24],
       [ 8,  9, 14, 46, 91, 75, 56],
       [26, 78, 52, 95, 45, 33, 78]], dtype=int32)>
attention_mask = None, encoder_hidden_states = None, encoder_attention_mask = None, inputs_embeds = None
head_mask = <tf.Tensor: shape=(5, 4), dtype=float32, numpy=
array([[0., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [0., 0., 0., 1.]], dtype=float32)>, encoder_head_mask = None
past_key_values = None, use_cache = False, output_attentions = True, output_hidden_states = True, return_dict = True, training = False, kwargs = {}
inputs = {'attention_mask': <tf.Tensor: shape=(13, 7), dtype=float32, numpy=
array([[1., 1., 1., 1., 1., 1., 1.],
       [1., 1..., 1.]], dtype=float32)>, 'encoder_attention_mask': None, 'encoder_head_mask': None, 'encoder_hidden_states': None, ...}
input_shape = [13, 7], batch_size = 13, seq_length = 7, mask_seq_length = 7

    def call(
        self,
        input_ids=None,
        attention_mask=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
        inputs_embeds=None,
        head_mask=None,
        encoder_head_mask=None,
        past_key_values=None,
        use_cache=None,
        output_attentions=None,
        output_hidden_states=None,
        return_dict=None,
        training=False,
        **kwargs,
    ) -> Tuple:
        inputs = input_processing(
            func=self.call,
            config=self.config,
            input_ids=input_ids,
            attention_mask=attention_mask,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
            inputs_embeds=inputs_embeds,
            head_mask=head_mask,
            encoder_head_mask=encoder_head_mask,
            past_key_values=past_key_values,
            use_cache=use_cache,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            training=training,
            kwargs_call=kwargs,
        )

        if inputs["input_ids"] is not None and inputs["inputs_embeds"] is not None:
            err_msg_prefix = "decoder_" if self.is_decoder else ""
            raise ValueError(
                f"You cannot specify both {err_msg_prefix}inputs and {err_msg_prefix}inputs_embeds at the same time"
            )
        elif inputs["input_ids"] is not None:
            input_shape = shape_list(inputs["input_ids"])
            inputs["input_ids"] = tf.reshape(inputs["input_ids"], (-1, input_shape[-1]))
        elif inputs["inputs_embeds"] is not None:
            input_shape = shape_list(inputs["inputs_embeds"])[:-1]
        else:
            err_msg_prefix = "decoder_" if self.is_decoder else ""
            raise ValueError(f"You have to specify either {err_msg_prefix}inputs or {err_msg_prefix}inputs_embeds")

        if inputs["inputs_embeds"] is None:
            assert self.embed_tokens is not None, "You have to intialize the model with valid token embeddings"
            inputs["inputs_embeds"] = self.embed_tokens(inputs["input_ids"])

        batch_size, seq_length = input_shape

        # required mask seq length can be calculated via length of past
        mask_seq_length = (
            shape_list(inputs["past_key_values"][0][0])[2] + seq_length
            if inputs["past_key_values"] is not None
            else seq_length
        )

        if inputs["attention_mask"] is None:
            inputs["attention_mask"] = tf.fill((batch_size, mask_seq_length), 1)
        if (
            self.is_decoder
            and inputs["encoder_attention_mask"] is None
            and inputs["encoder_hidden_states"] is not None
        ):
            encoder_seq_length = shape_list(inputs["encoder_hidden_states"])[1]
            inputs["encoder_attention_mask"] = tf.fill((batch_size, encoder_seq_length), 1)

        # initialize past_key_values with `None` if past does not exist
        if inputs["past_key_values"] is None:
            inputs["past_key_values"] = [None] * len(self.block)

        # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
        # ourselves in which case we just need to make it broadcastable to all heads.
        inputs["attention_mask"] = tf.cast(inputs["attention_mask"], dtype=tf.float32)
        num_dims_attention_mask = len(shape_list(inputs["attention_mask"]))
        if num_dims_attention_mask == 3:
            extended_attention_mask = inputs["attention_mask"][:, None, :, :]
        elif num_dims_attention_mask == 2:
            # Provided a padding mask of dimensions [batch_size, mask_seq_length]
            # - if the model is a decoder, apply a causal mask in addition to the padding mask
            # - if the model is an encoder, make the mask broadcastable to [batch_size, num_heads, mask_seq_length, mask_seq_length]
            if self.is_decoder:
                seq_ids = tf.range(mask_seq_length)
                causal_mask = tf.less_equal(
                    tf.tile(seq_ids[None, None, :], (batch_size, mask_seq_length, 1)),
                    seq_ids[None, :, None],
                )
                causal_mask = tf.cast(causal_mask, dtype=tf.float32)
                extended_attention_mask = causal_mask[:, None, :, :] * inputs["attention_mask"][:, None, None, :]
                if inputs["past_key_values"][0] is not None:
                    extended_attention_mask = extended_attention_mask[:, :, -seq_length:, :]
            else:
                extended_attention_mask = inputs["attention_mask"][:, None, None, :]

        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
        # masked positions, this operation will create a tensor which is 0.0 for
        # positions we want to attend and  -1e9 for masked positions.
        # Since we are adding it to the raw scores before the softmax, this is
        # effectively the same as removing these entirely.

        # T5 has a mask that can compare sequence ids, we can simulate this here with this transposition
        # Cf. https://github.com/tensorflow/mesh/blob/8d2465e9bc93129b913b5ccc6a59aa97abd96ec6/mesh_tensorflow/transformer/transformer_layers.py#L270
        # extended_attention_mask = tf.math.equal(extended_attention_mask,
        #                                         tf.transpose(extended_attention_mask, perm=(-1, -2)))

        extended_attention_mask = (1.0 - extended_attention_mask) * -1e9

        if self.is_decoder and inputs["encoder_attention_mask"] is not None:
            # If a 2D ou 3D attention mask is provided for the cross-attention
            # we need to make broadcastable to [batch_size, num_heads, mask_seq_length, mask_seq_length]
            # we need to make broadcastable to [batch_size, num_heads, seq_length, seq_length]
            inputs["encoder_attention_mask"] = tf.cast(inputs["encoder_attention_mask"], dtype=tf.float32)
            num_dims_encoder_attention_mask = len(shape_list(inputs["encoder_attention_mask"]))
            if num_dims_encoder_attention_mask == 3:
                encoder_extended_attention_mask = inputs["encoder_attention_mask"][:, None, :, :]
            if num_dims_encoder_attention_mask == 2:
                encoder_extended_attention_mask = inputs["encoder_attention_mask"][:, None, None, :]

            # T5 has a mask that can compare sequence ids, we can simulate this here with this transposition
            # Cf. https://github.com/tensorflow/mesh/blob/8d2465e9bc93129b913b5ccc6a59aa97abd96ec6/mesh_tensorflow/transformer/transformer_layers.py#L270
            # encoder_extended_attention_mask = tf.math.equal(encoder_extended_attention_mask,
            #                                         tf.transpose(encoder_extended_attention_mask, perm=(-1, -2)))

            encoder_extended_attention_mask = (1.0 - encoder_extended_attention_mask) * -1e9
        else:
            encoder_extended_attention_mask = None

>       assert inputs["head_mask"] is None, "Head mask not supported"
E       AssertionError: Head mask not supported

../src/transformers/models/t5/modeling_tf_t5.py:714: AssertionError
============================================================================================================== warnings summary ==============================================================================================================
../../../../../../miniconda3/envs/bart/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22
  /Users/daniel.stancl/miniconda3/envs/bart/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:22: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

tests/test_modeling_tf_t5.py: 44 warnings
  /var/folders/vs/4jsdk4nx1ds2m48ltfk3nmdc0000gn/T/tmpc35hmpmg.py:8: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    ag__.converted_call(ag__.ld(warnings).warn, ("The 'warn' method is deprecated, use 'warning' instead", ag__.ld(DeprecationWarning), 2), None, fscope)

tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
  /Users/daniel.stancl/Documents/PhD/Projects/test_transformers/transformers/src/transformers/modeling_tf_utils.py:293: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    tf_logger.warn(

tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
tests/test_modeling_tf_t5.py::TFT5EncoderOnlyModelTest::test_saved_model_creation
  /Users/daniel.stancl/Documents/PhD/Projects/test_transformers/transformers/src/transformers/modeling_tf_utils.py:302: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
    tf_logger.warn("The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.")

-- Docs: https://docs.pytest.org/en/stable/warnings.html
========================================================================================================== short test summary info ===========================================================================================================
FAILED test_modeling_tf_t5.py::TFT5ModelTest::test_headmasking - AssertionError: Head mask not supported
```

My contribution: I'm gonna try to take care of this tomorrow.

<hr>
Reviewer: @jplu


Remove redundant `test_head_masking  = True` flags in test files

This PR removes redundant `test_head_masking = True` flags from test files as this is set by default.

Reviewer: @LysandreJik


Pin memory in Trainer by default


Add head_mask and decoder_head_mask to PyTorch LED

This PR implements `head_mask` and `decoder_head_mask` for PyTorch LED (and Longformer as there's a copy dependency) and it is the follow-up to the open issue #9814.

**Motivation:** This PR is a part of an endeavour to enable the usage of `head_mask` and `decoder_head_mask` for all encoder-decoder transformers following the recent work on BART-like models (#9569).

<hr>

Fixes: https://github.com/huggingface/transformers/issues/9814

Reviewers: @patrickvonplaten @LysandreJik @stas00


About max_length in generation_utils.py

In `generation_utils.py`, the docstring of the `beam_search` function shows the example of usage.

```
>>> input_ids = torch.ones((num_beams, 1), device=model.device, dtype=torch.long)
>>> input_ids = input_ids * model.config.decoder_start_token_id

>>> something else that I omit here

>>> outputs = model.beam_search(input_ids, beam_scorer, logits_processor=logits_processor, **model_kwargs)
```

The `beam_search` function uses `while cur_len < max_length` to control the length of generated sequence. But the `cur_len` counts the length including the start token which is a special token. When the user sets `max_length = 1`, does it not mean that the user wants the model to generate one token **while not considering the start token** (I am not sure that is it just me or others may think that way too)? But the `cur_len` will be 1 at the beginning because of the start token and the statement below in the source code.

```
batch_beam_size, cur_len = input_ids.shape
```

The control flow will jump out of the `while` loop and not generate any token.

Maybe `while cur_len < max_length` should be changed to `while cur_len <= max_length`. And maybe other functions should also change the corresponding loop control statement if I am right.


Deprecate model_path in Trainer.train

# What does this PR do?

This PR deprecates `Trainer.train(model_path=xxx)` to be replaced by `Trainer.train(resume_from_checkpoint=xxx)` which (I think) is clearer and better. No breaking change, just a deprecation warning for now.


Fix computation of attention_probs when head_mask is provided.

Remove dead code path when computing `attention_probs` in case of `head_mask` is provided.

Masking was computed on `attention_scores` which is never used / returned afterwards.