Fix torch 1.8.0 segmentation fault

The ONNX test fails on PyTorch 1.8.0 due to a segmentation fault. This is a draft PR to try different things out.


Fixing conversation test for torch 1.8

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Handle padding in decoder_inputs_id when using generate

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #10478 for pytorch version of `generate()` in generate_utils.py

As described in the issue, when `decoder_input_ids` have different lengths and require padding, generation continues after the padding tokens. This PR modifies that behavior so that tokens are generated before the padding for each element in the batch.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

I am adding those who I see more often in the git blame of the file:
@patrickvonplaten, @patil-suraj, @yjernite

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

This is my first PR, so let me use it to thank everyone involved in this library for all the cool work :)


Similar issue like #1091 in Blenderbot

Tokenizer and model are not in sync. I am using "facebook/blenderbot-400M-distill"

Tokenizer has 8009 base tokens where as model has 8008.

Could you please help me with this?

`from transformers import BlenderbotTokenizer, BlenderbotForConditionalGeneration

mname = "facebook/blenderbot-400M-distill"

model = BlenderbotForConditionalGeneration.from_pretrained(mname)
tokenizer = BlenderbotTokenizer.from_pretrained(mname, local_files_only=True)

print(len(tokenizer))
print(model.config.to_dict()['vocab_size'])`

Here is the output that I get.
8009
8008

## Environment info

- `transformers` version: 4.3.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.1+cpu (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj


OSError: Can't load weights for 'facebook/mbart-large-cc25' when using TFMBartModel

## Environment info

- `transformers` version: 4.3.3
- Platform: Windows 10
- Python version: 3.8
- PyTorch version (GPU?): - No
- Tensorflow version (GPU?): 2.4.1, Yes
- Using GPU in script?: Yes (I think it is indifferent for this issue)
- Using distributed or parallel set-up in script?: No

## Issue Description
Firstly, I would to thank you this extraordinary contribution to NLP. We are starting to apply transformers to our NLP problem and we want to test the pretrained mBART model. I have tried to load the TF version of this model following your documentation: https://huggingface.co/transformers/master/model_doc/mbart.html#tfmbartmodel

Unfortunately, we are experiencing an error in _"model = TFMBartModel.from_pretrained('facebook/mbart-large-cc25')"_.

**Error Traceback:**

404 Client Error: Not Found for url: https://huggingface.co/facebook/mbart-large-cc25/resolve/main/tf_model.h5
Traceback (most recent call last):

  File "c:\users\vicen\anaconda3\envs\signon_2\lib\site-packages\transformers\modeling_tf_utils.py", line 1203, in from_pretrained
    resolved_archive_file = cached_path(

  File "c:\users\vicen\anaconda3\envs\signon_2\lib\site-packages\transformers\file_utils.py", line 1078, in cached_path
    output_path = get_from_cache(

  File "c:\users\vicen\anaconda3\envs\signon_2\lib\site-packages\transformers\file_utils.py", line 1216, in get_from_cache
    r.raise_for_status()

  File "c:\users\vicen\anaconda3\envs\signon_2\lib\site-packages\requests\models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/facebook/mbart-large-cc25/resolve/main/tf_model.h5

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "G:\Mi unidad\D4.2\Proof of Concept simCLR for MT\test_tokenizer.py", line 46, in <module>
    bart_model = TFMBartModel.from_pretrained('facebook/mbart-large-cc25')

  File "c:\users\vicen\anaconda3\envs\signon_2\lib\site-packages\transformers\modeling_tf_utils.py", line 1219, in from_pretrained
    raise EnvironmentError(msg)

OSError: Can't load weights for 'facebook/mbart-large-cc25'. Make sure that:

- 'facebook/mbart-large-cc25' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'facebook/mbart-large-cc25' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

It seems that .h5 file with pretrained weights are not available in your repository. If it is not possible to update it, do you any way to transform pytorch .bin to TF .h5?

Thank you in advance!!!


Facing Issue while running `run_tf_multiple_choice.py` from examples

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Colab
- Python version: NA
- PyTorch version (GPU?): NA
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

I was trying to train `bert-base-cased` on Multiple Choice task with below script provided on Readme of the task
```
export SWAG_DIR=/path/to/swag_data_dir
python ./examples/multiple-choice/run_tf_multiple_choice.py \
--task_name swag \
--model_name_or_path bert-base-cased \
--do_train \
--do_eval \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir models_bert/swag_base \
--per_gpu_eval_batch_size=16 \
--per_device_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output
```
I got below error
```
Invalid argument: ValueError: `generator` yielded an element of shape (4, 1, 80) where an element of shape (None, None) was expected.
Traceback (most recent call last):

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/script_ops.py", line 249, in __call__
    ret = func(*args)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/autograph/impl/api.py", line 620, in wrapper
    return func(*args, **kwargs)

  File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/data/ops/dataset_ops.py", line 938, in generator_py_func
    (ret_array.shape, expected_shape))

ValueError: `generator` yielded an element of shape (4, 1, 80) where an element of shape (None, None) was expected.
```
The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) swag
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Follow this colab [Notebook ](https://github.com/bhadreshpsavani/UnderstandingNLP/blob/master/CheckingTFScripts.ipynb) to run the script and reproduce the issue.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

It should execute the script and train the model without the given error.

Note: The colab notebook given in the readme is not working, It's outdated maybe!
**I removed `--logging-dir logs \` from the script because it was giving me another error**

Tagging SMEs:
@LysandreJik @jplu


🐛 Bug in attention head mask for cross-attention module in encoder-decoder models

Currently, encoder-decoder models use either `head_mask` or `decoder_head_mask` for masking attention heads in cross-attention modules. Both cases are not perfectly correct. Furthermore, MHA in cross-attention modules shares the parameters with the decoder, i.e. `shape = (decoder.num_layers, decoder.num_attention_heads)`, therefore, the usage of encoder `head_mask` in the cross-attention module may lead to errors due to the shape mismatch.

**My contribution:** I will take care of this issue this weekend.

**Reviewers:** @patil-suraj @patrickvonplaten


Wave2vec custom training tokenizer bug

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master
- Platform: win 10
- Python version: 3.8
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 -->
@patrickvonplaten

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
*  [X] the official example scripts: (give details below)
wave2vec Training example
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
I was just running the wave2vec Training on pretrained base modell with the german common voice dataset. I modified the dataset that it fits into the format of librispeech so there are no changes in the example script.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
While Training the wer on eval dataset is exactly 1.
After Training, while evaluation I recognized that the transcribed predictions are unk tokens only and the tokenizer is missing. I am using the pretrained tokenizer that's why.
The words and sentences seems to be correct (counting the same on both sides), only instead of words only the unk tokens are returned after detokenizing.


Transfomer-xl  padding token

When dealing with a batch consisting of sequences of different lengths, how do I choose parameters so that padding_token is not involved in the computation.


Fix example of custom Trainer to reflect signature of compute_loss

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes out-of-date example of custom `Trainer` in docs.

Since several people have asked about multi-label classification in the forum and in #10232 I thought it might be useful to use this as the example.

I also took the liberty of tightening the grammar a bit in the preceding text 😃

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Forum link: https://discuss.huggingface.co/t/custom-loss-compute-loss-got-an-unexpected-keyword-argument-return-outputs/4148?u=lewtun

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

@sgugger


Enabling multilingual models for translation pipelines.

# What does this PR do?

Enables mutlilingual translation for pipelines.

Some models can target multiple languages/ language pairs. Before this PR, there was no simple way to exploit that within the Translation pipeline.

## Move away from `translation_XX_to_YY`.

Because src_lang, tgt_lang pairs can be used for a single model, we need to move away from this task naming scheme.
Currently it was done, as being able to use `src_lang` and `tgt_lang` both within `pipeline(.... src_lang=XX, tgt_lang=yy)` and at call time `translator = pipeline(...); translation("input string", src_lang=XX, tgt_lang=YY)`

 ## Rely on the model's tokenizer to build the inputs.

We now have at least 3 (m2m. mbart50, T5) different models that prepare input ids in a different manner. In order to avoid switches within a pipeline that would depend on a model, instead `tokenizer` can optionally implement `_build_translation_input_ids`.  That enables to put model specific logic within their model files and use all their custom methods over there.

This is in line with https://github.com/huggingface/transformers/pull/10002.

## Misc

- `ensure_tensor_on_device` now supports non tensor members
- Added a `deep_round` test utility to enable testing nested structures that contain tensors, floats and so on.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


tensorflow model convert onnx

I use this code to transfer TFBertModel to onnx:
”convert(framework="tf", model=model, tokenizer=tokenizer, output=Path("onnx_tf/segment.onnx"), opset=12)”
and the output log as follows:
`Using framework TensorFlow: 2.2.0, keras2onnx: 1.7.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch'}
Found output output_1 with shape: {0: 'batch', 1: 'sequence'}
Found output output_2 with shape: {0: 'batch', 1: 'sequence'}
Found output output_3 with shape: {0: 'batch', 1: 'sequence'}
Found output output_4 with shape: {0: 'batch', 1: 'sequence'}
Found output output_5 with shape: {0: 'batch', 1: 'sequence'}
Found output output_6 with shape: {0: 'batch', 1: 'sequence'}
Found output output_7 with shape: {0: 'batch', 1: 'sequence'}
Found output output_8 with shape: {0: 'batch', 1: 'sequence'}
Found output output_9 with shape: {0: 'batch', 1: 'sequence'}
Found output output_10 with shape: {0: 'batch', 1: 'sequence'}
Found output output_11 with shape: {0: 'batch', 1: 'sequence'}
Found output output_12 with shape: {0: 'batch', 1: 'sequence'}
Found output output_13 with shape: {0: 'batch', 1: 'sequence'}`

but,I run the output onnx,an error is occured as follows:
 logits = session.run(None, inputs_onnx)
  File "/usr/lib64/python3.6/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 124, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids:0

transformers           3.0.2


[WIP] Add VisualBERT

This PR adds VisualBERT (See Closed Issue #5095). It is a work in progress.


RAG with RAY workers keep repetitive copies  of knowledge base as  .nfs files until the process is done.

As mentioned in [this PR](https://github.com/huggingface/transformers/pull/10410), I update the **my_knowledge_dataset**  object all the time. I save the new my_knowledge_dataset in the same place by removing previously saved stuff. But still, I see there are always some hidden files left.  Please check the screenshot below.

![image](https://user-images.githubusercontent.com/16892570/110067455-a0da4880-7dd8-11eb-95cd-5b908c7a04af.png)
![image](https://user-images.githubusercontent.com/16892570/110067527-ccf5c980-7dd8-11eb-9328-7a361f554454.png)

I did some checks and found that these .nfs files being used by RAY. But my local KB is 30GB, so I do not want to add a .nfs file in every iteration. Is there a way to get over this?

@amogkam
@lhoestq


Calling Inference API returns input text

- `transformers` version: 4.4.0dev0
- Platform: MACosx
- Python version: 3.7
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Library:

@patrickvonplaten
@LysandreJik
@sgugger

## Information

Model I am using (Bert, XLNet ...): TransformerXL

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Upload private model to hub
2. Follow the tutorial around calling the Inference API (pasted below)

I've trained a `TransformerXLLMHeadModel` and am using the equivalent tokenizer class `TransfoXLTokenizer` on a custom dataset. I've saved both of these classes and have verified that loading the directory using the Auto classes succeeds and that the model and tokenizer are usable. When attempting to call the Inference API, I only get back my input text.

### Specific code from tutorial
```python
import json
import requests

API_TOKEN = "api_1234"
API_URL = "https://api-inference.huggingface.co/models/private/model"
headers = {"Authorization": f"Bearer {API_TOKEN}"}

def query(payload):
    data = json.dumps(payload)
    response = requests.request("POST", API_URL, headers=headers, data=data)
    return json.loads(response.content.decode("utf-8"))

data = query({"inputs": "Begin 8Bars ", "temperature": .85, "num_return_sequences": 5})
# data = [{'generated_text': 'Begin First '}]
```

## Expected behavior

When calling the Inference API on my private model I would expect it to return additional output rather than just my input text.


Typo correction.

# What does this PR do?

Fix a typo: DEBERTA_PRETRAINED_MODEL_ARCHIVE_LIST => DEBERTA_V2_PRETRAINED_MODEL_ARCHIVE_LIST in line 31.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes  #10529

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Test/Predict on summarization task

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.3.0-53-generic-x86_64-with-glibc2.10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

@patrickvonplaten, @patil-suraj
Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
- maintained examples using bart: @patrickvonplaten, @patil-suraj

## Information

Model I am using (Bart):

The problem arises when using:
* [ ] only have train and eval example, do not have test/predict example script
* [ ] my own modified scripts: (give details below)
* [ ] CUDA_VISIBLE_DEVICES=2,3 python examples/seq2seq/run_seq2seq.py \
    --model_name_or_path /home/yxzhou/experiment/ASBG/output/xsum_bart_large/ \
    --do_predict \
    --task summarization \
    --dataset_name xsum \
    --output_dir /home/yxzhou/experiment/ASBG/output/xsum_bart_large/test/ \
    --num_beams 1 \

The tasks I am working on is:
* [ ] CNNDAILYMAIL , XSUM

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Use the above script to test on CNNDAILYMAIL and XSUM dataset, the program seems will be always stuck at training_step (e.g., 26/709)

Could you please kindly provide a test/predict example script of the summarization task (e.g., XSUM). Thank you so much!


Typo in deberta_v2/__init__.py

https://github.com/huggingface/transformers/blob/c503a1c15ec1b11e69a3eaaf06edfa87c05a2849/src/transformers/models/deberta_v2/__init__.py#L31
Should be '' DEBERTA_V2_PRETRAINED_MODEL_ARCHIVE_LIST ''.


Different vocab_size between model and tokenizer of mT5

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: ubuntu 18.04
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1
### Who can help
@patrickvonplaten
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoModelForSeq2SeqLM
from transformers import AutoTokenizer

mt5s = ['google/mt5-base', 'google/mt5-small', 'google/mt5-large', 'google/mt5-xl', 'google/mt5-xxl']

for mt5 in mt5s:
    model = AutoModelForSeq2SeqLM.from_pretrained(mt5)
    tokenizer = AutoTokenizer.from_pretrained(mt5)

    print()
    print(mt5)
    print(f"tokenizer vocab: {tokenizer.vocab_size}, model vocab: {model.config.vocab_size}")
```
This is problematic in case when one addes some (special) tokens to tokenizer and resizes the token embedding of the model with `model.resize_token_embedding(len(tokenizer))`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
vocab_size for model and tokenizer should be the same?
<!-- A clear and concise description of what you would expect to happen. -->


Refactoring checkpoint names for multiple models

Hi, @sgugger reupload without datasets dir and added tf_modeling files, removed extra decorator in distilbert.
Linked to #10193, this PR refactors the checkpoint names in one private constant.

one note: longformer_tf has two checkpoints "allenai/longformer-base-4096" & "allenai/longformer-large-4096-finetuned-triviaqa". I set the checkpoint constant to "allenai/longformer-base-4096" and left the one decorator with "allenai/longformer-large-4096-finetuned-triviaqa".

Fixes #10193

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Fix Adafactor documentation (recommend correct settings)

This PR fixes documentation to reflect optimal settings for Adafactor:
- fix an impossible arg combination erroneously proposed in the example
- use the correct link to the adafactor paper where `clip_treshold` is discussed
- document the recommended `scale_parameter=False`
- add other recommended settings combinations, which are quite different from the original
- re-org notes
- make the errors less ambiguous

Fixes #7789

@sgugger

(edited by @stas00 to reflect it's pre-merge state as the PR evolved since it's original submission)


fine-tune Pegasus with xsum using Colab but generation results have no difference

Hi. I tried to fine-tune pegasus large with xsum dataset using Colab (Pro). I was able to finish the fine-tuning with batch size 1, and 2000 epochs in about 40 minutes (larger batch size crashed colab). The working Colab notebook I used is shared at https://colab.research.google.com/drive/1RyUsYDAo6bA1RZICMb-FxYLszBcDY81X?usp=sharing

However, the generated summary seems to be the same for the pegasus large model (https://huggingface.co/google/pegasus-large) and the fine-tuned model. But the generated result using pegasus xsum model (https://huggingface.co/google/pegasus-xsum) is different and much better.

The training loss is already 0 and I am not sure what I have done wrong. Any help and pointers are highly appreciated.

@sshleifer


Change/remove default maximum length in run_glue.py

I propose one of the following:
1) A vast majority of models have maximum sequence length as 512. 128 as a maximum length is very misleading because of this reason  and hence I suggest we revise it to 512.
2) Totally remove this variable and set maximum length only based on model’s maximum length. The tokenizers library used in this example  already has code to take the minimum of (max_seq_length, max_model_seq_length). Why not just remove this redundancy and directly set it based on the model’s largest acceptable sequence length

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


BERT as encoder - position ids

Hello,

I have an EncoderDecoderModel.from_encoder_decoder_pretrained which is using BERT as both the decoder and encoder.
I would like to adjust position_ids for the encoder input, however, looking at [this documentation](https://huggingface.co/transformers/model_doc/encoderdecoder.html#transformers.EncoderDecoderModel.forward) it seems like there is no such argument.
How can I do this?
Sorry if this is an obvious question, I'm new to this stuff.
Thanks!


Inconsistent API output for Q&A models between eager mode and torchscripted

## Environment info

- `transformers` version: 4.3.3
- Platform: Linux-5.4.0-1037-aws-x86_64-with-glibc2.10
- Python version:3.8.6
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Information

@sgugger I am using question-answering models from auto class, have tried, Bert-base/Roberta-base/distilbert-base-cased-distilled-squad.

The problem arises when using:

Accessing the start and end logits through the output of the model is inconsistent between eager mode and Torchscripted(traced) model. For the eager mode, it is possible to use "outputs.start_logits, outputs.end_logits" however for
Torchscripted model, it returns a tuple, as it was in older version (i.e. 2.10).

## To reproduce

```
import transformers
import os
import torch
from transformers import (AutoModelForSequenceClassification, AutoTokenizer, AutoModelForQuestionAnswering,
 AutoModelForTokenClassification, AutoConfig)
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
pretrained_model_name = 'distilbert-base-cased-distilled-squad'
max_length = 30
config_torchscript = AutoConfig.from_pretrained(pretrained_model_name,torchscript=True)
model_torchscript = AutoModelForQuestionAnswering.from_pretrained(pretrained_model_name,config=config_torchscript)

tokenizer = AutoTokenizer.from_pretrained(pretrained_model_name,do_lower_case=True)

config = AutoConfig.from_pretrained(pretrained_model_name)
model = AutoModelForQuestionAnswering.from_pretrained(pretrained_model_name,config=config)

dummy_input = "This is a dummy input for torch jit trace"
inputs = tokenizer.encode_plus(dummy_input,max_length = int(max_length),pad_to_max_length = True, add_special_tokens = True, return_tensors = 'pt')
input_ids = inputs["input_ids"].to(device)

model.to(device).eval()
model_torchscript.to(device).eval()

traced_model = torch.jit.trace(model_torchscript, [input_ids])

outputs = model(input_ids)

print(outputs.start_logits, outputs.end_logits)

print("******************************* eager mode passed **************")

traced_outputs = traced_model(input_ids)

print(traced_outputs.start_logits, traced_outputs.end_logits)

print("******************************* traced mode passed **************")

```
Steps to reproduce the behavior:

1. install transformers.
2. Run the above code snippet

## Expected behavior

Being able to access the start and end logits consistently between eager and torchscript mode.


Removes overwrites for output_dir

# What does this PR do?

This PR removes the overwrites of the `output_dir` when running training on `SageMaker` and it removes the automatic save if `output_dir` is `None`.
The overwrites have been removed since it prevents saving checkpoints to a different dir like `opt/ml/checkpoints` impossible and it is not the "transformers" way. We can keep it the same as running training somewhere else and provide documentation on what you need to do.


Unable to translate Arabic to many other languages in MBart-50

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten
## Information

Model I am using (Bert, XLNet ...): Mbart-50

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
```
# translate Arabic to Hindi
article_ar = "الأمين العام للأمم المتحدة يقول إنه لا يوجد حل عسكري في سوريا."
tokenizer.src_lang = "ar_AR"
encoded_ar = tokenizer(article_ar, return_tensors="pt")
generated_tokens = model.generate(
    **encoded_ar,
    forced_bos_token_id=tokenizer.lang_code_to_id["hi_IN"]
)
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
```
**Result: ['Secretary of the United Nations says there is no military solution to Syria.']**

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The result should be in Hindi, but it weirdly it is in English. Arabic --> Hindi is an example. But if you test Arabic -->xx_XX you will obtain the same behavior except for a few ones (such as `fr_XX` or `de_DE`)
<!-- A clear and concise description of what you would expect to happen. -->


Adding option to truncation from beginning instead of end, for both longest_first and longest_second

Feature request

Request for a new feature

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

The current [truncation strategies](https://huggingface.co/transformers/preprocessing.html#everything-you-always-wanted-to-know-about-padding-and-truncation) only provide truncation from end of the input sentence. This may be enough for most cases, but is not applicable for dialog tasks.

Say, you are concatenating an input dialog context to BERT. If the input length > max_length (256/ 512), then the tokens are truncated from the ending i.e. the most recent utterances in the dialog.

In some cases, you want the most recent utterances to be included in the input and if truncation needs to be done, then the oldest utterances be truncated i.e. truncation from the beginning.

This can be done manually in own code base outside of Transformers library, but it's not ideal.

Truncation outside of model will be done most likely on full words to truncate from beginning and fit them to model max length (say you truncate from beginning and reduce input to 254 words). But, these words will be converted to subwords when fed to BertTokenizer and the final input will be > 256, thus resulting in words being dropped from the last utterance again.

To do this properly outside of the Transformers library, one would need to instantiate the same Tokenizer object, tokenize each input and then truncate from beginning, then convert ids back to tokens and then either reconstruct the input from the truncated tokenized version or skip the tokenizer call inside the Transformers library.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Converting models for tensoflowjs (node)

External

Using the library with external tools (onnx, tflite, ...)

## Environment info
- `transformers` version: 4.3.3
- Platform: Linux-5.11.2-zen1-1-zen-x86_64-with-glibc2.2.5
- Python version: 3.8.8 (also 3.9.2 but tensorflowjs is not available in 3.9.2)
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
- tensorflow: @jplu

## Information

Model I am using (Bert, XLNet ...): Any tensorflow ones

The problem arises when using:
* [X] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load a tensforflow model (e.g : `model = TFAutoModel.from_pretrained("distilbert-base-uncased")`)
2. Convert it to H5 with `model.save_pretrained(path)`
3. Try to load it in tensorflowjs and get `UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'model_config' of null`

## Expected behavior
Be able to load a model in the LayerModel format as it's the only one which allows finetuning.

## More informations.
I know the question was already posted in https://github.com/huggingface/transformers/issues/4073

I need to do finetuning so onnx, graphModels & cie should be avoided.

Seems that the H5 model just need the config file which seems saved on the side with the custom HF script.
I went to read some issues on tensorflowjs (exemple : [this](https://github.com/tensorflow/tfjs/issues/2455) or [that](https://github.com/tensorflow/tfjs/issues/931)) and the problem is that the HF model contains only the weights and not the architecture. The goal would be to adapt the `save_pretrained` function to save the architecture as well. I guess it's complex because of the `Saving the model to HDF5 format requires the model to be a Functional model or a Sequential model.` error in described bellow.

Seems also that only H5 model can be converted to a trainable LayerModel.

I'm willing to work on a PR or to help as i'm working on a web stack (nodejs) and I need this.

I made a drawing of all models (that I'm aware of) to summarize loading converting :
![image](https://user-images.githubusercontent.com/30974685/109985131-6edebe80-7cd2-11eb-8f3c-60080e98a115.png)

## Also tried :
Use the nodejs `tf.node.loadSavedModel` which return only a saved model which I cannot use as the base structure with something like :
```
const bert = await tf.loadLayersModel(`file://${this._bert_model_path}/model.json`)
this._clf = tf.sequential();
this._clf.add(bert); // Raise Error
this._clf.add(tf.layers.dense({ units: 768, useBias: true }));
etc...
this._clf.compile(....)
this._clf.train(...)
```

Look for other libraries to train models (libtorch : incomplete, onnx training: only in python etc..)

Should I also write an issue on tensorflowjs ?

Thanks in advance for you time and have a great day.


Not always consider a local model a checkpoint in run_glue

# What does this PR do?

In the `run_glue` script, a local model is automatically considered a checkpoint (which is there to enable users to do --model_path_or_name path_to_specific_checkpoint`) but when using a local model, it can crash if the number of labels is different (cf #10502). This PR fixes that by checking the number of labels inside the potential checkpoint before passing it to the `Trainer`. Another possible fix is to check for the presence of files like `trainer_state.json` (but this only works if the checkpoint was created with a recent version of Transformers).

Fixes #10502


Bug in Hosted inference API

Hello,I found that when I use the Hosted inference API, there will be some problems.
Some inference results show the entire sentence, but other inference results only show the mask token.
For example the model [uer/roberta-base-word-chinese-cluecorpussmall](https://huggingface.co/uer/roberta-base-word-chinese-cluecorpussmall).
When I input "中国的首都是北[MASK]",the results are:
![image](https://user-images.githubusercontent.com/59219579/109969085-72079800-7d2e-11eb-876a-2f66e0fd23b2.png)
![image](https://user-images.githubusercontent.com/59219579/109969434-dcb8d380-7d2e-11eb-9f15-84b3229f97cb.png)
Thanks


Add Vision Transformer + ViTFeatureExtractor

# What does this PR do?

This PR includes 2 things:

* it adds the [Vision Transformer (ViT)](https://arxiv.org/abs/2010.11929) by Google Brain. ViT is a Transformer encoder trained on ImageNet. It is capable of classifying images, by placing a linear classification head on top of the final hidden state of the [CLS] token. I converted the weights from the [timm](https://github.com/rwightman/pytorch-image-models) repository, which already took care of converting the weights of the original implementation (which is written in JAX) into PyTorch. Once this model is added, we can also add [DeIT](https://ai.facebook.com/blog/data-efficient-image-transformers-a-promising-new-technique-for-image-classification/) (Data-efficient Image Transformers) by Facebook AI, which improve upon ViT.

* it provides a design for the `ViTFeatureExtractor` class, which can be used to prepare images for the model. It inherits from `FeatureExtractionMixin` and defines a `__call__` method. It currently accepts 3 types of inputs: PIL images, Numpy arrays and PyTorch tensors. It defines 2 transformations using `torchvision`: resizing + normalization. It then returns a `BatchFeature` object with 1 key, namely `pixel_values`.

Demo notebook of combination of `ViTForImageClassification` + `ViTFeatureExtractor`: https://colab.research.google.com/drive/16TCM-tJ1Mfhs00Qas063kWZmAtVJcOeP?usp=sharing

Compared to NLP models (which accept `input_ids`, `attention_mask` and `token_type_ids`), this model only accepts `pixel_values`. The model itself then converts these pixel values into patches (in case of ViT) in the `ViTEmbeddings` class.

## Help needed

Would be great if you can help me with the following tasks:

- [x] Add and improve tests. Currently I have defined the following tests: `test_modeling_vit.py`, `test_feature_extraction_vit.py`. However, for the former, since ViT does not use `input_ids`/`input_embeds`, some tests are failing, so I wonder whether it should use all tests defined in `test_modeling_common.py`. For the latter, I also need some help in creating random inputs to test the feature extractor on.
- [x] Add support for `head_mask` in the forward of `ViTModel`. Possibly remove `attention_mask`?
- [x] Run `make fix-copies` (doesn't work right now for me on Windows)
- [x] Remove the `is_decoder` logic from `modeling_vit.py` (since the model was created using the CookieCutter template). I assume that things such as `past_key_values` are not required for an encoder-only model.

## Who can review?

@patrickvonplaten @LysandreJik @sgugger


Dynamic batch size for Seq2SeqTrainer

# 🚀 Feature request

In Fairseq it is possible to forego setting a constant batch-size in favor of a dynamic batch size with --max_tokens. This ensures that a batch always consists of at max N=max_tokens tokens. Fairseq tries to get to max_tokens by adding samples to the batch until N = max_tokens or just below.

I believe @sshleifer has implemented this for finetune.py here: #7030

Is it possible to add "--max_tokens_per_batch N" as a trainer argument to Seq2SeqTrainer?

## Motivation

This would an invaluable help when training/fine-tuning large models on data sequences (like sentences) of varying length. Long sequences/sentences might lead to OOM-Errors with a fixed batch-size.


Why the positional embeddings in bert are not inplemented by sin/cos as the original paper said? Are these embeddings trainable?


Error in run_squad.py with BartForQuestionAnswering model

I am using the  BartForQuestionAnswering model and getting the following error during evaluation
`Evaluating:   0%|          | 0/315 [00:10<?, ?it/s]
Traceback (most recent call last):
  File "../run_squad.py", line 831, in <module>
    main()
  File "../run_squad.py", line 820, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "../run_squad.py", line 325, in evaluate
    output = [to_list(output[i]) for output in outputs.to_tuple()]
  File "../run_squad.py", line 325, in <listcomp>
    output = [to_list(output[i]) for output in outputs.to_tuple()]
  File "../run_squad.py", line 73, in to_list
    return tensor.detach().cpu().tolist()
AttributeError: 'tuple' object has no attribute 'detach'`

It works perfectly fine with the bert-base-uncased model but fails in case of BART model.
What could resolve this issue?
Thanks


Stale Bot

Adds a stale bot based on GitHub Actions.

This bot is slightly different than the previous one in that it comments that it is closing the issue and closes it immediately, rather than waiting 7 days. From what I've seen up to now, this shouldn't be an issue at all.

I've commented out the code for now so that it doesn't actually close issues, but you can take a look at which would be closed [here](https://github.com/huggingface/transformers/runs/2027394567?check_suite_focus=true). Will un-comment the code before merging.


Loading tapas model into pipeline from directory gives different result

Hi,
   I am using the following versions of these packages :
transformers = 4.3.2
pytorch = 1.6.0

I am using the following code to download and save a pretrained model:

```py
from transformers import TapasConfig,TapasTokenizer,TapasForQuestionAnswering
import torch
config = TapasConfig.from_pretrained('google/tapas-base-finetuned-wtq',from_pt=True)
model = TapasForQuestionAnswering.from_pretrained('google/tapas-base', config=config)
tokenizer=TapasTokenizer.from_pretrained("google/tapas-base-finetuned-wtq",from_pt=True)
import sys

outdir = sys.argv[1]

model.save_pretrained(outdir)
tokenizer.save_pretrained(outdir)
config.save_pretrained(outdir)
```
When I then feed the model directory into pipeline, I don't get any result for the table illustrated in the documentation....
If I let pipeline download the model on-the-fly, I get results. Here is the code to feed model directory into pipeline...

```py
import sys
from transformers import pipeline

nlp = pipeline(task="table-question-answering",framework="pt",model="tapas_model_dir")
#nlp = pipeline(task="table-question-answering")

import pandas as pd

data= {    "actors": ["brad pitt", "leonardo di caprio", "george clooney"],
    "age": ["56", "45", "59"],
    "number of movies": ["87", "53", "69"],
    "date of birth": ["7 february 1967", "10 june 1996", "28 november 1967"]
}

import numpy as np
table = pd.DataFrame.from_dict(data)
print(np.shape(table))
result = nlp(query=["How many movies has Brad Pitt acted in","What is Leonardo di caprio's age"],table=table)

print(result)
```


I was trying to use run_mlm.py to fine-tune roberta-large with a custom dataset on Google Colab.

!python /content/transformers/examples/language-modeling/run_mlm.py \
    --model_name_or_path roberta-large \
    --train_file /content/traincorpus.txt \
    --validation_file /content/devcorpus.txt \
    --do_train \
    --do_eval \
    --per_device_train_batch_size 2 \
    --output_dir finetunedmodel \
    --overwrite_output_dir True

I am able to train the model however at the end I can not see the evaluation due to an error:

Traceback (most recent call last):
  File "/content/transformers/examples/language-modeling/run_mlm.py", line 442, in <module>
    main()
  File "/content/transformers/examples/language-modeling/run_mlm.py", line 416, in main
    trainer.log_metrics("train", metrics)
AttributeError: 'Trainer' object has no attribute 'log_metrics'

There is nothing extra on colab env. I wasn't getting this error a week ago or something.


[WIP][BIGBIRD] Add new conversion

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Remove unsupported methods from ModelOutput doc

# What does this PR do?

As said in the title :-)

Fixes #10469


Rework TPU checkpointing in Trainer

# What does this PR do?

This PR rewors a tiny bit the checkpointing mechanism in the `Trainer` and `PreTrainedModel` to get rid of the hack that stored something in the config (which was then forever present if the user decided to share their model on the hub).

The main problem is that the save on TPU has to be called on all processes because there is a synchronization inside (so if we only execute it in one process, the other do not reach the synchronization point and everything hangs). At the same time, the config should only be saved on one process to avoid race conditions. So this PR solves the problem by adding two new arguments to `save_pretrained`: `is_main` and `_save_function`.


Fine tuning a pipeline

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
[Already asked](https://github.com/huggingface/transformers/issues/8127)

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->
I would like to fine-tune a pipeline using a model (pre-trained or fine-tuned).


GLUE benchmark crashes with MNLI and STSB

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.8.0-44-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: no, running only on CPU or single GPU

### Who can help
Tags: @patrickvonplaten  @LysandreJik

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: GLUE
* [ ] my own task or dataset:

## To reproduce

Steps to reproduce the behaviour:

Create model with a very recent version of transformers:
```
>>> from transformers import RobertaForMaskedLM, RobertaConfig, RobertaTokenizer
>>> tok = RobertaTokenizer.from_pretrained('roberta-base')
>>> config = RobertaConfig.from_pretrained('roberta-base')
>>> model = RobertaForMaskedLM(config=config)
>>> tok.save_pretrained('tmp_model')
('tmp_model/tokenizer_config.json', 'tmp_model/special_tokens_map.json', 'tmp_model/vocab.json', 'tmp_model/merges.txt', 'tmp_model/added_tokens.json')
>>> model.save_pretrained('tmp_model')
```

Run GLUE benchmark on MNLI:
```
python ./examples/text-classification/run_glue.py \
            --model_name_or_path tmp_model \
            --task_name MNLI \
            --do_train \
            --do_eval \
            --max_seq_length 128 \
            --per_device_train_batch_size 32 \
            --per_device_eval_batch_size 32 \
            --learning_rate 1e-5 \
            --num_train_epochs 1 \
            --output_dir TMP/
```
or on STSB:
```
python ./examples/text-classification/run_glue.py \
            --model_name_or_path tmp_model \
            --task_name STSB \
            --do_train \
            --do_eval \
            --max_seq_length 128 \
            --per_device_train_batch_size 32 \
            --per_device_eval_batch_size 32 \
            --learning_rate 1e-5 \
            --num_train_epochs 1 \
            --output_dir TMP/
```

### Error logs
MNLI:
> Traceback (most recent call last):
>   File "./examples/text-classification/run_glue.py", line 480, in <module>
>     main()
>   File "./examples/text-classification/run_glue.py", line 415, in main
>     train_result = trainer.train(resume_from_checkpoint=checkpoint)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1048, in train
>     tr_loss += self.training_step(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1432, in training_step
>     loss = self.compute_loss(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1464, in compute_loss
>     outputs = model(**inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1168, in forward
>     loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
>     return F.cross_entropy(input, target, weight=self.weight,
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
>     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
>     ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
> IndexError: Target 2 is out of bounds.

The problem seems to be related to the incorrect initialisation of the classification head. However, with some prints I noticed that the shape of the classification head was `hidden_size x 3`, so I do not really understand where the problem comes from...

STSB:
> Traceback (most recent call last):
>   File "./examples/text-classification/run_glue.py", line 480, in <module>
>     main()
>   File "./examples/text-classification/run_glue.py", line 415, in main
>     train_result = trainer.train(resume_from_checkpoint=checkpoint)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1048, in train
>     tr_loss += self.training_step(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1432, in training_step
>     loss = self.compute_loss(model, inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/trainer.py", line 1464, in compute_loss
>     outputs = model(**inputs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/transformers/models/roberta/modeling_roberta.py", line 1168, in forward
>     loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
>     result = self.forward(*input, **kwargs)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 961, in forward
>     return F.cross_entropy(input, target, weight=self.weight,
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2468, in cross_entropy
>     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
>   File "/home/lucadiliello/anaconda3/envs/tmp/lib/python3.8/site-packages/torch/nn/functional.py", line 2264, in nll_loss
>     ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
> RuntimeError: expected scalar type Long but found Float

Here the problem seems to be related to the `dtype` of the targets.

Interestingly, loading an old model like `bert-base-cased` or `roberta-base` does not raise errors....


[ProphetNet] Bart-like Refactor

# What does this PR do?

This PR refactors ProphetNet similar to Bart in that it moves the time dimension to be always at the 2nd place and the batch dimensions always in the first place. Also, the cache is refactored to consists of tuples instead of a dict.

The model is thereby very much aligned with Bart (I cannot really add any " # Copied from" statements though because the weight names are different).

The PR is in spirit very similar to https://github.com/huggingface/transformers/pull/8900.

I've verified that all slow tests pass. In the next step, I want to make a short notebook, verifying that ProphetNet can be trained since there have been some issues on training: #9804

# Benchmarking

The PR doesn't change compute or memory complexity:

On this PR:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s
--------------------------------------------------------------------------------
microsoft/prophetnet-large-unc       8               8             0.029
microsoft/prophetnet-large-unc       8               32            0.044
microsoft/prophetnet-large-unc       8              128            0.175
microsoft/prophetnet-large-unc       8              512             N/A
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB
--------------------------------------------------------------------------------
microsoft/prophetnet-large-unc       8               8              2562
microsoft/prophetnet-large-unc       8               32             2756
microsoft/prophetnet-large-unc       8              128             3628
microsoft/prophetnet-large-unc       8              512             N/A
--------------------------------------------------------------------------------
```

on master:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s
--------------------------------------------------------------------------------
microsoft/prophetnet-large-unc       8               8             0.027
microsoft/prophetnet-large-unc       8               32            0.044
microsoft/prophetnet-large-unc       8              128            0.172
microsoft/prophetnet-large-unc       8              512             N/A
--------------------------------------------------------------------------------

====================      INFERENCE - MEMORY - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length    Memory in MB
--------------------------------------------------------------------------------
microsoft/prophetnet-large-unc       8               8              2562
microsoft/prophetnet-large-unc       8               32             2768
microsoft/prophetnet-large-unc       8              128             3740
microsoft/prophetnet-large-unc       8              512             N/A
--------------------------------------------------------------------------------
```


Fine tune of speaker embeddings model

# 🚀 Feature request

Provide a way to fine tune the X-vector speaker embeddings model using our own custom dataset. (https://huggingface.co/hbredin/SpeakerEmbedding-XVectorMFCC-VoxCeleb)

## Motivation

This will help in finetuning the model for new domain/speakers.


## Context
I have used the official example for Q&A [here](https://huggingface.co/dbmdz/bert-base-italian-cased), but slightly modified the `Pipeline` to use the `model` and `tokenizer` objects from pretrained model as exaplained [here](https://huggingface.co/transformers/main_classes/pipelines.html#the-pipeline-abstraction). This because I need a `cache_dir` that currently is not supported by `Pipeline` object.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.3
- Platform: Linux-4.19.121-linuxkit-x86_64-with-debian-10.1
- Python version: 3.7.4
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: <NO>
- Using distributed or parallel set-up in script?: <NO>

### Who can help

Model:
https://huggingface.co/dbmdz/bert-base-italian-cased
@patrickvonplaten, @patil-suraj

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (translation)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
question = 'Quale filosofia seguì Marco Aurelio ?'
context = 'Marco Aurelio era un imperatore romano che praticava lo stoicismo come filosofia di vita .'

tokenizer = AutoTokenizer.from_pretrained(
    'mrm8488/bert-italian-finedtuned-squadv1-it-alfa',
    cache_dir=os.getenv("cache_dir", "model"))
model = AutoModel.from_pretrained(
    'mrm8488/bert-italian-finedtuned-squadv1-it-alfa',
    cache_dir=os.getenv("cache_dir", "model"))

nlp_qa_bert = pipeline(
    'question-answering',
    model=model,
    tokenizer=tokenizer)

out = nlp_qa_bert({
    'question': question,
    'context': context
})
print(out)
````

ERROR:

```
Traceback (most recent call last):
  File "qa/run.py", line 26, in <module>
    tokenizer=tokenizer)
  File "/usr/local/lib/python3.7/site-packages/transformers/pipelines/__init__.py", line 418, in pipeline
    return task_class(model=model, tokenizer=tokenizer, modelcard=modelcard, framework=framework, task=task, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/transformers/pipelines/question_answering.py", line 135, in __init__
    TF_MODEL_FOR_QUESTION_ANSWERING_MAPPING if self.framework == "tf" else MODEL_FOR_QUESTION_ANSWERING_MAPPING
  File "/usr/local/lib/python3.7/site-packages/transformers/pipelines/base.py", line 577, in check_model_type
    f"The model '{self.model.__class__.__name__}' is not supported for {self.task}. Supported models are {supported_models}",
transformers.pipelines.base.PipelineException: The model 'BertModel' is not supported for question-answering. Supported models are ['ConvBertForQuestionAnswering', 'LEDForQuestionAnswering', 'DistilBertForQuestionAnswering', 'AlbertForQuestionAnswering', 'CamembertForQuestionAnswering', 'BartForQuestionAnswering', 'MBartForQuestionAnswering', 'LongformerForQuestionAnswering', 'XLMRobertaForQuestionAnswering', 'RobertaForQuestionAnswering', 'SqueezeBertForQuestionAnswering', 'BertForQuestionAnswering', 'XLNetForQuestionAnsweringSimple', 'FlaubertForQuestionAnsweringSimple', 'MobileBertForQuestionAnswering', 'XLMForQuestionAnsweringSimple', 'ElectraForQuestionAnswering', 'ReformerForQuestionAnswering', 'FunnelForQuestionAnswering', 'LxmertForQuestionAnswering', 'MPNetForQuestionAnswering', 'DebertaForQuestionAnswering', 'DebertaV2ForQuestionAnswering', 'IBertForQuestionAnswering']
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
no error
<!-- A clear and concise description of what you would expect to happen. -->


DeBERTa Fast Tokenizer

Good Second Issue

Issues that are more difficult to do than "Good First" issues - give it a try if you want!

Hi, I am interested in using the DeBERTa model that was recently implemented here and incorporating it into [FARM](https://github.com/deepset-ai/FARM) so that it can also be used in open-domain QA settings through [Haystack](https://github.com/deepset-ai/haystack).

Just wondering why there's only a Slow Tokenizer implemented for DeBERTa and wondering if there are plans to create the Fast Tokenizer too. Thanks in advance!

Hi @stefan-it! Wondering if you might have any insight on this?


Wav2Vec fine code

# 🚀 Feature request

@patrickvonplaten

Hi, I have the following data set I want to use to fine tune Wav2Vec:
[cv-valid-train.zip](https://github.com/huggingface/transformers/files/6074839/cv-valid-train.zip)

I'm using the current transformers library from github (4.4.0 dev).
And I wrote the following code based on the code in the PR https://github.com/huggingface/transformers/pull/10145:
1. ctc_trainer.py
```python
from typing import Dict, Union, Any

import torch
from transformers import Trainer

class CTCTrainer(Trainer):
    def training_step(self, model: torch.nn.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) -> torch.Tensor:
        """
        Perform a training step on a batch of inputs.
        Subclass and override to inject custom behavior.
        Args:
            model (:obj:`nn.Module`):
                The model to train.
            inputs (:obj:`Dict[str, Union[torch.Tensor, Any]]`):
                The inputs and targets of the model.
                The dictionary will be unpacked before being fed to the model. Most models expect the targets under the
                argument :obj:`labels`. Check your model's documentation for all accepted arguments.
        Return:
            :obj:`torch.Tensor`: The tensor with training loss on this batch.
        """

        model.train()
        inputs = self._prepare_inputs(inputs)
        loss = self.compute_loss(model, inputs)

        if self.args.n_gpu > 1:
            if model.module.config.ctc_loss_reduction == "mean":
                loss = loss.mean()
            elif model.module.config.ctc_loss_reduction == "sum":
                loss = loss.sum() / (inputs["labels"] >= 0).sum()
            else:
                raise ValueError(f"{model.config.ctc_loss_reduction} is not valid. Choose one of ['mean', 'sum']")

        if self.args.gradient_accumulation_steps > 1:
            loss = loss / self.args.gradient_accumulation_steps

        loss.backward()

        return loss.detach()
```

2. data_collector.py
```python
from dataclasses import dataclass
from typing import Union, Optional, List, Dict

import torch
from transformers import Wav2Vec2Processor

@dataclass
class DataCollatorCTCWithPadding:
    """
    Data collator that will dynamically pad the inputs received.
    Args:
        processor (:class:`~transformers.Wav2Vec2Processor`)
            The processor used for proccessing the data.
        padding (:obj:`bool`, :obj:`str` or :class:`~transformers.tokenization_utils_base.PaddingStrategy`, `optional`, defaults to :obj:`True`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
            among:
            * :obj:`True` or :obj:`'longest'`: Pad to the longest sequence in the batch (or no padding if only a single
              sequence if provided).
            * :obj:`'max_length'`: Pad to a maximum length specified with the argument :obj:`max_length` or to the
              maximum acceptable input length for the model if that argument is not provided.
            * :obj:`False` or :obj:`'do_not_pad'` (default): No padding (i.e., can output a batch with sequences of
              different lengths).
        max_length (:obj:`int`, `optional`):
            Maximum length of the ``input_values`` of the returned list and optionally padding length (see above).
        max_length_labels (:obj:`int`, `optional`):
            Maximum length of the ``labels`` returned list and optionally padding length (see above).
        pad_to_multiple_of (:obj:`int`, `optional`):
            If set will pad the sequence to a multiple of the provided value.
            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
            7.5 (Volta).
    """

    processor: Wav2Vec2Processor
    padding: Union[bool, str] = True
    max_length: Optional[int] = None
    max_length_labels: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    pad_to_multiple_of_labels: Optional[int] = None

    def __call__(self, features: List[Dict[str, Union[List[int], torch.Tensor]]]) -> Dict[str, torch.Tensor]:
        # split inputs and labels since they have to be of different lenghts and need
        # different padding methods
        input_features = [{"input_values": feature["input_values"]} for feature in features]
        label_features = [{"input_ids": feature["labels"]} for feature in features]

        batch = self.processor.pad(
            input_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors="pt",
        )
        with self.processor.as_target_processor():
            labels_batch = self.processor.pad(
                label_features,
                padding=self.padding,
                max_length=self.max_length_labels,
                pad_to_multiple_of=self.pad_to_multiple_of_labels,
                return_tensors="pt",
            )

        # replace padding with -100 to ignore loss correctly
        labels = labels_batch["input_ids"].masked_fill(labels_batch.attention_mask.ne(1), -100)

        batch["labels"] = labels

        return batch
```

3. fine tune model.py
```python
from pathlib import Path

import datasets
import librosa
import numpy
import pandas
import torch
from sklearn.model_selection import train_test_split
from torch.utils.data import TensorDataset
from tqdm import tqdm
from transformers import Wav2Vec2ForCTC, Wav2Vec2Processor, TrainingArguments

from ctc_trainer import CTCTrainer
from data_collector import DataCollatorCTCWithPadding

def map_to_array(batch):
    input_audio, _ = librosa.load(
        Path("__file__").parents[0].joinpath(batch["filename"]), sr=16000)
    return input_audio

def convert_to_dataset_torch(x: pandas.DataFrame, y: pandas.DataFrame) -> TensorDataset:
    input_values = []
    labels = []
    for _, row in tqdm(x.iterrows(), total=x.shape[0]):
        input_values.append(row["input_values"])
    for _, row in tqdm(y.iterrows(), total=y.shape[0]):
        labels.append(row["labels"])
    return TensorDataset(torch.cat(input_values, dim=0), torch.cat(labels, dim=0))

if __name__ == '__main__':
    dataset = pandas.read_csv(Path(__file__).parents[0].joinpath("cv-valid-train.csv"))
    X_train, X_test, y_train, y_test = train_test_split(dataset[["filename"]], dataset[["text"]], test_size=0.2,
                                                        random_state=42)
    X_train, X_validation, y_train, y_validation = train_test_split(X_train, y_train, test_size=0.2, random_state=42)

    model = Wav2Vec2ForCTC.from_pretrained("facebook/wav2vec2-base-960h")
    processor = Wav2Vec2Processor.from_pretrained("facebook/wav2vec2-base-960h")
    wer_metric = datasets.load_metric("wer")

    X_train["speech"] = X_train.apply(map_to_array, axis=1)
    X_train["input_values"] = X_train.apply(lambda row: processor(row["speech"], sampling_rate=16000).input_values,
                                            axis=1)
    X_validation["speech"] = X_validation.apply(map_to_array, axis=1)
    X_validation["input_values"] = X_validation.apply(
        lambda row: processor(row["speech"], sampling_rate=16000).input_values,
        axis=1)
    X_test["speech"] = X_test.apply(map_to_array, axis=1)
    X_test["input_values"] = X_test.apply(lambda row: processor(row["speech"], sampling_rate=16000).input_values,
                                          axis=1)
    with processor.as_target_processor():
        y_train["labels"] = y_train.apply(lambda row: processor(row["text"]).input_ids, axis=1)
        y_validation["labels"] = y_validation.apply(lambda row: processor(row["text"]).input_ids, axis=1)
        y_test["labels"] = y_test.apply(lambda row: processor(row["text"]).input_ids, axis=1)

    data_collator = DataCollatorCTCWithPadding(processor=processor, padding=True)

    def compute_metrics(pred):
        pred_logits = pred.predictions
        pred_ids = numpy.argmax(pred_logits, axis=-1)

        pred.label_ids[pred.label_ids == -100] = 0

        pred_str = processor.batch_decode(pred_ids)
        # we do not want to group tokens when computing the metrics
        label_str = processor.batch_decode(pred.label_ids, group_tokens=False)

        wer = wer_metric.compute(predictions=pred_str, references=label_str)

        return {"wer": wer}

    training_args = TrainingArguments(
        output_dir='./results',  # output directory
        num_train_epochs=2,  # total number of training epochs
        per_device_train_batch_size=16,  # batch size per device during training
        per_device_eval_batch_size=64,  # batch size for evaluation
        warmup_steps=500,  # number of warmup steps for learning rate scheduler
        weight_decay=0.01,  # strength of weight decay
        logging_dir='./logs',  # directory for storing logs
        logging_steps=10,
    )

    trainer = CTCTrainer(
        model=model,
        data_collator=data_collator,
        args=training_args,
        compute_metrics=compute_metrics,
        train_dataset=convert_to_dataset_torch(X_train, y_train),
        eval_dataset=convert_to_dataset_torch(X_validation, y_validation),
        tokenizer=processor.feature_extractor,
    )

    trainer.train()
```

I'm unable in the method ```convert_to_dataset_torch``` to create TensorDataset. I get the following error:
TypeError: expected Tensor as element 0 in argument 0, but got numpy.ndarray

1. How can I convert the 2d numpy to torch?
2. How can I control argument such  n_gpu and gradient_accumulation_steps?
4. What is model.module.config.ctc_loss_reduction, How It can be controled, and what is best for ASR task?
3. Is there any remorks over the code?


[T5] Fix speed degradation bug t5

# What does this PR do?

Checking every value of a tensor for `inf` is expensive. This was added to T5 to allow for fp16 training, but should then also be used when the model is in fp16 to not slow down normal fp32 mode.

Using @dsgissin  script:

```python
device = torch.device('cuda:0') if torch.cuda.is_available() else torch.device('cpu')
print(f"Using device: {device}")

t5_tokenizer = T5TokenizerFast.from_pretrained('t5-base')
t5_model = T5ForConditionalGeneration.from_pretrained('t5-base')
t5_model = t5_model.to(device)

t5_input_ids = t5_tokenizer("summarize: studies have shown that owning a dog is good for you ", return_tensors="pt").input_ids  # Batch size 1
t5_input_ids = t5_input_ids.to(device)

import time
import numpy as np
N = 100
times = []
for _ in range(N):
  start = time.time()
  t5_outputs = t5_model.generate(t5_input_ids)
  end = time.time()
  times.append(end-start)
print(f"transformers version: {transformers_version}")
print(f"torch version: {torch_version}")
print(f"{1000*np.mean(times):.0f} ms \u00B1 {1000*np.std(times):.2f} ms per loop (mean \u00B1 std of {N} runs)")
```

with:

- Python 3.8.5
- PyTorch 1.7.1
- CUDA 11.1 on a NVIDIA V100 GPU

The time was improved from:
```441 ms ± 41.67 ms per loop (mean ± std of 100 runs)```
to
```388 ms ± 44.75 ms per loop (mean ± std of 100 runs)```


Albert quantized

I use onnxruntime to optimize and quantize transformers model 'albert-base-v2' ,but the quantized result is different from original result,so,Does it support transformers albert quantized right now?