[Wav2Vec2] Improve SpecAugment function by converting numpy based fun…

…ction to pytorch based function

Implements #10459

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Generate can return cross-attention weights too

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #10335

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
@patrickvonplaten


Model Weights Fail to Load from Pre-Trained Model when Using `tf.name_scope`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.0
- Platform: Darwin-19.2.0-x86_64-i386-64bit
- Python version: 3.6.6
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

@LysandreJik @jplu

I can't load the pre-trained weights from BERT or Roberta when using a `tf.name_scope`.
Without defining a name scope, the code runs as expected.
```
text_model= TFBertModel.from_pretrained('bert-base-uncased')
Some layers from the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['nsp___cls', 'mlm___cls']
- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the layers of TFBertModel were initialized from the model checkpoint at bert-base-uncased.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.

```
By adding a name_scope, however, I'll get a warning, indicating that the pre-trained weights are not loaded.
```
import tensorflow as tf
from transformers import TFBertModel
with tf.name_scope("Model"):
    text_model_2 = TFBertModel.from_pretrained('bert-base-uncased')
Some layers from the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['nsp___cls', 'mlm___cls', 'bert/encoder/layer_._2/attention/self/key/bias:0', 'bert/encoder/layer_._7/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._4/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._6/attention/output/dense/kernel:0', 'bert/encoder/layer_._9/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._6/attention/self/key/bias:0', 'bert/encoder/layer_._5/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._3/output/LayerNorm/gamma:0', 'bert/encoder/layer_._5/attention/self/value/kernel:0', 'bert/encoder/layer_._9/attention/self/key/bias:0', 'bert/encoder/layer_._6/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._7/intermediate/dense/bias:0', 'bert/encoder/layer_._4/attention/self/value/kernel:0', 'bert/encoder/layer_._11/attention/self/key/bias:0', 'bert/encoder/layer_._5/attention/self/key/bias:0', 'bert/encoder/layer_._4/attention/self/query/bias:0', 'bert/encoder/layer_._7/output/dense/kernel:0', 'bert/encoder/layer_._8/output/dense/bias:0', 'bert/encoder/layer_._8/intermediate/dense/bias:0', 'bert/encoder/layer_._9/output/LayerNorm/gamma:0', 'bert/encoder/layer_._2/attention/self/key/kernel:0', 'bert/encoder/layer_._0/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._1/intermediate/dense/bias:0', 'bert/encoder/layer_._4/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._1/attention/self/key/kernel:0', 'bert/encoder/layer_._1/attention/self/query/kernel:0', 'bert/encoder/layer_._1/attention/output/dense/bias:0', 'bert/encoder/layer_._6/output/dense/kernel:0', 'bert/encoder/layer_._8/attention/self/key/kernel:0', 'bert/encoder/layer_._8/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._5/output/LayerNorm/beta:0', 'bert/encoder/layer_._1/attention/self/value/bias:0', 'bert/encoder/layer_._11/output/dense/kernel:0', 'bert/encoder/layer_._8/attention/output/dense/bias:0', 'bert/encoder/layer_._6/intermediate/dense/kernel:0', 'bert/encoder/layer_._8/attention/self/value/kernel:0', 'bert/encoder/layer_._7/attention/self/value/kernel:0', 'bert/encoder/layer_._11/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._5/attention/self/value/bias:0', 'bert/encoder/layer_._9/attention/output/dense/kernel:0', 'bert/encoder/layer_._1/attention/self/key/bias:0', 'bert/encoder/layer_._4/attention/output/dense/bias:0', 'bert/encoder/layer_._1/intermediate/dense/kernel:0', 'bert/encoder/layer_._6/output/LayerNorm/gamma:0', 'bert/encoder/layer_._2/intermediate/dense/kernel:0', 'bert/encoder/layer_._4/attention/self/query/kernel:0', 'bert/encoder/layer_._4/output/LayerNorm/gamma:0', 'bert/encoder/layer_._0/attention/self/query/bias:0', 'bert/encoder/layer_._11/attention/output/dense/bias:0', 'bert/encoder/layer_._2/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._10/attention/self/value/bias:0', 'bert/encoder/layer_._2/intermediate/dense/bias:0', 'bert/encoder/layer_._9/output/LayerNorm/beta:0', 'bert/encoder/layer_._0/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._3/attention/output/dense/bias:0', 'bert/encoder/layer_._2/attention/self/value/kernel:0', 'bert/encoder/layer_._3/output/dense/kernel:0', 'bert/encoder/layer_._1/attention/self/query/bias:0', 'bert/encoder/layer_._6/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._10/attention/output/dense/kernel:0', 'bert/encoder/layer_._3/intermediate/dense/kernel:0', 'bert/encoder/layer_._4/output/dense/kernel:0', 'bert/encoder/layer_._6/attention/self/key/kernel:0', 'bert/encoder/layer_._0/output/LayerNorm/beta:0', 'bert/encoder/layer_._7/attention/self/query/kernel:0', 'bert/encoder/layer_._1/output/dense/kernel:0', 'bert/encoder/layer_._0/output/dense/bias:0', 'bert/encoder/layer_._8/attention/output/dense/kernel:0', 'bert/encoder/layer_._11/attention/self/query/kernel:0', 'bert/encoder/layer_._9/attention/self/value/kernel:0', 'bert/encoder/layer_._2/attention/self/query/kernel:0', 'bert/encoder/layer_._4/intermediate/dense/bias:0', 'bert/encoder/layer_._4/intermediate/dense/kernel:0', 'bert/encoder/layer_._0/attention/self/key/kernel:0', 'bert/encoder/layer_._8/attention/self/query/bias:0', 'bert/encoder/layer_._5/attention/output/dense/bias:0', 'bert/encoder/layer_._10/attention/output/dense/bias:0', 'bert/encoder/layer_._11/attention/self/key/kernel:0', 'bert/encoder/layer_._2/output/LayerNorm/gamma:0', 'bert/encoder/layer_._10/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._3/attention/self/query/kernel:0', 'bert/encoder/layer_._10/intermediate/dense/kernel:0', 'bert/encoder/layer_._10/attention/self/query/bias:0', 'bert/encoder/layer_._7/output/LayerNorm/beta:0', 'bert/encoder/layer_._6/intermediate/dense/bias:0', 'bert/encoder/layer_._3/attention/self/key/kernel:0', 'bert/encoder/layer_._8/intermediate/dense/kernel:0', 'bert/encoder/layer_._5/intermediate/dense/kernel:0', 'bert/encoder/layer_._6/output/dense/bias:0', 'bert/encoder/layer_._0/attention/self/query/kernel:0', 'bert/encoder/layer_._6/attention/self/query/bias:0', 'bert/encoder/layer_._7/attention/output/dense/kernel:0', 'bert/encoder/layer_._8/output/LayerNorm/beta:0', 'bert/encoder/layer_._9/attention/self/query/bias:0', 'bert/encoder/layer_._3/output/dense/bias:0', 'bert/encoder/layer_._11/intermediate/dense/bias:0', 'bert/encoder/layer_._4/attention/output/dense/kernel:0', 'bert/encoder/layer_._6/output/LayerNorm/beta:0', 'bert/encoder/layer_._5/output/dense/kernel:0', 'bert/encoder/layer_._3/attention/self/value/kernel:0', 'bert/encoder/layer_._8/output/LayerNorm/gamma:0', 'bert/encoder/layer_._1/attention/output/dense/kernel:0', 'bert/encoder/layer_._11/output/dense/bias:0', 'bert/encoder/layer_._0/output/LayerNorm/gamma:0', 'bert/encoder/layer_._10/output/LayerNorm/beta:0', 'bert/encoder/layer_._0/intermediate/dense/bias:0', 'bert/encoder/layer_._9/output/dense/bias:0', 'bert/encoder/layer_._2/attention/self/value/bias:0', 'bert/encoder/layer_._5/output/LayerNorm/gamma:0', 'bert/encoder/layer_._1/output/dense/bias:0', 'bert/encoder/layer_._0/attention/self/value/kernel:0', 'bert/encoder/layer_._7/attention/output/dense/bias:0', 'bert/encoder/layer_._10/output/dense/bias:0', 'bert/encoder/layer_._11/attention/self/value/kernel:0', 'bert/encoder/layer_._3/intermediate/dense/bias:0', 'bert/encoder/layer_._8/attention/self/query/kernel:0', 'bert/encoder/layer_._10/intermediate/dense/bias:0', 'bert/encoder/layer_._6/attention/self/value/kernel:0', 'bert/encoder/layer_._5/attention/output/dense/kernel:0', 'bert/encoder/layer_._9/intermediate/dense/kernel:0', 'bert/encoder/layer_._4/attention/self/value/bias:0', 'bert/encoder/layer_._4/output/dense/bias:0', 'bert/encoder/layer_._5/attention/output/LayerNorm/beta:0', 'bert/embeddings/LayerNorm/gamma:0', 'bert/embeddings/position_embeddings/embeddings:0', 'bert/encoder/layer_._4/attention/self/key/kernel:0', 'bert/encoder/layer_._7/attention/self/query/bias:0', 'bert/encoder/layer_._10/attention/self/query/kernel:0', 'bert/encoder/layer_._10/attention/self/key/kernel:0', 'bert/encoder/layer_._11/attention/self/value/bias:0', 'bert/encoder/layer_._2/attention/self/query/bias:0', 'bert/encoder/layer_._4/attention/self/key/bias:0', 'bert/encoder/layer_._7/attention/self/key/kernel:0', 'bert/encoder/layer_._11/intermediate/dense/kernel:0', 'bert/encoder/layer_._3/attention/self/value/bias:0', 'bert/pooler/dense/kernel:0', 'bert/encoder/layer_._2/output/dense/bias:0', 'bert/encoder/layer_._7/intermediate/dense/kernel:0', 'bert/encoder/layer_._8/attention/self/key/bias:0', 'bert/embeddings/word_embeddings/weight:0', 'bert/encoder/layer_._11/output/LayerNorm/beta:0', 'bert/encoder/layer_._9/attention/self/value/bias:0', 'bert/embeddings/token_type_embeddings/embeddings:0', 'bert/encoder/layer_._1/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._6/attention/self/value/bias:0', 'bert/pooler/dense/bias:0', 'bert/encoder/layer_._8/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._1/output/LayerNorm/gamma:0', 'bert/encoder/layer_._9/intermediate/dense/bias:0', 'bert/encoder/layer_._1/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._9/output/dense/kernel:0', 'bert/encoder/layer_._0/output/dense/kernel:0', 'bert/encoder/layer_._3/output/LayerNorm/beta:0', 'bert/encoder/layer_._7/output/LayerNorm/gamma:0', 'bert/encoder/layer_._11/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._3/attention/output/dense/kernel:0', 'bert/encoder/layer_._3/attention/self/key/bias:0', 'bert/encoder/layer_._10/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._2/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._6/attention/output/dense/bias:0', 'bert/encoder/layer_._7/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._0/intermediate/dense/kernel:0', 'bert/encoder/layer_._10/attention/self/value/kernel:0', 'bert/encoder/layer_._5/attention/self/key/kernel:0', 'bert/encoder/layer_._10/attention/self/key/bias:0', 'bert/encoder/layer_._9/attention/self/key/kernel:0', 'bert/encoder/layer_._1/attention/self/value/kernel:0', 'bert/encoder/layer_._7/output/dense/bias:0', 'bert/encoder/layer_._11/output/LayerNorm/gamma:0', 'bert/encoder/layer_._0/attention/self/key/bias:0', 'bert/encoder/layer_._3/attention/output/LayerNorm/beta:0', 'bert/encoder/layer_._11/attention/output/dense/kernel:0', 'bert/encoder/layer_._7/attention/self/key/bias:0', 'bert/encoder/layer_._3/attention/output/LayerNorm/gamma:0', 'bert/encoder/layer_._10/output/dense/kernel:0', 'bert/encoder/layer_._10/output/LayerNorm/gamma:0', 'bert/encoder/layer_._5/output/dense/bias:0', 'bert/encoder/layer_._9/attention/output/dense/bias:0', 'bert/encoder/layer_._11/attention/self/query/bias:0', 'bert/encoder/layer_._2/output/LayerNorm/beta:0', 'bert/encoder/layer_._7/attention/self/value/bias:0', 'bert/encoder/layer_._1/output/LayerNorm/beta:0', 'bert/encoder/layer_._5/intermediate/dense/bias:0', 'bert/embeddings/LayerNorm/beta:0', 'bert/encoder/layer_._9/attention/self/query/kernel:0', 'bert/encoder/layer_._2/output/dense/kernel:0', 'bert/encoder/layer_._0/attention/output/dense/kernel:0', 'bert/encoder/layer_._2/attention/output/dense/bias:0', 'bert/encoder/layer_._4/output/LayerNorm/beta:0', 'bert/encoder/layer_._0/attention/self/value/bias:0', 'bert/encoder/layer_._8/output/dense/kernel:0', 'bert/encoder/layer_._2/attention/output/dense/kernel:0', 'bert/encoder/layer_._6/attention/self/query/kernel:0', 'bert/encoder/layer_._0/attention/output/dense/bias:0', 'bert/encoder/layer_._3/attention/self/query/bias:0', 'bert/encoder/layer_._5/attention/self/query/bias:0', 'bert/encoder/layer_._5/attention/self/query/kernel:0', 'bert/encoder/layer_._8/attention/self/value/bias:0', 'bert/encoder/layer_._9/attention/output/LayerNorm/gamma:0']
- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the layers of TFBertModel were initialized from the model checkpoint at bert-base-uncased.
If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.
```
The problem I think is rooted in https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_utils.py#L487.


ONNX Training for Transformers

# 🚀 Feature request

Is there a script for ONNX training of transformers on glue tasks ? If so, did anyone benchmark the training times ? If not, I can contribute.

## Motivation

Faster training.


Pipeline's QnA and run_qa predictions do not match

## Environment info

- `transformers` version: 4.3.0 or 4.4.0dev (tested in both versions)
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
pipelines: @LysandreJik
maintained examples (run_qa.py): @sgugger, @patil-suraj

## Information
I have noticed that the `pipeline("question-answering)` and `run_qa.py `evaluation functions (prediction + post_process) not only are built differently but also yield different results affecting the precision/recall numbers, `pipeline` being the worse performer. I ran the test on 2 different environments and got the same results.  Given the same exact model, the module and the script may affect precision/recall numbers up to 5%. The issue may manifest itself only in a set with negatives as I have not run a test on a set where negatives are not possible.

The problem arises when using:
* [X] the official example scripts:
`run_qa.py` and `pipeline`

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: SQuAD V2

## To reproduce
Steps to reproduce the behavior:

1. Take any QnA model and execute `run_qa.py`on the SQuAD V2 validation set
2. Get predictions on the same set via `pipeline` with the same model and parameters
3. Compare

Here is my notebook to follow the logic and see the results on a particular QnA model.

[Colab Notebook](https://colab.research.google.com/drive/1GMetvI6e0pkUPUHiNX6f46_TBO0LUKre?usp=sharing
)
## Expected behavior
The results need to match. `run_qa.py` code appears to be optimized for training and evaluation at training. Most researchers report QnA model performance on a script like this. The inference module `pipeline` should match what `run_qa.py` is producing.

### more observations:

1.  Pipeline produces more NULL answers than `run_qa` (Highest Concern)
2. About half the non-null answers do not match exactly (45%). Most of those are off by a character or so
3. More than negligible number of records do not match closely
4. pipeline produces shorter answers,
5. pipeline does not take care of apostrophe cases
6. pipeline answers may be in separate places in the context.


Fix typos

# What does this PR do?

I fixed a couple of typos in comments.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.


Smp grad accum

# What does this PR do?

This PR adds support for gradient accumulation in `SageMakerTrainer`. It has been tested on the glue script with success (with and without gradient accumulation passed along).


remap MODEL_FOR_QUESTION_ANSWERING_MAPPING classes to names auto-generated file

As discussed in https://github.com/huggingface/transformers/issues/10467 currently Trainer loads `models.auto.modeling_auto` which loads **all** modeling files, which is absolutely not needed most of the time. At times it could have unwanted side-effects, loading modeling files that require some 3rd party modules that could be a problem.

Similar to the autogenerated 3rd party version numbers look up dict, this PR autogenerates the names of the classes dict for `MODEL_FOR_QUESTION_ANSWERING_MAPPING` which can then be quickly loaded.

This of course can be extended in the future to generate other structures if need be.

@sgugger, @LysandreJik

Fixes: https://github.com/huggingface/transformers/issues/10467


Trainer not logging to WandB in SageMaker

- `transformers` version: 4.3.0
- wandb version: 0.10.20
- Platform: SageMaker hosted training with PyTorch estimator.
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

@stas00 @sgugger

I am using a SageMaker training environment to train `BertForSequenceClassification`. To do this, I'm passing the model into a `Trainer` instance and calling `trainer.train()`.

To train in SageMaker, I am using a PyTorch estimator:

```
estimator = PyTorch(
                    entry_point='train_classifier.py',
                    source_dir='./',
                    role=role,
                    sagemaker_session=sagemaker_session,
                    hyperparameters=hp,
                    subnets=subnets,
                    security_group_ids=sec_groups,
                    framework_version='1.6.0',
                    py_version='py3',
                    instance_count=1,
                    instance_type=instance_type,
                    dependencies=[ '../lib', '../db_conn'],
                    use_spot_instances=False,
                    volume_size=100,
                    #max_wait=max_wait_time_secs
                    )
estimator.fit()
```
I have tried this with different p2 and p3 instances.

In EC2 or in a SageMaker notebook, this does automated logging of training loss and evaluation loss and metrics to WandB. With the estimator, I get no training logs.

Anything that I manually log to WandB appears in my dashboard. The only info that doesn't show up is whatever used to get logged by the Trainer.

I tried `os.environ["WANDB_DISALBED"] = "false"` in my training script, no luck.


Constrained decoding?

Is it possible to implement constraints on the beam during decoding using a seq2seq model? [NeuroLogic Decoding](https://arxiv.org/abs/2010.12884), [Constrained Abstractive Summarization](https://arxiv.org/abs/2010.12723)

I see that there is a Callback feature in the library - but AFAIK it only lets modify the training state and not the beam.


Corrupted Relative Attention in T5 Decoder

## Environment info
platform: Mac/Ubuntu 14
transformers==2.11.0
torch==1.4.0 (GPU)
python 3.6
I know this is an old version but it supports important experiments in a paper under review. Would appreciate to know what's wrong. I checked the commit log and I don't think any following commits resolve it.

### Who can help
@patrickvonplaten (through slack) @patil-suraj (mentioned below)
Please let me know if there is anything else I can provide!
Thank you!

## Information

I made an artificial binary classification data where the input sequences are near-randomly generated tokens from the T5 vocab. The output sequences are balanced “`answer: correct/restaurant`” (two binary tag words randomly selected). A data sample [can be found here](https://github.com/Slash0BZ/t5-investigation/blob/main/sample_data.txt) in format (`input_seq \t output_seq`). The custom data reader parses this data with T5Tokenizer and is_pretokenized=True ([see here](https://github.com/Slash0BZ/t5-investigation/blob/main/train_t5.py#L129))

I feed the [T5ForConditionalGeneration model (v.2.11.0)](https://github.com/Slash0BZ/t5-investigation/blob/main/overrides.py#L60) with input_ids, lm_labels, and their corresponding attention_masks during training. The model should not learn anything because the sequences are near-random, but in reality, it converges to a zero loss, meaning that the lm_logits from decoder actually attend to future inputs (after `shift_right()`) and knows the label. During evaluation where I hide the binary tag, the model always predicts positive.

## To reproduce

Steps to reproduce the behavior:

1. Use the code in this repo: https://github.com/Slash0BZ/t5-investigation
2. Ran with sample data. I have tried both pre-trained T5-large and also randomly initialized T5-Large ([written like this](https://github.com/Slash0BZ/t5-investigation/blob/main/train_t5.py#L258))

I am not sure if the training data size affects the result. I ran with a training size of 5M. I am happy to provide the full data and a trained model if actual experiments are needed.

## Expected behavior

The training loss converges to near-zero and the lm_logits reflects predictions the same as the output sequence during training. However, in evaluation where the data reader hides the binary tag in the output sequence ([achieve through only providing "answer:" in decoder_input_ids](https://github.com/Slash0BZ/t5-investigation/blob/main/overrides.py#L46)), the prediction is uniform.

I also tried to change the decoder_input_ids. When it is [0, 1525, 10, 2024], the prediction at position 2 is 2024. When it is [0, 1525, 10, 2062], the prediction at position 2 is 2062.

Notes: 1525->"answer", 10->":", 2024->"correct", 2062->"restaurant"


add shift on BartForCausalLM

# What does this PR do?

Fixes  #10480

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patil-suraj
@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[examples] should all examples support the predict stage?

Examples

Which is related to examples in general

This is part of the ongoing effort to sync the example scripts.

In https://github.com/huggingface/transformers/issues/10437#issuecomment-789090858 it was flagged that some scripts have test/predict, whereas others don't.

Should we:
A. have all scripts have train/eval/predict
B. only have predict where it's desired

@sgugger, @patil-suraj, @LysandreJik


feat(docs): navigate with left/right arrow keys

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Enables docs navigation with left/right arrow keys. It can be useful for the ones who navigate with keyboard a lot.
More info : https://github.com/sphinx-doc/sphinx/pull/2064

You can try here : https://174105-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Different result in AutoModelForCausalLM

# 🚀 Feature request
Models inside AutoModelForCausalLM have different behavior on loss calculation.

In BartForCausalLM
there is no shift in loss calculation
https://github.com/huggingface/transformers/blob/b013842244df7be96b8cc841491bd1e35e475e36/src/transformers/models/bart/modeling_bart.py#L1745
```
loss_fct = CrossEntropyLoss()
loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1))
```

In RobertaForCausalLM
A shift is applied before loss calculation
https://github.com/huggingface/transformers/blob/b013842244df7be96b8cc841491bd1e35e475e36/src/transformers/models/roberta/modeling_roberta.py#L944
```
# we are doing next-token prediction; shift prediction scores and input ids by one
shifted_prediction_scores = prediction_scores[:, :-1, :].contiguous()
labels = labels[:, 1:].contiguous()
loss_fct = CrossEntropyLoss()
lm_loss = loss_fct(shifted_prediction_scores.view(-1, self.config.vocab_size), labels.view(-1))
```

## Motivation
I found a mistake when I switched the config from Roberta to BART in AutoModelForCausalLM. It turns out to be different labeling in loss. So, It would be nice to make CausalLM models handle label in the same way, either shift or not.

## Your contribution
I can make a PR to make sure that all the models will have a shift prediction.


Question regarding training of BartForConditionalGeneration

Hello Guys,

I am trying to fine-tune the BART summarization model but due to the lack of big dataset, having some difficulties with the fine-tuning.

Thus, I decided to look at the trainig process of BartForConditionalGeneration model in detail. I came across this article, [Introducing BART](https://sshleifer.github.io/blog_v2/jupyter/2020/03/12/bart.html) from one of the engineers, @sshleifer, at HuggingFace. It says that BartModel was directly fine-tuned for the summarisation task without **any new randomly initialized heads**.

**My question is about this fine-tuning process, especially on CNN-DailyMail dataset. Do you guys fine-tune the entire Bart model or only the decoder or something else?**

I looked at the example [fine-tuning script](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_seq2seq.py) provided on the GitHub but I didn't find anything related to freezing some part of the model.

I also tried to look at the source code of the BartForConditionalGeneration model and observed the following -

Its just adds a [linear layer](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bart/modeling_bart.py#L1207) on top of the BartModel (copy-pasting the `__init__` code here for quick reference).

```
self.model = BartModel(config)
self.register_buffer("final_logits_bias", torch.zeros((1, self.model.shared.num_embeddings)))
self.lm_head = nn.Linear(config.d_model, self.model.shared.num_embeddings, bias=False)
```

At first, I thought these are the new parameters that are being introduced and thus, being trained. Therefore, I tried the following code to check the number of trainable parameters while keeping the endoer and decoder fixed  -

```
from transformers import BartModel, BartForConditionalGeneration, BartTokenizer

def freeze_params(model):
    for par in model.parameters():
        par.requires_grad = False

model_sum = BartForConditionalGeneration.from_pretrained('facebook/bart-large')
freeze_params(model_sum.get_encoder()) ## freeze the encoder
freeze_params(model_sum.get_decoder()) ## freeze the decoder

model_sum.train() ## set the train mode
train_p = [p for p in model_sum.parameters() if p.requires_grad] ## get the trainable params
print(f'Length of train params in Summarization Model : {len(train_p)}')
```

But this code shows that the list is empty. One thing I can do is to explictly set the `requires_grad=True` for the paramters in the `model_sum.lm_head` and only fine-tune these parameters. But I am curious to understand the original training/fine-tuning process. **It would be of great help to me if you guys could answer my question.**

P.S. - Love the HuggingFace library.

Thanks,
<br />
Naman


generate() decoder_input_ids padding

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.0.dev0
- Platform: Linux-5.4.0-66-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes (but unrelevant)
- Using distributed or parallel set-up in script?: No

### Who can help

As it is a generation issue:
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

But also may benefit from better documentation:

Documentation: @sgugger

## Information

When using the `generate()` method from `generation_utils` for Bart (although other models will probably behave the same by checking the code), I am using `decoder_input_ids` to inform of the first tokens each sample in the batch should start with. Although not stated in the documentation (see [post in HF forum](https://discuss.huggingface.co/t/generate-continuation-for-seq2seq-models/)), `generate()` can take `decoder_input_ids` as the forward method would.

This works fine with batch size equal to one or if the `decoder_input_ids` would all have same length and not require padding. However, when padding is involved, the `generate` method does not ignore the padding tokens in order to generate text for each sample in the batch, generating instead after the padding tokens in `decoder_input_ids`.

## To reproduce

Steps to reproduce the behavior:

1. Load your favorite bart model for generation.
2. Prepare your `inputs_ids` for the encoder and the `decoder_input_ids` for your decoder, using sequences of different length.
3. Check the generated text.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
from transformers import AutoConfig, AutoModelForSeq2SeqLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-cnn-6-6")

model = AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distilbart-cnn-6-6")

model.to("cuda:0")
model.eval()
inputs_text = ['Hugging Face is taking its first step into machine translation this week with the release of more than 1,000 models. Researchers trained models using unsupervised learning and the Open Parallel Corpus (OPUS). OPUS is a project undertaken by the University of Helsinki and global partners to gather and open-source a wide variety of language data sets, particularly for low resource languages. Low resource languages are those with less training data than more commonly used languages like English.', 'Hugging Face has announced the close of a $15 million series A funding round led by Lux Capital, with participation from Salesforce chief scientist Richard Socher and OpenAI CTO Greg Brockman, as well as Betaworks and A.Capital.']
decoder_text = ['</s><s>Hugging Face released', '</s><s>Hugging Face closed a']
inputs = tokenizer(inputs_text, return_tensors = 'pt', padding=True)
inputs_decoder = tokenizer(decoder_text, return_tensors = 'pt', padding=True, add_special_tokens = False)
generated_tokens = model.generate(inputs['input_ids'].to(model.device),
                                  attention_mask=inputs['attention_mask'].to(model.device),
                                  decoder_input_ids = inputs_decoder['input_ids'].to(model.device))
decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)
print('\n'.join(decoded_preds), '\n')
Output:
</s><s>Hugging Face released<pad> models using unsupervised learning and the Open Parallel Corpus (OPUS) Low resource languages are those with less training data than more commonly used languages like English. The project was undertaken by the University of Helsinki and global partners to gather and open-source a wide variety of language data sets.</s>
</s><s>Hugging Face closed a $15 million series A funding round led by Lux Capital. Salesforce chief scientist Richard Socher and OpenAI CTO Greg Brockman participated. Betaworks and A.Capital also took part in the round, with participation from Betaworkorks.</s><pad><pad><pad><pad><pad><pad><pad>
```
In the first generated sentence, the `<pad>` token after released is kept, and would be worse if there was a higher length difference between the decoder inputs.

## Expected behavior

If one gives as `decoder_input_ids` same length sentences, for instance, we remove the a from "Hugging Face closed a":
```
decoder_text = ['</s><s>Hugging Face released', '</s><s>Hugging Face closed']
inputs = tokenizer(inputs_text, return_tensors = 'pt', padding=True)
inputs_decoder = tokenizer(decoder_text, return_tensors = 'pt', padding=True, add_special_tokens = False)
generated_tokens = model.generate(inputs['input_ids'].to(model.device),
                                  attention_mask=inputs['attention_mask'].to(model.device),
                                  decoder_input_ids = inputs_decoder['input_ids'].to(model.device))
decoded_preds = tokenizer.batch_decode(generated_tokens, skip_special_tokens=False)
print('\n'.join(decoded_preds), '\n')
Output:
</s><s>Hugging Face released its first step into machine translation this week. Researchers trained models using unsupervised learning and the Open Parallel Corpus (OPUS) OPUS is a project undertaken by the University of Helsinki and global partners. Low resource languages are those with less training data than more commonly used languages like English.</s>
</s><s>Hugging Face closed a $15 million series A funding round led by Lux Capital. Salesforce chief scientist Richard Socher and OpenAI CTO Greg Brockman were involved. Betaworks and A.Capital also participated in the round, which will take place in New York City.</s><pad><pad><pad><pad><pad>
```
Then the output does not include any `<pad>` token in between the generated text. It would be nice if this would be the same for different length in the decoder_input_ids (ie. ignore the `<pad>` tokens).


Facing NCCL error on Multi-GPU training(on single machine) using run_glue.py script

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.3.2
- Platform: Linux-4.19.0-14-cloud-amd64-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: 4xTesla T4 (GCP)
- Using distributed or parallel set-up in script?: torch.distributed.launch

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): DistilRoberta

The problem arises when using:
* [*] the official example scripts: (give details below)
* [  ] my own modified scripts: (give details below)

The tasks I am working on is:
* [  ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

Regression task with a single output, using BertForSequenceClassification

## To reproduce

Steps to reproduce the behavior:

1.python -m torch.distributed.launch --nproc_per_node 4 /home/run_glue.py   --train_file /home/data/train.csv   --validation_file /home/data/dev.csv   --test_file /home/data/test.csv   --model_name_or_path distilroberta-base   --output_dir /home/model   --num_train_epochs 5   --per_device_train_batch_size 1   --per_device_eval_batch_size 16   --do_train   --do_eval   --fp16   --gradient_accumulation_steps 2   --do_predict   --logging_steps 100   --evaluation_strategy steps   --save_steps 100   --overwrite_output_dir

File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 442, in init_process_group
732793de051f:1895:1925 [1] NCCL INFO transport/shm.cc:101 -> 2
732793de051f:1895:1925 [1] NCCL INFO transport.cc:30 -> 2
732793de051f:1895:1925 [1] NCCL INFO transport.cc:49 -> 2
732793de051f:1895:1925 [1] NCCL INFO init.cc:766 -> 2
732793de051f:1895:1925 [1] NCCL INFO init.cc:840 -> 2
732793de051f:1895:1925 [1] NCCL INFO group.cc:73 -> 2 [Async thread]
    barrier()
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1947, in barrier
Traceback (most recent call last):
  File "/home/run_text_classification.py", line 480, in <module>
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1603729138878/work/torch/lib/c10d/ProcessGroupNCCL.cpp:784, unhandled system error, NCCL version 2.7.8
    main()
  File "/home/run_text_classification.py", line 163, in main
    model_args, data_args, training_args = parser.parse_args_into_dataclasses()
  File "/opt/conda/lib/python3.7/site-packages/transformers/hf_argparser.py", line 180, in parse_args_into_dataclasses
    obj = dtype(**inputs)
  File "<string>", line 60, in __init__
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 478, in __post_init__
    if is_torch_available() and self.device.type != "cuda" and self.fp16:
  File "/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py", line 1346, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/lib/python3.7/site-packages/transformers/training_args.py", line 583, in device
    return self._setup_devices

732793de051f:1897:1927 [3] include/shm.h:28 NCCL WARN Call to posix_fallocate failed : No space left on device
  File "/opt/conda/lib/python3.7/site-packages/transformers/file_utils.py", line 1336, in __get__
732793de051f:1897:1927 [3] NCCL INFO include/shm.h:41 -> 2

732793de051f:1897:1927 [3] include/shm.h:48 NCCL WARN Error while creating shared memory segment nccl-shm-recv-b3d54cebe4167a34-0-2-3 (size 9637888)
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Expected model training to proceed smoothly using 4xGPU. When I run the said script with nproc_per_node=1(or even 2), it runs smoothly but setting it as 4 gives strange errors.

After updating to 1.9.0 I face a different error:

RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:832, unhandled system error, NCCL version 2.7.8 ncclSystemError: System call (socket, malloc, munmap, etc) failed.
<!-- A clear and concise description of what you would expect to happen. -->


The size of CoNLL-2003 is not consistant with the official release.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: -
- Python version: -
- PyTorch version (GPU?): -
- Tensorflow version (GPU?): -
- Using GPU in script?: -
- Using distributed or parallel set-up in script?: -

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->@sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: CoNLL-2003

## To reproduce

Steps to reproduce the behavior:

1. just run the code

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The official release of CoNLL-2003 is:
# train 14987
# dev 3466
# test 3684
While CoNLL-2003 in datasets is:
# train 14041
# dev 3250
# test 3453

Wish for your reply~ Thank you!


Fixes compatibility bug when using grouped beam search and constrained decoding together

Fixes #10415

## Who can review?

@patrickvonplaten


Continue pre-training using the example code "run_mlm.py"

## Environment info
- `transformers` version: 4.3.3
- Platform: Linux-5.4.0-65-generic-x86_64-with-glibc2.10  (Ubuntu 18.04)
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?:  Yes
- Using distributed or parallel set-up in script?:  No

### Who can help
- tokenizers: @LysandreJik
- maintained examples (not research project or legacy): @sgugger, @patil-suraj

## Information

Model I am using (Bert, XLNet ...): albert-xlarge-v2

The problem arises when using:
  transformers/examples/language-modeling/run_mlm.py
  (https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py)

The tasks I am working on is:
  continue pre-training on a specific dataset (glue/sst2)

## To reproduce

Steps to reproduce the behavior:
```=
CUDA_VISIBLE_DEVICES=0 python run_mlm.py \
    --model_name_or_path albert-xlarge-v2 \
    --dataset_name "glue" \
    --dataset_config_name "sst2" \
    --do_train \
    --do_eval \
    --output_dir ckpt/pre_training/glue
```

### Error message
```=
Traceback (most recent call last):
  File "src/run_mlm.py", line 447, in <module>
    main()
  File "src/run_mlm.py", line 353, in main
    tokenized_datasets = datasets.map(
  File "/home/robotlab/anaconda3/envs/thesis/lib/python3.8/site-packages/datasets/dataset_dict.py", line 369, in map
    {
  File "/home/robotlab/anaconda3/envs/thesis/lib/python3.8/site-packages/datasets/dataset_dict.py", line 370, in <dictcomp>
    k: dataset.map(
  File "/home/robotlab/anaconda3/envs/thesis/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1120, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/home/robotlab/anaconda3/envs/thesis/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1091, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "src/run_mlm.py", line 351, in tokenize_function
    return tokenizer(examples[text_column_name], return_special_tokens_mask=True)
  File "/home/robotlab/anaconda3/envs/thesis/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 2286, in __call__
    assert isinstance(text, str) or (
AssertionError: text input must of type `str` (single example), `List[str]` (batch or single pretokenized example) or `List[List[str]]` (batch of pretokenized exampl
es).
```

I skipped the message that relates to the model, as there's no problem with the loading of the model.
Dataset is successfully downloaded. There's a warning above the assertion error
```=
03/02/2021 14:03:44 - WARNING - datasets.builder -   Reusing dataset glue (/home/robotlab/.cache/huggingface/datasets/glue/sst2/1.0.0/7c99657241149a24692c402a5c3f34d
4c9f1df5ac2e4c3759fadea38f6cb29c4)
```

## Expected behavior
When I continue pre-training on other datasets such as 'ag_news', 'dbpedia_14', 'imdb', there's no error and everything is fine.
There are also no "dataset_config_name" in these three datasets.
However, there's no error when I use   `dataset_name=wikitext` and `dataset_config_name=wikitext-2-raw-v1`    in  `run_mlm.py`

Judging from the error message above,
it seems like the data format of the SST-2 is wrong so that the datasets can not handled the data correctly.
Any suggestion is highly appreciated!


Issue with converting my own BERT TF2 checkpoint to PyTorch and loading the converted PyTorch checkpoint for training

Hi,

I’m using huggingface to train my own bert model. I have checkpoints which are in TensorFlow2 and I have converted them successfully in PyTorch. The checkpoint conversion script has created **_.bin_** file which is having following subdirectories and pkl file.

![dfa9d353b4e25888726ebae99c19c048aaa9d4e4](https://user-images.githubusercontent.com/12840374/109604765-167cb500-7b4a-11eb-9e52-013762d02608.png)

I wanted to gain some more information on the following points:

- Am’I doing the conversation of TensorFlow checkpoint to PyTorch checkpoint correct? I can see the pre-trained bert-base-uncased model which is hosted on hugging face model repository is just having single [pytorch_model.bin](https://huggingface.co/bert-base-uncased/blob/main/pytorch_model.bin) file.
- How I can load this custom BERT-PyTorch model’s checkpoint using [modeling_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py) script?

Any help or suggestions will be helpful.

Thanks


Needed a feature to convert facebook mbart many - many model to ONNX runtime inorder to reduce the inference time

We need a feature which makes the facebook mbart many - many model to convert to ONNX runtime which reduces the inference time, current mbart many-many model takes 9 secs to translate and we need to quantize it further to reduce the inference time


Question: change location of cache datasets

This is not a bug, it's a question. I don't have too much space allowed in my home dir and I have to store most stuff elsewhere. I noticed that the library caches a lot of stuff under ~/.cache/huggingface (in particular under datasets). How do I change the location of the cache dir? I have `export TRANSFORMERS_CACHE=...` in my run script but that doesn't seem to be it.

I started using some large datasets and whatever caching happens because of these datasets is causing a disk quota issue. I hope it's possible to change the location of the cache dir.

Thanks!


(Sorry I can not visit the forum) BORT question: pre-training-using-knowledge-distillation is better than pre-training-only for downstream tasks?

The paper shows the MLM accuarcy comparision.

![image](https://user-images.githubusercontent.com/4702353/109590213-87ba6900-7b46-11eb-9fea-f046d23abeb2.png)

What is the **downstream tasks'** performance comparison for pre-training-using-knowledge-distillation and pre-training-only to the end?

Thank you very much.


The described function in docs was not implemented in source code

https://github.com/huggingface/transformers/blob/0c2325198fd638e5d1f0c7dcbdd8bf7f14c0ff7d/src/transformers/file_utils.py#L1512

The method of `update` in `ModelOutput` described as below,
https://huggingface.co/transformers/main_classes/output.html?highlight=modeloutput#transformers.file_utils.ModelOutput.update
but the source code will raise an exception when try to use it, which disagree with the docs.


run_ner.py training data file format

# 🚀 Feature request

Would it be possible to support text files as input files for the run_ner.py script similar to what was supported in examples/legacy/token-classification/run_ner.py ? This, I believe is the CoNLL-2003 format. The current version of the run_ner.py script in /examples/token-classification appears to support only json and csv format for training input. Could someone also tell me where I can find example files for aforementioned json and csv files ? Looking at some example files would help me format my labeled data in the required format if supporting text files is not possible.

## Motivation

I have some training data formatted as text files with one token and label per line and would like to use that as input to the run_ner.py script if possible.


modeling files loaded when they aren't being asked to be loaded

```
  File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/trainer_seq2seq.py", line 22, in <module>
    from .trainer import Trainer
  File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/trainer.py", line 65, in <module>
    from .trainer import Trainer
  File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/trainer.py", line 65, in <module>
    from .models.auto.modeling_auto import MODEL_FOR_QUESTION_ANSWERING_MAPPING
      File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/models/auto/modeling_auto.py", line 214, in <module>
from .models.auto.modeling_auto import MODEL_FOR_QUESTION_ANSWERING_MAPPING
  File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/models/auto/modeling_auto.py", line 214, in <module>
    from ..tapas.modeling_tapas import (
    from ..tapas.modeling_tapas import (  File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/models/tapas/modeling_tapas.py", line 51, in <module>

  File "/gpfsdswork/projects/rech/ajs/uiz98zp/stas/transformers-master/src/transformers/models/tapas/modeling_tapas.py", line 51, in <module>
    from torch_scatter import scatter
      File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.1/lib/python3.7/site-packages/torch_scatter/__init__.py", line 12, in <module>
```

Do we have to load `models/tapas/modeling_tapas.py` when we aren't using `tapas`?

There is some unrelated problem that gets triggered by this model loading `torch_scatter`, which is a 3rd party module.

I figured out the solution to the problem it triggered (binary incompatible `torch_scatter`), but still it might be a good idea not to pre-load model files until they are needed.

@LysandreJik, @sgugger


Fix the bug in constructing the all_hidden_states of DeBERTa v2

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR fixes the bug in constructing the `all_hidden_states` of DeBERTa v2. In the master branch, it keeps appending `hidden_states` to `all_hidden_states` which comes from the inputs and is never updated in each layer. This would make `all_hidden_states` a list of duplicated elements. Instead, it should append `output_states` (which is the counterpart of the `hidden_states` in DeBERTa v1) to `all_hidden_states` because it is the real output of each layer.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Tflite conversion error for TFMT5ForConditionalGeneration model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: Version: 4.4.0.dev0
- Platform: Not sure what this means
- Python version: 3
- PyTorch version (GPU?):
- Tensorflow version (GPU?):'2.4.1'
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

t5: @patrickvonplaten
tensorflow: @jplu

## Information

Model I am using (Bert, XLNet ...): TFMT5ForConditionalGeneration (it is already trained)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

converting the model to tflite

```
config = AutoConfig.from_pretrained(
    model_path
#     output_attentions=True,
#     output_hidden_states=True,
#     use_cache=True,
#     return_dict=True
)
tokenizer = AutoTokenizer.from_pretrained(
    model_path
)
model = TFMT5ForConditionalGeneration.from_pretrained(
    model_path,
    from_pt=True,
    config=config
)

conc_func = model.serving.get_concrete_function()
converter = tf.lite.TFLiteConverter.from_concrete_functions([conc_func])
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS,
                                       tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()
print("Converted successfully")

```

## To reproduce

Steps to reproduce the behavior:

1.run the above script on TFMT5ForConditionalGeneration model
2. will get the follow error
```
Exception: /home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/ops/array_ops.py:1047:0: error: 'tf.Reshape' op requires 'shape' to have at most one dynamic dimension, but got multiple dynamic dimensions at indices 0 and 3
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py:201:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/transformers/models/t5/modeling_tf_t5.py:703:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/autograph/operators/control_flow.py:1218:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/autograph/operators/control_flow.py:1165:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/transformers/models/t5/modeling_tf_t5.py:702:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/autograph/operators/control_flow.py:1218:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/autograph/operators/control_flow.py:1165:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/transformers/models/t5/modeling_tf_t5.py:694:0: note: called from
/home/python_user/.pyenv/versions/3.7.2/lib/python3.7/site-packages/tensorflow/python/keras/engine/base_layer.py:1012:0: note: called from
```

When I inspected : t5/modeling_tf_t5.py:703: i got
Line number: code
```
702:            if num_dims_encoder_attention_mask == 2:
703:                encoder_extended_attention_mask = inputs["encoder_attention_mask"][:, None, None, :]
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

convert the model to tflite successfully

<!-- A clear and concise description of what you would expect to happen. -->


[Deepspeed] Allow HF optimizer and scheduler to be passed to deepspeed

# Use HF optimizer and/or scheduler unless specified in deepspeed config

If HF is already creating an optimizer and LR scheduler, we should not try to match that config/implementation in a ds_config.json instead we pass it to deepspeed.initialize(..., lr_scheduler=hf_lr_scheduler)

* [x] This PR checks if ds_config has an optimizer or scheduler, if it does not, calls  `create_optimizer` or `create_cheduler` (after splitting it) to create an optimizer or scheduler.  Then HF optimizer and scheduler are passed it to deepspeed.initialize().
DeepSpeed can handle any optimizer and scheduler if these are passed directly to deepspeed.initialize() as an object.

Due to the chicken-n-egg init problem, the valid combinations are:

| Combos       | HF Scheduler | DS Scheduler |
|--------------|--------------|--------------|
| HF Optimizer | Yes          | Yes          |
| DS Optimizer | No           | Yes          |

but if `cpu_offload` is used all bets are off - we can only use DS optim/sched.

----------
added by @stas00 below:

Added:
* [x] make init_deepspeed support config dict, besides the config file - this makes the testing much easier
* [x] add tests for this PR using this new feature of passing the dict
* [x] various small clean ups
* [x] update the docs
* [x] check for `cpu_offload` - add test
* [x] recode the config overrides to have one true source of values
* [x] tweak one not working test

**blocking event: waiting for a new release 0.3.13 from DeepSpeed.**

@sgugger


Script for squad_v2 for custom data not working

**Environment info**
transformers version: 4.3.3
Platform: Linux-4.15.0-91-generic-x86_64-with-debian-buster-sid
Python version: 3.7.6
Using GPU in script?: True
Using distributed or parallel set-up in script?: True

**Who can help**
@gowtham1997 @patil-suraj

I am running the script from docs to train and evaluate squad_v2 data with custom arguments.
My dataset is as per the structure of squad format with every keys and values properly.
[colabJson2.zip](https://github.com/huggingface/transformers/files/6063805/colabJson2.zip)

```
python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --version_2_with_negative\
  --train_file=/content/colabJson.json \
  --validation_file=/content/dev-v2.0.json \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/debug_squad_16
```

Output of the script i am getting error as

```
Traceback (most recent call last):
  File "run_qa.py", line 507, in <module>
    main()
  File "run_qa.py", line 230, in main
    datasets = load_dataset(extension, data_files=data_files, field="data")
  File "/usr/local/lib/python3.7/dist-packages/datasets/load.py", line 750, in load_dataset
    ds = builder_instance.as_dataset(split=split, ignore_verifications=ignore_verifications, in_memory=keep_in_memory)
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 740, in as_dataset
    map_tuple=True,
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py", line 234, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py", line 234, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/usr/local/lib/python3.7/dist-packages/datasets/utils/py_utils.py", line 172, in _single_map_nested
    return function(data_struct)
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 757, in _build_single_dataset
    in_memory=in_memory,
  File "/usr/local/lib/python3.7/dist-packages/datasets/builder.py", line 831, in _as_dataset
    return Dataset(**dataset_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 250, in __init__
    self.info.features, self.info.features.type, inferred_features, inferred_features.type
ValueError: External features info don't match the dataset:
Got
{'title': Value(dtype='string', id=None), 'paragraphs': [{'qas': [{'question': Value(dtype='string', id=None), 'id': Value(dtype='string', id=None), 'answers': [{'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int64', id=None)}], 'is_impossible': Value(dtype='bool', id=None), 'plausible_answers': [{'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int64', id=None)}]}], 'context': Value(dtype='string', id=None)}]}
with type
struct<paragraphs: list<item: struct<context: string, qas: list<item: struct<answers: list<item: struct<answer_start: int64, text: string>>, id: string, is_impossible: bool, plausible_answers: list<item: struct<answer_start: int64, text: string>>, question: string>>>>, title: string>

but expected something like
{'title': Value(dtype='string', id=None), 'paragraphs': [{'context': Value(dtype='string', id=None), 'qas': [{'answers': [{'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int64', id=None)}], 'question': Value(dtype='string', id=None), 'id': Value(dtype='string', id=None)}]}]}
with type
struct<paragraphs: list<item: struct<context: string, qas: list<item: struct<answers: list<item: struct<answer_start: int64, text: string>>, id: string, question: string>>>>, title: string>
```
In my dataset I don't have any field as **plausible_answers**, **is_impossible** ..
how to match it with expected format when it is already in that format


Add I-BERT to README

This PR adds I-BERT to the readme, it was forgotten.


pass correct head mask to cross-attention layer

# What does this PR do?

 MBart, Blender, and Pegaus models decoder layers pass `layer_head_mask` head mask to cross attention layer, which is incorrect. This PR passes the correct `encoder_layer_head_mask` to cross-attention layer.


How to Reduce the inference time of Facebook/many to many model?

Facebook/many to many model takes 9s on cpu to translate , how to reduce the inference time on cpu ?
It will be helpful if an method is suggested .


[Wav2Vec2] Improve SpecAugment function by converting numpy based function to pytorch based function

# 🚀 Feature request

As can be seen here: https://github.com/huggingface/transformers/blob/11655fafdd42eb56ad94e09ecd84d4dc2d1041ae/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L47,

the function `_compute_mask_indices` (responsible for spec augment) of Wav2Vec2 is written in numpy which means that the function is not GPU compatible.
The function could simply be rewritten in PyTorch, which should make training on GPU faster.

This "Good First Issue" is about converting `_compute_mask_indices` to PyTorch while keeping the same functionality.

## Your contribution

I'm happy to guide the contributor through the PR!


Work towards fixing Flax tests

I still get failures that seem due to missing HTTP artifacts, e.g.

E               OSError: Can't load weights for 'roberta-large'. Make sure that:
E
E               - 'roberta-large' is a correct model identifier listed on 'https://huggingface.co/models'
E
E               - or 'roberta-large' is the correct path to a directory containing a file named pytorch_model.bin.

This is the command I used to run tests:

    RUN_SLOW=true python -m pytest -k flax -n 8 --dist=loadfile -rA -s --make-reports=tests_flax ./tests/ | tee tests_output.txt

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? (<-- this PR fixes existing tests)

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


[Wav2Vec2] Remove unused config

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Removes unused config variable

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @jplu

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


How to Improve inference time of facebook/mbart many to many model?

If we tried to run translation service on facebook mbart many to many on cpu it take 9 secs to translate, how do we reduce the inference time further


[Wav2Vec2FeatureExtractor] smal fixes

# What does this PR do?

This PR adds the `return_attention_mask` argument to `Wav2Vec2FeatureExtractor.__call__` method.


How can I make the logging utils log to a file as well?

# 🚀 Feature request

I want to make the logging utils log to a file in addition to the console. But I can't find an API that lets me add a handler to the logging utils.


pytorch Albert quantization error

I use  huggingface transformers 'albert_chinese_base',but in pytorch quantization,The following problem occurred:
File "test_simple.py", line 186, in <module>
    model_pt_quantized(input_ids=model_inputs["input_ids"], token_type_ids=model_inputs["token_type_ids"], attention_mask=model_inputs["attention_mask"])
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 563, in forward
    output_hidden_states=output_hidden_states,
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 346, in forward
    output_hidden_states,
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 299, in forward
    layer_output = albert_layer(hidden_states, attention_mask, head_mask[layer_index], output_attentions)
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 277, in forward
    attention_output = self.attention(hidden_states, attention_mask, head_mask, output_attentions)
  File "/work/runtime/torch/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/work/runtime/torch/lib/python3.6/site-packages/transformers/modeling_albert.py", line 251, in forward
    self.dense.weight.t()
AttributeError: 'function' object has no attribute 't'


BartForConditionalGeneration breaks with label smoothing loss

## Environment info

- `transformers` version: 4.3.3
- The other parameters are irrelevant

### Who can help

@patrickvonplaten @sgugger

## Information

I apologize for not using the provided template for this issue.

By generating entries with `PreTrainedTokenizer.prepare_seq2seq_batch`, collating with `DataCollatorForSeq2Seq` and
training a `BartForConditionalGeneration` with `Seq2SeqTrainer`, I ran into this particular error message.

```txt
Traceback (most recent call last):
  File "playground.py", line 42, in <module>
    trainer.train()
  File "/Users/mingrui.wang/miniconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer.py", line 940, in train
    tr_loss += self.training_step(model, inputs)
  File "/Users/mingrui.wang/miniconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer.py", line 1304, in training_step
    loss = self.compute_loss(model, inputs)
  File "/Users/mingrui.wang/miniconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer.py", line 1341, in compute_loss
    loss = self.label_smoother(outputs, labels)
  File "/Users/mingrui.wang/miniconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer_pt_utils.py", line 398, in __call__
    nll_loss = log_probs.gather(dim=-1, index=labels)
RuntimeError: Size does not match at dimension 1 expected index [1, 7, 1] to be smaller than src [1, 5, 50265] apart from dimension 2
```

Provided is a script to replicate the error

```python
from torch.utils.data import Dataset
from transformers import (BartForConditionalGeneration, BartTokenizer,
                          BatchEncoding, DataCollatorForSeq2Seq,
                          PreTrainedTokenizer, Seq2SeqTrainer,
                          TrainingArguments)

class DummySeq2SeqDataset(Dataset):

    def __init__(self, tokenizer: PreTrainedTokenizer):
        self.tokenizer = tokenizer
        self.data = [
            ("Hello world!", "Hallo welt!"),
        ]

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index: int) -> BatchEncoding:
        src_text, tgt_text = self.data[index]
        return self.tokenizer.prepare_seq2seq_batch(
            src_text, tgt_text,
            return_token_type_ids=False
        )

train_args = TrainingArguments(output_dir='tmp', label_smoothing_factor=0.1)

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')

train_dataset = DummySeq2SeqDataset(tokenizer)
data_collator = DataCollatorForSeq2Seq(tokenizer)

trainer = Seq2SeqTrainer(
    model=model,
    args=train_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    tokenizer=tokenizer,
)
trainer.train()
```

## Source of problem

The problem lies with the interaction between `BartForConditionalGeneration` and how label-smoothing is implemented in `Trainer`.

`BartForConditionalGeneration.forward` is highly tied to `labels`, since it's also used to generate `decoder_input_ids`.

In the behavior of label-smoothing as implemented in the `Trainer` class, the following is currently being done.

https://github.com/huggingface/transformers/blob/3c733f320870261ea948049505a30c30fd6ea23a/src/transformers/trainer.py#L1447-L1469

The whoopsie is that `labels` is removed from the arguments passed to `BartForConditionalGeneration.forward`. Computation of logits then defaulted to using `input_ids` as `decoder_input_ids`.

## Possible solutions

A possible way to fix this would be to shift smooth label loss into the loss computation of each model rather than in `Trainer`. Doing it this way comes with its own set of pros and cons.

Pros
- Backward compatibility can be completely maintained
- Removes the little bit of code smell where true training loss is not reflected in `model.forward` when `label_smoothing > 0`.

Cons
- Complicates configuration
  - label_smoothing loss defined in model config rather than training args
- Requires changes in many places in this repository (albeit, they are the same exact set of changes)


BART for generating sequence of length more than 1024 tokens

I was using pretraining code given in transformers/examples/seq2seq to finetune on my custom dataset containing summaries of the text of greater than 1024 tokens. But I am getting an error regarding index out of bounds error. Is it possible to fine-tune BART to generate summaries of more than 1024 tokens? I have added log file for reference.

[v100job.txt](https://github.com/huggingface/transformers/files/6059178/v100job.txt)


OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5'] When I try to use my model

an error occurred while importing my model from a folder. I cloned my repository and wanted to use the model, I got an error
https://huggingface.co/Fidlobabovic/beta-kvantorium-simple-small
Do I need to change files in my repository? What should I fix in code or model files?
#7370
#9667
````
from transformers import pipeline

nlp = pipeline("question-answering",  model='/content/beta-kvantorium-simple-small', tokenizer='/content/beta-kvantorium-simple-small')

context = r"""
   Цель текущего контроля и аттестаций - выявление уровня обученности, развития способностей обучающихся, приобретенных компетенций и их соответствие прогнозируемым результатам дополнительной общеобразовательной программы.
"""

print(nlp(question="Какая Цель текущего контроля аттестаций?", context=context))

-> 1046                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
   1047             except Exception:

9 frames
UnpicklingError: invalid load key, 'v'.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
OSError: Unable to load weights from pytorch checkpoint file for '/content/beta-kvantorium-simple-small' at '/content/beta-kvantorium-simple-small/pytorch_model.bin'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/transformers/modeling_tf_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1183                     raise EnvironmentError(
   1184                         "Error no file named {} found in directory {} or `from_pt` set to False".format(
-> 1185                             [WEIGHTS_NAME, TF2_WEIGHTS_NAME], pretrained_model_name_or_path
   1186                         )
   1187                     )

OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5'] found in directory /content/beta-kvantorium-simple-small or `from_pt` set to False