[setup] update/add setup targets

updating pip targets

* [x] adds `tokenizer`
* [x] adds `docs` to `dev`, since we need to have the tools to run `make docs`
* [x] adds `flax` to `dev`, since we need to have the libs to run flax tests - except when on windows - it skips it then
* [x] brings `all` up-to-date

@sgugger


Create README.md

model card

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Doc styling fixes

# What does this PR do?

This PR fixes a few of the docstrings left in a bad state by #8067 and a small bug in the styling script (it was matching lines of the form `.. note::` or `.. warning::` to kind of examples inside docstrings.

With this, everything should now be fine.


[breaking|pipelines|tokenizers] Adding slow-fast tokenizers equivalence tests pipelines - Removing sentencepiece as a required dependency

# What does this PR do?

**Breaking**: Auto-tokenizers and pipelines:
- switch to `use_fast=True` by default (Fast tokenizers by default)
=> The main expected breaking change is **the handling of overflowing tokens** which is different between slow and fast tokenizers.
- removing sentencepiece from the required dependencies (in some special case this may require you to install `sentencepiece` in addition to the normal install).

Pipelines:
- Add slow/fast tokenizers equivalence tests in pipelines
- upgrade QA/NER processing pipeline to handle fast tokenizers
- remove `test_pipelines_dialog.py` which was a duplicated test file

Tokenizers:
- Update and add new alignement method in `BatchEncoding`

Dependencies:
- upgrade to tokenizers==0.9.4 to allow QA processing with fast tokenizers
- remove sentencepiece from the required dependencies

Misc:
- Fix bug in RobertaFast and test for XLM-Prophetnet and RAG

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


`BartForConditionalGeneration.from_pretrained` suddenly fails

I have been using the same `BartForConditionalGeneration` model and `transformers==3.0.2` for weeks, but today the same code threw a new error that has never happened before. It says I am passing an unexpected `output_past` parameter to `from_pretrained`. I am loading the `facebook/bart-large` model.

The line throwing the error is `BartForConditionalGeneration.from_pretrained("facebook/bart-large", output_past=True)`

```
/usr/local/lib/python3.6/dist-packages/qaeval/generation/model.py in __init__(self, vocab, model_name, max_decoding_steps, beam_size)
     59                  beam_size: int = 4) -> None:
     60         super().__init__(vocab)
---> 61         self.bart = BartForConditionalGeneration.from_pretrained(model_name, output_past=True)
     62         self.tokenizer = PretrainedTransformerTokenizer(model_name)
     63

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    670
    671         # Instantiate model.
--> 672         model = cls(config, *model_args, **model_kwargs)
    673
    674         if state_dict is None and not from_tf:

TypeError: __init__() got an unexpected keyword argument 'output_past'
```

I am running this code in a Jupyter Notebook. This morning the code ran (I have all of the output saved). When I make a copy of the notebook and rerun it now without making any changes, it fails with the above error. I see that the `output_past` parameter was removed [here](https://github.com/huggingface/transformers/pull/3632/files/904b387af42744f9141a6dc4be698a5815ce5bbd), but that does not explain why it was working up until just a few hours ago.

I can see in the saved output between the two notebooks that one of the files that transformers first downloads when loading the model used to be 1.26kb in size, but now it's 1.52k. I assume this is the config file for `facebook/bart-large`.

Did anything change to that file within the past few hours? I don't know where to look for the copy of that file to see what happened. I am quite perplexed by this issue.


[All Seq2Seq model + CLM models that can be used with EncoderDecoder] Add cross-attention weights to outputs

# What does this PR do?

This PR causes models that support cross-attention to output the cross-attention tensors as well as the self-attention tensors when output_attentions is set

(from @patrickvonplaten)
This PR adds cross-attention outptus to all Seq2Seq models and to CLM models compatible with the `EncoderDecoderModel` framework.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR. @patrickvonplaten @Bharat123rox

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Pretraining for encoder of TF T5 model

Hi, for sequence to sequence task in tensorflow, I see that I can use TFT5 model. However, before sequence to sequence training, I need to perform masked language model pretraining on the encoder and initialise the weights of the encoder and decoder with the same weights that I obtain by masked language model pretraining. Is there a way to do this in the tensorflow version of T5?
I could do that with the EncoderDecoder, but it is only supported in PyTorch and not tensorflow.


DEP: pinned sentencepiece to 0.1.91 in setup.py to fix build issues with newer versions

# What does this PR do?
Pins `sentencepiece` to `0.1.91` to resolve build issues with newer versions
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)
https://github.com/huggingface/transformers/issues/8020

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@LysandreJik
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


seq2seq/finetune.py: remove useless check

Removed on built-in already
```python
        self.dataset_class = (
            Seq2SeqDataset if hasattr(self.tokenizer, "prepare_seq2seq_batch") else LegacySeq2SeqDataset
        )

```


Doc styling

# What does this PR do?

This PR introduces a doc styling script and applies it to the repo. The styling script runs similarly to black, with a an option that fixes and overwrites the files (put inside `make style`) and an option that only checks if there should be a restyle, failing with an error if that's the case (put inside `make quality`).

The script is applied to all rst files inside `docs/source` and all py files inside `src/transformers`. It will look for paragraphs and always reorganize them to use the most of the `max_len` passed (set at 119 for the repo, like for the code). It will remove all duplicate or trailing whitespace, make all blank lines empty, ignore blocks of code/math and properly take care of the indentation.

A few extra things are performed:
- making the underline of the titles in rst to the `max_len` and always adding a blank line after those titles.
- unifying the format of the triple docstrings in the files
- always adding a new line before the beginning of a list (because sphinx sometimes complains otherwise)

To make the script ignore a string inside triple quotes (like warnings or long regex expressions), put a `# docstyle-ignore` somewhere before (it has to be between the previous triple quotes and the ones of the string you want to ignore).

In general, if the script reformats atrociously a docstring, it was because it was badly formatted. Adding a blank line to clearly mark paragraphs can make the script happier. Properly indenting list of arguments (see examples on any of the files of the lib) is also important to get good outputs.


Missing Import

When trying to run this file:

https://github.com/huggingface/transformers/blob/3a10764574f252591eeaa5bbb10b778f623a4814/examples/language-modeling/run_language_modeling.py#L40

The following error occurs:
````
Traceback (most recent call last):
  File "run_language_modeling.py", line 32, in <module>
    from transformers import (
ImportError: cannot import name 'DataCollatorForWholeWordMask'
````
I attempted to run the following:
````
import transformers
for i in dir(transformers):
    if "data" in i.lower():
        print (i)
````
And I got the following:

````
CsvPipelineDataFormat
DataCollator
DataCollatorForLanguageModeling
DataCollatorForNextSentencePrediction
DataCollatorForPermutationLanguageModeling
DataCollatorForSOP
DataCollatorWithPadding
DataProcessor
GlueDataTrainingArguments
GlueDataset
JsonPipelineDataFormat
LineByLineTextDataset
LineByLineWithSOPTextDataset
PipedPipelineDataFormat
PipelineDataFormat
SquadDataTrainingArguments
SquadDataset
TextDataset
TextDatasetForNextSentencePrediction
data
default_data_collator
is_datasets_available
````
It appears that the **DataCollatorForWholeWordMask** is not a part of transformers for some reason.

I commented it out and this one also appears to have an issue (I'll list all that complain at the bottom of this post and edit it as I find more)

At this point I commented the below imports out and it is running (well it's downloading one of the models I believe). I'll update if it fails/succeeds.

Missing Imports from transformers:
* DataCollatorForWholeWordMask
* LineByLineWithRefDataset


load 'microsoft/unilm-base-cased' failed

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I use the following code in https://huggingface.co/microsoft/unilm-base-cased to load the model.
```
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("microsoft/unilm-base-cased")

model = AutoModel.from_pretrained("microsoft/unilm-base-cased")
```

And I got the traceback like this
 tokenizer = AutoTokenizer.from_pretrained("microsoft/unilm-base-cased")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\tokenization_auto.py", line 298, in from_pretrained
    config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
  File "C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_auto.py", line 341, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in microsoft/unilm-base-cased. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: retribert, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, bart, blenderbot, reformer, longformer, roberta, deberta, flaubert, fsmt, squeezebert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag
>>>
>>> model = AutoModel.from_pretrained("microsoft/unilm-base-cased")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\modeling_auto.py", line 623, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "C:\Users\666666\AppData\Local\Programs\Python\Python38\lib\site-packages\transformers\configuration_auto.py", line 341, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in microsoft/unilm-base-cased. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: retribert, t5, mobilebert, distilbert, albert, bert-generation, camembert, xlm-roberta, pegasus, marian, mbart, bart, blenderbot, reformer, longformer, roberta, deberta, flaubert, fsmt, squeezebert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm-prophetnet, prophetnet, xlm, ctrl, electra, encoder-decoder, funnel, lxmert, dpr, layoutlm, rag
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


[QOL] PretrainedConfig.to_diff_dict(other_config)

This PR allows users to compare two configs in a backwards compatible way.
### Solution
The following calls work:
```python
bart_large_config = BartConfig.from_pretrained('facebook/bart-large')
bart_base_config = BartConfig.from_pretrained('facebook/bart-base')
t5_config = T5Config.from_pretrained('t5-small')
bart_large_config.to_diff_dict() # unchanged
bart_large_config.to_diff_dict(bart_base_config) # compares configs
bart_large_config.to_diff_dict(t5_config) # can be across subtypes
bart_large_config.to_diff_dict(bart_base_config.to_dict()) # can be against dict
```

Adds test that outputs are reasonable.
### Problem
Current best way to compare configs is to define your own function. Here is the one I use. Also good for debugging conversion scripts:

```python
def dct_differences(dct_a, dct_b):
    SENTINEL = '__MissingKey'
    k1, k2 = set(dct_a), set(dct_b) # just the keys
    deltas = []
    for k in k1.union(k2):
        vala, valb = dct_a.get(k, SENTINEL), dct_b.get(k, SENTINEL)
        # TODO(SS): nested dicts? Maybe better to dump to json and compare (after sorting keys!)
        if vala == valb:
            if (vala == SENTINEL and valb == SENTINEL): raise AssertionError('Adversarial Sentinel Input!')
        else:
            deltas.append((k, vala, valb))
    return deltas
bart_large_config = BartConfig.from_pretrained('facebook/bart-large')
bart_base_config = BartConfig.from_pretrained('facebook/bart-base')
delta =  dct_differences(bart_large_config.to_dict(), bart_base_config.to_dict())
```
this implementation is almost as useful without breaking backwards compatibility.


Fix TF training arguments instantiation

Check that pytorch is installed before checking the device type.


Add AzureML in integrations via dedicated callback

# What does this PR do?

I propose this PR to allow transformers to call AzureML logging using https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.run(class)?view=azure-ml-py

The intended behaviour is to enable `transformers` users to track metrics in the AzureML UI in this fashion

![image](https://user-images.githubusercontent.com/4547987/97200283-4f2e0b80-17b1-11eb-8928-20d6062c69a4.png)

Contributors to https://github.com/microsoft/AzureML-BERT and folks @microsoft may well come up with a better implementation though!

I am glad to improve the following if reviewers like the idea, and update docs and tests if needed.
@reviewers feel free to add any suggestions as my contributions to transformers have been very limited so far :)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Yes, check for detais https://discuss.huggingface.co/t/how-to-integrate-an-azuremlcallback-for-logging-in-azure/1713/4
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@julien-c is aware of this and @sgugger participated in the thread on forums above and implemented callbacks with https://github.com/huggingface/transformers/pull/7596


Doc fixes in preparation for the docstyle PR

# What does this PR do?

This PR fixes a few docstrings and adds the `# docstyle-ignore` marker where necessary in preparation for the big docstyle PR.


a multitude of deprecations for pytorch-1.7+

This is not urgent. There is a ton of deprecation warnings across many modules with pytorch-1.7+ and a few with python-3.8:
(I hard-wrapped the lines to avoid the need to scroll, but it makes somewhat harder to see the warnings):
```
src/transformers/modeling_deberta.py:18 src/transformers/modeling_deberta.py:18
src/transformers/modeling_deberta.py:18 src/transformers/modeling_deberta.py:18
src/transformers/modeling_deberta.py:18
src/transformers/modeling_deberta.py:18:
DeprecationWarning: Using or importing the ABCs from 'collections' instead of
from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop
working from collections import Sequence

tests/test_logging.py::HfArgumentParserTest::test_integration
  tests/test_logging.py:40:
  DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn(msg)

tests/test_logging.py::HfArgumentParserTest::test_integration
  tests/test_logging.py:48:
  DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn(msg)

tests/test_benchmark.py::BenchmarkTest::test_inference_torchscript
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript_output_attentions
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_gpt2.py:164:
TracerWarning: Converting a tensor to a Python float might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! w = w / (float(v.size(-1)) ** 0.5)

tests/test_benchmark.py::BenchmarkTest::test_inference_torchscript
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript_output_attentions
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_gpt2.py:169:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! mask = self.bias[:, :, ns - nd : ns, :ns]

tests/test_modeling_auto.py::AutoModelTest::test_from_identifier_from_model_type
tests/test_modeling_auto.py::AutoModelTest::test_from_pretrained_identifier
src/transformers/modeling_auto.py:821:
FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed
in a future version. Please use `AutoModelForCausalLM` for causal language
models, `AutoModelForMaskedLM` for masked language models and
`AutoModelForSeq2SeqLM` for encoder-decoder models. warnings.warn(

tests/test_benchmark_tf.py::TFBenchmarkTest::test_train_no_configs
tests/test_benchmark_tf.py::TFBenchmarkTest::test_train_with_configs
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/tensorflow/python/framework/indexed_slices.py:432:
UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape.
This may consume a large amount of memory. warnings.warn(

tests/test_modeling_albert.py::AlbertModelTest::test_torchscript
tests/test_modeling_albert.py::AlbertModelTest::test_torchscript_output_attentions
tests/test_modeling_albert.py::AlbertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_albert.py:229:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_albert.py: 3 warnings tests/test_modeling_bert.py: 3
warnings tests/test_modeling_bert_generation.py: 3 warnings
tests/test_modeling_distilbert.py: 2 warnings tests/test_modeling_dpr.py: 3
warnings tests/test_modeling_flaubert.py: 3 warnings
tests/test_modeling_electra.py: 3 warnings tests/test_modeling_layoutlm.py: 3
warnings tests/test_modeling_roberta.py: 3 warnings tests/test_modeling_xlm.py:
3 warnings tests/test_modeling_xlnet.py: 3 warnings
src/transformers/modeling_utils.py:1670:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! input_tensor.shape == tensor_shape for input_tensor
in input_tensors

tests/test_modeling_bert_generation.py: 32 warnings
  src/transformers/modeling_bert_generation.py:417:
  DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
  logger.warn("If you want to use `BertGenerationDecoder` as a standalone, add
  `is_decoder=True.`")

tests/test_modeling_bert.py::BertModelTest::test_torchscript
tests/test_modeling_bert.py::BertModelTest::test_torchscript_output_attentions
tests/test_modeling_bert.py::BertModelTest::test_torchscript_output_hidden_state
tests/test_modeling_dpr.py::DPRModelTest::test_torchscript
tests/test_modeling_dpr.py::DPRModelTest::test_torchscript_output_attentions
tests/test_modeling_dpr.py::DPRModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bert.py:191:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_bart.py::BARTModelTest::test_torchscript
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_attentions
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bart.py:175:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! if decoder_padding_mask is not None and
decoder_padding_mask.shape[1] > 1:

tests/test_modeling_bart.py: 3 warnings tests/test_modeling_flaubert.py: 3
warnings tests/test_modeling_fsmt.py: 3 warnings tests/test_modeling_roberta.py:
3 warnings tests/test_modeling_xlm.py: 3 warnings
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/nn/functional.py:1836:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert padding_idx < weight.size(0), 'Padding_idx
must be within num_embeddings'

tests/test_modeling_bart.py::BARTModelTest::test_torchscript
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_attentions
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bart.py:720:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert key_padding_mask is None or
key_padding_mask.shape == (bsz, src_len)

tests/test_modeling_bart.py::BARTModelTest::test_torchscript
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_attentions
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bart.py:722:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert attn_weights.size() == (bsz * self.num_heads,
tgt_len, src_len)

tests/test_modeling_bart.py::BARTModelTest::test_torchscript
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_attentions
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bart.py:740:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert attn_output.size() == (bsz * self.num_heads,
tgt_len, self.head_dim)

tests/test_modeling_bart.py::BARTModelTest::test_torchscript
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_attentions
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bart.py:287:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! if torch.isinf(x).any() or torch.isnan(x).any():

tests/test_modeling_bart.py::BARTModelTest::test_torchscript
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_attentions
tests/test_modeling_bart.py::BARTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_bart.py:1190:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! if len(torch.unique(eos_mask.sum(1))) > 1:

tests/test_modeling_common.py::UtilsFunctionsTest::test_top_k_top_p_filtering
  tests/test_modeling_common.py:1196:
  UserWarning: This overload of nonzero is deprecated: nonzero() Consider using
  one of the following signatures instead: nonzero(*, bool as_tuple) (Triggered
  internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  non_inf_idx = (output != -float("inf")).nonzero().to(device=torch_device)

tests/test_modeling_bert_generation.py::BertGenerationEncoderTest::test_torchscript
tests/test_modeling_bert_generation.py::BertGenerationEncoderTest::test_torchscript_output_attentions
tests/test_modeling_bert_generation.py::BertGenerationEncoderTest::test_torchscript_output_hidden_state
src/transformers/modeling_bert_generation.py:156:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_flaubert.py: 14 warnings tests/test_modeling_xlm.py: 14
warnings
src/transformers/modeling_xlm.py:1220:
FutureWarning: The `lengths` parameter cannot be used with the XLM multiple
choice models. Please use the attention mask instead. warnings.warn(

tests/test_modeling_flaubert.py::FlaubertModelTest::test_flaubert_lm_head
tests/test_modeling_flaubert.py::FlaubertModelTest::test_model_outputs_equivalence
tests/test_modeling_xlm.py::XLMModelTest::test_model_outputs_equivalence
tests/test_modeling_xlm.py::XLMModelTest::test_xlm_lm_head
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/nn/_reduction.py:14:
UserWarning: reduction='elementwise_mean' is deprecated, please use
reduction='mean' instead. warnings.warn("reduction='elementwise_mean' is
deprecated, please use reduction='mean' instead.")

tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_attentions
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_flaubert.py:188:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.size(0) == bs

tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_attentions
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_flaubert.py:189:
TracerWarning: Converting a tensor to a Python number might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.max().item() <= slen

tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_attentions
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_flaubert.py:189:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.max().item() <= slen

tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_attentions
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_hidden_state
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:95:
TracerWarning: Converting a tensor to a Python number might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.max().item() <= slen

tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_attentions
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_hidden_state
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:95:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.max().item() <= slen

tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_attentions
tests/test_modeling_flaubert.py::FlaubertModelTest::test_torchscript_output_hidden_state
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:106:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert mask.size() == (bs, slen)

tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_attentions
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_fsmt.py:1224:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! if max_pos > self.weight.size(0):

tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_attentions
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_fsmt.py:763:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert embed_dim == self.embed_dim

tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_attentions
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_fsmt.py:764:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert list(query.size()) == [tgt_len, bsz,
embed_dim]

tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_attentions
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_fsmt.py:805:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert attn_weights.size() == (bsz * self.num_heads,
tgt_len, src_len)

tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_attentions
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_fsmt.py:814:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert key_padding_mask is None or
key_padding_mask.size()[:2] == (

tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_attentions
tests/test_modeling_fsmt.py::FSMTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_fsmt.py:833:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert attn_output.size() == (bsz * self.num_heads,
tgt_len, self.head_dim)

tests/test_modeling_gpt2.py::GPT2ModelTest::test_gpt2_model_att_mask_past
tests/test_modeling_gpt2.py::GPT2ModelTest::test_gpt2_model_past
tests/test_modeling_gpt2.py::GPT2ModelTest::test_gpt2_model_past_large_inputs
src/transformers/modeling_gpt2.py:530:
FutureWarning: The `past` argument is deprecated and will be removed in a future
version, use `past_key_values` instead. warnings.warn(

tests/test_modeling_electra.py::ElectraModelTest::test_torchscript
tests/test_modeling_electra.py::ElectraModelTest::test_torchscript_output_attentions
tests/test_modeling_electra.py::ElectraModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_electra.py:180:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_layoutlm.py::LayoutLMModelTest::test_torchscript
tests/test_modeling_layoutlm.py::LayoutLMModelTest::test_torchscript_output_attentions
tests/test_modeling_layoutlm.py::LayoutLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_layoutlm.py:87:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_hidden_state
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_hidden_state
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/tensor.py:547:
TracerWarning: torch.tensor results are registered as constants in the trace.
You can safely ignore this warning if you use this function to create tensors
out of constant variables that would be the same every time you call this
function. In any other case, this might cause the trace to be incorrect. return
torch.tensor(other, dtype=dtype, device=self.device) ** self

tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_hidden_state
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_funnel.py:314:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! num_remove = shift * len(pooled_pos)

tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_hidden_state
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_funnel.py:638:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! pooling_flag = pooling_flag and block_index > 0

tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_hidden_state
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_funnel.py:481:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! shift = 2 if q_head.shape[1] != context_len else 1

tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_hidden_state
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelBaseModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_funnel.py:431:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! positional_attn = positional_attn[..., :context_len]

tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_attentions
tests/test_modeling_funnel.py::FunnelModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_funnel.py:678:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! output = output[:, : target_len - 1]

tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript_output_attentions
tests/test_modeling_gpt2.py::GPT2ModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_gpt2.py:1058:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! pooled_logits = logits[range(batch_size),
sequence_lengths]

tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_attentions
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_openai.py:467:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[None, :
input_shape[-1]]

tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_attentions
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_openai.py:180:
TracerWarning: Converting a tensor to a Python float might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! w = w / math.sqrt(v.size(-1))

tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_attentions
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_openai.py:183:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! b = self.bias[:, :, : w.size(-2), : w.size(-1)]

tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_attentions
tests/test_modeling_openai.py::OpenAIGPTModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_openai.py:823:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! pooled_logits = logits[range(batch_size),
sequence_lengths]

tests/test_modeling_rag.py: 12 warnings tests/test_retrieval_rag.py: 1 warning
src/transformers/tokenization_utils_base.py:613:
UserWarning: To copy construct from a tensor, it is recommended to use
sourceTensor.clone().detach() or
sourceTensor.clone().detach().requires_grad_(True), rather than
torch.tensor(sourceTensor). tensor = as_tensor(value)

tests/test_modeling_reformer.py: 58 warnings tests/test_modeling_transfo_xl.py:
18 warnings
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/nn/modules/container.py:434:
UserWarning: Setting attributes on ParameterList is not supported.
warnings.warn("Setting attributes on ParameterList is not supported.")

tests/test_modeling_mobilebert.py::MobileBertModelTest::test_torchscript
tests/test_modeling_mobilebert.py::MobileBertModelTest::test_torchscript_output_attentions
tests/test_modeling_mobilebert.py::MobileBertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_mobilebert.py:192:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_mobilebert.py::MobileBertModelTest::test_torchscript
tests/test_modeling_mobilebert.py::MobileBertModelTest::test_torchscript_output_attentions
tests/test_modeling_mobilebert.py::MobileBertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_mobilebert.py:534:
TracerWarning: torch.tensor results are registered as constants in the trace.
You can safely ignore this warning if you use this function to create tensors
out of constant variables that would be the same every time you call this
function. In any other case, this might cause the trace to be incorrect.
torch.tensor(1000),

tests/test_modeling_reformer.py::ReformerLSHAttnModelTest::test_reformer_cached_inference
  src/transformers/modeling_reformer.py:899:
  UserWarning: This overload of nonzero is deprecated: nonzero() Consider using
  one of the following signatures instead: nonzero(*, bool as_tuple) (Triggered
  internally at /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  relevant_bucket_idx = (bucket_idx == (bucket_idx.shape[-1] - 1)).nonzero()

tests/test_modeling_t5.py::T5ModelTest::test_export_to_onnx
tests/test_modeling_t5.py::T5ModelTest::test_torchscript_output_attentions
tests/test_modeling_t5.py::T5ModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_utils.py:244:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! if causal_mask.shape[1] < attention_mask.shape[1]:

tests/test_modeling_t5.py: 95 warnings
  /home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/onnx/utils.py:760:
  DeprecationWarning: an integer is required (got type
  torch._C._onnx.TensorProtoDataType). Implicit conversion to integers using
  __int__ is deprecated, and may be removed in a future version of Python.
  return getattr(node, kind + "_")(name, value)

tests/test_modeling_t5.py::T5ModelTest::test_export_to_onnx
tests/test_modeling_t5.py::T5ModelTest::test_export_to_onnx
tests/test_modeling_t5.py::T5ModelTest::test_export_to_onnx
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/onnx/symbolic_opset9.py:1638:
DeprecationWarning: an integer is required (got type float). Implicit conversion
to integers using __int__ is deprecated, and may be removed in a future version
of Python. value_t=torch.tensor([fill_value],
dtype=sym_help.scalar_type_to_pytorch_type[dtype]))

tests/test_modeling_tf_auto.py::TFAutoModelTest::test_from_identifier_from_model_type
tests/test_modeling_tf_auto.py::TFAutoModelTest::test_from_pretrained_identifier
src/transformers/modeling_tf_auto.py:697:
FutureWarning: The class `TFAutoModelWithLMHead` is deprecated and will be
removed in a future version. Please use `TFAutoModelForCausalLM` for causal
language models, `TFAutoModelForMaskedLM` for masked language models and
`TFAutoModelForSeq2SeqLM` for encoder-decoder models. warnings.warn(

tests/test_modeling_squeezebert.py::SqueezeBertModelTest::test_torchscript_output_attentions
tests/test_modeling_squeezebert.py::SqueezeBertModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_squeezebert.py:78:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :seq_length]

tests/test_modeling_tf_flaubert.py: 9 warnings tests/test_modeling_tf_xlm.py: 9
warnings
src/transformers/modeling_tf_xlm.py:994:
FutureWarning: The `lengths` parameter cannot be used with the XLM multiple
choice models. Please use the attention mask instead. warnings.warn(

tests/test_modeling_tf_flaubert.py::TFFlaubertModelTest::test_graph_mode
tests/test_modeling_tf_xlm.py::TFXLMModelTest::test_graph_mode
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/tensorflow/python/autograph/impl/api.py:493:
FutureWarning: The `lengths` parameter cannot be used with the XLM multiple
choice models. Please use the attention mask instead. return
py_builtins.overload_of(f)(*args)

tests/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_compile_tf_model
tests/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_config
tests/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_keras_save_load
tests/test_modeling_xlnet.py::XLNetModelTest::test_config
tests/test_modeling_xlnet.py::XLNetModelTest::test_correct_missing_keys
tests/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_save_load
tests/test_modeling_tf_xlnet.py::TFXLNetModelTest::test_train_pipeline_custom_model
tests/test_modeling_xlnet.py::XLNetModelTest::test_save_load
src/transformers/configuration_xlnet.py:205:
FutureWarning: This config doesn't use attention memories, a core feature of
XLNet. Consider setting `mem_len` to a non-zero value, for example `xlnet =
XLNetLMHeadModel.from_pretrained('xlnet-base-cased'', mem_len=1024)`, for
accurate training performance as well as an order of magnitude faster inference.
Starting from version 3.5.0, the default parameter will be 1024, following the
implementation in https://arxiv.org/abs/1906.08237 warnings.warn(

tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:531:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.size(0) == bs

tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:532:
TracerWarning: Converting a tensor to a Python number might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.max().item() <= slen

tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:532:
TracerWarning: Converting a tensor to a Python boolean might cause the trace to
be incorrect. We can't record the data flow of Python values, so this value will
be treated as a constant in the future. This means that the trace might not
generalize to other inputs! assert lengths.max().item() <= slen

tests/test_modeling_xlm.py::XLMModelTest::test_torchscript
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_attentions
tests/test_modeling_xlm.py::XLMModelTest::test_torchscript_output_hidden_state
src/transformers/modeling_xlm.py:546:
TracerWarning: Converting a tensor to a Python index might cause the trace to be
incorrect. We can't record the data flow of Python values, so this value will be
treated as a constant in the future. This means that the trace might not
generalize to other inputs! position_ids = self.position_ids[:, :slen]

tests/test_optimization.py::OptimizationTest::test_adafactor
  src/transformers/optimization.py:512:
  UserWarning: This overload of add_ is deprecated: add_(Number alpha, Tensor
  other) Consider using one of the following signatures instead: add_(Tensor
  other, *, Number alpha) (Triggered internally at
  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  exp_avg_sq.mul_(beta2t).add_(1.0 - beta2t, update)

tests/test_optimization.py::ScheduleInitTest::test_schedulers
  /home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:247:
  UserWarning: To get the last learning rate computed by the scheduler, please
  use `get_last_lr()`. warnings.warn("To get the last learning rate computed by
  the scheduler, "

tests/test_optimization.py::ScheduleInitTest::test_schedulers
  /home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:131:
  UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`.
  In PyTorch 1.1.0 and later, you should call them in the opposite order:
  `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will
  result in PyTorch skipping the first value of the learning rate schedule. See
  more details at
  https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  warnings.warn("Detected call of `lr_scheduler.step()` before
  `optimizer.step()`. "

tests/test_optimization.py::ScheduleInitTest::test_schedulers
  /home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:216:
  UserWarning: Please also save or load the state of the optimizer when saving
  or loading the scheduler. warnings.warn(SAVE_STATE_WARNING, UserWarning)

tests/test_optimization.py::ScheduleInitTest::test_schedulers
  /home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:234:
  UserWarning: Please also save or load the state of the optimizer when saving
  or loading the scheduler. warnings.warn(SAVE_STATE_WARNING, UserWarning)

tests/test_tokenization_auto.py::AutoTokenizerTest::test_tokenizer_identifier_with_correct_config
tests/test_tokenization_mbart.py::MBartEnroIntegrationTest::test_batch_fairseq_parity
tests/test_tokenization_t5.py::T5TokenizationTest::test_empty_target_text
tests/test_tokenization_t5.py::T5TokenizationTest::test_eos_in_input
tests/test_tokenization_t5.py::T5TokenizationTest::test_max_target_length
tests/test_tokenization_t5.py::T5TokenizationTest::test_outputs_not_longer_than_maxlen
tests/test_tokenization_t5.py::T5TokenizationTest::test_prepare_seq2seq_batch
src/transformers/tokenization_utils_base.py:1421:
FutureWarning: The `max_len` attribute has been deprecated and will be removed
in a future version, use `model_max_length` instead. warnings.warn(

tests/test_tokenization_albert.py: 2 warnings tests/test_tokenization_bart.py: 2
warnings tests/test_tokenization_bert.py: 2 warnings
tests/test_tokenization_bert_generation.py: 1 warning
tests/test_tokenization_bertweet.py: 1 warning
tests/test_tokenization_blenderbot.py: 1 warning
tests/test_tokenization_ctrl.py: 1 warning tests/test_tokenization_camembert.py:
2 warnings tests/test_tokenization_distilbert.py: 4 warnings
tests/test_tokenization_dpr.py: 8 warnings tests/test_tokenization_fsmt.py: 1
warning tests/test_tokenization_funnel.py: 2 warnings
tests/test_tokenization_herbert.py: 2 warnings tests/test_tokenization_gpt2.py:
1 warning tests/test_tokenization_layoutlm.py: 2 warnings
tests/test_tokenization_marian.py: 1 warning tests/test_tokenization_lxmert.py:
2 warnings tests/test_tokenization_mbart.py: 2 warnings
tests/test_tokenization_pegasus.py: 2 warnings
tests/test_tokenization_openai.py: 1 warning tests/test_tokenization_phobert.py:
1 warning tests/test_tokenization_deberta.py: 1 warning
tests/test_tokenization_prophetnet.py: 1 warning
tests/test_tokenization_reformer.py: 1 warning
tests/test_tokenization_squeezebert.py: 4 warnings
tests/test_tokenization_t5.py: 2 warnings tests/test_tokenization_roberta.py: 2
warnings tests/test_tokenization_transfo_xl.py: 1 warning
tests/test_tokenization_xlm.py: 1 warning
tests/test_tokenization_xlm_prophetnet.py: 1 warning
tests/test_tokenization_xlnet.py: 2 warnings
tests/test_tokenization_xlm_roberta.py: 2 warnings
src/transformers/tokenization_utils_base.py:2025:
FutureWarning: The `pad_to_max_length` argument is deprecated and will be
removed in a future version, use `padding=True` or `padding='longest'` to pad to
the longest sequence in the batch, or use `padding='max_length'` to pad to a max
length. In this case, you can give a specific length with `max_length` (e.g.
`max_length=45`) or leave max_length to None to pad to the maximal input size of
the model (e.g. 512 for Bert). warnings.warn(

tests/test_tokenization_t5.py::T5TokenizationTest::test_eos_in_input
tests/test_tokenization_t5.py::T5TokenizationTest::test_eos_treatment
src/transformers/tokenization_t5.py:183:
UserWarning: This sequence already has </s>. In future versions this behavior
may lead to duplicated eos tokens being added. warnings.warn(

tests/test_trainer.py: 44 warnings
  /home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/nn/parallel/_functions.py:64:
  UserWarning: Was asked to gather along dimension 0, but all input tensors were
  scalars; will instead unsqueeze and return a vector. warnings.warn('Was asked
  to gather along dimension 0, but all '

tests/test_trainer.py::TrainerIntegrationTest::test_can_resume_training
tests/test_trainer_callback.py::TrainerCallbackTest::test_event_flow
/home/stas/anaconda3/envs/py38-pt17/lib/python3.8/site-packages/torch/cuda/nccl.py:48:
DeprecationWarning: Using or importing the ABCs from 'collections' instead of
from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop
working if not isinstance(inputs, collections.Container) or isinstance(inputs,
torch.Tensor):

-- Docs: https://docs.pytest.org/en/stable/warnings.html
```

@LysandreJik


infer entailment label id on zero shot pipeline

# What does this PR do?

Adds an optional argument to the zero shot pipeline constructor to specify the label id of the NLI model that corresponds to "entailment", which it needs to calculate each candidate label's score. Most models in the hub use the last last label id, but some differ (e.g. the recent [ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli](https://huggingface.co/ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli)).

If the argument is not passed, the pipeline will attempt to look up the entailment dimension in the model config's id2label mapping. If the config does not specify the entailment dimension, the value will be set to `-1`, indicating the last dimension of the model output.

With this logic in place, the arg only needs to be passed when both (1) the model's entailment label id is not the last id and (2) when the model config's `label2id` doesn't specify the entailment id.


[testing] port test_trainer_distributed to run with pytest

Extracting the request from https://github.com/huggingface/transformers/pull/7993#issuecomment-716508513 to this issue to make it easier to track.

Now that we have a framework to run distributed under `pytest` `test_trainer_distributed` needs to be ported there.

I will work on that.


[testing] fixing crash in deberta

This PR fixes a crash in deberta tests w/ pytorch-1.7+:
```
RuntimeError: diff_view_meta->output_nr_ == 0 INTERNAL ASSERT FAILED at "/opt/conda/conda- \
bld/pytorch_1603436966316/work/torch/csrc/autograd/variable.cpp":363, please report a bug to PyTorch.
```

All credits go to @gchanan, thank you! For details of why please see https://github.com/huggingface/transformers/issues/8022#issuecomment-716252599

Fixes: #8022


[TF] from_pt should respect authorized_unexpected_keys

This comes in handy when
```
        "model.encoder.embed_tokens.weight",
        "model.decoder.embed_tokens.weight",
```
are in the PT state dict but not the TF symbolic weights.


BertEncoder has no attribute 'bias' when convert tf checkpoint

# ❓ Questions & Help
BertEncoder object has no attribute 'bias' when convert tf checkpoint
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I tried to convert my pretrained BERT tf checkpoint but I got this error:
```
/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
    133                 pointer = getattr(pointer, "weight")
    134             elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
--> 135                 pointer = getattr(pointer, "bias")
    136             elif scope_names[0] == "output_weights":
    137                 pointer = getattr(pointer, "weight")
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    770                 return modules[name]
    771         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--> 772             type(self).__name__, name))
    773
    774     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:
ModuleAttributeError: 'BertEncoder' object has no attribute 'bias'
```

I used both load_tf_weights_in_bert function and BertForPreTraining but it was not working.

My code:
```
config = BertConfig.from_json_file('bertvn_base/bertvn_base_config.json')
model = BertForPreTraining.from_pretrained('bertvn_base/model.ckpt', from_tf=True, config=config)
```

My config:
```
{"attention_probs_dropout_prob": 0,
"hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "embedding_size": 768,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 192,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_hidden_groups": 12,
  "net_structure_type": 0,
  "gap_size": 0,
  "num_memory_blocks": 0,
  "inner_group_num": 1,
  "down_scale_factor": 1,
  "type_vocab_size": 2,
  "vocab_size": 120000
}
```

Thanks for your help!
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Add m2m 100 multilingual translation model from FAIR

Weights, code are available.

+ Fairseq Code: https://github.com/pytorch/fairseq/tree/master/examples/m2m_100?fbclid=IwAR304kICXsffdDMogK4MWf4D7Xeu_3Cbmgu8pBCU_jKcjijCuJfLK7CY9_I
+ Paper: https://arxiv.org/abs/2010.11125

+ This model will not run on 1 V100 GPU, so model parallelism will be needed.
+ I would expect the state dict to be very similar to mBART, but not sure yet.
+ All I've done is download the state dict, run their command, and asked for help https://github.com/pytorch/fairseq/issues/2772#issuecomment-716152453 when it broke.

Leaving this unassigned in case somebody else wants to take over.


Minor error fix of 'bart-large-cnn' details in the pretrained_models doc

I found that there seemed to be a mistake regarding "facebook/bart-large-cnn" in the pretrained_models doc.

# What does this PR do?

While I check the model explanations in the pretrained_models doc, I found that there seemed to be a mistake.

Regarding `facebook/bart-large-cnn`, the details of the model is as follows:

```
12-layer, 1024-hidden, 16-heads, 406M parameters (same as base)
bart-large base architecture finetuned on cnn summarization task
```

If my understanding is correct, it seems that `12-layer` and `(same as base)` should be `24-layer` and `(same as large)`.

I asked a question in the forum about this:
https://discuss.huggingface.co/t/there-seems-to-be-a-mistake-in-documentation-pretrained-models-html-regarding-bart/1746/

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

documentation: @sgugger
(Thank you for kindly answering my question in the forum!)


Fix a bug for `CallbackHandler.callback_list`

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fix a bug where `CallbackHandler.callback_list` fails when given callbacks are duplicated:

```
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-40-9605b122f4d1> in <module>()
      2 from transformers.trainer import DEFAULT_CALLBACKS
      3
----> 4 CallbackHandler(DEFAULT_CALLBACKS + [MLflowCallback], "model", "optimizer", "lr_scheduler")

2 frames
/usr/local/lib/python3.6/dist-packages/transformers/trainer_callback.py in __init__(self, callbacks, model, optimizer, lr_scheduler)
    277         self.callbacks = []
    278         for cb in callbacks:
--> 279             self.add_callback(cb)
    280         self.model = model
    281         self.optimizer = optimizer

/usr/local/lib/python3.6/dist-packages/transformers/trainer_callback.py in add_callback(self, callback)
    299                 f"You are adding a {cb_class} to the callbacks of this Trainer, but there is already one. The current"
    300                 + "list of callbacks is\n:"
--> 301                 + self.callback_list
    302             )
    303         self.callbacks.append(cb)

/usr/local/lib/python3.6/dist-packages/transformers/trainer_callback.py in callback_list(self)
    326     @property
    327     def callback_list(self):
--> 328         return "\n".join(self.callbacks)
    329
    330     def on_init_end(self, args: TrainingArguments, state: TrainerState, control: TrainerControl):

TypeError: sequence item 0: expected str instance, DefaultFlowCallback found
```

Code to reproduce the bug:

```python
from transformers.trainer_callback import CallbackHandler
from transformers.trainer import DEFAULT_CALLBACKS

CallbackHandler(DEFAULT_CALLBACKS + [MLflowCallback], "model", "optimizer", "lr_scheduler")
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


minor model card description updates

Related to pretrained model cards

# What does this PR do?

Makes a few minor updates to the [joeddav/xlm-roberta-large-xnli](https://huggingface.co/joeddav/xlm-roberta-large-xnli) model card, such as removing the reference to the deprecated zero shot demo when the user can play with zero shot classification via the embedded widget. Also links to distilled bart models.


invalid argument wwm passed to the run_language_modeling.py file

# What does this PR do?
--wwm cant be used as an argument given run_language_modeling.py and should be changed to --whole_word_mask
<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [x ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

@stefan-it


Fix + Test

Fix an edge case of the blenderbot-90 tokenizer.

Closes #8029

# Context

If the blenderbot-90 tokenizer is used to tokenize the following sequence:
```py
sequence = "Ok ."
```
It will split it in two tokens at first:
https://github.com/huggingface/transformers/blob/8bbe8247f13057b7df1b2c9abbfacb05b30020bf/src/transformers/tokenization_blenderbot.py#L221

Those two tokens will be `['Ok', '.']`

The issue is that, when passed the second token, the `bpe` method will convert it from `'.'` to `' .'` here:
https://github.com/huggingface/transformers/blob/8bbe8247f13057b7df1b2c9abbfacb05b30020bf/src/transformers/tokenization_blenderbot.py#L160

This then gets split on spaces here:
https://github.com/huggingface/transformers/blob/8bbe8247f13057b7df1b2c9abbfacb05b30020bf/src/transformers/tokenization_blenderbot.py#L166

This is where the issue lies, as it creates two strings: `["", "."]`, the first one being empty.

It then crashes a bit further as we try to index the empty string:
https://github.com/huggingface/transformers/blob/8bbe8247f13057b7df1b2c9abbfacb05b30020bf/src/transformers/tokenization_blenderbot.py#L171

## Proposal

Ensure that the token has a length > 0 before trying to manage it, otherwise ignore that token.

Added a test.


Fix label name in DataCollatorForNextSentencePrediction test

# What does this PR do?

Labels have been renamed in `DataCollatorForNextSentencePrediction` to go with the fact `masked_lm_labels` is a deprecated argument, but the corresponding test was not adjusted accordingly, this PR fixes that.

Fixes #8034


[T5] Unused `n_positions` and `max_position_embeddings`.

The T5Config has the parameter `n_positions` set to 512 and `max_position_embeddigs` referring to `n_positions`. However, neither `max_position_embeddigs` nor `n_positions` is used in the `T5Model` and T5 is not limited to `max_position_embeddings`. *E.g.*:

```python
from transformers import T5Model
model = T5Model.from_pretrained("t5-small")

model.config.max_position_embeddings # shows 512

input_ids = torch.tensor([600 * [0]])  # input of size > 512

model(input_ids, decoder_input_ids=input_ids)  # works fine
```

 I think we should delete the parameter.

@thomwolf - do you remember why we added `max_position_embeddigs` and `n_positions` to T5? The model does not seem to use these params and also should not be limited to 512 due to its relative position embeddings.


Minor typo fixes to the preprocessing tutorial in the docs

# What does this PR do?

Minor typo fixes to the tokenizer summary in the docs

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
 documentation: @sgugger


Minor typo fixes to the tokenizer summary in the docs

# What does this PR do?

Minor typo fixes to the tokenizer summary in the docs

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
 documentation: @sgugger


Automatically cuts input if the pipeline position_ids can't handle it.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Currently the failure is `index out of range in self`.  (it fails in pytorch position_embeddings call).
The current PR aims to simply cut the input to the pipeline if the pipeline can't handle it with a warning to the user.

It feels better than the bad error message. As for triggering an error, it seems correct for the underlying model (it can't handle
the input). But, it seems a bit off to trigger the error for the pipeline as there is no way of knowing how to trim the input
for the user as he does not inputs tokens himself. Automatically cutting with a warning seems a bit better from
a usage standpoint.

This PR also improve a docstring which had a typo.
It also adds a new `PipelineWarning` because there are now 2 instances of such warnings and there should be more in the
future so enabling users to catch those warnings seems like a good idea.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten
@lhoestq

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


[Seq2Seq Trainer] Make sure padding is implemented for models without pad_token

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR adds padding for models without padding token as well. The logic is the following:

1) If model predicts `targets` < `max_length` => model has to have at least an `eos_token_id`. If model has no `config.pad_token_id` defined than the model simply uses the `config.eos_token_id` for padding.

2) If the model has no `config.eos_token_id`, => model cannot generate predictions shorter than `max_length`. In this case padding will never happen.

@sshleifer @patil-suraj - you guys were right -> the `Trainer` requires padding in any case (also if model has no padding token).
Could you guys review this PR and see if these fixes in Seq2Seq Trainer are ok for you?


Tentative improvement on sequence_length error for position_ids

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Currently the failure is `index out of range in self`.  (it fails in pytorch position_embeddings call).
The current PR aims to simply improve the error message by saying that the position_ids are simply too large.

It does add an `if` statement and a `max` call in a forward pass which is not so great
but it might help other users understand better the failure mode.

If this PR is desirable (which I am not sure it is because it has a non zero overhead), we should probably look at all
position_ids and copy the behavior at some point.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten
@lhoestq

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Create model cards for guwenbert

Related to pretrained model cards

Add two model_cards: ethanyt/guwenbert-base and ethanyt/guwenbert-large


Converting Transformers model to Tensorflow model

I am trying to convert `dbmdz/bert-base-turkish-uncased` to tensorflow model (`.pb`). It contains `tf_model.h5` file.

I tried to convert from `tf_model.h5` to tensorflow model. However, I couldn't handle it. Is `tf_model.h5` file keras model, isn’t it?

Is there any instruction to convert huggingface non-tf models to tensorflow model(`.pb`)?


Why the functions "add_special_tokens()" and "resize_token_embeddings()" hurt the performance of 'gpt2' and 'gpt2-medium' but not 'gpt2-large' and 'gpt2-xl' ?

# ❓ Questions & Help

## Details
When I use add_special_tokens and resize_token_embeddings to expand the vocabulary, the LM loss would become very large in gpt2 and gpt2-medium models (loaded by from_pretrained('gpt2') and from_pretrained('gpt2-medium)). But it don't happen when I load the gpt2-large and gpt2-xl models (also loaded by from_pretrained).  Why?

Environment Info:
python 3.7.7
Linux 16.04
transformers 3.3.1
pytorch 1.6.0

Codes and results:
'''
import torch
from transformers import GPT2Tokenizer
from transformers import GPT2LMHeadModel
device = torch.device('cuda:3')
input_sentence = 'who win this game?'

gpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
gpt2model = GPT2LMHeadModel.from_pretrained('gpt2', return_dict=True)
gpt2model.to(device)
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(5.0102, device='cuda:3', grad_fn=<NllLossBackward>)

gpt2tokenizer.add_special_tokens({'additional_special_tokens': ['[first]', '[second]']})
gpt2model.resize_token_embeddings(len(gpt2tokenizer))
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(77.1513, device='cuda:3', grad_fn=<NllLossBackward>)

gpt2tokenizer = GPT2Tokenizer.from_pretrained('gpt2-large')
gpt2model = GPT2LMHeadModel.from_pretrained('gpt2-large', return_dict=True)
gpt2model.to(device)
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(5.1567, device='cuda:3', grad_fn=<NllLossBackward>)

gpt2tokenizer.add_special_tokens({'additional_special_tokens': ['[first]', '[second]']})
gpt2model.resize_token_embeddings(len(gpt2tokenizer))
input = gpt2tokenizer(input_sentence, return_tensors='pt').to(device)
outputs = gpt2model(**input, labels=input['input_ids'])
outputs.loss

tensor(5.1568, device='cuda:3', grad_fn=<NllLossBackward>)
'''


Model Card for Gujarati-XLM-R-Base

Related to pretrained model cards

This PR adds the model card for the Gujarati-XLM-R-Base.


RAG: Do we need to pretrained the doc-encoder when using a custom dataset?

Now the RAG consists of a [script](https://github.com/huggingface/transformers/blob/master/examples/rag/use_own_knowledge_dataset.py) where we can use a custom dataset other than the wiki-dataset.

Since, in the fine-tuning phase of the RAG, we do not update the doc-encoder (we update only BART and Question Encoder), what if our custom dataset consists of different distribution compared to the wiki dataset (Ex: medical records)?

Will it still work?


Add mixed precision evaluation

# What does this PR do?

Add flag and code to do mixed precision forward in trainer's `prediction_step` function.
Let evaluation (and prediction) to run faster.

## Who can review?

@sgugger