Corrected a typo in the ReadMe

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Fix typo in docstring in src/transformers/models/bert_japanese/tokenization_bert_japanese.py

# What does this PR do?
Only fix typo (thi -> this).

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

@sgugger, @LysandreJik


Using doc chunks without answer token during training ( BertForQuestionAnswering )

Hi!

When creating features from the squad examples, you use sliding window approach to generate doc chunks, when the input data is too long. There you state, **_if the document chunk does not contain an annotation, you throw it out, since there is nothing to predict._** Hence you set the start_position=0 and end_position=0. Then you re-set it to cls_index in training mode. When you do **CLS token at the beginning, then your cls_index = 0**. Finishing up, you add this item to the InputFeatures.
https://github.com/huggingface/transformers/blob/e768f2322abd2a2f60a3a6d64a6a94c2d957fe89/examples/utils_squad.py#L332-L351

During training - forward step, you calculate loss on these items:
Let's say the max seq length is 512, then the ignored_index = 512, meaning you only ignore those start/end positions which are >= 512. With cls_index = 0 we have start_position=0 and end_position=0. So we end up getting a loss calculated using the start_position with start_logits and end_position with end_logits.
https://github.com/huggingface/transformers/blob/a8c3f9aa760ed7b516ee00f602e8efc0e5d80285/src/transformers/models/bert/modeling_bert.py#L1651-L1665

Then **you return with the calculated loss and in run_squad.py you add this new loss to the training loss**:
https://github.com/huggingface/transformers/blob/a8c3f9aa760ed7b516ee00f602e8efc0e5d80285/examples/question-answering/run_squad.py#L219

**So actually you do not throw these chunks out, but use them for training.**

**Solution maybe:**
Why not set the start_position = end_position = **max_seq_length** instead of cls_index in the utils_squad.py, making sure they will be ignored during training and the loss calculated with them will be 0 ??

Hope you will understand my point, let me know what you think!

**Update:**
I got this idea from another repo using bits and pieces from your implementation (they only used labels with answer, so squad-v1 like dataset) :
In case of too long sequences the sliding window approach creates multiple doc chunks, some of those are without an annotated answer. The model learns to predict start and end positions as 0  (if CLS is on 0. pos) when the answer is NOT present in the document chunk. This will help reduce false predictions on test data where documents are too long and split into multiple chunks.

**So either you actually want to use those doc chunks with no annotation, but your comments are misleading,
OR you don't want to use those chunks, hence the comment, but you failed to implement it.**

Hope it helps!


[trainer] improve code readability

This PR:

* removes redundant code, as:
```
self.model = model if model is not None else None
```
and
```
self.model = model
```
are the same.

* decouples attribute assignment from code logic - which simplifies things further.

@sgugger, @LysandreJik


fix(pipeline): error when model not in AutoModel

# What does this PR do?

During pipeline initialize, it will call get_framework to check whether tf or pt model. get_framework will call AutoModel to load model and return where it depends on.

However, not all the models are in AutoModel. For example, `Helsinki-NLP/opus-mt-en-fr` is under AutoModelForSeq2SeqLM which will cause an error when calling pipeline.

get_framework should depend on task instead. This fix pass targeted_task
to get_framework,  if the model not in AutoModel, it will use the targeted_task model instead.

error example
```
from transformers import pipeline
model = pipeline(task="translation_en_to_fr",model="Helsinki-NLP/opus-mt-en-fr")
```

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Removing Head Layer/Model Conversion

I am currently working on some research in which I am to delve into the analysis of decision boundaries in text classification tasks and I am aiming to use recent work from the `ExBERT` paper, allowing me to visualise the importance of particular features across sentences.

Since the library is built on top of models from the Transformers library and requires that *The model architecture must be supported by the `AutoModelWithLMHead`*, I was wondering if it was possible to modify a fine-tuned model to work with that architecture. I am currently using `DistilBERTForSequenceClassification` in my pipeline and was wondering if it were possible to essentially fine-tune for a classification task and use the underlying `DistilBERT` model, as I assume all of the attention weights etc. will still be included in the model? ie. Could I change the loaded model after training to work with the library and to work with the `AutoModelWithLMHead` architecture so that I could inspect the attention heads?

I wasn't sure if I was only able to use models trained for Masked LM or if I could use models trained for downstream tasks? Apologies if this is a question best for the ExBERT github but since it was built into the library, thought I'd ask.


[Bart] Refactor - fix issues, consistency with the library, naming

# What does this PR do?

This PR refactors the Bart model. The goal is to fix a couple of bugs related to Bart, make Bart more consistent with other models in the library and make Bart the "default" Seq2Seq template model for other models. The PR may be a bit difficult to review, so the following sections lists the main changes and the reasons why they are taken.

## In-detail explanation of main changes

1. Fix a bug related to `past_key_values`, `use_cache` and `decoder_input_ids`. Previously it was assumed that if `use_cache=True`, then `decoder_input_ids` have to be of length 1. This is not always the case! E.g. If the first decoder_input_ids prompt is longer than 1 and `use_cache=True` this would have led to errors previously - see #7814, #6353. This is fixed now so that any length of `past_key_values` can be combined with any length of `decoder_input_ids`, just as it can be done for GPT2, T5, CTRL, ... In order to make the pt_tf_equivalence tests pass, some hotfixes are applied for TFBart. TFBart will be refactored in a later PR. A test `create_and_check_decoder_model_past_large_inputs` is added to ensure that this functionality works.

2. Allow to use `BartEncoder` and `BartDecoder` separately from the `BartModel`. Because Bart is the default seq2seq model it's a great opportunity to combine just the `BartDecoder` with other "encoder-only" models. E.g. if someone wants to run experiments on long-range summarization `Longformer-Bart` could be an interesting combination (@ibeltagy). This PR lays the groundwork to easily combine these models by making `BartEncoder` and `BartDecoder` fully functional models on their own. One should probably also add a `BartForCausalLM` class analogs to https://github.com/huggingface/transformers/blob/df311a5ccf50be3031474e289b43b1be43111144/src/transformers/models/prophetnet/modeling_prophetnet.py#L1882 (could be a good first issue). further improves how to handle an issue like #5282

3. Simplify query, key, value projections in attention layer. A rather difficult if-else cascade with a complex follow-up function to concat past_key_values is simplified to a single if-elif-elif-else clause. IMO, the code in `BartAttention.forward()` is much clearer now.

4. Change the cache from dict to tuple and make it stateless. The general design in the library is to have a stateless design for the cache. Bart previously used a dict -> this PR changes the cache to a tuple for consistency. It should also be a bit more efficient, more consistent and easier to use with torchscript and onnx.

5. Bart did a lot of dimensions transposing from time -> batch and batch -> time. This is not at all necessary IMO. We can just have the batch dimension in the first spot the whole time just like the other models do too. Therefore, I deleted a bunch of `transpose(0, 1)` operations.

6. Add inputs_embeds. Just like other models Bart can make use of inputs_embeds.

7. Rename all classes from `...Model` to `Bart...Model`. Public class names that needed to be renamed were depreciated for backwards compatibility. This is better for look-up and consistency with other models.

8. Simpler handling of attention_masks. Previously Bart moved many different masks with many different names through-out the model. This PR aligns the functionality with other models, but creating the full attention mask for each model in the beginning of `BartEncoder` and `BartDecoder` instead of doing it in the attention function. This simplifies the code and is more consistent with other models.

9. Re-structure order in `modeling_bart.py`. Usually, modeling files have helper functions in the beginning, followed by submodules, followed by docstring, followed by the pre-trained models. This PR re-orders `modeling_bart.py` accordingly.

10. Replace functionality to make lm head embeddings on-the-fly by the usual `_init_weights` tying mechanism that we have in PyTorch. This is a) much more consistent with other models and b) cleaner because we don't have to instantiate a new class each time `get_output_embeddings()` is called. Solves #5282.

11. (subjectively) better naming. Replace x -> hidden_states, etc...

## Breaking changes

- There are no breaking changes to the "public" API IMO (except if it corrects a bug). `BartModel`, `BartForConditionalGeneration` and all other `BartPretrainedModel`s have exactly the same as before except for the following case which was a bug:
Previously, the following code:

```python
from transformers import BartForConditionalGeneration, BartTokenizer
model = BartForConditionalGeneration.from_pretrained("facebook/bart-large-cnn")
tokenizer = BartTokenizer.from_pretrained("facebook/bart-large-cnn")

input_ids = tokenizer("the encoded sequence in all its beauty", return_tensors="pt").input_ids
decoder_input_ids = tokenizer("the decoder sequence", return_tensors="pt").input_ids

print(model(input_ids, decoder_input_ids=decoder_input_ids).logits.shape)
```

would have printed out only a single output because `use_cache` is enabled which was wrong because no causal-mask was used. This PR corrects the behavior so that the output seq length matches the decoder_input_ids seq lengths.

- BartEncoder and BartDecoder now have a rather different API. This is OK for me since it was not possible to import the models directly and there were only model components.

- sub modules of Bart are named differently, *e.g.* LayerNorm is now called BartLayerNorm. Since these modules are also not public I don't think we have to depreciate the names.

- The API of `BartModel`, ... is extended by `inputs_embeds` and `decoder_inputs_embeds`.

## Review:

Because Bart is the most important Seq2Seq model in the library (5 other models classes depend on it), I would be very happy for a couple of thorough reviews. Also all kinds of comments, improvements, discussions, questions are welcome! I ran all slow tests and tried to be careful with the changes. In case @sshleifer is interested I'd also be more than happy about some feedback from you ;-)

## TODO-List

- [x] Keep dims consistent within the model -> no switching around between time x batch_size and batch_size x time. We can just stick to batch_size x time throughout the whole forward pass just like other models do too.
- [x] Add same `lm_head` logic, other models have as well. Bart should make use of the `tie_weight_embeddings` function instead of doing weird `"on-the-fly"` output embeddings, #5282
- [x] Clean the Attention layer: Replace dict cache by past_key_values tuple (consistency with other models and stateless which is better IMO). Break up complicated if-else cascade and remove unnecessary parameters.
- [x] Make Encoder/Decoder stand-alone models to be used on their own: #7127, this way pretrained weights can be used in the Encoder-Decoder framework as well. If I remember correctly @ibeltagy was interested in this as well
- [x] Correct error with past_key_values/decoder_input_ids/use_cache: #7814, #6353,
- [x] Make Bart torchscriptable: #6348
- [x] Add input_embeds to Bart
- [x] (very subjectively) better naming
- [x] Check that all slow tests are passing - ran the following slow tests:
```
[
# assumes USE_CUDA is exported, rather than passed
RUN_SLOW=1 pytest tests/test_modeling_pegasus.py
RUN_SLOW=1 pytest tests/test_modeling_bart.py
RUN_SLOW=1 pytest tests/test_modeling_marian.py
RUN_SLOW=1 pytest tests/test_modeling_mbart.py
RUN_SLOW=1 pytest tests/test_modeling_fsmt.py
RUN_SLOW=1 pytest tests/test_modeling_blenderbot.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_conversational.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_text2text_generation.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_summarization.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_translation.py
RUN_PIPELINE_TESTS=1 RUN_SLOW=1 pytest tests/test_pipelines_dialog.py
]
```
=> MBartEnroIntegrationTest.test_enro_generate_batch fails on PR, but also on master with the same message, so that's ok for me!

- [x] Update docstring and final design change check
- [x] Refactor Bart tests
- [x] Check no speed regression
- [x] Check no training performance regression (Is there a good fine-tuning script I can run for this @patil-suraj, @sshleifer)?


Wrong Length of Dataset in examples/seq2seq/finetune_trainer.py

`train/validation/test examples. -1 means use all` may be not correct

https://github.com/huggingface/transformers/blob/693ac3594b96e86dd282fdf8e413f3a48b176892/examples/seq2seq/finetune_trainer.py#L97-L99

`n_train/val/test` is used to compute the length of the dataset, there will lack one line if set to -1. It should be None to use all examples.

https://github.com/huggingface/transformers/blob/693ac3594b96e86dd282fdf8e413f3a48b176892/examples/seq2seq/utils.py#L136-L137


Hi
I am trying with master branch getting this error when installign requirements inside examples
thanks.

Collecting seqeval (from -r ../requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9d/2d/233c79d5b4e5ab1dbf111242299153f3caddddbb691219f363ad55ce783d/seqeval-1.2.2.tar.gz (43kB)
    100% |████████████████████████████████| 51kB 13.4MB/s
Collecting psutil (from -r ../requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/33/e0/82d459af36bda999f82c7ea86c67610591cf5556168f48fd6509e5fa154d/psutil-5.7.3.tar.gz (465kB)
    100% |████████████████████████████████| 471kB 2.7MB/s
Collecting sacrebleu (from -r ../requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/b9/d6/258a1e63463b4731a387f0872dca759c330bf4845cc0464f2c65028674b6/sacrebleu-1.3.7.tar.gz
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-Q_fyRn/sacrebleu/setup.py", line 65, in <module>
        version = get_version(),
      File "/tmp/pip-install-Q_fyRn/sacrebleu/setup.py", line 56, in get_version
        with open(os.path.join(os.path.dirname(__file__), 'sacrebleu.py'), encoding='utf-8') as fin:
    TypeError: 'encoding' is an invalid keyword argument for this function

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-Q_fyRn/sacrebleu/


finetune_trainer with python -m torch.distributed.launch

Hi
I need to run the finetune_trainer with multiple gpus, I am getting the error of

"Default process group is not initialized"
AssertionError: Default process group is not initialized

I am using custom dataloader, might be hard to share all the parts of codes, but I defined sampler as DistributedSampler.
this is transformer 3.5.1, python 3.7 on GPU.

thanks
Best
Rabeeh


Create README.md

model card

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Create README.md

Related to pretrained model cards


custom prepare_inputs_for_generation for generation

Hi
I need to change  model_inputs used for the generation, I am using T5ForConditionalGeneration which has extra input parameter and this needs to be passed in each time I call model.generate(), I cannot see how to rewrite the generate function to also pass this argument, could you provide me with some explanation:

https://github.com/huggingface/transformers/blob/d5b3e56de5376aa85ef46e7f0325139d9e299a41/src/transformers/generation_utils.py#L676

thanks


[🚀 Feature request]  Performer support, tensorflow code, not jax.

https://arxiv.org/abs/2009.14794

Thank you thank you very much.


TFRag draft #1 (page BROKEN) - Should close and use #9002 instead

# What does this PR do?
Hi guys, this is the draft WIP of TFRag. It is runnable in eager mode with mostly proper outputs.
I really need help/consult at this stage especially from @Patrick and @jplu .
In this draft, only `modeling_tf_rag.py` is new here.

## What is done and test. (working on HF 4.0.0, but not on Master due to changes in TF input)
- TFRagModel
- TFRagTokenForGeneration
- generate function with no_beam_search
- work on eager mode
- Colab notebook to test/play around whether the code works properly : https://colab.research.google.com/drive/1CfCulkKGrneiQ0gV0Bgdo71gZ_kgMRIB?usp=sharing

## Main things not done yet
- TFRagSequenceForGeneration
- beam_search generation (may wait for TF generation refactor ?)
- Working in graph mode (due to the need of `.numpy()` for retriever calling, and this doesn't work on graph mode)
- Change input format for HF 4.1.0 (need help from @jplu)

## Need your suggestion on NEED_ADVICE, NEED_HELP
As stated, the code is mostly OK except on the points I marked TOFIX which will be clean later during finishing the draft.
However, there are 2 categories that I really need help especially from @Patrick:

1) There are some codes that work, but I am not sure if I meet Huggingface coding standard (marked by NEED_ADVICE)

2) There are 2 points that I need real help (marked by  NEED_HELP )
2.1) the aforementioned `.numpy()` on graph mode.
2.2) about `.from_pretrained` : Rag has two loading methods which are `.from_pretrained_question_encoder_generator` and `.from_pretrained` . While` .from_pretrained_question_encoder_generator` works , in `.from_pretrained` there are two weights that the names do not match , which I could not find any way to fix :

```
'rag.generator.model.shared.weight', 'rag.generator.final_logits_bias' --> Pytorch name
'model.shared.weight', 'final_logits_bias' --> TF name
```
So at the moment I made UGLY fix by overwrite .from_pretrained and manually loading these two weights.

## Who can review?
TFRag : @patrickvonplaten ,
new TF input for master / 4.1.0 : @jplu
about graph mode & retriever module : will need help from @lhoestq later once all other issues are fixed :)


providing an example with a dummy iterative dataloaders

Hi
I have tested the trainer.py with iterative datasets and this does not work in distributed case, I shard the data across the cores. Could you please assist me in providing me with a dummy iterative dataloader for finetune_seq2seq.py model which runs fine with xla_spawn.py on TPU so I get some understanding which functions needs to be implemented. I really need to make this work, and trainer.py does not seem to work with iterative datasets or I am missing how to do it properly. thanks

@sgugger


Update generation_beam_search.py

BeamHypotheses.add() now behave differently depending on whether it finished with or without EOS token.

# What does this PR do?

see the discussion here #8722

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


trainer.py does not handle distributed training for iterative datasets and is very slow

## Environment info
- `transformers` version: 3.5.1
- Platform: TPU
- Python version: 3.7
- using xla_spawn.py

### Who can help
@sgugger
@patrickvonplaten
@patrickvonplaten
@patil-suraj

## Information
I am running seq2seq_finetune.py with iterative datasets and I do not get any speed up for 8 TPU cores versus 1 TPU cores, the code is also even slower than 1 GPU.

## To reproduce
```
git clone git@github.com:google-research/ruse.git
go to iter branch
pip install -r requirements.txt
python setup.py develop
cd seq2seq
python xla_spawn.py finetune_t5_trainer.py  configs/mrpc_adapter_tpu.json

```

## Expected behavior
Being faster on TPU, to me trainer.py does not handle iterative datasets properly. could you have a look please? thank you for your help.


clip_grad_norm on Multiple GPUs: (CUDA error: device-side assert triggered)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.0
- Platform: Linux-5.4.0-53-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help
@LysandreJik
@sgugger

## Information

Model I am using (Bert, XLNet ...):
RoBERTa

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
trainer.train() runs for a bit, then fails with the following output:
```
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-3435b262f1ae> in <module>
----> 1 trainer.train()

~/anaconda3/envs/transformers/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path, trial)
    759                         torch.nn.utils.clip_grad_norm_(amp.master_params(self.optimizer), self.args.max_grad_norm)
    760                     else:
--> 761                         torch.nn.utils.clip_grad_norm_(model.parameters(), self.args.max_grad_norm)
    762
    763                     if is_torch_tpu_available():

~/anaconda3/envs/transformers/lib/python3.7/site-packages/torch/nn/utils/clip_grad.py in clip_grad_norm_(parameters, max_norm, norm_type)
     33         total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(), norm_type).to(device) for p in parameters]), norm_type)
     34     clip_coef = max_norm / (total_norm + 1e-6)
---> 35     if clip_coef < 1:
     36         for p in parameters:
     37             p.grad.detach().mul_(clip_coef.to(p.grad.device))

RuntimeError: CUDA error: device-side assert triggered
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Training RoBERTa for sequence classification from text to binary.

## To reproduce
Steps to reproduce the behavior:
1. Load pre-processed dataset from disk using datasets.Dataset.load_from_disk()
2. Instantiate RoBERTa from pretrained (roberta-base) with config mods (num_labels = 2)
3. Create and run trainer. See full code below (most imports omitted).

```
from transformers import (RobertaTokenizerFast)

BLOCK_SIZE = 512
tok = RobertaTokenizerFast.from_pretrained("./art_tok_onefile_roberta_tuned/")

ds_root = '/media/b/My Passport/datasets/'
tokenized = datasets.Dataset.load_from_disk(os.path.join(ds_root, 'art_unit_tokenized_balanced'))

columns_to_return = ['input_ids', 'attention_mask', 'labels']
tokenized.set_format(type='torch', columns=columns_to_return)

from transformers import RobertaForSequenceClassification

config = RobertaConfig(
    vocab_size=tok.vocab_size,
    max_position_embeddings=514,
    num_labels = 2
)

config = RobertaConfig.from_pretrained("roberta-base",
                                        vocab_size=tok.vocab_size,
                                        max_position_embeddings=514,
                                        num_labels = 2)

model = RobertaForSequenceClassification.from_pretrained('roberta-base', config=config)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-5)

for param in model.base_model.parameters():
    param.requires_grad = False

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./roberta_train_test",
    overwrite_output_dir=True,
    num_train_epochs=5,
    per_device_train_batch_size=128,
    save_steps=50,
    save_total_limit=2,
    logging_steps=10,
    #fp16 = True #Enable low-precision via AMP - omitted for now.
)

train_test_bal = tokenized.train_test_split(test_size=0.1)

trainer = Trainer(
    model=model,
    args=training_args,
    #data_collator=collate_fn,
    train_dataset=train_test_bal['train']
)

trainer.train()
```

## Expected behavior

The model trains for the duration of the training cycle.


Hi everyone,

I ran [ run_mlm.py ](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py) to continue pertaining uncased BERT directly from the examples on this repo, but once I load the newly saved pretrained Bert Model, I receive a warning - "'Some weights of BertModel were not initialized from the model checkpoint at ./model and are newly initialized: ['bert.pooler.dense.weight', 'bert.pooler.dense.bias']'"

I'm trying to fine-tune the model on a sentiment analysis task, but I'm getting horrible results and I wonder if it has something to do with this? Thanks for your help.


UnicodeEncodeError: surrogates not allowed with GPT2Tokenizer

## Environment info

- `transformers` version: 3.1.0
- Platform: EC2
- Python version: 3.6

### Who can help
@mfuntowicz @LysandreJik

## Information

Model I am using: GPT-2

The problem arises when using the `GPT2Tokenizer` on a piece of text from a file that was written `utf-8` strings and is being opened in `utf-8`.

## To reproduce

```
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-xl")
text = "some utf-8 string" # this string is loaded from a file containing a dictionary {"text": "<some text>"} in each row - the file itself was written by converting TFRecords to text and "<some text>" was decoded explicitly to "utf-8" prior to being dumped into this dictionary and written
text_ids = tokenizer.encode(text)
```

Stack trace I get:

```
Traceback (most recent call last):
text_ids = tokenizer.encode(text)
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 1730, in encode
**kwargs,
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 2045, in encode_plus
**kwargs,
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 448, in _encode_plus
first_ids = get_input_ids(text)
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 419, in get_input_ids
tokens = self.tokenize(text, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 350, in tokenize
tokenized_text = split_on_tokens(no_split_token, text)
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 344, in split_on_tokens
for token in tokenized_text
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py", line 344, in <genexpr>
for token in tokenized_text
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_gpt2.py", line 237, in _tokenize
self.byte_encoder[b] for b in token.encode("utf-8")
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 0-1: surrogates not allowed
```

## Expected behavior

It should tokenize and then convert the tokens to ids just fine, since `text`  is a `utf-8` string.

I'm trying to specifically identify the `text` itself from my file that leads to this error, but I am unable to print it either. I used a try-except block to catch the above `UnicodeEncodeError` and tried to print the `text`, but print itself expectedly failed because print is using the `ascii` codec. Is there a good way for me to identify the exact piece of text that led to this failure? Perhaps it'll help assist with debugging this issue.


[ci] skip doc jobs take #3

@LysandreJik found another edge case when a developer force-pushes a change and `pipeline.git.base_revision` is defined but bogus, resulting in a range that returns no files. https://github.com/huggingface/transformers/pull/8853#issuecomment-736781950

So the proposed logic for take 3 is:

1. if pipeline.git.base_revision and pipeline.git.revision are defined
2. if git diff --name-only range returns anything
3. if what it returned in 2 is just docs
4. then skip

Bottom line, we skip the test altogether if:
```
unless test -n "<< pipeline.git.base_revision >>" && test -n "<< pipeline.git.revision >>" \
&& test -n "$(git diff --name-only << pipeline.git.base_revision >>...<< pipeline.git.revision >>)"
```

@LysandreJik, @sgugger


[s2s finetune_trainer] add instructions for distributed training

This PR adds instructions for running finetune_trainer.py under dpp

@patrickvonplaten


Extracting important information

I'm trying to extract important information from a lecture transcript. What's the best way to go about doing this ? This would be without a particular query parameter, just generally important information in the global context of the lecture.


[trainer] start using training_args.parallel_mode

Following up on https://github.com/huggingface/transformers/pull/8877 which adds `training_args.parallel_mode` to make it easy to comprehend which mode the trainer is running under - this PR deploys the new property in a few places.

@sgugger, have I deployed it as you envisioned it?


Better warning when loading a tokenizer with AutoTokenizer w/o Sneten…

…cePiece

Currently, initializing a `sentencepiece` `AutoTokenizer` without having `sentencepiece` installed results in the following error:

```
AttributeError: 'NoneType' object has no attribute 'from_pretrained'
```
This improves the error message to:
```
 This tokenizer cannot be instantiated. Please make sure you have `sentencepiece` installed in order to use this tokenizer.
```

Fix #8864


[PyTorch] Refactor Resize Token Embeddings

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR extends the `resize_embeddings` function in PyTorch to models that have input/output embeddings that are **not** tied.
In PyTorch all models that have tied input/output embeddings by default can also untie those embeddings by setting `config.tie_word_embeddings=False`. This however requires the `_resize_token_embeddings` to be extended to also resize the `lm_head`. This PR does this extension by adding a `_get_resized_lm_head` method. Also, all models that have a `get_output_embedding()` function, now need a `set_output_embedding()` function. A test is added to make sure the new functionality works as expected. The Bart-like models currently skip this test because there is a rather weird `lm_head` behavior that I want to refactor in another PR.

In addition this PR:

- Fixes #8706: With MT5 and T5v1_1, T5 now has a configuration where input and output embeddings are not tied anymore. This PR fixes this.
- Refactors MobileBert

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


dropout(): argument 'input' (position 1) must be Tensor, not str With Bert

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: google colab
- Python version: 3
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): Bert

@LysandreJik
@jplu

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I am trying to do sentiment analysis using Bert. My code was working perfectly fine and then last night I tried to run it without changing anything and I am getting the following error message:

"dropout(): argument 'input' (position 1) must be Tensor, not str"

I trained my Bert model and saved the bin file. This occurs when I load the bin file into collab and try to predict the sentiment of any text.

## To reproduce

Steps to reproduce the behavior:

1. Loaded my model that was saved in a bin file in google colab
2. Ran the following code:

`def conclude_sentiment(text):
  encoded_review = tokenizer.encode_plus(
    text,
    max_length=MAX_LEN,
    add_special_tokens=True,
    return_token_type_ids=False,
    pad_to_max_length=True,
    return_attention_mask=True,
    return_tensors='pt',
  )

  input_ids = encoded_review['input_ids'].to(device)
  attention_mask = encoded_review['attention_mask'].to(device)

  output = model(input_ids, attention_mask)
  _, prediction = torch.max(output, dim=1)

  #print(f'Review text: {text}')
  #print(f'Sentiment  : {class_names[prediction]}')
  return class_names[prediction]`

3. Got an error that says

`/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in dropout(input, p, training, inplace)
    981     return (_VF.dropout_(input, p, training)
    982             if inplace
--> 983             else _VF.dropout(input, p, training))
    984
    985

TypeError: dropout(): argument 'input' (position 1) must be Tensor, not str`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
An output of either 'positive' or 'negative' when a string is passed into the method named 'conclude_sentiment'


Better support for resuming training

# What does this PR do?

This PR adds two things linked to resuming training:

1. It brings full reproducibility when resuming an interrupted training from a checkpoint (i.e., resuming a training from a checkpoint will give the exact same results as a training from the beginning with the same seeding). This was not currently the case because the dataloader shuffle was not triggered `epochs_already_trained` times, so the shuffle of the dataloader was the same as epoch 0. So the full reproducibility was only there for trainings resumed from an early checkpoint (during the first epoch).

2. It also adds the option to ignore that data skipping which can take a very long time on a large dataset. This will go faster but yield different results from a training from scratch.

Fixes #8874 and #8876


Add a `parallel_mode` property to TrainingArguments

# What does this PR do?

This PR adds a `distributed_env` property to the `TrainingArugments` making it clear if we are in:
- a single process (CPU or one GPU)
- a parallel setting (one process but several GPUs)
- a distributed parallel setting (several processes, one per GPU)
- a TPU setting

Fixes #8858


Resume training from checkpoint: not progressing

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.0
- Platform: Linux-3.10.0-514.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.2
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
### Who can help

@sgugger

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [x] the official example scripts: /examples/language-modeling/run_mlm.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: BERT MLM pre-training with own dataset

## To reproduce

Steps to reproduce the behavior:

1. Run script run_mlm.py, training from scratch, and save a checkpoint.
2. Stop the training.
3. Restore the training from the checkpoint, e.g. with the code below
4. When restoring, the pre-training process is not progressing (since hours).

```cmd
python run_mlm.py     --model_type bert     --model_name_or_path /bert-base-v2/checkpoint-204516/
--overwrite_output_dir     --config_name /bert-base-v2-config/     --tokenizer_name /bert-base-v2-config/
 --train_file /train_subset.txt     --validation_file /eval_subset.txt      --do_train     --do_eval     --line_by_line
--output_dir /bert-base-v2/     --cache_dir /tmp/     --save_total_limit 300     --num_train_epochs 10
  --warmup_steps 10000     --logging_steps 5000     --save_steps 11362
 --per_device_train_batch_size 128     --per_device_eval_batch_size 128     --seed 42
```
Output is
```
12/01/2020 15:43:28 - INFO - __main__ -   Loading tokenized dataset from file...
12/01/2020 15:47:22 - INFO - __main__ -   Done.
[INFO|trainer.py:357] 2020-12-01 15:47:29,458 >> The following columns in the training set don't have a corresponding argument in `BertForMaskedLM.forward` and have been ignored: special_tokens_mask.
[INFO|trainer.py:357] 2020-12-01 15:47:29,459 >> The following columns in the evaluation set don't have a corresponding argument in `BertForMaskedLM.forward` and have been ignored: special_tokens_mask.
[INFO|trainer.py:662] 2020-12-01 15:47:32,843 >> ***** Running training *****
[INFO|trainer.py:663] 2020-12-01 15:47:32,843 >>   Num examples = 145434960
[INFO|trainer.py:664] 2020-12-01 15:47:32,843 >>   Num Epochs = 10
[INFO|trainer.py:665] 2020-12-01 15:47:32,843 >>   Instantaneous batch size per device = 128
[INFO|trainer.py:666] 2020-12-01 15:47:32,843 >>   Total train batch size (w. parallel, distributed & accumulation) = 128
[INFO|trainer.py:667] 2020-12-01 15:47:32,843 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:668] 2020-12-01 15:47:32,843 >>   Total optimization steps = 11362110
[INFO|trainer.py:681] 2020-12-01 15:47:32,846 >>   Continuing training from checkpoint, will skip to saved global_step
[INFO|trainer.py:682] 2020-12-01 15:47:32,846 >>   Continuing training from epoch 0
[INFO|trainer.py:683] 2020-12-01 15:47:32,846 >>   Continuing training from global step 204516
[INFO|trainer.py:684] 2020-12-01 15:47:32,846 >>   Will skip the first 204516 batches in the first epoch
  0%|                                                                       | 0/11362110 [00:00<?, ?it/s]
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Would expect the training to restore from 204516 and continue training.


Fix mlflow parameter overflow

# What does this PR do?

This PR fixes the issue #8849 where MLflow logging failed due to parameters logged being too long. Now the MLflow logger also fetches the limits directly from MLflow validation utility.

<!-- Remove if not applicable -->

Fixes #8849

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger


Results are different when fine-tuning continues after loading model from checkpoint

## Environment info

- `transformers` version: 4.0.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes (device: cuda:0, n_gpu: 1)
- Using distributed or parallel set-up in script?: False

### Who can help

@sgugger
@stefan-it
## Information

Model I am using (Bert, XLNet ...): bert-base-cased

The problem arises when using:
* [x] the official example scripts: run_ner_old.py

The tasks I am working on is:
* [x] my own task or dataset:  token classification for a rhetoric device

## To reproduce

Steps to reproduce the behavior:

1. Run run_ner_old script and save model after one epoch (282 steps):

```
python3  ./run_ner_old.py \
--data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path bert-base-cased \
--output_dir ./output/ \
--max_seq_length  128 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--save_steps 282 \
--seed 1 \
--do_train \
--do_eval
```
2. Run ner_old_script from checkpoint-282:
```
python3  ./run_ner_old.py \
--data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path ./output/checkpoint-282 \
--tokenizer bert-base-cased \
--output_dir ./output2/ \
--max_seq_length  128 \
--num_train_epochs 2 \
--per_device_train_batch_size 16 \
--save_steps 282 \
--seed 1 \
--do_train \
--do_eval
```
3. Compare evaluation results

**First experiment:**
Run the script `run_ner_old.py` as showed above to fine-tune BERT.
I saved the model after the first epoch (282 steps).

**Second experiment:**
Run the script `run_ner_old.py` as showed above to fine-tune BERT, starting from checkpoint-282 from the first experiment:
```
[INFO|trainer.py:662] 2020-12-01 14:35:09,848 >> ***** Running training *****
[INFO|trainer.py:663] 2020-12-01 14:35:09,848 >>   Num examples = 4501
[INFO|trainer.py:664] 2020-12-01 14:35:09,848 >>   Num Epochs = 2
[INFO|trainer.py:665] 2020-12-01 14:35:09,849 >>   Instantaneous batch size per device = 16
[INFO|trainer.py:666] 2020-12-01 14:35:09,849 >>   Total train batch size (w. parallel, distributed & accumulation) = 16
[INFO|trainer.py:667] 2020-12-01 14:35:09,849 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:668] 2020-12-01 14:35:09,849 >>   Total optimization steps = 564
[INFO|trainer.py:681] 2020-12-01 14:35:09,851 >>   Continuing training from checkpoint, will skip to saved global_step
[INFO|trainer.py:682] 2020-12-01 14:35:09,851 >>   Continuing training from epoch 1
[INFO|trainer.py:683] 2020-12-01 14:35:09,851 >>   Continuing training from global step 282
[INFO|trainer.py:684] 2020-12-01 14:35:09,851 >>   Will skip the first 0 batches in the first epoch
```
This seems right as the training continues from step 282 and it trains one complete epoch ("skip the first 0 batches").

 But when I **compare the results**, they are slightly different:
1. experiment: eval_f1 = 0.9226747985188413
2. experiment: eval_f1 = 0.9211328976034858

Also the loss after 500 steps is already different:
1. experiment:
`{'loss': 0.09096851348876953, 'learning_rate': 5.673758865248227e-06, 'epoch': 1.773049645390071}
`
2. experiment:
`
{'loss': 0.010856814384460449, 'learning_rate': 5.673758865248227e-06, 'epoch': 1.773049645390071}
`

## Expected behavior

I would have expected that both trained models should produce the same results since the second experiment does exactly the same but in two steps. (The model is saved and loaded between the two epochs).

The *checkpoint-282* directory consists of the following files:
```
config.json
optimizer.pt
pytorch_model.bin
scheduler.pt
trainer_state.json
training_args.bin
vocab.txt
```
It does not seem that there is any random initialization since I added the seed and the results do not change when running again.

Did I forget to save or load anything?

Cheers


How to pass the attention mask as a param to model forward when using torchscript?

## Environment info

- `transformers` version:
- Platform: Ubuntu 16.04
- Python version: 3.6.9
- PyTorch version (GPU): 1.3.0+cu100
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

## Information
 I am using a Bert model downloaded from hugging face. I finetuned that model for a two-class classification task and convert that model to a `torchscript` through the `jit.trace`.

The following code show how I got the torchscript:

    from transformers import BertTokenizer, BertForSequenceClassification
    tokenizer = BertTokenizer.from_pretrained(tokenizer_dir)
    model = BertForSequenceClassification.from_pretrained(model_dir, num_labels=2, torchscript=True)
    model.eval()
    model = model.to("cuda:0")

    input_text = ["test this case", "test test this case"]
    encoding = tokenizer(text_batch, return_tensors='pt', padding=True, truncation=False)
    input_ids = encoding['input_ids']
    attention_mask = encoding['attention_mask']

    tokens_tensor = torch.tensor(input_ids)
    tokens_tensor = tokens_tensor.to("cuda:0")
    traced_model = torch.jit.trace(model, tokens_tensor)
    torch.jit.save(traced_model, str(pt_path))

The following code show how I use the torchscript, the input tensor is the same with the first code:

    pt_model = torch.jit.load(model_path))
    pt_model.eval()
    pt_label = pt_model(input_tensor)[0]

For the normal model, it needs to pass two params input tensor and attention mask like:

    model(input, attention_mask=attn_mask)

But for the `torchscript`, I can't pass the attention mask to the model.
So, what's the right way to use `torchscript` to do the forward with the attention mask?

Thanks!


Deberta Tokenizatiion

## Environment info

- `transformers` version: 4.0.0
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?): 1.7

### Who can help
@BigBird01 @LysandreJik

## Information

I'd like to use the new deberta model, but it seems that the tokens aren't mapped correctly?

```
from transformers import AutoTokenizer

test_string = 'hello, I am a dog'

tokenizer = AutoTokenizer.from_pretrained('roberta-base')
print('Roberta output is: ', tokenizer.tokenize(test_string))

tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-base')
print('Deberta output is: ', tokenizer.tokenize(test_string))
```

Roberta output is:  ['hello', ',', 'ĠI', 'Ġam', 'Ġa', 'Ġdog']
Deberta output is:  ['31373', '11', '314', '716', '257', '3290']

I'd expect deberta to give an output similar to roberta, rather than numbers.


Decrease Longformer window size / computational cost

Hi there,

I would like to use Longformer instead of BERT or ROBERTA for longer documents, e.g., 1024 subword units. My goal is to fit a batch of equal size in the same GPU card for all models. In my understanding, this cannot happen with the default configuration, which used windows of 512 subword-units for local attention. In other words, this is by default more computationally expensive than running BERT or ROBERTA. So I thought the solution would be to decrease the size of window proportionally to the increase of input sequence size. This will lead to equal or less computations.  I run the following experiments in a single RTX 2080Ti:

**Train ROBERTA with `batch_size=6` and `max_len=512` (SUCCESS)**

```python
from transformers import TFLongformerModel, LongformerConfig, TFRobertaModel
import tensorflow as tf
import numpy as np
import logging
logging.getLogger("tensorflow").setLevel(logging.ERROR)
logging.getLogger("transformers").setLevel(logging.ERROR)

class Classifier(tf.keras.Model):

    def __init__(self, bert_encoder, *args, **kwargs):
        super(Classifier, self).__init__(*args, **kwargs)
        self.classifier = tf.keras.layers.Dense(2)
        self.bert_encoder =  bert_encoder

    def call(self, inputs):
        bert_encodings = self.bert_encoder(inputs)
        return self.classifier(tf.squeeze(bert_encodings[0][:, 0:1, :], axis=1))

# Train ROBERTA for 512 TOKENS

roberta = TFRobertaModel.from_pretrained('roberta-base')
roberta_classifier = Classifier(bert_encoder=roberta)
dummy_inputs = np.zeros((6, 512), dtype=np.int32)
dummy_outputs =  np.zeros((6, 2), dtype=np.int32)

roberta_classifier.compile(optimizer='adam', loss='categorical_crossentropy')
roberta_classifier.fit(dummy_inputs, dummy_outputs, batch_size=8)
print('Roberta (512) trained successfully!')
```

**Train LONGFORMER with `batch_size=6` and `max_len=512` and `attention_window=512` (SUCCESS)**

```python
# Train LONG-FORMER for 512 TOKENS
config =  LongformerConfig.from_pretrained('allenai/longformer-base-4096')
config.attention_window = [512] * 12
longformer = TFLongformerModel(config)
longformer_classifier = Classifier(bert_encoder=longformer)
dummy_inputs = np.zeros((6, 512), dtype=np.int32)
dummy_outputs =  np.zeros((6, 2), dtype=np.int32)

longformer_classifier.compile(optimizer='adam', loss='categorical_crossentropy')
longformer_classifier.fit(dummy_inputs, dummy_outputs, batch_size=8)
print('Longformer (512) trained successfully!')
```

**Train LONGFORMER with `batch_size=6` and `max_len=1024` and `attention_window=128` (FAILED-OOM)**

```python
# Train LONG-FORMER for 1024 TOKENS
config =  LongformerConfig.from_pretrained('allenai/longformer-base-4096')
config.attention_window = [128] * 12
longformer = TFLongformerModel(config)
longformer_classifier = Classifier(bert_encoder=longformer)
dummy_inputs = np.zeros((6, 1024), dtype=np.int32)
dummy_outputs =  np.zeros((6, 2), dtype=np.int32)

longformer_classifier.compile(optimizer='adam', loss='categorical_crossentropy')
longformer_classifier.fit(dummy_inputs, dummy_outputs, batch_size=8)
print('Longformer (1024) trained successfully!')
```

The last script is failing with an OOM issue. The same happens for  `attention_window in [32,64]`

@patrickvonplaten  do I miss something? Thanks!


Token classification example only returns labels as -100 for longformer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.0
- Platform: Linux-5.4.0-1029-aws-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?:4 x V100 (16GB)
- Using distributed or parallel set-up in script?: parallel

Also:

tokenizers==0.9.4
datasets==1.1.3

I should note I had the same issue in transformers 3.5.1 too

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.  -->

 tokenizers: @mfuntowicz
 Longformer/Reformer: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

I have made very modifications to the token-classification example ([see here](https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py)) to allow me to use my own custom dataset for NER. I have 3 labels O, B-Org, I-ORG.

When processing my inputs with models like `bert-base-cased`, everything runs smoothly, however, when I make the switch to the `allenai/longformer-base-4096` model, the `tokenize_and_align_labels()` that runs via `datasets.map()` only returns labels of -100 for every token.

## To reproduce

Patch file for original `run_ner.py`

```
20d19
<
44d42
<
199d196
<
206,209c203,204
<     text_column_name = "tokens" if "tokens" in column_names else column_names[0]
<     label_column_name = (
<         f"{data_args.task_name}_tags" if f"{data_args.task_name}_tags" in column_names else column_names[1]
<     )
---
>     text_column_name = "words"
>     label_column_name = "ner"
213,217c208,213
<     def get_label_list(labels):
<         unique_labels = set()
<         for label in labels:
<             unique_labels = unique_labels | set(label)
<         label_list = list(unique_labels)
---
>     def get_label_list(label_lists):
>         label_list = list(set(
>             [label
>              for label_list in label_lists
>              for label in label_list]
>         ))
227a224
>         id_to_label = {i: l for i, l in enumerate(label_list)}
239a237,238
>         id2label=id_to_label,
>         label2id=label_to_id,
244a244
>         add_prefix_space=True,
```

minimal `train.json` data
```json
{
    "id": 169,
    "words": [
        "My", "favourite", "thing", "about", "the", "market", "was", "Manteigaria", "which", "sells", "the", "best", "pasteis", "de", "Nata", "in", "the", "city", "in", "my", "opinion", "."
    ],
    "ner": [
        "O", "O", "O", "O", "O", "O", "O", "B-Organisation", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O", "O"
    ]
}

```

Example `train-config.json`
```json
{
    "train_file": "./train.json",
    "model_name_or_path": "allenai/longformer-base-4096",
    "output_dir": "./output",
    "max_seq_length": 4096,
    "num_train_epochs": 3,
    "pad_to_max_length": false,
    "per_device_train_batch_size": 1,
    "per_device_eval_batch_size": 1,
    "save_steps": 250,
    "eval_steps": 250,
    "seed": 1,
    "do_train": true,
    "do_eval": false,
    "do_predict": false,
    "fp16": true,
    "evaluation_strategy": "steps",
    "save_total_limit": 1,
}

```

Steps to reproduce the behavior:

1. Apply the patch to the `run_ner.py` file found in the token_classification example on the master branch
2. run `python run_ner.py train-config.json`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
If you print `tokenized_inputs["labels"]` produced by `tokenize_and_align_labels()` you'll see that it is a list of `-100` when using `allenai/longformer-base-4096`. However, if you change this to `bert-base-cased`, it will produce the labels `[[-100, 1, 1, 1, 1, 1, 1, 1, 0, -100, -100, -100, 1, 1, 1, 1, 1, -100, 1, 1, -100, 1, 1, 1, 1, 1, 1, 1, -100]]`, which is correct as `1` is `O` and `0` is `B-Organisation`. (There won't be an `I-Organisation` as this minimal reproducible example doesn't have one).


Exporting ALBERT model to onnx increases model size by 7x

I'm trying to export `albert-base-v2` model to onnx using

`python -m transformers.convert_graph_to_onnx --framework pt --model albert-base-v2 --quantize albert.onnx --opset 12`

The original pytorch model size is around 45 MB (https://huggingface.co/albert-base-v2), but the exported model size is around 340 MB.

How do I keep the model's size same?


Transfoxl seq classification

This PR implements Sequence classification for Transformer XL model
TransfoxlForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-1,GPT-2) do.

Fixes #7623 (Partially)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@LysandreJik


length_penalty not influencing results (Bart, Pegasus)

Hello,

I am experimenting with the generative parameters of the two models Bart and Pegasus. In particular, I am having trouble with the `length_penalty` parameter, since changing it does not change the output of the model.

I am summarizing two different chapters of a book (# tokens around 1k) and this is the code I am using:

```
    model.generate(
        b0ch1sec1_text_enc,
        min_length = 150,
        max_length = 350,
        num_beams = 2,
        length_penalty = lp,
        early_stopping = True)[0]
```

With `lp` swiping from 0.1 to 2 and model being either `bart-large-cnn` or `pegasus-large`.

Do you have any idea why the output does not change at all?


different embedding weights for base-uncased with different transformers versions

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.0, 3.4.0 and 2.9.0
- Platform:
- Python version: 3.7.0
- PyTorch version: 1.4.0
- Tensorflow version: 2.2.0

## Information

Model I am using: Bert

The problem arises when using my own scripts. I trained a LayoutLM model by using the original Unilm repo (https://github.com/microsoft/unilm/tree/master/layoutlm) and obtained pretty good results (± 0.9 f1 score). When the Huggingface implementation came out, I retrained the model with the same dataset, parameters and seed and got rubbish results (less then 0.2 F1 score). After investigating, I found that the weights of the embeddings of the pretrained model, loaded at the beginning of training are different for different transformers versions. The weights are also different for the final trained model: a model trained with the original implementation gives different predict results for the same data when predicting using the Huggingface implementation, due to the weights being different after loading.

## To reproduce

Steps to reproduce the behavior:

Huggingface code:

```
from transformers import LayoutLMConfig, LayoutLMForTokenClassification

pretrained_model_path = "models/base-uncased"

config = LayoutLMConfig.from_pretrained(pretrained_model_path, num_labels=len(25))

model = LayoutLMForTokenClassification.from_pretrained(
        pretrained_model_path, from_tf=bool(".ckpt" in pretrained_model_path), config=config
    )

print(model.base_model._modules["embeddings"]._modules["word_embeddings"].weight)

"""transformers 4.0.0:
Parameter containing:
tensor([[-0.0211, -0.0056,  0.0198,  ...,  0.0119,  0.0074, -0.0048],
        [-0.0268,  0.0006,  0.0310,  ..., -0.0195, -0.0534,  0.0284],
        [ 0.0234,  0.0026, -0.0024,  ..., -0.0074, -0.0015, -0.0212],
        ...,
        [-0.0274, -0.0074,  0.0161,  ..., -0.0256,  0.0189, -0.0328],
        [-0.0350, -0.0304,  0.0087,  ..., -0.0349, -0.0086,  0.0229],
        [-0.0068, -0.0077, -0.0084,  ..., -0.0181, -0.0111,  0.0385]],
       requires_grad=True)
"""

"""transformers 3.4.0:
Parameter containing:
tensor([[ 0.0298, -0.0229, -0.0033,  ...,  0.0097, -0.0179, -0.0065],
        [-0.0098,  0.0150, -0.0283,  ..., -0.0424, -0.0031, -0.0135],
        [ 0.0122,  0.0038, -0.0066,  ..., -0.0261,  0.0167,  0.0176],
        ...,
        [ 0.0037,  0.0001,  0.0096,  ..., -0.0037, -0.0018,  0.0067],
        [ 0.0274,  0.0076,  0.0065,  ...,  0.0084, -0.0230, -0.0011],
        [-0.0155, -0.0155, -0.0028,  ..., -0.0140,  0.0084, -0.0016]],
       requires_grad=True)
"""
```

With original Layoutlm implementation, transformers 2.9.0:
```
from unilm.layoutlm.layoutlm import LayoutlmConfig, LayoutlmForTokenClassification

pretrained_model_path = "models/base-uncased"

config = LayoutlmConfig.from_pretrained(
        pretrained_model_path,
        num_labels=len(25),
    )
model = LayoutlmForTokenClassification.from_pretrained(
        pretrained_model_path,
        from_tf=bool(".ckpt" in pretrained_model_path),
        config=config,
    )

print(model.base_model._modules["embeddings"]._modules["word_embeddings"].weight)

"""
Parameter containing:
tensor([[-0.0111, -0.0777,  0.0293,  ..., -0.0323, -0.0190,  0.0403],
        [-0.0579, -0.0331, -0.0399,  ..., -0.0248, -0.0278, -0.0398],
        [-0.0261, -0.0383, -0.0225,  ...,  0.0011, -0.0803, -0.0019],
        ...,
        [-0.0186, -0.0593, -0.0167,  ..., -0.0243, -0.0096,  0.0050],
        [-0.0555, -0.0274,  0.0049,  ..., -0.0206, -0.0172, -0.0241],
        [-0.0328, -0.0788, -0.0211,  ..., -0.0187, -0.0497,  0.0444]],
       requires_grad=True)
"""
```

## Expected behavior

Get the same weights regardless the transformers version used.


can the BertModel convert to onnx? whether any one had done sucessfully ?


AttributeError: 'NoneType' object has no attribute 'from_pretrained'

This code was working yesterday but doesn't work today:

```py
from transformers import AutoTokenizer
AutoTokenizer("Helsinki-NLP/opus-mt-en-fr")
```


Unwanted left shift of target tokens in `get_nll`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4
- Platform: ubuntu
- Python version: 3.8
- PyTorch version (GPU?): 1.6
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 RAG: @patrickvonplaten, @lhoestq
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
I am training RAG on the FEVER dataset, trying to generate one token from `['[SUPPORTS]', '[REFUTES]', '[INCONCLUSIVE]'.  My loss function is always zero, I think because of the shift left that occurs in `RagTokenForGeneration.get_nll()`, which I think should only happen if special tokens are included.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
Training RAG on FEVER dataset

## To reproduce

Steps to reproduce the behavior:

1. Simply train `RAGTokenForGeneration` on anything, using only one token with no special tokens.  Loss is zero

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


TypeError: forward() got an unexpected keyword argument 'past'

```
        text1 = request.form['rawtext']
        m = text1
        text = tokenizer.encode(text1)
        myinput, past = torch.tensor([text]), None
        logits, past = model(myinput, past = past)
        logits = logits[0,-1]
        probabilities = torch.nn.functional.softmax(logits)
        best_logits, best_indices = logits.topk(780)
        best_words = [tokenizer.decode([idx.item()]) for idx in best_indices]
        text.append(best_indices[0].item())
        best_probabilities = probabilities[best_indices].tolist()
        for i in range(780):
            f = ('Generated {}: {}'.format(i, best_words[i]))
            print(f)
```

```
/content/GLPAPP
 * Serving Flask app "__main__" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Running on http://19bd405035a7.ngrok.io
 * Traffic stats available on http://127.0.0.1:4040
127.0.0.1 - - [30/Nov/2020 22:34:16] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/js/jquery.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/css/main.css HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/js/breakpoints.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/js/main.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/js/util.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/js/browser.min.js HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:19] "GET /static/css/fontawesome-all.min.css HTTP/1.1" 200 -
127.0.0.1 - - [30/Nov/2020 22:34:22] "GET /favicon.ico HTTP/1.1" 404 -
[2020-11-30 22:34:23,393] ERROR in app: Exception on /predict [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.6/dist-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.6/dist-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "<ipython-input-8-a5d8492f7c0c>", line 36, in predict
    logits, past = model(myinput, past = past)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'past'
127.0.0.1 - - [30/Nov/2020 22:34:23] "POST /predict HTTP/1.1" 500 -
```


Add warnings for incompatible generation parameters

# What does this PR do?
While testing various generation configurations, I got confused when beam_sample's outputs (i.e. `num_beams > 1, do_sample=True`) failed to change with different settings of `top_k, top_p, temperature`. Then I realized that in my call to `generate()`, `do_sample` was still set to false even though I was cycling through various settings of top_k, top_p and temperature.

Generate (and its helper methods) already contain some parameter compatibility checks. This adds a few more checks:
- `num_beams` must be >= 1
- when `do_sample` is not set, warn the user if any of `top_p, top_k, temperature` are not None (since they will have no effect)
- if `num_beams` is 1 (no beam search), warn user if any of `early_stopping, length_penalty` are not None (since they will have no effect)
- if an invalid set of params `num_beams` and `do_sample` are passed, raise ValueError. Note that since we also add a check for `num_beams < 1`, this final value error will never be raised, but this prevents the `generate` function from falling off the end of the ifelse chain if something is altered in the future.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
No doc changes necessary
- [ ] Did you write any new necessary tests?
ran tests, but no new functionality created, just warning messages

## Who can review?
 Please tag fewer than 3 people.
 Text Generation: @patrickvonplaten


Prevent BatchEncoding from blindly passing casts down to the tensors it contains

# What does this PR do?

This PR prevents `BatchEncoding.to` from passing down things which aren't devices to the tensors it contains. Previously it would pass down all the arguments, and as the `to` method in pytorch can also cast the arguments to different types it's used blindly by other packages (e.g. Nvidia's Apex). This caused an issue where when using Apex's AMP support with `O2` or greater it would cast the token indexes from a `LongTensor` to a `HalfTensor` truncating our vocab at 65k and rounding most of the words to the nearest 8th word (if you blindly insert the cast back in in the embedding layer, which the warning says to do).

The doc for `BatchEncoding.to` says it is only for moving the encoding and the tensors it contains between devices, but as the type checking isn't on by default it can behave like a regular pytorch `to` method and accept cast arguments that it passes down to the tensors it contains.

Fixes #6582

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case. #6582
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

There are no docs or tests changes as the change makes the method conform with its currently documented behaviour.

@LysandreJik


transformers/trainer.py stops after some iterations for iterative dataloaders.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.5.1
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): yes
- Tensorflow version (GPU?): -
- Using GPU in script?: -
- Using distributed or parallel set-up in script?: -

### Who can help
 Trainer: @sgugger
 Text Generation: @patrickvonplaten @TevenLeScao
 T5: @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner

## Information
Hi
I am using dataloader of below, after first epoch it finishes and trainer does not continue with max_steps, could you point me to the issue? I set is_sized_dataset to False.  Thank you.

```
class TaskDataLoader:
    """Wrapper around dataloader to keep the task names."""
    def __init__(self, task_name, dataset, batch_size=8,
                 collate_fn=None, drop_last=False, num_workers=0, sampler=None):
        self.dataset = dataset
        self.task_name = task_name
        self.data_loader = DataLoader(self.dataset,
                                      batch_size=batch_size,
                                      sampler=sampler,
                                      collate_fn=collate_fn,
                                      drop_last=drop_last,
                                      num_workers=num_workers)
    def __len__(self):
        return len(self.data_loader) #self.dataset.num_rows

    def __iter__(self):
        for batch in self.data_loader:
            yield batch

class MultiTaskDataLoader:
    """Given a dictionary of task: dataset, returns a multi-task dataloader
    which uses temperature sampling to sample different datasets."""

    def __init__(self,  tasks_to_datasets, batch_size=8, collate_fn=None,
                 drop_last=False, num_workers=0, temperature=100.0):

        # Computes a mapping from task to dataloaders.
        self.task_to_dataloaders = {}
        for task, dataset in tasks_to_datasets.items():
            dataloader = TaskDataLoader(task, dataset, batch_size,
                collate_fn=collate_fn, drop_last=drop_last, num_workers=num_workers)
            self.task_to_dataloaders.update({task: dataloader})
        self.tasknames = list(self.task_to_dataloaders.keys())

        # Computes the temperature sampling weights.
        self.sampling_weights = self.temperature_sampling(self.dataloader_sizes.values(), temperature)
        self.dataiters = {k: cycle(v) for k, v in self.task_to_dataloaders.items()}

    def temperature_sampling(self, dataset_sizes, temp):
        total_size = sum(dataset_sizes)
        weights = np.array([(size / total_size) ** (1.0 / temp) for size in dataset_sizes])
        return weights/np.sum(weights)

    @property
    def dataloader_sizes(self):
        if not hasattr(self, '_dataloader_sizes'):
            self._dataloader_sizes = {k: len(v) for k, v in self.task_to_dataloaders.items()}
        return self._dataloader_sizes

    def __len__(self):
        return sum(v for k, v in self.dataloader_sizes.items())

    def __iter__(self):
        outputs = {}
        for i in range(len(self)):
            taskname = np.random.choice(self.tasknames, p=self.sampling_weights)
            dataiter = self.dataiters[taskname]
            outputs["batch"] = next(dataiter)
            outputs["task"] = taskname
            yield outputs

class Trainer():
    """This is the trainer class which is responsible for distributing the data
    in case of multiple TPUs/GPUs."""
    def __init__(self, dataset_names_to_datasets):
        self.dataset_names_to_datasets = dataset_names_to_datasets
        self.batch_size = 8
        self.local_rank = -1 # this is not -1 in case of multi-gpu
        self.collate_fn = None
        self.drop_last = False
        self.num_workers = 0

    def get_sharded_data(self, num_replicas, rank):
        """Returns the sharded data belonging to the given rank."""
        sharded_dataset_names_to_datasets = {}
        for dataset_name, dataset in self.dataset_names_to_datasets:
            sharded_data = dataset.shard(num_replicas, rank)
            sharded_dataset_names_to_datasets.update({dataset_name: sharded_data})
        return sharded_dataset_names_to_datasets

    def get_train_dataset_shards(self):
        """In case of multiprocessing, returns the sharded data for the given rank."""
        if is_torch_tpu_available():
            if xm.xrt_world_size() > 1:
                return self.get_sharded_data(num_replicas=xm.xrt_world_size(), rank=xm.get_ordinal())
        elif self.local_rank != -1:
                return self.get_sharded_data(num_replicas=xm.xrt_world_size(), rank=xm.get_ordinal())
        else:
            return self.dataset_names_to_datasets

    def get_train_dataloader(self):
        """Returns the multi-task dataloader, each batch belongs
        to one task dataset."""
        dataset_names_to_datasets = self.get_train_dataset_shards()
        dataloader = MultiTaskDataLoader(dataset_names_to_datasets,
                                         batch_size=self.batch_size,
                                         collate_fn=self.collate_fn,
                                         drop_last=self.drop_last,
                                         num_workers=self.num_workers)
        return dataloader

```


[trainer] add distributed_env to TrainingArguments

As discussed in https://github.com/huggingface/transformers/pull/8823 it's not simple to check whether the downstream code is running under distributed mode or not (currently requires checking `self.args.local_rank != -1` which is far from obvious.

So we were discussing about adding a flag like `distributed_env` so that the downstream code could do a much simpler intuitive check.

I'm not sure whether we need just True/False for ddp or whether we also need to have another flag if we are under DP as well?

@sgugger


keys_to_ignore_at_inference -> output_keys_to_ignore_at_inference

# What does this PR do?

@patrickvonplaten mentioned in #8633 he was not happy with a name I picked, so I changed it.


Make the big table creation/check platform independent

# What does this PR do?

As @jplu mentioned in #8813, the check that the big table of models/tokenziers is up-to-date (done in `make quality`) requires all three backends installed (plus tokenizers and sentencepiece). This PR amends the script to use the objects in the init (that are always there, thanks to the dummies) instead of the dicts in the auto modules (which are set to None if a specific backend is not installed).

In passing, it adds aliases `MT5Tokenizer` and `MT5TokenizerFast` (to `T5Tokenizer` and `T5TokenizerFast` respectively) because otherwise the script does not detect the tokenizers associated to this model, cc @patrickvonplaten


KeyError: 'labels' in   training_step in transformers/trainer.py

## Environment info
- `transformers` version: 3.5.1
- Platform:
- Python version: 2.7
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): -
- Using GPU in script?: -
- Using distributed or parallel set-up in script?: -

### Who can help
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 T5: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information
I am using finetune_seq2seq model, the issue arises inside  training_step in trainer function, and it gives error that there is not "labels" inside the batch, below please find the stack trace, I spent the whole day I could not get why dataloader does not return back the labels, I would be grateful to point me to some possible reasons why this behaviour might happen and assisting me figuring this out. thanks

```
Traceback (most recent call last):
  File "finetune_t5_trainer.py", line 250, in <module>
    main()
  File "finetune_t5_trainer.py", line 183, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/trainers/trainer.py", line 784, in train
    tr_loss += self.training_step(model, inputs)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/trainers/trainer.py", line 1125, in training_step
    loss = self.compute_loss(model, inputs)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/trainers/t5_trainer.py", line 338, in compute_loss
    labels = inputs.pop("labels")
KeyError: 'labels'

```


Fix interaction of return_token_type_ids and add_special_tokens

Fix https://github.com/huggingface/transformers/issues/8578

It shouldn't raise a warning if `return_token_type_ids` is set to `False`. @thomwolf am I missing something here?


[CI] skip docs-only jobs take #2

So we discovered CircleCI has a problem and `pipeline.git.base_revision` is unreliable - not always set - breaking the test. https://github.com/huggingface/transformers/pull/8826#issuecomment-735972196

We had a few PRs incorrectly skipping the jobs, as in this example: https://app.circleci.com/pipelines/github/huggingface/transformers/16541/workflows/17b20230-8d7c-4b36-813c-2681f2c8a977/jobs/128232

It's missing `<< pipeline.git.base_revision >>`  in

```
if git diff --name-only << pipeline.git.base_revision >>...<< pipeline.git.revision >> | egrep -qv '\.(md|rst)$'
```
resulting in:
```
if git diff --name-only ...5170e5381b9fccdfb9405d665ecee0515efc6453 | egrep -qv '\.(md|rst)$'
```
and hence fails the test. (it's missing the first hash before `...`).

This PR checks that the external variables `pipeline.git.base_revision` and `pipeline.git.revision` are set before we do the test. Should one of them be not set, the whole test is skipped and the job continues normally, regardless of whether it's docs only or not.

Meanwhile I filed a question about why `pipeline.git.base_revision` is not always set:
https://discuss.circleci.com/t/pipeline-git-base-revision-is-often-empty-which-reliable-variable-to-use/38301

Let's merge it at a time that one of us can monitor the next few PRs in case we need to back it out again.

If you have to back it out - you only need to comment out this line: `circleci step halt` and leave the invocations in place.

@sgugger, @LysandreJik


Remove deprecated `evalutate_during_training`

# What does this PR do?

Replaces the `evaluate_during_training` in examples using the `Trainer` (as well as integrations and tf_trainer) by the new `evaluation_strategy`.

Fixes #8792


Transfoxl sequence classification

This PR implements Sequence classification for Transformer XL model
TransfoxlForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. GPT-1,GPT-2) do.

Fixes #7623 (Partially)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@LysandreJik


Add a direct link to the big table

# What does this PR do?

This PR adds an anchor to the big table of models/tokenizers to be able to generate a direct link to it, and it adds that link in the README.