Adding RAG to text-generation pipeline

# 🚀 Feature request

Thank you for the awesome work. I am working on https://github.com/deepset-ai/haystack/issues/443 and just wanted to check whether any plan to add RAG into `text-generation` pipeline.

## Motivation

`text-generation` already have other models, hence it I would be great to have it in there. And this will help keeping our code clean by not adding classes for each type of generators.

```
model = pipeline('text-generation', model="facebook/rag-token-nq", tokenizer=None, device=-1)
# ValueError: Unrecognized configuration class <class 'transformers.configuration_rag.RagConfig'> for this kind of AutoModel: AutoModelForCausalLM.
Model type should be one of CamembertConfig, XLMRobertaConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, TransfoXLConfig, XLNetConfig, XLMConfig, CTRLConfig, ReformerConfig, BertGenerationConfig.
```

## Your contribution

If you guide me I am happy to help.


Fix bert position ids in DPR convert script

https://github.com/huggingface/transformers/commit/614fef1691edb806de976756d4948ecbcd0c0ca3 introduced buffers for position ids for BERT that breaks the DPR convert script since the DPR weights don't have those.

To fix that I followed @LysandreJik 's suggestion to manually add the position ids to the state dict before loading it into the model.


Create README.md

model card

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


XLM-RoBERTa model for QA seems not properly work

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1 (and I've also tried installing Transformers from `master`, see details below)
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
albert, bert, GPT2, XLM: @LysandreJik

## Information

Model I am using (Bert, XLNet ...):
[deepset/xlm-roberta-large-squad2](https://huggingface.co/deepset/xlm-roberta-large-squad2)

The problem arises when using:
* [x] the official example scripts: [run_squad.py](https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: **SQuAD 2.0 dev set evaluation**
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
! wget https://raw.githubusercontent.com/rajpurkar/SQuAD-explorer/master/dataset/dev-v2.0.json
! python transformers/examples/question-answering/run_squad.py \
  --model_type xlm-roberta \
  --model_name_or_path 'deepset/xlm-roberta-large-squad2' \
  --do_eval \
  --do_lower_case \
  --predict_file 'dev-v2.0.json' \
  --output_dir 'output' \
  --overwrite_output_dir \
  --version_2_with_negative
```

## Expected behavior
There are some values mismatch between:
1. values reported in the model card [here](https://huggingface.co/deepset/xlm-roberta-large-squad2#performance)
2. values obtained when Transformers is installed using `pip install transformers`
3. values obtained when Transformers is installed from master

In particular:
- Reported metrics in the model card: `"exact": 79.45759285774446, "f1": 83.79259828925511`
- Transformers installed from pip: `'exact': 64.67615598416576, 'f1': 77.27580544355429`
- Transformers installed from master: `'exact': 60.11959090373114, 'f1': 76.13129575803934`


Error in run_ner.py - ModuleNotFoundError: No module named 'tasks'

## Environment info
- Google colab notebook.
- `transformers` version: 3.3.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
@stefan-it

## Information

Model I am using is NER model
Script: run_ner.py:
https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_ner.py

The problem arises when using:
* [X ] the official example scripts: (give details below)

The tasks I am working on is:
* building the NER model from model  "bert-base-multilingual-cased" using GermEval data with instructions here:
https://huggingface.co/transformers/v2.4.0/examples.html#named-entity-recognition

## To reproduce

Steps to reproduce the behavior:

```
# Eliminate --model_type; it creates an error and said it isn't used.
# sys.path.append(os.getcwd())
! python3 run_ner.py --data_dir ./ \
--labels ./labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_gpu_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_train \
--do_eval \
--do_predict

Error output:
/usr/local/lib/python3.6/dist-packages/transformers/training_args.py:332: FutureWarning: The `evaluate_during_training` argument is deprecated in favor of `evaluation_strategy` (which has more options)
  FutureWarning,
Traceback (most recent call last):
  File "run_ner.py", line 308, in <module>
    main()
  File "run_ner.py", line 118, in main
    module = import_module("tasks")
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'tasks'
```

## Expected behavior

According to the example page, the model should get fine-tuned and tested on the GermEval test dataset.


Added gpt2 model parallelism

Model Parallel

Model Parallelilsm Implementations

# Model Parallelism for GPT2LMHead
Addresses [issue 7526](https://github.com/huggingface/transformers/issues/7526)

Adds two new methods to `GPT2LMHead` and the `GPT2Model` classes to enable you to generate and fine-tune models using model parallelism. This feature is most applicable for `gpt2-large` and `gpt2-xl`. Minor modifications are made to the `TrainingArguments` and `Trainer` classes to avoid conflicting data parallelism behavior and related batch_size increases which would negate model parallelism. Note that nearly 64GB of GPU (4 Tesla v100s)  are needed to fine-tune `gpt2-xl` @ 1024 tokens.

It is critically important to provide users the ability to specify where to put the blocks of a model because the GPU sizes and numbers are likely to be very diverse. This is done with a dictionary called `device_map`. I am planning on providing some examples and guidelines for the p3, p2 and g3 AWS instances.

Model parallelism has to be baked into the model class itself. Currently working on the T5 model. From my calculations the 11B model cannot fit on the largest p3 instance that I have access to (8 Tesla v100 GPUs). The 3B model can.

The methods are:
- `parallelize`, which will distribute the attention blocks of the model across several devices according to a device map
- `deparallelize`, which will move the model back to cpu

# Example

```
model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
device_map = {0: [0, 1, 2, 3, 4, 5, 6, 7, 8],
                              1: [9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21],
                              2: [22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34],
                              3: [35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]}
model.parallelize(device_map) # Distributes the model's attention blocks across several devices
model.deparallelize() # Puts the model back on cpu and calls torch.cuda.empty_cache() to liberate GPU memory
```

## Reviewers
`TrainingArguments`: @sgugger
- Added a new parameter `model_parallel` and attribute `self.model_parallel` to control model parallelism behavior
- Slightly modified the `train_batch_size` and `eval_batch_size` calculations to avoid automatically increasing the batch size if `self.model_parallel` (automatically increasing the batch size defeats the purpose of model parallelism because you won't be able to train a larger model if the batch_size increases proportionally to the number of devices)

`Trainer`: @sgugger
- Minor changes controlled by new `args.model_parallel `attribute

`GPT2LMHead`: @patrickvonplaten
- Adds parallelize and deparallelize methods
- Adds new `self.model_parallel`  and `self.device_map` attributes
- Changes forward behavior when `self.model_parallel == True` to ensure tensors are on the right device


Cannot trace_module on models using model's generate function

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-5.8.14_1-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

@patrickvonplaten, @sshleifer
 -->

@patrickvonplaten, @sshleifer

## Information

Model I am using BART

The problem arises when using:
* [*] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [*] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. load any model that uses the generate function
2. try to trace it using trace_module

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Can be easily reproduced with the following snippet:

```
import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = 'sshleifer/bart-tiny-random'
tokenizer = AutoTokenizer.from_pretrained(model)
sqgen_model = AutoModelForSeq2SeqLM.from_pretrained(model, torchscript=True)
sqgen_model.eval()
dummy_input = ' '.join('dummy' for dummy in range(512))
batch = tokenizer(
    [dummy_input], return_tensors='pt', truncation=True, padding='longest',
)
with torch.no_grad():
    traced_model = torch.jit.trace_module(  # type: ignore
        sqgen_model,
        {
            'forward': (batch.input_ids, batch.attention_mask),
            'generate': (batch.input_ids, batch.attention_mask),
        },
    )

```

It throws an error:

```
  File "/home/void/.miniconda3/envs/lexml/src/transformers/src/transformers/generation_utils.py", line 288, in generate
    assert isinstance(max_length, int) and max_length > 0, "`max_length` should be a strictly positive integer."
AssertionError: `max_length` should be a strictly positive integer.
```

obviously because the generate function's second argument is supposed to be max_length and not attention_mask

## Expected behavior
Should be able to trace models that use the generate function.

<!-- A clear and concise description of what you would expect to happen. -->


How to create a QA model where the answer can be from the question text as well?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
SQuAD QA dataset has questions where the answer is a span in the context,
How do I create a QA system where the answer string can be from the context or question text?

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


from transformers import RagSequenceForGeneration gives ImportError

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux
- Python version:3.6.3
- PyTorch version (GPU?):
- Tensorflow version (GPU?):2.3.0
- Using GPU in script?:no
- Using distributed or parallel set-up in script?:no

### Who can help
@patrickvonplaten @sgugger

## Information

Model I am using ():

The problem arises when using:

## To reproduce
1.python
>> from transformers import RagSequenceForGeneration           <- RagTokenizer is imported without any errors

## Expected behavior
I shouldn't get
ImportError: cannot import name 'RagSequenceForGeneration'


Is there any way to control the input of a layer of `Longformer`?

Hello,

Is there any way that I can directly control the input to a layer of the `Longformer` model, similar to `GPT2.transformer.h[]`?

I tried `best_model_longformer.longformer.encoder.layer[layer_index](input_hidden_state_for_layer)` but it's giving this error:

```python
Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 426, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 388, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 205, in fill_MC_loss_accuracy_tensor
    best_model_longformer.longformer.encoder.layer[j](input_hidden_state)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 852, in forward
    output_attentions=output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 796, in forward
    output_attentions,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 241, in forward
    attention_mask = attention_mask.squeeze(dim=2).squeeze(dim=1)
AttributeError: 'NoneType' object has no attribute 'squeeze'
```
:S thank you,


Add predict step accumulation

# What does this PR do?

Currently, the Trainer accumulates all predictions on the host (GPU or TPU) before gathering across all hosts (in case of distributed training) and moving back to the CPU. This can result in OOM errors when users have a big dataset (the model is already taking up a lot of space on the host) as highlighted in #7232. However moving the predictions to the CPU at each prediction step is also inefficient (particularly on TPU).

This PR aims at fixing the OOM problem while retaining efficiency by introducing a new training argument called `eval_accumulation_step`. If left untouched, the behavior is the same as right now (all predictions accumulated on the host and moved at the end of the prediction loop). If set to an int, the predictions are gathered and moved every `eval_accumulation_step`. This required some clever reorganization of the predictions (see the docstring of `DistributedTensorGatherer` for more details).

In passing I cleaned up the code related to gathering tensors across multiple hosts and fixed the issue of the `loss.item()` (big slow down to do that at every step on TPUs) and accumulated the losses the same way predictions and labels are. This still works for any number of outputs/labels of the model.

To check those changes did not break anything, I ran `test_trainer_distributed.py` on my local setup and created an equivalent for TPUs that I also ran (they both pass).

This slightly change Seq2SeqTrainer (since we don't want the `loss.item()`) so cc @patil-suraj I don't think this should break anything in it.

<!-- Remove if not applicable -->

Fixes #7232


Use Marian-MT to evaluate translated outputs by printing out per-word log-probility

# ❓ Questions & Help

I am going to evaluate the Marian's opus-mt to evaluate the translated output y given the input x.
I want to get the model's log-probability of y given x, i.e., P(y|x).
I didn't find any usage example in the document [here](https://huggingface.co/transformers/model_doc/marian.html#multilingual-models).
@sshleifer do you know an example in this case? Thanks!

For example:
Source input (x):  I would like to run this experiment.
Translated output (y) in Chinese: 我     希望   跑        这个    实验   。
The word-level log probability:     -0.3  -0.4   -0.23  -0.43  -0.23 -0.8

BTW, fairseq supports this function in their [comment-line tools](https://fairseq.readthedocs.io/en/latest/command_line_tools.html).


Seq2seq finetune example: "Please save or load state of the optimizer"

When running the example scripts in examples/seq2seq/finetune_bart and finetune_t5, get warning messages:

## Environment info
- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-66-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Ran both with and without gpus; same result
- Using distributed or parallel set-up in script?: no

### Who can help
@sshleifer  for examples/seq2seq, Bart
@patrickvonplaten (maybe because this also happens in T5?)

## Information
Model I am using (Bert, XLNet ...):
Occurs when running bart and also when running T5 via the examples/seq2seq/finetune

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Steps to reproduce:
1) clone transformers into new directory
2) Set up environment (new): cd transformers && pip install .e; cd examples && pip install -r requirements.txt
3) cd seq2seq && ./finetune_t5_bart_tiny.sh

Observe that warnings are printed:

../python3.8/site-packages/pytorch_lightning/utilities/distributed.py:37: UserWarning: Could not log computational graph since the `model.example_input_array` attribute is not set or `input_array` was not given
warnings.warn(*args, **kwargs)
.../python3.8/site-packages/torch/optim/lr_scheduler.py:200: UserWarning: Please also save or load the state of the optimzer when saving or loading the scheduler.
  warnings.warn(SAVE_STATE_WARNING, UserWarning)

(There is both the optimizer warning and the computational graph logging warning)

## Expected behavior

Should not see warnings for the given example.

## Other notes:
There was a related issue where supplementary files / checkpoints were not being saved, but that seems to be fixed now.


Update of DialoGPT `max_length`

### Who can help
@patrickvonplaten

## Information

Following https://github.com/huggingface/transformers/pull/5516, the DialoGPT models `max_length` has not been updated, and defaults to the `generate` value of 20. This value is very low for a conversational pipeline and would lead to answers that completely ignore the history (truncation happens to ensure enough space is available for the response).

## Expected behavior

- The configuration files (e.g. https://s3.amazonaws.com/models.huggingface.co/bert/microsoft/DialoGPT-medium/config.json) need to store a `max_length` equal to 1000.
- If the suggested config file structure from the PR is adopted, the code should be updated to read this value instead
```json
  "task_specific_params" : {
      "dialogue": {
          "max_length": 1000
      }
  }
```


Allow Custom Dataset in RAG Retriever

As asked in #7725 , #7462 and #7631 , I added a way to let users build and load their own knowledge dataset for RAG.
I also added an example script that shows how to do that from csv files.

Before merging I'd like to make sure it creates the exact same embeddings that the one that were computed by the RAG team. I might need to do adjustments to the tokenization and maybe change the DPR encoder from the one trained on Natural Questions to the one trained ont the multiset/hybrid dataset.

Any feedbacks on the example and the HFIndex changes are welcome !

More details about the changes:
Previously the HFIndex only allowed to load existing datasets ("canonical" datasets) from the datasets library. So I split it into two classes `CanonicalHFIndex` to load canonical datasets and `CustomHFIndex` for custom user-defined ones.
Moreover the `config.index_name` used to accept any canonical dataset name, or "legacy" for the index the RAG team first provided. Now `config.index_name` can also accept "custom" for custom user-defined indexed datasets.


Faster pegasus tokenization test with reduced data size

This used to take 10s (and tokenize 100K words) now tokenizes 1k words and takes 1s.


Deutsch to English Translation Model by Google doesn't work anymore...

Hi, the model in: https://huggingface.co/google/bert2bert_L-24_wmt_de_en doesn't work anymore. It seems that the library has changed a lot since the model was added, therefore the classes themselves seem to have changed in names etc.
Can anyone tell me how could I apply with the current library functionality?
Thanks in advance! :)


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: bert-base-uncased
- Platform: pytorch
- Python version: 3.6
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 documentation: @sgugger
 -->

## Information

Model I am using (Bert,):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [.] my own modified scripts: (give details below)
    I am using bert for keyphrase extraction, based on the allenai-scibert code.
    When evaluating the model,

       `
        for _ in range(params.eval_steps):
        # fetch the next evaluation batch
        batch_data, batch_tags = next(data_iterator)
        batch_masks = batch_data.gt(0)

        loss, _ = model(batch_data, token_type_ids=None, attention_mask=batch_masks, labels=batch_tags)
        if params.n_gpu > 1 and params.multi_gpu:
            loss = loss.mean()
        loss_avg.update(loss.item())

        batch_output = model(batch_data, token_type_ids=None, attention_mask=batch_masks)  # shape: (batch_size, max_len, num_labels)

        batch_output = batch_output.detach().cpu().numpy()
        batch_tags = batch_tags.to('cpu').numpy()

        pred_tags.extend([idx2tag.get(idx) for indices in np.argmax(batch_output, axis=2) for idx in indices])
        true_tags.extend([idx2tag.get(idx) for indices in batch_tags for idx in indices])
    assert len(pred_tags) == len(true_tags)`

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [.] my own task or dataset: (give details below)
    SemEval 2017, task 1

## To reproduce

Steps to reproduce the behavior:

1. run the train.py script from [this repo](https://github.com/pranav-ust/BERT-keyphrase-extraction), but with `transformers` library instead of `pytorch-pretrained-bert`
2. The script gives the error:
    `Traceback (most recent call last):
  File "train.py", line 219, in <module>
    train_and_evaluate(model, train_data, val_data, optimizer, scheduler, params, args.model_dir, args.restore_file)
  File "train.py", line 106, in train_and_evaluate
    train_metrics = evaluate(model, train_data_iterator, params, mark='Train')
  File "/content/BERT-keyphrase-extraction/evaluate.py", line 54, in evaluate
    batch_output = batch_output.detach().cpu().numpy()
AttributeError: 'tuple' object has no attribute 'detach'`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The model should continue training after the first epoch


Adding optional trial argument to model_init

# Model structure optimization

This PR proposes to add the "trial" argument to the model_init function when using `trainer.hyperparameter_search`.
It's backward compatible using six to check the number of arguments of model_init through Python reflection.

```
def model_init(trial):
    if trial != None:
        layer_count = trial.suggest_int("layer_count", 2, 4)
    else:
        layer_count = 2
    return MyModel(layer_count)

trainer = Trainer(
   ...
    model_init=model_init,
   ...
)

trainer.hyperparameter_search(direction="maximize")
```


fixed lots of typos.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixed lots of typos in the documentation.

For the record, I used a chrome spell check extension to find common typos and used vim + ripgrep + fzf to do bulk corrections.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Help with finetuning mBART on an unseen language

 I wanted to know how could we finetune mBART on summarization task on a different language than that of English. Also, how can we finetune mBART on translation task where one of the languages is not present in the language code list that mBART has been trained on.
Appreciate any help!! Thank you.


[Rag] Fix loading of pretrained Rag Tokenizer

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #7710, #7690

A bug was introduced in https://github.com/huggingface/transformers/pull/7141#discussion_r503800703 that changes the loading of pre-trained special tokens files. This PR more or less reverts the critical changes so that RAG works again. This can be verified by running this test:

```
RUN_SLOW=1 pytest tests/test_modeling_rag.py::RagModelIntegrationTests::test_rag_sequence_generate_batch
```

A new RAG tokenizer test was added.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Running all slow tests to check that nothing breaks.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 --> @thomwolf @LysandreJik @sgugger


HfArgumentParser not support optional bools

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.3.1
- Platform: Linux-3.10.0
- Python version: 3.7.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

@sgugger

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Part of the code are copied from [examples/text-classification/run_glue.py](https://github.com/huggingface/transformers/blob/v3.3.1/examples/text-classification/run_glue.py)
## To reproduce

Steps to reproduce the behavior:

1.
Save the following code as 'test.py' then `python test.py --output_dir 123`
```python
from transformers import TrainingArguments, HfArgumentParser

parser = HfArgumentParser(TrainingArguments)
training_args, = parser.parse_args_into_dataclasses()
# All of the following fields has a default value of `None`
print(training_args.greater_is_better) # type=Optional[bool]
print(training_args.disable_tqdm) # type=Optional[bool]
print(training_args.metric_for_best_model) # type=Optional[str]
print(training_args.save_total_limit) # type=Optional[int]
```
2.
Got following output:

False
False
None
None

Notice the first two fields' default values are changed. They should be remain `None` as long as I don't pass the args from the cli.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
It should be the following output:

None
None
None
None

Because all of these four fields are **Optional** and their default values are `None` (See [training_args.py](https://github.com/huggingface/transformers/blob/v3.3.1/src/transformers/training_args.py)).

Also, in
https://github.com/huggingface/transformers/blob/1ba08dc221ff101a751c16462c3a256d726e7c85/src/transformers/training_args.py#L324
https://github.com/huggingface/transformers/blob/1ba08dc221ff101a751c16462c3a256d726e7c85/src/transformers/training_args.py#L342

both `greater_is_better` and `disable_tqdm` are detected whether they are None, which means the author intends them to be None when they are unspecific in passing args.

This may caused by
https://github.com/huggingface/transformers/blob/1ba08dc221ff101a751c16462c3a256d726e7c85/src/transformers/hf_argparser.py#L67-L68
Since they have a type of `Optional[bool]`，their `kwargs["action"]` is set to `store_true`. So when I don't pass the args from the cli, they have a default value of `False` instead of `None`.

I'm not sure if this is a intended design or a mistake, sorry for disturbing.

<!-- A clear and concise description of what you would expect to happen. -->


ElectraTokenizerFast

Before the fix, when loading an `ElectraTokenizerFast`:

```py
from transformers import ElectraTokenizerFast

tokenizer = ElectraTokenizerFast.from_pretrained("ahotrod/electra_large_discriminator_squad2_512")
```
```
Traceback (most recent call last):
  File "/Users/jik/Library/Application Support/JetBrains/PyCharm2020.2/scratches/7735.py", line 3, in <module>
    tokenizer = ElectraTokenizerFast.from_pretrained("ahotrod/electra_large_discriminator_squad2_512")
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_utils_base.py", line 1555, in from_pretrained
    resolved_vocab_files, pretrained_model_name_or_path, init_configuration, *init_inputs, **kwargs
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_utils_base.py", line 1623, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_bert.py", line 641, in __init__
    **kwargs,
  File "/Users/jik/Workspaces/python/transformers/src/transformers/tokenization_utils_fast.py", line 89, in __init__
    self._tokenizer = convert_slow_tokenizer(slow_tokenizer)
  File "/Users/jik/Workspaces/python/transformers/src/transformers/convert_slow_tokenizer.py", line 565, in convert_slow_tokenizer
    converter_class = CONVERTERS[transformer_tokenizer.__class__.__name__]
KeyError: 'ElectraTokenizer'
```


New TF model design

# What does this PR do?

This PR aims to improve the TensorFlow code base of transformers. For sake of simplicity the changes are made only on the BERT model, but the new features can be applied to all the others. The PR brings among some bug fix the following main new features;

1. It is now possible to train an LM model from scratch (see this [notebook ](https://colab.research.google.com/drive/1As9iz2_2eQp1Ct8trxRG3ScNfxeaVAH2?usp=sharing)as example).
2. The default outputs of the models are now dictionaries. One can only return tuples in eager execution otherwise a warning message will be displayed saying that only dictionaries are allowed in graph mode. This update fix two issues, the first one in graph mode where a layer cannot return different size of outputs, and a second one where an output cannot have a `None` value.
3. Better inputs handle. Now the inputs of each model and the main layer are parsed with a single generic function bringing a more robust parsing and a better error handling in case of wrong input. This feature fix an issue when the input was a list of symbolic inputs (i.e. `tf.keras.layers.Input`).
4. TensorFlow models looks now much more similar to what looks PyTorch models making easier for users to switch from a PyTorch model to its TensorFlow implementation and vice versa.
5. Old and new model implementations can coexist in the library making this new implementation 100% backward compatible. Including the tests.


Model Card

model card

Related to pretrained model cards

# What does this PR do?
New model card for model uploaded to https://huggingface.co/sentence-transformers

## Who can review?
 Model Cards: @julien-c


Unicode issue with tokenizer.decode()

## Environment info

- `transformers` version: 3.3.1
- Platform: Ubuntu 18.04.1 LTS
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 (No)
- Tensorflow version (GPU?): 1.14.0 (No)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

Since the issue regards tokenizers, tagging @mfuntowicz.

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run this (I'm running in a Python shell, but it's reproducible in a script)

```python
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained('t5-3b')
>>> sent = 'Luis Fonsi. Luis Alfonso Rodríguez López-Cepero, more commonly known by his stage name Luis Fonsi, (born April 15, 1978) is a Puerto Rican singer, songwriter and actor.'
>>> tokenizer.decode(tokenizer.encode(sent))
'Luis Fonsi. Luis Alfonso Rodr ⁇ guez López-Cepero, more commonly known by his stage name Luis Fonsi, (born April 15, 1978) is a Puerto Rican singer, songwriter and actor.'
```

The "í" character turns into "⁇", while other unicode characters like "ó" come out fine.

## Expected behavior

If I use a different set of functions that should end up with the same result, I get the expected:

```python
>>> tokenizer.convert_tokens_to_string(tokenizer.tokenize(sent))
'Luis Fonsi. Luis Alfonso Rodríguez López-Cepero, more commonly known by his stage name Luis Fonsi, (born April 15, 1978) is a Puerto Rican singer, songwriter and actor.'
```


Update pyarrow to meet datasets 1.1.2

# What does this PR do?

Update pyarrow to meet the requirement of y'alls awesome datasets 1.1.2 library for running the examples (tested on the finetune_tiny_bart.sh seq2seq example). You can see the colab I tested on here: https://colab.research.google.com/drive/12XSQoFRlpXLEd_tfvwC5wUVYJZUPKFwE?usp=sharing

Fixes #7691

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Issue #7691
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.
@sshleifer


Does bart need to cache prev_key_padding_mask?

This is only be used for self attention, presumably to remember to ignore if the last token generated was a pad. But if the last token generated was a pad, we are done with the hypothesis anyways so may not need this.
If this is unnec it will save 5-10 lines of annoying stuff.


Create README.md

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


BertTokenizer meet multilingual corpus, it fails to work.@mfuntowicz


Keep getting the same `Target 1 is out of bounds` error with `LongformerForMultipleChoice`

Hello,
I am a Transformer user who posted the similar question yesterday.
I am trying to use `LongformerForMultipleChoice` model, I've updated my code according to the answer that was provided. However, I am still getting the same error `Target out of bounds`. The correct answers are coded correctly, like my multiple-choice answers works very well with the `GPT2DoubleHeadsModel`. I am not sure why I am keep getting this error:
```python
# import the pre-trained HuggingFace Longformer tokenizer.
longformer_tokenizer = LongformerTokenizer.from_pretrained('allenai/longformer-base-4096')

# get the pre-trained HuggingFace Longformer
best_model_longformer = LongformerForMultipleChoice.from_pretrained('allenai/longformer-base-4096',
                                                         output_hidden_states = True)

# my multiple choice question has 4 options.
question_list = [main_question, main_question, main_question, main_question]
options_list = [option1, option2, option3, option4]
# unsqueeze the answer
mc_labels = torch.tensor(my_answer).unsqueeze(0)

encoded_dict = longformer_tokenizer(question_list, options_list,
                             return_tensors = 'pt',
                             add_prefix_space = True,
                             padding = True)
input_hidden_state = best_model_longformer(
                       **{k: v.unsqueeze(0) for k,v in encoded_dict.items()},
                        labels = mc_labels, return_dict=True)[2][0][:,:,:].detach()
```

and I am getting the error below:
```
Traceback (most recent call last):
  File "SEED_125_V20_15_LONGFORMER.py", line 427, in <module>
    main_function('/home/ec2-user/G1G2.txt','/home/ec2-user/G1G2_answer_num.txt', num_iter)
  File "SEED_125_V20_15_LONGFORMER.py", line 389, in main_function
    best_model_longformer)
  File "SEED_125_V20_15_LONGFORMER.py", line 198, in fill_MC_loss_accuracy_tensor
    input_hidden_state = best_model_longformer(**{k: v.unsqueeze(0) for k,v in encoded_dict.items()}, labels = mc_labels, return_dict = True)[2][0][:,:,:].detach()
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/transformers/modeling_longformer.py", line 1808, in forward
    loss = loss_fct(reshaped_logits, labels)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 948, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2422, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2218, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 1 is out of bounds.
```
When I do instead `mc_labels = torch.tensor([my_answer]).unsqueeze(0)` (note the square brackets around `my_answer`), another error occurs, the error is something like `cannot process multiple answers`.

How can I solve this issue?

Thank you,


Attention masks are ignored when using model.generate() in batch setting for GPT-2

## Environment info

- `transformers` version: '3.3.1' and '2.1.0' (Tested on both)
- Platform: Linux Azure VM
- Python version: 3.6.8
- PyTorch version (GPU?): 1.3.0 (Yes)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik  @TevenLeScao

## Information

Model I am using (Bert, XLNet ...): GPT-2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
import argparse
import logging
import os
import sys
import time
sys.path.append('transformers/src')
import numpy as np
import torch
import csv
import copy

from transformers import (
	GPT2LMHeadModel,
	GPT2Tokenizer
)

from multiprocessing import Pool, cpu_count
from tqdm import tqdm

MODEL_CLASSES = {
	"gpt2": (GPT2LMHeadModel, GPT2Tokenizer),
}

def set_seed():
	np.random.seed(42)
	torch.manual_seed(42)
	torch.cuda.manual_seed_all(42)

def generate_sequences_parallel(model, tokenizer, orig_prompt_list):
	set_seed()
	proc_cnt = cpu_count() - 2
	prompt_list = copy.deepcopy(orig_prompt_list)

	max_seq_len = 128

	requires_preprocessing = False
	if not requires_preprocessing:
		# GPT-2 doesn't require prepocess so we don't need to parallelize that

		inputs = tokenizer(orig_prompt_list, add_special_tokens=False, return_tensors="pt", padding=True)

		input_ids = inputs["input_ids"]
		attn_masks = inputs["attention_mask"]

		max_len_input_ids = max([len(input_id) for input_id in input_ids])

	input_ids = input_ids.to('cuda')
	attn_masks = attn_masks.to('cuda')

	output_sequences = model.generate(
		input_ids=input_ids,
		max_length=10 + max_len_input_ids,
		temperature=1.0,
		top_k=0,
		top_p=0.9,
		repetition_penalty=1.0,
		do_sample=True,
		num_return_sequences=1,
		attention_mask=attn_masks
	)

	return output_sequences

prompt_list_single = [['Good Morning Who is up with the sun Starting my morning routine with some Yoga and my mood was'], ['What do you all do to make it a great day and my mood was']]
prompt_list_batch = ['Good Morning Who is up with the sun Starting my morning routine with some Yoga and my mood was', 'What do you all do to make it a great day and my mood was']

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.to('cuda')
tokenizer.padding_side = "left"

# Define PAD Token = EOS Token = 50256
tokenizer.pad_token = tokenizer.eos_token
model.config.pad_token_id = model.config.eos_token_id

single = []
for elem in prompt_list_single:
	single.append(generate_sequences_parallel(model, tokenizer, elem))

print('BATCH')
print()

batch = generate_sequences_parallel(model, tokenizer, prompt_list_batch)

assert(torch.eq(single[0],batch[0]))
assert(torch.eq(single[1],batch[1]))
```

## Expected behavior

I expect the results of this script with batch size 1 to be the size as batch size 2 but it just ignores all the generated attention_ masks and position_ids. I've looked at #3021 and #3167 but those don't seem to offer a concrete solution. Is there some way to use GPT-2's batch generation?

Thanks!


cannot load "layoutlm-base-uncased"

hi,
im trying to do "LayoutLMTokenizer.from_pretrained('layoutlm-base-uncased')", and got an error saying "OSError: Model name 'layoutlm-base-uncased' was not found in tokenizers model name list (microsoft/layoutlm-base-uncased, microsoft/layoutlm-large-uncased). We assumed 'layoutlm-base-uncased' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url."
thank you!

- `transformers` version: 3.3.1
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0, GPU

@mfuntowicz


should PegasusTokenizer replace `/n` with `<n>`?

On the `encode`/text -> ids side I'm certain.
On the `decode`/ids -> text side, I'm worried about breaking `run_eval.py`, which reads the generations from disk before calculating rouge [here](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_eval.py#L141)

cc @stas00


Avoid unnecessary DDP synchronization when gradient_accumulation_steps > 1

# What does this PR do?

This PR avoid unnecessary ```DistributedDataParallel``` synchronization when gradient_accumulation_steps > 1 by using  ```DistributedDataParallel.no_sync```.

This lead to speedup when training with multiple gpu's for example the ```run_language_modeling.py``` complete wiki-2 epoch in 85 seconds instead of 111

```bash
python run_language_modeling.py --output_dir=runs --model_type=gpt2 --model_name_or_path=gpt2 --per_device_train_batch_size 6 --do_train --train_data_file=$TRAIN_FILE  --gradient_accumulation_steps=32 --fp16 --block_size 513 --overwrite_output_dir
```

## Before submitting
- [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@patrickvonplaten

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


examples/seq2seq/finetune_trainer.py: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`.

## Environment info

- `transformers` version: 3.3.1
   - I did `pip install -e .` in the repository cloned from
https://github.com/huggingface/transformers/tree/ba4bbd92bcb55febbfa06aaa1551738388ec7eb0
- Platform: Linux
- Python version: 3.8.3 (anaconda3-2020.07)
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 examples/seq2seq: @sshleifer

## Information

Model I am using (Bert, XLNet ...): Bart (facebook/bart-base)

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [x] an official XSUM summarization task

## To reproduce

Running examples/seq2seq/finetune_trainer.py as below.

```sh
$ CUDA_VISIBLE_DEVICES=3 python finetune_trainer.py \
    --learning_rate=3e-5 \
    --fp16 \
    --do_train --do_eval --do_predict --evaluate_during_training \
    --predict_with_generate \
    --n_val 1000 \
    --model_name_or_path facebook/bart-base \
    --data_dir ********/xsum/ \
    --output_dir ******** \
    2>&1 | tee test.log
```

Then, I get an UserWarning  message:

```sh
0%|          | 0/76506 [00:00<?, ?it/s]/********/.pyenv/versions/anaconda3-2020.07/lib/python3.8/site-packages/torch/optim/lr_scheduler.py:118: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`.  Failure to do this will result in PyTorch skipping the first value of the learning rate schedule. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
  warnings.warn("Detected call of `lr_scheduler.step()` before `optimizer.step()`. "
```

## Expected behavior

It's an UserWarning, not an Error. It may be not critical, but I want to know why this is caused.
I'm sorry if the "Bug Report" isn't the right choice for this issue.

I'm new to Transformers Trainer. I apologize that I can't distinguish whether this UserWarning belongs to examples/seq2seq/ or Trainer itself.

Thank you in advance.


Cannot load pretrained microsoft's layoutlm

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-4.15.0-112-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

@julien-c @sshleifer

## Information

Model I am using: LayoutLM

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import LayoutLMTokenizer, LayoutLMForTokenClassification
import torch

tokenizer = LayoutLMTokenizer.from_pretrained('microsoft/layoutlm-base-uncased')
model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased')
```

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/anaconda3/envs/nlp/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    926             try:
--> 927                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    928             except Exception:

~/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    526         if _is_zipfile(opened_file):
--> 527             with _open_zipfile_reader(f) as opened_zipfile:
    528                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)

~/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/serialization.py in __init__(self, name_or_buffer)
    223     def __init__(self, name_or_buffer):
--> 224         super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
    225

RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /pytorch/caffe2/serialize/inline_container.cc:132, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2. Your PyTorch installation may be too old. (init at /pytorch/caffe2/serialize/inline_container.cc:132)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7faec0a9f193 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1f5b (0x7faec3c279eb in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::string const&) + 0x64 (0x7faec3c28c04 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch.so)
frame #3: <unknown function> + 0x6c1ef6 (0x7faf0bb54ef6 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x295928 (0x7faf0b728928 in /home/user/anaconda3/envs/nlp/lib/python3.8/site-packages/torch/lib/libtorch_python.so)
frame #5: PyCFunction_Call + 0x56 (0x5640b0549f76 in /home/user/anaconda3/envs/nlp/bin/python)
frame #6: _PyObject_MakeTpCall + 0x22f (0x5640b050785f in /home/user/anaconda3/envs/nlp/bin/python)
frame #7: <unknown function> + 0x18bfdc (0x5640b0555fdc in /home/user/anaconda3/envs/nlp/bin/python)
frame #8: PyVectorcall_Call + 0x71 (0x5640b0507041 in /home/user/anaconda3/envs/nlp/bin/python)
frame #9: <unknown function> + 0x18c92a (0x5640b055692a in /home/user/anaconda3/envs/nlp/bin/python)
frame #10: _PyObject_MakeTpCall + 0x1a4 (0x5640b05077d4 in /home/user/anaconda3/envs/nlp/bin/python)
frame #11: _PyEval_EvalFrameDefault + 0x4596 (0x5640b058ef56 in /home/user/anaconda3/envs/nlp/bin/python)
frame #12: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #13: _PyObject_FastCallDict + 0x20c (0x5640b055648c in /home/user/anaconda3/envs/nlp/bin/python)
frame #14: _PyObject_Call_Prepend + 0x63 (0x5640b0556733 in /home/user/anaconda3/envs/nlp/bin/python)
frame #15: <unknown function> + 0x18c8ca (0x5640b05568ca in /home/user/anaconda3/envs/nlp/bin/python)
frame #16: _PyObject_MakeTpCall + 0x1a4 (0x5640b05077d4 in /home/user/anaconda3/envs/nlp/bin/python)
frame #17: _PyEval_EvalFrameDefault + 0x475 (0x5640b058ae35 in /home/user/anaconda3/envs/nlp/bin/python)
frame #18: _PyEval_EvalCodeWithName + 0x2d2 (0x5640b0554a92 in /home/user/anaconda3/envs/nlp/bin/python)
frame #19: _PyFunction_Vectorcall + 0x1e3 (0x5640b0555943 in /home/user/anaconda3/envs/nlp/bin/python)
frame #20: <unknown function> + 0x10011a (0x5640b04ca11a in /home/user/anaconda3/envs/nlp/bin/python)
frame #21: _PyEval_EvalCodeWithName + 0x7df (0x5640b0554f9f in /home/user/anaconda3/envs/nlp/bin/python)
frame #22: <unknown function> + 0x18bd20 (0x5640b0555d20 in /home/user/anaconda3/envs/nlp/bin/python)
frame #23: <unknown function> + 0x10077f (0x5640b04ca77f in /home/user/anaconda3/envs/nlp/bin/python)
frame #24: _PyEval_EvalCodeWithName + 0x2d2 (0x5640b0554a92 in /home/user/anaconda3/envs/nlp/bin/python)
frame #25: PyEval_EvalCodeEx + 0x44 (0x5640b0555754 in /home/user/anaconda3/envs/nlp/bin/python)
frame #26: PyEval_EvalCode + 0x1c (0x5640b05e3edc in /home/user/anaconda3/envs/nlp/bin/python)
frame #27: <unknown function> + 0x24f083 (0x5640b0619083 in /home/user/anaconda3/envs/nlp/bin/python)
frame #28: <unknown function> + 0x140699 (0x5640b050a699 in /home/user/anaconda3/envs/nlp/bin/python)
frame #29: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #30: _PyGen_Send + 0x149 (0x5640b054edc9 in /home/user/anaconda3/envs/nlp/bin/python)
frame #31: _PyEval_EvalFrameDefault + 0x49a3 (0x5640b058f363 in /home/user/anaconda3/envs/nlp/bin/python)
frame #32: _PyGen_Send + 0x149 (0x5640b054edc9 in /home/user/anaconda3/envs/nlp/bin/python)
frame #33: _PyEval_EvalFrameDefault + 0x49a3 (0x5640b058f363 in /home/user/anaconda3/envs/nlp/bin/python)
frame #34: _PyGen_Send + 0x149 (0x5640b054edc9 in /home/user/anaconda3/envs/nlp/bin/python)
frame #35: <unknown function> + 0x1701cd (0x5640b053a1cd in /home/user/anaconda3/envs/nlp/bin/python)
frame #36: <unknown function> + 0x10075e (0x5640b04ca75e in /home/user/anaconda3/envs/nlp/bin/python)
frame #37: _PyFunction_Vectorcall + 0x10b (0x5640b055586b in /home/user/anaconda3/envs/nlp/bin/python)
frame #38: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #39: _PyFunction_Vectorcall + 0x10b (0x5640b055586b in /home/user/anaconda3/envs/nlp/bin/python)
frame #40: <unknown function> + 0x10075e (0x5640b04ca75e in /home/user/anaconda3/envs/nlp/bin/python)
frame #41: _PyEval_EvalCodeWithName + 0x2d2 (0x5640b0554a92 in /home/user/anaconda3/envs/nlp/bin/python)
frame #42: _PyFunction_Vectorcall + 0x1e3 (0x5640b0555943 in /home/user/anaconda3/envs/nlp/bin/python)
frame #43: <unknown function> + 0x18be79 (0x5640b0555e79 in /home/user/anaconda3/envs/nlp/bin/python)
frame #44: PyVectorcall_Call + 0x71 (0x5640b0507041 in /home/user/anaconda3/envs/nlp/bin/python)
frame #45: _PyEval_EvalFrameDefault + 0x1fdb (0x5640b058c99b in /home/user/anaconda3/envs/nlp/bin/python)
frame #46: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #47: <unknown function> + 0x18bd20 (0x5640b0555d20 in /home/user/anaconda3/envs/nlp/bin/python)
frame #48: <unknown function> + 0x10011a (0x5640b04ca11a in /home/user/anaconda3/envs/nlp/bin/python)
frame #49: <unknown function> + 0x215056 (0x5640b05df056 in /home/user/anaconda3/envs/nlp/bin/python)
frame #50: <unknown function> + 0x1847f3 (0x5640b054e7f3 in /home/user/anaconda3/envs/nlp/bin/python)
frame #51: <unknown function> + 0x140699 (0x5640b050a699 in /home/user/anaconda3/envs/nlp/bin/python)
frame #52: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #53: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #54: _PyFunction_Vectorcall + 0x1e3 (0x5640b0555943 in /home/user/anaconda3/envs/nlp/bin/python)
frame #55: <unknown function> + 0x10075e (0x5640b04ca75e in /home/user/anaconda3/envs/nlp/bin/python)
frame #56: <unknown function> + 0x215056 (0x5640b05df056 in /home/user/anaconda3/envs/nlp/bin/python)
frame #57: <unknown function> + 0x1847f3 (0x5640b054e7f3 in /home/user/anaconda3/envs/nlp/bin/python)
frame #58: <unknown function> + 0x140699 (0x5640b050a699 in /home/user/anaconda3/envs/nlp/bin/python)
frame #59: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #60: _PyEval_EvalCodeWithName + 0x659 (0x5640b0554e19 in /home/user/anaconda3/envs/nlp/bin/python)
frame #61: <unknown function> + 0x18bd20 (0x5640b0555d20 in /home/user/anaconda3/envs/nlp/bin/python)
frame #62: <unknown function> + 0xfeb84 (0x5640b04c8b84 in /home/user/anaconda3/envs/nlp/bin/python)
frame #63: <unknown function> + 0x215056 (0x5640b05df056 in /home/user/anaconda3/envs/nlp/bin/python)

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-6-29a9c0941587> in <module>
      3
      4 tokenizer = LayoutLMTokenizer.from_pretrained('microsoft/layoutlm-base-uncased')
----> 5 model = LayoutLMForTokenClassification.from_pretrained('microsoft/layoutlm-base-uncased')
      6

~/anaconda3/envs/nlp/lib/python3.8/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    927                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    928             except Exception:
--> 929                 raise OSError(
    930                     "Unable to load weights from pytorch checkpoint file. "
    931                     "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "

OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
```
## Expected behavior

Load pretrained layoutlm


Add license info to nlptown/bert-base-multilingual-uncased-sentiment

Related to pretrained model cards

PR to close this thread: https://discuss.huggingface.co/t/what-is-the-license-of-nlptown-bert-base-multilingual-uncased-sentiment/1445/4


blenderbot-3B has wrong model card

Related to pretrained model cards

bb90 too


Make T5 Supports Gradient Checkpointing

# What does this PR do?

Since T5 3B and 11B models are really huge models to be fine-tuned on a single GPU, Gradient Checkpointing will allow this model to be fine-tuned on a single GPU but at the cost of more training time.

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?
 T5: @patrickvonplaten


Tokenizer Fast bug: ValueError: TextInputSequence must be str

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: In a Colab enviroment aswell as on my local windows version
- Python version: 3.7.4
- PyTorch version (GPU?): Yes and No
- Tensorflow version (GPU?): I didn't try with tensorflow, but I suspect that it has nothing to do with it
- Using GPU in script?: I used the automodeling on a GPU session in Colab
- Using distributed or parallel set-up in script?: Nope

### Who can help
@mfuntowicz

## Information
Model I am using: Initially Electra but I tested it out with BERT, DistilBERT and RoBERTa

It's using your scripts, but again, it believe it wouldn't work if I did it myself. The model is trained on SQuAD.

#### Error traceback
```
"""
Traceback (most recent call last):
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/usr/local/lib/python3.6/dist-packages/transformers/data/processors/squad.py", line 165, in squad_convert_example_to_features
    return_token_type_ids=True,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 2050, in encode_plus
    **kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_fast.py", line 473, in _encode_plus
    **kwargs,
  File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_fast.py", line 376, in _batch_encode_plus
    is_pretokenized=is_split_into_words,
  File "/usr/local/lib/python3.6/dist-packages/tokenizers/implementations/base_tokenizer.py", line 212, in encode
    return self._tokenizer.encode(sequence, pair, is_pretokenized, add_special_tokens)
ValueError: TextInputSequence must be str
"""
```

## To reproduce
Steps to reproduce the behavior:

1. Download model and tokenizer (fast)
2. Test it out with the transformers pipeline for a question answering task

I've also made a small notebook to test it out for yourself.  [here](https://colab.research.google.com/drive/11_qK3w7OWBTYC_GdspAjFna2XJkBgODU?usp=sharing)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Instead of giving an error, I would expect the tokenizer to work...
<!-- A clear and concise description of what you would expect to happen. -->


GLUE STS-B on longer sequence lengths doesn't work?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I have an issue where I tried to use the [standard GLUE finetuning script](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py) for task STS-B with longer sequence lengths and the results are **bad** (see below). Correlation decreases massively when using longer sequence lengths, and when I instead use binary classification with two classes instead of regression, it is the same situation. For 128 and with some data (e. g. Yelp) 256 works well but longer sequence lengths then simply fail.
My _assumption_ was that longer sequence lengths should results in similar or sometimes better results and that for shorter input sequences padding is added but not incorporated into the embedding because of masking (where the input sequence is and where it is not)?

Initially, I was using the Yelp Business Review dataset for sentiment prediction (which worked well for sequence lengths of 128, 256 and 512) but pairing same reviews sentiments for the same business should be similar to sequence pair classification (I know the task/data works) but it only gave good results for a sequence length of 128 and 256, but 400 or 512 just predicted zeros (as far as I observed). I then tried to just use this with the GLUE STS-B data with the same issue happening.

Background:
Before that, I was using GluonNLP (MXNet) and the [BERT demo finetuning script](https://gluon-nlp.mxnet.io/examples/sentence_embedding/bert.html) (also GLUE STS-B like) with the same data and basically same framework/workflow (even hyperparameters) as here in PyTorch but there all sequence lengths worked, and longer sequence length even improved results (even with smaller batch sizes because of GPU RAM and longer training durations). As the input texts were were smaller and longer (about a third of the data, I guess) this was not that surprising. I'm currently trying to switch to `transformers` because of the larger choice and support of models...

So, what am **I** doing wrong?
I tried using a constant learning rate schedule (using the default learning rate in the code) but it gave no improvements.
I tried different datasets also with almost similar end results. (even if input texts were longer than the maximum sequence length)

Can others reproduce this? (Just switch to seqlen 512 and batchsize 8 / seqlen 256 and batchsize 16)
Do I have to choose another padding strategy?

---

Results on GeForce RTX 2080 with `transformers` version `3.3.1` and CUDA 10.2:

```bash
# my script args (basically just changing the output dir and the sequence length (batch size for GPU memory reasons))
# transformers_copy being the cloned repo root folder
export GLUE_DIR=data/glue
export TASK_NAME=STS-B
python transformers_copy/examples/text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --data_dir data/sentiment/yelp-pair-b/   --max_seq_length 128 --per_device_train_batch_size 32   --learning_rate 2e-5   --num_train_epochs 3.0   --output_dir output/glue_yelp_128_32
CUDA_VISIBLE_DEVICES=1 python transformers_copy/examples/text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --data_dir data/glue/STS-B/   --max_seq_length 256 --per_device_train_batch_size 16 --per_device_eval_batch_size 16  --learning_rate 2e-5   --num_train_epochs 3.0   --output_dir output/glue_STS-B_256_16 --save_steps 1000
CUDA_VISIBLE_DEVICES=1 python transformers_copy/examples/text-classification/run_glue.py   --model_name_or_path bert-base-cased   --task_name $TASK_NAME   --do_train   --do_eval   --data_dir data/glue/STS-B/   --max_seq_length 512 --per_device_train_batch_size 8 --per_device_eval_batch_size 8  --learning_rate 2e-5   --num_train_epochs 3.0   --output_dir output/glue_STS-B_512_8 --save_steps 2000
```

```python
# cat glue_STS-B_128_32/eval_results_sts-b.txt
# seqlen 128
eval_loss = 0.5857866220474243
eval_pearson = 0.8675888610991327
eval_spearmanr = 0.8641174656753431
eval_corr = 0.865853163387238
epoch = 3.0
total_flos = 1434655122529536
```

```python
# cat glue_STS-B_256_16/eval_results_sts-b.txt
# seqlen 256
# this result should be bad, as far as I would think
eval_loss = 2.2562920122146606
eval_pearson = 0.22274851498729242
eval_spearmanr = 0.09065396938535858
eval_corr = 0.1567012421863255
epoch = 3.0
total_flos = 2869310245059072
```

```python
# cat glue_STS-B_512_8/eval_results_sts-b.txt
# seqlen 512
eval_loss = 2.224635926246643
eval_pearson = 0.24041184048438544
eval_spearmanr = 0.08133980923357159
eval_corr = 0.1608758248589785
epoch = 3.0
total_flos = 5738620490118144
```

Yelp (sentiment, single sequence) with sequence length of 512

```python
# cat yelp-sentiment-b_512_16_1/eval_results_sent-b.txt
eval_loss = 0.2301591751359403
eval_acc = 0.92832
eval_f1 = 0.945765994794504
eval_acc_and_f1 = 0.937042997397252
eval_pearson = 0.8404006160382227
eval_spearmanr = 0.8404006160382247
eval_corr = 0.8404006160382237
eval_class_report = {'not same': {'precision': 0.9099418011639767, 'recall': 0.8792393761957215, 'f1-score': 0.8943271612218422, 'support': 17249}, 'same': {'precision': 0.937509375093751, 'recall': 0.954169338340814, 'f1-score': 0.945765994794504, 'support': 32751}, 'accuracy': 0.92832, 'macro avg': {'precision': 0.9237255881288639, 'recall': 0.9167043572682677, 'f1-score': 0.920046578008173, 'support': 50000}, 'weighted avg': {'precision': 0.9279991134394574, 'recall': 0.92832, 'f1-score': 0.928020625988607, 'support': 50000}}
epoch = 0.08
total_flos = 26906733281280000
```

Yelp (sequence pairs) with 128, 256 and 512 (were 512 fails)

```python
# cat yelp-pair-b_128_32_3/eval_results_same-b.txt
# seqlen 128
eval_loss = 0.4788903475597093
eval_acc = 0.8130612708878027
eval_f1 = 0.8137388152678672
eval_acc_and_f1 = 0.813400043077835
eval_pearson = 0.6262220422479998
eval_spearmanr = 0.6262220422479998
eval_corr = 0.6262220422479998
eval_class_report = {'not same': {'precision': 0.8189660129967221, 'recall': 0.8058966668552996, 'f1-score': 0.8123787792355962, 'support': 35342}, 'same': {'precision': 0.8072925445249733, 'recall': 0.8202888622481018, 'f1-score': 0.8137388152678672, 'support': 35034}, 'accuracy': 0.8130612708878027, 'macro avg': {'precision': 0.8131292787608477, 'recall': 0.8130927645517008, 'f1-score': 0.8130587972517317, 'support': 70376}, 'weighted avg': {'precision': 0.8131548231814548, 'recall': 0.8130612708878027, 'f1-score': 0.8130558211583339, 'support': 70376}}
epoch = 3.0
total_flos = 71009559802626048
```

```python
# cat yelp-pair-b_256_16_1/eval_results_same-b.txt
# seqlen 256
eval_loss = 0.3369856428101318
eval_acc = 0.8494088893941116
eval_f1 = 0.8505977218901545
eval_acc_and_f1 = 0.850003305642133
eval_pearson = 0.6990572001217541
eval_spearmanr = 0.6990572001217481
eval_corr = 0.6990572001217511
eval_class_report = {'not same': {'precision': 0.8588791553054476, 'recall': 0.8377850715862147, 'f1-score': 0.8482009854474619, 'support': 35342}, 'same': {'precision': 0.840315302768648, 'recall': 0.8611348975281156, 'f1-score': 0.8505977218901545, 'support': 35034}, 'accuracy': 0.8494088893941116, 'macro avg': {'precision': 0.8495972290370477, 'recall': 0.8494599845571651, 'f1-score': 0.8493993536688083, 'support': 70376}, 'weighted avg': {'precision': 0.8496378513129752, 'recall': 0.8494088893941116, 'f1-score': 0.8493941090198912, 'support': 70376}}
epoch = 1.0
total_flos = 47339706535084032
```

```python
# cat yelp-pair-b_512_8_3/eval_results_same-b.txt
# seqlen 512
# here it basically just predicts zeros all the time (as fas as I saw)
eval_loss = 0.6931421184073636
eval_acc = 0.5021882459929522
eval_f1 = 0.0
eval_acc_and_f1 = 0.2510941229964761
eval_pearson = nan
eval_spearmanr = nan
eval_corr = nan
eval_class_report = {'not same': {'precision': 0.5021882459929522, 'recall': 1.0, 'f1-score': 0.6686089407669461, 'support': 35342}, 'same': {'precision': 0.0, 'recall': 0.0, 'f1-score': 0.0, 'support': 35034}, 'accuracy': 0.5021882459929522, 'macro avg': {'precision': 0.2510941229964761, 'recall': 0.5, 'f1-score': 0.33430447038347305, 'support': 70376}, 'weighted avg': {'precision': 0.25219303441347785, 'recall': 0.5021882459929522, 'f1-score': 0.3357675512189583, 'support': 70376}}
epoch = 3.0
total_flos = 284038239210504192
```

Side note:
I also ran Yelp with regression and it worked for 128 but for 512 the correlation was below 0.3 so it also failed again.
And I worked on another (private) dataset with similar results...

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**: n/a


[Prophetnet] Develop in parallel

Related to pretrained model cards


Fix #7731

Fixes #7731


Pytorch 1.6 DataParallel

I get an error similar to that of [#4189](https://github.com/huggingface/transformers/issues/4189) and [#3936](https://github.com/huggingface/transformers/issues/3936), when using DataParallel with GPT2.
The issue should be resolved in newer versions of transformers, but I still get an error.

## Environment info

- `transformers` version: 3.3.1
- Platform: Linux-3.10.0-1062.18.1.el7.x86_64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

CUDA compilation tools, release 10.2, V10.2.89

### Who can help
albert, bert, GPT2, XLM: @LysandreJik
TextGeneration: @TevenLeScao

## Information

Model I am using: GPT2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behaviour:
1. Run forward on GPT2 with 2 GPUs

### Small example

```python
import torch
from torch.nn import DataParallel
from transformers import GPT2Tokenizer, GPT2LMHeadModel

device = "cuda:0"

# Get model
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")
model = DataParallel(model, device_ids=list(range(torch.cuda.device_count())))
model.to(device=device)

# Run forward
inputs = tokenizer(["This is an example"], return_tensors="pt")
outputs = model(
    input_ids=inputs["input_ids"].to(device),
    attention_mask=inputs["attention_mask"].to(device),
    labels=inputs["input_ids"].to(device),
)

print(f"outputs: {outputs}")
print("Success.")
```

### Output

```
Traceback (most recent call last):
  File "minimum_example.py", line 15, in <module>
    outputs = model(
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
StopIteration: Caught StopIteration in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/transformers/modeling_gpt2.py", line 752, in forward
    transformer_outputs = self.transformer(
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/user/.conda/envs/main/lib/python3.8/site-packages/transformers/modeling_gpt2.py", line 587, in forward
    attention_mask = attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
StopIteration

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I simply expect to get an output and no error :)

<!-- A clear and concise description of what you would expect to happen. -->


Upgrading in pipelines TFAutoModelWithLMHead to new Causal/Masked/Seq2Seq LM classes

# What does this PR do?

Updates code to remove deprecation warning.
No tests as it simply removes current warnings from tests.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@mfuntowicz

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Fix DeBERTa integration tests

Fix the two DeBERTa integration tests.

- The classification test doesn't actually make sense, since there is no classification head saved in `microsoft/deberta-base`
- Fix to the way the attention was initialized according to the pos type.

closes https://github.com/huggingface/transformers/issues/7565
closes https://github.com/huggingface/transformers/pull/7645


Improving Pipelines by defaulting to framework='tf' when pytorch seems unavailable.

# What does this PR do?

When loading a model that was `tf` only and by passing only model by string without framework argument, it would fail with an odd error message:

```python
>>> transformers.AutoModel.from_pretrained('Narsil/small')
OSError: Can't load weights for 'Narsil/small'. Make sure that:
- 'Narsil/small' is a correct model identifier listed on 'https://huggingface.co/models' (It exists and contains tf_model.h5)
- or 'Narsil/small' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.
```

This PR corrects the `get_framework` that happens very early in the pipeline to detect the type of model
automatically. It does trigger an early download, but that will happen anyway later.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dimiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@mfuntowicz

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


what is the perplexity of distilbert-base-uncased ?

# ❓ Questions & Help

## Details

In the [readme](https://github.com/huggingface/transformers/tree/master/examples/distillation) , it is said that distilbert-base-uncased is pretraind on the same data used to pretrain Bert, so I wonder what is the final perplexity or cross entropy of the pretrain?


Do not softmax when num_labels==1

When `config.num_labels=1`, the text classification pipeline shouldn't softmax over the results as it will always return 1. Instead, when the number of labels is 1, this will run a sigmoid over the result.

cf https://github.com/huggingface/transformers/issues/7493#issuecomment-706413792


how we can replace/swap the Wikipedia data with our custom data for knowledge retrieval in the RAG  model and  the format of the retrieval data.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Fix tf text class

# What does this PR do?

This PR fixes an issue in the `run_tf_text_classification.py` where the tokenizer was raising an error when the CSV file had 3 columns.

Fixes #7706