Longformer `token_type_ids` Vocabulary Size is 1 But Documentation States Otherwise

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2 and latest (4.0.0)
- Platform: Linux (Ubuntu)
- Python version: 3.6.9 and 3.8.6
- PyTorch version (GPU?): 3.7.0 Tesla P100-PCIE-16GB and Nvidia RTX 3090
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes and No
- Using distributed or parallel set-up in script?: No

### Who can help
Possibly @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('allenai/longformer-base-4096')
model = AutoModel.from_pretrained('allenai/longformer-base-4096')

tokenizer_bert = AutoTokenizer.from_pretrained('bert-base-uncased')

inputs = tokenizer("How old are you?", "I'm 6 years old", return_tensors="pt", return_token_type_ids=True, return_attention_mask=True)
inputs_bert = tokenizer_bert("How old are you?", "I'm 6 years old", return_tensors="pt", return_token_type_ids=True, return_attention_mask=True)

print(inputs)
print(inputs_bert)
```

```python
inputs = {'input_ids': tensor([[   0, 6179,  793,   32,   47,  116,    2,    2,  100,  437,  231,  107,
          793,    2]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
inputs_bert = {'input_ids': tensor([[ 101, 2129, 2214, 2024, 2017, 1029,  102, 1045, 1005, 1049, 1020, 2086,
         2214,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
```

```python
inputs['token_type_ids'] = torch.tensor([[0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]])
model.forward(**inputs)
```

Stack Trace:

```
---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

<ipython-input-65-905d6a7d6135> in <module>()
----> 1 model.forward(**inputs)

6 frames

/usr/local/lib/python3.6/dist-packages/transformers/modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states)
    995
    996         embedding_output = self.embeddings(
--> 997             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    998         )
    999

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     66
     67         return super().forward(
---> 68             input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
     69         )
     70

/usr/local/lib/python3.6/dist-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    178             inputs_embeds = self.word_embeddings(input_ids)
    179         position_embeddings = self.position_embeddings(position_ids)
--> 180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181
    182         embeddings = inputs_embeds + position_embeddings + token_type_embeddings

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    124         return F.embedding(
    125             input, self.weight, self.padding_idx, self.max_norm,
--> 126             self.norm_type, self.scale_grad_by_freq, self.sparse)
    127
    128     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1850         # remove once script supports set_grad_enabled
   1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1853
   1854

IndexError: index out of range in self
```

## Expected behavior

The longformer documentation should be updated and state that the longformer does not support `token_type_ids` like RoBERTa. The `token_type_ids` [vocabulary size is 1](https://huggingface.co/allenai/longformer-base-4096/raw/main/config.json) (compared to [2 for BERT](https://huggingface.co/bert-base-uncased/raw/main/config.json)) for `allenai/longformer-base-4096`, which means `0` is the only valid input for `token_type_ids`. However, [the documentation](https://huggingface.co/transformers/model_doc/longformer.html#transformers.LongformerModel.forward) says `token_type_ids` can be selected in `[0, 1]` for the longformer. The documentation should to specify that the longformer doesn't support `token_type_ids`. For instance, the [RoBERTa documentation](https://huggingface.co/transformers/model_doc/roberta.html) states "RoBERTa doesn’t have `token_type_ids`, you don’t need to indicate which token belongs to which segment. Just separate your segments with the separation token `tokenizer.sep_token` (or `</s>`)." Should a similar message be added for the longformer since it is based on RoBERTa?


Not able to train RoBERTa language model from scratch

I tried training RoBERTa language model from scratch using

<code> !python ./run_mlm.py
    --model_name_or_path roberta-base
    --train_file './data_lm.txt'
    --do_train
    --line_by_line
    --num_train_epochs 3
    --output_dir ./roberta
</code>

Due to the limits on the colab storage, I deleted all the checkpoints generated during the training process. So finally my model directory has the following files -
1. config.json
2. merges.txt
3. pytorch_model.bin
4. special_tokens_map.json
5. tokenizer.config
6. vocab.json

But after loading my trained model using RobertaTokenizer, RobertaForSequenceClassification and fine-tuning it  for my classification task, i am receiving almost same accuracy as by loading and fine-tuning the readily available 'roberta-base'.

Also, when i try loading it as
`model = RobertaModel.from_pretrained('./roberta)
`

 I get the warning-

<code>
Some weights of RobertaModel were not initialized from the model checkpoint at ./roberta/ and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']

You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
</code>

So, my question is, Is there something wrong in the training procedure of the Language Model on my dataset and the loading process? Or the checkpoints that i deleted were important?


Cannot disable logging from trainer module

@sgugger @stas00

- `transformers` version: 3.2.0
- Platform:
- Python version: 3.7.6
- PyTorch version (GPU?): 1.6.0, Tesla V100
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Parallel

I am using the Hugging Face Trainer class for NER fine-tuning. Whenever I turn evaluation on with the `--do_eval` argument, my console gets overwhelmed with a printed dictionary that appears to be coming from evaluation that is happening inside of the Trainer.

The dictionary has the keys:

` 'eval_loss', 'eval_accuracy_score', 'eval_precision', 'eval_recall', 'eval_f1', 'eval_class_report', 'eval_predictions' `

It's especially hard to read the console with NER predictions, because `eval_predictions`  is a list with each token receiving an IOB tag.

I tried suppressing logging from transformers with this solution https://github.com/huggingface/transformers/issues/3050. I also tried disabling all logging below CRITICAL level. The problem persisted, and I noticed that the console output of the evaluation dictionary appeared to be coming from a print statement.

I tried suppressing all print statements from the Trainer's `train(...)` method, using this solution https://stackoverflow.com/questions/977840/redirecting-fortran-called-via-f2py-output-in-python/978264#978264. That worked, but now I have no logging of training at all :(.


Time for second encoding is much higher than first time

Hi,

using a bert model on a single gpu  to encode multiple times after each each other like
```
bert_model = TFBertModel.from_pretrained('bert-base-cased',output_hidden_states=True)

input = tokenizer(data , max_length=MAX_SEQ_lEN,padding="max_length",truncation=True, return_tensors="tf")
outputs1 = bert_model(input)
###time1 : 0.1 seconds
outputs2 = bert_model(input)
### time2: 1.7 seconds

```
gives a unproportional high time for the second encoding. If the first encoding time is just 0.1 second I would assume to have each other encoding afterweards also about 0.1 seconds. I run this multiple times and it seems a patterns that the encoding after the first encoding is significantly larger.

Can someone explain this behaviour? I assume it is due to gpu.

```
Env: win 10, python: 3.6, tensorflow 2.3, transformers 3.3.1
GPU: Nvidia mx 150
```


Fix T5 model parallel test

The model was defined in the wrong model tester.


Cannot load community model on local machine

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.6
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
Probably @LysandreJik

## Information

Model I am using: https://huggingface.co/huggingtweets/xinqisu

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

## To reproduce

Steps to reproduce the behavior: (this is the instruction on the model page)

```
from transformers import pipeline
generator = pipeline('text-generation', model='huggingtweets/xinqisu')
generator("My dream is", num_return_sequences=5)
```

It gives me

```
OSError: Can't load config for 'huggingtweets/xinqisu'. Make sure that:

- 'huggingtweets/xinqisu' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'huggingtweets/xinqisu' is the correct path to a directory containing a config.json file
```

## Expected behavior

The generator should work with the snippet above. I have trained other `huggingtweets` models and they still work with the same code, for example, the following still works, it downloaded the model successfully.

```
from transformers import pipeline
generator = pipeline('text-generation', model='huggingtweets/billgates')
generator("My dream is", num_return_sequences=5)
```


Added TF OpenAi GPT1 Sequence Classification

This PR implements Sequence classification for TF OpenAi GPT1 model.
TFOpenAIGPTForSequenceClassification uses the last token in order to do the classification, as other causal models (e.g. Transformer XL ,GPT-2) do.

Fixes #7623

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?
@LysandreJik @jplu


Cannot load custom tokenizer for Trainer

## Environment info
- `transformers` version: 4.0.0
- Platform: Linux-5.9.13-zen1-1-zen-x86_64-with-glibc2.2.5
- Python version: 3.8.6
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@mfuntowicz

## Information

Model I am using: My own

The problem arises when using:
* [X] the official example scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

I want to finetune a model on my own dataset. For now it doesn't matter if i finetune bert, DistilBert or other, I just want good embeddings for text similarity (cosine distance)
## To reproduce

Steps to reproduce the behavior:

1. Read the [How to train tutorial](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb)
2. Train your own tokenizer (working perfectly) and got `model-vocab.json` and `model-merges.txt`
3. Load and encode with :
```
tokenizer = ByteLevelBPETokenizer(./models/custom/my_model-vocab.json","./models/custom/my_model-merges.txt")
```
This works nicelly !
4. Try to do the same with a DistilBertTokenizerFast to use with the `Trainer` class
```
tokenizer = DistilBertTokenizerFast.from_pretrained('./models/custom', max_len=512)
```

5. Get the error `check that './models/custom' is the correct path to a directory containing relevant tokenizer files`

Note : I also tried to add a `config.json` file next to the merge and vocab which seemed missing but it doesn't change anything.
I also tried a RobertaTokenizerFast (and the 'not fast' version) but same problem

## Expected behavior
Train a custom tokenizer and be able to load it with a ModelTokenizer for the Trainer.
(The BPE tokenizer which works do not have the  `mask_token` attribute to work with the dataset loader)


Seq2Seq training calculate_rouge with precision and recall

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): pytorch-lightning==1.0.4
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
 Trainer: @sgugger
 examples/seq2seq: @patil-suraj

## Information

Model I am using (Bert, XLNet ...): bart-base

The tasks I am working on is:
summarization on XSUM

## To reproduce
Change `calculate_rouge` function in `utils.py` with `return_precision_and_recall=True`.
Fine-tune any seq2seq model with the official script `finetune.py`:
```
!python3 $finetune_script \
--model_name_or_path facebook/bart-base \
--tokenizer_name facebook/bart-base \
--data_dir $data_dir \
--learning_rate 3e-5 --label_smoothing 0.1 --num_train_epochs 2 \
--sortish_sampler --freeze_embeds --adafactor \
--task summarization \
--do_train \
--max_source_length 1024 \
--max_target_length 60 \
--val_max_target_length 60 \
--test_max_target_length 100 \
--n_train 8 --n_val 2 \
--train_batch_size 2 --eval_batch_size 2 \
--eval_beams 2 \
--val_check_interval 0.5 \
--log_every_n_steps 1 \
--logger_name wandb \
--output_dir $output_dir \
--overwrite_output_dir \
--gpus 1
```

Throws the error
```
Validation sanity check: 100%|██████████| 1/1 [00:01<00:00,  1.67s/it]Traceback (most recent call last):
  File "/content/drive/My Drive/MAGMA: Summarization/seq2seq/finetune.py", line 443, in <module>
    main(args)
  File "/content/drive/My Drive/MAGMA: Summarization/seq2seq/finetune.py", line 418, in main
    logger=logger,
  File "/content/drive/My Drive/MAGMA: Summarization/seq2seq/lightning_base.py", line 389, in generic_train
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 440, in fit
    results = self.accelerator_backend.train()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/gpu_accelerator.py", line 54, in train
    results = self.train_or_test()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/accelerators/accelerator.py", line 68, in train_or_test
    results = self.trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 462, in train
    self.run_sanity_check(self.get_model())
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 650, in run_sanity_check
    _, eval_results = self.run_evaluation(test_mode=False, max_batches=self.num_sanity_val_batches)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 597, in run_evaluation
    num_dataloaders=len(dataloaders)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 196, in evaluation_epoch_end
    deprecated_results = self.__run_eval_epoch_end(num_dataloaders, using_eval_result)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/evaluation_loop.py", line 247, in __run_eval_epoch_end
    eval_results = model.validation_epoch_end(eval_results)
  File "/content/drive/My Drive/MAGMA: Summarization/seq2seq/finetune.py", line 190, in validation_epoch_end
    k: np.array([x[k] for x in outputs]).mean() for k in self.metric_names + ["gen_time", "gen_len"]
  File "/content/drive/My Drive/MAGMA: Summarization/seq2seq/finetune.py", line 190, in <dictcomp>
    k: np.array([x[k] for x in outputs]).mean() for k in self.metric_names + ["gen_time", "gen_len"]
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/_methods.py", line 163, in _mean
    ret = ret / rcount
TypeError: unsupported operand type(s) for /: 'dict' and 'int'
```

From my understanding self.metric_names should be a list.


Unexpected logits shape on prediction with TFRobertaForSequenceClassification

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.0
- Platform: Linux-4.9.0-11-amd64-x86_64-with-debian-9.11
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0a0+bf2bbd9 (False)
- Tensorflow version (GPU?): 2.3.1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Distributed

I am using the the TFRobertaForSequenceClassification to create a classifier. According to the documentation, the logits output should have a shape of (batch_size, num_labels) which makes sense. I however get (batch_size, seq_length, num_labels).

Code to reproduce:

```
from transformers import TFRobertaForSequenceClassification, RobertaConfig
import numpy as np

seq_len = 512

classifier = TFRobertaForSequenceClassification(RobertaConfig())

#create random inputs for demo
input_ids = np.random.randint(0,10000, size=(seq_len,))
attention_mask = np.random.randint(0,2, size=(seq_len,))
token_type_ids = np.random.randint(0,2, size=(seq_len,))

#make a prediction with batch_size of 1
output = classifier.predict([input_ids, attention_mask, token_type_ids])

print(output.logits.shape) -> prints out (512,2)
```

## Expected behavior

Logits in the shape of (batch_size,num_labels) or (1,2)


Fix a broken link in documentation

# What does this PR do?

Fixes a broken link to the BERTology example in documentation

Fixes #9100

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

 documentation: @sgugger


Link to BERTology example is broken

Link to BERTology example is broken in Documentation (https://huggingface.co/transformers/bertology.html)


bug with _load_optimizer_and_scheduler in trainer.py

## Environment info
- `transformers` version: 3.5.1
- Platform: GPU
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 Trainer: @sgugger
 Text Generation: @patrickvonplaten

## Information
I am using finetune_trainer.py, what I am observing is that in trainer.py, when you call _load_optimizer_and_scheduler if the model_path folder exist, then this ignores the user's set learning rate, meaning it continues with some saved learning rate than actually use the learning rate user set, could you have a look please? thanks


[RAG, Bart] Align RAG, Bart cache with T5 and other models of transformers

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

In Transformers, the cache should always have the same structure. This becomes especially important for composite models like `RAG` and `EncoderDecoder` that expect all models to have the same cache.

Bart and T5 had different caches with Bart being most different from the standard cache of the library.
This PR aligns the `past_key_values` cache of Bart/Rag with all other models in the library. In general, the philosophy should be:

the past_key_value should have exactly one level for each layer, no matter whether the model is a decoder-only a.k.a. GPT2 or BART. This was not correctly refactored in BART (it should have been implemented 1-to-1 as in T5). No breaking changes here though.

- `past_key_value` tuple for each layer should always be a tuple of tensors, **not** a tuple of a tuple
- for decodre-only models (GPT2), the tuple for each layer contains 2 tensors: key and value states
- for seq2seq (BART/T5), the tuple for each layer contains 4 tensors: key and value states of uni-directional self-attention, saved key and value states for cross-attention

This doesn't break any backward compatibility and should fix some RAG problems (@ratthachat). All RAG, Bart slow tests are passing and changes correspond just to the tuple structure.

PR is blocking me for TFBart refactor -> will merge already.

cc @LysandreJik, @sgugger, @patil-suraj for info.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Is the LayoutLM working now?

Getting endless erros when trying to use the LayoutLMForTokenClassification from transformers for NER task, is just me doing wrong or the function still on work?
Really appreciate if anyone can give some information.


Fix variable name in TrainingArguments docstring

# What does this PR do?
Corrects a var name in the docstring for `TrainingArguments` (there is no `ignore_skip_data`)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?
@sgugger


[TorchScript] Received several warning during Summarization model conversion

## Environment info
Using Transformers 4.0.1 and PyTorch 1.6.0.

```pytorch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-cnn-6-6")

model = AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distilbart-cnn-6-6")

# model = BartModel.from_pretrained("sshleifer/bart-tiny-random")
input_ids = decoder_input_ids = torch.tensor([19 * [1] + [model.config.eos_token_id]])
traced_model = torch.jit.trace(model, (input_ids, decoder_input_ids), strict=False)

traced_model.save("distilbart.pt")
```

I have to disable the strict checking in order to pass. (Error message without disable the strict flag):

```
RuntimeError: Encountering a dict at the output of the tracer might cause the trace to be incorrect, this is only valid if the container structure does not change based on the module's inputs. Consider using a constant container instead (e.g. for `list`, use a `tuple` instead. for `dict`, use a `NamedTuple` instead). If you absolutely need this and know the side effects, pass strict=False to trace() to allow this behavior.
```

Here is the warning messages:
```
/Users/qingla/PycharmProjects/pytorch/venv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py:232: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if not padding_mask.any():
/Users/qingla/PycharmProjects/pytorch/venv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py:175: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if decoder_padding_mask is not None and decoder_padding_mask.shape[1] > 1:
/Users/qingla/PycharmProjects/pytorch/venv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py:716: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert key_padding_mask is None or key_padding_mask.shape == (bsz, src_len)
/Users/qingla/PycharmProjects/pytorch/venv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py:718: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_weights.size() == (bsz * self.num_heads, tgt_len, src_len)
/Users/qingla/PycharmProjects/pytorch/venv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py:736: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  assert attn_output.size() == (bsz * self.num_heads, tgt_len, self.head_dim)
/Users/qingla/PycharmProjects/pytorch/venv/lib/python3.7/site-packages/transformers/models/bart/modeling_bart.py:287: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if torch.isinf(x).any() or torch.isnan(x).any():
```

If these warning are indicated correctly, them the model I traced is highly tied to the dummy input I provided which would bring inaccurate inference result... Any thoughts on how to improve it? @sshleifer Thanks!


head mask issue  transformers==3.5.1

## Environment info

- `transformers` version: 3.5.1
- Platform: windows & linux
- Python version: python 3.7
- PyTorch version (GPU?): 1.7.0
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: Yes  both CPU and GPU
- Using distributed or parallel set-up in script?: No

### Who can help
albert, bert, GPT2, XLM: @LysandreJik

## Information
Hi
I am using  tiny albert Chinese as an encoder. and I've also tried to use albert transformer in my code.
Thing is I have to change  a little bit source code to avoid some head mask issues.
watch transformers/modeling_albert.py
line 387:  layer_output = albert_layer(hidden_states, attention_mask, **head_mask[layer_index]**, output_attentions)
however a few lines above, the defualt head_mask is None. So **TypeError: 'NoneType' object is not subscriptable** would be raised

It's not a deep bug and could be easily avoided if making a torch.ones  head_mask. Just want to bring it up so it might probably help the others who encounter the same problem.


Not able to load T5 tokenizer

Transformers==4.0.0
torch == 1.7.0+cu101
tensorflow == 2.3.0
Platform = Colab notebook

@julien-c  @patrickvonplaten

Not able to load T5 tokenizer using
 `tokenizer = T5Tokenizer.from_pretrained('t5-base')'

Getting error -
![image](https://user-images.githubusercontent.com/54103522/102049719-6c717480-3e07-11eb-8671-9c10c3169b17.png)

I am able to download the pre-trained model though.


Patch *ForCausalLM model with TF resize_token_embeddings

cc @jplu


Chinese

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


run_clm example gives `CUDA out of memory. Tried to allocate` error

## Environment info

Google Colab with GPU runtime.

- Python version: 3.6.9

## Information

I'm trying to run the GPT2 training example from `https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py`.

The problem arises when using:

- Run CLM language modeling example.

## To reproduce

Steps to reproduce the behavior:

1. Open Google Colab with GPU on

2. Run

```
!git clone https://github.com/huggingface/transformers
%cd transformers
!pip install .
%cd examples
%cd language-modeling
!pip install -r requirements.txt
```

```
!python run_clm.py \
    --model_name_or_path gpt2 \
    --dataset_name wikitext \
    --dataset_config_name wikitext-2-raw-v1 \
    --do_train \
    --do_eval \
    --output_dir test
```

Log:

```
[INFO|trainer.py:668] 2020-12-14 02:09:02,049 >> ***** Running training *****
[INFO|trainer.py:669] 2020-12-14 02:09:02,049 >>   Num examples = 2318
[INFO|trainer.py:670] 2020-12-14 02:09:02,049 >>   Num Epochs = 3
[INFO|trainer.py:671] 2020-12-14 02:09:02,049 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:672] 2020-12-14 02:09:02,049 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:673] 2020-12-14 02:09:02,049 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:674] 2020-12-14 02:09:02,049 >>   Total optimization steps = 870
  0% 0/870 [00:00<?, ?it/s]Traceback (most recent call last):
  File "run_clm.py", line 357, in <module>
    main()
  File "run_clm.py", line 327, in main
    trainer.train(model_path=model_path)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 767, in train
    tr_loss += self.training_step(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1096, in training_step
    loss = self.compute_loss(model, inputs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 1120, in compute_loss
    outputs = model(**inputs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 895, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 740, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 295, in forward
    output_attentions=output_attentions,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 239, in forward
    attn_outputs = self._attn(query, key, value, attention_mask, head_mask, output_attentions)
  File "/usr/local/lib/python3.6/dist-packages/transformers/models/gpt2/modeling_gpt2.py", line 166, in _attn
    w = torch.matmul(q, k)
RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 15.90 GiB total capacity; 14.75 GiB already allocated; 185.88 MiB free; 14.81 GiB reserved in total by PyTorch)
  0% 0/870 [00:00<?, ?it/s]
```

## Expected behavior

Outputs the model in the output_dir with no memory error.


Fix a bug in eval_batch_retrieval of eval_rag.py

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Following the instructions in [RAG example](https://github.com/huggingface/transformers/tree/master/examples/research_projects/rag#retrieval-evaluation), I was trying to evaluate retrieval against DPR evaluation data.

`pipenv run python eval_rag.py     --model_name_or_path facebook/rag-sequence-nq     --model_type rag_sequence     --evaluation_set output/biencoder-nq-dev.questions     --gold_data_path output/biencoder-nq-dev.pages     --predictions_path output/retrieval_preds.tsv      --eval_mode retrieval     --k 1`

With the above command, I faced the following error and confirmed that `question_enc_outputs` is a tuple whose length is 1.

```
...
loading weights file https://huggingface.co/facebook/rag-sequence-nq/resolve/main/pytorch_model.bin from cache at /home/ubuntu/.cache/huggingface/transformers/9456ce4ba210322153f704e0f26c6228bd6c0caad60fe1b3bdca001558adbeca.ee816b8e716f9741a2ac602bb9c6f4d84eff545b0b00a6c5353241bea6dec221
All model checkpoint weights were used when initializing RagSequenceForGeneration.

All the weights of RagSequenceForGeneration were initialized from the model checkpoint at facebook/rag-sequence-nq.
If your task is similar to the task the model of the checkpoint was trained on, you can already use RagSequenceForGeneration for predictions without further training.
initializing retrieval
Loading index from https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/
loading file https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/hf_bert_base.hnswSQ8_correct_phi_128.c_index.index.dpr from cache at /home/ubuntu/.cache/huggingface/transformers/a481b3aaed56325cb8901610e03e76f93b47f4284a1392d85e2ba5ce5d40d174.a382b038f1ea97c4fbad3098cd4a881a7cd4c5f73902c093e0c560511655cc0b
loading file https://storage.googleapis.com/huggingface-nlp/datasets/wiki_dpr/hf_bert_base.hnswSQ8_correct_phi_128.c_index.index_meta.dpr from cache at /home/ubuntu/.cache/huggingface/transformers/bb9560964463bc761c682818cbdb4e1662e91d25a9407afb102970f00445678c.f8cbe3240b82ffaad54506b5c13c63d26ff873d5cfabbc30eef9ad668264bab4
7it [00:00, 212.77it/s]
Traceback (most recent call last):
  File "eval_rag.py", line 315, in <module>
    main(args)
  File "eval_rag.py", line 301, in main
    answers = evaluate_batch_fn(args, model, questions)
  File "eval_rag.py", line 99, in evaluate_batch_retrieval
    question_enc_pool_output = question_enc_outputs.pooler_output
AttributeError: 'tuple' object has no attribute 'pooler_output'
```

With this simple change (`question_enc_outputs.pooler_output` -> `question_enc_outputs[0]`), I got to run the evaluation code and confirmed
`INFO:__main__:Precision@1:  70.74`

## Environments
- Ubuntu 18.04 LTS
- Python 3.7.7
- transformers 4.0.1
- torch: 1.7.1

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->
@ola13 (confirmed by `git blame`) @patrickvonplaten @lhoestq


run_clm.py Early stopping with ^C

- `transformers` version: 4.0.1
- Platform: Colab
- Python version: 3.6.9
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger

## Information

Model I am using: GPT2

The problem arises when using:
`run_clm.py`

## To reproduce
`!python ./transformers/examples/language-modeling/run_clm.py \
    --model_name_or_path ./GPT2_PRETRAINED_LOCAL \
    --dataset_name bookcorpusopen \
    --dataset_config_name plain_text \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 2 \
    --block_size 128 \
    --gradient_accumulation_steps 1 \
    --overwrite_output_dir \
    --do_train \
    --do_eval \
    --num_train_epochs 20 \
    --save_steps 50000 \
    --save_total_limit 1 \
    --output_dir ./GPT2-trained-save`

Output:
`2020-12-13 20:02:51.391764: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
12/13/2020 20:02:53 - WARNING - __main__ -   Process rank: -1, device: cpu, n_gpu: 0distributed training: False, 16-bits training: False
12/13/2020 20:02:53 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='./RPT-trained-save', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=2, per_device_eval_batch_size=2, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=20.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Dec13_20-02-53_7d34d2e22bee', logging_first_step=False, logging_steps=500, save_steps=50000, save_total_limit=1, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='./RPT-trained-save', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None)
Downloading: 4.00kB [00:00, 2.73MB/s]
Downloading: 2.10kB [00:00, 1.47MB/s]
Downloading and preparing dataset book_corpus_open/plain_text (download: 2.24 GiB, generated: 6.19 GiB, post-processed: Unknown size, total: 8.43 GiB) to /root/.cache/huggingface/datasets/book_corpus_open/plain_text/1.0.0/5cc3e4620a202388e77500f913b37532be8b036287436f3365e066671a1bd97e...
Downloading: 100% 2.40G/2.40G [02:41<00:00, 14.9MB/s]
9990 examples [01:04, 149.50 examples/s]^C`

The ^C automatically appears and the script stops.

## Expected behavior
The training process takes place as normal.


BertForSequenceClassification finetune training loss and accuracy have some problem

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: v4.0.0
- Platform: colab pro
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0+cu101
- Tensorflow version (GPU?): 2.3.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

@sgugger
@JetRunner

## Information
I follow the paper https://arxiv.org/pdf/2003.02245.pdf to do augmentation and test the performance

Model I am using Bert for BertTokenizer, BertForMaskedLM, BertForSequenceClassification

The problem arises when using:
Use Trainer to fine-tuning on both training set and concatenated set of training set and augmentation set, the Training log is No log or 0.683592, and accuracy is always 0.8

The tasks I am working on is:
An official GLUE  task: sst2, using by huggingface datasets package

The details:
Trainer setting I follow the examples/text_classification.ipynb to build the compute_metrics function and tokenize mapping function, but the training loss and accuracy have bug

my tokenized datasets format:
![擷取](https://user-images.githubusercontent.com/39672039/102020487-7d44ca80-3db4-11eb-8f48-590f885f6957.JPG)

compute_function, little modify by examples/text_classification.ipynb
![image](https://user-images.githubusercontent.com/39672039/102020602-24c1fd00-3db5-11eb-9ed3-5c3a55e50ac9.png)

bert_finetuned_setting
![image](https://user-images.githubusercontent.com/39672039/102020585-05c36b00-3db5-11eb-9440-aa715b3f4c36.png)

fine_tuned result
![image](https://user-images.githubusercontent.com/39672039/102020616-3c00ea80-3db5-11eb-9bcd-95380c8166fc.png)


Getting a 404 error when loading TFXLMRobertaModel from 'xlm-roberta-large'

Getting a 404 when trying to load the model.
Manyually  checked the https://huggingface.co repository for the xlm-roberta-large, was only able to find the Pytorch models, why aren't the TF models available for this, if its not, why is it not explicitly mentioned in the documentation?


Adding to docs how to train CTRL Model with control codes.

# 🚀 Feature request

At the moment there is no explanation in the docs how to train a CTRL model with user defined `control codes`.

## Motivation

At the moment there is no explanation in the docs how to train a CTRL model with user defined `control codes`. I think it should be added because control codes are an important part of CTRL model.

## Your contribution

 I am currently struggling on coming up with ideas on how to do that using transformer interface, but I'd love to open a PR after I understand how to do that.


Problem with Token Classification models

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1
- Platform: Win10
- Python version:3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.3 CPU
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

 examples/token-classification: @stefan-it

## Information

I followed this tutorial https://huggingface.co/transformers/custom_datasets.html#token-classification-with-w-nut-emerging-entities for token classification but results were really bad. So I changed the dataset to conll2003 and simplified the data a little (remove sentences without entities, keep only sentences with a certain length) as I saw that the model should perform well on this data.  Unfortunately the results are still bad for example after epoch two with the bert model set trainable=True:

Conf mat: (rows are prediction, columns are the  labels)
[[  1   3  21  10   5  10   1  16 172]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   1   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  0   0   0   0   0   0   0   0   0]
 [  2   7   7  13   7   6   0   9 162]]

Classification report
('              precision    recall  f1-score   support\n'
 '\n'
 '           0       0.33      0.00      0.01       239\n'
 '           1       0.00      0.00      0.00         0\n'
 '           2       0.00      0.00      0.00         0\n'
 '           3       0.00      0.00      0.00         1\n'
 '           4       0.00      0.00      0.00         0\n'
 '           5       0.00      0.00      0.00         0\n'
 '           6       0.00      0.00      0.00         0\n'
 '           7       0.00      0.00      0.00         0\n'
 '           8       0.49      0.76      0.59       213\n'
 '\n'
 '    accuracy                           0.36       453\n'
 '   macro avg       0.09      0.08      0.07       453\n'
 'weighted avg       0.40      0.36      0.28       453\n')

I tried a lot of things and checked pre-processing and post-processing multiple times and can't find a bug in there.
The model is close to the tutorial(in the tutorial it's a DistilBert model but as if performed in the same manner I changed to the bigger brother) but it seems like it's not learning at all. Though it should perform well with conll data and in other tutorials this model has shown good results (for example: https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Here's the code is use (not complete preprocessing):

```

import tensorflow as tf
from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')

train_encodings = tokenizer(train_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)
val_encodings = tokenizer(val_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)

train_dataset = tf.data.Dataset.from_tensor_slices((dict(train_encodings), train_labels))
val_dataset = tf.data.Dataset.from_tensor_slices((dict(val_encodings), val_labels))

from transformers import TFBertForTokenClassification

model = TFBertForTokenClassification.from_pretrained('bert-base-cased', num_labels=len(unique_tags)) #unique tags are infered from training data
model.layers[0].trainable = True
optimizer = tf.keras.optimizers.Adam(learning_rate=5e-5)

model.compile(optimizer=optimizer, loss=model.compute_loss)  #or tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)

for epoch in range(8):
    model.fit(train_dataset.shuffle(64).batch(16), batch_size=16, verbose=1, epochs=10)
    predictions = model.predict(val_dataset)
    # some post-processing. predictions is a list with all logits. for calculating the metrics I only consider
    # the tags that are not -100 (which are supposed to be ignored ).
    good_indexes = [i for i, l in enumerate(val_labels) if l != -100]
    list_preds = []
    for logi in predictions['logits']:
        list_preds.append(np.argmax(logi))
    pred_post = [list_preds[j] for j in good_indexes]
    print(confusion_matrix(pred_post, label_post))
    report = classification_report(pred_post, label_post)
    pprint(report)

```

## Expected behavior

Better performance  i.e. reasonable F1 Scores   (for example: https://www.depends-on-the-definition.com/named-entity-recognition-with-bert/)


Image rendering not working in example notebook

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: n/a
- Platform: n/a
- Python version: n/a
- PyTorch version (GPU?): n/a
- Tensorflow version (GPU?): n/a
- Using GPU in script?: n/a
- Using distributed or parallel set-up in script?: n/a

### Who can help
As advised looking at the git blame @mfuntowicz @n1t0 could you advise?

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/huggingface/transformers/blob/master/notebooks/02-transformers.ipynb
2. Go to section `Want it lighter? Faster? Let's talk distillation!`
3. You should see that there is an image which is not rendering like the below
![image](https://user-images.githubusercontent.com/30328618/101995456-cff18a00-3cc1-11eb-81c2-3685e96c5f52.png)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Should be an image to explain something

<!-- A clear and concise description of what you would expect to happen. -->


Add parallelization support for T5EncoderModel

# What does this PR do?

Extend T5EncoderModel to support model parallization across different GPUs.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

T5: @patrickvonplaten


Segmentation fault (core dumped) running run_qa.py

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.0.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1+cu110 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten @TevenLeScao
 Blenderbot: @patrickvonplaten
 Bart: @patrickvonplaten
 Marian: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 RAG: @patrickvonplaten, @lhoestq
 FSMT: @stas00
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): distilbert-base-uncased (but other bert variants  do the same)

The problem arises when using:
* [ X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.mkdir squad
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -O squad/train-v2.0.json
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O squad/dev

2. python run_qa.py \
  --model_name_or_path distilbert-base-uncased  \
  --do_train \
  --train_file ./squad/train-v2.0.json \
  --per_device_train_batch_size 2 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ./models/ \
  --overwrite_output_dir

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

12/12/2020 21:22:50 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1distributed training: False, 16-bits training: False
12/12/2020 21:22:50 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='./models/', overwrite_output_dir=True, do_train=True, do_eval=False, do_predict=False, evaluation_strategy=<EvaluationStrategy.NO: 'no'>, prediction_loss_only=False, per_device_train_batch_size=2, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, eval_accumulation_steps=None, learning_rate=3e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=2.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Dec12_21-22-50_piero-laptop', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=500, dataloader_num_workers=0, past_index=-1, run_name='./models/', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=None)
Using custom data configuration default
Downloading and preparing dataset json/default-0b904584a9578d6f (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/piero/.cache/huggingface/datasets/json/default-0b904584a9578d6f/0.0.0/70d89ed4db1394f028c651589fcab6d6b28dddcabbe39d3b21b4d41f9a708514...
0 tables [00:00, ? tables/s]Segmentation fault (core dumped)

## Note:
I would like to test the script on the downloaded SQUAD dataset to apply the script after to my own dataset. If I run as below, everything works fine
python run_qa.py \
  --model_name_or_path bert-base-uncased \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 4 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir ./models \
  --overwrite_output_dir


Fine tune GPT-2 pytorch

Hello,

I want to fine tune GPT-2 (PyTorch version) on a custom dataset. Words or small phrases of the dataset are marked, for example:

_some text [ss] word / small phrase [se] some other text._

I want to generate this kind of text with GPT-2, so firstly I thought to add [ss] and [se] as special tokens.

I am looking for a training script sample for GPT-2, to see how to prepare data as input for the model (if preprocessing, or specific format is needed), which type of loss to use, etc. but I cannot find any. I also looked through the [library](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_clm.py) and also didn't find the training script.

Are there any suggestions?

Thank you in advance.
P.s. if this is not the appropriate place for this question, feel free to direct me accordingly.