AutoTokenizer not able to load saved Roberta Tokenizer

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

I'm trying to run `run_language_modelling.py` with my own tokenizer

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
!pip install transformers --upgrade

from transformers import AutoTokenizer
my_tokenizer = AutoTokenizer.from_pretrained("distilroberta-base")
!mkdir my_tokenizer
my_tokenizer.save_pretrained("my_tokenizer")
my_tokenizer2 = AutoTokenizer.from_pretrained("./my_tokenizer")
```

### Stack Trace:
```
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    246                 resume_download=resume_download,
--> 247                 local_files_only=local_files_only,
    248             )

4 frames
/usr/local/lib/python3.6/dist-packages/transformers/file_utils.py in cached_path(url_or_filename, cache_dir, force_download, proxies, resume_download, user_agent, extract_compressed_file, force_extract, local_files_only)
    260         # File, but it doesn't exist.
--> 261         raise EnvironmentError("file {} not found".format(url_or_filename))
    262     else:

OSError: file ./my_tokenizer/config.json not found

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-16-b699c4abfe8e> in <module>()
      3 get_ipython().system('mkdir my_tokenizer')
      4 my_tokenizer.save_pretrained("my_tokenizer")
----> 5 my_tokenizer2 = AutoTokenizer.from_pretrained("./my_tokenizer")

/usr/local/lib/python3.6/dist-packages/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    184         config = kwargs.pop("config", None)
    185         if not isinstance(config, PretrainedConfig):
--> 186             config = AutoConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
    187
    188         if "bert-base-japanese" in pretrained_model_name_or_path:

/usr/local/lib/python3.6/dist-packages/transformers/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    185         """
    186         config_dict, _ = PretrainedConfig.get_config_dict(
--> 187             pretrained_model_name_or_path, pretrained_config_archive_map=ALL_PRETRAINED_CONFIG_ARCHIVE_MAP, **kwargs
    188         )
    189

/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    268                     )
    269                 )
--> 270             raise EnvironmentError(msg)
    271
    272         except json.JSONDecodeError:

OSError: Can't load './my_tokenizer'. Make sure that:

- './my_tokenizer' is a correct model identifier listed on 'https://huggingface.co/models'

- or './my_tokenizer' is the correct path to a directory containing a 'config.json' file
```

## Expected behavior

AutoTokenizer should be able to load the tokenizer from the file.

Possible duplicate of #1063 #3838

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Model card for spanish electra small

model card

Related to pretrained model cards


add XLMRobertaForQuestionAnswering

1. add XLMRobertaForQuestionAnswering in modeling_xlm_roberta.py
2. add XLMRobertaForQuestionAnswering in modeling_auto.py


Optimize PyTorch's GPT2 computation graph

Brings various optimizations to make the computation graph more efficient.


Language model training with NSP using run_language_modeling.py

When using "run_language_modeling.py" for fine-tuning BERT is only the MLM task used or also the NSP task as described in the paper?


[Reformer] Doctsring: fix examples again

Forgot a fix.


[Reformer] Fix example and error message

Fix typos and improve error message.


[Reformer] fix docstring

Fix small typo in docstring


Bug: can not use pretrained BERT on multiple GPUs with DataParallel (PyTorch 1.5.0)

Python: 3.6.10
PyTorch: 1.5.0
Transformers: 2.8.0 and 2.9.0

In the following code, I wrap the pretrained BERT with a DataParallel wrapper so as to run it on multiple GPUs:

> import torch, transformers
> model = transformers.AutoModel.from_pretrained("bert-base-multilingual-cased")
> model = torch.nn.DataParallel(model)
> model = model.cuda()
> input = torch.ones([16, 10], dtype=torch.long)
> input = input.cuda()
> model(input)

But I got the following error:

> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
>     outputs = self.parallel_apply(replicas, inputs, kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
>     return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
>     output.reraise()
>   File "/home/anaconda/lib/python3.6/site-packages/torch/_utils.py", line 395, in reraise
>     raise self.exc_type(msg)
> StopIteration: Caught StopIteration in replica 0 on device 0.
> Original Traceback (most recent call last):
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
>     output = module(*input, **kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
>     result = self.forward(*input, **kwargs)
>   File "/home/anaconda/lib/python3.6/site-packages/transformers/modeling_bert.py", line 734, in forward
>     extended_attention_mask = extended_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
> StopIteration
>

But it will work if I remove the DataParallel wrapper.


Fix Albert Attention

This PR simplifies  the code of class AlbertAttention.


How to apply Torchtext convenience classes to prepare data for a Transformer?

Hello,
Reading the tutorial [Language Translation with torchText](https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html) I wondered how someone could use those convenience classes (`Field, BucketIterator`) to train/fine-tune a `Transformer`.

For instance, I'm currently working with a large dataset distributed in jsonl files which looks like:
```python
{ "query": "this is a query 1", "doc": "relevant document regarding query 1" },
{ "query": "this is a query 2", "doc": "relevant document regarding query 2" },
               ...

```

Now, to forward this data into a transformer like Bert, it is necessary to convert this dataset into the format:

```python3
(
#queries
 {
    'input_ids': tensor([
        [  101,  2023,  2003,  1037, 23032,  1015,   102,     0],
        [  101,  2023,  2003,  1037, 23032,  1016,   102,     0]]),
    'attention_mask': tensor([
        [1, 1, 1, 1, 1, 1, 1, 0],
        [1, 1, 1, 1, 1, 1, 1, 0]])
 },

 #docs
 {
    'input_ids': tensor([
        [ 101, 2023, 2003, 2028, 7882, 6254, 4953,  102],
        [ 101, 2023, 2003, 2028, 7882, 6254, 4953,  102]]),
    'attention_mask': 'input_ids': tensor([
        [1, 1, 1, 1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1]])
}
```
So, what would be a clear and efficient approach to apply those convenience classes to tokenize a text dataset to fit it in the required format of a transformer?


Add patience argument to Trainer

This closes #4894.

# Summary
Often, we want to stop training if loss does not improve for a number of epochs. This PR adds a "patience" argument, which is a limit on the number of times we can get a non-improving eval loss before stopping training early.

It is implemented by other NLP frameworks, such as AllenNLP (see [trainer.py](https://github.com/allenai/allennlp/blob/master/allennlp/training/trainer.py#L95) and [metric_tracker.py](https://github.com/allenai/allennlp/blob/1a8a12cd1b065d74fec3d2e80105a684736ff709/allennlp/training/metric_tracker.py#L6)).

# Motivation
This feature allows faster fine-tuning by breaking the training loop early and avoids users the toil of checking metrics on Tensorboard.

# Caveats
Often, models are evaluated once per epoch, but run_lm_finetuning.py has an option to evaluate after a set number of model update steps (dictated by `--logging_steps` if `--evaluate_during_training` is true). Because of this, I've elected to tie patience to the number of evaluations without improvement in loss.


New model request: MobileBERT from Google

# 🌟 New model addition
This issue is to request adding the MobileBERT model, which was recently released by Carnegie Mellon University, Google Research, and Google Brain.

## Model description
MobileBERT is a more computationally-efficient model for achieving BERT-base level accuracy on a smartphone.

<!-- Important information -->

## Open source status

* [x] the model implementation is available: https://github.com/google-research/google-research/tree/master/mobilebert
* [x] the model weights are available: [uncased_L-24_H-128_B-512_A-4_F-4_OPT](https://storage.googleapis.com/cloud-tpu-checkpoints/mobilebert/uncased_L-24_H-128_B-512_A-4_F-4_OPT.tar.gz) (MobileBERT Optimized Uncased English)
* [x] who are the authors: Zhiqing Sun, Hongkun Yu (@saberkun), Xiaodan Song, Renjie Liu, Yiming Yang, Denny Zhou


Model card for allegro/herbert-klej-cased-tokenizer-v1

Related to pretrained model cards


Model card for allegro/herbert-klej-cased-v1

Related to pretrained model cards


Is it possible to get document embeddings using GPT-2? If so, how?

# ❓ Questions & Help

I posted a question here on SO: https://stackoverflow.com/questions/61641257/how-to-get-document-embeddings-using-gpt-2?noredirect=1#comment109035633_61641257

But unfortunately was informed that "Please notice that SO is not a tutorial service, and also that recommendation requests for external resources are explicitly off-topic."

## Details
I'm curious if using GPT-2 might yield a higher accuracy for document vectors (with greatly varying length) or not (would it surpass the state of the art?)

Really I'm most interested in document embeddings that are as accurate as possible. I'm wondering if using GPT-2 will get results that are more accurate than Paragraph Vectors for example.

I heard that in order to get vectors from GPT-2 "you can use a weighted sum and/or concatenation of vector outputs at its hidden layers (typically the last few hidden layers) as a representation of its corresponding words or even "meaning" of the entire text, although for this role BERT is used more often as it is bi-directional and takes into account of both forward and backward contexts."

As a machine learning and NLP beginner, I'd love to know how to go about this, or to be pointed in the right direction to learn more about how to attempt this in Python.

I've tried fine-tuning GPT-2 before but I have no idea how to extract vectors from it for text.

**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/61641257/how-to-get-document-embeddings-using-gpt-2?noredirect=1#comment109035633_61641257


[Marian] Key-Error for some languages

model -> symbol that caused `KeyError`.

```python

{'ha-en': '|',
 'ber-es': '▁Be',
 'pis-fi': '▁|',
 'es-mt': '|',
 'fr-he': '₫',
 'niu-sv': 'OGI',
 'fi-fse': '▁rentou',
 'fi-mh': '|',
 'hr-es': '|',
 'fr-ber': '▁devr',
 'ase-en': 'olos'}
```

Reproduce
```python
pair = 'ber-es'
mname = f'Helsinki-NLP/opus-mt-{pair}'
tok = MarianTokenizer.from_pretrained(mname)
tok.prepare_translation_batch(['Bessif kan ay aɣ-d-iṣaḥ wakud akken ad necc imensi.'])
```

Traceback:
```python
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-55-5da2d9c189be> in <module>
      2 mname = f'Helsinki-NLP/opus-mt-{pair}'
      3 tok = MarianTokenizer.from_pretrained(mname)
----> 4 tok.prepare_translation_batch(['Bessif kan ay aɣ-d-iṣaḥ wakud akken ad necc imensi.'])

~/transformers_fork/src/transformers/tokenization_marian.py in prepare_translation_batch(self, src_texts, tgt_texts, max_length, pad_to_max_length, return_tensors)
    145             max_length=max_length,
    146             pad_to_max_length=pad_to_max_length,
--> 147             src=True,
    148         )
    149         if tgt_texts is None:

~/transformers_fork/src/transformers/tokenization_utils.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, max_length, stride, truncation_strategy, pad_to_max_length, is_pretokenized, return_tensors, return_token_type_ids, return_attention_masks, return_overflowing_tokens, return_special_tokens_masks, return_offsets_mapping, return_lengths, **kwargs)
   1729                 ids, pair_ids = ids_or_pair_ids, None
   1730
-> 1731             first_ids = get_input_ids(ids)
   1732             second_ids = get_input_ids(pair_ids) if pair_ids is not None else None
   1733             input_ids.append((first_ids, second_ids))

~/transformers_fork/src/transformers/tokenization_utils.py in get_input_ids(text)
   1697             if isinstance(text, str):
   1698                 tokens = self.tokenize(text, add_special_tokens=add_special_tokens, **kwargs)
-> 1699                 return self.convert_tokens_to_ids(tokens)
   1700             elif isinstance(text, (list, tuple)) and len(text) > 0 and isinstance(text[0], str):
   1701                 return self.convert_tokens_to_ids(text)

~/transformers_fork/src/transformers/tokenization_utils.py in convert_tokens_to_ids(self, tokens)
   1340         ids = []
   1341         for token in tokens:
-> 1342             ids.append(self._convert_token_to_id_with_added_voc(token))
   1343         return ids
   1344

~/transformers_fork/src/transformers/tokenization_utils.py in _convert_token_to_id_with_added_voc(self, token)
   1349         if token in self.added_tokens_encoder:
   1350             return self.added_tokens_encoder[token]
-> 1351         return self._convert_token_to_id(token)
   1352
   1353     def _convert_token_to_id(self, token):

~/transformers_fork/src/transformers/tokenization_marian.py in _convert_token_to_id(self, token)
     90
     91     def _convert_token_to_id(self, token):
---> 92         return self.encoder[token]
     93
     94     def _tokenize(self, text: str, src=True) -> List[str]:

KeyError: '▁Be'

```


considering empty input text to avoid string index out of range error

When the input string is empty, the above condition in the tokenization_roberta will encounter string index out of range error. For example, a pair input string exists in the QQP dataset that has an empty string:

{'idx': 362246,
 'label': 0,
 'question1': b'How can I develop android app?',
 'question2': b''}


Create README.md

Related to pretrained model cards

model card for my De Novo Drug discovery model using MLM


[Model Cards] Add 1010 model cards for Helsinki-NLP

Related to pretrained model cards

- Generated using tools in `convert_marian_to_pytorch.py`.
- Takes the bottom most entry in each opus-mt-train/models/*/README.md. This assumes that the bottom entry is most recent. (Created by @jorgtied).
- Example below is at `model_cards/Helsinki-NLP/opus-mt-fr-en/README.md`
__________________________

# opus-2020-02-26.zip

* dataset: opus
* model: transformer-align
* pre-processing: normalization + SentencePiece
* download: [opus-2020-02-26.zip](https://object.pouta.csc.fi/OPUS-MT-models/fr-en/opus-2020-02-26.zip)
* test set translations: [opus-2020-02-26.test.txt](https://object.pouta.csc.fi/OPUS-MT-models/fr-en/opus-2020-02-26.test.txt)
* test set scores: [opus-2020-02-26.eval.txt](https://object.pouta.csc.fi/OPUS-MT-models/fr-en/opus-2020-02-26.eval.txt)

## Benchmarks

| testset               | BLEU  | chr-F |
|-----------------------|-------|-------|
| newsdiscussdev2015-enfr.fr.en 	| 33.1 	| 0.580 |
| newsdiscusstest2015-enfr.fr.en 	| 38.7 	| 0.614 |
| newssyscomb2009.fr.en 	| 30.3 	| 0.569 |
| news-test2008.fr.en 	| 26.2 	| 0.542 |
| newstest2009.fr.en 	| 30.2 	| 0.570 |
| newstest2010.fr.en 	| 32.2 	| 0.590 |
| newstest2011.fr.en 	| 33.0 	| 0.597 |
| newstest2012.fr.en 	| 32.8 	| 0.591 |
| newstest2013.fr.en 	| 33.9 	| 0.591 |
| newstest2014-fren.fr.en 	| 37.8 	| 0.633 |
| Tatoeba.fr.en 	| 57.5 	| 0.720 |


Not able to import certain packages

# 🐛 Bug

## Information
- I am using `run_glue.py`.
- Working with GLUE

## To reproduce

Steps to reproduce the behavior:
Even though these functions are imported in `__init__.py` (I have thoroughly checked myself), I'm not able to import `EvalPrediction`, `HfArgumentParser`, `Trainer`, `TrainingArguments`

## Expected behavior
Should be able to import them. I'm sharing this [colab notebook](https://colab.research.google.com/drive/1CqWzt9A96iUT2vh47qwnMSDMsQHQkRSx) link.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8
- Platform: Ubuntu 18.04
- Python version: 1.5
- PyTorch version (GPU?): 1.5
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


ONNX conversion script.

This PR adds conversion script to export our models to ONNX IR.

Plan is to support both PyTorch and TensorFlow:

- [x] PyTorch
- [ ] TensorFlow

TensorFlow currently blocked because of an issue in the conversion script provided by ONNX (seems fixed on master : https://github.com/onnx/tensorflow-onnx/issues/876)


args.output_dir seems like been ignored

https://github.com/huggingface/transformers/blob/a638e986f45b338c86482e1c13e045c06cfeccad/examples/run_squad.py#L814
It seems that args.output_dir is been ignored in checkpoints when args.eval_all_checkpoints is True.
It probably should be like this:
global_step = checkpoint.split("-")[-1] if (len(checkpoints) > 1 and checkpoint != args.output_dir) else ""


Make ElectraPreTrainedModel importable

# 🚀 Feature request

Make `ElectraPreTrainedModel` importable

## Motivation

Consistency.

`DistilBertPreTrainedModel` and `AlbertPreTrainedModel` are importable but I cannot import `ElectraPretrainedModel` from v2.8.0
(see https://github.com/huggingface/transformers/issues/1968)

## Your contribution

https://github.com/huggingface/transformers/pull/4173


Include ElectraPreTrainedModel into __init__


ImportError: cannot import name 'AutoModel' from 'transformers'

# 🐛 Bug

(Not sure that it is a bug, but it is too easy to reproduce I think)

## Information

I couldn't run `python -c 'from transformers import AutoModel'`, instead getting the error on the titile.

## To reproduce

Steps to reproduce the behavior:

1. `$ sudo docker run -it --rm python:3.6 bash`
2. `# pip install tensorflow==2.0 transformers==2.8.0`
3. `# python -c 'from transformers import AutoModel'`

```
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: cannot import name 'AutoModel'
```

Initially I got this error with `transformers-cli download` :

```
# transformers-cli download t5-large
Traceback (most recent call last):
  File "/usr/local/bin/transformers-cli", line 32, in <module>
    service.run()
  File "/usr/local/lib/python3.6/site-packages/transformers/commands/download.py", line 29, in run
    from transformers import AutoModel, AutoTokenizer
ImportError: cannot import name 'AutoModel'
```

## Expected behavior

no import error.

## Environment info

- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-99-generic-x86_64-with-debian-10.3
- Python version: 3.6.10
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

Also tested on 3.6@Ubuntu 18.04, 3.8@Docker(python:3.8).
The Docker backend is Ubuntu 18.04.

Thank you.


Encoder/Decoder generation

Hello! I tried to train a Bert2Bert model for QA generation, however when I try the generate function it returns gibberish. I also tried using the example code below, and that also generated gibberish(the output is "[PAD] leon leon leon leon leonieieieieie shall shall shall shall shall shall shall shall shall"). Is the generate function supposed to work for EncoderDecoder models, and what am I doing wrong?
```
from transformers import EncoderDecoderModel, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert
generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)
```


How to position encode a sentence?

BertTokenizer provide us with input_ids, attention_mask and token_type_ids but i'm unable to get so called "Position ids".
I have read the documentation as well but unable to find anything.


Update __init__.py for AlbertMLMHead


# 🌟 New model addition

## Model description

MLM model head
```
No name 'AlbertMLMHead' in module 'transformers'
```

## Open source status

* [x] the model implementation is available:
```
class AlbertMLMHead(nn.Module):
    def __init__(self, config):
        super().__init__()

        self.LayerNorm = nn.LayerNorm(config.embedding_size)
        self.bias = nn.Parameter(torch.zeros(config.vocab_size))
        self.dense = nn.Linear(config.hidden_size, config.embedding_size)
        self.decoder = nn.Linear(config.embedding_size, config.vocab_size)
        self.activation = ACT2FN[config.hidden_act]

        # Need a link between the two variables so that the bias is correctly resized with `resize_token_embeddings`
        self.decoder.bias = self.bias

    def forward(self, hidden_states):
        hidden_states = self.dense(hidden_states)
        hidden_states = self.activation(hidden_states)
        hidden_states = self.LayerNorm(hidden_states)
        hidden_states = self.decoder(hidden_states)

        prediction_scores = hidden_states

        return prediction_scores

[DOCS]@add_start_docstrings(
    "Albert Model with a `language modeling` head on top.", ALBERT_START_DOCSTRING,
)
```
Include in the __initial__ file and make it available for use.

* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


change order pytorch/tf in readme


Tapas

# 🌟 New model addition

## Model descriptin

Tapas extends Bert architecture and is a transformer-based Table QA model

Paper: https://arxiv.org/abs/2004.02349

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/google-research/tapas
* [X] the model weights are available: (give details)
The pretrained weights and data for fine-tuning are linked in the repository readme

* [x] who are the authors:
@thomasmueller-google is one of the authors


[RFC] Sampling transform function for generation

See https://github.com/huggingface/transformers/issues/4164


Add a sampling_transform callback to generation for arbitrary probability-warps

# 🚀 Feature request
I'd like to add `sampling_transform` callback argument to all `generate` functions in `modeling_utils` that allows arbitrary sampling from the probability distribution during sequence generation. The signature of this function would be `(input_ids, next_probs, next_token) -> next_token`.

## Motivation
The modeling_utils's generation function is getting pretty hairy -- including parameters for top p, top k, bad tokens, temperature, repetition penalty, etc.. Every new way of sampling means that we have to add more parameters and it further complicate the function. I believe the right way of solving this is to provide a function-argument that allows users to express arbitrary rules for the next sample. In the long run, we could replace all the other parameters with a set of pre-baked sampling_transform functions that can be composed at will.

This method scales better to strange warps -- for example, in a project I'm working on (https://github.com/turtlesoupy/this-word-does-not-exist) I need to early-terminate sequences if they generate from a large set of bad tokens and need to continue generating if an EOS token is sampled too early. An example is here https://github.com/turtlesoupy/this-word-does-not-exist/blob/260e33a8f420b9be8b1e7260cb03c74d6231686e/title_maker_pro/datasets.py#L386

## Your contribution

I'll attach a sample PR that makes this work for pytorch and non-beam samples. If people like the idea, it should be easy to refine into a full PR that generalize to beam search and tensorflow.


Cannot use camembert for question answering

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Camembert
Language I am using the model on (English, Chinese ...):
French
The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: Squad
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load camembert for Q&A
2. Use the script for Q&A from the HuggingFace Doc
3. Get a Runtimeerror

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

from [https://huggingface.co/transformers/usage.html#question-answering](https://huggingface.co/transformers/usage.html#question-answering) :

Note : I tried with `camembert-base`, `illuin/camembert-base-fquad` and `fmikaelian/camembert-base-fquad`
```
from transformers import AutoTokenizer, CamembertForQuestionAnswering
import torch

tokenizer = AutoTokenizer.from_pretrained("camembert-base")
model = CamembertForQuestionAnswering.from_pretrained("camembert-base")

text = r"""Some text in french"""
questions = ["Just one question in french"]

for question in questions:
    inputs = tokenizer.encode_plus(question, text, add_special_tokens=True, return_tensors="pt")
    input_ids = inputs["input_ids"].tolist()[0]

    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
    answer_start_scores, answer_end_scores = model(**inputs)

    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score

    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

    print(f"Question: {question}")
    print(f"Answer: {answer}\n")
```
It fails as well with the `pipeline` method
```
q_a_pipeline = pipeline("question-answering", model=model, tokenizer=tokenizer)

q_a_pipeline({'question': question, 'context': text}
```
Stack trace :
```
Traceback (most recent call last):
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/sanic/app.py", line 976, in handle_request
    response = await response
  File "test_server_sanic.py", line 72, in get_answer
    results = [q_a_pipeline({'question': question, 'context': doc})
  File "test_server_sanic.py", line 72, in <listcomp>
    results = [q_a_pipeline({'question': question, 'context': doc})
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/pipelines.py", line 1109, in __call__
    start, end = self.model(**fw_args)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_roberta.py", line 663, in forward
    outputs = self.roberta(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_bert.py", line 728, in forward
    embedding_output = self.embeddings(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_roberta.py", line 64, in forward
    return super().forward(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/transformers/modeling_bert.py", line 175, in forward
    token_type_embeddings = self.token_type_embeddings(token_type_ids)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/modules/sparse.py", line 112, in forward
    return F.embedding(
  File "/home/covid_nlu/.local/lib/python3.8/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Run the exemple in french as with Bert in english

Note : I am able to run the exemple with a Hugging Face Pipeline (with all the different camembert model, comunity or not)
```
bert_tok = AutoTokenizer.from_pretrained("camembert-base")
bert = CamembertForQuestionAnswering.from_pretrained("camembert-base")

nlp = pipeline('question-answering', model=bert, tokenizer=bert_tok)

answer = nlp({'question': "A question in french",
    'context': a_big_string_in_french})
print(answer)
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-5.6.10-arch1-1-x86_64-with-glibc2.2.5
- Python version: 3.8.2
- PyTorch version (GPU?): 1.4.0 (True) (Same with 1.5)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Add model card for the NER model

Related to pretrained model cards

Add the model-card for a new NER model.


run_generation.py - use of < redirect for input text file only reads first line - solution requested

Example:

python3 examples/run_generation.py --model_type=gpt2 --model_name_or_path="models/custom-774M" --length=400 < test.txt > test_gpt2_400.txt

Version 2.0.3 and earlier accepts a simple linux redirect < test.txt and reads the file line-by-line while generating the corresponding output to > test_gpt2_400_out.txt

This works perfectly and does not require any changes. The test.txt is a simple 3 line text file with 1 sentence on each line.

Versions 2.0.4 to 2.0.8 do not accept the < redirect in the same manner. The above only reads the first line of 'test.txt', generates, then exits. Output is the same for the first line.

Is there a simple change to the 2.0.8 version of 'run_generation.py' that will return this useful terminal function under ubuntu 18.04?

Have tried 'while - read' loops and only managed to generate the Model prompt >>> 3x requesting terminal input:

exec 3<"test.txt"

while IFS= read -r line <&3; do

python3 examples/run_generation.py --model_type=gpt2 --model_name_or_path="models/custom-774M" --length=400

done > test_gpt2_400_out.txt

Any help is appreciated.


[Marian] Multilingual models require language codes

[ ] figure out codes
[ ] update/subclass `MarianTokenizer`


Tokenizer.batch_decode convenience method

convenience method that turns the list returned by `model.generate` into a list of sentences.


[Marian] @-@ symbol causes strange generations

good fr-en test case:
```
Veuillez m' apporter une demi @-@ bouteille de vin .
```


[Marian] Readme parser defaults to porting oldest model

- logic should be newest model or all models with unique names.
- the download URL (and probably other metadata) should be put in a `model_cards/` file.


Removed the use of deprecated Variable API in PPLM example


num_samples=0 when using pretrained model

When training the "run_language_modelling.py" example with a pretrained bert model, I get the error:

"""
Traceback (most recent call last):                                                                                File "run_language_modeling.py", line 284, in <module>                                                                                main()                                                                                File "run_language_modeling.py", line 254, in main                                                                                trainer.train(model_path=model_path)                                                                                File "C:\Users\AppData\Roaming\Python\Python38\site-packages\transformers\trainer.py", line 243, in train                                                                                train_dataloader = self.get_train_dataloader()                                                                                File "C:\Users\AppData\Roaming\Python\Python38\site-packages\transformers\trainer.py", line 179, in get_train_dataloader                                                                                RandomSampler(self.train_dataset) if self.args.local_rank == -1 else DistributedSampler(self.train_dataset)                                                                                File "C:\Users\Anaconda3\envs\kompetence\lib\site-packages\torch\utils\data\sampler.py", line 93, in __init__                                                                                raise ValueError("num_samples should be a positive integer "                                                                                ValueError: num_samples should be a positive integer value, but got num_samples=0
"""
The cached_lm_berttokenizer that gets saved to the data folder looks like this:

"""
€]”.
"""
In the folder for my bert model, I have a config.json, pytorch_model.bin, special_tokens_map.json, vocab.txt, tokenizer_config.json. I run the following command

python run_language_modeling.py --output_dir=. --model_type=bert --model_name_or_path="H:\danish_bert_uncased_v2\\" --mlm --train_data_file="H:\\data\train.txt" --eval_data_file="H:\data\test.txt" --do_eval --do_train --overwrite_output_dir -block_size=1000

Is anything missing here? The data is composed of sentences seperated by newlines and an empty line between documents.


Rewritten batch support in pipelines.

Batch support in Pipeline was confusing and not well tested.

This PR rewrites all the content of `DefaultArgumentHandler` which handles most of the input conversions (args, kwargs, batched, etc.) and brings unit tests on this specific class.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


Embedding index getting out of range while running camemebert model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Camembert

Language I am using the model on (English, Chinese ...):
French

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
Steps to reproduce the behavior:

1. Take a file with french text
2. Load pretrained Camembert Model and tokenizer as in the doc
3. Run inference

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

* Initialisation :
`bert = CamembertModel.from_pretrained("camembert-base")`
`bert_tok = CamembertTokenizer.from_pretrained("camembert-base")`

* Inference : like [https://huggingface.co/transformers/usage.html#question-answering](https://huggingface.co/transformers/usage.html#question-answering)

```
inputs = bert_tok.encode_plus(question, context, add_special_tokens=True, return_tensors="pt")
input_ids = inputs["input_ids"].tolist()[0]
text_tokens = bert_tok.convert_ids_to_tokens(input_ids)
answer_start_scores, answer_end_scores = bert(**inputs)
```

It works by removing the context argument (text_pair argument) but I need it to do question answering with other models and it lead to the same error with pipelines

* Stack trace :

```
IndexError                                Traceback (most recent call last)
<ipython-input-9-73762e6cf69b> in <module>
      2     for utterances in file.readlines():
      3         input_tensor = bert_tok.batch_encode_plus([utterances], pad_to_max_length=True, return_tensors="pt")
----> 4         last_hidden, pool = bert(input_tensor["input_ids"], input_tensor["attention_mask"])
      5
      6

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548                     functools.update_wrapper(wrapper, hook)
    549                     grad_fn.register_hook(wrapper)
--> 550         return result
    551
    552     def __setstate__(self, state):

~/.local/lib/python3.8/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask)
    780             head_mask = [None] * self.config.num_hidden_layers
    781
--> 782         embedding_output = self.embeddings(
    783             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    784         )

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548                     functools.update_wrapper(wrapper, hook)
    549                     grad_fn.register_hook(wrapper)
--> 550         return result
    551
    552     def __setstate__(self, state):

~/.local/lib/python3.8/site-packages/transformers/modeling_roberta.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
     62                 position_ids = self.create_position_ids_from_inputs_embeds(inputs_embeds)
     63
---> 64         return super().forward(
     65             input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
     66         )

~/.local/lib/python3.8/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    172         if inputs_embeds is None:
    173             inputs_embeds = self.word_embeddings(input_ids)
--> 174         position_embeddings = self.position_embeddings(position_ids)
    175         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    176

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548                     functools.update_wrapper(wrapper, hook)
    549                     grad_fn.register_hook(wrapper)
--> 550         return result
    551
    552     def __setstate__(self, state):

~/.local/lib/python3.8/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    110
    111     def forward(self, input):
--> 112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
    114             self.norm_type, self.scale_grad_by_freq, self.sparse)

~/.local/lib/python3.8/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1722     if dim == 3:
   1723         div = pad(div, (0, 0, size // 2, (size - 1) // 2))
-> 1724         div = avg_pool2d(div, (size, 1), stride=1).squeeze(1)
   1725     else:
   1726         sizes = input.size()

IndexError: index out of range in self
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Run inference without any error

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

`transformers` version: 2.8.0
- Platform: Linux-5.6.10-arch1-1-x86_64-with-glibc2.2.5
- Python version: 3.8.2
- PyTorch version (GPU?): 1.4.0 (True)  (Same with 1.5)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


[Marian] documentation and AutoModel support

- Adds integration tests for en-fr, fr-en.
- Easier bulk conversion
- remove unused pretrained_model_archive_map constant.
- boilerplate to make AutoModelWithLMHead, AutoTokenizer, AutoConfig work.

### Metrics:

For [fr-en test set](https://object.pouta.csc.fi/OPUS-MT-models/fr-en/opus-2020-02-26.test.txt):
- BLEU score from posted translations: 57.4979
- BLEU score from `MarianMTModel.from_pretrained('Helsinki-NLP/opus-mt-fr-en')`: 57.4817
- no perf change for fp16
- can fit batch_size=512 on 16GB card in fp16
- speed: 89s for 5k examples = 56 examples/second


How to pre-train BART model

Ex: LM (Finetuning)

Related to language modeling fine-tuning

Ex: LM (Pretraining)

Related to language modeling pre-training

How to pre-train BART model in an unsupervised manner. any example?


Config File

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi everybody, is there a way to automatically obtain the configuration file for pre-trained TensorFlow T5 network? Specifically the config.json file which I want to use to convert a tf model into pytorch using the script provided by HuggingFace.
Just to be more clear, I want something like that: https://s3.amazonaws.com/models.huggingface.co/bert/t5-small-config.json, with the specifications for my model.

Thanks in advance :)


Fine-tuning T5 in Tensorflow

I wonder whether there is an example/tutorial about how to fine-tune T5 for a particular task (say translation) in Tensorflow. On the doc website it says that `This model is a tf.keras.Model tf.keras.Model sub-class`. However I am not really sure what the output is for the Keras model. It seems like the output (the target sequence, in the case of translation) are considered input for the T5 model as well.

Thank you in advance.


DistilBertForQuestionAnswering returns [UNK]

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):Distilbert

Language I am using the model on (English, Chinese ...):English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
Distilber for question answering is return a sentence of [UNK]

Steps to reproduce the behavior:
Code snippet I am using :
```python
import torch
from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased-distilled-squad')
model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')

torch_device = "cuda" if torch.cuda.is_available() else "cpu"

def answer(question, text):
    input_dict = tokenizer.encode_plus(
        question, text, return_tensors='pt', max_length=512)
    input_ids = input_dict["input_ids"].to('cpu')
    start_scores, end_scores = model(input_ids)

    start = torch.argmax(start_scores)
    end = torch.argmax(end_scores)

    all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
    answer = ''.join(all_tokens[start: end + 1]).replace('▁', ' ').strip()
    answer = answer.replace('[SEP]', '')
    return answer if answer != '[CLS]' and len(answer) != 0 else 'could not find an answer'

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Using the Albert is giving a good results, but DistilBert tokenizer is not working.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.2.1
- Platform: Linux 18.04
- Python version: 3.6.9
- PyTorch version (GPU?): pytorch-pretrained-bert==0.6.2
- Tensorflow version (GPU?):NO
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Error in Calculating Sentence Perplexity for GPT-2 model

Hi,

I am using a following code to calculate the perplexity of sentences on my GPT-2 pretrained model:

```
       tokenizer = GPT2Tokenizer.from_pretrained('gpt-model')
       config = GPT2Config.from_pretrained('gpt-model')
       model = GPT2LMHeadModel.from_pretrained('gpt-model', config=config)
       model.eval()

  def calculatePerplexity(sentence,model,tokenizer):
        input_ids = torch.tensor(tokenizer.encode(sentence)).unsqueeze(0)
        input_ids = input_ids.to('cpu')
        with torch.no_grad():
            outputs = model(input_ids, labels=input_ids)
        loss, logits = outputs[:2]
        return math.exp(loss)
```
For some of the sentences from my testing corpus, I am getting following error:

**Token indices sequence length is longer than the specified maximum sequence length for this model (1140 > 1024). Running this sequence through the model will result in indexing errors**

How can I resolve this error? Kindly advise.


Tpu trainer

This PR aims to bring TPU support to the trainer. It runs on GLUE/MRPC but is yet untested on others.

Left to do:

- [ ] Saving and reloading mid-training
- [ ] Check it runs on a few examples (Language modeling, NER, other GLUE tasks)
- [ ] Write down training speed-ups
- [ ] Write down evaluation speed-ups

The API is not final.


tokenizer.batch_encoder_plus do not return input_len

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

```
 batch_tokens = self.tokenizer.batch_encode_plus(
                [example.text_a for example in examples],
                max_length=max_seq_len,
                pad_to_max_length=True,
            )
            for ex_id, (input_ids, segment_ids, input_mask) in enumerate(zip(batch_tokens["input_ids"],
                                                                             batch_tokens["token_type_ids"],
                                                                             batch_tokens["attention_mask"])):
                input_len = sum(1 for id in input_ids if id != self.tokenizer.pad_token_id)
```

if return input_len, it will be great.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Use finetuned-BART large to do conditional generation

Hi

I am using a slightly old tag of ur repo where BART had run_bart_sum.py. I finetuned bart-large on a custom data set and want to do conditional generation

```
from transformers import BartTokenizer, BartForConditionalGeneration
import torch

model = BartForConditionalGeneration.from_pretrained('bart-large')
tokenizer = BartTokenizer.from_pretrained('bart-large')

ARTICLE_TO_SUMMARIZE = "President Donald Trump's senior adviser and son-in-law, Jared Kushner, praised the administration's response to the coronavirus pandemic as a \"great success story\" on Wednesday -- less than a day after the number of confirmed coronavirus cases in the United States topped 1 million. Kushner painted a rosy picture for \"Fox and Friends\" Wednesday morning, saying that \"the federal government rose to the challenge and this is a great success story and I think that that's really what needs to be told.\""

# model = BartForConditionalGeneration.from_pretrained('./bart_sum/checkpointepoch=2.ckpt')
# tokenizer = BartTokenizer.from_pretrained('./bart_sum/checkpointepoch=2.ckpt')

model = BartForConditionalGeneration.from_pretrained('bart-large')
tokenizer = BartTokenizer.from_pretrained('bart-large')
state = torch.load('./bart_sum/checkpointepoch=2.ckpt',map_location='cpu')
model.load_state_dict(state)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
model.eval()

inputs = tokenizer.batch_encode_plus(
    [ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
summary_ids = model.generate(
    inputs['input_ids'], num_beams=1, max_length=512, early_stopping=True)

print([tokenizer.decode(g, skip_special_tokens=True,
                        clean_up_tokenization_spaces=False)
       for g in summary_ids])

```

I tried both loading the finetuned checkpoint directly as well as loading bart-large and setting state dict

For former it gives me

```
Traceback (most recent call last):
  File "generate.py", line 10, in <module>
    model = BartForConditionalGeneration.from_pretrained('./bart_sum/checkpointepoch=2.ckpt')
  File "/datastor/Softwarez/miniconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 438, in from_pretrained
    **kwargs,
  File "/datastor/Softwarez/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 200, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/datastor/Softwarez/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 252, in get_config_dict
    config_dict = cls._dict_from_json_file(resolved_config_file)
  File "/datastor/Softwarez/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 344, in _dict_from_json_file
    text = reader.read()
  File "/datastor/Softwarez/miniconda3/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
```

For latter
`	Unexpected key(s) in state_dict: "epoch", "global_step", "checkpoint_callback_best", "optimizer_states", "lr_schedulers", "state_dict", "hparams", "hparams_type". `


Camembert-large-fquad model card

model card

Related to pretrained model cards

Description for the model card describing the camembert-large-fquad model.


New model addition: Blender (Facebook chatbot)

# 🌟 New model addition

## Model description
- Dialogue response generation model for multiturn conversations, open-sourced by Facebook (28 April 2020)
- 3 models: 90m, 2.7b and 9.4b parameters
- New state-of-the-art "in terms of engagingness and humanness measurements"
- Blog post: https://parl.ai/projects/blender/
- Paper: https://arxiv.org/abs/2004.13637
- Code: Included in Facebook dialogue framework ParlAI https://github.com/facebookresearch/ParlAI

## Open source status

* [ ] the model implementation is available:
- possibly yes, together with the weights
* [x] the model weights are available:
- Need to install ParlAI (https://github.com/facebookresearch/ParlAI). Model weights are then downloaded when running `python parlai/scripts/safe_interactive.py -t blended_skill_talk -mf zoo:blender/blender_3B/model`. Compare https://github.com/facebookresearch/ParlAI/tree/master/parlai/zoo/blender
* [x] who are the authors: (mention them, if possible by @gh-username)

- Facebook AI Research:
    - Stephen Roller, Emily Dinan, Jason Weston
    - and Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott,Kurt Shuster, Eric M. Smith, Y-Lan Boureau,


🚀 An advice about changing variable name from "attention_mask" to "adder"

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation
I noticed that some users are pretty confused when reading source codes about variable `attention_mask`
like:
[What is the meaning of Attention Mask #205](https://github.com/huggingface/transformers/issues/205)
[Clarifying attention mask #542](https://github.com/huggingface/transformers/issues/542)
And I refer to the origional BERT repository - [google-research/bert](https://github.com/google-research/bert). Compared to the origin, I find in this repo sometimes the concepts of `attention_mask` and `adder` are mixed.

refering original BERT: [./modeling.py#L707](https://github.com/google-research/bert/blob/master/modeling.py#L707)
```python
attention_mask = tf.expand_dims(attention_mask, axis=[1])
adder = (1.0 - tf.cast(attention_mask, tf.float32)) * -10000.0
attention_scores += adder
```

But in this repo: take [src/transformers/modeling_tf_openai.py#L282](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_openai.py#L282) as an example:
```python
attention_mask = attention_mask[:, tf.newaxis, tf.newaxis, :]
attention_mask = tf.cast(attention_mask, tf.float32)
attention_mask = (1.0 - attention_mask) * -10000.0
```
and inside the method `TFAttention._attn()` [src/transformers/modeling_tf_openai.py#L112](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_openai.py#L112):
```python
if attention_mask is not None:
  # Apply the attention mask
  w = w + attention_mask
```
<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
may be changing its name is way better?
like:
```python
attention_mask = attention_mask[:, tf.newaxis, tf.newaxis, :]
attention_mask = tf.cast(attention_mask, tf.float32)
adder = (1.0 - attention_mask) * -10000.0
```
and then:
```python
if adder is not None:
  # Apply the attention mask
  attention_score = w + adder
```
<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


How to make run_language_modeling.py work for transformer-xl?

The code in [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) does Language modeling for any pre-trained model. However, for transformer-xl, we need additional arguments to the forward function like `mems` and even [run_transfo_xl.py](https://github.com/huggingface/transformers/blob/master/examples/contrib/run_transfo_xl.py) performs `model.reset_length()` operation using `mem_len` and `tgt_len` as arguments.

I am not sure how these functions are performed in this generic code for runningLM for any pre-trained model. Any suggestions will be helpful.


Cannot set max_position_embeddings to any desired value in T5Config

# 🐛 Bug

## Information

The model I am using (T5):

Language I am using the model on (English):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

```
config = T5Config(max_position_embeddings=1024)
```

## To reproduce

Steps to reproduce the behaviour:

1.Initiate T5Config
2. set max_position_embeddings any desired value like 1024, 2048 (as mentioned in docs)

This works perfectly fine with BertConfig

This is an error:
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
Can't set max_position_embeddings with value 1024 for T5Config {
  "_num_labels": 2,
  "architectures": null,
  "bad_words_ids": null,
  "bos_token_id": null,
  "decoder_start_token_id": null,
  "do_sample": false,
  "early_stopping": false,
  "eos_token_id": 1,
  "finetuning_task": null,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "is_decoder": false,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "length_penalty": 1.0,
  "max_length": 20,
  "min_length": 0,
  "model_type": "t5",
  "no_repeat_ngram_size": 0,
  "num_beams": 1,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "prefix": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "task_specific_params": null,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false
}

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-5-44823d82a38e> in <module>()
----> 1 config = T5Config(max_position_embeddings=1024)

2 frames
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in __init__(self, **kwargs)
    106         for key, value in kwargs.items():
    107             try:
--> 108                 setattr(self, key, value)
    109             except AttributeError as err:
    110                 logger.error("Can't set {} with value {} for {}".format(key, value, self))

AttributeError: can't set attribute
```
After further investigating in docs here is what I found (If I am wrong, then please correct me):
```
self.vocab_size = vocab_size
self.n_positions = n_positions
self.d_model = d_model
self.d_kv = d_kv
self.d_ff = d_ff
self.num_layers = num_layers
self.num_heads = num_heads
self.relative_attention_num_buckets = relative_attention_num_buckets
self.dropout_rate = dropout_rate
self.layer_norm_epsilon = layer_norm_epsilon
self.initializer_factor = initializer_factor

@property
def max_position_embeddings(self):
    return self.n_positions
```
Looking at code snippets I think we have to directly use n_positions instead of max_position_embeddings. I have tried this approach & **it worked**.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): 2.2.0-rc3 (True)
- Using GPU in script?: True


[Roberta] fix hard wired pad token id

This might break backward compatibility for users that have uploaded a roberta model and use a different pad token than 1 in the configs.


Error: `cannot import name 'TFBertForMaskedLM'`

When I try the following import:

```python
from transformers import TFBertForMaskedLM
```

I get the following error:

```
Traceback (most recent call last):
  File "/Users/danielk/ideaProjects/farsi-language-models/src/try_bert_lm.py", line 3, in <module>
    from transformers import TFBertForMaskedLM
ImportError: cannot import name 'TFBertForMaskedLM' from 'transformers' (/usr/local/lib/python3.7/site-packages/transformers/__init__.py)
```

My versions are:
```
torch                              1.4.0
transformers                       2.8.0
```


Cannot loading SpanBERT pre-trained model

I have some questions regarding SpanBert loading using the transformers package.

I downloaded the pre-trained file from [SpanBert](https://github.com/facebookresearch/SpanBERT) GitHub Repo and ```vocab.txt``` from Bert. Here is the code I used for loading:

```python
model = BertModel.from_pretrained(config_file=config_file,
                                  pretrained_model_name_or_path=model_file,
                                  vocab_file=vocab_file)
model.to("cuda")
```

where

- ```config_file``` -> ```config.json```
- ```model_file``` -> ```pytorch_model.bin```
- ```vocab_file``` -> ```vocab.txt```

But I got the ```UnicodeDecoderError``` with the above code saying that ```'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte```

I also tried loading SpanBert with the method mentioned [here](https://huggingface.co/SpanBERT/spanbert-large-cased). But it returned ```OSError: file SpanBERT/spanbert-base-cased not found```.

Do you have any suggestions on loading the pre-trained model correctly? Any suggestions are much appreciated. Thanks!


Why does ALBERT use einsum in PyTorch implementation while in TF one it does not?

# ❓ Questions & Help
I wanted to learn internals of ALBERT model from your implementation (which is BTW really clean in comparison to the original one - good job!), but I've stumbled upon weird looking part in the `AlbertAttention`: https://github.com/huggingface/transformers/blob/6af3306a1da0322f58861b1fbb62ce5223d97b8a/src/transformers/modeling_albert.py#L258

Why does PyTorch version use `einsum`-based notation while calculating hidden state (with manual usage of `dense` layer's weights), while the TensorFlow version just reshapes the `context_layer` and does standard "forward" on dense layer?

https://github.com/huggingface/transformers/blob/6af3306a1da0322f58861b1fbb62ce5223d97b8a/src/transformers/modeling_tf_albert.py#L296

I would really like to know the explanation of this implementation - @LysandreJik cloud you shed some light here?


# 🐛 jplu/tf-xlm-roberta-large showing random performance on tpu model training

## i have observed several times that jplu/tf-xlm-roberta-large showing random performance on several experiments,for example if i train a model and get validation accuracy 61% then if i run the model again (without changing anythin or doing 0% change in code) then second i get over 96% accuracy and sometimes i observed that validation accuracy stay same for all epoch and again this same code will get me totally different train statistics next time,why is this happening?

Model I am using jplu/tf-xlm-roberta-large

Language I am using the model on (English, Chinese ,turkish,spanish etc):

The problem arises when using:
jplu/tf-xlm-roberta-large for finetuning

The tasks I am working on is:
kaggle ongoing competition : jigsaw multilingual toxic comment classification where train set contails all english comments and 0/1(toxic/non toxic) label and validation+test data contains non English comments

## To reproduce

Steps to reproduce the behavior:

1. just simply fork and run this kernel : https://www.kaggle.com/mobassir/understanding-cross-lingual-models and you will never get same result and probably will get bit more worst result

## Expected behavior

same model training several times should always produce close result if not exact same, the randomness is high,,sometimes model with 91% validation accuracy can get us 61% validation accuracy on next run and sometimes validation accuracy of that same model get frozen and seems like the model is not learning anything (i repeat the model name : it's tf xlm roberta large)

## Environment info
kaggle kernels/notebooks using tpu


problem about change from pytorch-pretrained-bert to transformers

I am doing the sentiment analysis with pytorch-pretrained-bert, it work correct, but when I change to transformers, the accuracy on the dev data is poor (but the accuracy with pytorch-pretrained-bert is quite higher). Besides, I notice the Migrating from pytorch-pretrained-bert to transformers module.
the input is:
[CLS] sentence [SEP] aspect [SEP]
and just using BertForSequenceClassification (also try add an classifier after BertModel), but all fail on transformers.
Can Sombody figure out what's wrong with it? Thanks!


Create README.md

Related to pretrained model cards


Make transformers-cli cross-platform

Using `scripts` is a useful option in `setup.py` particularly when you want to get access to non-python scripts. However, in this case we want to have an entry point into some of our own Python scripts. To do this in a concise, cross-platfom way, we can use `entry_points.console_scripts`. This change is necessary to provide the CLI on different platforms (particularly also including Windows), which "scripts" does not ensure.

Usage remains the same, but the "transformers-cli" script has to be moved (be part of the library) and renamed (underscore + extension). So even though the script itself has been renamed to `transformers_cli.py` and moved to `src/commands`, its usage is still the same: `transformers-cli <args>`.

Tested on Ubuntu and Windows.


GPT-2 past behaves incorrectly when attention_mask is used

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import GPT2LMHeadModel
import torch

model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

# w/ `attention_mask` w/ `past`
input_ids      = torch.tensor([[0, 0, 1, 2]])
position_ids   = torch.tensor([[0, 0, 1, 2]])
attention_mask = torch.tensor([[0, 1, 1, 1]])
output = model(input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask)
input_ids      = torch.tensor([[3]])
position_ids   = torch.tensor([[3]])
attention_mask = torch.tensor([[1]])
output = model(input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask, past=output[1])
print(output[0][0, -1, :].detach().numpy())

# w/ `attention_mask` w/o `past`
input_ids      = torch.tensor([[0, 0, 1, 2, 3]])
position_ids   = torch.tensor([[0, 0, 1, 2, 3]])
attention_mask = torch.tensor([[0, 1, 1, 1, 1]])
output = model(input_ids=input_ids, position_ids=position_ids, attention_mask=attention_mask)
print(output[0][0, -1, :].detach().numpy())

# w/o `attention_mask` w/ `past`
input_ids      = torch.tensor([[0, 1, 2]])
position_ids   = torch.tensor([[0, 1, 2]])
output = model(input_ids=input_ids, position_ids=position_ids)
input_ids      = torch.tensor([[3]])
position_ids   = torch.tensor([[3]])
output = model(input_ids=input_ids, position_ids=position_ids, past=output[1])
print(output[0][0, -1, :].detach().numpy())

# w/o `attention_mask` w/o `past`
input_ids      = torch.tensor([[0, 1, 2, 3]])
position_ids   = torch.tensor([[0, 1, 2, 3]])
output = model(input_ids=input_ids, position_ids=position_ids)
print(output[0][0, -1, :].detach().numpy())
```

## Expected behavior

All printed outputs should be equal. But this is what I see:

```
[-63.085373 -62.34121  -61.08647  ... -76.70221  -73.235085 -65.88774 ]
[-67.13892  -66.132126 -64.81706  ... -80.93089  -77.780525 -70.23946 ]
[-67.1389   -66.13211  -64.81704  ... -80.930855 -77.780495 -70.239426]
[-67.13892  -66.132126 -64.81706  ... -80.93089  -77.780525 -70.23946 ]
```
Specifically, it seems like `past` does not work correctly when used with `attention_mask`.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-5.4.19-100.fc30.x86_64-x86_64-with-fedora-30-Thirty
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Parse in tensorflow strings as well as normal strings

# 🚀 Feature request
So this is something I came across while trying to get a (fast) streaming dataset via the tensorflow data API for TPUs. This is a crosspost of [this SO question](https://stackoverflow.com/questions/61555097/mapping-text-data-through-huggingface-tokenizer/).

I have my encode function that looks like this:
```python
from transformers import BertTokenizer, BertModel

MODEL = 'bert-base-multilingual-uncased'
tokenizer = BertTokenizer.from_pretrained(MODEL)

def encode(texts, tokenizer=tokenizer, maxlen=10):
#     import pdb; pdb.set_trace()
    inputs = tokenizer.encode_plus(
        texts,
        return_tensors='tf',
        return_attention_masks=True,
        return_token_type_ids=True,
        pad_to_max_length=True,
        max_length=maxlen
    )

    return inputs['input_ids'], inputs["token_type_ids"], inputs["attention_mask"]
```
I want to get my data **encoded on the fly** by doing this:
```python
x_train = (tf.data.Dataset.from_tensor_slices(df_train.comment_text.astype(str).values)
           .map(encode))
```
However, this chucks the error:
```
ValueError: Input is not valid. Should be a string, a list/tuple of strings or a list/tuple of integers.
```

Now from my understanding when I set a breakpoint inside `encode` it was because I was sending a non-numpy array. How do I get huggingface transformers to play nice with tensorflow strings as inputs?

If you need a dummy dataframe here it is:
```
df_train = pd.DataFrame({'comment_text': ['Today was a good day']*5})
```
## What I tried
So I tried to use `from_generator` so that I can parse in the strings to the `encode_plus` function. However, this does not work with TPUs.
```python
AUTO = tf.data.experimental.AUTOTUNE

def get_gen(df):
    def gen():
        for i in range(len(df)):
            yield encode(df.loc[i, 'comment_text']) , df.loc[i, 'toxic']
    return gen

shapes = ((tf.TensorShape([maxlen]), tf.TensorShape([maxlen]), tf.TensorShape([maxlen])), tf.TensorShape([]))

train_dataset = tf.data.Dataset.from_generator(
    get_gen(df_train),
    ((tf.int32, tf.int32, tf.int32), tf.int32),
    shapes
)
train_dataset = train_dataset.batch(BATCH_SIZE).prefetch(AUTO)
```

My final solution was to encode the all the strings using `batch_encode_plus` but to write those tensors out to disk. This currently takes ~1 hour, but seems like an inefficient way of prepping data for training.
## Version Info:
`transformers.__version__, tf.__version__` => `('2.7.0', '2.1.0')`


Add decoder specific error message for T5Stack.forward

#### This PR adds a decoder specific error message in `T5Stack.forward` in the case that `self.is_decoder == True`

This was requested in #4126 and suggested by @patrickvonplaten in #3626 .

Confirmed that this works as expected in this colab [notebook](https://colab.research.google.com/drive/1j1mdtOylXZClH-ikZthDiOxXg4DOis_4?usp=sharing).

Missing `decoder_input_ids` example:

```
## Setup t5-small and tokenized inputs
from transformers import T5Tokenizer, T5Model
tokenizer = T5Tokenizer.from_pretrained('t5-small')
model = T5Model.from_pretrained('t5-small')
input_ids = tokenizer.encode("Hello, my dog is cute", return_tensors="pt")  # Batch size 1

# Test missing `decoder_input_ids`
model(input_ids=input_ids)[0]
# ValueError: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

Missing `input_ids` example:

```
model(decoder_input_ids=input_ids)[0]
# ValueError: You have to specify either input_ids or inputs_embeds
```


Model output should be dict

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->
I think model output should be a dict. Since dict are more clearly than tuple, we do not know what output[1] means, but we do know output["pooled_ouptut"] means.

https://github.com/huggingface/transformers#models-always-output-tuples

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


No decoder specific error message for T5Stack.forward

When I run a forward pass with `T5Model` with a missing value for `decoder_input_ids`, it still gives me the same error message compared to when there is a missing value for `input_ids` (below).

`ValueError: You have to specify either input_ids or inputs_embeds`

I've confirmed with @patrickvonplaten that it makes sense to have a specific error message for the case when `self.is_decoder == True` in issue #3626.


Is it possible to have high perplexity of some individual sentences, as compared to overall testing corpus perplexity?

Hi,

I have calculated an overall perplexity of my testing corpus after finetuning GPT-2 pretrained model
Similarly, I have also calculated perplexity of individual sentences, after extracting it from my testing corpus.

**I notice that some individual sentence perplexity is even higher than the overall perplexity.**

What will be the reason for it? Will it be possible? Have I done something wrong in my calculation?

Kindly share your opinion.


convert pytorch_model.pt [pretrained Bert model] to pytorch_model.onnx (ONNX)

i use this to convert pytorch_model.pt to pytorch_model.onnx

```
from torch.autograd import Variable
import torch.onnx
import torchvision
import torch

dummy_input = Variable(torch.randn(1, 3, 256, 256))
model = torch.load('./pytorch_model.pt')
torch.onnx.export(model, dummy_input, "pytorch_model.onnx")
```

ERROR :- `The expanded size of the tensor (256) must match the existing size (3) at non-singleton dimension 3. Target sizes: [1, 3, 256, 256]. Tensor sizes: [1, 3]`

Are my `dummy_inputs` incorrect ?


Text corrections


# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):GPT2

Language I am using the model on (English, Chinese ...):English

The problem arises when using:

my own modified scripts: (give details below)
python examples/run_language_modeling.py \
    --train_data_file=temp_gpt2/gpt2.train \
    --output_dir=checkpoints/gpt2 \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --eval_data_file=temp_gpt2/test.txt \
    --line_by_line \
    --do_train \
    --do_eval \
    --evaluate_during_training \
    --per_gpu_train_batch_size=20 \
    --per_gpu_eval_batch_size=20 \
    --gradient_accumulation_steps=1 \
    --learning_rate=8e-5 \
    --weight_decay=0.075  \
    --adam_epsilon=1e-8 \
    --warmup_steps=500 \
    --max_grad_norm=5.0 \
    --num_train_epochs=20 \
    --logging_steps=500 \
    --save_steps=500

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: language modeling
* [ ] my own task or dataset: (give details below) Conll2014 GEC

## To reproduce

Steps to reproduce the behavior:

1. run the script
I get the following error:
```
bash train_gpt2.sh
05/02/2020 10:14:25 - INFO - transformers.training_args -   PyTorch: setting up devices
05/02/2020 10:14:25 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: False
05/02/2020 10:14:25 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='checkpoints/gpt2', overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=True, per_gpu_train_batch_size=20, per_gpu_eval_batch_size=20, gradient_accumulation_steps=1, learning_rate=8e-05, weight_decay=0.075, adam_epsilon=1e-08, max_grad_norm=5.0, num_train_epochs=20.0, max_steps=-1, warmup_steps=500, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1)
05/02/2020 10:14:35 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /home/zixi/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
05/02/2020 10:14:35 - INFO - transformers.configuration_utils -   Model config GPT2Config {
  "activation_function": "gelu_new",
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "vocab_size": 50257
}

05/02/2020 10:14:39 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /home/zixi/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.4c1d7fc2ac6ddabeaf0c8bec2ffc7dc112f668f5871a06efcff113d2797ec7d5
05/02/2020 10:14:39 - INFO - transformers.configuration_utils -   Model config GPT2Config {
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "vocab_size": 50257
}

05/02/2020 10:14:42 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /home/zixi/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
05/02/2020 10:14:42 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /home/zixi/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
05/02/2020 10:14:43 - INFO - transformers.modeling_utils -   loading weights file https://cdn.huggingface.co/gpt2-pytorch_model.bin from cache at /home/zixi/.cache/torch/transformers/d71fd633e58263bd5e91dd3bde9f658bafd81e11ece622be6a3c2e4d42d8fd89.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
05/02/2020 10:14:47 - INFO - transformers.modeling_utils -   Weights of GPT2LMHeadModel not initialized from pretrained model: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
05/02/2020 10:14:47 - INFO - transformers.data.datasets.language_modeling -   Creating features from dataset file at temp_gpt2/gpt2_train.txt
05/02/2020 10:16:41 - INFO - transformers.data.datasets.language_modeling -   Creating features from dataset file at temp_gpt2/gpt2_test.txt
05/02/2020 10:16:44 - INFO - transformers.trainer -   ***** Running training *****
05/02/2020 10:16:44 - INFO - transformers.trainer -     Num examples = 1130686
05/02/2020 10:16:44 - INFO - transformers.trainer -     Num Epochs = 20
05/02/2020 10:16:44 - INFO - transformers.trainer -     Instantaneous batch size per GPU = 20
05/02/2020 10:16:44 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 40
05/02/2020 10:16:44 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
05/02/2020 10:16:44 - INFO - transformers.trainer -     Total optimization steps = 565360
Epoch:   0%|                                                                                | 0/20 [00:00<?, ?it/sTraceback (most recent call last):                                                                                | 0/28268 [00:00<?, ?it/s]
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/transformers/trainer.py", line 307, in train
    for step, inputs in enumerate(epoch_iterator):
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/tqdm/std.py", line 1107, in __iter__
    for obj in iterable:
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/transformers/data/data_collator.py", line 91, in collate_batch
    batch = self._tensorize_batch(examples)
  File "/home/zixi/anaconda3/envs/ROC/lib/python3.7/site-packages/transformers/data/data_collator.py", line 106, in _tensorize_batch
    "You are attempting to pad samples but the tokenizer you are using"
ValueError: You are attempting to pad samples but the tokenizer you are using (GPT2Tokenizer) does not have one.
Epoch:   0%|                                                                                | 0/20 [00:00<?, ?it/s]
Iteration:   0%|
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:master
- Platform:Ubuntu 18.04LTS
- Python version:3.7
- PyTorch version (GPU?):1.5
- Tensorflow version (GPU?):--
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:No


`eos_token_id` breaks the `T5ForConditionalGeneration`

Ex: Generation

Natural Language Generation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `T5ForConditionalGeneration`

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

```python
from transformers import T5Tokenizer, T5Config, T5ForConditionalGeneration

T5_PATH = 't5-base' # "t5-small", "t5-base", "t5-large", "t5-3b", "t5-11b"

DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu') # My envirnment uses CPU

t5_tokenizer = T5Tokenizer.from_pretrained(T5_PATH)
t5_config = T5Config.from_pretrained(T5_PATH)
t5_mlm = T5ForConditionalGeneration.from_pretrained(T5_PATH, config=t5_config).to(DEVICE)

text = 'India is a <extra_id_0> of the world. </s>'
encoded = t5_tokenizer.encode_plus(text, add_special_tokens=True, return_tensors='pt')
input_ids = encoded['input_ids'].to(DEVICE)

# I want to set <extra_id_1> as end of a sequence.
eos_token_id = t5_tokenizer.encode('<extra_id_1>')[0]

outputs = t5_mlm.generate(input_ids=input_ids,
                          num_beams=200, num_return_sequences=20,
                          eos_token_id=eos_token_id,  # <-- This is causes the error. Just removing it works fine.
                          max_length=5)

results = [t5_tokenizer.decode(output, skip_special_tokens=False, clean_up_tokenization_spaces=False) for output in outputs]
print(results)
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
* [x] Just playing with the `T5` model

## To reproduce

Steps to reproduce the behavior:

1. Start a new Google Colab notebook,
2. Install transformers using `pip install git+https://github.com/huggingface/transformers`,
3. Restart the runtime, and
4. Run the above snippet.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
AssertionError                            Traceback (most recent call last)

<ipython-input-86-aae5bd45e59c> in <module>()
      2                           num_beams=200, num_return_sequences=20, max_length=5,
      3                         #   repetition_penalty=1.2,
----> 4                           eos_token_id=eos_token_id,
      5                           )
      6 # print(outputs)

2 frames

/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id)
    993                 vocab_size=vocab_size,
    994                 encoder_outputs=encoder_outputs,
--> 995                 attention_mask=attention_mask,
    996             )
    997         else:

/usr/local/lib/python3.6/dist-packages/transformers/modeling_utils.py in _generate_beam_search(self, input_ids, cur_len, max_length, min_length, do_sample, early_stopping, temperature, top_k, top_p, repetition_penalty, no_repeat_ngram_size, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, decoder_start_token_id, batch_size, num_return_sequences, length_penalty, num_beams, vocab_size, encoder_outputs, attention_mask)
   1359                     next_scores[batch_idx, :num_beams] == beam_scores.view(batch_size, num_beams)[batch_idx]
   1360                 ), "If batch_idx is not done, final next scores: {} have to equal to accumulated beam_scores: {}".format(
-> 1361                     next_scores[:, :num_beams][batch_idx], beam_scores.view(batch_size, num_beams)[batch_idx],
   1362                 )
   1363

AssertionError: If batch_idx is not done, final next scores: tensor([-3.5015, -4.4583, -4.5677, -4.5808, -4.7325, -4.8779, -5.1127, -5.2106,
        -5.2147, -5.3292, -5.3575, -5.5227, -5.6846, -5.7290, -5.8227, -5.8522,
        -5.8537, -5.9930, -6.0119, -6.0440, -6.0469, -6.0496, -6.0739, -6.0959,
        -6.1007, -6.1125, -6.1519, -6.2492, -6.2850, -6.3033, -6.3035, -6.3677,
        -6.3798, -6.4438, -6.4508, -6.4511, -6.4993, -6.5149, -6.5269, -6.5330,
        -6.5335, -6.5662, -6.6071, -6.6222, -6.6331, -6.6437, -6.6454, -6.6660,
        -6.7163, -6.7690, -6.7841, -6.8139, -6.8147, -6.8157, -6.8297, -6.8297,
        -6.8326, -6.8448, -6.8493, -6.8763, -6.8866, -6.8871, -6.9026, -6.9046,
        -6.9078, -6.9475, -6.9544, -6.9654, -6.9672, -6.9735, -6.9749, -6.9795,
        -7.0020, -7.0024, -7.0124, -7.0310, -7.0837, -7.1405, -7.1407, -7.1420,
        -7.1652, -7.1675, -7.1788, -7.1800, -7.1875, -7.2146, -7.2148, -7.2294,
        -7.2538, -7.2765, -7.2775, -7.2859, -7.3110, -7.3120, -7.3189, -7.3403,
        -7.3438, -7.3479, -7.3535, -7.3592, -7.3733, -7.3740, -7.3842, -7.3848,
        -7.3882, -7.3889, -7.4192, -7.4192, -7.4211, -7.4657, -7.4662, -7.4774,
        -7.4776, -7.4862, -7.4999, -7.5150, -7.5156, -7.5187, -7.5214, -7.5253,
        -7.5365, -7.5376, -7.5547, -7.5566, -7.5815, -7.5922, -7.5940, -7.5960,
        -7.5996, -7.6115, -7.6128, -7.6406, -7.6485, -7.6619, -7.6744, -7.6752,
        -7.6804, -7.6875, -7.6928, -7.6979, -7.7064, -7.7172, -7.7229, -7.7429,
        -7.7431, -7.7482, -7.7510, -7.7576, -7.7619, -7.7641, -7.7655, -7.7721,
        -7.7787, -7.7792, -7.7842, -7.7859, -7.8005, -7.8083, -7.8087, -7.8128,
        -7.8145, -7.8184, -7.8185, -7.8197, -7.8228, -7.8275, -7.8381, -7.8478,
        -7.8572, -7.8598, -7.8677, -7.8712, -7.8755, -7.8820, -7.8867, -7.9135,
        -7.9308, -7.9485, -7.9613, -7.9629, -7.9649, -7.9706, -7.9714, -7.9739,
        -7.9740, -7.9757, -7.9779, -7.9809, -7.9858, -7.9964, -7.9992, -8.0047,
        -8.0086, -8.0105, -8.0255, -8.0500, -8.0550, -8.0604, -8.0811, -8.0858]) have to equal to accumulated beam_scores: tensor([-4.7325, -5.1127, -5.2106, -5.3292, -5.8227, -5.8522, -6.0739, -6.0959,
        -6.1007, -6.1125, -6.2492, -6.3033, -6.3035, -6.3798, -6.4438, -6.4508,
        -6.4511, -6.4993, -6.5269, -6.5330, -6.5662, -6.6331, -6.6660, -6.7163,
        -6.8139, -6.8157, -6.8297, -6.8297, -6.8493, -6.8763, -6.8871, -6.9026,
        -6.9078, -6.9475, -6.9735, -6.9749, -7.0020, -7.0024, -7.0837, -7.1405,
        -7.1420, -7.1652, -7.1800, -7.1875, -7.2146, -7.2294, -7.2538, -7.2765,
        -7.2775, -7.2859, -7.3189, -7.3403, -7.3438, -7.3535, -7.3592, -7.3733,
        -7.4192, -7.4192, -7.4657, -7.4662, -7.4774, -7.4999, -7.5150, -7.5156,
        -7.5187, -7.5253, -7.5365, -7.5547, -7.5566, -7.5815, -7.5922, -7.5940,
        -7.5996, -7.6115, -7.6128, -7.6406, -7.6619, -7.6744, -7.6752, -7.6804,
        -7.6875, -7.7172, -7.7229, -7.7429, -7.7431, -7.7482, -7.7510, -7.7619,
        -7.7641, -7.7655, -7.7787, -7.7792, -7.7842, -7.8083, -7.8087, -7.8145,
        -7.8184, -7.8185, -7.8197, -7.8228, -7.8275, -7.8478, -7.8572, -7.8598,
        -7.8677, -7.8712, -7.8755, -7.8820, -7.8867, -7.9613, -7.9629, -7.9706,
        -7.9714, -7.9739, -7.9740, -7.9757, -7.9779, -7.9809, -7.9858, -7.9964,
        -7.9992, -8.0047, -8.0086, -8.0255, -8.0500, -8.0550, -8.0604, -8.0811,
        -8.0858, -8.0862, -8.0880, -8.1197, -8.1198, -8.1261, -8.1331, -8.1337,
        -8.1364, -8.1509, -8.1633, -8.1681, -8.1759, -8.1843, -8.2007, -8.2019,
        -8.2094, -8.2223, -8.2236, -8.2504, -8.2580, -8.2653, -8.2748, -8.2778,
        -8.2782, -8.2860, -8.2931, -8.2932, -8.2963, -8.3047, -8.3111, -8.3161,
        -8.3215, -8.3255, -8.3393, -8.3439, -8.3567, -8.3608, -8.3706, -8.3810,
        -8.3858, -8.4023, -8.4076, -8.4111, -8.4153, -8.4198, -8.4335, -8.4365,
        -8.4390, -8.4397, -8.4480, -8.4511, -8.4532, -8.4594, -8.4663, -8.4681,
        -8.4686, -8.4720, -8.4726, -8.4778, -8.4781, -8.4786, -8.4885, -8.5027,
        -8.5173, -8.5181, -8.5226, -8.5298, -8.5308, -8.5341, -8.5463, -8.5466])
```

## Expected behavior

If I remove the `eos_token_id` argument from `t5_mlm.generate(...)` (refer the given code snippet), it returns the following (expected) output:
```
['<extra_id_0> cornerstone<extra_id_1>',
 '<extra_id_0> part<extra_id_1> developing',
 '<extra_id_0> huge part<extra_id_1>',
 '<extra_id_0> big part<extra_id_1>',
 '<extra_id_0> beautiful part<extra_id_1>',
 '<extra_id_0> very important part',
 '<extra_id_0> part<extra_id_1> larger',
 '<extra_id_0> unique part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> developed',
 '<extra_id_0> part<extra_id_1> large',
 '<extra_id_0> beautiful country in',
 '<extra_id_0> part of the',
 '<extra_id_0> small part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> great',
 '<extra_id_0> part<extra_id_1> bigger',
 '<extra_id_0> country in the',
 '<extra_id_0> large part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> middle',
 '<extra_id_0> significant part<extra_id_1>',
 '<extra_id_0> part<extra_id_1> vast']
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Google Colab
- Python version: 3.6
- PyTorch version (GPU?): 1.5.0+cu101
- Tensorflow version (GPU?): N/A
- Using GPU in script?:  No
- Using distributed or parallel set-up in script?: No


fixed utils_summarization import path

As per issue #3827 I've removed the dot in the import.


Fix markdown to show the results table properly

model card

Related to pretrained model cards


Update run_pl_ner.py


Update run_pl_glue.py


Updated


Create model card

Related to pretrained model cards

Create Model card for distilroberta-base-finetuned-sentiment


model card for surajp/albert-base-sanskrit

Related to pretrained model cards

Added Model card for `surajp/albert-base-sanskrit`


[Reformer] Move model card to google model


Albert large QA model pretrained from baidu webqa and baidu dureader datasets.

Related to pretrained model cards


BERT as encoder and a transformer as a decoder.

Usage

General questions about the library

Hi, there
Is there probability to build a model of the BERT as the encoder and the transformer as the decoder?

Thanks.


NER: parse args from .args file or JSON

Hi,

thanks to @julien-c , args parsing from `.args` or json-based configuration files were introduced in #3934 into the internal argparser class.

This PR adds support for it in the `run_ner.py` script.

It also extends the NER documentation and shows how to use a json-based configuration with `run_ner.py`.


Fix #2941

Reshaped score array to avoid `numpy` ValueError. This should allow the sentiment analyzer pipeline to run.


Feature/torchserve interface [WIP]

Work in progress PR to add a CLI interface for easy packaging of transformers models
for serving in [pytorch/serve](https://github.com/pytorch/serve).

Intended usage example:
`transformers-cli torchserve --checkpoint="distilbert-base-uncased-finetuned-sst-2-english" --tokenizer="distilbert-base-uncased" --model-name="distilbert" --task="sentiment-analysis"`

The call above produces a MAR (model archive) file that can be served directly by the `torchserve` binary.


Fix `RobertaClassificationHead` style consistency.

There's a slight inconsistency in `RobertaClassificationHead` in that it takes in the whole sequence output from the `RobertaModel`, and extracts the pooled output inside its own forward method, seen [here](https://github.com/huggingface/transformers/blob/d713cfc5ebfb1ed83de1fce55dd7279f9db30672/src/transformers/modeling_roberta.py#L573).

This is different from other models, where the pooled output is computed beforehand and directly passed on to the classifier. E.g. in [`BertForSequenceClassification`](https://github.com/huggingface/transformers/blob/d713cfc5ebfb1ed83de1fce55dd7279f9db30672/src/transformers/modeling_bert.py#L1147), [`DistilBertForSequenceClassification`](https://github.com/huggingface/transformers/blob/d713cfc5ebfb1ed83de1fce55dd7279f9db30672/src/transformers/modeling_distilbert.py#L614), [`BartForSequenceClassification`](https://github.com/huggingface/transformers/blob/d713cfc5ebfb1ed83de1fce55dd7279f9db30672/src/transformers/modeling_bart.py#L1097), etc.


FIXME(Actually test multi input pipelines)


model from path 16-bits training:True but float16 false

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Fix pytorch lighting examples

It throws the exception 'Trainer' object has no attribute 'avg_loss' because since version 0.7.2 they removed the avg_loss field from the Trainer class.

Also `get_tqdm_dict` is deprecated since 0.7.3.

See https://github.com/huggingface/transformers/pull/2890#issuecomment-613066707