T5 fp16 forward yields nan

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

I use pytorch-lightning to manage fp16. This is the minimal example that reproduces the result.

```python
from transformers import T5Model, T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("t5-base")
model = T5Model.from_pretrained("t5-base").cuda().half()
text = "hello world!"
inputs = tokenizer.encode(text, return_tensors="pt").cuda()
out = model(input_ids=inputs, decoder_input_ids=inputs)
print(out[0][:, :, :10])
```
output:
```
tensor([[[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
         [nan, nan, nan, nan, nan, nan, nan, nan, nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Get non-nan values.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.0
- Platform: Linux-4.15.0-88-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Reformer enwik8 - Model card

model card

Related to pretrained model cards

Create a model card with char level tokenizer for reformer enwik8.


Default code snippet for shared model doesn't work because of different model and tokenizer types

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2 with AlbertTokenizer

Language I am using the model on (English, Chinese ...): Ukrainian

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

When I share GPT2 model trained with AlbertTokenizer on huggingface.co the default code snippet doesn't work. The reason is that AutoTokenizer detects tokenizer type by model type.

## To reproduce

Steps to reproduce the behavior:

1. Go to https://huggingface.co/Tereveni-AI/gpt2-124M-uk-fiction
2. Copy code from default snippet `How to use this model directly from the 🤗/transformers library`
3. Try to execute snippet code

```
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("Tereveni-AI/gpt2-124M-uk-fiction")
Downloading: 100%|████████████████████████████████████████████████████████████████████████████| 645/645 [00:00<00:00, 89.4kB/s]
Downloading: 100%|████████████████████████████████████████████████████████████████████████████| 121/121 [00:00<00:00, 18.8kB/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_auto.py", line 200, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 898, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1051, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/Users/oleksandrbushkovskyi/Projects/transformers-ua/venv/lib/python3.7/site-packages/transformers/tokenization_gpt2.py", line 151, in __init__
    with open(vocab_file, encoding="utf-8") as vocab_handle:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

## Expected behavior

I would expect `AutoTokenizer` to detect `spiece.model` and initialize `AlbertTokenizer`.
Even better if `AutoTokenizer` read tokenizer type explicitly defined in `tokenizer_config.json`.

## Environment info
- `transformers` version: 2.9.0
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


[Marian] Fix typo in docstring


[TF 2.2 compat]  use tf.VariableAggregation.ONLY_FIRST_REPLICA

This PR updates the TF gradient accumulator optimization to make it compliant with the new TF 2.2.


[Reformer] Add Enwiki8 Reformer Model - Adapt convert script

This PR adapts the reformer conversion script to the newest trax code and adds `reformer-enwik8` to the docs.


Model card for bert-turkish-question-answering question-answering model

Related to pretrained model cards

First model card for bert-turkish-question-answering question-answering model


Update model-card

Related to pretrained model cards

This PR is to:
- Fix the XTREME link.
- add the number of evaluation documents for each language.
- fix the usage piece of code to specify which framework to use.


Documentation: fix links to NER examples

Hi,

this PR just fixes some links related to the NER examples :)


Could you help with a huggingface/transformers pretrained models download link?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
  I am a natural language processing researcher from the Lenovo Research in Beijing. My colleagues and I really like your huggingface/transfromers framework and having been using it. I am currently writing a series of Chinese blogs about transformers, hoping to promote it to more Chinese friends. I want to download all of your pretrained models for study and experiments, but due to network problems in China, the download will be very slow and unstable. At the same time, manual downloading one by one is very troublesome. Could you kindly provide a unified download link which I can download all models?
        Thank you.
@julien-c
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Use BART for longer documents

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Fairseq folks say we can finetune BART model with longer seq_len on our custom training data. They pre-trained bart on 512 seq_len and during fine-tuning, they use 1024 seq_len. One can raise it further (let's say 2048) and finetune.

For above, we would need to adjust positional embeddings by either:

1. learning them from start.
2. copy 512 from pre-trained BART to the first 512 of your 2048 positional embedding.

with 2 being recommended. If I have to do it any pointers where to change the code ?

https://github.com/pytorch/fairseq/issues/1685


Update README.md

Related to pretrained model cards


training script for any tensorflow based pre-trained model for question-answer

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

I am trying to train a question-answer model from scratch using TensorFlow backend. But I am getting unexpected result. It would be a great help if Hugging Face team or anyone can share training script of any TensorFlow based pre-trained model for question-answer (like 'bert-large-uncased-whole-word-masking-finetuned-squad' or 'distilbert-base-uncased-distilled-squad', etc)
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

**A link to original question on Stack Overflow**:
https://stackoverflow.com/questions/61708227/how-to-train-any-hugging-face-transformer-model-eg-distilbert-for-question-ans


Wrong returns of API:get_special_tokens_mask()

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
I use transformers tokenizer, and created mask using API: get_special_tokens_mask.
[My Code](https://github.com/ishikawa-takumi/transformers-sample/blob/master/tokenizer.ipynb)

In [RoBERTa Doc](https://huggingface.co/transformers/model_doc/roberta.html), returns of this API is "A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token". But I seem that this API returns "0 for a sequence token, 1 for a special token".
## To reproduce

Steps to reproduce the behavior:

1. Use get_special_tokens_mask().
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Returns of this API is "A list of integers in the range [0, 1]: 0 for a special token, 1 for a sequence token".

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Add migrating from `pytorch-transformers`

"Migrating from pytorch-transformers to transformers" is missing in the main document. It is available in the main `readme` thought. Just move it to the document.


FileNotFoundError when running distributed Trainer

# 🐛 Bug

## Information

I'm running the language modeling example in distributed mode and am getting FileNotFoundError in the Trainer.

The error :
```
Traceback (most recent call last):
  File "run_language_modeling.py", line 292, in <module>
    main()
  File "run_language_modeling.py", line 257, in main
    trainer.train(model_path=model_path)
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/transformers/trainer.py", line 451, in train
    torch.save(optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 327, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 212, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/home/shaoyent/anaconda3/envs/bert/lib/python3.7/site-packages/torch/serialization.py", line 193, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/home/shaoyent/transformers/examples/language-modeling/output/checkpoint-500/optimizer.pt'
```

The error seems to be the Trainer saving checkpoints in each [local master](https://github.com/huggingface/transformers/blob/b290c32e1617fa74f44ccd8b83365fe764437be9/src/transformers/trainer.py#L413) while the directory is created by [world master](https://github.com/huggingface/transformers/blob/b290c32e1617fa74f44ccd8b83365fe764437be9/src/transformers/trainer.py#L448) causing a race condition.

I think the suggested method is to handle save checkpoints in world master, since weights should be synchronized after backprop.


Evaluating a checkpoint

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I have trained a NER model with the following command
python3.6 examples/token-classification/run_ner.py --save_steps 100 --data_dir $DIR --model_type bert --model_name_or_path bert-base-cased --output_dir $DIR/model --do_train --do_eval --do_predict --overwrite_output_dir --labels $DIR/labels.txt --evaluate_during_training --max_seq_length 300

Now, I am trying to evaluate and predict at checkpoint 500 (already evaluated during training using --evaluate_during_training argument) using the following command
python3.6 examples/token-classification/run_ner.py --data_dir $DIR --model_type bert --model_name_or_path bert-base-cased --output_dir $DIR/model/checkpoint-500 --do_eval --do_predict --labels $DIR/labels.txt --max_seq_length 300

However, I don't get same performance as during training (as evaluated at the same checkpoint using --evaluate_during_training argument), I get lower performance (by far).

Any idea about it?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Add MultipleChoice to TFTrainer [WIP]

TensorFlow

Anything TensorFlow

Working on getting the TF trainer running on TPUs for multiple choice. I'm quite the newb so input is needed.

- [x]  Add 1 GPU strategy
- [x]  Args for TPU Name
- [x]  Add MPC example for TFTrainer
- [x]  Fix docstrings

Examples:
GPU on Swag
https://colab.research.google.com/drive/1X9d1TY1I7r8d3rCJ9vWiy0UgvJM6BqQ4

TPU on Swag (fails, havent looked into it might be my misstake):
https://colab.research.google.com/drive/1X9d1TY1I7r8d3rCJ9vWiy0UgvJM6BqQ4#scrollTo=2r1Rng8NWzit

Glue on TPU (fails, havent looked into it, might be my mistake)
https://colab.research.google.com/drive/1f2x5wPSaBV0NbGCpMyUPXGUSMEGjSHeX#scrollTo=YDhXmegN3WWj


[Jax] top_k_top_p


KlDivBackward

Hi,
I fine-tune bert and using KLDivLoss instead of CrossEntropyLoss in binary classification. when I call
loss.backward()
I got this error !?

![error-KLDiv](https://user-images.githubusercontent.com/55197626/81508647-4dd85780-92d3-11ea-9547-a9690b733a61.PNG)

Thank you in advance


useless chg


[CI] remove run_tests_torch_and_tf

Problem:
This job is rerunning all tests, after `run_tests_torch` and `run_tests_tf` with both packages installed. Meaning that it ~doubles the runtime of CI and doesn't find very many bugs.

Solution: This changes `run_tests_torch_and_tf` to only run the tests that require both TF and torch


[DOCS] BertModel documentation doesn't mention the usage of tokenizers for creating input_ids

Documentation of https://huggingface.co/transformers/model_doc/bert.html#transformers.BertModel.forward currently doesn't mention the usage of `tokenizers.BertWordPieceTokenizer` for instance when I am using BertModel.


Align sentiment-analysis' tokenizer (currently cased) to the model (uncased)

`sentiment-analysis` pipeline has a tokenizer using a cased vocab while the model is uncased.

This PR makes the tokenizer uncased to match the actual model's vocab.


Sentiment Analysis Pipeline is predicting incorrect sentiment in 2.9

# 🐛 Bug
Sentiment Analysis Pipeline is predicting incorrect sentiment. Everything seems to be NEGATIVE.

## To reproduce

Steps to reproduce the behavior:
Left a Colab Notebook here for reproducing: https://colab.research.google.com/drive/1hIwyBOGFpcLMUNmaKf5UYb0wPeYb4yKS?usp=sharing

`pip install -qU transformers`

`from transformers import pipeline`
`sentiment_model29 = pipeline('sentiment-analysis')`
`sentiment_model29('I hate this movie')`
Returns: [{'label': 'NEGATIVE', 'score': 0.703369677066803}]


NER Models not using BIO tagging scheme

Hello,

I have noticed that most (all?) pre-trained models uploaded seem not to use the BIO tagging scheme, instead returning `I-xxx` and `O` for all categories (`B-xxx` is not used). I believe this is just caused by the way these models have been trained. This makes using these models in actual pipelines difficult as entity parsing becomes impossible (two consecutive entities cannot be differentiated from a multi-tokens entity).

Are there any plans for updating these models with a BIO labelling scheme?
@stefan-it including you in the discussion as you have uploaded standard models used in the default pipelines


I am training a bert model for question classification task, now it is a binary classifier. 0 for Descriptive question and 1 for not descriptive. While testing I am getting the following error :-

I have generated my model with the help of around 1700 questions

But when I am testing it for one question at a time and running
      outputs = model(b_input_ids, token_type_ids=None, attention_mask=b_input_masks)

It is giving me the following error
ValueError: Wrong shape for input_ids (shape torch.Size([1])) or attention_mask (shape torch.Size([1]))

b_input_ids are the padded id of the input question
b_input_masks are the attention mask of the b_input_ids

Any idea how this error can be resolved


Strange Behaviour using Transformer pipline with FLASK

# ❓ Questions & Help

I am building a web-frame  with Flask to run Transformer pipeline ( sentiment - analysis ) over Excel columns text.
However ,  In first try  running Flask app , gives following error as :

ModuleNotFoundError: No module named 'tensorflow_core._api'

after few days, without any success, As mentioned in other pages , issue will be resolved or better saying ignored by  change debug Flask run flag from TRUE to FALSE as below :

  app.run(host='0.0.0.0', debug=False)

It runs ! Webpage opens, Excel sheet  uploaded to server and then after calling pipline Transformer

!!  Now i am facing very strange!!! behaviour !

It seems that pipline works !! but results are different than stand alone run!! !

In light be better clarification
For instance :

Downloading: 100%|█████████████████████████████████████████████████████████████| 230/230 [00:00<00:00, 255kB/s]
amazing job
----------------
[{'label': 'POSITIVE', 'score': 0.99987316}]
sounds misleading
----------------
[{'label': 'NEGATIVE', 'score': 0.9996892}]
amazing job
----------------
[{'label': 'POSITIVE', 'score': 0.99987316}]
sounds cheap
----------------
[{'label': 'NEGATIVE', 'score': 0.99980205}]
Uninteresting.
----------------
[{'label': 'NEGATIVE', 'score': 0.9997152}]
This is an Interesting item.
----------------
[{'label': 'POSITIVE', 'score': 0.99984246}]
very good
----------------
[{'label': 'POSITIVE', 'score': 0.999852}]
this is inappropriate for you
----------------
[{'label': 'NEGATIVE', 'score': 0.99970394}]
this is appropriate for you
----------------
[{'label': 'POSITIVE', 'score': 0.99979025}]
very bad
----------------
[{'label': 'NEGATIVE', 'score': 0.99978846}]

will be like as below after calling in FLASK app

amazing job | 0.793442070484161 | NEGATIVE
sounds misleading | 0.940393328666687 | POSITIVE
amazing job | 0.793442070484161 | NEGATIVE
sounds cheap | 0.934843122959137 | NEGATIVE
Uninteresting. | 0.995108544826508 | NEGATIVE
This is an Interesting item. | 0.946804642677307 | NEGATIVE
very good | 0.77636182308197 | POSITIVE
this is inappropriate for you | 0.945333242416382 | NEGATIVE
this is appropriate for you | 0.942687690258026 | NEGATIVE
very bad | 0.623483657836914 | NEGATIVE

Your help and comments are greatly appreciated.

## Details

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Fix bug in sample code


Model call with `inputs_embeds` does not match that with `input_ids`

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using: my own modified scripts: (details below)

## To reproduce

Steps to reproduce the behavior:

```
import tensorflow as tf
from transformers import BertConfig, BertTokenizer, TFBertModel

bert_tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

input_ids = tf.constant(bert_tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]
attention_mask = tf.stack([tf.ones(shape=(len(sent),)) for sent in input_ids])
token_type_ids = tf.stack([tf.ones(shape=(len(sent),)) for sent in input_ids])

config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
bert_model = TFBertModel.from_pretrained('bert-base-uncased', config=config)

result = bert_model(inputs={'input_ids': input_ids,
                            'attention_mask': attention_mask,
                             'token_type_ids': token_type_ids})
inputs_embeds = result[-1][0]
result2 = bert_model(inputs={'inputs_embeds': inputs_embeds,
                            'attention_mask': attention_mask,
                             'token_type_ids': token_type_ids})

print(tf.reduce_sum(tf.abs(result[0] - result2[0])))  # 458.2522, should be 0
```

## Expected behavior

The HuggingFace BERT TensorFlow implementation allows us to feed in a precomputed embedding in place of the embedding lookup that is native to BERT. This is done using the model's `call` method's optional parameter `inputs_embeds` (in place of `input_ids`). To test this out, I wanted to make sure that if I did feed in BERT's embedding lookup, I would get the same result as having fed in the `input_ids` themselves.

The result of BERT's embedding lookup can be obtained by setting the BERT configuration parameter `output_hidden_states` to `True` and extracting the first tensor from the last output of the call method. (The remaining 12 outputs correspond to each of the 12 hidden layers.)

Thus, I wrote the above code to test my hypothesis. The output of the `call` method is a tuple, and the first element of this tuple is the output of the last layer of BERT. Thus, I expected `result[0]` and `result2[0]` to match. Why is this not the case?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-debian-stretch-sid
- Python version: 3.6.10
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


added functionality for electra classification head

Added functionality for an electra classification head following the google repo.
Tested and trains fine with the run_glue script.  I was testing on my own repurposed classification problems so i didnt evaluate the actual GLUE accuracies, but can confirm that it is getting competitive results on my own problems.
Saw a few requests for this, thought it could benefit the community.

Originally I used a tanh activation in the classification head.  (which in my tests reliably performed better than a gelu activation)  But looking over the google code, it seems that they used gelu, so to keep the integrity of their paper im committing gelu here.

I also opted to make a ClassificationHead class instead of integrating a pooling layer in the base model class because i wasn't sure if that would affect other things, or run time in non sequence-classification tasks.


Vaswani's Transformer (Encoder-Decoder) decoding methods

Hi,

I recently studied this blog https://huggingface.co/blog/how-to-generate for using different decoding methods with PreTrained Models.

I would like to ask if it is possible to use those methods with the simple Encoder-Decoder Transformer model (as mentioned in Vaswani's paper)  (as it is available in pytorch library "torch.nn") or I have to implement those decoding methods by my own.

Thank you in advance.


bert uncased models needs lower casing option turned on in the example code.

The command to replicate bert-base-uncased model and bert-large-uncased-whole-word-masking model results on Squad requires --do_lower_case to be used. If not, the result is low.


'utf-8' codec can't decode byte 0x80 in position 229888: invalid start byte

Usage

General questions about the library

os : google colab
from transformers import *
tokenizer = AutoTokenizer.from_pretrained('https://s3-us-west-2.amazonaws.com/ai2-s2-research/scibert/huggingface_pytorch/scibert_scivocab_uncased.tar')
![SCI_BERT](https://user-images.githubusercontent.com/51091425/81477029-d94ddc00-9232-11ea-90ca-827a0e0ecc80.PNG)


Conversion script to export transformers models to ONNX IR.

This PR adds conversion script to export our models to ONNX IR.

Plan is to support both PyTorch and TensorFlow:

- [x] PyTorch
- [x] TensorFlow

TensorFlow currently blocked because of an issue in the conversion script provided by ONNX (seems fixed on master : https://github.com/onnx/tensorflow-onnx/issues/876)

_PS: PR recreated because the first one got corrupted (#4176)._


How to specify a TPU IP for run_tf_glue.py?

Hi team!

I have a VM on Google Cloud and a TPU. When running your example script `run_tf_glue.py` there is no option for specifying a TPU IP. It seems like the TPU cannot be auto-discovered and then training falls back to training on CPU 🐢


Encoder - Decoder loading wrong weights and missing decoders

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta as encoder and Bert as decoder

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. https://github.com/ThilinaRajapakse/simpletransformers/pull/360

## Expected behavior
Predicting the same string before saving and after loading the saved model are completely different.
Expected that i get the same results both times

Furthermore your tutorial on medium is not working, caused missing decoder models.
Link to medium: https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8
There is missing an argument in the forward function:
`TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Linux
- Python version: 3.7
- PyTorch version (GPU?): GPU 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


[Request] NER Scripts on CoNLL 2003 dataset

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation
Current NER scripts (data preparation, train, and evaluation) is for the GermEval 2014 dataset.
It is very nice. However, according to the original BERT paper, experiments are done on the CoNLL-2003 dataset. It would be much better if the NER script could reproduce the results on this dataset.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
Looking forward to the feedback. Thanks in advance!

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


[docs] fix typo

replace sequence classification with question answering in AutoModelForQuestionAnswering class


sentiment analysis pipeline provides option to output the original rating than positive/negative tags

It'd be helpful to provide an option for sentiment analysis pipeline output the original ratings rather than the positive/negative tags.

Also the documentation doesn't really explain what the output scores mean..


removed variable api from pplm example

Same as previous [PR](https://github.com/huggingface/transformers/pull/4156)


Structuring Dataset for Fine Tuning GPT-2 with Song Lyrics?

I'm looking to fine tune a GPT-2 model with several songs from the same artist. However, I'm not exactly sure how I should be feeding nearly 300 songs into the model to learn from.

Currently, I'm breaking the lyrics into blocks of 100 and feeding them into my model:

```
lyrics = open('/JayZ_lyrics.txt', "r", encoding="utf-8").read()
lyrics = lyrics.split('\n')
lyrics = ''.join(lyrics)

tokenizer = GPT2Tokenizer.from_pretrained('distilgpt2')
tokenized_text = tokenizer.encode(lyrics)

examples = []
block_size = 100
for i in range(0, len(tokenized_text)-block_size+1, block_size):
    examples.append(tokenized_text[i:i+block_size])

inputs, labels = [], []
for ex in examples:
    inputs.append(ex[:-1])
    labels.append(ex[1:])
```

Then, I feed my dataset into the model and train on it.

```
dataset= tf.data.Dataset.from_tensor_slices((inputs,labels))
BATCH_SIZE = 16
dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)

optimizer = tf.keras.optimizers.Adam(epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model = TFGPT2LMHeadModel.from_pretrained('distilgpt2')
model.compile(optimizer=optimizer, loss=[loss, None, None, None, None, None, None], metrics=[metric])
model.fit(dataset, epochs=4, callbacks=[LRFinder()])
```

Is this the right approach? Or, should I turn every song into a vector, find the len of longest song, pad the other songs to keep uniformity amongst vector lengths, feed in dataset.

Also, should I include a BOS and/or EOS token `<|endoftext|>` for each block that I create?


Add back option --do_lower_case in SQuAD for uncased models

The option `--do_lower_case` is currently needed by the uncased models (i.e., bert-base-uncased, bert-large-uncased). The results w/ and w/o this option are shown below.

The option was deleted in the current v2.9.0 version so this PR tries to add it back. This would not affect the cased models (i.e., RoBERTa, XLNet).

Results of SQuAD v1.1:
bert-base-uncased **without** `--do_lower_case`:  'exact': 73.83, 'f1': 82.22
bert-base-uncased **with** `--do_lower_cas`e:  'exact': 81.02, 'f1': 88.34


Allow gpt2 to be exported to valid ONNX

Problem: gpt2 model and other models using gelu_new or gelu_fast cannot be exported to valid ONNX model.

Causes:
PyTorch will export some constants like 1 or 2 as int64, while ONNX operators like [Add](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Add) and [Pow](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Pow) requires data type matches between two inputs. It is invalid to have one input as float and another as int64.

For GPT2 model, another issue is the ONNX operator [Where](https://github.com/onnx/onnx/blob/master/docs/Operators.md#Where). It requires the first input to be bool tensor but currently it is uint8 tensor.

Add explicit data type cast could solve these problems. It is verified in exporting 3 pretrained models to ONNX model and loading by ONNX Runtime: albert-base-v2, distilgpt2, gpt2.


Distributed eval: SequentialDistributedSampler + gather all results

As discussed in #3944

In our [**`Trainer`**](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py), in torch.distributed mode, eval/predict are currently running on a single node. This PR attempts to fix this.

This relies on:
- a custom `DistributedSampler` named SequentialDistributedSampler that makes it easier to collate inference results at the end of the loop
- some tricky code to gather numpy arrays back to a single node (or to all of them).

I might be missing something simpler here. For instance in `pytorch/xla` there seems to be a [built-in function to do this](https://github.com/huggingface/transformers/blob/7b75aa9fa55bee577e2c7403301ed31103125a35/src/transformers/trainer.py#L656-L659).

Please let me know if that's the case 😁


Reformer self attention mask value not converted in apex half precision

reformer

Everything related to the reformer model

# 🐛 Bug

## Information

Model I am using Reformer:

Language I am using the model on English:

The problem arises when using:
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Install the latest transformers 2.9.0 and apex

```
from transformers import ReformerModel, ReformerTokenizer
import torch
from apex import amp

tokenizer = ReformerTokenizer.from_pretrained('google/reformer-crime-and-punishment')
model =  ReformerModel.from_pretrained('google/reformer-crime-and-punishment').cuda()

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

model, optimizer = amp.initialize(model, optimizer, opt_level='O1')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0).cuda()  # Batch size 1
outputs = model(input_ids)

last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

```

## Expected behavior

```
File "/home/model/env/lib/python3.7/site-packages/transformers/modeling_reformer.py", line 840, in forward
    query_key_dots = torch.where(mask, query_key_dots, mask_value)
RuntimeError: expected scalar type Half but found Float
```

- `transformers` version:
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0 cuda
- Tensorflow version (GPU?): No tensorflow
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


# 🐛 Bug

## Information

Model I am using Reformer:

Language I am using the model on English:

The problem arises when using:
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Install the latest transformers 2.9.0 and apex

```
from transformers import ReformerModel, ReformerTokenizer
import torch
from apex import amp

tokenizer = ReformerTokenizer.from_pretrained('google/reformer-crime-and-punishment')
model =  ReformerModel.from_pretrained('google/reformer-crime-and-punishment').cuda()

optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

model, optimizer = amp.initialize(model, optimizer, opt_level='O1')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0).cuda()  # Batch size 1
outputs = model(input_ids)

last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

```

## Expected behavior

```
File "/home/model/env/lib/python3.7/site-packages/transformers/modeling_reformer.py", line 840, in forward
    query_key_dots = torch.where(mask, query_key_dots, mask_value)
RuntimeError: expected scalar type Half but found Float
```

- `transformers` version: 2.9.0 (latest source from github)
- Platform: Ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5.0 cuda
- Tensorflow version (GPU?): No tensorflow
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


I am traing a roberta model and running the script examples/run_language_modeling.py
The following error occurs when i am trying to resume training.
Traceback (most recent call last):
  File "examples/run_language_modeling.py", line 284, in <module>
    main()
  File "examples/run_language_modeling.py", line 254, in main
    trainer.train(model_path=model_path)
  File "/home/socian-pc1/anaconda3/envs/XformerEnv/lib/python3.6/site-packages/transformers/trainer.py", line 326, in train
    optimizer.step()
  File "/home/socian-pc1/anaconda3/envs/XformerEnv/lib/python3.6/site-packages/torch/optim/lr_scheduler.py", line 67, in wrapper
    return wrapped(*args, **kwargs)
  File "/home/socian-pc1/anaconda3/envs/XformerEnv/lib/python3.6/site-packages/transformers/optimization.py", line 155, in step
    exp_avg.mul_(beta1).add_(1.0 - beta1, grad)
RuntimeError: expected device cpu but got device cuda:0

My config
python examples/run_language_modeling.py \
    --train_data_file $TRAIN_FILE \
    --eval_data_file $TEST_FILE \
    --output_dir ./MyRobertaOutput \
    --model_name_or_path ./MyRoBERTa/checkpoint-570000 \
    --config_name ../xformer_output \
    --tokenizer_name ../xformer_output \
    --mlm \
    --do_train \
    --do_eval \
    --line_by_line \
    --learning_rate 1e-5 \
    --num_train_epochs 2 \
    --save_total_limit 20 \
    --save_steps 5000 \
    --per_gpu_train_batch_size 6 \
    --warmup_steps=10000 \
    --logging_steps=100 \
    --gradient_accumulation_steps=4 \
    --seed 666 --block_size=512


distilbert-base-uncased

# 🐛 Bug/Question

## Information
 am trying to execute :

**import ktrain

from ktrain import text

MODEL_NAME='distilbert-base-uncased'

t=text.Transformer(MODEL_NAME, maxlen=500, classes=np.unique(y_train))**

I get the following error:

OSError: Model name 'distilbert-base-uncased' was not found in tokenizers model name list (distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-cased, distilbert-base-cased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased). We assumed 'distilbert-base-uncased' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

ktrain version 0.14.6 transformers version 2.8.0

Libraries was installed using pip install. I am following the tutorial : https://towardsdatascience.com/text-classification-with-hugging-face-transformers-in-tensorflow-2-without-tears-ee50e4f3e7ed

Any help would be appreciated.


[tests] make pipelines tests faster with smaller models

- speedup: 8mins -> 90s on 1 CPU

- besides `test_fill_mask` the removed coverage was not meaningful, only shapes were being checked.
- Would be happy to continue for TF or get help.
- could use these in some examples tests also.


Only self-attention returned when output_attentions = True for BERT

# 🐛 Bug
```
if self.output_attentions:
                all_attentions = all_attentions + (layer_outputs[1],)
```
should be
```
if self.output_attentions:
                all_attentions = all_attentions + (layer_outputs[1:],)
```
## Information
Line 412 [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py
)

I assume that there is no reason to exclude cross-attention from all_attentions.


Pipelines: use tokenizer.max_len

fix #4224 by passing tokenizer.max_len to `tokenizer.batch_encode_plus`


Error importing Reformer model

# 🐛 Bug

## Information

I am trying to import the Reformer model as in here:
https://huggingface.co/transformers/model_doc/reformer.html
'''
from transformers import ReformerModel, ReformerConfig
'''
However, I am getting the following error:
ImportError: cannot import name 'ReformerModel' from 'transformers'

The problem arises when using:
* [ √] the official example scripts: (give details below)
* [ X ] my own modified scripts: (give details below)

- `transformers` version: 2.8.0


[infra] make a tiny "distilroberta-base" to speed up test_pipelines.py and test_examples.py

put it in a user namespace on s3 like `sshleifer/tiny-distilroberta`
doing the same for distilbert-base would probably also help.


How can I mute the info from BertTokenizerFast.encode_plus

# 🐛 Bug

## Information

Model I am using:
Bert

Language I am using the model on (English, Chinese ...):
Chinese

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
```python
tokenizer.encode_plus(...)
```
Every time I run this line, the info "Fast tokenizers add special tokens by default. To remove special tokens, please specify`add_special_tokens=False` during the initialisation rather than when calling `encode`,`encode_plus` or `batch_encode_plus`." will come out.

Even though I have already set the add_special_tokens to False
```python
tokenizer = BertTokenizerFast.from_pretrained(model_path, add_special_tokens = False)
```
The info can be very annoying when I run an iteration loop. How can I mute this info.

## Environment info

- `transformers` version: 2.8.0
- Platform: Linux-4.4.0-87-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cu100 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


MarianMTModel: Runtime Errors

11/1010 models cannot me loaded because of state_dict mismatches (not tokenizer).

```python
failing = ['sv-tpi',
 'fi-en',
 'sv-ty',
 'sv-toi',
 'en-st',
 'en-tl',
 'sv-ts',
 'sv-tvl',
 'en-fi',
 'en-ml',
 'fi-sv']
for mname in failing:
	slug = f'opus-mt-{mname}'
	MarianMTModel.from_pretrained(slug)
```

`en-st` is a slightly different error:
```
  'Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.'),
```


Strange difference between NER results in 2.8.0 and previous versions

# 🐛 Bug

## Information

Model I am using: Albert

Language I am using the model on: English and Portuguese

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Not sure if this is a bug, but for the same seed and hyperparameters, I'm getting quite different results for different benchmarks on transformers 2.8.0 and 2.7.0.

Steps to reproduce the behavior:

1. For 2.7.0: `python examples/ner/run_ner.py --data_dir <data-dir> --labels <labels-path> --model_type albert --model_name_or_path <albert-model> --output_dir <output-dir> --max_seq_length 512 --overwrite_output_dir --do_train --do_eval --do_predict --per_gpu_train_batch_size 16 --per_gpu_eval_batch_size 8 --learning_rate 5e-05 --num_train_epochs 5 --save_steps 878 --seed 1`

2. For 2.8.0: `python examples/ner/run_ner.py config.json`. Here's the config file for each:

2.1. For CoNLL:
```
{
    "data_dir": "/home/corpora/ner/conll2003",
    "labels": "/home/corpora/ner/conll2003/labels.txt",
    "model_name_or_path": "albert-base-v2",
    "output_dir": "/home/models/albert/conll2003",
    "max_seq_length": 512,
    "num_train_epochs": 5,
    "per_gpu_train_batch_size": 16,
    "save_steps": 878,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true,
    "overwrite_output_dir": true
}
```
2.2. For HAREM:
```
{
    "data_dir": "/home/corpora/ner/harem/selective",
    "labels": "/home/corpora/ner/harem/selective/labels.txt",
    "model_name_or_path": "/home/models/albert/base_pt_brwac_en/albert_ckp_1176k",
    "output_dir": "/home/models/albert/base_pt_brwac_en/albert_ckp_1176k/harem",
    "max_seq_length": 512,
    "num_train_epochs": 5,
    "per_gpu_train_batch_size": 16,
    "save_steps": 878,
    "seed": 1,
    "do_train": true,
    "do_eval": true,
    "do_predict": true,
    "overwrite_output_dir": true
}
```
3. For English CoNLL-2003 used albert-base-v2, and for Portuguese HAREM used a checkpoint for a version we're training.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Following are the results I'm getting for each of the 4 runs. I think the results should be a lot closer to each other, considering the same hyperparameters and seeds are being used, right?

1. 2.7.0 / CoNLL-2003:

```
INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='/home/corpora/ner/conll2003', device=device(type='cuda'), do_eval=True, do_lower_case=False, do_predict=True, do_train=True, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, keep_accents=None, labels='/home/corpora/ner/conll2003/labels.txt', learning_rate=5e-05, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_seq_length=512, max_steps=-1, model_name_or_path='albert-base-v2', model_type='albert', n_gpu=1, no_cuda=False, num_train_epochs=5.0, output_dir='/home/models/albert/conll2003-2.7.0', overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=16, save_steps=878, seed=1, server_ip='', server_port='', strip_accents=None, tokenizer_name='', use_fast=None, warmup_steps=0, weight_decay=0.0)

INFO - __main__ -     f1 = 0.8424940638466273
INFO - __main__ -     loss = 0.184873234364002
INFO - __main__ -     precision = 0.836973615236764
INFO - __main__ -     recall = 0.8480878186968839
```

2. 2.8.0 / CoNLL-2003:

```
INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/home/models/albert/conll2003', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=878, save_total_limit=None, no_cuda=False, seed=1, fp16=False, fp16_opt_level='O1', local_rank=-1)

INFO - __main__ -     precision = 0.890553401480437
INFO - __main__ -     recall = 0.8946529745042493
INFO - __main__ -     f1 = 0.892598480833775
INFO - __main__ -     loss = 0.15480460636704224
```
3. 2.7.0 / HAREM

```
INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, cache_dir='', config_name='', data_dir='/home/corpora/ner/harem/selective', device=device(type='cuda'), do_eval=True, do_lower_case=False, do_predict=True, do_train=True, eval_all_checkpoints=False, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, keep_accents=None, labels='/home/corpora/ner/harem/selective/labels.txt', learning_rate=5e-05, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_seq_length=512, max_steps=-1, model_name_or_path='albert-base-v2', model_type='albert', n_gpu=1, no_cuda=False, num_train_epochs=5.0, output_dir='/home/models/albert/eduardo/base_pt_brwac_en/albert_ckp_1176k/harem-2.7.0', overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=8, per_gpu_train_batch_size=16, save_steps=878, seed=1, server_ip='', server_port='', strip_accents=None, tokenizer_name='', use_fast=None, warmup_steps=0, weight_decay=0.0)

INFO - __main__ -     f1 = 0.805560264451602
INFO - __main__ -     loss = 0.11168152010335869
INFO - __main__ -     precision = 0.8184636582845333
INFO - __main__ -     recall = 0.7930574098798397
```

4. 2.8.0 / HAREM

```
INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/home/models/albert/eduardo/base_pt_brwac_en/albert_ckp_1176k/harem', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=5, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=878, save_total_limit=None, no_cuda=False, seed=1, fp16=False, fp16_opt_level='O1', local_rank=-1)

INFO - __main__ -     precision = 0.7228915662650602
INFO - __main__ -     recall = 0.7209612817089452
INFO - __main__ -     f1 = 0.7219251336898397
INFO - __main__ -     loss = 0.14575799524457683
```

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-99-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes, Tesla V100-SXM2
- Using distributed or parallel set-up in script?: No


example updated to use generation pipeline

model card

Related to pretrained model cards


Problem with BertTokenizer using additional_special_tokens

# 🐛 Bug

## Information

Model I am using: Bert (bert-base-uncased)

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. import transformers
2. from transformers import BertTokenizer
3. specify additional_special_tokens as ["[E11]", "[E12]", "[E21]", "[E22]"]
4. instantiate tokenizer as `tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True, additional_special_tokens=additional_special_tokens)`
5. tokenize test string with the special tokens `'[E11] Tom Thabane [E12] resigned in October last year to form the [E21] All Basotho Convention [E22] -LRB- ABC -RRB- , crossing the floor with 17 members of parliament , causing constitutional monarch King Letsie III to dissolve parliament and call the snap election .'` using `tokenizer.tokenize(test_string)`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I would expect that the tokenization would give me regular tokens from wordpiece and the special tokens would be kept intact as [E11], [E12], etc, but instead I get:

`['[', 'e', '##11', ']', 'tom', 'tha', '##bane', '[', 'e', '##12', ']', 'resigned', 'in', 'october', 'last', 'year', 'to', 'form', 'the', '[', 'e', '##21', ']', 'all', 'bas', '##otho', 'convention', '[', 'e', '##22', ']', '-', 'l', '##rb', '-', 'abc', '-', 'rr', '##b', '-', ',', 'crossing', 'the', 'floor', 'with', '17', 'members', 'of', 'parliament', ',', 'causing', 'constitutional', 'monarch', 'king', 'lets', '##ie', 'iii', 'to', 'dissolve', 'parliament', 'and', 'call', 'the', 'snap', 'election', '.']`

I'm trying to run a training from [https://github.com/mickeystroller/R-BERT](https://github.com/mickeystroller/R-BERT), and reported this to the author, but he seems to get the proper results, even though we're both using transformers 2.8.0:

His results:

![image](https://user-images.githubusercontent.com/12713359/81397632-1b293600-90fe-11ea-8d91-a7af43aeabae.png)

My results:

![image](https://user-images.githubusercontent.com/12713359/81397653-241a0780-90fe-11ea-9ba6-1d86c2435dfd.png)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-99-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.3.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes, GTX 1070 with CUDA 10.1.243
- Using distributed or parallel set-up in script?: No

Here's the output from R-BERT's author as well:

- `transformers` version: 2.8.0
- Platform: Linux-4.15.0-72-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

The author from R-BERT seems to be using `tokenizers version 0.5.2`, and mine is 0.7.0. I tried downgrading mine to 0.5.2 to see if I would get the same results he did, but this doesn't seem to work because it's not compatible with transformers 2.8.0, as can be seen below. I have no Idea how he was able to use both together:

```
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-514ca6d60059> in <module>
----> 1 import transformers
      2 from transformers import BertTokenizer

/media/discoD/anaconda3/envs/fast-bert/lib/python3.7/site-packages/transformers/__init__.py in <module>
     53 from .configuration_xlm_roberta import XLM_ROBERTA_PRETRAINED_CONFIG_ARCHIVE_MAP, XLMRobertaConfig
     54 from .configuration_xlnet import XLNET_PRETRAINED_CONFIG_ARCHIVE_MAP, XLNetConfig
---> 55 from .data import (
     56     DataProcessor,
     57     InputExample,

/media/discoD/anaconda3/envs/fast-bert/lib/python3.7/site-packages/transformers/data/__init__.py in <module>
      4
      5 from .metrics import is_sklearn_available
----> 6 from .processors import (
      7     DataProcessor,
      8     InputExample,

/media/discoD/anaconda3/envs/fast-bert/lib/python3.7/site-packages/transformers/data/processors/__init__.py in <module>
      3 # module, but to preserve other warnings. So, don't check this module at all.
      4
----> 5 from .glue import glue_convert_examples_to_features, glue_output_modes, glue_processors, glue_tasks_num_labels
      6 from .squad import SquadExample, SquadFeatures, SquadV1Processor, SquadV2Processor, squad_convert_examples_to_features
      7 from .utils import DataProcessor, InputExample, InputFeatures, SingleSentenceClassificationProcessor

/media/discoD/anaconda3/envs/fast-bert/lib/python3.7/site-packages/transformers/data/processors/glue.py in <module>
     21
     22 from ...file_utils import is_tf_available
---> 23 from ...tokenization_utils import PreTrainedTokenizer
     24 from .utils import DataProcessor, InputExample, InputFeatures
     25

/media/discoD/anaconda3/envs/fast-bert/lib/python3.7/site-packages/transformers/tokenization_utils.py in <module>
     27 from typing import List, Optional, Sequence, Tuple, Union
     28
---> 29 from tokenizers import AddedToken, Encoding
     30 from tokenizers.decoders import Decoder
     31 from tokenizers.implementations import BaseTokenizer

ImportError: cannot import name 'AddedToken' from 'tokenizers' (/media/discoD/anaconda3/envs/fast-bert/lib/python3.7/site-packages/tokenizers/__init__.py)
```


Reformer tokenizer not working

# 🐛 Bug
`TextDataset` throws out of index error.
## Information
```
model_args = ModelArguments(model_name_or_path = 'Reformer',
                            config_name = 'ReformerConfig',
                           tokenizer_name = 'ReformerTokenizer')
training_args = TrainingArguments(output_dir = '/tmp', do_train = True,
                                  evaluate_during_training = True,
                                  do_eval = True,
                                  do_predict = False)
data_args = DataTrainingArguments(train_data_file = TRAIN_DATA_DIR,
                                  eval_data_file = EVAL_DATA_DIR,
                                 line_by_line = True)
tokenizer = AutoTokenizer.from_pretrained("google/reformer-crime-and-punishment",
                                         cache_dir=model_args.cache_dir)
train_dataset = (
    get_dataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank)
    if training_args.do_train
    else None
)
eval_dataset = (
    get_dataset(data_args, tokenizer=tokenizer, local_rank=training_args.local_rank, evaluate=True)
    if training_args.do_eval
    else None
)
```
It throws this:
```
~/.local/lib/python3.6/site-packages/transformers/tokenization_utils.py in truncate_sequences(self, ids, pair_ids, num_tokens_to_remove, truncation_strategy, stride)
   2044             for _ in range(num_tokens_to_remove):
   2045                 if pair_ids is None or len(ids) > len(pair_ids):
-> 2046                     overflowing_tokens = [ids[-1]] + overflowing_tokens
   2047                     ids = ids[:-1]
   2048                 else:

IndexError: list index out of range
```
When `line_by_line` is kept False,
```
train_dataset[0]
```
returns
```
~/.local/lib/python3.6/site-packages/transformers/data/datasets/language_modeling.py in __getitem__(self, i)
     72
     73     def __getitem__(self, i) -> torch.Tensor:
---> 74         return torch.tensor(self.examples[i], dtype=torch.long)
     75
     76

IndexError: list index out of range
```
Model I am using (Bert, XLNet ...): Reformer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) wikitext-2
* [ ] my own task or dataset: (give details below)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

`TextDataset` should work as per standard Pytorch `Dataset` class.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9
- Platform: Ubuntu 18.04
- Python version: 3.8
- PyTorch version (GPU?): 1.5
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


2.9.0 Padding Bug

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): **Roberta with RobertaForSequenceClassification head**

Language I am using the model on (English, Chinese ...): **Polish**

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

**The problem arises when performing a forward pass while training. I assume something is wrong with padding, though with 2.8.0 version it works correctly.**

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

**I'm using pre-trained roBERTa model for classification task. Details can be found in the notebook below.**

## To reproduce

Steps to reproduce the behavior:

1. Run https://colab.research.google.com/drive/1w54ScQ3-dh99-cO4OGG6-IdP4JRfki59?usp=sharing

Error message:

```
     56         # Perform a forward pass
---> 57         loss, logits = model(b_input_ids, token_type_ids=None, attention_mask=b_input_mask, labels=b_labels)

...

   2033     """
   2034     # The series of casts and type-conversions here are carefully balanced to both work with ONNX export and XLA.
-> 2035     mask = input_ids.ne(padding_idx).int()
   2036     incremental_indices = torch.cumsum(mask, dim=1).type_as(mask) * mask
   2037     return incremental_indices.long() + padding_idx

TypeError: ne() received an invalid combination of arguments - got (NoneType), but expected one of:
 * (Tensor other)
      didn't match because some of the arguments have invalid types: (!NoneType!)
 * (Number other)
      didn't match because some of the arguments have invalid types: (!NoneType!)
```

## Expected behavior

It should work without errors. The problem arises after installing 2.9.0 version. With 2.8.0 version it works correctly.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: **2.9.0**
- Platform: Google colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101
- Tensorflow version (GPU?): 2.2.0rc4
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: n/a


Simplify cache vars and allow for TRANSFORMERS_CACHE env

As it currently stands, "TRANSFORMERS_CACHE" is not an accepted environment variable. It seems that the these variables were not updated when moving from version pytorch_transformers to transformers. In addition, the fallback procedure could be improved. and simplified. Pathlib seems redundant here.

This PR:

- adds the TRANSFORMERS_CACHE environment variable
- change the fall-back order of the cache env variables

File_utils (and perhaps the rest of the library) might benefit from using pathlib more widely. In a future endeavour I might take this on.


[Reformer/Longformer] Add cross-attention layers for Encoder-Decoder setting

# ❓ Questions & Help

## Details
It seems that Reformer model has been added in Transformers. But it's a standard auto-regressive language model. How can we use Reformer in Encoder-Decoder settings ?


Bart now enforces maximum sequence length in Summarization Pipeline

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bart (bart-large-cnn)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

Based on example code in docs, though.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load default summarization pipeline
2. Try to use model to summarize text that has > 1024 tokens

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Example code:
```
from transformers import pipeline
summarizer = pipeline('summarization')
text = '=' * 102570    # Happened to be the length of the file I was testing, my actual file produced 25,257 tokens
print(summarizer(text))
```

Output:
```
Token indices sequence length is longer than the specified maximum sequence length for this model (1605 > 1024). Running this sequence through the model will result in indexing errors
Traceback (most recent call last):
  File "ex.py", line 4, in <module>
    print(summarizer(text, max_length=250))
  File ".../lib/python3.7/site-packages/transformers/pipelines.py", line 1330, in __call__
    inputs["input_ids"], attention_mask=inputs["attention_mask"], **generate_kwargs,
  File ".../lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File ".../lib/python3.7/site-packages/transformers/modeling_utils.py", line 1047, in generate
    encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask)
  File ".../lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File ".../lib/python3.7/site-packages/transformers/modeling_bart.py", line 292, in forward
    embed_pos = self.embed_positions(input_ids)
  File ".../lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File ".../lib/python3.7/site-packages/transformers/modeling_bart.py", line 763, in forward
    return super().forward(positions)
  File ".../lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File ".../lib/python3.7/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self
```

## Expected behavior

As of last week (week of 4/26/2020) this caused no issue. Today (5/7/2020) I tried to run the exact same code, a new model was downloaded (no change in transformers module, just the model itself), and now it enforces a token limit.

Expected behavior is to summarize document regardless of size.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0 (also occurs in 2.9.0)
- Platform: Both macOS 10.15.4 and Windows 10
- Python version: 3.7.5 (Mac) and 3.6.3/Anaconda (Windows)
- PyTorch version (GPU?): 1.5.0, no GPU
- Tensorflow version (GPU?): n/a
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
- Model (from associated JSON file downloaded): `{"url": "https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/pytorch_model.bin", "etag": "\"6eeacfe81d9304a6c5015424912f8df8\""}`
- Model config:
```
{
  "_num_labels": 3,
  "activation_dropout": 0.0,
  "activation_function": "gelu",
  "add_final_layer_norm": false,
  "attention_dropout": 0.0,
  "bos_token_id": 0,
  "classif_dropout": 0.0,
  "d_model": 1024,
  "decoder_attention_heads": 16,
  "decoder_ffn_dim": 4096,
  "decoder_layerdrop": 0.0,
  "decoder_layers": 12,
  "decoder_start_token_id": 2,
  "dropout": 0.1,
  "early_stopping": true,
  "encoder_attention_heads": 16,
  "encoder_ffn_dim": 4096,
  "encoder_layerdrop": 0.0,
  "encoder_layers": 12,
  "eos_token_id": 2,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1",
    "2": "LABEL_2"
  },
  "init_std": 0.02,
  "is_encoder_decoder": true,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1,
    "LABEL_2": 2
  },
  "length_penalty": 2.0,
  "max_length": 142,
  "max_position_embeddings": 1024,
  "min_length": 56,
  "model_type": "bart",
  "no_repeat_ngram_size": 3,
  "normalize_before": false,
  "num_beams": 4,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 1,
  "prefix": " ",
  "scale_embedding": false,
  "task_specific_params": {
    "summarization": {
      "early_stopping": true,
      "length_penalty": 2.0,
      "max_length": 142,
      "min_length": 56,
      "no_repeat_ngram_size": 3,
      "num_beams": 4
    }
  },
  "vocab_size": 50264
}
```

EDIT: Tagging @sshleifer as recommended by docs


[TPU] Doc, fix xla_spawn.py, only preprocess dataset once

- Fixed the `xla_spawn.py` launcher script which I broke earlier today when moving files around. It should be reasonably more robust now.
- Started to add a little documentation
- Changed the dataset logic so that multiple workers don't concurrently preprocess the dataset.


TPU Trainer's PerDeviceLoader has no len()

# 🐛 Bug

The new TPU Trainer fails when running on a single TPU core in a colab notebook since the XLA ParallelLoader's  PerDeviceLoader does not implement a length method.

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I used a custom script, but I would imagine it's not hard to reproduce using one of the example scripts. The error trace is below, with the relevant line in bold:

`Exception in device=TPU:0: object of type 'PerDeviceLoader' has no len()
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch_xla/distributed/xla_multiprocessing.py", line 119, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-6-3aa4c6105066>", line 264, in main
    trainer.train()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 286, in train
    **t_total = int(len(train_dataloader) // self.args.gradient_accumulation_steps *** self.args.num_train_epochs)
TypeError: object of type 'PerDeviceLoader' has no len()
An exception has occurred, use %tb to see the full traceback.`

## Expected behavior

Based on the conversation in the issue linked below, the PerDeviceLoader does not implement length, so the Trainer would need to be aware of that and get the length another way.

https://github.com/pytorch/xla/issues/1191

## Environment info

- `transformers` version: 2.9.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0a0+ab660ae (False)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: Using TPU
- Using distributed or parallel set-up in script?: XLA is used by the Trainer, but it's a single core TPU in a colab notebook


feat(trainer): log epoch and final metrics

This PR adds new logging capabilities to `Trainer`:

* logs epoch -> this let us compare runs with different batch sizes, using "epoch" as the x-axis instead of "global_step"
* logs final metrics -> metrics are also logged by `Trainer.evaluate` at the end of the script (or at the start if we just use `--do_eval` and not train).

In below example, we see `epoch` has been used for x-axis and `eval_loss` has one extra point corresponding to the end of the script.
![image](https://user-images.githubusercontent.com/715491/81350982-d57e5600-9088-11ea-8ce6-400bda09bea7.png)


Improvements to the wandb integration

Hey guys,

I'm one of the founders of W&B and have been working with @borisdayma on the wandb integration. I made a few changes to address a few cases we thought through:

1. If wandb is installed but the user hasn't logged in we now print a warning instead of prompting for an api_key
2. Made messaging more clear on how to install wandb and linked to more documentation
3. Added an environment variable for disabling wandb. This is important for users using fastai or lightning with their own wandb callback.
4. Added an environment variable for disabling wandb.watch. Because we're pulling gradients and parameters during training there is a small performance impact on training. We wanted to make it easy for users to disable this feature.

Happy to iterate on any feedback you guys have. Really excited about this integration!


Need correction in GPT-2 perplexity equation based on computation

# ❓ Questions & Help

I would like to formulate GPT-2 perplexity [calculation  ](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L266)in terms of equation as under:

![image](https://user-images.githubusercontent.com/64096105/81352017-2d8b7b80-90ce-11ea-8cdd-b44fdccf0ea3.png)

![image](https://user-images.githubusercontent.com/64096105/81352211-ab4f8700-90ce-11ea-990d-524c9756addf.png)

I am confused which equation can best describe the process.

Following are the representation:
1. N is orignal word length
2. M is the BPE word length after tokenization
3. w represents `n_position` size, which is 1024 for GPT-2 pretrained model.
4. i=0 till M

I try to make the first equation, as GPT-2 cannot calculate perplexity in one go, if M is greater than w. It needs to calculate mean of cross-entropy loss.

Can you please share your opinion in formulating an equation?


feat(Trainer): log epoch and final metrics


[Pipeline, Generation]  tf generation pipeline bug

Generation can currently not be used for TF due to a small bug. This PR fixes it and adds tests.

I will update the usage docs and pipeline notebook tomorrow.


[examples/summarization] run_train_tiny.sh: remove unused kwarg


Examples readme.md


[examples] text_classification/run_pl.sh error

Examples

Which is related to examples in general

Help wanted

Extra attention is needed, help appreciated

```bash
cd examples/text-classification
./run_pl.sh
```
trains for 1 epoch, tests 12/13 steps, then fails with the following traceback

```python
/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/utilities/warnings.py:18: UserWarning: The dataloader, test dataloader 0, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` in the `DataLoader` init to improve performance.
  warnings.warn(*args, **kwargs)
Testing:  92%|████████████████████████████████████   | 12/13 [00:00<00:00, 14.67it/s]Traceback (most recent call last):
  File "run_pl_glue.py", line 195, in <module>
    trainer.test(model)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 894, in test
    self.fit(model)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 704, in fit
    self.single_gpu_train(model)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 477, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 819, in run_pretrain_routine
    self.run_evaluation(test_mode=True)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 374, in run_evaluation
    eval_results)
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/logging.py", line 107, in process_output
    for k, v in output.items():
AttributeError: 'NoneType' object has no attribute 'items'
  File "/home/shleifer/.conda/envs/nb/lib/python3.7/site-packages/pytorch_lightning/trainer/logging.py", line 107, in process_output
    for k, v in output.items():
AttributeError: 'NoneType' object has no attribute 'items'
```


BIG Reorganize examples


GPT2Tokenizer.decode() slow for individual tokens

`GPT2Tokenizer.decode()` is considerable slow when invoking it for each token individually, not for a set of tokens... I need to pass in a set of tokens and get back a set of strings for each token. I tried convert_ids_to_tokens() but I'm getting results with the Ġ character. Should I simply remove that character from the results?

I also tried `GPT2TokenizerFast`, but it failed with `"encoder() takes from 2 to 3 positional arguments but 6 were given"` when I was trying to use batch_encode_plus() to pad my tokens with the following arguments: tokens (that were already encoded), pad_to_max_length=True and return_tensors='pt' . transformers ver 2.6.0 - is this a bug?


W&B integration improvements

Hey guys,

I'm one of the founders of W&B and have been working with @borisdayma on the wandb integration.  I made a few changes to address a few cases we thought through:

1. If wandb is installed but the user hasn't logged in we now print a warning instead of prompting for an api_key
2. Made messaging more clear on how to install wandb and linked to more documentation
3. Added an environment variable for disabling wandb.  This is important for users using fastai or lightning with their own wandb callback.
4. Added an environment variable for disabling wandb.watch.  Because we're pulling gradients and parameters during training there is a small performance impact on training.  We wanted to make it easy for users to disable this feature.

Happy to iterate on any feedback you guys have.  Really excited about this integration!


Perplexity in T5/BART

Hi,
Is there a way to compute perplexity for T5 or BART outputs?


Fix for IndexError when Roberta Tokenizer is called on empty text

Check if `text` is set to avoid IndexError

Fix for https://github.com/huggingface/transformers/issues/3809


Remove comma that turns int into tuple

cf @stefan-it's error on Slack


BertForTokenClassification, logits on sequence tagging task

# ❓ Questions & Help

## Details
<!-- Description of your issue -->
I'm trying to use BertForTokenClassification to do a sequence tagging task. During training, everything seems fine and loss is decreasing per epoch, but when I put the model in evaluation mode and feed it inputs, the logits for each token are all the same.

My code looks like this:
```
class PretrainedBert(nn.Module):

    def __init__(self,
                 device,
                 pretrained_path='bert-base-multilingual-cased',
                 output_dim=5,
                 learning_rate=0.01,
                 eps=1e-8,
                 max_grad_norm=1.0,
                 model_name='bert_multilingual',
                 cased=True
                 ):
        super(PretrainedBert, self).__init__()

        self.device = device
        self.output_dim = output_dim
        self.learning_rate = learning_rate
        self.max_grad_norm = max_grad_norm
        self.model_name = model_name
        self.cased = cased

        # Load pretrained bert
        self.bert = BertForTokenClassification.from_pretrained(pretrained_path, num_labels=output_dim, output_attentions = False, output_hidden_states = False)
        self.tokenizer = BertTokenizer.from_pretrained(pretrained_path, do_lower_case=not cased)

        param_optimizer = list(self.bert.named_parameters())
        no_decay = ['bias', 'gamma', 'beta']
        optimizer_grouped_parameters = [
            {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
             'weight_decay_rate': 0.01},
            {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
             'weight_decay_rate': 0.0}]
        self.optimizer = AdamW(optimizer_grouped_parameters, lr=learning_rate, eps=eps)

    def forward(self, input_ids, attention_mask, labels=None):
        output = self.bert(input_ids, attention_mask=attention_mask, labels=labels)
        return output

    def fit(self, train_loader, dev_loader, epochs=10):
        """
        :param test_loader: torch.utils.data.DataLoader object
        :param test_loader: torch.utils.data.DataLoader object
        :param epochs: int
        """
        scheduler = get_linear_schedule_with_warmup(self.optimizer, num_warmup_steps=0,num_training_steps=len(train_loader)*epochs)

        for epoch in range(epochs):
            # Iterate over training data
            epoch_time = time.time()
            self.train()
            epoch_loss = 0
            num_batches = 0

            for raw_text, x, y, idx in tqdm(train_loader):
                self.zero_grad()

                # Get gold labels and extend them to cover the wordpieces created by bert
                y = self._extend_labels(raw_text, y)
                y, _ = pad_packed_sequence(y, batch_first=True)
                batches_len, seq_length = y.size()
                y.to(self.device)

                # Run input through bert and get logits
                bert_tokenized = self.tokenizer.batch_encode_plus([' '.join(
                    text) for text in raw_text], max_length=seq_length, pad_to_max_length=True)

                input_ids, token_type_ids, attention_masks = bert_tokenized[
                    'input_ids'], bert_tokenized['token_type_ids'], bert_tokenized['attention_mask']

                loss, logits = self.forward(torch.LongTensor(
                    input_ids), torch.LongTensor(attention_masks), labels=y)

                # Help prevent the "exploding gradients" problem.
                torch.nn.utils.clip_grad_norm_(parameters=self.bert.parameters(), max_norm=self.max_grad_norm)

                # update parameters and learning rate
                loss.backward()
                self.optimizer.step()
                scheduler.step()

                epoch_loss += loss.item()
                num_batches += 1

            epoch_time = time.time() - epoch_time

            print()
            print("Epoch {0} loss: {1:.3f}".format(epoch + 1,
                                                   epoch_loss / num_batches))

            print("Dev")
            binary_f1, propor_f1 = self.evaluate(dev_loader)

    def predict(self, test_loader):
        """
        :param test_loader: torch.utils.data.DataLoader object with
                            batch_size=1
        """
        self.eval()
        predictions, golds, sents = [], [], []

        for raw_text, x, y, idx in tqdm(test_loader):
            # Run input through bert and get logits
            bert_tokenized = self.tokenizer.encode_plus(' '.join(raw_text[0]))

            input_ids, token_type_ids, attention_masks = bert_tokenized[
                'input_ids'], bert_tokenized['token_type_ids'], bert_tokenized['attention_mask']

            with torch.no_grad():
                logits = self.forward(torch.LongTensor(
                    input_ids).unsqueeze(0), torch.LongTensor(attention_masks).unsqueeze(0))

            # remove batch dim and [CLS]+[SEP] tokens from logits
            logits = logits[0].squeeze(0)[1:-1:]

            # mean pool wordpiece rows of logits and argmax to get predictions
            preds = self._logits_to_preds(logits, y[0], raw_text[0])

            predictions.append(preds)
            golds.append(y[0])
            sents.append(raw_text[0])

        return predictions, golds, sents

    def evaluate(self, test_loader):
        """
        Returns the binary and proportional F1 scores of the model on the examples passed via test_loader.
        :param test_loader: torch.utils.data.DataLoader object with
                            batch_size=1
        """
        preds, golds, sents = self.predict(test_loader)

        flat_preds = [int(i) for l in preds for i in l]
        flat_golds = [int(i) for l in golds for i in l]

        analysis = get_analysis(sents, preds, golds)
        binary_f1 = binary_analysis(analysis)
        propor_f1 = proportional_analysis(flat_golds, flat_preds)

        return binary_f1, propor_f1

    def _extend_labels(self, text, labels):
        extended_labels = []

        for idx, (text_seq, label_seq) in enumerate(zip(text, labels)):
            extended_seq_labels = []

            for word, label in zip(text_seq, label_seq):
                n_subwords = len(self.tokenizer.tokenize(word))
                extended_seq_labels.extend([label.item()] * n_subwords)

            extended_labels.append(torch.LongTensor(extended_seq_labels))

        extended_labels = pack_sequence(extended_labels, enforce_sorted=False)

        return extended_labels

    def _logits_to_preds(self, logits, y, raw_text):
        preds = torch.zeros(len(y))
        head = 0
        for idx, word in enumerate(raw_text):
            n_subwords = len(self.tokenizer.tokenize(word))

            if n_subwords > 1:
                preds[idx] = torch.argmax(torch.mean(
                    logits[head:head+n_subwords, :], dim=0))
            else:
                preds[idx] = torch.argmax(logits[head, :])

            head = head + n_subwords

        return preds
```

Example input:

```
raw_text = ['Donkey', 'Kong', 'Country', ':']
y =  [tensor([0, 0, 0, 0])]
```

Produces these results:
```
tokenized_words = ['Don', '##key', 'Kong', 'Country', ':']

logits = tensor([[-3.2811,  1.1715,  1.2381,  0.5201,  1.0921],
        [-3.2813,  1.1715,  1.2382,  0.5201,  1.0922],
        [-3.2815,  1.1716,  1.2383,  0.5202,  1.0923],
        [-3.2814,  1.1716,  1.2383,  0.5202,  1.0922],
        [-3.2811,  1.1715,  1.2381,  0.5201,  1.0921]])

preds = tensor([2., 2., 2., 2.])
```

Any ideas for what I'm doing wrong? Thanks for any help in advance.


# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
How to get output from BERTNextSentencePrediction by passing One Sentence and Next Sentence getting predicted ?

I got the Prediction score but how and where should I pass the first sentence so I can get the next sentence predicted ?

Codefrom torch.nn.functional import softmax
from transformers import BertForNextSentencePrediction, BertTokenizer

seq_A = 'I like cookies !'
seq_B = 'Do you like them ?'

# load pretrained model and a pretrained tokenizer
model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

# encode the two sequences. Particularly, make clear that they must be
# encoded as "one" input to the model by using 'seq_B' as the 'text_pair'
encoded = tokenizer.encode_plus(seq_A, text_pair=seq_B, return_tensors='pt')
print(encoded)
# {'input_ids': tensor([[  101,   146,  1176, 18621,   106,   102,  2091,  1128,  1176,  1172, 136,   102]]),
#  'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]]),
#  'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
# NOTE how the token_type_ids are 0 for all tokens in seq_A and 1 for seq_B,
# this way the model knows which token belongs to which sequence

# a model's output is a tuple, we only need the output tensor containing
# the relationships which is the first item in the tuple
seq_relationship_logits = model(**encoded)[0]

# we still need softmax to convert the logits into probabilities
# index 0: sequence B is a continuation of sequence A
# index 1: sequence B is a random sequence
probs = softmax(seq_relationship_logits, dim=1)

print(seq_relationship_logits)
print(probs)
# tensor([[9.9993e-01, 6.7607e-05]], grad_fn=<SoftmaxBackward>)
# very high value for index 0: high probability of seq_B being a continuation of seq_A

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Issues with RobertaTokenizer and unicode characters

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Roberta

Language I am using the model on (English, Chinese ...):
English

## To reproduce

Steps to reproduce the behavior:

1. Insert a jumble of some weird unicode characters into the vocabulary file. These unicode characters must exist on their own somewhere else in the vocab. (Note: I didn't actually do this manual vocab editing when I encountered the bug. Since I'm using a custom vocabulary I'm simply detailing the steps to reproduce the behavior)
```
# vocab.txt
...
ĠOHÃ¢
...
```
(Let's say this was inserted on line 1001)
(Don't forget to account for this extra word with a change in vocab size in your config file)

2. Load an instance of `RobertaTokenizer` with this vocab file.
3. Call
```python3
input_ids = tokenizer.encode_plus('OHÃ¢',
                                  add_special_tokens=True,
                                  add_prefix_space=True,
                                  max_length=max_seq_length,
                                  return_token_type_ids=True)['input_ids']
print(input_ids)
```

## Expected behavior

The special start-of-word character `Ġ` will be added by the tokenizer, so the input ids we'd expect would be `[0, 1000, 2]` (given that `0` and `2` correspond to your start/end tokens, and `1000` corresponds to the index where we inserted the unicode jumble `ĠOHÃ¢`). Instead, I'm getting some seemingly unrelated input ids corresponding to other weird unicode characters not included in the inserted unicode jumble.

## Environment info
- `transformers` version: 2.8
- Platform: Linux and MacOS
- Python version: 3.7
- PyTorch version (GPU?): torch 1.4.
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


KeyError with a fine-tuned model

I've fine-tuned the allenai's scibert to do some NER using the `run_ner.py` script, but I keep bumping into a key error. My code is the following:
```
m = AutoModel.from_pretrained("models/sciBERT_ner_e50/")
tok = AutoTokenizer.from_pretrained("allenai/scibert_scivocab_uncased")
nlp = pipeline("ner", model=m, tokenizer=tok)
```
where models/scibert_ner_e50 contains my model and the files it needs to run properly.

when i run something like `nlp(abstract)`, the code crashes and sends me the following error:
```
KeyError                                  Traceback (most recent call last)
<ipython-input-86-793f89d86633> in <module>
      3         if(len(row.abstract.split())<512):
      4             abst = row.abstract
----> 5             print(nlp(abst))
      6
      7     except ValueError as ve:

c:\users\tbabi\appdata\local\programs\python\python37\lib\site-packages\transformers\pipelines.py in __call__(self, *texts, **kwargs)
    792             answer = []
    793             for idx, label_idx in enumerate(labels_idx):
--> 794                 if self.model.config.id2label[label_idx] not in self.ignore_labels:
    795                     answer += [
    796                         {

KeyError: 422
```

Has anyone else been meeting this error ? I've been struggling with this for quiet some time now, and any advice/thoughts/insights on what's going on would be greatly appreciated :)

Thank you !


Use with_suffix to change the extension of the path

As per https://github.com/huggingface/transformers/pull/3934#discussion_r421307659. Note that this does not rename the file, it simply changes the path 'string' to another one with a different suffix.


Create README.md

model card

Related to pretrained model cards


Ensure fast tokenizer can construct single-element tensor without pad token


[Proposal] Small HfAPI Cleanup

- Implement __dict__ and __str__ so that one need not guess the class attributes of `ModelInfo` and `S3Object`
- Delete redundant `S3Obj` class. (`S3Object` seems to be a superset).
- Rename to `HFApi`?

Thoughts @julien-c @LysandreJik ?


[README] Corrected some grammatical mistakes

Corrected some grammatical mistakes, like added required punctuation marks, corrected singular-plural mistakes, etc.


[Benchmark] Memory benchmark utils

## Description

Currently, we have a benchmarks file in examples `examples/benchmarks.py` which measures time and speed for inference and one `src/transformers/benchmark_utils.py` file which including the memory tracing from @thomwolf.

This PR generalizes the benchmarks much more:
- We have `benchmark_args_utils, benchmark_args, benchmark_tf_args` similar to `trainer_args`. Because there is such a big overlap between tf and torch, I decided to have a`benchmark_args_utils`
- We have a `benchmark_utils` including the previous code of `benchmark_utils` and a new abstract`Benchmark` class. `PyTorchBenchmark` and `TensorflowBenchmark` inherit from this class and implement the abstract methods `inference` and `train`.
- Instead of being able to just measure our "default" models identified by their `model_name`, we can create a special config and measure the memory and time usage of the corresponding model before starting to train
- In addition to inference we can also measure train
- **IMPORTANT**: The line-by-line tracing implemented by @thomwolf is super useful to optimize one's code and see exactly which line produces most of the code, but it does not give the peek memory usage as one can see in `nvidia-smi` and which is relevant to decide how much RAM is needed for the model. I decided to not use the line-by-line tracing to plot the overall memory usage on GPU. It should rather only be used when one want to optimize his/her code instead of benchmarking the model. For Benchmarks the PyTorch method `torch.cuda.max_memory_reserved` is a better fit - it's much faster and gives the correct value.
- Also I added a `run_script` and a `plot_script` to the examples.
- @julien-c - I think it would be super nice to have such plots on our website somewhere. If people can see quickly how much more efficient model x is over y in for a given batch size and sequence length, I think it would be super useful.

## TODO:
- [x] Write tests
- [x] Make pretty (more type hints + smaller refactoring)

## Review

@thomwolf, @julien-c - It would be awesome if you could take a look here and check if you are ok with the general design choice. To get a better feeling how the Benchmark can be used, you can checkout this notebook: https://colab.research.google.com/drive/1GM3WPXmZ5wZHwhdPyf7RykJS3Y12iceJ?usp=sharing

## UPDATE

I think the PR is ready for merge now. The notebook: https://colab.research.google.com/drive/1GM3WPXmZ5wZHwhdPyf7RykJS3Y12iceJ?usp=sharing is updated and shows what environment and version infos are shown @thomwolf . Tests are included. The next PR should take care of pytorch jit tracing and TPU and TF @LysandreJik @julien-c

## Future PR:
- [ ] Support Pytorch Tracing (discuss with Lysandre a bit before)
- [ ] Support TPU (discuss with Lysandre a bit before)
- [ ] TF version