KeyError in Camembert in QuestionAnsweringPipeline

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
`Camembert ("illuin/camembert-large-fquad")`
Language I am using the model on (English, Chinese ...):
`French`
The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
Load model and create a question answering pipeline
Steps to reproduce the behavior:

1.
```
from transformers import (QuestionAnsweringPipeline, CamembertForQuestionAnswering, CamembertModel, CamembertTokenizer)
```
2.
```
QA_model = "illuin/camembert-large-fquad"
CamTokQA = CamembertTokenizer.from_pretrained(QA_model)
CamQA = CamembertForQuestionAnswering.from_pretrained(QA_model)
```
3.
```
device_pipeline = 0 if torch.cuda.is_available() else -1
q_a_pipeline = QuestionAnsweringPipeline(model=CamQA,
                                         tokenizer=CamTokQA,
                                         device=device_pipeline)
```
4.
```
res = q_a_pipeline({'question': question, 'context': ctx})
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
  File "/mnt/Documents/Projets/BotPress/R_D/R_D_q_a/sdk/readers.py", line 15, in get_answers
    res = q_a_pipeline({'question': question, 'context': ctx})
  File "/home/pedro/.local/lib/python3.8/site-packages/transformers/pipelines.py", line 1213, in __call__
    answers += [
  File "/home/pedro/.local/lib/python3.8/site-packages/transformers/pipelines.py", line 1216, in <listcomp>
    "start": np.where(char_to_word == feature.token_to_orig_map[s])[0][0].item(),
KeyError: 339
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Get answer from the Q_a pipeline

*Is working on transformers version 2.8.0* (but not on next ones)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.10.0
- Platform: Linux-5.6.14-arch1-1-x86_64-with-glibc2.2.5
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): 2.2.0-rc4 (False)
- Using GPU in script?: Yes (but same problem with cpu only)
- Using distributed or parallel set-up in script?: No


QUESTION: How do I know what type of positional encoding to input during fine-tuning or pretrained BERT?

Hello HuggingFace team,

I am familiarizing myself with the HuggingFace tutorials and understand the functionality of the various methods. However, I have a general question for example when using models like BERT.

Considering that I am doing sentiment classification and I want to fine-tune the whole BERT based on pre-trained weights, how do I know what should be the positional encoding as input during the `forward ( )` method?

I know it has a default value of None, but doesn't it mean that during fine-tuning I need to input the same value that it was originally trained on in the first place? If so, how do I know what it was trained on during its original training from scratch? Is there a documentation for that somewhere?

Following to this, if I am freezing the weights of BERT for sentiment classification, I again have the same question on what should be my positional encoding input to the `forward ( )` method.

Please clarify this. Thanks for your time!


[Longformer] Better handling of global attention mask vs local attention mask

This PR extends Longformer's API to also take a `global_attention_mask` besides the usual `attention_mask` as an input as discussed with @thomwolf @ibeltagy.

Docs are updated.


get_from_cache in file_utils.py gobbles up error in making url requests

# 🐛 Bug
No information from the package on SSL error encountered, making it difficult to troubleshoot or figure out a workaround

## Information
When trying to do:
`TFAutoModelWithLMHead.from_pretrained("t5-small")`

Get an error:
`TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType`

The above is a result of an SSL error encountered when trying to fetch the model, however, since the exception handling isn't proper within file_utils.py I don't come to know of it, unless I debug.

Model I am using (Bert, XLNet ...): T5

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

https://huggingface.co/transformers/usage.html#summarization

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x] my own task or dataset: (give details below)
Just getting familiar with transformers for summarization

## To reproduce
You need a machine with an expired certificate for proxy etc.

Steps to reproduce the behavior:

1. See information above

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
If there has been an issue in fetching the pre-trained model from s3 bucket etc. I should get an error to that effect.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.10.0
- Platform: Windows 10
- Python version: 3.7.4
- PyTorch version (GPU?): NA
- Tensorflow version (GPU?): 2.0.0 (Yes)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Coversion between tokenizers

Core: Tokenization

Internals of the library; Tokenization.

I want to convert GPT2 tokens to BERT tokens. Is there any API that can directly convert between the Tokenizers?


Cannot load labels from old models

Core: Modeling

Internals of the library; Models.

# ❓ Questions & Help

If I load a model from 2.8 or older in 2.9 o newer, the labels from my model are changed automatically so all the tests in my code start to fail because instead of predicting `I-PER` it predicts `LABEL_2`.

After reading the source code I think I found what happens but I'm not quite sure.  I think it all started here in #3967

https://github.com/huggingface/transformers/blob/e7cfc1a313cc928e962bb8699868f5dcf46f11eb/src/transformers/configuration_utils.py#L123

In the code above you can see that the labels dicts are modified if we set `num_labels`. I couldn't find the place in the code where that is done, but it definetly modifies 2 attributes from the class. I don't really think that a setter for attribute A, should change attributes B and C.

Am I missing something when loading my models?

Thanks you so much for reading and for the library, we all love it <3


Colab crashes due to tcmalloc large allocation

I am pretraining a RoBERTa model on the Newsroom dataset on colab. I have trained a custom tokenizer on the text data. I am using the Text Dataset LinebyLineTextDataset as I have a single file and each line is the text of a news article. The colab crashes when I run this code
_%%time
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(tokenizer=tokenizer,
                                file_path="/content/drive/My Drive/Newsroom Dataset/newsroom-firsthalf.txt",
                                 block_size=128)_

I tried with the full dataset and reduced it to half and have also tried it by reducing the block size.
The config is
_config = RobertaConfig(
    vocab_size=52000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1)_

and the error log is
_tcmalloc: large alloc 7267041280 bytes == 0x9d916000 @ 0x7f3ea16311e7 0x5aca9b 0x4bb106 0x5bcf53 0x50a2bf 0x50bfb4 0x507d64 0x509042 0x594931 0x549e5f 0x5513d1 0x5a9cbc 0x50a5c3 0x50cd96 0x507d64 0x516345 0x50a2bf 0x50bfb4 0x507d64 0x588d41 0x59fc4e 0x50d356 0x507d64 0x509a90 0x50a48d 0x50bfb4 0x507d64 0x509a90 0x50a48d 0x50bfb4 0x509758._

Additional Notes: The increase RAM message doesnt come when colab crashes so I am essentially working 12.72 GB RAM.
Please help me


HooshvareLab readme parsbert-peymaner

model card

Related to pretrained model cards

Readme for HooshvareLab/bert-base-parsbert-peymaner-uncased


HooshvareLab readme parsbert-armananer

Related to pretrained model cards

Readme for HooshvareLab/bert-base-parsbert-armananer-uncased


HooshvareLab readme parsbert-ner

Related to pretrained model cards

Readme for HooshvareLab/bert-base-parsbert-ner-uncased


run_tf_ner.py TFTrainer logdir cannot be none

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): bert

Language I am using the model on (English, Chinese ...): bert-base-multilingual-cased

The problem arises when using:
* [x] the official example scripts: (give details below)
run_tf_ner.py

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
germeval2014ner
* [ ] my own task or dataset: (give details below)

## To reproduce
run_tf_ner.py (original)
run.sh (see below)

Steps to reproduce the behavior:

1.
run.sh

```
export MAX_LENGTH=128
export BERT_MODEL=bert-base-multilingual-cased
export OUTPUT_DIR=germeval-model
export BATCH_SIZE=32
export NUM_EPOCHS=1
export SAVE_STEPS=750
export SEED=1
export data_dir=data

python run_tf_ner.py --data_dir ./data/ \
--labels ./data/labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_gpu_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_train \
--do_eval \
--do_predict
```

2. error message
```
Traceback (most recent call last):
  File "run_tf_ner.py", line 295, in <module>
    main()
  File "run_tf_ner.py", line 220, in main
    compute_metrics=compute_metrics,
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 48, in __init__
    self._setup_training()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 65, in _setup_training
    self._create_summary_writer()
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py", line 88, in _create_summary_writer
    self.writer = tf.summary.create_file_writer(self.args.logging_dir)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/summary_ops_v2.py", line 377, in create_file_writer_v2
    raise ValueError("logdir cannot be None")
ValueError: logdir cannot be None
```

## Expected behavior
I did not change the code or the data. I was trying to reproduce exactly the German NER tf2.0 example: [https://github.com/huggingface/transformers/tree/master/examples/token-classification](url)
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
google colab gpu
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?): tensorflow gpu 2.2.0
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?: no


End-to-end object detection with Transformers

# 🚀 Feature request

A modular Transformer Encoder-Decoder block that can be attached to ConvNets for many tasks.

## Motivation
As shown in the recent paper [End-to-end Object Detection with Transformers](https://ai.facebook.com/research/publications/end-to-end-object-detection-with-transformers) which used Transformer for Object Detection. [https://github.com/facebookresearch/detr](https://github.com/facebookresearch/detr)
## Your contribution


run_tf_ner.py cannot run

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...):bert-base-multilingual-cased

The problem arises when using:
* [ ] the official example scripts: (give details below)
original code in _run_tf_ner.py_
original data from _germaeval2014eval_
original data process from _preprocess.py_

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
_germaeval2014eval_ NER
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. this is the structure
--data
---train.txt
---dev.txt
---test.txt
---label.txt
run.sh (see below)
run_tf_ner.py (orignal)
utils_ner.py (original)
preprocess.py(original)

2. run.sh
export MAX_LENGTH=128
export BERT_MODEL=bert-base-multilingual-cased
export OUTPUT_DIR=germeval-model
export BATCH_SIZE=32
export NUM_EPOCHS=1
export SAVE_STEPS=750
export SEED=1
export data_dir=data

python3 run_tf_ner.py \
--data_dir . \
--labels $data_dir/labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_gpu_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_train \
--do_eval \
--do_predict

3. sh run.sh will return the invalid configuration error

```
--05/29/2020 11:21:48 - INFO - __main__ -   n_gpu: 2, distributed training: True, 16-bits training: False
05/29/2020 11:21:48 - INFO - __main__ -   Training/evaluation parameters TFTrainingArguments(output_dir='germeval-model', overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=False, per_gpu_train_batch_size=32, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=750, save_total_limit=None, no_cuda=False, seed=1, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, optimizer_name='adam', mode='text-classification', loss_name='SparseCategoricalCrossentropy', tpu_name=None, end_lr=0, eval_steps=1000, debug=False)
05/29/2020 11:21:50 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json from cache at /home/ll/.cache/torch/transformers/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0
05/29/2020 11:21:50 - INFO - transformers.configuration_utils -   Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "B-LOC",
    "1": "B-LOCderiv",
    "2": "B-LOCpart",
    "3": "B-ORG",
    "4": "B-ORGderiv",
    "5": "B-ORGpart",
    "6": "B-OTH",
    "7": "B-OTHderiv",
    "8": "B-OTHpart",
    "9": "B-PER",
    "10": "B-PERderiv",
    "11": "B-PERpart",
    "12": "I-LOC",
    "13": "I-LOCderiv",
    "14": "I-LOCpart",
    "15": "I-ORG",
    "16": "I-ORGderiv",
    "17": "I-ORGpart",
    "18": "I-OTH",
    "19": "I-OTHderiv",
    "20": "I-OTHpart",
    "21": "I-PER",
    "22": "I-PERderiv",
    "23": "I-PERpart",
    "24": "O"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "label2id": {
    "B-LOC": 0,
    "B-LOCderiv": 1,
    "B-LOCpart": 2,
    "B-ORG": 3,
    "B-ORGderiv": 4,
    "B-ORGpart": 5,
    "B-OTH": 6,
    "B-OTHderiv": 7,
    "B-OTHpart": 8,
    "B-PER": 9,
    "B-PERderiv": 10,
    "B-PERpart": 11,
    "I-LOC": 12,
    "I-LOCderiv": 13,
    "I-LOCpart": 14,
    "I-ORG": 15,
    "I-ORGderiv": 16,
    "I-ORGpart": 17,
    "I-OTH": 18,
    "I-OTHderiv": 19,
    "I-OTHpart": 20,
    "I-PER": 21,
    "I-PERderiv": 22,
    "I-PERpart": 23,
    "O": 24
  },
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 119547
}

05/29/2020 11:21:51 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-config.json from cache at /home/ll/.cache/torch/transformers/45629519f3117b89d89fd9c740073d8e4c1f0a70f9842476185100a8afe715d1.65df3cef028a0c91a7b059e4c404a975ebe6843c71267b67019c0e9cfa8a88f0
05/29/2020 11:21:51 - INFO - transformers.configuration_utils -   Model config BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 119547
}

05/29/2020 11:21:52 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt from cache at /home/ll/.cache/torch/transformers/96435fa287fbf7e469185f1062386e05a075cadbf6838b74da22bf64b080bc32.99bcd55fc66f4f3360bc49ba472b940b8dcf223ea6a345deb969d607ca900729
05/29/2020 11:21:54 - INFO - transformers.modeling_tf_utils -   loading weights file https://cdn.huggingface.co/bert-base-multilingual-cased-tf_model.h5 from cache at /home/ll/.cache/torch/transformers/273ed844d60ef1d5a4ea8f7857e3c3869d05d7b22296f4ae9bc56026ed40eeb7.1b4841f14bf42137fc7ecee17a46c1b2f22b417f636347e4b810bd06dd9c45ea.h5
2020-05-29 11:21:55.520823: F ./tensorflow/core/kernels/random_op_gpu.h:232] Non-OK-status: GpuLaunchKernel(FillPhiloxRandomKernelLaunch<Distribution>, num_blocks, block_size, 0, d.stream(), gen, data, size, dist) status: Internal: invalid configuration argument
Aborted (core dumped)
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I was trying to run the training of transformer ner extraction

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Ubuntu 16.04.4 LTS
- Python version: Python 3.6.10 :: Anaconda, Inc.
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.2.0 GPU
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:


Write With Transformer: PPLM page is broken

Need more information

Further information is requested

# 🐛 Bug

Triggering autocomplete results in endless spinning.

iOS safari But also desktop Safari, Firefox and Chrome.


Assert message error in Reformer chunking

reformer

Everything related to the reformer model

# 🐛 Bug

In the function `apply_chunking_to_forward`, an assertion checking the input tensor shape is trying to print the contents of the tensor itself instead of its shape:

https://github.com/huggingface/transformers/blob/b5015a2a0f4ea63035a877f5626cb0c3ce97e25d/src/transformers/modeling_utils.py#L2195

I'm pretty sure that line should be `input_tensors[0].shape[chunk_dim], chunk_size`


Add support for gradient checkpointing in BERT

This PR adds support for gradient checkpointing in `modeling_bert.py` to save memory at training time at the expense of a slower backward pass. This is particularly useful if we want to pretrain a version of BERT for sequences longer than 512. It is also useful for long-document models like Longformer.

Stats:
```
Forward/backward - no grad checkpointing: 40.1GB memory, 25.3 seconds.
Forward/backward - with grad checkpointing: 8.2GB memory (~5x less), 33.5 seconds (~1.3x more)
Forward pass only - with/without gradient checkpointing: 4GB memory, 6.1 seconds.
```


Add upcoming GPT-3 model

# 🌟 New model addition

## Model description

The GPT-3 paper just landed on ArXiv: https://arxiv.org/abs/2005.14165.

Would be great to integrate it into Transformers, whenever models are available.

> Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general.

## Open source status

* [x] GitHub repository is available: [here](https://github.com/openai/gpt-3)
* [ ] the model implementation is available: (give details)
* [ ] the model weights are available: (give details)
* [ ] who are the authors: (mention them, if possible by @gh-username)


--fp causes an issue when running example scripts in distributed mode

PyTorch

Anything PyTorch

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
`roberta-large`
Language I am using the model on (English, Chinese ...):
`English`

The problem arises when using:
* the official example scripts

The tasks I am working on is:
* Finetuning a LM with `run_language_modeling.py` and the SST-2 task with `run_glue.py`
* my own dataset

## To reproduce
If I run either of the following commands, I get the error included below. However, if I remove `--fp`, everything works normally. Also, if I add `--fp`, but run it non-distributed, everything works normally. So, it appears there is an issue with my running `-fp`  in a distributed fashion. I haven't had an issue with this before; so, I'm not sure what the problem is. Any ideas? Thanks in advance.

I installed apex in two different way, but still get the same results.
```
#Install package required for fp16 computations
RUN git clone https://github.com/NVIDIA/apex.git \
    && cd apex \
    && python3 setup.py install --cuda_ext --cpp_ext
```
```
Install package required for fp16 computations
RUN git clone https://github.com/NVIDIA/apex.git \
    && cd apex \
    && pip3 install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```
```
python3 -m torch.distributed.launch --nproc_per_node 2 run_language_modeling.py --output_dir=/ptcc/shared/lm_roberta_20200528_164228 --model_type=roberta --do_train --train_data_file=/ptcc/data/train.txt --do_eval --eval_data_file=/ptcc/data/test.txt --evaluate_during_training --per_gpu_train_batch_size=2 --per_gpu_eval_batch_size=2 --learning_rate=5e-06 --model_name_or_path=roberta-large --mlm --max_steps=120000 --warmup_steps=10000 --save_steps=12000 --seed=42 --fp16 --logging_dir=/ptcc/shared/roberta_20200528_164228_tf_logs'
```
```
python3 -m torch.distributed.launch --nproc_per_node 2 run_glue.py --model_type roberta --task_name SST-2 --do_train --do_eval --evaluate_during_training --data_dir /ptcc/data/ --per_gpu_train_batch_size 2 --per_gpu_eval_batch_size 2 --learning_rate 1e-06 --output_dir clf_roberta_20200528_162937 --model_name_or_path /ptcc/shared/lm_roberta_20200528_113420 --num_train_epochs 2.0 --save_steps 1000 --seed 42 --fp16 --logging_dir=/ptcc/shared/roberta_20200528_162937_tf_logs
```

```
ptcc_1  | 05/28/2020 20:30:38 - INFO - transformers.trainer -     Starting fine-tuning.
Epoch:   0%|          | 0/2 [00:00<?, ?it/s]       Traceback (most recent call last):
ptcc_1  |   File "/ptcc/run_glue.py", line 228, in <module>
ptcc_1  |     main()
ptcc_1  |   File "/ptcc/run_glue.py", line 160, in main
ptcc_1  |     model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
ptcc_1  |   File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 470, in train
ptcc_1  |     tr_loss += self._training_step(model, inputs, optimizer)
ptcc_1  |   File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 577, in _training_step
ptcc_1  |     scaled_loss.backward()
ptcc_1  |   File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
ptcc_1  |     next(self.gen)
ptcc_1  |   File "/usr/local/lib/python3.6/dist-packages/apex-0.1-py3.6-linux-x86_64.egg/apex/amp/handle.py", line 127, in scale_loss
ptcc_1  |     should_skip = False if delay_overflow_check else loss_scaler.update_scale()
ptcc_1  |   File "/usr/local/lib/python3.6/dist-packages/apex-0.1-py3.6-linux-x86_64.egg/apex/amp/scaler.py", line 200, in update_scale
ptcc_1  |     self._has_overflow = self._overflow_buf.item()
ptcc_1  | RuntimeError: CUDA error: an illegal memory access was encountered
ptcc_1  | /usr/local/lib/python3.6/dist-packages/torch/optim/lr_scheduler.py:114: UserWarning: Seems like `optimizer.step()` has been overridden after learning rate scheduler initialization. Please, make sure to call `optimizer.step()` before `lr_scheduler.step()`. See more details at https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate
ptcc_1  |   "https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate", UserWarning)
ptcc_1  |                                                  terminate called after throwing an instance of 'c10::Error'
ptcc_1  |   what():  CUDA error: an illegal memory access was encountered (insert_events at /pytorch/c10/cuda/CUDACachingAllocator.cpp:771)
ptcc_1  | frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x46 (0x7f69777f6536 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
ptcc_1  | frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x7ae (0x7f6977a39fbe in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
ptcc_1  | frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f69777e6abd in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
ptcc_1  | frame #3: std::vector<c10d::Reducer::Bucket, std::allocator<c10d::Reducer::Bucket> >::~vector() + 0x1d9 (0x7f69c3926ef9 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #4: c10d::Reducer::~Reducer() + 0x23a (0x7f69c391c84a in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #5: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x12 (0x7f69c38fb7c2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #6: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7f69c32be466 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #7: <unknown function> + 0x87146b (0x7f69c38fc46b in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #8: <unknown function> + 0x240500 (0x7f69c32cb500 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #9: <unknown function> + 0x24174e (0x7f69c32cc74e in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
ptcc_1  | frame #10: /usr/bin/python3() [0x572a27]
ptcc_1  | frame #11: /usr/bin/python3() [0x54eef2]
ptcc_1  | frame #12: /usr/bin/python3() [0x588948]
ptcc_1  | frame #13: /usr/bin/python3() [0x5ad438]
ptcc_1  | frame #14: /usr/bin/python3() [0x5ad44e]
ptcc_1  | frame #15: /usr/bin/python3() [0x5ad44e]
ptcc_1  | frame #16: /usr/bin/python3() [0x56b276]
ptcc_1  | frame #17: PyDict_SetItemString + 0x153 (0x5709f3 in /usr/bin/python3)
ptcc_1  | frame #18: PyImport_Cleanup + 0x76 (0x4f2fc6 in /usr/bin/python3)
ptcc_1  | frame #19: Py_FinalizeEx + 0x5e (0x637e2e in /usr/bin/python3)
ptcc_1  | frame #20: Py_Main + 0x395 (0x638e95 in /usr/bin/python3)
ptcc_1  | frame #21: main + 0xe0 (0x4b0d00 in /usr/bin/python3)
ptcc_1  | frame #22: __libc_start_main + 0xe7 (0x7f69e4727b97 in /lib/x86_64-linux-gnu/libc.so.6)
ptcc_1  | frame #23: _start + 0x2a (0x5b250a in /usr/bin/python3)
```

## Environment info
- `transformers` version: 2.10.0
- Platform: Linux-5.3.0-26-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Y,  2 Tesla V100-SXM2
- Using distributed or parallel set-up in script?: Y,  2 Tesla V100-SXM2


Electra training from scratch

# Electra Trainer

_**Still in testing process. Feedback welcome!**_

This PR introduces:

## New features
- A new language modeling script based on the [ELECTRA pre-training method](https://github.com/google-research/electra).

### Combined model

- Combines `ElectraForMaskedLM` and `ElectraForPreTraining` with embedding sharing + custom masking/replaced token detection

### Lazy Dataset for OpenWebText

- Tokenizes text into multiple files
- Lazy loads files into memory

## Trainer

- Introduces `IterableDataset` handling to the trainer
- New evaluation [_to be discussed_]:

Up to now the evaluation was only possible when the trainer was provided `preds` and `label_ids`. The `compute_metrics` function allowed the user to compute specific metrics, but not to customize which inputs to use for this function.

This re-vamped evaluation works with DataParallel and with TPU.

- Better logging when a limiting a training to a specific amount of steps (both training and evaluation)
- `max_eval_steps` flag

## Bugfix
- Fixes a bug in the ELECTRA model for batch sizes of 1

Left to do:

- [ ] Post wandb graphs here as they complete
- [ ] Allow all models to be used as discriminator/generator (same tokenizer, same embedding matrix)
- [ ] Better way to handle dataset building when using TPUs


Tokenization_utils doesn't work with Pytorch-Lightning on 2.10.0 version

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert with pytorch-lightning

Language I am using the model on (English, Chinese ...): English

The problem arises when using
*  my own modified scripts: Pytorch Dataset with tokenizer inside

The tasks I am working on is:
* my own task or dataset

## To reproduce

Take a look at [this colab link](https://colab.research.google.com/drive/1SH1xRzhNwgnSn382OLCoMFi_-8CCqk5y?usp=sharing).

I've copied the method from pytorch-lightning which shows an error on 2.10.0 transformers when process a batch. Doing the same with transformers 2.8.0 cause no error.

```
/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils.py in __getattr__(self, item)
    201
    202     def __getattr__(self, item: str):
--> 203         return self.data[item]
    204
    205     def keys(self):

KeyError: 'cuda'
```

## Expected behavior

No error

- `transformers` version: 2.10.0
- Platform: Linux
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


TfElectraForSequenceClassification

This pull request add functionality for sequence classification with Electra. The only missing bit is that I have put the activation function as "tanh" instead of "gelu" (in the orig. implementation)


[Longformer] fix model name in examples

This PR fixes the model identifier of Longformer to the new standard fomat <organisation/model_name>


[Community notebooks] add longformer-for-qa notebook

This PR adds a community notebook to showcase how to fine-tune Longformer for QA task.

@ibeltagy  @patrickvonplaten Please provide feedback if you think this notebook can be further improved.


Update modeling_electra for adding ability to using electra as decoder

Adapt electra for using as decoder(code for it borrowed from modeling_bert)


Allow pathlib.Path to be used on save_pretrained and save_vocabulary

Related to #4541

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


I added some small change(borrow it from modeling_bert). Now, electra able to work as decoder.


Update modeling_electra for using it as decoder

I add some adding(borrow it from modeling_bert). Now, electra able to work as decoder.


Encode-Decode after training, generation gives the same results regardless of the input

# ❓ Questions & Help

Hi, everyone. I need help with the encoding-decoding model. I'm trying to train the model to create a title for a small text.

I'm creating a basic Encode-Decode model with Bert

```
from transformers import EncoderDecoderModel, BertTokenizer
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')
```

After training on my data, when generate I get the same results independent of the input data in model.eval () mode. If you convert model to train, then different results will be generated.

The code I use for training.

```
tokenized_texts = [tokenizer.tokenize(sent) for sent in train_sentences]
tokenized_gt = [tokenizer.tokenize(sent) for sent in train_gt]

input_ids = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_texts]
input_ids = pad_sequences(
    input_ids,
    maxlen=max_len_abstract,
    dtype="long",
    truncating="post",
    padding="post"
)
attention_masks = [[float(i>0) for i in seq] for seq in input_ids]
input_ids_decode = [tokenizer.convert_tokens_to_ids(x) for x in tokenized_gt]
input_ids_decode = pad_sequences(
    input_ids_decode,
    maxlen=max_len_title,
    dtype="long",
    truncating="post",
    padding="post"
)

attention_masks_encode = [[float(i>0) for i in seq] for seq in input_ids]
attention_masks_decode = [[float(i>0) for i in seq] for seq in input_ids_decode]

input_ids = torch.tensor(input_ids)
input_ids_decode = torch.tensor(input_ids_decode)
attention_masks_encode = torch.tensor(attention_masks_encode)
attention_masks_decode = torch.tensor(attention_masks_decode)

train_data = TensorDataset(input_ids, input_ids_decode, attention_masks_encode, attention_masks_decode)
train_dataloader = DataLoader(train_data, sampler=RandomSampler(train_data), batch_size=4)

model.cuda()

param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'gamma', 'beta']
optimizer_grouped_parameters = [
    {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.01},
    {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
     'weight_decay_rate': 0.0}
]

optimizer = AdamW(optimizer_grouped_parameters, lr=2e-5)

model.train()
train_loss_set = []
train_loss = 0
for i in range(4):
    for step, batch in enumerate(train_dataloader):
        batch = tuple(t.to(device) for t in batch)
        b_input_ids, b_input_ids_de, b_attention_masks_encode, b_attention_masks_decode = batch
        optimizer.zero_grad()
        model.zero_grad()
        loss, outputs = model(input_ids=b_input_ids, decoder_input_ids=b_input_ids_de, lm_labels=b_input_ids_de)[:2]
        train_loss_set.append(loss.item())
        loss.backward()
        optimizer.step()
        train_loss += loss.item()

        clear_output(True)
        plt.plot(train_loss_set)
        plt.title("Training loss")
        plt.xlabel("Batch")
        plt.ylabel("Loss")
        plt.show()
        if step != 0 and step % 20 == 0:
              torch.save(model.state_dict(), model_weigth)
    print(f'Epoch {i}')
```
Maybe I'm doing something wrong? I would be grateful for any advice.


add longformer docs


[Longformer] Multiple choice for longformer

## Description:

- This PR adds Multiple Choice for Longformer as in: #4644 (Sorry for not telling you earlier @patil-suraj). @ibeltagy

- The documentation is updated for all models using MultipleChoice since their `input_ids` have 3 dimensions. It is done by use of `{}.format()` for `INPUTS_DOCSTRING`. @LysandreJik

- Adds a couple of models that were missing to the respective `models_page` @LysandreJik

@ibeltagy Regarding global attention - I think we probably should automatically add global attention here since the multiple inputs are flattened across the dimension `num_choices` and then should attend each other. Maybe add global attention always on the first token of each input of the dim `num_choices`?

- Global attention should probably still be implemented. Waiting for @ibeltagy answer.


LongformerForMultipleChoice

This PR adds `LongformerForMultipleChoice` following `RobertaForMultipleChoice`

@patrickvonplaten , @ibeltagy
Same question as before, do we need any automatic global attention here ?


question-answering examples bug in pipelines document

Regarding
https://github.com/huggingface/transformers/blob/96f57c9ccb6363623005fb3f05166dfd7acb3f53/src/transformers/pipelines.py#L1739

it will cause a reproducible bug:

```python
from transformers import pipeline
nlp_qa = pipeline('question-answering', model='distilbert-base-cased-distilled-squad', tokenizer='bert-base-cased')
nlp_qa(context='Hugging Face is a French company based in New-York.', question='Where is based Hugging Face ?')

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/my_name/Library/Python/3.7/lib/python/site-packages/transformers/pipelines.py", line 1188, in __call__
    start, end = self.model(**fw_args)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'token_type_ids'
```

but it should use `tokenizer='distilbert-base-cased'`

```python
from transformers import pipeline
nlp_qa = pipeline('question-answering', model='distilbert-base-cased-distilled-squad', tokenizer='distilbert-base-cased')
nlp_qa(context='Hugging Face is a French company based in New-York.', question='Where is based Hugging Face ?')

{'score': 0.9632966867654424, 'start': 42, 'end': 50, 'answer': 'New-York.'}
```


[Longformer] Notebook to train Longformer

This PR adds a community notebook that demonstrates how we pretrained Longformer starting from the RoBERTa checkpoint. The same procedure can be followed to convert other existing pretrained models into their Long version.

@patrickvonplaten, @patil-suraj, it would be great if you also check the notebook. Any comments are welcomed.


Fix onnx export input names order

This PR makes it possible to export custom bert models to onnx.

It resolves the issue addressed [here](https://github.com/huggingface/transformers/issues/4523#issuecomment-634920569).

TODO:
- [x] update ensure_valid_input function
- [x] update test ensure_valid_input_function
- [x] update convert_pytorch
- [x] add a test of exporting a custom pytorch models

@mfuntowicz


Error when loading a trained Encoder-Decoder model.

# 🐛 Bug Report
When loading the config the in configuration_auto.py the model_type is expected on the form encoder-decoder
but in configuration_encoder_decoder.py
model_type is on the form encoder_decoder which raises a KeyError.


How to generate prediction/answer from a custom model fined-tuned/trained for self-defined questions?

I used run_squad.py to fine-tuned a pre-trained DistilledBERT, and I wonder how exactly to implement my model to answer a list of self-defined questions. It seems to me that the pipeline only works with one of the "regular" models (BERT DistBert, XLM, XLNET, etc.), or a model that has already been uploaded to the community. I spent a lot of time researching this but couldn't find a solution that suits the best for my case. Would anyone please explain, and if possibly, provide a demo? Here are all the files generated after my fine-tuning and evaluation:

config.json
pytorch_model.bin
training_args.bin
nbest_predictions_.json
special_tokens_map.json
vocab.txt
predictions_.json
tokenizer_config.json

Thank you!

SO Link (no answer yet): https://stackoverflow.com/questions/62057333/how-exactly-to-generate-prediction-answer-from-a-custom-model-fined-tuned-traine


LongformerForTokenClassification

This PR adds `LongformerForTokenClassification`

@patrickvonplaten  @ibeltagy
do we need any automatic global attention here ?


Movement pruning

This PR adds the code to reproduce the results of our recent work on Movement pruning.
Some supplemental treats:
- A notebook showcasing how to efficiently store an extremely sparse model
- Sharing a couple of fine-pruned checkpoints (PruneBERT)
- Details of all hyper-parameters and results


Kill model archive maps

Links to model weights inside the code are not useful anymore, on the contrary, defining those shortcuts to URLs in code tend to lead to discrepancies with the canonical naming scheme of models at huggingface.co

As an example, we cannot cleanly load [`facebook/bart-large-cnn`](https://huggingface.co/facebook/bart-large-cnn) from either huggingface.co or the inference API because it's aliased to bart-large-cnn in the code.

If this PR is approved, before merging I will:
- do the same thing for configs (easy)
- just rename the renamed model identifiers for tokenizers. (completely getting rid of "archive maps" for tokenizers is way harder because of the hardcoded maps like `PRETRAINED_POSITIONAL_EMBEDDINGS_SIZES` and `PRETRAINED_INIT_CONFIGURATION`)

---

⚠  **Note that the change in this PR is breaking for the names of the following models:**

```
"cl-tohoku/bert-base-japanese"
"cl-tohoku/bert-base-japanese-whole-word-masking"
"cl-tohoku/bert-base-japanese-char"
"cl-tohoku/bert-base-japanese-char-whole-word-masking"
"TurkuNLP/bert-base-finnish-cased-v1"
"TurkuNLP/bert-base-finnish-uncased-v1"
"wietsedv/bert-base-dutch-cased"
"flaubert/flaubert_small_cased"
"flaubert/flaubert_base_uncased"
"flaubert/flaubert_base_cased"
"flaubert/flaubert_large_cased"

all variants of "facebook/bart"
```

^^ You'll need to specify the organization prefix for those models from now on. However, no files were moved on S3 so this doesn't change anything for all current versions of the library.

However it's a breaking change so we'll be sure to pinpoint to it in the next release.


03-pipelines.ipynb on Colab: error on "Summarization"


tensorflow2_gpt2 Slow speed

# ❓ The speed of pytorch_gpt2 based on transformers is 4-5 times faster than that of tensorflow_gpt2, which is unreasonable. Where is my problem

**pytorch_gpt2**

```
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

model.to(dev)
raw_text_2 = "And if so be ye can descrive what ye bear,"

inputs = tokenizer.encode(raw_text_2, add_special_tokens=False, return_tensors="pt")
inputs = inputs.to(dev)
generated = model.generate(inputs, top_k=0, max_length=512)

```

**tensorflow2_gpt2**
```

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = TFGPT2LMHeadModel.from_pretrained('gpt2')

raw_text = "And if so be ye can descrive what ye bear,"

tokens = tokenizer.encode(raw_text,  return_tensors="tf")
output_ids = model.generate(tokens, top_k = 0, max_length=512)

```


Merge pull request #1 from huggingface/master


Pipelines: miscellanea of QoL improvements and small features...

...needed for inference API.

see individual commits for description


Numpy format string issue in TFTrainer

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: tf_ner
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Running the `run_tf_ner` example raises the following exception:

`Traceback (most recent call last):
  File "run_tf_ner.py", line 282, in <module>
    main()
  File "run_tf_ner.py", line 213, in main
    trainer.train()
  File "venv/lib/python3.7/site-packages/transformers/trainer_tf.py", line 308, in train
    logger.info("Epoch {} Step {} Train Loss {:.4f}".format(epoch, step, training_loss.numpy()))
TypeError: unsupported format string passed to numpy.ndarray.__format__`

This issue was reported by multiple people:
https://github.com/numpy/numpy/issues/12491
https://github.com/numpy/numpy/issues/5543

I think the easiest solution is to avoid using the numpy format string this way in `TFTrainer`.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.1.0
- Platform: Ubuntu-18.04
- Python version: 3.7.7
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): 2.1.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Model evaluated at each checkpoint, but results not in checkpoint file

# ❓ Questions & Help

I start `run_language_modeling.py` and `run_glue.py` with `--do_eval` and `--evaluate_during_training` arguments. While it checkpoints and evaluates the model at each saving step, the evaluation results are not output to the checkpoint folder but merely the terminal at each saving step, as well as logging step. However, the final model is evaluated and its results placed in the appropriate location. I'd like the performance of each checkpoint. Am I doing something wrong? Is there an additional argument I must specify?


gpt2 typo

From #4572


[Longformer] more models + model cards

model card

Related to pretrained model cards

This PR adds the following:

- Longformer models trained with frozen-roberta weights

- Model cards

- Model names start with `allenai/`

- Remove unnecessary type casting


[WIP] lightning glue example uses nlp package

Main goal here was to use `nlp` library to load datasets instead of custom scripts. I've made some changes to the GLUE example, and will reflect those changes elsewhere if the patterns used seem reasonable. Feedback would be greatly appreciated.

**So far, this PR:**
- Uses `nlp` library instead of manual processing scripts to download/process the benchmark datasets (Ref #4494)
- Uses lightning `Trainer` default arguments instead of previous custom logic. See[ list of available args](https://pytorch-lightning.readthedocs.io/en/latest/trainer.html#trainer-class) in their documentation. Resolves #3925
- Generates submission files on test sets (partially resolves #3692)
- Fixes bug mentioned in #4214
- Pins lightning version to latest stable release, as master branch is a little to volatile for my taste.
- upgrades to pl=.76

**TODOs**
- Validate multi-gpu logic is up to date with lightning base practices
- Validate TPU logic is up to date with lightning best practices
- See if there's a better way to save dataset with `nlp` lib directly instead of current `torch.save` logic
- Run exhaustive test over model types over each dataset
- Optionally generate larger output table reporting benchmarks across models

**Note on Trainer args**

The argument parser will now accept any kwargs from the `Trainer` class's init function. For example, to load a checkpoint and run predictions to get submission files, you could run something like this to get a submission file at `./submissions/mrpc_submission.csv`:

```bash
export TASK=mrpc
export DATA_DIR=./cached_glue_data
export MAX_LENGTH=128
export LEARNING_RATE=2e-5
export BERT_MODEL=bert-base-cased
export BATCH_SIZE=32
export NUM_EPOCHS=3
export SEED=2
export GPUS=1
export NUM_WORKERS=4

# Add parent directory to python path to access lightning_base.py
export PYTHONPATH="../":"${PYTHONPATH}"

python3 -i run_pl_glue.py \
    --model_name_or_path $BERT_MODEL \
    --task $TASK \
    --data_dir $DATA_DIR \
    --max_seq_length  $MAX_LENGTH \
    --max_epochs $NUM_EPOCHS \
    --learning_rate $LEARNING_RATE \
    --seed $SEED \
    --gpus $GPUS \
    --num_workers $NUM_WORKERS \
    --train_batch_size $BATCH_SIZE \
    --resume_from_checkpoint ./lightning_logs/version_0/checkpoints/epoch=1.ckpt \
    --output_dir ./submissions/
    --do_predict
```

CC: @srush @williamFalcon


How to use run_glue.py with tensorboard?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

I'm running `run_glue.py` script, where I added a new task_name in `data/metrics` and `data/processors`. The training happens OK, the checkpoints are being saved, but no tfevent file is being written. Shouldn't it be written in checkpoints folder during training?


[Model Card] model card for longformer-base-4096-finetuned-squadv1

Related to pretrained model cards

@patrickvonplaten


Can't see logger output

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RoBERTa

Language I am using the model on (English, Chinese ...): Sanskrit

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Can't see logger output showing model config and other parameters in Trainer that were printed in training_scripts.

1.
```
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./model_path",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=128,
    per_gpu_eval_batch_size =256,
    save_steps=1_000,
    save_total_limit=2,
    logging_first_step = True,
    do_train=True,
    do_eval = True,
    evaluate_during_training=True,
    logging_steps = 1000
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset = valid_dataset,
    prediction_loss_only=True,
)
```
2.
```
%%time
trainer.train(model_path="./model_path")
```

Is it It is overriden by tqdm?
but I can still see `Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.`

## Environment info

- `transformers` version: 2.10.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0a0+916084d (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: TPU
- Using distributed or parallel set-up in script?: No


`train.jsonl` file missing in MM-IMDb task

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MM-IMDb
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Download the **raw** MM-IMDb dataset from http://lisi1.unal.edu.co/mmimdb/
2. Download `run_mmimdb.py` and `utils_mmimdb.py` from `/examples/contrib/mm-imdb`
3. Run the command given in [README.md](https://github.com/huggingface/transformers/blob/master/examples/contrib/mm-imdb/README.md#training-on-mm-imdb)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
Traceback (most recent call last):
  File "run_mmimdb.py", line 614, in <module>
    main()
  File "run_mmimdb.py", line 555, in main
    train_dataset = load_examples(args, tokenizer, evaluate=False)
  File "run_mmimdb.py", line 339, in load_examples
    dataset = JsonlDataset(path, tokenizer, transforms, labels, args.max_seq_length - args.num_image_embeds - 2)
  File ".../mmimdb/utils_mmimdb.py", line 50, in __init__
    self.data = [json.loads(l) for l in open(data_path)]
FileNotFoundError: [Errno 2] No such file or directory: '.../mmimdb/dataset/train.jsonl'
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
+ `train.jsonl` should be present in the script directory

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Linux
- Python version: Python 3.6.5
- PyTorch version (GPU?): `1.4.0` (yes)
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


GPU memory usage

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am training albert from scratch. I am using 8 V100. Issue is gpu 0 is almost completely used but others have around 50% ram unused. I am getting only 85 batch size on this system  and above this OOM.
Using transformers from source 2.10.0
```
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.33.01    Driver Version: 440.33.01    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:16.0 Off |                    0 |
| N/A   77C    P0   291W / 300W |  30931MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   71C    P0   255W / 300W |  18963MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   71C    P0    95W / 300W |  18963MiB / 32510MiB |     98%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   68C    P0    89W / 300W |  18963MiB / 32510MiB |     72%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:00:1A.0 Off |                    0 |
| N/A   68C    P0    78W / 300W |  18963MiB / 32510MiB |    100%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:00:1B.0 Off |                    0 |
| N/A   69C    P0    96W / 300W |  18963MiB / 32510MiB |     65%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:00:1C.0 Off |                    0 |
| N/A   69C    P0    79W / 300W |  18963MiB / 32510MiB |     95%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:00:1D.0 Off |                    0 |
| N/A   74C    P0    80W / 300W |  18963MiB / 32510MiB |     12%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     28066      C   python                                     30917MiB |
|    1     28066      C   python                                     18949MiB |
|    2     28066      C   python                                     18949MiB |
|    3     28066      C   python                                     18949MiB |
|    4     28066      C   python                                     18949MiB |
|    5     28066      C   python                                     18949MiB |
|    6     28066      C   python                                     18949MiB |
|    7     28066      C   python                                     18949MiB |
+-----------------------------------------------------------------------------+
 ```


Cleanup glue

* Make sure that MNLI acc metrics are named differently for match vs mismatch
* Allow writing to `cache_dir` from Glue dataset in case where datasets live in readOnly filesystem
* Flush TB writer at end to not miss any metrics


The new abstractions in /master are counterproductive

I understand the desire to use abstraction to make the library easier to use out-of-the-box, especially for newer users who just want to call run_model.py with some hyparameters.

However, as someone who considers himself a novice-intermediate and frequently needs to look at or modify the source code to achieve what I want, it's been a huge pain adapting to the new changes.

I'll try to be as concrete as possible but here are some big painpoints:

(I will say, these painpoints may come from inexpeience with the library rather than something is just hard/impossible to achieve but either way, as a novice-intermediate, the end result is the same: I have a hard time navigating the library)

- arguments are hidden, so far example in run_language_modeling.py I'm not able to see what all the available parameters are. I feel using abstraction over argument types like training_args, model_args, etc. is overkill and just programming fluff

- model source code is hidden under many layers of abstraction

- support for basic functionality is not implemented when code is refactored. One example is the new run_language_modeling.py doesn't have support for training continuation from a checkpoint

Like for example, it's very hard to tell what any of the XXX.from_pretrained classes are actually doing.

These are just the painpoints I can think of from the top of my head. It's overall been just unsmooth using the new example files like run_language_modeling.py and the rest of the API.

My main suggestion is to rethink the trade-off beween simplicity, abstraction versus flexibility and ability to hack/modify the code. Generally I think abstraction is good, but it seems overly excessive in certain places and I wonder if you can achieve the same goal while doing a bit less.

A tangential observation is the example scripts get way too cumbersome when you try to support all these things within the same file (e.g. run_language_modeling.py): apex, tpu, tensorflow, distributed. There are flags everywhere.


removed deprecated use of Variable API from pplm example

This is same as previous [PR](https://github.com/huggingface/transformers/pull/4156) which I closed due to code styling issue. Didn't know that specific isort version was supposed to be used.


per_device instead of per_gpu/error thrown when argument unknown

Modified the trainer argument so that `per_device_train_batch_size` and `per_device_eval_batch_size` are preferred over `per_gpu_*`.

`per_gpu_*` still works when `per_device_*` isn't used, but is deprecated.

The trainer argument parser now throws an error if an argument is unknown, only if the `return_remaining_strings` flag is kept to `False`.


run evalation after every epoch in Trainer

# 🚀 Feature request

With the current Trainer implementation:
`trainer.train(..)` is called first followed by `trainer.evaluate(..)`. It would be nice if the user can pass the flag `--run_eval` (something similar) to run evaluation after every epoch. It would be nice for users who want to see how model performs on validation set as training progresses. In some cases, this is the general norm (run evaluation after every epoch).


[testing] LanguageModelGenerationTests require_tf or require_torch


[Longformer] longformer in question-answering pipeline

This PR adds `LongformerForQuestionAnswering` in `QuestionAnsweringPipeline`

@patrickvonplaten @ibeltagy


[Contributing Doc] Update version command when contributing

According to PR: #4131, the `CONTRIBUTING.md` should be updated a bit. @BramVanroy


What does the output of feature-extraction pipeline represent?

I am using the feature-extraction pipeline:
```
nlp_fe = pipeline('feature-extraction')
nlp_fe('there is a book on the desk')
```
As an output I get a list with one element - that is a list with 9 elements - that is a list of 768 features (floats).
What is the output represent? What is every element of the lists, and what is the meaning of the 768 float values?
Thanks


Use fill-mask pipeline to get probability of specific token

Hi,
I am trying to use the fill-mask pipeline:
```
nlp_fm = pipeline('fill-mask')
nlp_fm('Hugging Face is a French company based in <mask>')
```

And get the output:
```
[{'sequence': '<s> Hugging Face is a French company based in Paris</s>',
  'score': 0.23106734454631805,
  'token': 2201},
 {'sequence': '<s> Hugging Face is a French company based in Lyon</s>',
  'score': 0.08198195695877075,
  'token': 12790},
 {'sequence': '<s> Hugging Face is a French company based in Geneva</s>',
  'score': 0.04769458621740341,
  'token': 11559},
 {'sequence': '<s> Hugging Face is a French company based in Brussels</s>',
  'score': 0.04762236401438713,
  'token': 6497},
 {'sequence': '<s> Hugging Face is a French company based in France</s>',
  'score': 0.041305914521217346,
  'token': 1470}]

```

But let's say I want to get the score & rank on other word - such as London - is this possible?


Key error while evaluating the Language Model finetuning

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): DistilBert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. python run_language_modeling.py \
    --output_dir=output \
    --model_type=distilbert\
    --model_name_or_path=distilbert-base-uncased \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```code
05/27/2020 08:44:42 - INFO - __main__ -   *** Evaluate ***
05/27/2020 08:44:42 - INFO - transformers.trainer -   ***** Running Evaluation *****
05/27/2020 08:44:42 - INFO - transformers.trainer -     Num examples = 329
05/27/2020 08:44:42 - INFO - transformers.trainer -     Batch size = 8
Evaluation: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42/42 [00:04<00:00,  9.09it/s]
Traceback (most recent call last):
  File "run_language_modeling.py", line 281, in <module>
    main()
  File "run_language_modeling.py", line 259, in main
    perplexity = math.exp(eval_output["eval_loss"])
KeyError: 'eval_loss'

```

## Expected behavior

Evaluation of the validation data and output the perplexity.

Upon debugging the code, the eval_output doesn't have the key `eval_loss`

```code
-> perplexity = math.exp(eval_output["eval_loss"])
(Pdb) eval_output
{'loss': 1.8573534346762157}
```
Please change the key value accordingly.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.0
- Platform: RHEL
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


README for HooshvareLab

Related to pretrained model cards

HooshvareLab/bert-base-parsbert-uncased


How to deal with summarization task to long sequences input?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I am going to carry out the sumarization task using the 'transformers' module you provided. But there's a problem. The sequence which i have is too long, so an error occurs when inserting it. Is there any way to summarize the entire document by sliding it?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


uncased readme

Related to pretrained model cards

updates to the model card for uncased model with more evaluation results and recommendation to switch to cased model


Create README.md

Related to pretrained model cards

Model card for cased model


Inconsistency in how Electra doing sentence level prediction

In `ElectraForSequenceClassification`:
We have docstring says, `ELECTRA Model transformer with a sequence classification/regression head on top (a linear layer on top of
    the pooled output) e.g. for GLUE tasks.`
This is also what I observed in the official repository.
https://github.com/google-research/electra/blob/81f7e5fc98b0ad8bfd20b641aa8bc9e6ac00c8eb/finetune/classification/classification_tasks.py#L270
https://github.com/google-research/electra/blob/79111328070e491b287c307906701ebc61091eb2/model/modeling.py#L254
which is
```
nn.Sequential( nn.Dropout(config.hidden_dropout_prob),
                        nn.Linear(config.hidden_size, config.num_labels)
```

**But** the implementation of `ElectraClassificationHead` (used by `ElectraForSequenceClassification`) is
```
def forward(self, features, **kwargs):
        x = features[:, 0, :]  # take <s> token (equiv. to [CLS])
        x = self.dropout(x)
        x = self.dense(x)
        x = get_activation("gelu")(x)  # although BERT uses tanh here, it seems Electra authors used gelu here
        x = self.dropout(x)
        x = self.out_proj(x)
        return x
```

Is there something I overlooked in the official repository ? Hot to explain the inconsistency in doc and implementaion of `ElectraForSequenceClassification`.


Glue task cleanup

* Enable writing cache to cache_dir in case dataset lives in readOnly
filesystem
* Differentiate match vs mismatch for MNLI metrics
* Manually flush tensorboard writer to avoid missing metrics.


updated model cards for both models at aubmindlab

Related to pretrained model cards

- added AraBERT image.
- updated usage examples
- updated results


Creating a readme for ALBERT in Mongolian

Related to pretrained model cards

Here I am uploading Mongolian masked language model (ALBERT) on your platform.
https://en.wikipedia.org/wiki/Mongolia


Remove MD emojis

Related to pretrained model cards


Which models can be using for encoder-decoder?

Hi, I trying to use EncoderDecoderModel. I tried google/electra-base-discriminator, google/electra-small-discriminator, albert-base-v2 as encoder and decoder:

```python
from transformers import EncoderDecoderModel

model = EncoderDecoderModel.from_encoder_decoder_pretrained('google/electra-small-discriminator', 'google/electra-small-discriminator')
```
but, I always gate the same error

```python
TypeError: forward() got an unexpected keyword argument 'encoder_hidden_states'
```

Only Bert can be using as encoder and decoder? If so, can you add list of available models for encoder-decoder in documentation?


Functionality for addressing imbalance data points?

Is there yet any functionality from transformers library to address or tackle imbalance classes in data points?


Hi friends,

I would like to use  transformers library. But while import I received this error.

This code:
```
from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
import torch
 # LOAD MODEL
tokenizer = AutoTokenizer.from_pretrained("savasy/bert-base-turkish-squad")
model = AutoModelForQuestionAnswering.from_pretrained("savasy/bert-base-turkish-squad")

```
**Error**

**ImportError: cannot import name 'AutoModelForQuestionAnswering' from 'transformers' (C:\Users\oguzk\anaconda3\lib\site-packages\transformers\__init__.py)**


[Reformer] automate axial_pos_shape

This PR automates the calculation of the axial_pos_shape.
checked that every combination of 2**n works using this sheet: https://docs.google.com/spreadsheets/d/19gnP1ve2fT2F59LNiky44SPpmtmOegfIlk5_3OFXjyU/edit?usp=sharing

@patrickvonplaten


[Draft] Bharaax outputattentions


AttributeError: 'Namespace' object has no attribute 'to_json_string'

trainer.train()
don't know how to set parameters self.args.to_json_string()


KeyError when loading a trained EncoderDecoder model

# 🐛 Bug

## Information

Error when loading a trained EncoderDecoder model. When loading the config the in `configuration_auto.py` the `model_type` is expected on the form `encoder-decoder` but in   `configuration_encoder_decoder.py` `model_type` is on the form `encoder_decoder` which raises a KeyError.

The hyphen version seems to be convention in the other model configuration files.

I guess this is something for @patrickvonplaten


hi, i have create a new class named ClsNerModel inherited from TFBertPreTrainedModel
i have trained it successfully and save the model using model.save_pretrained to some dir

however, when i loaded from that dir using  ClsNerModel.from_pretrained, it fails and report some error below.
sorry to report this, and any advice is appreciated.

  File "/home/yuanhao/anaconda3/envs/tf2/lib/python3.7/site-packages/transformers/modeling_tf_utils.py", line 410, in from_pretrained
    model.load_weights(resolved_archive_file, by_name=True)
  File "/home/yuanhao/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 250, in load_weights
    return super(Model, self).load_weights(filepath, by_name, skip_mismatch)
  File "/home/yuanhao/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py", line 1264, in load_weights
    f, self.layers, skip_mismatch=skip_mismatch)
  File "/home/yuanhao/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 753, in load_weights_from_hdf5_group_by_name
    weight_values = [np.asarray(g[weight_name]) for weight_name in weight_names]
  File "/home/yuanhao/anaconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/keras/saving/hdf5_format.py", line 753, in <listcomp>
    weight_values = [np.asarray(g[weight_name]) for weight_name in weight_names]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/yuanhao/anaconda3/envs/tf2/lib/python3.7/site-packages/h5py/_hl/group.py", line 264, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'bias:0' doesn't exist)"

my code is something like this:
##############################################################################
class ClsNerModel(TFBertPreTrainedModel):
    def __init__(self, config, *inputs, cls_num_labels:int=2, **kwargs):
        super().__init__(config, *inputs, **kwargs)
        self.num_labels = config.num_labels
        self.cls_num_labels = cls_num_labels

        self.bert = TFBertMainLayer(config, name="bert")
        self.dropout = tf.keras.layers.Dropout(config.hidden_dropout_prob)
        self.classifier = tf.keras.layers.Dense(
            config.num_labels, kernel_initializer=get_initializer(config.initializer_range), name="classifier"
        )
        self.cls_dropout = tf.keras.layers.Dropout(config.hidden_dropout_prob)
        self.cls_classifier = tf.keras.layers.Dense(
            self.cls_num_labels, kernel_initializer=get_initializer(config.initializer_range), name="cls_classifier"
        )

    def call(self, inputs, **kwargs):
        outputs = self.bert(inputs, **kwargs)

        sequence_output = outputs[0] # (b, t, d)
        pool_output = outputs[1] # (b, d) only for cls token

        sequence_output = self.dropout(sequence_output, training=kwargs.get("training", False))
        token_logits = self.classifier(sequence_output)
        pool_output = self.cls_dropout(pool_output, training=kwargs.get("training", False))
        cls_logits = self.cls_classifier(pool_output)

        outputs = (token_logits, cls_logits) + outputs[2:]  # add hidden states and attention if they are here

        return outputs  # scores, (hidden_states), (attentions)

##############################################################################
    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,
        id2label=label_map,
        label2id={label: i for i, label in enumerate(label_map)},
        cache_dir=model_args.cache_dir,
    )
    model = ClsNerModel.from_pretrained(
        model_path,
        from_pt=bool(".bin" in model_args.model_name_or_path),
        output_loading_info=True,
        config=config,
        cls_num_labels=cls_num_labels,
        cache_dir=model_args.cache_dir,
    )


[Longformer For Question Answering] Conversion script, doc, small fixes

This PR adds
- Longformer For Question Answering doc
- Adds the link to the (official) uploaded model: https://huggingface.co/allenai/longformer-large-4096-finetuned-triviaqa
- Some minor refactoring
- Conversion script

@ibeltagy @patil-suraj


# 🐛 Bug
I am running Huggingface's nlp pipeline. The code is below:

```
nlp = pipeline("question-answering",
               model = 'distilbert-base-cased-distilled-squad',
               tokenizer='distilbert-base-cased-distilled-squad')
```

Model I am using is pipeline.

I try run an example using docker and I get the following error:

```
convert squad examples to features: 100%|██████████| 1/1 [00:00<00:00, 360.86it/s]
add example index and unique id: 100%|██████████| 1/1 [00:00<00:00, 3120.76it/s]
myimage_1  | [2020-05-26 08:38:22,060] ERROR in app: Exception on /deep/search [POST]
myimage_1  | Traceback (most recent call last):
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
myimage_1  |     response = self.full_dispatch_request()
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
myimage_1  |     rv = self.handle_user_exception(e)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
myimage_1  |     reraise(exc_type, exc_value, tb)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
myimage_1  |     raise value
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
myimage_1  |     rv = self.dispatch_request()
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
myimage_1  |     return self.view_functions[rule.endpoint](**req.view_args)
myimage_1  |   File "main.py", line 50, in launch_app
myimage_1  |     answers = nlp(question=ques0, context=abstract, topk = 10)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/transformers/pipelines.py", line 1010, in __call__
myimage_1  |     start, end = self.model(**fw_args)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
myimage_1  |     result = self.forward(*input, **kwargs)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/transformers/modeling_distilbert.py", line 720, in forward
myimage_1  |     input_ids=input_ids, attention_mask=attention_mask, head_mask=head_mask, inputs_embeds=inputs_embeds
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
myimage_1  |     result = self.forward(*input, **kwargs)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/transformers/modeling_distilbert.py", line 482, in forward
myimage_1  |     inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
myimage_1  |     result = self.forward(*input, **kwargs)
myimage_1  |   File "/usr/local/lib/python3.7/site-packages/transformers/modeling_distilbert.py", line 86, in forward
myimage_1  |     seq_length = input_ids.size(1)
myimage_1  | IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
```

My python version in 3.7.4.

Please help in fixing this.


Create README.md

Related to pretrained model cards


[Model hub web parsing MD code error]

# 🐛 Bug

## Information

Hi guys!
If you navigate to https://github.com/huggingface/transformers/blob/master/model_cards/mrm8488/bert-italian-finedtuned-squadv1-it-alfa/README.md

You will see the emojis without problem. But if you go to its HTML page on the model hub: https://huggingface.co/mrm8488/bert-italian-finedtuned-squadv1-it-alfa
Emojis are not shown


[LongformerForQuestionAnswering] fix qa example in docstring

@patrickvonplaten


Help Wanted: Predict Next Two Tokens

Is it possible to change this in order to predct the next two tokens?

```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel
import torch.nn.functional as F
import re

def grow_branches(sentence_so_far, probs, input_probability,past, h):
    #recursive function to find all sentence completions
    global branch_list
    global leaf_list
    global complete_list
    global model
    sorted_probability_list = sorted(enumerate(probs), key=lambda x: x[1], reverse=True)
    has_children = False
    for (this_token,this_probability) in sorted_probability_list:
        next_probability = this_probability * input_probability
        out_sentence = sentence_so_far.copy()
        sentence_and_probability = (out_sentence, input_probability)
        pattern = ' [A-Z]{1,1}'
        pattern2 = '[A-Z]{1,1}'
        test_string = tokenizer.decode(out_sentence[-1])
        result = re.match(pattern, test_string) or re.match(pattern2, test_string)
        if not (result or (out_sentence[-1] in {1583,1770,6997,19090,9074,7504})) and (this_token == 13):
            #if the next token is going to be a period, then no need to carry out that step.
            #except allow Mr., Dr., Mrs., Ms., Lt., Sgt., Jr. or single initials.
            sentence_and_probability = (out_sentence, next_probability)
            complete_list.append(sentence_and_probability)
            return
        if next_probability < h:
            if has_children == True:
                branch_list.append(sentence_and_probability)
            else:
                leaf_list.append(sentence_and_probability)
            return
        else:
            has_children = True
            next_sentence = sentence_so_far.copy()
            next_sentence.append(this_token)
            (next_probability_list,next_past) = expand_node(next_sentence,past)
            grow_branches(next_sentence,next_probability_list, next_probability, next_past, h)

def expand_node(sentence, past):
    #finds probabilities for the next token using gpt-2
    global model
    if past == None:
        input_ids = torch.tensor(sentence).unsqueeze(0)
    else:
        input_ids = torch.tensor([sentence[-1]]).unsqueeze(0)
    inputs = {'input_ids': input_ids}
    with torch.no_grad():
        logits, past = model(**inputs, past=past)
        logits = logits[:, -1, :]
        probs = F.softmax(logits, dim=-1).tolist()[0]
        return (probs, past)

# globals here
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

leaf_list = []
branch_list = []
complete_list = []
probability_threshhold=float(input("probability cutoff (e.g. .001 or less):"))
raw_prompt = input("partial sentence to complete:")

prompt=tokenizer.encode(raw_prompt)

(probs, past) = expand_node(prompt, None)
grow_branches(prompt,probs,1,past,probability_threshhold)

sorted_complete_list = sorted(complete_list, reverse=True,key=lambda x: x[1])
sorted_leaf_list = sorted(leaf_list, reverse=True,key=lambda x: x[1])
sorted_branch_list = sorted(branch_list, reverse=True,key=lambda x: x[1])

# to get the most probable completed sentence:
#tokenizer.decode(sorted_complete_list[0])

#print just the completions
for (sentence, prob) in sorted_complete_list:
    #print(round(prob,6),end=':')
    if prob>probability_threshhold - 1:
       print(repr(tokenizer.decode(sentence[len(prompt):])).strip("'"),end='|')
    else:
       print(repr(tokenizer.decode(sentence[len(prompt):])).strip("'"),end='\\')

for (sentence, prob) in sorted_leaf_list:
    if prob>probability_threshhold:
        print(repr(tokenizer.decode(sentence[len(prompt):])).strip("'"),end='|')
    else:
       print(repr(tokenizer.decode(sentence[len(prompt):])).strip("'"),end='\\')
```


ensure_ascii=False


T5Model in fp16 still yield nan with more complex examples

# 🐛 Bug

Hello, thank you for the recent [PR](https://github.com/huggingface/transformers/pull/4436) with fp16 fixes. It seems to work well with short inputs, but once the model is fed with some more complex data it still yields nans.

## Information

Model I am using: T5

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
Run the code:
```
from transformers import T5Model
import torch

model = T5Model.from_pretrained("t5-base").cuda().half().eval()
inputs = torch.tensor([[37,423,215,1504,13,8,1186,10670,11,10449,49,1152,11363,15465,1514,5,4433,399,7863,24766,15,17,965,594,5386,14286,28,8,6,5,755,5781,32099,993,3744,21,8,2367,18,458,53,16616,32098,16,32097,7660,16409,77,19,3,107,13164,1054,32096,993,1970,9368,948,147,8,15465,5861,87,25481,788,12,8,32095,1300,61,37,423,215,1504,13,3,24151,40,3,19668,594,5386,14286,28,8,3,115,13164]]).cuda()
decoder_input_ids = torch.tensor([[21820, 296, 55]]).cuda()

out = model(input_ids=inputs, decoder_input_ids=decoder_input_ids)
# encoder outputs
out[2][:,:2]
```

output:
```
tensor([[[nan, nan, nan,  ..., nan, nan, nan],
         [nan, nan, nan,  ..., nan, nan, nan]]], device='cuda:0',
       dtype=torch.float16, grad_fn=<SliceBackward>)
```

## Expected behavior
Output with non-nan values.

## Environment info

- `transformers` version: 2.10.0
- Platform: Linux-4.15.0-88-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Introduce a new tensor type for return_tensors on tokenizer for NumPy

Two changes in this PR:

- As we're introducing more than two tensor backend alternatives I created an enum `TensorType` listing all the possible tensor we can create `TensorType.TENSORFLOW`, `TensorType.PYTORCH`, `TensorType.NUMPY`. This might help newcomers who don't know about `"tf"`, `"pt"`.

   _->Note: TensorType are compatible with previous  `"tf"`, `"pt"` and now `"np"` str to allow backward compatbility (+unittest)_

- Numpy is now a possible target when creating tensors. This is usefull for JAX :)


[ci] fix 3 remaining slow GPU failures


Provide simple way to train a new translation model from scratch

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

Huggingface just released a huge pile of pretrained translation models. I just want to train a completely custom model on a custom language pair, without pretraining etc.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Improve model card for Tereveni-AI/gpt2-124M-uk-fiction

Related to pretrained model cards

Add language metadata, training and evaluation corpora details.
Add example output. Fix inconsistent use of quotes.


[GPT2, CTRL] Allow input of input_ids and past of variable length

## Description

This PR reverts the automatic cutting of input ids as introduced in PR: https://github.com/huggingface/transformers/pull/3734 and fixes issue https://github.com/huggingface/transformers/issues/4368 .

Currently, when `past` is used in combination with `input_ids`, the `input_ids` are cut to just the last token. This breaks certain functionality as explained in Issue: #4368.

Also, the documentation is made more precise for GPT2 and CTRL.

## Backward Compatibility

This PR slightly breaks backward compatibility since `input_ids` now have to be input according to `past` and are **not** cut automatically for example for automatic language generation. So the functionality is as it was before version 2.8.0.


LongformerForSequenceClassification

This PR adds `LongformerForSequenceClassification`

@patrickvonplaten @ibeltagy
All the changes here are as per we discussed  in `LongformerForQuestionAnswering`.
`forward` method automatically sets global attention on cls token.


How to save tokenize data when training from scratch

# ❓ Questions & Help

I am training Allbert from scratch following the blog post by hugging face. As it mentions that :

> If your dataset is very large, you can opt to load and tokenize examples on the fly, rather than as a preprocessing step.

How this can be done any suggestions?
As of now , using the method given in the notebook:
```
from transformers import TextDataset

dataset = TextDataset(
    tokenizer=tokenizer,
    file_path="./oscar.eo.txt",
    block_size=128,
)
```
 there is no method to save tokenize data, can anyone suggest how to save that as its already taking long enough before starting training.


Create model card

Related to pretrained model cards


Using whole word masking on training LM from scratch

# ❓ Questions & Help
## Details
Hello everyone,
I wanted to use _whole-word-masking_ in training LM from scratch. I could not have found how to apply this option using Trainer.
I thought this option should be managed in  "class DataCollatorForLanguageModeling", but I could not find options for _whole-word-masking._
Am I looking at wrong place OR it is not implemented yet?
If not, is it possible to do with run_language_modeling.py?

**A link to original question on Stack Overflow**:  https://stackoverflow.com/questions/62061578/how-to-use-whole-word-masking-on-training-lm-from-scratch

Any help is appreciated!
Thanks


OSError: Model name 'transfo-xl-wt103' was not found in tokenizers model name list (transfo-xl-wt103). We assumed 'transfo-xl-wt103' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.bin', 'vocab.txt'] but couldn't find such vocabulary files at this path or url.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.10.0
- Platform: ubantu
- Python version:3.6
- PyTorch version (GPU?):yes
- Tensorflow version (GPU?):yes
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:


Onnx notebook problem

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I saw the issue related to mine in #260 so I changed env to python 3.6 and torch 1.1 but I didn't help.
When I run onnx notebook I get an error [TypeError: export() got an unexpected keyword argument 'dynamic_axes']
Does anyone have guess what's wrong?
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Fix longformer attention mask type casting when using apex

Fix for issue [#4525](https://github.com/huggingface/transformers/issues/4525).