Error in run_language_modeling on TPU: Transferring data with element type U8 has not been implemented on TPUs

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-4.19.0-10-cloud-amd64-x86_64-with-debian-9.13
- Python version: 3.7.7
- PyTorch version (GPU?): 1.7.0a0+626e410 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No (TPU)
- Using distributed or parallel set-up in script?: No

### Who can help
 albert, bert, GPT2, XLM: @LysandreJik

<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): GPT2 - (Not pretrained)

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce
I am trying to train a GPT model from scratch with TPU v2-8 in GCP. I have tried several setups but consistently getting the error below. I have simplified the arguments as much as I can do to increase the reproducibility.

Steps to reproduce the behavior:

1. Pytorch/XLA installed through docker: gcr.io/tpu-pytorch/xla:nightly_3.7
2. Transformers 3.1.0 installed through pip.
3. I used the latest xla_spawn.py with previous version of [run_language_modeling.py](https://github.com/huggingface/transformers/blob/a75c64d80c76c3dc71f735d9197a4a601847e0cd/examples/language-modeling/run_language_modeling.py) because the latest version of run_language_modeling gives error about cache_dir.
4. Simple txt file with 10k lines given as input.

```bash
python xla_spawn.py --num_cores=1 run_language_modeling.py --output_dir=/home/GPT/output --model_type=gpt2 --tokenizer_name=gpt2 --do_train --train_data_file=/home/GPT/corpus/oscar_wiki_opus_eval.txt --per_device_train_batch_size=128
```

Output:

```bash
09/08/2020 11:28:50 - WARNING - run_language_modeling -   Process rank: -1, device: xla:1, n_gpu: 0, distributed training: False, 16-bits training: False
09/08/2020 11:28:50 - INFO - run_language_modeling -   Training/evaluation parameters TrainingArguments(output_dir='/home/GPT/output', overwrite_output_dir=False, do_train=True, do_eval=False, do_predict=False, evaluate_during_training=False, prediction_loss_only=False, per_device_train_batch_size=128, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Sep08_11-28-39_b0b61c9b1d63', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=1, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1, run_name=None, disable_tqdm=False, remove_unused_columns=True)
09/08/2020 11:28:50 - WARNING - run_language_modeling -   You are instantiating a new config instance from scratch.
09/08/2020 11:28:50 - INFO - run_language_modeling -   Training new model from scratch
/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/transformers/modeling_auto.py:732: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/transformers/tokenization_utils_base.py:1321: FutureWarning: The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.
  FutureWarning,
09/08/2020 11:28:54 - INFO - filelock -   Lock 140191457088144 acquired on /home/GPT/corpus/cached_lm_GPT2Tokenizer_1024_oscar_wiki_opus_eval.txt.lock
09/08/2020 11:29:09 - INFO - filelock -   Lock 140191457088144 released on /home/GPT/corpus/cached_lm_GPT2Tokenizer_1024_oscar_wiki_opus_eval.txt.lock
Traceback (most recent call last):
  File "xla_spawn.py", line 72, in <module>
    main()
  File "xla_spawn.py", line 68, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 387, in spawn
    _start_fn(0, pf_cfg, fn, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 324, in _start_fn
    fn(gindex, *args)
  File "/home/GPT/run_language_modeling.py", line 294, in _mp_fn
    main()
  File "/home/GPT/run_language_modeling.py", line 252, in main
    prediction_loss_only=True,
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/transformers/trainer.py", line 229, in __init__
    self.model = model.to(args.device) if model is not None else None
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 402, in _apply
    self._buffers[key] = fn(buf)
  File "/root/anaconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 610, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:383 : Check failed: session->session()->Run( session_work->feed_inputs, session_work->outputs_handles, &outputs) == ::tensorflow::Status::OK() (Unimplemented: From /job:tpu_worker/replica:0/task:0:
Attempted to transfer array of shape u8[1,1,1024,1024] to a TPU device. Transferring data with element type U8 has not been implemented on TPUs.
	 [[{{node XRTAllocateFromTensor_6}}]] vs. OK)
*** Begin stack trace ***
	tensorflow::CurrentStackTrace()

	xla::util::MultiWait::Complete(std::function<void ()> const&)

	clone
*** End stack trace ***

terminate called after throwing an instance of 'std::runtime_error'
  what():  tensorflow/compiler/xla/xla_client/xrt_computation_client.cc:1110 : Check failed: session->session()->Run( feed_inputs, {}, {cached_node.operations[0]}, &outputs) == ::tensorflow::Status::OK() (Not found: Op type not registered 'XRTMemoryInfo' in binary running on n-cec96fe8-w-0. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. vs. OK)
*** Begin stack trace ***
	tensorflow::CurrentStackTrace()
	xla::XrtComputationClient::ReleaseHandles(std::vector<xla::XrtComputationClient::DeviceHandle, std::allocator<xla::XrtComputationClient::DeviceHandle> >*, std::function<xla::XrtSession::CachedNode const& (xla::XrtSession*, tensorflow::Scope const&, std::string const&)> const&, xla::metrics::Metric*, xla::metrics::Counter*)
	xla::XrtComputationClient::HandleReleaser()
	xla::util::TriggeredTask::Runner()

	clone
*** End stack trace ***

Aborted (core dumped)
```

## Expected behavior
I would expect it to train GPT model on TPU with given txt file from scratch.

P.S.:
I have also tried with the stable version of XLA (gcr.io/tpu-pytorch/xla:r1.6) and different model_type and tokenizer_name (bert). I am getting the same error in both cases.

I have tried several things to change the input type to float in the source code of transformers, but I was not successful. Any tips for that direction would be good for me, too.


getting 'ValueError-TextInputSequence must be str' in 'train_dataset = train_dataset.map(convert_to_features)'

Hello friends,
I want to fine tune longformer QnA for different data. I found this code very relevant to my requirement, so was trying to understand the same thoroughly. but getting an error.
i am trying to run the same code that Suraj has on github.

**

> 'ValueError-TextInputSequence must be str' in 'train_dataset = train_dataset.map(convert_to_features)'.

**

`train_dataset  = nlp.load_dataset('squad', split=nlp.Split.TRAIN)
valid_dataset = nlp.load_dataset('squad', split=nlp.Split.VALIDATION)

train_dataset = train_dataset.map(convert_to_features)
valid_dataset = valid_dataset.map(convert_to_features, load_from_cache_file=False)

#patil_suraj


Huggingface "sentiment-analysis" pipeline always output "POSITIVE" label even for negative sentences

## Environment info
- `transformers` version: 3.1
- Platform: AWS , Colab etc.
- Python version: 3.6
- PyTorch version (GPU?): NA
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

 examples/token-classification: @stefan-it

## Information

Model I am using (Bert: https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english):

The problem arises when using:
* [T ] the official example scripts: (give details below)
from transformers import pipeline
classifier = pipeline('sentiment-analysis')

classifier("I hate this thing") -->Returns Postive label
In fact when tested with a large bacth it never returned Negative label at all

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Get transformer 3.1
2. Use pipeline("sentiment-analysis")
3. Get label and score

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

from transformers import pipeline
classifier = pipeline('sentiment-analysis')

classifier("I hate this thing") -->Returns Negative label

Updates post some more experiments:

- I am using  distilbert-base-uncased-finetuned-sst-2-english following instruction from :https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
- When I download the model and save it in local before using it through AutoModel and AutoTokenizer the score for sentence nlp("i dislike it") is 0.5453
- when I directly use it without manually downloading the model to local the same score jumps to 0.99 [same score in hosted modelhub inference api as well]
- another observation is that the tokenizer_config.json has both special_tokens_map_file and full_tokneizer_file as Null. Not sure if it supposed to be like this
- for my purpose I have to use the downloaded and manually loaded style of inferencing

<!-- A clear and concise description of what you would expect to happen. -->


Index out of range in Bart-large-xsum

Questions & Help

Hello to everyone!!

I am facing a problem summarizing long articles. I mean very long text with larger vocab size than it is pre-trained already i guess.
I see that many of the models have a limitation of maximum input and trying to execute results in error of index out of range.
I am particularly using "BART-large-xsum". Please suggest what is the correct way of using these models with long documents shall I finetuning to increase the vocabsize or do anything else.

A code snippet with an example of how to handle long documents with the "BART-large-xsum" would be perfect to start with!

Thanks in advance,
Teja


Diverse Beam Search decoding

# 🚀 Feature request

Currently, with all the decoding strategies available in this library, there isn't any decoding strategy which helps in producing diverse outputs across the beam of the decoder. Such diversity is useful for tasks such as noisy data augmentation where you want to generate multiple possible outputs. [Diverse Beam Search](https://arxiv.org/abs/1610.02424) paper introduces an extremely simple trick to accomplish this and it works really well.
It is already implemented in the fairseq library and would be cool to have it in transformers too.

## Motivation

Having a decoding strategy which promotes more diversity across the beam.

## Your contribution

I would like to submit a PR for it, but before that I would like to know if there has already been some work internally on exploring this. I would appreciate if someone from the team could guide me on what parts of the code are relevant for this to work.

cc @patrickvonplaten would love some feedback from you since your wrote this [blog](https://huggingface.co/blog/how-to-generate) and you may have seen some work around this.


MLM performance difference between bert-base-cased and Conversational BERT

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help

albert, bert, GPT2, XLM: @LysandreJik
Authors of Conversational BERT: @dilyararimovna @yoptar from DeepPavlov

## Information

I am using `bert-base-cased` and [Conversational BERT](https://huggingface.co/DeepPavlov/bert-base-cased-conversational) for MLM using the pipeline tool. The only difference between the two is that Conversational BERT was initialized with `bert-base-cased` and further pre-trained on social media text data (Twitter and the like) with a custom vocabulary (more info on the model card).

As I'm using models that were not yet finetuned, I understand why I'm getting an output telling me that some weights were not used and that some weights are newly initialized (cf [this issue](https://github.com/huggingface/transformers/issues/5421)). What I don't understand is that even though the two models seem to have the same architecture and were trained on similar tasks (MLM and next sentence prediction), the performance on MLM is really poor for Conversational BERT and pretty good for `bert-base-cased`. This is all the more weird that I used Conversational BERT for tweet classification and found that it performed better than `bert-base-cased` on this downstream task.

For example, when giving '[MASK] lost his job yesterday' as input, the output for `bert-base-cased` (`pipeline_bert('[MASK] lost his job yesterday')`) is

[{'score': 0.8036763072013855,
  'sequence': '[CLS] he lost his job yesterday [SEP]',
  'token': 2002,
  'token_str': 'he'},
 {'score': 0.005656923167407513,
  'sequence': '[CLS] i lost his job yesterday [SEP]',
  'token': 1045,
  'token_str': 'i'},
 {'score': 0.005227738991379738,
  'sequence': '[CLS] dad lost his job yesterday [SEP]',
  'token': 3611,
  'token_str': 'dad'},
 {'score': 0.0032391520217061043,
  'sequence': '[CLS] david lost his job yesterday [SEP]',
  'token': 2585,
  'token_str': 'david'},
 {'score': 0.0028738391119986773,
  'sequence': '[CLS] and lost his job yesterday [SEP]',
  'token': 1998,
  'token_str': 'and'}]

and the output for Conversational BERT (`pipeline_convbert('[MASK] lost his job yesterday')`) is:

[{'score': 0.0005837088683620095,
  'sequence': '[CLS]As lost his job yesterday [SEP]',
  'token': 23390,
  'token_str': '##As'},
 {'score': 0.0004703140293713659,
  'sequence': '[CLS] rock lost his job yesterday [SEP]',
  'token': 2067,
  'token_str': 'rock'},
 {'score': 0.0004569509474094957,
  'sequence': '[CLS] ACT lost his job yesterday [SEP]',
  'token': 21111,
  'token_str': 'ACT'},
 {'score': 0.0004535183834377676,
  'sequence': '[CLS] colour lost his job yesterday [SEP]',
  'token': 5922,
  'token_str': 'colour'},
 {'score': 0.0004286181356292218,
  'sequence': '[CLS]inas lost his job yesterday [SEP]',
  'token': 16924,
  'token_str': '##inas'}]

I think this has to do with weight initializations but I'm not grasping everything in the output. Some points that are unclear:
- `['cls.seq_relationship.weight', 'cls.seq_relationship.bias']` are not used in the case of `bert-base-cased` (see output) but are used (or missing?) in the case of Conversational BERT
- Only `['cls.predictions.decoder.bias']` is newly initialized in the case of `bert-base-cased` but all of the following weights are initialized in the case of Conversational BERT: `['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias']`

## To reproduce

Steps to reproduce the behavior:

### bert-base-cased:

#### Input:

`pipeline_bert = pipeline('fill-mask', model='bert-base-uncased', tokenizer='bert-base-uncased', config='bert-base-uncased')`

#### Output:
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForMaskedLM were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['cls.predictions.decoder.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

### Conversational BERT

#### Input:

`pipeline_convbert = pipeline('fill-mask', model='DeepPavlov/bert-base-cased-conversational', tokenizer='DeepPavlov/bert-base-cased-conversational', config='DeepPavlov/bert-base-cased-conversational')`

#### Output:

Some weights of BertForMaskedLM were not initialized from the model checkpoint at DeepPavlov/bert-base-cased-conversational and are newly initialized: ['cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.bias', 'cls.predictions.bias', 'cls.predictions.transform.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.


I used command

!python /content/transformers/examples/language-modeling/run_language_modeling.py \
    --output_dir=/content/output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=/content/input.txt \
    --do_eval \
    --eval_data_file=/content/dev.txt

and the error occurs

2020-09-08 06:02:43.113931: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
09/08/2020 06:02:45 - WARNING - __main__ -   Process rank: -1, device: cuda:0, n_gpu: 1, distributed training: False, 16-bits training: False
09/08/2020 06:02:45 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='/content/output', overwrite_output_dir=False, do_train=True, do_eval=True, do_predict=False, evaluate_during_training=False, prediction_loss_only=False, per_device_train_batch_size=8, per_device_eval_batch_size=8, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=3.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Sep08_06-02-45_58d9f15c989e', logging_first_step=False, logging_steps=500, save_steps=500, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=1000, past_index=-1, run_name=None, disable_tqdm=False, remove_unused_columns=True)
09/08/2020 06:02:45 - INFO - filelock -   Lock 140608954702032 acquired on /root/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.db13c9bc9c7bdd738ec89e069621d88e05dc670366092d809a9cbcac6798e24e.lock
Downloading: 100% 665/665 [00:00<00:00, 556kB/s]
09/08/2020 06:02:46 - INFO - filelock -   Lock 140608954702032 released on /root/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.db13c9bc9c7bdd738ec89e069621d88e05dc670366092d809a9cbcac6798e24e.lock
09/08/2020 06:02:46 - INFO - filelock -   Lock 140608954701528 acquired on /root/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71.lock
Downloading: 100% 1.04M/1.04M [00:00<00:00, 2.47MB/s]
09/08/2020 06:02:47 - INFO - filelock -   Lock 140608954701528 released on /root/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71.lock
09/08/2020 06:02:48 - INFO - filelock -   Lock 140608954701640 acquired on /root/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda.lock
Downloading: 100% 456k/456k [00:00<00:00, 1.37MB/s]
09/08/2020 06:02:48 - INFO - filelock -   Lock 140608954701640 released on /root/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda.lock
/usr/local/lib/python3.6/dist-packages/transformers/modeling_auto.py:821: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
09/08/2020 06:02:48 - INFO - filelock -   Lock 140608954702312 acquired on /root/.cache/torch/transformers/d71fd633e58263bd5e91dd3bde9f658bafd81e11ece622be6a3c2e4d42d8fd89.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1.lock
Downloading: 100% 548M/548M [00:16<00:00, 33.1MB/s]
09/08/2020 06:03:06 - INFO - filelock -   Lock 140608954702312 released on /root/.cache/torch/transformers/d71fd633e58263bd5e91dd3bde9f658bafd81e11ece622be6a3c2e4d42d8fd89.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1.lock
/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py:1321: FutureWarning: The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.
  FutureWarning,
Traceback (most recent call last):
  File "/content/transformers/examples/language-modeling/run_language_modeling.py", line 313, in <module>
    main()
  File "/content/transformers/examples/language-modeling/run_language_modeling.py", line 242, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "/content/transformers/examples/language-modeling/run_language_modeling.py", line 143, in get_dataset
    cache_dir=cache_dir,
TypeError: __init__() got an unexpected keyword argument 'cache_dir'

I'm working on the Google Colab environment


[Community notebooks] Add notebook on fine-tuning GPT-2 Model with Trainer Class

Adding a link to a community notebook containing an example of fine-tuning a German GPT-2 Model with the Trainer Class


[seq2seq Examples] Use _step instead of generate for val, test

Addresses a workaround I proposed in #6589 for limiting the memory consumption in `.generate` steps.

@sshleifer


On Gpu sharing mechanism, specify model.to(' CPU '), but still use the Gpu

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
You know that The GPU device(K8s) only supports one container exclusive GPU, In the inferencing stage, it is extremely wasteful. For Tiny-Albert model,It's only using about 500MiB。We try to use  GPU share device, support more containers use one GPU device。We expect using  torch.cuda.is_available() to control Using CUDA or Not.
Here is My Code:
`
use_cuda = bool(os.environ.get("USE_CUDA"))
if use_cuda and torch.cuda.is_available():
    self._device = torch.device("cuda")
else:
   self._device = torch.device("cpu")
`

1. USE_CUDA== False and  torch.cuda.is_available() is True:
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|============================================================|
|    0     11538      C   /usr/bin/python3                             **342MiB** |
+-----------------------------------------------------------------------------+
2. USE_CUDA== True and  torch.cuda.is_available() is True:
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|============================================================|
|    0     11538      C   /usr/bin/python3                             **342MiB** |
+-----------------------------------------------------------------------------+

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


How to bypass "Special tokens have been added in the vocabulary..." warning?

Is there a way to avoid always getting:

> Special tokens have been added in the vocabulary, make sure the associated word embedding are fine-tuned or trained

@ https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L1614

(other than turning logging off)

As far as I can see from stepping through that code, there are always special tokens (e.g. eos, pad, etc.), i.e. there is nothing special about it. What purpose does this warning serve when loading a tokenizer?

I'm not sure how the end user can act on the suggestion:

> make sure the associated word embedding are fine-tuned or trained

when they just want to run, say, the `generate` function on a pre-trained model, other than just learning to ignore this warning and not paying heed to when a warning is really saying something crucial.

Thoughts?


typo

apologies for the tiny PRs, just sending those as I find them.


access to the embeddings for query and text used in a downstream NLP task

Newbie question -

is there any way to access the embeddings  that are generated for the passage(and query) that is fed to a BertForQuestionAnswering model ?


fixed trainer tr_loss memory leak

Fixes #6939

The Trainer class contains the memory leak described [here](https://discuss.pytorch.org/t/cpu-ram-usage-increasing-for-every-epoch/24475/6). It is not specific to any particular model type and will occur with any model trained using the trainer class.

It is fixed in this pull request.

The issue is demonstrated in [this Colab notebook](https://colab.research.google.com/drive/1KQZEiZtfY14sDiAQnfw0gkKj9eC6XpTm?usp=sharing). The model trains for around 20 000 steps (this takes around 10 minutes on a t4) before using up all 12GB of RAM.


Fix TF Trainer loss calculation

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6968

## Description
Issue #6968 is about the incorrect loss calculation due to dividing the per example losses by the number of sentences, rather than by the number of tokens (not ignored, i.e. label != -100) - if the task is a token level task.

## Implementation
Before a whole batch being distributed to replicas, we compute the number of instances in that batch. Depending on the task types (sentence level or token level), the word `instance` means different things:
- sentence level task: it means examples
- token level task: it means the tokens with label != -100

This information (number of instances) is injected into global batches. While each replica receives a small batch, it use this information to correctly computing the scaled losses.

If no information is provided in the dataset, the default behavior is to use the number of examples in a global batch.
This way, the code change is minimal.

## Test code

    import os
    import random
    import shutil

    shutil.rmtree("./tmp/", ignore_errors=True)
    os.mkdir("./tmp/")

    nb_sentences = 70
    words = ["i", "like", "dogs", "but", "you", "prefer", "cats"]
    with open("./tmp/train.txt", "w", encoding="UTF-8") as fp:
        for i in range(nb_sentences):
            if i == 0:
                for word in words:
                    fp.write(f"{word} O\n")
            else:
                fp.write(f" \n")

            fp.write("\n")

    os.system("cp ./tmp/train.txt ./tmp/dev.txt")

    labels = ["O", "B-MISC", "I-MISC", "B-PER", "I-PER", "B-ORG", "I-ORG", "B-LOC", "I-LOC"]
    with open("./tmp/test.txt", "w", encoding="UTF-8") as fp:
        for i in range(nb_sentences):
            for word in words:
                label = random.choice(labels)
                fp.write(f"{word} {label}\n")

            fp.write("\n")

    command = (
        "python run_tf_ner.py "
        + "--model_name_or_path distilbert-base-uncased "
        + "--data_dir ./tmp/ --seed 2020 --output_dir ./tmp/ "
        + "--overwrite_output_dir --logging_steps 1 "
        + "--do_train --do_eval --do_predict "
        + "--num_train_epochs 1 "
        + f"--per_device_train_batch_size {nb_sentences} "
        + f"--per_device_eval_batch_size {nb_sentences} --labels '' "
        + "--max_seq_length 16"
    )

    print(command)

    os.system(command)

Testing it against master, you will see the loss values is smaller (~0.3) than testing against this PR code, you will (1.0 ~ 2.0), because on master, the denominator is `70` (but 69 of them having only ignored tokens) while on this PR, the denominator is `7` (the number of tokens).


run_squad.py not working on 3.1.0 version

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-91-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: True

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->@LysandreJik @sshleifer

## Information

Model I am using (Bert, XLNet ...): Bert

The problem arises when using:
* [x] the official example scripts: (give details below)
 run_squad.py is working with 2.9.1 but when I update it to 3.1.0 it gives error

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. !pip install transformers==3.1.0
It works when using 2.9.1 but not with this
2.
!mkdir dataset \
&& cd dataset \
&& wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json \
&& wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json
3.
!export SQUAD_DIR=/content/dataset \
&& python transformers/examples/run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v2.0.json \
  --predict_file $SQUAD_DIR/dev-v2.0.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 1.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /content/model_output \
  --save_steps 1000 \
  --threads 4 \
  --version_2_with_negative

Error message is:
2020-09-07 19:33:26.850641: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "transformers/examples/run_squad.py", line 74, in <module>
    (),
  File "transformers/examples/run_squad.py", line 73, in <genexpr>
    (tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, RobertaConfig, XLNetConfig, XLMConfig)),
AttributeError: type object 'BertConfig' has no attribute 'pretrained_config_archive_map'

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. --> It should start training without error just like older versions


[generation] decoder priority for choosing decoder_start_token_id value

`config.decoder` needs to be checked first before model's `config` to set `decoder_start_token_id`.

This is needed for https://github.com/huggingface/transformers/pull/6940 where I think for the first time there is an actual `config.decoder`


[from_pretrained] Allow tokenizer_type ≠ model_type

For an exemple usage of this PR, see the `tokenizer_class` attribute in this config.json: https://s3.amazonaws.com/models.huggingface.co/bert/julien-c/dummy-diff-tokenizer/config.json

Instead of a class, we could have used a `tokenizer_type` belonging to the set of all `model_type`s, like `"bert"`, etc. but it feels more restrictive, especially in case we start having tokenizer classes that are not obviously linked to a "model", like a potential "TweetTokenizer"

Context: https://github.com/huggingface/transformers/pull/6129

**Update: documented by @sgugger in https://github.com/huggingface/transformers/pull/8152**


Fix typo

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


PegasusForConditionalGeneration stops at unknown token

Hi all,

When following the code snippet from the [huggingface documentation](https://huggingface.co/transformers/master/model_doc/pegasus.html) but replacing the text, I have found that the summary stops when it reaches an unknown token. Is there a way around this?

src_text = [
"""As a child, the documentary maker, who was born in the Indian state of Kerala but now lives in Toronto, saw ceremonial elephants being paraded and thought they were beautiful. Later, she learned about the ordeal the animals are subjected to. "So many elephants had ghastly wounds on their hips, massive tumours and blood oozing out of their ankles, because chains had cut into their flesh and many of them were blind," Iyer told the BBC. She has made a documentary, Gods in Shackles, in an attempt to draw attention to the treatment of temple elephants she saw in India. "They were so helpless and the chains were so heavy," she said. "It was absolutely heart-breaking for me to witness this."""
]

Using this sample text, for example, my summary is:
As a child, the documentary maker, who was born in the Indian state of Kerala but now lives in Toronto, saw ceremonial elephants being paraded and thought they were beautiful, but later, she learned about the ordeal the animals are subjected to in India, in an attempt to draw attention to the treatment of templeunk_9


Mobile Bert Tiny model

# 🚀 Feature request

Can you add support for variants of MobileBERT?

## Motivation

The package currently provides various varients of bert - 'bert-base-cased', 'bert-base-uncased', 'bert-large'...
Can you also similarly provide Mobile Bert Tiny as well?


Conversion scripts shouldn't have relative imports


README for HooshvareLab/bert-fa-base-uncased

model card

Related to pretrained model cards

ParsBERT v2.0 is a fine-tuned and vocab-reconstructed version of ParsBERT, and it's able to be used in other scopes!

Some features:
- We added some unused-vocab for use in summarization and other scopes.
- We fine-tuned the model on vast styles of writing in the Persian language.


I'm fine tuning with the example `run_language_modeling.py`as follows.
```shell
python run_language_modeling.py --output_dir=output_dir --model_type gpt2 --model_name_or_path distilgpt2  --do_train  --train_data_file=xxx.data.txt
```
It failed with error
```shell
Traceback (most recent call last):
  File "run_language_modeling.py", line 313, in <module>
    main()
  File "run_language_modeling.py", line 242, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "run_language_modeling.py", line 143, in get_dataset
    cache_dir=cache_dir,
TypeError: __init__() got an unexpected keyword argument 'cache_dir'
```
Could you please tell me how to use it ? Thanks a lot.
<!-- A clear and concise description of what you would expect to happen. -->


t5 embed_tokens

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

I'm using the `run_language_modeling.py` to fine tuning the model `distilgpt2`. The command i used as follows:
```shell
python run_language_modeling.py --output_dir=output_dir --model_type gpt2 --model_name_or_path distilgpt2  --do_train  --train_data_file=data/data.txt  --overwrite_output_dir
```
But it core with following error
```shell
File "/data1/xxx/transformers/src/transformers/activations.py", line 30, in gelu_new
    return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 100663296 bytes. Error code 12 (Cannot allocate memory)
```
Does it core  duing to the insufficient memory on my machine? Hope for your help. Thanks a lot.

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Demoing LXMERT with raw images by incorporating the FRCNN model for roi-pooled extraction and bounding-box predction on the GQA answer set.

This is a follow up PR from initially incorporating LXMERT. This PR includes the Faster-RCNN code to convert raw-images into usable roi-pooled features downstream in lxmert or any other suitable vision model.


Enhance a MarianMT pretrained model from HuggingFace with more training data

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**: https://stackoverflow.com/questions/63774619/enhance-a-marianmt-pretrained-model-from-huggingface-with-more-training-data


Cannot index `None`

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6950


[generation] multiple eos/pad asserts/ifs in generate search functions

In `_generate_no_beam_search` `eos_token_id` is required: https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L731 (that code always get hit)

```
                    assert (
                        eos_token_id is not None and pad_token_id is not None
                    ), "generated beams >= num_beams -> eos_token_id and pad_token have to be defined"
```

why do we assert and check `eos_token_id is not None` multiple times through the code, why not assert once at the top of the function and then just use it?

Moreover, all those `if eos_token_id is not None` can be then removed (or reduced if there are other parts to them).

Also a larger question - is there a model where `eos_token_id` is not defined? If there is none, then why not assert once at the top of `generate` and then just use it everywhere in sub-calls without testing its definition?

Oh, I also see `pad_token_id` is used in `_generate_no_beam_search` w/o testing whether it's defined: https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L571
```
                tokens_to_add = next_token * unfinished_sents + (pad_token_id) * (1 - unfinished_sents)
```
is it the same situation as `eos_token_id` - that is it is always needed?

I see it's may be defined here: https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L355 but only if  `eos_token_id` is defined.
```
        if pad_token_id is None and eos_token_id is not None:
            logger.warning(
                "Setting `pad_token_id` to {} (first `eos_token_id`) to generate sequence".format(eos_token_id)
            )
            pad_token_id = eos_token_id
```
my thinking is that if this worked until now for all models, it's another proof that `eos_token_id` has to be required again.

in `_generate_no_beam_search` `pad_token_id` is required and similarly to `eos_token_id` can be asserted once on top and not multiple times through the code.

Thank you for reviewing my observations. It's possible that some (all?) are incorrect if I missed something.


[generation] consistently add eos tokens

Currently beam search returns inconsistent outputs - if hypos have different lengths we get eos, if they are the same - we don't. I found a sentence where if I change one letter in one of the input words the beam search outputs all suddenly lack eos, which is an inconsistent behavior.

This PR makes the output more consistent. (but not 100%, please see below)

---------

Also why not replace:

```
            if sent_lengths[i] < max_length:
                decoded[i, sent_lengths[i]] = eos_token_id
```
with:
```
            decoded[i, sent_lengths[i]] = eos_token_id
```
Shouldn't eos always be there? If generated data gets truncated, the caller needs to use a larger `max_length`.

Currently, if the hypos lengths are on the cusp of `max_length`, some of them will get eos, whereas others won't, which is again inconsistent.

Please correct me if my logic is flawed.

-----

I also looked at `_generate_no_beam_search` - there eos adding logic uses a somewhat different logic.

Should the two functions (beam/no_beam) be consistent eos-injection wise?


LongformerForQuestionAnswering sample code error

## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-4.19.112+-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

 Longformer/Reformer: @patrickvonplaten

## Information

Model I am using LongformerForQuestionAnswering
The problem arises when using:
[https://huggingface.co/transformers/v2.11.0/model_doc/longformer.html#longformerforquestionanswering](url)

## To reproduce

Steps to reproduce the behavior:

1. I ran the following code on Kaggle kernel

from transformers import LongformerTokenizer, LongformerForQuestionAnswering
import torch
tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
model = LongformerForQuestionAnswering.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa", return_dict=True)
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
encoding = tokenizer(question, text, return_tensors="pt")
input_ids = encoding["input_ids"]
attention_mask = encoding["attention_mask"]
outputs = model(input_ids, attention_mask=attention_mask)
start_logits = outputs.start_logits
end_logits = outputs.end_logits
all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())
answer_tokens = all_tokens[torch.argmax(start_logits) :torch.argmax(end_logits)+1]
answer = tokenizer.decode(tokenizer.convert_tokens_to_ids(answer_tokens)) # remove space prepending space token

**Following error occurred**

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-70f3a4bf4161> in <module>
      2 import torch
      3 tokenizer = LongformerTokenizer.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
----> 4 model = LongformerForQuestionAnswering.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa", return_dict=True)
      5 question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
      6 encoding = tokenizer(question, text, return_tensors="pt")

/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    653
    654         # Instantiate model.
--> 655         model = cls(config, *model_args, **model_kwargs)
    656
    657         if state_dict is None and not from_tf:

TypeError: __init__() got an unexpected keyword argument 'return_dict'

> and after doing the following changes:

 _encoding = tokenizer(question, text, return_tensors="pt")  ->   encoding = tokenizer(question, text)_

the following error occured

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-54aabf4ba7c0> in <module>
      1 question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
----> 2 encoding = tokenizer(question, text, return_tensors="pt")

TypeError: 'LongformerTokenizer' object is not callable


[gen utils] missing else case

1. `else` is missing - I hit that case while porting a model. Probably needs to assert there?
2. also the comment on top seems to be outdated (just vocab_size is being set there)


RunTime Error: CUDA out of memory when running trainer.train()

## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: P100 GPU instance (Google Colab)
- Using distributed or parallel set-up in script?: No

### Who can help
@sgugger @julien-c @LysandreJik

## Information

Model I am using (Bert, XLNet ...): RoBERTa with Byte-Pair Encoder (loading from the checkpointed pre-trained model on HuggingFace model hub).

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x ] my own modified scripts: (give details below)
I modify a variant of the [`01_how_to_train.ipynb`,](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb) replacing the `LinebyLineTextDataset` as it results in Out of Memory Issues with my text corpus. I built a hugging face NLP dataset which tokenizes the corpus.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below): the PubChem 1M SELFIES set, a set of one million SELFIES Strings. "SELFIES" is a 100% chemically valid molecular string representation. You can view the library [here](https://github.com/aspuru-guzik-group/selfies) (I'm one of the developers).

## To reproduce

Steps to reproduce the behavior:

Reproducing requires a copy of the `shard_00_selfies.txt` dataset (click [here](https://drive.google.com/file/d/1DRq8UgBaKSNfyYtqNMyQ4WimG64sSOFR/view?usp=sharing) for a drive link), as well as the tokenizer's files which can be loaded from the HuggingFace hub with the following name: `seyonec/BPE_SELFIES_PubChem_shard00_50k`

From there, you can just run a variant of the following colab file, with modified file paths of course: https://colab.research.google.com/drive/1a4edCW1b2rSVA_bkqEaywhvGMM_nQzRC?usp=sharing

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Strangely, the Trainer doesn't output a link to the Weights and Biases run page like it normally did before (using a near-identical script only a couple of days ago). It throws a CUDA out of memory error once I run the `trainer.train()` command:
![Screen Shot 2020-09-06 at 10 42 44 PM](https://user-images.githubusercontent.com/46096704/92343684-621a3e80-f092-11ea-8163-3cff1cd03e6c.png)
<img width="1200" alt="Screen Shot 2020-09-06 at 10 45 58 PM" src="https://user-images.githubusercontent.com/46096704/92343851-d8b73c00-f092-11ea-8c89-a0034191975d.png">

Thanks for the help! Any advice or help is desperately welcome, I've been stuck for the past days with various memory issues with tokenization, and now with running the trainer class for some reason. 😄

You can check out the main public repository for this project, alongside with the abstract, and more [here](https://github.com/seyonechithrananda/bert-loves-chemistry)!


[gen utils] missing else case

This code: https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L361-L369
```
        # current position and vocab size
        if hasattr(self.config, "vocab_size"):
            vocab_size = self.config.vocab_size
        elif (
            self.config.is_encoder_decoder
            and hasattr(self.config, "decoder")
            and hasattr(self.config.decoder, "vocab_size")
        ):
            vocab_size = self.config.decoder.vocab_size
```
1. `else` is missing - I hit that case while porting a model. Probably needs to assert there?
```
raise ValueError("either self.config.vocab_size or self.config.decoder.vocab_size need to be defined")
```

2. also the comment on top seems to be outdated (just `vocab_size` is being set there)


[s2s] warn if --fp16 for torch 1.6


LXMERT imports

# ❓ Questions & Help
Hello, I was very happy to see that LXMERT is being integrated into this library and I wanted to try it out. I got a KeyError in configuration_auto.py, as in CONFIG_MAPPING there was no 'lxmert'. Then, I re-installed transformers from source. The KeyError went away, but this time I encountered issues in modeling_auto.py as below:

```model = AutoModelWithLMHead.from_pretrained("unc-nlp/lxmert-base-uncased")```

/transformers/src/transformers/modeling_auto.py", line 841, in from_pretrained
    raise ValueError(

ValueError: Unrecognized configuration class <class 'transformers.configuration_lxmert.LxmertConfig'> for this kind of AutoModel: AutoModelWithLMHead.
Model type should be one of T5Config, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, MarianConfig, BartConfig, LongformerConfig, RobertaConfig, BertConfig, OpenAIGPTConfig, GPT2Config, MobileBertConfig, TransfoXLConfig, XLNetConfig, FlaubertConfig, XLMConfig, CTRLConfig, ElectraConfig, EncoderDecoderConfig, ReformerConfig.
`

In the example [here](https://huggingface.co/unc-nlp/lxmert-base-uncased), AutoModelWithLMHead is imported. However, because of the error above, I also tried LxmertPreTrainedModel, which led to some other error concerning the initialization of the weights. Importing LxmertModel instead seems to work out without errors. Would that be the correct way if I want to extract features from a pretrained model?

I would appreciate any help regarding this issue!

Best regards,
Ece


Created README for labse_bert model card

Related to pretrained model cards

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Create README.md

Related to pretrained model cards

<!-- This line specifies which issue to close after the pull request is merged. -->
Model Card for https://huggingface.co/akhooli/mbart-large-cc25-ar-en


Fixed the default number of attention heads in Reformer Configuration

<!-- This line specifies which issue to close after the pull request is merged. -->
Just a simple fix. The default number of attention heads was 2 instead of 12.


The configuration of 3.0.2 and 3.1.0 is not compatible

The configuration of **3.0.2** and **3.1.0** is not compatible.

For example, in https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L388, `config.chunk_size_feed_forward` should change to `config.get('chunk_size_feed_forward', 0)`, because there is not `chunk_size_feed_forward` config in former version.

```
class BertLayer(nn.Module):
    def __init__(self, config):
        super().__init__()
        # self.chunk_size_feed_forward = config.chunk_size_feed_forward
        self.chunk_size_feed_forward = config.get('chunk_size_feed_forward', 0)
        self.seq_len_dim = 1
        self.attention = BertAttention(config)
        self.is_decoder = config.is_decoder
        self.add_cross_attention = config.add_cross_attention
        if self.add_cross_attention:
            assert self.is_decoder, f"{self} should be used as a decoder model if cross attention is added"
            self.crossattention = BertAttention(config)
        self.intermediate = BertIntermediate(config)
        self.output = BertOutput(config)
```


distilled bart-large/bart-base

@sshleifer Is there a distilled bart model (not CNN/XSUM) model available?

Thanks!


Error installing transformers 3.1.0

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):
pipeline("zero-shot-classification")

The problem arises when using:
pip3 install transformers=="3.1.0"

The tasks I am working on is:
Just installing the package to use zero-shot-classification

## To reproduce

Steps to reproduce the behavior:

1. pip3 install transformers=="3.1.0"

Alternatively:
1. pip3 install tokenizers=="0.8.1.rc2"

Notes:
It seems that the tokenizers version '0.8.1.rc2' is the issue. I can install the system just fine on different systems by changing the version to '0.8.0' in transformers/setup.py. Alternatively pip3 install transformers=="3.1.0" tokenizers=="0.8.0" seems to be a working method of installation, but tokenizers version "0.8.1.rc2" still has the error.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

  error: build failed
  /tmp/pip-build-env-mey29riz/overlay/lib/python3.6/site-packages/setuptools/dist.py:452: UserWarning: Normalizing '0.8.1.rc2' to '0.8.1rc2'
    warnings.warn(tmpl.format(**locals()))
  cargo rustc --lib --manifest-path Cargo.toml --features pyo3/extension-module --release --verbose -- --crate-type cdylib
  error: cargo failed with code: 101
  ERROR: Failed building wheel for tokenizers
  Running setup.py clean for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The expected behaviour would be to install properly without error.


Incorrect loss calculation for the last batch in TFTrainer if dataloader_drop_last is False

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-115-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.7
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 -->
 tensorflow: @jplu

## Description

In [training_step()](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py#L595) in `trainer_tf.py`, we have

    scaled_loss = per_example_loss / self.total_train_batch_size

However, if `dataloader_drop_last=False`, the last bacth (before being distributed to replicas) won't necessary have `self.total_train_batch_size` examples. If we allow `dataloader_drop_last=False`, we need a way to dynamically calculate the actual number of examples in a global batch, and pass this information in some way to the replicas.


Potential incorrect loss calculation for TFTokenClassification in TFTrainer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-115-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.7
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 -->

 Trainer: @sgugger
 tensorflow: @jplu
 examples/token-classification: @stefan-it

Mostly for @jplu, potentially for @stefan-it (because the workaround I have in mind requires a bit change in the token classification dataset).

## Information

The problem arises when using:
* [x] The official example scripts:
    The involved scripts are:
    - https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py
    - https://github.com/huggingface/transformers/blob/master/examples/token-classification/run_tf_ner.py

  However, in order to demonstrate the issue in a more clear way, I use a minimal example which doesn't use directly these two scripts. See the description and code snippet below.

The tasks I am working on is:
* [x] Official token classification task in TensorFlow

## Description

In [trainer_tf.py](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py#L595), the loss calculation is calculated from `per_example_loss` divided by `total_train_batch_size`.

        per_example_loss, _ = self.run_model(features, labels, True)
        scaled_loss = per_example_loss / self.total_train_batch_size

Here `total_train_batch_size` is the size of a whole batch that will be distributed to (potentially) different replicas and optionally consisting of several smaller batches for accumulation steps.

For sentence level tasks, where each example (i.e. sentence) corresponds to a label (for example, sentence classification), the above loss calculation is correct.

However, for token level tasks like token classification, the the above loss seems incorrect to me. For such tasks, the loss should be the per example losses **divided by the number of real tokens involved in the batch**.

In [utils_ner](https://github.com/huggingface/transformers/blob/master/examples/token-classification/utils_ner.py#L75), `convert_examples_to_features` set labels to `-100` for padding tokens and other special tokens (`[CLS]`, `[SEP]`, etc), which are the places to be ignored for loss calculation. Therefore, the loss calculation should be the per example losses **divided by the number of labels that are not -100 in the \*_batch_\***.

By **\*_batch_\***, it should be careful that it is not the batch received by a single replica, and neither the smaller batch in a single accumulation step. It means `the whole batch that will be distributed to (potentially) different replicas and optionally consisting of several smaller batches for accumulation steps.` More precisely, it means a batch passed to [distributed_training_steps()](https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py#L651) - for the same reason as we divide per example losses by `total_train_batch_size` for sentence level tasks, rather than dividing it by the size of batch received by a single replica.

In order to calculate the correct loss values, we have to pass the global information - the number of labels that are not `-100` in a `global batch` to each replica. I don't know a clean way to do it, but for my own personal projects, I inject this extra information into global batches as a constant, and each replica receiving a distributed smaller batch will have this information to calculate the correct scaled losses.

(I have a notebook showing how to perform it, if you want to look it, let me know.)

## Code Snippets
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is a minimal example to demonstrate the issue.

Here, we have only one real example (sentence) and `n_empty_string` empty sentences.
Each empty sentence will give only [CLS], [SEP] and [PAD] tokens that will be ignored for token classification.

    import os
    os.environ['TF_DETERMINISTIC_OPS'] = '1'

    SEED = 42
    name = 'distilbert-base-uncased'
    seq_len = 8
    num_labels = 2
    n_empty_string = 10

    import tensorflow as tf
    tf.random.set_seed(SEED)

    strategy = tf.distribute.OneDeviceStrategy(device="/cpu:0")

    from transformers import TFTrainer, AutoConfig, AutoTokenizer, TFAutoModelForTokenClassification
    from transformers.training_args_tf import TFTrainingArguments

    text = [
        'My dog is cute'
    ]
    text.extend([''] * n_empty_string)
    n_examples = len(text)

    config = AutoConfig.from_pretrained(
        name,
        num_labels=num_labels
    )

    tokenizer = AutoTokenizer.from_pretrained(name)

    model = TFAutoModelForTokenClassification.from_pretrained(
        name
    )
    training_args = TFTrainingArguments(
        output_dir='./tmp/',
        per_device_train_batch_size=n_examples,
        gradient_accumulation_steps=1,
        seed=SEED
    )

    # Initialize our Trainer
    trainer = TFTrainer(
        model=model,
        args=training_args,
        train_dataset=None,
        eval_dataset=None,
        compute_metrics=None
    )
    trainer.total_train_batch_size = strategy.num_replicas_in_sync \
                                     * training_args.per_device_train_batch_size \
                                     * training_args.gradient_accumulation_steps
    trainer.train_loss = tf.keras.metrics.Sum()

    features = tokenizer.batch_encode_plus(text, max_length=seq_len, padding='max_length', return_tensors='tf')
    # Set all labels to `1`, except for special tokens: cls/sep/pad, where the labels are `-100`.
    labels = tf.constant(1, shape=[n_examples, seq_len])
    for token_id in [tokenizer.pad_token_id] + tokenizer.all_special_ids:
        labels = labels * tf.cast(features['input_ids'] != token_id, dtype=tf.int32) + \
                 -100 * tf.cast(features['input_ids'] == token_id, dtype=tf.int32)

    # Only the first example `features[0]` has real tokens, the other examples have only [PAD].
    print(features['input_ids'])

    # Only the first example has labels that won't be ignored.
    print(labels)

    # Copy from:
    #     https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py#L601
    per_example_loss, _ = trainer.run_model(features, labels, True)
    scaled_loss = per_example_loss / trainer.total_train_batch_size

    print(scaled_loss)

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
When `n_empty_string = 0`, we get `scaled_loss`

    tf.Tensor([0.56047076 0.46507886 0.51456743 0.50131255], shape=(4,), dtype=float32)

When `n_empty_string = 9`, we get `scaled_loss`

    tf.Tensor([0.05604707 0.04650789 0.05145674 0.05013125], shape=(4,), dtype=float32)

However, in both case, we should get the same value, which should be

    tf.Tensor([0.56047076 0.46507886 0.51456743 0.50131255], shape=(4,), dtype=float32)


hack to extract cross attention for bart decoder


SPM Tokenizer confusion with fairseq Roberta

Hi,

I have pre-trained a custom Roberta Model from scratch with a unigram sentencepiece model (also trained from scratch). I have converted the model from fairseq to huggingface with this [script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py) successfully.

I have tried loading the model in huggingface, which was successful, but the issue lies with the tokenizer, I tried using the Roberta tokenizer but it screamed at me because it was looking for merges and vocab. I then loaded my spm model with AlbertTokenizer, but when I try to test it out with a simple fill_mask, the answer tokens are incorrect. How do I correctly use my SPM model with Roberta? I also have dict.txt from fairseq.

Any help would be appreciated!


transformers-cli upload individual files simplification

Currently it's not possible to upload an individual file in a simple:
```
transformers-cli upload fsmt-wmt19-ru-en/vocab-src.json
```
Getting error:
```
Filename invalid, every file must be nested inside a "model_name" folder.
```
but, instead, have to do:
```
transformers-cli upload fsmt-wmt19-ru-en/vocab-src.json --filename fsmt-wmt19-ru-en/vocab-src.json
```
But this is silly as the exact same input is repeated twice and I'm more likely to make an error while copy-n-pasting when providing an explicit destination filename. Why not look at the relative path and use that? And only give that error when there is no "model_name" folder in the args. i.e. definitely give error on:
```
transformers-cli upload vocab.json
```
I understand the ` --filename` is useful for renaming, but there is no renaming here.

Additionally, if the first suggestion is acceptable, would it be OK to support multiple filenames? I want to be able to update several files in one go (just the config files, but not the whole folder, since the model is huge)

```
transformers-cli upload fsmt-wmt19-ru-en/vocab-*.json
```
gives error:
```
Transformers CLI tool: error: unrecognized arguments: fsmt-wmt19-ru-en/vocab-ru.json fsmt-wmt19-ru-en/vocab-src.json fsmt-wmt19-ru-en/vocab-tgt.json
```
Thanks.


Create README.md model card

Related to pretrained model cards

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}
model card for https://huggingface.co/rjbownes/Magic-The-Generating?text=Once+upon+a+time%2C


Longformer config - vocabulary size

## Environment info
- `transformers` version: 3.1.0

### Who can help
 Longformer/Reformer: @patrickvonplaten

## Information

Why does the LongformerConfig's vocab_size defaults to 30522 while the Longformer has embeddings first dimension 50265? It reuses the same config as RoBERTa which also has embeddings shape (50265,768) due to the BPE tokenization (and also defaults to 30522).

```python
LongformerConfig {
  "attention_probs_dropout_prob": 0.1,
  "attention_window": 512,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "longformer",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "sep_token_id": 2,
  "type_vocab_size": 2,
  "vocab_size": 30522
}
```


Tokenizers became slow compared to 2.8.0

## Environment info

- `transformers` version: 3.1.0
- Platform: Ubuntu 20.04
- Python version: 3.7.9
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 tokenizers: @mfuntowicz

## To reproduce

Steps to reproduce the behavior:

```python
import timeit
import numpy as np
from transformers import __version__ as trans_version
from transformers import (
    CTRLTokenizer,
    GPT2Tokenizer,
    RobertaTokenizer,
    XLMTokenizer,
    XLNetTokenizer
)

Tok_CLASSES = {
    "gpt2": GPT2Tokenizer,
    "ctrl": CTRLTokenizer,
    "roberta-large": RobertaTokenizer,
    "xlnet-large-cased": XLNetTokenizer,
    "xlm-mlm-100-1280": XLMTokenizer,
}

print(trans_version)
for k, v in Tok_CLASSES.items():
    tokenizer_class = v
    tokenizer = tokenizer_class.from_pretrained(k)

    text = """<|endoftext|>🤗 Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides
                general-purpose
                architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet…) for Natural Language Understanding (NLU)
                and Natural Language Generation (NLG) with over 32+ pretrained models in 100+ languages and deep
                interoperability between TensorFlow 2.0 and PyTorch.</s> <eos>"""

    print(tokenizer.__class__)
    r = timeit.repeat(stmt="tokenizer.encode(text)", repeat=100, number=500, globals=globals())
    rounding = 4
    print('Time taken (mean ± 3std):',
          str(np.round(np.mean(r), rounding)) + '±' + str(np.round(3 * np.std(r), rounding)))

```
In 3.1.0 output is:
```
3.1.0
<class 'transformers.tokenization_gpt2.GPT2Tokenizer'>
Time taken (mean ± 3std): 0.1808±0.0115
<class 'transformers.tokenization_ctrl.CTRLTokenizer'>
Time taken (mean ± 3std): 0.0678±0.0015
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
Time taken (mean ± 3std): 0.2051±0.0024
<class 'transformers.tokenization_xlnet.XLNetTokenizer'>
Time taken (mean ± 3std): 0.1567±0.002
<class 'transformers.tokenization_xlm.XLMTokenizer'>
Time taken (mean ± 3std): 0.3601±0.0248
```

## Expected behavior

In 2.8.0 output is (and i hope, even these times can be improved without using Fast versions).
```
<class 'transformers.tokenization_gpt2.GPT2Tokenizer'>
Time taken (mean ± 3std): 0.1808±0.0115
<class 'transformers.tokenization_ctrl.CTRLTokenizer'>
Time taken (mean ± 3std): 0.0678±0.0015
<class 'transformers.tokenization_roberta.RobertaTokenizer'>
Time taken (mean ± 3std): 0.2051±0.0024
<class 'transformers.tokenization_xlnet.XLNetTokenizer'>
Time taken (mean ± 3std): 0.1567±0.002
<class 'transformers.tokenization_xlm.XLMTokenizer'>
Time taken (mean ± 3std): 0.3601±0.0248
```
## TLDR
With GPT2 and CTRL 3.1.0 tokenizer.encode (with default options) takes ~1.3x time compared to 2.8.0 code. I think this can be improved|solved if code like https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L254-L256 was not executed every tokenization call but for example once tokens added (and result stored in self.all_special_tokens_extended). Maybe this is not only place with unnecessary calculations per tokenization.
For example `self.encoder.get(self.unk_token)` from https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_gpt2.py#L244 could be stored once in some property updated with unk_token change to reduce time to call it every time token-to-id conversion happens.
Same storage idea for https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L266


adding TRANSFORMERS_VERBOSITY env var

Per discussion at https://github.com/huggingface/transformers/pull/6816#issuecomment-686347433, this PR:

- adds `TRANSFORMERS_VERBOSITY` env var
- docs
- tests
- new test utils

I'm open to a different name if that one doesn't work.

Thank you.


create model card for astroGPT

Related to pretrained model cards


typo

there is no var `decoder_input_ids`, but there is `input_ids` for decoder :)


[testing] add dependency: parametrize

unittest doesn't support pytest's super-handy `@pytest.mark.parametrize`, I researched and there are many proposed workarounds, most are tedious at best. If we include https://pypi.org/project/parameterized/ in dev's testing dependencies - it will provide a very easy to write parameterization in tests. It provides the same functionality as pytest's fixture, plus quite a few other ways.

Example:
```
from parameterized import parameterized
@parameterized([
    (2, 2, 4),
    (2, 3, 8),
    (1, 9, 1),
    (0, 9, 0),
])
def test_pow(base, exponent, expected):
   assert_equal(math.pow(base, exponent), expected)
```
(extra `self`var if inside a test class)

To remind the pytest style is slightly different:
```
    @pytest.mark.parametrize("test_input,expected", [("3+5", 8), ("2+4", 6), ("6*9", 42)])
    def test_eval(test_input, expected):
```
More examples here: https://pypi.org/project/parameterized

May I suggest that it will make it much easier to write some types of tests?

And I have an immediate use for it, in the current PR I'm working on. So it's not just nice to have request.

Thank you.


PRETRAINED_INIT_CONFIGURATION for local model path

Tokenizers have a special dict `PRETRAINED_INIT_CONFIGURATION`, which tells the tokenization_utils_base, which extra args to pass to the tokenizer's `__init__`, except it doesn't work for the local model, as the hash is for online s3 models.

I have:

```
PRETRAINED_INIT_CONFIGURATION = {
    "stas/fsmt-wmt19-ru-en": {
        "langs": ["ru", "en"],
    },
    "stas/fsmt-wmt19-en-ru": {
        "langs": ["en", "ru"],
    },
    "stas/fsmt-wmt19-de-en": {
        "langs": ["de", "en"],
    },
    "stas/fsmt-wmt19-en-de": {
        "langs": ["en", "de"],
    },
}

```

So in my own code I use

```
   if LOCAL:
        path = "/code/huggingface/transformers-fair-wmt/data/fsmt-wmt19-ru-en/"
        mname = path
        mname_tok = f"stas/fsmt-wmt19-{src}-{tgt}"
        tokenizer = FSMTTokenizer.from_pretrained(mname_tok)
        model = FSMTForConditionalGeneration.from_pretrained(mname)
    else:
        # # s3 uploaded model
        mname = f"stas/fsmt-wmt19-{src}-{tgt}"
        tokenizer = FSMTTokenizer.from_pretrained(mname)
        model = FSMTForConditionalGeneration.from_pretrained(mname)
```

So `mname_tok` overrides to look up the dict above, since it fails to find that entry using local path.

This, however, doesn't work in tools that aren't under my control, `run_eval.py` in seq2seq for example.

**edit**, more to it - it doesn't pass `PRETRAINED_VOCAB_FILES_MAP` args either for the same reason - it fails to look up those entries for local path. I need to change them all.

Any suggestions on how to fix this problem?


[doc] remove the implied defaults to :obj:`None`, s/True/ :obj:`True/, etc.

as discussed at https://github.com/huggingface/transformers/pull/6932#issuecomment-687362952
**edit**: I also threw in  :obj:`True/False - anything else?
```
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e 's|, defaults to :obj:.None.||'  {} \;
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e 's|, defaults to True|, defaults to :obj:`True`|'  {} \;
find . -type d -name ".git" -prune -o -type f -exec perl -pi -e 's|, defaults to False|, defaults to :obj:`False`|'  {} \;
```
@sgugger


[WIP] Language modeling example for TF Trainer

To support language modeling for TF Trainer.


How to insert a hidden output from GPT2 model directly into a BERT layer?

Hello,

I am trying to do the following:
1. Feed in the input_ids into the embedding layer of the pre-trained GPT2 model, and get the resulting embedding.
2. directly feed the embedding from 1. to each layer of the pre-trained GPT2, by using `GPT2_model.transformers.h`.
Store the resulting hidden output from each layer of the GPT-2 in a tensor named `layer_hidden_state`.
3. Directly input the `layer_hidden_state` into the 1st layer (which is on the top of the embedding layer) of the pre-trained BertModel, and let BertModel process the `layer_hidden_state` until it reaches the uppermost layer of BertModel.

My attempts for carrying out the above steps are shown below.
But I am getting an error when I try to do the step 3...how can I fix my error? the error is shown at the bottom of my code.

Thank you for the help,

```Python
    # turn on the evaluation mode
    # (to prevent the dropout for evaluation purpose).
    gpt2DoubleHeadsModel.eval()
    len_input_ids = len(input_ids)

    # get the hidden state vector from the embedding layer.
    # we will use this hidden state vector as an input to each layer.
    input_hidden_state = gpt2DoubleHeadsModel(input_ids=input_ids,
                                                         mc_token_ids = mc_token_ids,
                                                         token_type_ids = token_type_ids,
                                                         attention_mask = attention_mask)[3][0][:,:,:].detach()

    for j in range(num_layer_gpt2):

        # directly feed in the embedding hidden state vector into each layer of the GPT2DoubleHeadsModel,
        # and retrieve the resulting hidden state vector from each layer.
        layer_hidden_state = \
                    gpt2DoubleHeadsModel.transformer.h[j](input_hidden_state)[0][:,(len_input_ids-1),:]

        # store the hidden state vectors of the last token from each layer in `last_hidden_output_tensor`.
        last_hidden_output_tensor[:,j,:] = layer_hidden_state

    last_hidden_output_tensor = tuple(last_hidden_output_tensor)

    best_model_bert = BertModel.from_pretrained('bert-large-uncased', output_hidden_states=True)

    # an error is generated here the error is shown below:
    for k in range(nlayer_bert):
        last_hidden_output_tensor = best_model_bert.encoder.layer[k]((last_hidden_output_tensor)[0])

"""
 error:
  File "/Users/hyunjindominiquecho/opt/anaconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 239, in transpose_for_scores
    return x.permute(0, 2, 1, 3)

RuntimeError: number of dims don't match in permute
"""
```


[s2s] run_eval supports --prefix clarg.

- Useful for multilingual models.
- `model.config.prefix` will still be used if prefix not passed. This prefix is added to the beginning of each example from the  source document before calling `generate`.
- `decoder_start_token_id` is a different thing and unaffected.

Usage:

```bash
export dd=wmt_en_de
python run_eval.py Helsinki-NLP/opus-mt-en-gem \
	$dd/val.source \
	$dd/marian_multi_val_gens.txt \
	--reference_path $dd/val.target \
	--task translation --fp16  --bs 128  \
	--score_path $dd/marian_multi_val_bleu.json \
	--prefix ">>deu<<"
```


typo


How to enable grad_fn when calling the generate() method of a T5 model

I was trying to attribute a prediction by a T5 model to the words of an input by gradient method. For a ```T5_model```, I call ```T5_model(**inputs)``` when training, and call ```T5_model.generate(**inputs)``` when doing inference. In training, the grad_fn is enabled for the loss but not in inference.

So how do I enable grad_fn when calling the ```generate()``` method to generate a prediction? So that I can get the gradient of the prediction with respect to each word of the input sentence.


head_mask in modeling_bert.py

Should change https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L487 `head_mask[i]` to `head_mask[i] if head_mask is not None else None`.

full code:

```
class BertEncoder(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.layer = nn.ModuleList([BertLayer(config) for _ in range(config.num_hidden_layers)])

    def forward(
        self,
        hidden_states,
        attention_mask=None,
        head_mask=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
        output_attentions=False,
        output_hidden_states=False,
        return_dict=False,
    ):
        all_hidden_states = () if output_hidden_states else None
        all_attentions = () if output_attentions else None
        for i, layer_module in enumerate(self.layer):
            if output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            if getattr(self.config, "gradient_checkpointing", False):

                def create_custom_forward(module):
                    def custom_forward(*inputs):
                        return module(*inputs, output_attentions)

                    return custom_forward

                layer_outputs = torch.utils.checkpoint.checkpoint(
                    create_custom_forward(layer_module),
                    hidden_states,
                    attention_mask,
                    head_mask[i] if head_mask is not None else None,
                    encoder_hidden_states,
                    encoder_attention_mask,
                )
            else:
                layer_outputs = layer_module(
                    hidden_states,
                    attention_mask,
                    head_mask[i] if head_mask is not None else None,
                    encoder_hidden_states,
                    encoder_attention_mask,
                    output_attentions,
                )
            hidden_states = layer_outputs[0]
            if output_attentions:
                all_attentions = all_attentions + (layer_outputs[1],)

        if output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        if not return_dict:
            return tuple(v for v in [hidden_states, all_hidden_states, all_attentions] if v is not None)
        return BaseModelOutput(
            last_hidden_state=hidden_states, hidden_states=all_hidden_states, attentions=all_attentions
        )
```


Refactoring the generate() function

## Generation refactor

This is a possible design that IMO would make generate much more readable and flexible for future code changes.
The code shown in this PR is more or less pseudo code, but I'm sure that this design is fully backwards compatible (for the `generate()` function, I don't plan on keep `beam_search_generation` and `_no_beam_search_generation` -> I don't think anybody directly accessed these functions).

It's probably better to look at the code directly, here: https://github.com/huggingface/transformers/blob/3c8a12a3bf44aa7845675852e28c47d0a9cb808e/src/transformers/generation_utils.py then at the diff.

The following major changes are made:

### Split the generate method into four generate functions:

a) `greedy_search` (corresponds to num_beams = 1, do_sample=False)
b) `sample` (corresponds to num_beams = 1, do_sample=True)
c) `beam_search` (corresponds to num_beams > 1, do_sample = False)
d) `beam_sample` (corresponds to num_beams > 1, do_sample = True)

It is split in a way that the functions can be used on their own and don't necessarily have to be accessed by the `generate()` method. This allows for much more flexibility for the user - he can decide what kind of distribution warper he wants to use and what kind of "Beam scorer" he wants to use (more on "distribution warper" and "beam scorer" later.
     Also mostly a model only uses one of these methods, `EncoderDecoder` models usually use `beam_search()`, `...ForCausalLM` models usually use `sample()`. Because `generate` is now split to the corresponding functions relevant for a model, it makes the code much more readable for the users because they will mostly only look into  one of the four functions.
    Splitting `generate()` into four functions removes **a lot** of `if-else` statements

### Create `LogitsProcessor` and `LogitsWarper` objects.

Instead of adding each of these "logit warpers / processors", such as `bad_token_words` with `if-else` statements, a list of these objects is created in the beginning and then called in the code. This is largely copied from this very nice PR: https://github.com/huggingface/transformers/pull/5420 . The arguments are 1) easier to test, 2) adds more flexbility because users can easily add their own "logit warpers / processors".
Note that we need both a `pre_processor` and a `dist_warper` list to make `beam_sample` work correctly. This comment explains why in more detail: https://github.com/huggingface/transformers/pull/5420#discussion_r449779867

### Create `beam_scorer` class.

The `generate_beam_search` function has become extremely hard to read because most of the beam search logic is directly written into the function. I would propose to move this code into a `beam_scorer` class that would also replace the `BeamHypotheses` class. The class would essentially expose the following functions:

```python
beam_scorer.update(next_scores, next_tokens)

next_beam_scores = beam_scorer.get_next_scores()
next_beam_tokens = beam_scorer.get_next_tokens()
next_beam_idx = beam_scorer.get_next_beam_idx()

beam_scorer.is_done()
```

and IMO all the beam search relevant code can be handled in the `beam_scorer.update(...)` function.
Besides better readability, such a class could also be replaced by another "Beam seach scorer" logic, which makes the beam search code much easier to extend IMO.

The con of this PR is obviously that some code would be copy pasted along the four functions (but it should not be too much).

Let me know if something is not clear in the design. Would be super happy to hear your feedback @LysandreJik @thomwolf @sshleifer @yjernite @JetRunner @sgugger

## 1st Review:

The current state shows how the refactored code would look like. All important tests are passing now (the slow GPT2, Bart & T5 tests), but the PR is not at all finished yet. It would be nice if @sshleifer @yjernite @thomwolf (and @LysandreJik @sgugger) could take a first look at the complete "new" architecture and give some feedback. If you guys are ok with the new design, I will add a bunch of tests and clean up the code more to make sure we have 100% backward compatibily.

## TODO:
- [x] Make all slow tests pass
- [x] Rename according to discussion
- [x] Add tests for all processors
- [x] Add more and better generation tests
- [x] Do speed comparison
- [x] Add docstring
- [x] Final thoughts about design

## Final review:

I'm happy with the changes I've done now. Better tests (more aggressive and should also be faster since only 3 tokens are generated instead of 10 now) have been added and docstrings have hopefully become more understandable.
A complete explanation of new "generation" philosophy is described in the forum: https://discuss.huggingface.co/t/big-generate-refactor/1857


[s2s] run_eval.py parses generate_kwargs

You can now run

```bash
td=test_data/wmt_en_ro
python run_eval.py t5-base $td/val.source preds.txt --reference_path $td/val.target \
        --score_path metrics.json --length_penalty 0.2 \
         --task translation_en_to_ro --num_beams 2 --n_obs 2 --bs 1 --length_penalty 0.6
```

h/t @stas00 in #6369


Training script for other language (except English)

Dear Friends. I have found this very nice and great.
I would like to ask, is there any script which can be used to train other language model?
and how much minimum GPU power is required for such training?

If you can guide . ! I am thinking to test for German language.


[LXMERT] Fix tests on gpu

Some tensors and models were created in tests without specifying the device.