'pad_to_max_length' in Pipeline should be set to True by default

# 🐛 Bug
pad_to_max_length is set by default False in Piplene class' _parse_and_tokenize() function
## Information

Model I am using (Bert):

Language I am using the model on (English):

The problem arises when using: my own modified scripts:
```
import numpy as np
from transformers import AutoTokenizer, pipeline, TFDistilBertModel
model = TFDistilBertModel.from_pretrained('distilbert-base-uncased')

# model = AutoModel.from_pretrained('distilbert-base-uncased')
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-uncased', pad_to_max_length=True)
pipe = pipeline('feature-extraction', model=model, tokenizer=tokenizer)

features = pipe(train_data['comment_text'][:100].values.tolist())
features = np.squeeze(features)
print(features.shape)
```
As there are about 100 input of variable length, tokenizer should perform padding. But even after giving ```pad_to_max_length=True``` , padding operation is not perform.

I get the following error
```
~/anaconda3/envs/tf2/lib/python3.7/site-packages/transformers/pipelines.py in predict(self, X)
    392         Scikit / Keras interface to transformers' pipelines. This method will forward to __call__().
    393         """
--> 394         return self(X=X)
    395
    396     @contextmanager

~/anaconda3/envs/tf2/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    551
    552     def __call__(self, *args, **kwargs):
--> 553         return super().__call__(*args, **kwargs).tolist()
    554
    555

~/anaconda3/envs/tf2/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *texts, **kwargs)
    465
    466     def __call__(self, *texts, **kwargs):
--> 467         inputs = self._parse_and_tokenize(*texts, **kwargs)
    468         return self._forward(inputs)
    469

~/anaconda3/envs/tf2/lib/python3.7/site-packages/transformers/pipelines.py in _parse_and_tokenize(self, pad_to_max_length, *texts, **kwargs)
    456             return_tensors=self.framework,
    457             max_length=self.tokenizer.max_len,
--> 458             pad_to_max_length=pad_to_max_length,
    459         )
    460

~/anaconda3/envs/tf2/lib/python3.7/site-packages/transformers/tokenization_utils.py in batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, max_length, stride, truncation_strategy, pad_to_max_length, return_tensors, return_token_type_ids, return_attention_masks, return_overflowing_tokens, return_special_tokens_masks, return_offsets_mapping, return_input_lengths, **kwargs)
   1260                             raise ValueError(self.NO_PAD_TOKEN_FOR_BATCH_MSG)
   1261                         else:
-> 1262                             raise ValueError(self.UNEVEN_SEQUENCES_FOR_BATCH_MSG)
   1263                 elif return_tensors == "pt" and is_torch_available():
   1264                     try:

ValueError: The sequences building the batch are not of the same size, no tensor can be built. Set `pad_to_max_length=True` to pad the smaller sequencesup to the larger sequence's length.
```
The tasks I am working on is:
* [ ] my own task or dataset: (give details below)
kaggle toxic tweet dataset.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Pipeline should perform padding operation.
<!-- A clear and concise description of what you would expect to happen. -->
When I made pad_to_max_length=True inside function  _parse_and_tokenize() of class Pipeline,
I got expected result. Pipeline perform its task perfectly (padding operation was also done) and feature extraction was executed on all inputs (in my case 100)
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-5.3.0-46-generic-x86_64-with-debian-buster-sid
- Python version:3.8
- Tensorflow version (GPU?): 2.1
- Using GPU in script?: Yes


Bug in run_glue

Hi
I am getting this error when running run_glue.py

ImportError: cannot import name 'TrainingArguments' from 'transformers' (/idiap/user/rkarimi/libs/anaconda3/envs/iborn/lib/python3.7/site-packages/transformers/__init__.py)

Traceback (most recent call last):
  File "run_glue.py", line 34, in <module>
    from transformers import (
ImportError: cannot import name 'HfArgumentParser' from 'transformers' (/idiap/user/rkarimi/libs/anaconda3/envs/iborn/lib/python3.7/site-packages/transformers/__init__.py)

To fix I searched in the repo on how you use the hf_argparser, I modified it as below:
from transformers.hf_argparser import HfArgumentParser

again, getting  the error:
Traceback (most recent call last):
  File "run_glue.py", line 44, in <module>
    from transformers.hf_argparser import HfArgumentParser
ModuleNotFoundError: No module named 'transformers.hf_argparser'

however, this is how you call it in the tests but this does not work. Seems to me that things have been changed but the codes are not updated.  thanks


Electra for question answering

# 🚀 Feature request

Electra for question answering

## Motivation

Electra is the highest rated single model (non essemble) on the Squad leaderboard

## Your contribution

I am not sure if I have the skills, but I'm willing to take a crack at it! Looking the other QA architectures, it seems that I'll need to put a single linear layer (two outputs) on top of the Electra discriminator ?


Share more details on fine-tuning GPT-2 on WikiText-2 ?

Hello! Regarding https://github.com/huggingface/transformers/tree/master/examples#gpt-2gpt-and-causal-language-modeling, would you mind sharing what hyper-parameters you use to get this result ? How many epochs, what's the batch size? etc...


Roberta (and BERT) tokenization converts "do not" to "don't"

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Execute the following:
```python
import transformers

tokenizer = transformers.RobertaTokenizer.from_pretrained('roberta-base')
print(tokenizer.decode(tokenizer.encode('is not')))
print(tokenizer.decode(tokenizer.encode('do not')))
```

The output is
```
<s> don't</s>
<s> is not</s>
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The detokenization should not incorrectly introduce a contraction.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-5.5.3-arch1-1-x86_64-with-glibc2.2.5
- Python version: 3.8.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


list index out of range error when I execute a command with examples/run_glue.py

 Hi, I am new to transformers, and I got a list index out of range error when I execute a command for examples/run_glue.py.
 I want to do Fine-Turning to classify Japanese words, and I modified some files following a web site.

**The process was:**
**1. I used these commands below to install transformers and to use examples following github's instructions.**

```
$ pip install transformers
$ git clone https://github.com/huggingface/transformers
$ cd transformers
$ pip install .
$ pip install -r ./examples/requirements.txt
```

**2. I changed two files(transformers/data/processors/glue.py, transformers/data/metrics/__init__.py)**
 I will show you them last of this question.

**3. I made train.tsv and dev.tsv under the data/original/ directory after making this directory.**
 I will show you them last of this question.

**4. I executed a command below.**

```
$ python ./examples/run_glue.py
    --data_dir=./src/transformers/data/original/
    --model_type=bert
    --model_name_or_path=bert-base-japanese-whole-word-masking
    --task_name=original
    --do_train
    --do_eval
    --output_dir=output/original

```

**5. list index out of range error occurred.**

```
Traceback (most recent call last):
  File "./examples/run_glue.py", line 562, in <module>
    main()
  File "./examples/run_glue.py", line 510, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "./examples/run_glue.py", line 358, in load_and_cache_examples
    processor.get_dev_examples(args.data_dir) if evaluate else processor.get_train_examples(args.data_dir)
  File "/home/haoki/Bert1/lib/python3.6/site-packages/transformers/data/processors/glue.py", line 519, in get_train_examples
    return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")
  File "/home/haoki/Bert1/lib/python3.6/site-packages/transformers/data/processors/glue.py", line 538, in _create_examples
    label = line[1]
IndexError: list index out of range

```

**My environment:**
OS: linux
IDE: pycharm
python: python 3.6

I stacked almost for a day with problem... Pleeease help meeee.

--------------------------------------------------------------------------
**Codes of procedure 2 and 3:**
**transformers/data/processors/glue.py　(Transformers are installed with pip):**
```

~~~
#added this class
class OriginalProcessor(DataProcessor):
    """Processor for the original data set."""
    def get_example_from_tensor_dict(self, tensor_dict):
        """See base class."""
        return InputExample(
            tensor_dict["idx"].numpy(),
            tensor_dict["sentence"].numpy().decode("utf-8"),
            None,
            str(tensor_dict["label"].numpy()),
        )

    def get_train_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "train.tsv")), "train")

    def get_dev_examples(self, data_dir):
        """See base class."""
        return self._create_examples(self._read_tsv(os.path.join(data_dir, "dev.tsv")), "dev")

    def get_labels(self):
        """See base class."""
        return ["0", "1"]

    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            # If TSV file has a header, it will take off it.
            # if i == 0:
            #     continue
            guid = "%s-%s" % (set_type, i)
            text_a = line[0]
            label = line[1]
            examples.append(InputExample(guid=guid, text_a=text_a, text_b=None, label=label))
        return examples

glue_tasks_num_labels = {
    "cola": 2,
    "mnli": 3,
    "mrpc": 2,
    "sst-2": 2,
    "sts-b": 1,
    "qqp": 2,
    "qnli": 2,
    "rte": 2,
    "wnli": 2,

    "original": 2, # added
}

glue_processors = {
    "cola": ColaProcessor,
    "mnli": MnliProcessor,
    "mnli-mm": MnliMismatchedProcessor,
    "mrpc": MrpcProcessor,
    "sst-2": Sst2Processor,
    "sts-b": StsbProcessor,
    "qqp": QqpProcessor,
    "qnli": QnliProcessor,
    "rte": RteProcessor,
    "wnli": WnliProcessor,

    "original": OriginalProcessor, # added
}

glue_output_modes = {
    "cola": "classification",
    "mnli": "classification",
    "mnli-mm": "classification",
    "mrpc": "classification",
    "sst-2": "classification",
    "sts-b": "regression",
    "qqp": "classification",
    "qnli": "classification",
    "rte": "classification",
    "wnli": "classification",

    "original": "classification", # added
}

```

**transformers/data/metrics/__init__.py (Transformers are installed with pip)**

```
    def glue_compute_metrics(task_name, preds, labels):
        assert len(preds) == len(labels)
        if task_name == "cola":
            return {"mcc": matthews_corrcoef(labels, preds)}
        elif task_name == "sst-2":
            return {"acc": simple_accuracy(preds, labels)}
        elif task_name == "mrpc":
            return acc_and_f1(preds, labels)
        elif task_name == "sts-b":
            return pearson_and_spearman(preds, labels)
        elif task_name == "qqp":
            return acc_and_f1(preds, labels)
        elif task_name == "mnli":
            return {"acc": simple_accuracy(preds, labels)}
        elif task_name == "mnli-mm":
            return {"acc": simple_accuracy(preds, labels)}
        elif task_name == "qnli":
            return {"acc": simple_accuracy(preds, labels)}
        elif task_name == "rte":
            return {"acc": simple_accuracy(preds, labels)}
        elif task_name == "wnli":
            return {"acc": simple_accuracy(preds, labels)}

        # added
        elif task_name == "original":
            return {"acc": simple_accuracy(preds, labels)}

        else:
            raise KeyError(task_name)

````

**train.tsv**
```

面白かった  0 #interesting
楽しかった  0 #fun
退屈だった  1 #boring
悲しかった  1 #sad
```

**dev.tsv**
```

満喫した  0 #satisfied
辛かった  1 #hard

````


[TF T5] Higher tolerance for past testing in TF T5

Higher tolerance to be certain that tests pass


[T5] Higher tolerance for past testing in T5


Fix bug in run_*.py scripts: double wrap into DataParallel during eval

This bug is present in several scripts in `examples`:
* `examples/run_language_modeling.py`
* `examples/run_multiple_choice.py`
* `examples/run_xnli.py`
* `examples/ner/run_ner.py`
* `examples/mm-imdb/run_mmimdb.py`
* `examples/hans/test_hans.py`

The problem is exactly the same as it was in #1801 and in #1504:

During the evaluation, we are trying to wrap the `model` into `DataParallel` second time (we did it already during training). As a result we have:
> "RuntimeError: module must have its parameters and buffers on device cuda:0 (device_ids[0]) but found one of them on device: cuda:1" (ids of devices may differ)

The fix is straightforward:
Before:
```python
# multi-gpu eval
if args.n_gpu > 1:
    model = torch.nn.DataParallel(model)
```

After:
```python
# multi-gpu eval
if args.n_gpu > 1 and not isinstance(model, torch.nn.DataParallel):
    model = torch.nn.DataParallel(model)
```


Reproducing squad score with TFXLMRoberta?

# ❓ Questions & Help

Hello all,

first of all, thanks for the library that is very helpful. There have been several discussions regarding XLM-Roberta and questions answering (#3732  #3694 ). On my side, I added a TFXLMRobertaForQuestionAnswering but never reproduced a decent squad score (I was always below 50% f1). The base LM I was using was xlm-roberta-base converted to tf, or the community ones (jplu), I tried with type_vocab_size=1 or type_vocab_size=2 (in order to use segment_ids as for Bert, I did it by overwriting the create_token_type_ids_from_sequences in the tokenizer). This did not really change anything. I am using a AdamWeightDecay as for Bert finetuning and I start to believe that there is no problem with my code since I saw yesterday the PR #3812 which is basically the same code as mine. For info, I am not only training on squad, I am training on squad + mlqa. Using the exact same approach, I got very good score with Bert multilingual. Since I am a bit stuck I was wondering if someone here (from huggingface or not) managed to properly train XLMRoberta with tensorflow and get good squad results. If so, it would be super helpful for me if you can share the parameters that was used (learning rate, number of epochs, etc).

**


Decoding predictions for masked language modeling task using custom BPE

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
https://stackoverflow.com/questions/61232399/decoding-predictions-for-masked-language-modeling-task-using-custom-bpe


Different output encode and encode_plus

Hi everyone,

I'm struggling with the following scenario:

I have the following input:

```
sentence = "Sheldon : If a photon is directed through a plane with two slits in it and either is observed . Sheldon : . it will not go through both . If unobserved , it will . UNKNAME : If it 's observed after it left the plane , before it hits its target . Sheldon : . it will not have gone through both slits . Agreed . Leonard : What 's your point ? Sheldon : There 's no point , I just think it 's a good idea for a T shirt . UNKNAME : Excuse me . Hang on . Leonard : One across is Aegean , eight down is Nabokov . Leonard : Twenty six across is MCM . Leonard : Fourteen down is . Leonard : Move your finger . UNKNAME : . phylum , which makes 14 across Port Au Prince . Leonard : See , Papa Doc 's capital idea , that 's Port Au Prince . Leonard : Haiti . UNKNAME : Can I help you ? Yes . UNKNAME : Um , is this the high IQ sperm bank ?"
```

I am using bert-base-uncased and I get different length of tokens depending if I use `tokenizer.encode` or `tokenizer.encode_plus`.

Below is an example:

```
test = tokenizer.encode_plus(sentence, add_special_tokens=True, max_length=512)["input_ids"]
print(test)
214
```
```
test2 = test2 = tokenizer.encode_plus(sentence, add_special_tokens=True, max_length=512)["input_ids"]
print(test2)
189
```

In above scenario I expect the amount of tokens to be the same length. I looked at the documentation but I cannot find an explanation for the difference. It is problematic for me because I need to use tokenizer.batch_encode_plus but for my model I expect and need the length of 189 instead of 214.

Can someone please explain why the output is different and how to make encode.plus output the same as encode?

Thanks in advance ;)


Cutom tokenizer not loaded in AutoTokenizer

I am training a language model from scratch. I trained **ByteLevelBPETokenizer** and then when I am trying to load this tokenizer using **AutoTokenizer**, it is giving me the following error.

OSError: Model name './MyRobertaConfig/' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). We assumed './MyRobertaConfig/' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

I have not added tokenizer_config.json file in config directory, I don't think it should be an issue. Now do I need to migrate or transform my custom tokenizer to make it compatible with transformers tokenizers or what.


PretrainedTokenizer cleanup: Typehints, decode_batch

This is very very minor cleanup

- adds `decode_batch` which calls `.decode` on every entry in a list.
- A few cosmetic changes to tokenization_utils.py (type hints using defaultdict)
- adds type hints in files I touched.


Update camembert-base-README.md

model card

Related to pretrained model cards


Transfo-XL cannot generate long texts. Using run_generation.py to generate texts

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Trying to generate long text with Transfo-XL Text-generator but I continuously get a
        warning which may be the reason I am unable to generate the long text.
        Here is the Warning: WARNING - transformers.modeling_utils -   Setting `pad_token_id` to 0 (first `eos_token_id`) to generate sequence -->

This is how I run the code:
cd transformers/
python examples/run_generation.py  --model_type transfo-xl --model_name_or_path transfo-xl-wt103 \
                                  --prompt "China wants to take a victory lap over its handling of the coronavirus outbreak" --repetition 2.2 \
                                  --length 500   \
                                  --temperature 0.8 --k 8


i want help to create saved model(.pth) from Pytorch Dump(pytorch_model.bin) if possible!

i have, PROJECT(folder)
                     ├── pytorch_model.bin
                     ├── bert_config.json
                     └── vocab.txt

i tried saving it with
`torch.save( pytorch_model.bin , PATH)`

but came with error
`-bash: syntax error near unexpected token `pytorch_model.bin,'`

what am i doing wrong ?
and please help me convert pretrained model to saved model( .pth)!!


Remove tqdm logging when using pipelines.

Attempt to fix #3744

Introduce `tqdm_enabled` parameter on `squad_convert_examples_to_features()` default to `True` and set to `False` in QA pipelines.


The issue I met when do the NER task for Universal language by using XLM-R

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

**A link to original question on Stack Overflow**:
Firstly, go through the roughly logic when we do slot tagging task for English query, take query “what’s the weather in Beijing” and detect location slot tagging as example:
1. We are using space to split the query and label it as “O O O O B-location” when generate the training data
2. By using sentence piece to tokenize the query to ['_what', "'", 's', '_the', '_weather', '_in', '_bei', 'jing'], and corresponding tagging mask [1, 0, 0, 1, 1, 1, 1, 0] (take first token in the word as 1, other tokens in the word as 0)
3. By using NER task to do the predict, and then extract the token’s prediction result as the word prediction result if the token’s mask is 1

When we do the NER task as above logic for CJK language, take “今天北京天气怎么样？” as example, after sentence piece tokenize, the tokens is ['▁', '北京', '今天', '天气', '怎么样', '?'], since there are no space in the query, so it will take the whole query as one slot and the corresponding tagging mask is [1, 0, 0, 0, 0, 0], so we can’t get the location slot “北京”

The cause is that the split methods for English and CJK are different, In French, “Je t’aime” is means “I love you”, “Je” means “I”, “t’” means “you”, “aime” means “love”, it is using “’” as splitter.

I think we can solve it by using below solutions:
1. Detect what language it is firstly for the query
2. For the language likes English, we don’t change the logic as above, for the language likes CJK, after split by using space, we can use tokens generated by sentence piece as word,
a. For example “北京今天天气怎么样，西雅图呢?”, split it as [“北京今天天气怎么样”, “西雅图呢?”] as space
b. For “北京今天天气怎么样”,  after tokenized is: ['▁', '北京', '今天', '天气', '怎么样', ',’], we mark each token as word we need to predict, the correspond tagging mask is [0, 1, 1, 1, 1, 0]
By using NER task to do the predict, and then extract the token’s prediction result as the word prediction result if the token’s mask is 1, so we can predict ‘北京’ as location slot

But for this, it needs to have specific language knowledge and language classification detector, any better ideas do you have?


AlbertModel output is not HalfTensor when using apex fp16

# 🐛 Bug
Despite the fact that I turned the model to fp16 with apex, the hidden representation output is not half tensor (see code snippet for details) while class heads are half tensors.

## Information

Model I am using AlbertModel.

Language I am using the model on arbitrary:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```py
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import AlbertModel, AlbertConfig
from transformers.modeling_albert import AlbertMLMHead
import apex

pad_token_id = 0
bos_token_id = 2
eos_token_id = 3
vocab_size = 20

config = {
             "attention_probs_dropout_prob": 0.1,
             "hidden_act": "gelu_new",
             "hidden_dropout_prob": 0.1,
             "embedding_size": 64,
             "hidden_size": 256,
             "initializer_range": 0.02,
             "intermediate_size": 1024,
             "max_position_embeddings": 512,
             "num_attention_heads": 2,  # smaller than usual
             "num_hidden_layers": 2,  # smaller than usual
             "num_hidden_groups": 1,
             "net_structure_type": 0,
             "gap_size": 0,
             "num_memory_blocks": 0,
             "inner_group_num": 1,
             "down_scale_factor": 1,
             "type_vocab_size": 2,
             "vocab_size": vocab_size,
             "pad_token_id": pad_token_id,
             "bos_token_id": bos_token_id,
             "eos_token_id": eos_token_id,
         }

albert_config = AlbertConfig(**config)
encoder: AlbertModel = AlbertModel(albert_config)
masked_lm = AlbertMLMHead(albert_config)
optimizer = torch.optim.Adam(encoder.parameters(), lr=0.0001)
encoder = encoder.cuda()
model, optimizer = apex.amp.initialize(encoder, optimizer, opt_level="O1", )

# When giving LongTensor as input, the class heads are half tensors,
# but hidden representations are not half!
long_input = torch.randint(1, 10, (10,5)).cuda()
f1, m1= encoder(long_input)

f1.type()
"""
'torch.cuda.FloatTensor'
"""
m1.type()
"""
'torch.cuda.HalfTensor'
"""

# When giving HalfTensor as input, it crashes
half_input = long_input.half()
f2, m2= encoder(half_input)

"""
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/transformers/modeling_albert.py", line 570, in forward
    input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/transformers/modeling_bert.py", line 173, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/mnt/castor/seas_home/r/rasooli/torch_env/lib64/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError:
"""
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.7.0
- Platform:
- Python version: Python 3.6.10
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: Yes, NVIDIA-SMI 418.67; GeForce RTX 208
- Using distributed or parallel set-up in script?: Optional parallel


Faster mask computation

# 🚀 Feature request

Currently, the masking is done using the full prediction matrix which is both memory and computation inefficient. One example is in [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_albert.py#L675). I think with Pytorch indexing, first and foremost, we don't need to do full mask matrix construction. Second, it can be much faster. Fairseq does a similar thing in [here](https://github.com/pytorch/fairseq/blob/cce6dcb1cca85955a82879ea5064fe8202e8f412/fairseq/models/roberta/model.py#L217)

## Motivation

If input is [n,m], currently the code creates a clone of [n,m] mask where non-masked inputs are -100. Then it does a full output projection: if input in [n, m] and hidden representation is [n, m, h], the final output will be a huge [n, m, v] if v is the vocabulary size. Instead we can think of mask indices of size k<< n* m, and thus the we can extract a [k, h] submatrix from [n, m, h], then can have a smaller output result [k, v].

## Your contribution
This a sample code (I extracted from a bigger code of mine; sorry if some variables are not defined in the snippet):
```py
mask_prob = 0.15
mask = torch.empty(input_ids.size()).uniform_(0, 1) < mask_prob
mask[pads] = False  # We should not mask pads.
masked_ids = input_ids[mask]
replacements = masked_ids.clone()
for i in range(len(replacements)):
    r = random.random()
    if r < 0.8:
        replacements[i] = mask_id
    elif r < 0.9:
        # Replace with another random word.
        random_index = random.randint(0, vocab_size - 1)
        replacements[i] = random_index
    else:
        # keep the word
        pass
input_ids[mask] = replacements

text_hidden, text_cls_head = albert_model(texts, attention_mask=pads)
masked_hidden_state = text_hidden[mask]
output_predictions = F.log_softmax(albertMLMHead(masked_hidden_state), dim=1)
```


Can't install transformers in conda environment

# 🐛 Bug

I tried to install transformers into a conda environment

```
pip install transformers
Collecting transformers
  Using cached transformers-2.8.0-py3-none-any.whl (563 kB)
Collecting tokenizers==0.5.2
  Downloading tokenizers-0.5.2-cp38-cp38-macosx_10_15_x86_64.whl (1.1 MB)
     |████████████████████████████████| 1.1 MB 1.6 MB/s
Collecting tqdm>=4.27
  Downloading tqdm-4.45.0-py2.py3-none-any.whl (60 kB)
     |████████████████████████████████| 60 kB 23.6 MB/s
Collecting filelock
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Collecting requests
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting regex!=2019.12.17
  Using cached regex-2020.4.4.tar.gz (695 kB)
Collecting boto3
  Using cached boto3-1.12.39-py2.py3-none-any.whl (128 kB)
Requirement already satisfied: numpy in ./Anaconda/anaconda3/envs/nlp/lib/python3.8/site-packages (from transformers) (1.18.2)
Collecting sentencepiece
  Using cached sentencepiece-0.1.83.tar.gz (497 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/chen_bowen/Anaconda/anaconda3/envs/nlp/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/lz/65hfhw790_z85w_09kvvm37r0000gn/T/pip-install-lezphia4/sentencepiece/setup.py'"'"'; __file__='"'"'/private/var/folders/lz/65hfhw790_z85w_09kvvm37r0000gn/T/pip-install-lezphia4/sentencepiece/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/lz/65hfhw790_z85w_09kvvm37r0000gn/T/pip-install-lezphia4/sentencepiece/pip-egg-info
         cwd: /private/var/folders/lz/65hfhw790_z85w_09kvvm37r0000gn/T/pip-install-lezphia4/sentencepiece/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/lz/65hfhw790_z85w_09kvvm37r0000gn/T/pip-install-lezphia4/sentencepiece/setup.py", line 29, in <module>
        with codecs.open(os.path.join('..', 'VERSION'), 'r', 'utf-8') as f:
      File "/Users/chen_bowen/Anaconda/anaconda3/envs/nlp/lib/python3.8/codecs.py", line 905, in open
        file = builtins.open(filename, mode, buffering)
    FileNotFoundError: [Errno 2] No such file or directory: '../VERSION'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.```

python version: Python 3.8.2
OS: Mac OSX 10.15.3
Anaconda version: conda 4.8.0


Tanh torch warnings

This pull request the warning generated by using torch.nn.functional.tanh (which is deprecated). This pull request changes it to torch.tanh to remove the warning.


# 🐛 Bug

## Information

I am using ./examples/summarization/bertabs/

`python run_summarization.py \
    --documents_dir $data_dir\
    --summaries_output_dir$output_dir \
    --no_cuda true \
    --batch_size 4 \
    --min_length 50 \
    --max_length 200 \
    --beam_size 5 \
    --alpha 0.95 \
    --block_trigram true
`
returns
`ModuleNotFoundError: No module named '__main__.utils_summarization'; '__main__' is not a package`

My environment
- `transformers` version: 2.8.0
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.3
- PyTorch version (GPU?): 1.1.0 (False)
- Tensorflow version (GPU?): 2.0.0-beta1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

Thank you!


[readability] consolidate examples/summarization/bart  and examples/summarization/t5

Examples

Which is related to examples in general

Help wanted

Extra attention is needed, help appreciated

This involves

- [ ]  consolidating README.md
- [ ]  consolidating evaluate_cnn.py scripts.
- [ ]  evaluate_wmt.py should also work
- [ ]  consolidating unittests
- [ ]  updating bash scripts
- [ ]  checking that consolidated stuff works, adding appropriate test coverage.

the resulting code should probably all be in `summarization/` with only a `bertabs/` subdirectory.


[readability] Consolidate prune_heads logic to PretrainedModel.

PyTorch

Anything PyTorch

Many models have identical implementations of `prune_heads` it would be nice to store that implementation as a method on `PretrainedModel` and reduce the redundancy.


[examples] summarization/bart/finetune.py supports t5

- we were passing attention_mask as an arg, not a kwarg, causing `test_step` to break.
- That case is now covered in the unittest, unittests also cover the t5 model.
- renamed run_bart_sum.py to finetune.py since it is model agnostic.
- The `bart/` and `t5` subdirectories should be consolidated in a future PR.

This took 15 mins because of underlying infrastructure: unittests for examples and tiny models on S3 :)


lowercasing on LM with cased models

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Execute 'run_language_modeling.py' with one of the following cased models.
   (example of cased models: sciBert cased, BioBert, ClinicalBert)
2. Check the tokenizer's do_lower_case option.

## Expected behavior

I'm language modeling with my own data on the top of a pre-trained model. I'm using cased models, but the tokenizer lower case input data since its default lowercase option is True. When I used 'bert-base-cased', the tokenizer didn't lower case, but it happened with other cased models mentioned above.
- tokens with 'bert-base-cased' model
['[CLS]', 'This', 'text', 'is', 'included', 'to', 'make', 'sure', 'Uni', '##code',...
- tokens with 'scibert_scivocab_cased' model
['[CLS]', 'this', 'text', 'is', 'included', 'to', 'make', 'sure', 'unic', '##ode',...

Is it a bug? or am I missing something?

As an alternative, I'm using a workaround code by adding additional command parameter.
```python
parser.add_argument("--do_lower_case", action="store_true", help="Should be added for uncased models.")

tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path, cache_dir=args.cache_dir, do_lower_case=args.do_lower_case)
```

Thanks in advance.


getting random results when running run_glue

Hi
I am running run_glue.py script on RTE dataset with BERT base model, on different gpus and I am getting very random results, like changing a lot by changing the gpu. I am using python 3.6 and transformer version 2.5.0. I tried with two gpu types like
Kepler , GTX1080ti and P40, ...
such randomness really affects the benchmarking and appreciate your help. thanks
I have a deadline and appreciate your prompt response.
thanks.
Best
Rabeeh


Typo fix


#3787 Fixing the pip install issue by installing from git


Tokenizers Notebook  Issue

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

Hello everyone,
While playing around with tokenizers notebook : https://github.com/huggingface/transformers/blob/master/notebooks/01-training-tokenizers.ipynb

`---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in
----> 1 tokenizer = Tokenizer(BPE()) # byte-pair encoding model
      2 # now using normalizers
      3 tokenizer.normalizer = Sequence([
      4     NFKC(),
      5     Lowercase()

TypeError: cannot create 'BPE' instances`

Could not find a resolution for this.  thanks


What are the GPU RAM requirements of popular models?

# ❓ Questions & Help

What are the GPU RAM requirement of `gpt2`, `gpt2-medium`, `distilgpt2`, `bert-base-uncased` and/or `distilroberta-base`

* for training?
* for inference?

Additionally, how do you calculate or find this information for other models?

original StackOverflow question: https://stackoverflow.com/questions/61226569/what-are-the-gpu-ram-requirements-of-popular-huggingface-transformers-models
related: #1750


[Examples, T5] Change newstest2013 to newstest2014 and clean up

This PR just adds a small change to #3802 to make the code quality happy.


Aborted (core dumped) or Kernel dies

Whenever I am trying to import tranfomers my kernel dies off in jupyter notebook .
tranformer version - 2.8.0
python version -3.7.7


How to speed up getting answers?

Hi ,

I'm facing the issues using BertForQuestionAnswering ,can you please help me fixing these:

1. I'm trying to use BertForQuestionAnswering pretrained model to get answers from news,
answer_question is the function where in you pass the context and the question and get the relevant answers,but if I have 100 contexts its taking around 100 seconds to get the answer,may I please know by any chance I can get the answers in much lesser time.

2. Some times I get the answer where in start index and end index are pointing to [SEP],that means the whole context,can I avoid this.

```
BERT_SQUAD = 'bert-large-uncased-whole-word-masking-finetuned-squad'

model = BertForQuestionAnswering.from_pretrained(BERT_SQUAD)
tokenizer = BertTokenizer.from_pretrained(BERT_SQUAD)

def answer_question(question, context):
    """
    Answer questions
    """
    try:
        print("Type:",type(context))
        print(context)

        encoded_dict = tokenizer.encode_plus(
                            question, context, # Sentence to encode.
                            add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                            max_length = 256,  # Pad & truncate all sentences.
                            pad_to_max_length = True,
                            return_attention_mask = True,   # Construct attn. masks.
                            return_tensors = 'pt'     # Return pytorch tensors.
                       )
        print(encoded_dict)
        input_ids = encoded_dict['input_ids'].to(torch_device)
        token_type_ids = encoded_dict['token_type_ids'].to(torch_device) # segments

        start_scores, end_scores = model(input_ids, token_type_ids=token_type_ids)
        print('Start Scores:',start_scores)
        print('End Scores:',end_scores)
        all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0])
        print(all_tokens)
        answer = tokenizer.convert_tokens_to_string(all_tokens[torch.argmax(start_scores) : torch.argmax(end_scores)+1])

        answer = answer.replace('[CLS]', '')
        #answer= answer.replace('[SEP]','')
        print("Start index:",all_tokens[torch.argmax(start_scores)])
        print("End index:",all_tokens[torch.argmax(end_scores)])
        print(answer)
    except ValueError:
        print("Error in fetching answer")
        answer=''

    return answer
```

Thanks in advance!!


A bug in the padding of input examples in the NER fine-tuning example

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. TODO

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

https://github.com/huggingface/transformers/blob/c59b1e682d6ebaf7295c63418d4570228904e690/examples/ner/utils_ner.py#L123
This line is supposed to return 3 for Roberta models but it's just returning 2 causing the length of the input_ids to be more than the max_seq_len.
This might be the reason for that: https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_roberta.py#L288
TODO: Share the notebook.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.2.0-rc2 (True)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


T5 prediction using fine-tuned model

After fine-tuning the T5 model on my own dataset, when I use the fine-tuned model to predict for test set using the following command:

python '/content/transformers-master/examples/summarization/bart/run_bart_sum.py' --data_dir='/content/drive/My Drive/two_keywords/' --model_type=t5 --output_dir=/content/t5 --do_predict --model_name_or_path=t5-small
Error generated:
<img width="1010" alt="Screenshot 2020-04-13 at 6 18 47 PM" src="https://user-images.githubusercontent.com/30004110/79394604-31692980-7f78-11ea-8c87-3c04e542e962.png">

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Question Answering support for Albert and Roberta in TF

This PR simply adds `TFRobertaForQuestionAnswering` and `TFAlbertForQuestionAnswering` classes (I needed them to do some model conversions!)


Pre-trained BART performance on XSum lower than expected

Greetings,

I am trying to reproduce BART's results on xsum using 'bart-large-xsum' and modified `examples/summarization/bart/evaluate_cnn.py` (max_length=60, min_length=10, beam=6, lenpen=1)  but got lower ROUGE scores than reported.

I first obtained comparable results on CNNDM using 'bart-large-cnndm'  and the dataset on s3:

CNNDM | R-1 | R-2 | R-L
-- | -- | -- | --
BART (Lewis et al., 2019) | 44.16 | 21.28 | 40.9
BART (ours) | 44.32 | 21.12 | 41.13

I then obtained the raw xsum dataset from the original authors and saved them to test.source and test.target (cased) as for CNNDM. Then I ran evaluate_cnn.py with the new parameters above. Is there anything that I am missing? Thank you!

XSum | R-1 | R-2 | R-L
-- | -- | -- | --
BART (Lewis et al., 2019) | 45.14 | 22.27 | 37.25
BART (ours) | 44.7 | 21.04 | 35.64


run_glue.py example doesn't work for distilbert models

# 🐛 Bug

## Information
Hi all,

I am succesfully able to run the run_glue.py example with BERT, XLNet and other architectures. However, when I try distilbert I got the following error:
```

Traceback (most recent call last):
  File "run_glue.py", line 562, in <module>
    main()
  File "run_glue.py", line 510, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "run_glue.py", line 373, in load_and_cache_examples
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
TypeError: an integer is required (got type NoneType)
```

Model I am using (Bert, XLNet ...):
distilbert (distilbert-base-cased)
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: GLUE/SST-2
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
export GLUE_DIR=./glue/glue_data
export TASK_NAME=SST-2
CUDA_VISIBLE_DEVICES=2,3 python run_glue.py \
    --model_type DistilBERT \
    --model_name_or_path distilbert-base-cased \
    --task_name $TASK_NAME \
    --do_train \
    --do_eval \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_train_batch_size 32 \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --output_dir ./output/$TASK_NAME/
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
```
     - `transformers` version: 2.8.0
- Platform: Linux-5.5.15-200.fc31.x86_64-x86_64-with-fedora-31-Thirty_One
- Python version: 3.6.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO
```


Roberta Tokenizer crashes when tokenizing  empty string in 2.8.0

Core: Tokenization

Internals of the library; Tokenization.

Should Fix

This has been identified as a bug and should be fixed.

```
import transformers

tokenizer = transformers.AutoTokenizer.from_pretrained("roberta-large")
tokenzer.tokenize("")
```

  File "/scratch/wh629/nlu/env/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1054, in encode_plus
    first_ids = get_input_ids(text)
  File "/scratch/wh629/nlu/env/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1028, in get_input_ids
    tokens = self.tokenize(text, add_special_tokens=add_special_tokens, **kwargs)
  File "/scratch/wh629/nlu/env/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 759, in tokenize
    text = self.prepare_for_tokenization(text, **kwargs)
  File "/scratch/wh629/nlu/env/lib/python3.7/site-packages/transformers/tokenization_roberta.py", line 239, in prepare_for_tokenization
    if add_prefix_space and not text[0].isspace():
IndexError: string index out of range


typo: fine-grained token-leven

Changing from "fine-grained token-leven" to "fine-grained token-level"


isort ignores examples directory

Temporary solution while we wait for an isort release.
I have my local alias hacked in this way, but I figure new contributors might get confused by the circleci/local isort discrepancy.


[cleanup] factor out get_head_mask, invert_attn_mask, get_extended_attention_mask

This is three changes applied all over:

1) `get_head_mask` from @LysandreJik is used instead of redundant snippet
2) `get_extended_attention_mask` is used instead of redundant snippet (that also makes causal mask)
3) `invert_attention_mask` is used instead of redundant snippet that doesn't make causal mask.

These changes make the forward passes more readable and allow us to update common logic in one place moving forward! I was reading code last night to try to get a sense of what all the models/tokenizers do and was frustrated with the amount of time spent scrolling through this stuff. Especially for new people, it makes getting to the meet of the `forward` pass much harder to have 100 lines repeated lines of input manipulation at the beginning.

Very open to suggestions.
Open to doing this for TF in a separate PR.
Also if `prune_heads` or other opportunities catch your eye, let me know.


Using fill-mask pipeline to get the “score” for a result it didn't suggest

Hi lovely huggingface people,

I'm trying to use your fill-mask pipeline in order to get the score for a result it didn't suggest.

For example, if my sentence is `"I ate bacon and <mask> for breakfast"` I can use `pipeline('fill-mask')` to get back predictions and their scores e.g. it might give me back `["eggs", 0.1]`. But what I would like to do is **provide my own guess and then get back the score it assigns to my own guess.** e.g. i might want to know what score it gives to the word "pancakes" in the situation.

Is this possible? If not can I register it as a feature request?

Stack overflow [question](https://stackoverflow.com/questions/61168513/using-huggingface-fill-mask-pipeline-to-get-the-score-for-a-result-it-didnt-s)


Calculated offsets are wrong in squad.py

- when tokenizer.padding_side == "left" and if tokenizer.pad_token_id in span["input_ids"]:
doc_offset should add last_padding_id_position+1 instead of 0.

        start_position = 0
        end_position = 0
        if is_training and not span_is_impossible:
            # For training, if our document chunk does not contain an annotation
            # we throw it out, since there is nothing to predict.
            doc_start = span["start"]
            doc_end = span["start"] + span["length"] - 1
            out_of_span = False

            if not (tok_start_position >= doc_start and tok_end_position <= doc_end):
                out_of_span = True

            if out_of_span:
                start_position = cls_index
                end_position = cls_index
                span_is_impossible = True
            else:
                if tokenizer.padding_side == "left":
                    doc_offset = 0
                else:
                    doc_offset = len(truncated_query) + sequence_added_tokens

                start_position = tok_start_position - doc_start + doc_offset
                end_position = tok_end_position - doc_start + doc_offset


Fix bug in max_seq_length for preprocessing in ner example

**Summary**

If you try to use a value for `max_seq_length` that is less than 128 in the NER example, the maximum sequence length is exceeded when the predictions are made. There is a warning logged for this "Maximum sequence length exceeded: No prediction for.." and predictions cannot be made for these tokens.

**Changes**

Two changes are made:

- In `preprocess.py`, `subword_len_counter` is set to `current_subwords_len` when a blank line is inserted to split up sequences that exceed the maximum sequence length.
- `tokenizer.num_added_tokens()` is subtracted from `max_len` to account for the additional tokens inserted by the BERT tokenizer.


Fix examples/translation/t5 to use newstest2014 rather than newstest2013

Resolves #3759, in addition to minor nits: fixed a bug with argparse arguments + more pythonic file handling + formatted with black and isort.

Please note that I have not yet run the evaluation script against the full newstest2014 test set, as it is rather compute intensive, so the disclaimer at the top of the README.md about the score gap between the pre-trained and fine-tuned models is only ostensibly accurate to the score gap on newstest2013, not newstest2014.


Fix bug in GLUE example for models that do not require token_type_ids

If you try to run the `run_glue.py` example with e.g. roberta from a fresh install of the library, it errors out with the following error:

```
Traceback (most recent call last):
  File "examples/run_glue.py", line 564, in <module>
    main()
  File "examples/run_glue.py", line 512, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "examples/run_glue.py", line 373, in load_and_cache_examples
    all_token_type_ids = torch.tensor([f.token_type_ids for f in features], dtype=torch.long)
TypeError: an integer is required (got type NoneType)
```

To reproduce, run e.g.

`python examples/run_glue.py --model_name_or_path roberta-base --task_name SST-2 --per_gpu_eval_batch_size=8 --per_gpu_train_batch_size=8 --data_dir ./glue_data/SST-2/ --output_dir ./blah --model_type roberta --do_train --do_eval --max_seq_length 128 --learning_rate 2e-5 --num_train_epochs 3.0`

The reason is obviously that roberta does not have segment ids so `token_type_ids` is set to null in the data loader, causing `torch.tensor` to freak out. There's probably a more elegant long-term solution for this, but it's easy to fix by just setting it to 0 instead of null for those models.


Trainer

This is a bottom-up refactor of the example scripts (currently `run_glue.py`, `run_language_modeling.py`, `run_ner.py` and `run_multiple_choice.py`) into a Trainer class and associated utilities, as described in [trainer-proposal](https://github.com/julien-c/trainer-proposal).

# Regression test and benchmark on `run_glue.py`

📊 All training metrics are in [**this TensorBoard**](https://tensorboard.dev/experiment/7LuNpqw3Q8y147WxfyOqdA/#scalars&_smoothingWeight=0&runSelectionState=eyJkYXRhcGFyYWxsZWxfMl9ncHUiOnRydWUsImRkcF8yX2dwdSI6dHJ1ZX0%3D):

The Trainer class supports PyTorch's backends for parallel/distributed training so we performed the following experiment:

- Experiment: MNLI
  - Train set: 100_000 first samples
  - Dev set: Full (~9_800 samples)
- Backends: DataParallel, DistributedDataParallel, single-GPU, CPU

You can compare speed of convergence by clicking on "Relative" in the TensorBoard and comparing loss and accuracy curves:

## Results

### DataParallel

```
--model_name_or_path distilbert-base-cased
--task_name mnli
--data_dir ./data/glue_data/MNLI
--output_dir ./models/dataparallel_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--num_train_epochs 1
--per_gpu_train_batch_size 32
--per_gpu_eval_batch_size 128
--logging_steps 100
--logging_dir ./runs/dataparallel_2_gpu
--logging_first_step
--save_steps 1000
--evaluate_during_training
```

1 Epoch = 21 mins

```
04/14/2020 23:16:34 - INFO - __main__ -   ***** Eval results mnli *****
04/14/2020 23:16:34 - INFO - __main__ -     acc = 0.7406011207335711
04/14/2020 23:16:34 - INFO - __main__ -     loss = 0.6281169515389663

04/14/2020 23:17:02 - INFO - __main__ -   ***** Eval results mnli-mm *****
04/14/2020 23:17:02 - INFO - __main__ -     acc = 0.7507119609438568
04/14/2020 23:17:02 - INFO - __main__ -     loss = 0.6062953961201203
```

### DistributedDataParallel

```
python -m torch.distributed.launch --nproc_per_node 2 ./examples/run_glue.py
--model_name_or_path distilbert-base-cased
--task_name mnli
--data_dir ./data/glue_data/MNLI
--output_dir ./models/ddp_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--num_train_epochs 1
--per_gpu_train_batch_size 32
--per_gpu_eval_batch_size 128
--logging_steps 100
--logging_dir ./runs/ddp_2_gpu
--logging_first_step
--save_steps 1000
--evaluate_during_training
```

Speed: about same speed as DataParallel on this workload and machine.

Pre-existing issue (to fix in future PR): when using DDP, the eval is not GPU-parallelized.

### single-GPU

`CUDA_VISIBLE_DEVICES=0 python ...`

04/15/2020 00:52:24 - INFO - __main__ -   ***** Eval results mnli *****
04/15/2020 00:52:24 - INFO - __main__ -     acc = 0.7383596535914416
04/15/2020 00:52:24 - INFO - __main__ -     loss = 0.631212914144838
04/15/2020 00:53:16 - INFO - __main__ -   ***** Eval results mnli-mm *****
04/15/2020 00:53:16 - INFO - __main__ -     acc = 0.7534580960130187
04/15/2020 00:53:16 - INFO - __main__ -     loss = 0.6002480050960144

Speed: Twice slower

### CPU

`--no_cuda`

Speed: too slow to benchmark

# Regression test on `run_ner.py`

The arguments below:

```
--model_name_or_path bert-base-multilingual-cased
--data_dir ./data/germeval
--labels ./data/germeval/labels.txt
--max_seq_length 128
--output_dir ./models/ner_dp_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--do_predict
--num_train_epochs 3
--per_gpu_train_batch_size 32
--logging_dir ./runs/ner_dp_2_gpu
--logging_steps 100
--evaluate_during_training
--save_steps 750
--seed 1
```

yield the following results, consistent with the ones in the README:

```
04/17/2020 16:12:30 - INFO - __main__ -     f1 = 0.8634538152610443
04/17/2020 16:12:30 - INFO - __main__ -     loss = 0.07145964812514359
04/17/2020 16:12:30 - INFO - __main__ -     precision = 0.8434379457917262
04/17/2020 16:12:30 - INFO - __main__ -     recall = 0.8844427823485415
```

Shape of F1 and eval loss:

# Regression test on `run_language_modeling.py`

Reproducing the training described in the [how to train blogpost](https://huggingface.co/blog/how-to-train):

```
--train_data_file ./data/oscar.eo.txt
--eval_data_file ./data/oscar.eo.eval.txt
--evaluate_during_training
--output_dir ./models/EsperBERTo-small-v1
--overwrite_output_dir
--mlm
--config_name ./models/EsperBERTo-small
--tokenizer_name ./models/EsperBERTo-small
--do_train
--do_eval
--line_by_line
--logging_first_step
--logging_steps 10
--logging_dir ./runs/EsperBERTo
--num_train_epochs 1
--save_total_limit 2
--save_steps 2000
--per_gpu_train_batch_size 16
--seed 42
```

# Regression test on `run_multiple_choice.py`

```
--model_name_or_path distilroberta-base
--task swag
--data_dir ./data/swag
--output_dir ./models/swag_dp_2_gpu
--overwrite_output_dir
--do_train
--do_eval
--per_gpu_train_batch_size 32
--per_gpu_eval_batch_size 512
--logging_dir ./runs/swag_dp_2_gpu
--logging_steps 100
--logging_first_step
--evaluate_during_training
```


Clarification about GPT2LMHeadModel lm_head weights

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->
Each time the GPT2LMHeadModel is loaded from pretrained weights, the following is logged:
```
Weights of GPT2LMHeadModel not initialized from pretrained model: ['lm_head.weight']
```

Just to clarify, is this OK because we tie the output (`lm_head`) weights to the input weights?


Error when using run_generation.py to generate texts with long prompts, specifically for models -XLM and Openai-GPT

# 🐛 Bug

## Information

Model I am using (XLM, OPENAI-GPT):

Language I am using the model on (English):

The problem arises when using:
* [ ] The models to generate texts with long prompts

The tasks I am working on is:
* [ ] Text generation

## To reproduce

Steps to reproduce the behavior:

1. cd transformers/
python examples/run_generation.py  --model_type xlm --model_name_or_path xlm-mlm-en-2048 \
                                  --prompt "China wants to take a victory lap over its handling of the coronavirus outbreak" \
                                  --repetition 2.2 --k 5 \
                                  --length 500

2.
3.

<!-- Error:   RuntimeError: The size of tensor a (513) must match the size of tensor b (512) at non-singleton dimension 3. \
This leads to the next error - RuntimeError: CUDA error: device-side assert triggered.-->

## Expected behavior

<!-- The expected behavior is a generated piece of texts of about 500 words -->

## Environment info
<!-- cd transformers/
python examples/run_generation.py  --model_type xlm --model_name_or_path xlm-mlm-en-2048 \
                                  --prompt "China wants to take a victory lap over its handling of the coronavirus outbreak" \
                                  --repetition 2.2 --k 5 \
                                  --length 500  -->

I think the problem is that there is a bug somewhere in the input_embed or input_id or vocabulary because seems to out of index given certain prompts. This may suggest that the vocab list is limited or maybe not.

- `transformers` version:
- Platform: Google Colab
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): no
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


[Config, Serialization] more readable config serialization

Given the discussion in PR: #3433, we want to make the serialized model conf more readable.

### Problem:

E.g. `bert-base-cased` has the following config on S3:
```
{
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 28996
}
```

But when saved all default params are saved as well (which is unnecessary):

```
{
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "type_vocab_size": 2,
  "vocab_size": 28996
}
(which is readable imo) and once it's saved it now looks like this:

{
  "_num_labels": 2,
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "early_stopping": false,
  "eos_token_id": null,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "is_encoder_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-12,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 512,
  "min_length": 0,
  "model_type": "bert",
  "no_repeat_ngram_size": 0,
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 2,
  "use_bfloat16": false,
  "vocab_size": 28996
}
```

### Solution:

We should only save the difference of the actual config to either **v1**) to the model class' config or **v2**) to the PretrainedConfig() (which contains most of the unnecessary default params).

This PR implements either **v1**) or **v2**) - up for discussion!

**v1**) for `bert-base-cased` would look like this:

```
{
  "architectures": [
    "BertForMaskedLM"
  ],
  "vocab_size": 28996
}
```

**v2**) for `bert-base-cased` would look like this:

```
{
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 28996
}
```


Calculated offsets are wrong

This is on the latest `master` (from 2020-04-13):
```Python
import transformers
text = 'A, <mask> AllenNLP sentence.'
t = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True, add_special_tokens=True)
x2 = t.encode_plus(text, return_offsets_mapping=True)
print(repr(t.convert_ids_to_tokens(x2['input_ids']))
print(repr([text[start:end] for start, end in x2['offset_mapping']]))
```

This prints (with some manual alignment):
```
['<s>', 'ĠA', ',', '<mask>',   'ĠAllen', 'N', 'LP', 'Ġsentence', '.', '</s>']
['',    'A',  ',', ', <mask>', ' Alle',  'n', 'NL', 'P sentenc', 'e', '']
```


[Pipelines] Clean pipelines test and remove unnecessary code

This PR cleans up pipelines a bit:

1) Fixes non-working pipeline creation test
2) Remove unnecessary code (due to PR #3116) in pipelines as discussed with @thomwolf in PR #3413

Note: Tested on QA pipelines slow tests.


Getting large alloc error while evaluating bert-base on NER task

# 🐛 Bug

## Information

Model I am using (Bert-base-multilingual-cased,):

Language I am using the model on (English):

The problem arises when using:
Evaluation

The tasks I am working on is:
My own custom dataset having the same as GermEval task format.

Running on **Colab**.

I believe this is due to memory error. But why should there be memory **error while testing when things were fine during training **?

Max_seq_length during train was 128 and batch_size 8

04/14/2020 19:06:13 - INFO - transformers.modeling_utils -   loading weights file germeval-model/checkpoint-21750/pytorch_model.bin
04/14/2020 19:07:03 - INFO - __main__ -   Loading features from cached file ./cached_dev_bert-base-multilingual-cased_128
04/14/2020 19:07:05 - INFO - __main__ -   ***** Running evaluation  *****
04/14/2020 19:07:05 - INFO - __main__ -     Num examples = 22026
04/14/2020 19:07:05 - INFO - __main__ -     Batch size = 2
Evaluating:   0% 3/11013 [00:01<1:07:38,  2.71it/s]tcmalloc: large alloc 1110605824 bytes == 0x3e706000 @  0x7f207d0051e7 0x7f207a3995e1 0x7f207a3fde88 0x7f207a3fdfa3 0x7f207a49c098 0x7f207a49c8f4 0x7f207a49ca42 0x5678b3 0x5a067e 0x7f207a3e970d 0x50a8af 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50d390 0x508245 0x58958c 0x5a067e 0x7f207a3e970d 0x50a8af 0x50c5b9 0x508245 0x50a080 0x50aa7d 0x50d390 0x508245 0x50a080 0x50aa7d 0x50d390 0x508245


[Bert] remove hard-coded pad token id

tiny change to remove hard coded `pad_token_id` in Bert.


Using run_glue.py on external datasets for fine-tuning a RoBERTa classification model --> Is this possible?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I recently uploaded the model weights for RoBERTa trained on a chemical benchmark dataset called ZINC15K for masked-language modelling of individual atoms in each molecule. The model performed pretty decently, so I thought it would be interesting to apply it to a downstream task of toxicity prediction on the Tox21 dataset (balanced dataset I created here: https://github.com/seyonechithrananda/bert-loves-chemistry/blob/master/tox21_balanced_revised.csv)

^The link above is in the repo with all the HuggingFace notebooks relating to this task I have created for more details.

As you can see in the CSV above, the 'SR-p53' value represents the labels for the dataset, whereas the 'SMILES' column represents the text representation for each molecule. Is there a way that `run_glue.py` can be repurposed alongside `RobertaforSequenceClassification` to train a pre-trained model for this classification task? And if so, could anyone give me a couple pointers on where to start? I'm relatively new to HuggingFace (more familiar with CV + graph NN's for chemistry) but have been enjoying using it so far!

Link to model weights (in Huggingface hub): https://huggingface.co/seyonec/ChemBERTa-zinc-base-v1

Thanks for the help!

*Given that this was a more library-centric question and not a bug I felt it would be better to post here then on SO.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


XLM tokenizer should encode with bos token

XLM tokenizer should behave according to the documentation

closes https://github.com/huggingface/transformers/issues/3788


Fix token_type_id in BERT question-answering example

`token_type_id` wasn't being set correctly in the code examples for BERT Question answering. It's being turned into the sequence embedding, hence needs to highlight whether each token belongs to sequence 0 or 1.

For this small example the model returns the correct answer even though the parameter was incorrectly set, but for bigger paragraphs that is not the case.

I changed the code to use encode_plus which returns the correct `token_type_id`.


Is there a classical transformer model in the project?

Hi,
I am studying in some domain that need the original transformer which is from 《attention is what you all need》. Is there an implementation?


Inconsistencies and possible bugs in different tokenizers

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Over in [Flair](https://github.com/flairNLP/flair/pull/1494) we are integrating your awesome library in our embeddings interfaces. We are using the `AutoTokenizer` class to create one interface for all embeddings. We use the tokenizers to encode strings with special tokens.

However, we note some inconsistencies: (1) Most, but not all encodings do not include the BOS token and (2) some encodings behave differently depending on how the tokenizer is called. In both cases, this is detrimental to downstream task performance.

This can be reproduced with the following script:

```python
from transformers import AutoTokenizer

# example string to tokenize
text = "CRICKET1 MATCH"

# different models
for tokenizer_name in [
    'bert-base-cased',
    'openai-gpt',
    'transfo-xl-wt103',
    'gpt2',
    'xlnet-base-cased',
    'xlm-mlm-ende-1024',
    'roberta-base',
    'distilbert-base-uncased',
    'ctrl',
    'camembert-base',
    'albert-base-v2',
    'xlm-roberta-base',
    'distilgpt2',
    'bart-large',
    'distilroberta-base',
]:

    # for each tokenizer model, print name and result of checks
    print('------------')
    print(tokenizer_name)
    print('------------')

    # get tokenizer
    tokenizer = AutoTokenizer.from_pretrained(tokenizer_name)

    # method 1: tokenizer.encode() with add_special_tokens=True
    ids = tokenizer.encode(text, add_special_tokens=True)
    subtokens_encode_special = tokenizer.convert_ids_to_tokens(ids)

    # method 2: tokenizer.encode() with add_special_tokens=False and subsequent build_inputs_with_special_tokens()
    ids = tokenizer.encode(text, add_special_tokens=False)
    ids_extended = tokenizer.build_inputs_with_special_tokens(ids)
    subtokens_encode_and_build = tokenizer.convert_ids_to_tokens(ids_extended)

    # check if both methods yield the same result
    if subtokens_encode_special != subtokens_encode_and_build:
        print("DIFFERENCE IN ENCODING!")
        print(f'Method 1 - Encode (+ special):\t{str(subtokens_encode_special)}')
        print(f'Method 2 - Encode and build:  \t{str(subtokens_encode_and_build)}')

    # check if the BOS token is included
    bos_token = tokenizer.bos_token
    if bos_token and bos_token not in subtokens_encode_and_build:
        print("DOES NOT CONTAIN BOS TOKEN!")
        print(f"BOS token '{bos_token}' not in {str(subtokens_encode_and_build)}")
```

This outputs the following inconsistencies, at least some of which likely are bugs.

There are two encodings that do not contain the BOS token:

```console
------------
xlm-mlm-ende-1024
------------
DOES NOT CONTAIN BOS TOKEN!
BOS token '<s>' not in ['</s>', 'crick', 'et', '1</w>', 'match</w>', '</s>']
```

So, the XLM encoding of the string "CRICKET1 MATCH" strangely starts with a **`</s>`** (EOS) even though it should probably start with a **`<s>`**.

```console
------------
xlnet-base-cased
------------
DOES NOT CONTAIN BOS TOKEN!
BOS token '<s>' not in ['▁CR', 'ICK', 'ET', '1', '▁M', 'ATCH', '<sep>', '<cls>']
```

XLNet encoding does not contain BOS and EOS at all. This is consistent with the documentation but is detrimental to performance. In our experiments, it works a lot better if we include `<s>` and `</s>` in the sequence.

There are also two tokenizers for which the two methods (encode with special tokens and encode and build) give slightly different results, namely  RoBERTa and BART:

```console
------------
roberta-base
------------
DIFFERENCE IN ENCODING!
Method 1 - Encode (+ special):	['<s>', 'ĠCR', 'ICK', 'ET', '1', 'ĠM', 'ATCH', '</s>']
Method 2 - Encode and build:   	['<s>', 'CR', 'ICK', 'ET', '1', 'ĠM', 'ATCH', '</s>']
```

This was already noted by @stefan-it in #1196 and strangely, even though the tokenization output by method 1 seems to make more sense, method 2 gives better results.

## Expected behavior

Consistent output :)

## Environment info
- `transformers` version: 2.8
- Platform: Ubuntu
- Python version: 3.7
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: o


In just fouth blocks of the code of the colab notebook "01-training notebook", it just failed.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
https://colab.research.google.com/drive/1gamxcO5AHioIHFVhTr1x71mn8SYPNkPz

`---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-559dbb5a1852> in <module>()
      7
      8 # First we create an empty Byte-Pair Encoding model (i.e. not trained model)
----> 9 tokenizer = Tokenizer(BPE())
     10
     11 # Then we enable lower-casing and unicode-normalization

TypeError: cannot create 'BPE' instances`

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Why force tokens in Bart decoding

https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L955

What's the meaning of this line? Why decode \<S\>  when cur_len = 1? Why not when cur_len = 0?


How to fine tune EncoderDecoder model for training a new corpus of data ?

is there any documentation available for the same?


Convert pytorch-pretrained-bert to new version (transformers)

So I have this code for albert:

```py
from transformers import AlbertTokenizer, AlbertForMaskedLM

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForMaskedLM.from_pretrained('albert-base-v2')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, masked_lm_labels=input_ids)
loss, prediction_scores = outputs[:2]
```

How do i convert it to old format:

```py
from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForMaskedLM

 tokenizer = BertTokenizer.from_pretrained('bert-large-cased')
 model = BertForMaskedLM.from_pretrained('bert-large-cased')
 tokenized_text_tmp = tokenizer.tokenize(text)
 indexed_tokens_tmp = tokenizer.convert_tokens_to_ids(text_tokens_tmp)
 predictions = model(tokens_tensors, segments_tensors, attention_mask_tensors)
```

How do i get functions tokenizer.convert_tokens_to_ids and tokenizer.tokenize from old version into new one.
I need to first tokenize text and only after convert it to ids. Because in old code I do this way, changing in new format would take a very long time, because I did a lot of speed optimization and adding of padding.


# 🌟 New model addition

## Model description
This is an incredible project from the awesome https://github.com/allenai team that solves a big problem in transformers.

From https://twitter.com/i_beltagy/status/1249750021811011591

Excited to share our work on Longformer, a scalable transformer model for long-document NLP tasks without chunking/truncation to fit the 512 limit.
Work with @mattthemathman
, @armancohan

Code and pretrained model: http://github.com/allenai/longformer

We replace the standard self-attention with one that scales linearly with sequence length and that can flexibly adapt to downstream tasks. We continue pretraning from the RoBERTa checkpoint and evaluate on QA, coref, classification. Pretrained model supports seqlen 4,096

The small model archives sota results on enwik8 and text8 and large model gets close with half the parameters. Longformer's self-attention uses an efficient CUDA kernel that minimizes memory usage (char-lm large model, 23k tokens at training and 32k tokens at evaluation)

<!-- Important information -->

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/allenai/longformer

* [X] the model weights are available: (give details)
Yes, at https://github.com/allenai/longformer

* [ X] who are the authors: (mention them, if possible by @gh-username)
@ibeltagy @schmmd


Importing horovod.tensorflow crashes AlbertTokenizer but not BertTokenizer

# 🐛 Bug

## Information

Albert tokenizers began crashing after I reorded my import statements with `isort`. Tracked down the bug to very strange behavior: importing `horovod.tensorflow` before `AlbertTokenizer` causes a crash, while importing `AlbertTokenizer` first does not. This behavior does not occur with `BertTokenizer`, only with `AlbertTokenizer`.

## To reproduce

Steps to reproduce the behavior:

```bash
docker run -it nvcr.io/nvidia/tensorflow:20.03-tf2-py3 /bin/bash # TF 2.1, horovod 0.19.0
pip install transformers==2.8.0
```

```python
import horovod.tensorflow as hvd
from transformers import AlbertTokenizer, BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
print("BERT success!") # this succeeds
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
print("ALBERT success!") # this causes a CoreDump
```
outputs
```error
BERT success!
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/stubs/common.cc:86] This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the installed version (3.8.0).  Contact the program author
for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  This program was compiled against version 3.6.1 of the Protocol Buffer runtime library, which is not compatible with the
installed version (3.8.0).  Contact the program author for an update.  If you compiled the program yourself, make sure that your headers are from the same version of Protocol Buffers as your link-time library.  (Version verification failed in "/sentencepiece/src/builtin_pb/sentencepiece_model.pb.cc".)
Aborted (core dumped)
```

However, the code below succeeds. The only difference is that the transformers import comes first:
```python
from transformers import AlbertTokenizer, BertTokenizer
import horovod.tensorflow as hvd

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
print("BERT success!") # this succeeds
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
print("ALBERT success!") # this succeeds
```

This bug is a bit bewildering, to be honest. I can stop sorting my imports, I guess... Hoping that someone can identify the root cause.


Create model card

model card

Related to pretrained model cards


language modeling other models