Memory issue in Transformers

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello everyone,
I am using distilBert from hugging face transformers and loading its tokenizer and model in my class. I have already quantised it to reduce the size of the model but still this class is taking nearly 500 MB of the memory though the model is taking only 100 MB. Then I looked into github repo of HuggingFace Transformers they are not using __all__ and __slots__ to their classes and functions, to reduce the memory size of the classes. My question is how do I reduce the memory size while loading any transformer model and why hugging face is not using __all__ and __slots__ in their codebase to make it more efficient.

Thank you in advance.
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


[Generate] beam search should generate without replacement

When doing beam search decoding and sampling instead of argmax (edge case, probably very rarely used), we need to sample **without** replacement. This is implemented by  default in torch, but not in TF, see https://pytorch.org/docs/master/generated/torch.multinomial.html#torch.multinomial (torch). An easy solution is to use the Gumbel max trick instead: https://github.com/tensorflow/tensorflow/issues/9260

This will fix the sometimes flaky TFBeamSearchGenerate Tests as well: #4447


Add support for Funnel-Transformer

# 🌟 New model addition

## Model description

The recently introduced Funnel-Transformer architecture and models would be a great feature for Transformers:

>Funnel-Transformer is a new self-attention model that gradually compresses the sequence of hidden states to a shorter one and hence reduces the computation cost. More importantly, by re-investing the saved FLOPs from length reduction in constructing a deeper or wider model, Funnel-Transformer usually has a higher capacity given the same FLOPs. In addition, with a decoder, Funnel-Transformer is able to recover the token-level deep representation for each token from the reduced hidden sequence, which enables standard pretraining.

The paper can be found [here](https://arxiv.org/abs/2006.03236).

## Open source status

* [x] the model implementation is available: [official GitHub repo](https://github.com/laiguokun/Funnel-Transformer)
* [x] the model weights are available: [Google Cloud Bucket](https://github.com/laiguokun/Funnel-Transformer/blob/master/download_all_ckpts.sh)
* [x] who are the authors: @zihangdai and @laiguokun


remove words from vocabulary

Is there any way to remove the words from the vocabulary of pretrained model? and is there a way to see the vocabulary of the model?


[Benchmark] Add optimization notebook

# 🖥 Benchmarking `transformers`

## Benchmark

This notebook is about benchmarking model training with/without dynamic padding optimization.
https://github.com/ELS-RD/transformers-notebook

**Would it be possible to add it to the [community notebook](https://github.com/huggingface/transformers/tree/master/notebooks) list?** (a link to the Google collab version is provided) @julien-c @patrickvonplaten

## Set-up

GPU : Nvidia P100 provided by Google Collab

## Results

Using dynamic padding on MNLI provides a **4.7 times training time reduction**, with max pad length set to 512. The effect is strong because few examples are >> 400 tokens in this dataset. IRL, it will depend of the dataset, but it always bring improvement and, after more than 20 experiments listed in this [article](https://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e?source=friends_link&sk=10a45a0ace94b3255643d81b6475f409), it seems to not hurt performance.


Multi-output regression support for Transformer models

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

I am trying to build a shipping address to geo-code predictor using RoBERTa. Here the shipping address would be the text input and the output would be a gecode (latitude and longitude).
I tried using _robertaforsequenceclassification_ but it mentions that when the final layer consists of more than one class, a cross entropy loss would be used automatically. But I want to perform regression using the RMSE loss. It would be great if we can add the multi-output regression feature to the existing sequence classification pipeline.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->


BUG while calculate LM loss in AlbertForMaskedLM

# 🐛 Bug

Error Code Here: https://github.com/huggingface/transformers/blob/e33fdc93b4ecb571dd7a8002a74789ec8bfffc09/src/transformers/modeling_albert.py#L822

Here calculate the loss of all tokens in a seq. While MLM actually only calculate loss on [MASK]


[Longformer] Remove redundant code

This PR fixes class  LongformerSelfAttention as follow:

1. Since the method  **_mask_invalid_locations()** was already run in  **_sliding_chunks_matmul_qk()**, it should be removed from **forward()** to avoid code duplication.

2. In the method  **_mask_invalid_locations()**,  since the size of the variable **beginning_mask** is (1, w, 1, w+1) and w is always less than **seqlen**, the index **beginning_mask[:, :seqlen]** is needless.


[Bert Model] ValueError: not enough values to unpack (expected 3, got 2)

# 🐛 Bug: ValueError: not enough values to unpack (expected 3, got 2)

## Information
I am using Bert initialized with 'bert-base-uncased', as per the [documentation](https://huggingface.co/transformers/model_doc/bert.html), the forward step is suppose to yield 4 outputs:
- last_hidden_state
- pooler_output
- hidden_states
- attentions

But when I try to intialize BERT and call forward method, it yields only 2 results. Based on the shape, I feel they are the hidden_states and pooler_output.
```
self.bert_model = BertModel.from_pretrained('bert-base-uncased')
_, _, hidden_states = self.bert_model(input_ids, attn_masks, token_type_ids)
```

**Error**
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-69-6d2cb1238cab> in <module>
     45 for i, data in enumerate(trainloader):
     46     input_ids, attn_mask, token_type_ids = data['tokens'], data['attention_mask'], data['token_type_ids']
---> 47     start_logits, end_logits = model.forward(input_ids, attn_mask, token_type_ids)
     48     print(start_logits.shape)
     49     print(end_logits.shape)

<ipython-input-69-6d2cb1238cab> in forward(self, input_ids, attn_masks, token_type_ids)
     23
     24         # Feeding the input to BERT model to obtain hidden_states of all the layers
---> 25         _, _, hidden_states = self.bert_model(input_ids, attn_masks, token_type_ids)
     26
     27         # Shape of hidden_states is (1, 50, 768)

ValueError: not enough values to unpack (expected 3, got 2)
```
Model I am using (Bert, XLNet ...): Bert
Language I am using the model on English

The problem arises when using:
* [ ] the official example scripts: NA
* [x] my own modified scripts: Below are scripts details.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: NA
* [x] my own task or dataset: Fine tuning for my own task.

## To reproduce

Steps to reproduce the behavior:

1. Copy paste the full code below in a notebook.
2. Run as is.

Complete code:
```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

# Dataset definition
class TweetDataset(Dataset):
    def __init__(self, data, maxlen, tokenizer):
        self.df = data
        self.tokenizer = tokenizer
        self.maxlen = maxlen

    def __len__(self):
        return len(self.df)

    def __getitem__(self, index):
        """
        Returns the token_ids_tensors, attn_mask for the item and text denoting the sentiment.

        :param index:
        :return:
        """

        # Selecting the sentence and label at the specified index in the data frame
        orig_sentence = self.df.iloc[index]['text']
        sentiment = self.df.iloc[index]['sentiment']
        selected_text = self.df.iloc[index]['selected_text']

        # Preprocessing the text to be suitable for BERT

        # Encode the sentence. Does the following:
        # 1. Inserting the CLS and SEP token in the beginning and end of the sentence
        # 2. Generates attention mask
        # 3. Generate token_type_ids used to differentiate first part of the sentence from the second
        encoded_dict = self.tokenizer.encode_plus(
            sentiment,
            orig_sentence,
            max_length=self.maxlen,
            truncation_strategy='only_second',
            add_special_tokens=True,
            pad_to_max_length=True,
            return_tensors='pt',
            return_token_type_ids=True,
            return_attention_mask=True
        )
        tokens = encoded_dict['input_ids'][0]
        token_type_ids = encoded_dict['token_type_ids'][0]
        attn_mask = encoded_dict['attention_mask'][0]

        # Determine the beginning and end of the sentence
        def phrase_start_finder(sentence, phrase):
            if phrase not in sentence:
                raise ValueError('s2 not substring of s1')
            start = sentence.find(phrase)
            return len(sentence[:start].strip().split(' '))

        def phrase_end_finder(sentence, phrase):
            if phrase not in sentence:
                raise ValueError('s2 not substring of s1')
            return phrase_start_finder(sentence, phrase) + len(phrase.strip().split(' ')) - 1

        start = phrase_start_finder(orig_sentence, selected_text)
        end = phrase_end_finder(orig_sentence, selected_text)

        return {
            'tokens': tokens,
            'attention_mask': attn_mask,
            'token_type_ids': token_type_ids,
            'start': float(start),
            'end': float(end),
            'sentence': orig_sentence,
            'selected_text': selected_text,
            'sentiment': sentiment
        }

# Defining the loader
dataset = TweetDataset(train_data, 50, tokenizer)

trainloader = DataLoader(
    dataset,
    batch_size=BATCH_SIZE,
    shuffle=True,
    num_workers=4
)

# Defining the model
class TweetModel(nn.Module):
    def __init__(self, freeze_bert=True):
        super(TweetModel, self).__init__()
        # Instantiating BERT model object
        self.bert_model = BertModel.from_pretrained('bert-base-uncased')

        # TODO(Viman): Before training on GPUs and finalization, remove this
        # Freeze bert layers
        # In first experiment, not training the previous layers
        if freeze_bert:
            for p in self.bert_model.parameters():
                p.requires_grad = False

        # Final layer. Needs two outputs which are supposed to be logits: startIndex and endIndex
        self.dropout = nn.Dropout(0.2)
        # 768 because output is a vector of size 768 (Dimensionality of the encoder layer)
        self.fc = nn.Linear(768, 2)
        # Intialize the fc layer
        nn.init.normal_(self.fc.weight, std=0.02)
        nn.init.normal_(self.fc.bias, 0)

    def forward(self, input_ids, attn_masks, token_type_ids):

        # Feeding the input to BERT model to obtain hidden_states of all the layers
        _, _, hidden_states = self.bert_model(input_ids, attn_masks, token_type_ids)

        # Shape of hidden_states is (1, 50, 768)
        # TODO(Viman): Try mean as opposed to max
        # hidden_states, _ = torch.max(hidden_states, dim=1)

        # last_hidden_state = hidden_states[-1]
        print(hidden_states.shape)

        X = self.dropout(hidden_states)
        logits = self.fc(X)

        start_logits, end_logits = logits.split(1, dim=-1)
        start_logits = start_logits.squeeze(-1)
        end_logits = end_logits.squeeze(-1)

        return start_logits, end_logits

model = TweetModel()

# Testing the model forward implementation
for i, data in enumerate(trainloader):
    input_ids, attn_mask, token_type_ids = data['tokens'], data['attention_mask'], data['token_type_ids']
    start_logits, end_logits = model.forward(input_ids, attn_mask, token_type_ids)
    print(start_logits.shape)
    print(end_logits.shape)
    if i == 1:
        break
```

## Expected behavior
The self.bert_model(input_ids, attn_masks, token_type_ids) line should return the a tuple containing 4 elements however it seems to return 2 only.

## Environment info
- `transformers` version: 2.9.0
- Platform: Linux-4.19.112+-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: Not yet
- Using distributed or parallel set-up in script?: No

- `transformers` version: 2.11.0
- Platform: Mac/Kaggle notebook (Tried in both)
- Python version: 3.7
- PyTorch version (GPU?): No
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


[examples] consolidate summarization examples

Consolidating summarization examples of T5 & Bertabs models into one - [#3826](https://github.com/huggingface/transformers/issues/3826)
@sshleifer


Remove unneeded call convert_ids_to_tokens.

On the base `PreTrainedTokenizer` object, calling `convert_tokens_to_string` returns an error because the method tries to call `convert_ids_to_tokens.` however the input should already be a list of tokens not ids.

I noticed this when creating a tokenizer that inherits from the Pretrained Tokenizer, and trying to test it with:

```python
assert transformers_tokenizer.decode([1, 15, 22, 15, 2]) == ">MVM<"
```

which eventually results in:

```python
self = <gcgc.third_party.GCGCTransformersTokenizer object at 0x16539ead0>, ids = ['>', 'M', 'V', 'M', '<'], skip_special_tokens = False

    def convert_ids_to_tokens(
        self, ids: Union[int, List[int]], skip_special_tokens: bool = False
    ) -> Union[int, List[int]]:
        """ Converts a single index or a sequence of indices (integers) in a token "
            (resp.) a sequence of tokens (str), using the vocabulary and added tokens.

            Args:
                skip_special_tokens: Don't decode special tokens (self.all_special_tokens). Default: False
        """
        if isinstance(ids, int):
            if ids in self.added_tokens_decoder:
                return self.added_tokens_decoder[ids]
            else:
                return self._convert_id_to_token(ids)
        tokens = []
        for index in ids:
>           index = int(index)
E           ValueError: invalid literal for int() with base 10: '>'
```

But I don't think this should get called in the first place?

I think I'm actually go override this now that I understand the process better, but this still seemed incorrect.


Any reason why BART does not have a ForTokenClassification variant?

Core: Modeling

Internals of the library; Models.

Are there any theoretical constraints to create a ForTokenClassification variant for BART? In my current project I am using a sequence classification head + a token classification head, so I would like to implement the token classification part manually. However, since it is not implemented in the repository, I wonder if there are any particular reasons why one should not do this.


Why init specific layers rather than whole model  in BART

Internals of the library; Models.

In BartForSequenceClassification I can see that rather than calling `self.init_weights()` (which most other models use) only specifically the classification head is initialized.

https://github.com/huggingface/transformers/blob/c58e6c129a153ca1a5021e5d7e642d00bf011e20/src/transformers/modeling_bart.py#L1046-L1047

Is there any advantage of doing this for the head(s) only rather than for the whole model? I can think of a speed improved, but apart from that I'm not sure.


GlossBert adding

Here is a link https://github.com/HSLCY/GlossBERT - it should be able to do better for word vectors representations and there is already pretrained model than maybe need to be converted to a different format.


Why exclude LayerNorm.bias from weight decay when finetuning?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L306
In the original BERT implementation and in earlier versions of this repo, both LayerNorm.weight and LayerNorm.bias are decayed.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


TF Checkpoints

Align how the checkpoints are managed the same way than in the PyTorch trainer.


Add diagnostic dataset of glue tasks for prediction

As diagnostic dataset didn't have train set, it's very common in current research work to use model fine tuned on MNLI task to conduct prediction of diagnostic dataset.

As experimented, by adding logic in this request, we can achieve 47.1% on diagnostic dataset after fine-tuning 10k steps on MNLI dataset.


[examples] Add trainer support for question-answering

This PR adds trainer support for question-answering task. Regarding issue #4784

**TODOs**
- [ ]  Add automatic data loading. Right now it requires the user to specify data directory. Decided not to use `tfds` because I think it will be soon replaced by `nlp` here
- [ ] Add evaluation
- [ ] Test all models.

@julien-c  @patrickvonplaten


`run_glue.py` fails with models `bert-base-cased`, `distil-bert-cased`, others

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `bert-base-cased`

Language I am using the model on (English, Chinese ...): `English`

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. download GLUE data
2. run `python run_glue.py --model_name_or_path bert-base-cased`

Error message:

```
06/07/2020 00:01:06 - INFO - transformers.trainer -   ***** Running training *****
06/07/2020 00:01:06 - INFO - transformers.trainer -     Num examples = 3668
06/07/2020 00:01:06 - INFO - transformers.trainer -     Num Epochs = 3
06/07/2020 00:01:06 - INFO - transformers.trainer -     Instantaneous batch size per device = 4
06/07/2020 00:01:06 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 4
06/07/2020 00:01:06 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
06/07/2020 00:01:06 - INFO - transformers.trainer -     Total optimization steps = 2751
Iteration:   1%|█▊                                                                                | 9/917 [00:24<40:42,  2.69s/it]
Epoch:   0%|                                                                                | 0/3 [00:24<?, ?it/s]
Traceback (most recent call last):
  File "./run_glue.py", line 247, in <module>
    main()
  File "./run_glue.py", line 174, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 471, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/trainer.py", line 571, in _training_step
    outputs = model(**inputs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 1143, in forward
    inputs_embeds=inputs_embeds,
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 727, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/u/.conda/envs/torch/lib/python3.7/site-packages/torch/nn/functional.py", line 1484, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: index out of range: Tried to a
ccess index 29597 out of table with 28995 rows. at /opt/conda/conda-bld/pytorch_1579040055865/work/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```
## Expected behavior

Model successfully trains. The script works well on my machine for many other models, including `bert-base-uncased` ~~and `distilbert-base-cased`~~.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
```
- `transformers` version: 2.11.0
- Platform: Linux-3.10.0-693.el7.x86_64-x86_64-with-centos-7.4.1708-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: [yes]
- Using distributed or parallel set-up in script?: [yes, parallel, but I received the same error when I ran the script just using the CPU]
```s


How to remove token ?

I only know how to add token, but how to remoce some special token


Fix use of mems in Transformer-XL text generation

In Transformer-XL, when ```mems``` is being used to save computation with the ```generate``` function, the inputs are not properly truncated, so that ```mems``` does not actually speed things up, and also seems to create inaccuracies in the output. I have attempted to fix this by changing Transformer-XL's ```prepare_inputs_for_generation``` function to make it more like that function as used in GPT-2.

See the issue at https://github.com/huggingface/transformers/issues/4752 for more details.


Onnx converted model has its output shape modified when compared to original (finetuned) model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `mrm8488/distilroberta-base-finetuned-sentiment` from the hub

Language I am using the model on (English, Chinese ...): `English`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

I use the `04-onnx-export.ipynb` Notebook and have only change the model name and the tokenizer:
![image](https://user-images.githubusercontent.com/1029874/83956354-6a799800-a85d-11ea-9b0b-b2febdd8e91a.png)

The issue appeared on all finetuned model I tried, being classification or multichoice questions.

The tasks I am working on is:
* [ ] my own task or dataset: (give details below)
* [X] an official GLUE/SQUaD task: classification

## To reproduce

Steps to reproduce the behavior:

Import AutoTokenizer, AutoModelForSequenceClassification and change tokenizer and model name, the section we are interested into:

```python
# ...
!rm -rf onnx/
from transformers.convert_graph_to_onnx import convert

# Handles all the above steps for you
convert(framework="pt", model="mrm8488/distilroberta-base-finetuned-sentiment", output="onnx/bert-base-cased.onnx", opset=11)
# ...

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("mrm8488/distilroberta-base-finetuned-sentiment")
cpu_model = create_model_for_provider("onnx/bert-base-cased.onnx", "CPUExecutionProvider")

# Inputs are provided through numpy array
model_inputs = tokenizer.encode_plus("My name is Bert", return_tensors="pt")
inputs_onnx = {k: v.cpu().detach().numpy() for k, v in model_inputs.items()}

# Run the model (None = get all the outputs)
sequence, pooled = cpu_model.run(None, inputs_onnx)

# Print information about outputs

print(f"Sequence output: {sequence.shape}, Pooled output: {pooled.shape}")

pytorch_model = AutoModelForSequenceClassification.from_pretrained("mrm8488/distilroberta-base-finetuned-sentiment")
a, = pytorch_model(**model_inputs)
print(f"finetune non onnx pytorch model output: {a.shape}")
# ...
```

## Expected behavior

I was expecting that the onnx output shape would be the same than the non converted model output shape, but that's not the case:
```text
Sequence output: (1, 6, 768), Pooled output: (1, 768)
finetune non onnx pytorch model output: torch.Size([1, 6])
```
It is like the last layer of the model related to the classification task is not taken in onnx.
Does it make sense? @mfuntowicz

## Environment info
Google Colab with a GPU


Top-k sampling and top-p sampling for generating phrases on batches with GPT-2?

How can I generate on batches with GPT-2 if I want to make use of these awesome sampling techniques: [top-k sampling and top-p sampling](https://huggingface.co/blog/how-to-generate)?.

There is this implementation for generating phrases on batches already in issue [#3021](https://github.com/huggingface/transformers/issues/3021).

Any advice? thanks!

@patrickvonplaten


Discriminative fine-tuning for new (added) words

Good afternoon,
I have a question about the way of changing the learning rate for some parameters of the model. Let's say we have a BERT model and we have added a few new tokens. Consequently, we need to resize the embedding layer and initialize the embeddings for new words randomly. Meanwhile the learning rate of 3e-5 is used to train the model (otherwise we "overjump" the global minimum). If we use this LR for the embeddings of the new words as well, they will hardly change and thus will be close to random, thereby, the reasonable way is to change the learning rate only for their embeddings (as it is done, for instance, in ULMFiT). The question is: is there a simple way to do it in HuggingFace? Or probably are there some examples of doing it? Thanks in advance!


EncoderDecoderModel forwards return different values every time.

# 🐛 Bug in EncoderDecoderModel

I am using EncoderDecoderModel and I have tested the sample code of it which is written in its page.

```
from transformers import EncoderDecoderModel, BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = EncoderDecoderModel.from_encoder_decoder_pretrained(
    'bert-base-uncased', 'bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)

output = model(input_ids=input_ids, decoder_input_ids=input_ids)[0]
```

but every time I run this code i will get different values for output! I also have used model.eval() but it also couldn't help.


Enable multiprocessing in glue datasets

The preprocessing of glue datasets is too slow. This change enables multiprocessing to speed up the process of converting examples to features by utilizing multiple cpu cores.


Updates args in tf squad example.

Updates example for execution of `run-tf-squad.py` due to changes in https://github.com/huggingface/transformers/pull/4530, particularly removal of `mode` and `optimizer_name`.


Export PretrainedBartModel from __init__

`PretrainedBartModel ` is currently not being exported so one has to manually do

```python
from transformers.modeling_bart import PretrainedBartModel
```

This required behaviour is different from other models which do expose their PretrainedModel in `__init__`.


Enable multiprocessing in glue datasets


Question: Where do I find the Transformer model from the paper "Attention is all you need" ?

Hello

  Firstly, thanks for supporting all questions here.

I read the paper "Attention is all you need" and wondering which class should I use in the HuggingFace library to use the Transformer architecture used in the paper.

Can you please advise?

Thanks
Abhishek


NER pipeline: Inconsistent entity grouping

# 🐛 Bug

## Information

"mrm8488/bert-spanish-cased-finetuned-ner"

Language I am using the model on (English, Chinese ...):  Spanish

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. create a `ner` pipeline
2. pass flag `grouped_entities`
3. entities are not grouped as expected see sample below

```python
NER_MODEL = "mrm8488/bert-spanish-cased-finetuned-ner"
nlp_ner = pipeline("ner", model=NER_MODEL,
                   grouped_entities=True,
                   tokenizer=(NER_MODEL, {"use_fast": False}))

t = """Consuelo Araújo Noguera, ministra de cultura del presidente Andrés Pastrana (1998.2002) fue asesinada por las Farc luego de haber permanecido secuestrada por algunos meses."""
ner(t)
>>>
[ {'entity_group': 'B-PER', 'score': 0.901019960641861, 'word': 'Consuelo'},
 {'entity_group': 'I-PER', 'score': 0.9990904808044434, 'word': 'Araújo Noguera'},
 {'entity_group': 'B-PER', 'score': 0.9998136162757874, 'word': 'Andrés'},
 {'entity_group': 'I-PER', 'score': 0.9996985991795858, 'word': 'Pastrana'},
 {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Far'}]
```

## Expected behavior

### Inconsistent grouping
I expect the first two items of the given sample( `B-PER`, and `I-PER`) to be grouped. As they are contiguous tokens and correspond to a single entity spot.   It seems the current code does not take into account `B` and `I` tokens.

expected output:
```
 {'entity_group': 'I-PER', 'score': 0.9990904808044434, 'word': ' Consuelo Araújo Noguera'},
 {'entity_group': 'I-PER', 'score': 0.9998136162757874, 'word': 'Andrés Pastrana'},
 {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Farc'}]
```

### Lost tokens?

for the same input, passing `grouped_entities=False` generates the following output:

```
[
{'word': 'Cons', 'score': 0.9994944930076599, 'entity': 'B-PER', 'index': 1},
{'word': '##uelo', 'score': 0.802545428276062, 'entity': 'B-PER', 'index': 2},
{'word': 'Ara', 'score': 0.9993102550506592, 'entity': 'I-PER', 'index': 3},
{'word': '##új', 'score': 0.9993743896484375, 'entity': 'I-PER', 'index': 4},
{'word': '##o', 'score': 0.9992871880531311, 'entity': 'I-PER', 'index': 5},
{'word': 'No', 'score': 0.9993029236793518, 'entity': 'I-PER', 'index': 6},
{'word': '##guera', 'score': 0.9981776475906372, 'entity': 'I-PER', 'index': 7},
{'word': 'Andrés', 'score': 0.9998136162757874, 'entity': 'B-PER', 'index': 15},
{'word': 'Pas', 'score': 0.999740719795227, 'entity': 'I-PER', 'index': 16},
{'word': '##tran', 'score': 0.9997414350509644, 'entity': 'I-PER', 'index': 17},
{'word': '##a', 'score': 0.9996136426925659, 'entity': 'I-PER', 'index': 18},
{'word': 'Far', 'score': 0.9989739060401917, 'entity': 'B-ORG', 'index': 28},
{'word': '##c', 'score': 0.7188423275947571, 'entity': 'I-ORG', 'index': 29}]
```

when using `grouped_entities`  the last entity `word` (`##c`) got lost, it is not even considered as a different group

` {'entity_group': 'B-ORG', 'score': 0.9989739060401917, 'word': 'Far'}]`

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: OSX
- Python version: 3.7
- PyTorch version (GPU?):  1.5.0
- Tensorflow version (GPU?):
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


[marian tests ] pass device to pipeline

fixes self-hosted-runner failure


TPU Training fails with --evaluate_during_training

# 🐛 Bug

TPU Trainer does not seem to support `--evaluate_during_training`. When the training loop goes into logging part, the whole process just hangs up stalling training. The same code/dataset with a multi-gpu setup works well.

I am trying to move my company to Huggingface so want to train models on TPUs on our dataset which hung during the logging step. I was able to replicate the behavior with run_langugage_modelling.py, and the steps to replicate this are shown below.

Other observations are -
I felt that multiprocessing way of doing TPU training wastes a lot of CPU memory because with large datasets one has to use a machine with 100s of GBs of RAM because the features are being replicated 8 times in memory.
Another bug is that with TPU training there are 8 WandB runs generated and it creates a lot of clutter. Suggestions to fix this would be to only do wandb logging from a single process. If its unavoidable to generate 8 wandb runs, tag all the runs to belong to a single 'group' that leads to better organization of the runs. (https://docs.wandb.com/library/advanced/grouping)

## Information

Model I am using (Bert, XLNet ...): Roberta with run_language_modelling.py to replicate, T5 with our internal data.

Language I am using the model on (English, Chinese ...):  English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a new n1-highmem-32 machine with debian-9-torch-xla OS image in us-central1-c zone
2. `conda activate torch-xla-nightly` and start a v2-8 TPU in us-central1-c zone. Set the TPU env vars
3. Use the master branch of transformers
4. Download Wikitext 103 raw char level data from https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-103-raw-v1.zip (according to examples for run_language_modelling). Extract it
5. Run the example script
```
export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python xla_spawn.py --num_cores 8 language_modelling/run_language_modeling.py \
    --output_dir=output \
    --model_type=roberta \
    --model_name_or_path=roberta-base \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE \
    --mlm
    --evaluate_during_training
    --per_device_train_batch_size=4
    --per_device_eval_batch_size=4
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
When it hangs, the tqdm counter is stuck at step 499 (with 500 as the logging interval) and nothing happens. When I do a Keyboard Interrupt, I get this stack trace.

```
main()
  File "../../../vendor/transformers/examples/xla_spawn.py", line 68, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 296, in spawn
    start_method=start_method)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 78, in join
    timeout=timeout,
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
KeyboardInterrupt
```
## Expected behavior
Being able to log validation set loss during training
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 `transformers` version: 2.11.0
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0a0+03eca38 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: Yes, 8 core parallelism with xla_spawn.py


Is albert lm finetuning with SOP in Pytorch supported?

# ❓ Questions & Help

Hello,

I am trying to use transfer learning on the albert language model. Before i train it on Squad.

Does run_language_modeling.py support albert models and SOP ?

Thank you

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


[cleanup/marian] pipelines test and new kwarg

avoids DeprecationWarning (because `max_len` kwarg is being deprecated)


Add model and doc badges

Add badges at each model for:
- the page with all community models
- the documentation of the model

Remove the manual doc links as a result.


[Benchmark] Add encoder decoder to benchmark and clean labels

This PR cleans the benchmark utils a bit more:
- tracing is made independent from CPU memory benchmarking
- possibility to benchmark encoder-decoder models is added
- 3 new tests
- general refactoring


[EncoderDecoderConfig] automatically set decoder config to decoder

When instantiating an encoder-decoder configuration from two pretrainied configs, the decoder config should automatically set to `config.is_decoder=True`.

In general, whenever we instantiate an encoder decoder model, no matter how, the resulting decoder config should have the attribute `decoder.is_decoder=True`.

This PR also adds a couple of tests to make sure that an encoder-decoder model can be instantiated from two configs over the encoder decoder config class.


Expose classes used in documentation

Currently, the documentation page of the tokenizers has three methods lacking documentation (see [here](https://huggingface.co/transformers/main_classes/tokenizer.html#pretrainedtokenizerfast)). This PR adds them to the `__init__` so sphynx can see them.

If there is one that should not be public, we should remove it from the documentation.


Use labels to remove deprecation warnings

This is a follow-up from #4722 and remove the deprecated arguments in the tests.


Albert pretrained weights change across runs.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): TFAlbertModel

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)
```
import tensorflow as tf
from transformers import AlbertTokenizer, TFAlbertModel

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = TFAlbertModel.from_pretrained('albert-base-v2')
model.summary()

print(len(model.trainable_weights))

print(model.trainable_weights[23])

input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]

outputs = model(input_ids)

print(outputs[0].shape, outputs[1].shape, len(outputs))
last_hidden_states = outputs[0]

print(last_hidden_states)
```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)
Trying to load pre-trained weights
## To reproduce
Run the code above two times and you will see that the weights of the model are not the same across the two runs
Steps to reproduce the behavior:

1. Run the code the first time and log the output
2. Run the code a second time and log the output
3. Check that the two logs are not the same.

## Expected behavior

Since the model is loading pre-trained weights the results should be the same across runs.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-4.4.0-179-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.0.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

I apologize if the issue is due to me misusing your library, first time using Albert.


Invalid Argument for Onnxruntime Inference on GPT2

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
I've been following the ipython notebook provided [here](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb)

1. Take an off-the-shelf pretrained `gpt` model and export to onnx format using the following invocation:
```
python convert_graph_to_onnx.py --framework pt --model gpt2 gpt2.onnx
```
2. Run inference on the exported onnx model, following the steps [here](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb). After invoking the appropriate provider, run inference using something like the following
```
model.run(None, {"input_ids": np.array([blah]), "token_type_ids": np.array([blah]), "attention_mask": np.array([blah])
```
Note above, `blah` is replaced with actual data.

After invoking the above, I get the error:
```
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Invalid Feed Input Name:token_type_ids
```

## Expected behavior

I would expect this to work successfully.

My hypothesis is that the `convert_graph_to_onnx.py` is not exporting all the inputs from the `gpt2` model. In particular in line 43-48:
```
for arg_name in model_args_name[1:]:  # start at index 1 to skip "self" argument
        if arg_name in input_names:
            ordered_input_names.append(arg_name)
            model_args.append(tokens[arg_name])
        else:
            break
```

`model_args` is only populated with `input_ids` because the order of arguments in the `forward` method of `gpt2` is `input_ids, past, attention_mask, token_type_ids` so the for loop breaks early.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: Commit 0e1869cc286d607f1598506be7bd1312b76ca82c
- Onnxruntime: 1.3.0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0+cu101
- Using GPU in script?: Yes

Thanks for your help! @mfuntowicz @tianleiwu


Add link to community models


[WIP] Blenderbot

**UPDATES - 14 AUGUST 2020**

- Blenderbot-3B is working exactly in the same way as parlai and they have the same generation outputs

- Blenderbot-90M also generates the same output as parlai in most cases but it can sometime generate a sequence with a small difference.
For example:

Parlai: `i' m not sure . i just feel like i ' m going to throw up . `
hf: `i don ' t know . i just feel like i ' m going to throw up .`
Discrepancy could be from length penalty or some other beam search param.

**Update Sep 17**

@sshleifer  taking over

### TODO:
- check eos generation
- test distilled 2b model
- `AutoTokenizer`/`AutoModelForSeq2SeqLM` coverage
- debug failing 3b integration test
- implement backwards compatibility for the variant change by checking `config.model_type`
- document `layernorm_variant` nicely or pursue alternative solution.


[cleanup] MarianTokenizer: delete unused constants

slow tests pass.
The only needed constant is `vocab_files_names`


pip install -e does not always install the correct isort version

```bash
pip install -e .["dev"]
make quality
```
### Output
```bash
black --check --line-length 119 --target-version py35 examples templates tests src utils
All done! ✨ 🍰 ✨
306 files would be left unchanged.
isort --check-only --recursive examples templates tests src utils
ERROR: /Users/shleifer/transformers_fork/examples/benchmarking/plot_csv_file.py Imports are incorrectly sorted.
ERROR: /Users/shleifer/transformers_fork/templates/adding_a_new_example_script/run_xxx.py Imports are incorrectly sorted.
ERROR: /Users/shleifer/transformers_fork/templates/adding_a_new_example_script/utils_xxx.py Imports are incorrectly sorted.
ERROR: /Users/shleifer/transformers_fork/src/transformers/__init__.py Imports are incorrectly sorted.
make: *** [quality] Error 1
```
relevant packages:
```python
flake8==3.8.1
isort==4.3.21
black==19.10b0
```

Env:
```
- `transformers` version: 2.11.0
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
```

Would also be good to add more verbose error messages if possible


[isort] add matplotlib to known 3rd party dependencies

Many people (like me) have it installed locally, so this will synchronize local isort and circleci.


[cleanup] consolidate some prune_heads logic

factors out 5 repetitions  of the following logic
```python
def find_pruneable_heads_and_indices(
    heads: List, n_heads: int, head_size: int, already_pruned_heads: set
) -> Tuple[set, "torch.LongTensor"]:
    mask = torch.ones(n_heads, head_size)
    heads = set(heads) - already_pruned_heads  # Convert to set and remove already pruned heads
    for head in heads:
        # Compute how many pruned heads are before the head and move the index accordingly
        head = head - sum(1 if h < head else 0 for h in already_pruned_heads)
        mask[head] = 0
    mask = mask.view(-1).contiguous().eq(1)
    index: torch.LongTensor = torch.arange(len(mask))[mask].long()
    return heads, index

```


[ctrl] has broken code for pruning that is not tested

references an `attn` parameter than should be `multiheaded_attn`. Easy-ish fix.


Write With Transformer Request:

# 🚀 Feature request

I understand if this is unattainable for reason of money, but I was wondering if you could replace three autocomplete suggestions with five.


Ignore simlink

Didn't get an answer to my question on #4774 so asking again in the form of a PR ;-)

Currently, building the docs require making a simlink to the examples README (as per the [instructions](https://github.com/huggingface/transformers/tree/master/docs#building-the-documentation)) and that file then becomes untracked bit git. We should either ignore it (as proposed in this PR) or add it once for all (might be OS-dependent though).

Happy to amend this PR to the second solution, I just don't like untracked files. :-)


Explain how to preview the docs in a PR

As discussed offline, add the instructions to check how to preview the docs.


enable multiprocessing in glue dataset

enable multiprocessing when converting examples to features utilizing the multiple cpu cores. N time faster...


🐛 run_ner.py runtime error linked to TPU training

# 🐛 Bug

## Information

Model I am using **Longformer For Token Classification**

Language I am using the model on **German**:

The problem arises when using:
* [x] the official example scripts:
T**he problem arises when trying to run run_ner.py on google colab in TPU fp16 mode.**

The tasks I am working on is:
* [x] an official GLUE/SQUaD task:
**CoNLL NER**

## To reproduce

I have a colab up if you want to see exactly what I did. You just need to upload data files in a new folder in /Content/<YOUR_FOLDER> to use in training and make sure to modify the run_ner paths correspondingly. [Google Colab](https://github.com/vinmorel/transformers/blob/master/run_ner_TPU.ipynb)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Runtime error after running the following :
```
!python3 "/content/transformers/examples/token-classification/run_ner.py" --data_dir "/content/CoNLL/" \
--labels "/content/CoNLL/labels.txt" \
--model_name_or_path "allenai/longformer-base-4096" \
--output_dir "xlnet-base-cased" \
--max_seq_length  200 \
--num_train_epochs 2 \
--per_device_train_batch_size 1 \
--save_steps 750 \
--seed 1 \
--do_train \
--do_eval  \
--do_predict \
--fp16
```
```
Traceback (most recent call last):
  File "/content/transformers/examples/token-classification/run_ner.py", line 303, in <module>
    main()
  File "/content/transformers/examples/token-classification/run_ner.py", line 228, in main
    model_path=model_args.model_name_or_path if os.path.isdir(model_args.model_name_or_path) else None
  File "/usr/local/lib/python3.6/dist-packages/transformers/trainer.py", line 390, in train
    model, optimizer = amp.initialize(model, optimizer, opt_level=self.args.fp16_opt_level)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/frontend.py", line 358, in initialize
    return _initialize(models, optimizers, _amp_state.opt_properties, num_losses, cast_model_outputs)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py", line 171, in _initialize
    check_params_fp32(models)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/_initialize.py", line 93, in check_params_fp32
    name, param.type()))
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/_amp_state.py", line 32, in warn_or_err
    raise RuntimeError(msg)
RuntimeError: Found param longformer.embeddings.word_embeddings.weight with type torch.FloatTensor, expected torch.cuda.FloatTensor.
When using amp.initialize, you need to provide a model with parameters
located on a CUDA device before passing it no matter what optimization level
you chose. Use model.to('cuda') to use the default device.
```

## Expected behavior

Would expect the code to run and start training on TPU without the runtime error.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0.dev20200528 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Fix argument label

After #4722 the labels are called just `labels` now, not `masked_lm_labels`. The fact it wasn't caught by the tests probably means we have some test missing...


parse arguments from dict

This PR adds `parse_dict` method to `HfArgumentParser`  to allow parsing arguments from `dict`.

I find this necessary for notebook workflows where I'm not using `Trainer` from command line. Otherwise I need to write arguments to json file and  use that path with `parse_json_file` or pass a list of strings to `parse_args_into_dataclasses`.

@julien-c @patrickvonplaten


Clean-up code

Looks like #4747 introduced some bad formatting, fixing so CI is happy again.


Add model summary

This PR adds a high-level summary of all the models in the documentation.


Onnx conversion for bert models with classification layers

Hi,
  I am trying to convert a NER model trained with BertForTokenClassification to Onnx format. Am able to convert it using the convert_graph_to_onnx.py script using the following params

convert(framework="pt", model="bert_small_ner", output="bert_small_onnx/bert_small_ner.onnx", opset=11)

There are no errors thrown either. However, when i run the inference, I see only the Bert layer outputs but not the output of the Linear classification layer as it is in BertForTokenClassification.

Am I missing something? Or this is a know issue with some work around? Kindly let me know

Thanks in advance


🚀 [Feature Request] Add self-contained browsable examples/notebooks in the docs

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation and feature request
Recently I'm seeing lots of beginner-level issues asking help on how to fine-tune a particular model for a particular task, how to use already trained model from the hub or how to do inference for certain task, what some parameters mean etc. See #4744, #4406, #4677, #4639

While the examples in the /examples directory are really awesome, it seems that they are bit hard to understand from a beginner's perspective. Also even though there is notebooks section, some people still seem to not find it. And as the library is getting very popular lots of students/beginners are starting their NLP journey with Transformers.

So IMO it would be really awesome if we have self-contained end-to-end browsable notebook examples with clear task and model descriptions  in the docs itself (like the ones in pytorch and keras docs)

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
I have few notebooks in community notebooks section and I would be happy to contribute more examples with clear descriptions  for individual task with both fine-tuning and inference details.

@patrickvonplaten @julien-c


Usage of Ġ in BPE tokenizer

Hello,
I want to add new words to my BPE tokenizer. I know the symbol Ġ means the end of a new token and the majority of tokens in vocabs of pre-trained tokenizers start with Ġ. Assume I want to add the word **Salah** to my tokenizer. I tried to add both **Salah** token and **ĠSalah**:
`tokenizer.add_tokens(['Salah', 'ĠSalah'])` # they get 50265 and 50266 values respectively.
However, when I tokenize a sentence where **Salah** appears, the tokenizer will never return me the second number (neither when using `.tokenize` nor `.encode`), for instance:
`tokenizer.tokenize('I love Salah and salad')` returns `['I', 'Ġlove', 'Salah', 'Ġand', 'Ġsalad']`.
The question is: should I use the symbol `Ġ` when adding new tokens or the tokenizer does it itself? Or, probably, it must be specified manually?
Thanks in advance!


Question Answering Pipeline with big texts.

# ❓ Questions & Help

I am searching for some ideas of how to use QA pipeline with big data texts and get a good time response. (I am already using GPU)


Can I train question-answering on TPU using Huggingface

# ❓ Questions & Help
I am trying to run run_squad from question answering on TPU but I get the following error.

## Details
export SQUAD_DIR=/path/to/SQUAD
python examples/xla_spawn.py --num_cores 8 \
 run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-uncased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/

**run_squad.py: error: unrecognized arguments: --tpu_num_cores 1**


question about tokenizer

hi, is there any difference between initializing BertTokenizer and load it using from_pretrained,
such as:
1、tokenizer = BertTokenizer(bert-base-uncased-vocab)
2、tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

i want to pretrain for some new language by myself, so i need to create the vocabulary, and how could i integrate it to the transformers's tokenizer
thanks a lot!


❓ How to use Gradient Accumulator in TF_Trainer ?

# ❓ Questions & Help

According to the documentation of Gradient Accumulator :

> the accumulator should be called in a
  replica context. Gradients will be accumulated locally on each replica and
  without synchronization. Users should then call ``.gradients``, scale the
  gradients if required, and pass the result to ``apply_gradients``.

The default optimizer for TF_Trainer does not provide gradient accumulation.

**Is there any example available showing how to use Gradient Accumulator with TF_Trainer ?**


Regarding generate method used in BART

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarily intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiasts can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you can always open a question on GitHub. You should then link to the SO question that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if you didn't get an answer ask it here on GitHub. -->

I am new to NLP and text generation so please help me understand some basic things.
I fine-tuned BART on CNN/DM dataset using provide scripts in the examples section and it works fine.
I have some understanding of how model.generate() method works. But need to clarify some basic questions.
Is the generate() method used at the time of training/validation?
If yes, then how loss is computed if the beam size is greater than 1?
Or do we even use the parameters like in generate() method during training?

Please help. Thank you.


Reformer hidden_size of output is doubled.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

The default hidden_size is 256 but the output dim is 512. Similarly, when I change the config and set hidden_size to 512, the output dim is 1024.

```python

enc_config = {
    "attention_head_size": 64,
    "attn_layers": ["local", "lsh", "local", "lsh", "local", "lsh"],
    "axial_pos_embds": True,
    "sinusoidal_pos_embds": False,
    "axial_pos_embds_dim": [256, 256],
    "axial_pos_shape": [64, 64],
    "lsh_attn_chunk_length": 64,
    "local_attn_chunk_length": 64,
    "feed_forward_size": 256,
    "hidden_act": "relu",
    "hidden_size": 512,
    "is_decoder": False,
    "max_position_embeddings": 4096,
    "num_attention_heads": 12,
    "num_buckets": [64, 64],
    "num_hashes": 4,
    "lsh_attention_probs_dropout_prob": 0.0,
    "lsh_num_chunks_before": 1,
    "lsh_num_chunks_after": 0,
    "local_num_chunks_before": 1,
    "local_num_chunks_after": 0,
    "local_attention_probs_dropout_prob": 0.025,
    "hidden_dropout_prob": 0.025,
    "pad_token_id": tokenizer.pad_token_id,
    "eos_token_id": tokenizer.eos_token_id,
    "vocab_size": tokenizer.vocab_size,
}

reformer = ReformerModel(ReformerConfig(**enc_config))
input_ids = tokenizer.encode(doc, max_length=4096, pad_to_max_length=True, return_tensors='pt')
out = reformer.forward(input_ids)
out[0].shape
```
```
torch.Size([1, 4096, 1024])
```

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


🐛 [BART] Pipeline OOM

# 🐛 Bug

I try to run BART model myself versus running the model through `pipeline`.

Running the BART model myself is fine, but I have OOM on my GPU if I try to run the same model through pipeline.

Please see the following code : https://gist.github.com/Colanim/4fae6ab52c05716062a0f20c4a6b9737

_(It assume you have a file `cnndm/test.source` with an article on each line)_

Run with :
`python pipeline_oom.py --model HuggingFace --batch-size 32`
(Should **not** produce OOM on 11G-GPU)

and `python pipeline_oom.py --model Pipeline --batch-size 32`
(Should produce OOM on 11G-GPU)

---

**Why the pipeline use more memory ?**

@sshleifer


Updated path "cd examples/text-generation/pplm"


The purpose of files merges.txt, special_tokens_map.json, training_args.bin and add_tokens.json

Good evening!

After I have my RoBERTa model pre-trained, I get the list of the following files:
`merges.txt`, `special_tokens_map.json`, `training_args.bin`. I have also seen if you add extra tokens to the tokenizer, the file `add_tokens.json` appears. Could I ask to clarify the meaning of the first three files - how they are used and what they contain? And also how can I add extra tokens when pre-training RoBERTa or any BERT-type model? Million of thanks in advance!

Be safe,
Akim


Correcting path to pplm examples

# 🐛 Bug

On https://github.com/huggingface/transformers/tree/master/examples/text-generation/pplm#setup

It says

```
git clone https://github.com/huggingface/transformers && cd transformers
pip install .
pip install nltk torchtext # additional requirements.
cd examples/pplm

```

and as you can guess from the url, the correct path is

```
git clone https://github.com/huggingface/transformers && cd transformers
pip install .
pip install nltk torchtext # additional requirements.
cd examples/text-generation/pplm
```

cd examples/**text-generation**/pplm


Create model card for tblard/allocine

model card

Related to pretrained model cards

Model card for: https://huggingface.co/tblard/tf-allocine

This is a french sentiment analysis model, trained from camembert-base, finetuned on Allociné.fr data.

Original repo: https://github.com/TheophileBlard/french-sentiment-analysis-with-bert


Add .vs to gitignore

My VS code writes stuff there so added it to the .gitignore.

I also have other untracked files:
- in tests/fixtures/ I have some cached_lm_\*Tokenizer_\*.txt and .txt.lock after running the tests
- in docs/ I have the simlink to examples.md as indicated in [here](https://github.com/huggingface/transformers/tree/master/docs#building-the-documentation)

Should I add stuff in .gitignore to ignore those as well?


Don't access pad_token_id if there is no pad_token

When using the `encode`  method of a fast tokenizer, we end up here and accessing `pad_token_id` may log an error even when no padding is done. This PR corrects that.

This fixes #4764 .


Fix the __getattr__ method in BatchEncoding

Fix the issue where the `__getattr__` method in `BatchEncoding` was raising a `KeyError` instead of an `AttributeError` when the attribute was accessed with `getattr()`.

Example:
```
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")
features = tokenizer.encode_plus("Hello here")
getattr(features, "attr", False)
```

Previous output:
```
/home/jplu/transformers/src/transformers/tokenization_utils.py:204 __getattr__
        return self.data[item]
    KeyError: 'attr'
```

New output:
```
False
```


Remove unnecessary model_type arg in example


Add note about doc generation

Just make it explicit that doc generation is only for local inspection.


Bert (sentence classification) output is non-deterministic for PyTorch (not for TF)

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): German

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load model:
`config = BertConfig.from_json_file(config_filename)
model = BertForSequenceClassification(config)
state_dict = torch.load(model_filename)
model.load_state_dict(state_dict)`
2. Do inference twice on the same input + compare results.
3. Alternatively, save the first output, load the model from scratch, and run the same inference. Even in this case, the first output will not be the same as the next time.

## Expected behavior

The prediction value should be deterministic. Note that it *is* deterministic when the model parameters are loaded from a TensorFlow file (with `from_tf=True`).

## Environment info
- `transformers` version: 2.10.0
- Platform: Linux-5.3.0-55-generic-x86_64-with-Ubuntu-19.10-eoan
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Codecov setup

Setup codecov


Model is running on special characters and word pieces for token classification

['last', 'completed', 'interactions', 'on', 'wed', '##nes', '##day']
and it will return for 9 labels, but should really on return 5


Issue with HANS evaluation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Albert, Bert

Language I am using the model on (English, Chinese ...): GLUE (MNLI), HANS

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce
I have tried to evaluate ALBERT and BERT (pretrained, trained on MNLI) on HANS. I'm using the official example.

Everytime I run evaluation, I'm getting
```
Heuristic entailed results:
lexical_overlap: 0.0
subsequence: 0.0
constituent: 0.0

Heuristic non-entailed results:
lexical_overlap: 1.0
subsequence: 1.0
constituent: 1.0
```
Results can be reproduced by:
```
python3 hans/test_hans.py --task_name hans --model_type bert/albert --do_eval --data_dir $HANS_DIR --model_name_or_path $MODEL_PATH --max_seq_length 128 --output_dir $MODEL_PATH --per_gpu_eval_batch_size 1024 --overwrite_cache
```

## Expected behavior
It shouldn't be exactly 0 and 1 always and entailment score should be higher than non entailment.

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux 5.4.0-29-generic
- Python version: 3.8.2
- PyTorch version (GPU?): 1.5.0+cu101
- Tensorflow version (GPU?): NA
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Cannot load pretrained model from repo.

Hi, I'm trying to load a model from the repository:

```
tokenizer = AutoTokenizer.from_pretrained("NeuML/bert-small-cord19-squad2")
model = AutoModelForQuestionAnswering.from_pretrained("NeuML/bert-small-cord19-squad2")
```
but I'm receiving the error:
`OSError: Model name 'NeuML/bert-small-cord19-squad2' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed 'NeuML/bert-small-cord19-squad2' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
`

I've tried a few other models with the same result. The vocab.txt file is there in each of them. Am I missing something?

EDIT: Current libraries -
beautifulsoup4==4.9.1
boto3==1.10.30
botocore==1.13.30
certifi==2019.11.28
chardet==3.0.4
Click==7.0
docutils==0.15.2
fsspec==0.6.1
future==0.18.2
html2text==2020.1.16
idna==2.8
jmespath==0.9.4
joblib==0.14.0
nltk==3.4.5
numpy==1.17.4
pandas==0.25.3
Pillow==7.1.2
python-dateutil==2.8.0
pytz==2019.3
regex==2019.11.1
requests==2.22.0
s3fs==0.4.0
s3transfer==0.2.1
sacremoses==0.0.35
scikit-learn==0.21.3
scipy==1.3.2
sentencepiece==0.1.83
six==1.13.0
soupsieve==2.0.1
torch==1.5.0
torchvision==0.6.0
tqdm==4.40.0
transformers==2.2.0
urllib3==1.25.7
wikipedia==1.4.0


GPT2TokenizerFast raises pad_token error even if not used

# 🐛 Bug

## Information

Model: GPT2

Language: English

Encoding with the `GPT2TokenizerFast` causes a `pad_token` error to be sent to stderr, despite not attempting to access that property.

## To reproduce

```import transformers
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')

res = tokenizer.encode("This is a sentence")

print(transformers.__version__)
```

Output:
```
Using pad_token, but it is not set yet.
2.11.0
```

## Expected behavior

I'm aware that GPT-2 doesn't include a pad token (#2630) - I haven't tried to use it. I would expect no error to be displayed until I try to access that property.

## Environment info

- `transformers` version: 2.11.0
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 with CUDA
- Tensorflow version (GPU?): n/a
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Model Card for RoBERTa trained on Sanskrit

Related to pretrained model cards


KeyError in Pipeline Question Answering with LongFormer

I'm trying to do QA with LongFormer in a Pipeline. First of all, I generate the pipeline:
`
MODEL_STR = "mrm8488/longformer-base-4096-finetuned-squadv2"
tokenizer = AutoTokenizer.from_pretrained(MODEL_STR)
model = AutoModelForQuestionAnswering.from_pretrained(MODEL_STR)
QA = pipeline('question-answering', model=model, tokenizer=tokenizer)
`

Then, I get the paper text from which I want the answer to come from, named my_article, that's a string containing the full body of the article (around 3000 words). Then, I try:

`
with torch.no_grad():
    answer = QA(question=question, context=articles_abstract.body_text.iloc[0])
`

And it throws the following error:

`
eyError                                  Traceback (most recent call last)
<ipython-input-53-b5f8dc0503c8> in <module>
      1 with torch.no_grad():
----> 2     answer = QA(question=question, context=articles_abstract.body_text.iloc[0])

~/miniconda/envs/transformers_env/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
   1225                         ),
   1226                     }
-> 1227                     for s, e, score in zip(starts, ends, scores)
   1228                 ]
   1229

~/miniconda/envs/transformers_env/lib/python3.7/site-packages/transformers/pipelines.py in <listcomp>(.0)
   1225                         ),
   1226                     }
-> 1227                     for s, e, score in zip(starts, ends, scores)
   1228                 ]
   1229

KeyError: 382
`

How can I solve this issue? More importantly, what do you think is causing the issue?

Thanks in advance! :)


[cleanup] PretrainedModel.generate: remove unused kwargs

`_generate_beam_search` and `_generate_no_beam_search` do not use `bos_token_id` or `decoder_start_token_id`


Fine-tuning of RoBERTa

Good afternoon,
I'm trying to fine-tune RoBERTa on my own dataset, following the instruction provided here [https://huggingface.co/transformers/v1.2.0/examples.html](url). However, I cannot find the file `run_lm_finetuning.py` - could you please clarify, is the instruction valid or the file has really been deleted? Thanks in advance!

Be safe,
Akim


Fix resize_token_embeddings for Transformer-XL

Fixes #3554

As discussed in the issue above, the fix ensures that per default the last layer of the `AdaptiveEmbedding` is resized. Otherwise the target layer can be passed to the `resize_token_embeddings()` method as the parameter `layer`.

After the resizing is done, the cutoffs are adjusted accordingly.


run_tf_ner.py output_dir/saved_model empty

# ❓ Questions & Help

https://github.com/huggingface/transformers/tree/master/examples/token-classification

I followed the example here and saw the logging message:
` Saving model in model/saved_model`

Then I went to the folder saved_model and found it is empty.

Is this expected?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Add drop_last arg for data loader

Add an extra argument to `TrainingArguments` that would be passed on to `Trainer` for use in DataLoader.

I ran into a problem while using the `Trainer` this week and the GPU expecting the full batch size of vector inputs, and put a workaround in place in the dataset class I was using, but would be useful to have this as an optional argument.


[WIP] feat(wandb): add logging to TFTrainer

Bring logging feature parity from `Trainer` to `TFTrainer`.
Code has been refactored to share logging utilities.


run_ner.py crashes with RoBERTa because of incorrect sequence length

# 🐛 Bug

## Information

I'm running `examples/token-classification/run_ner.py` with RoBERTa. An assert statement fails:

```
assert len(input_ids) == max_seq_length
AssertionError
```

Looks like the cause is a mismatch between the value of roberta's `tokenizer.num_special_tokens_to_add()` and the number of special tokens that is actually added to the sequence in `utils_ner.py::convert_examples_to_features()`.

Specifically, `tokenizer.num_special_tokens_to_add()` is 2 (presumably for `<s>` and `</s>`). However, `convert_examples_to_features()` adds an extra `</s>` token at line 331, in addition to the `<s>` token and first `</s>` token. So the result is that there are three special tokens, and the sequence ends with `</s> </s>`.

`convert_examples_to_features()` relies on `num_special_tokens_to_add()` to determine how many content tokens from the sequence to use, but because of the mismatch above, you can end up with a sequence length of 129 even when the sequence length was set to a max of 128.

To reproduce this, just follow the instructions at `examples/token-classification/README.md` except use the flag `--model_name_or_path roberta-base`.

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

## Environment info
- `transformers` version: 2.11.0
- Platform: Linux-4.15.0-101-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: fails in both cases
- Using distributed or parallel set-up in script?: no


bart-large-cnn model weights updated?

Hi,

Today, I noticed that the bart model path has been changed to 'facebook/bart-large-cnn' rather than 'bart-large-cnn'. And I run the demo below, but the result changed since then. (seems worse)

https://colab.research.google.com/drive/11hKBPfsfBXPKo-dK_gHsPklF4PcNflQZ#scrollTo=dyTJ_ZavDp1q

So, is that weights updated?

Thanks,
Max


Can I use TorchText Iterator output as the input_ids for Hugging Face Transformer?

Hello,

For a given text, `TorchText` iterator, such as the `BPTTIterator`, returns the string of the text after each token is converted to their respective integer ID.

So for example, if the integer ids are assigned as the following manner:
"I" = 53, "like"=753, "dogs" = 2

Then for the string "I like dogs", a `TorchText` iterator would return `[53, 753, 2]`.

Is it okay to use this type of TorchText Iterator output directly as an `input_ids` for Hugging Face Transformers, providing that the Transformer models I use is not the Hugging Face pre-trained ones?

Thank you,


Batching not speeding up Transformer-XL

I have modified the example `[run_generation.py](https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py)` so that it can use batches. My code (pared down for the example) is below, called `batch_gen.py`:

```
#!/usr/bin/env python3
# coding=utf-8

import argparse
import logging

import numpy as np
import torch

from transformers import (
    GPT2LMHeadModel,
    GPT2Tokenizer,
    TransfoXLLMHeadModel,
    TransfoXLTokenizer,
)

logging.basicConfig(
    format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s", datefmt="%m/%d/%Y %H:%M:%S", level=logging.INFO,
)
logger = logging.getLogger(__name__)

MAX_LENGTH = int(10000)  # Hardcoded max length to avoid infinite loop
MODEL_CLASSES = {
    "gpt2": (GPT2LMHeadModel, GPT2Tokenizer),
    "transfo-xl": (TransfoXLLMHeadModel, TransfoXLTokenizer),
}

# Convert a list of prompts (strings) into batches (lists of strings,
# where each list is of size batch_size). The final batch might be
# smaller than batch_size
def batchify_prompts(prompt_list, batch_size):
    batches = []

    this_batch = []
    for prompt in prompt_list:
        this_batch.append(prompt)
        if len(this_batch) == batch_size:
            batches.append(this_batch[:])
            this_batch = []
    if len(this_batch) > 0:
        batches.append(this_batch)

    return batches

parser = argparse.ArgumentParser()
parser.add_argument("--model_type",default=None,type=str,required=True,help="Model type selected in the list: " + ", ".join(MODEL_CLASSES.keys()),)
parser.add_argument("--model_name_or_path",default=None,type=str,required=True,help="Path to pre-trained model or shortcut name selected in the list: " + ", ".join(MODEL_CLASSES.keys()),)
parser.add_argument("--length", type=int, default=20)
parser.add_argument("--prompt_file", type=str, default=None, help="File of prompts, 1 prompt per line.")
parser.add_argument("--batch_size", type=int, default=10, help="Number of prompts to include in a batch.")
args = parser.parse_args()

args.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
args.n_gpu = torch.cuda.device_count()

# Create file to print to
output_filename = "_".join([str(x) for x in [args.model_type, args.prompt_file.split("/")[-1]]]) + ".generated"
fo = open(output_filename, "w", encoding="utf-8")

args.model_type = args.model_type.lower()
model_class, tokenizer_class = MODEL_CLASSES[args.model_type]

tokenizer = tokenizer_class.from_pretrained(args.model_name_or_path)
model = model_class.from_pretrained(args.model_name_or_path)
model.to(args.device)

# Read in prompts from file
prompt_file = open(args.prompt_file, "r", encoding="utf-8")
prompt_list = []
for prompt_line in prompt_file:
    prompt_list.append(prompt_line);

prompt_batches = batchify_prompts(prompt_list, args.batch_size)

# Generate text for each prompt
for prompt_batch in prompt_batches:
    tokenizer.pad_token = "<PADDINGTOKEN>"
    tokenizer.padding_side = "left"
    encoding = tokenizer.batch_encode_plus(prompt_batch, add_special_tokens=False, return_tensors="pt", pad_to_max_length=True, add_space_before_punct_symbol=True)
    encoded_prompt = encoding["input_ids"]

    # Attention mask is not automatically returned by batch_encode_plus, so here we generate it manually
    attention_mask = 1 - (encoded_prompt == tokenizer.pad_token_id).type(torch.LongTensor)

    encoded_prompt = encoded_prompt.to(args.device)

    if encoded_prompt.size()[-1] == 0:
        input_ids = None
    else:
        input_ids = encoded_prompt

    output_sequences = model.generate(
        input_ids=input_ids,
        max_length=50 + len(encoded_prompt[0]),
        min_length=50 + len(encoded_prompt[0]),
        temperature=1.0,
        top_k=40,
        top_p=1,
        repetition_penalty=1.0,
        do_sample=True,
        num_return_sequences=1,
        attention_mask=attention_mask,
    )

    # Write the generations to the output file
    for generated_sequence_idx, generated_sequence in enumerate(output_sequences):
        fo.write("=== PROMPT ===\n")
        generated_sequence = generated_sequence.tolist()

        # Decode text
        text = tokenizer.decode(generated_sequence, clean_up_tokenization_spaces=True)

        # Add the prompt at the beginning of the sequence. Remove the excess text that was used for pre-processing
        generated_sequence = (
            text[len(tokenizer.decode(encoded_prompt[0], clean_up_tokenization_spaces=True)) :]
        )

        fo.write(prompt_batch[generated_sequence_idx] + "\n=== GENERATED ===\n")

        fo.write(generated_sequence + "\n\n")
```

To test the speedup provided by batching, I use a text file called `prompts.txt` with the following prompts:

```
The accompanying music video , directed by Vaughan Arnell ,
Inspired by the Beach Boys , cult surfing films ,
Premiering worldwide on Vevo on 7 January 2013 , the
The video features scenes reminiscent of the films South Pacific
The music video garnered 10 @.@ 4 million views in
Despite a 34 % gain in weekly activity to their
191 @,@ 000 Twitter followers added contributed to their overall
Rebecca <unk> of E ! Online praised its " intentionally
Molly Chance , writing for Zap2it , was convinced that
Mikael Wood , the critic for Los Angeles Times ,
It is said that when he died in Osaka during
A variety of styles have been used in efforts to
As Burton Watson remarks in The Selected Poems of Du
The translators have had to contend with bringing out
One extreme on each issue is represented by Kenneth Rexroth
His are free translations , which seek to conceal the <unk>
Other translators have placed much greater weight on trying to
Vikram Seth in Three Chinese Poets uses English @-@ style
In The Selected Poems of Du Fu , Burton Watson follows the
Traditional Chinese literary criticism emphasized the life of the author
Since many of Du Fu 's poems feature morality and
Another reason , identified by the Chinese historian William Hung
For modern Western readers , " The less accurately we
Stephen Owen suggests a third factor particular to Du Fu
Most of what is known of Du Fu 's life
His paternal grandfather was Du <unk> , a noted politician
Du Fu was born in 712 ; the exact birthplace
In later life , he considered himself to belong to
He also had three half brothers and one half sister
The son of a minor scholar @-@ official , his
```

The following command is used to run the code with GPT-2:

```
python batch_gen.py --model_type=gpt2 --model_name_or_path=gpt2 --prompt_file prompts.txt --batch_size 10
```

With GPT-2, batching speeds up the runtime as expected: Each batch takes approximately 1 second, regardless of whether the batch size is 1, 5, or 10. However, with Transformer-XL, this is not the case. Here is the command to run with Transformer-XL:

```
python batch_gen.py --model_type=transfo-xl --model_name_or_path=transfo-xl-wt103 --prompt_file prompts.txt --batch_size 1
```

With a batch size of 1, each batch takes 3 seconds. With a batch size of 5, each batch takes 12 seconds. With a batch size of 10, each batch takes 21 seconds. Thus, batching is not providing much of a speedup compared to generating examples serially. (You can see the amount of time each batch takes by looking at the time stamps on the log messages that are printed out).

Therefore, I am wondering if there is a bug in the batching for Transformer-XL? Or is there some reason why the architecture cannot support efficient batching?

I am running this code on a p100 GPU through Ubuntu version 18.04 with PyTorch version 1.5.0 and Python version 3.7.7.

Thank you!


Update encode documentation

closes #4750


Tokenizer.encode documentation not correct

# 🐛 Bug

## Information

Model I am using Bert (bert-base-german-cased):

Language I am using the model on German:

The problem arises when using:
* tokenizer of this model

The tasks I am working on is:
* encoding

## To reproduce

Steps to reproduce the behavior:

1. lang_model = 'bert-base-german-cased'
2. tokenizer = BertTokenizer.from_pretrained(lang_model)
3. test_sentence = 'Das war gut'
4. tokenizer.encode(test_sentence)
output: [3, 295, 185, 1522, 4]
5. tokenizer.convert_tokens_to_ids(tokenizer.tokenize(test_sentence))
output: [295, 185, 1522]

## Expected behavior

According to the documentation of the encoding method in

https://huggingface.co/transformers/main_classes/tokenizer.html

these two outputs should be the same. The problem is, that the _encode_ method adds special tokens [CLS] and [SEP] at the beginning and the end. However, the transformation in line 5. does not. This is not a problem at all but one should consider correcting the online-documentation. That is, the hint

**Same as doing self.convert_tokens_to_ids(self.tokenize(text))**

in the encode method is misleading. Instead, one could also add, that these two commands are the same if _add_special_tokens_ is set _False_ in the _encode_ method.

## Environment info

- `transformers` version: 2.8.0
- Platform: Windows 10
- Python version: 3.8.3
- PyTorch version (GPU?): 1.5.0+cpu
- Tensorflow version (GPU?): None
- Using GPU in script?:No
- Using distributed or parallel set-up in script?:No


Hugging Face GPT-2 Tokenizer

Hello,

I know that if I choose to add any new "special token" onto the pre-made GPT-2 tokenizer, and if I want to use the pre-trained GPT-2 model for my analysis, I will need to re-train the pre-trained GPT-2 to make the model learn that new special token.

But what if I just add an extra non-special token? for example, a word "paradox" is not included in the existing GPT-2 tokenizer, so say I add the word "paradox" to the existing set of GPT-2 vocabulary, like below:

```python
# load the pre-trained GPT2-tokenizer
gpt2_tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

# adding a new word (not special token) to the existing vocabulary,
# but I am not making any changes to the pre-assigned special tokens
gpt2_tokenizer.add_tokens("paradox")

# get the pre-trained HuggingFace GPT2DoubleHeadsModel
model_gpt2DoubleHeadsModel = GPT2DoubleHeadsModel.from_pretrained('gpt2', output_hidden_states = True)

# resize the token embeddings
# (not sure what this function does)
model_gpt2DoubleHeadsModel.resize_token_embeddings(len(gpt2_tokenizer))
```
Given that I didn't make any changes to the special tokens in the GPT-2-tokenizer, do I still need to train the already pre-trained `GPT2DoubleHeadsModel` before I start using it,  just because I added a new word to the set of vocabulary?

Thank you,


QuestionAnsweringPipeline query performance

This is my first issue posted here, so first off thank you for building this library, it's really pushing NLP forward.

The current [QuestionAnsweringPipeline](https://github.com/huggingface/transformers/blob/master/src/transformers/pipelines.py#L1187) relies on the method [squad_convert_examples_to_features](https://github.com/huggingface/transformers/blob/ed4df85572924871758ca32133b46116121c706f/src/transformers/data/processors/squad.py#L269) to convert question/context pairs to SquadFeatures. In reviewing this method, it looks like it spawns a process for each example.

This is causing performance issues when looking to support near real-time queries or bulk queries. As a workaround, I can directly issue the queries against the model but the pipeline has a lot of nice logic to help format answers properly and pulling the best answer vs start/end argmax.

Please see the results of a rudimentary performance test to demonstrate:

```python
import time

from transformers import pipeline

context = r"""
The extractive question answering process took an average of 36.555 seconds using pipelines and about 2 seconds when
queried directly using the models.
"""
question = "How long did the process take?"

nlp = pipeline("question-answering", model="distilbert-base-cased-distilled-squad", tokenizer="distilbert-base-cased-distilled-squad")

start = time.time()
for x in range(100):
    answer = nlp(question=question, context=context)

print("Answer", answer)
print("Time", time.time() - start, "s")
```
```
Answer {'score': 0.8029816785368773, 'start': 62, 'end': 76, 'answer': '36.555 seconds'}
Time 36.703474044799805 s
```
```python
import torch

from transformers import pipeline, AutoModelForQuestionAnswering, AutoTokenizer

model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")
tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")

start = time.time()
for x in range(100):
    inputs = tokenizer.encode_plus(question, context, add_special_tokens=True, return_tensors="pt")
    input_ids = inputs["input_ids"].tolist()[0]

    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
    answer_start_scores, answer_end_scores = model(**inputs)

    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score

    answer = tokenizer.convert_tokens_to_string(tokenizer.convert_ids_to_tokens(input_ids[answer_start:answer_end]))

print("Answer", answer)
print("Time", time.time() - start, "s")
```
```
Answer 36 . 555 seconds
Time 2.1718859672546387 s
```
I believe the 10x slowdown is that the first example had to spawn 100 processes. I also tried passing a list of 100 question/context pairs to see if that was better and that took ~28s. But for this use case, all 100 questions wouldn't be available at once.

The additional logic for answer extraction doesn't come for free but it doesn't add much overhead. The third test below uses a [custom pipeline component](https://github.com/neuml/cord19q/blob/master/src/python/cord19q/pipeline.py) to demonstrate.

```python
from cord19q.pipeline import Pipeline

pipeline = Pipeline("distilbert-base-cased-distilled-squad", False)

start = time.time()
for x in range(100):
    answer = pipeline([question], [context])

print("\nAnswer", answer)
print("Time", time.time() - start, "s")
```
```
Answer [{'answer': '36.555 seconds', 'score': 0.8029860216482803}]
Time 2.219379186630249 s
```
It would be great if the QuestionAnsweringPipeline could either not use the squad processor or the processor is changed to have an argument to not spawn processes.


No silent error when d_head already in the configuration

closes #4696


Why I can't generate phrases in batches if I include an attention mask? (GPT2)

Assuming these are my input phrases and model:

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<PAD>')

prompt_text = [
    "are there any good coaching institutes for civil services preparations in bangalore? ->"]
```

If I try to generate phrases in batches with the corresponding attention mask it doesn't work. It outputs the input phrase without any new words on it:

```
# encode plus batch handles multiple batches and automatically creates attention_masks
seq_len = 100
encodings_dict = tokenizer.batch_encode_plus(prompt_text, max_length=seq_len, pad_to_max_length=True)

input_ids = torch.tensor(encodings_dict['input_ids'])
attn_mask = torch.tensor(encodings_dict['attention_mask'])

encoded_result = model.generate(input_ids, attention_mask=attn_mask, eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id, num_return_sequences=10, top_k=50, top_p=0.95, do_sample=True, max_length=100)

for er in encoded_result:
    print(tokenizer.decode(er, skip_special_tokens=True))
```

However, if I generate phrases one by one (without batches) then it works:

```
encoded_text = tokenizer.encode(prompt_text[0], return_tensors='pt')
encoded_result = model.generate(encoded_text,eos_token_id=tokenizer.eos_token_id, pad_token_id=tokenizer.eos_token_id, num_return_sequences=10, top_k=50, top_p=0.95, do_sample=True, max_length=100)
print(tokenizer.decode(encoded_result[0], skip_special_tokens=True))
```

## Details
Any ideas what could be causing this problem?

Thanks!!


[Generation Beam Search] Fix bug when changing the <EOS> token for generate

This PR fixes https://github.com/huggingface/transformers/issues/4121 . When comparing int `!=` should be used here instead of `is not`.


How to use pretrained model for inference?

I have a bert model finetuned for pytorch and have trouble actually using it. Like how can I get to model("<some sentance>") outputting results for the tokens?


Create README.md

Related to pretrained model cards

Main issue is to refer to the fairseq website.


Perform evaluation on HANS with Trainer (like GLUE example)

Current [HANS](https://github.com/huggingface/transformers/tree/master/examples/adversarial) evaluation implementation is carried out in the old way. It'd be good to do it in the same manner as other examples are implemented now with the Trainer class.


Implemented resizing of token embeddings for TensorFlow models

As mentioned in [this issue](https://github.com/huggingface/transformers/issues/1838) transformers currently does not support resizing token embeddings with TensorFlow. I have implemented this functionality for ALBERT, BERT, DistilBERT, and GPT2.

**Note**
All of the respective TF[...]MainLayer[s] inherit from a utility class (TFLayerUtilsMixin; similar to the already existing TFModelUtilsMixin; both of them live in `modeling_tf_utils.py`) to avoid code duplication. This has to be done because TensorFlow models are structured differently than the corresponding PyTorch models - i.e., all the TF{ModelName} classes have a corresponding TF{ClassName}MainLayer which itself inherits from tf.keras.layers.Layer, whereas the PyTorch {ModelName} classes implement all the functionality themselves.

**Usage**
The usage is exactly the same as for the PyTorch models.
```
import transformers
bert = transformers.TFAutoModelForSequenceClassification.from_pretrained("bert-base-uncased")
tokenizer = transformers.AutoTokenizer.from_pretrained("bert-base-uncased")
tokenizer.add_tokens("[E1]"); tokenizer.add_tokens("[/E1]");
bert.resize_token_embeddings(len(tokenizer))
```

**Tests**
There are no tests in the existing transformers code for the `resize_token_embeddings` methods of the PyTorch models (as far as I can tell). This might be a separate issue.


Can't find config.json

# ❓ Questions & Help

Hello!When I use transformers I get this error Make sure that 'mrm8488/bert-multi-cased-finetuned-xquadv1' is the correct path to a directory containing a config.json file.How to solve it?


Extending run_language_modeling.py for XLNet

# 🚀 Feature request

The run_language_modeling.py script in examples/language-modeling/ currently works for BERT, RoBERTa, GPT and related models. It would be helpful if it also allowed XLNet. I believe this would involve writing functions to generate `perm_mask`, `target_mapping` and `labels` for the inputs sequences as per the paper and `https://github.com/zihangdai/xlnet/blob/master/data_utils.py`, but I'm not a 100% sure about this.

## Motivation

I have to adapt XLNet onto a specialized domain (finance) and am yet to find a decent guide or implementation which discusses how to do this. I've decided to do this myself (with the help of this library), and would like to share my code in case others find it useful.

## Your contribution

Since I will be trying to do this anyway, I would like to submit a relevant PR when it is done. I was also hoping to receive guidance or feedback as appropriate by other contributors to ensure correctness and utility.