Beam search sometimes fails this assert error

# 🐛 Bug

## Information

Model I am using:
GPT2 with custom config (vocab=27)

Language I am using the model on (English, Chinese ...):
Molecules... (see https://en.wikipedia.org/wiki/Simplified_molecular-input_line-entry_system)

The problem arises when using:
Using the .generate(beam=2) function

The tasks I am working on is:
Generating molecules

## The Problem
Essentially, every so

## To reproduce

Steps to reproduce the behavior:

Just do the generation with these args:

```args = { "num_beams": 3, "max_length": 50, "temperature": 1, "repetition_penalty": 1, "length_penalty": 1, "do_sample": true, "top_k": 50, "top_p": 1} ```

The generation runs fine for several batches, but then after like 100s of iterations, it sometimes bugs out with this error:

```File "/Users/laithani/anaconda3/envs/TransformerVAE/lib/python3.7/site-packages/transformers/modeling_utils.py", line 979, in _generate_beam_search
    assert len(next_batch_beam) == num_beams * (batch_ex + 1), f"{next_batch_beam}, {num_beams}, {batch_ex}"
```

I then added print statements to modeling_utils to try and see what is going on. I changed the assert line to:

```
assert len(next_batch_beam) == num_beams * (batch_ex + 1), f"{next_batch_beam}, {num_beams}, {batch_ex}"
```

And with this I got:
```
AssertionError: [(tensor(-19.8421), tensor(26), tensor(0)), (tensor(-20.9710), tensor(26), tensor(0)), (tensor(-30.5064), tensor(5), tensor(0)), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (0, 0, 0), (tensor(-17.4236), tensor(11), tensor(9)), (tensor(-26.3645), tensor(16), tensor(9)), (tensor(-23.9410), tensor(16), tensor(9)), (0, 0, 0), (0, 0, 0), (0, 0, 0), (tensor(-58.0648), tensor(0), tensor(15))], 3, 5
```

If you count up the length of the list, it is length 16, which =/= 3 * (5 + 1)

Not sure what is going on here, looking into the code now to try and figure out what is going on.

- `transformers` version: 2.4.1
- Platform: Ubuntu and Mac (problem occurs in both)
- Python version: 3.6 and 3.7 (problem occurs in both)
- PyTorch version (GPU?): 1.3.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: Yes, either V100 or K80
- Using distributed or parallel set-up in script?: No


Knowing the specific data set used for DistilBertForQuestionAnswering

Distillation

Related to model distillation

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi,
I am using the pipeline for question answering and am wondering what dataset was used to train the underlying model. The loaded model is DistilBertForQuestionAnswering. But I need to know if it used Squad 1.1 or Squad 2.0 which includes the possibility of no answer?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


CPU/GPU memory benchmarking utilities - Remove support for python 3.5 (now only 3.6+)

This PR add some utilities to benchmark (RAM) memory consumption of the models.
This is actually a generic utility that can work with any arbitrary python code

Ex:
```python
import torch
from transformers import GPT2Model, GPT2Tokenizer
from transformers import start_memory_tracing, stop_memory_tracing

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2Model.from_pretrained('gpt2')

sequence = tokenizer.encode("Hello how are you", return_tensors='pt')

# Line by line memory tracing (all code in the module `transformers`).
trace = start_memory_tracing(modules_to_trace="transformers")
output = model(sequence)
summary = stop_memory_tracing(trace)

# Summary contain three fields:
# `sequential`: list of line by line consumption (with line code and location)
# `cumulative`: list of cumulative line by line consumption (when lines are executed several times) ordered from the most memory consuming line to the least (also with line code and location)
# `total`: total memory consumption of the script (default to sum memory increase at each line and ignore released mem, can be seet to count increase and release by less reliable on ubuntu).
# Each `Memory` object contain CPU, GPU and CPU + GPU memory, each both in int and human readable string

print(f"Total memory consumption: {summary.total}")
top_line = summary.cumulative[0]
print(f"Consumed {top_line.cpu_gpu}: {top_line.frame.line_text} at {top_line.frame.filename}:{top_line.frame.line_number}")
```

Incorporated in the `./examples/benchmark.py` script. Example of command-line run:
``` bash
(py37) bash-3.2$ python ./examples/benchmarks.py --models gpt2 --torch --batch_sizes 1 --slice_sizes 64 256 512 512 512 --no_speed --verbose
Running with arguments Namespace(amp=False, average_over=30, batch_sizes=[1], csv_filename=None, fp16=False, keras_predict=False, models=['gpt2'], no_memory=False, no_speed=True, save_to_csv=False, slice_sizes=[64, 256, 512, 512, 512], tensorflow=False, torch=True, torch_cuda=False, torchscript=False, verbose=False, xla=False)
1 / 1
Token indices sequence length is longer than the specified maximum sequence length for this model (2708 > 1024). Running this sequence through the model will result in indexing errors
....
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:487: mem 0.000B:                 presents = presents + (present,)
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:489: mem 0.000B:             if self.output_attentions:
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:477: mem 0.000B:         for i, (block, layer_past) in enumerate(zip(self.h, past)):
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:492: mem 0.000B:         hidden_states = self.ln_f(hidden_states)
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:494: mem 0.000B:         hidden_states = hidden_states.view(*output_shape)
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:496: mem 0.000B:         if self.output_hidden_states:
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:499: mem 0.000B:         outputs = (hidden_states,)
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:500: mem 0.000B:         if self.output_past:
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:501: mem 0.000B:             outputs = outputs + (presents,)
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:502: mem 0.000B:         if self.output_hidden_states:
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:504: mem 0.000B:         if self.output_attentions:
/Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:509: mem 0.000B:         return outputs  # last hidden state, (presents), (all hidden_states), (attentions)

Top 5 script lines consuming the most memory:
0 => /Users/thomwolf/Documents/GitHub/transformers/src/transformers/activations.py:31: mem 276.004MB:     return 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
1 => /Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_utils.py:1311: mem 151.520MB:         x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight)
2 => /Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:146: mem 146.004MB:         w = w * b - 1e4 * (1 - b)
3 => /Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:143: mem 132.004MB:             w = w / math.sqrt(v.size(-1))
4 => /Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:187: mem 36.000MB:         present = torch.stack((key.transpose(-2, -1), value))  # transpose to have same shapes for stacking
5 => /Users/thomwolf/Documents/GitHub/transformers/src/transformers/modeling_gpt2.py:159: mem 33.000MB:         outputs = [torch.matmul(w, v)]

Memory increase computed by summing traced script lines: 843.758MB
=========== RESULTS ===========
        ======= MODEL CHECKPOINT: gpt2 =======
                ===== BATCH SIZE: 1 =====
                gpt2/1/64: N/A 75.176MB
                gpt2/1/256: N/A 349.695MB
                gpt2/1/512: N/A 843.758MB
                gpt2/1/512: N/A 843.758MB
                gpt2/1/512: N/A 843.758MB
```


Tokenizers v3.0.0


`Failed to build tokenizers` when installing 2.5.1 version.

`Failed to build tokenizers` when i try to install transformers==2.5.1

`Failed to build tokenizers` when i try to install tokenizers == 0.5.2

so, i want to know `tokenizers == 0.5.2` is must ?

thanks~


About the examples document of bert with SQuAD 2.0

I'm wondering if there is a mistake.
In the README document of /examples, the training parameters setting for training on SQuAD dataset, the first code block:
```
export SQUAD_DIR=/path/to/SQUAD

python run_squad.py \
  --model_type bert \
  --model_name_or_path bert-base-cased \
  --do_train \
  --do_eval \
  --do_lower_case \
  --train_file $SQUAD_DIR/train-v1.1.json \
  --predict_file $SQUAD_DIR/dev-v1.1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /tmp/debug_squad/
```
Because this model is bert-base-cased, which means it's cased, I think the ``--do_lower_case`` shouldn't be here. Is that a mistake?


How can i use pipeline with pretrained automodel ?

how can i use pipeline with pretrained auto model and tokenizer ?


The implementation of GPT2 masked attention mechanism will cause errors when the model was trained after some iterations.

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information
Model I am using (GPT2):
Language I am using the model on ( Chinese ...):

File in modeling_gpt2.py：
146        w = w * b - 1e4 * (1 - b)  # here the bias "-1e4" is too big, suggest replace it to "-1e10"

The item value of "query * key" may be close to or lower to "-1e4" ,  then after the softmax operation, the weights of masked attention may be attend to "unseen" context. This will cause inference errors when to evaluate the model.


NER - pl example

1. Solves most of the issues raised in #3159
2. Streamlines shell script pipeline
3. pl logs related changes
4. added in Readme


I can not import transformers

# 🐛 Bug

## Information
when I execute "from transformers import TFBertModel, BertModel" in ipython, the error, "ImportError: cannot import name 'BartConfig' from 'transformers.configuration_auto'" was raised.
This error occured after the tensorflow update from version 2.0 to version 2.1 and the python update from version 3.6 to 3.7.

In addition, after I update the python from version 3.6 to 3.7, I installed torch1.4 and tensorflow2.1 in a same env, the "import" is always failed but when I "import transformers" in other env which only include torch1.4 and python3.7, it succeed
I want to know how to make it. Thank you.

- `transformers` version:2.5.1
- Platform: windows10
- Python version: 3.7.6
- PyTorch version (GPU?):GPU1.4
- Tensorflow version (GPU?):GPU2.1


Pretraining QA corpora from scratch with sentence pairs

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I would like to pretrain a non-factoid QA corpora (question and answer passages) from scratch using BERT. I have looked at both:

https://huggingface.co/blog/how-to-train and
https://gist.github.com/aditya-malte/2d4f896f471be9c38eb4d723a710768b#file-smallberta_pretraining-ipynb

I would like to confirm that what I am doing is correct:

1. I concatenated the question and answers with a [SEP] token, so each line in my input data file looks like:
question [SEP] answer
2. I am running the script with --line_by_line

I am uncertain if this is correct because how would the script know which is sentence A and which is sentence B, or is this not necessary?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Distilgpt2 finetuning and text generation

This ipynb notebook contains a finetuning and text generation tutorial for distilgpt2. The tutorial also have used code from run_generation.py file to make generation faster than using the original file for every iteration.


GLUE test set predictions

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation
The `run_glue` script is super helpful.  But it currently doesn't implement producing predictions on the test datasets for the GLUE tasks.  I think this would be extremely helpful for a lot of people.  I'm sure there are plenty of people who have implemented this functionality themselves, but I haven't found any.  Since `transformers` already provides train and dev for GLUE, it would be cool to complete the feature set with providing test set predictions.

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

I'm personally working on a branch that extends the `glue_processors` to support the test sets (which are already downloaded by the recommended `download_glue.py` script.  I also update the `run_glue.py` script to produce the `*.tsv` files required by the GLUE online submission interface.

I think I'm a couple days out from testing/completing my implementation.  I'm also sure plenty of implementations exist of this.  If there are no other plans to support this in the works, I'm happy to submit a PR.

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Updated `Tokenw   ise` in print statement to `Token wise`


How can I assign a specific gpu when using examples/run_language_modeling.py?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello, I'm wondering if I can assign a specific gpu when using examples/run_language_modeling.py to train a language model?
Lots of thanks!


Get the CNN/Daily Mail Data for BART

@sshleifer
In the README for the BART summarization it says:

> download both CNN and Daily Mail datasets from Kyunghyun Cho's website
`tar -xvf cnn_stories.tgz && tar -xvf dailymail_stories.tgz`
> this should make a directory called cnn_dm/ with files like test.source. To use your own data, copy that files format. Each article to be summarized is on its own line.

This doesn't produce a cnn_dm directory, it produces two different folders. The contents of the folders are `text.story` files, not `test.source`.

Did you use [this repo](https://github.com/artmatsak/cnn-dailymail)?
Or did you get the data from somewhere else?

Happy to submit a PR either way!
Thanks!


Quick tour TF 2.0

# 🐛 Bug

## Information

I am trying **Quick tour TF 2.0**.

The problem arises when using a quick example: **How a TensorFlow 2.0 model can be trained in 12 lines of code**:

```python
import tensorflow as tf
import tensorflow_datasets
from transformers import *

# Load dataset, tokenizer, model from pretrained model/vocabulary
tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
data = tensorflow_datasets.load('glue/mrpc')

# Prepare dataset for GLUE as a tf.data.Dataset instance
train_dataset = glue_convert_examples_to_features(data['train'], tokenizer, max_length=128, task='mrpc')
valid_dataset = glue_convert_examples_to_features(data['validation'], tokenizer, max_length=128, task='mrpc')
train_dataset = train_dataset.shuffle(100).batch(32).repeat(2)
valid_dataset = valid_dataset.batch(64)

# Prepare training: Compile tf.keras model with optimizer, loss and learning rate schedule
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
model.compile(optimizer=optimizer, loss=loss, metrics=[metric])

# Train and evaluate using tf.keras.Model.fit()
history = model.fit(train_dataset, epochs=2, steps_per_epoch=115,
                    validation_data=valid_dataset, validation_steps=7)
```
which produces the output below:
```python
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-ea94e17f2f79> in <module>()
      1 tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
----> 2 model = TFBertForSequenceClassification.from_pretrained('bert-base-cased')
      3 data = tensorflow_datasets.load('glue/mrpc')

NameError: name 'TFBertForSequenceClassification' is not defined
```

The above behavior can be reproduced using this [Colab ](https://colab.research.google.com/drive/1aAmOVlvkuP9PLOuGKVx7-k0vsVBoD506)notebook.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Any help will be much appreciated.


Do we have a whole-word-masked version of BERT?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Thanks for the excellent work! But I'm wonder is there a whole-word-masked version of BERT? Moreover, how can I adapt the Tokenizer class to make it support other parsing method (e.g. In Chinese the BERTTokenizer simply parses the sequences in character level, while parser like jieba can parse them into Chinese words. How can I keep the features of Tokenizer class while using other parsing method?)?


Semantic Code Retrieval using Transformers

I am entering the world of transformers and would like to use some architectures to create a semantic search engine to retrieve source code (Python, Javascript, Ruby, Go, Java, and PHP code).

Currently, the [dataset ](https://github.com/github/CodeSearchNet#data-details)contains 2 million pairs **(code, docstring)**, where code is a list of tokens from a method or function and docstring is a short description of the code in natural language.

As a starting point, it would be interesting to construct a model architecture that receives the code and the docstring **([ [code], [docstring] ])** as input example and outputs the code embedding and docstring embedding. Using cosine similarity as loss function the model could be fine-tuned to encode both code and docstring to the same embedding space. As shown in the figure below:

<pre>            <img src="https://i.stack.imgur.com/4fx3h.png" width="480">
</pre>

I started reading and tokenizing the dataset:

```python
    from transformers import BertTokenizer
    # reads a list of [[code], [docstring]]
    reader = CodeDocstringReader(dataset_path)

    # loads tokenizer
    model_name = "bert-base-uncased"
    tokenizer = BertTokenizer.from_pretrained(model_name, do_lower_case=True)

    # returns a list of tokenized examples
    # [[code_tokes_ids], [docstring_tokens_ids]]
    tokenized_features = tokenizer_examples(
        reader.get_examples(),
        tokenizer
    )
```

The definition and training of the model are still incomplete, but it is outlined as:

```python
import tensorflow as tf
from transformers import BertModel

class JointEncoder(tf.keras.Model):
    """Encodes the code and docstring into an same space of embeddings."""

    def __init__(self,
                 path,
                 name="jointencoder"):
        super(JointEncoder, self).__init__(name=name)
        self.bert = BertModel.from_pretrained(path)

    def call(self, inputs):
        """Returns code and docstring embeddings"""
         ...
         code_embedding = ..
         docstring_embedding = ..

        return code_embedding, docstring_embedding

```

However, I'm stuck on how to code this simple architecture. Could you give me some directions?

Thanks in advance.


issues while modifying modeling_roberta.py file

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
I need the CLS representation from RobertaForMultipleChoice

For this, I changed
 outputs = (reshaped_logits,) + outputs[2:]
To
 outputs = (reshaped_logits,) + outputs[1:]
in file https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_roberta.py
since outputs[1] gives the CLS representation.

Now to incorporate this change
I need to import RobertaForMultipleChoice from this file  and replace "from transformers import ..." in https://github.com/huggingface/transformers/blob/master/examples/run_multiple_choice.py

to "from modeling_roberta.py import ..."

I am getting import issues
![image](https://user-images.githubusercontent.com/19836137/76137311-8d937200-5ff8-11ea-98fc-4d361eee0b32.png)

Can somebody help in resolving this?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Can we use GPT-2 sentence embedding for classification tasks?

Discussion

Discussion on a topic (keep it focused or open a new issue though)

I am experimenting on the use of transformer embeddings in sentence classification tasks **without finetuning them**. I have used BERT embeddings and those experiments gave me very good results. Now I want to use GPT-2 embeddings (without fine-tuning). So I have two questions,

1. Can I use GPT-2 embeddings like that (because I know Gpt-2 is trained on the left to right)
2. Is there any example uses of GPT-2 in classification tasks other than generation tasks?
3.  If I can use GPT-2 embeddings, how should I do it ?


padding and attention mask does not work as intended in batch input in GPT2 language model

The following code is without batch:
```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()
context=torch.tensor([tokenizer.encode("This is")])
output, past = model(context)
token = torch.argmax(output[..., -1, :])
print(tokenizer.decode(token.item()))

output: ' a'
```
Now, I extended this to batch setting:
```

from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is ")),torch.tensor(tokenizer.encode("Hello How are "))]
context=pad_sequence(context,batch_first=True)

mask=torch.tensor([[1,1,0],[1,1,1]])
output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: '\n you'
```
Here `\n` is next token for the first context and `you` is next token for second context of the batch.
But The expected next token for the first context is "a", since all the setting are same. Futhermore, if you reduce the second context to 2 token you will get `'a'` in this batch setting. So clearly, model can not understand the padding.
Also, **the attention mask does not work**. Because,
after padding the next token of sequence "`this is`" is  0 (zero). And according to the attention mask (`[1,1,0]`), this zero should be avoided and only the tokens `this` and `is` should be attended. The proofs that this attention masking is not working are:

- Use attention mask [1,1,1], that means attend even on the padding zero, you get the same output
which is  `\n'.

-  Use the the string `this is!`.  Here `!` has the zero index in the vocabulary matrix. Again you get  the same output which is `\n'.

Only time, it is possible to get desirable output without the batch settings and attention mask ( now it seems, it does not matter because it has no effect anyway)

Then I found [this](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.pad_token), which suggest to use `pad_token`.  So I used like following:

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch
from torch.nn.utils.rnn import pad_sequence

tokenizer = GPT2Tokenizer.from_pretrained("gpt2",pad_token="<PAD>")
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

context=[torch.tensor(tokenizer.encode("This is <PAD> ")),torch.tensor(tokenizer.encode("Hello How are"))]
context=torch.stack(context)
print(context)
mask=torch.tensor([[1,1,0],[1,1,1]])

output, past = model(context,attention_mask=mask)
token = torch.argmax(output[..., -1, :],dim=1)
tokenizer.decode(token)

output: 'The you'

```
Here `The` is next token for the first context and `you` is next token for second context of the batch.  This is also not working. Because `The` is not expected for the first context.

How do I use variable length sequence in batch setting in gpt/gpt2 model?


[BERT] Implementation of the sliding window for long sequences

I was trying to look for the references where the sliding window is implemented to process long sequences. How do we split a long sequence and then after getting the embeddings, how do we unpack them? I am unable to find the code segments that handle these operations.

Also, is it possible to describe the main trick? I am trying to implement it in plain PyTorch. I am unable to implement in Batches without running any loops.

Any help would be appreciated.


Reason for speedup

I used to develop a code based on the pytorch_transformers package (v1.1) and now transfering to the current version. I see a 2x speedup in Bert run_glue.py. Wondering what is the major reason? Looking at the code, I couldn't find major differences.


wrong configuration of ALBERT xlarge

# 🐛 Bug

## Information
If you are **reproducing** the  [ALBERT](https://github.com/google-research/albert), the `num_attention_heads` of albert xlarge should be 32 instead of 16 as
https://github.com/huggingface/transformers/blob/db9279dedbb9c5e7d24569a1ac3f74f9d5c3eb18/src/transformers/configuration_albert.py#L27


Inference is slow with

Hi @julien-c,
I know that @loretoparisi already talked with you guys about some problems we have with inference using BertTokenizer and BertForSequenceClassification Classes.
So for starting to discuss about that, we would like to show you the most important lines of our inference and training code, that probably can have some bugs and slowdowns.

Briefly, we are developing a language classifier trained on 93 languages, so we started from multilingual bert ( both tokenizer and model ) and then we want to finetune it on our dataset that has about 8M lines.

# INFERENCE CODE
```
# tokenize entire text
    tokens = tokenizer.encode(original_text.strip())
    # remove bos and eos tokens
    tokens = tokens[1:-1]

    # get number of slices we have to insert in DL model
    number_of_slices = len(tokens) // (MAX_SEQUENCE_LENGTH - 2)
    if len(tokens) % (MAX_SEQUENCE_LENGTH - 2) != 0: number_of_slices +=1

    # create slices to be inserted
    slices = []
    for index in range(number_of_slices):
        slice_ = tokens[ index*(MAX_SEQUENCE_LENGTH - 2) : (index+1)*(MAX_SEQUENCE_LENGTH - 2)]
        slice_ = [tokenizer.bos_token_id] + slice_ + [tokenizer.eos_token_id]
        slices.append(slice_)

    # for every slice, preprocess data creating mask and padding
    texts = []
    masks = []

    for text in slices:
        padding = [tokenizer.pad_token_id] * (MAX_SEQUENCE_LENGTH - len(text))
        mask = torch.zeros(MAX_SEQUENCE_LENGTH, dtype=torch.int32).tolist()
        mask[:len(text)] = [1]*len(text)
        text = text + padding
        texts.append(text)
        masks.append(mask)

    # texts to tensor pytorch
    texts = torch.tensor(texts)
    # masks to tensor pytorch
    masks = torch.tensor(masks)

    #inference from DL model
    logits = model(texts, attention_mask=masks)

    # stack logits
    logits = torch.stack(logits).mean(dim=0)
    #sum logits in order to have a single logits
    logits = torch.sum(logits, dim=0)
```

# TRAINING CODE
```
tokenizer = BertTokenizer.from_pretrained( os.path.join(data_path, 'model') )

    special_tokens_dict = {'eos_token': '[CLS]',
                            'unk_token' : '[UNK]',
                            'eos_token' : '[SEP]',
                            'bos_token' : '[CLS]',
                            'pad_token' : '[PAD]' }

    num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)

    train_loader, validation_loader = load_datasets(data_path,
                                                    tokenizer,
                                                    batch_size,
                                                    max_sequence_length,
                                                    random_sequence_length,
                                                    epoch_size,
                                                    token_dropout,
                                                    seed)

    model = BertForSequenceClassification.from_pretrained(os.path.join(data_path, 'model'))

    print("Num_labels:")
    print(model.num_labels)

    if torch.cuda.is_available():
        model.cuda()

    if rank == 0:
        summary(model)
        if distributed():
            dist.barrier()

    if world_size > 1:
        model = DistributedDataParallel(model, [rank], output_device=rank, find_unused_parameters=True)

    optimizer = Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)
    epoch_loop = count(1) if max_epochs is None else range(1, max_epochs + 1)

    logdir = os.environ.get("LOGDIR", "logs")
    os.makedirs(logdir, exist_ok=True)

    from torch.utils.tensorboard import SummaryWriter
    writer = SummaryWriter(logdir) if rank == 0 else None
    best_validation_accuracy = 0

    for epoch in epoch_loop:

        try:

            if world_size > 1:
                train_loader.sampler.set_epoch(epoch)
                validation_loader.sampler.set_epoch(epoch)

            train_metrics = train(model, optimizer, device, train_loader, f'Epoch {epoch}')
            validation_metrics = validate(model, device, validation_loader)
```

Do you have some suggestion about that our inference is slow and loading model is very long?
We do not use hub actually, we pre download model on local disk and we load with local path.
Thanks


Does this project have this function ?

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

can we use this project to calculate the probability that a input text as a real/resonable sentence base on the corpus we trained


urgent - ROBERTA on WSC

Hi
I am looking for how to run ROBERTA on WSC data, similar to the example in fairseq:
https://github.com/pytorch/fairseq/tree/master/examples/roberta

However, fairseq is hard to use and modify and I really appreciate if you could add this to your great repo
thanks


[Bart] add imports to examples


NER: some issues in PyTorch Lightning example

Hi,

I wanted to try out the new NER example script (`./ner/run_pl_ner.py`) that uses PyTorch Lightning.

Here are some bugs that I've found:

Dataset preparation method is not called. Usually, InputBatch batches or input features are written and stored in a file. However, the `prepare_data()` [1] method is not called and no input features are written. I fixed that adding this method to the `train_dataloader()` [2] function, but I'm not sure if it's the right place.

Model training will work then.

Evaluation is currently not working correctly. The checkpoint output file name is:

```bash
# ls
'checkpointepoch=0.ckpt'  'checkpointepoch=1.ckpt'  'checkpointepoch=2.ckpt'
```

so the pattern `checkpointepoch=<number_epoch>.ckpt` is used, whereas the main script expects an output checkpoint pattern of `checkpoint_<number_epoch>.ckpt` [3]

[1] https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py#L56-L80
[2] https://github.com/huggingface/transformers/blob/master/examples/ner/transformer_base.py#L126-L139
[3] https://github.com/huggingface/transformers/blob/master/examples/ner/run_pl_ner.py#L220


[Bart] _prepare_decoder_inputs should use large negative

Also renames some things and adds a nice test.
I suspect that this didn't break integration tests because we don't have a serious integration test with decoder_input_ids set (e.g. calculating loss for a summarization example)


[model_cards]Add albert chinese model

model card

Related to pretrained model cards

Hi,

this PR adds the model card for albert-chinese model
- albert_chinese_tiny
- albert_chinese_small
- albert_chinese_base
- albert_chinese_large
- albert_chinese_xlarge
- albert_chinese_xxlarge


Partially fix space only input without special tokens added int the output

Core: Tokenization

Internals of the library; Tokenization.

Original issue #3091.

It fixes the issue for all non BPE-based tokenizers. For BPE ones, the output is different from Python and Rust:

GPT2:
- Python : `[]`
- Rust: `['Ġ']`

Roberta:
- Python: `[]`
- Rust: `['<s>', 'Ġ', '</s>']`

Rust seems the right one here. I should have a look at Roberta why it include the special_tokens even if not asked to do so.

cc @n1t0 cc @LysandreJik fyi.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


i want to browse and store image in data base in tkinter can u give me suggessions?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


seed parameter for model generate()

# 🚀 Feature request

There should be a `seed` parameter for the `generate()` function of a model.

Although a seed can be manually set before calling `generate()` (as tested in #3063), using it as a parameter is more intuitive (and covers all the bases)

## Motivation

Generation reproducibility (also good for CI tests)

## Your contribution

The implementation `set_seed()` functions around the repo (e.g. https://github.com/huggingface/transformers/blob/6b1ff250842f52136d5159bb67a26b50ba01485d/examples/run_generation.py#L74) should be sufficient.


Fresh macOS install errors out on import

# 🐛 Bug
Fresh Install errors out on import

## Information
Following from the README instructions, created a venv, installed torch and tensorflow. Installed transformers. Upon import

Model I am using (Bert, XLNet ...):
transformers==2.5.1

Language I am using the model on (English, Chinese ...):
N/A
The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create a new virtual env ```$ virtualenv -p python3 venv```
2. Source your new venv ```$ source venv/usr/local/bin/activate```
3. Pip install packages ``` (venv) $ pip install torch tensorflow transformers```
4. import package ``` python -m transformers```

```python

>>> import transformers
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 183, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 142, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 109, in _get_module_details
    __import__(pkg_name)
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/transformers/__init__.py", line 22, in <module>
    from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/transformers/configuration_albert.py", line 18, in <module>
    from .configuration_utils import PretrainedConfig
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/transformers/configuration_utils.py", line 25, in <module>
    from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_url
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/transformers/file_utils.py", line 53, in <module>
    import tensorflow as tf
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/tensorflow/__init__.py", line 101, in <module>
    from tensorflow_core import *
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/tensorflow_core/__init__.py", line 40, in <module>
    from tensorflow.python.tools import module_util as _module_util
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 959, in _find_and_load_unlocked
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/tensorflow/__init__.py", line 50, in __getattr__
    module = self._load()
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/tensorflow/__init__.py", line 44, in _load
    module = _importlib.import_module(self.__name__)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/tensorflow_core/python/__init__.py", line 64, in <module>
    from tensorflow.core.framework.graph_pb2 import *
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/tensorflow_core/core/framework/graph_pb2.py", line 7, in <module>
    from google.protobuf import descriptor as _descriptor
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/google/protobuf/__init__.py", line 37, in <module>
    __import__('pkg_resources').declare_namespace(__name__)
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/__init__.py", line 84, in <module>
    __import__('pkg_resources.extern.packaging.requirements')
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 9, in <module>
    from pkg_resources.extern.pyparsing import stringStart, stringEnd, originalTextFor, ParseException
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/extern/__init__.py", line 43, in load_module
    __import__(extant)
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/_vendor/pyparsing.py", line 4756, in <module>
    _escapedPunc = Word( _bslash, r"\[]-*.$+^?()~ ", exact=2 ).setParseAction(lambda s,l,t:t[0][1])
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/_vendor/pyparsing.py", line 1284, in setParseAction
    self.parseAction = list(map(_trim_arity, list(fns)))
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/_vendor/pyparsing.py", line 1066, in _trim_arity
    this_line = extract_stack(limit=2)[-1]
  File "/Users/kiran/Library/Python/3.7/lib/python/site-packages/pkg_resources/_vendor/pyparsing.py", line 1050, in extract_stack
    frame_summary = traceback.extract_stack(limit=-offset+limit-1)[offset]
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py", line 211, in extract_stack
    stack = StackSummary.extract(walk_stack(f), limit=limit)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py", line 363, in extract
    f.line
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/traceback.py", line 285, in line
    self._line = linecache.getline(self.filename, self.lineno).strip()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py", line 16, in getline
    lines = getlines(filename, module_globals)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/linecache.py", line 48, in getlines
    for mod in sys.modules.values():
RuntimeError: dictionary changed size during iteration
```

## Expected behavior

Importing transformers into the python runtime should import transformers.

- `transformers` version: 2.5.1
- Platform: macOS Catalina 10.15.3
- Python version: Python 3.7.3
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A


BART.generate: possible to reduce time/memory?

# 🐛 Performance issues

I did a quick benchmark between HuggingFace's implementation of **BART** and FairSeq's implementation.

You can find the benchmark code [here](https://gist.github.com/Colanim/cc418d19e5e107f462bac306f53ba994).

---

Here is my results, on a single GPU GTX 1080 (12 GiB of memory) :

| FP16 - Batch size 16 | s/batch | s/sample |
|----------------------|---------|----------|
| FairSeq              | 8.8676  | 0.5664   |
| HuggingFace          | 12.3358 | 0.7879   |

| FP16 - Batch size 32 | s/batch | s/sample |
|----------------------|---------|----------|
| FairSeq              | 17.1247 | 0.5469   |
| HuggingFace          | OOM     | OOM      |

| FP16 - Batch size 1 | s/sample |
|---------------------|----------|
| FairSeq             | 1.6743   |
| HuggingFace         | 1.8856   |

| FP32 - Batch size 1 | s/sample |
|---------------------|----------|
| FairSeq             | 1.7865   |
| HuggingFace         | 2.0670   |

---

**FairSeq is consistently faster than HuggingFace on all my experiments.**

---

This sparks a few questions :

* Do you have similar results on your side ? Did I mess my benchmark ?
* Why HuggingFace's implementation is significantly slower ?
* Why HuggingFace's implementation takes more space in memory (illustrated by `OOM` with batch size of 32) ?
* Is the release of the `Summarization Pipeline` going to improve this ?

@sshleifer


How to train a distilled gpt2

Hi! Thanks for everything.
I'm interested in training a distilled version of gpt2, because I want it to be even smaller than the distilled gpt2 model.
In https://github.com/huggingface/transformers/tree/master/examples/distillation , there is a tutorial about how to get a distilled bert. Could you give instructions about how to train a distilled gpt2?
Thanks!


Padding changes model outputs (even with attention_mask)

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Consider the following example code which gets a BERT embedding, and compares the BERT embedding on the same input with padding + masking.
```
bertModel=BertModel.from_pretrained('bert-base-uncased')
bertModel=bertModel.eval()
print('token id:'+str(id))
print('token types:'+str(typeout))

bert_out = torch.mean(bertModel(id, token_type_ids = typeout, attention_mask = mask)[0],1)

add_pad = lambda x: torch.cat((x, torch.zeros(1, 10, dtype=x.dtype)),1)
print('mask:'+str(mask))
print('padded id:'+str(add_pad(id)))
print('padded type:'+str(add_pad(typeout)))
print('padded mask:'+str(add_pad(mask)))
bert_out_2 = torch.mean(bertModel(add_pad(id), token_type_ids = add_pad(typeout), attention_mask = add_pad(mask))[0],1)

print(bert_out[0,0:10])
print(bert_out_2[0,0:10])
```
The output here is
```
token id:tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102]])
token types:tensor([[0, 0, 0, 0, 1, 1, 1]])
mask:tensor([[1., 1., 1., 1., 1., 1., 1.]])
padded id:tensor([[ 101, 5292, 3270,  102, 8638, 2060,  102,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0]])
padded type:tensor([[0, 0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])
padded mask:tensor([[1., 1., 1., 1., 1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]])
tensor([ 0.2600,  0.1568, -0.2576, -0.0110,  0.1021, -0.1336,  0.2546, -0.5071,
         0.0462, -0.2283], grad_fn=<SliceBackward>)
tensor([ 0.0996,  0.0061, -0.3331, -0.0237, -0.1110, -0.0050,  0.2755, -0.3335,
        -0.0565, -0.2542], grad_fn=<SliceBackward>)
```
The two tensors which were generated by the same input but have different padding have drastically different embeddings.

## Expected behavior

The last two lines of the output should be the same
## Environment info

- `transformers` version: 2.1.1.
- Platform: osx
- Python version: 3.7.3
- PyTorch version (GPU?): 1.2.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


fix missed BartForMaskedLM renaming

Quick fix @sshleifer


refactored beam search according to torch implementation

1:1 translation of PR #3135 from Pytorch to TF.


Pass kwargs to configuration

**kwargs were not passed to the PreTrainedConfiguration when using `from_pretrained`

closes #3093


Create README.md for mrm8488/bert-multi-uncased-finetuned-xquadv1

model card

Related to pretrained model cards


[Bart] FP16 Support


[Question]: Why does model.__call__ return the loss too?

Core: Modeling

Internals of the library; Models.

# ❓ Questions & Help

In PyTorch, model.__call__ returns the output tensor and users have to call a loss function to get the loss. I wonder why models in transformers doesn't follow this convention? Any specific reason?


Correct missing keys

closes #3142

The problem came from the fact that `from_pretrained` set the model on which to load the weights to the base model: it detected that the state dict was made for the base, therefore loading only onto the base.

It didn't look for the weights it didn't load. Here, both state dicts are analyzed and the difference (keys present in the state dict with the head and not in the base state dict) are added to the missing keys.

Added a test.


Missing `missing_keys` when loading from saved base model checkpoint

Internals of the library; Models.

# 🐛 Bug

## Information

If a base model (e.g. `BertModel`, `DistilBertModel`, ...) is saved using `save_pretrained` and a model with an additional head (e.g. `BertForSequenceClassification`, `DistilBertForQuestionAnswering`, ...) is loaded from that checkpoint, it will not detect that it is missing layers.

## To reproduce

Steps to reproduce the behavior:

1. Instantiate base model from configuration or from `from_pretrained`
2. Save model using `save_pretrained`
3. Load checkpoint in model with head
4. No warning is output. Furthermore, if `output_loading_info=True` in step 3), will output `{'missing_keys': [], 'unexpected_keys': [], 'error_msgs': []}`

Here's a reproducible example:

```py
from transformers import BertForSequenceClassification, BertModel, BertConfig

config = BertConfig()
base_model = BertModel(config)
base_model.save_pretrained(directory)

model, loading_info = BertForSequenceClassification.from_pretrained(directory, output_loading_info=True)
print(loading_info)

# {'missing_keys': [], 'unexpected_keys': [], 'error_msgs': []}
# Should output {'missing_keys': ['classifier.weight', 'classifier.bias'], 'unexpected_keys': [], 'error_msgs': []}

```

## Expected behavior

Should detect the missing keys, as it does when loading from a full checkpoint:

```py
from transformers import BertForSequenceClassification

model, loading_info = BertForSequenceClassification.from_pretrained("bert-base-cased", output_loading_info=True)
print(loading_info)

# {'missing_keys': ['classifier.weight', 'classifier.bias'], 'unexpected_keys': ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias'], 'error_msgs': []}

```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: master branch
- Platform: Linux-5.5.7-arch1-1-x86_64-with-arch
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


GPU memory getting out of bound

Internals of the library; Models.

I am trying to run the pre-trained small GPT model with language head with batch size 16. Problem is, after each iteration about 440MB of memory is allocated and quickly the GPU memory is getting out of bound. I am not running the pre-trained model in training mode.
In my understanding, in each iteration a single word (16 word for batch size 16) is going as input (from the second iteration) and the new attention is calculated and the `past` variable will be updated and increased for 16 word. So, a little bit of memory usage is expected but I don't understand  why it is almost half a GB. I ran the following code to measure the memory usage in each iteration:
```
before=torch.cuda.max_memory_allocated(device=device)
output, past = model(b_train_contexts,past=past)
print("memory usage")
after=torch.cuda.max_memory_allocated(device=device)
print(after-before)

```
Output:
```
memory
0
memory
270742528
memory
442328576
memory
443433472
memory
444525056
memory
445629952
memory
446721536
memory
447826432
memory
448918016
.
.
.
```


Merge bart generate into default generate

This PR is a first version of how the bart generation() code can be merged into the "default" generation function. I think it's actually much better feasible than we originally thought.
Please not that this change also includes all the changes from PR #3135 , so the code changes will be much less cluttered after #3135 merged.

This version was passes the general random language generation tests in found in `test_modeling_common.py` and the easy integration test with the original fairseq model (renamed to `test_cnn_summarization_same_as_fairseq_easy` in `test_modeling_bart`)

There are a couple of things we should discuss:

1. Both Bart generate() and default generate(), encoder-decoder models **must** have a BOS token and an EOS token.

2. Two new parameters were added: `min_length` and `no_repeat_ngram_size` . I think these parameters should be added generally as it is done now.

3. There was one hack which initializes the `decoder_input_ids` to the EOS token and then forces the model to generate the BOS token afterwards (see comment in code line). I changed it to simply start with the BOS token (which makes more sense) and it also passed the "easy integration tests". This hack might be needed to pass the hard integration test though.

4. Fairseq forces the last token of all beam hypotheses to be the EOS token (see comment in line). This is probably necessary to pass the integration tests. It's up for debate whether this the correct way. I would prefer not to do it this way because it will then be impossible to generate unfinished sentences (sentence that end because they hit `max_length`). If one really wants all beam hypotheses to be finished, one could set the `max_length` higher than usual and set the parameter:
`self.early_stopping` in the Beam Hypotheses class to `True`. Up for debate how to handle this.

5. In order to also pass the hard integration tests (which has a padded batch as an input), we will have to add `attention_masks` to the `generate`() function. Here I see three possibilities:
         a) add the `attention_mask` as a parameter to the generation() function.
         b) automatically calculate the `attention_mask` from the `input_ids` **if** the model has a `pad_token_id` **and** there is a `pad_token_id` in the input_ids.
         c) Not allow padded batches for the moment.
I would prefer option b) because some models do have a set `pad_token_id` (such as Bart) so we should be able to allow padded generation.


Remove excess line breaks in DeepPavlov model cards

model card

Related to pretrained model cards


Add model cards for DeepPavlov models

Related to pretrained model cards

Sorry about the Cyrillic `с` yesterday.


Refactor BartModel so that input checks are handled within enc/dec

Implementing #3133

I've left the code that creates dummy inputs and checks/filters the outputs, this could potentially also be moved to `BartEncoder` and `BartDecoder`.


links of the model's pretrained weights

where can I find all the links of the model's pretrained weights,please?to download them by IDE is too slow.


Refactoring and bug fixing beam search generate

This PR cleanes the beam_search decoding part of language generation. It simplifies the code and fixes a small bug for do_sample=True (see comments in code).
It was also tested on all language generation slow tests.

### Future PR

- [x] Do the same change for TF 2.0 if ok -> #3148


Cant import my pretrained bert model from NVIDIA/DeepLearningExamples/

 Trained with **NVIDIA/DeepLearningExamples/** and got my_bert_model/ckpt_1000.pt

**.pt to .bin**

 Edit my_bert_model/ckpt_1000.pt  to my_bert_model/pytorch_model.bin

**My code**
```configuration = BertConfig(3258, 768, 12, 12, 3072)
model = BertModel(configuration)
model = model.from_pretrained('/home/my_bert_model', state_dict)
for key, weight in model.state_dict().items():
    print (weight)
```
**Output**
Every execution of this code produces different ouput


BART: move boilerplate code inside encoder/decoder

# 🚀 Feature request

Could we move the boilerplate code in the `BartModel` `forward()` method inside the `forward()` methods for the encoder and decoder? That way, the encoder and decoder can be called separately as independent modules more easily.

## Motivation

Currently, there's some cleanup work that is done before calling `BartEncoder.forward()` and `BartDecoder.forward()` (looks like attention mask flipping and preparing dummy outputs). If we want to call the encoder and decoder separately from our own code (eg to use Bart as an encoder, or limit which parts are fine tuned) we currently have to re-implement this code.

If we move this logic inside the encoder and decoder, such that `BartModel.forward()` is a thin wrapper around the encoder+decoder, this kind of work would be much easier.

Example usage:
```
model = BartModel.from_pretrained(...)
inputs = tokenizer.encode('I want to classify this text.')
encoding = model.encoder(inputs)
predictions = my_classifier(encoding)
```

## Your contribution

I could put together a PR for this if you agree?

(cc @sshleifer )


[hf_api] Get the public list of all the models on huggingface


Converting tf weights:  AttributeError: 'GPT2Model' object has no attribute 'zeLoss'

# 🐛 Bug

## Information

Model I am using: gpt-2
I am trying to convert a fine-tuned gpt-2 model on tensroflow to pytorch state_dict. I used the nice script [here](https://github.com/huggingface/transformers/blob/ce50305e5b8c8748b81b0c8f5539a337b6a995b9/src/transformers/convert_gpt2_original_tf_checkpoint_to_pytorch.py).

## To reproduce

Steps to reproduce the behavior:

1. I run the following command:
```
python convert_gpt2_original_tf_checkpoint_to_pytorch.py --gpt2_checkpoint_path /home/finetune/model_best.ckpt --pytorch_dump_folder_path /home/gpt2-fine-tf2torch/
```
after running this command, I see the logs showing that the tf weights are being loaded but suddently I got the following error:

```
INFO:transformers.modeling_gpt2:Loading TF weight transformer_decoder/layer_9/self_attention/multihead_attention/value/bias with shape [1024]
INFO:transformers.modeling_gpt2:Loading TF weight transformer_decoder/layer_9/self_attention/multihead_attention/value/kernel with shape [1024, 1024]
INFO:transformers.modeling_gpt2:Loading TF weight word_embedder/w with shape [50257, 1024]
INFO:transformers.modeling_gpt2:Loading TF weight word_embedder_1/w with shape [50257, 1024]
Traceback (most recent call last):
  File "convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 67, in <module>
    convert_gpt2_checkpoint_to_pytorch(args.gpt2_checkpoint_path, args.gpt2_config_file, args.pytorch_dump_folder_path)
  File "convert_gpt2_original_tf_checkpoint_to_pytorch.py", line 38, in convert_gpt2_checkpoint_to_pytorch
    load_tf_weights_in_gpt2(model, config, gpt2_checkpoint_path)
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 84, in load_tf_weights_in_gpt2
    pointer = getattr(pointer, scope_names[0])
  File "/home/anaconda3/envs/torch03/lib/python3.6/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'GPT2Model' object has no attribute 'zeLoss'
```

I tried to load the tf checkpoint and inspect variables, there were no 'zeLoss' but ''OptimizeLoss' which I guess the script mistakenly matched only the 'zeLoss':

```
>>> import tensorflow as tf
>>> import os
>>> tf_path = os.path.abspath('model_best.ckpt')
>>> tf_vars = tf.train.list_variables(tf_path)
```
here is part of the ```tf_vars```:
```
[('OptimizeLoss/beta1_power', []), ('OptimizeLoss/beta2_power', []), ('OptimizeLoss/position_embedder/w/Adam', [1024, 1024]), ('OptimizeLoss/position_embedder/w/Adam_1', [1024, 1024]), ('OptimizeLoss/transformer_decoder/beta/Adam', [1024]), ('OptimizeLoss/transformer_decoder/beta/Adam_1', [1024]), ('OptimizeLoss/transformer_decoder/gamma/Adam', [1024]), ('OptimizeLoss/transformer_decoder/gamma/Adam_1', [1024]), ('OptimizeLoss/transformer_decoder/layer_0/beta/Adam', [1024]), ('OptimizeLoss/transformer_decoder/layer_0/beta/Adam_1', [1024]), ('OptimizeLoss/transformer_decoder/layer_0/ffn/conv1/bias/Adam', [4096]), ('OptimizeLoss/transformer_decoder/layer_0/ffn/conv1/bias/Adam_1', [4096]), ('OptimizeLoss/transformer_decoder/layer_0/ffn/conv1/kernel/Adam', [1024, 4096]),
```

I would appreciate if you can help me fix this .
Thanks


Performance Issue about pretrained bert migration from tensorflow to pytorch

# ❓ Questions & Help

Hi,
I have a question about migration from tensorflow to pytorch.
if I change pretrained BERT with google tensorflow 1.x version to pytorch transformers BERT through transformers-cli, performance drops a lot. I wonder why.

Is there any similar case?


Load pretrained roberta model from fairseq?

# ❓ Questions & Help
Would it be possible to load pretrained model of roberta-base that is trained by myself using fairseq, following the instructions in https://github.com/pytorch/fairseq/blob/master/examples/roberta/README.pretraining.md?

Since the current transformers library is not suitable for pretraining from scratch, I think it would be nice to be able to load pre-trained model trained on fairseq. I think it might be possible but I am not sure how the current transformers' roberta pretrained model is translated/loaded?

thanks!


Add Summarization to Pipelines

Choices:

1) This is not TextGenerationPipeline, so it only supports bart-large-cnn.
1) It doesn't return the input back to the caller because it is annoyingly long.


Create README.md

Related to pretrained model cards


BartTokenizer and 'bart-large-cnn' out of sync

tok.encode('<mask>') -> 52064, but BartForMaskedLM.from_pretrained('bart-large-cnn') does not support a mask token.


NER tutorial: run_tf_ner.py reports an entity number not matching the one in the .txt files

TensorFlow

Anything TensorFlow

Hi everyone,

I'm following this tutorial https://github.com/huggingface/transformers/tree/master/examples/ner .

Generated results are the same but I noticed that the reported entity numbers ("support" column) are not the same B-entity numbers I can count through Notepad++ in *.txt files used as source for training, validation or test.

Which is my mistake?

Thank you.

UPDATE: run_ner.py works correctly. I suppose there is a bug in run_tf_ner.py.

I also tag @stefan-it who is the only mentioned in tutorial. Thank you Stefan.


[Broken Proposal] CircleCI runs tests with torch=1.0.0

Goal: maintain backwards compatibility!


fix sklearn release circle ci [temporary]

new sklearn release (https://github.com/scikit-learn/scikit-learn/releases) seems to be broken or leads to errors for PR since this morning - go back to previous version for now to avoid circle ci errors


include tf gpt2 tests for attn mask and past variable

Test TF GPT2 for correct behavior regarding the past and attn mask variable. Translated code from torch to TF 2.0.


A better way to process extended_attention_mask in BertModel.forward()

Core: Modeling

Internals of the library; Models.

In the `forward()` method of the `BertModel` (https://huggingface.co/transformers/_modules/transformers/modeling_bert.html#BertModel.forward), the `extended_attention_mask` is processed in the following way:

> extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0

I know this is to make an additive mask so that the unmasked positions will be unchanged (by adding 0) and the masked positions will be very small (by subtracting 10000).

But I think it is better to achieve this goal by the following way:

> extended_attention_mask = torch.log(extended_attention_mask)


Making past and mems variables have batch size as their first output dimension.

# 🚀 Feature request

And the moment, the variables **past** / **mems** have the shape:
`(2, batch_size, num_heads, sequence_length, embed_size_per_head)` and `(mem_len, batch_size, embed_size)` , respectively.

meaning that the variable `batch_size` dim is not on first position. Change the variables structure to have `batch_size` on first position.

## Motivation

This might be confusing as all other variables have the `batch_size` dim no first position.
Being certain that the first dimension is always the `batch_size` would be very helpful for the user.
Normally `mems` and `past` variables are just used to speed up decoding, and not too much changed by the user (or even looked at) I think, but consistency should be good anyways.

## Your contribution

Changing this for GPT2/CTRL is very straightforward (changing three lines of code), but for xlnet and transfo-xl would probably take a slightly bigger change.


rename variables named 'word' to 'token' in generate fn

Rename `word` to `token` in generate() function.


Add beam search to generation tf 2 0

Add beam search to the TF generate function as it is done for torch at the moment. Use same TF syntax that was used in PR #3063

EDIT: Also included a quick fix that `TF GPT2 past.shape == PT GPT2 past.shape`


BART FP16

# 🚀 Feature request

I would like to use BART in FP16 mode, but it seems impossible for now :

```
config = BartConfig(vocab_size=50264, output_past=True)
model = AutoModelWithLMHead.from_pretrained('bart-large-cnn', config=config).cuda().half()
tokenizer = AutoTokenizer.from_pretrained('bart-large-cnn')
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer.batch_encode_plus([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
generated_ids = model.generate(inputs['input_ids'].cuda(), attention_mask=inputs['attention_mask'].cuda(), num_beams=4, max_length=5)
```

>   File "/data/user/.venv/bartqg/lib/python3.6/site-packages/transformers/modeling_bart.py", line 647, in forward
    attn_output = torch.bmm(attn_probs, v)
RuntimeError: Expected object of scalar type Float but got scalar type Half for argument #2 'mat2' in call to _th_bmm

@sshleifer Do you plan to implement a FP16-friendly version of BART ?


Skipping outputs

Currently, `encode_plus` and `batch_encode_plus` return the same outputs for different models.

This is sub-optimal as we can't do the following for each model:

```py
inputs = tokenizer.encode_plus(sequence, return_tensors="pt")
model(**inputs)
```

This will crash for DistilBERT as the tokenizer would return `token_type_ids` which can't be handled by the model.

In order to fix this, each tokenizer has to return model-specific arguments. Usually there are the same default arguments, and some models handle less (e.g. DistilBERT, RoBERTa).

This is a mock PR offering a solution using a ~`skip_outputs`~ `return_outputs` argument to tokenizers.

```py
from transformers import DistilBertTokenizer

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased")
print(tokenizer.encode_plus("Hey, how are you?"))
```

Returns a dictionary without the token type ids:

```py
{'input_ids': [101, 4403, 117, 1293, 1132, 1128, 136, 102], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
```

Specifying a custom ~`skip_outputs`~ `return_outputs` at initialisation works as expected:

```py
from transformers import DistilBertTokenizer

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased", return_outputs=["attention_mask", "token_type_ids"])
print(tokenizer.encode_plus("Hey, how are you?"))
```

```py
{'input_ids': [101, 4403, 117, 1293, 1132, 1128, 136, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
```

or with a custom ~skipped~ output:

```py
from transformers import DistilBertTokenizer

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased", return_outputs=["token_type_ids"])
print(tokenizer.encode_plus("Hey, how are you?"))
```

```py
{'input_ids': [101, 4403, 117, 1293, 1132, 1128, 136, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0]}
```

This also works with saving/reloading:

```py
from transformers import DistilBertTokenizer

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased", return_outputs=["token_type_ids"])
print(tokenizer.encode_plus("Hey, how are you?"))
tokenizer.save_pretrained("xxx")

tokenizer = DistilBertTokenizer.from_pretrained("xxx")
print(tokenizer.encode_plus("Hey, how are you?"))
```

Returns the following:

```py
{'input_ids': [101, 4403, 117, 1293, 1132, 1128, 136, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0]}
{'input_ids': [101, 4403, 117, 1293, 1132, 1128, 136, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0]}
```


fix: passing config as Layer trainable param

Lurking bugs discovered while working on other stuff.


Rename BartForMaskedLM -> BartForConditionalGeneration


model cards for both aubmindlab/bert-base-arabert models

model card

Related to pretrained model cards


Adds failing tests for the fast tokenizers

Core: Tokenization

Internals of the library; Tokenization.

Tests

Related to tests

This ports some of the tests over that started failing on the AllenNLP side when the new fast tokenizers came out.

Note: These tests are failing right now. They will need updates to the fast transformers before this can be merged. Maybe it would be better to merge this branch into the branch where the tokenizers are being fixed?


Create README.md

model card

Related to pretrained model cards

- Thumbnail is not set!


BartForSequenceClassification: fix num_labels, add test


[WIP] Add tests that ensure that copied functions remain in sync

Adding some tests that use ast to check that functions that were originally copied stay in sync.


BART: <mask> token ID is outside vocab bounds

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BART

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
from transformers import BartForMaskedLM, BartTokenizer
from transformers.configuration_bart import BartConfig

config = BartConfig(vocab_size=50264, output_past=True)
model = AutoModelWithLMHead.from_pretrained('bart-large-cnn', config=config)
tokenizer = AutoTokenizer.from_pretrained('bart-large-cnn')

ARTICLE_TO_SUMMARIZE = "My friends are <mask> but they eat too many carbs."
inputs = tokenizer.batch_encode_plus([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')

generated_ids = model.generate(inputs['input_ids'], attention_mask=inputs['attention_mask'], num_return_sequences=4)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in generated_ids])
```

## Expected behavior

I'd expect some sort of infilling to occur, but instead I see the error:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-13-bad65359ada6> in <module>
     10 inputs = tokenizer.batch_encode_plus([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
     11
---> 12 generated_ids = model.generate(inputs['input_ids'], attention_mask=inputs['attention_mask'], num_return_sequences=4)
     13 print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in generated_ids])

~/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py in decorate_no_grad(*args, **kwargs)
     47         def decorate_no_grad(*args, **kwargs):
     48             with self:
---> 49                 return func(*args, **kwargs)
     50         return decorate_no_grad
     51

~/.local/lib/python3.6/site-packages/transformers/modeling_bart.py in generate(self, input_ids, attention_mask, max_length, num_beams, repetition_penalty, length_penalty, num_return_sequences, min_len, no_repeat_ngram_size)
   1106                 input_ids, decoder_cache, decoder_input_ids, attention_mask,
   1107             )
-> 1108             outputs = self(**model_inputs)
   1109             lprobs = F.log_softmax(outputs[0][:, -1, :], dim=-1)
   1110

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/.local/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, decoder_cached_states, lm_labels, **unused)
    932             encoder_outputs=encoder_outputs,
    933             decoder_attention_mask=decoder_attention_mask,
--> 934             decoder_cached_states=decoder_cached_states,
    935         )
    936         lm_logits = self.lm_head.forward(outputs[0])

~/.local/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, encoder_outputs, decoder_attention_mask, decoder_cached_states)
    837         assert decoder_input_ids is not None
    838         if encoder_outputs is None:
--> 839             encoder_outputs = self.encoder.forward(input_ids=input_ids, attention_mask=attention_mask)
    840         assert isinstance(encoder_outputs, tuple)
    841         # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn)

~/.local/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask)
    272                 During training might not be of length n_layers because of layer dropout.
    273         """
--> 274         inputs_embeds = self.embed_tokens(input_ids)
    275         embed_pos = self.embed_positions(input_ids)
    276         x = inputs_embeds + embed_pos

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~/.local/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/.local/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: index out of range: Tried to access index 50264 out of table with 50263 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

Looks to me like the `<mask>` token ID (50264) is out of bounds?

## Environment info

- `transformers` version: a088d75e510d5641808ccd72f5dca4df36d95b8e
- Platform: Ubuntu 18.04
- Python version: 3.6.9
- PyTorch version (GPU?): 1.3.1 (Y)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


BertTokenizer.save_pretrained() ignores do_lower_case

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

When saving a tokenizer with the purpose of sharing, `init` arguments are not saved to a config.

## To reproduce

Steps to reproduce the behavior:

Initialize a tokenizer with `do_lower_case=False`, save pretrained, initialize from pretrained. The default `do_lower_case=True` will not be overwritten and further tokenization will be incorrect.

```python3
In[1]: import transformers
In[2]: tokenizer = transformers.BertTokenizer('my/model/vocab.txt', do_lower_case=False)
In[3]: tokenizer.basic_tokenizer.do_lower_case
Out[3]: False
In[4]: tokenizer.save_pretrained('dumped/model/')
Out[4]:
('dumped/model/vocab.txt',
 'dumped/model/special_tokens_map.json',
 'dumped/model/added_tokens.json')
In[5]: tokenizer = transformers.BertTokenizer.from_pretrained('dumped/model/')
In[6]: tokenizer.basic_tokenizer.do_lower_case
Out[6]: True
```

## Expected behavior

```python3
In[1]: import transformers
In[2]: tokenizer = transformers.BertTokenizer('my/model/vocab.txt', do_lower_case=False)
In[3]: tokenizer.basic_tokenizer.do_lower_case
Out[3]: False
In[4]: tokenizer.save_pretrained('dumped/model/')
Out[4]:
('dumped/model/vocab.txt',
 'dumped/model/special_tokens_map.json',
 'dumped/model/added_tokens.json')
In[5]: tokenizer = transformers.BertTokenizer.from_pretrained('dumped/model/')
In[6]: tokenizer.basic_tokenizer.do_lower_case
Out[6]: False
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.0


fix beam_search behavior when sampling

This PR aims to fix the beam search behavior when sampling for language generation.
For once, when doing beam_search decoding for language generation, one would usually do greedy decoding (do_sample=False), so this case should not be used very often, but it should be logical nevertheless.

It's kind of hard to see what happens when doing beam_search decoding with sampling=True, so here a quick example. Running this code:

```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
inputs_dict = tokenizer.encode_plus('The dog', return_tensors='pt')
outputs = model.generate(inputs_dict['input_ids'], num_beams=3, max_length=10)
```

and putting the following print statement:
`print("Sorted hyps: {}".format([x[1] for x in sorted_hyps]))`
after line: https://github.com/huggingface/transformers/blob/a088d75e510d5641808ccd72f5dca4df36d95b8e/src/transformers/modeling_utils.py#L1087

would print the following beam hypothesis before this PR:

```
# printed sorted_hyps from line: 1088
# Beam_idx: 1 - tensor([ 464, 3290,  635,  468,  257, 2041, 3895,  326,  481, 1037])
# Beam_idx: 2 - tensor([ 464, 3290,  635,  468,  257, 2041, 3895,  326,  481, 1037])
# Beam_idx: 3 - tensor([ 464, 3290,  635,  468,  257, 2041, 3895,  326,  481, 1037])

=> Result: best beam hypothesis: "The dog, named T.H., was recently"
```

It can be seen that they are all equal even the last word. And they will always be equal. The reason for this is that currently we sample only word_idx in the interval [0, vocab_size]
(see https://github.com/huggingface/transformers/blob/a088d75e510d5641808ccd72f5dca4df36d95b8e/src/transformers/modeling_utils.py#L975)

which forces that all beam_idx computed in this line:
https://github.com/huggingface/transformers/blob/a088d75e510d5641808ccd72f5dca4df36d95b8e/src/transformers/modeling_utils.py#L1023

always all equal 0. This means that we only consider the first (0-idx) beam and disregard all other beams no matter what.

After this PR: we sample from `[0, num_beams * vocab_size]` (as it's done in greedy decoding so that the beam_idx can be in the range `[0, num_beams]` - as it should be). Same print statement would produce:

```
# printed sorted_hyps from line: 1088
# Beam_idx: 1 - tensor([ 464, 3290,  373,  788, 3888,  284,  257, 6716, 2156,  351])
# Beam_idx: 2 - tensor([ 464, 3290,  373,  788, 3888,  284,  257, 6716, 2156,  11])
# Beam_idx: 3 - tensor([ 464, 3290,  373,  788, 3888,  284,  257, 6716, 2156,  1566])

=> Result: best beam hypothesis: "The dog was then moved to a nearby house until"
```

I discussed with @thomwolf and think this is the best solution for beam_search sampling for language generation.


Change back pipeline signatures

As discussed with @julien-c in the merged #3055.


BART -- RuntimeError: expected device cuda:0 but got device cpu

# 🐛 Bug
@sshleifer

I'm using BART model (bart-large), when I try to use the BartForMaskedLM i'm getting the above error. The reason is that in the _combine_masks (line 146 in modeling_bart) is creating a tensor without the device. so by default it is on CPU.  To reproduce - simply use BartForMaskedLM model with GPU.
can you help? am I missing anything?

Additional details:
----------------------
- `transformers` version: 2.5.1
- Python version: 3.7.4
- PyTorch version (GPU?): 1.3.0 (with GPU)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

stuck trace:
File "/specific/netapp5_2/gamir/adi/git/BERTese/lama/training.py", line 151, in train_and_eval
    outputs = model(b_in_tensor, lm_labels=b_label_tensor)
  File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 925, in forward
    decoder_cached_states=decoder_cached_states,
  File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 844, in forward
    decoder_cached_states=decoder_cached_states,
  File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 499, in forward
    need_attn_weights=self.output_attentions,
  File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 372, in forward
    attn_mask=attention_mask,
  File "/specific/netapp5_2/gamir/adi/miniconda3/envs/trans_py37/lib/python3.7/site-packages/transformers/modeling_bart.py", line 629, in forward
    attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
RuntimeError: expected device cuda:0 but got device cpu

Thanks,
Adi.


Support keras JSON/HDF5 serialization of main layers

Fixes #3101


bert-base-arabic model card

model card

Related to pretrained model cards

Please add model card file to the newly added asafaya/bert-base-arabic model


Keras layers should override get_config to be JSON-serializable

# 🚀 Feature request

Support JSON serialization of Keras layers by overriding `get_config`, so that they can be sent to Tensorboard to display a conceptual graph of the model.

## Motivation

### 1. Without this, can't write model graph to Tensorboard

From https://github.com/tensorflow/tensorflow/blob/d1786ea19eb41922c0d433d71ca13b123b69b4be/tensorflow/python/ops/summary_ops_v2.py#L1004-L1009

> Writing the Keras model configuration allows the TensorBoard graph plugin to render a conceptual graph, as opposed to graph of ops. In case the model fails to serialze as JSON, it ignores and returns False.

### 2. Without this, can't save model with Keras `model.save`

The base class `get_config` method actually refuses to run if the subclass initializer has positional arguments; from `tensorflow/python/keras/engine/base_layer.py`:

```python
  @base_layer_utils.default
  def get_config(self):
    [...]
    if len(extra_args) > 1 and hasattr(self.get_config, '_is_default'):
      raise NotImplementedError('Layer %s has arguments in `__init__` and '
                                'therefore must override `get_config`.' %
                                self.__class__.__name__)
```

and all the `TF*MainLayer` classes have a `config` positional argument, so this says they “must” all override `get_config`.

And sure enough, if I make a simple Keras model using a TFBertMainLayer inside:

```python
import tensorflow as tf
from transformers import TFBertMainLayer, BertConfig

def create_model(max_sequence_len: int) -> tf.keras.Model:
    cfg = BertConfig.from_pretrained('bert-base-cased')
    bert = TFBertMainLayer(cfg)
    input_ids = tf.keras.Input(shape=(max_sequence_len,), dtype=tf.int32, name='wp_input_token_ids')
    input_mask = tf.keras.Input(shape=(max_sequence_len,), dtype=tf.bool, name='wp_input_mask')
    pooled = bert(input_ids, input_mask)[1]
    out = tf.keras.layers.Dense(units=3, activation='softmax',
                                kernel_initializer=tf.keras.initializers.glorot_uniform(),
                                use_bias=False,
                                name='classification'
                                )(pooled)
    return tf.keras.Model(inputs=[input_ids, input_mask], outputs=[out])

model = create_model(40)
model.save(filepath="tf_model.h5")
```

... then `model.save` fails:

```
Traceback (most recent call last):
  File "trysave.py", line 32, in <module>
    model.save(filepath="tf_model.h5")
  File ".../tensorflow_core/python/keras/engine/network.py", line 1008, in save
    signatures, options)
  File ".../tensorflow_core/python/keras/saving/save.py", line 112, in save_model
    model, filepath, overwrite, include_optimizer)
  File ".../tensorflow_core/python/keras/saving/hdf5_format.py", line 99, in save_model_to_hdf5
    model_metadata = saving_utils.model_metadata(model, include_optimizer)
  File ".../tensorflow_core/python/keras/saving/saving_utils.py", line 172, in model_metadata
    raise e
  File ".../tensorflow_core/python/keras/saving/saving_utils.py", line 169, in model_metadata
    model_config['config'] = model.get_config()
  File ".../tensorflow_core/python/keras/engine/network.py", line 918, in get_config
    return copy.deepcopy(get_network_config(self))
  File ".../tensorflow_core/python/keras/engine/network.py", line 1993, in get_network_config
    layer_config = serialize_layer_fn(layer)
  File ".../tensorflow_core/python/keras/utils/generic_utils.py", line 198, in serialize_keras_object
    config = instance.get_config()
  File ".../tensorflow_core/python/keras/engine/base_layer.py", line 499, in get_config
    raise NotImplementedError('Layers with arguments in `__init__` must '
NotImplementedError: Layers with arguments in `__init__` must override `get_config`.
```

## Your contribution

I got this working for the one layer I was experimenting with, like this:

```patch
diff --git a/src/transformers/modeling_tf_bert.py b/src/transformers/modeling_tf_bert.py
index 19046235..74ad621c 100644
--- a/src/transformers/modeling_tf_bert.py
+++ b/src/transformers/modeling_tf_bert.py
@@ -21,6 +21,7 @@ import logging
 import numpy as np
 import tensorflow as tf

+from . import PretrainedConfig
 from .configuration_bert import BertConfig
 from .file_utils import MULTIPLE_CHOICE_DUMMY_INPUTS, add_start_docstrings, add_start_docstrings_to_callable
 from .modeling_tf_utils import TFPreTrainedModel, get_initializer, shape_list
@@ -474,12 +475,20 @@ class TFBertNSPHead(tf.keras.layers.Layer):
 class TFBertMainLayer(tf.keras.layers.Layer):
     def __init__(self, config, **kwargs):
         super().__init__(**kwargs)
+        if isinstance(config, dict):
+            config = PretrainedConfig.from_dict(config)
+        self.config = config
         self.num_hidden_layers = config.num_hidden_layers

         self.embeddings = TFBertEmbeddings(config, name="embeddings")
         self.encoder = TFBertEncoder(config, name="encoder")
         self.pooler = TFBertPooler(config, name="pooler")

+    def get_config(self):
+        cfg = super().get_config()
+        cfg['config'] = self.config.to_dict()
+        return cfg
+
     def get_input_embeddings(self):
         return self.embeddings

```

and I didn't need to modify any other layer classes, just the main layer.

So maybe it's enough to do this for all the `MainLayer` classes:

```
$ rg 'class .*MainLayer\(tf.keras.layers.Layer\)' src | cat
src/transformers/modeling_tf_openai.py:class TFOpenAIGPTMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_transfo_xl.py:class TFTransfoXLMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_xlm.py:class TFXLMMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_xlnet.py:class TFXLNetMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_distilbert.py:class TFDistilBertMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_bert.py:class TFBertMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_albert.py:class TFAlbertMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_ctrl.py:class TFCTRLMainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_t5.py:class TFT5MainLayer(tf.keras.layers.Layer):
src/transformers/modeling_tf_gpt2.py:class TFGPT2MainLayer(tf.keras.layers.Layer):
```

... or, neater, to extract a single `TFMainLayer(tf.keras.layers.Layer)` superclass for all of them, to do this in one place.


Fix QA models binding for Flaubert, XLNet and XLM.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>

Fix #2893


Don't crash if fine-tuned model doesn't end with a number

That's the same fix applied in https://github.com/huggingface/transformers/issues/2258 , but for the GLUE example


why BertModel' object has no attribute 'bias'

```py
from torch.nn.functional import softmax
from transformers import BertForNextSentencePrediction, BertTokenizer,BertConfig,BertModel
seq_A = 'I like cookies !'
seq_B = 'Do you like them ?'
'''
model = BertForNextSentencePrediction.from_pretrained('bert-base-cased')
'''

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')
config = BertConfig.from_json_file('E:\\work\\pycharm\\transformers-master\\tf_model\\bert_config.json')

model = BertModel.from_pretrained('E:\\work\\pycharm\\transformers-master\\tf_model\\model.ckpt.index', from_tf=True, config=config)

encoded = tokenizer.encode_plus(seq_A, text_pair=seq_B, return_tensors='pt')
print(encoded)
seq_relationship_logits = model(**encoded)[0]
probs = softmax(seq_relationship_logits, dim=1)
print(seq_relationship_logits)
print(probs)

the above demo can be used for next sentence prediction provided by BramVanroy,thank you again,
Now I want to use my own bert pre-training model or google's model  in this task ,so I find the examples in modeling_utils.py LINE 366

         # Loading from a TF checkpoint file instead of a PyTorch model (slower)
          config = BertConfig.from_json_file('./tf_model/my_tf_model_config.json')
          model = BertModel.from_pretrained('./tf_model/my_tf_checkpoint.ckpt.index', from_tf=True, config=config)
```

bert_config.json :

```
{
  "attention_probs_dropout_prob": 0.1,
  "directionality": "bidi",
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "max_position_embeddings": 512,
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pooler_fc_size": 768,
  "pooler_num_attention_heads": 12,
  "pooler_num_fc_layers": 3,
  "pooler_size_per_head": 128,
  "pooler_type": "first_token_transform",
  "type_vocab_size": 2,
  "vocab_size": 21128
}
```

But when I run this demo,some errors happen,
```
AttributeError: 'BertModel' object has no attribute 'bias'

Traceback (most recent call last):
  File "E:/work/pycharm/transformers-master/src/transformers/test.py", line 18, in <module>
    model = BertModel.from_pretrained('E:\\work\\pycharm\\transformers-master\\tf_model\\model.ckpt.index', from_tf=True, config=config)
  File "E:\work\pycharm\transformers-master\src\transformers\modeling_utils.py", line 485, in from_pretrained
    model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
  File "E:\work\pycharm\transformers-master\src\transformers\modeling_bert.py", line 106, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "E:\Users\Administrator\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertModel' object has no attribute 'bias'
```

It took a long time to fix it, but failed ,can you help me to solve it.thanks a lot!!


Some question about training BERT after change the Vocab.txt size

The original bert-base-chinese-vocab.txt size is 21128.
I use my own vocab.txt size is 44900
When I try to train Fine-tune BERT Model using BertForMaskedLM it have some promble about size mismatch.
I have try to change the BertEmbeddings its self.word_embeddings config.vocab_size to 44900 like this
`self.word_embeddings = nn.Embedding(44900, config.hidden_size, padding_idx=0) `
But it still have problem like this

```
RuntimeError: Error(s) in loading state_dict for BertForMaskedLM:
        size mismatch for bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([21128, 768]) from checkpoint, the shape in current model is torch.Size([44900, 768]).
        size mismatch for cls.predictions.bias: copying a param with shape torch.Size([21128]) from checkpoint, the shape in current model is torch.Size([44900]).
        size mismatch for cls.predictions.decoder.weight: copying a param with shape torch.Size([21128, 768]) from checkpoint, the shape in current model is torch.Size([44900, 768]).
```
I do not sure how to fix it.
I have think about if I need to change the pre trained BERT config.json file? Its vocab_size from 21128 to 44900


BART BartForSequenceClassification example

# 🐛 Bug
I'm trying the run the code on the documentation.
https://huggingface.co/transformers/model_doc/bart.html#bartforsequenceclassification

```
from transformers import BartTokenizer, BartForSequenceClassification
import torch

tokenizer = BartTokenizer.from_pretrained('bart-large')
model = BartForSequenceClassification.from_pretrained('bart-large')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute",
add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)
loss, logits = outputs[:2]
```

output:
```
Traceback (most recent call last):
  File "/ssd-playpen/yixin1/projects/cleartheuncertainty/utest/utest_transformer/utest_bart.py", line 11, in <module>
    outputs = model(input_ids, labels=labels)
  File "/ssd-playpen/yixin1/projects/cleartheuncertainty/ENV/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/ssd-playpen/yixin1/projects/cleartheuncertainty/ENV/lib/python3.7/site-packages/transformers/modeling_bart.py", line 1327, in forward
    loss = F.cross_entropy(logits.view(-1, self.num_labels), labels.view(-1))
  File "/ssd-playpen/yixin1/projects/cleartheuncertainty/ENV/lib/python3.7/site-packages/torch/nn/modules/module.py", line 576, in __getattr__
    type(self).__name__, name))
AttributeError: 'BartForSequenceClassification' object has no attribute 'num_labels'
```

I guess this should be a quick fix or so.