Getting different topk results when using past + attention mask for more than 1 sentence

# ❓ Questions & Help

Hi. I'm having some issues when using the past + attention mask functionality. Results are not the ones I was expecting to get... I'm using latest master as the latest release 2.5.1 is failing (`RuntimeError: The size of tensor a (6) must match the size of tensor b (3) at non-singleton dimension 3` at modeling_gpt2.py:150). With master there's no error event being thrown.

My code works with a single sentence, but it doesn't work with more sentences because I get different predictions as I'm expecting to get the same topk results by using past vs no using past at all.

Code snippet below:
```
from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch

tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Complete phrases are: "I like to drink soda and" and "Please help me with this"
docs = ["I like to", "Please help me"]
# note: comment the above line and uncomment the following line to make it work with 1 document
#docs = ["I like to"]
docs_tensors = tokenizer.batch_encode_plus(
    [d for d in docs], pad_to_max_length=True, return_tensors='pt')

docs_next = ["soda and ", "with this"]
# note: comment the above line and uncomment the following line to make it work with 1 document
#docs_next = ["soda and "]
docs_next_tensors = tokenizer.batch_encode_plus(
    [d for d in docs_next], pad_to_max_length=True, return_tensors='pt')

# predicting the first part of each phrase
_, past = model(docs_tensors['input_ids'], attention_mask=docs_tensors['attention_mask'])

# predicting the rest of the phrase
attn_mask = torch.cat([docs_tensors['attention_mask'], docs_next_tensors['attention_mask']], dim=-1)
logits, _ = model(docs_next_tensors['input_ids'], attention_mask=attn_mask, past=past)
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Results with past:", words)

#####################
docs_full_tensors = tokenizer.batch_encode_plus(
    [d + n for d, n in zip(docs, docs_next)], pad_to_max_length=True, return_tensors='pt')
logits, _ = model(docs_full_tensors['input_ids'], attention_mask=docs_full_tensors['attention_mask'])
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Results without past:", words)

```

Output results (please note the inconsistence between both results - I'm expecting them to match like the next test):
```
Results with past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just', 'The', 'A', 'I', '"', 'This', 'In', 'It', 'B', 'As', 'S', 'We', 'M', 'P', 'C', 'There', 'If', 'T', '1', 'By', 'F', 'You', 'D', 'Image', 'An', 'When', '(', 'On', 'What', 'For', 'L', 'H', 'R', 'About', '[', 'From', 'G', 'After', 'E', 'One', 'K', 'With', 'Still', 'So', 'W', 'by', 'N', 'My', 'Please', 'How', 'O']
Results without past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just', '.', ' site', ' page', '!', ' project', ',', ' website', ' article', ' thread', ':', ' story', ' one', ' and', ' message', ' post', ' issue', ' great', ' blog', ' amazing', ' thing', ' little', ' problem', '\n', '?', ' book', ' game', ' by', ' to', ' wonderful', ' in', ' awesome', ' guy', ' community', ' new', ' mod', ' information', ' web', ' beautiful', '...', ' man', ' stuff', ' work', ' place', ' video', '."', ' app', ' kind', ' piece', '!"', ' world']
```

If I were uncommenting the lines stated in the code I would get same results (...but for only 1 sentence):
```
Results with past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just']
Results without past: [' I', ' the', ' a', ' t', ' s', ' k', ' it', ' c', ' my', ' other', ' to', ' b', ' d', ' we', ' make', ' then', ' o', ' m', ' have', ' you', ' do', ' all', ' l', ' some', ' so', ' can', ' i', ' j', ' p', ' that', ' be', ' get', ' he', ' take', ' st', ' this', ' also', ' n', ' ch', ' is', ' use', ' h', ' they', ' f', ' put', ' go', ' g', ' w', ' not', ' just']
```

## Details
<!-- Description of your issue -->
Original stackoverflow question that @patrickvonplaten had answered initially:
https://stackoverflow.com/questions/60459292/using-past-and-attention-mask-at-the-same-time-for-gpt2/


[Bart] dont call .forward


wrong 'label2id' and 'id2label' in config when loading from pretrained

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.
```python
from transformers import BertConfig
config = BertConfig.from_pretrained('bert-base-cased', num_labels=3)
print(config.id2label)
```
2. Prints: {0: 'LABEL_0', 1: 'LABEL_1'}

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

prints {0: 'LABEL_0', 1: 'LABEL_1', 2: 'LABEL_2'}

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Ubuntu 16.04
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


ċ in gpt2

Core: Tokenization

Internals of the library; Tokenization.

Discussion

Discussion on a topic (keep it focused or open a new issue though)

There's a 'ċ' in gpt2.

```
vocab = list(tokenizer_gpt2.encoder.keys())
vocab[198]
```
output: ċ

Based on some examples, I guess it means "with_break". But I can't find this parameter in gpt2 tokenizer document. Can anyone tell me the meaning? thank you.


Fast tokenizers fail when the input is just spaces

Core: Tokenization

Internals of the library; Tokenization.

Slow tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
>>> t.encode_plus(" ", add_special_tokens=False)
{'input_ids': [], 'token_type_ids': []}
```

Fast tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> t.encode_plus(" ")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1894, in encode_plus
    return {key: value[0] if isinstance(value[0], list) else value for key, value in batched_output.items()}
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1894, in <dictcomp>
    return {key: value[0] if isinstance(value[0], list) else value for key, value in batched_output.items()}
IndexError: list index out of range
```

The `add_special_tokens=False` bit is critical. Otherwise, there is no failure because the results aren't empty.


Cuda error during evaluation - CUBLAS_STATUS_NOT_INITIALIZED

# 🐛 Bug

## Information
Overview:
I am using the Bert pre-trained model and trying to finetune it using a customized dataset which requires me to add new tokens so that the tokenizer doesn't wordpiece them (these tokens are of the form <1234> and </1234> where 1234 can be any int converted to string).
I was able to go through the train step but when it comes to evaluating the perplexity I get :
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] my own modified scripts: (give details below)
The only bit of tweak I made was to use tokenizer.add_tokens("<my_new_token>")
before tokenizing using tokenizer.batch_encode_plus

The tasks I am working on is:
* [x] my own task or dataset: (give details below)
facebook messages dataset

## To reproduce
Steps to reproduce the behavior:

1. In LineByLineTextDataset - add new tokens by using tokenizer.add_tokens("<new_token>") for each line that is added in lines list.
(The only other change I made was to fetch the text directly from DB instead of using the text files)
2. I limited the run to use only 3 instances of text line to debug
3. Run the regular examples script to train and evaluate

Error:
Exception has occurred: RuntimeError
Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 987, in forward
    encoder_attention_mask=encoder_attention_mask,
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 790, in forward
    encoder_attention_mask=encoder_extended_attention_mask,
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 407, in forward
    hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 368, in forward
    self_attention_outputs = self.attention(hidden_states, attention_mask, head_mask)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 314, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/transformers/modeling_bert.py", line 216, in forward
    mixed_query_layer = self.query(hidden_states)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 87, in forward
    return F.linear(input, self.weight, self.bias)
  File "/data/nisoni/anaconda3/envs/trans/lib/python3.6/site-packages/torch/nn/functional.py", line 1372, in linear
    output = input.matmul(weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_NOT_INITIALIZED when calling `cublasCreate(handle)`
  File "/data/nisoni/transformers/transformers/examples/run_language_modeling.py", line 550, in evaluate
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/data/nisoni/transformers/transformers/examples/run_language_modeling.py", line 910, in main
    result = evaluate(args, model, tokenizer, prefix=prefix)
  File "/data/nisoni/transformers/transformers/examples/run_language_modeling.py", line 918, in <module>
    main()

## Expected behavior
A regular examples run giving a perplexity score as it gives without adding new tokens

## Environment info
- `transformers` version: 2.5.1
- Platform: Linux-4.4.0-171-generic-x86_64-with-debian-stretch-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: not explicitly
- Using distributed or parallel set-up in script?:  not explicitly


add models cards for camembert-base-fquad camembert-base-squad

model card

Related to pretrained model cards

Following #2893

Model links:

- [`fmikaelian/camembert-base-fquad`](https://huggingface.co/fmikaelian/camembert-base-fquad)
- [`fmikaelian/camembert-base-squad`](https://huggingface.co/fmikaelian/camembert-base-squad)


Fast tokenizers can't `encode_plus` a list of ids; slow tokenizers can

Core: Tokenization

Internals of the library; Tokenization.

With the slow tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=False)
>>> t.encode_plus([1000])
{'input_ids': [101, 1000, 102],
 'token_type_ids': [0, 0, 0],
 'attention_mask': [1, 1, 1]}
```

With the fast tokenizers:
```
>>> import transformers
>>> t = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True)
>>> t.encode_plus([1000])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1889, in encode_plus
    **kwargs,
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/transformers/tokenization_utils.py", line 1815, in batch_encode_plus
    tokens = self._tokenizer.encode(*batch_text_or_text_pairs[0])
  File "/Users/dirkg/anaconda3/envs/allennlp/lib/python3.7/site-packages/tokenizers/implementations/base_tokenizer.py", line 141, in encode
    return self._tokenizer.encode(sequence, pair)
TypeError
```


Error when runnig run_tf_ner.py

# 🐛 Bug

## Information
```python
!python3 /content/transformers/examples/ner/run_tf_ner.py --data_dir /content/ner_dataset \
--model_type bert \
--labels /content/labels.txt \
--model_name_or_path dccuchile/bert-base-spanish-wwm-cased \
--output_dir model_output \
--max_seq_length 256 \
--num_train_epochs 5\
--per_gpu_train_batch_size 42 \
--save_steps 2000\
--do_train \
--do_eval

Traceback (most recent call last):
  File "/content/transformers/examples/ner/run_tf_ner.py", line 14, in <module>
    from transformers import (
ImportError: cannot import name 'GradientAccumulator'
```

## Transformer version ```transformers==2.5.1```


Disabling Eager Mode Prevents Loading Pre-Trained BERT

Core: Modeling

Internals of the library; Models.

TensorFlow

Anything TensorFlow

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**BERT**
Language I am using the model on (English, Chinese ...):
**English**

## To reproduce

Steps to reproduce the behavior:

1. Disable Tensorflow eager mode `tf.compat.v1.disable_v2_behavior()`
2. Create a pretrained BERT instance `model = transformers.TFBertModel.from_pretrained("bert-base-uncased")`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
`model` contains a pre-trained BERT model

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.6
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


TF GPU CI


Update README.md

model card

Related to pretrained model cards

- Add example of usage
- Update metrics


Memory error : load 200GB file in run_language_model.py

Core: Tokenization

Internals of the library; Tokenization.

line 107 in run_language_model.py

with open(file_path, encoding="utf-8") as f:
                text = f.read()

any idea how to use generators to load large files?


Summarization Examples: add Bart CNN Evaluation


Why there is no TransfoXLForSequenceClassification class?

Core: Modeling

Internals of the library; Models.

Hello, huggingface! Thank you for the great work with doing neat interfaces for transformers family!

I'm analyzing performance of transformers on a task of texts classification (like SST-2 in GLUE). I found that many architectures has `<ModelName>ForSequenceClassification` class, But `transformerXL` has restricted set of auxiliary classes (https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_transfo_xl.py).

Is there any reasons for missing these useful class? Is it planned to be implemented or is there any restrictions for such implementation?


Docker Hub automatic tests failing

Should Fix

This has been identified as a bug and should be fixed.

Tests

Related to tests

The automated docker tests are failing on HEAD and when you click details you get 404, so tough to debug


Bart CUDA not working

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BART - bart-large

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load model
2. Tokenize text
3. Send model and tensor to cuda
4. Forward the model

```python
from transformers import BartConfig, BartTokenizer, BartForMaskedLM, BartModel

configuration = BartConfig()
tokenizer_class = BartTokenizer
model_class = BartForMaskedLM(configuration)

tokenizer = tokenizer_class.from_pretrained('bart-large')
model = model_class.from_pretrained('bart-large')

model.eval()
model.to('cuda')

tokens = tokenizer.encode("Text example to test natural language generation with bart.")
input_ids = torch.tensor([tokens])
input_ids = input_ids.to('cuda')

with torch.no_grad():
    last_hidden_states = model(input_ids)[0]

print("Len tokens:", len(tokens))
print("Shape last hidden states:", last_hidden_states.shape)
```

This code raises the following error:
```python
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-24-3bba66ed6aeb> in <module>
      3
      4 with torch.no_grad():
----> 5     last_hidden_states = model(input_ids)[0]
      6
      7 print("Len tokens:", len(tokens))

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, encoder_outputs, decoder_input_ids, decoder_attention_mask, decoder_cached_states, lm_labels, **unused)
    923             encoder_outputs=encoder_outputs,
    924             decoder_attention_mask=decoder_attention_mask,
--> 925             decoder_cached_states=decoder_cached_states,
    926         )
    927         lm_logits = self.lm_head.forward(outputs[0])

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, attention_mask, decoder_input_ids, encoder_outputs, decoder_attention_mask, decoder_cached_states)
    842             attention_mask,
    843             decoder_attn_mask,
--> 844             decoder_cached_states=decoder_cached_states,
    845         )
    846         # Attention and hidden_states will be [] or None if they aren't needed

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, input_ids, encoder_hidden_states, encoder_padding_mask, combined_mask, decoder_cached_states, **unused)
    497                 decoder_cached_states=layer_state,
    498                 attention_mask=combined_mask,
--> 499                 need_attn_weights=self.output_attentions,
    500             )
    501             if self.output_past:

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, x, encoder_hidden_states, encoder_attn_mask, decoder_cached_states, attention_mask, need_attn_weights)
    370             decoder_cached_states=decoder_cached_states,
    371             need_weights=need_attn_weights,
--> 372             attn_mask=attention_mask,
    373         )
    374         x = F.dropout(x, p=self.dropout, training=self.training)

~/projects/p130-nlg/.conda/envs/p130-nlg-env-kgpubart/lib/python3.6/site-packages/transformers/modeling_bart.py in forward(self, query, key, value, key_padding_mask, decoder_cached_states, need_weights, static_kv, attn_mask)
    627
    628         if attn_mask is not None:
--> 629             attn_weights = attn_weights.view(bsz, self.num_heads, tgt_len, src_len) + attn_mask
    630             attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)
    631

RuntimeError: expected device cuda:0 but got device cpu
```

But I have tested this code with another examples (like GPT-2) and it works.

## Expected behavior

I would expect to get the tensor size, as with another models I have tested this code.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux
- Python version: 3.6.7
- PyTorch version (GPU?): 1.4.0. GPU: Yes
- Tensorflow version (GPU?): No used
- Using GPU in script?: Trying to
- Using distributed or parallel set-up in script?: No


correct greedy generation when doing beam search

This PR changes the behavior of greedy beam search generation as discussed and wished in #2415 .
Also two assertion statements are added:

1. It is not allowed to generate multiple sequences from the same input_ids when greedy generation (`num_return_sequences > 1`, `do_sample=False`, `num_beams` == 1 => `AssertionError`) because it would always lead to the same output sequence for all `num_return_sequences`.

2. It is not allowed to generate more sequences when doing greedy beam serach generation than the number of beams that are used (`num_return_sequences` <= `num_beams`, `do_sample=False` => `AssertionError`) because this is not possible or would also lead to the same output sequences.

Discussion:

- [x] the generation function becomes bigger and bigger handling more and more exceptions - might need a big refactoring at some point which modularizes it for more flexibility and more readability. Also when thinking about including the encoder-decoder models in the model.generate() function.
Also maybe the `no_beam_search_generation` fn could simply be handled by `beam_search_generation(num_beams=1)` ?

- [x] beam search when do_sample=True still does not work really (see PR #2317  ). Should discuss how exactly it should be implemented.

@thomwolf, @LysandreJik, @sshleifer


fix n_gpu count when no_cuda flag is activated

As I understand it, `no_cuda` should prevent the use of GPU in the `run_*` example scripts. However, `n_gpu` doesn't take it into account and count the numbers of GPUs available on the machine. It sends the model on the GPUs while the tensors are still on CPUs...


XLNet multiple sentence modeling token type ids

Core: Modeling

Internals of the library; Models.

Help wanted

Extra attention is needed, help appreciated

XLNet was designed to handle multiple segment modeling (i.e. > 2 sentences) by using relative segment encodings. For sentence-level classification tasks with arbitrary sentence counts, what is the structure of the segment (token type) ids? I’ve found from the documentation that 2-sequence classification is supported by using `create_token_type_ids` but what about more than two segments?

If more than two segments are supported, would it be correct to infer (from examples in the documentation) that a 3-sentence input with `<cls>` after each sentence (`<sep>` token) should have the form:

0, 0, 0, 0, 0, 1, 2, 2, 2, 2, 3, 4, 4, 4, 4, 5.

where 1, 3, 5 are classification token segment ids? Would the transformers XLNet implementation support segment ids of this form?


Training TFBertForSequenceClassification with custom X and Y data

Usage

General questions about the library

I am working on a TextClassification problem, for which I am trying to traing my model on TFBertForSequenceClassification given in huggingface-transformers library.

I followed the example given on their github page, I am able to run the sample code with given sample data using tensorflow_datasets.load('glue/mrpc'). However, I am unable to find an example on how to load my own custom data and pass it in  model.fit(train_dataset, epochs=2, steps_per_epoch=115, validation_data=valid_dataset, validation_steps=7).

How can I define my own X, do tokenization of my X and prepare train_dataset with my X and Y. Where X represents my input text and Y represents classification category of given X.

Sample Training dataframe :
```
    text    category_index
0   Assorted Print Joggers - Pack of 2 ,/ Gray Pri...   0
1   "Buckle" ( Matt ) for 35 mm Width Belt  0
2   (Gagam 07) Barcelona Football Jersey Home 17 1...   2
3   (Pack of 3 Pair) Flocklined Reusable Rubber Ha...   1
4   (Summer special Offer)Firststep new born baby ...   0
```
```
Question already asked on SO :
https://stackoverflow.com/questions/60463829/training-tfbertforsequenceclassification-with-custom-x-and-y-data
```


General questions about the library

# ❓ Questions & Help
Only enable some labels in BERT fine-tuned NER

## Details
I would be interested to enable only some labels in BERT fine-tuned NER prediction.
For example, I know what are the entities and I would like to train a model that will classify entities - but I am not interested to train/predict O tag.
What would be the best way to do it?
Thanks


Finetuned BERT model does not seem to predict right labels/work properly?

# ❓ Questions & Help
I am trying out a finetuned BERT model for token classification (--> https://huggingface.co/bert-base-cased-finetuned-conll03-english), but when I observe the model output (i.e. the logits after applying the softmax) and compare it with the true label_ids, they are totally uncorrelated (see pictures).

https://i.stack.imgur.com/gVyMn.png
https://i.stack.imgur.com/qS62L.png

## Details

I assume that the finetuned model (bert-base-cased-finetuned-conll03-english) is correctly pretrained, but I don't seem to understand why its predictions are off. I think one issue is that the pretrained model has another labelling scheme than I made myself during data prep (so that the tag2name dict is different), but I don't know how I can find out what label-index map the model uses for its predictions. Even then it is not the case that the model consistently makes the same mistakes, it is outputting things quite randomly.

Any idea what the issue could be?

``


# ❓ Questions & Help

I want to ask about the Chinese BERT model can be used represented by words instead of character?
Because when I do BERT visual for Chinese can only see its attention from character to character.
I want to see attention from words to words. Can I change this?
Thanks a lot for your help


Predict the next word in sentence context from the list of possible words in Russian

General questions about the library

Hello from Russia. I have a task of predicting the next word from the list of possible words in Russian. How can I do this?


load_tf_weights_in_bert : 'BertModel' object has no attribute 'bias'

Core: Modeling

Internals of the library; Models.

```
AttributeError                            Traceback (most recent call last)
<ipython-input-14-0d66155b396d> in <module>
     12
     13         K.clear_session()
---> 14         model = create_model()
     15         optimizer = tf.keras.optimizers.Adam(learning_rate=2e-5)
     16         model.compile(loss='categorical_crossentropy', optimizer=optimizer, metrics=['acc', 'mae'])

<ipython-input-13-4f429fe61419> in create_model()
      5     config = BertConfig.from_pretrained(BERT_PATH + 'bert_config.json')
      6     config.output_hidden_states = False
----> 7     bert_model = BertModel.from_pretrained(BERT_PATH + 'bert_model.ckpt.index', from_tf=True, config=config)
      8     # if config.output_hidden_states = True, obtain hidden states via bert_model(...)[-1]
      9     embedding = bert_model(input_id, attention_mask=input_mask, token_type_ids=input_atn)[0]

~/anaconda3/envs/fasterai/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    482             if resolved_archive_file.endswith(".index"):
    483                 # Load from a TensorFlow 1.X checkpoint - provided by original authors
--> 484                 model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
    485             else:
    486                 # Load from our TensorFlow 2.0 checkpoints

~/anaconda3/envs/fasterai/lib/python3.7/site-packages/transformers/modeling_bert.py in load_tf_weights_in_bert(model, config, tf_checkpoint_path)
    103                 pointer = getattr(pointer, "weight")
    104             elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
--> 105                 pointer = getattr(pointer, "bias")
    106             elif scope_names[0] == "output_weights":
    107                 pointer = getattr(pointer, "weight")

~/anaconda3/envs/fasterai/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    574                 return modules[name]
    575         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 576             type(self).__name__, name))
    577
    578     def __setattr__(self, name, value):

AttributeError: 'BertModel' object has no attribute 'bias'
```

related libs&version:
transformers 2.5.1
tensorflow 2.1.0

environment:
NVIDIA-SMI 440.59       Driver Version: 440.59       CUDA Version: 10.2


No Causal Attention Masking in GPT-2 LM Finetuning Script

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
running run_language_modeling.py on WikiText-2 dataset

* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. attention_mask is None at each forward step of the GPT-2 model (GPT2LMHeadModel)

## Expected behavior
attention_mask should reflect causal attention masking for the LM objective in finetuning GPT-2 so that outputs (t) only attend to inputs at previous time steps (1,..,t-1) instead of relying on input at the same time-step of output (t) where GPT-2 can simply learn to copy the input as output to optimize the LM objective.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux-4.15.0-76-generic-x86_64-with-glibc2.10
- Python version: 3.8.1
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


Problem with using pretrained BertTokenizer for Korean

Core: Tokenization

Internals of the library; Tokenization.

I have a corpus that contains Korean sentences. Here is the output of a berttokenizer for a token:

`tok = BertTokenizer.from_pretrained('bert-base-multilingual-uncased')`
`tok.convert_tokens_to_ids(tok.tokenize('잘해놨습니다'))`
`[44628, 14840, 97071, 97089, 97104, 13212, 79427]`
`tok = BertTokenizer.from_pretrained('bert-base-multilingual-cased')`
`tok.convert_tokens_to_ids(tok.tokenize('잘해놨습니다'))`
`[100]`
transformers version: 2.4.1

Totally, 'cased' tokenizer produces more 'unknown' token than 'uncased' one for Korean.
Is it a bug?


No speed diference when doing prediction between BERT and ALBERT

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I'm comparing two trained models, one using BERT and the other using ALBERT. I'm using the following code to do a prediction in both models: it tokenizes a list of phrases, padding them according to biggest max length per tokenized phrase, and then it applies a prediction:

```
def padding(phrase_list):
    """
    Add padding to phrases in phrase list
    """
    max_size = 0
    for phrase in phrase_list:
        max_size = max(max_size, len(tokenizer_phrase.encode(phrase)))
    print(f"Max_size: {max_size}")
    padded_list = []
    for phrase in phrase_list:
        phrase_encoded = tokenizer_phrase.encode_plus(phrase, max_length=max_size, pad_to_max_length=True)
        padded_list.append(phrase_encoded)
    return padded_list
```

```
def predict_batch_outputs(phrase_list):
    """
    Receive list of phrases and return model prediction WITHOUT softmax
    """
    with torch.no_grad():
        phrases_padded = padding(phrase_list)
        input_ids = torch.tensor([pad['input_ids'] for pad in phrases_padded])
        token_type_ids = torch.tensor([pad['token_type_ids'] for pad in phrases_padded])
        attention_mask = torch.tensor([pad['attention_mask'] for pad in phrases_padded])
        labels = torch.tensor([[1] for i in range(0, len(input_ids))])
        outputs = model_phrase(input_ids.to('cuda'), token_type_ids = token_type_ids.to('cuda'), attention_mask = attention_mask.to('cuda'), labels = labels.to('cuda'))
    return outputs[1].tolist()
```

My question is: I'm not seeing any difference in speed between two models. For instance, I have a script that read a csv dataset, break every row into a list of phrases using nltk, and then send such frases to a model (either Bert or Albert) and print it prediction. Using the same script, with the same methods, same dataset, and changing only which model is doing the prediction, I have the opposite result than expected: Bert can predict 11.89 docs per second, while Albert can predict 9.13 docs per second. I've done other tests and they also showed Albert being SLOWER than Bert.

Can someone share their experiences between Bert and Albert in matters of speed? Thanks.


Documentation and code mismatch in BertForMaskedLM forward method

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

BertForMaskedLM

Language I am using the model on (English, Chinese ...):

Not important.

The problem arises when using:
* [x] my own modified scripts: (give details below)

The tasks I am working on is:

I am trying to obtain LM probabilities for research purposes.

## To reproduce

Steps to reproduce the behavior:

1. According to the documentation, the code below should work as expected.

However, masked_lm_loss and ltr_lm_loss values are in fact in 2nd and 1st positions. This is apparent if the code is inspected, i.e. lm_labels related code is executed after masked_lm_labels related code.

See https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_bert.py#L1001-L1014

and

https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_bert.py#L1014

The following code is an excerpt from my code.

```python
model = BertForMaskedLM.from_pretrained('bert-base-cased')
with torch.no_grad():
    outputs = model(token_ids_in_sentence_tensor,
                    masked_lm_labels=token_ids_in_sentence_tensor,
                            lm_labels=token_ids_in_sentence_tensor,
                            token_type_ids=segments_tensors)
    masked_lm_loss = outputs[0]
    ltr_lm_loss = outputs[1]
    predictions = outputs[2]
```

## Expected behavior

Explained above.

## Environment info

- `transformers` version:
- Platform: Darwin XXX-3.local 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64
- Python version: 3.7.4
- PyTorch version (GPU?): no GPU, 1.3.1
- Tensorflow version (GPU?): 2.0.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


XLM-RoBERTa can't add new tokens.

Internals of the library; Tokenization.

# 🐛 Bug

Model I am using (Bert, XLNet ...): XLM-RoBERTa

## To reproduce
Steps to reproduce the behavior:

1. tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large', do_lower_case=True)
2. tokenizer.add_tokens(['<a1>', '</a1>', '<a2>', '</a2>'])
3. tokenizer.convert_tokens_to_ids('<a1>')

It always respond 1 as ids for the new tokens.
The problem seems connected to:
if (
                token != self.unk_token
                and self.convert_tokens_to_ids(token) == self.convert_tokens_to_ids(self.unk_token)
                and token not in to_add_tokens
            ):

in the second condition.
When it tries to get self.convert_tokens_to_ids(token) always returns 1 instead of 3.
3 is the id for unk_token.
1 is the id for pad.


Add LM capabilities to TFTransfoXLLMHead

Core: Modeling

Internals of the library; Models.

# 🚀 Feature request

It is currently not possible to generate language with the TFTransfoXLLMHeadModel because
the `lm_head` is not implemented (see https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_tf_transfo_xl.py#L745)

Doing:

```
from transformers import TFTransfoXLLMHeadModel
model = TFTransfoXLLMHeadModel.from_pretrained('transfo-xl-wt103')
model(tf.convert_to_tensor([[1, 5, 8]]))
```

currently leads to an error.

## Motivation

Pytorch has TransfoXLLMHead model working - TF should has well.

## Your contribution

@LysandreJik , @thomwolf  I could implement it if needed.


Add generate() functionality to TF 2.0

I added the `_generate_no_beam_search` functionality for TF 2.0. It works for the following models:
'gpt2', 'openai', 'xlnet', 'xlm', 'ctrl'. Only for the model 'transfo-xl' it doesn't work because the
lm_head is not implemented yet in TF 2.0 (added an issue here: #3064 ).

Also I checked whether the pytorch 'distilgpt2' and TF 2.0 'distilgpt2' generate the same output (added one Integration test for this). Will add other integration tests in a future PR.

Setting only certain indices to values is much less straight forward in TF 2.0 than in pytorch, which is why I added more code for the TF 2.0 version.

Would be very happy about some feedback @LysandreJik @thomwolf

EDIT: There was also a bug in TFCTRL where tf.concat uses a pytorch argument 'dim' instead of 'axis'

## TODO
- [x] Discuss how to change the test_torch_tf_conversion.py() @LysandreJik @thomwolf

## Future PR:
- [ ] Adapt all LMHead Integration Tests to greedy generate to be able to compare PT & TF
- [ ] Add generate() to TFTransfoXL (see Issue #3064 )


## ❓Should weight distribution change more when fine-tuning transformers-based classifier?
This question was posted on DataScience stack exchange:
[https://datascience.stackexchange.com/questions/68641/should-weight-distribution-change-more-when-fine-tuning-transformers-based-class](https://datascience.stackexchange.com/questions/68641/should-weight-distribution-change-more-when-fine-tuning-transformers-based-class)

## Details
I'm using pre-trained DistilBERT model with custom classification head, which is almost the same as in the [reference implementation ](https://github.com/huggingface/transformers/blob/fb560dcb075497f61880010245192e7e1fdbeca4/src/transformers/modeling_distilbert.py#L579)

```python
class PretrainedTransformer(nn.Module):
    def __init__(
        self, target_classes):
        super().__init__()
        base_model_output_shape=768
        self.base_model = DistilBertModel.from_pretrained("distilbert-base-uncased")
        self.classifier = nn.Sequential(
            nn.Linear(base_model_output_shape, out_features=base_model_output_shape),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(base_model_output_shape, out_features=target_classes),
        )

        for layer in self.classifier:
            if isinstance(layer, nn.Linear):
                layer.weight.data.normal_(mean=0.0, std=0.02)
                if layer.bias is not None:
                    layer.bias.data.zero_()

    def forward(self, input_, y=None):
        X, length, attention_mask = input_
        base_output = self.base_model(X, attention_mask=attention_mask)[0]
        base_model_last_layer = base_output[:, 0]
        cls = self.classifier(base_model_last_layer)
        return cls
```

During training, I use linear LR warmup schedule with max LR=5-e5 and cross entropy loss. In general, the model is able to learn on my dataset and reach high precision/recall metrics.

**My question is:**

Should weights distributions and biases in classification layers change more during training? It seems like the weights almost do not change at all, even when I do not initialize them as in the code (to mean=0.0 and std=0.02). Is this an indication that something is wrong with my model or it's just because the layers I've added are redundant and model does not learn nothing new?

Take look at the image of weight from the tensorboard:
<img width="1021" alt="Screenshot 2020-02-24 at 20 56 36" src="https://user-images.githubusercontent.com/6958772/75526050-4bbf6600-5a11-11ea-8d62-37407f968e06.png">


Bad word list for text generation

# 🚀 Feature request
Add a word of lists that you do not want the model to generate for whatever reason.

## Motivation
When creating a text generation model, especially if you will serve that model publicly, it is desirable to have assurances that a model is physically incapable of outputting certain tokens.

Such tokens would include profanity of all kinds, prejudicial language, or even just out-of-domain vocabulary.

## Your contribution

I am unsure as to your coding style guide, but I will detail how I would implement this below.

Firstly, you cannot simply set the offending word's values to `-Inf` as done here https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_utils.py#L1114 and here https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_utils.py#L1131
when calculating `top_k_top_p_filtering` as it would not capture multi-token words.

As an example, I will use the common Irish exclamation "feck" (see [here ](https://en.wikipedia.org/wiki/Feck) for an off-topic discussion of the word).

'feck' is tokenized into `['fe', 'ck']` in the `gpt2-xl` tokenizer. It would be unreasonable to simply set the logit values of any mention of the token `'fe'` or `'ck'` to -Inf as that would stop the creation of purely harmless words such as 'feasible' (`['fe', 'as', 'ible']`) and 'peck' (`['pe', 'ck']`). Therefore, I suggest a list of tokens that are not allowed at any one instance, and are updated, depending on the previous tokens.
This functionality would mean that a model would be allowed to generate the `'fe'` token, but would not be able to follow it with a `'ck'` token straight after.

I would operationalize this by having:
* a `bad_words_list` which the user has passed into the generate function
* a `possible_bad_words` dict which describes all the possible bad words that you can make with the current token history and how far along they are
* a `prohibited_tokens_list` which would prevent the model from choosing those tokens at a given time.

E.g.
Let's say that we pass the following list to the .generate function:
`bad_words_list = ["feck", "arse", "booze", "up yours"]`
which would be then tokenized and made into ids in the .generator function before the generation starts:
`bad_words_list = [tokenizer.encode(x, add_special_tokens=False) for x in bad_words_list]`

Then, the following function would be run just before the model outputs are obtained from both beam and no beam generators

```
def update_possible_bad_words(previous_token, bad_words_list, possible_bad_words):
    # Start with an empty list of prohibited bad words
    prohibited_tokens_list = []
    unmatched_bad_words= []

    # We cycle through the provided list of bad words, and if a bad word has only one token, then we add it to our prohibited list
    # Else, if a bad word starts with the token of our previous token, then we add it to our list of possible_bad_words
    for bad_word in bad_words_list:
        if len(bad_word) == 1:
            unmatched_bad_words.append(bad_word[0])
        elif previous_token == bad_word[0] and bad_word not in possible_bad_words.keys():
            possible_bad_words[bad_word] = 0

    # We cycle through all our possible bad words
    for bad_word in possible_bad_words.keys():
        bad_word_index = possible_bad_words[bad_word]

        # if the previous token matches the token currently indicated by the stored bad word index, then we increase this index by one.
        if previous_token == bad_word[bad_word_index]:
            new_bad_word_index = bad_word_index +1

            # If the length of the bad word is one greater than the currently stored bad word index, then that means that we need to stop the next token from being generated. Thus, we add it to the prohibited list. We also add this word to the unmatched_bad_words, as we can now consider deleting it from possible bad words as it has been potentially mitigated.
            if len(bad_word) == new_bad_word_index+1:
                prohibited_tokens_list.append(bad_word[-1])
                unmatched_bad_words.append(bad_word)

            # We set the dict value to be this new incremented index
            possible_bad_words[bad_word] = new_bad_word_index
        else:
           # Else if there is no match (I.e. the pattern of tokens created is different to the bad word) then we can mark this word as unmatched
            unmatched_bad_words.append(bad_word)

    # We cycle through all words that were not matched and we delete them from the possible_bad_words dict, as long as their starting token was not the previous token, in which case it could still be possible bad word.
    for unmatched_bad_word in unmatched_bad_words:
        if previous_token == unmatched_bad_word[0]:
            possible_bad_words[unmatched_bad_word] = 0
        else:
            del possible_bad_words[unmatched_bad_word]

    return prohibited_tokens_list
```
and I would call this function like so:
`prohibited_tokens_list = update_possible_bad_words(input_ids[-1], bad_words_list, possible_bad_words)`
here
https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_utils.py#L827
and here
https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_utils.py#L934
where `possible_bad_words` has been initialized as an empty dict directly before the generation loop.

Finally, we would pass `prohibited_tokens_list` to `top_k_top_p_filtering`
https://github.com/huggingface/transformers/blob/908fa43b543cf52a3238129624f502240725a6a6/src/transformers/modeling_utils.py#L1100
and would simply perform `logits[prohibited_tokens_list] = filter_value` before or after the top p and k filtering in that function.


How to init a subclass of BertForTokenClassification

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I want to build a subclass of BertForTokenClassification and also want to use the weights of pretrained model
`
class SeqLabelClassifier(BertForTokenClassification):

    def __init__(self,  pretrained_model_name, config):
        super(SeqLabelClassifier,self).__init__(config)
        self.lstm=nn.LSTM(...)

config = BertConfig()
model = SeqLabelClassifier(pretrained_model_name, config)
model = model.from_pretrained(args.pretrained_model_name, config=config)
`
But I get this error

>   File "/home/haoran/anaconda3/envs/nsd/lib/python3.8/site-packages/transformers/modeling_utils.py", line 466, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: __init__() missing 1 required positional argument: 'config'
How to correctly pass the args?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Bart-CNN

## Sources

- copy pastes code from generate but does not share very much in an effort to simplify. there is no big abstraction yet.
- also copy pastes some code from fairseq
- encoder_outputs and previous decoder attentions are cached.

## Differences with PretrainedModel.generate

these are all bc thats the way fairseq does it!
- BeamHypotheses(early_stopping=True)
- assumptions about various token_ids being present
- force decoder to start with EOS, then predict BOS
- decoder only considers the most recently generated token bc everything else is cached.
- prevent predictions of various special tokens at inopportune moments (all the -inf stuff)
- force eos if you hit max length
- max_length is about how many tokens you want to generate. Doesn't matter how many you have.
- min_len parameter to prevent short summaries
- no_ngram_repetition parameter (set to 3 in Bart-CNN) to prevent repetition.

## TODO
- [ ] docstrings
- [ ] Mystery: results are identical to fairseq 98.6% of the time, 1.4% of the time they differ by a few words.
- [ ] run rouge metrics, compare run time to fairseq.
- [ ] Resist pressure to make big seq2seq abstraction before there are more callers
- [ ] Deeper dive on the MaskedLM.tie_weights hack, what is right way to do it?


Fast tokenizers calculate wrong offsets when special characters are present

Core: Tokenization

Internals of the library; Tokenization.

Example:

```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("bert-base-cased", use_fast=True, add_special_tokens=False)
>>> sentence = "A, naïve [MASK] AllenNLP sentence."
>>> tokenized = t_fast.encode_plus(sentence, add_special_tokens=False, return_offsets_mapping=True)
>>> for start, end in tokenized['offset_mapping']:
...     print(repr(sentence[start:end]))
'A'
','
'naïve'
' [MASK'
' Alle'
'nN'
'L'
' sentenc'
'e'
```

As you can see, after the word "naïve", the offsets go off the rails.


Fast tokenizers don't properly tokenize special tokens

Internals of the library; Tokenization.

Slow tokenizers:

```
>>> import transformers
>>> t_slow = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=False)
>>> t_slow.encode_plus("A <mask> sentence.")
{'input_ids': [0, 83, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1]}

>>> t_slow.convert_ids_to_tokens([0, 83, 50264, 3645, 4, 2])
['<s>', 'ĠA', '<mask>', 'Ġsentence', '.', '</s>']
```

Fast tokenizers:
```
>>> import transformers
>>> t_fast = transformers.AutoTokenizer.from_pretrained("roberta-base", use_fast=True)
>>> t_fast.encode_plus("A <mask> sentence.")
{'input_ids': [0, 250, 1437, 50264, 3645, 4, 2],
 'token_type_ids': [0, 0, 0, 0, 0, 0, 0],
 'attention_mask': [1, 1, 1, 1, 1, 1, 1]}

>>> t_fast.convert_ids_to_tokens([0, 250, 1437, 50264, 3645, 4, 2])
['<s>', 'A', 'Ġ', '<mask>', 'Ġsentence', '.', '</s>']
```


(Gross WIP) Bart-CNN


Pipeline doc

This PR adds documentation to the pipelines and slightly modifies their behavior:

- `modelcard` is no longer available when using the `pipeline` factory method. As discussed with @thomwolf and @mfuntowicz, it doesn't serve any purpose for the user.
- All task-specific pipelines can now be instantiated without any model/tokenizer, instead relying on the defaults defined for the `pipeline` factory method.


Bart: Use bool attention_mask for encoder

Wasn't breaking because bool(-1e4) is True, but clearer this way.


Changes to NER examples for PLT and TPU

* Simplify the NER example to support new features added for us by pytorch-lightning.
* Pull out all rank and backend special casing in the code base.
* Setup data so that TPU examples work using the new code base.

Testing:
* Verify that standard examples of training work.
* Confirm that new TPU code works and runs https://colab.research.google.com/drive/1dBN-wwYUngLYVt985wGs_OKPlK_ANB9D

In my example, I see a 4x speedup over colab GPU and multi-gpu k40, but a slow down on loading and saving model. So certainly a win for larger datsets.


Is ALBERT the right implement from paper ?

I read ALBERT paper and code form Google: github.com/google-research/ALBERT. One of the main contribute of ALBert is cross-layer parameter sharing and i can see it on Google's code. But i can't see sharing on this code. Every Layers(or Blocks) are new object and their parameter will be different after fine-tuning.

Is the implement is wrong or i have misunderstanding about parameter sharing?


Adding Docker images for transformers + notebooks

Docker images are as follow:

- transformers-cpu (PyTorch + TF)
- transformers-gpu (PyTorch + TF)
- transformers-pytorch-cpu
- transformers-pytorch-gpu
- transformers-tensorflow-cpu
- transformers-tensorflow-gpu

Images are tagged according to the version of the library they bring and always use the latest version of DL frameworks.

Notebooks introduce:

- How to use tokenizers
- Overall transformers overview
- How to use pipelines

Currently notebooks are added to the repo, let's discuss internally if it's a good idea.


Should be able to turn off logging

# 🚀 Feature request

When doing a simple pipeline, I want to supress:
```
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 230/230 [00:00<00:00, 136kB/s]
convert squad examples to features: 100%|█████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 241.08it/s]
add example index and unique id: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 7037.42it/s]
```
## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

My code is pretty straightforward:
```
    args = parse_args()
    f = open(args.text_path, "r")
    context = f.read()
    # print(context)

    tokenizer = AutoTokenizer.from_pretrained(args.model)
    model = AutoModelForQuestionAnswering.from_pretrained(args.model)

    qa = pipeline('question-answering',
                  model='distilbert-base-uncased-distilled-squad', tokenizer='bert-base-cased')
    response = qa(context=context,
                  question=args.question)

    print(response['answer'])
```


Regarding attention received by the distilbert model

I receive the attention of all the 6 layers with all the 12 heads while exporting the tfdistilbert model. I just want to take the attention of dimension equal to the sequence length. Which layer and head attention is the most effective attention value that I should take in order to get the best value of attention scores with respect to my sentence.


Set specific hidden_size for ClassificationHead

# 🚀 Feature request

Add a config term `head_hidden_size` to the model configurations that will be used for the head of models such as `RobertaForSequenceClassification`

## Motivation

HuggingFace transformers library provides a very accessible API and PyTorch models that can be used "plug n play" for various task such as classification.
In many case, varying the hidden size of the last layer (that outputs the logits) is one of the first thing we tweak to improve the performance on such a task.

Currently, the dense layer uses the `hidden_size` config parameter, which is the same as the one used in the transformer (BERT). One cannot change the hidden size of the last layer without changing the hidden size of the entire transformer model behind it.

This means we have to code a new PyTorch module in order to do something as simple as that.

## Your contribution

I could PR this should the change be welcomed.


[WIP] deleted special tokens as attributes from model config

Given our conversation in #3011 , I thought about deleting all model special config attributes to make sure that no data is duplicated between the tokenizer of a model and the model.

There are two instances where config special attributes were used (e.g. self.config.pad_token_id)

1. the generate() function. But the generate function also takes all those tokens as attributes, so it should not at all rely on self.config.pad_token_id. This one is trivial to fix.

2. the bart model. It seems like the pad_token_id is actually an integral part of the bart model itself, so to me it seems very hard to disentangle the pad_token_id from the bart model.

I see three options:

1) Leave the code as it is and leave default attributes self.config.pad_token_id, self.config.bos_token_id, self.config.eos_token_id = None in the PretrainedConfig class.

2) Remove the self.config.pad_token_id, ... from the PretrainedConfig class, make the generate function independent of those variables, but add those variables to the BartConfig only.

3) Make the models completely independent from all special tokens. This probably would mean that the bart model class needs quite a lot of changes.

I tend to option 2) or 3). I like the idea of separation token ids from internal model behavior completely, but I cannot really estimate whether this is feasible for Bart (@sshleifer you probably have a better opinion on this).

What do you think? @LysandreJik , @julien-c , @thomwolf , @sshleifer


[WIP] add generate tests to more models

- [ ] add prepare_input_for_generation() to all bert masked lm models
- [ ] add slow integration tests to check results (also include camembert for this one)


[docs] Provide a barebones GPT-2 colab notebook

Please provide a barebones "pick up and go" GPT-2 colab notebook for text generation, [just like gpt-2-simple does](https://colab.research.google.com/drive/1VLG8e7YSEwypxU-noRNhsv5dW4NfTGce)


How can I use the this result?

Usage

General questions about the library

I wan to call the next sentence prediction function on new data.
 and this webpage tell it
https://stackoverflow.com/questions/55111360/using-bert-for-next-sentence-prediction

 input_ids = torch.LongTensor([[31, 51, 99], [15, 5, 0]])
    input_mask = torch.LongTensor([[1, 1, 1], [1, 1, 0]])
    token_type_ids = torch.LongTensor([[0, 0, 1], [0, 1, 0]])
    config = BertConfig(vocab_size_or_config_json_file=32000, hidden_size=768,
        num_hidden_layers=12, num_attention_heads=12, intermediate_size=3072)
    model = BertForNextSentencePrediction(config)
    seq_relationship_logits = model(input_ids, token_type_ids, input_mask)

but when I run this demo and get the result ( seq_relationship_logits) ,
(tensor([[-0.0728,  0.1863],
        [ 0.3190, -0.1528]], grad_fn=<AddmmBackward>),)
how to use it as the predict result （like if sentence B follows sentence A, so the predict label is 0,else the label is 1）


Issue with Makefile

# 🐛 Bug

## Information

Model I am using (Bert ...):

Language I am using the model on (English):

The problem arises when using:
* [x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. pip install -e ".[testing]"
2. pip install -r examples/requirements.txt
3. make test-examples

Running 3 throws a few errors and warnings including a   AttributeError: 'TestCaseFunction' object has no attribute 'get_marker'. Wasn't happening a few days ago. What should I do?


Fix spelling of strictly in error messages


Fix batch_encode_plus

Fix #3037


Knowledge distillation from internal representation GPT2

# ❓ Questions & Help

I am trying to implement above paper for the GPT2 model. The attention outputs are softmax probabilities as seen from modeling_gpt2.py line (152-162). If KLD loss is computed on these values from teacher and student I am getting negative value of KLD indicating that the inputs are not probability distributions.
The attention output is of  dimensions (bs, nh, sl, sl) nh=12 I am just flattening the output and computing kld loss.

Is my understanding correct.?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Paragraph re-ranking using MS MARCO dataset

General questions about the library

How can I use transformers package to train on MS MARCO dataset  by adding my own domain data. Or can I use pre trained model to add my own domain data?

SO link: https://stackoverflow.com/questions/60424723/paragraph-re-ranking-using-ms-marco-dataset


AttributeError: 'Model2Model' object has no attribute 'prepare_model_kwargs' in 2.5.1

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Model2Model

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)
based on the quick start guide for Model2Model.
if I create a model using any of the following, it get an exception during the forward call :
```
model = Model2Model.from_pretrained('bert-base-uncased')

model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased','bert-base-uncased')

decoder_config = BertConfig.from_pretrained('bert-base-uncased', is_decoder=True)
model = PreTrainedEncoderDecoder.from_pretrained('bert-base-uncased', 'bert-base-uncased', decoder_config=decoder_config)
```

```
model(torch.tensor([[10,20,300,4,500,600]]).cuda(), torch.tensor([[400,500]]).cuda(), decoder_lm_labels=torch.tensor([[400,500]]).cuda())[0]
```
this started happening in 2.5.1
2.5.0 didn't throw the error

stacktrace:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-13-77add3526cdd> in <module>()
----> 1 model(torch.tensor([[10,20,300,4,500,600]]).cuda(), torch.tensor([[400,500]]).cuda(), decoder_lm_labels=torch.tensor([[400,500]]).cuda())[0]

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_encoder_decoder.py in forward(self, encoder_input_ids, decoder_input_ids, **kwargs)
    221             kwargs: (`optional`) Remaining dictionary of keyword arguments.
    222         """
--> 223         kwargs_encoder, kwargs_decoder = self.prepare_model_kwargs(**kwargs)
    224
    225         # Encode if needed (training, first prediction pass)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    574                 return modules[name]
    575         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 576             type(self).__name__, name))
    577
    578     def __setattr__(self, name, value):

AttributeError: 'Model2Model' object has no attribute 'prepare_model_kwargs'
```
## Expected behavior

No error should occur.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: google colab
- Python version: 3.6
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Wrong logic in `batch_encode_plus()`

# 🐛 Bug

I'm trying to use `batch_encode_plus()` with already tokenized ID as input.

So I have a `list of list of int` as input. For example :

```python
x = [[0, 83, 313, 11, 10551, 2278, 16, 2183, 1958, 804, 7, 916, 11, 13933, 982, 4, 286, 68, 5046, 6, 37, 40, 3627, 231, 2697, 9, 1958, 11, 41, 36027, 27894, 1001, 21466, 424, 2233, 4, 2], [0, 20, 291, 212, 13989, 191, 3772, 42, 983, 4, 815, 34, 1714, 8617, 187, 63, 17692, 11, 8008, 4, 993, 864, 549, 1492, 2624, 5391, 9686, 8, 12291, 240, 7, 464, 4, 2]]
```

When I call `batch_encode_plus()` with this input, I receive `AssertionError` :

```
tokenizer.batch_encode_plus(x)
# >>> line 1130, in batch_encode_plus
#                    assert len(ids_or_pair_ids) == 2
```

---

After reviewing the code, it seems there is an error in the logic. Here :

https://github.com/huggingface/transformers/blob/b370cc7e99c5b8c7436154d4694c33b461ea0f08/src/transformers/tokenization_utils.py#L1128-L1137

since I gave only tokenized ID and not a tuple of tokenized ID (not a pair), the logic should bring me to the `else` clause. But instead, I enter the `if` clause, leading to the assertion error.

**It's currently not possible to use `batch_encode_plus()` with already tokenized text / IDs** if the input is not a pair.

---

I think this line :

https://github.com/huggingface/transformers/blob/b370cc7e99c5b8c7436154d4694c33b461ea0f08/src/transformers/tokenization_utils.py#L1129

Should be changed to this :

```python
if isinstance(ids_or_pair_ids, (list, tuple)) and len(ids_or_pair_ids) == 2:
```


Cannot use `transformers.GradientAccumulator` with `tf.function`

TensorFlow

Anything TensorFlow

# 🐛 Bug

I'm using gradient accumulation to get larger batch sizes when pretraining BERT, and GradientAccumulator works wonderfully. Thanks for including that!

However, tf.AutoGraph fails when I wrap in the gradient accumulation step in tf.function. I have to write code like this:

```code
@tf.function
def batch_step():
    step_grads = ...
    return step_grads

@tf.function
def allreduce():
    grads = [hvd.allreduce(grad) for grad in gradient_accumulator.gradients]
    optimizer.apply_gradients(grads)
    gradient_accumulator.reset()

def step():
    for i in range(gradient_accumulation_steps):
        step_grads = batch_step(dataset[i])
        gradient_accumulator(step_grads)

    allreduce()
```

This code works. However, it is very inefficient because TensorFlow stores the GradientAccumulator on CPU rather than GPU. My PCI throughput is around 9 GiB/s (on 8-GPU single-node training) when using gradient accumulation, as opposed to in the kilobytes when everything is wrapped in a tf.function. GPU utilization also drops because the bottleneck is transfer of gradients between GPU and CPU. I also suspect that it's causing a memory leak somewhere, but that's another story :) Being able to wrap the GradientAccumulator in a tf.function (or somehow pin it to the GPU, using `tf.device()` doesn't seem to be working) would be wonderful. Any tips?

Sample PCI throughput, ideally RX (GPU-to-CPU) should be close to 0.
<img width="682" alt="Screen Shot 2020-02-26 at 11 21 47 AM" src="https://user-images.githubusercontent.com/4564897/75380114-5223d580-588b-11ea-9594-e85755ac0db0.png">

## Information

Model I am using (Bert, XLNet ...): TFAlbert

Language I am using the model on (English, Chinese ...): English

## Environment info

- `transformers` version: 2.5.0
- Platform: Ubuntu 18.04
- Python version: 3.6.9
- PyTorch version (GPU?): None
- Tensorflow version (GPU?): 2.1
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes, but it also applies to single-node


Force pad_token_id to be set before padding for standard tokenizer

I think batch_encode_plus with a proper padding strategy should not be allowed if the pad_token_id is not set. I don't feel like it helps the user to have a python list of lists with None values that he can't transform to a torch.Tensor anyways.

As a remedy I think it is alright if a new pad_token is added or whether it is set to an existing special_token.

This behavior is already enforced for FastTokenizer, so the PR should also make it easier to transition from Tokenizer to FastTokenizer.

I will fix the tests and add a new one if you guys agree.

@mfuntowicz @thomwolf @LysandreJik


fix several typos in Distil* readme

Hi, feel free to reject if you think these changes aren't necessary. Just saw a couple typos while reading the documentation and wanted to help 😄. The first change looks a bit hard to tell what changed, but it was 'superseeds' to 'supersedes'.


Fix attn mask gpt2 when using past

- fixed issue #3031
- updated doc-string for GPT2
- added two test for GPT2 that I think are important:
        - check that when using past as an input to speed up decoding, the results are equivalent to not using past.
       - check that when using past and attn_mask as an input to speed up decoding, the results are equivalent to not using past where as an input_id slice was corrupted that is masked by the attn_mask.


Loading custom weights for BERT in pytorch

Is it possible to load custom pre-trained weights for BERT, other than the one you provide ?


Forward pass with GPT2 using both past and attention_mask as an input leads to dimension error

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce run the following code:

```
from transformers import GPT2LMHeadModel
import torch

model = GPT2LMHeadModel.from_pretrained('gpt2')

input_ids = torch.tensor([8, 8, 0, 50256, 50256]).unsqueeze(0)
attn_mask = torch.tensor([1, 1, 1, 0, 0]).unsqueeze(0)

# first step there is no past so lets get it from the model and append new embedding id to inputs and extend the attn_mask
logits_output, past = model(input_ids, attention_mask=attn_mask)
next_token = torch.argmax(logits_output[:, -1, :]).unsqueeze(0)
input_ids = torch.cat([input_ids, next_token.unsqueeze(-1)], dim=-1)
attn_mask = torch.cat([attn_mask, torch.ones((attn_mask.shape[0], 1)).long()], dim=1)

# now we have a past so we can use it to speed up training
model_inputs = model.prepare_inputs_for_generation(input_ids=input_ids, past=past)
logits_output, past = model(**model_inputs, attention_mask=attn_mask)  # this leads to an error which it should not
```

## Expected behavior

No error should be thrown and the forward pass to get `logits_output` and `past` should work

## Environment info

- `transformers` version: 2.5.1
- Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


run_tf_glue with AdamW optimizer and distributed training

# ❓ Questions & Help
Can you please update the run_tf_glue example (Tensorflow version) that includes the Adam optimizer with Warm-up and weight decay.  If you can also show how to run the code on multiple GPUs and on TPU.


How to add data to  pretrained model.

Usage

General questions about the library

I want to add wiki data to pretrained model.

   i.e., kor wiki data is added to XLMRoberta Model weight.

I think that files in /transformers/templates/adding_a_new_model do

Is it right?

please let me know way to add data.


TensorFlow

Anything TensorFlow

This is the model that I have defined:

```
input_layer = keras.layers.Input(shape = (attention_mask.shape[1],), dtype='int64')
bert = DistilBertModel.from_pretrained("distilbert-base-cased")(input_layer)
bert = bert[0][:,0,:]
# bert = keras.layers.Dense(units=10, activation='relu')(bert)
classifier = keras.layers.Dense(units=1, activation='sigmoid')(bert)
model = keras.models.Model(inputs=input_layer, outputs=classifier)
model.summary()
```

This is the error I am getting.
```
AttributeError                            Traceback (most recent call last)

<ipython-input-12-6d7e88036056> in <module>()
      1 input_layer = keras.layers.Input(shape = (attention_mask.shape[1],), dtype='int64')
----> 2 bert = DistilBertModel.from_pretrained("distilbert-base-cased")(input_layer)
      3 bert = bert[0][:,0,:]
      4 # bert = keras.layers.Dense(units=10, activation='relu')(bert)
      5 classifier = keras.layers.Dense(units=1, activation='sigmoid')(bert)

1 frames

/usr/local/lib/python3.6/dist-packages/transformers/modeling_distilbert.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds)
    449             raise ValueError("You cannot specify both input_ids and inputs_embeds at the same time")
    450         elif input_ids is not None:
--> 451             input_shape = input_ids.size()
    452         elif inputs_embeds is not None:
    453             input_shape = inputs_embeds.size()[:-1]

AttributeError: 'Tensor' object has no attribute 'size'
```

The same code works fine when distilbert is replaced with bert. What to do in this case.


This class and module cannot be found

Core: Tokenization

Internals of the library; Tokenization.

I want to use BertForNextSentencePrediction in ' modeling_bert.py' line 1020  ,but when I run Examples in line 1065,  error happens

  File "E:\work\pycharm\transformers-master\src\transformers\tokenization_bert.py", line 24, in <module>
    from tokenizers import BertWordPieceTokenizer
ImportError: No module named 'tokenizers'

where I can find this module tokenizers,
thanks !!


language_modeling.py doesn't continue from last global step

Ex: LM (Finetuning)

Related to language modeling fine-tuning

# 🐛 Bug
Hey, The checkpoint's suffix is the last optimization step rather than the last global step (I'm working with accumulation steps)
## Information

The problem arises when using:
* [ ] the official example scripts: language_modeling.py

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: language_modeling.py

## To reproduce
Steps to reproduce the behavior:

1. run a model on language_modeling.py script with an accumulation step > 0
2. save a checkpoint after x > 0 steps and exit
3. try to continue training and it will continue from the last optimization step rather than global step

```
roberta-base-openai-detector, roberta-large-openai-detector). Assuming 'tmlm_roberta_output/checkpoint-480' is a path, a model identifier, or url to a directory containing tokenizer files.
02/26/2020 07:39:49 - INFO - transformers.tokenization_utils -   Didn't find file tmlm_roberta_output/checkpoint-480/added_tokens.json. We won't load it.
02/26/2020 07:39:49 - INFO - transformers.tokenization_utils -   loading file tmlm_roberta_output/checkpoint-480/vocab.json
02/26/2020 07:39:49 - INFO - transformers.tokenization_utils -   loading file tmlm_roberta_output/checkpoint-480/merges.txt
02/26/2020 07:39:49 - INFO - transformers.tokenization_utils -   loading file None
02/26/2020 07:39:49 - INFO - transformers.tokenization_utils -   loading file tmlm_roberta_output/checkpoint-480/special_tokens_map.json
02/26/2020 07:39:49 - INFO - transformers.tokenization_utils -   loading file tmlm_roberta_output/checkpoint-480/tokenizer_config.json
02/26/2020 07:39:49 - INFO - transformers.modeling_utils -   loading weights file tmlm_roberta_output/checkpoint-480/pytorch_model.bin
init tud head
02/26/2020 07:40:08 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=512, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=True, do_train=True, eval_all_checkpoints=False, eval_data_file='/specific/netapp5_2/gamir/advml19/yuvalk/project/transformers/examples/lm_data/wiki.test.raw.time_filter.normalized', evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=64, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=True, mlm_probability=0.15, model_name_or_path='tmlm_roberta_output/checkpoint-480', model_type='roberta', n_gpu=4, no_cuda=False, num_train_epochs=1.0, output_dir='tmlm_roberta_output', overwrite_cache=False, overwrite_output_dir=True, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=1, save_steps=80, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=True, tokenizer_name=None, train_data_file='/specific/netapp5_2/gamir/advml19/yuvalk/project/transformers/examples/lm_data/wiki.train.raw.time_filter.normalized', warmup_steps=0, weight_decay=0.0)
02/26/2020 07:40:08 - INFO - __main__ -   Loading features from cached file /specific/netapp5_2/gamir/advml19/yuvalk/project/transformers/examples/lm_data/roberta_cached_lm_510_wiki.train.raw.time_filter.normalized
02/26/2020 07:40:16 - INFO - __main__ -   ***** Running training *****
02/26/2020 07:40:16 - INFO - __main__ -     Num examples = 163046
02/26/2020 07:40:16 - INFO - __main__ -     Num Epochs = 1
02/26/2020 07:40:16 - INFO - __main__ -     Instantaneous batch size per GPU = 1
02/26/2020 07:40:16 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 256
02/26/2020 07:40:16 - INFO - __main__ -     Gradient Accumulation steps = 64
02/26/2020 07:40:16 - INFO - __main__ -     Total optimization steps = 636
02/26/2020 07:40:16 - INFO - __main__ -     Continuing training from checkpoint, will skip to saved global_step
02/26/2020 07:40:16 - INFO - __main__ -     Continuing training from epoch 0
02/26/2020 07:40:16 - INFO - __main__ -     Continuing training from global step 480
02/26/2020 07:40:16 - INFO - __main__ -     Will skip the first 480 steps in the first epoch
```

## Expected behavior
I expect it to run from the last global step. I.e. optimization steps * gradient accumulation steps. Note that optimization steps == checkpoint suffix.

## I made the following changes and it seems to work ok:

Former code:
```global_step = int(checkpoint_suffix)
            epochs_trained = global_step // (len(train_dataloader) // args.gradient_accumulation_steps)
            steps_trained_in_current_epoch = global_step % (len(train_dataloader) // args.gradient_accumulation_steps)```
New code:
            ```global_step = int(checkpoint_suffix) * args.gradient_accumulation_steps
            epochs_trained = global_step // len(train_dataloader)
            steps_trained_in_current_epoch = global_step % (len(train_dataloader))```

- `transformers` version: latest
- Platform:
- Python version: 3.7
- PyTorch version (GPU?): GPU
- Using GPU in script?: yes


Why do I run example/run_ner.py with no output

Need more information

Further information is requested

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
### This is the log I ran. Any questions? Please help me

![image](https://user-images.githubusercontent.com/9248572/75306701-c1e29200-5884-11ea-9ffe-07c02375413e.png)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Fix (non-slow) tests on GPU (torch)

Fixes ~29 failing tests

Also cf previous commit from december: https://github.com/huggingface/transformers/pull/2055/commits/61978c1dd3f340a545e74537c3dae41a4514e867

(not sure why T5 and the common methods were not failing back then)


BART : host `bart-large-cnn`

# 🌟 New model addition

BART pretrained model `bart-large` is currently provided, as well as fine-tuned BART model `bart-large-mnli`.

**How about [`bart-large-cnn`](https://github.com/pytorch/fairseq/tree/master/examples/bart#pre-trained-models) ?**


Make format consistent with that of PreTrainedTokenizer


Can GPT2LMHeadModel do batch inference with variable sentence lengths?

Given GPT2 tokenizer do not have an internal pad_token_id, how do I pad sentences and do batch inference using GPT2LMHeadModel?
Specifically my code as:
```
prompt_text = [
    'in this paper we',
    'we are trying to',
    'The purpose of this workshop is to check whether we can', ]

tokens = [tokenizer.convert_tokens_to_ids(tokenizer.tokenize(x, add_prefix_space=True)) for x in prompt_text]

inputs = pad_sequence([torch.LongTensor(x) for x in tokens], batch_first = True, padding_value=tokenizer.eos_token_id)

outputs, past = model(input_ids=inputs, attention_mask=None)
```
This will return non-relevant predictions since GPT2 will consider the eos_tokens and start a new sentence in the batch.

Can anyone please share sample codes that using GPT2LMHeadModel to do batch inference with various sentence lengths?

Thanks!


[ci] Run all tests on (self-hosted) GPU


Delete Model2Model

- the quickstart code doesn't work
- the tests don't test a forward pass

if you need it `git checkout e8ce63ff`


[WIP] Updates to simplify PLT example and use new features

Some changes to PLT to support our workflow. Prelim support for TPUs.

Still testing.


[WIP] Update to use new pytorch-lightning features

Updates to utilize a bunch of new features the PLT wrote to support us. Also support for TPU. Still in testing phase.


Use my own pretrained BERT model

Usage

General questions about the library

How can I use my own pretrained BERT model in SQUAD finetuing? How to do this in the code? Can anyone provide any instructions?


Latest version of transformers available via conda?

# 🚀 Feature request

I notice that the version of transformers available via conda-forge is quite outdated (v2.1.1). Could you make later versions available there too? Thanks!


Add integration tests for xlm roberta modelling and xlm roberta tokenzier

It's quite easy to get real numbers for the XLM-R moder from [fairseq](https://github.com/pytorch/fairseq/tree/master/examples/xlmr), so I added integration tests for `xlm_roberta_modeling.py` and `xlm_roberta_tokenization.py`

Since `XLMRobertaModel` is the same model as `RobertaModel`, I think intergration tests for `xlm_roberta_modeling.py` are enough.
Regarding `XLMRobertaTokenizer` there were no tests so far, so in this file there should definitely be more "fast" tests. I would need some help on those (@mfuntowicz ?).

Regarding the results of the integration tests:

The tests for `xlm_roberta_modeling.py` all pass.
One of the two tests (called hard_token_symbols) for `xlm_roberta_tokenization.py' fails. @LysandreJik @mfuntowicz


Generation with gpt-2

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Hi @julien-c and all,
we are developing a training with GPT-2 language model based on our custom dataset.
Firstly, we train a custom Tokenizer with tokenizers ( as described [here](https://github.com/huggingface/tokenizers/issues/166) ) .
Now I have a dummy GPT-2 model trained on 1 epoch but my problem is on inference.
The result is always the same when I feed the model with the starting text.

Code (is the same at link )
```
from transformers import GPT2LMHeadModel, GPT2Tokenizer
import torch

tokenizer = GPT2Tokenizer.from_pretrained("my-gpt2-tokenizer")
model = GPT2LMHeadModel.from_pretrained('my-gpt2-model')

generated = tokenizer.encode("The Manhattan bridge")
context = torch.tensor([generated])
past = None

for i in range(100):
    print(i)
    output, past = model(context, past=past)
    token = torch.argmax(output[..., -1, :])

    generated += [token.tolist()]
    context = token.unsqueeze(0)

sequence = tokenizer.decode(generated)
print(sequence)
```
Using this code, the output is repeated:

```
The Manhattan bridge<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit<nl> I'm a little bit
```
I think that problems are two basically:
1. the first is that for every inference we take only the argument that maximizes probability, and I don't use any type of sampler based on some type of distribution.
2. the second is that the content fed after the first step is always the next word predicted , so it goes in loop.

I don't have any other ideas. Is there an example of inference with huggingface lib?
Thanks


batch_encode_plus with pad_to_max_length is not padding the output

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (BertTokenizer,):

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts.

The tasks I am working on is:
* [ ] Simple batch tokenization task

## To reproduce
```
from transformers import BertTokenizer, BertForQuestionAnswering
import torch

import numpy as np

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

qt1 = ('What is the name of the repository?', 'pipeline have been included in the huggingface/transformers repository')
qt2 = ('What is the name of the repository?', 'What can be done with this???')

inp_raw = [qt1, qt2]
tck_temp = tokenizer.batch_encode_plus(inp_raw, max_length=20, pad_to_max_length=True)

print(tck_temp)
inp_ids = tck_temp['input_ids']
tck_type_ids = tck_temp['token_type_ids']

print(len(inp_ids[0]) == len(inp_ids[1]))
## This is coming false
```

## Expected behavior

The code snippet should print True
But it prints False

## Environment info

- `transformers` version: 2.3
- Platform: Linux
- Python version: 3.6
- PyTorch version (1.4CPU)

[EDIT] : correct transformers version


Add models special tokens to its pretrained configs

I think the token_ids for each specific model should also be added to their pretrain configs. This would also make the function generate() much easier to use for the user.

If ok, I can also add these tokens for models not having a LMHead.


Possible improvement of padding logic in generate

I wanted to try out a approach to the padding logic (previously discussed with @thomwolf and @mfuntowicz) which is **different** to what is currently done (see #2885 description).

Instead of setting the `pad_token_id` to `eos_token_id` if `pad_token_id` is not defined and `eos_token_id` is defined, the following logic could be applied:

If there is no `pad_token_id`, the user has to add the `pad_token_id` to the tokenizer and resize the model token embedding matrix to add an additional token vector to the weight matrix.

During my PR today, I encountered two problems:

1. Adding a token embedding vector causes some pretrained model (`gpt2` and `openai-gpt`) often generate the newly added token embedding vector (since input & output token embeddings are tied). A remedy for this is to simply always set the produced logits of the pad_token_id to -Inf. This works as it doesn't generate the `pad_token`, but it changes the internal behavior of `gpt2` and `openai-gpt` however.

2. TransfoXL uses an adaptive token embedding matrix, which means that the function `resize_embedding_matrix` produces an error when used with `TransfoXLHeadModel`. This means the `token_embedding_matrix` of TransfoXL cannot be changed! Therefore, this approach doesn't work at all for TransfoXL (which is a pity as TransfoXL generates very good language).

In the current PR, some tests fail (TransfoXL and the slow tests of gpt2 and openai-gpt). I wanted to hear your opinion on this logic vs. the logic that is implemented now (see #2885) @thomwolf, @LysandreJik and @mfuntowicz


add crf layer to BERT, RoBERTa, XLMR

Building on pull request [ bert（+lstm）+crf #2249 ], I make it more generic and similar to run_ner.py . There is a CRF layer adapted from https://github.com/jiesutd/LatticeLSTM , which does not have additional package requirements, and was published in ACL 2018.


[WIP] Remove tokenizers dependency

In v2.5.1 the tokenizers don't default to the fast implementations (as introduced in v2.5.0), which I think is a good thing when looking at the issues that have arisen from it. Even though the performance of the new tokenizers is phenomenal (and I complement everyone who has been working on it!), it seems a bit premature to make `tokenizers` a dependency. (In addition, see for instance this topic concerning installation issues: https://github.com/huggingface/transformers/issues/2980.)

Even though the fast implementation isn't the default any more, it is still part of the dependencies in setup.py. This PR removes `tokenizers` from the dependency list but indicates in the documentation that having `tokenizers` installed and using `use_fast` can result in great performance improvements.

Generally, I am not satisfied with how this PR has been implemented (quite some duplication across the different tokenizers), but I don't really see another way. Alternative ideas are definitely welcome. If, on the other hand, you decide to keep the dependency, that is fine too.

Note: my flake8 keeps failing with an obscure error so can't do the manual code checking now. Might try again later.

Note: tests will fail for the fast tokenizers (and perhaps on some other imports). I'll look further into this if you decide that this PR is welcome.


[Benchmark] Pipeline for question answering

Core: Pipeline

Internals of the library; Pipeline.

# 🖥 Benchmarking `transformers`

## Benchmark

I'm trying to benchmark QA model with `bert-large-uncased-whole-word-masking-finetuned-squad`. But it seems it is extremely slow e.g. 3-4 sec for 1 question with 2 contexts.

I feel there is something I'm missing in pipeline.

## Sample Code:
```
def answer(self, contexts:List[str], question:str, **kwargs):
   ## tokenizer, model, pipeline all are cached in actual implementation
  ## via [reify](https://docs.pylonsproject.org/projects/pyramid/en/latest/api/decorator.html)
  ## so model loading is not a problem.
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', max_len=500)
    model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-
        finetuned-squad')
    pipeline('question-answering', model=model, tokenizer=tokenizer)
    pipeline_input = []
    for c in contexts:
        pipeline_input.append({
            'question' : question,
            'context' : c
        })
    answers = pipeline(pipeline_input)
```

## Set-up
CPU: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
memory: 16GB


[FIX] not training when epoch is small

Closes https://github.com/huggingface/transformers/issues/2995

Solving bug where for small epochs and large gradient_accumulation_steps we never train.
Explained further here: https://github.com/huggingface/transformers/issues/2995


Output of pipeline feature extraction

Internals of the library; Pipeline.

Hi,

I am using the new pipeline feature of transformers for feature extraction and I have to say it's amazing. However I would like to alter the output of the pipeline slightly but I am not sure how to and I was hoping some people of the community could point me into the right direction.

I am using the following code snippet in my script:

```
nlp = pipeline('feature-extraction', model=args.model, config=args.config, tokenizer=args.tokenizer, device=args.device)

features = nlp(value)
features = np.squeeze(features)
features = features.astype('float32')

h5f = h5py.File(args.output_path, 'a')
h5f.create_dataset(key, data=features)
h5f.close()
```
The output for every value I put into the pipeline has the shape of (512, 768) no matter the length of the sentence I put into the pipeline.

I understand the shape of 512 is caused by padding and 768 because of the number of hidden states.

However I would like to have the output to be (15, 768) or (312, 768) for example depending on the input instead of always (512, 768). I know this is not standard but for my purpose I need this format.

Could someone please point me to the right direction how to achieve this?

Thanks a lot! I'm really at a loss here.


BART : How can I train and evaluate BART on CNN/DM dataset

# ❓ Questions & Help

@sshleifer

I found examples for text summarization on CNN/DM dataset using BERT, but I couldn't find any example using BART.

**Are you going to add it later, or update the existing example to add BART, or it's not scheduled ?**


Test correct tokenizers after default switch


Tokenizer Fast False by default


Changing the loss function in BERT

Usage

General questions about the library

# ❓ Questions & Help

Hello,

I'm trying to replace the _CrossEntropyLoss_ with the _KLDivLoss_ for a sentence classification task using some golden logits and the logits from the BERT model.

```
golden_logits = …

outputs = model(**inputs)

# the logits from BERT
logits = outputs[1]

loss_fct = KLDivLoss(reduction='sum')
loss = loss_fct(F.log_softmax(logits, dim=-1), F.softmax(golden_logits, dim=-1))
```

Am I doing this correctly ?