cannot import name 'MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING'

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): ENglish

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
Run the run_ner.py script on examples/ner/

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Add t5 summarization example

Adds TF 2.0 Example for T5 summarization.

Adds dataset download file via `tensorflow_datasets` and a rouge scorer.

Example is currently being tested on T5-large on GPU to see how rouge scorer performs in comparsion to `examples/summarization/bart` rouge scorer.


Added precisions in SciBERT-NLI model card

model card

Related to pretrained model cards

Sorry that I have to do a second PR for this model card, but I forgot to include some precisions about the training process that are undoubtedly useful in order to reproduce my results!

- Added training time and training hardware
- Added lowercasing and Max. Seq. Length to parameters table


Add right model and tokenizer path in example

Related to pretrained model cards


[model_cards] 🇹🇷 Add new BERTurk models

Related to pretrained model cards

Hi,

this PR adds three new BERT models for Turkish:

* `dbmdz/bert-base-turkish-uncased` - uncased model with a vocab size of 32k
* `dbmdz/bert-base-turkish-128k-cased` - cased model with a vocab size of 128k
* `dbmdz/bert-base-turkish-128k-uncased` - uncased model with a vocab size of 128k

Models (incl. `tokenizer_config.json`) are already uploaded to the model hub :)

Results are coming soon in the [BERTurk repository](https://github.com/stefan-it/turkish-bert)!


AdamW in HuggingFace is different from AdamW in Pytorch

# ❓ Question

I just noticed that the implementation of AdamW in HuggingFace is different from PyTorch. The previous AdamW first updates the gradient then apply the weight decay. However, in the paper (Decoupled Weight Decay Regularization, link: https://arxiv.org/abs/1711.05101) and the implementation of Pytorch, the AdamW first apply the weight decay then update the gradient.

I was wondering if the two approaches are the same. Thanks! (In my opinion, they are not the same procedure.)

HuggingFace:
```python
for group in self.param_groups:
    for p in group["params"]:
        ...
        # Decay the first and second moment running average coefficient
        # In-place operations to update the averages at the same time
        exp_avg.mul_(beta1).add_(1.0 - beta1, grad)
        exp_avg_sq.mul_(beta2).addcmul_(1.0 - beta2, grad, grad)
        denom = exp_avg_sq.sqrt().add_(group["eps"])

        step_size = group["lr"]
        if group["correct_bias"]:  # No bias correction for Bert
            bias_correction1 = 1.0 - beta1 ** state["step"]
            bias_correction2 = 1.0 - beta2 ** state["step"]
            step_size = step_size * math.sqrt(bias_correction2) / bias_correction1

        p.data.addcdiv_(-step_size, exp_avg, denom)

        # Just adding the square of the weights to the loss function is *not*
        # the correct way of using L2 regularization/weight decay with Adam,
        # since that will interact with the m and v parameters in strange ways.
        #
        # Instead we want to decay the weights in a manner that doesn't interact
        # with the m/v parameters. This is equivalent to adding the square
        # of the weights to the loss with plain (non-momentum) SGD.
        # Add weight decay at the end (fixed version)
        if group["weight_decay"] > 0.0:
            p.data.add_(-group["lr"] * group["weight_decay"], p.data)
```
Pytorch:
```python
for group in self.param_groups:
    for p in group['params']:
        ...
        # Perform stepweight decay
        p.data.mul_(1 - group['lr'] * group['weight_decay'])

        exp_avg, exp_avg_sq = state['exp_avg'], state['exp_avg_sq']
        if amsgrad:
            max_exp_avg_sq = state['max_exp_avg_sq']
        beta1, beta2 = group['betas']

        state['step'] += 1
        bias_correction1 = 1 - beta1 ** state['step']
        bias_correction2 = 1 - beta2 ** state['step']

        # Decay the first and second moment running average coefficient
        exp_avg.mul_(beta1).add_(1 - beta1, grad)
        exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
        if amsgrad:
            # Maintains the maximum of all 2nd moment running avg. till now
            torch.max(max_exp_avg_sq, exp_avg_sq, out=max_exp_avg_sq)
            # Use the max. for normalizing running avg. of gradient
            denom = (max_exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])
        else:
            denom = (exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])

        step_size = group['lr'] / bias_correction1

        p.data.addcdiv_(-step_size, exp_avg, denom)
```


Model cards for CS224n SQuAD2.0 models

Related to pretrained model cards

For the following models:
* elgeish/cs224n-squad2.0-albert-base-v2
* elgeish/cs224n-squad2.0-albert-large-v2
* elgeish/cs224n-squad2.0-albert-xxlarge-v1
* elgeish/cs224n-squad2.0-distilbert-base-uncased
* elgeish/cs224n-squad2.0-roberta-base


Glue test processors and predictions

Adress #3176

- Adds a function to load the test dataset for each GLUE task processor.
- Update `run_glue.py` example script to add a `--do_test` flag for producing test set predictions in a `.tsv` file, submittable to the [GLUE scoreboard](https://gluebenchmark.com/).
- Adds a couple extra feature flags to `run_glue.py` that don't need to stay.


[Bart]example---BartForConditionalGeneration

**when I run your example:**
from transformers import BartTokenizer, BartForConditionalGeneration, BartConfig
model = BartForConditionalGeneration.from_pretrained('bart-large-cnn')
tokenizer = BartTokenizer.from_pretrained('bart-large-cnn')
ARTICLE_TO_SUMMARIZE = "My friends are cool but they eat too many carbs."
inputs = tokenizer.batch_encode_plus([ARTICLE_TO_SUMMARIZE], max_length=1024, return_tensors='pt')
print(inputs)
summary_ids = model.generate(inputs['input_ids'], attention_mask=inputs['attention_mask'], num_beams=4, max_length=5)
print([tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids])

**model :**
https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/pytorch_model.bin

**the results:**
{'input_ids': tensor([[    0,  1308,   964,    32,  3035,    53,    51,  3529,   350,   171,
         33237,     4,     2]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
Traceback (most recent call last):
  File "/home/qwh/桌面/OpenNMT/bart.py", line 17, in <module>
    summary_ids = model.generate(inputs['input_ids'], attention_mask=inputs['attention_mask'], num_beams=4, max_length=5)
  File "/home/qwh/.local/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
**TypeError: generate() got an unexpected keyword argument 'attention_mask'**

thank you!


[examples] Use AutoModels in more examples

still to-do (non-exhaustive list):
- [ ] run_multiple_choice
- [ ] run_xnli
- [ ] test_hans
- [ ] run_mmimdb
- [ ] (maybe) run_generation


[WIP] seq2seq example

This PR presents an example seq2seq use case and bug fixes necessary for this to execute with reasonable accuracy.

The utils_seq2seq.py file defines the data format for training data, and the run_seq2seq.py file takes training, development, and test data and produces a model. The README.md discusses how to execute this toy problem. The specific toy problem in use here is formatting a date string to the American style, which is a trivial example. On my local setup using GPUs, this example executes within 5 minutes. Production models should include more learnings.

I welcome feedback about how to strengthen performance here and the best route to increase testing.

This relies on a few bug fixes which have been incorporated in this branch
- Without a fix for #3038, PreTrainedEncoderDecoder won't instantiate at all.
- Without a fix for #2435, BERT models fail completely on this use case as the BERT decoder isn't instantiated correctly without CrossAttention in that case.
- ~I strongly suspect that the input to the decoder in the PreTrainedEncoderDecoder class is incorrect as present in the code base, and commit https://github.com/huggingface/transformers/commit/9fcf73afbcfa18918234592039da7bd409820431 has a proposed fix. It doesn't make sense to have the expected token ids as input to the decoder when the decoder needs to learn how to decode from the embeddings.~ Incomplete understanding - will fix


added_tokens.json is used for splitting texts

Core: Tokenization

Internals of the library; Tokenization.

Should Fix

This has been identified as a bug and should be fixed.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Run this script to save a pre-trained vocab, add some vocab using `added_tokens.json`. Then create a new tokenizer with the combined vocabs and use it to tokenize a sentence.

```
import os

from transformers import BertTokenizer

model_name = 'bert-base-uncased'
tokenizer_path = 'tmp'

if not os.path.exists(tokenizer_path):
    os.makedirs(tokenizer_path)

tokenizer = BertTokenizer.from_pretrained(model_name)

tokenizer.save_vocabulary(tokenizer_path)

with open(tokenizer_path + '/added_tokens.json', 'w') as f:
    f.write('{"ver": 30522, "rw": 30523}')

tokenizer = BertTokenizer.from_pretrained(tokenizer_path)

s = "i want to overwrite ubuntu with windows"

a = tokenizer.tokenize(s)
print(a)
```
Output run 1:
```
['i', 'want', 'to', 'o', '##ve', 'rw', 'rite', 'u', '##bu', '##nt', '##u', 'with', 'windows']
```

Ouptut run 2:
```
['i', 'want', 'to', 'o', 'ver', 'write', 'u', '##bu', '##nt', '##u', 'with', 'windows']
```
Cause of the problem:

`added_tokens.json` is [merged](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L668) with `all_special_tokens`, and then used to [split](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L815) the input text. Since the merged tokens is stored in an unordered set, the splitting process is non-deterministic for each run.

For example, these are how the text is split in different runs:

Split Run 1 (use `rw`):
```
['i want to ove', 'rw', 'rite ubuntu with windows']
```

Split Run 2 (use `ver`):
```
['i want to o', 'ver', 'write ubuntu with windows']
```
Possible solution:
Instead of `self.unique_added_tokens_encoder`, use `set(self.all_special_tokens)` to [split the text](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L814)

## Expected behavior
Using `added_tokens.json` to split the text seems to be a bug to me. I expect only a small set of special tokens in the `special_tokens_map.json` should be used for this purpose.

In general it is helpful to let a tokenizer behave deterministically across multiple runs. Otherwise it will be bad for certain downstream task such as [sentence embedding ](https://github.com/UKPLab/sentence-transformers) because one sentence can be encoded in many different ways. This is in particular problematic if the number of added_vocab is big.

## Environment info

wget https://download.pytorch.org/whl/cu100/torch-1.4.0%2Bcu100-cp36-cp36m-linux_x86_64.whl
wget https://files.pythonhosted.org/packages/7e/90/6141bf41f5655c78e24f40f710fdd4f8a8aff6c8b7c6f0328240f649bdbe/torchvision-0.5.0-cp36-cp36m-manylinux1_x86_64.whl

virtualenv -p /usr/bin/python3.6 venv && . venv/bin/activate && find . -maxdepth 1 -name "*.whl" | xargs pip install && pip install -r requirements.txt

requirements.txt:
transformers==2.5.1
tensorboardX==2.0
scikit-learn==0.22.2

- `transformers` version: 2.5.1
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): Y
- Tensorflow version (GPU?): N
- Using GPU in script?: Y
- Using distributed or parallel set-up in script?: N


[Bart: example] drop columns that are exclusively pad_token_id from input_ids

reasoning: These columns slow down computation, but do not change output.
impact: this reduces the runtime to compute EVAL on the CNN examples from 2h to 1:37 before any other changes.

I'm open to putting this as a method on `PretrainedTokenizer` if others find it useful.

@joeddav you might find this useful.

### Code for easy copy paste
```python
def trim_batch(
    input_ids, pad_token_id, attention_mask=None,
):
    """Remove columns that are populated exclusively by pad_token_id"""
    keep_column_mask = input_ids.ne(pad_token_id).any(dim=0)
    if attention_mask is None:
        return input_ids[:, keep_column_mask]
    else:
        return (input_ids[:, keep_column_mask], attention_mask[:, keep_column_mask])
```


Trying to train a GPT2 from scratch

Hi !

I am trying to use a GPT2 architecture for musical applications and consequently need to train it from scratch. After a bit of googling I found that the issue #1714 already had "solved" the question but when I try the to run
```Python
from transformers import GPT2Config, GPT2Model

NUMLAYER = 4
NUMHEAD = 4
SIZEREDUCTION = 10 #the factor by which we reduce the size of the velocity argument.
VELSIZE = int(np.floor(127/SIZEREDUCTION)) + 1
SEQLEN=40 #size of data sequences.
EMBEDSIZE = 5

config = GPT2Config(vocab_size = VELSIZE, n_positions = SEQLEN, n_embd = EMBEDSIZE, n_layer = NUMLAYER, n_ctx = SEQLEN, n_head = NUMHEAD)
model = GPT2Model(config)
 ```
I get the following error :
```Python
Traceback (most recent call last):

  File "<ipython-input-7-b043a7a2425f>", line 1, in <module>
    runfile('C:/Users/cnelias/Desktop/PHD/Swing project/code/script/GPT2.py', wdir='C:/Users/cnelias/Desktop/PHD/Swing project/code/script')

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 786, in runfile
    execfile(filename, namespace)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\spyder_kernels\customize\spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/Users/cnelias/Desktop/PHD/Swing project/code/script/GPT2.py", line 191, in <module>
    model = GPT2Model(config)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 355, in __init__
    self.h = nn.ModuleList([Block(config.n_ctx, config, scale=True) for _ in range(config.n_layer)])

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 355, in <listcomp>
    self.h = nn.ModuleList([Block(config.n_ctx, config, scale=True) for _ in range(config.n_layer)])

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 223, in __init__
    self.attn = Attention(nx, n_ctx, config, scale)

  File "C:\Users\cnelias\Anaconda3\lib\site-packages\transformers\modeling_gpt2.py", line 109, in __init__
    assert n_state % config.n_head == 0
```
What does it mean and how can I solve it ?

Also more generally, is there a documentation on how to do a forward call with the GPT2 ? Can I define my own ```train()``` function or do I have to use the model's build-in function ? Am I forced to use a ```Dataset``` to do the training or can I feed it individual tensors ?
I looked for it but couldn't find answer to these on the doc, but maybe I missed something

EDIT : Yes, I have already read the blogpost on ```huggingface.co``` but it omits too much informations and details to be usefull for my application :(


[Bart] Fix: put dummy_inputs on correct device

This fixes `test_dummy_inputs`, which was failing on GPU because dummy inputs were put on CPU even if model was on GPU.


Supported language information by model

Hi there,

Another feature/documentation request. I am evaluating the language support of all pre-trained models distributed via huggingface :)

I have quickly looked into the code and happily found that XLNet and FlauBERT models have that information: https://github.com/huggingface/transformers/search?q=lang&unscoped_q=lang

Do you plan in the short term to add an `available_languages` attribute to all pre-trained models?

If not, happy to do that investigation and share results.

Cheers,

Alex


Cannnot Import from transformers

# 🐛 Bug

## Information
I am trying to import  TFGPT2LMHeadModel in transformers.Python giving below error.
cannot import name 'TFGPT2LMHeadModel' from 'transformers
## To reproduce
import tensorflow as tf
from transformers import TFGPT2LMHeadModel,GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")

# add the EOS token as PAD token to avoid warnings
model = TFGPT2LMHeadModel.from_pretrained("gpt2", pad_token_id=tokenizer.eos_token_id)

# encode context the generation is conditioned on
input_ids = tokenizer.encode('I enjoy walking with my cute dog', return_tensors='tf')

# generate text until the output length (which includes the context length) reaches 50
greedy_output = model.generate(input_ids, max_length=50)

print("Output:\n" + 100 * '-')
print(tokenizer.decode(greedy_output[0], skip_special_tokens=True))

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


🚀 Feature request Multimodal BERT Models

Hello, it would be great if more **multimodal** BERT models are included in the library. I have noticed that MMBT from Facebook is provided; however, I was unable to find some guidelines about how to make it work with the help of 🤗 Transformers.

Possible models can be [VilBERT](https://arxiv.org/abs/1908.02265), [VL-BERT](https://arxiv.org/abs/1908.08530), [VisualBERT](https://arxiv.org/abs/1908.03557), [VideoBERT](https://arxiv.org/abs/1904.01766) and so on.

Best regards.


Add comparison table with new models

model card

Related to pretrained model cards


Create README.md

Related to pretrained model cards


Add comparison table with older brother in family

Related to pretrained model cards


Create card for the model

Related to pretrained model cards


adding --fp16 to run_language_modeling and increase batch size but cuda out of memory error

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


🐛Bugs in run_tf_ner.py

Found a bug in [run_tf_ner.py](https://github.com/huggingface/transformers/blob/master/examples/ner/run_tf_ner.py) at line 170 and 325:

```python
loss_fct = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE)
```
refer to [tf.keras.losses.SparseCategoricalCrossentropy](https://www.tensorflow.org/api_docs/python/tf/keras/losses/SparseCategoricalCrossentropy)

```python
tf.keras.losses.SparseCategoricalCrossentropy(
    from_logits=False, reduction=losses_utils.ReductionV2.AUTO,
    name='sparse_categorical_crossentropy'
)
```

> `from_logits`: Whether y_pred is expected to be a logits tensor. By default, we assume that y_pred encodes a probability distribution. Note: Using from_logits=True may be more numerically stable.

So I think `loss_fct` should be init with `from_logits=True` if the `TFBertForTokenClassification` just return pure logits rather than softmax output.


Lazy text dataset loading for language modelling with PyTorch

#3083

Added a lazy text dataset using linecache to run_language_modeling.py. Slightly refactored collate_fn construction to accommodate the different collate functions needed for a lazy dataset vs a in-memory dataset.


Finetuning of T5 on SQuAD 1.1 including code examples

Hi, I am implementing T5 model on the SQuAD 1.1 dataset.

When I fine-tune the model with the Adam or AdaFactor optimizier, validation accuracy goes down.
But, the training accuracy goes up.

Could you give to me any advice for me?

I feed into the model as below:
  input_ids = ['(QUESTION)', Q_W1, Q_W2, ..., '(CONTEXT)', C_W1, C_W2, ..., '(PAD)', ...]
  attention_masks = [1, 1, 1, ..., 1, 1, 1, ..., 0, ...]
  decoder_input_ids = ['(PAD)', W1, W2, ..., '(PAD)', '(PAD)', ...]
  decoder_attention_masks = [1, 1, 1, ..., 0, 0, ...]
  lm_labels = [W1, W2, ..., '(EOS)', '(PAD)', ...]

I matched the shape between 'decoder_input_ids' and 'lm_labels'. (Shift doesn't be used.)
And, '(PAD)' in 'lm_labels' is converted into -100 in a process of the loss calculation.

For the generation, 'decoder_input_ids' is generated from the decoder except for an initial token '(PAD)'.

The result from the 'T5-Small' pretrained weights on the dataset is

{
  "exact": 71.03122043519394,
  "f1": 81.08158598580584,
  "total": 10570,
  "HasAns_exact": 71.03122043519394,
  "HasAns_f1": 81.08158598580584,
  "HasAns_total": 10570
}


Model conversion from PyTorch to TF2 doesn't work properly for ALBERT

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

The model conversion script `convert_pytorch_checkpoint_to_tf2.py` seems to be not working properly for ALBERT models.
It fails on the pre-trained models officially released by Google which are converted to PyTorch models with `convert_albert_original_tf_checkpoint_to_pytorch.py`.

## To reproduce

```
$ wget https://storage.googleapis.com/albert_models/albert_base_v2.tar.gz
$ tar xzf albert_base_v2.tar.gz
$ cd albert_base/
$ python -m transformers.convert_albert_original_tf_checkpoint_to_pytorch --tf_checkpoint_path model.ckpt-best --pytorch_dump_path ./pytorch_model.bin --albert_config_file albert_config.json
$ python -m transformers.convert_pytorch_checkpoint_to_tf2 --tf_dump_path ./ --model_type albert --pytorch_checkpoint_path ./pytorch_model.bin --config_file albert_config.json --compare_with_pt_model
...
Max absolute difference between models outputs 17.709423065185547
Traceback (most recent call last):
  File "/home/m-suzuki/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/m-suzuki/.pyenv/versions/3.7.4/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 499, in <module>
    only_convert_finetuned_models=args.only_convert_finetuned_models,
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 428, in convert_all_pt_checkpoints_to_tf
    compare_with_pt_model=compare_with_pt_model,
  File "/home/m-suzuki/.pyenv/versions/Python-3.7.4/lib/python3.7/site-packages/transformers/convert_pytorch_checkpoint_to_tf2.py", line 351, in convert_pt_checkpoint_to_tf
    assert diff <= 2e-2, "Error, model absolute difference is >2e-2: {}".format(diff)
AssertionError: Error, model absolute difference is >2e-2: 17.709423065185547
```

Same error for ALBERT v1 models.

## Expected behavior

Max absolute difference between models outputs should be <= 2e-2

## Environment info

Observed on both of the following environments:

- `transformers` version: 2.5.1
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
<!-- -->
- `transformers` version: 2.5.1
- Platform: Linux-4.15.0-58-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


minor website fix

On this page [https://huggingface.co/transformers/notebooks.html](https://huggingface.co/transformers/notebooks.html)

The first link is fine. The others give 404. Just thought you would like to know


gpt2 - convert examples to features(tensorflow 2)

I'm trying to fine-tune GPT2 to generate shakespeare text.
I have variable "train_examples", which is list of InputExamples:
```
>> print(train_examples)
 <__main__.InputExample at 0x7f55e0fafd68>,
 <__main__.InputExample at 0x7f55e0fafda0>,
 <__main__.InputExample at 0x7f55e0fafef0>,
 <__main__.InputExample at 0x7f55e0fafeb8>,
 <__main__.InputExample at 0x7f55e0f6aeb8>,
```
I created the examples using the following function:
```
    def _create_examples(self, lines, set_type):
        """Creates examples for the training and dev sets."""
        examples = []
        for (i, line) in enumerate(lines):
            guid = "%s-%s" % (set_type, i)
            #guid = i
            text_a = line[1]
            examples.append(
                InputExample(guid=guid, text_a=text_a))
        return examples

class InputExample(object):
    def __init__(self, guid, text_a):
        self.guid = guid
        self.text_a = text_a
```

As I understand I need to convert the type to 'features', before I call the fit function. But how can I convert the examples to features? I saw many cases for BERT, but I couldn't find for gpt2.

I tried:
```
from transformers import glue_convert_examples_to_features

input_train_tensor_data = glue_convert_examples_to_features(train_examples, gpt2_tokenizer, max_length=128, task='mrpc')
```

But got:
```
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-118-d04221b84b4f> in <module>()
      1 from transformers import glue_convert_examples_to_features
      2
----> 3 input_train_tensor_data = glue_convert_examples_to_features(train_examples, gpt2_tokenizer, max_length=128, task='mrpc')

/usr/local/lib/python3.6/dist-packages/transformers/data/processors/glue.py in glue_convert_examples_to_features(examples, tokenizer, max_length, task, label_list, output_mode, pad_on_left, pad_token, pad_token_segment_id, mask_padding_with_zero)
    120
    121         if output_mode == "classification":
--> 122             label = label_map[example.label]
    123         elif output_mode == "regression":
    124             label = float(example.label)

KeyError: None
```


Clean Encoder-Decoder models with Bart/T5-like API and add generate possibility

Bert-Bert Encoder-Decoder models can now be used as is shown in the test cases:
`tests/test_modeling_encoder_decoder.py`.
Tests include:
- forward `input_ids` and `decoder_input_ids` for Bert-Bert
- backprop using masked language model loss for Bert-Bert
- backprop using "conventional" language model loss for Bert-Bert
- using the `generate()` fn with Bart-Bart
- saving and loading of Encoder-Decoder models.

Before merging a couple of things have to be agreed on as mentioned in the comments below.

UPDATE:

This branch is IMO now fully functional for Bert-2-Bert models.
I will finish the PR (clean the code, make a pretty docstring, etc...) once we agreed on the issues I mentioned further down. Would be very happy if you can review @thomwolf @LysandreJik @sshleifer @julien-c @yjernite


When I used the add_special_tokens function in the BertTokenizer, it assigns 2 different tokens with the same ID. Is this done on purpose?

Need more information

Further information is requested


[BART] test_dummy_inputs fails on GPU

```
RuntimeError: Expected object of device type cuda but got device type cpu for argument #3 'index' in call to _th_index_select

```
Easy fix, but putting here so I don't forget!


Can't save DistilBert model.

Model:
```
input_layer = tf.keras.layers.Input(shape = (attention_mask.shape[1],), dtype='int64')
bert = TFDistilBertModel.from_pretrained("distilbert-base-cased")(input_layer)
bert = bert[0][:,0,:]
bert = tf.keras.layers.Dense(units=20, activation='relu')(bert)
classifier = tf.keras.layers.Dense(units=train_y.shape[1], activation='softmax')(bert)
model = tf.keras.models.Model(inputs=input_layer, outputs=classifier)
model.summary()
```

After training when I try to save the model using:
```
# serialize model to JSON
model_json = model.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("model.h5")
print("Saved model to disk")
```

I am getting this error:
```
File "train_DistilBERT_model.py", line 138, in <module>
    model_json = model.to_json()
  File "/home/v-mohit.saini/anaconda3/envs/flask/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1254, in to_json
    model_config = self._updated_config()
  File "/home/v-mohit.saini/anaconda3/envs/flask/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1232, in _updated_config
    config = self.get_config()
  File "/home/v-mohit.saini/anaconda3/envs/flask/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 918, in get_config
    return copy.deepcopy(get_network_config(self))
  File "/home/v-mohit.saini/anaconda3/envs/flask/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1993, in get_network_config
    layer_config = serialize_layer_fn(layer)
  File "/home/v-mohit.saini/anaconda3/envs/flask/lib/python3.6/site-packages/tensorflow_core/python/keras/utils/generic_utils.py", line 198, in serialize_keras_object
    config = instance.get_config()
  File "/home/v-mohit.saini/anaconda3/envs/flask/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 917, in get_config
    raise NotImplementedError
NotImplementedError
```

How to fix this?


Data Processor should not include in the package

# 🚀 Feature request

For a clearly, flexible packages, I think there is no need including data processor for certain data.

## Motivation

In the past, it was easy using transformers to do research, since the module is explicit and clear.
When reading example, it was easy tracing the code to apply on a new dataset.
However, recently transformers merged some unrelated module into the package like Data Processor, making hard to modify the preprocess stage.


test_resize_tokens_embeddings does not inspect `get_output_embeddings`

Help wanted

Extra attention is needed, help appreciated

Here is the existing logic: copied below for convenience
https://github.com/huggingface/transformers/blob/bbf26c4e619cf42106163e1e2cd5ff98b936ff93/tests/test_modeling_common.py#L489)

```python
model_embed = model.resize_token_embeddings(config.vocab_size)
cloned_embeddings = model_embed.weight.clone()
# Check that resizing the token embeddings with a larger vocab size increases the model's vocab size
model_embed = model.resize_token_embeddings(model_vocab_size + 10)
self.assertEqual(model.config.vocab_size, model_vocab_size + 10)
self.assertEqual(model_embed.weight.shape[0], cloned_embeddings.shape[0] + 10)
```

Since we never test the return value of `get_output_embeddings` after resizing, the model can avoid setting it to the new vocab size.
BART did this by overwriting `tie_weights` to do nothing (fix proposed in https://github.com/huggingface/transformers/pull/3323)


RobertaTokenizer doesn't have 'batch_encode_plus'

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RoBERTa

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Following the tutorial on how to train your own RoBERTa model in this [link](https://huggingface.co/blog/how-to-train)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

Fill mask

## To reproduce
Steps to reproduce the behavior:

1. I pretrained my own tokenizer and roberta model
2.  The tokenizer and model are loaded fine and both seems to have the training information
3. However, when I join them together in the  pipeline step as in:
`tokenizer = RobertaTokenizer.from_pretrained('./eo_data')`
`rmodel = RobertaForMaskedLM.from_pretrained('./output_dir')`
`fill_mask = pipeline(
    "fill-mask",
    model=rmodel,
    tokenizer= tokenizer
)
`

 I get the following error:
`
AttributeError: 'RobertaTokenizer' object has no attribute 'batch_encode_plus'
`
It seems that RobertaTokenizer doesn't have the batch_encode_plus function as BertTokenizer

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Ubuntu
- Python version: 3.6.8
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

Thank you


Added scibert-nli model card

model card

Related to pretrained model cards


Add camembert integration tests

Add integration tests for camembert comparing results to original fairseq code.


closed

closed


Add example code for CRF heads

# Add example `crf/` with heads that force structural output dependencies.

(Mostly a note to myself as a side project)

## Model description

As requested in: https://github.com/huggingface/transformers/pull/3009 there are some tasks and languages where it is useful to have final layer structural dependencies.

Using https://github.com/harvardnlp/pytorch-struct/ we can add these with minimal changes to the code and no new model parameters.

Target:
* example code for ner / parsing (sota).


BERT pretrained checkpoints

# ❓ Questions & Help

Did you pretrain the BERT cased checkpoints with huggingface or convert it from google's checkpoints?


[Bart/Memory] Two separate, smaller decoder attention masks

### Background

The bart decoder requires two masks: one to ignore padding tokens, the other (`causal_mask`), to avoid attending to future tokens during training.
Previously, `_prepare_bart_decoder_inputs` combined these two masks into one  float_mask of shape `(bsz, 1, tgt_len, tgt_len)` filled with -inf for tokens that should be ignored. This mask was subsequently added to the attention activations.

Now, we return the two masks separately:
`decoder_padding_mask`: shape `(bs, tgt_len)`, `bool`
`causal_mask`: shape `(tgt_len, tgt_len)`, `float`

### Impact
saves 800 MB for bs=6, tgt_len=1024, with negligible speed impact.

### Notes
- The distinct data types (bool and float) are used to minimize code change.


[Seq2Seq Generation] Call encoder before expanding input_ids

Proposing to call model.encoder before expanding `input_ids` to `effective_batch_size*num_beams`.

For Bart, this saves 1.5 GB of GPU mem on batch_size=6. Savings probably similar for T5 (untested).

Requires knowing which index of the encoder_outputs is associated with the batch dim (we need to expand this dimension), which is different between `Bart` and `T5`. This difference is encoded in the `self.encoder_outputs_batch_idx` variable.

This PR is WIP because `encoder_outputs_batch_idx` could be avoided if we transposed Bart's encoder_outputs, which I haven't tried.


[Bart/Memory] SelfAttention only returns weights if config.output_attentions

**Previously**, `SelfAttention` would always return `attn_weights`, and then `BartDecoder` and `BartEncoder` would decide whether to return them to the user.
The `attn_weights` tensor is fairly large, with shape = `(bs, num_heads, tgt_len, src_len)`
This meant that the memory allocated for `attn_weights` could not be freed until after the forward pass of `BartDecoder`.

Now: `SelfAttention` returns (output, None) if `config.output_attentions=False` and the memory can be freed

Impact: memory can be freed after SelfAttention returns. -600MB peak GPU consumption for batch_size=6, tgt_len=src_len=1024, num_heads=16

Speed impact: negligible


Why does huggingface bert pooler hack make mixed precission training stable?

# ❓ Questions & Help

## Details

Huggigface BERT implementation has a hack to remove the pooler from optimizer.

https://github.com/huggingface/transformers/blob/b832d5bb8a6dfc5965015b828e577677eace601e/examples/run_squad.py#L927
```
# hack to remove pooler, which is not used
# thus it produce None grad that break apex
param_optimizer = [n for n in param_optimizer if 'pooler' not in n[0]]
```
We are trying to run pretraining on huggingface bert models. The code always diverges later during the training if this pooler hack is not applied. Everytime the reason is apex loss scaler becomes zero.
After using the above hack, there is no divergence issue seen.

The pooler layer is a FFN with tanh activation
```
class BertPooler(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.dense = nn.Linear(config.hidden_size, config.hidden_size)
        self.activation = nn.Tanh()

    def forward(self, hidden_states):
        # We "pool" the model by simply taking the hidden state corresponding
        # to the first token.
        first_token_tensor = hidden_states[:, 0]
        pooled_output = self.dense(first_token_tensor)
        pooled_output = self.activation(pooled_output)
        return pooled_output
```
I even tried replacing the tanh acrivation with GELU and adding layer norm in the pooler layer. But the loss scaler became zero even faster.

My question is why this pooler hack solves numeric instability?

**https://stackoverflow.com/questions/60743907/why-does-huggingface-bert-pooler-hack-make-mixed-precission-training-stable**:


[Generate] Add bad words list argument to the generate function

The `bad_words_ids` argument allows to insert a list of lists of `input_ids` that cannot be generated, *e.g.* bad words.

That's a proposed feature request (I think there were actually multiple ones):
#3061

Also adds tests for all language models to verify behavior.


GPT2TokenizerFast does not preserve special tokens' ids after a save and load.

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2 Fast Tokenizer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

This problem only happens with `GPT2TokenizerFast` and not `GPT2Tokenizer`

## To reproduce

Steps to reproduce the behavior:

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
from transformers import GPT2TokenizerFast

tokenizer = GPT2TokenizerFast.from_pretrained('gpt2')
print('special tokens: ', tokenizer.additional_special_tokens, tokenizer.additional_special_tokens_ids)
tokenizer.add_special_tokens({'additional_special_tokens': ['<special_token>'], 'pad_token': '<pad>'})
print('special tokens: ', tokenizer.additional_special_tokens, tokenizer.additional_special_tokens_ids)
print(tokenizer.pad_token, tokenizer.convert_tokens_to_ids(tokenizer.pad_token))
tokenizer.save_pretrained('./save_dir/')

tokenizer = GPT2TokenizerFast.from_pretrained('./save_dir/')
print('special tokens: ', tokenizer.additional_special_tokens, tokenizer.additional_special_tokens_ids)
print(tokenizer.pad_token, tokenizer.convert_tokens_to_ids(tokenizer.pad_token))
```

It outputs

```
special tokens:  [] []
special tokens:  ['<special_token>'] [50257]
<pad> 50258
special tokens:  ['<special_token>'] [50258]
<pad> 50257
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

```
special tokens:  [] []
special tokens:  ['<special_token>'] [50257]
<pad> 50258
special tokens:  ['<special_token>'] [50257]
<pad> 50258
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux-5.4.19-100.fc30.x86_64-x86_64-with-fedora-30-Thirty
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False


fixes lr_scheduler warning

For more details, see https://pytorch.org/docs/stable/optim.html#how-to-adjust-learning-rate


Generate all possible sentences using a fine-tuned GPT-2 model

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

Is there a way to generate all possible sentences using a fine-tuned GPT-2 model given a certain sampling technique? For some reason I wan to exhaust all possible combination of tokens given a fine-tuned GPT-2 model with certain sampling technique. Is it doable? If it is not, how do we get an estimate of how many possible sentences are there in the latent space?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Added total_save_limit feature similar to run_langauge_modeling.py

Added args.total_save_limit in order to save only the last specific checkpoints similar to the feature in run_langauge_modeling.py. This might be helpful for a student like me who has limited space storage quota on the school's remote server.


New model, new model cards

model card

Related to pretrained model cards

Trained another squad model! Added details in card.


TF Camembert not improving over epochs

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
`jplu/tf-camembert-base`

Language I am using the model on (English, Chinese ...):
French

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.  Get a custom multi-class dataset with imbalanced data
2. Train TFCamembertForSequenceClassification on this dataset
3. Try with and without `class_weight` or under-sample biggest classes (accuracy and loss change but still don't improve over epochs)

```python
import tensorflow as tf
from transformers import TFCamembertForSequenceClassification, CamembertTokenizer

model = TFCamembertForSequenceClassification.from_pretrained("jplu/tf-camembert-base", num_labels=len(labels))
tokenizer = CamembertTokenizer.from_pretrained("jplu/tf-camembert-base")

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08, clipnorm=1.0),
    loss=tf.keras.losses.CategoricalCrossentropy(from_logits=True),
    metrics=["accuracy"],
)

model.fit(
    custom_generator(), # generator yield encoded sample (by tokenizer) and encoded label (by OneHotEncoder)
    epochs=10,
    max_queue_size=2,
    steps_per_epoch=25,
    #class_weight=class_weights,
    validation_data=custom_generator(),
    validation_steps=4
)
```

## Expected behavior

The classifier should improve over each epochs. In this case it stay at the same accuracy and loss, it just vary with more or less 5% accuracy.
To compare, I tried to run the same code but with `TFFlaubertForSequenceClassification.from_pretrained("jplu/tf-flaubert-base-cased")` and it worked as expected.

## Environment info

- `transformers` version: 2.5.1
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12 (Google AI Platform)
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

For information, I already posted this problem on [Stack Overflow](https://stackoverflow.com/questions/60761761/hugging-face-transformer-classifier-fail-on-imbalance-dataset) which lead me here.


# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2
Language I am using the model on (English, Chinese ...): eng

**The tasks I am working on is:**

!python /content/transformers/examples/run_language_modeling.py --train_data_file=shakespeare.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=output --do_train

## To reproduce

Steps to reproduce the behavior:

import os
import requests
file_name = "shakespeare.txt"
if not os.path.isfile(file_name):
	url = "https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt"
	data = requests.get(url)

	with open(file_name, 'w') as f:
		f.write(data.text)
!python /content/transformers/examples/run_language_modeling.py --train_data_file=shakespeare.txt --model_type=gpt2 --model_name_or_path=gpt2 --output_dir=output --do_train

03/20/2020 13:36:52 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 1, distributed training: False, 16-bits training: False
03/20/2020 13:36:53 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at /root/.cache/torch/transformers/4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.699bbd1c449e9861456f359d6daa51bd523ac085b4b531ab0aad5a55d091e942
03/20/2020 13:36:53 - INFO - transformers.configuration_utils -   Model config GPT2Config {
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": null,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": null,
  "finetuning_task": null,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_epsilon": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_beams": 1,
  "num_labels": 2,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

03/20/2020 13:36:54 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at /root/.cache/torch/transformers/f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
03/20/2020 13:36:54 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at /root/.cache/torch/transformers/d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
03/20/2020 13:36:54 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at /root/.cache/torch/transformers/4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
03/20/2020 13:37:02 - INFO - __main__ -   Training/evaluation parameters Namespace(adam_epsilon=1e-08, block_size=1024, cache_dir=None, config_name=None, device=device(type='cuda'), do_eval=False, do_train=True, eval_all_checkpoints=False, eval_data_file=None, evaluate_during_training=False, fp16=False, fp16_opt_level='O1', gradient_accumulation_steps=1, learning_rate=5e-05, line_by_line=False, local_rank=-1, logging_steps=500, max_grad_norm=1.0, max_steps=-1, mlm=False, mlm_probability=0.15, model_name_or_path='gpt2', model_type='gpt2', n_gpu=1, no_cuda=False, num_train_epochs=1.0, output_dir='output', overwrite_cache=False, overwrite_output_dir=False, per_gpu_eval_batch_size=4, per_gpu_train_batch_size=4, save_steps=500, save_total_limit=None, seed=42, server_ip='', server_port='', should_continue=False, tokenizer_name=None, train_data_file='shakespeare.txt', warmup_steps=0, weight_decay=0.0)
03/20/2020 13:37:02 - INFO - __main__ -   Loading features from cached file gpt2_cached_lm_1024_shakespeare.txt
03/20/2020 13:37:02 - INFO - __main__ -   ***** Running training *****
03/20/2020 13:37:02 - INFO - __main__ -     Num examples = 330
03/20/2020 13:37:02 - INFO - __main__ -     Num Epochs = 1
03/20/2020 13:37:02 - INFO - __main__ -     Instantaneous batch size per GPU = 4
03/20/2020 13:37:02 - INFO - __main__ -     Total train batch size (w. parallel, distributed & accumulation) = 4
03/20/2020 13:37:02 - INFO - __main__ -     Gradient Accumulation steps = 1
03/20/2020 13:37:02 - INFO - __main__ -     Total optimization steps = 83
Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/83 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/content/transformers/examples/run_language_modeling.py", line 799, in <module>
    main()
  File "/content/transformers/examples/run_language_modeling.py", line 749, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "/content/transformers/examples/run_language_modeling.py", line 353, in train
    outputs = model(inputs, masked_lm_labels=labels) if args.mlm else model(inputs, labels=labels)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_gpt2.py", line 602, in forward
    shift_logits = lm_logits[..., :-1, :].contiguous()
RuntimeError: CUDA out of memory. Tried to allocate 786.00 MiB (GPU 0; 14.73 GiB total capacity; 13.33 GiB already allocated; 575.88 MiB free; 13.38 GiB reserved in total by PyTorch)

Epoch:   0% 0/1 [00:00<?, ?it/s]
Iteration:   0% 0/83 [00:00<?, ?it/s]


Some community models are broken and can't be downloaded

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Community Models

Language I am using the model on (English, Chinese ...): Multiple different ones

Quite some community models can't be loaded. The stats are here:

## Stats

1. **68** can't load either their config (n)or their tokenizer:

  - a) **34** models can't even load their config file. The reasons for this are either:

      - i. **11/34**: Model identifier is wrong, e.g. `albert-large` does not exist anymore, it seems like it was renamed to `albert-large-v1`. These models have saved the another name online than how it is saved on AWS.

      - ii. **23/34**: There is an unrecognized `model_type` in the config.json, `e.g.`
> "Error: Message: Unrecognized model in hfl/rbtl3. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: t5, distilbert, albert, camembert, xlm-roberta, bart, roberta, flaubert, bert, openai-gpt, gpt2, transfo-xl, xlnet, xlm, ctrl
> "

   - b) **33** models can load their config, but cannot load their tokenizers. The error message is almost always the same:

> TOK ERROR: clue/roberta_chinese_base tokenizer can not be loaded
> Message: Model name 'clue/roberta_chinese_base' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed 'clue/roberta_chinese_base' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.

   - i. Here: the model has neither of:
    - `vocab_file`
    - `added_tokens_file`
    - `special_tokens_map_file`
    - `tokenizer_config_file`

 2.  **79** currently have wrong `pad_token_id`, `eos_token_id`, `bos_token_id` in their configs. IMPORTANT: The reason for this is that we used to have the wrong defaults saved in `PretrainedConfig()` - see e.g. [here](https://github.com/huggingface/transformers/pull/2885/commits/77d958ac7f0b008df17656e3652246f602aef095)
the default value for **any** model for `pad_token_id` was 0. People trained a model with the lib, saved it and the resulting config.json now had a `pad_token_id = 0` saved. This was then uploaded. But it's wrong and should be corrected.

3. For **162** models everything is fine!

Here the full analysis log [here](https://github.com/patrickvonplaten/files_to_link_to/blob/master/results.txt)
Here the code that created this log (simple comparison of loaded tokenizer and config with default config): [here](https://github.com/patrickvonplaten/files_to_link_to/blob/master/test_all_community_models.py)

### HOW-TO-FIX-STEPS (in the following order):

- [x] Fix 1 a) i. first: All models that have a wrong model identifier path should get the correct one. Need to update some model identifier paths on `https://huggingface.co/models` like changing `bertabs-finetuned-xsum-extractive-abstractive-summarization` to `remi/bertabs-finetuned-xsum-extractive-abstractive-summarization`. Some of those errors are very weird, see #3358

- [ ] Fix 1 a) ii. shoud be quite easy to add the correct `model_type` to the config.json

- [ ] Fix 1 b) Not sure how to fix the lacking tokenizer files most efficiently @julien-c

- [x] Fix 2) Create automated script that:
 - 1. `If tokenizer.pad_token_id != default_config.pad_token_id` -> `config.pad_token_id = tokenizer.pad_token_id else` remove `pad_token_id`.
 - 2. Removes all `eos_token_ids` -> they don't exist anymore


Downloading mlm-17-1280 community model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): mlm-17-1280

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```
from transformers import AutoConfig
conf = AutoConfig.from_pretrained('mlm-17-1280')
```

## Expected behavior

The config should be loaded correctly.
All files exist are seem to be correct. There seems to be a problem with the etag.
When debugging, the call jumps into this statement:
https://github.com/huggingface/transformers/blob/8becb732931bbab5dd75cca5f5e7c75b2516d10b/src/transformers/file_utils.py#L449

and never manages to store the `config.json` file. Not sure what's going on here.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Linux-5.3.0-40-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0+cpu (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


License information by model

Hi,

First of all, thanks for the good work. Very useful.

Would it be possible to add the license information for each model listed on https://huggingface.co/transformers/pretrained_models.html?
The reason is that for production, I need to know which models can be bundled in my app. Some licenses do not allow bundling...

I may have missed it, but I could not find licensing information in the doc or code.

If that information is not centralized, I am happy to do the research myself (and share results!). I would be interested in hints if you have some.

Cheers,

Alex


Update run_language_modeling.py to handle writes on networked filesystem better

In the case of multi-node distributed training, reads and writes typically happen to a common networked filesystem.

In the current version of the `run_language_modeling.py` script, processes that have `local_rank` as 0 perform the writes to disk (tensorboard, dataset cache and model checkpointing). In the case of multi-node distributed training, there ends up being one process per node having `local_rank` as 0, hence multiple processes try writing to the filesystem at one point, resulting on errors being thrown depending on the filesystem.

This pull request updates the script such that only the process having a `global_rank` of 0 does the writing. `global_rank` isn't a variable directly accessible in the script, it is obtained by calling `torch.distributed.get_rank()`.

I've tested the script in 4 different cases and they work without any error in these cases: multi-node training with DDP, single-node training with DDP, single-node training with DP and single gpu training.


Bug? NaN loss after training for a while using for BERT Encoded sentences.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I have a model which is taken from the HF Examples and slightly modified.

```
from transformers import TFBertModel, TFBertForSequenceClassification, BertTokenizer
# configuration = BertConfig()
def build_bert(batch_size=1, use_logits=True):
    num_labels = max(max(train_label), max(test_label))
    print(f"Number of labels: {num_labels}")

    optimizer = tf.keras.optimizers.Adam(learning_rate=1e-5, epsilon=1e-06, clipnorm=1.0)

    if num_labels == 1:
        loss = tf.keras.losses.MeanSquaredError()
    else:
        loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=use_logits)

    print(f"loss used: {loss}")
    macc = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')
    vacc = tf.keras.metrics.SparseCategoricalAccuracy('val_accuracy')

    config = BertConfig.from_pretrained("bert-base-cased", num_labels=num_labels, batch_size=batch_size)
    bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-cased", config=config)
    bert_model.compile(optimizer=optimizer, loss=loss, metrics=[macc, vacc])
    return bert_model
```

I am training on data which is encoded by BERT into 48 token arrays and encoded with the HF Bert Encoder.

```bert_model = build_bert(1000, False)
bert_model.fit([encodings[30000:40000], train_attn_mask[30000:40000]], classes[30000:40000],
               epochs=1, validation_split=.1, shuffle=False)
```

![image](https://user-images.githubusercontent.com/14914/77113444-d683f700-69f8-11ea-8f4e-b9991e777b16.png)

My model will train for a while (Please pardon the output... I have no idea why jupyter lab does this.)
![image](https://user-images.githubusercontent.com/14914/77113488-ee5b7b00-69f8-11ea-9029-fdda4ec26a31.png)

Then at some point (different every run) the loss drops to NaN.
![image](https://user-images.githubusercontent.com/14914/77113529-016e4b00-69f9-11ea-94d4-010388ff0f35.png)

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
The solution for this problem on SO is varied. I tried changing the optimizer learning rates as well as altering the epsilon. I have validated that my data does not contain nan values, negative classifications,  or invalid encodings. I have removed all non unicode characters.

My concern is that this has uncovered a bug with in the framework.


Export ALBERT main layer in TensorFlow

closes #3262


Handle pinned version of isort

The CONTRIBUTING file pins to a specific version of isort, so we might as well install that in `dev` . This makes it easier for contributors so they don't have to manually install the specific commit.


Add model cards for huseinzol05/bert-base-bahasa-cased

Related to pretrained model cards


Reformer

## Add the Reformer
Paper: (https://arxiv.org/pdf/2001.04451.pdf)

### First steps to take:

- [x] Copy Bert PT code to Reformer PT file.
- [x] Replace self-attention with LSH attention
- [x] Make forward pass work for Bert Layer

### Forward-Pass: Get 1-to-1 same outputs as original Flax code for forward pass
   - [x] for LSH attention layer
   - [x] for Bert Layer with RevNet
   - [x] for different attention masks
   - [x] for feed forward chunking layer
   - [x] for whole Reformer model
   - [x] for sinusoidal position encodings
   - [x] for axial position encodings
   - [x] for local blocking attention (chunked attention)
   - [x] for pretrained weights from official reformer model: ReformerLM model was trained using https://colab.research.google.com/github/google/trax/blob/master/trax/models/reformer/text_generation.ipynb and weights were loaded into https://huggingface.co/patrickvonplaten/reformer-crime-and-punish and checked that a single forward pass is identical. `predict_mem_len` had to be adapted to make functions equal.
   - [x] Add optional attention mask
   - [x] Add support for fp16
   - [ ] Speed up incremental generation. This is needed for generation and will not be trivial since the buckets have to ordered correctly and there is a chunk length parameter.

### Backpropagation:
   - [x] Make backpropagation work
   - [x] Check that backpropagation works with chunked feed forward layers
   - [x] Implement RevResLayers for backprop
      - [x] Implement code using  and
      - [x] Get identical results for forward pass
      - [x] Make sure backprop works
      - [x] Implement bucket caching
      - [x] Implement random seed caching to have deterministic dropout for backward pass: https://github.com/RobinBruegger/RevTorch/pull/4
      - [ ] Make rev resnet work for multi-gpu training
   - [x] Check that RevReslayers backprop works on CPU
   - [x] Check that RevReslayers backprop works on GPU
   - [x] Get same gradients as original trax code
   - [x] Train model on crime-and-punishment text and check that model performs reasonable afterwards

### Tokenizer
   - [x] Copy sentence piece tokenizer from T5
   - [x] Add vanilla sentence piece tokenizer for crime-and-punishment pretrained tokenizer: https://console.cloud.google.com/storage/browser/_details/trax-ml/reformer/cp.320.model
   - [ ] Check how many tokenizers are needed
   - [ ] Get pretrained tokenizers

### Optimize time and memory efficiency
   - [x] Compare memory & time complexity to standard Bert: check https://github.com/huggingface/transformers/pull/3186
   - [x] Check and improve memory and speed when training
   - [ ] Move "on-the-fly" created masks in LSHSelfAttention to using them as an input
   - [ ] Optimizie all unnecessary calculations

### Pretrained Models
   - [ ] Check if pretrained model on C4 is added soon: https://github.com/google/trax/commit/b1f0c176a281d35e285137a45ff117b8c5495173

- [ ] Add Reformer / Bert in trax

Useful code resources:
- Original trax code: https://github.com/google/trax/tree/master/trax/models/reformer
- Working trax notebook: https://github.com/google/trax/blob/master/trax/models/reformer/machine_translation.ipynb
- Working PyTorch implementation: https://github.com/lucidrains/reformer-pytorch
- Great to implement for backprop. https://github.com/lucidrains/reformer-pytorch/blob/master/reformer_pytorch/reversible.py
- Pretrained weights: https://console.cloud.google.com/storage/browser/trax-ml/reformer

Useful blog/paper resources:
- Original paper: https://arxiv.org/pdf/2001.04451.pdf
- Google AI blog: https://ai.googleblog.com/2020/01/reformer-efficient-transformer.html
- Good blog post 1: https://www.pragmatic.ml/reformer-deep-dive/
- Good blog post 2: https://towardsdatascience.com/illustrating-the-reformer-393575ac6ba0

Previous Discussions:
- #2341

## Update

The code is clean and ready for review now.
Small ToDos before merging:
- [x] Fill in TODOs in docs
- [ ] Check whether more pre-trained weights can be used
- [ ] Train on fp16 once
- [ ] Update notebook showing how to use Reformer

### Review

I added quite some docstrings to explain the new methods introduced by the Reformer (Axial Position Encoding, LSH Attention, Local Attention, Feed Forward chunking), so it might be better to first go through the doctsrings. Docstrings are easier to read when switching to this branch and creating the docs locally.


Reproducing SQuAD v1.1 with xlnet-base cased?

Hi, first of all, thanks for the great library you guys are providing. I'm currently using the latest version of huggingface/transformers, and I'm trying to get a score for the SQuAD V1.1 with XLNET base-cased. However, it seems that the performance I get is only 0.10 for EM and 0.64 for F1.

When getting the score with BERT base-cased, the score comes out appropriately. (F1 about 88.5)

Is there any bugs or sth else I should be aware of?


Create model card for bert-small-finetuned-squadv2

Related to pretrained model cards


Create card for BERT-Mini finetuned on SQuAD v2

Related to pretrained model cards


Create card for BERT-Tiny fine-tuned on SQuAD v2

Related to pretrained model cards

- Only 17MB of Model weights!!
- The smalles model fine-tuned on SQUaD v2?


Created card for spanbert-finetuned-squadv1

Related to pretrained model cards


Fix input ids can be none attn mask

Make sure `batch_size` is correct for gpt2 and ctrl - these models need a slightly different behavior since the shape of `input_ids` can change depending on whether the past variable is inserted or not.
See also:
PR: https://github.com/huggingface/transformers/pull/3033
and its issue:
https://github.com/huggingface/transformers/issues/3031


Fix wrong link for the notebook file

For the tutorial of "How to generate text", the URL link was wrong (it was linked to the tutorial of "How to train a language model").

I fixed the URL.


Update 01-training-tokenizers.ipynb (typo issue)

I found there are two grammar errors or typo issues in the explanation of the encoding properties.

The original sentences:
- **If your was** made of multiple \"parts\" such as (question, context), then this would be a vector with for each token the segment it belongs to
- **If your has** been truncated into multiple subparts because of a length limit (for BERT for example the sequence length is limited to 512), this will contain all the remaining overflowing parts.

I think "**input**" should be inserted after the phrase "If your".


No Module named Transformers

# 🐛 Bug
 No module found transformers
## Information
Package                  Version
------------------------ ----------
absl-py                  0.9.0
astor                    0.8.1
boto3                    1.12.22
botocore                 1.15.22
cachetools               4.0.0
certifi                  2019.11.28
chardet                  3.0.4
click                    7.1.1
docutils                 0.15.2
filelock                 3.0.12
gast                     0.2.2
google-auth              1.11.3
google-auth-oauthlib     0.4.1
google-pasta             0.2.0
grpcio                   1.27.2
h5py                     2.10.0
idna                     2.9
jmespath                 0.9.5
joblib                   0.14.1
Keras-Applications       1.0.8
Keras-Preprocessing      1.1.0
Markdown                 3.2.1
numpy                    1.18.1
oauthlib                 3.1.0
opt-einsum               3.2.0
pandas                   1.0.2
Pillow                   7.0.0
pip                      20.0.2
protobuf                 3.11.3
pyasn1                   0.4.8
pyasn1-modules           0.2.8
python-dateutil          2.8.1
pytorch-transformers     1.2.0
pytz                     2019.3
pywin32                  227
regex                    2020.2.20
requests                 2.23.0
requests-oauthlib        1.3.0
rsa                      4.0
s3transfer               0.3.3
sacremoses               0.0.38
scipy                    1.4.1
sentencepiece            0.1.85
setuptools               41.2.0
six                      1.14.0
tensorboard              2.1.1
tensorflow               2.1.0
tensorflow-estimator     2.1.0
tensorflow-gpu           2.1.0
tensorflow-gpu-estimator 2.1.0
termcolor                1.1.0
tokenizers               0.5.2
torch                    1.4.0
torchvision              0.5.0
tqdm                     4.43.0
transformers             2.5.1
urllib3                  1.25.8
Werkzeug                 1.0.0
wget                     3.2
wheel                    0.34.2
wrapt                    1.12.1
Using Bert on English language
## To reproduce

Steps to reproduce the behavior:
I just run the following code.
from transformers import BertTokenizer

# Load the BERT tokenizer.
print('Loading BERT tokenizer...')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-ff68f42f17c9> in <module>
----> 1 from transformers import BertTokenizer
      2
      3 # Load the BERT tokenizer.
      4 print('Loading BERT tokenizer...')
      5 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)

ModuleNotFoundError: No module named 'transformers'

## Expected behavior

Do the tokenization.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
C:\Users\David\anaconda3\python.exe: can't open file 'transformers-cli': [Errno 2] No such file or directory

- `transformers` version:transformers             2.5.1
- Platform: Windows 10
- Python version: 3.7.3b
- PyTorch version (GPU?):1.4
- Tensorflow version (GPU?):2.1
- Using GPU in script?:yes
- Using distributed or parallel set-up in script?:distributed


Simpler Error message when loading config/model with .from_pretrained()


Create README.md

Related to pretrained model cards

roberta_chinese_large card


Create README.md

Related to pretrained model cards

xlnet_chinese_large card


Create README.md

roberta_chinese_base card


Create README.md

albert_chinese_tiny card


Create README.md

albert_chinese_small card


Fix #3305: run_ner only possible on ModelForTokenClassification models

Also, #3305 breaks (if i'm not mistaken) the ability to run the example script from a pip-installed instance of transformers (vs. from an instance installed from source) (This PR does not fix that second issue)


transformers.PreTrainedTokenizer.tokenize does lower case work all the time and discards space and tab. Want this changed.

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):'albert_xxlarge_zh'

Language I am using the model on (English, Chinese ...):Chinese

Two problems are all related to method `transformers.PreTrainedTokenizer.tokenize`:

1. How can I not let this method automatically lower the case of the English words in the input sentence? `tokenizer.init_kwargs["do_lower_case"]=True` doesn't work...
2. How can I not let this method discard '\t' and space in default? Or is there any method that can solve this problem?

## To reproduce

Steps to reproduce the behavior:

`
tokenizer=BertTokenizer.from_pretrained("./albert_pytorch/prev_trained_model/albert_xxlarge_zh/")`
`print(tokenizer.init_kwargs.get("do_lower_case")) #output None`
`tokenizer.init_kwargs["do_lower_case"]=True`
`print(tokenizer.init_kwargs.get("do_lower_case")) #output True`
`seq=tokenizer.tokenize("我喜欢\tAPP和WIFI。")`
`print(seq)
`

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected output is:
['我', '喜', '欢',**'\t'**, **'APP'**, '和', **'WIFI'**, '。']
While the actual output is:
['我', '喜', '欢', **'app'**, '和', **'wifi'**, '。']

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:2.5.1
- Platform:ubuntu
- Python version:3.6.1
- PyTorch version (GPU?):1.1.0 cuda9
- Using GPU in script?:nope
- Using distributed or parallel set-up in script?:nope

**BTW**, `python transformers-cli env` didn't work, the callback :

> python: can't open file 'transformers-cli': [Errno 2] No such file or directory


Finetuning T5 Model

Usage

General questions about the library

Hi there. I am trying to fine tune T5, but I have noticed your documentation gives conflicting instructions.

In modeling_t5.py you say
> To match pre-training, T5 input sequence should be formatted with [CLS] and [SEP] tokens

but then in tokenization_t5.py both of those tokens are set to None and the only tokens that are defined are EOS, UNK, and PAD.
Additionally, the actual T5 implementation makes no mention of SEP and CLS as far as I can tell.
Given that, could you clarify how we should be formatting our training data for HuggingFace's T5 implementation? Thank you!


run_tf_ner.py doesn't work with unlabelled test data

When running `run_tf_ner.py` in `predict` mode if all the labels in test data are `O`, script errors out with
`  File "/home/himanshu/.local/lib/python3.7/site-packages/numpy/lib/function_base.py", line 423, in average
    "Weights sum to zero, can't be normalized")
ZeroDivisionError: Weights sum to zero, can't be normalized
`
This is because `pad_token_label_id` https://github.com/huggingface/transformers/blob/cae334c43c49aa770d9dac1ee48319679ee8c72c/examples/ner/run_tf_ner.py#L511 , `label_id` for `O` are both zero, resulting in empty `y_pred`
https://github.com/huggingface/transformers/blob/cae334c43c49aa770d9dac1ee48319679ee8c72c/examples/ner/run_tf_ner.py#L364-L367 Shouldn't the `pad_token_label_id` be different?


Add model cards for FinBERT.

model card

Related to pretrained model cards

These are a copy of https://github.com/TurkuNLP/FinBERT/blob/master/README.md.


Added model cards for SciBERT models uploaded under AllenAI org

Related to pretrained model cards


CUDA Error when running run_language_modeling.py

I am trying to run this script[run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py)

I can load the pre-trained roberta model without a problem. However, when it starts training (loss.backward()), then there are issues like these:

> /pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [30,0,0] Assertion `t >= 0 && t < n_classes` failed.
> /pytorch/aten/src/THCUNN/ClassNLLCriterion.cu:106: void cunn_ClassNLLCriterion_updateOutput_kernel(Dtype *, Dtype *, Dtype *, long *, Dtype *, int, int, int, int, long) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [31,0,0] Assertion `t >= 0 && t < n_classes` failed.

I assume it was caused by the `loss.backward()` line:

>
> File "2_fine_tune_bert.py", line 386, in train
>     loss.backward()
>   File "/home/lily/zl379/anaconda2/envs/py36/lib/python3.7/site-packages/torch/tensor.py", line 195, in backward
>     torch.autograd.backward(self, gradient, retain_graph, create_graph)
>   File "/home/lily/zl379/anaconda2/envs/py36/lib/python3.7/site-packages/torch/autograd/__init__.py", line 99, in backward
>     allow_unreachable=True)  # allow_unreachable flag
> RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)

Is it caused by CUDA version or pytorch version? My pytorch is 1.4.0, CUDA: release 10.1, V10.1.243.
Thanks.


Create README.md

Related to pretrained model cards


improve doctstring for tf and pt generate() method


add link to blog post


Cubla Error on DistilBert

# 🐛 Bug
When using DistilBert I get `CUBLAS_STATUS_ALLOC_FAILED` when trying to forward

## Information
Exact error is : `RuntimeError : Cuda Error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)` and traceback indicates i happens on `output = input.matmul(weight.t())` which is probably not informative but the whole stack is filled with forward on a transformer.

I'm using distilbert-base-multilingual-cased with French

## Environment info
- `transformers` version: 2.5.1
- Platform: Linux-5.4.0-3amd-64-x89_64-with-debian-bullseye-sid
- Python version: 3.7.0
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


Error loading finetuned bert model AttributeError: 'NoneType' object has no attribute 'endswith'

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!--  -->
Unable to load finetuned bert model for language modeling task using run_language_modeling.py script
<!---->
I used the following code to load a fine-tuned model saved on my disk:
`BertForMaskedLM.from_pretrained('outputs/pytorch_model.bin', config=config, from_tf=True).` however I am getting the following error:
Traceback (most recent call last):
  File "/home/mahdi/Desktop/pycharm-community-4.5.3/helpers/pydev/pydevd_vars.py", line 342, in evaluateExpression
    compiled = compile(expression, '<string>', 'eval')
  File "<string>", line 1
    from transformers import WEIGHTS_NAME, BertForMaskedLM
       ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/mahdi/Desktop/pycharm-community-4.5.3/helpers/pydev/pydevd_comm.py", line 1071, in doIt
    result = pydevd_vars.evaluateExpression(self.thread_id, self.frame_id, self.expression, self.doExec)
  File "/home/mahdi/Desktop/pycharm-community-4.5.3/helpers/pydev/pydevd_vars.py", line 344, in evaluateExpression
    Exec(expression, updated_globals, frame.f_locals)
  File "/home/mahdi/Desktop/pycharm-community-4.5.3/helpers/pydev/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<string>", line 2, in <module>
  File "/home/mahdi/anaconda3/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 482, in from_pretrained
    if resolved_archive_file.endswith(".index"):
AttributeError: 'NoneType' object has no attribute 'endswith'

Transformers version:2.5.1
Pytorch version: 1.3.0
**A link to original question on Stack Overflow**:


[Bart/Memory] don't create lm_head

### Summary

previously we created a `self.lm_head = nn.Linear()` with the exact same weight matrix as the input embeddings, and then skipped `tie_weights`.
This presented 3 problems:
1) 200 MB of extra GPU RAM
2) Can't `tie_weights`
3) Can't `resize_token_embeddings`

This PR alleviates all the concerns by using `lm_logits = F.linear(decoder_outputs[0], self.shared)`. It also adds more aggressive test coverage that `resize_embeddings` is changing the shape of both input and output embeddings.

### Concerns
1) If I recall from an earlier PR, tying the input and output embeddings to a single parameter is unfriendly to torchscript.
However, neither `Bart` before this change nor `T5ForConditionalGeneration`, which uses the `self.lm_head = nn.Linear` technique, pass the common torchscript tests, which suggests that the weight tying in this PR is not removing functionality that existed before it.

2) The failing test here is caused by fact that S3 has `lm_head` in `state_dict`. I will update S3 right before this PR gets merged.

3) To pass unit tests and use `.generate`, `get_output_embeddings` must return `nn.Linear`. To satisfy this constraint, this PR makes the `nn.Linear` module on the fly when `get_output_embeddings` is called. I think (but am not sure) that this is fine because `resize_token_embeddings` works by resizing the input_embeddings then calling `tie_weights`, and we have stopped skipping `tie_weights`

(Note there is a separate but related issue that `test_common.py::test_resize_embeddings` is shallow, detailed in https://github.com/huggingface/transformers/issues/3378)


[BART] torch 1.0 compatibility

This PR contains two minor fixes and one piece of cleanup for the BartModel.

**1.** Previously, Bart's encoder padding mask used -10000. to represent tokens that should be ignored, then called `masked_fill(mask.to(torch.bool), -inf)` to use the mask.

There are two problems with this:
- it's confusing to set a value to a large negative and then call `bool`. Why not just invert the mask and call `bool` immediately.
- `torch.bool` is released in pytorch 1.2, so this code breaks on earlier versions.
- Fix: let `torch.eq` make the mask the correct dtype at the beginning.

**2.** explicit use of `F.gelu` is not allowed/or broken in earlier torch versions. Let `ACT2FN` handle this logic.

**3.** An unreachable code branch is deleted.

Supplementary Material: torch v 1.2.0 [release notes](https://github.com/pytorch/pytorch/releases/tag/v1.2.0)


Init card for model

Related to pretrained model cards

We create card for this model.


TF BERT not FP16 compatible?

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): TFBertForQuestionAnswering

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] my own modified scripts:

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQUaD

## To reproduce
Simple example to reproduce error:
```
import tensorflow as tf
from transformers import TFBertForQuestionAnswering

# turn on mp (fp16 operations)
tf.keras.mixed_precision.experimental.set_policy('mixed_float16')

model = TFBertForQuestionAnswering.from_pretrained('bert-base-uncased')
```

The error occurs here:
transformers/modeling_tf_bert.py", line 174, in _embedding
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings

And this is the error:
tensorflow.python.framework.errors_impl.InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a half tensor but is a float tensor [Op:AddV2] name: tf_bert_for_question_answering/bert/embeddings/add/

## Expected behavior
I want to use TF BERT with mixed precision (for faster inference on tensor core GPUs). I know that full fp16 is not working out-of-the-box, because the model weights need to be in fp16 as well. Mixed precision, however, should work because only operations are performed in fp16.

I get some dtype issue. Seems the mode is not fp16 compatible yet? Will this be fixed in the future?

## Environment info
- `transformers` version: 2.5.0
- Platform:  ubuntu 16.04
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (GPU)
- Tensorflow version (GPU?): 2.1.0 (GPU)
- Using GPU in script?: sort of
- Using distributed or parallel set-up in script?: nope


[BART] cleanup: remove redundant kwargs, improve docstrings

Small Bart code cleanups before pip release.

### Cleanup
- Deletes unused `value` argument for SelfAttention. (`value` is always the same as `key`.) This might be moderately controversial as most attention modules take query, key, value as arguments, but this change reduces the signature to just query and key (since key always the same as value).
- Deletes redundant `static_kv` argument for SelfAttention. It is always the same as `self.encoder_decoder_attention`.
  - Context: the `static_kv` variable decides whether we want to extend the keys and values in the cache or, if `True`, use them without modification.
  - This PR keeps a local `static_kv` variable because that variable name describes the purpose of the variable better than `self.encoder_decoder_attention`. But `static_kv` is no longer a kwarg. This simplifies the API and avoids having the same logic in two places.

#### Two new fast tests
- test coverage for `dummy_inputs` (previously broken)
- test coverage for the default generate kwargs.


pipelines.ipynb mask should be [MASK]

transformers/notebooks/03-pipelines.ipynb fill-mask task, use newest version, <mask> is error, should be [MASK].


output value of  XLNetModel changes for the same input

I was trying to use the pre-trained `XLNetModel` model. I found that for the same input, each time I run the model and the output values are different, which is wired to me. Here is a code that I just used:
```
tokenizer =XLNetTokenizer.from_pretrained("xlnet-base-cased")
model=XLNetModel.from_pretrained('xlnet-base-cased')
sent=tokenizer.encode("I love my dog")
test=torch.tensor(sent)
test=test.view(1,test.shape[0])
```
Now If I run the model, each time the output is different.
```
out=model(test)
print(out[0])

```
Why this be behaviour?

I have another question, isn't this model the pre-trained transformer trained on language modeling task (transformer without the LM head)?


TextClassificationPipeline does not work with pretrained BERT model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): 'nlptown/bert-base-multilingual-uncased-sentiment'

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

```
model = BertModel.from_pretrained(pretrained_model_name_or_path='nlptown/bert-base-multilingual-uncased-sentiment')
tokenizer = BertTokenizer.from_pretrained(pretrained_model_name_or_path='nlptown/bert-base-multilingual-uncased-sentiment')
sentiment_analyzer = TextClassificationPipeline(model=model, tokenizer=tokenizer)
sentiment_analyzer('This is awesome!')

/usr/local/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    504     def __call__(self, *args, **kwargs):
    505         outputs = super().__call__(*args, **kwargs)
--> 506         scores = np.exp(outputs) / np.exp(outputs).sum(-1)
    507         return [{"label": self.model.config.id2label[item.argmax()], "score": item.max()} for item in scores]
    508

ValueError: operands could not be broadcast together with shapes (1,8,768) (1,8)
```

## Expected behavior

A sentiment score.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.5.1
- Platform: Ubuntu 16.04
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


how does masked_lm_labels  work ?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Mismatch in the accuracy figures

# ❓ Questions & Help

Hi Just wanted to know if "bert-base-multilingual-cased" model and https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip are same.

I tried using the tokenizer from the multi_cased_L-12_H-768_A-12 and bert-base-multilingual-cased model which performed better than using the tokenizer and model from pretrained  "bert-base-multilingual-cased" .

I was trying to fine tune a sentiment analysis task and the performance of using tokenizers above provide different performances.

Can anybody shed light on this ?


KeyError in GLUE data tokenization with RoBERTA

# 🐛 Bug

I'm getting a KeyError [here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/glue.py#L94) when using RoBERTa in [examples/run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py) and trying to access `'token_type_ids'` while preprocessing the data, maybe from [this commit](https://github.com/huggingface/transformers/commit/5164ea91a7b4d35cb03867233527fa383a651775) removing `'token_type_ids'` from RoBERTa (and DistilBERT)?

I get the error when fine-tuning RoBERTa on CoLA and RTE. I haven't tried other tasks, but I think you'd get the same error.
I don't get the error when fine-tuning XLNet (presumably, since XLNet does use `'token_type_ids'`), and I don't get the error when I do `pip install transformers` instead of `pip install .` (which I think means the issue is coming from a recent commit).

Here's the full error message:
```bash
03/17/2020 11:53:58 - INFO - transformers.data.processors.glue -   Writing example 0/13997
Traceback (most recent call last):
  File "examples/run_glue.py", line 731, in <module>
    main()
  File "examples/run_glue.py", line 679, in main
    train_dataset = load_and_cache_examples(args, args.task_name, tokenizer, evaluate=False)
  File "examples/run_glue.py", line 419, in load_and_cache_examples
    pad_token_segment_id=4 if args.model_type in ["xlnet"] else 0,
  File "/home/ejp416/cmv/transformers/src/transformers/data/processors/glue.py", line 94, in glue_convert_examples_to_features
    input_ids, token_type_ids = inputs["input_ids"], inputs["token_type_ids"]
KeyError: 'token_type_ids'
```

## Information

Model I am using (Bert, XLNet ...): RoBERTa. I think DistilBERT may run into the same issue as well.

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

I've made slight modifications to the training loop in the official [examples/run_glue.py](https://github.com/huggingface/transformers/blob/master/examples/run_glue.py), but I did not touch the data pre-processing, which is where the error occurs (before any training).

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

I've run into the error on CoLA and RTE, though I think the error should happen on all GLUE tasks.

## To reproduce

Steps to reproduce the behavior:

1. Install `transformers` using the latest clone (use `pip install .` not `pip install transformers`)
2. Download the RTE data (e.g., into `data/RTE` using the GLUE download scripts in this repo)
3. Run a command to train RoBERTa (base or large). I'm using:
```
python examples/run_glue.py --model_type roberta --model_name_or_path roberta-base --output_dir models/debug --task_name rte --do_train --evaluate_during_training --data_dir data/RTE --max_seq_length 32 --max_grad_norm inf --adam_epsilon 1e-6 --adam_beta_2 0.98 --weight_decay 0.1 --logging_steps 874 --save_steps 874 --num_train_epochs 10 --warmup_steps 874 --per_gpu_train_batch_size 1 --per_gpu_eval_batch_size 2 --learning_rate 1e-5 --seed 12 --gradient_accumulation_steps 16 --overwrite_output_dir
```

## Expected behavior

`load_and_cache_examples` (and specifically, the call to `convert_examples_to_features`) in `examples/run_glue.py` should run without error, to load, preprocess, and tokenize the dataset.

## Environment info

- `transformers` version: 2.5.1
- Platform: Linux-3.10.0-1062.12.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Error happens with both GPU and CPU
- Using distributed or parallel set-up in script?: No


GPT2Tokenizer doesn't include BOS or EOS token

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT-2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Script:
```
from transformers import GPT2Tokenizer

tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
encoded_dict = tokenizer.encode_plus(text="Hello I am Moin", add_special_tokens=True, \
    max_length=512, truncation_strategy="longest_first", pad_to_max_length=False, \
    return_tensors=None, return_token_type_ids=True, return_attention_mask=True, \
    return_overflowing_tokens=False, return_special_tokens_mask=False)

print(tokenizer.bos_token_id)
print(encoded_dict['input_ids'])
```

You should see that the `input_ids` do not include the `bos_token_id`. Shouldn't `encode_plus` be doing this?

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
The <|endoftext|> token would appear, since I included to `add_special_tokens`.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Linux-4.15.0-54-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.2
- PyTorch version (GPU?): 1.3.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


GPT2 -- build_inputs_with_special_tokens lacking BOS and EOS tokens.


Add sample softmax possibility to TransfoXL model for TransfoXL training

# 🚀 Feature request

TransfoXL samples the logits during training if required. At the moment TransfoXL can only be used whithout sampling from the logits during training. A partrly finished implementation can be found under the branch: `add_sampling_and_training_to_transfo_xl_models` .

## Motivation

To be able to train tranfoXL correctly.

## Your contribution

Already looked into the issue. Could try to implement it correctly with help from @thomwolf and @LysandreJik . Not a priority at the moment though.


Create model card for CodeBERTaPy