Post-padding affects the Bert embedding output

# 🐛 Bug

## Information

Model: BertModel
Language: English

The problem arises when using:

```
# Load model
from transformers import *
import torch
model_class = BertModel
tokenizer_class = BertTokenizer
pretrained_weights = 'bert-base-uncased'
tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
model = model_class.from_pretrained(pretrained_weights).to('cuda')

# First example
batch_id = [[101, 1996, 3035, 2038, 2741, 1037, 1056, 28394, 2102, 2000, 1996, 3035, 2012, 17836, 4186, 2000, 8439, 2014, 3938, 2705, 5798, 102]]
batch_id = torch.tensor(batch_id).to('cuda')
with torch.no_grad():
    last_hidden_states = model(batch_id)[0].cpu().numpy()
print (last_hidden_states[0][:10])

# Second example
batch_id = [[101, 1996, 3035, 2038, 2741, 1037, 1056, 28394, 2102, 2000, 1996, 3035, 2012, 17836, 4186, 2000, 8439, 2014, 3938, 2705, 5798, 102, 0, 0]]
batch_id = torch.tensor(batch_id).to('cuda')
with torch.no_grad():
    last_hidden_states = model(batch_id)[0].cpu().numpy()
print (last_hidden_states[0][:10])
```

Output for the first example
```
array([[ 0.00197573, -0.06912418,  0.24121636, ..., -0.13239928,
         0.13210389,  0.3860737 ],
       [ 0.18745837, -0.15252575,  0.16234997, ..., -0.34497464,
         1.0031146 ,  0.20545363],
       [ 0.40690556, -0.7345518 ,  1.1162403 , ..., -1.148023  ,
        -0.38943186, -0.6397534 ],
       ...,
       [ 1.3574413 , -0.87637144,  1.007168  , ..., -0.7466023 ,
        -0.5337318 , -0.02415964],
       [ 0.0907229 , -1.0051603 ,  0.7100666 , ..., -0.00599465,
        -0.37829682,  0.4773703 ],
       [-0.00619348, -0.34730428,  0.9920887 , ...,  0.28678447,
         0.2980772 ,  0.8005251 ]], dtype=float32)
```

Output for the second example
```
array([[-0.10877508,  0.0271297 ,  0.17947783, ..., -0.2650592 ,
         0.15821457,  0.35017303],
       [-0.1396759 , -0.25098413,  0.3990493 , ..., -0.52468735,
         0.8060062 ,  0.42330667],
       [ 0.18865047, -1.0035415 ,  1.3446846 , ..., -1.1652598 ,
        -0.60856164, -0.419513  ],
       ...,
       [ 1.3687737 , -0.9032434 ,  1.0184443 , ..., -0.7951573 ,
        -0.56618035, -0.00522863],
       [ 0.02363256, -0.962884  ,  0.68822455, ..., -0.03798304,
        -0.34567115,  0.5442954 ],
       [-0.00341167, -0.33559048,  1.0627198 , ...,  0.31898227,
         0.2941662 ,  0.7981017 ]], dtype=float32)
```

I also checked that the output for `tokenizer.convert_tokens_to_ids(tokenizer.pad_token)` is `0`

## Expected behavior

The embeddings for the padded sequence should be same with the ones without padding.

## Environment info

- `transformers` version: transformers 2.4.1
- Platform: Linux
- Python version: Python 3.7.5
- PyTorch version (GPU?): PyTorch 1.4.0 (CUDA Version 10.1.243)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Added model card for bert-base-multilingual-uncased-sentiment

model card

Related to pretrained model cards

Added the model card for nlptown/bert-base-multilingual-uncased-sentiment


is right?

Ex: LM (Pretraining)

Related to language modeling pre-training

https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py
in 225 row

![image](https://user-images.githubusercontent.com/20811685/74525332-d1480e00-4f5b-11ea-8918-ae22507de8e8.png)

# 10% of the time, we replace masked input tokens with random word
but write 0.5
is ok?


Fix typos


Fix typo


Create model card for 'distill-bert-base-spanish-wwm-cased-finetuned-spa-squad2-es'


[pipeline] Alias NerPipeline as TokenClassificationPipeline


Update with additional information

Added a "Pre-training details" section


Create model card for the newly released 'nlpaueb/bert-base-greek-uncased-v1'

model card

Related to pretrained model cards


Adding usage examples for common tasks

Adding a documentation page detailing usage for common tasks (inference, not training


PreTrainedEncoderDecoder does not work for LSTM

Core: Modeling

Internals of the library; Models.

Should Fix

This has been identified as a bug and should be fixed.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
If we want to have a BERT-based encoder and LSTM encoder, that is not currently possible with the current huggingface implementation, mostly because torch.nn.LSTM does not contain a config class variable.

## Stack Trace
  File "/beegfs/yp913/anaconda3/envs/jiant_new/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 349, in from_pretrained
    model = super().from_pretrained(*args, **kwargs)
  File "/beegfs/yp913/anaconda3/envs/jiant_new/lib/python3.6/site-packages/transformers/modeling_encoder_decoder.py", line 153, in from_pretrained
    decoder.config.is_decoder = True
  File "/beegfs/yp913/anaconda3/envs/jiant_new/lib/python3.6/site-packages/torch/nn/modules/module.py", line 539, in __getattr__
    type(self).__name__, name))
AttributeError: 'LSTM' object has no attribute 'config'

## To reproduce

You can reproduce this by:
import transformers
from transformers.modeling_encoder_decoder import  Model2LSTM
model =  Model2LSTM.from_pretrained("roberta-large", decoder_config={"hidden_size":512, "input_size":1024, "num_layers": 2})

(When you initialize Model2LSTM like the above it runs into a separate error. I believe a ** is missing from the Model2LSTM decoder LSTM initialization).


Add `masked_lm_labels` argument to `TFAlbertForMaskedLM`

TensorFlow

Anything TensorFlow

# 🚀 Feature request

The PyTorch `AlbertForMaskedLM` model has support for the `masked_lm_labels` parameter, while `TFAlbertForMaskedLM` does not. I'd like to bring feature parity.

It looks like a similar feature is also missing for `TFBertForMaskedLM`, `TFRobertaForMaskedLM`, `TFDistilBertForMaskedLM`. I'd be happy to add support for those models as well.

## Motivation

I'm pretraining TF NLP models, and this would simplify the training script by encapsulating the loss function.

## Your contribution

I'm happy to contribute the code. I'll follow CONTRIBUTING.md, any gotchas I should be aware of?


BART/T5 seq2seq example

# 🚀 Feature request

Can we have a seq2seq example with training/fine-tuning and generation for BART/T5 models?


Error reported when running ''run_language_modeling.py" file

Core: Modeling

Internals of the library; Models.

Need more information

Further information is requested

# 🐛 Bug

## Information

Model I am using (Bert and RoBerta):

Language I am using the model on (English).

The problem arises when using:
* [ ] the official example scripts: (give details below)
I followed the tutorial of how to fine tuning the Bert model on own corpus data, and used recommended corpus 'wiki-text-2' corpus to fine tune the Bert Model.
However there is always an error appears: "RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at ../aten/src/THNN/generic/ClassNLLCriterion.c:97".
Thus I am not sure are there something wrong with the "run_language_modeling.py" file, because I did do any change of the original code and used recommended wiki corpus. Could you help me check this error?

* [ ] my own modified scripts: (give details below)
None

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
Fine tuning the language model of Bert on our own customer corpus data.

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Skip flaky test_tf_question_answering

Reasoning: While we diagnose the problem, better to keep circleci from randomly failing.


Attempt to increase timeout for circleci slow tests

@LysandreJik can you help me test this?


Model card: Literary German BERT

model card

Related to pretrained model cards

This PR adds a model card for [severinsimmler/literary-german-bert](https://huggingface.co/severinsimmler/literary-german-bert), a domain-adapted and fine-tuned BERT for named entity recognition in German literary texts.


when will add XLMRobertaForQuestionAnswering package

I will study squad of multilingual.

I found that question answer package did include in run_squad.py.

I wanna release that package

Are you plan to release XLMRobertaForQuestionAnwering?

please let me know.


cannot find model in model name list

Core: Tokenization

Internals of the library; Tokenization.

Usage

General questions about the library

Hi , thank you for developing well-made pytorch version of BERT !

I am new to NLP area and have problem while coding like this:
```python
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
```

The error discription is below:
```
INFO:pytorch_pretrained_bert.file_utils:https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache, downloading to C:\Users\zxr\AppData\Local\Temp\tmpb3lgzjlo
ERROR:pytorch_pretrained_bert.tokenization:Model name 'bert-base-uncased' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt' was a path or url but couldn't find any file associated to this path or url.
```

I searched it and thought it might be poor internet connect.

I downloaded the model i want but donot know how to load it with code.

Thank you very much for your attention!


[WIP] Add patience argument to run_language_modeling script

# Summary
Often, we want to stop training if loss does not improve for a number of epochs. This PR adds a "patience" argument, which is a limit on the number of times we can get a non-improving eval loss before stopping training early.

It is implemented by other NLP frameworks, such as AllenNLP (see [trainer.py](https://github.com/allenai/allennlp/blob/master/allennlp/training/trainer.py#L95) and [metric_tracker.py](https://github.com/allenai/allennlp/blob/1a8a12cd1b065d74fec3d2e80105a684736ff709/allennlp/training/metric_tracker.py#L6)).

# Motivation
This feature allows faster fine-tuning by breaking the training loop early and avoids users the toil of checking metrics on Tensorboard.

# Caveats
Often, models are evaluated once per epoch, but run_lm_finetuning.py has an option to evaluate after a set number of model update steps (dictated by `--logging_steps` if `--evaluate_during_training` is true). Because of this, I've elected to tie patience to the number of evaluations without improvement in loss.

# To-do
- Add tests
- Fix long lines


Fine-tuning the model using classification tasks

General questions about the library

Hello All,

Could anyone tell how can I fine-tune the language model using classification tasks, however not using any GLUE data, as I have my own custom dataset?
Is there any solution and/or method to do the classification using a custom dataset?


A small model for CTRL

Distillation

Related to model distillation

# 🚀 Feature request

A smaller version of the pre-trained model CTRL, related to the stack overflow question;
[https://stackoverflow.com/questions/60142937/huggingface-transformers-for-text-generation-with-ctrl]

## Motivation

I've been trying to generate text using CTRL and I run to a memory insufficiency, since it is a large model. And was wondering whether there would be a small version of CTRL like a Distill version like some of the other transformer models do.


Pretrained TFAlbertForMaskedLM returns seemingly random token predictions

Core: Modeling

Internals of the library; Models.

TensorFlow

Anything TensorFlow

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT, ALBERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [x] my own task or dataset: toy data.

## To reproduce

```
import tensorflow as tf
from transformers import BertTokenizer, TFBertForMaskedLM, AlbertTokenizer, TFAlbertForMaskedLM

tf.random.set_seed(1)
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = TFBertForMaskedLM.from_pretrained("bert-base-uncased")
input_ids = tokenizer.encode(f"I {tokenizer.mask_token} you", return_tensors="tf")
outputs = model(input_ids)
prediction_scores = outputs[0]
predicted_ids = tf.reshape(tf.argmax(prediction_scores, -1), [-1])
predicted_tokens = tokenizer.convert_ids_to_tokens(predicted_ids)
print(predicted_tokens)
# ['.', 'i', 'love', 'you', '.']

tf.random.set_seed(1)
tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
model = TFAlbertForMaskedLM.from_pretrained("albert-base-v2")
input_ids = tokenizer.encode(f"I {tokenizer.mask_token} you", return_tensors="tf")
outputs = model(input_ids)
prediction_scores = outputs[0]
predicted_ids = tf.reshape(tf.argmax(prediction_scores, -1), [-1])
predicted_tokens = tokenizer.convert_ids_to_tokens(predicted_ids)
print(predicted_tokens)
# ['_pawn', '_addressing', '_fundraising', '_george', '_hybrid']
```

## Expected behavior

I would expect both commands to return the same result, filling in the middle with "love" or some other word. BERT performs correctly, while ALBERT seems to return nonsense. Any idea why this is happening?

## Environment info
- `transformers` version: 2.4.1
- Platform: Linux
- Python version: 3.6.5
- PyTorch version (GPU?): not installed
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Getting value of [UNK] labels

Core: Tokenization

Internals of the library; Tokenization.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

I have created a NER model based ob Bert with this library, but I have a problem when  I run my model due to `[UNK]`. Sometimos there are entities that aren't on my vocab so they are marked as unkowns so I cant know what they are.

I know I can't revert `[UNK]` label, so would like to be able to define the words that would be unknown before the processing of the sentence.

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:  [https://stackoverflow.com/questions/60192523/get-the-value-of-unk-in-bert](url)


Failing slow RobertaModelIntegrationTest

Core: Modeling

Internals of the library; Models.

Tests

Related to tests

```
RUN_SLOW=1 pytest tests/test_modeling_roberta.py::RobertaModelIntegrationTest::test_inference_masked_lm
```
Have not investigated at all, but wanted to record.

Traceback:
```
self = <tests.test_modeling_roberta.RobertaModelIntegrationTest testMethod=test_inference_masked_lm>

    @slow
    def test_inference_masked_lm(self):
        model = RobertaForMaskedLM.from_pretrained("roberta-base")

        input_ids = torch.tensor([[0, 31414, 232, 328, 740, 1140, 12695, 69, 46078, 1588, 2]])
        output = model(input_ids)[0]
        expected_shape = torch.Size((1, 11, 50265))
        self.assertEqual(output.shape, expected_shape)
        # compare the actual values for a slice.
        expected_slice = torch.Tensor(
            [[[33.8843, -4.3107, 22.7779], [4.6533, -2.8099, 13.6252], [1.8222, -3.6898, 8.8600]]]
        )
>       self.assertTrue(torch.allclose(output[:, :3, :3], expected_slice, atol=1e-3))
E       AssertionError: False is not true
```


Failing slow AutoModelTest/BertForPreTraining

Core: Pipeline

Internals of the library; Pipeline.

Tests

Related to tests

```
RUN_SLOW=1 pytest tests/test_modeling_auto.py::AutoModelTest::test_model_for_pretraining_from_pretrained
```
Have not investigated at all, but wanted to record since the slow test failures are evasive :)

Clues:
model: `transformers.modeling_bert.BertForPreTraining`
```
loading_info = {'missing_keys': ['cls.predictions.decoder.bias'], 'unexpected_keys': [], 'error_msgs': []}
```
Likely related to this funkiness

https://github.com/huggingface/transformers/blob/ee5de0ba449d638da704e1c03ffcc20a930f5589/src/transformers/modeling_bert.py#L482-L483


add model_card flaubert-base-uncased-squad

model card

Related to pretrained model cards

A baseline model for question-answering in french ([flaubert](https://github.com/getalp/Flaubert) model fine-tuned on [french-translated SQuAD 1.1 dataset](https://github.com/Alikabbadj/French-SQuAD))

Small error when trying it with the pipeline though:

```python-traceback
>>> nlp = pipeline('question-answering', model='fmikaelian/flaubert-base-uncased-squad', tokenizer='fmikaelian/flaubert-base-uncased-squad')

nlp({
    'question': "Qui est Claude Monet?",
    'context': "Claude Monet, né le 14 novembre 1840 à Paris et mort le 5 décembre 1926 à Giverny, est un peintre français et l’un des fondateurs de l'impressionnisme."
})

Model name 'fmikaelian/flaubert-base-uncased-squad' was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-japanese, bert-base-japanese-whole-word-masking, bert-base-japanese-char, bert-base-japanese-char-whole-word-masking, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased, openai-gpt, transfo-xl-wt103, gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2, ctrl, xlnet-base-cased, xlnet-large-cased, xlm-mlm-en-2048, xlm-mlm-ende-1024, xlm-mlm-enfr-1024, xlm-mlm-enro-1024, xlm-mlm-tlm-xnli15-1024, xlm-mlm-xnli15-1024, xlm-clm-enfr-1024, xlm-clm-ende-1024, xlm-mlm-17-1280, xlm-mlm-100-1280, roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector, distilbert-base-uncased, distilbert-base-uncased-distilled-squad, distilbert-base-cased, distilbert-base-cased-distilled-squad, distilbert-base-german-cased, distilbert-base-multilingual-cased, distilbert-base-uncased-finetuned-sst-2-english, albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2, camembert-base, umberto-commoncrawl-cased-v1, umberto-wikipedia-uncased-v1, t5-small, t5-base, t5-large, t5-3b, t5-11b, xlm-roberta-base, xlm-roberta-large, xlm-roberta-large-finetuned-conll02-dutch, xlm-roberta-large-finetuned-conll02-spanish, xlm-roberta-large-finetuned-conll03-english, xlm-roberta-large-finetuned-conll03-german, flaubert-small-cased, flaubert-base-uncased, flaubert-base-cased, flaubert-large-cased). We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/fmikaelian/flaubert-base-uncased-squad/modelcard.json' was a path or url to a model card file named modelcard.json or a directory containing such a file but couldn't find any such file at this path or url.
Creating an empty model card.
>>>
>>> nlp({
...     'question': "Qui est Claude Monet?",
...     'context': "Claude Monet, né le 14 novembre 1840 à Paris et mort le 5 décembre 1926 à Giverny, est un peintre français et l’un des fondateurs de l'impressionnisme."
... })
convert squad examples to features: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  3.25it/s]
add example index and unique id: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 4181.76it/s]
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/usr/local/lib/python3.7/site-packages/transformers/pipelines.py", line 815, in __call__
    start, end = self.model(**fw_args)
ValueError: too many values to unpack (expected 2)
```


'distilbert-base-cased-distilled-squad' was not found error

# 🐛 Bug

## Information

Model I am using: distilbert-base-cased-distilled-squad

The problem arises when using: AutoTokenizer or AutoModelForQuestionAnswering

Steps to reproduce the behavior:
0. make sure you have everything on colab installed and imported
```
!pip install transformers
import torch
from transformers import AutoModelForQuestionAnswering, AutoTokenizer
```
1. run the code on google colab:
```

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")

model = AutoModelForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")
```

and the error:
```
OSError                                   Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    224             if resolved_config_file is None:
--> 225                 raise EnvironmentError
    226             config_dict = cls._dict_from_json_file(resolved_config_file)

OSError:

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
3 frames
/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, pretrained_config_archive_map, **kwargs)
    239                     )
    240                 )
--> 241             raise EnvironmentError(msg)
    242
    243         except json.JSONDecodeError:

OSError: Model name 'distilbert-base-cased-distilled-squad' was not found in model name list. We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/distilbert-base-cased-distilled-squad/config.json' was a path, a model identifier, or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.
```

## Environment info
- `transformers` version: 2.4.1
- Platform: Google Colab
- PyTorch version :1.4.0
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Installation Error - Failed building wheel for tokenizers

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): N/A

Language I am using the model on (English, Chinese ...): N/A

The problem arises when using:
* [X] the official example scripts: (give details below)

Problem arises in transformers installation on Microsoft Windows 10 Pro, version 10.0.17763

After creating and activating the virtual environment, installing transformers is not possible, because the following error occurs:

"error: can not find Rust Compiler"
"ERROR: Failed building wheel for tokenizers"
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed d

The tasks I am working on is:
[X ] transformers installation

## To reproduce

Steps to reproduce the behavior:

1. From command line interface, create and activate a virtual environment by following the steps in this URL: https://packaging.python.org/guides/installing-using-pip-and-virtual-environments/
2. Install transformers from source, by following the example in the topic From Source on this URL: https://github.com/huggingface/transformers

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
-m pip --version
-m pip install --upgrade pip
-m pip install --user virtualenv
-m venv env
.\env\Scripts\activate
pip install transformers

ERROR: Command errored out with exit status 1:
   command: 'c:\users\vbrandao\env\scripts\python.exe' 'c:\users\vbrandao\env\lib\site-packages\pip\_vendor\pep517\_in_process.py' build_wheel 'C:\Users\vbrandao\AppData\Local\Temp\tmpj6evjmze'
       cwd: C:\Users\vbrandao\AppData\Local\Temp\pip-install-sza2_lmj\tokenizers
  Complete output (10 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib
  creating build\lib\tokenizers
  copying tokenizers\__init__.py -> build\lib\tokenizers
  running build_ext
  running build_rust
  error: Can not find Rust compiler
  ----------------------------------------
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers which use PEP 517 and cannot be installed directly

```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Installation of transformers should be complete.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: N/A - installation step
- Platform: Command Line Interface / Virtual Env
- Python version: python 3.8
- PyTorch version (GPU?): N/A
- Tensorflow version (GPU?): N/A
- Using GPU in script?: N/A
- Using distributed or parallel set-up in script?: N/A
![tokenizers_intallation_error](https://user-images.githubusercontent.com/17074908/74371705-06b3f680-4db8-11ea-8a2d-5f920cb3caab.PNG)


Reusing states for sequential decoding in BERTForMaskedLM

PyTorch

Anything PyTorch

# 🚀 Feature request

I am using Bert as a decoder (by setting is_decoder=True). However, during sequential decoding, there is no way of reusing the hidden states, so for every word to be generated we need to rerun the model on the ENTIRE decoded sequence, which renders decoding inefficient. Can you add something similar to the keyword `past=` in GPT2 model to BERT's forward function (https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L938)?

## Generalization of my Issue
More generally, to the best of my knowledge, there's no model in this library that simultaneously supports 1) cross attention (by feeding `encoder_hidden_states=` or `memory=`), and 2) reusing decoder states during sequential decoding (by feeding `past=`). 1) rules out models like GPT-2 and XLNet which only supports language modeling (although in theory we can just use a decoder to do translation, I want to use a separate encoder and decoder); and 2) rules out models like BERT and T5 which supports 1) but not 2). For example, the point of T5 is to use it for text-to-text translation problems, but since we cannot reuse hidden states, sequential decoding (beam search) would be extremely inefficient.

## Example
In the provided summarization example, both 1) and 2) are supported. However, the decoder is defined in its own code (examples/summarization/modeling_bertabs.py) and cannot be used directly in the library. Besides, supporting incremental state update is a basic function that every decoder shall support.

## Relevant Issues
I checked the suggested similar issues and did not find the same issue. Please let me know if my issue duplicates others'.


BERT generating prediction in 120sec approx using squad 2.0 in prediction.json

I am using the below command to predict the question answer using BERT with squad but it taking too long to generate the prediction.json approx 120 sec. I wanna reduce this time to 10secs.

run_squad.py --vocab_file=uncased_L-12_H-768_A-12/vocab.txt --bert_config_file=uncased_L-12_H-768_A-12/bert_config.json --init_checkpoint=model.ckpt-21899 --do_train=False --train_file=train-v1.1.json --do_predict=True --train_batch_size=32 --learning_rate=5e-5 --num_train_epochs=3.0 --max_seq_length=384 --doc_stride=128 --version_2_with_negative=True --output_dir=/ --predict_file=input.json --use_tpu=False

Please suggest me some solution.


[WIP] Create a Trainer class to handle TF2 model training

**EDIT**
Close this PR to create a cleaner, and more on purpose one.

Hello,

I'm opening the pull request I was talking about in the issue #2783. Here the proposed features in this PR:

- [x] add checkpoint manager in order to make a training fault-tolerant
- [x] add custom fit method to take into account the specific training steps in distributed mode or not
- [x] add optimizer creation method depending of its name
- [ ] add loss method in order to be able to custom the loss computation
- [x] add a Tensorboard summary writer to make the logs available in Tensorboard

For now I have created the definition of the methods with their documentation but with a `raise NotImplementedError` as first I would like to have your opinion on the signatures of these methods. Also I know that @julien-c you have recently worked on a `TFModelUtilsMixin` class. Do you think that some of these methods should go into it instead of directly in `TFPreTrainedModel`?

ping also @sshleifer


OOM risk in RobertaTokenizer/GPT2Tokenizer

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using: Roberta (_roberta-base_)

Language I am using the model on: English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts

I am using a modified version of the [examples/distillation/scripts/binarized_data.py](https://github.com/huggingface/transformers/blob/master/examples/distillation/scripts/binarized_data.py)  file.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset

I am tokenizing a gzipped JSON file that contains 2B inputs resulting in 250Gb of uncompressed data.
The tokenization function is divided across _n_ processes to make the tokenization part faster.
The resulting _token_ids_ are written as a list of integers in an output file.
While tokenization was done batch by batch, I noticed that my RAM was increasing.
It caused an OOM error (I have 64GB or RAM) while only 1.5B inputs had been processed.

I identified the problem to be the `cache` attribute of the `GPT2Tokenizer` ([link](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_gpt2.py#L191)) that is never flushed so its size will potentially grow infinitely.
Tokenizers inheriting from `GPT2Tokenizer` (such as `RobertaTokenizer` ) are thus also impacted.

## To reproduce

Steps to reproduce the behavior:

Run tokenization: `tokenizer.encode()` on a very big file using `GPT2Tokenizer` or `RobertaTokenizer`.

## Expected behavior

The memory footprint of the tokenizer should be constant while processing an infinite stream of inputs.

## Suggestion
I made a quick and dirty fix in my script by flushing the `cache` (tokenizer.cache.clear()) if its size reaches an arbitrarily set threshold (100k in my case) with no significant loss in terms of performance.
However I think that there are smarter solutions rather than flushing the whole cache content. One can use a LRU cache instead of a Python dict. You can also define a private method that checks if cache size reaches a threshold and perform flushing in an "elegant" way.

I know that for production purposes [Tokenizers](https://github.com/huggingface/tokenizers) lib would be more appropriate but I wanted to notice you this behavior.

## Environment info

- `transformers` version: 2.4.1
- Platform: Ubuntu 16.04 LTS
- Python version: 3.6.9
- PyTorch version: 1.4.0 (with GPU)
- Tensorflow version : 2.0.0 (with GPU)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: using python 3.6 [multiprocessing](https://docs.python.org/3.6/library/multiprocessing.html) lib


Why only use the hidden state of last token of last layer is used for predicting the next word?

I was trying to generate text and also reading the code to understand how it is working.
I found that, after providing some text as context (first iteration), it goes through the transformer and the output of the transformer (`output[0] `of `GPT2Model` ) is for each token position there is a vector. To my understanding, these vectors are the context-aware representation of each token position.
Now for generating the next word, the representation of the last token of last layers is being used.
This is the case for the first iteration.
Then for each subsequent iteration, only the representation of the last predicted word is used to predict the next word.

My question is, why the only the representation of the last word is used to predict the next word?

This raises another question, is the last token-position representation hold the context of the whole sequence (like LSTM)?


binarized_data.py in distillation uses incorrect type casting

Distillation

Related to model distillation

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): possibly affected model is DistilBert(distilbert-base-multilingual-cased)

Language I am using the model on (English, Chinese ...): multiple

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [x] my task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Open line [84 in examples/distillation/scripts/binarized_data.py](https://github.com/huggingface/transformers/blob/21da895013a95e60df645b7d6b95f4a38f604759/examples/distillation/scripts/binarized_data.py#L84)
2. See typecast into np.uint16 (possibly added to produce smaller output file size)
3. Realize, that multilingual model has vocab size of 119547, so a large portion of tokens(54012, 45%), which has id > uint16 max value(65535), receives the wrong id after binarization

```python
# Some code to demonstrate the thing
import transformers as tr
import numpy as np

tok = tr.DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
print("UInt16 max value", np.iinfo(np.uint16).max) ## 65535
print("Vocab size:", tok.vocab_size) ## 119547

## code to produce table i've included into issue
def table_row(tok_id):
    print(f"|{tok_id:^15}|{tok.decode([tok_id]):^18}|{np.uint16(tok_id):^18}|{tok.decode([np.uint16(tok_id)]):^30}|")

print("|Actual token id|Actual token value|Token id in uint16|Token value by uint16 token id|")
print("|---------------|------------------|------------------|------------------------------|")
for i in range(65535, 65700):
    table_row(i)

```

## Examples
|Actual token id|Actual token value|Token id in uint16|Token value by uint16 token id|
|---------------|------------------|------------------|------------------------------|
|     65535     |        PD        |      65535       |              PD              |
|     65536     |       ##्ग       |        0         |            [PAD]             |
|     65537     |     označava     |        1         |          [unused1]           |
|     65538     |     ##gården     |        2         |          [unused2]           |
|     65539     |      ##чном      |        3         |          [unused3]           |
| ....               | ....                 | ....              | ....                                |
|     65635     |      siege       |        99        |          [unused99]          |
|     65636     |      ##lën       |       100        |            [UNK]             |
|     65637     |      dotato      |       101        |            [CLS]             |
|     65638     |     madeira      |       102        |            [SEP]             |
|     65639     |      ##μίας      |       103        |            [MASK]            |
|     65640     |     ##muggen     |       104        |             <S>              |
|     65641     |     ##льним      |       105        |             <T>              |
|     65642     |      Crimea      |       106        |              !               |
|     65643     |      altor       |       107        |              "               |
|     65644     |      chefo       |       108        |              #               |
|     65645     |     persoon      |       109        |              $               |
|     65646     |      ##зія       |       110        |              %               |
|     65647     |      новое       |       111        |              &               |
|     65648     |       ##šť       |       112        |              '               |
|     65649     |       ##황        |       113        |              (               |
|     65650     |      fisica      |       114        |              )               |
|     65651     |     ##ținut      |       115        |              *               |
|     65652     |      Woche       |       116        |              +               |
|     65653     |    angesehen     |       117        |              ,               |
|     65654     |       Mach       |       118        |              -               |
|     65655     |       TNT        |       119        |              .               |
|     65656     |    obiettivo     |       120        |              /               |
|     65657     |      ##ceno      |       121        |              0               |
|     65658     |      ##מכון      |       122        |              1               |
|     65659     |    Tallinnas     |       123        |              2               |
|     65660     |      graet       |       124        |              3               |
|     65661     |      straal      |       125        |              4               |
|     65662     |     Pulitzer     |       126        |              5               |
|     65663     |       прво       |       127        |              6               |
|     65664     |     ##laska      |       128        |              7               |
|     65665     |      Actors      |       129        |              8               |
|     65666     |     Daimler      |       130        |              9               |
|     65667     |     estadual     |       131        |              :               |
|     65668     |       ##ಃ        |       132        |              ;               |
|     65669     |     resultó      |       133        |              <               |
|     65670     |      Tokom       |       134        |              =               |
|     65671     |  Parliamentary   |       135        |              >               |
|     65672     |       Phật       |       136        |              ?               |
|     65673     |      liście      |       137        |              @               |
|     65674     |      ##ерна      |       138        |              A               |

## Expected behavior

binarize_data.py should use typecasting to int32 at least to avoid incorrect behavior

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.3.0 (distillation code from current master branch)
- Platform: GNU/Linux Fedora 5.4.13-201.fc31.x86_64
- Python version: Python 3.6.9 :: Anaconda, Inc.
- PyTorch version (GPU?): 1.4.0 GPU
- Tensorflow version (GPU?): not applicable
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


GPT-2 language model: multiplying decoder-transformer output with token embedding or another weight matrix

Core: Modeling

Internals of the library; Models.

PyTorch

Anything PyTorch

I was reading the code of GPT2 language model. The transformation of hidden states to the probability distribution over the vocabulary has done in the following line:

`lm_logits = self.lm_head(hidden_states)`
Here,

`self.lm_head = nn.Linear(config.n_embd, config.vocab_size, bias=False)`
However, In the original paper, they suggested multiplying hidden states with the token embedding matrix whereas huggingface implementation used another matrix.

Is there any advantage of this? Am I missing something?


Update run_tf_squad.py


bugs in xlnet XLNetLMHeadModel

Core: Modeling

Internals of the library; Models.

PyTorch

Anything PyTorch

# 🐛 Bug

## Information

Model I am using  XLNet :

Language I am using the model on English :

The problem arises when using:
* [True ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. just deep into modeling_xlnet.py code;
2. browsing  https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_xlnet.py#L1057;
3. comparing with other LMHead,such as https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py#L603;
4.you'll found inputs and labels are not shifted.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
the code should be refactored to the same style as other LMHeadModels
and because xlnet tokenizer appends <'sep'> , <'cls'> on the end, so the num tokens shifted should be 2, so the think the code should be:
        if labels is not None:
            # Shift so that tokens < n predict n
            shift_logits = logits[..., :-2, :].contiguous()
            shift_labels = labels[..., 1:-1].contiguous()
            # Flatten the tokens
            loss_fct = CrossEntropyLoss()
            loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1))
            # loss = loss_fct(logits.view(-1, logits.size(-1)), labels.reshape(-1))
            outputs = (loss,) + outputs

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.4.1
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


CUDA out of memory issue in the middle of training in run_language_modeling.py (say after 1000 steps).

Ex: LM (Finetuning)

Related to language modeling fine-tuning

Need more information

Further information is requested

# 🐛 Bug
CUDA OOM in run_language_modeling.py after many steps.
## Information
It seems strange to get them so late in the training procedure.
Model I am using (Bert, XLNet ...):
roberta-large
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: run_language_modeling.py, with line_by_line parameter

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: Wikitext after some filter of sentences (attached)
[wiki.train.raw.time_filter.normalized.text.txt](https://github.com/huggingface/transformers/files/4191154/wiki.train.raw.time_filter.normalized.text.txt)

## To reproduce
just run the script with the line_by_line parameter
Steps to reproduce the behavior:

- `transformers` version: latest
- Platform:
- Python version: 3.7.0
- PyTorch version (GPU?): latest
- Tensorflow version (GPU?):
- Using GPU in script?: yes - TITAN Xp
- Using distributed or parallel set-up in script?: no


transformers==2.4.1;
tensoflow==2.1.0;
torch==1.4.0;
when i start with follow code get some error and i didn't have tensoflow-gpu.

**code**
---------------------------------------------------------------------------
from transformers import (
    TF2_WEIGHTS_NAME,
    BertConfig,
    BertTokenizer,
    DistilBertConfig,
    DistilBertTokenizer,
    GradientAccumulator,
    RobertaConfig,
    RobertaTokenizer,
    TFBertForTokenClassification,
    TFDistilBertForTokenClassification,
    TFRobertaForTokenClassification,
    create_optimizer,
)

**error**
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-2-4dd6bfff15e9> in <module>()
     12 from seqeval import metrics
     13
---> 14 from transformers import (
     15     TF2_WEIGHTS_NAME,
     16     BertConfig,

ImportError: cannot import name 'GradientAccumulator'


Create card for model bert-base-spanish-wwm-cased-finetuned-spa-squad2-es.md

model card

Related to pretrained model cards


Albert multilingual

Need more information

Further information is requested

# 🚀 Feature request

Provide multilingual pre-trained albert model.

## Motivation
Albert is a light weighted bert. It would be nice if it has the multilingual version.

## Your contribution


GPT2LMHeadModel with variable length batch input

Help wanted

Extra attention is needed, help appreciated

PyTorch

Anything PyTorch

I'm trying to repurpose the GPT2LMHeadModel for a seq2seq-like task, where I have an input prompt sequence of length L and I'm trying to ask the model to output a sequence to match a target sequence/sentence.
For a single input-output pair, I simply change the original code of
`shift_logits = lm_logits[..., :-1, :].contiguous()`
to
`shift_logits = lm_logits[..., L-1:-1, :].contiguous()`

But I'm a bit lost on how can I do this for a batch of variable length input. Even if I pad the shorter sequences, I would need to shift the logits by different amount for each input. I'm also uncertain if I need to do something about the attention mask. Any tip is appreicated!


Proposal: Update examples to utilize a new format.

This PR creates a new example coding style for the pytorch code.

* Uses pytorch-lightning for the underlying training.
* Separates out the base transformer loading from the individual training.
* Moves each individual example to its own directory.
* Move the code in the readme to bash scripts.

The only two new files are `run_pl_ner.py` and `transformers_base.py`.

The goal is to keep the same format as the original command-line. Most of the argument names are preserved. I have verified that for NER the results of the same on GPU.

There are several nice benefits of lightning -> somewhat nicer logging and library integration (e.g. wandb), auto-checkpointing. Mostly the goal though is code readability with identical functionality.

Todo:
* make sure that the output file format is identical.
* print test results after training.
* test multi-gpu and apex (in theory these should work)


Add more specific testing advice to Contributing.md


Repository with recipes how to pretrain model from scratch on my own data

# 🚀 Feature request

It would very useful to have documentation on how to train different models, not necessarily with the use of transformers, but  with use external libs (like original BERT, fairseq, etc)

Maybe another repository with readmes or docs with recipes from those who already pretrain their model in order to reproduce procedure for other languages or domain.
There are many external resources (blogs, articles in arxiv) but without any details and very often they are not reproducible.

## Motivation
Have a proven recipe for training the model. Make it easy for others to train a custom model. The community will easily train language or domain-specific models.
More models available in transformers library.

There are many issues related to this:
* https://github.com/huggingface/transformers/issues/1283
* https://github.com/huggingface/transformers/issues/2301
* https://github.com/huggingface/transformers/issues/1672
* https://github.com/huggingface/transformers/issues/1714
* https://github.com/huggingface/transformers/issues/1638
* https://github.com/huggingface/transformers/issues/2279
* https://github.com/huggingface/transformers/issues/1108
* https://github.com/huggingface/transformers/issues/1175
* https://github.com/huggingface/transformers/issues/1381
* https://github.com/huggingface/transformers/issues/1547
* https://github.com/huggingface/transformers/issues/1999
* #1908
* #417
* #170


PreTrainedModel.generate do_sample default argument is wrong in the documentation

Core: Modeling

Internals of the library; Models.

PyTorch

Anything PyTorch

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2LMHeadModel

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create GPT2LMHeadModel and GPT2Tokenizer objects using the from_pretrained('gpt2') method
2. Use the generate function to generate sequences without any input argument multiple times, and then repeat by setting do_sample = True
3. When do_sample is set to False (or is not supplied at all), the generate method constantly generates the following string:
'!\n\nThe first thing I did was to make a list of all the things I would!'

When generating with do_sample set to True, changing results are outputted. This is consistent with the behaviour described in the code, except for the default value of do_sample

Code sample from a python shell:
'''python
>>> model = transformers.GPT2LMHeadModel.from_pretrained('gpt2')
>>> g2t = transformers.GPT2Tokenizer.from_pretrained('gpt2')
>>> g2t.decode(model.generate()[0])
'!\n\nThe first thing I did was to make a list of all the things I would!'
>>> g2t.decode(model.generate()[0])
'!\n\nThe first thing I did was to make a list of all the things I would!'
>>> g2t.decode(model.generate()[0])
'!\n\nThe first thing I did was to make a list of all the things I would!'
>>> g2t.decode(model.generate(do_sample=True)[0])
"!, I can't help but wonder how she's doing. I really have no idea.!"
>>> g2t.decode(model.generate(do_sample=True)[0])
'!\n\nThe other guy is trying to take something away from the guy before you even start!'
>>> g2t.decode(model.generate(do_sample=True)[0])
'! are you kidding me?\n\n\nBut maybe you should wait for his own "last act!'
'''

Similarly, you can do print((transformers.GPT2LMHeadModel.from_pretrained('gpt2')).config.do_sample) to verify that the 'default' argument is in fact False

## Expected behavior
The documentation should say do_sample is False by default OR the config should be updated to be in line with the documentation

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.4.1
- Platform: Ubuntu GNU/Linux 18.04
- Python version: Python 3.6.9 (default, Nov  7 2019, 10:44:02) [GCC 8.3.0] on linux
- PyTorch version (GPU?): 1.4.0 GPU version with Nvidia RTX 2080Ti
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


How can I finetune the BERTModel on my own corpus?

Core: Modeling

Internals of the library; Models.

Usage

General questions about the library

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Thanks for your code!

I want to fine-tune the BERT model on my own corpus which has a smaller vocabulary than the default size of 30522, and my final goal is to obtain a fine-tuned and personalized BERT model which can provide proper word embedding for future top tasks. In short, I need to fine-tune the BERTModel for providing word embedding based on my own corpus.

How can I build a new vocabulary and then fetch the embeddings from the provided pre-trained model, e.g., bert-base-uncased, and then fine-tune the model on my own corpus?

Have you provided functions for building vocabulary and further fine-tuning?


How to use a batch size bigger than zero in Bert Sequence Classification

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
[Hugging Face documentation describes](https://huggingface.co/transformers/model_doc/bert.html#bertforsequenceclassification) how to do a sequence classification using a Bert model:

```
from transformers import BertTokenizer, BertForSequenceClassification
import torch

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained('bert-base-uncased')

input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
labels = torch.tensor([1]).unsqueeze(0)  # Batch size 1
outputs = model(input_ids, labels=labels)

loss, logits = outputs[:2]
```

However, there is only example for batch size 1. How to implement it when we have a list of phrases and want to use a bigger batch size?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/60170037/how-to-use-a-batch-size-bigger-than-zero-in-bert-sequence-classification


How to get longer output for summary?

General questions about the library

#Question
https://stackoverflow.com/questions/60157959/transformers-summarization-with-python-pytorch-how-to-get-longer-output

Should i train it myself to get summary output longer than used in original training script?
:

    python run_summarization.py \
        --documents_dir $DATA_PATH \
        --summaries_output_dir $SUMMARIES_PATH \ # optional
        --no_cuda false \
        --batch_size 4 \
        --min_length 50 \
        --max_length 200 \
        --beam_size 5 \
        --alpha 0.95 \
        --block_trigram true \
        --compute_rouge true

When i do inference with

    --min_length 500 \
    --max_length 600 \

I got a good output for 200 tokens, but the rest of the text is

    . . . [unused7] [unused7] [unused7] [unused8] [unused4] [unused7] [unused7]  [unused4] [unused7] [unused8]. [unused4] [unused7] . [unused4] [unused8] [unused4] [unused8].  [unused4]  [unused4] [unused8]  [unused4] . .  [unused4] [unused6] [unused4] [unused7] [unused6] [unused4] [unused8] [unused5] [unused4] [unused7] [unused4] [unused4] [unused7]. [unused4] [unused6]. [unused4] [unused4] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8] [unused6] [unused4]   [unused4] [unused4]. [unused4].  [unused5] [unused4] [unused8] [unused7] [unused4] [unused7] [unused9] [unused4] [unused7]  [unused4] [unused7] [unused5] [unused4]  [unused5] [unused4] [unused6]  [unused4]. .  . [unused5]. [unused4]  [unused4]   [unused4] [unused6] [unused5] [unused4] [unused4]  [unused6] [unused4] [unused6]  [unused4] [unused4] [unused5] [unused4]. [unused5]  [unused4] . [unused4]  [unused4] [unused8] [unused8] [unused4]  [unused7] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8]  [unused4]  [unused8] [unused4] [unused6]


Fix typo in src/transformers/data/processors/squad.py

end end -> and end


Multiple Choice BERT, SWAG task, failure to test

PyTorch

Anything PyTorch

Usage

General questions about the library

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x] the official example scripts: (give details below)
* [ x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x ] an official GLUE/SQUaD task: (give the name)
* [x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Download SWAG dataset and put it in some directory and set the path by `export SWAG_DIR=path/to/swag/dir`
2. Copy `run_multiple_choice.py` and `utils_multiple_choice.py`
3. Run the code only for testing with the following command
`./run_multiple_choice.py --model_type bert --task_name swag --model_name_or_path bert-base-uncased --do_lower_case --max_seq_length 80 --output_dir models_bert/swag_testing --data_dir $SWAG_DIR --do_test`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
```
Traceback (most recent call last):
  File "./run_multiple_choice.py", line 678, in <module>
    main()
  File "./run_multiple_choice.py", line 669, in main
    result = evaluate(args, model, tokenizer, prefix=prefix, test=True)
  File "./run_multiple_choice.py", line 248, in evaluate
    eval_dataset = load_and_cache_examples(args, eval_task, tokenizer, evaluate=not test, test=test)
  File "./run_multiple_choice.py", line 354, in load_and_cache_examples
    examples = processor.get_test_examples(args.data_dir)
  File "utils_multiple_choice.py", line 168, in get_test_examples
    "For swag testing, the input file does not contain a label column. It can not be tested in current code"
ValueError: For swag testing, the input file does not contain a label column. It can not be tested in current codesetting!
```

In the code it says for testing there is no need for label column, but it doesn't work with or without it. It does not work with the default `test.csv` file (it will be called by default for testing if it is in the directory), but it also does not work with `val.csv` (has label column).

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `Transformers` version: current
- Platform: Linux
- Python version: 3.7.5
- Using GPU in script?:  yes
- Using distributed or parallel set-up in script?:


get_activation('relu') provides a simple mapping from strings in configs to activation functions


TFBertModel.from_pretrained('neuralmind/bert-base-portuguese-cased') -> TypeError

Need more information

Further information is requested

TensorFlow

Anything TensorFlow

Usage

General questions about the library

I just installed the library on a TensorFlow environment (2.0.0-rc1) and there is no `BertModel` in `transformers`.

Is `TFBertModel` equivalent? If so, then I get the error `TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType` when loading the model with `model = TFBertModel.from_pretrained('neuralmind/bert-base-portuguese-cased')`.

- `transformers` version: 2.4.1
- Platform: Windows 10
- Python version: 3.7.6
- Tensorflow version (GPU?): 2.0.0-rc1 (it automatically uses GPU now)
- Using GPU in script?: No, just importing.
- Using distributed or parallel set-up in script?: No.


[model_cards] Add new German Europeana BERT models

model card

Related to pretrained model cards

Hi,

this PR adds the model cards for two new BERT models for Historic German.

The cased and uncased BERT models were trained on a huge corpus: newspapers from [Europeana](http://www.europeana-newspapers.eu/). Time period of these (noisy) OCRed newspapers is 18th - 20th century.

More information can be found [here](https://github.com/dbmdz/berts) and more detailed results on downstream tasks [here](https://github.com/stefan-it/europeana-bert).


Fix a few issues regarding the language modeling script

The language modeling script currently has a few issues.

- in the line by line dataset, no special tokens are added (that's due to the fact the `batch_encode_plus` has the `add_special_token` flag `False` by default, which is misleading).
- the max length is ill computed in that same dataset, as it doesn't take into account the fact that `encode_plus` is aware of the special tokens and their impact on the sequence length.


Support DeepSpeed for language modeling finetuning

External

Using the library with external tools (onnx, tflite, ...)

# 🚀 Feature request

https://github.com/microsoft/DeepSpeed

This was just released, and given the code flow in `run_language_modeling.py` it seems like it would not be too difficult to drop-in, and it has a permissible license (MIT).

However, given the dependencies and difficulty installing them, it would likely have to be done in a separate file.

## Motivation

![](https://i.kym-cdn.com/entries/icons/mobile/000/006/360/gottago.jpg)


FlauBERT lang embeddings only when n_langs > 1


Can't load pre-trained Flaubert model

Core: Tokenization

Internals of the library; Tokenization.

Need more information

Further information is requested

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Flaubert

Language I am using the model on (English, Chinese ...): French

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load a pre-trained model
2.
3.

I'm following the guide from https://huggingface.co/transformers/model_doc/flaubert.html#flaubertmodel:
```
import transformers
tokenizer = transformers.FlaubertTokenizer.from_pretrained('flaubert-base-cased')
```

```
Traceback (most recent call last):
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\IPython\core\interactiveshell.py", line 3326, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-05c64572fe39>", line 2, in <module>
    tokenizer = transformers.FlaubertTokenizer.from_pretrained('flaubert-base-cased')
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\tokenization_utils.py", line 309, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\tokenization_utils.py", line 410, in _from_pretrained
    list(cls.vocab_files_names.values()),
OSError: Model name 'flaubert-base-cased' was not found in tokenizers model name list (flaubert-large-cased, flaubert-base-uncased, flaubert-small-cased, flaubert-base-cased). We assumed 'flaubert-base-cased' was a path, a model identifier, or url to a directory containing vocabulary files named ['merges.txt', 'vocab.json'] but couldn't find such vocabulary files at this path or url.

```

## Expected behavior

`tokenizer` should be a `FlaubertTokenizer` object

## Environment info

Well calling `python transformers-cli env` gave me another error:

```
(venv) C:\Users\PLHT09191\Documents\work\dev\Classif_Annonces\venv\Scripts>python transformers-cli env
Traceback (most recent call last):
  File "transformers-cli", line 4, in <module>
    __import__('pkg_resources').run_script('transformers==2.4.1', 'transformers-cli')
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\setuptools-40.8.0-py3.5.egg\pkg_resources\__init__.py", line 666, in run_script
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\setuptools-40.8.0-py3.5.egg\pkg_resources\__init__.py", line 1446, in run_script
  File "c:\users\myself\documents\work\dev\classif_annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\EGG-INFO\scripts\transformers-cli", line 6, in <module>
    from transformers.commands.user import UserCommands
  File "C:\Users\myself\Documents\work\dev\Classif_Annonces\venv\lib\site-packages\transformers-2.4.1-py3.5.egg\transformers\commands\user.py", line 163
    entries: List[os.DirEntry] = list(os.scandir(rel_path))
           ^
SyntaxError: invalid syntax
```

- `transformers` version: 2.4.1
- Platform: Windows 64 bits
- Python version: Python 3.5.2
- PyTorch version (GPU?): torch.__version__ = 1.4.0+cpu
- Tensorflow version (GPU?):
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


CircleCI doesn't run slow tests

circle_ci.cfg says `RUN_SLOW: yes`, but all my circleci runs have the slow tests skipped.
Is this expected behavior?
@LysandreJik


Add model readme for bert-base-german-cased

model card

Related to pretrained model cards

Adding a readme for our German BERT model. Not sure if the file location is correct, as the model was added before model hub / user name spaces were created.


Reduce the CamemBERT dimensions

Usage

General questions about the library

I want to reduce the output dimension by adding a linear layer at the end of the camem model.

code :
```python
from transformers import CamembertTokenizer, CamembertModel
import torch
from torch.nn import Sequential, Linear

tokenizer = CamembertTokenizer.from_pretrained('camembert-base')
model = CamembertModel.from_pretrained('camembert-base')
input_ids = torch.tensor(tokenizer.encode("La pose d'un panneau stop.", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
# labels = torch.tensor([1] * input_ids.size(1)).unsqueeze(0)  # Batch size 1
model = Sequential(model, Linear(768, 256))

outputs = model(input_ids)

print(input_ids)
print(outputs[1].size())
print(outputs[0].size())
```
I got this :

```shell
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1366         - Output: :math:`(N, *, out\_features)`
   1367     """
-> 1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
   1370         ret = torch.addmm(bias, input, weight.t())

AttributeError: 'tuple' object has no attribute 'dim'
```

Additionally, I want to do a word-level embedding, however, the 768 dimensions is too big from my point.

Thanks for your helps.


Add model readme for deepset/roberta-base-squad2

model card

Related to pretrained model cards

Adding a model readme for https://huggingface.co/deepset/roberta-base-squad2


output padding different to zero in hidden layers with attention mask

Core: Modeling

Internals of the library; Models.

PyTorch

Anything PyTorch

Usage

General questions about the library

# 🐛 Bug
On the last layer, the token corresponding to the padding does not return 0, even if attention masking is used.
## Information

Model I am using (Bert, XLNet ...):
roberta and roberta XLM

Language I am using the model on (English, Chinese ...):
english

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
looking at the output of the last layer of RobertaModel

## To reproduce

Steps to reproduce the behavior:

1. use some padding in your inputs data
2. create accordingly the attention mask
3.

example of code :
```

def tokenize_sentences_Bert(sentences, tokenizer, maxlen):
    tokens = []
    lengths = []
    for s in sentences:
        token = tokenizer.encode(s, add_special_tokens=True, max_length=maxlen)
        lengths.append(len(token))
        token = token + [tokenizer.pad_token_id]  * (maxlen - len(token) )

        tokens.append(token)

    return tokens, lengths
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = RobertaModel.from_pretrained('roberta-base',
                                          output_hidden_states=False,
                                          output_attentions=True)
max_length = 10
sequence = ["I eat a green apple", "I am playing football tomorrow"]
tokens, lengths = tokenize_sentences_Bert(sequence, tokenizer, maxlen=max_length)

lengths = torch.tensor(lengths)
tokens = torch.tensor(tokens)
attention_mask = (torch.arange(max_length).expand(len(lengths), max_length) < lengths.unsqueeze(1)).float()
print(attention_mask)
outputs = model(tokens, attention_mask=attention_mask)
print(outputs[0][:, :, :2]) # last step should return 0 last hidden layers
```

- `transformers` version:
- Platform: ubuntu
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Tensorflow version (GPU?):
- Using GPU in script? no:
- Using distributed or parallel set-up in script?: no


Probably a bug in XLMRobertaTokenizer

Core: Tokenization

Internals of the library; Tokenization.

(Everything goes perfect when I did experiment with MultilingualBert, but seems only the base-model is released.)

When using XLM-R, the according tokenizer (XLMRobertaTokenizer) converts \<unk\> and every OOV token into id = 1. However, 1 should be the number of \<pad\>. (And the tokenizer can convert 1 to \<pad\>, 3 to \<unk\>).


TensorFlow

Anything TensorFlow

Usage

General questions about the library

# You must specify an aggregation method to update a MirroredVariable in Replica Context.

##
   <ipython-input-28-7cf32baaf070>:52 step_fn  *
        gradient_accumulator(grads)
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/distribute/distribute_lib.py:763 experimental_run_v2
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/transformers/optimization_tf.py:229 __call__  *
        accum_gradient.assign_add(gradient)
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/distribute/values.py:1124 assign_add
        return self._assign_func(f=assign_add_fn, *args, **kwargs)
    /tensorflow-2.1.0/python3.6/tensorflow_core/python/distribute/values.py:1108 _assign_func
        variable_type="MirroredVariable"))

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

using GPU: Yes

The problem arises when using:
 Training on multiple gpus and accumulating gradient as given in run_tf_ner.py


Fix circleci cuInit error on Tensorflow >= 2.1.0.

Tests

Related to tests

Tensorflow 2.1.0 introduce a new dependency model where `pip install tensorflow` would install tf **with GPU support**. Before 2.1.0 it would just install with CPU support only.

CircleCI machines are running without GPU hardware so, at initialisation, TensorFlow tests are looking for NVidia driver version but fails as their is no NVidia Driver running.

This PR introduces an extra (optional) dependency **tf-cpu** which explicitly requires **tensorflow-cpu** and makes sure it `pip install tf-cpu` instead of `pip install tf` while running unit tests.

It should remove the following error on CircleCI:

```bash
tests/test_modeling_tf_bert.py::TFBertModelTest::test_attention_outputs 2020-02-10 11:14:08.280770: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory
2020-02-10 11:14:08.280808: E tensorflow/stream_executor/cuda/cuda_driver.cc:351] failed call to cuInit: UNKNOWN ERROR (303)
2020-02-10 11:14:08.280837: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (40403e139ccb): /proc/driver/nvidia/version does not exist
2020-02-10 11:14:08.281093: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 AVX512F FMA
```

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


tiny issue with distilbertconfig docs

Usage

General questions about the library

# 🐛 Bug (barely)

Discrepancy in variable names between docs and code:
I presume [``intermediate_size``](https://github.com/huggingface/transformers/blob/520e7f211926e07b2059bc8e21b668db4372e4db/src/transformers/configuration_distilbert.py#L63) refers to [``hidden_dim``](https://github.com/huggingface/transformers/blob/520e7f211926e07b2059bc8e21b668db4372e4db/src/transformers/configuration_distilbert.py#L109)?


Create BERT-of-Theseus model card

model card

Related to pretrained model cards


Is there any way that I can directly feed the hidden output of the embedding layer into each of the transformer's layer?

PyTorch

Anything PyTorch

Usage

General questions about the library

Hello,

For an original sequence ```X``` that has length of ```n```, I am interested in feeding the embedding of the original sequence ```X``` (```E```) as an input to the self-attention block of each layer of ```GPT2LMHeadModels``` (here, layer = self-attention block + feedforward block), and examine the layer output generated by ```E```.

Is there any way that I can carry out this task with HuggingFace ```GPT2LMHeadsModel``` transformers?

Thank you,


Is there any way that I can extract the hidden output from the self-attention layer?

PyTorch

Anything PyTorch

Usage

General questions about the library

Hello,

From my understanding, for the ```GPT2LMHeadModel```, the output ```past``` allows me to retrieve the key and value vectors that are used in the self-attention block (which is prior to the feedforward block).

Is there any way I can extract the output of the self-attention block **at a particular head of a single layer** of ```GPT2LMHeadModel``` (if I am understanding this correctly, the output ```hidden_states``` only returns the output after the input had gone into the feedforward block... but what I want is to extract the output from the self-attention block, which happens before the feedforward block).

Thank you,


SQuAD preprocessing not working for roberta (wrong p_mask)

Core: Pipeline

Internals of the library; Pipeline.

Usage

General questions about the library

**Description**
The pipeline for QA crashes for roberta models.
It's loading the model and tokenizer correctly, but the SQuAD preprocessing produces a wrong `p_mask` leading to no possible prediction and the error message below.

The observed `p_mask` for a roberta model is
```[0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...] ```

while it should only mask the question tokens like this
``` [0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, ...]```

I think the deeper root cause here is that roberta's `token_type_ids` returned from `encode_plus` are now all zeros (introduced in https://github.com/huggingface/transformers/pull/2432) and the creation of `p_mask` in `squad_convert_example_to_features` relies on this information:
https://github.com/huggingface/transformers/blob/520e7f211926e07b2059bc8e21b668db4372e4db/src/transformers/data/processors/squad.py#L189-L202
Haven't checked yet, but this might also affect training/eval if `p_mask` is used there.

**How to reproduce?**
```
model_name = "deepset/roberta-base-squad2"
nlp = pipeline('question-answering', model=model_name, tokenizer=model_name)
res = nlp({
    'question': 'What is roberta?',
    'context': 'Roberta is a language model that was trained for a longer time, on more data, without NSP'
})
```
results in
```
  File "/home/mp/deepset/dev/transformers/src/transformers/pipelines.py", line 847, in __call__
    for s, e, score in zip(starts, ends, scores)
  File "/home/mp/deepset/dev/transformers/src/transformers/pipelines.py", line 847, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
KeyError: 0
```

**Environment**
- Ubuntu 18.04
- Python 3.7.6
- PyTorch 1.3.1


Distillation code loss functions

Core: Modeling

Internals of the library; Models.

Distillation

Related to model distillation

PyTorch

Anything PyTorch

Usage

General questions about the library

# ❓ Questions & Help
Why compute cross entropy loss from the hard labels in distillation code?

if self.alpha_clm > 0.0:
            shift_logits = s_logits[..., :-1, :].contiguous()
            shift_labels = lm_labels[..., 1:].contiguous()
            loss_clm = self.lm_loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
            loss += self.alpha_clm * loss_clm

The model outputs loss when passed with the labels.

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


SequenceSummary: config.summary_activation = 'relu' would be ignored

Core: Modeling

Internals of the library; Models.

This isn't a bug, but merely an unintuitive argument name.

`summary_activation` sounds like it can be general, e.g. "relu" or "gelu" or something, but if its not "tanh" its ignored.

Since I assume it's annoying to go through all the configs and rename a field to use_tanh=True, I propose that we raise if summary_activation is a string thats not tanh, instead of silently just using no activation.

Another approach could integrate an `ACT2FN` dictionary (see  https://github.com/huggingface/transformers/issues/1347)
to actually support the other activation functions.

Happy to do either approach if others think it would be useful.


Create README.md

model card

Related to pretrained model cards

Albert xxlarge version 1 language model fine-tuned on SQuAD2.0 with the following results:
```
 {'exact': 85.65653162637918,
 'f1': 89.260458954177,
 'total': 11873,
 'HasAns_exact': 82.6417004048583,
 'HasAns_f1': 89.85989020967376,
 'HasAns_total': 5928,
 'NoAns_exact': 88.66274179983179,
 'NoAns_f1': 88.66274179983179,
 'NoAns_total': 5945,
 'best_exact': 85.65653162637918,
 'best_exact_thresh': 0.0,
 'best_f1': 89.2604589541768,
 'best_f1_thresh': 0.0}
```
with script:
```
python -m torch.distributed.launch --nproc_per_node=2 ${RUN_SQUAD_DIR}/run_squad.py \
  --model_type albert \
  --model_name_or_path albert-xxlarge-v1 \
  --do_train \
  --train_file ${SQUAD_DIR}/train-v2.0.json \
  --predict_file ${SQUAD_DIR}/dev-v2.0.json \
  --version_2_with_negative \
  --num_train_epochs 3 \
  --max_steps 8144 \
  --warmup_steps 814 \
  --do_lower_case \
  --learning_rate 3e-5 \
  --max_seq_length 512 \
  --doc_stride 128 \
  --save_steps 2000 \
  --per_gpu_train_batch_size 1 \
  --gradient_accumulation_steps 24 \
  --output_dir ${MODEL_PATH}

CUDA_VISIBLE_DEVICES=0 python ${RUN_SQUAD_DIR}/run_squad.py \
  --model_type albert \
  --model_name_or_path ${MODEL_PATH} \
  --do_eval \
  --train_file ${SQUAD_DIR}/train-v2.0.json \
  --predict_file ${SQUAD_DIR}/dev-v2.0.json \
  --version_2_with_negative \
  --do_lower_case \
  --max_seq_length 512 \
  --per_gpu_eval_batch_size 48 \
  --output_dir ${MODEL_PATH}
```
using the following system & software:
```
OS/Platform: Linux-4.15.0-76-generic-x86_64-with-debian-buster-sid
GPU/CPU: 2 x NVIDIA 1080Ti / Intel i7-8700
Transformers: 2.3.0
PyTorch: 1.4.0
TensorFlow: 2.1.0
Python: 3.7.6
```
Inferencing/prediction works with the current Transformers v2.4.1

Access this `albert_xxlargev1_sqd2_512` fine-tuned model with "tried & true" code:
```
config_class, model_class, tokenizer_class = \
            AlbertConfig, AlbertForQuestionAnswering, AlbertTokenizer

model_name_or_path = "ahotrod/albert_xxlargev1_squad2_512"
config = config_class.from_pretrained(model_name_or_path)
tokenizer = tokenizer_class.from_pretrained(model_name_or_path, do_lower_case=True)
model = model_class.from_pretrained(model_name_or_path, config=config)
```
or the AutoModels (AutoConfig, AutoTokenizer & AutoModel) should also work, however I
have yet to use them in my app & confirm:
```
from transformers import AutoConfig, AutoTokenizer, AutoModel

model_name_or_path = "ahotrod/albert_xxlargev1_squad2_512"
config = AutoConfig.from_pretrained(model_name_or_path)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, do_lower_case=True)
model = AutoModel.from_pretrained(model_name_or_path, config=config)
```


ERROR:CUDA out of memory when using GPT2 tour

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

I follow the tour in the program.Everything is OK . When i start the train,an error occurred.

RuntimeError:
CUDA out of memory. Tried to allocate 12.00 MiB (GPU 0; 6.00 GiB total capacity; 4.44 GiB already allocated; 3.06 MiB free; 4.57 GiB reserved in total by PyTorch)
I'm using GTX2060(6GB) i'd like to know whether this GPU is quite qualified for the work.
Thanks


Features proposals to simplify training Tensorflow model

TensorFlow

Anything TensorFlow

# 🚀 Feature request
Hello,

On my side I thought to implement some features to simplify the way we can train Tensorflow models (I think it can certainly be adapted to Pytorch as well), and I wanted to know if it might be useful for you. Here a non exhaustive list of features I have in mind:

1. Augment the tranining pipeline with some useful functions such as:
- An [LR finder](https://docs.fast.ai/callbacks.lr_finder.html) that will try to find the best LR for a specific dataset
- [Approach to help to set better hyperparameters](https://arxiv.org/abs/1803.09820)
- [Cyclical LR during training](https://arxiv.org/abs/1506.01186)
- Augment the config.json file of each model with specific training parameters (epochs, LR, batch size, GPUs, etc...) in order to better reproduce a specific training.

2. Modify a bit the `TFPretrainedModel` class in order to better handle:
- multiple GPU training
- Custom training loop
- Custom optimizer creation
- Gradient Accucmulation
- Add a checkpoint manager
- Handle Tensorboard

3. Modify few model classes to add custom loss computation such as for the NER as I have done in the TF example.

I don't know if it sounds interesting for you @thomwolf, @julien-c and @LysandreJik ?


RoBERTaMultiChoice does not work with `roberta-large`

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**roberta-large**
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [* ] the official example scripts:
https://github.com/huggingface/transformers/tree/6c1b23554f8bb5b5e1f6c80969acab764c755678/examples#multiple-choice
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ *] an official GLUE/SQUaD task: **SWAG**
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
just follow the example code of running swag dataset but with `robert-large` instead of `roberta-base` (which works well)

```
export SWAG_DIR=~/swagaf-master/data/
python ./examples/run_multiple_choice.py \
--model_type roberta \
--task_name swag \
--model_name_or_path roberta-large \
--do_train \
--do_eval \
--do_lower_case \
--data_dir $SWAG_DIR \
--learning_rate 5e-5 \
--num_train_epochs 3 \
--max_seq_length 80 \
--output_dir tmp/swag_base \
--per_gpu_eval_batch_size=16 \
--per_gpu_train_batch_size=16 \
--gradient_accumulation_steps 2 \
--overwrite_output
```

And it will say:

02/08/2020 00:46:23 - INFO - transformers.modeling_utils -   **Weights of RobertaForMultipleChoice not initialized from pretrained model:** ['classifier.weight', 'classifier.bias']

02/08/2020 00:46:23 - INFO - transformers.modeling_utils -   **Weights from pretrained model not used in RobertaForMultipleChoice:** ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']

Consequently, the script is to learn a model from the scratch instead of fine-tuning pre-trained roberta-large.

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
It should load the pre-trained weights of roberta-large model.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.4.1
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): n/a
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Flaky TF pipelines test on CircleCI

Core: Pipeline

Internals of the library; Pipeline.

TensorFlow

Anything TensorFlow

Tests

Related to tests

Environment: CircleCI

Test: `tests/test_pipelines.py::MultiColumnInputTestCase::test_tf_question_answering`

Traceback: https://circleci.com/gh/huggingface/transformers/15691?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link

Diff where I changed nothing relevant and the test started passing: https://github.com/huggingface/transformers/pull/2745/commits/a4edf2e878d23346f45715ac213f1f870ae8ec0c

Happy to look deeper if helpful!


Pipelines- if initial model download is interrupted, everything is ruined

Core: Pipeline

Internals of the library; Pipeline.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): pipeline('ner') and pipeline('feature-extraction')

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
  - [x] my own modified scripts: (give details below)

The tasks I am working on is:
 - [x] (mostly NA) my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. fresh install transformers from source
2.  run:

```python
from transformers import pipeline
model =pipeline('feature-extraction')
```

3. interrupt download.  rerun #2

Error on reload:

```python
Downloading: 100%
230/230 [00:01<00:00, 136B/s]

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
~/miniconda3/envs/hugging/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    466             try:
--> 467                 state_dict = torch.load(resolved_archive_file, map_location="cpu")
    468             except Exception:

~/miniconda3/envs/hugging/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module)
    357     try:
--> 358         return _load(f, map_location, pickle_module)
    359     finally:

~/miniconda3/envs/hugging/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module)
    548         assert key in deserialized_objects
--> 549         deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
    550         offset = None

RuntimeError: unexpected EOF. The file might be corrupted.

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-26-2fd4b689c1db> in <module>
----> 1 featify=pipeline('feature-extraction')

~/miniconda3/envs/hugging/lib/python3.7/site-packages/transformers/pipelines.py in pipeline(task, model, config, tokenizer, modelcard, **kwargs)
   1084                 "Trying to load the model with Tensorflow."
   1085             )
-> 1086         model = model_class.from_pretrained(model, config=config, **model_kwargs)
   1087
   1088     return task(model=model, tokenizer=tokenizer, modelcard=modelcard, framework=framework, **kwargs)

~/miniconda3/envs/hugging/lib/python3.7/site-packages/transformers/modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    375         for config_class, model_class in MODEL_MAPPING.items():
    376             if isinstance(config, config_class):
--> 377                 return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
    378         raise ValueError(
    379             "Unrecognized configuration class {} for this kind of AutoModel: {}.\n"

~/miniconda3/envs/hugging/lib/python3.7/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    468             except Exception:
    469                 raise OSError(
--> 470                     "Unable to load weights from pytorch checkpoint file. "
    471                     "If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True. "
    472                 )

OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
```

## Expected behavior

Model should (download and) load.

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.4.1
- Platform: WSL
- Python version:  3.7.6.final.0
- PyTorch version (GPU?): 0.4.1 (no)
- Tensorflow version (GPU?): none
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


configuration from custom config file not working

Usage

General questions about the library

I am trying to get the configuration from a custom config file by the following line :
`config = GPT2Config.from_pretrained("./lm/gpt2-xl/lm/my_config.json")`

This is similar to the example of this [page]. But I am getting the following error:
```
OSError: Model name './lm/gpt2-xl/lm/my_config.json' was not found in model name list. We assumed 'https://s3.amazonaws.com/models.huggingface.co/bert/./lm/gpt2-xl/lm/my_config.json/config.json' was a path, a model identifier, or url to a configuration file named config.json or a directory containing such a file but couldn't find any such file at this path or url.
(https://huggingface.co/transformers/main_classes/configuration.html#transformers.PretrainedConfig)
```
Am I missing something?


Preserve spaces in GPT-2 tokenizers

**The issue**: The GPT-2 and RoBERTa tokenizers are incorrectly stripping whitespace following special characters, preventing the BPE encoder from correctly encoding spaces in tokens following RoBERTa `<mask>` and `<unk>` tokens.

```
tokenizer.convert_ids_to_tokens(tokenizer.encode('She likes <mask> cats.'))
# output:    ['<s>', 'She', 'Ġlikes', '<mask>', 'cats', '.', '</s>']
# should be: ['<s>', 'ĠShe', 'Ġlikes', '<mask>', 'Ġcats', '.', '</s>']
```

This makes the model inputs (and therefore outputs) incorrect. This issue manifests itself in the `fill-mask` pipeline where the model erroneously thinks the mask is a prefix to the following word when using RoBERTa:

```
roberta_fillmask = pipeline("fill-mask")
sentence = "She likes <mask> cats."
roberta_fillmask(sentence)
# top predictions: "She likes bobcats.", "She likes pussycats."
```

This PR makes the following changes:
- Preserves trailing whitespace following special tokens
- Inserts a space after the prepended start token when `add_special_tokens` is `True` in `encode()` so that the user doesn't have to include a leading space in the string. This can be overriden with the `add_prefix_space` argument.
- Adds a `framework` argument to the `pipeline` factory function, allowing users to easily specify TF vs PyTorch

After making these changes, the top predictions from the above example become 'She likes cute cats.' and 'She likes her cats.'


distilbert-base-cased

- Weights
- Readmes and docs
- Previous omissions

weights are uploaded on s3, along with modelcards.

@LysandreJik Could you make sure I didn't forget anything?
@mfuntowicz Could have a check on the pipeline part?


Pipeline for text classification

Core: Pipeline

Internals of the library; Pipeline.

Usage

General questions about the library

# 🚀 Feature request
Could  you please add a text classification pipeline?
<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Using fast tokenizers with pipelines

Core: Pipeline

Internals of the library; Pipeline.

Core: Tokenization

Internals of the library; Tokenization.

# 🚀 Feature request
Currently tokenizers are not working with QA pipeline, because they do not have the tokenize method implemented. Speeding up the tokenization would be really beneficial for my application.
<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


embedding index getting out of range while running gpt2-xl model

Core: Modeling

Internals of the library; Models.

Ex: Generation

Natural Language Generation

PyTorch

Anything PyTorch

Usage

General questions about the library

I am trying to run [hugginface][1] gpt2-xl model. I ran code from the [quickstart][2] page that load the small gpt2 model and generate text by the following code:

    ```
    from transformers import GPT2LMHeadModel, GPT2Tokenizer
    import torch

    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained('gpt2')

    generated = tokenizer.encode("The Manhattan bridge")
    context = torch.tensor([generated])
    past = None

    for i in range(100):
        print(i)
        output, past = model(context, past=past)
        token = torch.argmax(output[0, :])

        generated += [token.tolist()]
        context = token.unsqueeze(0)

    sequence = tokenizer.decode(generated)

    print(sequence)
```
This is running perfectly. Then I try to run `gpt2-xl` model.
I changed `tokenizer` and `model` loading code like following:

    ```
tokenizer = GPT2Tokenizer.from_pretrained("gpt2-xl")
    model = GPT2LMHeadModel.from_pretrained('gpt2-xl')
```

The `tokenizer` and `model` loaded perfectly. But I a getting error on the following line:

   ` output, past = model(context, past=past)`
The error is:

   ` RuntimeError: index out of range: Tried to access index 204483 out of table with 50256 rows. at /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
`
Looking at error it seems that the embedding size is not correct. So I write the following line to specifically fetch the config file of `gpt2-xl`:

`    config = GPT2Config.from_pretrained("gpt2-xl")`
But, here `vocab_size:50257`
So I changed explicitly the value by:

  `  config.vocab_size=204483`
Then after printing the `config`, I can see that the previous line took effect in the configuration. But still, I am getting the same error.


How to load a pretrained TF model using AutoModel?

Run the following code:
```
import tensorflow as tf
from transformers import AutoModel, TFBertModel

auto_model = AutoModel.from_pretrained("bert-base-uncased")
tfbert_model = TFBertModel.from_pretrained("bert-base-uncased")

print(auto_model.__class__)
print(tfbert_model.__class__)
```
Then the output is:
```
<class 'transformers.modeling_bert.BertModel'>
<class 'transformers.modeling_tf_bert.TFBertModel'>
```
It seems that AutoModel defaultly loads the pretrained PyTorch models, but how can I use it to load a pretrained TF model?