Pipelines do not control input sequences longer than those accepted by the model

Core: Pipeline

Internals of the library; Pipeline.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): DistilBERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

1. Create a "sentiment-analysis" pipeline with a DistilBERT tokenizer and model
2. Prepare a string that will produce more than 512 tokens upon tokenization
3. Run the pipeline over such input string

```python
from transformers import pipeline

pipe = pipeline("sentiment-analysis", tokenizer='distilbert-base-uncased', model='distilbert-base-uncased')
very_long_text = "This is a very long text" * 100
pipe(very_long_text)
```

## Expected behavior

The pipeline should control in some way that the input string will not overflow the maximum number of tokens the model can accept, for instance by limiting the number of tokens generated in the tokenization step. The user can't control this beforehand, as the tokenizer is run by the pipeline itself and it can be hard to predict into how many tokens a given text will be broken down to.

One possible way of addressing this might be to include optional parameters in the pipeline constructor that are forwarded to the tokenizer.

The current error trace is:

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-ef48faf7ffbb> in <module>
      3 pipe = pipeline("sentiment-analysis", tokenizer='distilbert-base-uncased', model='distilbert-base-uncased')
      4 very_long_text = "This is a very long text" * 100
----> 5 pipe(very_long_text)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    714
    715     def __call__(self, *args, **kwargs):
--> 716         outputs = super().__call__(*args, **kwargs)
    717         scores = np.exp(outputs) / np.exp(outputs).sum(-1, keepdims=True)
    718         return [{"label": self.model.config.id2label[item.argmax()], "score": item.max().item()} for item in scores]

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    469     def __call__(self, *args, **kwargs):
    470         inputs = self._parse_and_tokenize(*args, **kwargs)
--> 471         return self._forward(inputs)
    472
    473     def _forward(self, inputs, return_tensors=False):

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/transformers/pipelines.py in _forward(self, inputs, return_tensors)
    488                 with torch.no_grad():
    489                     inputs = self.ensure_tensor_on_device(**inputs)
--> 490                     predictions = self.model(**inputs)[0].cpu()
    491
    492         if return_tensors:

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/transformers/modeling_distilbert.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds, labels)
    609         """
    610         distilbert_output = self.distilbert(
--> 611             input_ids=input_ids, attention_mask=attention_mask, head_mask=head_mask, inputs_embeds=inputs_embeds
    612         )
    613         hidden_state = distilbert_output[0]  # (bs, seq_len, dim)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/transformers/modeling_distilbert.py in forward(self, input_ids, attention_mask, head_mask, inputs_embeds)
    464
    465         if inputs_embeds is None:
--> 466             inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)
    467         tfmr_output = self.transformer(x=inputs_embeds, attn_mask=attention_mask, head_mask=head_mask)
    468         hidden_state = tfmr_output[0]

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/transformers/modeling_distilbert.py in forward(self, input_ids)
     89
     90         word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)
---> 91         position_embeddings = self.position_embeddings(position_ids)  # (bs, max_seq_length, dim)
     92
     93         embeddings = word_embeddings + position_embeddings  # (bs, max_seq_length, dim)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/torch/nn/modules/sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~/anaconda3/envs/deeplearning-labs-gpu/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows. at /tmp/pip-req-build-808afw3c/aten/src/TH/generic/THTensorEvenMoreMath.cpp:418
```

## Environment info

```
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_pytorch_select           0.2                       gpu_0
_tflow_select             2.1.0                       gpu
absl-py                   0.9.0                    py36_0
asn1crypto                1.3.0                    py36_0
astor                     0.8.0                    py36_0
attrs                     19.3.0                     py_0
backcall                  0.1.0                    py36_0
blas                      1.0                         mkl
bleach                    3.1.4                      py_0
boto3                     1.12.47                  pypi_0    pypi
botocore                  1.15.47                  pypi_0    pypi
c-ares                    1.15.0            h7b6447c_1001
ca-certificates           2020.1.1                      0
certifi                   2020.4.5.1               py36_0
cffi                      1.14.0           py36h2e261b9_0
chardet                   3.0.4                 py36_1003
click                     7.1.2                    pypi_0    pypi
cloudpickle               1.3.0                      py_0
cryptography              2.8              py36h1ba5d50_0
cudatoolkit               10.1.243             h6bb024c_0
cudnn                     7.6.5                cuda10.1_0
cupti                     10.1.168                      0
cycler                    0.10.0                   py36_0
cytoolz                   0.10.1           py36h7b6447c_0
dask-core                 2.15.0                     py_0
dataclasses               0.7                      pypi_0    pypi
dbus                      1.13.12              h746ee38_0
decorator                 4.4.2                      py_0
defusedxml                0.6.0                      py_0
docutils                  0.15.2                   pypi_0    pypi
eli5                      0.10.1                   pypi_0    pypi
entrypoints               0.3                      py36_0
expat                     2.2.6                he6710b0_0
filelock                  3.0.12                   pypi_0    pypi
fontconfig                2.13.0               h9420a91_0
freetype                  2.9.1                h8a8886c_1
gast                      0.3.3                      py_0
glib                      2.63.1               h5a9c865_0
gmp                       6.1.2                h6c8ec71_1
google-pasta              0.2.0                      py_0
grpcio                    1.27.2           py36hf8bcb03_0
gst-plugins-base          1.14.0               hbbd80ab_1
gstreamer                 1.14.0               hb453b48_1
h5py                      2.10.0           py36h7918eee_0
hdf5                      1.10.4               hb1b8bf9_0
icu                       58.2                 h9c2bf20_1
idna                      2.8                      py36_0
imageio                   2.8.0                      py_0
importlib_metadata        1.5.0                    py36_0
intel-openmp              2020.0                      166
ipykernel                 5.1.4            py36h39e3cac_0
ipython                   7.13.0           py36h5ca1d4c_0
ipython_genutils          0.2.0                    py36_0
ipywidgets                7.5.1                      py_0
jedi                      0.16.0                   py36_1
jinja2                    2.11.1                     py_0
jmespath                  0.9.5                    pypi_0    pypi
joblib                    0.14.1                     py_0
jpeg                      9b                   h024ee3a_2
json5                     0.9.4                    pypi_0    pypi
jsonschema                3.2.0                    py36_0
jupyter                   1.0.0                    py36_7
jupyter_client            6.1.2                      py_0
jupyter_console           6.1.0                      py_0
jupyter_core              4.6.3                    py36_0
jupyterlab                2.1.2                    pypi_0    pypi
jupyterlab-server         1.1.4                    pypi_0    pypi
keras-applications        1.0.8                      py_0
keras-base                2.3.1                    py36_0
keras-gpu                 2.3.1                         0
keras-preprocessing       1.1.0                      py_1
kiwisolver                1.1.0            py36he6710b0_0
ld_impl_linux-64          2.33.1               h53a641e_7
libedit                   3.1.20181209         hc058e9b_0
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 9.1.0                hdf63c60_0
libgfortran-ng            7.3.0                hdf63c60_0
libpng                    1.6.37               hbc83047_0
libprotobuf               3.11.4               hd408876_0
libsodium                 1.0.16               h1bed415_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.1.0                h2733197_0
libuuid                   1.0.3                h1bed415_2
libxcb                    1.13                 h1bed415_1
libxml2                   2.9.9                hea5a465_1
markdown                  3.1.1                    py36_0
markupsafe                1.1.1            py36h7b6447c_0
matplotlib                2.2.2            py36hb69df0a_2
mistune                   0.8.4            py36h7b6447c_0
mkl                       2020.0                      166
mkl-service               2.3.0            py36he904b0f_0
mkl_fft                   1.0.15           py36ha843d7b_0
mkl_random                1.1.0            py36hd6b4f25_0
nb_conda                  2.2.1                    py36_0
nb_conda_kernels          2.2.3                    py36_0
nbconvert                 5.6.1                    py36_0
nbformat                  5.0.4                      py_0
ncurses                   6.2                  he6710b0_0
networkx                  2.4                        py_0
ninja                     1.9.0            py36hfd86e86_0
notebook                  6.0.3                    py36_0
numpy                     1.18.1           py36h4f9e942_0
numpy-base                1.18.1           py36hde5b4d6_1
olefile                   0.46                     py36_0
openssl                   1.1.1g               h7b6447c_0
packaging                 20.3                       py_0
pandas                    0.23.0           py36h637b7d7_0
pandoc                    2.2.3.2                       0
pandocfilters             1.4.2                    py36_1
parso                     0.6.2                      py_0
pcre                      8.43                 he6710b0_0
pexpect                   4.8.0                    py36_0
pickleshare               0.7.5                    py36_0
pillow                    7.0.0            py36hb39fc2d_0
pip                       19.3.1                   py36_0
prometheus_client         0.7.1                      py_0
prompt-toolkit            3.0.4                      py_0
prompt_toolkit            3.0.4                         0
protobuf                  3.11.4           py36he6710b0_0
ptyprocess                0.6.0                    py36_0
pycparser                 2.20                       py_0
pygments                  2.6.1                      py_0
pyopenssl                 19.1.0                   py36_0
pyparsing                 2.4.6                      py_0
pyqt                      5.9.2            py36h05f1152_2
pyrsistent                0.16.0           py36h7b6447c_0
pysocks                   1.7.1                    py36_0
python                    3.6.10               hcf32534_1
python-dateutil           2.8.1                      py_0
python-graphviz           0.14                     pypi_0    pypi
pytorch                   1.4.0           cuda101py36h02f0884_0
pytz                      2019.3                     py_0
pywavelets                1.1.1            py36h7b6447c_0
pyyaml                    5.3.1            py36h7b6447c_0
pyzmq                     18.1.1           py36he6710b0_0
qt                        5.9.7                h5867ecd_1
qtconsole                 4.7.3                      py_0
qtpy                      1.9.0                      py_0
readline                  8.0                  h7b6447c_0
regex                     2020.4.4                 pypi_0    pypi
requests                  2.22.0                   py36_1
s3transfer                0.3.3                    pypi_0    pypi
sacremoses                0.0.41                   pypi_0    pypi
scikit-image              0.14.2           py36he6710b0_0
scikit-learn              0.22.1           py36hd81dba3_0
scikit-optimize           0.5.2                    pypi_0    pypi
scipy                     1.4.1            py36h0b6359f_0
send2trash                1.5.0                    py36_0
sentencepiece             0.1.86                   pypi_0    pypi
setuptools                46.1.3                   py36_0
sip                       4.19.8           py36hf484d3e_0
six                       1.14.0                   py36_0
sqlite                    3.31.1               h62c20be_1
tabulate                  0.8.7                    pypi_0    pypi
tensorboard               1.14.0           py36hf484d3e_0
tensorflow                1.14.0          gpu_py36h3fb9ad6_0
tensorflow-base           1.14.0          gpu_py36he45bfe2_0
tensorflow-estimator      1.14.0                     py_0
tensorflow-gpu            1.14.0               h0d30ee6_0
termcolor                 1.1.0                    py36_1
terminado                 0.8.3                    py36_0
testpath                  0.4.4                      py_0
tk                        8.6.8                hbc83047_0
tokenizers                0.7.0                    pypi_0    pypi
toolz                     0.10.0                     py_0
torchvision               0.5.0                py36_cu101    pytorch
tornado                   6.0.4            py36h7b6447c_1
tqdm                      4.45.0                   pypi_0    pypi
traitlets                 4.3.3                    py36_0
transformers              2.9.1                    pypi_0    pypi
urllib3                   1.25.8                   py36_0
wcwidth                   0.1.9                      py_0
webencodings              0.5.1                    py36_1
werkzeug                  1.0.1                      py_0
wheel                     0.34.2                   py36_0
widgetsnbextension        3.5.1                    py36_0
wrapt                     1.12.1           py36h7b6447c_1
xz                        5.2.5                h7b6447c_0
yaml                      0.1.7                had09818_2
zeromq                    4.3.1                he6710b0_3
zipp                      2.2.0                      py_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.3.7                h0b5b093_0
```

<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- Platform: Linux matrix 4.4.0-174-generic #204-Ubuntu SMP Wed Jan 29 06:41:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
- Python version: Python 3.6.10 :: Anaconda, Inc.
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Longformer for question answering

This PR adds `LongformerForQuestionAnswering` following `RobertaForQuestionAnswering`.

The code is almost identical to `RobertaForQuestionAnswering`, just had to remove `head_mask` parameter from forward method of `RobertaForQuestionAnswering`.

Also trained the model to verify this. You can check inference [here](https://colab.research.google.com/drive/1WGgYuBEzGvkvhGOrQxB94Jr1nwCPyblu?usp=sharing)

@patrickvonplaten


[T5] Fix Cross Attention position bias

This PR fixes the Cross Attention position bias assignment in Class T5Stack.


Pre-trained electra-large model doesn't converge when fine-tuned on SST-2

Need more information

Further information is requested

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ELECTRA

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SST-2

## To reproduce

Steps to reproduce the behavior:

1. Load the large pre-trained ELECTRA model using ElectraModel.from_pretrained('google/electra-large-discriminator', output_hidden_states=True)
2. fine-tune it on SST-2 using a simple binary classification head (linear, ReLU, linear, Sigmoid) on top of the [CLS] hidden state with BCEWithLogitsLoss and AdamW for 3-4 epochs.
3. Model is stuck on ~55% accuracy during all training and the loss increases

Important: When I do the same with 'google/electra-base-discriminator', and 'google/electra-small-discriminator' I'm getting an accuracy of ~93% and 89% (respectively) on the first epoch.

Hyperparameters:
batch_size: 16
lr: 0.000005
adam_epsilon: 0.0000001
max_len: 32

## Expected behavior

I expected that the fine-tuned electra-large model will outperform the base and small electra model and have better results than ~50% accuracy

## Environment info

- `transformers` version: 2.9.0
- Platform: Linux-5.3.0-1017-aws-x86_64-with-debian-buster-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


🐛 Bug
Information
Model I am using (Bert, XLNet ...): XLNet

Language I am using the model on (English, Chinese ...): Chinese
The problem arises when using: Tokenizer

 the official example scripts: (give details below) N/A
 my own modified scripts: (give details below) N/A
The tasks I am working on is: Any

 an official GLUE/SQUaD task: (give the name) N/A
 my own task or dataset: (give details below) N/A
To reproduce
Steps to reproduce the behavior:

Load any BERT tokenizer
Tokenize something with a "." in between
Decode these ids, you will find it mismatch
x = tokenizer.encode('AN.C', add_special_tokens=False)
z = tokenizer.decode(x)
It prints:

AN. C
Expected behavior
AN.C
Environment info
transformers version:
Platform: CentOS
Python version: 3.6
PyTorch version (GPU?): GPU
Tensorflow version (GPU?): GPU
Using GPU in script?: NO
Using distributed or parallel set-up in script?: NO


python run_glue.py with the AttributeError: 'NoneType' object has no attribute 'seek'

Further information is requested

# 🐛 Bug

Traceback (most recent call last):
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/torch/serialization.py", line 191, in _check_seekable
    f.seek(f.tell())
AttributeError: 'NoneType' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/transformers/modeling_utils.py", line 659, in from_pretrained
    state_dict = torch.load(resolved_archive_file, map_location="cpu")
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/torch/serialization.py", line 549, in _load
    _check_seekable(f)
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/torch/serialization.py", line 194, in _check_seekable
    raise_err_msg(["seek", "tell"], e)
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/torch/serialization.py", line 187, in raise_err_msg
    raise type(e)(msg)
AttributeError: 'NoneType' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./examples/text-classification/run_glue.py", line 202, in <module>
    main()
  File "./examples/text-classification/run_glue.py", line 133, in main
    cache_dir=model_args.cache_dir,
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/transformers/modeling_auto.py", line 874, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/gpfs/home/bsub/anaconda3/envs/abc/lib/python3.6/site-packages/transformers/modeling_utils.py", line 662, in from_pretrained
    "Unable to load weights from pytorch checkpoint file. "
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.


❓ [BART] Why Decoder Layer Normalization is applied only at the last layer ?

# ❓ Questions & Help

It seems this line :
https://github.com/huggingface/transformers/blob/efbc1c5a9d96048ab11f8d746fe51107cb91646f/src/transformers/modeling_bart.py#L524

was modified when MBART was added.

---

Before, Layer Normalization was applied after **all** layers of the decoder (_similar to the encoder, if the config was set appropriately_).

But now, Layer Normalization is applied **only at the end**, even for other BART models (_not MBART_).

---

**Is it expected ? What's the reason behind this logic ?**

@sshleifer


Incorporate HuggingFace 'nlp' library in examples

External

Using the library with external tools (onnx, tflite, ...)

# 🚀 Feature request

I propose we replace the custom data downloading/preprocessing logic found within the examples directory with the new [**HuggingFace `nlp` Library**](https://github.com/huggingface/nlp) where applicable.

## Motivation

The examples directory is filled with custom shell scripts that download and process common research datasets. These scripts work great, but are at times tricky to follow. I'm sure this can be discouraging for new users looking to try out `transformers` for the first time.

I'm hoping `nlp` will make the examples generally more accessible for both new and experienced users. And yeah...I guess it's probably not too bad for the brand either. 😉

## Your contribution

I'll get a WIP PR pushed up this weekend. I'll focus on the `pytorch_lightning` examples for now.


Use args.per_gpu_train_batch_size instead of args.train_batch_size in…

… Trainer.

It appears that this is preferred, per https://github.com/huggingface/transformers/blob/master/src/transformers/training_args.py. This also matches the calculation which is printed referring to batch size at https://github.com/huggingface/transformers/blob/master/src/transformers/trainer.py#L432.

As a side note, the GPT-2 example in https://github.com/huggingface/transformers/blob/master/examples/language-modeling/README.md no longer works. There is a default `per_gpu_train_batch_size=8`, which throws OOM on a Tesla V100 with 32GB RAM. I ran it successfully with `--per_gpu_train_batch_size=1`, and it used 7GB of RAM. So we probably want to add that hyperparameter to the example command.


Cannot load reformer-enwik8 tokenizer

Core: Tokenization

Internals of the library; Tokenization.

reformer

Everything related to the reformer model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Reformer tokenizer

## To reproduce

Steps to reproduce the behavior:

1. Try to load the pretrained reformer-enwik8 tokenizer with `AutoTokenizer.from_pretrained("google/reformer-enwik8")`

This is the error I got:
```
OSError                                   Traceback (most recent call last)
<ipython-input-51-ab9a64363cc0> in <module>
----> 1 AutoTokenizer.from_pretrained("google/reformer-enwik8")

~/.virtualenvs/sparseref/lib/python3.7/site-packages/transformers-2.9.0-py3.7.egg/transformers/tokenization_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *inputs, **kwargs)
    198                     return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    199                 else:
--> 200                     return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
    201
    202         raise ValueError(

~/.virtualenvs/sparseref/lib/python3.7/site-packages/transformers-2.9.0-py3.7.egg/transformers/tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    896
    897         """
--> 898         return cls._from_pretrained(*inputs, **kwargs)
    899
    900     @classmethod

~/.virtualenvs/sparseref/lib/python3.7/site-packages/transformers-2.9.0-py3.7.egg/transformers/tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1001                     ", ".join(s3_models),
   1002                     pretrained_model_name_or_path,
-> 1003                     list(cls.vocab_files_names.values()),
   1004                 )
   1005             )

OSError: Model name 'google/reformer-enwik8' was not found in tokenizers model name list (google/reformer-crime-and-punishment). We assumed 'google/reformer-enwik8' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.
```

I tried with and without `google/`, same result. However, it did print the download progress bar. Trying to load the `crime-and-punishment` Reformer tokenizer works.

- `transformers` version: 2.9.0
- Platform: macOS
- Python version: 3.7
- PyTorch version (GPU?): 1.4.0, no GPU
- Using distributed or parallel set-up in script?: no


Windows: Can't find vocabulary file for MarianTokenizer

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug MarianTokenizer.from_pretrained() fails in Python 3.6.4 in Windows 10

## Information
Occurs with using the example here: [https://huggingface.co/transformers/model_doc/marian.html?highlight=marianmtmodel#transformers.MarianMTModel](huggingface.co/transformers/model_doc/marian.html?highlight=marianmtmodel#transformers.MarianMTModel)

Model I am using (Bert, XLNet ...): MarianMTModel

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X ] my own task or dataset: (give details below)

## To reproduce
Paste code from example and run:
```Python
from transformers import MarianTokenizer, MarianMTModel
from typing import List
src = 'fr'  # source language
trg = 'en'  # target language
sample_text = "où est l'arrêt de bus ?"
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'

model = MarianMTModel.from_pretrained(mname)
tok = MarianTokenizer.from_pretrained(mname)
batch = tok.prepare_translation_batch(src_texts=[sample_text])  # don't need tgt_text for inference
gen = model.generate(**batch)  # for forward pass: model(**batch)
words: List[str] = tok.batch_decode(gen, skip_special_tokens=True)  # returns "Where is the the bus stop ?"
print(words)
```
Steps to reproduce the behavior:

1. Run the example
2. Program terminates on `tok = MarianTokenizer.from_pretrained(mname)`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```Python
stdbuf was not found; communication with perl may hang due to stdio buffering.
Traceback (most recent call last):
  File "C:\Program Files\Python\lib\site-packages\transformers\tokenization_utils.py", line 1055, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "C:\Program Files\Python\lib\site-packages\transformers\tokenization_marian.py", line 89, in __init__
    self._setup_normalizer()
  File "C:\Program Files\Python\lib\site-packages\transformers\tokenization_marian.py", line 95, in _setup_normalizer
    self.punc_normalizer = MosesPunctuationNormalizer(self.source_lang)
  File "C:\Program Files\Python\lib\site-packages\mosestokenizer\punctnormalizer.py", line 47, in __init__
    super().__init__(argv)
  File "C:\Program Files\Python\lib\site-packages\toolwrapper.py", line 64, in __init__
    self.start()
  File "C:\Program Files\Python\lib\site-packages\toolwrapper.py", line 108, in start
    env=env,
  File "C:\Program Files\Python\lib\subprocess.py", line 709, in __init__
    restore_signals, start_new_session)
  File "C:\Program Files\Python\lib\subprocess.py", line 997, in _execute_child
    startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Development/Research/COVID-19-Misinfo2/src/translate_test_2.py", line 9, in <module>
    tok = MarianTokenizer.from_pretrained(mname)
  File "C:\Program Files\Python\lib\site-packages\transformers\tokenization_utils.py", line 902, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "C:\Program Files\Python\lib\site-packages\transformers\tokenization_utils.py", line 1058, in _from_pretrained
    "Unable to load vocabulary from file. "
OSError: Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.
```

## Expected behavior
prints ["Where is the the bus stop ?"]
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.1
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.4
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): 2.1.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


How to load a pruned Albert model with from_pretrained()?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I pruned Albert during the fine-tuning phase. I was unable to load the pruned model after saving it. I tried using:
```
output_model_file = os.path.join(args.output_dir, "pytorch_model.bin")
model_state_dict = torch.load(output_model_file)
model = model_class.from_pretrained(args.output_dir,state_dict=model_state_dict)
```
but still got the same error:
```
 File "run_glue.py", line 526, in main
    model = model_class.from_pretrained(args.output_dir,state_dict=model_state_dict)
  File "/home/user/.local/lib/python3.7/site-packages/transformers/modeling_utils.py", line 471, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for AlbertForSequenceClassification:
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.query.weight: copying a param with shape torch.Size([64, 768]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.query.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.key.weight: copying a param with shape torch.Size([64, 768]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.key.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.value.weight: copying a param with shape torch.Size([64, 768]) from checkpoint, the shape in current model is torch.Size([768, 768]).
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.value.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for albert.encoder.albert_layer_groups.0.albert_layers.0.attention.dense.weight: copying a param with shape torch.Size([768, 64]) from checkpoint, the shape in current model is torch.Size([768, 768]).
```
transformer version == 2.2.1
torch-1.4.0
Anybody can help?


bugfix: pass on tokenizer to pipeline in load_graph_from_args

I think I found a small bug in the `load_graph_from_args` function in `convert_graph_to_onnx.py` as it accepts a tokenizer as input but doesn't pass it on to the pipeline inside the function.

Love the library 🤗


Make changes to german-bert vocab file more prominent

model card

Related to pretrained model cards

We have been approached by researchers because the expected behavior of their bert-base-german-cased models changed without code modifications.

- So we wanted to make the changes to the vocab more prominent in the model card
- and also support a solution where people can easily use the old version through https://huggingface.co/deepset/bert-base-german-cased-oldvocab


Fix slow gpu tests lysandre

Fixes three tests of the slow + gpu suite cc @sshleifer


tokenizer.vocab has not changed after using add_tokens

Core: Tokenization

Internals of the library; Tokenization.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->

When I use the add_tokens, I have the following problems
```python3
# len(tokenizer)  == 30522
 tokens_dict = ['[HL]']
 num_added_toks = tokenizer.add_tokens(tokens_dict)
# len(tokenizer)  == 30523
# But tokenizer.vocab_size == 30522
```
Should I change the dictionary by myself ?


Can't find vocabulary file or is corrupted for MarianTokenizer

# 🐛 Bug

## Information

Model I am using MarianMT:

The problem arises when using the tokenizer with from_pretrained

## To reproduce

from transformers import MarianTokenizer, MarianMTModel
src = 'fr'  # source language
trg = 'en'  # target language
sample_text = "où est l'arrêt de bus ?"
mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'
model = MarianMTModel.from_pretrained(mname)
tok = MarianTokenizer.from_pretrained(mname)

stdbuf was not found; communication with perl may hang due to stdio buffering.
FileNotFoundError                         Traceback (most recent call last)
~\Anaconda3\envs\pytorch\lib\site-packages\transformers\tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1054         try:
-> 1055             tokenizer = cls(*init_inputs, **init_kwargs)
   1056         except OSError:

~\Anaconda3\envs\pytorch\lib\site-packages\transformers\tokenization_marian.py in __init__(self, vocab, source_spm, target_spm, source_lang, target_lang, unk_token, eos_token, pad_token, max_len)
     88
---> 89             self.punc_normalizer = MosesPunctuationNormalizer(source_lang)
     90         except ImportError:

~\Anaconda3\envs\pytorch\lib\site-packages\mosestokenizer\punctnormalizer.py in __init__(self, lang)
     46         argv = ["perl", program, "-b", "-l", self.lang]
---> 47         super().__init__(argv)
     48

~\Anaconda3\envs\pytorch\lib\site-packages\toolwrapper.py in __init__(self, argv, encoding, start, cwd, stdbuf, stderr, env)
     63         if start:
---> 64             self.start()
     65

~\Anaconda3\envs\pytorch\lib\site-packages\toolwrapper.py in start(self)
    107             cwd=self.cwd,
--> 108             env=env,
    109         )

~\Anaconda3\envs\pytorch\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
    799                                 errread, errwrite,
--> 800                                 restore_signals, start_new_session)
    801         except:

~\Anaconda3\envs\pytorch\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
   1206                                          os.fspath(cwd) if cwd is not None else None,
-> 1207                                          startupinfo)
   1208             finally:

FileNotFoundError: [WinError 2] The system cannot find the file specified

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-10-56dab20251f1> in <module>
      6 mname = f'Helsinki-NLP/opus-mt-{src}-{trg}'
      7 model = MarianMTModel.from_pretrained(mname)
----> 8 tok = MarianTokenizer.from_pretrained(mname)

~\Anaconda3\envs\pytorch\lib\site-packages\transformers\tokenization_utils.py in from_pretrained(cls, *inputs, **kwargs)
    900
    901         """
--> 902         return cls._from_pretrained(*inputs, **kwargs)
    903
    904     @classmethod

~\Anaconda3\envs\pytorch\lib\site-packages\transformers\tokenization_utils.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1056         except OSError:
   1057             raise OSError(
-> 1058                 "Unable to load vocabulary from file. "
   1059                 "Please check that the provided vocabulary is accessible and not corrupted."
   1060             )

OSError: Unable to load vocabulary from file. Please check that the provided vocabulary is accessible and not corrupted.

----------------------------------------
I'm working on a windows 10 environment


Bug using Roberta models in QA Transformers pipeline.

Core: Modeling

Internals of the library; Models.

Core: Pipeline

Internals of the library; Pipeline.

# 🐛 Bug

Hello, I cant use any roberta model with pipeline('question-answering'), someone can help me in how to fix this issue?

OBS=This error appears just when I use Roberta models.

ERROR:

![image](https://user-images.githubusercontent.com/64150563/82443055-38072700-9a77-11ea-99ac-58316d3e0636.png)


Trying to add support for GPT2 as decoder in EncoderDecoder model

# 🚀 Feature request

Hi,
I am trying to add the option of using GPT2 as the decoder in the EncoderDecoder model, which only support

## Motivation

For a generation problem, it usually better to use GPT2 as the decoder, over BERT.

## Your contribution

I've made the following changes in `modeling_gpt2.py` file:

- Added crossattention layer if the model is a decoder, to the `Block` class:
```python
class Block(nn.Module):
    def __init__(self, n_ctx, config, scale=False):
        super().__init__()
        nx = config.n_embd
        self.ln_1 = nn.LayerNorm(nx, eps=config.layer_norm_epsilon)
        self.attn = Attention(nx, n_ctx, config, scale)
        self.ln_2 = nn.LayerNorm(nx, eps=config.layer_norm_epsilon)
        self.mlp = MLP(4 * nx, config)
        self.is_decoder = config.is_decoder
        if self.is_decoder:
            self.crossattention = Attention(nx, n_ctx, config, scale)
...
    def forward(self, x, layer_past=None, attention_mask=None, head_mask=None, use_cache=False, encoder_hidden_states=None,
                encoder_attention_mask=None):
        output_attn = self.attn(
            self.ln_1(x),
            layer_past=layer_past,
            attention_mask=attention_mask,
            head_mask=head_mask,
            use_cache=use_cache,
        )
        a = output_attn[0]  # output_attn: a, present, (attentions)
        outputs = []
        if self.is_decoder and encoder_hidden_states is not None:
            cross_attention_outputs = self.crossattention(
                a, layer_past, attention_mask, head_mask, encoder_hidden_states=encoder_hidden_states,
                                            encoder_attention_mask=encoder_attention_mask
            )
            a = cross_attention_outputs[0]
            outputs = outputs + cross_attention_outputs[1:]  # add cross attentions if we output attention weights

        x = x + a
        m = self.mlp(self.ln_2(x))
        x = x + m

        outputs = [x] + output_attn[1:] + outputs

        return outputs  # x, present, (attentions)
```

- Added 3 Linear layers instead of the Conv1d layer:
```python
class Attention(nn.Module):
    def __init__(self, nx, n_ctx, config, scale=False):
...
        # self.c_attn = Conv1D(n_state * 3, nx)
        self.query = nn.Linear(n_state, nx)
        self.key = nn.Linear(n_state, nx)
        self.value = nn.Linear(n_state, nx)
...
```

- Added `encoder_attention_mask` and `encoder_hidden_states` to the forward function of the `Attention` class, and using them for the key and the value if they are provided:
```python
def forward(self, x, layer_past=None, attention_mask=None, head_mask=None, use_cache=False, encoder_hidden_states=None,
                encoder_attention_mask=None):
        query = self.query(x)
        if encoder_hidden_states is not None:
            key = self.key(encoder_hidden_states)
            value = self.value(encoder_hidden_states)
            attention_mask = encoder_attention_mask
        else:
            key = self.key(x)
            value = self.value(x)
        query = self.split_heads(query)
        key = self.split_heads(key, k=True)
        value = self.split_heads(value)
...
```

- Added the `encoder_attention_mask` and `encoder_hidden_states` arguments to the `GPT2Model` forward function, and processed `encoder_attention_mask` same as attention_mask:
```python
class GPT2Model(GPT2PreTrainedModel):
...
    def forward(
        self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        use_cache=True,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
    ):
...
        # Encoder attention mask. (same action as for regular attention mask)
        if encoder_attention_mask is not None:
            assert batch_size > 0, "batch_size has to be defined and > 0"
            encoder_attention_mask = encoder_attention_mask.view(batch_size, -1)
            encoder_attention_mask = encoder_attention_mask.unsqueeze(1).unsqueeze(2)
            encoder_attention_mask = encoder_attention_mask.to(dtype=next(self.parameters()).dtype)  # fp16 compatibility
            encoder_attention_mask = (1.0 - encoder_attention_mask) * -10000.0
...
        for i, (block, layer_past) in enumerate(zip(self.h, past)):
            if self.output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states.view(*output_shape),)

            outputs = block(
                hidden_states,
                layer_past=layer_past,
                attention_mask=attention_mask,
                head_mask=head_mask[i],
                use_cache=use_cache,
                encoder_hidden_states=encoder_hidden_states,
                encoder_attention_mask=encoder_attention_mask,
            )
...
```

- Added the `encoder_attention_mask` and `encoder_hidden_states` arguments to the `GPT2LMHeadModel`forward function, as well as `lm_lables` and `masked_lm_labels` for EncoderDecoder model compatibility (probably it's better to use `GPT2DoubleHeadsModel`):
```python
class GPT2LMHeadModel(GPT2PreTrainedModel):
...
    def forward(
        self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        use_cache=True,
        lm_labels=None,
        masked_lm_labels=None,
        encoder_hidden_states=None,
        encoder_attention_mask=None,
    ):
...
        if lm_labels is not None:
            if labels is not None:
                raise ValueError("You cannot specify both labels and lm_labels at the same time")
            labels = lm_labels

        transformer_outputs = self.transformer(
            input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
            encoder_hidden_states=encoder_hidden_states,
            encoder_attention_mask=encoder_attention_mask,
        )
...
```
My biggest concern is with the second bullet, and I wanted to ask you if this implementation seems right (for now it's look like I am able to train and test an EncoderDecoder with BERT2GPT architecture).
Of course that if needed, I can provide the full code to all of my changes, but all of my changes is listed above.
Most (if not all) of the code I've add is adapted from huggingface `modeling_bert.py`file, so all of the credit goes to them.

Thanks


Create model card for RuPERTA-base-finetuned-pos

model card

Related to pretrained model cards


Add mecab dependency on slow tests.

Solves the following error:

```
2020-05-19T17:01:17.3352437Z [gw0] linux -- Python 3.7.6 /home/hf/actions-r
2020-05-19T17:01:17.3354221Z     @slow
2020-05-19T17:01:17.3354825Z     def test_sequence_builders(self):
2020-05-19T17:01:17.3356512Z >       tokenizer = self.tokenizer_class.from_pretrained("bert-base-japanese-char")
2020-05-19T17:01:17.3356685Z
2020-05-19T17:01:17.3357374Z tests/test_tokenization_bert_japanese.py:192:
2020-05-19T17:01:17.3359012Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-05-19T17:01:17.3360868Z .env/lib/python3.7/site-packages/transformers/tokenization_utils.py:902: in from_pretrained
2020-05-19T17:01:17.3361266Z     return cls._from_pretrained(*inputs, **kwargs)
2020-05-19T17:01:17.3363161Z .env/lib/python3.7/site-packages/transformers/tokenization_utils.py:1055: in _from_pretrained
2020-05-19T17:01:17.3363615Z     tokenizer = cls(*init_inputs, **init_kwargs)
2020-05-19T17:01:17.3365382Z .env/lib/python3.7/site-packages/transformers/tokenization_bert_japanese.py:139: in __init__
2020-05-19T17:01:17.3366229Z     do_lower_case=do_lower_case, never_split=never_split, **(mecab_kwargs or {})
2020-05-19T17:01:17.3367669Z _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2020-05-19T17:01:17.3367895Z
2020-05-19T17:01:17.3369474Z self = <transformers.tokenization_bert_japanese.MecabTokenizer object at 0x7f433565b9d0>
2020-05-19T17:01:17.3371043Z do_lower_case = False, never_split = None, normalize_text = True
2020-05-19T17:01:17.3371414Z mecab_option = None
2020-05-19T17:01:17.3371564Z
2020-05-19T17:01:17.3373681Z     def __init__(self, do_lower_case=False, never_split=None, normalize_text=True, mecab_option: Optional[str] = None):
2020-05-19T17:01:17.3373909Z         """Constructs a MecabTokenizer.
2020-05-19T17:01:17.3374082Z
2020-05-19T17:01:17.3374357Z         Args:
2020-05-19T17:01:17.3375149Z             **do_lower_case**: (`optional`) boolean (default True)
2020-05-19T17:01:17.3375850Z                 Whether to lower case the input.
2020-05-19T17:01:17.3376666Z             **never_split**: (`optional`) list of str
2020-05-19T17:01:17.3377692Z                 Kept for backward compatibility purposes.
2020-05-19T17:01:17.3378953Z                 Now implemented directly at the base class level (see :func:`PreTrainedTokenizer.tokenize`)
2020-05-19T17:01:17.3379578Z                 List of token not to split.
2020-05-19T17:01:17.3380559Z             **normalize_text**: (`optional`) boolean (default True)
2020-05-19T17:01:17.3381677Z                 Whether to apply unicode normalization to text before tokenization.
2020-05-19T17:01:17.3382985Z             **mecab_option**: (`optional`) string passed to `MeCab.Tagger` constructor (default "")
2020-05-19T17:01:17.3383398Z         """
2020-05-19T17:01:17.3384034Z         self.do_lower_case = do_lower_case
2020-05-19T17:01:17.3385088Z         self.never_split = never_split if never_split is not None else []
2020-05-19T17:01:17.3385841Z         self.normalize_text = normalize_text
2020-05-19T17:01:17.3386284Z
2020-05-19T17:01:17.3386881Z >       import MeCab
2020-05-19T17:01:17.3388516Z E       ModuleNotFoundError: No module named 'MeCab'
```


[Reformer] Include char lm to Trainer

Trainer currently expects every model to have a tokenizer. The reformer: `google/reformer-enwik8` is a char lm which does not require a tokenizer.


[examples] fix no grad in second pruning in run_bertology

the `new_head_mask` index assignment operation makes it become a non-leaf node in the following gradient computation, resulting in grad is None bug as mentioned in  #3895


❓ [TPU] [Trainer] Moving model to device before setting optimizer slow the training

# ❓ Questions & Help

On master, after applying the fix of #4450, the training on 8 TPU cores is much slower.

* Before the fix : **20 min / epoch (8 iterations / s)**
* After the fix : **1h30 / epoch (2~3 iterations / s)**

Of course the training before the fix was not working (loss was not decreasing).

But this slow is not expected : for the TF2 equivalent with the same dataset, it takes **20 min / epoch**.

---

Anyone meeting the same problem ?


Remove warning of deprecation

Remove warning of deprecated overload of addcdiv_

Fix #4451


Tokenizer encode to have an option to overflow from left

Core: Tokenization

Internals of the library; Tokenization.

# 🚀 Feature request

Current tokenizer encode variants ( encode, batch_encode, batch_encode_plus) handle longer sequences than max_length by overflowing tokens from the right hand side and thus restricting the length to max_length. This feature request is to allow an option for the tokenizer encode methods to overflow tokens from the left hand side as well.

## Motivation

For problems dealing with dialog, if one were to train an intent classification or next sentence prediction model and the dialog was longer than max_length, one would like to throw away the tokens from the beginning of the conversation as they are less relevant than the more recent messages.

This motivates the need for a encoder that works well with dialog data where more recent tokens are more valuable.

## Your contribution

I could change the function `truncate_sequences` by adding a new truncation_strategy option that will truncate from left. But want to get feedback from the Huggingface team about this proposal.


Request for hosting model files in a Virtual Hosted-Style S3 buckets

Is there any plans for the s3 buckets currently hosting the model files to migrate from the current "Path-Style Request" format to a "Virtual Hosted-Style Request" format?

Path-Style URLs follow the following format (s3.amazonaws.com/* OR s3.Region.amazonaws.com/*). For example, today the model config file for 'bert_uncased_L-2_H-128_A-2' is accessed via the Path-Style URL:

https://s3.amazonaws.com/models.huggingface.co/bert/google/bert_uncased_L-2_H-128_A-2/config.json

According to AWS, they will be deprecating Path-Style requests (though there will be legacy support) - but one major reason for the migration to the Virtual Hosted-Style URL (which takes the form bucket-name.s3.amazonaws.com or bucket-name.s3.Region.amazonaws.com) is for security reasons (e.g. if companies/organizations need to whitelist sites in their servers to utilize transformer models, the virtual hosted style will reduce the "blast radius" in cases of security breaches).

More details on "Path" vs "Virtual-Hosted" style requests:

https://docs.aws.amazon.com/AmazonS3/latest/dev/VirtualHosting.html#path-style-access


Remove warning of deprecation

Remove warning of deprecated overload of `addcdiv_`

Fix #4451


Add Fine-tune DialoGPT on new datasets notebook

Here is a tutorial notebook I created for fine-tuning the DialoGPT on a Spanish conversation dataset. It shows how to prepare a dataset that conforms to the necessary style of the original DialoGPT dataset and how to train it using a GPU provided by Google Colab. Sadly it is not using the newer Trainer that Huggingface provides, but I thought it might be useful for others trying to work with conversational AI so wanted to share.

Thanks for the amazing library and hugs to all of y'all 🤗!


[gpu slow tests] fix mbart-large-enro gpu tests


batch_encode_plus returns same lengths when enable pad_to_max_length

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the following script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
sents = [
    "I can eat glass without harm",
    "I cannot eat glass"
]
resp = tokenizer.batch_encode_plus(sents, pad_to_max_length=True, return_lengths=True)
print(resp['length'])
# >>> get [8, 8], should be [8, 6]
```
## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The function batch_encode_plus should return correct lengths of sentences before padded to max length. Which should be [8, 6] in the above example. Otherwise, we can just get the length from the last dimension of the mask.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Ubuntu
- Python version: 3.7
- PyTorch version (GPU?): CPU 1.5
- Tensorflow version (GPU?): No
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Model card for Tereveni-AI/gpt2-124M-uk-fiction

model card

Related to pretrained model cards

Create model card for "Tereveni-AI/gpt2-124M-uk-fiction" model


Better None gradients handling in TF Trainer

Update the TF Trainer to better handle None gradients in order to have something generic and not anymore task dependent.


[Tests, GPU, SLOW] fix a bunch of GPU hardcoded tests in Pytorch

in almost all tests I forgot to put the model on gpu via
`model = model.to(torch_device)`


TPU hangs when saving optimizer/scheduler

Similarly to when saving a model state dict, the optimizer and scheduler should be saved using `xm.save` and behind an `xm.rendezvous`.

Additional fix: `pl.ParallelLoader` is not a `torch.utils.data.DataLoader`, and, therefore, must be reinitialized at each epoch.


Model card for RuPERTa-base fine-tuned for NER

Related to pretrained model cards


[ci] Slow GPU tests run daily

Could be useful to fix a few of the warnings and deprecation warnings too


[Longformer] Docs and clean API

This PR:

- adds a documentation page for Longformer. @ibeltagy - it's best to read it using this link I think: https://github.com/huggingface/transformers/pull/4464/files?short_path=3909947#diff-3909947f36862a1731195bf05c85c64c.

- fixes a typo to correctly render the pretrained models doc page

- changes the API of Longformer slightly. I removed the `attention_mode` from Longformer because I don't think it should be used. The modus should always be `Longformer` since it is a `Longformer` model. The user should not be able to create a `RobertaModel` using `LongformerModel`.
For comparisons people should use `RobertaModel` vs `LongformerModel` and not different modi of Longformer which is essentially the same as `RobertaModel` (correct me if I'm wrong here @ibeltagy).


Adds predict stage for glue tasks, and generate result files which can be submitted to gluebenchmark.com

By simply fine-tune robterta-large with 3k steps on several tasks, achieved:

Task | Metrics | Score
-- | -- | --
Microsoft Research Paraphrase Corpus | F1 / Accuracy | 91.5/88.6
Semantic Textual Similarity Benchmark | Pearson-Spearman Corr | 90.7/90.2
Quora Question Pairs | F1 / Accuracy | 69.5/87.3
Recognizing Textual Entailment | Accuracy | 82.0
Winograd NLI | Accuracy | 65.1


add T5 fine-tuning notebook [Community notebooks]

@patrickvonplaten
This is the second notebook which shows how to fine-tune T5 for multiple tasks with text-to-text approach (IMDB, emotion classification, SWAG) that we discussed in issue #4426 . I didn't find emotion and SWAG dataset in the `nlp` library so I decided to keep my original dataset code to keep everything unified.

Also there's growing interest in `pytoch-lightning` so I decided to keep the `lightning`  trainer. But if you think I should use HF Trainer then I can add that as well.


ProjectedAdaptiveLogSoftmax.log_prob raises Exception

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Transformer-XL

Language I am using the model on (English, Chinese ...): WikiText-103 (English)

The problem arises when using:
* [x] the official example scripts: `run_transfo_xl.py` (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official task: WikiText-103
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fix `run_transfo_xl.py` by removing the unused `work_dir` argument (line 52) and changing `lm_labels=target` to `labels=target` (line 108).
2. Add `logits = self.crit.log_prob(pred_hid.flatten(0, 1))` right before the model output, e.g. in line 919 of `modeling_transfo_xl.py`.
3. Run `run_transfo_xl.py`.
4. Look at error message:
```
The size of tensor a (1280) must match the size of tensor b (20000) at non-singleton dimension 1
  File ".../transformers/src/transformers/modeling_transfo_xl_utilities.py", line 246, in log_prob
    logprob_i = head_logprob[:, -i] + tail_logprob_i
  File ".../transformers/src/transformers/modeling_transfo_xl.py", line 920, in forward
    logits = self.crit.log_prob(pred_hid.flatten(0, 1))
  File ".../run_transfo_xl.py", line 107, in evaluate
    ret = model(data, labels=target, mems=mems)
  File ".../run_transfo_xl.py", line 124, in main
    test_loss = evaluate(te_iter)
  File ".../run_transfo_xl.py", line 143, in <module>
    main()
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
`log_prob` should return the log probabilities instead of raising an Exception.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: Master (commit 384f0eb)
- Platform: Ubuntu
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (CUDA 10.1, CuDNN 7.6.3)
- Tensorflow version (GPU?): -
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Attempt to do some optimizations for BERT models

- Use functional as much as possible instead of creating a class instance everytime
- Precompute and store the attention scaling factor to avoid `1/sqrt(...)` every forward
- Refactor Self-Attention to group QKV weights and and increase hardware density


Pretrained Transformer-XL gives unreasonable result on WikiText-103

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Transformer-XL (`transfo-xl-wt103`)

Language I am using the model on (English, Chinese ...): WikiText-103 (English)

The problem arises when using:
* [x] the official example scripts: `run_transfo_xl.py` (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official task: WikiText-103 (not GLUE/SQUaD)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Fix `run_transfo_xl.py` by removing the unused `work_dir` argument (line 52) and changing `lm_labels=target` to `labels=target` (line 108).
2. Run `run_transfo_xl.py`.
3. Observe the result: `test loss 10.20 | test ppl 26951.114`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

A reasonable result around the order of PPL=18.3, as reported in the paper. I know that the result will not be exactly the same, but something is definitely wrong here.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: Both 2.9.1 and Master (commit 384f0eb)
- Platform: Ubuntu
- Python version: 3.7.6
- PyTorch version (GPU?): 1.4.0 (CUDA 10.1, CuDNN 7.6.3)
- Tensorflow version (GPU?): -
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


layer name change to match compatibility with pytorch layer name in BertForQuestionAnswering

As pointed out in #438 When using BertForQuestionAnswering to load a tensorflow model using from_pretrained, one runs into an error
`AttributeError: 'BertForQuestionAnswering' object has no attribute 'classifier'`

As pointed out in the thread it should be "qa_outputs" and not "classifier" for this functionality to work as expected.


FastTokenizer add_special_tokens also adding individual characters for multi character tokens

# 🐛 Bug

## Information
Model I am using (Bert, XLNet ...): BERT FastTokenizer

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] my own task or dataset: (give details below)

As far as I can tell, adding tokens (e.g. '[EOS]') to the FastTokenizer  will result in all the single characters of the token to be added to the tokenizer ('E', 'O', 'S'). This seems to only occur when using tokenizer.add_special_tokens() as described [here](https://huggingface.co/transformers/main_classes/tokenizer.html#transformers.PreTrainedTokenizer.add_special_tokens). However, adding the tokens via the constructor seem to work just fine.
This bug results in problems using the uncased models as we don't want uppercase letters. Also the unwanted additions to the vocab don't seem to show up in len(tokenizer), which results in index of out range errors when feeding into BERT.

This would be a breaking case:
```
    tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased', use_fast=True)
    print('Tokenizer len before: {}'.format(len(tokenizer)))
    num_added = tokenizer.add_special_tokens({'eos_token': '[EOS]', 'bos_token': '[BOS]'})
    print('Tokenizer len after: {}'.format(len(tokenizer)))
    print('Number tokens added: ', num_added)
    print(tokenizer.bos_token)
    print(tokenizer.eos_token)
    # We can see that the tokens have been added successfully

    # However, encoding the same sequence as before, we run into problems:
    encoded = tokenizer.encode('This is a big S!')
    print(encoded)
    print(tokenizer.convert_ids_to_tokens(encoded))
    # If you look carefully, you can see that the 'S' in the sequence is not lowercase.
    # Also the id in the line above (30526) should not be higher than the tokenizer len (30524)
    # If we feed this into bert (after model.resize_token_embeddings(len(tokenizer))) this will crash
    with an index out of range exception.
```
Outputs:
```
Tokenizer len before: 30522
Tokenizer len after: 30524
Number tokens added:  2
[BOS]
[EOS]
[101, 2023, 2003, 1037, 2502, 30526, 999, 102]
['[CLS]', 'this', 'is', 'a', 'big', 'S', '!', '[SEP]']
```

Edit: My proposed workaround of adding the special tokens via the constructor also does not work. The tokens are accessible via tokenizer.<eos/bos>_token but adding tokens this way does not change the number of tokens in the vocab. I.e. using len(tokenizer) doesn't reflect the newly added tokens.

## To reproduce

Steps to reproduce the behavior:
Please find this [colab notebook](https://colab.research.google.com/drive/1hMEr0gpbyGJCZvIzFB22eKlmb9I-vKuu?usp=sharing) investigating the bug

## Expected behavior
add_special_tokens() of the fast tokenizer should behave the same as for the regular tokenizer: Only adding the full special tokens '[EOS]' and not also single characters of it. Furthermore, I would expect that if something is added, this would also be reflected in the length of the tokenizer.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.1
- Platform: Colab, linux
- Python version: 3.7
- PyTorch version (GPU?): 1.5, gpu
- Tensorflow version (GPU?):
- Using GPU in script?: tried both, no difference
- Using distributed or parallel set-up in script?: No


Problems About Using the Run_language_modeling with Tf2.

# 🐛 Bug

### Model I am using Bert.

### The problem arises when using:
I'm using the run_language_modling.py to fine-tuning with (Tf2.0.0/Tf2.2.0). Of course i modify the script.
I use TFAutoModelWithLMHead,TFTrainer in this repo to build my script.
When i'm traning,here is the problem.
`ValueError: Trying to create optimizer slot variable under the scope for tf.distribute.Strategy (<tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x15b49bed0>), which is different from the scope used for the original variable (<tf.Variable 'tf_bert_for_masked_lm/bert/embeddings/word_embeddings/weight:0' shape=(21128, 768) dtype=float32, numpy=array(),dtype=float32)>). Make sure the slot variables are created under the same strategy scope. This may happen if you're restoring from a checkpoint outside the scope`
I move initializing the model to TFTrainer().args.strategy.scope() and work well!

### Someing to say
- It seems that when using tf2.0, there's some problems about TFTrainer's design. Will you improve this design so that one can use tf2.0 more conveniently in this repo?
- Thanks for this repo and your contribution.


get output from a particular layer of pre-trained transformer (xlnet)

As the title, how can I do this in PyTorch version of pre-trained transformer?


DMOZ - web page classification / multi-language

Hi,

Hope you are all well !

Still quite a newbie with transformers, I wanted to know how could it be possible to build a web page classifier with the DMOZ dump and to classify them into categories in several languages.

Thanks in advance for any insights or inputs on that question.

Cheers,
X


Bug - TFBertForSequenceClassification on SQUaD data

# 🐛 Bug

## Information
I'm using TFBertForSequenceClassification on SQUaD data v1 data.

The problem arises when using:
* [ ] Both official example scripts and my own modified scripts

The tasks I am working on is:
* [ ] an official SQUaD v1 data and my own SQUaD v1 data.

## To reproduce

### Try 1 - with official squad via `tensorflow_datasets.load("squad")`, trying to mimic the following official reference -
https://github.com/huggingface/transformers#quick-tour-tf-20-training-and-pytorch-interoperability
```python
import tensorflow as tf
from transformers import TFBertForSequenceClassification, BertTokenizer, \
    squad_convert_examples_to_features, SquadV1Processor
import tensorflow_datasets

model = TFBertForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

data = tensorflow_datasets.load("squad")
processor = SquadV1Processor()
examples = processor.get_examples_from_dataset(data, evaluate=False)

dataset_features = squad_convert_examples_to_features(examples=examples, tokenizer=tokenizer, max_seq_length=384, doc_stride=128, max_query_length=64, is_training=True, return_dataset='tf')
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
opt = tf.keras.optimizers.Adam(learning_rate=3e-5)

model.compile(optimizer=opt,
              loss={'start_position': loss_fn, 'end_position': loss_fn},
              loss_weights={'start_position': 1., 'end_position': 1.},
              metrics=['accuracy'])

model.fit(dataset_features, epochs=3)
```
**Stacktrace:**  - the bug is at the `squad_convert_examples_to_features` part
```python
convert squad examples to features:   0%|             | 0/10570 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 95, in squad_convert_example_to_features
    cleaned_answer_text = " ".join(whitespace_tokenize(example.answer_text))
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/transformers/tokenization_bert.py", line 112, in whitespace_tokenize
    text = text.strip()
AttributeError: 'NoneType' object has no attribute 'strip'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/examples_git/huggingface_tf_example_squad.py", line 18, in <module>
    dataset_features = squad_convert_examples_to_features(examples=examples, tokenizer=tokenizer, max_seq_length=384, doc_stride=128, max_query_length=64, is_training=True, return_dataset='tf')
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/transformers/data/processors/squad.py", line 327, in squad_convert_examples_to_features
    disable=not tqdm_enabled,
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tqdm/std.py", line 1129, in __iter__
    for obj in iterable:
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/multiprocessing/pool.py", line 320, in <genexpr>
    return (item for chunk in result for item in chunk)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/multiprocessing/pool.py", line 735, in next
    raise value
AttributeError: 'NoneType' object has no attribute 'strip'
```

### Try 2 - readine data from file, trying to mimic the following official reference- https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb
```python
import tensorflow as tf
from transformers import TFBertForSequenceClassification, BertTokenizer, \
    squad_convert_examples_to_features, SquadV1Processor
import tensorflow_datasets

model = TFBertForSequenceClassification.from_pretrained("bert-base-cased")
tokenizer = BertTokenizer.from_pretrained("bert-base-cased")

data = tensorflow_datasets.load("squad", data_dir='/data/users/yonatab/zero_shot_data/datasets_refs')
processor = SquadV1Processor()
examples = processor.get_examples_from_dataset(data, evaluate=True)

dataset_features = squad_convert_examples_to_features(examples=examples, tokenizer=tokenizer, max_seq_length=384, doc_stride=128, max_query_length=64, is_training=True, return_dataset='tf')
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
opt = tf.keras.optimizers.Adam(learning_rate=3e-5)

model.compile(optimizer=opt,
              loss={'start_position': loss_fn, 'end_position': loss_fn},
              loss_weights={'start_position': 1., 'end_position': 1.},
              metrics=['accuracy'])

model.fit(dataset_features, epochs=3)
```
**Stacktrace:**  - the bug is at the `fit` method
```python
Traceback (most recent call last):
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/examples_git/minimal_example_for_git.py", line 97, in <module>
    main()
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/examples_git/minimal_example_for_git.py", line 69, in main
    history = model.fit(tfdataset, epochs=1, steps_per_epoch=3)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 235, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 593, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 706, in _process_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/data_adapter.py", line 702, in __init__
    x = standardize_function(x)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 660, in standardize_function
    standardize(dataset, extract_tensors_from_dataset=False)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2360, in _standardize_user_data
    self._compile_from_inputs(all_inputs, y_input, x, y)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2580, in _compile_from_inputs
    target, self.outputs)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_utils.py", line 1341, in cast_if_floating_dtype_and_mismatch
    if target.dtype != out.dtype:
AttributeError: 'str' object has no attribute 'dtype'
```

### Try 3
```python
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

    processor = SquadV1Processor()
    examples = processor.get_train_examples(args.data_dir, filename=args.train_file)
    dataset_features = squad_convert_examples_to_features(examples=examples, tokenizer=tokenizer, max_seq_length=384,
                                                          doc_stride=128, max_query_length=64, is_training=True,
                                                          return_dataset='tf')

    model = TFBertForQuestionAnswering.from_pretrained("bert-base-cased")

    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
    opt = tf.keras.optimizers.Adam(learning_rate=3e-5)

    model.compile(optimizer=opt,
                  loss={'start_position': loss_fn, 'end_position': loss_fn},
                  loss_weights={'start_position': 1., 'end_position': 1.},
                  metrics=['accuracy'])

    history = model.fit(dataset_features, epochs=1)
```
**Stacktrace:**  - the bug is at the `fit` method
```python
Traceback (most recent call last):
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/examples_git/reading_from_file.py", line 39, in <module>
    main()
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/examples_git/reading_from_file.py", line 32, in main
    history = model.fit(dataset_features, epochs=1)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 235, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 593, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 706, in _process_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/data_adapter.py", line 702, in __init__
    x = standardize_function(x)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 660, in standardize_function
    standardize(dataset, extract_tensors_from_dataset=False)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2360, in _standardize_user_data
    self._compile_from_inputs(all_inputs, y_input, x, y)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2580, in _compile_from_inputs
    target, self.outputs)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_utils.py", line 1341, in cast_if_floating_dtype_and_mismatch
    if target.dtype != out.dtype:
AttributeError: 'str' object has no attribute 'dtype'
```

### Try 4 - (after first comment here)
I'm using the code of `run_tf_squad.py` and instead of the `VFTrainer` i'm trying to use `fit`.
This is the only change I made - same dataset, same examples, same features. Just trying to use `fit`.

```python
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
    opt = tf.keras.optimizers.Adam(learning_rate=3e-5)

model.compile(optimizer=opt,
                  loss={'output_1': loss_fn, 'output_2': loss_fn},
                  loss_weights={'output_1': 1., 'output_2': 1.},
                  metrics=['accuracy'])

history = model.fit(train_dataset, validation_data=eval_dataset, epochs=1)

```
And it's the same problem that occurs:
```python
Traceback (most recent call last):
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/run_squad_tf.py", line 257, in <module>
    main()
  File "/home/ec2-user/yonatab/ZeroShot/transformers_experiments/src/run_squad_tf.py", line 242, in main
    history = model.fit(train_dataset, validation_data=eval_dataset, epochs=1)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 819, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 235, in fit
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 593, in _process_training_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 706, in _process_inputs
    use_multiprocessing=use_multiprocessing)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/data_adapter.py", line 702, in __init__
    x = standardize_function(x)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 660, in standardize_function
    standardize(dataset, extract_tensors_from_dataset=False)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2360, in _standardize_user_data
    self._compile_from_inputs(all_inputs, y_input, x, y)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2580, in _compile_from_inputs
    target, self.outputs)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_utils.py", line 1341, in cast_if_floating_dtype_and_mismatch
    if target.dtype != out.dtype:
AttributeError: 'str' object has no attribute 'dtype'

```

## Expected behavior

I want to be able to use fit on my own squad data.

## Environment info

- `transformers` version: 2.9.1
- Platform: Linux
- Python version: 3.6.6
- PyTorch version (GPU?): - Using tensorflow
- Tensorflow version (GPU?): 2.1.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

Edit:
Keras has a new tutorial for it:
https://keras.io/examples/nlp/text_extraction_with_bert/


Value matrix of self-attention


❓ Warning : This overload of addcdiv_ is deprecated

# ❓ Questions & Help

When running the [official Colab example of GLUE](https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/trainer/01_text_classification.ipynb), during training I receive a `UserWarning` :

```
/pytorch/torch/csrc/utils/python_arg_parser.cpp:756: UserWarning: This overload of addcdiv_ is deprecated:
	addcdiv_(Number value, Tensor tensor1, Tensor tensor2)
Consider using one of the following signatures instead:
	addcdiv_(Tensor tensor1, Tensor tensor2, *, Number value)
```

---

**Is it expected ?**


[Trainer] move model to device before setting optimizer

Fixes #4240

Thanks @shaoyent for diagnosing the issue


[Questions & Help] The loss doesn't decrease correctly while training BERT from scratch

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

I am now using [huggingface/transformers](https://github.com/huggingface/transformers) to train a BERT model on **1m** wiki data **from scratch**, but the training loss looks so weird. Before showing the details of the training process, I would first share the scripts and configs I used:

```
python run_language_modeling.py --output_dir $OUTPUT_DIR \
                                --model_type bert \
                                --mlm \
                                --config_name $CONFIG_AND_DATA_DIR \
                                --tokenizer_name $CONFIG_AND_DATA_DIR \
                                --do_train \
                                --do_eval \
                                --num_train_epochs 20 \
                                --learning_rate 1e-4 \
                                --save_steps 250 \
                                --per_gpu_train_batch_size 64 \
                                --evaluate_during_training \
                                --seed 404 \
                                --block_size 256 \
                                --train_data_file $DATA_DIR/train.txt \
                                --eval_data_file $DATA_DIR/valid.txt \
                                --evaluate_during_training \
                                --logging_steps 250 > log.bert
```

where [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py) is the python script provided by huggingface. I didn't change the config of BERT, except for the vocabulary size. The vocabulary, or the tokenizer, was trained using [huggingface/tokenizers](https://github.com/huggingface/tokenizers).

I put the log of training loss [here](https://gist.github.com/ghrua/01fd859707923f80f1e16af5c2bd3f6a). We can see that, after 20 epochs, the training loss decreases from `7.84` to `7.60`. That's so weird, since the number of lines of the raw data is just 1 million, the training loss should decrease sharply with such an amount of data. Note that I also used the same python script to train a GPT-2 from scratch with the same data, and it worked very well, the loss decreased as expected.

I have tried several ways to address this issue:

1. Set the batch size as big as the GPU can afford. Since BERT just predicts 15% tokens at each time step, big batch size may give the model more error signals while training. However, it doesn't help.
2. Maybe the learning rate is too small. I tried to adjust the `learning_rate` to 5e-4, unfortunately, the converged loss even became worse.
3. Maybe the `vocab.txt` has some problem, which is extracted from my training data using the toolkit [huggingface/tokenizer](https://github.com/huggingface/tokenizers). Then, I used the `vocab.txt` and `config.json` downloaded from this repo to run the python script, but it met the same problem.
4. I also run the same script on 2k data. I trained the BERT model for 200 epochs, and the converged training loss was at around `6.8`, which cannot even overfit on a toy dataset.

Thanks for your kindly help!!!

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
[Link to the question asked on SO](https://stackoverflow.com/questions/61873435/the-loss-doesnt-decrease-correctly-while-training-bert-from-scratch)


Correct TF formatting to exclude LayerNorms from weight decay

Fixes #4360

Layer Norm is formated in the wrong way for TensorFlow. This causes it not to be excluded from weight decay in the [run_tf_glue.py](https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_tf_glue.py) script.

This pr simply formats the string to fit the TensorFlow naming.

Not sure if you want a test for this? One option would be to check that all elements in `_exclude_from_weight_decay` trigger a regexp match. But seems a bit overkill.


TF Beam Search generation seems to be flaky sometimes

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): ALL TF generate models

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
* [ ] all generate beam search tests in TF

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Some commits are failing due to `Beam size should alway be full` in circle ci - this should actually never happen. See a failed circle ci here: https://circleci.com/gh/huggingface/transformers/39780?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link .

## Expected behavior

Circle ci should not fail with this message.


Make get_last_lr in trainer backward compatible

Fixes https://github.com/huggingface/transformers/issues/3959 .

@julien-c


Generation with EncoderDecoder Model

# ❓ Questions & Help

Hi,
I am using the EncoderDecoder Model and I would like to use the generate method for sequence generation. As I have read in docs, the generate method can be used with any Pretrained HF model with  LM head on top.
So wrapping a pretrained LM model (e.g. GPT2LMHeadModel, BertForMaskedLM) (and having a encoder too) with the Encoder-Decoder class gives null bos_token_id and the generate method does not work properly. However using only a pretrained LM model (without wrapping with the Encoder-Decoder class) gives a valid bos_token_id (because the config file contains bos_token_id).
How I should handle the above issue?

Thank you in advance.