BertTokenizerFast.convert_tokens_to_string converts ids to string, not tokens to string

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

The `BertTokenizerFast.convert_tokens_to_string` function expects a list of integers instead of a list of strings as the function implies. This does not happen for the normal `BertTokenizer`.

The [BertTokenizerFast](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_bert.py#L550) does not override `convert_tokens_to_string` as it is defined in [tokenization_utils_fast.py](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_fast.py#L206), which causes this issue. Within `tokenization_utils_fast.py`, the `convert_tokens_to_string` function calls `self._tokenizer.decode` which expects ids (integers not strings).

This issue does not arise when using the normal BertTokenizer because that class overrides `convert_tokens_to_string` as can be seen [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_bert.py#L230). However, the implementation in [tokenization_utils.py](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils.py#L839) is incorrect according to the docstring. The function should return `" ".join(tokens)` by default and the call to `convert_ids_to_tokens` should be removed because that function accepts ids not tokens.

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```
from transformers import BertTokenizerFast, BertTokenizer

# Error
tokenizer = BertTokenizerFast.from_pretrained("bert-base-uncased")

tokens = tokenizer.tokenize("This is a sentence.")
print(tokens)

output = tokenizer.convert_tokens_to_string(tokens)

# No Error because `convert_tokens_to_string` overridden
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

tokens = tokenizer.tokenize("This is a sentence.")
print(tokens)

output = tokenizer.convert_tokens_to_string(tokens)
```

Output:
```
['this', 'is', 'a', 'sentence', '.']
Traceback (most recent call last):
  File "test.py", line 7, in <module>
    output = tokenizer.convert_tokens_to_string(tokens)
  File "/home/user/anaconda3/envs/testing/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 209, in convert_tokens_to_string
    return self._tokenizer.decode(tokens, skip_special_tokens=skip_special_tokens)
  File "/home/user/anaconda3/envs/testing/lib/python3.8/site-packages/tokenizers/implementations/base_tokenizer.py", line 267, in decode
    return self._tokenizer.decode(ids, skip_special_tokens=skip_special_tokens)
TypeError: 'str' object cannot be interpreted as an integer
```

## Expected behavior

The `BertTokenizerFast.convert_tokens_to_string` function converts a list of tokens (which are strings) to a single string.

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


BertAbs run_summarization.py example fails with errors

# 🐛 Bug

## Information

Attempting to use BertAbs with the official example script for summarization: https://github.com/huggingface/transformers/tree/master/examples/summarization/bertabs#summarize-any-text

The language I am attempting to summarize for is English.

Simply attempting to run a command like
```python run_summarization.py --documents_dir ../../../../test-summaries/ --no_cuda true --min_length 50 --max_length 200 --alpha 0.95```
fails with the following error:
```
Traceback (most recent call last):
  File "run_summarization.py", line 15, in <module>
    from .utils_summarization import (
ModuleNotFoundError: No module named '__main__.utils_summarization'; '__main__' is not a package
```
I thought the import line was strange and changed it to `from utils_summarization import (` (note that I removed the `.` which preceded `utils_summarization`. This seemed to fix the error, although I am unsure if it is the correct fix.

Nevertheless, even with this temporary fix that I made, the `run_summarization.py` script fails with the following error:
```
INFO:filelock:Lock 140401652398456 acquired on /home/nikhil/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock
INFO:transformers.file_utils:https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to /home/nikhil/.cache/torch/transformers/tmpjgcj6x3w
Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 232k/232k [00:00<00:00, 919kB/s]
INFO:transformers.file_utils:storing https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt in cache at /home/nikhil/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
INFO:transformers.file_utils:creating metadata file for /home/nikhil/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
INFO:filelock:Lock 140401652398456 released on /home/nikhil/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084.lock
INFO:transformers.tokenization_utils_base:loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /home/nikhil/.cache/torch/transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
Traceback (most recent call last):
  File "/home/nikhil/.pyenv/versions/huggingface/lib/python3.6/site-packages/transformers/configuration_utils.py", line 243, in get_config_dict
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_summarization.py", line 324, in <module>
    main()
  File "run_summarization.py", line 309, in main
    evaluate(args)
  File "run_summarization.py", line 33, in evaluate
    model = BertAbs.from_pretrained("bertabs-finetuned-cnndm")
  File "/home/nikhil/.pyenv/versions/huggingface/lib/python3.6/site-packages/transformers/modeling_utils.py", line 602, in from_pretrained
    **kwargs,
  File "/home/nikhil/.pyenv/versions/huggingface/lib/python3.6/site-packages/transformers/configuration_utils.py", line 201, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/nikhil/.pyenv/versions/huggingface/lib/python3.6/site-packages/transformers/configuration_utils.py", line 252, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load config for 'bertabs-finetuned-cnndm'. Make sure that:

- 'bertabs-finetuned-cnndm' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bertabs-finetuned-cnndm' is the correct path to a directory containing a config.json file

```
Based on the error message, I looked up `bertabs-finetuned-cnndm` on https://huggingface.co/models to find that there is no exact match for this model name. The closest match is called `remi/bertabs-finetuned-cnndm-extractive-abstractive-summarization`. Should the script be updated to include this model name instead?

## Environment info
Output of `transformers-cli env`:

```
- `transformers` version: 2.11.0
- Platform: Linux-4.4.0-18362-Microsoft-x86_64-with-debian-bullseye-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No
```


Fix convert_graph_to_onnx script

- Remove all references to `args` in methods, using arguments instead. This lets us use the `convert` method directly by importing it in another script.
- Check that the wanted framework is installed before creating the pipeline, otherwise, it might fail to instantiate it.


Cleaning TensorFlow models

While writing docstrings for #5036, I stumbled upon a few bugs in TensorFlow, especially related to the loss computation. I'm patching them in this PR.

Here's the list of the bugs solved:

### Loss computation

- The loss is computed differently than it is with the PyTorch models. Here it is returned example-wise (therefore with a shape of `(batch_size,)`, whereas the PyTorch models return the loss as a scalar. @jplu is there a reason for this implementation?
- The TensorFlow models should be able to handle three types of scenarios: keyword arguments, dictionary, and tuple/list. Right now the `labels` can only be passed through the keyword argument. This PR changes that, and adds a test.

### Missing models in the test files

A few models were implemented but were not tested. Some of these models were not working as expected, therefore they've been updated.

- TF DistilBERT for multiple choice (added test and patched)
- TF DistilBERT for token classification
- TF Electra for QA
- TF RoBERTa for multiple choice
- TF XLNet for multiple choice (added test and patched)

### Misc

- Most of the QA models had `is_impossible`, `cls_index` and `p_mask` in their signature while not making use of them. These have been removed. **Users relying on the order of arguments in the signature will be affected by this**
- The `labels` were generally placed before the `output_attentions` and `output_hidden_states` that have recently been added to the models. This resulted in an error in the documentation as the `labels` (part of the head model) were added after the `output_attentions` and `output_hidden_states` (part of the base model). The arguments have been re-ordered to once again respect the order of `**base_arguments, **head_arguments`


Embedding index out of range in self

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert cased (size=768)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below) When I feed the ids converted by BERTtokenizer to BERT embedding layer, it shows that the dimension does not match.

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below) task: text binary classification. dataset: ICLR2020 peer reviews

## To reproduce

Steps to reproduce the behavior:

1. use BERT model and BERT tokenizer
2. convert the text of any datasets to ids
3. feed to the BERT model

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

The dimension should match. Actually, the code works two days ago. I did not change anything and today it does not work.

The error is: return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) and index out of range in self.

## Environment info

2020-06-23 23:18:22.738569: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/transformers/commands/env.py:36: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2020-06-23 23:18:24.794569: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX512F
2020-06-23 23:18:24.849719: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2000160000 Hz
2020-06-23 23:18:24.850210: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x42a8bc0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-06-23 23:18:24.850259: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-06-23 23:18:24.857345: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-06-23 23:18:24.860619: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-06-23 23:18:24.860659: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (68cbbf79e491): /proc/driver/nvidia/version does not exist

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 2.11.0
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


[pl_examples] revert deletion of optimizer_step

using default `optimizer_step` has at least 2 issues:
The default version...
1) doesn't call `lr_scheduler.step()`
2) does call `self.trainer.scaler.step(optimizer)`

I haven't diagnosed which of these is the main culprit of the issue I was seeing (very high loss, not going down). This fixes that issue.

I suspect it is mostly the latter: @williamFalcon


Self documenting Payload instead of Tuples as output of Transformer

Core: Modeling

Internals of the library; Models.

I propose a replacing the default Tuple outputs with payloads where every field in the tuple is accessible by a name. This has the following benefits:

1. Being more accessible to newcomers of the library
2. Eliminating suspicious comments in the source code describing the output (see below)
3. Make it easier to extract particular values from different models for inspection -- adding a named field is easier than mangling the order of an existing Tuple
4. Like 3, where new models can output their own metadata without reordering expected structure
5. A consistent output structure makes it easier to compare different models in applications that involve multiple architectures.

## Motivation
This will be my first formal feature request, as I am getting weary of forgetting what the Tuple output in the forward pass of a Transformer means. I find myself constantly returning to the source code for each model and playing with the shapes and values of the different fields to check whether something is what I expect it to be.

### The Problem

Currently, the output of a Transformer (I use Bert as an example here, potentially not the most recent version) is structured and documented in the source code as follows:

``` python
...
    for i, layer_module in enumerate(self.layer):
        if self.output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        layer_outputs = layer_module(
            hidden_states, attention_mask, head_mask[i], encoder_hidden_states, encoder_attention_mask
        )
        hidden_states = layer_outputs[0]

        if self.output_attentions:
            all_attentions = all_attentions + (layer_outputs[1],)

    # Add last layer
    if self.output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    outputs = (hidden_states,)
    if self.output_hidden_states:
        outputs = outputs + (all_hidden_states,)
    if self.output_attentions:
        outputs = outputs + (all_attentions,)
        if self.output_additional_info:
            outputs = outputs + (all_additional_info,)

    return outputs  # last-layer hidden state, (all hidden states), (all attentions)
```

Of course, that last comment is highly dependent on what you pass to the configuration. What if you desire all the attentions but not the hidden_states? Now, attention is at `outputs[1]` instead of `outputs[2]`. Utterly confusing.

And what if you have an architecture that has different outputs? Here is an example from the output of a T5 model.

``` python
...
    # Add last layer
    if self.output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)

    outputs = (hidden_states,)
    if use_cache is True:
        assert self.is_decoder, "`use_cache` can only be set to `True` if {} is used as a decoder".format(self)
        outputs = outputs + (present_key_value_states,)
    if self.output_hidden_states:
        outputs = outputs + (all_hidden_states,)
    if self.output_attentions:
        outputs = outputs + (all_attentions,)
    return outputs  # last-layer hidden state, (presents,) (all hidden states), (all attentions)
```

There's this new field `presents` that again confuses the order. It starts to get a bit confusing.
In addition, one of my projects the past many months has been to visually compare and interpret different Transformer models [exbert](http://exbert.net/). This means that I often want to edit the source code and extract, for example, the keys / values / head embeddings prior to the final projection into the embeddings that are passed to the next layer. Extracting these and passing them through the model is more complicated than it should be -- there are no hooks that I can use to catch arbitrary information within a module's forward pass (potentially a separate feature request, but I feel this would slow the prototyping speed of this library quite a bit), and I worry about messing with an expected order to the Tuple output. It is also really easy to forget which field in a list of 8 items is the one I want. Architectures are also incorporated quickly into Transformers (kudos!), and it would be great to know what inference information I have available for a model simply by looking at the object outputted by the forward pass.

### Possible Solutions

I would like the return object of every Transformer's forward pass to be a Payload where the information outputted is easily identified by the fields. E.g., for a LMHead Transformer:

``` python
{
logits: __,
past: __,
hidden_states: __,
attentions: __,
}
```
Where a non-LMHead Transformer would not include logits and offset indexing into the output. This also allows models like T5 to unambiguously add additional fields without compromising the structure of the tuple.

``` python
{
last_layer_hidden_state: __,
presents: __,
}
```

It would be trivial to add fields to this at the output:

``` python
    output = {"last_hidden_state": hidden_states}
    if self.output_hidden_states:
        all_hidden_states = all_hidden_states + (hidden_states,)
        outputs['all_hidden_states'] = all_hidden_states

    if self.output_attentions:
        outputs['attentions'] = all_attentions

    return outputs  # Self documenting fields, no comment needed! :D
```

Payloads like this would also work for intermediate modules, though some naming convention to indicate the main output intended to be used by the next module/layer would be necessary.

Python's `namedtuples` could also be an option, though easily adding fields to this immutable structure is a bit more challenging for interpretability work. Additional alternatives could be [`namedlist`](https://pypi.org/project/namedlist/) or [`recordclass`](https://pypi.org/project/recordclass/).

## Contribution

I am willing to continue thinking of solutions and work towards this goal, but making a single PR would be both a sweeping change across library and the way each module is coded. However, I believe this change to be important for increasing the accessibility of the library for both newcomers and those who want to build applications around the different models.


Add hugs

Enforce that there are not transformers, Transformers, `transformers` but only 🤗 Transformers in the documentation.


Use the script in utils

Since we have the script `download_glue_data` in the utils folder, changing the instructions in the README for the GLUE example to use it for now (of course nlp will ultimately make this even easier) since it's easier than copying the gist in a local file.


Only put tensors on a device

Fix Trainer when users have inputs containing non-tensor values.


Add version control menu

This PR adds at the top of the navigation bar a menu to pick a version of the docs.

A few comments:

When switching version, the reader is sent on the same page of the docs in the older version (so it gives an error if the same page did not exist in this version of the docs). I don't know if this is preferable to the alternative (sending back to the index of the other version in the docs). Let me know what you think.

The menu will disappear once the reader goes to an older version of the docs (since it did not exist back then) unless we find a way to cherry pick in each release (but I doubt it's worth it).

Preview is [here](https://51918-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html). Side note, on a local build (like this one), the version appears as 'html', but it will be the right one once merged. Also, the preview/local build only has one version, so the links don't work there.


gpt2.generate breaks on FP16 Apex training.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): GPT2

Language I am using the model on (English, Chinese ...): Molecule

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Run Pytorch lightning with Apex (or just default apex training), then during validation try and generate samples with the model (which is on fp16).

## Trace

```
2020-06-23T18:30:00.337+01:00 | THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCReduceAll.cuh line=327 error=716 : misaligned address
-- | --
  | 2020-06-23T18:30:00.338+01:00 | Validation sanity check: 0it [00:00, ?it/s] Validation sanity check: 50% 1/2 [00:01<00:01, 1.08s/it]generating smiles
  | 2020-06-23T18:30:00.341+01:00 | Traceback (most recent call last):
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/bin/transformervae", line 11, in <module>
  | 2020-06-23T18:30:00.341+01:00 | load_entry_point('exs-transformervae', 'console_scripts', 'transformervae')()
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 829, in __call__
  | 2020-06-23T18:30:00.341+01:00 | return self.main(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 782, in main
  | 2020-06-23T18:30:00.341+01:00 | rv = self.invoke(ctx)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return _process_result(sub_ctx.command.invoke(sub_ctx))
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return ctx.invoke(self.callback, **ctx.params)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/click/core.py", line 610, in invoke
  | 2020-06-23T18:30:00.341+01:00 | return callback(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/cli.py", line 404, in pretrain
  | 2020-06-23T18:30:00.341+01:00 | trainer.fit(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/uwandb: Waiting for W&B process to finish, PID 28
  | 2020-06-23T18:30:00.341+01:00 | ser/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in fit
  | 2020-06-23T18:30:00.341+01:00 | self.single_gpu_train(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 176, in single_gpu_train
  | 2020-06-23T18:30:00.341+01:00 | self.run_pretrain_routine(model)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1076, in run_pretrain_routine
  | 2020-06-23T18:30:00.341+01:00 | False)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 330, in _evaluate
  | 2020-06-23T18:30:00.341+01:00 | eval_results = model.validation_epoch_end(outputs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/base.py", line 95, in validation_epoch_end
  | 2020-06-23T18:30:00.341+01:00 | return self._shared_eval_end(output, "val")
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/lm.py", line 152, in _shared_eval_end
  | 2020-06-23T18:30:00.341+01:00 | use_cache=True,
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | return func(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/app/transformervae/models/lm.py", line 173, in generate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | generated_ids = self.encoder.generate(input_ids, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
  | 2020-06-23T18:30:00.341+01:00 | return func(*args, **kwargs)
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1181, in generate
  | 2020-06-23T18:30:00.341+01:00 | model_specific_kwargs=model_specific_kwargs,
  | 2020-06-23T18:30:00.341+01:00 | File "/home/user/miniconda/envs/py36/lib/python3.6/site-packages/transformers/modeling_utils.py", line 1285, in _generate_no_beam_search
  | 2020-06-23T18:30:00.341+01:00 | if unfinished_sents.max() == 0:
  | 2020-06-23T18:30:00.341+01:00 | RuntimeError: cuda runtime error (716) : misaligned address at /opt/conda/conda-bld/pytorch_1579022034529/work/aten/src/THC/THCReduceAll.cuh:327

```

## Expected behavior

Should generate samples.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Ubuntu
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 cu101
- Tensorflow version (GPU?): n/a
- Using GPU in script?: V100
- Using distributed or parallel set-up in script?: no but FP16 training


run_language_modeling.py does not output vocab/config/etc files until training completes

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
GPT2

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* the official example scripts:
I began training using the following command (from Jupyter Lab) - output_100k is the specified (new) output folder for my fine tuned model (which in being trained on 100k US patents):
`!python run_language_modeling.py \
    --output_dir=output_100k \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --block_size 100 \
    --per_device_train_batch_size 3 \
    --do_train \
    --train_data_file=./train_100k.txt \
    --do_eval \
    --eval_data_file=./test_100k.txt`

This works fine; HOWEVER, no instructions are given in the example [README](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) on how to resume training later on.
**Including this information would be extremely helpful**. After some personal research, I deduced (perhaps incorrectly?) that I should point 'model_name_or_path' at the output_100k checkpoint folder like so (note: the README also doesn't specify whether or not it's okay to keep the original output-dir name, so I made a new one):
`!python run_language_modeling.py \
    --output_dir=output_100k_resumed \
    --model_name_or_path=./output_100k/checkpoint-12500 \
    --block_size 100 \
    --per_device_train_batch_size 3 \
    --do_train \
    --train_data_file=./train_100k.txt \
    --do_eval \
    --eval_data_file=./test_100k.txt`

This raises the following error:
`OSError: Model name './output_100k/checkpoint-12500' was not found in tokenizers model name list (gpt2, gpt2-medium, gpt2-large, gpt2-xl, distilgpt2). We assumed './output_100k/checkpoint-12500' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.`

I checked the directory, and indeed, there are no vocab.json, merges.txt, etc files included in the specified output directory from the initial training.

It appears that **run_language_modeling.py** does not output these files until the **end** of training (an examination of other models training using the script shows that these files are present in completed training sessions.

**What am I doing wrong?** Or is this a bug in the script itself? I'd imagine these vocab/merges files can be outputted relatively early in training. Note, when training on a new dataset starts, a file with the name of the form `cached_lm_GPT2Tokenizer_[SOME NUMBER]_train_clean.txt.lock` (and one without the .lock) is generated. Perhaps this contains some information that I'd need to resume training? Regardless, I see no documentation explaining this file's purpose linked in the README, which - again - could definitely provide more context.

The tasks I am working on is:
* my own task or dataset: (give details below)
The idea is to build an effective context-aware text generator for legal purposes. The dataset simply consists of a bunch of patent document texts.

## To reproduce

Steps to reproduce the behavior:
**See description above**

## Expected behavior

Model resumes training at the specified checkpoint.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform: Linux-5.3.0-59-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No modifications to script, but yes, this computer uses GPU.
- Using distributed or parallel set-up in script?: No.


[Longformer] Major Refactor

## Longformer Refactor

This PR does a major refactoring of Longformer. Mainly, the Roberta abstraction is removed and compositionally is chosen instead. This has the following advantages:

- It's easier now to implement a `cross_attention_layer`
- The code is more readable and the logic stays in this file only
- A bug was corrected regarding the attention mask. @ibeltagy - maybe you can check this as well. Previously, if **no** `attention_mask` was inserted, the padding function that became before `super.forward()` in `LongformerModel` was not used, **but** if instead an `attention_mask = torch.tensor([1, ..., 1])` (attend to all tokens was passed, the padding function was applied and could lead to different outputs as when no `attention_mask` is passed. This should not be the case. `model(input_ids)` and `model(input_ids, attention_mask=torch.ones(input_ids.shape))` should always yield the same result. Removing the `super.forward()` abstraction makes the code much cleaner here so that a `attention_mask = torch.ones(input_ids.shape)` can be calculated before calling the longformer encoder. **IMPORTANT** Since in almost all tasks longformer somehow passes either a `global_attention_mask` or `attention_mask` to `LongformerModel`, this bug did not really become visible before.
- We don't have to "inject" a `self-attention layer` into another model anymore, which I did not like very much.
- Unnecessary code can be removed (head_mask, prev cross-attention layer inputs that do not work yet), ...

**Additionally**:
- Variable names are made more explicit and dead code (If statements that would have never occurred) was removed and code is simplified.
- The forward function of the self-attention layer is broken up into multiple helper functions. The advantage here is that quite some memory should be saved before `attention_probs` go out of scope after they are not used anymore and thus the memory bottleneck should be reduced.
- All longformer models are added to the tests (@sgugger) and a couple more tests are added.

Next step is to add cross attention layers to longformer.

**Review**

I made sure that besides the bug with `attention_mask = None` vs `attention_mask = torch.ones(...)` all outputs stay the same.
Would be great if @thomwolf  @LysandreJik @sgugger @sshleifer @ibeltagy can do a quick review.


TensorFlow

Anything TensorFlow

I tried to run the pipeline task 'summarization', but get a error with "module **'tensorflow' has no attribute 'repeat' "** Does anyone encountered the same problem? How to fix it?

**my installed tensorflow == 2.0.0**

error messages:
/home/ww/anaconda3/envs/environment_name/lib/python3.6/site-packages/transformers/pipelines.py", line 1446, in __call__
    inputs["input_ids"], attention_mask=inputs["attention_mask"], **generate_kwargs,
  File "/home/ww/anaconda3/envs/environment_name/lib/python3.6/site-packages/transformers/modeling_tf_utils.py", line 747, in generate
    tf.repeat(tf.expand_dims(tf.range(batch_size), -1), repeats=num_beams * effective_batch_mult, axis=1),
AttributeError: module 'tensorflow' has no attribute 'repeat'


Create README.md

model card

Related to pretrained model cards

electra_large_discriminator_squad2_512 Question Answering LM


[WIP - Don't merge yet][Pipeline] Make "task" a static class variable

This PR removes "task" from the "init" of the class and adds it as a static variable.
In my opinion, it is cleaner to have "task" as a static variable instead of an object attribute. This would also solve #5210 .

The problem we would run into then is that for the "Translation" pipeline we would need a class for each translation. I think this is still the better option though because it is less prone to errors (see #5210) and we could add a "get translation factory design" or something.

After some discussion with @mfuntowicz, I think the best option is to add two additional
parameters `src_lang` and `tgt_lang` to the pipelines function and delete the task names
`translation_en_to_fr` in favor of just `translation`.

The new recommended way of instantiating a translation pipeline is

```python
translation_en_to_fr = pipeline("translation", src_lang="en", tgt_lang="fr")
```

The option:
```python
translation_en_to_fr = pipeline("translation_en_to_fr")
```

is still supported with a future warning.

@mfuntowicz @julien-c @LysandreJik @sshleifer
What do you think?

**Backward Compatibility**

It should be fully backward compatible, but I added some warning statements to let the user know of Future Depreciation

### TODO:
If this PR is ok for you, I will update the docs and add tests.


TF2 support for Longformer

# 🚀 Feature request

Hi,

I'm currently working on a project involving long documents (6000+ tokens).  I normally work with Tensorflow and I was wondering if there are any plans for adding a Longformer TF model in the near future? My PyTorch knowledge is fairly limited, but given the potential of the Longformer for my project, I would want to learn the basics if there are no plans for adding TF support.

Kind regards


How to predict on a batch?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
[Link](https://stackoverflow.com/questions/62533181/huggingface-transformers-library-predict-in-batches)


Train EncoderDecoder Models for question generation

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
### How to train models for text generation
Hi Everyone,

I am trying to finetune an Encoder Decoder model on question generation task on SQuAD.
Input data are a concatenation of answer span and context and outputs are the question.

`inputs = tokenizer.encode_plus(example.answer, example.context, add_special_tokens=True, max_length=max_length, truncation='only_second')`

`label = tokenizer.encode_plus(example.question, add_special_tokens=True, max_length=max_length_label, truncation=True)`

`decoder_input_ids, label_ids = data_collator.mask_tokens(torch.tensor(label_ids).unsqueeze(0))`

I add padding to all of these arguments if necessary and pass them to the model which can be:

- an encoder decoder model:
`model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')`
`inputs = {'input_ids': batch[0], 'attention_mask': batch[1], 'token_type_ids' : batch[2], 'decoder_input_ids': batch[3], 'lm_labels' : batch[5]}`
`outputs = model(**inputs)`

- a BART model :
`model = BartForConditionalGeneration.from_pretrained(model_name)`
`inputs = {'input_ids': batch[0], 'attention_mask' : batch[1], 'decoder_input_ids': batch[2], 'labels' : batch[3] }`

I thought that everything was alright and I started training my two models. As the training progressed, the mlm_probability of the datacollator object increased from 0.20 to 0.40 and then to 1.
The learning rate and the optimizer are as follows: (lr around 3e-5)

`optimizer = AdamW(model.parameters(), lr=learning_rate, eps=adam_epsilon)`
`scheduler = get_cosine_with_hard_restarts_schedule_with_warmup(optimizer, num_warmup_steps=warmup_steps, num_training_steps=t_total, num_cycles=num_cycles)`

The eval loss was decreasing all along the 100 epochs for the BERT2BERT but it didn't looked like the questions were improving:
epoch 50:
what country did the french support in libya????? - 2013, 2014??
what country did nasser end to the coup? in 1989, 2007 and 2008 - 2011's
what country did the us state have to use a particular prohibition of fuel in its oil? 2007

epoch 100:
where was the fisafat for? islamic party in libya and al - farabut movement
what did the unfyadi want to end in 1990? - 1991, 2003 and gulf
what country did the oil industry stop its fuel and coaling? in a world, which countries

The observation remains the same for BART model:
100 steps:
? what was the name of normnormandy in frfrance.
? when did people in the first half of what began to give their
? who were the people that did not to swear fealty oath in

4 epochs:
 normnormnaandyanye gave given offered name namesNames to forfor
 normnormNormansons gave given granted their own original initial ancestral native
 normnormdonaldansons descended originated originate originating from origins origin?ers

My questions are:
**Do you think that something is wrong with my training?
What do you think about the performances?
Do you have any suggestions for the question generation task?
How does the decoder input ids is supposed to change for Next word prediction loss ?
Should I use Next word prediction loss or should I use Masked lm loss ?
How to use dropout with pretrained model?**

Thank you in advance for your help and I hope that my post will be usefull to others, if need be I can share a bigger part of my code :)


BartConfig wrong decoder_start_token_id?

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bart

Language I am using the model on (English, Chinese ...): English

## To reproduce

Steps to reproduce the behavior:

```
from transformers import BartConfig, BartTokenizer
config = BartConfig.from_pretrained('facebook/bart-large')
tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
config.decoder_start_token_id
>>> 2
tokenizer.bos_token_id
>>> 0 # != config.decoder_start_token_id
tokenizer.eos_token_id
>>> 2
```

It is misleading in the documentation of the function ```generate````

*decoder_start_token_id=None – (optional) int If an encoder-decoder model starts decoding with a different token than BOS. Defaults to None and is changed to BOS later.*

## Expected behavior

I expect that decoder_start_token_id = tokenizer.bos_token_id, but maybe the model is designed to start decoding with EOS token.


Remove wandb warning as it is unnecessary. If wandb is installed, this throws a warning which just makes noise.
Some images might have wandb installed but the user doesn't want to use it. If the user knows what wandb is they will have the API key set anyways.


Increase the default max_length parameter when using TransfoXL & XLnet.

This is needed because the prepended PADDING_TEXT constant is already bigger than the default max_length parameter on the generate method, thus leading to no token being generated.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


[Reformer] Axial Pos Emb Improve mem usage reformer

This PR improves memory usage of Axial Position Encodings by cutting position encodings only to the required length before applying contiguous pytorch operations.


Train RobertaModel from scratch for my dataset

I am trying to train RobertaModel from scratch. I am following [this](https://huggingface.co/blog/how-to-train) blog but instead of `model = RobertaForMaskedLM(config=config)`, I am starting with `configuration = RobertaConfig()
model = RobertaModel(configuration)` and then continuing with other steps. But I am getting error `TypeError: forward() got an unexpected keyword argument 'labels'`. The whole code piece:
```
configuration = RobertaConfig()
model = RobertaModel(configuration)
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./train.txt",
    block_size=128,
)
from transformers import DataCollatorForLanguageModeling

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=False, mlm_probability=0.15
)
from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./Model1",
    overwrite_output_dir=True,
    num_train_epochs=1,
    per_gpu_train_batch_size=64,
    save_steps=10_000,
    save_total_limit=2,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)
trainer.train()
```
Is there some other way to do pre-training? Am i missing something here?


How to build Bimodel to search code snippets? [CodeBERTa]

Discussion

Discussion on a topic (keep it focused or open a new issue though)

Hi,

I would like build a code search engine model. The main purpose is that when I pass docstring, it should give me top-k associated code snippets as results.

I have a data in the form of (docstring, code), which means each docstring is associated with mentioned code snippet.

I have seen CodeBERTa fine tune [code](https://huggingface.co/huggingface/CodeBERTa-language-id), but it is not using docstring in it. Is it possible to use this model?

Can you please give me some entry points to solve this problem by using hugging-face library?


[fix] remove unused import


[fix] mobilebert had wrong path, causing slow test failure

Also deleted redundant slow test. `test_inference_no_head` covers this completely.


T5 Model : What is maximum sequence length that can be used with   pretrained T5 (3b model) checkpoint?

As the paper described, T5 uses a relative attention mechanism and the answer for this [issue](https://github.com/google-research/text-to-text-transfer-transformer/issues/273) says, T5 can use any sequence length were the only constraint is memory.

According to this, can I use T5 to summarize inputs that have more than 512 tokens in a sequence?


Can you release the code for Write For Transformer?

I'd like to use my own model, so is this possible?


examples/seq2seq supports translation

Examples

Which is related to examples in general

translation

machine translation utilities and models

- renames `examples/summarization` -> `examples/seq2seq`
- finetune.py and run_eval.py support mbart, marian and t5.
- task_specific_params are used
- if you specify task='translation', then your metric becomes BLEU instead of ROUGE.
- improved `README.md`
- lots of test coverage
- scripts to reproduce distilbart results

TODO:
- [x] verified distilbart commands replicate posted results.
- [x] new xsum shared task URL.
- [x] mini models for marian
- [x] mbart finetuning unittests.

Postponed and made issues for:
- [ ] check bleu scores for translation models with run_eval.py


Linformer

# 🌟 New model addition

## Model description

https://arxiv.org/pdf/2006.04768.pdf

This model is very simple, it just projects the key tensor into a lower dimensional space (e.g k=128) along the sequence axis, then computes attention (seq_len x k), softmax, and matmul with the value tensor (note, the value tensor must also be projected to dimension k along the length dimension).

Could this be added?

## Open source status

Pytorch sketch implementation: https://github.com/tatp22/linformer-pytorch


tokenizer.convert_ids_to_tokens(tokenizer.convert_tokens_to_ids(x)) returning a different result

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I have a trained Esperanto Tokenizer, the training code was taken from the how_to_train example. I'm using Roberta Tokenizer as RobertaTokenizerFast doesn't work with trainer.py yet (or last time I checked).

```python
from transformers import RobertaTokenizer

alt_tokenizer = RobertaTokenizer.from_pretrained("./drive/My Drive/EsperBERTo", max_len=512)

alt_tokenizer.convert_ids_to_tokens(alt_tokenizer.convert_tokens_to_ids(alt_tokenizer.tokenize('Neniu atendas la Hispanan Inkvizicion.')))

```
output is: ['Neniu', 'Ġatendas', 'Ġla', 'ĠHispan', 'an', 'ĠIn', 'kvizi', 'cion', '.']

## Expected behavior

the same output as input

## Environment info
tokenizers               0.8.0rc1
transformers             2.11.0

- `transformers` version: 2.11.0
- tokenizers version: 0.8.0rc1
- Platform: colab
- Python version: 3.6
- Using GPU in script?: No


Update README.md

model card

Related to pretrained model cards

Fix/add information in README.md


[Reformer classification head] Implement the reformer model classification head for text classification

This PR is an effort to implement the classification head on top of the plain vanilla reformer model.
References have been taken from the Roberta and XLNet classification head to implement the reformer classification head.

Tried testing the implementation changes, but it is failing error, link to google collab https://colab.research.google.com/drive/1KFsQxLqsMB6vBF4_bRmTFGhdGwkgx0zI?usp=sharing;
Test scenario details;

- IMDB movie review dataset.

-  The cell3 has the reformer model classification head code.

- Since the length of the reviews is not more than 512 words( assuming there would be a few outliers, but that could be skipped)

- Initializing the model with the config.axial_pos_shape to (16,32) but the setting does not take effect and throws runtime error.

Please help.


batch_encode_plus() causes OOM, while encode_plus does not

Core: Tokenization

Internals of the library; Tokenization.

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I am running a sequence classification task using `DistilBertForSequenceClassfication`. I follow `examples/text_classfication/run_glue.py` and `src/transformers/data/processors/glue.py` to implement my data loading process. My dataset is a rather large one (~2.5 GB with 7M+ examples), compared to those of the GLUE tasks.

In the current `glue.py`, `_glue_convert_examples_to_features()` reads all the examples into a list, and then call `batch_encode_plus()` on that list. On my large dataset, this implementation caused an out-of-memory (OOM) error. Therefore, I switched to `encode_plus()`, and called it on individual data example while looping through the dataset. `encode_plus()` did not cause OOM.

I wonder if there is something wrong with `batch_encode_plus()` so that it cannot handle all the examples in a dataset at once? If that is the case, it might be a good idea to add a corresponding note to the documentation.


[HANS] Fix label_list for RoBERTa/BART (class flipping)

the MNLI checkpoints ported from fairseq (more specifically RoBERTa and BART) has flipping classes. @sshleifer  fixed it for `run_glue.py` last week (see https://github.com/huggingface/transformers/pull/5141), this PR fixes the same problem for `run_hans.py`.

Now the HANS evaluation for [bart-large-mnli](https://huggingface.co/facebook/bart-large-mnli) gives:
```
Heuristic entailed results:
lexical_overlap: 0.9932
subsequence: 0.9996
constituent: 0.9982

Heuristic non-entailed results:
lexical_overlap: 0.8002
subsequence: 0.263
constituent: 0.2074
```

For [roberta-large-mnli](https://huggingface.co/roberta-large-mnli):
```
Heuristic entailed results:
lexical_overlap: 1.0
subsequence: 1.0
constituent: 1.0

Heuristic non-entailed results:
lexical_overlap: 0.8746
subsequence: 0.2804
constituent: 0.1056
```


Add link to new comunity notebook (optimization)

related to https://github.com/huggingface/transformers/issues/4842#event-3469184635

This notebook is about benchmarking model training with/without dynamic padding optimization.
https://github.com/ELS-RD/transformers-notebook

Using dynamic padding on MNLI provides a **4.7 times training time reduction**, with max pad length set to 512. The effect is strong because few examples are >> 400 tokens in this dataset. IRL, it will depend of the dataset, but it always bring improvement and, after more than 20 experiments listed in this [article](https://towardsdatascience.com/divide-hugging-face-transformers-training-time-by-2-or-more-21bf7129db9q-21bf7129db9e?source=friends_link&sk=10a45a0ace94b3255643d81b6475f409), it seems to not hurt performance.

Following advice from @patrickvonplaten I do the PR myself :-)


[Use cache] Align logic of `use_cache` with output_attentions and output_hidden_states

This PR cleans the usage for `use_cache` the same way it is done for `output_hidden_states` and `output_attentions`.

The logic as for all non-tensor function arguments is that
1) if the function argument `use_cache` is specified use this one
2) if not then use the `config.use_cache`.

**Break in Backward Compatibilty**
There is a small break in backward compatibility, in that Bart now uses "use_cache=True" as a default value *if* both `input_ids` and `decoder_input_ids` are passed as an argument. Bart always used `use_cache=True` as a default for generation, so this should not concern many people IMO.


Switch master/stable doc and add older releases

Switch the doc base url to the latest stable release and add master docs as well as some intermediate docs that were missing.


Using segments ids in encoder-decoder model in generate function

# 🐛 Bug

## Information

I have implemented an encoderdecoder model where both of them are bert, in encoder module i'm using segment ids (token_type_ids) and give it to model easily in this way and works correctly:
```
kwargs = {'token_type_ids':segments_tensors}
outputs = model(input_ids=encoder_input, decoder_input_ids=decoder_input, **kwargs)[0]
```
but in generate function i use below code to give segment ids:
```
kwargs = {'token_type_ids':segments_tensors}
generated = model.generate(encoder_input, decoder_start_token_id=101
num_return_sequences=16,num_beams=32, **kwargs)
```
I thought this should work but commenting or uncommenting kwargs shows me that no changes in output have happened. Is there another correct approach to give arguments of encoder module and token_type_ids through generate function or not?


[Benchmark] Jetson Nano DistillBERT SQuAD benchmark

# 🖥 Benchmarking `transformers`

## Benchmark

DistillBERT SQuAD

## Set-up

- Jetson Nano
- Jetpack 4.4
- onnxruntime 1.13.1

Benchmark script https://gist.github.com/arijitx/1400d3d4e07fc517d6c5bfea506c2353

## Results

tokens/sec

|                    | Pytorch-GPU | Pytorch-CPU(4 cores) | onnx-CPU (4 core) | onnx-CUDA | onnx-TRT |
|--------------------|-------------|----------------------|-------------------|-----------|----------|
| Distill BERT SQuAD |         570 |                   61 |               107 |  605      |     fail     |


[bart] add config.extra_pos_embeddings to facilitate reuse

Previously bart's LearnedPositionalEmbedding basically assumed pad_token_id>=1, and added 2 extra spaces of empty embeddings at the beginning. These positions were never used.

Since the blenderbot state dict has only 128 entries (exactly as many as it needs, whereas bart has 1026, 2 more than it needs). it cannot use `LearnedPositionalEmbedding` unless we either allow this offset to bet set to 0, or add empty indices.

I prefer the former since the offset's motivation is not super clear to me.

I also move `create_position_ids_from_input_ids` into modeling_roberta.py, the only place it is used.


Have documentation fail on warning


Simplify LearnedPositionalEmbedding

`create_position_ids_from_input_ids` and `LearnedPositionalEmbedding`
were both copied from fairseq and need either much more documentation or much simpler logic. They were originally copied for Roberta and now are also used for bart.


Add TF auto model to the docs + fix sphinx warnings (again)

The goal of this PR was just to add the automodel to the docs (as pointed out in #5145) but then I had 100 sphinx warnings that made me grumpy so I had to fix them (some of them from the auto model docs, but most of them from #4978).

As usual a few of them were harmless, others had a real impact on the docs, so I think we should make sure in the CI that there are no sphinx warnings to avoid making the docs bad by mistake.


Trouble with PL Checkpoint loading after finetuning bart-large

Examples

Which is related to examples in general

Hi there,

I recently ran "finetune_bart.sh" with data from cnn_dm/train.source and saved checkpoints. I only have GPU with 12G memory, then I trained this model with batch_size 1 without distributed training.

I predicted test summaries results on cnn_dm/test.source with:

```python
        tokenizer = AutoTokenizer.from_pretrained(args.model_name_or_path)
        checkpoints = list(sorted(glob.glob(os.path.join(args.output_dir, "checkpointepoch=*.ckpt"), recursive=True)))
        model = model.load_from_checkpoint(checkpoints[-1])
        model.eval()
        model.freeze()
        fout = Path("test_generation.txt").open("w", encoding="utf-8")
        for batch in tqdm(list(chunks(examples, 1))):
            inputs = tokenizer.batch_encode_plus([batch[0]], max_length=1024, return_tensors='pt')['input_ids']
            outputs = model.text_predictions(inputs)
            fout.write(outputs[0] + "\n")
            fout.flush()

        output_lns = [x.rstrip() for x in open("test_generation.txt").readlines()]
        reference_lns = [x.rstrip() for x in open("cnn_dm/test.target").readlines()]
        calculate_rouge(output_lns, reference_lns, "rouge_scores.txt")
```

After that I received results:
ROUGE_1:
AggregateScore(low=Score(precision=0.428813534233686, recall=0.38720984475275855, fmeasure=0.3951340300593005), mid=Score(precision=0.4311467253592325, recall=0.38942524434531234, fmeasure=0.3970368964972586), high=Score(precision=0.4336709710273236, recall=0.3917389347240298, fmeasure=0.39884900115657695))

 ROUGE_2:
AggregateScore(low=Score(precision=0.166422532946031, recall=0.14864999757241987, fmeasure=0.15229217101061118), mid=Score(precision=0.16833011753361882, recall=0.1504195716038362, fmeasure=0.15396139813339343), high=Score(precision=0.17058789969308358, recall=0.15232853455768658, fmeasure=0.15590467051587376))

 ROUGE_L:
AggregateScore(low=Score(precision=0.2791458979499142, recall=0.2529897770384392, fmeasure=0.25757140506162507), mid=Score(precision=0.28120564591223207, recall=0.25488090117768447, fmeasure=0.2593799995678622), high=Score(precision=0.2834430254354239, recall=0.25683467237590635, fmeasure=0.2611850779055858))

This result is worse than the reported cnn_dm results in paper. I haven't changed any parameters yet. What is wrong with my training?


[T5] add missing docstring for some configurations

Add missing docs for T5 config.


More clear error message in the use-case of #5169

Supplying `is_pretokenized=True` to an encoding method means that the sequences are given as a list of words (words being strings).

Make the error message more clear in this case.

Fix #5169


Unable to load the reformer pre-trained model, connection broken after X%

# ❓ Questions & Help

I am trying to load the Reformer pre-trained model, But I am not able to do so. The load fails after some random x%. The error stack trace is;

ChunkedEncodingError: ('Connection broken: OSError("(10054, \'WSAECONNRESET\')",)', OSError("(10054, 'WSAECONNRESET')",))

Code to load the model, a simple cell with imports and just the code below to load the model;

modelWiki =  ReformerModelWithLMHead.from_pretrained('google/reformer-enwik8')

The notebook is running behind the proxy, could that be an issue?
The SO - https://stackoverflow.com/questions/27333671/how-to-solve-the-10054-error
But it talks about connection refused by server.
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


MarianTokenizer.prepare_translation_batch uses new tokenizer API

new params:

```python
        truncation_strategy="only_first",
        padding="longest",
```
The old behavior was to pad to model_max_len then call `trim_batch`.

I also added two tests than ensure that it works.


Is it possible to mimic trim_batch using new tokenizer strategies?

Core: Tokenization

Internals of the library; Tokenization.

I am trying to replace the old workflow of
calling batch_encode_plus to make tensors of shape
`(n_examples, model_max_length)` and then calling `trim_batch` to reduce padding computation, with the new tokenizers kwargs.
Is this possible?
The following code does not seem to truncate inputs longer than 512 (the second assert breaks).

Attempt:
```python
from transformers import BartTokenizer

tokenizer = BartTokenizer.from_pretrained('facebook/bart-large')
kw = dict(max_length=512,
          pad_to_max_length=True, padding=True, return_tensors='pt', truncation='only_first')
batch = tokenizer(['tiny sentence 1', 'tiny_sentence2'],**kw)
assert batch.input_ids.shape[1] == 7, batch.input_ids.shape[1]
input_ids, mask = trim_batch(**batch, pad_token_id=tokenizer.pad_token_id)
assert input_ids.shape[1] == 7, batch.input_ids.shape[1]

batch_overflow = tokenizer(['tiny sentence 1'*1000, 'tiny_sentence2'], **kw)

assert batch_overflow.input_ids.shape[1] == 512, batch_overflow.input_ids.shape[1]
```

Traceback:
```python
assert batch_overflow.input_ids.shape[1] == 512, batch_overflow.input_ids.shape[1]

AssertionError: 3002
```

Help much appreciated, @mfuntowicz @thomwolf


Which Marian version was used to train the Helsinki-NLP/* checkpoints?

Trying to find the **Marian version used to train Marian MT transformers**. This would help me understand the **benchmarks for translation times**. I see that multiple papers mention various benchmarks for CPU translation :

- https://www.aclweb.org/anthology/D19-5632.pdf - has the benchmarks for the latest update Marian v1.9
- https://www.aclweb.org/anthology/P18-4020.pdf - does not mention the version but I presume it uses versions between  Marian v1.7 and Marian v1.5

**Marian v1.9 suggests considerably faster translation times than Marian v1.7 and Marian v1.5.**

Finding the version used for the huggingface transformers both in the huggingface and Helsinki NLP documentation hasn't been fruitful. Would be really helpful to have this answered.


Model card for t5-base-finetuned-emotion (recognition)

model card

Related to pretrained model cards


Add model cards for Microsoft's MiniLM

Related to pretrained model cards


When is 2.12 coming out?

Hi when is 2.12 coming out? Cheers.

(Interested in AutoModelForCausalLM)


object returned by  RobertaTokenizerFast() class are not serializable,

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): RobertaTokenizerFast()
Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
yes

## To reproduce

[Steps to reproduce the behavior: https://github.com/huggingface/tokenizers/issues/313#issue-642829111 ](https://github.com/huggingface/tokenizers/issues/313#issue-642829111)


Update model card for COVID-QA model

model card

Related to pretrained model cards

Specify exact dataset that was used for crossvalidation.


Add README.md (nyu-mll)

Related to pretrained model cards

Add README.md on organization nyu-mll.


Trying to made a keras model with transformer layers defined in hf-transformers, keep running into `AttributeError: Tensor.op is meaningless when eager execution is enabled, when trying to make a keras model`

# ❓ Questions & Help

## Details

I am trying to build models using huggingface transformer layers. However, I keep running into `AttributeError: Tensor.op is meaningless when eager execution is enabled.`

I don't want to disable eager execution, as I heard it interfere's with some of keras' other functionality (feel free to correct if this is wrong).

Here's the abridged code for one attempt (full code here https://colab.research.google.com/drive/1pnFDEQB4EuxNM1pSgbWJNKD2208dIIN0?usp=sharing )

```
from transformers.modeling_tf_bert import TFBertLayer

class TFBertEncoderAlter(tf.keras.layers.Layer):
    def __init__(self, config, **kwargs):
        super().__init__(**kwargs)
        self.output_hidden_states = config.output_hidden_states
        self.layer = [TFBertLayer(config, name="layer_._{}".format(i)) for i in range(config.num_hidden_layers)]

    def call(self, inputs, training=False):
        hidden_states, attention_mask, output_attentions = inputs

        all_hidden_states = ()
        all_attentions = ()
        for i, layer_module in enumerate(self.layer):
            if self.output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            layer_outputs = layer_module(
                [hidden_states, attention_mask, output_attentions], training=training
            )
            hidden_states = layer_outputs[0]

            if cast_bool_to_primitive(output_attentions) is True:
                all_attentions = all_attentions + (layer_outputs[1],)

        # Add last layer
        if self.output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        outputs = (hidden_states,)
        if self.output_hidden_states:
            outputs = outputs + (all_hidden_states,)
        if cast_bool_to_primitive(output_attentions) is True:
            outputs = outputs + (all_attentions,)
        return outputs  # outputs, (hidden states), (attentions)

P_trans11 = TFBertEncoderAlter(config, name='Encoder')

inputHiddenVals = tf.keras.Input(shape=[None, None], dtype=tf.float32, name='input_Q',
                                batch_size=None)

P_outputs = P_trans11((outt, None, None))
modelNew = tf.keras.Model(inputHiddenVals,P_outputs)

```

Here is the output

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-35-8cd9393cb573> in <module>()
      5
      6 P_outputs = P_trans11((outt, None, None))
----> 7 modelNew = tf.keras.Model(inputHiddenVals,P_outputs)

6 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in __init__(self, *args, **kwargs)
    165
    166   def __init__(self, *args, **kwargs):
--> 167     super(Model, self).__init__(*args, **kwargs)
    168     _keras_api_gauge.get_cell('model').set(True)
    169     # Model must be created under scope of DistStrat it will be trained with.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in __init__(self, *args, **kwargs)
    171         'inputs' in kwargs and 'outputs' in kwargs):
    172       # Graph network
--> 173       self._init_graph_network(*args, **kwargs)
    174     else:
    175       # Subclassed network

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    454     self._self_setattr_tracking = False  # pylint: disable=protected-access
    455     try:
--> 456       result = method(self, *args, **kwargs)
    457     finally:
    458       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs)
    252
    253     if any(not hasattr(tensor, '_keras_history') for tensor in self.outputs):
--> 254       base_layer_utils.create_keras_history(self._nested_outputs)
    255
    256     self._base_init(name=name, **kwargs)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py in create_keras_history(tensors)
    184     keras_tensors: The Tensors found that came from a Keras Layer.
    185   """
--> 186   _, created_layers = _create_keras_history_helper(tensors, set(), [])
    187   return created_layers
    188

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    210     if getattr(tensor, '_keras_history', None) is not None:
    211       continue
--> 212     op = tensor.op  # The Op that created this Tensor.
    213     if op not in processed_ops:
    214       if op.type.startswith('Sparse'):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in op(self)
   1111   def op(self):
   1112     raise AttributeError(
-> 1113         "Tensor.op is meaningless when eager execution is enabled.")
   1114
   1115   @property

AttributeError: Tensor.op is meaningless when eager execution is enabled.
```

Here is a different approach ( full code here https://colab.research.google.com/drive/1bieigPh98l9POzT3Tdz8DWDN_nh18YG1?usp=sharing )

```
from transformers.modeling_tf_bert import TFBertEncoder, TFBertMainLayer, TFBertLayer
from transformers.modeling_tf_utils import (
    get_initializer,
    keras_serializable,
    shape_list,
)
from transformers.configuration_bert import BertConfig

class TFBertEncoder0(tf.keras.layers.Layer):
    def __init__(self, config, **kwargs):
        super().__init__(**kwargs)
        self.output_attentions = config.output_attentions
        self.output_hidden_states = config.output_hidden_states
        self.layer = [TFBertLayer(config, name="layer_._{}".format(i)) for i in range(config.num_hidden_layers)]

    def call(self, inputs, training=False):
        hidden_states, attention_mask, head_mask = inputs

        all_hidden_states = ()
        all_attentions = ()
        for i, layer_module in enumerate(self.layer):
            if self.output_hidden_states:
                all_hidden_states = all_hidden_states + (hidden_states,)

            layer_outputs = layer_module([hidden_states, attention_mask, head_mask], training=training)
            hidden_states = layer_outputs[0]

            if self.output_attentions:
                all_attentions = all_attentions + (layer_outputs[1],)

        # Add last layer
        if self.output_hidden_states:
            all_hidden_states = all_hidden_states + (hidden_states,)

        outputs = (hidden_states,)
        if self.output_hidden_states:
            outputs = outputs + (all_hidden_states,)
        if self.output_attentions:
            outputs = outputs + (all_attentions,)
        return outputs  # outputs, (hidden states), (attentions)

@keras_serializable
class TFBertMainLayerAlter4(tf.keras.layers.Layer):
    config_class = BertConfig

    def __init__(self, config, **kwargs):
        super().__init__(**kwargs)
        self.num_hidden_layers = config.num_hidden_layers
        self.initializer_range = config.initializer_range
        self.output_attentions = config.output_attentions

        self.encoder = TFBertEncoder0(config, name="encoder")

    def _prune_heads(self, heads_to_prune):
        """ Prunes heads of the model.
            heads_to_prune: dict of {layer_num: list of heads to prune in this layer}
            See base class PreTrainedModel
        """
        raise NotImplementedError

    def call(
        self,
        inputs,
        training=False,
    ):

        encoder_outputs = self.encoder(
            [inputs, None, None], training=training
        )

        sequence_output = encoder_outputs[0]

        outputs = (sequence_output,) + encoder_outputs[
            1:
        ]  # add hidden_states and attentions if they are here
        return outputs  # sequence_output, pooled_output, (hidden_states), (attentions)

P_trans11 = TFBertMainLayerAlter4(config3, name="roberta")

inputHiddenVals = tf.keras.Input(shape=[None, None], dtype=tf.float32, name='input_Q',
                                batch_size=None)

P_outputs = P_trans11(outt)
modelNew = tf.keras.Model(inputHiddenVals,P_outputs)
```

Again, same result

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-43-79b3b73e5f5f> in <module>()
      5
      6 P_outputs = P_trans11(outt)
----> 7 modelNew = tf.keras.Model(inputHiddenVals,P_outputs)

6 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py in __init__(self, *args, **kwargs)
    165
    166   def __init__(self, *args, **kwargs):
--> 167     super(Model, self).__init__(*args, **kwargs)
    168     _keras_api_gauge.get_cell('model').set(True)
    169     # Model must be created under scope of DistStrat it will be trained with.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in __init__(self, *args, **kwargs)
    171         'inputs' in kwargs and 'outputs' in kwargs):
    172       # Graph network
--> 173       self._init_graph_network(*args, **kwargs)
    174     else:
    175       # Subclassed network

/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    454     self._self_setattr_tracking = False  # pylint: disable=protected-access
    455     try:
--> 456       result = method(self, *args, **kwargs)
    457     finally:
    458       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/network.py in _init_graph_network(self, inputs, outputs, name, **kwargs)
    252
    253     if any(not hasattr(tensor, '_keras_history') for tensor in self.outputs):
--> 254       base_layer_utils.create_keras_history(self._nested_outputs)
    255
    256     self._base_init(name=name, **kwargs)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py in create_keras_history(tensors)
    184     keras_tensors: The Tensors found that came from a Keras Layer.
    185   """
--> 186   _, created_layers = _create_keras_history_helper(tensors, set(), [])
    187   return created_layers
    188

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py in _create_keras_history_helper(tensors, processed_ops, created_layers)
    210     if getattr(tensor, '_keras_history', None) is not None:
    211       continue
--> 212     op = tensor.op  # The Op that created this Tensor.
    213     if op not in processed_ops:
    214       if op.type.startswith('Sparse'):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in op(self)
   1111   def op(self):
   1112     raise AttributeError(
-> 1113         "Tensor.op is meaningless when eager execution is enabled.")
   1114
   1115   @property

AttributeError: Tensor.op is meaningless when eager execution is enabled.
```

Here's another attempt (full code here https://colab.research.google.com/drive/1UVJ7XSx0vXpgApe6E7ECVb9LNSJN_D9-?usp=sharing )

```
from transformers.modeling_tf_bert import TFBertLayer

l1 = TFBertLayer(config)
l2 = TFBertLayer(config)

inputHiddenVals = tf.keras.Input(shape=[None, None], dtype=tf.float32, name='input_Q',
                                batch_size=None)

P_outputs = l1((outt, None, None))[0]
P_outputs2 = l2((outt, None, None))[0]

modelNew = tf.keras.Model(inputHiddenVals,P_outputs2)
```

Again, same results.

**A link to original question on Stack Overflow**:

https://stackoverflow.com/questions/62504903/attributeerror-tensor-op-is-meaningless-when-eager-execution-is-enabled-when-t


Load a T5ForConditionalGeneration's encoder into a T5Model

Hi,
I know that T5ForConditionalGeneration is a T5Model with decoding. I've got a T5ForConditionalGeneration and that I've fine-tuned on a seq2seq task and now I want to use its T5 encoder to initialize the parameters in a T5Model( to further train it on some other task.) I read the code and I didn't understand what should I do. Can you please help me?


Fixing docs for Encoder Decoder Config

This is a small contribution to the EncoderDecoderConfig documentation.
While trying to use this class, I noticed the documentation said that we could pass 2 "encoder" parameters to the class constructor.
I am almost 99.99% certain it meant we could pass an encoder and a decoder parameter to the class constructor.