Model utils doc

This PR documents all the model utilities and fixes a few things in the `PreTrainedModel`/`TFPreTrainedModel`.

It introduces a new category I called "internal" that comes at the very end of the package documentation, which is aimed at documenting all our public internal methods (e.g., public in their modules, but not in the project init). This should be useful for people who tweak our code and reuse those pieces.

Preview of the new file in the docs is [here](https://64241-155220641-gh.circle-artifacts.com/0/docs/_build/html/internal/modeling_utils.html) (and you can navigate on the other pages if you want to check the smaller changes).


CI/Examples: ModuleNotFoundError: No module named '_sqlite3'

This is a common issue,  all solutions I have found to address involve recompiling python.

like [1](https://stackoverflow.com/questions/1210664/no-module-named-sqlite3) [2](https://www.virtualizationteam.com/cloud/running-vcd-cli-fail-with-the-following-error-modulenotfounderror-no-module-named-_sqlite3.html)

Full Traceback:
```
2020-07-23T00:30:23.1433578Z ==================================== ERRORS ====================================
2020-07-23T00:30:23.1435598Z ____________ ERROR collecting examples/seq2seq/test_bash_script.py _____________
2020-07-23T00:30:23.1437122Z ImportError while importing test module '/home/hf/actions-runner_transformers/_work/transformers/transformers/examples/seq2seq/test_bash_script.py'.
2020-07-23T00:30:23.1437481Z Hint: make sure your test modules/packages have valid Python names.
2020-07-23T00:30:23.1437737Z Traceback:
2020-07-23T00:30:23.1438102Z examples/seq2seq/finetune.py:21: in <module>
2020-07-23T00:30:23.1438342Z     from .utils import (
2020-07-23T00:30:23.1438573Z examples/seq2seq/utils.py:14: in <module>
2020-07-23T00:30:23.1438821Z     from rouge_score import rouge_scorer, scoring
2020-07-23T00:30:23.1439560Z .env/lib/python3.7/site-packages/rouge_score/rouge_scorer.py:41: in <module>
2020-07-23T00:30:23.1440092Z     from nltk.stem import porter
2020-07-23T00:30:23.1440789Z .env/lib/python3.7/site-packages/nltk/__init__.py:149: in <module>
2020-07-23T00:30:23.1441053Z     from nltk.translate import *
2020-07-23T00:30:23.1441694Z .env/lib/python3.7/site-packages/nltk/translate/__init__.py:23: in <module>
2020-07-23T00:30:23.1441956Z     from nltk.translate.meteor_score import meteor_score as meteor
2020-07-23T00:30:23.1442637Z .env/lib/python3.7/site-packages/nltk/translate/meteor_score.py:10: in <module>
2020-07-23T00:30:23.1442916Z     from nltk.stem.porter import PorterStemmer
2020-07-23T00:30:23.1443523Z .env/lib/python3.7/site-packages/nltk/stem/__init__.py:29: in <module>
2020-07-23T00:30:23.1443794Z     from nltk.stem.snowball import SnowballStemmer
2020-07-23T00:30:23.1444468Z .env/lib/python3.7/site-packages/nltk/stem/snowball.py:29: in <module>
2020-07-23T00:30:23.1444741Z     from nltk.corpus import stopwords
2020-07-23T00:30:23.1445379Z .env/lib/python3.7/site-packages/nltk/corpus/__init__.py:66: in <module>
2020-07-23T00:30:23.1445750Z     from nltk.corpus.reader import *
2020-07-23T00:30:23.1446438Z .env/lib/python3.7/site-packages/nltk/corpus/reader/__init__.py:105: in <module>
2020-07-23T00:30:23.1446711Z     from nltk.corpus.reader.panlex_lite import *
2020-07-23T00:30:23.1447355Z .env/lib/python3.7/site-packages/nltk/corpus/reader/panlex_lite.py:15: in <module>
2020-07-23T00:30:23.1447591Z     import sqlite3
2020-07-23T00:30:23.1447839Z /home/hf/.pyenv/versions/3.7.6/lib/python3.7/sqlite3/__init__.py:23: in <module>
2020-07-23T00:30:23.1448311Z     from sqlite3.dbapi2 import *
2020-07-23T00:30:23.1448584Z /home/hf/.pyenv/versions/3.7.6/lib/python3.7/sqlite3/dbapi2.py:27: in <module>
2020-07-23T00:30:23.1448845Z     from _sqlite3 import *
2020-07-23T00:30:23.1449485Z E   ModuleNotFoundError: No module named '_sqlite3'
2020-07-23T00:30:23.1449638Z

```


MBART: support summarization tasks where max_src_len > max_tgt_len

Previously, MBartTokenizer.prepare_translation_batch always truncated `src_texts` and `tgt_texts` to the same length.
This PR exposes a `max_target_length` kwarg, which, if specified, will control truncation of target_texts.


Deebert Examples test failure

`DeeBertTests.test_glue_deebert`
Excerpt from [CI](https://pipelines.actions.githubusercontent.com/SFFqAjp6ciVZiZmfZfjie9y9Q96dfpUE8sJvWAtTDWoFlixGkf/_apis/pipelines/1/runs/4783/signedlogcontent/3?urlExpires=2020-07-23T20%3A02%3A20.7871420Z&urlSigningMethod=HMACV1&urlSignature=XL%2Fj%2F2NSBD3Q174ifxC4jTOQUloQMx3UFZ%2ByaM%2Beomk%3D)

```python
2020-07-23T00:30:23.1537872Z         with patch.object(sys, "argv", train_args):
2020-07-23T00:30:23.1538109Z             result = run_glue_deebert.main()
2020-07-23T00:30:23.1538324Z             for value in result.values():
2020-07-23T00:30:23.1538573Z >               self.assertGreaterEqual(value, 0.75)
2020-07-23T00:30:23.1538856Z E               AssertionError: 0.6666666666666666 not greater than or equal to 0.75
2020-07-23T00:30:23.1539030Z
```


MbartDataset can support Summarization

need to allow max_source_length != max_seq_length


Bert german dbmdz uncased sentence stsb

model card

Related to pretrained model cards

Model Card for bert-german-dbmdz-uncased-sentence-stsb


Fix #5974

Small bug introduced by the model outputs PR.


MbartTokenizer: do not hardcode vocab size

language codes should start at the end of the standard vocab. If the standard vocab is smaller, this number is not 250,001.


bug in trainer.py line297

# 🐛 Bug

## Information

I found this bug while running example/token_classification/run_ner.py.
I fixed that by delete the 'self.' prefix.

## Reproduce
 just execute `run_ner.py`


T5 pre training on different languages from scratch

Hi Team,

I was exploring the pre training procedure/document for T5 models(small,base, large) models on different languages from scratch. However did not come across anything which could help me. Please share if there is any resource for the same. If not, could you please consider to include it .

Thank you .


[WIP] Trainer supports all Datasets as train_dataset, with/without __len__ #5990

A first PR.

Passed:
- `make test`
- `make style`
- `make quality`

Modifies:
- trainer.py: fixes issue
- test_trainer.py: calls `Trainer.train()`

It fixes only the case where the TRAINING dataset has not the `__len__` method.

The distinction is not between `Dataset` and `IterableDataset`, but between objects that are instances of class where `__len__` is implemented or not. This is pointed in [pytorch source](https://pytorch.org/docs/stable/_modules/torch/utils/data/dataset.html#Dataset), implementation of `__len__` is up to the user.
The test is therefore: `isinstance(dataset, collections.Sized)`

NB: fixing for EVAL and TEST dataset will require more code refactor: all get funneled to `Trainer._prediction_loop()` without keeping track of whether it is EVAL or TEST, which makes the usage of `TrainingArguments.eval_steps` impossible to assume. (not to mention, there is no `test_steps` field in `TrainingArguments`)

Not all datasets have an implementation of `__len__` method, therefore the trainer should not assume it is available.

The use case is:
- Dataset with `__len__`: use `num_train_epochs` or `max_steps` to specify how long should training run
- Dataset without `__len__`: use only `max_steps`

The limitation is still valid on the EVAL / TEST dataset who still has to implement `__len__`


[examples (seq2seq)] fix preparing decoder_input_ids for T5

possible fix for #5987
`</s>` should be added at the end of target text

@sshleifer


# 🚀 Feature request

Store Predictions on CPU in Every Prediction Iteration (Trainer)

## Motivation

Currently, in  [`Trainer._prediction_loop`,](https://github.com/huggingface/transformers/blob/33d7506ea10ca92886fd1bb3b5306a1a720c58fe/src/transformers/trainer.py#L785) the predictions (logits) of the model is stored on GPU/TPU in each iteration. After all iterations are finished, they will be concatenated together and sent to CPU. In this way, the GPU/TPU memory usage will increase linearly during prediction. If the test set is very large, there may be insufficient GPU/TPU memory to finish the prediction phase. To save GPU/TPU memory and allow larger-scale inference, the trainer should instead send a batch of prediction to CPU after each iteration.

However, if we do inference on multiple devices, we would need a `distributed_concat` function to aggregate all predictions from all devices. If the predictions are already stored on CPU, we can no longer aggregate them using NCCL. Therefore, this issue is unsolvable unless we introduce another CPU-based distributed communication mechanism. Still, we can at least solve it in a single-GPU scenario.

## Your contribution

If approved, I can write codes to solve this problem when the inference is not running in the distributed mode.


ONNX documentation

- Rename the current **torchscript.rst** to **serialization.rst**
- Move torchscript documentation into a subsection of the above **serialization.rst**
- Introduce documentation for ONNX/ONNXRuntime in the above section

Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>


T5 Tensorflow: _shift_right returns wrong result

# 🐛 Bug

## Information

The [_shift_right](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_t5.py#L777) method of the `TFT5PreTrainedModel` returns all zeros instead of shifting the `input_ids`.

## To reproduce

```
import tensorflow as tf

def shape_list(x):
    """Deal with dynamic shape in tensorflow cleanly."""
    static = x.shape.as_list()
    dynamic = tf.shape(x)
    return [dynamic[i] if s is None else s for i, s in enumerate(static)]

def _shift_right(input_ids):
    decoder_start_token_id = 0
    pad_token_id = 0

    assert (
        decoder_start_token_id is not None
    ), "self.model.config.decoder_start_token_id has to be defined. In TF T5 it is usually set to the pad_token_id. See T5 docs for more information"

    # shift inputs to the right
    shifted_input_ids = tf.zeros_like(input_ids, dtype=tf.int32)
    shifted_input_ids = tf.roll(shifted_input_ids, 1, axis=-1)
    start_tokens = tf.fill((shape_list(shifted_input_ids)[0], 1), decoder_start_token_id)
    shifted_input_ids = tf.concat([start_tokens, shifted_input_ids[:, 1:]], -1)

    assert pad_token_id is not None, "self.model.config.pad_token_id has to be defined."
    # replace possible -100 values in labels by `pad_token_id`
    shifted_input_ids = tf.where(
        shifted_input_ids == -100, tf.fill(shape_list(shifted_input_ids), pad_token_id), shifted_input_ids
    )

    assert tf.math.reduce_any(
        shifted_input_ids >= 0
    ).numpy(), "Verify that `labels` has only positive values and -100"

    return shifted_input_ids

input_ids = tf.convert_to_tensor([[32000, 1, 2, 3, 0, 0, 0]])
print(_shift_right(input_ids))
```

## Expected behavior

Should shift the tensor to the right.

## Suggested solution

Replace line `shifted_input_ids = tf.zeros_like(input_ids, dtype=tf.int32)` with `shifted_input_ids = tf.cast(input_ids, tf.int32)`.
Further I'd recommend removing the assertion for positive label values, as it depends on the `numpy()` method, which is not available in some cases (e.g. when using datasets loaded from tfrecord files) and will throw an error then.
Shall I open a PR directly?

## Environment info
- `transformers` version: Master (file commit `4dc6559`)
- Tensorflow version: 2.1.0


Trainer: exception raised when calling len() on IterableDataset

# 🐛 Bug

## Information
While pre-training a Longformer model from scratch, the text is delivered through an `IterableDataset` object. The code which is called by `Trainer.train()` still calls `len()` on this object, which raises an exception.
#5829 addressed the proper creation of the Dataloader.

The problem arises when using:
* [x] my own modified scripts: see code

The tasks I am working on is:
* [x] my own task or dataset: pre-train a LM from scratch

## To reproduce
Here is my entire code, but it can be reproduced with any `PreTrainedModel` by using an `IterableDataset`.

```python
import logging
import random

from dataclasses import dataclass, field

from transformers import LongformerConfig, LongformerForMaskedLM, LongformerTokenizerFast
from transformers import Trainer, TrainingArguments
from transformers import TextDataset, DataCollatorForLanguageModeling
from transformers import HfArgumentParser

from sklearn.model_selection import train_test_split

from pathlib import Path

from utils_pretrain import MultiTextDataset

logger = logging.getLogger(__name__)

@dataclass
class ModelArguments:
    """
    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
    """

    max_seq_len: int = field(
        metadata={"help": "Input Sequence Length"}
    )
    num_hidden_layers: int = field(
        metadata={'help': 'Number of transformer layers in Longformer'}
    )
    tok_dir: str = field(
        metadata={
            'help': 'Folder with tokenizer files'
        }
    )
    txt_dir: str = field(
        metadata={"help": "Folder with txt files for tokenizer training"}
    )
    filter_files: str = field(
        default='[a-c]*.txt',
        metadata={"help": "regex to select specific files"}
    )
    test_size: float = field(
        default=0.05,
        metadata={'help': 'proportion of the data that will be used for evaluation'}
    )

def main():
    parser = HfArgumentParser((ModelArguments, TrainingArguments))
    model_args, train_args = parser.parse_args_into_dataclasses()
    model_args: ModelArguments

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        level=logging.WARN,
    )
    logger.warning(
        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, 16-bits training: %s",
        train_args.local_rank,
        train_args.device,
        train_args.n_gpu,
        bool(train_args.local_rank != -1),
        train_args.fp16,
    )
    logger.info("Training/evaluation parameters %s", train_args)

    MODEL_NAME = 'allenai/longformer-base-4096'
    tokenizer: LongformerTokenizerFast = LongformerTokenizerFast.from_pretrained(model_args.tok_dir)

    # Customize an existing config rather than create from scratch
    config: LongformerConfig = LongformerConfig.from_pretrained(MODEL_NAME)
    config.max_position_embeddings = model_args.max_seq_len + 2
    config.num_hidden_layers = model_args.num_hidden_layers
    config.attention_window = [512] * model_args.num_hidden_layers
    config.vocab_size = tokenizer.vocab_size
    model = LongformerForMaskedLM(config)

    data_files = list(Path(model_args.txt_dir).glob(model_args.filter_files))
    shuffled_files = random.sample(data_files, len(data_files))
    train_files, val_files = train_test_split(shuffled_files, test_size=model_args.test_size)

    train_ds, val_ds = list(
        map(
            lambda x: MultiTextDataset(
                files=x,
                tokenizer=tokenizer,
                block_size=model_args.max_seq_len
            ),
            [train_files, val_files]
        )
    )

    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm=True,
        mlm_probability=0.15
    )

    train_args: TrainingArguments
    train_args.do_train = True
    train_args.evaluate_during_training = True

    trainer = Trainer(
        model=model,
        args=train_args,
        data_collator=data_collator,
        train_dataset=train_ds,
        eval_dataset=val_ds,
    )

    trainer.train(train_args.output_dir)
```

The class `MultiTextDataset` inherits `IterableDataset`. It has no `__len__` method, and the length would require the whole dataset to be parsed at once to be known.

Here is the exception and stack trace:
```
Traceback (most recent call last):
  File "longformer_pretrain.py", line 131, in <module>
    main()
  File "longformer_pretrain.py", line 122, in main
    trainer.train(train_args.output_dir)
  File "/home/jrossi/anaconda3/envs/COLIEE/lib/python3.7/site-packages/transformers/trainer.py", line 392, in train
    self.args.max_steps // (len(train_dataloader) // self.args.gradient_accumulation_steps) + 1
  File "/home/jrossi/anaconda3/envs/COLIEE/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 313, in __len__
    length = self._IterableDataset_len_called = len(self.dataset)
TypeError: object of type 'MultiTextDataset' has no len()
```

## Expected behavior

The call to `Trainer.train()` starts the training. A case has to be made in the code to accomodate the usage of `IterableDataset`, which means not assuming that `len()` can be called on the dataset at any point.

- If a number of epochs is given, one epoch corresponds to consuming the iterable dataset until StopIteration
- If a number of steps is given, training stops after performing MAX_STEPS or catching a StopIteration, whichever comes first
- During training, the progress bar should be either a % of epochs performed, or a % of steps performed
- (optional) If a number of epochs is given, register how many steps it took to consume the iterator so a better progress bar can be shown for the next epochs (each epoch will consume the same iterator once)

With regards to [Pytorch documentation](https://pytorch.org/docs/stable/data.html#), there is no certainty that `__len__` method will be implemented, even on `Dataset` objects.
The distinction should be made between objects that implement `__len__` and those that do not implement it.
The current code __assumes__ that the `Dataset` objects given when creating a `Trainer` implement `len()`, but there is no guarantee of this.

```python
import collections
if isinstance(bar, collections.Sized): (...)
```

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.7.8-1.el7.elrepo.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: YES
- Using distributed or parallel set-up in script?: NO (for the moment)

## Fix
I can contribute. I will suggest a PR to fix this.


Create README.md

Related to pretrained model cards

Hello, we are making this pull request to add model card for our Italian sentiment political model. Thanks!


EncoderDecoderModel:  weight can not be init from the checkpoint

I try to use EncoderDecoderModel to train a Chinese summary model.

```Python
from transformers import BertConfig, EncoderDecoderConfig, EncoderDecoderModel

encoder_config = BertConfig.from_pretrained('bert-base-chinese')
decoder_config = BertConfig.from_pretrained('bert-base-chinese', is_decoder=True)

encoder_config.max_length = 512
decoder_config.max_length = 128

model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-chinese', 'bert-base-chinese',
                                                            encoder_config=encoder_config,
                                                            decoder_config=decoder_config)
```

However, I get a warning, the whole encoder model doesn't init from checkpoint:
```
WARNING:transformers.modeling_utils:Some weights of the model checkpoint at bert-base-chinese were not used when initializing BertLMHeadModel: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertLMHeadModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertLMHeadModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
WARNING:transformers.modeling_utils:Some weights of BertLMHeadModel were not initialized from the model checkpoint at bert-base-chinese and are newly initialized: ['bert.encoder.layer.0.crossattention.self.query.weight', 'bert.encoder.layer.0.crossattention.self.query.bias',
 'bert.encoder.layer.0.crossattention.self.key.weight', 'bert.encoder.layer.0.crossattention.self.key.bias',
'bert.encoder.layer.0.crossattention.self.value.weight', 'bert.encoder.layer.0.crossattention.self.value.bias', 'bert.encoder.layer.0.crossattention.output.dense.weight', 'bert.encoder.layer.0.crossattention.output.dense.bias', 'bert.encoder.layer.0.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.0.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.1.crossattention.self.query.weight', 'bert.encoder.layer.1.crossattention.self.query.bias', 'bert.encoder.layer.1.crossattention.self.key.weight', 'bert.encoder.layer.1.crossattention.self.key.bias', 'bert.encoder.layer.1.crossattention.self.value.weight', 'bert.encoder.layer.1.crossattention.self.value.bias', 'bert.encoder.layer.1.crossattention.output.dense.weight', 'bert.encoder.layer.1.crossattention.output.dense.bias', 'bert.encoder.layer.1.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.1.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.2.crossattention.self.query.weight', 'bert.encoder.layer.2.crossattention.self.query.bias', 'bert.encoder.layer.2.crossattention.self.key.weight', 'bert.encoder.layer.2.crossattention.self.key.bias', 'bert.encoder.layer.2.crossattention.self.value.weight', 'bert.encoder.layer.2.crossattention.self.value.bias', 'bert.encoder.layer.2.crossattention.output.dense.weight', 'bert.encoder.layer.2.crossattention.output.dense.bias', 'bert.encoder.layer.2.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.2.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.3.crossattention.self.query.weight', 'bert.encoder.layer.3.crossattention.self.query.bias', 'bert.encoder.layer.3.crossattention.self.key.weight', 'bert.encoder.layer.3.crossattention.self.key.bias', 'bert.encoder.layer.3.crossattention.self.value.weight', 'bert.encoder.layer.3.crossattention.self.value.bias', 'bert.encoder.layer.3.crossattention.output.dense.weight', 'bert.encoder.layer.3.crossattention.output.dense.bias', 'bert.encoder.layer.3.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.3.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.4.crossattention.self.query.weight', 'bert.encoder.layer.4.crossattention.self.query.bias', 'bert.encoder.layer.4.crossattention.self.key.weight', 'bert.encoder.layer.4.crossattention.self.key.bias', 'bert.encoder.layer.4.crossattention.self.value.weight', 'bert.encoder.layer.4.crossattention.self.value.bias', 'bert.encoder.layer.4.crossattention.output.dense.weight', 'bert.encoder.layer.4.crossattention.output.dense.bias', 'bert.encoder.layer.4.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.4.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.5.crossattention.self.query.weight', 'bert.encoder.layer.5.crossattention.self.query.bias', 'bert.encoder.layer.5.crossattention.self.key.weight', 'bert.encoder.layer.5.crossattention.self.key.bias', 'bert.encoder.layer.5.crossattention.self.value.weight', 'bert.encoder.layer.5.crossattention.self.value.bias', 'bert.encoder.layer.5.crossattention.output.dense.weight', 'bert.encoder.layer.5.crossattention.output.dense.bias', 'bert.encoder.layer.5.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.5.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.6.crossattention.self.query.weight', 'bert.encoder.layer.6.crossattention.self.query.bias', 'bert.encoder.layer.6.crossattention.self.key.weight', 'bert.encoder.layer.6.crossattention.self.key.bias', 'bert.encoder.layer.6.crossattention.self.value.weight', 'bert.encoder.layer.6.crossattention.self.value.bias', 'bert.encoder.layer.6.crossattention.output.dense.weight', 'bert.encoder.layer.6.crossattention.output.dense.bias', 'bert.encoder.layer.6.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.6.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.7.crossattention.self.query.weight', 'bert.encoder.layer.7.crossattention.self.query.bias', 'bert.encoder.layer.7.crossattention.self.key.weight', 'bert.encoder.layer.7.crossattention.self.key.bias', 'bert.encoder.layer.7.crossattention.self.value.weight', 'bert.encoder.layer.7.crossattention.self.value.bias', 'bert.encoder.layer.7.crossattention.output.dense.weight', 'bert.encoder.layer.7.crossattention.output.dense.bias', 'bert.encoder.layer.7.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.7.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.8.crossattention.self.query.weight', 'bert.encoder.layer.8.crossattention.self.query.bias', 'bert.encoder.layer.8.crossattention.self.key.weight', 'bert.encoder.layer.8.crossattention.self.key.bias', 'bert.encoder.layer.8.crossattention.self.value.weight', 'bert.encoder.layer.8.crossattention.self.value.bias', 'bert.encoder.layer.8.crossattention.output.dense.weight', 'bert.encoder.layer.8.crossattention.output.dense.bias', 'bert.encoder.layer.8.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.8.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.9.crossattention.self.query.weight', 'bert.encoder.layer.9.crossattention.self.query.bias', 'bert.encoder.layer.9.crossattention.self.key.weight', 'bert.encoder.layer.9.crossattention.self.key.bias', 'bert.encoder.layer.9.crossattention.self.value.weight', 'bert.encoder.layer.9.crossattention.self.value.bias', 'bert.encoder.layer.9.crossattention.output.dense.weight', 'bert.encoder.layer.9.crossattention.output.dense.bias', 'bert.encoder.layer.9.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.9.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.10.crossattention.self.query.weight', 'bert.encoder.layer.10.crossattention.self.query.bias', 'bert.encoder.layer.10.crossattention.self.key.weight', 'bert.encoder.layer.10.crossattention.self.key.bias', 'bert.encoder.layer.10.crossattention.self.value.weight', 'bert.encoder.layer.10.crossattention.self.value.bias', 'bert.encoder.layer.10.crossattention.output.dense.weight', 'bert.encoder.layer.10.crossattention.output.dense.bias', 'bert.encoder.layer.10.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.10.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.11.crossattention.self.query.weight', 'bert.encoder.layer.11.crossattention.self.query.bias', 'bert.encoder.layer.11.crossattention.self.key.weight', 'bert.encoder.layer.11.crossattention.self.key.bias', 'bert.encoder.layer.11.crossattention.self.value.weight', 'bert.encoder.layer.11.crossattention.self.value.bias', 'bert.encoder.layer.11.crossattention.output.dense.weight', 'bert.encoder.layer.11.crossattention.output.dense.bias', 'bert.encoder.layer.11.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.11.crossattention.output.LayerNorm.bias', 'cls.predictions.decoder.bias']
```

So, how to fix this warning


Possible bug in preparing deocder_input_ids for T5 in seq2seq finetune.y

In finetune.py in `_step` method ([link](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py#L131)) `decoder_input_ids` are prepared like this
```python3
source_ids, source_mask, target_ids = batch["input_ids"], batch["attention_mask"], batch["decoder_input_ids"]
decoder_input_ids = target_ids[:, :-1].contiguous()  # Why this line?
lm_labels = target_ids[:, 1:].clone()  # why clone?
```

The `T5ForConditionalGeneration` automatically prepares `decoder_input_ids` using `lables` when they are not passed. It uses the _shift_right [method](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_t5.py#L620).
```python3
decoder_start_token_id = self.config.decoder_start_token_id
pad_token_id = self.config.pad_token_id
# shift inputs to the right
shifted_input_ids = input_ids.new_zeros(input_ids.shape)
shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()
shifted_input_ids[..., 0] = decoder_start_token_id
```

so finetune.py doesn't add decoder start token id as required by T5 model. This works for bart because for bart input starts with `bos` (`<s>`) which the tokenizer adds automatically. So for T5 model this removes the first token from `lm_labels` and adds no start token in `decoder_input_ids`.

### To reproduce
```python3
from transformers import T5ForConditionalGeneration, T5Tokenizer, T5Config
tokenizer = T5Tokenizer.from_pretrained("t5-base")
config = T5Config.from_pretrained("t5-base")

batch = tokenizer(["simple is better than complex </s>"], return_tensors="pt")

# from finetune.py
pad_token_id = tokenizer.pad_token_id
target_ids = batch["input_ids"]
decoder_input_ids = target_ids[:, :-1].contiguous()  # Why this line?
lm_labels = target_ids[:, 1:].clone()  # why clone?

print(decoder_input_ids[0])
# => tensor([ 650,   19,  394,  145, 1561])
print(tokenizer.convert_ids_to_tokens(decoder_input_ids[0]))
# => ['▁simple', '▁is', '▁better', '▁than', '▁complex']
print(tokenizer.decode(lm_labels[0]))
# => is better than complex

# from T5PreTrainedModel._shift_right
decoder_start_token_id = config.decoder_start_token_id
input_ids = batch["input_ids"]
shifted_input_ids = input_ids.new_zeros(input_ids.shape)
shifted_input_ids[..., 1:] = input_ids[..., :-1].clone()
shifted_input_ids[..., 0] = decoder_start_token_id

print(shifted_input_ids[0])
# => tensor([   0,  650,   19,  394,  145, 1561])
print(tokenizer.convert_ids_to_tokens(shifted_input_ids[0]))
# => ['<pad>', '▁simple', '▁is', '▁better', '▁than', '▁complex']
print(tokenizer.decode(input_ids[0]))
# => simple is better than complex
```
@sshleifer


how can I download T5-11B pretrained model?

It gives me the following error.

**OSError: Can't load weights for 't5-11b'. Make sure that:

- 't5-11b' is a correct model identifier listed on 'https://huggingface.co/models'

- or 't5-11b' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.**


Update doc of the model page

Fixes docstring to conform to sphinx syntax and rephrase when needed. Also fixed bad copy-pastes from PyTorch to TF.

Preview is [here](https://64023-155220641-gh.circle-artifacts.com/0/docs/_build/html/main_classes/model.html).


Albert pre-train from scratch convergence problem

# 🐛 Bug
Albert pre-train convergence problem
- The model training loss converged at 6.6 when using AlbertForMaskedLM as model class
- negative training loss when using AlbertForPretrain as model class

notice: I was deliberately set the eval dataset the same as training set for checking training loss at last run.
## Information
Using AlbertForMaskedLM as model class, figure showed below:
![image](https://user-images.githubusercontent.com/23205976/88234831-7ca56f00-cc2e-11ea-8ab7-75ce3e841b88.png)

Using AlbertForPretrain as model class, figure showed below:
![image](https://user-images.githubusercontent.com/23205976/88230876-0867cd00-cc28-11ea-90aa-a7412dd9c0c6.png)

Besides, when I was using the official `run_lanugage_modeling.py`, the training loss on wikiText-2 is also not converge to 0, it converged at 6.6 for several epochs.

Model I am using (Bert, XLNet ...):
Albert
Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official task: wikiText-2
* [ ] my own task or dataset: (give details below)

## To reproduce
```
from transformers import (
    AlbertConfig,
    AlbertTokenizer,
    BertTokenizer,
    AlbertForPreTraining,
    AlbertForMaskedLM,
    LineByLineTextDataset,
    TextDataset,
    DataCollatorForLanguageModeling,
    Trainer,
    TrainingArguments
)
import math

albert_base_configuration = AlbertConfig(
    hidden_size=768,
    num_attention_heads=12,
    intermediate_size=3072,
)

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
# model = AlbertForPreTraining(config=albert_base_configuration)
model = AlbertForMaskedLM(config=albert_base_configuration)
train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="/home/ubuntu/data_local/wikitext-2-raw/wiki.train.raw",
    block_size=512,
)
eval_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="/home/ubuntu/data_local/wikitext-2-raw/wiki.test.raw",
    block_size=512,
)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

training_args = TrainingArguments(
    output_dir="./results/new",
    overwrite_output_dir=True,
    num_train_epochs=5,
    per_gpu_train_batch_size=5,
    save_steps=10_000,
    save_total_limit=1,
    logging_steps=100,
    learning_rate=1.76e-3
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=train_dataset,
    eval_dataset=eval_dataset,
    prediction_loss_only=True,
)

trainer.train()
trainer.save_model("./results/new")

eval_output = trainer.evaluate()
perplexity = math.exp(eval_output["eval_loss"])
print({"loss": eval_output["eval_loss"]})
result = {"perplexity": perplexity}
print(result)
```
Steps to reproduce the behavior:

1.Download Albert here https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/
2.Run the script

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
- The training loss should converge to 0 in tiny datasets.
- Cross entropy should be always positive and eventually converge to zero.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: v3.02
- Platform: AWS instance
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 with GPU
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes, 8 GPU in total
- Using distributed or parallel set-up in script? Not sure.


pipeline does not do truncation on long texts input, error message found

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): No specified

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1.  I have tried using pipeline on my own purpose, but I realized it will cause errors if I input long sentence on some tasks, it should do truncation automatically, but it does not. And the pipeline function does not take extra argument so we cannot add something like `truncation=True`. Here is an example on sentiment-analysis task:
```python
from transformers import pipeline
nlp  = pipeline('sentiment-analysis')
text = "This is an example"*300
nlp(text)
```
2. The error message is below:
```
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-31-0a8ce849cc29> in <module>()
      2 nlp  = pipeline('sentiment-analysis')
      3 text = "This is an example"*300
----> 4 nlp(text)

11 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1722         # remove once script supports set_grad_enabled
   1723         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1724     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1725
   1726

IndexError: index out of range in self
```
3. same error on NER task, maybe some other task as well.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Google Colab
- Python version:  Python 3.6.9
- PyTorch version (GPU?):1.5.1+cu101, no GPU
- Tensorflow version (GPU?): no
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Cleanup Trainer and expose customization points

Clean up some parts of the code of `Trainer` and expose some function as customization points (doc will follow if you agree on this).


[WIP] Proposal for TF model outputs

This picks up on the work @thomwolf did on #5740 to have self-documented outputs in TensorFlow that are compatible with the AutoGraph system.
`TFModelOuput` subclasses `OrderedDict` while still being a dataclass, with some tweaks in the `post_init`:
- only the not-None attributes are set as values for the dictionary because tensorflow refuses None as outputs.
- a `TFModelOutput` can be instantiated with the regular keyword arguments but also with an iterator passed as the first argument (as a dict would) like @thomwolf suggested in #5740, with a fix to make sure the first input is not a tensor (because tensors are iterables).

This breaks two things for the TensorFlow side of the library:
1. when unpacking `outputs`, a slice needs to be used, otherwise the keys of the dictionary are returned, not the values:
```
loss, logits = outputs
```
will fail, it needs to be changed to
```
loss, logits = outputs[:2]
```
2. when loading and saving a model using `SavedModel`, the subclass is lost and the output becomes a regular dictionary (see the change in `test_keras_save_load`).

Apart from those, these model outputs are fully backward compatible (you can index with an int or a slice and get the same behavior as before).

If this is accepted, I would strongly recommend using the same base class for PyTorch model outputs, which would imply the breaking change number 1 but would have the added benefit of:
1. fixing the problem with `DataParallel`
2. have consistent outputs between TF and PyTorch.


add fine-tuned mobilebert squad v1 and squad v2 model cards

Related to pretrained model cards


dynamic masking for RoBERTa model

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->
<!-- Description of your issue -->
I saw that "dynamic masking" was mentioned on the README file for language modeling:
"In accordance to the RoBERTa paper, we use dynamic masking rather than static masking. The model may, therefore, converge slightly slower (over-fitting takes more epochs)."

I couldn't find which class this method is implemented in and how to enable this feature during pre-training using the Trainer class. Could someone please help me?

Thank you very much in advance.


Training data format

I have text on which I want to fine tune the gpt2 model for text autocompletion on my text the text sentences are separated by new line is there any format I should follow. When I trained on the data as it is it is not giving me proper results with the default training parameters. I have nearly after split 25k sentences for training. Please suggest. The training data looks like this
<img width="1220" alt="Screenshot 2020-07-22 at 10 24 01 PM" src="https://user-images.githubusercontent.com/33617789/88205241-18a09c00-cc6a-11ea-924e-a8df103c8b94.png">


[demo] Broken fp16 test


[test] partial coverage for train_mbart_enro_cc25.sh

CI covers all the examples logic that doesn't run on CUDA.
Adds coverage for:
- user facing bash script train_mbart_cc25_enro.sh
- the idea that seq2seq/finetune.py should lead to models getting better/ val BLEU increasing.

This only takes 10s to run on brutasse, so I gave it a minute for github actions CI, the only automated tester that will run this.

Will add coverage for `--do_predict` once [this](https://github.com/PyTorchLightning/pytorch-lightning/issues/2673) is fixed.


Transformer-XL: Fixed returned outputs when using `return_tuple=True`

Fixes #5974

`mems` are returned again when using `return_tuple=True`.


Transformer-XL: no mems are return when using 'return_tuple'

# 🐛 Bug

## Information
The forward pass of the `TransfoXLLMHeadModel` returns no `mems` when using `return_tuple=True`.

Model I am using: Transformer-XL

Language I am using the model on: English

The problem arises when using:
* [x] my own modified scripts: (give details below)

## To reproduce

```Python
from transformers import TransfoXLLMHeadModel, TransfoXLTokenizer

model = TransfoXLLMHeadModel.from_pretrained("transfo-xl-wt103")
model.train()
tokenizer = TransfoXLTokenizer.from_pretrained("transfo-xl-wt103")

encoded = tokenizer("Max is walking the dog in the streets", return_tensors='pt')
outputs = model(input_ids=encoded['input_ids'], mems=None, labels=encoded['input_ids'], return_tuple=True)
loss, _, mems = outputs

print(loss.size())
print(len(mems))      # should be 18 due to the 18 layers
```
Output:
```
Traceback (most recent call last):
  File "user/script.py", line 10, in <module>
    loss, _, mems = outputs
ValueError: not enough values to unpack (expected 3, got 2)
```

## Expected behavior
Output:
```
torch.Size([1, 7])
18
```

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.6.10
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


[cleanup] much cruft in unittests

Help wanted

Extra attention is needed, help appreciated

Anti patterns:
- making a result dict and then using each of it's keys. Why use the dict?
- delete all mentions of `check_loss_output`
- use tuple equality: `self.assertEqual(tensor.shape, (bs, seq_len)` instead of
```python
self.assertListEqual(list(tensor.size()), [bs, seq_len])
```

This does not need to be done for all test files at once.

fix `templates/testing_xxx ` to reflect the new best practice.


Update to match renamed attributes in fairseq master

Fix #5917 : RobertaModel no longer have model.encoder and args.num_classes attributes as of 5/28/20.


ImportError: cannot import name 'MODEL_WITH_LM_HEAD_MAPPING'

such error really annoys me.


[WIP] Ner pipeline grouped_entities fixes

There are many issues with ner pipeline using grouped_entities=True
https://github.com/huggingface/transformers/issues/5077
https://github.com/huggingface/transformers/issues/4816
https://github.com/huggingface/transformers/issues/5730
https://github.com/huggingface/transformers/issues/5609
https://github.com/huggingface/transformers/issues/6514
https://github.com/huggingface/transformers/issues/5541

- [x]  [Bug Fix] add an option `ignore_subwords` to ignore subsequent ##wordpieces in predictions. Because some models train on only the first token of a word and not on the subsequent wordpieces (BERT NER default). So it makes sense doing the same thing at inference time.

    - The simplest fix is to just group the subwords with the first wordpiece.
      - [TODO] how to handle ignored scores? just set them to 0 and calculate zero invariant mean ?
      - [TODO] handle different wordpiece_prefix ## ? possible approaches:
get it from tokenizer? but currently most tokenizers dont have a wordpiece_prefix property?
have an _is_subword(token)

- [x] [Feature add] added option to `skip_special_tokens`. Cause It was harder to remove them after grouping.

- [x] [Additional Changes] remove B/I prefix on returned grouped_entities

Edit: Ignored subwords' scores are also ignored by setting them to nan and using nanmean
Edit: B entities of different type are separated (as per BIO tag definition)
Edit: skip_special_tokens is now the default behavior
Edit: ignore_subwords is now the default behavior
Edit: more flexibility for custom non-standard tokenizers through tokenizer.is_subword_fn, tokenizer.convert_tokens_to_string
Edit: [fix UNK token related bugs by mapping UNK tokens to the correct original string] Use fast tokenizer or pass offset_mapping

# Usage
`pipeline('ner', model=model, tokenizer=tokenizer, ignore_labels=[], grouped_entities=True, ignore_subwords=True)`


run_squad example doesn't work with XLM model

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): XLM

Language I am using the model on (English, Chinese ...): Korean

The problem arises when using:
* [ ] the official example scripts: run_squad.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: KorQuAD v1.0

## To reproduce

Steps to reproduce the behavior:

1. Download the KorQuAD v1.0 data (it is formatted exactly as SQuAD v1.0)
2. Run the run_squad script with KorQuAD instead of SQuAD
3. Use the XLM model type (with specific model: xlm-mlm-100-1280_384)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

Since the data is formatted exactly the same as the english version, except its in Korean now, I expect the script to be able to fully train and evaluate on the new data. In fact, the script successfully does this when you use multi-lingual BERT instead of XLM. However, when you use XLM, you get invalid input in the forward call:

"cls_index" and "p_mask" are unexpected inputs.

I tried hacking in a fix for this, specifically just deleting cls_index and p_mask from the inputs when using XLM instead of Bert. This caused the model to be able to train correctly, but when we try to evaluate, we end up crashing on a new error. I don't know if its related or not, but this is in the squad metrics, not within run_squad.py, so I was less excited to start messing with that and instead decided to put up this issue.

It specifically crashes here:
 transformers/data/metrics/squad_metrics.py", line 629, in compute_predictions_log_probs
    cur_null_score = result.cls_logits
AttributeError: 'SquadResult' object has no attribute 'cls_logits'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.7.3
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No (Just 1 GPU)


Loss becoming nearly zero in first 5K steps when training LM from scratch

I am training the ALBERT LM model from scratch.

I have already trained it for Hindi and Bangla and it was working fine but when I am training on Gujarati, the loss is becoming zero in 5K steps.

What could be the reason for the sudden drop in the loss? Can anyone suggest what could be the cause or how to debug such issue?
Any suggestions?


Actually the extra_id are from 0-99 and not from 1-100

```
a = tokenizer.encode("we got a <extra_id_99>", return_tensors='pt',add_special_tokens=True)
print(a)
>tensor([[   62,   530,     3,     9, 32000]])
a = tokenizer.encode("we got a <extra_id_100>", return_tensors='pt',add_special_tokens=True)
print(a)
>tensor([[   62,   530,     3,     9,     3,     2, 25666,   834,    23,    26,
           834,  2915,  3155]])
```


Bug fix: NER pipeline shouldn't group separate entities of same type

## Effects
nlp=pipeline('ner', ... , grouped_entities=True)

## Fixes
Separate entities of same type shouldn't be grouped together even if they are same type
( B-type1 B-type1 ) != ( B-type1 I-type1 )

## Example
"something something Istanbul Los Angeles something something"

Current output: [ (O O) (B-type1 B-type1 I-type1) (O O) ]
Fixed output: [ (O O) (B-type1) (B-type1 I-type1) (O O) ]


BerTweet tokenizer issue

Core: Modeling

Internals of the library; Models.

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Bertweet

## To reproduce

Steps to reproduce the behavior:

1. tokenizer = AutoTokenizer.from_pretrained("vinai/bertweet-base")
2.
3.

OSError: Model name 'vinai/bertweet-base' was not found in tokenizers model name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). We assumed 'vinai/bertweet-base' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.json', 'merges.txt'] but couldn't find such vocabulary files at this path or url.
## Expected behavior

the tokenizer should be loaded correctly.
https://huggingface.co/vinai/bertweet-base?text=Paris+is+the+%3Cmask%3E+of+France.

## Environment info

- `transformers` version: 2.10.0
- Platform: ubuntu 18.04
- Python version: 3.7
- PyTorch version (GPU?): 1.5
- Tensorflow version (GPU?):
- Using GPU in script?: v100
- Using distributed or parallel set-up in script?: no


text classification reuse without classifier

thanks in advance.
I want to train first more data, it mans more label.
(its like pretraining)

after that I choose some label only, and train again.

But error happened during below code

tokenizer = RobertaTokenizer.from_pretrained(pretrained_path, do_lower_case=False)
model = RobertaForSequenceClassification.from_pretrained(pretrained_path, num_labels=10)

error message is like below.
Error(s) in loading state_dict for RobertaForSequenceClassification:
	size mismatch for classifier.out_proj.weight: copying a param with shape torch.Size([19, 768]) from checkpoint, the shape in current model is torch.Size([10, 768]).
	size mismatch for classifier.out_proj.bias: copying a param with shape torch.Size([19]) from checkpoint, the shape in current model is torch.Size([10]).

how can I do it??


Bert forward reports error on GPU; but runs fine on CPU

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I would like to encode articles using Bert. My code runs fine on CPU, but failed on GPU. The GPU is a on a remote server.

```
def assign_gpu(token):
    token_tensor = token['input_ids'].to('cuda')
    token_typeid = token['token_type_ids'].to('cuda')
    attention_mask = token['attention_mask'].to('cuda')
    output = {'input_ids': token_tensor,
              'token_type_ids': token_typeid,
              'attention_mask': attention_mask}
    return output

bs = 16
data_dl = DataLoader(PatentDataset(df), batch_size=bs, shuffle=False)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased/')
model = BertModel.from_pretrained('bert-base-uncased/')
inputs_list = []
for a in data_dl:
     inputs_list.append(tokenizer.batch_encode_plus(a, pad_to_max_length=True, return_tensors='pt'))

# GPU part
model.cuda()
model.eval()
out_list = []
with torch.no_grad():
    for i, inputs in enumerate(inputs_list):
        inputs = assign_gpu(inputs)
        output = model(**inputs)[0][:, 0, :]
        out_list.append(output)
```
The error message is

```
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-3-f6979d077f60> in <module>
      6     for i, inputs in enumerate(inputs_list):
      7         inputs = assign_gpu(inputs)
----> 8         output = model(**inputs)[0][:, 0, :]
      9         out_list.append(output)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/volta_pypkg/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, encoder_hidden_states, encoder_attention_mask, output_attentions, output_hidden_states)
    751
    752         embedding_output = self.embeddings(
--> 753             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
    754         )
    755         encoder_outputs = self.encoder(

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

~/volta_pypkg/lib/python3.6/site-packages/transformers/modeling_bert.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    180         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    181
--> 182         embeddings = inputs_embeds + position_embeddings + token_type_embeddings
    183         embeddings = self.LayerNorm(embeddings)
    184         embeddings = self.dropout(embeddings)

RuntimeError: CUDA error: device-side assert triggered
```

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**: https://stackoverflow.com/questions/63026701/bert-forward-reports-error-on-gpu-but-runs-fine-on-cpu


tensorflow转为pytorch的两个文件在哪里？

![image](https://user-images.githubusercontent.com/41299010/88131997-e31b9480-cc10-11ea-9966-f63a1b425878.png)
大佬您好：
请问这两个文件在哪里？discriminator.json和model.bin


[docs] Add integration test example to copy pasta template

Encourage testing practices that have been encouraged since last update, namely:

- @slow tests that run on cuda and fp16 if possible and show that your model produces good outputs.
- more tests ~= better
- unindent the ModelTester
- call get_extended_attention_mask and delete massive comment.

My code is probably broken because this thing isn't tested!

Add:
"""
Try to make this test take a string and check that a resultant string == desired_result using your tokenizers encode and decode functions
""


Adding Minimal Reproducible Usage Example For TPU support on examples/seq2seq

Attempt to resolve https://github.com/huggingface/transformers/issues/5895.

Minimal Working [Colab Example](https://colab.research.google.com/drive/16q2GWrnZ0Tjg1OxJQUcaWKCWwn3Jh5z0?usp=sharing).

For using more than a single core, one needs to ensure that enough RAM is available else wait for PyTorch-XLA to release a stable version. They have also [released](https://github.com/pytorch/xla/issues/1870#issuecomment-623603323) a fix way back that prevent excessive memory usage for nightly.


Can't load weights of models

When I run below codes, I can successfully load the tokenizer but fail with loading the models.
from transformers import AutoTokenizer, AutoModelWithLMHead
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModelWithLMHead.from_pretrained("bert-base-uncased")

Here is the error:
OSError: Can't load weights for 'bert-base-uncased'. Make sure that:
- 'bert-base-uncased' is a correct model identifier listed on 'https://huggingface.co/models'
- or 'bert-base-uncased' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.

I can load models after manually downloading them but I do want to directly load them via transformers.


Add functioning early stopping (patience) and weighted random sampling

This is to fix the issues in #4186 (hopefully) to get it merged in. Also adds weighted random sampling for imbalanced classes.


NoneType error when using Trainer

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 2.11.0
Python: 3.7.6

## Background Info
I wasn't sure what the `training_dataset` parameter of `Trainer` was so I opted to create a custom Pytorch `DataSet` using this [tutorial](https://pytorch.org/tutorials/beginner/data_loading_tutorial.html).
```python
from torch.utils.data import Dataset
import pandas as pd
import torch

class SDAbstractsDataset(Dataset):
    def __init__(self, csv_file):
        self.sd_abstracts_df = pd.read_csv(csv_file, encoding='ISO-8859-1')

    def __len__(self):
        return len(self.sd_abstracts_df)

    def __getitem__(self, idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()
        sample = {'abstract_text': self.sd_abstracts_df.iloc[idx, 1]}
        return sample
```

I instantiate the `SDAbstractsDataset` object, create the `TrainingArguments` object, create an object based of a customized model, then instantiate the `Trainer` object with `sd_dataset`.

```python
from text_gen_w_transformers.finetune_gpt2 import GPT2FinetunedWithNgrams
from text_gen_w_transformers.custom_dataset import SDAbstractsDataset
from transformers import TrainingArguments, Trainer

sd_dataset = SDAbstractsDataset('/path/to/samples_64.csv')

training_args = TrainingArguments(
    output_dir='/path/to/output/dir',
    do_train=True,
    per_device_train_batch_size=4,
    learning_rate=1e-3,
    num_train_epochs=1
)

model = GPT2FinetunedWithNgrams.from_pretrained('gpt2')

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=sd_dataset
)

trainer.train()
```

Whenever I run the `trainer.train()` command, I get the following error:
```python
Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Iteration:   0%|          | 0/16 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/path/to/project/finetune_test.py", line 37, in <module>
    trainer.train()
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/trainer.py", line 622, in _training_step
    outputs = model(**inputs)
  File "/path/to/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/path/to/project/finetune_gpt2.py", line 42, in forward
    orig_input_str = self.tokenizer.decode(input_ids[0], skip_special_tokens=True)
TypeError: 'NoneType' object is not subscriptable

Epoch:   0%|          | 0/1 [00:00<?, ?it/s]
Iteration:   0%|          | 0/16 [00:00<?, ?it/s]
```

I did a little debugging and found that `inputs` in the line `tr_loss += self._training_step(model, inputs, optimizer)` was empty. Any thoughts on how to fix this?

Thanks in advance!


[CI] Install examples/requirements.txt


# 🐛 Bug

A bunch of tf tests fail with:

```module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'```

```
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
ERROR tests/test_benchmark_tf.py - AttributeError: module 'tensorflow_core._api.v2.config' has no attribute 'list_physical_devices'
```

`tf.config.list_physical_devices` seems to be added in tf-2.1, so unless `transformers` starts to require tf >= 2.1, this breaks for tf < 2.1.

Depending on what you decide I can send a PR to fix this with either:

a. require  tf-2.1+ (simplest)
b. write a wrapper `list_physical_devices` that uses `tf.config.experimental.list_physical_devices` for tf < 2.1, and `tf.config.list_physical_devices` otherwise, and switch to using it.
c. do nothing?

## Environment info

```
- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


[pack_dataset] don't sort before packing, only pack train

this is better than sorting.
But for best metrics, don't pack.


CL util to convert models to fp16 before upload

This should probably go to `commands/`


Create README.md

model card

Related to pretrained model cards


seq2seq: checkpoint callback seems messed up

![image](https://user-images.githubusercontent.com/6045025/88097457-2967e980-cb66-11ea-85fb-9ea4ae1101e8.png)

^^ lightning checkpoint saved much later than `best_tfmr`


seq2seq/run_eval.py can take decoder_start_token_id

Also: batch_decode: document kwargs for autocomplete


Exporting T5 to ONNX

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): t5-small (T5ForConditionalGeneration)

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)
The problem arises when running `convert_graph_to_onnx.py`

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)
I am using a T5ForConditionalGeneration for machine translation.
## To reproduce

Steps to reproduce the behavior:

1. Run `python transformers/convert_graph_to_onnx.py --framework pt --model t5-small --tokenizer t5-small --opset 12 t5-small.onnx`

```
ONNX opset version set to: 12
Loading pipeline (model: t5-small, tokenizer: t5-small)
/Users/joshuasirota/onnx_env/lib/python3.6/site-packages/transformers/modeling_auto.py:798: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 242M/242M [01:09<00:00, 3.50MB/s]
Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at t5-small and are newly initialized: ['encoder.embed_tokens.weight', 'decoder.embed_tokens.weight', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Using framework PyTorch: 1.5.1
Error while converting the model: You have to specify either decoder_input_ids or decoder_inputs_embeds
```

## Expected behavior

An ONNX export should be created

## Environment info

- `transformers` version: 3.0.2
- Platform: Darwin-18.6.0-x86_64-i386-64bit
- Python version: 3.6.5
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Test on a new string of gpt2 fine tuned

I fine tuned GPT2 on my own dataset now I have fine tuned model how can I test it on new string??

GPT-2/GPT and causal language modeling
The following example fine-tunes GPT-2 on WikiText-2. We're using the raw WikiText-2 (no tokens were replaced before the tokenization). The loss here is that of causal language modeling.

```
export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE
```


Update doc to new model outputs

This is a follow-up from #5438, adapting doc pages and examples.


…

consistently use True/False for `return_tuple` (currently it's sometimes None, sometimes False)


process stuck at LineByLineTextDataset. training not starting

# ❓ Questions & Help

I am using below python code.
BASE_MODEL  = "/data/nlp/bert-base-uncased"
CACHE_DIR  = "/data/nlp/cache"

model       = AutoModelWithLMHead.from_pretrained(BASE_MODEL,cache_dir=CACHE_DIR)
t_tokenizer = ByteLevelBPETokenizer()
t_tokenizer.train(files=paths, vocab_size=52_000, min_frequency=2, special_tokens=[
    "<s>",
    "<pad>",
    "</s>",
    "<unk>",
    "<mask>",
])

new_vocab = t_tokenizer.get_vocab()

tokenizer = BertTokenizerFast.from_pretrained(BASE_MODEL)
num_added_toks = tokenizer.add_tokens(list(new_vocab.keys()))
model.resize_token_embeddings(len(tokenizer))
dataset = LineByLineTextDataset(tokenizer=tokenizer,file_path=DATA_FILE,block_size=64)

it works fine for small files but for file > 600 MB the process gets stuck at dataset = LineByLineTextDataset(tokenizer=tokenizer,file_path=DATA_FILE,block_size=64)


[Doc] explaining romanian postprocessing for MBART BLEU hacking


Converting GPT2 logits to token ids directly

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 2.11.0
Python: 3.7.6

## Background Info
Here is the constructor and forward method for the model I am trying to build. Ultimately it will finetune GPT2 with a loss from a ngrams model I made. The loss isn't implemented yet because I want to test the GPT2 generation part first.

```python
class GPT2FinetunedWithNgrams(GPT2LMHeadModel):
    def __init__(self, config):
        super().__init__(config)
        self.tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

def forward(
        self,
        input_ids=None,
        past=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        use_cache=True,
):
    temperature = 0.85
    tmp_input_ids = input_ids
    max_gen_length = 30
    counter = 0
    orig_input_str = self.tokenizer.decode(input_ids[0], skip_special_tokens=True)
    strs_to_join = orig_input_str.split()
    while counter < max_gen_length:
        transformer_outputs = self.transformer(
            tmp_input_ids,
            past=past,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            use_cache=use_cache,
        )

        hidden_states = transformer_outputs[0]
        lm_logits = self.lm_head(hidden_states) / (temperature)
        last_token = lm_logits[:, -1]
        last_token_softmax = torch.softmax(last_token, dim=-1).squeeze()

        next_token = torch.argmax(last_token_softmax).tolist()
        next_gen_token_str = self.tokenizer.decode(next_token, clean_up_tokenization_spaces=True).strip()
        strs_to_join.append(next_gen_token_str)

        new_str_input = ' '.join(strs_to_join)
        tmp_input_ids = self.tokenizer.encode(new_str_input, return_tensors='pt')
        counter += 1
    return new_str_input
```
Right now the code will take the `lm_logits`, calculate the softmax, and then get the next token predicted by GPT2. I then add that next token to the original input sequence and feed that combination back into GPT2, until the `max_gen_length` is reached. Finally it returns the original input sequence with the generated sequence appended to it.

## Question 1
Is there a way to directly go from logits to token ids that I am missing in HF Transformers? Or better yet, is there a better, more efficient way of doing this?

## Question 2
Given that I am using things like a `max_length` and `temperature`, I feel like the `generate()` method would be useful here. However, I am not exactly sure how to implement that. Since ultimately I want my model to finetune GPT2 based off a ngrams loss, I don't know if calling `generate()` will call that method from the GPT2 model that is involved in the finetuning or some fixed GPT2 model coming elsewhere. Part of this is my nonfamiliarity with how pytorch trains (which I'm working on understanding better).

Said differently, if I can call `generate()` within the above forward method (I imagine something like `super().generate()`) will that generate a sequence using the GPT2 model whose weights are currently being modified based off my dataset and finetuning, or will it generate a sequence from some static version of GPT2 that is not having its weights modified?

I hope those questions aren't too convoluted. I can elaborate more if needed. Thanks in advance for your help!


a transparent solution for  DataParallel.gather not supporting ModelOutput (dataclass)

1. Modify torch/nn/parallel/scatter_gather.gather function to support ModelOutput (dataclass) outputs. We override the torch.nn.DataParallel.gather method with this custom method.

2. remove previously committed workarounds

implementation: @sgugger
integration/testing: me

This should transparently solve https://github.com/huggingface/transformers/issues/5693


What is the difference between the function of add_tokens() and add_special_tokens() in tokenizer

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
When I read the code of tokenizer, I have a problem if I want to use a pretrained model in NMT task, I need to add some tag tokens, such as '2English' or '2French'. I think these tokens are special tokens, so which function should I use: add_tokens() or add_special_tokens(). What is the difference between them?


Can't use BatchEncoding in the fit function

I had a [few problems with the `transformers`](https://github.com/huggingface/transformers/issues/5555) and the related [`tensorflow` functionality](https://github.com/tensorflow/tensorflow/issues/41204), but now I've made some progress using trial and error method.
You can see the issue in this [gist](https://colab.research.google.com/drive/125jJ0qrXGIe6goNrH_Ja7XPZtYp7nMXU?usp=sharing), now I found out that the problem was due to `model.fit(...)` function.
Here is the first version which causes the error:
```
model.fit(
    x=X_train, #transformers.tokenization_utils_base.BatchEncoding
    y=targets,
    epochs=3
)
```
Output:
```
ValueError: too many values to unpack (expected 2)
```
Second version which at least works:
```
model.fit(
    x=X_train.values(), #dict_values
    y=targets,
    epochs=3
)
```
**So, now the main question is why does it work and is my code correct?** I'm asking because I have seen many code fragments which used `BatchEncoding` as `x` in the `fit`(e.g. [this](https://www.kaggle.com/definedennis/pretrained-bert-with-huggingface-tensorflow-2-1/) and [this](https://github.com/thakursc1/NLPKernels/blob/3bb1fcac60ab8cdc6f2f68a2d9b5b7a477873811/DisasterTweetClassification/Transformer.py)), but I just can't do the same thing.


How does AdamW weight_decay works for L2 regularization?

Hello I have some questions about weight regularization in Adam.
Apparently the `weight_decay` in the AdamW function https://huggingface.co/transformers/main_classes/optimizer_schedules.html#adamw-pytorch has the same impact as an `L2 regularization`.

My questions are: is that parameter the same as `lambda` that we have in the regularization term?

![image](https://user-images.githubusercontent.com/36989932/88071850-3f78a880-cb6c-11ea-990c-2d97620e4b5c.png)

How does it exactly work? And what is its impact on the model complexity?


typos in seq2seq/readme

same-as-title.


Easy selection of a learning rate scheduler when initializing a Trainer

# 🚀 Feature request

Please consider adding a new argument to the Trainer constructor to specify which among the available learning rate schedulers should be used during training.

## Motivation

Even though Trainer already has the option to specify a given optimizer and learning rate scheduler, you need to explicitly initialize both (even when you only want to change the scheduler) with parameters already available to the Trainer itself via TrainingArguments. It would be nicer and smoother to just provide Trainer with a string specifying which scheduler to use (e.g. 'constant_schedule', 'cosine_schedule_with_warmup', ...) and have `get_optimizers` implement the choice.


Getting "AttributeError: 'Tensor' object has no attribute 'numpy'" while fine-tuning BERT for NER

As per https://github.com/huggingface/transformers/tree/master/examples/token-classification after doing the required setup and installing required libraries, when I run

```
python3 run_tf_ner.py --data_dir ./ \
--labels ./labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_device_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_train \
--do_eval \
--do_predict
```
it stops at one point with error

```
    /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:488 _accumulate_next  *
        return self._accumulate_gradients(per_replica_features, per_replica_labels)
    /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:498 _accumulate_gradients  *
        per_replica_loss = self.args.strategy.experimental_run_v2(
    /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:511 _forward  *
        per_example_loss, _ = self._run_model(features, labels, True)
    /usr/local/lib/python3.6/dist-packages/transformers/trainer_tf.py:534 _run_model  *
        outputs = self.model(features, labels=labels, training=training)[:2]
    /usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_distilbert.py:879 call  *
        loss = self.compute_loss(labels, logits)
    /usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py:142 compute_loss  *
        if tf.math.reduce_any(labels == -1).numpy() is True:

    AttributeError: 'Tensor' object has no attribute 'numpy'
```

Tensorflow version: 2.2.0
Numpy version: 1.19.0
CUDA version: 10.2

As per some possible solutions I have checked that tf.executing_eagerly() is True.

Tried on own computer and colab both and it ends up at the same point with same error.


# 🐛 Bug

## Information

Model I am using (Bert):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```python
def build_model(item_dim, num_layers, num_heads, max_len):
    config = BertConfig(hidden_size=item_dim, num_hidden_layers=num_layers,
                        num_attention_heads=num_heads, intermediate_size=item_dim*4,
                        max_position_embeddings=max_len)
    bert = TFBertMainLayer(config=config)
    inputs = Input(shape=(max_len, item_dim), dtype=tf.float32, name='inputs')
    # pre-training vectors to bert
    seq_emb = bert(inputs=None, inputs_embeds=inputs)[0]
    print(seq_emb)
    print(seq_emb[:, -1, :])
    last_token_emb = seq_emb[:, -1, :]
    outputs = Dense(1, activation='sigmoid')(last_token_emb)
    model = Model(inputs=inputs, outputs=outputs)
    return model
```

Errors with
`InvalidArgumentError: Cannot convert a Tensor of dtype resource to a NumPy array.`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
* Run the code successfully when feed pre-training vectors to bert.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
https://colab.research.google.com/gist/Douboo/adf6e136e45f8406b1070d88f4041a49/untitled2.ipynb

- `transformers` version: 3.0.2
- Platform: google colab
- Python version: 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.2.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no


How to get a language model score in BertModel?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
Hi, thanks for your awesome works!

How to get a language model score which could judge whether a sentence conforms to grammatical rules by using BertModel ?
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Expose padding_strategy on squad processor to fix QA pipeline performance regression

**Before this PR**:
The squad processor was padding the sequence up to the provided `max_length` parameter which results in a tensor with 512 tokens, mostly padding, making the QA pipeline very slow.

- QA Pipeline (`model="distilbert-base-uncased-distilled-squad"`) = 4.8secs

**After this PR**:
The processor will not be padding at all when coming from the QA pipeline.

- QA Pipeline (`model="distilbert-base-uncased-distilled-squad"`) = 1.29secs


ALBERT tokenizer is not callable

# 🐛 Bug

## Information
when running the follwoing given example:
from transformers import AlbertTokenizer, AlbertModel
import torch
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertModel.from_pretrained('albert-base-v2')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

it alerts "*** TypeError: 'AlbertTokenizer' object is not callable"

examples link: https://huggingface.co/transformers/model_doc/albert.html#alberttokenizer

Model I am using (Bert, XLNet ...):
ALBERT

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [X ] the official example scripts: (give details below)
from transformers import AlbertTokenizer, AlbertModel
import torch
tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertModel.from_pretrained('albert-base-v2')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
 link: https://huggingface.co/transformers/model_doc/albert.html#alberttokenizer'

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

*** TypeError: 'AlbertTokenizer' object is not callable

## Expected behavior

Should given a workable example of using ALBERT ?

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:  2.3.0
- Platform:  Linux
- Python version:   3.7.4
- PyTorch version (GPU?):   1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?:  Yes
- Using distributed or parallel set-up in script?:   No


code copy button on the website doesn't copy `...` lines

# 🐛 Bug

## Information

When the copy button is clicked on a code like [this](https://huggingface.co/transformers/quicktour.html#using-the-tokenizer):

```
>>> pt_batch = tokenizer(
...     ["We are very happy to show you the 🤗 Transformers library.", "We hope you don't hate it."],
...     padding=True,
...     truncation=True,
...     return_tensors="pt"
... )
```

only the first line is copied.

All lines starting with `>>>` get copied, but `...` lines are ignored. So for example, [this](https://huggingface.co/transformers/quicktour.html#customizing-the-model) gets fully copied:

```
>>> from transformers import DistilBertConfig, DistilBertTokenizer, DistilBertForSequenceClassification
>>> config = DistilBertConfig(n_heads=8, dim=512, hidden_dim=4*512)
>>> tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased')
>>> model = DistilBertForSequenceClassification(config)
```

I was told that this is the issue with `sphinx_copybutton` and I found an already opened issue there:
https://github.com/executablebooks/sphinx-copybutton/issues/65

So hopefully it gets fixed over there, and then the website can be updated to include the fix.


Add DeBERTa model

Related to pretrained model cards

Add DeBERTa model to hf transformers. DeBERTa applies two techniques to improve RoBERTa, one is disentangled attention, the other is enhanced mask decoder.  With 80GB training data, DeBERTa outperform RoBERTa on a majority of NLU tasks, e.g. SQUAD, MNLI and RACE. Paper link: https://arxiv.org/abs/2006.03654


Feed forward chunking for all pretrained models

Based on this card: [Feed forward chunking](https://github.com/huggingface/transformers/projects/9#card-39483681)

@patrickvonplaten

I'd like to help contribute and implement this for all the other models if this is still pending?


[CI] self-scheduled runner tests examples/


DataParallel fix: multi gpu evaluation

The DataParallel training was fixed in https://github.com/huggingface/transformers/pull/5733, this commit also fixes the evaluation. It's more convenient when the user enables both `do_train` and `do_eval`.


Allow user to see actual error if a download has failed

Fixes #5869. Adds new logic of exception handling.
* If there was a caught exception in `cached_path` during download, and `force_download` is True, then raise it
* If `force_download` is False, save this exception and raise it later, it file is not in local cache
* If download exception in `cached_path` was raised, then re-raise it in calling function. If there's no exception, but resolved file is None, keep old message ("...is a correct model identifier...")

It would be great to know if there's a better way to dealing with this issue.


Create README.md

Related to pretrained model cards


how (if at all) are those models related...

https://github.com/huggingface/transformers/tree/master/examples/seq2seq

and...

https://github.com/nlpyang/PreSumm

are they the same?


Avoid unnecessary warnings when loading pretrained model

Currently, the following commands will produce a lot of warnings, because some weights are not saved with the model by design.

GPT-2 (see #5800, #5814)
```
from transformers import GPT2LMHeadModel
model = GPT2LMHeadModel.from_pretrained('gpt2')
```

T5 (see #3553, #5348)
```
from transformers import T5ForConditionalGeneration
model = T5ForConditionalGeneration.from_pretrained('t5-small')
```

Bart
```
from transformers import BartForConditionalGeneration
model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')
```

This PR introduces a new class attributes that you can tune per model to ignore some keys during loading and avoid those warnings (that scare users something went wrong for no reason). It fixes the above issues and gives us an API to fix any similar issues on other models.

Pinging @patrickvonplaten since you mentioned this recently.


Create README.md

Related to pretrained model cards

- Maybe the result of this query answers the question You did some days ago @julien-c ;-)


[examples/seq2seq]: add --label_smoothing option

cc @patil-suraj
This seems to improve BLEU score by ~2pts!
- also adds --early_stopping_patience command line arg
- fixes MBartDataset src,tgt flipping bug
- adds `--early_stopping_patience` command line arg for PL.
- wandb now looks for shell variable `$WANDB_PROJECT_NAME`


Add Fast Transformers - Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

# 🌟 New model addition

## Model description

The Fast Transformers repo introduces a fast transformer model based on work to improve attention published in two papers:

- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (https://arxiv.org/abs/2006.16236)
- Fast Transformers with Clustered Attention (https://arxiv.org/abs/2007.04825)

## Open source status

* [X] the model implementation is available: (give details)
https://github.com/idiap/fast-transformers

* [x] the model weights are available: (give details)
* [X] who are the authors: (mention them, if possible by @gh-username)
@angeloskath


convert_roberta: AttributeError when converting CamemBERT model.pt to pytorch_model.bin

Hi,

I trained a CamemBERT model with the fairseq library which gave me the following files:
- dict.txt: vocabulary coming from the sentencepiece model
- sentencepiece.bpe.model
- model.pt

Now I am trying to convert the model.pt into pytorch_model.bin and config.json as mentionned here ([fairseq/issues#1514](https://github.com/pytorch/fairseq/issues/1514)) and here ([transformers/issue#1850](https://github.com/huggingface/transformers/issues/1850)), by using the conversion script of the transformers library ([transfomers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py)). The goal is to use those files with fast-bert.

However, using this command line:
```shell
python convert_roberta_original_pytorch_checkpoint_to_pytorch.py --roberta_checkpoint_path ./ --pytorch_dump_folder_path ./ --classification_head
```

I get the following error:

```python
AttributeError                            Traceback (most recent call last)
<ipython-input-27-ea791887ff26> in <module>
----> 1 convert_roberta_original_pytorch_checkpoint_to_pytorch.convert_roberta_checkpoint_to_pytorch(CAMEMBERT_PATH, CAMEMBERT_PATH, True)

~/anaconda3/envs/NLP/lib/python3.7/site-packages/transformers/convert_roberta_original_pytorch_checkpoint_to_pytorch.py in convert_roberta_checkpoint_to_pytorch(roberta_checkpoint_path, pytorch_dump_folder_path, classification_head)
     48     roberta = FairseqRobertaModel.from_pretrained(roberta_checkpoint_path)
     49     roberta.eval()  # disable dropout
---> 50     roberta_sent_encoder = roberta.model.decoder.sentence_encoder
     51     config = RobertaConfig(
     52         vocab_size=roberta_sent_encoder.embed_tokens.num_embeddings,

~/anaconda3/envs/NLP/lib/python3.7/site-packages/torch/nn/modules/module.py in __getattr__(self, name)
    592                 return modules[name]
    593         raise AttributeError("'{}' object has no attribute '{}'".format(
--> 594             type(self).__name__, name))
    595
    596     def __setattr__(self, name, value):

AttributeError: 'RobertaModel' object has no attribute 'decoder'
```

And indeed when I check the fairseq/pytorch RobertaModel has no decoder attribute.

Am I doing this wrong ? I see no other conversion script to fit my CamemBERT model so I guess the RoBERTa one is the good one.


Clarify arg class

Just clarifying which dataset we're talking about.


Incompatible tensor type when running BART on TPU

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): `facebook/bart-large`

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: XSUM
* [ ] my own task or dataset: (give details below)

## To reproduce

1. Setup a Google VM with the XLA image and configure it to use TPUs
2. Follow the instrcutions in `seq2seq` example for downloading XSUM
3. Then run
```
export PYTHONPATH="../":"${PYTHONPATH}"

python finetune.py \
    --learning_rate=3e-5 \
    --gpus 0 \
    --n_tpu_cores 8 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.1 \
    --data_dir ${PWD}/xsum \
    --train_batch_size=1 \
    --eval_batch_size=1 \
    --output_dir=xsum_results \
    --num_train_epochs 1 \
    --model_name_or_path facebook/bart-large
```
...and you get something like
```
Exception in device=TPU:5: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
Traceback (most recent call last):
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 119, in _start_fn
    fn(gindex, *args)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/distrib_parts.py", line 222, in tpu_train
    self.run_pretrain_routine(model)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/trainer.py", line 1196, in run_pretrain_routine
    False)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 293, in _evaluate
    output = self.evaluation_forward(model, batch, batch_idx, dataloader_idx, test_mode)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 470, in evaluation_forward
    output = model.validation_step(*args)
  File "/home/martongyorgy/transformers/examples/seq2seq/finetune.py", line 145, in validation_step
    return self._generative_step(batch)
  File "/home/martongyorgy/transformers/examples/seq2seq/finetune.py", line 176, in _generative_step
    decoder_start_token_id=self.decoder_start_token_id,
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/generation_utils.py", line 248, in generate
    if self.get_output_embeddings() is None:
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/modeling_bart.py", line 1113, in get_output_embeddings
    return _make_linear_from_emb(self.model.shared)  # make it on the fly
  File "/anaconda3/envs/myenv/lib/python3.6/site-packages/transformers/modeling_bart.py", line 190, in _make_linear_from_emb
    lin_layer.weight.data = emb.weight.data
RuntimeError: Attempted to call `variable.set_data(tensor)`, but `variable` and `tensor` have incompatible tensor type.
```
## Environment info
```
- `transformers` version: 3.0.2
- Platform: Linux-4.9.0-12-amd64-x86_64-with-debian-9.12
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.0a0+ab660ae (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: TPU setup following Google Cloud tutorial for PyTorch
```


Add AlbertForPretraining to doc

Document models that were absent.


[Fix] seq2seq pack_dataset.py actually packs

Added stronger test (that failed before small code fixes).


Improve doc of use_cache

Followup from #5883


[WIP] Add Pegasus

* Add PEGASUS in TF2


QA Pipeline: Key Error due to predicting a token in question

Core: Pipeline

Internals of the library; Pipeline.

# 🐛 Bug

## Information

Model: deepset/roberta-base-squad2
Language: English
The problem arises when using: QA inference via pipeline

This seems to be a very similar issue to #5711
The pipeline throws an exception when the model predicts a token that is not part of the document, in this case it seems to be in the question.
In the example below, the model predicts token 3 to be the start and end of the answer span. But these tokens are a part of the question I believe. Therefore, we get a key error when trying to access
feature.token_to_orig_map[3]) in here:

https://github.com/huggingface/transformers/blob/ce374ba87767d551f720242d5e64bfa976531079/src/transformers/pipelines.py#L1370-L1380

## To reproduce

```
nlp = pipeline("question-answering",model="deepset/roberta-base-squad2",
                                  tokenizer="deepset/roberta-base-squad2",
                                  device=-1)

nlp(question="Who is the father of Sansa Stark?", context="===''A Game of Thrones''===\
Sansa Stark begins the novel by being betrothed to Crown Prince Joffrey Baratheon, believing Joffrey to be a gallant prince. While Joffrey and Sansa are walking through the woods, Joffrey notices Arya sparring with the butcher's boy, Mycah. A fight breaks out and Joffrey is attacked by Nymeria (Arya's direwolf) after Joffrey threatens to hurt Arya. Sansa lies to King Robert about the circumstances of the fight in order to protect both Joffrey and her sister Arya.  Since Arya ran off with her wolf to save it, Sansa's wolf is killed instead, estranging the Stark daughters.\
During the Tourney of the Hand to honour her father Lord Eddard Stark, Sansa Stark is enchanted by the knights performing in the event.  At the request of his mother, Queen Cersei Lannister, Joffrey spends a portion of the tourney with Sansa, but near the end he commands his guard Sandor Clegane, better known as The Hound, to take her back to her quarters. Sandor explains how his older brother, Gregor, aka "Mountain that Rides" pushed his face into a brazier of hot coals, for playing with one of his wooden toys.\
After Eddard discovers the truth of Joffrey's paternity, he tells Sansa that they will be heading back to Winterfell. Sansa is devastated and wishes to stay in King's Landing, so she runs off to inform Queen Cersei of her father's plans, unwittingly providing Cersei with the information needed to arrest her father. After Robert dies, Sansa begs Joffrey to show mercy on her father and he agrees, if Ned will swear an oath of loyalty, but executes him anyway, in front of Sansa.  Sansa is now effectively a hostage in King's Landing and finally sees Joffrey's true nature, after he forces her to look at the tarred head of her now-deceased father.")
```
results in
```
Traceback (most recent call last):
  File "/Users/deepset/deepset/haystack/tutorials/Tutorial1_Basic_QA_Pipeline.py", line 145, in <module>
    prediction = finder.get_answers(question="Who is the father of Sansa Stark?", top_k_retriever=1, top_k_reader=5)
  File "/Users/deepset/deepset/haystack/haystack/finder.py", line 57, in get_answers
    top_k=top_k_reader)  # type: Dict[str, Any]
  File "/Users/deepset/deepset/haystack/haystack/reader/transformers.py", line 80, in predict
    predictions = self.model(query, topk=self.n_best_per_passage)
  File "/Users/deepset/deepset/environments/haystack/lib/python3.7/site-packages/transformers/pipelines.py", line 1316, in __call__
    for s, e, score in zip(starts, ends, scores)
  File "/Users/deepset/deepset/environments/haystack/lib/python3.7/site-packages/transformers/pipelines.py", line 1316, in <listcomp>
    for s, e, score in zip(starts, ends, scores)
KeyError: 3
```
## Expected behavior
Predictions that are pointing to tokens that are not part of the "context" (here: tokens in question) should be filtered out from possible answers.

## Environment info
transformers version: latest master (82dd96cae74797be0c1d330566df7f929214b278)
Platform: Mac OS Catalina
Python version: 3.7.5
PyTorch version (GPU?): 1.5.1, CPU
Using GPU in script?: No
Using distributed or parallel set-up in script?: No


Make Tokenizers Faster When There Are Many Additional Special Tokens

`PreTrainedTokenizer.unique_no_split_tokens` used to be a list that contains all special tokens.  During tokenization, the tokenizer will repeatedly check `if sub_text not in self.unique_no_split_tokens` or `if token not in self.unique_no_split_tokens`. List lookups will significantly slow down tokenization if the list is large, i.e., there are many additional special tokens added to `unique_no_split_tokens`. To resolve this issue, this commit will change `PreTrainedTokenizer.unique_no_split_tokens` to be an ordered dict (actually an ordered set, since all values are `None`), such that lookups can be done very efficiently while still keeping its original ordering.