Resource exhausted when training in loop

## Environment info
- `transformers` version: 3.4.0
- Platform: Linux-4.15.0-109-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyTorch version (GPU?): 1.4.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik , @thomwolf or @jplu maybe?

## Information

Model I am using Bert:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce
To reproduce this error, one can use the script below. I have taken it from a blog post I made and altered it a bit - so it is a bit long. The main thing however, seems to be that if I load and fit a model several times it will at some point make my GPU run out of memory. Here I use kerastuner for hyperparam optimization and after 7 trials my GPU runs out of memory and thoughs the error below:
`  (0) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
`

### Reproducible script
```
#######################################
### -------- Load libraries ------- ###

# Load Huggingface transformers
from transformers import TFBertModel, BertConfig, BertTokenizerFast

# Then what you need from tensorflow.keras
from tensorflow.keras.layers import Input, Dropout, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.callbacks import EarlyStopping
from tensorflow.keras.initializers import TruncatedNormal
from tensorflow.keras.losses import CategoricalCrossentropy
from tensorflow.keras.metrics import CategoricalAccuracy
from tensorflow.keras.utils import to_categorical

# And pandas for data import + sklearn because you allways need sklearn
import pandas as pd
from sklearn.model_selection import train_test_split

#######################################
### --------- Import data --------- ###

# Import data from csv
data = pd.read_csv('dev/Fun with BERT/complaints.csv')

# Select required columns
data = data[['Consumer complaint narrative', 'Product', 'Issue']]

# Remove a row if any of the three remaining columns are missing
data = data.dropna()
data = data.sample(n = 5000)

# Remove rows, where the label is present only ones (can't be split)
data = data.groupby('Issue').filter(lambda x : len(x) > 1)
data = data.groupby('Product').filter(lambda x : len(x) > 1)

# Set your model output as categorical and save in new label col
data['Issue_label'] = pd.Categorical(data['Issue'])
data['Product_label'] = pd.Categorical(data['Product'])

# Transform your output to numeric
data['Issue'] = data['Issue_label'].cat.codes
data['Product'] = data['Product_label'].cat.codes

# Split into train and test - stratify over Issue
data, data_test = train_test_split(data, test_size = 0.2, stratify = data[['Issue']])

#######################################
### --------- Setup BERT ---------- ###

# Name of the BERT model to use
model_name = 'bert-base-uncased'

# Max length of tokens
max_length = 100

# Load transformers config and set output_hidden_states to False
config = BertConfig.from_pretrained(model_name)
config.output_hidden_states = False

# Load BERT tokenizer
tokenizer = BertTokenizerFast.from_pretrained(pretrained_model_name_or_path = model_name, config = config)

#######################################
### ------- Build the model ------- ###

def model_build(hp):
    # TF Keras documentation: https://www.tensorflow.org/api_docs/python/tf/keras/Model

    # Load the Transformers BERT model
    transformer_model = TFBertModel.from_pretrained(model_name, config = config)

    # Build your model input
    input_ids = Input(shape=(max_length,), name='input_ids', dtype='int32')
    # attention_mask = Input(shape=(max_length,), name='attention_mask', dtype='int32')
    # inputs = {'input_ids': input_ids, 'attention_mask': attention_mask}
    inputs = {'input_ids': input_ids}

    # Load the Transformers BERT model as a layer in a Keras model
    bert_model = transformer_model(inputs)[1]
    dropout = Dropout(hp.Float('hp_dropout', min_value=0, max_value=0.99, default = 0.2), name='pooled_output')
    pooled_output = dropout(bert_model, training=False)

    # Then build your model output
    issue = Dense(units=len(data.Issue_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=hp.Float('hp_stddev_issue', min_value=0, max_value=1, default = 0.02)), name='issue')(pooled_output)
    product = Dense(units=len(data.Product_label.value_counts()), kernel_initializer=TruncatedNormal(stddev=hp.Float('hp_stddev_product', min_value=0, max_value=1, default = 0.02)), name='product')(pooled_output)
    outputs = {'issue': issue, 'product': product}

    # And combine it all in a model object
    model = Model(inputs=inputs, outputs=outputs, name='BERT_MultiLabel_MultiClass')

    # Set an optimizer
    optimizer = Adam(
        learning_rate=5e-05,
        epsilon=1e-08,
        decay=0.01,
        clipnorm=1.0)

    # Set loss and metrics
    loss = {'issue': CategoricalCrossentropy(from_logits = True), 'product': CategoricalCrossentropy(from_logits = True)}
    metric = {'issue': CategoricalAccuracy('accuracy'), 'product': CategoricalAccuracy('accuracy')}

    # Compile the model
    model.compile(
        optimizer = optimizer,
        loss = loss,
        metrics = metric)

    return model

from kerastuner import HyperModel
class ClsModel(HyperModel):

    def build(self, fitting_param = None):

        model = model_build()

        return model

#######################################
### ------- Train the model ------- ###

# Ready output data for the model
y_issue = to_categorical(data['Issue'])
y_product = to_categorical(data['Product'])

# Tokenize the input (takes some time)
x = tokenizer(
    text=data['Consumer complaint narrative'].to_list(),
    add_special_tokens=True,
    max_length=max_length,
    truncation=True,
    padding=True,
    return_tensors='tf',
    return_token_type_ids = False,
    return_attention_mask = True,
    verbose = True)

from kerastuner.tuners import BayesianOptimization
tuner = BayesianOptimization(
    model_build,
    objective='loss',
    max_trials=50,
    executions_per_trial=5,
    project_name='modeltest')

tuner.search(
    x={'input_ids': x['input_ids']},
    y={'issue': y_issue, 'product': y_product},
    epochs=5)
```

### Full error log
Or allmost - removed some 100 lines of `2020-11-04 16:53:37.6569xx: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size xxxxxxxx totalling xxxMiB `

```
2020-11-04 16:53:37.656906: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2405376 totalling 2.29MiB
2020-11-04 16:53:37.656913: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2457600 totalling 2.34MiB
2020-11-04 16:53:37.656919: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2950912 totalling 2.81MiB
2020-11-04 16:53:37.656926: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 2967552 totalling 2.83MiB
2020-11-04 16:53:37.656933: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3091200 totalling 2.95MiB
2020-11-04 16:53:37.656939: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3104512 totalling 2.96MiB
2020-11-04 16:53:37.656946: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3161088 totalling 3.01MiB
2020-11-04 16:53:37.656953: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3555840 totalling 3.39MiB
2020-11-04 16:53:37.656960: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 3575808 totalling 3.41MiB
2020-11-04 16:53:37.656966: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4012032 totalling 3.83MiB
2020-11-04 16:53:37.656973: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 18 Chunks of size 4110336 totalling 70.56MiB
2020-11-04 16:53:37.656980: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4111104 totalling 3.92MiB
2020-11-04 16:53:37.656987: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4207872 totalling 4.01MiB
2020-11-04 16:53:37.656994: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4236544 totalling 4.04MiB
2020-11-04 16:53:37.657001: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4245504 totalling 4.05MiB
2020-11-04 16:53:37.657008: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 4354816 totalling 4.15MiB
2020-11-04 16:53:37.657015: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 809 Chunks of size 9437184 totalling 7.11GiB
2020-11-04 16:53:37.657023: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 143 Chunks of size 9830400 totalling 1.31GiB
2020-11-04 16:53:37.657030: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10106624 totalling 9.64MiB
2020-11-04 16:53:37.657037: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10291200 totalling 9.81MiB
2020-11-04 16:53:37.657044: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10386432 totalling 9.91MiB
2020-11-04 16:53:37.657051: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10658560 totalling 10.16MiB
2020-11-04 16:53:37.657059: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10850048 totalling 10.35MiB
2020-11-04 16:53:37.657066: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10895872 totalling 10.39MiB
2020-11-04 16:53:37.657073: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 10896384 totalling 10.39MiB
2020-11-04 16:53:37.657080: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11045120 totalling 10.53MiB
2020-11-04 16:53:37.657087: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11188224 totalling 10.67MiB
2020-11-04 16:53:37.657093: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 11581440 totalling 11.04MiB
2020-11-04 16:53:37.657100: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 19 Chunks of size 11796480 totalling 213.75MiB
2020-11-04 16:53:37.657108: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12137472 totalling 11.58MiB
2020-11-04 16:53:37.657115: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12404736 totalling 11.83MiB
2020-11-04 16:53:37.657122: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 12939264 totalling 12.34MiB
2020-11-04 16:53:37.657129: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 13547520 totalling 25.84MiB
2020-11-04 16:53:37.657136: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13621248 totalling 12.99MiB
2020-11-04 16:53:37.657144: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13785856 totalling 13.15MiB
2020-11-04 16:53:37.657151: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 13865984 totalling 13.22MiB
2020-11-04 16:53:37.657160: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 14155776 totalling 40.50MiB
2020-11-04 16:53:37.657164: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 14304768 totalling 13.64MiB
2020-11-04 16:53:37.657168: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 14649600 totalling 13.97MiB
2020-11-04 16:53:37.657172: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 35 Chunks of size 15360000 totalling 512.70MiB
2020-11-04 16:53:37.657176: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 3 Chunks of size 15906816 totalling 45.51MiB
2020-11-04 16:53:37.657181: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 17 Chunks of size 16515072 totalling 267.75MiB
2020-11-04 16:53:37.657186: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 16575488 totalling 15.81MiB
2020-11-04 16:53:37.657194: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17123328 totalling 16.33MiB
2020-11-04 16:53:37.657198: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 17658112 totalling 16.84MiB
2020-11-04 16:53:37.657202: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18265088 totalling 17.42MiB
2020-11-04 16:53:37.657206: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18265856 totalling 17.42MiB
2020-11-04 16:53:37.657211: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 31 Chunks of size 18266112 totalling 540.02MiB
2020-11-04 16:53:37.657215: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 2 Chunks of size 18266368 totalling 34.84MiB
2020-11-04 16:53:37.657219: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18315776 totalling 17.47MiB
2020-11-04 16:53:37.657223: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18501888 totalling 17.64MiB
2020-11-04 16:53:37.657229: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18659328 totalling 17.79MiB
2020-11-04 16:53:37.657234: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 18659584 totalling 17.79MiB
2020-11-04 16:53:37.657240: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 24899584 totalling 23.75MiB
2020-11-04 16:53:37.657245: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 44 Chunks of size 39321600 totalling 1.61GiB
2020-11-04 16:53:37.657248: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 28 Chunks of size 93763584 totalling 2.44GiB
2020-11-04 16:53:37.657252: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 95564544 totalling 91.14MiB
2020-11-04 16:53:37.657256: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 99225600 totalling 94.63MiB
2020-11-04 16:53:37.657260: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 102592512 totalling 97.84MiB
2020-11-04 16:53:37.657264: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 103200768 totalling 98.42MiB
2020-11-04 16:53:37.657268: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 122075136 totalling 116.42MiB
2020-11-04 16:53:37.657273: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 143917056 totalling 137.25MiB
2020-11-04 16:53:37.657278: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 173402880 totalling 165.37MiB
2020-11-04 16:53:37.657283: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 174149120 totalling 166.08MiB
2020-11-04 16:53:37.657287: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 184383744 totalling 175.84MiB
2020-11-04 16:53:37.657291: I tensorflow/core/common_runtime/bfc_allocator.cc:1034] 1 Chunks of size 184633344 totalling 176.08MiB
2020-11-04 16:53:37.657295: I tensorflow/core/common_runtime/bfc_allocator.cc:1038] Sum Total of in-use chunks: 19.95GiB
2020-11-04 16:53:37.657299: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] total_region_allocated_bytes_: 21468487680 memory_limit_: 21468487808 available bytes: 128 curr_region_allocation_byt
es_: 42936975872
2020-11-04 16:53:37.657313: I tensorflow/core/common_runtime/bfc_allocator.cc:1046] Stats:
Limit:                     21468487808
InUse:                     21422771968
MaxInUse:                  21432602368
NumAllocs:                    46783782MaxAllocSize:                187526912Reserved:                            0
PeakReserved:                        0
LargestFreeBlock:                    0

2020-11-04 16:53:37.657683: W tensorflow/core/common_runtime/bfc_allocator.cc:439] ****************************************************************************************************
2020-11-04 16:53:37.657727: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at matmul_op.cc:481 : Resource exhausted: OOM when allocating tensor with shape[3200,3072] and type float o
n /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
  File "myLib/python3.7/site-packages/kerastuner/engine/base_tuner.py", line 131, in search
    self.run_trial(trial, *fit_args, **fit_kwargs)
  File "myLib/python3.7/site-packages/kerastuner/engine/multi_execution_tuner.py", line 98, in run_trial
    history = model.fit(*fit_args, **copied_fit_kwargs)
  File "myLib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 108, in _method_wrapper
    return method(self, *args, **kwargs)
  File "myLib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1098, in fit
    tmp_logs = train_function(iterator)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 780, in __call__
    result = self._call(*args, **kwds)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 840, in _call
    return self._stateless_fn(*args, **kwds)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2829, in __call__
    return graph_function._filtered_call(args, kwargs)  # pylint: disable=protected-access
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1848, in _filtered_call
    cancellation_manager=cancellation_manager)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1924, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "myLib/python3.7/site-packages/tensorflow/python/eager/function.py", line 550, in call
    ctx=ctx)
  File "myLib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
  (0) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

         [[clip_by_norm_1/truediv/_546]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

  (1) Resource exhausted:  OOM when allocating tensor with shape[3200,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc         [[node BERT_MultiLabel_MultiClass/tf_bert_model/bert/encoder/layer_._11/intermediate/dense/Tensordot/MatMul (defined at myLib/python3.7/site-packages/transformers/modeling_tf_bert.py:327) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_1394299]

Function call stack:
train_function -> train_function
```


Exception: process 0 terminated with exit code 17 when using xla_spawn

Hi
I am calling finetune_trainer with xla_spawn command, on TPU, I am getting this error on wmt, any idea on this? thanks

```
Traceback (most recent call last):
  File "/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workdir/seq2seq/xla_spawn.py", line 113, in <module>
    app.run(main, flags_parser=parse_args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "/workdir/seq2seq/xla_spawn.py", line 108, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=xla_args.num_cores)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 395, in spawn
    start_method=start_method)
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/root/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 112, in join
    (error_index, exitcode)
Exception: process 0 terminated with exit code 17
```


How can we freeze the last few layers of a  BERT model using tf 2.0(or higher)?
I just want to take the pre-trained layers and train the last few layers for fine-tuning on my own data in min time.


Fix path to old run_language_modeling.py script

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Speedup doc build

# What does this PR do?

This PR speeds the doc build by pinning the version of sphinx to 3.2.1


adding model cards for distilled models

model card

Related to pretrained model cards

# Model cards for distilled models

As discussed on Slack, a bunch of model cards for distilled models (at least the ones I contributed to).

cc @julien-c

I never know which tags are auto-generated, so please correct me if I did something useless!


Model card: T5-base fine-tuned on QASC

Related to pretrained model cards


Fix validation file loading in scripts

# What does this PR do?

As pointed out in #8295, the validation file was not properly loaded in all the examples scripts (one typo copy-pasted several times). This PR fixes that.

<!-- Remove if not applicable -->

Fixes #8295


[s2s] 1 GPU test for run_distributed_eval

Help wanted

Extra attention is needed, help appreciated

Add test coverage for run_distributed_eval.py that can run on 1 GPU.
The command:
```bash
python -m torch.distributed.launch --nproc_per_node=1 run_distributed_eval.py --model_name Helsinki-NLP/opus-mt-en-ro --save_dir opus_wmt_en_ro_gens --data_dir wmt_en_ro
```

works on 1 GPU.

After adding test coverage, we could try to improve API consistency between run_distributed_eval.py and run_eval.py .


Update README.md

model card

Related to pretrained model cards

fix website address

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Validation data in `run_mlm.py` is the same as train data

Inspecting the script I found the following:

https://github.com/huggingface/transformers/blob/cb966e640b8b9d0f6e9c06c1655d078a917e5196/examples/language-modeling/run_mlm.py#L204

Am I missing something? Otherwise, I could send a PR.


pipelines: Tentative fix for AutoModel for PegasusConfig.

# What does this PR do?

Original error lies in `pipeline(task='summarization',
model='google/pegasus-xsum')`.
 - Code fails while trying to infer framework from model_name (str).
 - It attempts to determine framework by running `AutoModel.from_pretrained(..)` then  `TFAutoModel.from_pretrained(...)` and decides by seeing whichever works first.

Proposed fix by:
-  implementing `AutoModel.from_pretrained('google/pegasus-xsum')` that is a
`PegasusConfig`. and returning a `PegasusForConditionalGeneration`.

Not sure if that's desirable as we are loading a
`ForConditionalGeneration` model by default (but it's the only available
anyway).

Other options that are available:

- load `BartModel` (Pegasus inherits from BartForConditionalGeneration)
from `PegasusConfig`, but unsure about side effects and odd to load
`Bart` from `Pegasus`.
- Change `get_framework` function from pipeline. That was my initial
choice but it seems understanding if a config is for a TF or Pytorch
model would require replicating some of `AutoModel` logic anyway
so doing that would lead to a discrepancy between the 2 code paths just
for Pegasus (and maybe BartConfig which also suffers some issues, but
that will be in a follow-up PR).

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@LysandreJik
@sshleifer

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


[Generate Test] fix greedy generate test

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

`greedy_search` test was flaky. This PR should fix it.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Fine Tune Bert Ner using TFBertForTokenClassification.from_pretrained

Hey,

I am new to the transformers Bert Training world and am trying to fine tune Bert model for NER on a coNll like dataset. But its not training the model and giving the below error

ValueError: No gradients provided for any variable: ['tf_bert_for_token_classification_8/classifier/kernel:0', 'tf_bert_for_token_classification_8/classifier/bias:0'].

Below is my code

```py
tr_inputs = tf.convert_to_tensor(tr_inputs)
val_inputs = tf.convert_to_tensor(val_inputs)
tr_tags = tf.convert_to_tensor(tr_tags)
val_tags = tf.convert_to_tensor(val_tags)
tr_masks = tf.convert_to_tensor(tr_masks)
val_masks = tf.convert_to_tensor(val_masks)
tr_segs = tf.convert_to_tensor(tr_segs)
val_segs = tf.convert_to_tensor(val_segs)

input_features_dict = {"input_ids":tr_inputs, "attention_mask":tr_masks, "token_type_ids":tr_segs, 'labels':tr_tags}
val_features_dict = {"input_ids":val_inputs, "attention_mask":val_masks, "token_type_ids":val_segs, 'labels':tr_tags}
train_data = tf.data.Dataset.from_tensor_slices(input_features_dict)
batch_train_data = train_data.batch(batch_num)
valid_data = tf.data.Dataset.from_tensor_slices(val_features_dict)
batch_valid_data = valid_data.batch(batch_num)
modell = TFBertForTokenClassification.from_pretrained('bert-base-uncased',num_labels=len(tag2idx))
modell.layers[2].activation = tf.keras.activations.softmax
modell.layers[0].trainable = False
modell.compile(optimizer=optimizer, loss=loss, metrics=[metrics])
modell.fit(batch_train_data, epochs=epochs, validation_data=batch_val_data)
```

Not sure what needs to be done. Any advice/pointers on this would be highly helpful for me.


could you please give me a torch example of xlm-roberta-(base/large) for multilingual-text question?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
I read the document of [XLM-RoBERTa](https://huggingface.co/transformers/model_doc/xlmroberta.html#overview), but i also have a lots of question. For example, `should be able to determine the correct language from the input ids.` how to determine?

If you have a example for how to use xlm-roberta about multilingual text question, please show me, Thank you very much!


finetuning T5 on translation on TPU, questions about clarifying the setup

Hi,
I'd like to run finetune.py with wmt dataset in TPU to train from scratch not finetune. I appreciate the response to some questions:
1) Why there are two versions of fine-tuning using Seq2SeqTrainer and finetune.py and which one is suitable for my usecase?
2) Seq2SeqTrainer does not support predict on TPU, is this the case for finetune.py as well?
3) when running on TPU, it is written to use  xla_spawn.py, since  the codes of finetune.py are with pytorch lightening, is this needed to launch them with xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)?
4) in the finetune.py dataloader, I see this is distributed based on number of gpus, but I cannot see this is distributed also when one uses tpus, Is this taking care of making dataloader distributed in case of using TPU automatically?
5) If using finetune.py with TPU, is there any specific setup I need to add for finetuning/training T5 on WMT?
6) I assume one need to use maybe something like this as a sampler for TPU distributed dataloaders, I see this is not the case in the codes of finetune.py, does data parallelism works in finetune.py?
7) when to use sortish_sampler/dynamic_sampler, do they work on TPUs?
```
  sampler = torch.utils.data.distributed.DistributedSampler(
        dataset,
        num_replicas=xm.xrt_world_size(),
        rank=xm.get_ordinal(),
        shuffle=True)

```
8. fp16 does it work with TPUs as well?
9. with iterable datasets seems dataloader in finetune_trainer is not working with distributed detup on TPU, do you know how to implement it? am I missing something? thanks
thank you very much.


Why do I use XLMRobertaTokenizer and return an error on token_type_ids?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
```python
encoded_pair = self.tokenizer(sent_ko, sent_cn,
                                      padding='max_length',  # Pad to max_length
                                      truncation=True,       # Truncate to max_length
                                      max_length=self.maxlen,
                                      return_tensors='pt')  # Return torch.Tensor objects
token_ids = encoded_pair['input_ids'].squeeze(0)
attn_masks = encoded_pair['attention_mask'].squeeze(0)
token_type_ids = encoded_pair['token_type_ids'].squeeze(0)
```
```
File "/usr/local/lib/python3.6/dist-packages/transformers/tokenization_utils_base.py", line 234, in __getitem__
    return self.data[item]
KeyError: 'token_type_ids'
```


Training T5-large model for Question Answering

Are there are any specific documents that I can follow, to do the training of the t5 model for Question answering?
I found this (https://huggingface.co/transformers/custom_datasets.html#qa-squad) on your website and it is not allowing me to use t5 model instead of DistilBert.


Fix typo in language-modeling README.md

# What does this PR do?

Fix the typo in `README.md` in the `language-modeling` folder.

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Improve QA pipeline error handling

# What does this PR do?

- The issue is that with previous code we would have the following:

```python
qa_pipeline = (...)
qa_pipeline(question="Where was he born ?", context="")
-> IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
```

The goal here is to improve this to actually return a ValueError
wherever possible.

While at it, I tried to simplify QuestionArgumentHandler's code to
make it smaller and more compat while keeping backward compat.

Quick note: For the tests, I feel they would be more readable if it was possible to write

```python
self.assertEqual(qa(.....), [SquadExample(None, Q, C, None, None,...)])
```
as it would cover both types, and length and deep equality.
However, it's not possible because SquadExample does not implement `__eq__`. It felt out of scope, but
if reviewers think it would be a nice addition, I'd be happy to implement it and change the test.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@mfuntowicz
@LysandreJik

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


RAG performance on Open-NQ dataset much lower than expected

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
One peculiar finding is that when we ran the rag-sequence-nq model along with the provided wiki_dpr index, all models and index files were used as is, on the open-NQ test split (3610 questions, https://github.com/google-research-datasets/natural-questions/tree/master/nq_open), we observed EM=27.2 performance, which was rather different from that in the paper, namely EM=44.5.

We are baffled. Has anyone seen lower performance using the transformers RAG models?

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


[rag] missing a working End-to-end evaluation example

Feature request

Request for a new feature

I'm going to try to write tests for `examples/rag` (https://github.com/huggingface/transformers/issues/7715), but first I'm trying to figure out how it works.

Would it be possible to add a full `End-to-end evaluation` invocation example in https://github.com/huggingface/transformers/blob/master/examples/rag/README.md#end-to-end-evaluation? i.e. with the correct data.

I tested https://github.com/huggingface/transformers/blob/master/examples/rag/README.md#retrieval-evaluation and it worked, but if I try to adapt the same params for e2e it crashes with:
```
$ python eval_rag.py --model_name_or_path facebook/rag-sequence-nq --model_type rag_sequence \
--evaluation_set output/biencoder-nq-dev.questions --gold_data_path output/biencoder-nq-dev.pages \
--predictions_path output/retrieval_preds.tsv --eval_mode e2e --gold_data_mode qa --n_docs 5 \
--print_predictions
2020-11-03 22:07:33.124277: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
INFO:__main__:Evaluate the following checkpoints: ['facebook/rag-sequence-nq']
INFO:__main__:Calculating metrics based on an existing predictions file: output/retrieval_preds.tsv
Traceback (most recent call last):
  File "eval_rag.py", line 314, in <module>
    main(args)
  File "eval_rag.py", line 280, in main
    score_fn(args, args.predictions_path, args.gold_data_path)
  File "eval_rag.py", line 46, in get_scores
    data = pd.read_csv(gold_data_path, sep="\t", header=None)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 458, in _read
    data = parser.read(nrows)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 1196, in read
    ret = self._engine.read(nrows)
  File "/home/stas/anaconda3/envs/main-38/lib/python3.8/site-packages/pandas/io/parsers.py", line 2155, in read
    data = self._reader.read(nrows)
  File "pandas/_libs/parsers.pyx", line 847, in pandas._libs.parsers.TextReader.read
  File "pandas/_libs/parsers.pyx", line 862, in pandas._libs.parsers.TextReader._read_low_memory
  File "pandas/_libs/parsers.pyx", line 918, in pandas._libs.parsers.TextReader._read_rows
  File "pandas/_libs/parsers.pyx", line 905, in pandas._libs.parsers.TextReader._tokenize_rows
  File "pandas/_libs/parsers.pyx", line 2042, in pandas._libs.parsers.raise_parser_error
pandas.errors.ParserError: Error tokenizing data. C error: Expected 5 fields in line 2, saw 6
```

I think it needs a different input data.

And we need 2 functional examples: for `qa` and `ans` each.

I can handle adding this to the doc if you tell me what to add.

Thanks.

@patrickvonplaten, @lhoestq


[tokenizers] convert_to_tensors: don't reconvert when the type is already right

I was trying to fix this warning:
```
src/transformers/tokenization_utils_base.py:608: UserWarning: To copy construct from a tensor,
it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True),
rather than torch.tensor(sourceTensor).
  tensor = as_tensor(value)
```
which appeared when running:
```
python eval_rag.py --model_name_or_path facebook/rag-sequence-nq --model_type rag_sequence --evaluation_set output/biencoder-nq-dev.questions --gold_data_path output/biencoder-nq-dev.pages --predictions_path output/retrieval_preds.tsv --eval_mode retrieval --k 1
```

This appears to have happened since `convert_to_tensors` was called with data which was already a tensor of the right type.

* [x] and ended up fixing it for pt and also adding the same fix for tf/jax/np. basically skip the conversion if the value is already of the required type and avoid the pytorch warning.
* [x] added tests for converting the already converted
* [x] while at it added a missing test for `test_batch_encoding_with_labels_jax`

I understand `lambda` isn't welcome, so I had to define a few helper functions for numpy/jax. `partial` would have done the trick, but `isinstance` doesn't accept keyword args.

@LysandreJik, @mfuntowicz


[blenderbot] regex fix

This PR fixes:

```
src/transformers/tokenization_blenderbot.py:163: DeprecationWarning: invalid escape sequence \s
    token = re.sub("\s{2,}", " ", token)
```

@LysandreJik


Create README.md

model card

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Translation finetuning error : TypeError: '>' not supported between instances of 'function' and 'int'

Dear huggingface team,
I'd like to train from scratch T5 on wmt19 (de-en), and I see these instructions in your page:
- you provided the script for  finetune mbart_cc25, could I just change the model path and it works out of the box for training T5 on a translation task? any changes needed?
- when you use sortish sampler (line 256 finetune.py) you check the number of gpus, in case using tpus, shall I check the number of cores of tpus for distributed version of dataloader in line 256?
- does distributed tpu training works for seq2seq model? I wonder why the dataloader is not modified for tpu cores, is this by purpose and works fine for tpus too?
- I also gets these errors running the provided script, thank you for your help.
Best
Rabeeh

```
(test) rabeeh@brain1:~/ruse/hf/transformers/examples/seq2seq$ ./train_mbart_cc25_enro.sh --output_dir enro_finetune_baseline --label_smoothing 0.1 --fp16_opt_level=O1 --logger_name wandb --sortish_sampler
2020-11-04 01:41:53.720772: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64
2020-11-04 01:41:53.720823: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "finetune.py", line 442, in <module>
    main(args)
  File "finetune.py", line 383, in main
    model: SummarizationModule = TranslationModule(args)
  File "finetune.py", line 367, in __init__
    super().__init__(hparams, **kwargs)
  File "finetune.py", line 57, in __init__
    if hparams.sortish_sampler and hparams.gpus > 1:
TypeError: '>' not supported between instances of 'function' and 'int'

```


Finetuning T5 on translation wmt19(de-en)

Dear huggingface team,
I'd like to train from scratch T5 on wmt19 (de-en), and I see these instructions in your page:
you provided the script for  finetune mbart_cc25, could I just change the model path and it works out of the box for training T5 on a translation task?
I also gets these errors running the provided script, thank you for your help.
Best
Rabeeh

```
(test) rabeeh@brain1:~/ruse/hf/transformers/examples/seq2seq$ ./train_mbart_cc25_enro.sh --output_dir enro_finetune_baseline --label_smoothing 0.1 --fp16_opt_level=O1 --logger_name wandb --sortish_sampler
2020-11-04 01:41:53.720772: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64
2020-11-04 01:41:53.720823: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Traceback (most recent call last):
  File "finetune.py", line 442, in <module>
    main(args)
  File "finetune.py", line 383, in main
    model: SummarizationModule = TranslationModule(args)
  File "finetune.py", line 367, in __init__
    super().__init__(hparams, **kwargs)
  File "finetune.py", line 57, in __init__
    if hparams.sortish_sampler and hparams.gpus > 1:
TypeError: '>' not supported between instances of 'function' and 'int'

```


[commit #29b536a]AttributeError: module 'numpy.random' has no attribute 'Generator'

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: commit #29b536a
- Platform: Linux
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?): N/A
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes

### Who can help
I don't know, anyone?

## Information

The problem arises when using:
```
import transformers
```

The tasks I am working on is:
(ANY)

## To reproduce

Steps to reproduce the behavior:

```
import transformers
```

## Error message
```
File "/home01/a1204a01/.local/bin/transformers-cli", line 6, in <module>
    from transformers.commands.transformers_cli import main
  File "/home01/a1204a01/.local/lib/python3.7/site-packages/transformers/__init__.py", line 22, in <module>
    from .integrations import (  # isort:skip
  File "/home01/a1204a01/.local/lib/python3.7/site-packages/transformers/integrations.py", line 81, in <module>
    from .file_utils import is_torch_tpu_available  # noqa: E402
  File "/home01/a1204a01/.local/lib/python3.7/site-packages/transformers/file_utils.py", line 87, in <module>
    import datasets  # noqa: F401
  File "/home01/a1204a01/.local/lib/python3.7/site-packages/datasets/__init__.py", line 27, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/home01/a1204a01/.local/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 175, in <module>
    class Dataset(DatasetInfoMixin, IndexableMixin):
  File "/home01/a1204a01/.local/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1889, in Dataset
    new_fingerprint: Optional[str] = None,
AttributeError: module 'numpy.random' has no attribute 'Generator'
```

## Expected behavior

Import


SqueezeBert does not appear to properly generate text

## Environment info
Google Colab
Using CPU with High Ram

### Who can help
 @sgugger @forresti @LysandreJik

## Information
Model I am using: Squeezebert-uncased, squeezebert-mnli, etc.

The problem arises when using:

Trying to generate the likely output of the input sequence and predicting masked tokens.

## To reproduce

```

from torch import nn
from transformers import AutoModelForMaskedLM, AutoTokenizer
model = AutoModelForMaskedLM.from_pretrained('squeezebert/squeezebert-mnli')
tokenizer = AutoTokenizer.from_pretrained('squeezebert/squeezebert-mnli')
#model.tie_weights()
input_txt = ["[MASK] was an American [MASK]  and lawyer who served as the 16th president  of the United States from 1861 to 1865. [MASK] led the nation through the American Civil War, the country's greatest [MASK], [MASK], and [MASK] crisis. ", \
             "George [MASK], who served as the first  president of the United States from [MASK] to 1797, was an American political leader, [MASK] [MASK], statesman, and Founding Father. Previously, he led Patriot forces to [MASK] in the nation's War for Independence. ", \
             "[MASK], the first African-American [MASK] of the [MASK] [MASK], is an American politician and attorney who served as the 44th [MASK] of the United States from [MASK] to 2017.  [MASK] was a member of the [MASK] [MASK]. "]
#input_txt =
input_txt= [i.replace("[MASK]", tokenizer.mask_token) for i in input_txt] #
inputs = tokenizer(input_txt, return_tensors='pt', add_special_tokens=True, padding=True)
inputs['output_attentions'] = True
inputs['output_hidden_states'] = True
inputs['return_dict'] = True
outputs = model(**inputs)
if True:
  predictions = outputs.logits
  for pred in predictions:
    print ("**")
    sorted_preds, sorted_idx = pred.sort(dim=-1, descending=True)
    for k in range(2):
        predicted_index = [sorted_idx[i, k].item() for i in range(0,len(predictions[0]))]
        predicted_token = ' '.join([tokenizer.convert_ids_to_tokens([predicted_index[x]])[0] for x in range(1,len(predictions[0]))]).replace('Ġ', ' ').replace('  ', ' ').replace('##', '')
        print(predicted_token)

```
## Expected behavior

I expected at least the input to be echoed out, with the slots filling with Lincoln, Washington and Obama. This works for bert, distlbert, roberta, etc.

## Actual output

Some weights of the model checkpoint at squeezebert/squeezebert-mnli were not used when initializing SqueezeBertForMaskedLM: ['classifier.weight', 'classifier.bias']
- This IS expected if you are initializing SqueezeBertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing SqueezeBertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of SqueezeBertForMaskedLM were not initialized from the model checkpoint at squeezebert/squeezebert-mnli and are newly initialized: ['lm_head.weight', 'lm_head.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
odict_keys(['logits', 'hidden_states', 'attentions'])
**
tone lani rce soto olar rce ux wer lani bal bal vus novice rce rce rce lani owe frey owe gent naire tres che lani lani nae ui territories accusing oaks accusing ois lor resulting resulting rce lor rce rendering rce rce tres assist ois accusing rendering warns accusing gent
culture bowls hectares awan rce bal ade wd an rce mole hoe yde lani lani lani rce tres resulted bal resulted resulting tone consequently bowls fellow wo ois crafts oaks withdrew nations wu resulting fellow rce resulting verses motivated lori motivated motivated gent vus naire dealt warns gent warns tres
**
culture sas hari lani rce gaa lani novice rce rce rce rce tres nae jan thal rce rce rce awan olar v8 rce olar example rce select rce rce hore rden resulting lori resulting drive led bon peoples jal gau nae hoe lies lies lies lies lins lies resulting tone
continuum tone repeat gaa lani wo rce coven lani lani lani lani gle aw aw awan sco lani yde rce yde olar ux rce rce trait xie xie cao particular elder lani lani naturally blend lie aman commando folding rendering helps ois lete wi lins lins hoe independence sons tones
**
tone acts attribute trait pour pour trait % sities ub azi % acts lani rce awan act cao yde wd hoe hoe hoe hoe % vos vos rce hort hoe sept jan vers naire hum candle therefore lists chen hoe lie side mut hen mor lungs zoo lie side side
hum fever acts pour shropshire cz % sities isson penalties lie sities act acts bble pour yde ave shropshire yde lto ango ango pour lden rce hoe gil hoe tres aw nae dha therefore bisexual therefore lb mates rden too zoo forum naire dealt lag mole mess pore forum ior


Support various BERT relative position embeddings (2nd)

Related to pretrained model cards

# What does this PR do?

Creating a new PR for https://github.com/huggingface/transformers/pull/8108 to keep cleaner git history/commits.

The default BERT model `bert-base-uncased` was pre-trained with absolute position embeddings. We provide three pre-trained models which were pre-trained on the same training data (BooksCorpus and English Wikipedia) as in the BERT model training, but with different relative position embeddings (Shaw et al., Self-Attention with Relative Position Representations, https://arxiv.org/abs/1803.02155 and Huang et al., Improve Transformer Models with Better Relative Position Embeddings, https://arxiv.org/abs/2009.13658, accepted in findings of EMNLP 2020). We show how to fine-tune these pre-trained models on SQuAD1.1 data set. Our proposed relative position embedding method can boost the BERT base model (with default absolute position embedding) from f1 score of 88.52 to 90.54 with similar training/inference speed. It also boosts the `bert-large-uncased-whole-word-masking` model from 93.15 to 93.52 with 3 additional fine-tune epochs. See examples/question-answering/README.md for more details.

Fixes # (issue)

#8108

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten @LysandreJik @julien-c


[CIs] Better reports everywhere

Continuing the work in https://github.com/huggingface/transformers/pull/8110 and https://github.com/huggingface/transformers/pull/8163 this PR does the following:

* [x] rename `pytest --make_reports` to `pytest --make-reports` for consistency with the rest of `pytest` opts that don't use `_`
* [x] move the `--make_reports` opt adding to a shared location and load it only once to avoid `pytest` failure
   - some pytest plugins like `pytest-instafail` load `tests/conftest.py` even when running `examples`
   - now we can run tests from both test suites at once
* [x] rename `reports/report_foo` to `reports/foo` - avoid repetition
* [x] install `--make_reports` in all CIs: circleci and github actions
* [x] make the reports available via artifacts
* [x] always cat short failure reports for github actions in its own "tab" since getting to artifacts there is a cumbersome process. I'm not sure this is needed in circleci jobs since each report in artifacts is available in 2 clicks, so I left the `cat *failures_short.txt` out on CircleCI jobs.
* [x] fixed a few issues in the github actions job configuration

@sgugger, @LysandreJik, @sshleifer


Data collator for token classification

# What does this PR do?

This PR adds a `DataCollatorForTokenClassification`, very similar to `DataCollatorWithPadding` but whose job is to pad the labels to the same size as the inputs.
In passing, it adds tests of `DataCollatorWithPadding` and cleans all the tests of various data collators that were marked as slow because they required a pretrained tokenizer. For the unit testing, no real tokenizer is needed since we just need the pas/mask token.


add evaluate doc - trainer.evaluate returns 'epoch' from training

Improved documentation - see #8184


Saving and reloading DistilBertForTokenClassification fine-tuned model

I am trying to reload a fine-tuned DistilBertForTokenClassification model. I am using transformers 3.4.0 and pytorch version 1.6.0+cu101. After using the Trainer to train the downloaded model, I save the model with trainer.save_model() and during my trouble shooting I saved the model in a **different** directory via model.save_pretrained(). I am using Google Colab and saving the model to my Google drive. Before closing out my session, I evaluated the model and got good test results, however, when I return to the notebook (or Factory restart the colab notebook) and try to reload the model, the predictions are terrible. Upon checking the both directories, the config.json file is there as is the pytorch_mode.bin. It seems the trained model is not getting saved in this directories, but rather just the original model is? The model will work just fine if I don't close out my notebook session, but upon returning (or factory resetting) the reloading of the model yields a model that does not give good predictions. Is the trained model getting saved in a cache temporarily? But the save_model() function saves the original model?

```
from transformers import DistilBertForTokenClassification

# load the pretrained model from huggingface
model = DistilBertForTokenClassification.from_pretrained('distilbert-base-uncased', num_labels=len(uniq_labels))

model.to('cuda');

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir = model_dir +  'mitmovie_pt_distilbert_uncased/results',          # output directory
    #overwrite_output_dir = True,
    evaluation_strategy='epoch',
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir = model_dir +  'mitmovie_pt_distilbert_uncased/logs',            # directory for storing logs
    logging_steps=10,
    load_best_model_at_end = True
)

trainer = Trainer(
    model = model,                         # the instantiated 🤗 Transformers model to be trained
    args = training_args,                  # training arguments, defined above
    train_dataset = train_dataset,         # training dataset
    eval_dataset = test_dataset             # evaluation dataset
)

trainer.train()

trainer.evaluate()

model_dir = '/content/drive/My Drive/Colab Notebooks/models/'
trainer.save_model(model_dir + 'mitmovie_pt_distilbert_uncased/model')

# alternative saving method and folder
model.save_pretrained(model_dir + 'distilbert_testing')
```

Coming back to the notebook after restarting...

```from transformers import DistilBertForTokenClassification, DistilBertConfig, AutoModelForTokenClassification

# retreive the saved model
model = DistilBertForTokenClassification.from_pretrained(model_dir + 'mitmovie_pt_distilbert_uncased/model',
                                                        local_files_only=True)

model.to('cuda')
```
Model predictions are now terrible loading the model from either of the directories.


Low accuracy after load custom pretrained model in a text binary classification problem

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.4.0
- Platform: Linux-4.15.0-122-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: Distributed (not really sure)

### Who can help
@LysandreJik

## Information
Posted in StackOverflow. Received a comment with two similar issues regarding save and load custom models. The original question can be found at: https://stackoverflow.com/questions/64666510/huggingface-transformers-low-accuracy-after-load-custom-pretrained-model-in-a-t?noredirect=1#comment114344159_64666510

In a nutshell I am using BertForSequenceClassification (PyTorch) with ```dccuchile/bert-base-spanish-wwm-cased``` for solving a binary classification problem. I have trained the network and evaluate the model with a testing dataset (different from the training dataset). I have achieved an ```acc``` and ```val_acc``` between 0.85 and 0.9. However, after I save the model and retrieve it again in another script, the accuracy is similar to a random classifier (0.41).

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce
This is the code I am using for training and evaluating (during training):
```
criterion = torch.nn.CrossEntropyLoss ()
criterion = criterion.to (device)
optimizer = AdamW (model.parameters(), lr=5e-5)

for epoch in range (4):

    i = 0

    # Train this epoch
    model.train ()
    for batch in train_loader:
        optimizer.zero_grad ()
        input_ids = batch['input_ids'].to (device)
        attention_mask = batch['attention_mask'].to (device)
        labels = batch['label'].to (device)

        loss, _ = model (input_ids, attention_mask=attention_mask, labels=labels)
        _, preds = torch.max (_, dim=1)
        correct_predictions += torch.sum (preds == labels)
        i += 1
        acc = correct_predictions.item ()  / (batch_size * i)

        loss.backward ()
        optimizer.step ()

    # Eval this epoch with the testing dataset
    model = model.eval ()
    correct_predictions = 0
    with torch.no_grad ():
        for batch in test_loader:
            input_ids = batch['input_ids'].to (device)
            attention_mask = batch['attention_mask'].to (device)
            labels = batch['label'].to (device)

            loss, _ = model (input_ids, attention_mask=attention_mask, labels=labels)
            _, preds = torch.max (_, dim=1)
            correct_predictions += torch.sum (preds == labels)

model.bert.save_pretrained ("my-model")
tokenizer.save_pretrained ("my-model")
```

After this step, I got good accuracy after the first epoch

Then, I load the model again in another script
```
model = BertForSequenceClassification.from_pretrained ("my-model")

# Eval this epoch with the testing dataset
model = model.eval ()
correct_predictions = 0
with torch.no_grad ():
    for batch in test_loader:
        input_ids = batch['input_ids'].to (device)
        attention_mask = batch['attention_mask'].to (device)
        labels = batch['label'].to (device)

        loss, _ = model (input_ids, attention_mask=attention_mask, labels=labels)
        _, preds = torch.max (_, dim=1)
        correct_predictions += torch.sum (preds == labels)

print (correct_predictions.item () / len (test_df))
```

but the accuracy is similar as If I retrieved a non-trained model.

## Expected behavior
After load a model saved with ```save_pretrained```, the model should provide similar accuracy and loss for the same data.


improve documentation of training_args.py

Documentation for the following fields has been improved:

- do_train
- do_eval
- do_predict

Also see #8179


[wip/s2s/pl] attempt to sync metrics in DDP

This is broken.
Attempted to add
`AverageMetric` where you just dump python floats and they get averaged and the end, but not working on DDP.

### Failing command
(fails quickly at val sanity check)

```bash
cd examples/seq2seq
wget https://cdn-datasets.huggingface.co/translation/wmt_en_ro.tar.gz
tar -xzvf wmt_en_ro.tar.gz
export WANDB_PROJECT=dmar
export BS=64
export m=sshleifer/mar_enro_6_3_student
export MAX_LEN=128
python finetune.py \
  --learning_rate=3e-4 \
  --do_train \
  --do_predict \
  --fp16 \
  --val_check_interval 0.25 \
  --data_dir wmt_en_ro \
  --max_source_length $MAX_LEN --max_target_length $MAX_LEN --val_max_target_length $MAX_LEN --test_max_target_length $MAX_LEN \
  --freeze_encoder --freeze_embeds \
  --train_batch_size=$BS --eval_batch_size=$BS \
  --tokenizer_name Helsinki-NLP/opus-mt-en-ro --model_name_or_path $m \
  --warmup_steps 500 --sortish_sampler --logger_name wandb \
  --gpus 2 --fp16_opt_level=O1 --task translation --num_sanity_val_steps=1 --output_dir dmar_met_test_2gpu \
  --num_train_epochs=2 --overwrite_output_dir
```

### Traceback

```bash
  File "/home/shleifer/transformers_fork/examples/seq2seq/finetune.py", line 206, in <dictcomp>
    pl_metrics = {f"pl_{prefix}_avg_{k}": v.compute().item() for k, v in self.metric_stores.items()}
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/metrics/metric.py", line 214, in wrapped_func
    self._sync_dist()
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/metrics/metric.py", line 177, in _sync_dist
    output_dict = apply_to_collection(
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 53, in apply_to_collection
    return elem_type({k: apply_to_collection(v, dtype, function, *args, **kwargs)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 53, in <dictcomp>
    return elem_type({k: apply_to_collection(v, dtype, function, *args, **kwargs)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py", line 49, in apply_to_collection
    return function(data, *args, **kwargs)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/pytorch_lightning/utilities/distributed.py", line 100, in gather_all_tensors_if_available
    torch.distributed.all_gather(gathered_result, result, group)
  File "/home/shleifer/miniconda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 1185, in all_gather
    work = _default_pg.allgather([tensor_list], [tensor])
RuntimeError: Tensors must be CUDA and dense

```


Clean Trainer tests and datasets dep

# What does this PR do?

This PR removes the installation of datasets from master and uses the dependency already in `testing` instead. It also cleans up a bit the tests in Trainer by:
- using the decorator `requires_datasets` when needed
- using a temp dir for the output of one test, to avoid some files to be created when the user has optuna installed


[Seq2Seq] Make Seq2SeqArguments an independent file

# What does this PR do?

By putting all `Seq2SeqTrainingArguments` logic in a separate file, the `Seq2SeqTrainer` and `Seq2SeqTrainingArguments` can be used as standalone files without having to download any additional files because of other dependencies.


german medbert model details

Related to pretrained model cards

# What does this PR do?

- added details for German MedBERT

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Is there a pre-trained BERT model with the sequence length of 2048?

Hello,
I want to use the pre-trained BERT model because I do not want to train the entire BERT model to analyze my data. Is there a pre-trained BERT model with sequence length 2048?

or are all pre-trained BERT model only have the sequence length of 512?

Thank you.


New TensorFlow trainer version

Hello,

This PR is a proposal for an updated version of the current TensorFlow trainer. This new trainer brings the following improvements:

- Uses the Keras methods `.compile()` + `.fit()` instead of the custom training loop. This change brings a better integration with TensorFlow and the different strategies that can be used for training a model. It takes also advantages of all the optimizations done by the Google team for a proper training.
- Uses the Keras methods `.evaluate()` and `predict()` instead of the custom evaluation loop. Same advantages than for the training part.
- Uses the Keras callbacks and metrics. We can takes advantages of the callback and metric features proposed by default when training/evaluate a model with Keras. Also one can create its own callbacks and metrics and use them for the training/evaluation.
- Big reduction in terms of line of codes which makes it easier to maintain.
- Create a new optimizer for gradient accumulation to move the logic inside this new optimizer than inside the trainer instead.

Of course this is still far to be finished and there is still work to do but you can easily see the direction I'm thinking of.

@LysandreJik @sgugger I will be happy to ear your comments.

@thomwolf @lhoestq Here I have created a file where I put all the Keras metrics, but we should definitely think a way to integrate such metrics directly inside `datasets` where they will be better suited.


GPT2 is not jit-traceable

## Information

I would like to use Pytorch tracing on a pretrained GPT2 model, but I run into these warnings for the attention layers:
```
python3.8/site-packages/transformers/modeling_gpt2.py:164: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  w = w / (float(v.size(-1)) ** 0.5)
python3.8/site-packages/transformers/modeling_gpt2.py:169: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  mask = self.bias[:, :, ns - nd : ns, :ns]
```
The first warning concern the same line as the one reported in #3954 (and fixed by #3955).

## To reproduce

You can run the following:
```
import torch
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')
model.eval()

tokens=tokenizer('The cat is on the table.', return_tensors='pt')['input_ids']

with torch.jit.optimized_execution(True):
    traced_model = torch.jit.trace(model, tokens)
```

## Environment info

- `transformers` version: 3.4.0
- Platform: Linux-5.4.0-52-generic-x86_64-with-glibc2.29
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.0+cpu (False)
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO


[distributed testing] forward the worker stderr to the parent process

As discussed on slack, this PR:

* on distributed failure reproduces the combined `stderr` of the worker processes in the exception of the test invoking the distributed process

This is so that the CI's new optimized reports will include the full error message.

@sgugger


Encoder Decoder Model

Hi,

I am following the instructions written on the HuggingFace website to use an encoder-decoder model:

from transformers import EncoderDecoderModel, BertTokenizer
import torch

```
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints

#model.save_pretrained('/content/drive/My Drive/NLP/'+'model_1')

# forward
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)

# training
outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids, return_dict=True)
#print(type(outputs)) #Seq2SeqLMOutput
loss, logits = outputs.loss, outputs.logits

# save and load from pretrained
#model.save_pretrained("bert2bert")
#model = EncoderDecoderModel.from_pretrained("bert2bert")

# generation
generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)

generated
tensor([[   0, 1012, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010,
         1010, 1010, 1010, 1010, 1010, 1010, 1010, 1010]])
```

However, I have no idea how to decode the generated output, can anybody pls help?
Thank you


[fix] Skip tatoeba tests if Tatoeba-Challenge not cloned


Disable default sigmoid function for single label classification Inference API

# 🚀 Feature request

Allow people to disable the default sigmoid function in TextClassificationPipeline (maybe via model cards?).

## Motivation

When we use the sequence classification model (e.g. RobertaForSequenceClassification) for regression tasks, the output may have different ranges other than [0,1], it would be better to allow configurations for the sigmoid function in TextClassificationPipeline.


Create README.md

Related to pretrained model cards

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->


Create README.md


[FIX] TextGenerationPipeline is currently broken.

# What does this PR do?

It's most likely due to #8180. What's missing is a multi vs single string
handler at the beginning of the pipe. And also there was no testing of this
pipeline.

This also changes Conversational pipeline tests which seemed to have also test failures
This was linked to having state within the input that get consumed and the tests did not recreate
them so we had a stale `Conversation` object for the new test.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to the it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@thomwolf

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @sshleifer
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->