Unable to establish Lock on cached tokenizer output from RobertaTokenizer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-3.10.0-1127.13.1.el7.x86_64-x86_64-with-redhat-7.8-Maipo
- Python version: 3.7.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

 tokenizers: @mfuntowicz

## Information

Model I am using (Bert, XLNet ...): Roberta (`roberta-large-mnli`)

The problem arises when using:
* [X] the official example scripts: `run_glue.py`
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: MNLI
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. download GLUE data using official download script and place it in the root of `transformers`
2. `python run_glue.py --model_name_or_path roberta-large-mnli --data_dir ../glue_data --output_dir tmp --task_name MNLI --do_eval

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```bash
Traceback (most recent call last):
  File "./run_glue.py", line 247, in <module>
    main()
  File "./run_glue.py", line 143, in main
    if training_args.do_eval
  File "/home/USER/anaconda3/envs/nlu/lib/python3.7/site-packages/transformers/data/datasets/glue.py", line 106, in __init__
    with FileLock(lock_path):
  File "/home/USER/anaconda3/envs/nlu/lib/python3.7/site-packages/filelock.py", line 323, in __enter__
    self.acquire()
  File "/home/USER/anaconda3/envs/nlu/lib/python3.7/site-packages/filelock.py", line 271, in acquire
    self._acquire()
  File "/home/USER/anaconda3/envs/nlu/lib/python3.7/site-packages/filelock.py", line 384, in _acquire
    fd = os.open(self._lock_file, open_mode)
FileNotFoundError: [Errno 2] No such file or directory: '../glue_data/cached_dev_RobertaTokenizer_128_mnli.lock'
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

The script should be able to create the above-mentioned cached file if one doesn't exist, and acquire lock and load it if it does.


[tests] fix typos in inputs

fixes a typo in inputs and the corresponding ids


Tensorflow 2 Finetuning TF T5 using keras fit

## The Problem
I have been trying to finetune the T5 model model using tensorflow and keras. there is no documentation/ offcial+community or **notebook** for finetuning  T5 in tensorflow. There are a bunch of lines  [here](https://huggingface.co/transformers/model_doc/t5.html#tft5forconditionalgeneration) and some finetuning instructions [here](https://huggingface.co/transformers/model_doc/t5.html#tft5forconditionalgeneration) other than that there is nothing for tensorflow.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: `3.0.2`
- Platform: `Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid`
- Python version: `Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid`
- PyTorch version (GPU?): `1.6.0 (True)`
- Tensorflow version (GPU?): `2.2.0 (True)`
- Using GPU in script?: `yes`
- Using distributed or parallel set-up in script?: `no`

### Who can help
@patrickvonplaten  @jplu
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): `TFT5ForConditionalGeneration (TFAutoModelWithLMHead) pretrained`

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (SQuad from tfds)
* [ ] my own task or dataset: (give details below)

## To reproduce
```
model = TFAutoModelWithLMHead.from_pretrained("t5-base")
tokenizer = AutoTokenizer.from_pretrained("t5-base")

train_dataset, info = tfds.load('squad', split='train', with_info=True)

def encode_tf(inputs):
    """Encodes the squad inputs and uses the tokenizer to encode inputs returns
        the appropriate model 'input_ids', 'attention masks`, `decoder_attention_mask`, 'labels'
   Returns:
       dict: returns a dictionary with keys:  'input_ids', 'attention masks`, `decoder_attention_mask`,
        'labels' with appropriate tensor values
    """
    pass

dataset = train_dataset.map(encode_tf)
dataset = dataset.shuffle(1000)
dataset = dataset.batch(8)

```
### Sample data output:
```
data = next(iter(dataset))
data
```
```
{'input_ids': <tf.Tensor: shape=(8, 200), dtype=int32, numpy=
  array([[  987,   834,  7771, ...,     0,     0,     0],
         [  987,   834,  7771, ...,  2749,  3385, 12187],
         [  987,   834,  7771, ...,     0,     0,     0],
         ...,
         [  987,   834,  7771, ...,     0,     0,     0],
         [  987,   834,  7771, ...,     0,     0,     0],
         [  987,   834,  7771, ...,     6,    30,     8]], dtype=int32)>,
  'labels': <tf.Tensor: shape=(8, 200), dtype=int32, numpy=
  array([[ 363,   19,   80, ...,    0,    0,    0],
         [4504,  149,  186, ...,    0,    0,    0],
         [ 571,   54, 3298, ...,    0,    0,    0],
         ...,
         [2645, 2832, 4599, ...,    0,    0,    0],
         [ 571,  103, 7000, ...,    0,    0,    0],
         [ 366,  410,    8, ...,    0,    0,    0]], dtype=int32)>,
  'attention_mask': <tf.Tensor: shape=(8, 200), dtype=int32, numpy=
  array([[1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 1, 1, 1],
         [1, 1, 1, ..., 0, 0, 0],
         ...,
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 1, 1, 1]], dtype=int32)>,
  'decoder_attention_mask': <tf.Tensor: shape=(8, 200), dtype=int32, numpy=
  array([[1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         ...,
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0],
         [1, 1, 1, ..., 0, 0, 0]], dtype=int32)>}
```

### Training
```
optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5)
loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
model.compile(optimizer=optimizer, loss=loss)

model.fit(dataset, epochs=10)
```

model.fit result in the following error about **ValueError: No gradients provided for any variable**

### The Stacktrace
```
Epoch 1/10
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-163-f8c5e0c71664> in <module>
----> 1 model.fit(dataset, epochs=10)

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in _method_wrapper(self, *args, **kwargs)
     64   def _method_wrapper(self, *args, **kwargs):
     65     if not self._in_multi_worker_mode():  # pylint: disable=protected-access
---> 66       return method(self, *args, **kwargs)
     67
     68     # Running inside `run_distribute_coordinator` already.

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)
    846                 batch_size=batch_size):
    847               callbacks.on_train_batch_begin(step)
--> 848               tmp_logs = train_function(iterator)
    849               # Catch OutOfRangeError for Datasets of unknown size.
    850               # This blocks until the batch has finished executing.

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds)
    578         xla_context.Exit()
    579     else:
--> 580       result = self._call(*args, **kwds)
    581
    582     if tracing_count == self._get_tracing_count():

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds)
    625       # This is the first call of __call__, so we have to initialize.
    626       initializers = []
--> 627       self._initialize(args, kwds, add_initializers_to=initializers)
    628     finally:
    629       # At this point we know that the initialization is complete (or less

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to)
    504     self._concrete_stateful_fn = (
    505         self._stateful_fn._get_concrete_function_internal_garbage_collected(  # pylint: disable=protected-access
--> 506             *args, **kwds))
    507
    508     def invalid_creator_scope(*unused_args, **unused_kwds):

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2444       args, kwargs = None, None
   2445     with self._lock:
-> 2446       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2447     return graph_function
   2448

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _maybe_define_function(self, args, kwargs)
   2775
   2776       self._function_cache.missed.add(call_context_key)
-> 2777       graph_function = self._create_graph_function(args, kwargs)
   2778       self._function_cache.primary[cache_key] = graph_function
   2779       return graph_function, args, kwargs

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2665             arg_names=arg_names,
   2666             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2667             capture_by_value=self._capture_by_value),
   2668         self._function_attributes,
   2669         # Tell the ConcreteFunction to clean up its graph once it goes out of

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    979         _, original_func = tf_decorator.unwrap(python_func)
    980
--> 981       func_outputs = python_func(*func_args, **func_kwargs)
    982
    983       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py in wrapped_fn(*args, **kwds)
    439         # __wrapped__ allows AutoGraph to swap in a converted function. We give
    440         # the function a weak reference to itself to avoid a reference cycle.
--> 441         return weak_wrapped_fn().__wrapped__(*args, **kwds)
    442     weak_wrapped_fn = weakref.ref(wrapped_fn)
    443

~/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/framework/func_graph.py in wrapper(*args, **kwargs)
    966           except Exception as e:  # pylint:disable=broad-except
    967             if hasattr(e, "ag_error_metadata"):
--> 968               raise e.ag_error_metadata.to_exception(e)
    969             else:
    970               raise

ValueError: in user code:

    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:571 train_function  *
        outputs = self.distribute_strategy.run(
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:951 run  **
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:2290 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/distribute/distribute_lib.py:2649 _call_for_each_replica
        return fn(*args, **kwargs)
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:541 train_step  **
        self.trainable_variables)
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:1804 _minimize
        trainable_variables))
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:521 _aggregate_gradients
        filtered_grads_and_vars = _filter_grads(grads_and_vars)
    /home/ml/anaconda3/envs/hugging/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:1219 _filter_grads
        ([v.name for _, v in grads_and_vars],))

    ValueError: No gradients provided for any variable: ['shared/shared/weight:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._0/SelfAttention/relative_attention_bias/embeddings:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._0/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._1/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._2/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._3/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._4/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._5/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._6/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._7/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._8/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._9/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._10/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._1/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._1/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/encoder/block_._11/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/encoder/final_layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._0/SelfAttention/relative_attention_bias/embeddings:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._1/EncDecAttention/relative_attention_bias/embeddings:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._0/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._1/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._2/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._3/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._4/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._5/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._6/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._7/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._8/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._9/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._10/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._0/SelfAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._0/SelfAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._0/SelfAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._0/SelfAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._0/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._1/EncDecAttention/q/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._1/EncDecAttention/k/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._1/EncDecAttention/v/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._1/EncDecAttention/o/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._1/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._2/DenseReluDense/wi/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._2/DenseReluDense/wo/kernel:0', 'tf_t5for_conditional_generation/decoder/block_._11/layer_._2/layer_norm/weight:0', 'tf_t5for_conditional_generation/decoder/final_layer_norm/weight:0'].

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Should be able to run the training loop for the specified epochs.


control framework loglevel in scripts and tests

There is too much logging going on at times under `transformers` and friends. One needs to be able to turn the noise off easily. This is a follow up to https://github.com/huggingface/transformers/issues/3050#issuecomment-682167272

**edit**: The default was changed yesterday to`logging.WARNING` https://github.com/huggingface/transformers/commit/4561f05c5fafc2b636a2fc1d0dded9057d439745 so there is much less noise now.

**edit**: this PR has evolved since it was initially submitted, so this OP has been updated to reflect the current state of things.

This PR introduces 2 things.

# 1.  new function: `set_verbosity_all`

Usage:

a. override all module-specific loggers to a desired level (except whatever got logged during modules importing)
```
import everything, you, need
import transformers
transformers.testing_utils.set_verbosity_all(transformers.logging.ERROR)
```

b. If you want to disable specific loggers you can call it with specific top level names:
```
import transformers, torch, ...
transformers.testing_utils.set_verbosity_all(transformers.logging.ERROR, ["transformers", "nlp", "torch", "tensorflow", "tensorboard", "wandb"])
```
add/remove module name prefices as needed.

I initially placed it under `transformers.utils.logging` but then since it's beyond the core functionality, I moved it to testing_utils, since this is where we really want it. Please correct me if it should better belong elsewhere.

# 2.  new pytest option: `--log-level-all=error`

when debugging tests, sometimes framework-wide logging gets seriously in the way, e.g. try:

```
RUN_SLOW=1 pytest -sv --disable-warnings tests/test_modeling_bart.py::BartModelIntegrationTests::test_inference_no_head
```
gives a lot of noise. (it was so until `s/info/warn` recent change mention above, but there is noise still)

Now you will be able to turn it off, focusing only on the debug you want, by adding `--loglevel=error ` to the `pytest` options (or another level of your choice):

```
RUN_SLOW=1 pytest -sv --log-level-all=error --disable-warnings tests/test_modeling_bart.py::BartModelIntegrationTests::test_inference_no_head
```
voila - the noise is gone, while you can still do debug printing, etc.

```
pytest -h
  [...]
  --log-level-all={debug,info,warning,error,critical}
                        set global logger level before each test
```

# 3. new test + `CaptureLogger` context manager

While working on this a few integration tests were added and a helper `CaptureLogger` context manager
 to easily test the logger outputs.

# 4. cleaned up one test

removed verbosity setting in one test, which impacted other tests as it wasn't resetting the level to the original

-----

Quite a few testing features were added recently, I guess it's time to start `testing.md` or something.

----

Fixes: https://github.com/huggingface/transformers/issues/3050


make the tmp dir configurable/persistent in tokenizer tests

Currently, debugging tokenizers is difficult since the temp dir is random and it gets wiped out at the end of the test run. It can be done, but it takes so much repetitive work.

This PR uses the recently added [`TestCasePlus`](https://github.com/huggingface/transformers/pull/6494) which automatically sets up temp dirs and optionally doesn't remove them at the end of the test. It makes it very easy to configure the temp dir to be fixed rather than random, and also not delete itself.

As a side-effect of inheriting from `TestCasePlus`, the mixin approach of `FooTest(TokenizerTesterMixin, unittest.TestCase)` doesn't work - as it now tries to run the super-class tests directly and not from within the subclass.  Therefore, the code switches to normal sub-classing and instructs `unittest` not to run super-class' tests on its own, using the following machinery as explained [here](https://stackoverflow.com/a/50922971/9201239). Specifically here:

```
from transformers.testing_utils import TestCasePlus
class TokenizerCommonTester(TestCasePlus):
    __test__ = False

# and then the sub-class:
from .test_tokenization_common import TokenizerCommonTester
class BartTokenizationTest(TokenizerCommonTester):
    __test__ = True
```

The PR  makes the code ready to debug by changing just one flag:

```
       DEBUG = False
        # if you need to debug the contents of the tmpdirname, set DEBUG to True, which will then use
        # a hardcoded path and won't delete it at the end of the test
        if not DEBUG:
            self.tmpdirname = self.get_auto_remove_tmp_dir()
        else:
            self.tmpdirname = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/token-test", after=False)
```
So just make `DEBUG` `True` and nothing else needs to be tweaked.

I hope this is useful for developers.

There are a few other test mixins that could be improved the same way, but let's see if this approach is welcomed first.

----

## An alternative solution

If mixin is preferable, then let's leave everything as is and do this instead:

```
# transformers/testing_utils.py
from pathlib import Path
def make_dir(path):
    Path(path).resolve().mkdir(parents=True, exist_ok=True)
    return path

# tests/test_tokenization_common.py
from transformers.testing_utils import make_dir
DEBUG = True
class TokenizerTesterMixin:

    tokenizer_class = None
    test_rust_tokenizer = False

    def setUp(self):
        if not DEBUG:
            self.tmpdirname = tempfile.mkdtemp()
        else:
            self.tmpdirname = make_dir("./tmp/test-tok")

    def tearDown(self):
        if not DEBUG:
            shutil.rmtree(self.tmpdirname)
```


Create smaller number of heads in attn without pruning using shared parameters

# 🚀 Feature request

Instead of prunning heads or masking heads, create new linear layers with views of the larger linear layers that contains all heads.

## Motivation

For various models (Bert, Distilbert, etc.) I would like to be able to experiment on using separate heads without having to prune the heads. For example, training layers with separate heads. There is a way to masks heads, but the computation is still performed over all of the k, q, v tensors, and then masking is performed.

Is there a way to create views of the heads that you would otherwise prune, so that the extra computation per unused heads is not performed. My understanding is that the attention computation is one of the more expensive operation, since it is quadratic in the seq_len.

For example, with DistilBert, I was thinking of replacing, q_lin, v_lin, and k_lin with a shared linear layer that is a view into the original q_lin, k_line, v_lin.

## Contribution

I am thinking of adapting this code from prune_linear_layer from modeling_utils.py. Do you think this will work? Is there a better way to do this?
``
def share_linear_layer_by_index(layer, index, dim=0):
    """ Create a new linear layer (a model parameters) with shared parameters from index of the old layer
        Return new layer with requires_grad=True.
    """
    index = index.to(layer.weight.device)
    W = layer.weight.index_select(dim, index)
    if layer.bias is not None:
        if dim == 1:
            b = layer.bias
        else:
            b = layer.bias[index]
    new_size = list(layer.weight.size())
    new_size[dim] = len(index)
    new_layer = nn.Linear(new_size[1], new_size[0], bias=layer.bias is not None).to(layer.weight.device)
    new_layer.bias.requires_grad = False
    new_layer.weight = W
    new_layer.weight.requires_grad = True
    if layer.bias is not None:
        new_layer.bias.requires_grad = False
        new_layer.bias = b
        new_layer.bias.requires_grad = True
    return new_layer

``


RAG

# Intro
This pull request implements a RAG-sequence and a RAG-token models, as defined in [the paper](https://arxiv.org/pdf/2005.11401.pdf).

RAG (for Retrieval Augmented Generation) is an architecture combining a retriever with a generator model. During a forward pass, the input sequence is used as a question to the retriever, which surfaces relevant context documents. The documents are then prepended to the input and such contextualized input is passed to the generator. In the paper, we experiment with DPR-based retrieval and BART generator.

# Implementation
RAG is a seq2seq model which encapsulates three core components:
- a retriever - a wrapper around a faiss index of the documents,
- a question encoder which encodes the input sequence before passing it to the retriever,
- a generator which learns to generate the output from the contextualized input

as well as respective tokenizers (we need to be able to decode the input sequence, encode it with the question  encoder tokenizer, and then encode contextualized input with the generator tokenizer again).

---
We implement two variants of the model, both presented in the paper:
- `RagSequence`, which uses `DPRQuestionEncoder` as the question encoder. As for the generator, two compatible architectures have been tested: `BartForConditionalGeneration`  and `T5ForConditionalGeneration`.
- `RagToken`, which uses `DPRQuestionEncoder` and `BartForConditionalGeneration` as the generator.

---
Key files in the pull request:
- `modeling_rag.py`, `tokenization_rag.py`, `configuration_rag.py` the core model implementation
-  `retrieval_rag.py` - a distributed retriever built on top of the `torch.distributed` communication package. The retriever is an interface between the model and the faiss index of the encoded documents. During training, all workers initialize their own instance of the retriever, however, only the main worker loads the index into memory, which prevents OOMs on machines with multiple GPUs (we store the index in RAM). The index itself is based on the `nlp.Datasets`. We also implement a variant compatible with indices built using the original DPR implementation (https://github.com/facebookresearch/DPR)
- `eval_rag.py` - an evaluation script which allows to perform the evaluation end to end (measures the exact match and F1 on the downstream task) as well as the evaluation of the retrieval component alone (measures precision@k).
- `finetune.py` - a training script for finetuning RAG models.

# Testing
We have successfully managed to reproduce original paper results on Natural Questions for a couple of scenarios:
- converting original `fairseq` checkpoints to `HuggingFace` and evaluating them on `HuggingFace`
- converting original `fairseq` checkpoints to `HuggingFace` and continuing fine-tuning on `HuggingFace`
- training from scratch on `HuggingFace`

# Pretrained, ready-to-use models (after PR is merged).

- RagToken: https://huggingface.co/facebook/rag-token-nq
- RagSequence: https://huggingface.co/facebook/rag-sequence-nq
- RagTokenBase: https://huggingface.co/facebook/rag-token-base
- RagSequneceBase: https://huggingface.co/facebook/rag-sequence-base

# Future PR
- [ ] Add and test distributed Pytorch and possible Ray - Retriever @lhoestq
- [ ] Upload more rag model combinations.
- [ ] Clean examples


Potential bug in PLM training

There seems to be a bug in `mask_tokens` method of `DataCollatorForPermutationLanguageModeling`. Based on the comment, [this line](https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L294) is supposed to compute mask for non-functional tokens, ie. anything but padding and special tokens. So there should be an OR between the `padding_mask` and `special_tokens_mask`, and not AND. For reference, the [corresponding line](https://github.com/zihangdai/xlnet/blob/master/data_utils.py#L602) in the original XLNet code also has an OR.

I should acknowledge that I haven't understood the permutation masking code properly yet. But raising an issue, because it seems wrong to me.

-----

Besides the above problem, I'm also getting a very bad perplexity (**296.0**) on evaluating (w/o finetuning) `xlnet-base-cased` PLM model on plain wikitext2 dataset (`wiki.test.raw`). I've used XLNet example from [here](https://github.com/huggingface/transformers/tree/master/examples/language-modeling) (without `--do-train` flag) to get the perplexity.

The PLM code only works if the sequence lengths are even. To workaround this, I append a padding token when sequence length is odd. Concretely, I replaced the [error here](https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py#L255) with:

```
padding = inputs.new_ones((inputs.size(0), 1))*self.tokenizer.pad_token_id
inputs = torch.cat([inputs, padding], dim=1)
```

For comparison, the perplexity of BERT in this dataset is around 10.

Transformer Version: from master.

@patrickvonplaten @TevenLeScao @LysandreJik @shngt


Pegasus finetune script: add --adafactor


[s2s] save first batch to json for debugging purposes

helps debugging and understanding at a very low cost, by writing `text_batch.json` and `tok_batch.json` to `output_dir/`


[s2s] Test hub configs in self-scheduled CI

Tests

Related to tests

<!-- This line specifies which issue to close after the pull request is merged. -->
Regression test for #6806


bart-large-cnn ROUGE-L scores

## Environment info

### Who can help
BART + Summarization @sshleifer

## Information

Model I am using is BART.

The problem arises when:
verifying accuracy numbers of facebook/bart-large-cnn on CNN+Daily Mail. The paper reports R1, R2, RL of 44.16, 21.28, 40.90 but I can get only 44.05, 21.07, 30.62. I used [this](https://github.com/abisee/cnn-dailymail) to make my dataset. Is this expected?

The tasks I am working on is:
* CNN-Dm summarization task

## To reproduce

Steps to reproduce the behavior:

1. Follow instructions to download dataset
2. Run with `python run_summarization.py --reference_path=data/cnn_dm/test.target data/cnn_dm/test.source results/test.log`


[WIP] Added token_type_id support to GPT2Model

Fixes #6794 #4922 #1339

Added token type embedding support to GPT2Model.

To use `token_type_ids` with `GPT2Model` the user can now supply `type_vocab_size` to the `GPT2Config` constructor and then pass `token_type_ids` as input to the `forward` method as with other models.

Using all the default parameters, the model should continue to work as before, since I have set `type_vocab_size=None` as the default of the `GPT2Config` class, which disables my token type embeddings.

If the user tries to pass `token_type_ids` to the `forward` method when token type embeddings have not been enabled for a model, it will raise an informative error message. The same is true for when the user does not pass `token_type_ids` to the `forward` method when token type embeddings have been enabled for a model.


distilbart-cnn reproduction

@sshleifer
Unable to read `config.json` for `sshleifer/student_cnn_12_6`

[config.json](https://s3.amazonaws.com/models.huggingface.co/bert/sshleifer/student_cnn_12_6/config.json) Line 64 has a format error :
```
  "force_bos_token_to_be_generated", true
}
```

This should be:
 ```
 "force_bos_token_to_be_generated" : true
}
```
This is causing an issue in loading `sshleifer/student_cnn_12_6` using `.from_pretrained()`.


Fixes #6794 #4922 #1339

Added token type embedding support to GPT2Model.


BART ce loss ignores pad_token_id instead of -100

cc @ibeltagy who noticed this.

I think it is better to ignore pad_token_id then -100, but this is semi-breaking because old training code might have replaced pad token id with -100 in labels.

Maybe I should check for both?


Fix style

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Only access loss tensor every logging_steps

model card

Related to pretrained model cards

* tensor.item() was being called every step. This must not be done for XLA:TPU tensors as it's terrible for performance causing TPU<>CPU communication at each step. On RoBERTa MLM for example, it reduces step time by 30%, should be larger for smaller step time models/tasks.
* Train batch size was not correct in case a user uses the `per_gpu_train_batch_size` flag
* Avg reduce loss accross eval shards
* Log TPU debug metrics before last epoch break


Model card for primer/BART-Squad2

Related to pretrained model cards

Adds model card (model is already uploaded).


t5 model should make decoder_attention_mask

<!-- This line specifies which issue to close after the pull request is merged. -->
This undoes an aggressive change that was merged in #6654. I should discuss w Patrick first when he returns.


Marian distill scripts + integration test

- Fix 2 integration tests in examples
- Add new integration test for marian distillation script
- add 2 marian distillation scripts (but dont document them, will do that once they are slightly better).


[s2s] round runtime in run_eval


2 Slow Test Failures That Sam Can Fix

Tests

Related to tests

Both related to fp16

1)
```
        if torch.cuda.is_available():
            testargs += ["--fp16", "--gpus=1"]

        with patch.object(sys, "argv", testargs):
>           result = run_pl_glue.main()
```

2)
```
examples/seq2seq/test_bash_script.py::test_train_mbart_cc25_enro_script
```
needs rerun/ cannot use fp16


Problem 1:
When I load tokenizer from local directory.  And use it according MarianMT tutorial. There will be an error occurs.
AttributeError: 'MarianTokenizer' object has no attribute 'prepare_translation_batch'

Problem 2:
When I downloads models and tokenizers from the web, such as running 'tokenizer = AutoTokenizer.from_pretrained("Helsinki-NLP/opus-mt-zh-en")'. There will be some errors.
RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(818) [model_proto->ParseFromArray(serialized.data(), serialized.size())]


Maybe global step should be initialized to 0

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2

in script：https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py
location： line 143


Unexpected behavior encoding token_type_ids in GPT models

The issues #4922 and #1339 have been closed, but bug is still present in the `GPT2Model` code resulting in unexpected behaviour as described in #4922 .

I am trying to use token_type_ids to annotate different languages being input to a multilingual model. I am using my own tokenization code which outputs token_type_ids indicating the language of the tokens. I have run into unexpected behaviour where the embeddings for the first few tokens in my vocabulary are also being used for my token type embeddings. I am using version 3.0.2.

The issue was marked as closed since the GPT2 Tokenizer was no longer outputting token_type_ids, but the `GPT2Model` class still accepts token_type_ids in its forward method.

# Expected Behavior

Either:

1. addition of a separate `nn.Embedding` matrix for token_type_ids [here](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_gpt2.py#L352) (such as [in BERTModel](https://github.com/huggingface/transformers/blob/3ae2e86baffc1fea8b8b93695fb5a10941fd63dc/src/transformers/modeling_bert.py#L153))
2. discourage or throw a warning when `token_type_ids` are passed to GPTModel instances


Update README of my model

model card

Related to pretrained model cards


F16 support for DistilBert

# 🚀 Feature request
DIstilBert model donesn't support f16 or mixed precision training.
The hard-coded uses of float32 from the tensorflow implementation of BERT and ELECTRA has been fixed by #3320 .
The same problem in DistilBert needs a fix.


Enable wandb logging for Ray Tune HPO runs

<!-- This line specifies which issue to close after the pull request is merged. -->
Currently wandb logging does not work with Ray Tune, as each process tries to call `wandb.log` without initializing wandb first.

This PR fixes this by wrapping the objective in Ray Tune's `wandb_mixin`, which makes sure each trial initializes wandb and logs to it.


What is the size of the context window in the 'openai-gpt' pre-trained model?
# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


How to fine-tune T5 with some additional special tokens ?

I wanna to fine-tune T5 with seq2seq task, but there are some special tokens in this seq2seq task. T5 performs bad without these tokens.  How could I use some additional special tokens to fine-tune T5 with scripts https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune_t5.sh?


Update multilingual passage rereanking model card

Related to pretrained model cards

Fix range of possible score, add inference .


Dear transformers team, how could I use bert to NER task?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:

Dear  transformers team,
I have train and test corpus with BIO tags, like below:
The	O
patient	O
was	O
noted	O
with	O
first	O
depressive	O
episode	O
at	O
aged	O
36	O
.	O

How could I use bert to train my data to produce models and to predict the BIO tags of test data?
The resource have many programs, but I have no ideas that which program is what I need.

I would like to use google colab to run the program that could save python environmental problems.

Could you offer the tutorial of Bert NER task?

Thank you
   Best regards;


T5 batch inference same input data gives different outputs?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
Hi, I am using a T5 model from a tf check point. I have set model.eval(), but every time I forward the exactly same input data, the outputs were always different. here is my code:
```
import torch
from transformers import *

tokenizer = T5Tokenizer.from_pretrained('t5-base')
config = T5Config.from_pretrained('t5-base')
model = T5ForConditionalGeneration.from_pretrained(
    "../model/t5-base/model.ckpt-1004000", from_tf=True, config=config)
model.eval()

def prediction(documents, query):
    querys = [query] * len(documents)
    encoded_encoder_inputs = tokenizer(documents, padding=True, truncation=True, return_tensors="pt")
    encoded_decoder_inputs = tokenizer(querys, padding=True, truncation=True, return_tensors="pt")
    with torch.no_grad():
        outputs = model(input_ids=encoded_encoder_inputs["input_ids"],
                        labels=encoded_decoder_inputs["input_ids"],
                        attention_mask=encoded_encoder_inputs["attention_mask"])
        batch_logits = outputs[1]
        print(batch_logits)

documents = ['my dog is cute!', "so am I."]
query = "who am I?"
prediction(documents, query)
```

here is my twice outputs (run the above code twice):

```
tensor([[[-14.6796,  -6.2236, -13.3517,  ..., -42.7422, -42.7124, -42.8204],
         [-18.5999,  -4.9112, -10.6610,  ..., -40.7506, -40.7313, -40.8373],
         [-17.3894,  -5.3482, -11.4917,  ..., -41.2223, -41.1643, -41.3228],
         [-18.4449,  -5.9145, -12.0056,  ..., -42.3857, -42.3579, -42.4859]],
        [[-15.7967,  -6.9833, -14.5827,  ..., -41.3168, -41.0326, -40.9567],
         [-18.4241,  -5.7193, -12.0748,  ..., -40.1744, -40.0635, -39.9045],
         [-19.5852,  -5.1691, -12.7764,  ..., -42.2655, -42.0788, -41.9885],
         [-20.3673,  -3.6864, -12.5264,  ..., -40.1189, -40.0787, -39.8976]]])
```

```
tensor([[[-14.6796,  -6.2236, -13.3517,  ..., -42.7422, -42.7124, -42.8204],
         [-18.5848,  -4.9116, -10.6607,  ..., -40.7443, -40.7251, -40.8300],
         [-17.3248,  -5.3050, -11.4988,  ..., -41.1802, -41.1236, -41.2818],
         [-18.3950,  -5.8967, -11.9756,  ..., -42.3553, -42.3273, -42.4553]],
        [[-15.7967,  -6.9833, -14.5827,  ..., -41.3168, -41.0326, -40.9567],
         [-18.5133,  -5.7466, -12.1301,  ..., -40.2199, -40.1078, -39.9497],
         [-19.3636,  -5.1246, -12.7134,  ..., -42.1881, -41.9993, -41.9106],
         [-20.3247,  -3.6335, -12.4559,  ..., -40.0285, -39.9912, -39.8106]]])
```
Am I did anything wrong?


[help] Add multigpu evalulation script for seq2seq

Examples

Which is related to examples in general

Help wanted

Extra attention is needed, help appreciated

translation

machine translation utilities and models

### Goal
At the end the command,
```
cd examples/seq2seq
export DATA_DIR=test_data  # use wmt_en_ro for more data, download instructions in README.md
export save_dir=translations
python multigpu_eval.py Helsinki-NLP/opus-mt-en-ro $DATA_DIR/test.source $save_dir/test_translations.txt --reference_path $DATA_DIR/test.target --score_path $save_dir/test_bleu.json --fp16 --task translation --bs 16
```
On wmt_en_ro, thissShould get ~27.7 (same result as run_eval.py on 1 GPU) and should take less time.
On test_data I don't know the 1 GPU result.

side note: The motivation here is actually summarization, where the validation sets are larger and on 1 GPU for pegasus running eval takes an hour.

### Implementation

This looks similar, https://github.com/facebookresearch/ParlAI/blob/00efcbebb49524918692638ab580cadeebe70cf8/parlai/scripts/multiprocessing_eval.py#L34 but I'm open to many strategies. Lots of freedom. You will probably need to use one of the datasets in utils + an unshuffled dataloader of some sort. Try to keep the command line as similar to run_eval.py as possible, with the exception of a --gpus flag. Alternatively, use all available gpus and tell the user to use CUDA_VISIBLE_DEVICES=0,1 to restrict.

### Process

1. You can claim the bug here, but don't wait for me to reply to get started. This is hard enough that it is very unlikely that two people will be working on this at once. If I start working on this (unlikely) I will post here and remove the help-wanted label.
2. When you send a PR, tag me and link this issue.
3. It will be difficult to test your solution in CI, but try to make the code work with 1 GPU so we can test it that way in CI. You can make a new test_.py file or add to test_seq2seq_examples.py
4. Report BLEU scores in PR description.

Happy hacking!


[style] set the minimal required version for `black`

`make style` with `black < 20.8b1` is a no go (in case some other package forced a lower version) - so make it explicit to avoid confusion


Why does examples/seq2seq/finetune.py only use sortish sampler for train?

Examples

Which is related to examples in general

Would it make eval faster?
Would it make validation sanity check obnoxiously slow?
This should be a ~1 line PR with a longer PR description.


[style] a new `black` version is out there (20.8)

it now seems to respect `--line-length 119` in the `make style/quality` command

so a big reformat - no code change

These are just the changes after running `make style`.


Typo in modling_tf_bert

end_pos will be overwritten in some case
https://github.com/huggingface/transformers/blob/858b7d5873577d7997918439eb7429d424d07dd5/src/transformers/modeling_tf_bert.py#L1422


How can I get rid of this message every time I run GPT2?

```
Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2 and are newly initialized: ['h.0.attn.masked_bias', 'h.1.attn.masked_bias', 'h.2.attn.masked_bias', 'h.3.attn.masked_bias', 'h.4.attn.masked_bias', 'h.5.attn.masked_bias', 'h.6.attn.masked_bias', 'h.7.attn.masked_bias', 'h.8.attn.masked_bias', 'h.9.attn.masked_bias', 'h.10.attn.masked_bias', 'h.11.attn.masked_bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
```


Character-level tokenization?

# ❓ Questions & Help

In a project I'm using the Hugging Face library for, I would like to use transformers to perform arithmetic (framed as a text-to-text problem). However, it seems that the current method of tokenization adopted by most existing models in the Hugging Face library (BPE, WordPiece, etc.) are not character-level, and thereby, would not allow for the network to properly process the input for later arithmetic. Are there any ways to get this sort of analysis to work properly with the existing models in Hugging Face (and their accompanying tokenizers), otherwise, what recommendations do you have?

For reference, I am currently using the BART model from Hugging Face.


Bertology-like Analysis for BART, T5?

# ❓ Questions & Help

In my current project, I am working on training encoder-decoder models (BART, T5, etc.) and the Transformers library has been absolutely invaluable! After seeing several Bertology analyses (i.e. looking at the information the model's attention mechanism learns to attend to), I would like to know if a similar analysis is possible with the BART and T5 models in the Hugging Face library. Any recommendations are certainly appreciated!


[transformers-cli] fix logger getter

`transformers-cli` is currently broken:
```
  File "src/transformers/commands/transformers_cli.py", line 8, in <module>
    from transformers.commands.serving import ServeCommand
  File "/mnt/nvme1/code/huggingface/transformers-master/src/transformers/commands/serving.py", line 28, in <module>
    logger = logging.getLogger("transformers-cli/serving")
AttributeError: module 'transformers.utils.logging' has no attribute 'getLogger'
```
this fixes it.

plus added a basic test.


PL: --adafactor option

CC @moscow25, @patil-suraj

I used the "External LR" setup and verified that is saves a significant amount of memory on pegasus finetuning.

Happy to add to Trainer.


Support fast tokenizer in Question Answering Pipeline

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6545

The problem is that the behavior of the python tokenizer and the rust based fast tokenizer is very different.
The python tokenizer handles cases where inputs are in different formats (str tokens and int tokens and vice versa), where as the fast tokenizer is unable to do so.

Some edge cases where the fast tokenizer fails because of limited functionality (compare to python tok) have been added and I've included comments in those places.


Task specific params for pegasus-large to allow finetuning with correct generation_parameters

```
'task_specific_params': {'summ_xsum': {'max_length': 56, 'length_penalty': 0.8}}}
```


Error in run_ner.py

## Environment info

- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help

@stefan-it

## Information

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: Rare Entities task: WNUT’17 (English NER) dataset
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

When I try to execute run_ner.py, the following error occurs:

08/27/2020 16:10:10 - INFO - filelock -   Lock 2174122787120 acquired on ./data_wnut_17\cached_dev_BertTokenizer_128.lock
08/27/2020 16:10:24 - INFO - utils_ner -   Creating features from dataset file at ./data_wnut_17
08/27/2020 16:12:04 - INFO - filelock -   Lock 2174122787120 released on ./data_wnut_17\cached_dev_BertTokenizer_128.lock
Traceback (most recent call last):
  File "C:\Users\Monique\Documents\IA\NLP\transformers\examples\token-classification\utils_ner.py", line 247, in __init__
    examples = token_classification_task.read_examples_from_file(data_dir, mode)
  File "C:\Users\Monique\Documents\IA\NLP\transformers\examples\token-classification\tasks.py", line 27, in read_examples_from_file
    for line in f:
  File "C:\ProgramData\Anaconda3\lib\codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb4 in position 46: invalid start byte

My only local change was the following in preprocess.py:

`with open(dataset, "rt", encoding='utf-8') as f_p:`

(I added encoding='utf-8' due to a similar error in preprocessing).

## Expected behavior

The dataset should have been created without encoding errors.


Unable to install Transformers Master version

@patil-suraj @joeddav
Is there some other way to install master version of transformers?
I tried using the URL to install the master version but it installed v3.0.2

_Originally posted by @AnkitVarshney02 in https://github.com/huggingface/transformers/issues/6752#issuecomment-681716615_


Exported TF Bert model is much slower than that exported from Google's Bert

I benchmarked the Bert model exported from Huggingface TF Bert code and observed that it is much slower than Google's TF2 Bert model(https://github.com/tensorflow/models/tree/master/official/nlp/bert) on TF Serving (GPU device). When batch_size=1, Huggingface Bert model can be more than 200% slower than Google's one. Here are my benchmark results:

**latency: (ms/batch) on 1 GPU-V100**
Batch Size | google official | HF_TF2 | relative change |
-- | -- | -- | --
1 | 6.7 | 21.3 | 238.10%
2 | 9.4 | 24.2 | 171.91%
4 | 14.4 | 28.1 | 99.29%
8 | 24.0 | 36.9 | 55.70%
16 | 46.6 | 58.6 | 26.57%
64 | 171.5 | 171.4 | 0.35%
128 | 338.5 | 324.5 | -3.71%

I used the latest Tensorflow profiling tool (https://www.tensorflow.org/tfx/serving/tensorboard) to compare these two:

**Hugging-face Bert-Base FP32 Batch_size = 128**

**Google Bert-Base FP32 Batch_size = 128**

Comparing the event traces of HF's and google's, we see that before each *MatMul* operation on GPU, there are always some CPU processes running (wasn’t able to get more info about these CPU processes because the Profiling tool doesn’t provide meaningful labels or description for those CPU processes but it seems like they are doing some gather/reduce operations) and the GPU is inactive for a while. If you look at the trace of google official, you can find that all GPU ops are compact and there are no idle time for GPU. In parallel, only one CPU process keeps running and doesn’t cause GPU ops to wait. The short idle time will hurt the performance more when batch_size or model_size is small because short time is spent on GPU calculations and idle time is not ignorable anymore. See time comparison grouped by type below:

**Hugging-face Bert-Base FP32 Batch_size = 4**

**Google Bert-Base FP32 Batch_size = 4**

Under small batch_size=4, Hugging-face model has 60% GPU idle time while that of google model is only 25%. This also explains the reason why for small batch_size, there is over 200% slow down and for large batch_size=128, the situation gets better.

The *MatMul* op is triggered by tf.keras.layers.Dense(), which is widely used in Transformer encoder self-attention, intermediate layer and output layer. In comparison, Google's Bert uses DenseEinsum()([link](https://github.com/tensorflow/models/blob/1ebff962832ec8a599bd67ac8c43b9ae1b05c3fa/official/nlp/transformer/attention_layer.py#L59-L69)) to replace all usage of Dense layer. I personally modified Huggingface's TF Bert code base to use DenseEinsum(). And the slow down issue got solved:

**latency: (ms/batch)  on 1 GPU-V100**
Batch Size | google official | HF_TF2 | relative change | After Fixing | relative change
-- | -- | -- | -- | -- | --
1 | 6.7 | 21.3 | 238.10% | 6.60 | 4.76%
2 | 9.4 | 24.2 | 171.91% | 8.90 | 0.00%
4 | 14.4 | 28.1 | 99.29% | 13.40 | -4.96%
8 | 24.0 | 36.9 | 55.70% | 22.10 | -6.75%
16 | 46.6 | 58.6 | 26.57% | 43.20 | -6.70%
64 | 171.5 | 171.4 | 0.35% | 158.80 | -7.03%
128 | 338.5 | 324.5 | -3.71% | 313.10 | -7.09%

Profile after fixing:

I noticed that multiple issues may be related to this (https://github.com/huggingface/transformers/issues/6264).

Do you plan to solve this issue? Thanks. @patrickvonplaten


Roberta Large not working with SNLI dataset always gives random baseline that is 33 percent accuracy.

I have been working with parts of SNLI dataset, even after hours and hours of pretraining the model always gives 33.24 percent accuracy no matter the amount of the data used in pretraining. This issue is fixed if I revert my transformers version to 2.11.0


Seq2SeqTrainer

This PR adds `Seq2SeqTrainer` to use `Trainer` for summarization, translation and other seq2seq tasks.

`Seq2SeqTrainer` includes:
- label smoothing loss
- sortish sampler
- predict from generate to allow calculating generative metrics

`finetune_trainer.py` includes:
- `Seq2SeqDataCollator` to correctly prepare `labels` and `decoder_input_ids`
- `Seq2SeqTrainingArguments` for extra training arguments.
- encoder and embedding freezing.
- `compute_metrics_fn` for translation(BLEU) and summarization (ROUGE)
- evaluation

Most of the data and model arguments are kept identical to `finetune.py` file

Seq2Seq4Eva!🔥


Floating-point operations logging in trainer

First of two PRs to implement #4847 :

 - logging loss vs floating-point operations
 - using the results for scaling laws analysis

This directly logs floating-point operations in wandb and comet, and creates a `log_history.json` file with training metrics. To do so, it adds methods to `PretrainedModel` to count parameters with and without embeddings, and the number of floating-point operations. It also has a few Trainer fixes, most importantly averaging the eval loss across processes rather than logging the one in process 0, and a bug with checkpoint folder creation.


Add NLP install to self-scheduled CI

Same change as @sgugger already made for `self-push.yml`


Distributed training doesn’t work

Even if CUDA detects all the GPU of the machine, there is no distributed training :

`RuntimeError: CUDA out of memory. Tried to allocate 3.82 GiB (GPU 0; 11.17 GiB total capacity; 7.59 GiB already allocated; 594.31 MiB free; 10.28 GiB reserved in total by PyTorch) (malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:289)`

The training is launched only on the first GPU.
I’m using language modeling code. I tried to set local_rank at 0 and to use the Torch environment variables MASTER_ADDR, MASTER_PORT, WORLD_SIZE and RANK, but without success. Is there another way to do distributed training with transformers ?


Adds Adafactor to the docs and slightly fixes the formatting


Add ProtBert model card

model card

Related to pretrained model cards

This is a model card for our ProtBert:
https://huggingface.co/Rostlab/prot_bert


Cuda out of memory when fp16 training

# ❓ Questions & Help
I have error when run train with fp16
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I want to train Albert model on Russian. I use run_language_modeling.py with some changes in dataset creating. If i run training with fp32 (without --fp flag) and batch_size=2, I will ok. But if i run training with --fp16 flag, i will obtain cuda out of memory error. I tried to decdeare batch size to 1, however, I obtained same error.


Checklist for Model Hub

# 🚀 Feature request

I want to propose a checklist for model hub. Model hub is increasingly being used by `transformers` users. It is imperative that users exactly know what model they are downloading and importing for transparency and research-related reasons. It should be mandatory for users to upload a `REAME.md` while uploading model on model hub. Additionally, it will reduce the need to send a seperate PR for model cards.

Checklist should contain the following elements:
- Details of training and validation set used. Specify the split if not using default.
- A table with results on the validation set
- Model name (can be inferred from `config.json`)
- Objective used for training
- Hyperparameter configurations if default ones are not being used
- Link to implemented code (if available, and is different from default examples)
- For how many epochs the model was trained for ?
- Associated paper (if published)

These are just the general ones. Happy to hear about other suggestions. This is just a suggestion which would help many users. Thanks for this feature.


s2s distillation uses AutoModelForSeqToSeqLM

hard coding bart caused subsequent AutoTokenizer to fail for marian/mbart distillation.


Question: Differentiable Search in generate

Hi,
This might be a very dumb question to ask, but is there a way that we can have differentiable search and pickings in the "generate" method in models like T5 and GPT-2? If not, is there an example of using the pure real-valued model output in a task? Or an article giving some intuition on what the real-valued output means?
I'd really appreciate your help.


added model card for flexudys t5 model

Related to pretrained model cards

- Created a model card for a fine tuned model.


Bart can make decoder_input_ids from labels

mimics a nice t5 feature


Seq2Seq tokenization and training fixes

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Fix run_squad.py to work with BART

BART model should be among the set that doesn't use `token_type_ids`.


Model Card for Multilingual Passage Reranking BERT

Related to pretrained model cards


add __init__.py to utils

Adds `__init__.py` to the `utils` dir introduced in #6434, which caused a namespace issues breaking imports in some cases. Mentioned in #6752.

To reproduce, pip install from source without the editable `-e` flag.

```
>>> from transformers import pipeline
...
~/.virtualenvs/transformers/lib/python3.7/site-packages/transformers/file_utils.py in <module>
---> 32 from .utils import logging
ModuleNotFoundError: No module named 'transformers.utils'
```


Removing memory/deleting a model: how to properly do this

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.11.0
- Platform:
- Python version: 3.6.7
- PyTorch version (GPU?): 1.4.0
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): T5-large, T5-3b, bert-base-uncased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load a model
2. Try to remove it via `del`, clear GPU memory and cache

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```
import torch
from transformers import AutoTokenizer, AutoModelWithLMHead
model = AutoModelWithLMHead.from_pretrained("t5-large") # same behavior for `bert-base-uncased`, larger T5 models..
model = model.cuda()
model = model.train()

## delete model
del model
torch._C._cuda_emptyCache()
## alternatively
# with torch.cuda.device("cuda:0"):
#   ...:     torch.cuda.empty_cache()

## (as per the discussion here: https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3, seeing all the hanging tensors)
import gc
for obj in gc.get_objects():
    try:
        if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
            print(type(obj), obj.size())
    except:
        pass
```

## Expected behavior

I would expect this to clear the GPU memory, though the tensors still seem to linger (fuller context: In a larger Pytorch-Lightning script, I'm simply trying to re-load  the best model after training (and exiting the pl.Trainer) to run a final evaluation; behavior seems the same as in this simple example (ultimately I run out of memory when loading the best model because the model is the absolutely massive T5-3b).).

<!-- A clear and concise description of what you would expect to happen. -->


I want to run Pegasus Model for which master version of transformers is required but I am unable install it.
I tried installing with pip command but it always install version 3.0.2.


Add Language Agnostic Bert Sentence Embedding

# 🌟 New model addition

## Model description
Google released a new model LaBSe, which is a Language agnostic Bert sentence embedding with 109 language support.

https://ai.googleblog.com/2020/08/language-agnostic-bert-sentence.html?utm_source=Deep+Learning+Weekly&utm_campaign=cdb8afedb7-EMAIL_CAMPAIGN_2019_04_24_03_18_COPY_01&utm_medium=email&utm_term=0_384567b42d-cdb8afedb7-72940109

The model is also hosted on tfhub as well:https://tfhub.dev/google/LaBSE/1

<!-- Important information -->

## Open source status

* [ ] the model implementation is available: (https://arxiv.org/abs/2007.01852)
* [ ] the model weights are available: https://tfhub.dev/google/LaBSE/1
* [ ] who are the authors: (mention them, if possible by @gh-username)


## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 3.0.2
Tokenizers: 0.8.1rc1
Python: 3.7.6
Pretrained Model: GPT2
Pretrained Tokenizer: GPT2

## Question
I'm getting the following error when I try to evaluate a model while it's training:
```python
wandb: Waiting for W&B process to finish, PID 242876
Traceback (most recent call last):
File "run_finetune_gpt2.py", line 163, in <module>
    main()
File "run_finetune_gpt2.py", line 150, in main
    trainer.train()
File "/path/to/venvs/my_venv/lib/python3.6/site-packages/transformers/trainer.py", line 537, in train
    self.evaluate()
File "/path/to/venvs/my_venv/lib/python3.6/site-packages/transformers/trainer.py", line 745, in evaluate
    output = self._prediction_loop(eval_dataloader, description="Evaluation")
File "/path/to/venvs/my_venv/lib/python3.6/site-packages/transformers/trainer.py", line 838, in _prediction_loop
    preds = torch.cat((preds, logits.detach()), dim=0)
RuntimeError: zero-dimensional tensor (at position 0) cannot be concatenated
```

I printed out the `preds` and `logits` shapes inside `trainer.py`:
```python
preds.shape = torch.Size([])
logits.shape = torch.Size([])
```
I can't recall exactly how my `TrainingArguments` object was set up but I think it was something like this:
```python
training_args = TrainingArguments(
         output_dir=output_dir,
         do_train=True,
         evaluate_during_training=True,
         eval_steps=5,
         logging_dir=logging_dir,
         per_device_train_batch_size=32,
         num_train_epochs=1,
         save_steps=100)

trainer = Trainer(
         model=model,
         args=training_args,
         train_dataset=sd_dataset_train,
         eval_dataset=sd_dataset_test,
         data_collator=sd_data_collator)
```

I was using `GPT2LMHeadModel`. I tried to put together a code sample but couldn't reproduce the error. Has anyone run into this before when trying to evaluate while training?