model.save() does not save keras model that includes DIstillBert layer

# 🐛 Bug

## Information
I am trying to build a Keras Sequential model, where, I use DistillBERT as a non-trainable embedding layer. The model complies and fits well, even predict method works. But when I want to save it using model.save(model.h5),  It fails and shows the following error:

```
> ---------------------------------------------------------------------------
> NotImplementedError                       Traceback (most recent call last)
> <ipython-input-269-557c9cec7497> in <module>
> ----> 1 model.get_config()
>
> /usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in get_config(self)
>     966     if not self._is_graph_network:
>     967       raise NotImplementedError
> --> 968     return copy.deepcopy(get_network_config(self))
>     969
>     970   @classmethod
>
> /usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in get_network_config(network, serialize_layer_fn)
>    2117           filtered_inbound_nodes.append(node_data)
>    2118
> -> 2119     layer_config = serialize_layer_fn(layer)
>    2120     layer_config['name'] = layer.name
>    2121     layer_config['inbound_nodes'] = filtered_inbound_nodes
>
> /usr/local/lib/python3.7/site-packages/tensorflow/python/keras/utils/generic_utils.py in serialize_keras_object(instance)
>     273         return serialize_keras_class_and_config(
>     274             name, {_LAYER_UNDEFINED_CONFIG_KEY: True})
> --> 275       raise e
>     276     serialization_config = {}
>     277     for key, item in config.items():
>
> /usr/local/lib/python3.7/site-packages/tensorflow/python/keras/utils/generic_utils.py in serialize_keras_object(instance)
>     268     name = get_registered_name(instance.__class__)
>     269     try:
> --> 270       config = instance.get_config()
>     271     except NotImplementedError as e:
>     272       if _SKIP_FAILED_SERIALIZATION:
>
> /usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/network.py in get_config(self)
>     965   def get_config(self):
>     966     if not self._is_graph_network:
> --> 967       raise NotImplementedError
>     968     return copy.deepcopy(get_network_config(self))
>     969
>
> NotImplementedError:
```

The language I am using the model in English.

The problem arises when using my own modified scripts: (give details below)

```
from transformers import DistilBertConfig, TFDistilBertModel, DistilBertTokenizer
max_len = 8
distil_bert = 'distilbert-base-uncased'
config = DistilBertConfig(dropout=0.2, attention_dropout=0.2)
config.output_hidden_states = False
transformer_model = TFDistilBertModel.from_pretrained(distil_bert, config = config)

input_word_ids = tf.keras.layers.Input(shape=(max_len,), dtype = tf.int32, name = "input_word_ids")
distill_output =  transformer_model(input_word_ids)[0]

cls_out = tf.keras.layers.Lambda(lambda seq: seq[:, 0, :])(distill_output)
X = tf.keras.layers.BatchNormalization()(cls_out)
X = tf.keras.layers.Dense(256, activation='relu')(X)
X = tf.keras.layers.Dropout(0.2)(X)

X = tf.keras.layers.BatchNormalization()(X)
X = tf.keras.layers.Dense(128, activation='relu')(X)
X = tf.keras.layers.Dropout(0.2)(X)

X = tf.keras.layers.BatchNormalization()(X)
X = tf.keras.layers.Dense(64, activation='relu')(X)
X = tf.keras.layers.Dropout(0.2)(X)

X = tf.keras.layers.Dense(2)(X)
model = tf.keras.Model(inputs=input_word_ids, outputs=X)

for layer in model.layers[:3]:
    layer.trainable = False
```

The tasks I am working on is my own dataset.

## To reproduce

Steps to reproduce the behavior:

1. Run the above code
2. You will get the error when saving the model as

```
model.save('model.h5')
```

You can get the same error if you try:
```
model.get_config()
```

**_An interesting observation:_**
if you save the model without specifying ".h5" like
```
model.save('./model')
```
it saves the model as TensorFlow saved_model format and creates folders (assets (empty), variables, and some index files). But if you try to load the model, it produces different errors related to the DistillBert/Bert. It may be due to some naming inconsistency (input_ids vs. inputs, see below) inside the DistillBert model.

```

new_model = tf.keras.models.load_model('./model)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.7/site-packages/tensorflow/python/util/nest.py in assert_same_structure(nest1, nest2, check_types, expand_composites)
    377     _pywrap_utils.AssertSameStructure(nest1, nest2, check_types,
--> 378                                       expand_composites)
    379   except (ValueError, TypeError) as e:

ValueError: The two structures don't have the same nested structure.

First structure: type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')}

Second structure: type=TensorSpec str=TensorSpec(shape=(None, 8), dtype=tf.int32, name='inputs')

More specifically: Substructure "type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')}" is a sequence, while substructure "type=TensorSpec str=TensorSpec(shape=(None, 8), dtype=tf.int32, name='inputs')" is not

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-229-b46ed71fd9ad> in <module>
----> 1 new_model = tf.keras.models.load_model(keras_model_path)

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py in load_model(filepath, custom_objects, compile)
    188     if isinstance(filepath, six.string_types):
    189       loader_impl.parse_saved_model(filepath)
--> 190       return saved_model_load.load(filepath, compile)
    191
    192   raise IOError(

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py in load(path, compile)
    114   # TODO(kathywu): Add saving/loading of optimizer, compiled losses and metrics.
    115   # TODO(kathywu): Add code to load from objects that contain all endpoints
--> 116   model = tf_load.load_internal(path, loader_cls=KerasObjectLoader)
    117
    118   # pylint: disable=protected-access

/usr/local/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py in load_internal(export_dir, tags, loader_cls)
    602       loader = loader_cls(object_graph_proto,
    603                           saved_model_proto,
--> 604                           export_dir)
    605       root = loader.get(0)
    606       if isinstance(loader, Loader):

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py in __init__(self, *args, **kwargs)
    186     self._models_to_reconstruct = []
    187
--> 188     super(KerasObjectLoader, self).__init__(*args, **kwargs)
    189
    190     # Now that the node object has been fully loaded, and the checkpoint has

/usr/local/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py in __init__(self, object_graph_proto, saved_model_proto, export_dir)
    121       self._concrete_functions[name] = _WrapperFunction(concrete_function)
    122
--> 123     self._load_all()
    124     self._restore_checkpoint()
    125

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py in _load_all(self)
    213
    214     # Finish setting up layers and models. See function docstring for more info.
--> 215     self._finalize_objects()
    216
    217   @property

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py in _finalize_objects(self)
    504         layers_revived_from_saved_model.append(node)
    505
--> 506     _finalize_saved_model_layers(layers_revived_from_saved_model)
    507     _finalize_config_layers(layers_revived_from_config)
    508

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py in _finalize_saved_model_layers(layers)
    675       call_fn = _get_keras_attr(layer).call_and_return_conditional_losses
    676       if call_fn.input_signature is None:
--> 677         inputs = infer_inputs_from_restored_call_function(call_fn)
    678       else:
    679         inputs = call_fn.input_signature[0]

/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py in infer_inputs_from_restored_call_function(fn)
    919   for concrete in fn.concrete_functions[1:]:
    920     spec2 = concrete.structured_input_signature[0][0]
--> 921     spec = nest.map_structure(common_spec, spec, spec2)
    922   return spec
    923

/usr/local/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, **kwargs)
    609   for other in structure[1:]:
    610     assert_same_structure(structure[0], other, check_types=check_types,
--> 611                           expand_composites=expand_composites)
    612
    613   flat_structure = [flatten(s, expand_composites) for s in structure]

/usr/local/lib/python3.7/site-packages/tensorflow/python/util/nest.py in assert_same_structure(nest1, nest2, check_types, expand_composites)
    383                   "Entire first structure:\n%s\n"
    384                   "Entire second structure:\n%s"
--> 385                   % (str(e), str1, str2))
    386
    387

ValueError: The two structures don't have the same nested structure.

First structure: type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')}

Second structure: type=TensorSpec str=TensorSpec(shape=(None, 8), dtype=tf.int32, name='inputs')

More specifically: Substructure "type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')}" is a sequence, while substructure "type=TensorSpec str=TensorSpec(shape=(None, 8), dtype=tf.int32, name='inputs')" is not
Entire first structure:
{'input_ids': .}
Entire second structure:
.
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

I expect to have a normal saving and loading of the model.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.1
- Platform:
- Python version: 3.7.6
- Tensorflow version (CPU): 2.2.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Issues with the EncoderDecoderModel for sequence to sequence tasks

# ❓ Questions & Help

I have been attempting with various models to try to build an encoder-decoder, sequence to sequence transformer model. For the most part, I have been using BERT (bert-base-cased), but have encountered issues with various models.

The model is intended for an English to English sequence to sequence problem.

For reference, I had been trying to use the seq2seq example in this pull request as a template :

https://github.com/huggingface/transformers/pull/3402

But have needed to make some modifications to it to account for other recent changes in the EncoderDecoderModel class.

I have a hit a few main issues, three are posted here. I think at least some of them are possibly bugs in the EncoderDecoderModel code.

1. A recent commit made some major changes to the forward method, and I've been hitting issues with the section that defines the decoder_outputs (around line 253 of modeling_encoder_decoder.py.) The example in the pull request I linked does not provide decoder_input_ids when setting up the model, but that is now required by this code in your recent commit. When training, I modified the code to provide decoder_token_ids as the target tokens shifted one to the right with a PAD token in front, as described in various papers. However, I don't understand why this is required when in eval mode -- shouldn't the model not have any decoder input tokens when in test/eval mode, and only be able to see what the previous tokens it actually output were? I don't understand what I'm supposed to provide as decoder_input_ids when in evaluation mode, and haven't been able to find documentation on it.

The code I'm currently using for training looks something like this :
```
        for step, batch in enumerate(epoch_iterator):

            # Skip past any already trained steps if resuming training
            if steps_trained_in_current_epoch > 0:
                steps_trained_in_current_epoch -= 1
                continue

            model.train()
            batch = tuple(t.to(args.device) for t in batch)
            input_ids, output_ids, input_mask, output_mask, _, decoder_ids = batch

            # add other inputs here, including kwargs
            **inputs = {"input_ids": input_ids, "attention_mask": input_mask, 'decoder_input_ids': decoder_ids}**

            # The output tuple structure depends on the model used and the arguments invoked
            # For BERT-type models, this is
            # decoder_predictions, encoded_embeddings, encoded_attention_mask = model(**inputs)
            # For GPT2-type models, this at least starts with the decoder predictions
            # See the EncoderDecoderModel class for more details
            **output = model(**inputs)**
```

More context is given in the linked pull request, since again this is being copied from there. The initial pull request does not provide the 'decoder_input_ids' parameter, but it seems that is now required. My code is similar in eval mode, but without decoder_input_ids, and this code fails :

```
**for batch in tqdm(eval_dataloader, desc="Evaluating"):
        batch = tuple(t.to(args.device) for t in batch)
        input_ids, output_ids, input_mask, output_mask, _, decoder_ids = batch
        with torch.no_grad():
            inputs = {"input_ids": input_ids, "attention_mask": input_mask}

            # The output tuple structure depends on the model used and the arguments invoked
            # For BERT-type models, this is
            # decoder_predictions, encoded_embeddings, encoded_attention_mask = model(**inputs)
            # For GPT2-type models, this at least starts with the decoder predictions
            # See the EncoderDecoderModel class for more details
            output = model(**inputs)**
```

This code fails in modeling_encoder_decoder, line 283 with

ValueError: You have to specify either input_ids or inputs_embeds

2. The pull request uses a GPT2 model as an example, but that no longer works because the code mentioned from #1 requires some parameters like encoder_hidden_states that GPT2 does not take at initialization. When I try to create a GPT2 model I get exceptions regarding this extra parameter. In other words, when I switch from a bert-bert model to a gpt2-gpt2 model, the code posted above fails in the "forward" method of the EncoderDecoderModel (line 283 of modeling_encoder_decoder) because "encoder_hidden_states" is an unexpected param for GPT2. Is this intended / is GPT2 no longer supported for an encoder decoder architecture using this code?

3. This one is just more of a general question... but since I'm posting the above 2 as issues anyways, I figured I'd add it here in case anybody can clarify and save a separate issue being created..

I believe I'm doing this part correctly, but it was not handled in the example code so want to verify if possible... For the attention mask for the decoder, during training all non-PAD tokens are expected to be unmasked, and during evaluation no mask should be provided and a default causal mask will be used, right?

@patrickvonplaten , tagging you in this issue as requested.

Thank you for your time!! Let me know if you need more code, again my code is 95% or so identical to the run_seq2seq.py example in the linked PR, just with some changes to account for recent modifications in modeling_encoder_decoder.py


[Communtiy notebooks] Fine-tuning / Training

Proposal of how notebooks for training could be added.


[Community notebooks] General notebooks

A proposal how we could link community notebooks.

I'm using the awesome notebook of @patil-suraj (`nlp` + `Trainer` + `transformers` :-)) as an example of how community models can be added.

@patil-suraj - could you maybe review the PR and see whether it's ok for you?


Reformer training error

# 🐛 Bug

When training a Reformer model from scratch, I got the following error:

**TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'**

## Information

I am trying to train a Reformer model from scratch on English documents. My data is one document per line:

The problem arises when using:
* [ x ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Split my training documents on '.' to create a corpus of sentences. Use Google SentencePiece script to train a tokenization model.
2. Use run_language_modeling.py with --mlm --tokenizer_name=path/to/pretrained_SP_tokenizer to train the model.
3. My config.json looks like this:

```
{
  "architectures": [
    "ReformerModelWithLMHead"
  ],
  "model_type": "reformer",
  "vocab_size": 32000
}
```

File "/Users/a9dvzzz/.virtualenvs/cf-mlc/lib/python3.7/site-packages/transformers/trainer.py", line 506, in _training_step
    outputs = model(**inputs)
  File "/usr/local/Cellar/python/3.7.4_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'masked_lm_labels'

## Expected behavior

The training process produces a saved Reformer model.

## Environment info
- `transformers` version: 2.9.1
- Platform: Darwin-19.4.0-x86_64-i386-64bit
- Python version: 3.7.4
- PyTorch version (GPU?): 1.3.0 (False)
- Tensorflow version (GPU?): 2.0.0 (False)
- Using GPU in script?: no, but it doesn't seem to matter, both failed.
- Using distributed or parallel set-up in script?: no


Avoid abort due to missing paths in case of '--save_total_limit' argument

Checkpoint path will be deleted when using --save_total_limit. torch.save() would not be able to store and abort.


BERT Fine-tuning problems

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Hello. I'm going to do a fine-tuning of BERT-base-uncased using the QA Dataset I made. However, the following error occurs: Could you tell me how to solve this problem?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:

```
Traceback (most recent call last):
  File "./examples/question-answering/run_squad.py", line 830, in <module>
    main()
  File "./examples/question-answering/run_squad.py", line 768, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False)
  File "./examples/question-answering/run_squad.py", line 452, in load_and_cache_examples
    examples = processor.get_train_examples(args.data_dir, filename=args.train_file)
  File "/home/address/anaconda3/lib/python3.7/site-packages/transformers/data/processors/squad.py", line 525, in get_train_examples
    return self._create_examples(input_data, "train")
  File "/home/address/anaconda3/lib/python3.7/site-packages/transformers/data/processors/squad.py", line 552, in _create_examples
    title = entry["title"]
TypeError: string indices must be integers
Traceback (most recent call last):
  File "/home/address/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/address/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/address/anaconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 253, in <module>
    main()
  File "/home/address/anaconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 249, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/home/address/anaconda3/bin/python', '-u', './examples/question-answering/run_squad.py', '--local_rank=1', '--model_type', 'bert', '--model_name_or_path', 'bert-base-uncased', '--do_train', '--do_eval', '--train_file', '/home/address/Desktop/address/train_split.json', '--predict_file', '/home/address/Desktop/address/val_split.json', '--learning_rate', '3e-5', '--num_train_epochs', '2', '--max_seq_length', '384', '--doc_stride', '128', '--output_dir', '../models/wwm_uncased_finetuned_squad/', '--per_gpu_eval_batch_size=3', '--per_gpu_train_batch_size=3']' returned non-zero exit status 1.
```


Added model cards for Romanian BERT models

model card

Related to pretrained model cards

Added model card for ``dumitrescustefan/bert-base-romanian-cased-v1`` and ``dumitrescustefan/bert-base-romanian-uncased-v1``


[T5 fp16] Fix fp16 in T5

This PR fixes the issue: #4287.

- A test for T5 is added.
- The function self.invert_attention_mask included a if statement now so that no errors will occur when using the function in `fp16` mode.


added model card for german-sentiment-bert

Related to pretrained model cards

I added a description for my german sentiment model. If you have any feedback or questions please let me know.


albertModel object has no attribute bias

transformers version:2.9.0

model = AlbertModel.from_pretrained("xxx", from_tf=True)

Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'attention', 'key', 'kernel'] from bert/encoder/transformer/group_0/inner_group_0/attention_1/self/key/kernel
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'attention', 'query', 'bias'] from bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/bias
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'attention', 'query', 'kernel'] from bert/encoder/transformer/group_0/inner_group_0/attention_1/self/query/kernel
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'attention', 'value', 'bias'] from bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/bias
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'attention', 'value', 'kernel'] from bert/encoder/transformer/group_0/inner_group_0/attention_1/self/value/kernel
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'ffn', 'bias'] from bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/bias
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'ffn', 'kernel'] from bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/dense/kernel
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'ffn_output', 'bias'] from bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/bias
Initialize PyTorch weight ['albert', 'encoder', 'albert_layer_groups', '0', 'albert_layers', '0', 'ffn_output', 'kernel'] from bert/encoder/transformer/group_0/inner_group_0/ffn_1/intermediate/output/dense/kernel
Initialize PyTorch weight ['albert', 'pooler', 'bias'] from bert/pooler/dense/bias
Initialize PyTorch weight ['albert', 'pooler', 'kernel'] from bert/pooler/dense/kernel
Traceback (most recent call last):
  File "d:\python_workbase\project\transformers_test\test.py", line 7, in <module>
    model = AlbertModel.from_pretrained("D:\work\model\\albert_tiny_zh_google", from_tf=True)
  File "D:\Programs\Python\Python37\lib\site-packages\transformers\modeling_utils.py", line 640, in from_pretrained
    model = cls.load_tf_weights(model, config, resolved_archive_file[:-6])  # Remove the '.index'
  File "D:\Programs\Python\Python37\lib\site-packages\transformers\modeling_albert.py", line 139, in load_tf_weights_in_albert
    pointer = getattr(pointer, "bias")
  File "D:\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 591, in __getattr__
    type(self).__name__, name))
AttributeError: 'AlbertModel' object has no attribute 'bias'


Create README.md

Related to pretrained model cards


Tag onnx export tests as slow

TensorFlow ONNX export test is very slow as it makes many many optimizations passes of the graph.

This PR marks both PyTorch & TensorFlow as slow, and keeps all the others (fast) as non-slow.


Adding optimizations block from ONNXRuntime.

cc @EmmaNingMS


🐛 Weird learning rate with TPU Trainer

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): **ELECTRA**

Language I am using the model on (English, Chinese ...): **English**

The problem arises when using:
* [ ] the official example scripts:
* [x] my own modified scripts:

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task:
* [x] my own task or dataset: CNN/DM

## To reproduce

I use `run_glue.py` as example to build a training script for TPU, with the Trainer API. My task is sequence classification with CNN/DM dataset.

I initialized the Trainer with following optimizer / scheduler :

```python
optimizer = AdamW(optimizer_grouped_parameters, lr=training_args.learning_rate, eps=training_args.adam_epsilon)
scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=training_args.warmup_steps, num_training_steps=287113)

trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=cnn_dm["train"] if training_args.do_train else None,
        eval_dataset=cnn_dm["validation"],
        data_collator=MyCollator(),
        prediction_loss_only=True,
        optimizers=(optimizer, scheduler),
    )
```

Now, the training procedure is working : the code run fine on 8 TPU core.

**But the loss is not decreasing.**

After looking into Tensorboards logs, I found the learning rate to be very weird :

![image](https://user-images.githubusercontent.com/43774355/82189972-3e888800-992b-11ea-8887-6715500f388d.png)

A few points to note :
* I specified a learning rate of **1e-4** (with the command argument `--learning_rate 1e-4`), but as you can see, the maximum value for the learning rate is **3.5e-6**.
* The shape of learning rate is not what I expected : After warmup, learning rate is supposed to decrease linearly, but instead, it stays fixed.

I don't know why the learning rate is being like this. Any idea what I might be doing wrong ?

_I can't share my notebook, but this seems to be the exact same issue with the official script `run_glue.py`, as described in #4358_

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: **2.9.1**
- Platform: **Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic**
- Python version: **3.6.9**
- PyTorch version (GPU?): **1.6.0a0+176174a (False)**
- Tensorflow version (GPU?): **2.2.0 (False)**
- Using GPU in script?: **No**
- Using distributed or parallel set-up in script?: **Yes : `xla_spawn.py`**

@julien-c


mbart config.json missing

I downloaded mbart from fairseq, there are dict.txt, model.pt, sentence.bpe.model in it but no config.json, where can we get it (and other necessary missing files)?
I use mbart as pretrained model.

Is bart-large trained on multilingual data?
 Anyone has compared bart with t5?


How to extract the best candidate after token classification?

Let's assume the model predicts the following for an input sequence.
```
The O
creation. O
date O
is O
27 B-DATE
Aug I-DATE
2020 I-DATE
and  O
update. O
date.  O
is  O
01-09-2020 B-DATE
```

How do you pick the best candidate for **creation date** from logist values?


Refactored the README.md file

Related to pretrained model cards


Lack of funetune examples for T5 model

# 🚀 Feature request

It seems like examples under transformers/examples doesn't support T5 except for translation.

## Motivation

We need more examples! It should be easy for some simple benchmarks.

## Your contribution

None currently.. But I am working on it!


BERT and other models pretraining from scratch example

Hi,
I've been finetuning lots of tasks using this repo. Thanks :)
But I couldn't find any pretraining from scratch examples.
Please let me know if you guys have any advices on that.
It would be very helpful for me to do my research.


Update README.md (model_card)

Related to pretrained model cards

- add a citation.
- modify the table of the BLUE benchmark.

The table of the first version was not displayed correctly on https://huggingface.co/seiya/oubiobert-base-uncased.
Could you please confirm that this fix will allow you to display it correctly?


How to change transformers model embedding layer weights

I trained my own tokenizer and added new words. Now I need to change the embedding size from the pretrained model. What I do is like this:

```
import transformers as tfm
import tensorflow as tf

backbone = tfm.TFRobertaModel.from_pretrained(PRETRAINED_PATH, output_hidden_states=True)
add_emb = tf.random.uniform(shape=(new_vocab_size, 768), minval=-1., maxval=1.)
new_emb = tf.concat((backbone.roberta.embeddings.word_embeddings, add_emb), 0)
backbone.roberta.weights[194] = new_emb
```
However, the shape of embedding weight is still the original vocab size.

But if I do

```
backbone = tfm.TFRobertaModel.from_pretrained(PRETRAINED_PATH, output_hidden_states=True)
add_emb = tf.random.uniform(shape=(new_vocab_size, 768), minval=-1., maxval=1.)
backbone.roberta.embeddings.word_embeddings= tf.concat((backbone.roberta.embeddings.word_embeddings, add_emb), 0)
```
Then the embedding weights will be removed from model ```trainable_weights``` and has only 198 elements instead of 199 in the original list.

Am I doing something wrong to change the embedding weights? Thanks!

The original stack overflow question is also posted:
https://stackoverflow.com/questions/61860156/how-to-change-transformers-model-embedding-layer-weights


[T5 Conf] rename docstring to acuatly argument names

As mentioned in issue: #4139, the docstring names of the T5 Config are confusing since those names cannot be used to set the arguments.

This PR renames the arguments in the docstring and adds an explanation that those arguments can also be accessed via their properties.

To not break backward compatibility, renaming the docstring is better than renaming the actual variables IMO.


[test_pipelines] Mark tests > 10s @slow, small speedups

- pass in num_beams=2 to `SummarizationPipelines`


BERT Tokenization problem when the input string has a "." in the string, like floating number

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using: Tokenizer
* [ ] the official example scripts: (give details below) N/A
* [ ] my own modified scripts: (give details below) N/A

The tasks I am working on is: Any
* [ ] an official GLUE/SQUaD task: (give the name) N/A
* [ ] my own task or dataset: (give details below) N/A

## To reproduce

Steps to reproduce the behavior:

1.  Load any BERT tokenizer
2. Tokenize something with a "." in between
3. Decode these ids, you will find it mismatch

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->
```
x = tokenizer.encode('AN.C', add_special_tokens=False)
z = tokenizer.decode(x)
```
It prints:
```
AN. C
```

## Expected behavior
```
AN.C
```
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform: Ubuntu
- Python version: 3.6
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): GPU
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO


[TF generate] Fix issue for batch output generation of different output length.

This PR fixes the issue: #4088.

A wrong variable was used in TF generate to determine the sentence length in the case when multiple outputs have different sentence lengths and the max sentence lengths is < the user defined `max_length`.

Also, both PT and TF generate are refactored a bit so that the `cur_length` variable is incremented directly after the `input_ids` are incremented.


Scaling text classification / reusing models

If I have a system, where I want to train many text classifiers for many users, how could I go about it with the transformers library in a scalable way?
Right now I would have to run let's say a 10min training per user on a RTX 2080 ti for Albert for the dataset I have. That doesn't scale if I have thousands of users.

If I understand correctly, in the sequence classification models, the whole transformer model is being trained, so the backpropagation happens through the whole network.

However, if I now want to reuse the model for another user, maybe just passing in a bit more labeled data to customize a base classifier, how could I go about that?

It seems to me, that I would have to basically "freeze" the whole "Bert" model, no matter which one I would use, and then only train a thin layer on top.

One possibility I see would be KNN using sentence transformers, I already asked in the repo there https://github.com/UKPLab/sentence-transformers/issues/209

Maybe someone here has an idea which approach would make sense for such a situation.

Thanks!


# 🐛 Bug

## Information

I was trying to reproduce the GLUE fine-tuning example (https://huggingface.co/transformers/examples.html#fine-tuning-example) when I ran into this error:

```
  File "~/anaconda3/lib/python3.7/site-packages/transformers/optimization.py", line 155, in step
    exp_avg.mul_(beta1).add_(grad, 1.0 - beta1)
TypeError: add_() takes 1 positional argument but 2 were given
```

The problem arises when using:
* [x] the official example scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)

## To reproduce

Steps to reproduce the behavior:
Follow the official guide example here:  https://huggingface.co/transformers/examples.html#fine-tuning-example

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.1
- Platform: Linux 5.4.0-29-generic x86_64 Ubuntu 20.04 LTS
- Python version: 3.7.6
- PyTorch version (GPU?): 1.5.0 (Yes)
- Tensorflow version (GPU?):
- Using GPU in script?: Yes, 1
- Using distributed or parallel set-up in script?:


Fixed spelling of training

Spelling of training was incorrect. Fixed it. Sorry for such a bad PR :(


GPT2 perplexity rolling/striding way for evaluating a document.

As I understand GPT2 uses TextDataset as a loader and it produces the example list in block sizes. So say we have a sentence "**we are in a climate crisis**" and have block size 3. So this will produce the example list as

`ex = [["we","are","in"],["a","climate","crisis"]]`

So in such a scenario when calculating overall perplexity for the document the word "a" has no previous context and "climate" only has "a" as context. What I would want would ideally the context to be a in rolling/striding way. So  I edited the text loader to produce a list like:

`ex = [["we","are","in"],["are","in","a"],["in,"a","climate],["a","climate","crisis"]]`

Now, if I calculate perplexity for this ex list surely a lot of words will be counted multiple times as they are in the lists in multiple times. But in in a way for every instance from 2nd onwards (ex[1] above) i Would only want to consider last word for perplexity/loss calculation.  So my query is how to tackle it, should I use attention mask so that say in the case of

`ex[1] =["are","in","a"] `
with mask of [0,0,1] for loss calculation (and hence perplexity) it only takes "a" into account but for getting the context of "a' will also take previous 2 words in account?

Any help on the best way to approach this problem will be much appreciated.


Get BERT sentence encoding

I am trying to access to encoding of sentences in the vaious layers in pre-trained BERT model.
So it should be something like this:

```
sentence = 'We bought a new car'
bert_encoder = load_encoder('bert-base-uncased')
enc = bert_encoder.encode(sentence)
enc.get_layer[0] #this is the first layer
enc.get_layer[-1] #this is the last layer

```
What is the best way to do it?
Thanks!


Modify example of usage

Related to pretrained model cards

I followed the google example of usage for its electra small model but i have seen it is not meaningful, so i created a better example


# 🐛 Bug
Tensorflow NER Training script Not working
## Information
I am following the exact guide at https://github.com/huggingface/transformers/tree/master/examples/token-classification
Model I am using (Bert, XLNet ...): BERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
python run_tf_ner.py --data_dir ./ \
--labels ./labels.txt \
--model_name_or_path $BERT_MODEL \
--output_dir $OUTPUT_DIR \
--max_seq_length  $MAX_LENGTH \
--num_train_epochs $NUM_EPOCHS \
--per_device_train_batch_size $BATCH_SIZE \
--save_steps $SAVE_STEPS \
--seed $SEED \
--do_train \
--do_eval \
--do_predict

The tasks I am working on is:
Training NER on germeval data

Steps to reproduce the behavior:
Follow the official guide at https://github.com/huggingface/transformers/tree/master/examples/token-classification

<!-- ValueError: Variable <tf.Variable 'tf_bert_for_token_classification/bert/pooler/dense/kernel:0' shape=(768, 768) dtype=float32> has `None` for gradient. Please make sure that all of your ops have
 a gradient defined (i.e. are differentiable). Common ops without gradient: K.argmax, K.round, K.eval.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!--
- `transformers` version: 2.9.1
- Platform: Linux-3.10.0-1062.9.1.el7.x86_64-x86_64-with-centos-7.7.1908-Core
- Python version: 3.6.10
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.2.0 (True)
- Using GPU in script?: Yes, both Single and multi-GPU training
- Using distributed or parallel set-up in script?: Yes-->

- `transformers` version:

The documentation i guess is not updated for Tensorflow training, an additional parameter "logging_dir" is required in case of TF.


Pipeline for Conditional Generation (T5 type models)

As text-to-text models (like T5) increase the accessibility of multi-task learning, it also makes sense to have a flexible "Conditional Generation" pipeline.

For example, I should be able to use this pipeline for a multitude of tasks depending on how I format the text input (examples in Appendix D of the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf)). As a baseline, this should be able to work on `T5ForConditionalGeneration` and allow for any of the tasks that are learned by the open sourced T5 model.

Since T5 isn't usable for `TextGenerationPipeline`, I propose we add a `ConditionalGenerationPipeline`.

Please do let me know if there is an existing way to perform the above via pipelines, or if adding a pipeline doesn't makes sense for this; otherwise, I can submit a PR for the above `ConditionalGenerationPipeline` 🙂


Remove pytorch codes in Class TFXLNetMainLayer

This PR removes pytorch codes in Class TFXLNetMainLayer.


add model card for t5-base-squad

Related to pretrained model cards

Model card for https://huggingface.co/valhalla/t5-base-squad


Request to add MobileBert

# 🌟 New model addition
MobileBERT
## Model description

MobileBERT is a thin version of BERT_LARGE, while equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.

## Open source status

* [ ] the model implementation is available: (give details)
https://github.com/google-research/google-research/tree/master/mobilebert

* [ ] the model weights are available: (give details)
https://storage.googleapis.com/cloud-tpu-checkpoints/mobilebert/uncased_L-24_H-128_B-512_A-4_F-4_OPT.tar.gz

* [ ] who are the authors: (mention them, if possible by @gh-username)
Google LLC
Xiaodan Song
Zhiqing Sun
Hongkun Yu
Denny Zou


fix(run_language_modeling): use arg overwrite_cache

In `run_language_modeling.py`, arg `overwrite_cache` was unused.


Summarization Fine Tuning

Discussion

Discussion on a topic (keep it focused or open a new issue though)

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I tried using T5 and Bart but the abstraction summarization on scientific texts does not seem to give the results I want since I think they are both trained on news corpora. I have scraped all of the free PMC articles and I am thinking about fine-tuning a seq2seq model between the articles and their abstracts to make an abstractive summarizer for scientific texts. This Medium article (https://medium.com/huggingface/encoder-decoders-in-transformers-a-hybrid-pre-trained-architecture-for-seq2seq-af4d7bf14bb8) provides a bit of an introduction to how to approach this but does not quite go into detail so I am wondering how to approach this.

I'm not really asking for help being stuck but I just don't really know how to approach this problem.

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:
https://stackoverflow.com/questions/61826443/train-custom-seq2seq-transformers-model


add BERT trained from review corpus.

model card

Related to pretrained model cards

add BERT trained from review corpus.


feat(wandb): display logger

Logger info when `wandb` not installed was set to `info` which does not display by default.
It has been changed to `warning`.


Map optimizer to correct device after loading from checkpoint.

Loading from `optimizer.pt` causes `optimizer` to be mapped to the same device as the saved `optimizer.pt`. In most cases it's `cuda:0`(saved by local master), which puts all optimizers on
gpu0, causing OOM more easily in multi-gpu training.

Might fix issues like [#3730](https://github.com/huggingface/transformers/issues/3730).


Run Language Modeling on 8 TPU cores doesn't seem to terminate

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): DistilGPT2 & GPT2

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I'm trying to test ```run_language_modeling.py``` on DistilGPT2 using all 8 TPU cores. Running on 1 core executes fine, but when I attempt to run on all 8 cores, it finishes finetuning then gets stuck on "Training completed. Do not forget to share your model on huggingface.co/models =)" and doesn't terminate.

When I check the output directory, I only see two files: config.json and pytorch_model.bin. There should be seven files in the output directory.

I'm running this on a Colab TPU Notebook.

## To reproduce

Steps to reproduce the behavior:
```
VERSION = "nightly"  #@param ["1.5" , "20200325", "nightly"]
!curl https://raw.githubusercontent.com/pytorch/xla/master/contrib/scripts/env-setup.py -o pytorch-xla-env-setup.py
!python pytorch-xla-env-setup.py --version $VERSION

!wget https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-v1.zip
!unzip wikitext-2-v1.zip && rm wikitext-2-v1.zip
!pip install transformers
!git clone https://github.com/huggingface/transformers.git

!python transformers/examples/xla_spawn.py --num_cores 8 \
	transformers/examples/language-modeling/run_language_modeling.py \
	--output_dir=output \
        --model_type=distilgpt2 \
        --model_name_or_path=distilgpt2 \
        --train_data_file=wikitext-2/wiki.train.tokens \
        --do_train \
        --overwrite_output_dir

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Script terminates and 7 files are in the output folder:
* config.json
* pytorch_model.bin
* tokenizer_config.json
* vocab.json
* merges.txt
* special_tokens_map.json
* training_args.bin

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.1
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0a0+83df3be (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: yes


[TF T5] More coherent naming for inputs

In TF we have to name the first argument of the `call` function "inputs", due to some inner keras logic (I think), see: https://github.com/huggingface/transformers/pull/3547 . Having both names `inputs` and `input_ids` can thus lead to confusion, see #3626 .

This PR adopts the consistent naming `inputs` in the whole file.


BertWordPieceTokenizer cannot be pickled

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**Bert**

Language I am using the model on (English, Chinese ...):
**English**

The problem arises when using:
* [X] my own modified scripts:

The tasks I am working on is:
* [X] my own task or dataset:

## To reproduce

``` python
import torch
import tokenizers
import pandas as pd
from torch.utils import data

class config:
    MAX_LEN = 128
    TRAIN_BATCH_SIZE = 64
    VALID_BATCH_SIZE = 16
    EPOCHS = 5
    BERT_PATH = "../input/bert-base-uncased/"
    MODEL_PATH = "model.bin"
    TRAINING_FILE = "../input/tweet-sentiment-extraction/train_folds.csv"
    TOKENIZER = tokenizers.BertWordPieceTokenizer(
        f"{BERT_PATH}/vocab.txt",
        lowercase=True
    )

def process_data(tweet, selected_text, sentiment, tokenizer, max_len):
    len_st = len(selected_text)
    idx0 = -1
    idx1 = -1
    for ind in (i for i, e in enumerate(tweet) if e == selected_text[0]):
        if tweet[ind: ind+len_st] == selected_text:
            idx0 = ind
            idx1 = ind + len_st - 1
            break

    char_targets = [0] * len(tweet)
    if idx0 != -1 and idx1 != -1 :
        for ct in range(idx0, idx1 + 1):
            char_targets[ct] = 1

    tok_tweet = tokenizer.encode(tweet)
    input_ids_orig = tok_tweet.ids[1:-1]
    tweet_offsets = tok_tweet.offsets[1:-1]

    target_idx = []
    for j, (offset1, offset2) in enumerate(tweet_offsets):
        if sum(char_targets[offset1: offset2]) > 0:
            target_idx.append(j)

    targets_start = target_idx[0]
    targets_end = target_idx[-1]

    sentiment_id = {
        'positive': 3893,
        'negative': 4997,
        'neutral': 8699
    }

    input_ids = [101] + [sentiment_id[sentiment]] + [102] + input_ids_orig + [102]
    token_type_ids = [0, 0, 0] + [1] * (len(input_ids_orig) + 1)
    mask = [1] * len(token_type_ids)
    tweet_offsets = [(0, 0)] * 3 + tweet_offsets + [(0, 0)]
    targets_start += 3
    targets_end += 3

    padding_length = max_len - len(input_ids)
    if padding_length > 0:
        input_ids = input_ids + ([0] * padding_length)
        mask = mask + ([0] * padding_length)
        token_type_ids = token_type_ids + ([0] * padding_length)
        tweet_offsets = tweet_offsets + ([(0, 0)] * padding_length)

    return {
        'ids': input_ids,
        'mask': mask,
        'token_type_ids': token_type_ids,
        'targets_start': targets_start,
        'targets_end': targets_end,
        'orig_tweet': tweet,
        'orig_selected': selected_text,
        'sentiment': sentiment,
        'offsets': tweet_offsets
    }

class TweetDataset(data.Dataset):
    def __init__(self, tweet, sentiment, selected_text):
        self.tweet = tweet
        self.sentiment = sentiment
        self.selected_text = selected_text
        self.tokenizer = config.TOKENIZER
        self.max_len = config.MAX_LEN

    def __len__(self):
        return len(self.tweet)

    def __getitem__(self, item):
        data = process_data(
            self.tweet[item],
            self.selected_text[item],
            self.sentiment[item],
            self.tokenizer,
            self.max_len
        )

        return {
            'ids': torch.tensor(data["ids"], dtype=torch.long),
            'mask': torch.tensor(data["mask"], dtype=torch.long),
            'token_type_ids': torch.tensor(data["token_type_ids"], dtype=torch.long),
            'targets_start': torch.tensor(data["targets_start"], dtype=torch.long),
            'targets_end': torch.tensor(data["targets_end"], dtype=torch.long),
            'orig_tweet': data["orig_tweet"],
            'orig_selected': data["orig_selected"],
            'sentiment': data["sentiment"],
            'offsets': torch.tensor(data["offsets"], dtype=torch.long)
        }

dfx = pd.read_csv(config.TRAINING_FILE)
fold = 4
df_train = dfx[dfx.kfold != fold].reset_index(drop=True)
df_valid = dfx[dfx.kfold == fold].reset_index(drop=True)

train_dataset = TweetDataset(
        tweet=dfx.text.values,
        sentiment=dfx.sentiment.values,
        selected_text=dfx.selected_text.values
    )

train_data_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=config.TRAIN_BATCH_SIZE,
        num_workers=1
    )

if __name__ =='__main__':
    a = enumerate(train_data_loader)
```

## Expected behavior

The enumerate should return the iterable.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

Output of `transformers-cli env`

transformers version: 2.9.1
Platform: Windows-10-10.0.18362-SP0
Python version: 3.8.2
PyTorch version (GPU?): 1.5.0 (True)
Tensorflow version (GPU?): not installed (NA)
Using GPU in script?: Yes
Using distributed or parallel set-up in script?: No


Pipeline for question generation

# 🚀 Feature request

I can see there are pipelines for question answering, text summarisation and text generation. In my field I'm researching how question generation can be used in education research. I would love to see this pipeline added. I imagine it's a variation of question answering and text summarisation.

The paper 'Question Generation by Transformers' by Kettip Kriangchaivech, Artit Wangperawong and provides a good overview of using the SQUAD data set using questions as the output sequence with reference questions and contexts.

Artit also has an implementation called text2text https://github.com/artitw/text2text

## Motivation

It would be useful to have an official pipeline part of the hunggingface library for this use case.

## Your contribution

I'm happy to contribute some funds to pay some developers if need be but I don't have enough Python technical expertise to contribute an PRs myself.


Trainer is missing sampler.set_epoch for distributed mode

# 🐛 Bug

## Information

`train_dataloader.sampler.set_epoch(epoch)` is missing before the start of each epoch in [trainer.py](https://github.com/huggingface/transformers/blob/3e0f06210646a440509efa718b30d18322d6a830/src/transformers/trainer.py#L406).

According to [here](https://pytorch.org/docs/stable/data.html#torch.utils.data.distributed.DistributedSampler) :

> In distributed mode, calling the `set_epoch` method is needed to make shuffling work; each process will use the same random seed otherwise.


Training TFBertForQuestionAnswering on custom SquadV1 data

Hello.

TLDR: If there is any minimal code that trains a TFBertForQuestionAnswering on custom squad-v1 (not from `nlp.load_dataset`)

I've tried in several ways and encountered some problems.

This is the minimal code i'm trying to activate:
```python
    args = argparse.Namespace(**bert_config)
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

    processor = SquadV1Processor()
    # processor = SquadV2Processor()
    examples = processor.get_train_examples(args.data_dir, filename=args.train_file)
    train_dataset = squad_convert_examples_to_features(
        examples=examples,
        tokenizer=tokenizer,
        max_seq_length=args.max_seq_length,
        doc_stride=args.doc_stride,
        max_query_length=args.max_query_length,
        is_training=True,
        return_dataset="tf"
    )

    model = TFBertForQuestionAnswering.from_pretrained("bert-base-cased")
    loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
    opt = tf.keras.optimizers.Adam(learning_rate=3e-5)

    model.compile(optimizer=opt,
                  loss={'start_position': loss_fn, 'end_position': loss_fn},
                  loss_weights={'start_position': 1., 'end_position': 1.},
                  metrics=['accuracy'])

    # Now let's train our model
    try:
        history = model.fit(train_dataset, epochs=1, steps_per_epoch=3)
    except Exception as ex:
        print(f"Failed using fit, {ex}")
        history = model.fit_generator(train_dataset, epochs=1, steps_per_epoch=3)

```
The current errors are:
with fit:
```python
    x = standardize_function(x)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 660, in standardize_function
    standardize(dataset, extract_tensors_from_dataset=False)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2360, in _standardize_user_data
    self._compile_from_inputs(all_inputs, y_input, x, y)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training.py", line 2580, in _compile_from_inputs
    target, self.outputs)
  File "/home/ec2-user/anaconda3/envs/yonatan_env_tf2/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/training_utils.py", line 1341, in cast_if_floating_dtype_and_mismatch
    if target.dtype != out.dtype:
AttributeError: 'str' object has no attribute 'dtype'
```
with fit_generator:
```python
ValueError: Unknown entries in loss dictionary: ['start_position', 'end_position']. Only expected following keys: ['output_1', 'output_2']
```
The dataset that returns from squad_convert_examples_to_features is of type- `tensorflow.python.data.ops.dataset_ops.FlatMapDataset` and i'm not sure how to change it's columns from start_position to output_1 and end_position to output_2. I've also asked it on stackoverflow: https://stackoverflow.com/questions/61830361/how-the-change-column-name-in-tensorflow-flatmapdataset

I've seen the colab tutorial of the nlp package. It has simple code:
```python
train_tf_dataset = nlp.load_dataset('squad', split="train")
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')
def convert_to_tf_features(example_batch):
    # Tokenize contexts and questions (as pairs of inputs)
    input_pairs = list(zip(example_batch['context'], example_batch['question']))
    encodings = tokenizer.batch_encode_plus(input_pairs, pad_to_max_length=True)

    # Compute start and end tokens for labels using Transformers's fast tokenizers alignement methods.
    start_positions, end_positions = [], []
    for i, (context, answer) in enumerate(zip(example_batch['context'], example_batch['answers'])):
        start_idx, end_idx = get_correct_alignement(context, answer)
        start_positions.append([encodings.char_to_token(i, start_idx)])
        end_positions.append([encodings.char_to_token(i, end_idx-1)])

    if start_positions and end_positions:
      encodings.update({'start_positions': start_positions,
                        'end_positions': end_positions})
    return encodings

train_tf_dataset = train_tf_dataset.map(convert_to_tf_features, batched=True)

columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_tf_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_tf_dataset[x] for x in columns[:3]}
labels = {"output_1": train_tf_dataset["start_positions"]}
labels["output_2"] = train_tf_dataset["end_positions"]
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
# Let's load a pretrained TF2 Bert model and a simple optimizer
from transformers import TFBertForQuestionAnswering

model = TFBertForQuestionAnswering.from_pretrained("bert-base-cased")
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
opt = tf.keras.optimizers.Adam(learning_rate=3e-5)
model.compile(optimizer=opt,
              loss={'output_1': loss_fn, 'output_2': loss_fn},
              loss_weights={'output_1': 1., 'output_2': 1.},
              metrics=['accuracy'])
# Now let's train our model
model.fit(tfdataset, epochs=1, steps_per_epoch=3)
```

I can't do the same as this code, because the dataset here is of type - `nlp.arrow_dataset.Dataset`.
I've tried to convert my `tensorflow.python.data.ops.dataset_ops.FlatMapDataset` to `nlp.arrow_dataset.Dataset` (and then mimic the last code here) but didn't find suitable way.

Edit:
I've succeeded to change the names of the output in the `FlatMapDataset` to output_1 and output_2, and now I receive the following error:
```
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument:  logits and labels must have the same first dimension, got logits shape [384,1] and labels shape [1]
	 [[node loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at /yonatab/ZeroShot/transformers_experiments/src/minimal_example_for_git.py:53) ]]
	 [[Reshape_820/_546]]
  (1) Invalid argument:  logits and labels must have the same first dimension, got logits shape [384,1] and labels shape [1]
	 [[node loss/output_1_loss/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits (defined at /yonatab/ZeroShot/transformers_experiments/src/minimal_example_for_git.py:53) ]]
```
How can I create a tf dataset with `squad_convert_examples_to_features` (and return type `tf`) and train a TF model on it?

Thanks


Wrong model or tokenizer for MarianMT

While we [can't save](https://github.com/huggingface/transformers/issues/4371) `MarianTokenizer` to local directory, I found model weights and configs on [this page](https://huggingface.co/Helsinki-NLP/opus-mt-en-ROMANCE), in `List all files in model ` link.
I downloaded  these files, but I think this is wrong configs, because I can't reproduce even simplest example from [here](https://huggingface.co/transformers/model_doc/marian.html)
This text
```
1)  '>>fr<< this is a sentence in english that we want to translate to french',
2) '>>pt<< This should go to portuguese',
3) '>>es<< And this to Spanish'
should became
1) "c'est une phrase en anglais que nous voulons traduire en français",
2) 'Isto deve ir para o português.',
3) 'Y esto al español'
```
with model and configs downloaded from the link above produces these results:
```
1) "c'est une phrase en anglais que nous voulons traduire en français" (as expected)
2) "Questo deve ir in portughese"
3) "E questo a spagnol"
```
This is definitely wrong, can you help?


MarianMT = How to return 5 best candidates for a translation.

This is a code for normal translation. This is the most probable translation. How can I return lets says 5 best candidates for translation for every single word (beam size would be 1)?

This model only return just the best word, which gives us a better translation. But I want to use it as a language model.

Is this even possible. I was looking at classes and code but I am not sure how would I do it.

    from transformers import MarianMTModel, MarianTokenizer
    src_text = [
        '>>fr<< this is a sentence in english that we want to translate to french',
    ]
    model_name = 'Helsinki-NLP/opus-mt-en-ROMANCE'
    tokenizer = MarianTokenizer.from_pretrained(model_name)
    print(tokenizer.supported_language_codes)
    model = MarianMTModel.from_pretrained(model_name)
    translated = model.generate(**tokenizer.prepare_translation_batch(src_text))
    tgt_text = [tokenizer.decode(t, skip_special_tokens=True) for t in translated]


the special token of XLNet

hello,

may i ask you the special tokens of XLNet are same as BERT, in which they are '[CLS]' and '[SEP]'? Because i found the special tokens of XLNet are '<cls>' and '<sep>' in the original code, however, many public introduction about Xlnet still use the same tokens '[CLS]' and '[SEP]' as BERT. is it ok? are they same and don't matter?


BertTokenizerFast does not load custom vocab created from Tokenizer library

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
Bert ["iuliaturc/bert_uncased_L-2_H-128_A-2"]
https://huggingface.co/iuliaturc/bert_uncased_L-2_H-128_A-2

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
my own modified scripts: (give details below)

The tasks I am working on is:
my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
see comments in the code

1. Train model and tokenizer on new tokenizers library
2. try to add new tokens to tokenizer
3. Resize token embeddings for model

### Problem 1 :
If Loading with AutoTokenizer, can not use BertTokenizerFast but able to use custom vocabs

4. save tokenizer using save_tokenizer
5. Load using  AutoTokenizer

### Problem 2:
If Loading with BertTokenizerFast, can use the functioanlity of BertTokenizerFast but losed the custom vocabs

4. save tokenizer using save_tokenizer
5. Load using  BertTokenizerFast

```python

# Loaded Model and Tokenizer Default
tokenizer = BertTokenizerFast.from_pretrained("""iuliaturc/bert_uncased_L-2_H-128_A-2""")
model = AutoModel.from_pretrained("""iuliaturc/bert_uncased_L-2_H-128_A-2""")

# Added Custom vocab to the tokenizer

def add_vocab_to_model(df, model, tokenizer, old_vocab, vocab_size=30000):
    """Adds new vocab to tokenizer and randomly initialises rows for new vocab in the model"""
    PATH = Path('./lm_data')
    PATH.mkdir(exist_ok=True)

    df.dropna(inplace=True)
    lm_text = ' '.join(df['text'])
    lm_text = lm_text.replace("■","")
    lm_text = re.sub(r'[^\x00-\x7F]+',' ', lm_text)
    lm_text = re.sub(r'[^\u0000-\u007F]+',' ', lm_text)

    with open('./lm_data/data.txt', mode='w') as f:
        f.write(lm_text)

    paths = [str(x) for x in Path("./lm_data/").glob("**/*.txt")]

    # Initialize a tokenizer
    tokenizer_new = BertWordPieceTokenizer(old_vocab, lowercase=True)

    # Customize training
    tokenizer_new.train(files=paths, vocab_size=vocab_size, min_frequency=20)
    tokenizer_new.save(".", "./lm_data/new")

    new_vocab = open('./lm_data/new-vocab.txt', 'r').read().split('\n')
    new_vocab.remove('')

    print('Adding new tokens to vocab')
    n_orig_tokens = len(tokenizer)
    tokenizer.add_tokens(new_vocab)
    print('Original no. of tokens: %s'%n_orig_tokens)
    print('Final no. of tokens: %s'%len(tokenizer))

    print('Initialised random emb for new tokens')
    model.resize_token_embeddings(len(tokenizer))

    return model, tokenizer

# Just needs to pass a dataframe which is having a column with name "text"
model1, tokenizer1 = add_vocab_to_model(dataframe, model, tokenizer, 'bert-base-uncased-vocab.txt', vocab_size=10000)
#Adding new tokens to vocab
#Original no. of tokens: 30522
#Final no. of tokens: 34340
#Initialised random emb for new tokens

print(len(tokenizer1))
# Got the result : 34340
#Finalized Result

print(type(tokenizer1))
# transformers.tokenization_bert.BertTokenizerFast
# Final Tokenizer

# Problem 1 :
# If Loading with AutoTokenizer, can not use BertTokenizerFast but able to use custom vocabs

# Saved and Loaded Model Again
tokenizer1.save_pretrained("./tokenizer")
tokenizer2 = AutoTokenizer.from_pretrained("./tokenizer")

print(len(tokenizer2)
# Got 34340
# Which is Correct.

print(type(tokenizer2))
# transformers.tokenization_bert.BertTokenizer
# Not able to load BertTokenizerFast defaulting back to BertTokenizer

# Problem 2:
# If Loading with BertTokenizerFast, can use the functioanlity of BertTokenizerFast but losed the custom vocabs

# Saved and Loaded Model Again
tokenizer1.save_pretrained("./tokenizer")
tokenizer3 = BertTokenizerFast.from_pretrained("./tokenizer")

print(len(tokenizer3)
# Got 30522
# Losing the added custom vocabs

print(type(tokenizer3))
# transformers.tokenization_bert.BertTokenizerFast
#which is correct

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
1. If Loading tokenizer using AutoTokenizer it should load the BertTokenizerFast
2. If Loading tokenizer using BertTokenizerFast, it should maintain custom vocabs

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.9.1
- Platform: Windows
- Python version:  3.7.7
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): NA
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


MarianMTModel translate {tgt}-{src}

Hello!
It is possible to use `MarianMTModels` for translation from the target language to source, i.e. backward?


[PretrainedTokenizer] is  <unk> a special token?

Core: Tokenization

Internals of the library; Tokenization.

This confused me a lot as I was trying to add common tests for Marian.

I believe special_tokens_mask [must](
https://github.com/huggingface/transformers/blob/master/tests/test_tokenization_common.py#L445) return 0 for positions with unk_token_id, but `<unk>` is in `SpecialTokensMixin.all_special_ids`.

Relatedly, this [RobertaTokenizer](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_roberta.py#L204) code suggests there should be a relationship between `all_special_ids` and `special_token_mask` logic.

Maybe `<unk>` shouldnt be in `SpecialTokensMixin.all_special_ids`?


[cleanup] test_tokenization_common.py

- TesterMixin implements some oft repeated functions
- split long tests into multiple tests cases for better tracebacks
- misc typos
- type hints


[MarianTokenizer] implement save_vocabulary and other common methods

- adds `test_tokenization_marian.py` which runs the common tokenizer tests.
- adds `save_vocabulary` and other methods to `MarianTokenizer` to make the common tests pass.


[docs] AutoModelWithLMHead(model_name, **kwargs)

`AutoModelWithLMHead.from_pretrained` cannot accept `output_attentions=True`

This [docstring](https://huggingface.co/transformers/model_doc/auto.html?highlight=transformers%20automodelwithlmhead#transformers.AutoModelWithLMHead) suggests that it can.

Thanks @jrvc for discovering!

```python
import transformers
modelname='Helsinki-NLP/opus-mt-en-de'
config_overrider={'output_attentions':True, 'output_hidden_states':True}
self.model = transformers.AutoModelWithLMHead.from_pretrained(modelname, **config_overrider)
=>
*** TypeError: __init__() got an unexpected keyword argument 'output_attentions'
```

I will dig deeper!


[Bart/Marian] ignore output_attentions when invoked through AutoModel

[ ] should mimic behavior of other models
[ ] backwards compat?


Finetuning BERT classifier on a non-GLUE dataset in GLUE format

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
Hello,

I have a dataset of my own, formatted in GLUE format. Is it possible to use the script `run_glue.py` to finetune BERT on this dataset? If that's not possible, is there an alternative script for finetuning a classifier on arbitrary GLUE-formatted dataset?

Thanks!

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Should return overflowing information for the log

closes #4380


Attempt to unpin torch version for Github Action.


Issue with lr during training from scratch

I'm trying to resume my pretraining of RoBERTa model and I noticed that everything is OK except learning rate. I thought that it should continue to decrease from last checkpoint (lr = 7e-7) but lr is much bigger (lr = 5e-5) than it has to be.

this is my parameters to run script:

- 	--model_name_or_path ./ROBERTA-small-v1/checkpoint-30000
-   --train_data_file TEXT.txt
-   --output_dir ./ROBERTA-small-v2
- 	--mlm
- 	--config_name ./ROBERTA_SMILES-v1/checkpoint-30000
- 	--tokenizer_name ./ROBERTA
- 	--do_train
- 	--line_by_line
- 	--num_train_epochs 10
- 	--save_total_limit 1
- 	--save_steps 2000
- 	--per_gpu_train_batch_size 16
- 	--seed 42
- 	--logging_steps 250


Need clarity on training Albert from scratch

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->
Using transformers 2.9.1
I am following EsperBERTo tutorial for Training Albert LM from scratch. I am able to start training for roberta but for Albert getting tokenizer issue:
```
05/15/2020 16:23:29 - INFO - transformers.training_args -   PyTorch: setting up devices
05/15/2020 16:23:29 - WARNING - __main__ -   Process rank: -1, device: cuda, n_gpu: 2, distributed training: False, 16-bits training: False
05/15/2020 16:23:29 - INFO - __main__ -   Training/evaluation parameters TrainingArguments(output_dir='./EsperBERTo-small-v1', overwrite_output_dir=False, do_train=True, do_eval=False, do_predict=False, evaluate_during_training=False, per_gpu_train_batch_size=16, per_gpu_eval_batch_size=8, gradient_accumulation_steps=1, learning_rate=0.0001, weight_decay=0.0, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=1.0, max_steps=-1, warmup_steps=0, logging_dir=None, logging_first_step=False, logging_steps=500, save_steps=2000, save_total_limit=2, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False)
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   loading configuration file ./EsperBERTo/config.json
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   Model config AlbertConfig {
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 52000
}

05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   loading configuration file ./EsperBERTo/config.json
05/15/2020 16:23:29 - INFO - transformers.configuration_utils -   Model config AlbertConfig {
  "architectures": [
    "AlbertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0,
  "bos_token_id": 2,
  "classifier_dropout_prob": 0.1,
  "down_scale_factor": 1,
  "embedding_size": 128,
  "eos_token_id": 3,
  "gap_size": 0,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "inner_group_num": 1,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "albert",
  "net_structure_type": 0,
  "num_attention_heads": 12,
  "num_hidden_groups": 1,
  "num_hidden_layers": 12,
  "num_memory_blocks": 0,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 52000
}

05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Model name './EsperBERTo' not found in model shortcut name list (albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2). Assuming './EsperBERTo' is a path, a model identifier, or url to a directory containing tokenizer files.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/spiece.model. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/added_tokens.json. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   Didn't find file ./EsperBERTo/special_tokens_map.json. We won't load it.
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file None
05/15/2020 16:23:29 - INFO - transformers.tokenization_utils -   loading file ./EsperBERTo/tokenizer_config.json
Traceback (most recent call last):
  File "/home/gamut/Downloads/transformers_source/examples/language-modeling/run_language_modeling.py", line 289, in <module>
    main()
  File "/home/gamut/Downloads/transformers_source/examples/language-modeling/run_language_modeling.py", line 188, in main
    tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_auto.py", line 203, in from_pretrained
    return tokenizer_class_py.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 902, in from_pretrained
    return cls._from_pretrained(*inputs, **kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 1055, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/transformers/tokenization_albert.py", line 155, in __init__
    self.sp_model.Load(vocab_file)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/sentencepiece.py", line 275, in _sentencepiece_processor_load
    return self._Load_native(model_file)
  File "/home/gamut/anaconda2/envs/xyz/hugging_source/lib/python3.6/site-packages/sentencepiece.py", line 75, in Load
    return _sentencepiece.SentencePieceProcessor_Load(self, filename)
TypeError: not a string
CPU times: user 13.6 ms, sys: 27.4 ms, total: 41.1 ms
Wall time: 1.12 s
```
I guess it was looking for spiece model. So I added spice.model in EsperBERTo folder and it started training.
Now I confuse as vocab it is using now is generated using Huggingface tokenizer and taking spice.model also... can any one clarify whats going on here?


Unknown task fill-mask

# 🐛 Bug

## Information

I am trying to create a "Pipeline" for fill-mask task

Language I am using the model on is English

The problem arises when using:
My own modified scripts: (give details below)
```
import transformers
transformers.__version__
```
>>'2.3.0'
```
from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in <mask>')
```
>>---------------------------------------------------------------------------
>>KeyError                                  Traceback (most recent call last)
>><ipython-input-22-d415d13f7528> in <module>
>>----> 1 nlp_fill = pipeline('fill-mask')
>>     2 nlp_fill('Hugging Face is a French company based in <mask>')
>>
>>~\Anaconda3\lib\site-packages\transformers\pipelines.py in pipeline(task, model, config, tokenizer, modelcard, **kwargs)
>>    848     # Retrieve the task
>>    849     if task not in SUPPORTED_TASKS:
>>--> 850         raise KeyError("Unknown task {}, available tasks are {}".format(task, list(SUPPORTED_TASKS.keys())))
>>    851
>>    852     framework = get_framework(model)
>> **KeyError**: "Unknown task fill-mask, available tasks are ['feature-extraction', 'sentiment-analysis', 'ner', 'question-answering']"

The tasks I am working on is:
My own task or dataset. Trying using Pipeline.

## To reproduce

Steps to reproduce the behavior:

from transformers import pipeline
nlp_fill = pipeline('fill-mask')
nlp_fill('Hugging Face is a French company based in <mask>')

## Expected behavior
Should have got a filled output with a score.

## Environment info
'transformers-cli' is not recognized as an internal or external command, operable program or batch file.

- `transformers` version: 2.3.0
- Platform: Windows Jupyter notebook
- Python version: 3.7.4


max_qa_length is needed for funetune on multiple-choice problems

# 🚀 Feature request
In examples/multiple-choice/run_multiple_choice.py, the max_qa_length is not implemented as that in google/Albert, which would cause trouble when funetuning RACE.
This should be easy becauce there are similar codes in codes for run_squad.py.
### Your contribution
I have done the script and get similar RACE performance compared to google/Albert.

# Current Bug
In examples/multiple-choice/utils_multiple_choice.py
```python
# From 537 lines:
            # It should be =>
            # inputs = tokenizer.encode_plus(
            #    text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True, return_overflowing_tokens=True
            # )
            # Or the log info below would never be activated.
            inputs = tokenizer.encode_plus(
                text_a, text_b, add_special_tokens=True, max_length=max_length, pad_to_max_length=True,
            )
            if "num_truncated_tokens" in inputs and inputs["num_truncated_tokens"] > 0:
                logger.info(
                    "Attention! you are cropping tokens (swag task is ok). "
                    "If you are training ARC and RACE and you are poping question + options,"
                    "you need to try to use a bigger max seq length!"
                )
```


MNLI finetuning results affected by values of max_steps

# 🐛 Bug
If you follow my bash script and change MAX from 40000 to 50000, the performance is dramatically affected. However, 50000 may not be the smallest number that have this behavior and I am not sure if this is a bug or something expected; the performance is severely affected, so I decided to report.
## Information

Model I am using (Bert, XLNet ...): bert-base-uncased
Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Set MAX=40000, Run my provided code
2. Set MAX=50000, Run my provided code
3.

```
export GLUE_DIR=downstream_datasets/glue
export TASK_NAME=QQP
SAVING_PERIOD=1000
MAX=40000
BS=32
SEED=42
#EPOCH=1
export CUDA_VISIBLE_DEVICES=0
for TASK_NAME in MNLI # QQP SST-2
do
  echo $TASK_NAME
  python run_glue.py \
    --model_type bert \
    --model_name_or_path bert-base-uncased \
    --task_name $TASK_NAME \
    --do_train \
    --save_steps $SAVING_PERIOD \
    --logging_steps $SAVING_PERIOD \
    --max_steps $MAX \
    --do_eval \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_gpu_train_batch_size $BS \
    --per_gpu_eval_batch_size $BS \
    --learning_rate 2e-5 \
    --seed $SEED \
    --output_dir model_output/"$TASK_NAME"_maxlen128_seed"$SEED"_savesteps"$SAVING_PERIOD"_maxsteps"$MAX"/ \
    --logging_dir runs/"$TASK_NAME"_maxlen128_seed"$SEED"_savesteps"$SAVING_PERIOD"_maxsteps"$MAX"/ \
    --logging_first_step \
    --evaluate_during_training
done
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
From my limited experiments, in the tensorboard dashboard:
MAX=40000: the acc will keep climbing.
MAX=50000: the acc will fluctuate throughout training. It can not exceed 40%.

I suspect MNLI's performance is affecteed by learning rate set by optimizer and scheduler, which are affected by the max_steps. However, this behaviour **does not** hold on other similar-sized datasets like QQP and SST-2. This inconsistency is confusing.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform: Linux-5.3.0-46-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: single gpu


Implement SuperGLUE tasks and baselines

WIP implementation of the SuperGLUE tasks, following the style of the GLUE data processors. This PR includes:
- classification with marked spans: most tasks are classification and straightforward, but WiC and WSC have special spans. This PR implements `*SpanClassification` models (currently for just BERT and RoBERTa) and relevant data classes
- scripts for using and extracting results from the [experiment impact tracker](https://github.com/Breakend/experiment-impact-tracker) library, which tracks energy usage and is being used for the SustaiNLP 2020 competition

TODO
- clean up some of my development scripts and cruft


Added README huseinzol05/t5-base-bahasa-cased

model card

Related to pretrained model cards


Error on instantiating pipeline.summarizer

# 🐛 Bug

## Information

The problem arises when using:
I'm just testing out the different modules in the package. and following the examples on the [usage page](https://huggingface.co/transformers/usage.html)

## To reproduce

Run the following snippet with the environment set up as. described in the relevant section below
`from transformers import pipeline`
`summarizer = pipeline("summarization")`

<!-->

This is the traceback:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
 in
----> 1 summarizer = pipeline("summarization")

~/Projects/amlg/playground/venv/lib/python3.7/site-packages/transformers/pipelines.py in pipeline(task, model, config, tokenizer, framework, **kwargs)
   1759         model = model_class.from_pretrained(model, config=config, **model_kwargs)
   1760
-> 1761     return task_class(model=model, tokenizer=tokenizer, modelcard=modelcard, framework=framework, task=task, **kwargs)

~/Projects/amlg/playground/venv/lib/python3.7/site-packages/transformers/pipelines.py in __init__(self, model, tokenizer, modelcard, framework, task, args_parser, device, binary_output)
    392
    393         # Update config with task specific parameters
--> 394         task_specific_params = self.model.config.task_specific_params
    395         if task_specific_params is not None and task in task_specific_params:
    396             self.model.config.update(task_specific_params.get(task))

AttributeError: 'NoneType' object has no attribute 'config'
```

## Environment info
- `transformers` version: 2.9.1
- Platform: Darwin-19.3.0-x86_64-i386-64bit
- Python version: 3.7.3
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: Nope
- Using distributed or parallel set-up in script?: Nope


[docs] Restore examples.md symlink


Speed on various cards

This is not a bug, but I wonder if someone know what is the speed difference between different cards.

I now run bert base on 1080.

How much speedup will I get using 1080 ti  and 2080. Which one is faster for Transformers?

I know 2080 ti would be the best but probably the best one I can get would be 1080 ti or 2080.


Specify tensorboard logging in BART finetuning

Allows you to specify the directory in which to store tensorboard logs when running the BART finetuning script.

Issue: #4349


Allow for None gradients in GradientAccumulator.

The [TFTrainer commit](https://github.com/huggingface/transformers/commit/aad50151f35b934039c455242c433a24f011cc93#diff-2a765415189cf1edda4414e1758b0e79) seems to have reverted the ability for gradients to be None in GradientAccumulator. This behavior is necessary when pretraining a model, such as AlbertForPreTraining, and using only one loss (only SOP or only MLM).

The [OpenNMT repository](https://github.com/OpenNMT/OpenNMT-tf/blob/master/opennmt/optimizers/utils.py) the code was taken from lacks this ability, but it's important. This commit restores the ability for gradients to be None.


Save pretrained MarianTokenizer

Hello!
I can't save pretrained MarianTokenizer, it's raising NotImplementedError as a subclass of PretrainedTokenizer.


run_squad with early stopping on a validation set

Hello.
Is there a way to use run_squad with early stopping as a validation set?

I have 3 files: train-v1.1.json, dev-v1.1.json, and test-v1.1.json.
I want to train on the train file, stop the training when the loss on the dev file starts to increase, and then do the final prediction and answers output on the test set.

At Keras it's pretty straight forward:
`history = model.fit(X_train, y_train, validation_data=(X_dev, y_dev), epochs=4000, verbose=0, callbacks=[EarlyStopping(monitor='dev_loss', mode='min'])`

How can I have a similar functionallity in huggingface's run_squad? I'm asking whether is there a built in way before implementing it from scratch.

I thought of changing the code in the point of:
`--evaluate_during_training` - it activates the evaluation during training - I can change the fie to the dev-v1.1.json file in this point, and stop the training if the validation loss increases.

I would like to know if there is a better way.

Thanks


demo website: i info icon should link to resource about parameters

The website/demo at https://transformer.huggingface.co does have a "Model & decoder settings" box at the bottom left.
Next to it, at the right, there is an "(i)" sign.

STR: Click on it.
**What happens:** It links to this repo.
**What should happen:** It should link to somewhere, where the parameters are actually explained.

Reason:
The generic link to this repo is way to – well – generic. I've searched the Readme for the parameter names and could not find anything useful/an explanation of what they actually do.
The reason I click on that icon is to know what I can adjust there/what happens if I change an option, I don't just want to see the general project.

There should be a page that just explains, what each option means (in simple terms):
* Model size: what model to use (from small to big), the bigger the more useful/accurate(?) it may be
* Top p:
* Temperature:
* Max time: how long the model should – at most – compute to search for results


past functionality broken in release 2.9.0 and 2.9.1

# 🐛 Bug

## Information

Model: gpt2

## To reproduce

Steps to reproduce the behavior:

```
from transformers.tokenization_gpt2 import GPT2Tokenizer
from transformers.modeling_gpt2 import GPT2LMHeadModel
import torch

# Remember to run transformers with latest master (not release 2.5.1)
tokenizer = GPT2Tokenizer.from_pretrained('gpt2', pad_token='<|endoftext|>')
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Complete phrases are: "I like to drink soda without sugar" and "Go watch TV alone, I am not going"
doc = "I like to"
# note: comment the above line and uncomment the following line to make it work with 1 document
docs_tensors = tokenizer.batch_encode_plus([doc], pad_to_max_length=True, return_tensors='pt')

docs_next = [" soda and ", " with this"]
# note: comment the above line and uncomment the following line to make it work with 1 document
docs_next_tensors = tokenizer.batch_encode_plus(
    [d for d in docs_next], pad_to_max_length=True, return_tensors='pt')

# predicting the first part of each phrase
_, past = model(docs_tensors['input_ids'], attention_mask=docs_tensors['attention_mask'])

# manipulating the past
past_expanded = [torch.repeat_interleave(layer, torch.LongTensor([2]), dim=1) for layer in past]
past_attention_mask = torch.ones(docs_next_tensors['attention_mask'].shape[0], len(docs_tensors['input_ids'][0]), dtype=torch.int64)
attn_mask = torch.cat([past_attention_mask, docs_next_tensors['attention_mask']], dim=-1)

# predicting the rest of the phrase with past
logits, _ = model(docs_next_tensors['input_ids'], attention_mask=attn_mask, past=past_expanded)
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Predictions for:", [doc + n for n in docs_next])
print("Results with past:", words)

#####################
docs_full_tensors = tokenizer.batch_encode_plus(
    [doc + n for n in docs_next], pad_to_max_length=True, return_tensors='pt')
logits, _ = model(docs_full_tensors['input_ids'], attention_mask=docs_full_tensors['attention_mask'])
logits = logits[:, -1]
_, top_indices_results = logits.topk(50)

words = [tokenizer.decode([idx.item()]) for tir in top_indices_results for idx in tir]

print("Predictions for:", [doc + n for n in docs_next])
print("Results without past:", words)

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
I am expecting the past functionality to work. But I'm getting the following error:
```
RuntimeError: The size of tensor a (4) must match the size of tensor b (5) at non-singleton dimension 3
```
This was working correctly up to release 2.8.0

## Environment info

- `transformers` version:
- Platform: Macos
- Python version: 1.4.0 / 1.5.0
- PyTorch version (GPU?): 2.9.0 cpu / 2.9.1 cpu
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no


Fix: unpin flake8 and fix cs errors


[pipelines] Failing @slow test for TF Summarization

Command:
```bash
RUN_SLOW=1 pytest tests/test_pipelines.py --durations 0 -s -k tf_defaults
```

Issue caused by trying to use Bart in TF.


Has anyone successfully used TF2.0 to load pre-trained transformer-XL (wt103) weights and reproduce their sota results

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
I tried to reproduce, but the final result was 35.6 (vs paper 18.3), I carefully checked and found that the matrix multiplication result of TF2.x and TF1.12 (paper use) have some differences, about 0.0001, and then Many layers of linear transformation and multiplication, the difference in output characterization of the last layer has reached about 0.1, which is enough to affect the results, and if the mem_len and tgt_len values in the experiment are swapped, ppl actually drops to 19.5, I can ensure that all loaded The parameters are correct, the code is completely referenced to the author's source code and changed to 2.0 version, so I want to ask if anyone has reproduced it with TF2.0
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Can not reproduce article numbers

# 🐛 Bug
https://github.com/huggingface/transformers/tree/master/examples/summarization/bertabs

## Information
Bertabs Rouge1/2 F1 evaluation numbers that I am getting are much less than in their article. About twice less!

Model I am using (Bert, XLNet ...):
Bertabs

Language I am using the model on (English, Chinese ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
Summarization

## To reproduce

Steps to reproduce the behavior:

1.
2.
3.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Fix trainer evaluation

Fix eval loss calculation in Trainer for models with `lm_labels` parameter, related to issue #4361

Fix trainer crash at eval time on TPU, related to issue #4362