Has anyone used the run_language_modeling.py to train a gpt2 in a different language? is it possible?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


[examples] Generate argparsers from type hints on dataclasses


❓ In BART, why forcing the first token to BOS ?

# ❓ Questions & Help

In the generation method, the method `prepare_scores_for_generation` is called :
https://github.com/huggingface/transformers/blob/0a9d09b42a9c7c1ccc00da48486a1188078e8594/src/transformers/modeling_utils.py#L1208

And in this method, if it's the first decoding step, BOS token is forced :
https://github.com/huggingface/transformers/blob/0a9d09b42a9c7c1ccc00da48486a1188078e8594/src/transformers/modeling_bart.py#L924-L926

---

I don't understand why it's necessary, because anyway the decoder input ids already contain BOS :
https://github.com/huggingface/transformers/blob/0a9d09b42a9c7c1ccc00da48486a1188078e8594/src/transformers/modeling_utils.py#L866-L868


Any Ideas on how to generate in bulk with CTRL?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Is it possible to generate multiple articles with a list of prompts using CTRL?
Any ideas will be greatly appreciated
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


Created README.md for model card ChemBERTa

model card

Related to pretrained model cards

The README.md is added to explain an overview of SMILES, as this is the only model card trained on a non-language dataset. The documentation also explains potential use-cases for utilizing RoBERTa trained on masked language modelling for SMILES, and links to a repository with the original notebooks for evaluations, running predictions and some applications of the models for curiosity.


Fix mlm

The way the texts is being split up into blocks right now

```
tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text))

for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
```

Results in double [CLS] tokens at the beginning, since special tokens are being added at `tokenizer.convert_tokens_to_ids` And at `tokenizer.build_inputs_with_special_tokens`. Somehow, double [SEP] tokens are not occuring.

The following eliminates the double [CLS] token.

```
tokenized_text = tokenizer.convert_tokens_to_ids(tokenizer.tokenize(text)[1:-2])

for i in range(0, len(tokenized_text) - block_size + 1, block_size):  # Truncate in block of block_size
self.examples.append(tokenizer.build_inputs_with_special_tokens(tokenized_text[i : i + block_size]))
```


Unable to serialize/save TF2.0 RobertaSequenceClassification model to saved model format

Core: Modeling

Internals of the library; Models.

Should Fix

This has been identified as a bug and should be fixed.

TensorFlow

Anything TensorFlow

# 🐛 Bug
I am getting an error while trying to serialize/save TF2.0 RobertaSequenceClassification Keras model to saved model format. I do not see this issue with Bert or Albert model architecture. Please see below for my test script that can be used to reproduce this issue.
## Information

Model I am using (Bert, XLNet ...):  Roberta

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

```python
import tensorflow as tf
from transformers import *
tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
model = TFRobertaForSequenceClassification.from_pretrained('roberta-base')

##########Uncomment the following 2 lines for testing with BERT ############
#tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
#model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')

input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True))[None, :]
outputs = model(input_ids)
logits = outputs[0]
tf_saved_model_path= "/tmp/saved_model/"
tf.keras.models.save_model(model, tf_saved_model_path, overwrite=True, include_optimizer=False, save_format='tf')
```

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)
I need to export/serialize a TF Keras model to TF saved model format

## To reproduce

Steps to reproduce the behavior:

1. Run the script pasted above to reproduce the issue with Roberta
2. Uncomment the 2 lines as mentioned in the script for using Bert (no error seen with Bert)

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

***Stack Trace for Roberta***
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-87e63ee0b3ac> in <module>
     15
     16 tf_saved_model_path= "/tmp/saved_model/"
---> 17 tf.keras.models.save_model(model, tf_saved_model_path, overwrite=True, include_optimizer=False, save_format='tf')

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options)
    136   else:
    137     saved_model_save.save(model, filepath, overwrite, include_optimizer,
--> 138                           signatures, options)
    139
    140

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/save.py in save(model, filepath, overwrite, include_optimizer, signatures, options)
     76     # we use the default replica context here.
     77     with distribution_strategy_context._get_default_replica_context():  # pylint: disable=protected-access
---> 78       save_lib.save(model, filepath, signatures, options)
     79
     80   if not include_optimizer:

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py in save(obj, export_dir, signatures, options)
    949
    950   _, exported_graph, object_saver, asset_info = _build_meta_graph(
--> 951       obj, export_dir, signatures, options, meta_graph_def)
    952   saved_model.saved_model_schema_version = constants.SAVED_MODEL_SCHEMA_VERSION
    953

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py in _build_meta_graph(obj, export_dir, signatures, options, meta_graph_def)
   1035
   1036   object_graph_proto = _serialize_object_graph(saveable_view,
-> 1037                                                asset_info.asset_index)
   1038   meta_graph_def.object_graph_def.CopyFrom(object_graph_proto)
   1039

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py in _serialize_object_graph(saveable_view, asset_file_def_index)
    695   for obj, obj_proto in zip(saveable_view.nodes, proto.nodes):
    696     _write_object_proto(obj, obj_proto, asset_file_def_index,
--> 697                         saveable_view.function_name_map)
    698   return proto
    699

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/saved_model/save.py in _write_object_proto(obj, proto, asset_file_def_index, function_name_map)
    735           version=versions_pb2.VersionDef(
    736               producer=1, min_consumer=1, bad_consumers=[]),
--> 737           metadata=obj._tracking_metadata)
    738       # pylint:enable=protected-access
    739     proto.user_object.CopyFrom(registered_type_proto)

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py in _tracking_metadata(self)
   2727   @property
   2728   def _tracking_metadata(self):
-> 2729     return self._trackable_saved_model_saver.tracking_metadata
   2730
   2731   def _list_extra_dependencies_for_serialization(self, serialization_cache):

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/base_serialization.py in tracking_metadata(self)
     52     # TODO(kathywu): check that serialized JSON can be loaded (e.g., if an
     53     # object is in the python property)
---> 54     return json_utils.Encoder().encode(self.python_properties)
     55
     56   def list_extra_dependencies_for_serialization(self, serialization_cache):

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/json_utils.py in encode(self, obj)
     42
     43   def encode(self, obj):
---> 44     return super(Encoder, self).encode(_encode_tuple(obj))
     45
     46

/usr/local/opt/pyenv/versions/3.6.7/lib/python3.6/json/encoder.py in encode(self, o)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

/usr/local/opt/pyenv/versions/3.6.7/lib/python3.6/json/encoder.py in iterencode(self, o, _one_shot)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
    258
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/json_utils.py in default(self, obj)
     39       items = obj.as_list() if obj.rank is not None else None
     40       return {'class_name': 'TensorShape', 'items': items}
---> 41     return serialization.get_json_type(obj)
     42
     43   def encode(self, obj):

~/huggingface/transformers/env/lib/python3.6/site-packages/tensorflow/python/util/serialization.py in get_json_type(obj)
     74     return obj.__wrapped__
     75
---> 76   raise TypeError('Not JSON Serializable:', obj)

TypeError: ('Not JSON Serializable:', RobertaConfig {
  "_num_labels": 2,
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bad_words_ids": null,
  "bos_token_id": 0,
  "decoder_start_token_id": null,
  "do_sample": false,
  "early_stopping": false,
  "eos_token_id": 2,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "id2label": {
    "0": "LABEL_0",
    "1": "LABEL_1"
  },
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "is_decoder": false,
  "is_encoder_decoder": false,
  "label2id": {
    "LABEL_0": 0,
    "LABEL_1": 1
  },
  "layer_norm_eps": 1e-05,
  "length_penalty": 1.0,
  "max_length": 20,
  "max_position_embeddings": 514,
  "min_length": 0,
  "model_type": "roberta",
  "no_repeat_ngram_size": 0,
  "num_attention_heads": 12,
  "num_beams": 1,
  "num_hidden_layers": 12,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 1,
  "prefix": null,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "task_specific_params": null,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "type_vocab_size": 1,
  "use_bfloat16": false,
  "vocab_size": 50265
}
)

## Expected behavior
There should be no error when saving/serializing the TF Keras Model for Roberta. I do not see any error with Bert or Albert.
<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.7.0
- Platform: Darwin-19.2.0-x86_64-i386-64bit
- Python version: 3.6.7
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): 2.2.0-rc1 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

I also see the same issue with TF 2.1.0.


Speedup torch summarization tests

Speedup torch summarization tests  by using small models that are faster to download and instantiate.


Create model card  for NLP4H/ms_bert

model card

Related to pretrained model cards


fixed TransfoXLLMHeadModel documentation


# ❓ Questions & Help

i was using this notebook : https://www.kaggle.com/theoviel/bert-pytorch-huggingface-with-tpu-multiprocessing

to finetune huggingface’s xlm roberta base model on jigsaw multilingual (ongoing kaggle competition)

this is my first time with torch xla and TPU multiprocessing…!

the code i am trying is exactly this one : https://pastebin.com/fS94MKYc   on a kaggle kernel which gives TPU v3-8

but even for batch_size = 8 i see my jupyter notebook crashes after giving this error message : **Your notebook tried to allocate more memory than is available. It has restarted.**

where i can see other people are using same model with even batch_size = 64

full error message looks like this :

```
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
<timed exec> in <module>

/opt/conda/lib/python3.6/site-packages/torch_xla/distributed/xla_multiprocessing.py in spawn(fn, args, nprocs, join, daemon, start_method)
    180         join=join,
    181         daemon=daemon,
--> 182         start_method=start_method)

/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in start_processes(fn, args, nprocs, join, daemon, start_method)
    156
    157     # Loop on join until it returns True or raises an exception.
--> 158     while not context.join():
    159         pass
    160

/opt/conda/lib/python3.6/site-packages/torch/multiprocessing/spawn.py in join(self, timeout)
    106                 raise Exception(
    107                     "process %d terminated with signal %s" %
--> 108                     (error_index, name)
    109                 )
    110             else:

Exception: process 0 terminated with signal SIGKILL
```

same problem is occuring also when i try bert base multilingual of huggingface.
so i am not understanding exactly where in my code i need to make change so that it can work? it seems like the problem is not with the batch size but something else that i am unable to catch.please help,thanks in advance


Add model card

Related to pretrained model cards


Weird summarization results - the summary is longer than the input

# 🐛 Bug

## Information
Summarization task is returning an unexpected results. For an input of

> "We have a telephony partner who is very interested in this program and may be able to help identify pilot customers."

The results is

> [{'summary_text': '"We have a telephony partner who is very interested in this program and may be able to help identify pilot customers," the company says. "We are looking at a number of different ways to get people talking to each other," it adds. "It\'s a very exciting time for us," says the company\'s chief operating officer.'}]

Model I am using (Bert, XLNet ...): Summarization pipeline

Language I am using the model on (English, Chinese ...): Eng

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [V ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [V ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Execute below script

```python
!pip install -q transformers --upgrade
from transformers import pipeline
summarizer = pipeline(task="summarization")

data = "We have a telephony partner who is very interested in this program and may be able to help identify pilot customers."
print(summarizer(data))
```

## Expected behavior

Would expect the summary to 1) not add contextual information that doesn't exist, and 2) to not be longer than the input.
Arguably the input is short but still...

## Environment info
Colab


Create model card

Related to pretrained model cards


Create README.md for ktrapeznikov/biobert_v1.1_pubmed_squad_v2

Related to pretrained model cards


❓Adding new tokens to pre-trained tokenizer

## Details
Hi, I am working with DistilBERT multilingual model for sequence classification tasks where I need to add some additional languages apart from mentioned [here](https://github.com/google-research/bert/blob/master/multilingual.md#list-of-languages). And for that, I am struggling to find the correct way to update tokenizer. From the documentation, I inferred that first, i have to get all new tokens in a list, call `tokenizer.add_tokens()` and then again i have to pass those new sentences to tokenizer and get them tokenized. So the real question: is there any method which i use to update tokenizer and tokenize sentence at the same time (when tokenizer sees unknown token it adds the token to the dictionary). Thanks in advance.


How can I judge whether is in the dictionary?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Will different word get the same ids in tokenizer?
Cause I just meet this situation.
And it looks like this
candidate is ['charge', 'greet', 'treat', 'reward']
 and candidate_ids is [10813, 1, 13581, 1]
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Add model card for BERTeus

Related to pretrained model cards

This PR includes the model card for the BERTeus model which has been recently uploaded to the huggingface repository.


Chatbot QnA feature for given text corpus

Is there a way we can have a chatbot specifically which can answer questions on a given text corpus?
Is there any transformer model which we can train for this and how?


Bertabs metrics lower than paper

I tested abstractive summarization pre-trained model using the source under transformers/examples/summarization/bertabs/...

My dataset are CNN & Daily mail, which are 30 thousands of docs.

However, the result of rouge score is as follows

****** ROUGE SCORES ******

** ROUGE 1
F1        >> 0.275
Precision >> 0.299
Recall    >> 0.260

** ROUGE 2
F1        >> 0.161
Precision >> 0.184
Recall    >> 0.149

** ROUGE L
F1        >> 0.305
Precision >> 0.326
Recall    >> 0.290

why is the result different from that of the article, Text Summarization with Pretrained Encoders ?


Allow token regression in ForTokenClassification models

# 🚀 Feature request

Current `ForTokenClassification` class implementation for all models does not support regression. I propose to adapt the current implementation for all the `ForTokenClassification` models in order to enable out-of-the-box token regression when `num_labels == 1`, similarly to what is currently available in the `ForSentenceClassification` models.

Concretely, this would mean converting this (taken from `AlbertForTokenRegression` as an example, line 873 in `modeling_albert.py`):

```python
if labels is not None:
    loss_fct = CrossEntropyLoss()
    # Only keep active parts of the loss
    if attention_mask is not None:
        active_loss = attention_mask.view(-1) == 1
        active_logits = logits.view(-1, self.num_labels)[active_loss]
        active_labels = labels.view(-1)[active_loss]
        loss = loss_fct(active_logits, active_labels)
    else:
        loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
    outputs = (loss,) + outputs
return outputs  # (loss), logits, (hidden_states), (attentions)
```

into something like this:

```python
if labels is not None:
    if self.num_labels == 1:
        # We are doing regression
        loss_fct = MSELoss()
        logits_view = logits.view(-1)
    else:
        # We are doing classification
        loss_fct = CrossEntropyLoss()
        logits_view = logits.view(-1, self.num_labels)
    # Only keep active parts of the loss
    if attention_mask is not None:
        active_loss = attention_mask.view(-1) == 1
        active_logits = logits_view[active_loss]
	active_labels = labels.view(-1)[active_loss]
	loss = loss_fct(active_logits, active_labels)
    else:
	loss = loss_fct(logits_view, labels.view(-1))
    outputs = (loss,) + outputs

return outputs  # (loss), logits, (hidden_states), (attentions)
```

## Motivation

I am currently working with token-level regression using multiple transformer models to predict eye-tracking metrics that are commonly considered as proxies for cognitive processing in psycholinguistics (e.g. word reading times, fixation counts, etc.). Given that most of those are continuous metrics, the ability to use `transformers` for token regression would make my work much faster. Moreover, I believe that this functionality can benefit other researchers working with token-level continuous metrics.

## Your contribution

If this feature is regarded as interesting by maintainers, I can submit a PR with the suggested changes applied to all currently supported models.


❓ How to run pipeline (summarization) in FP16 mode ?

# ❓ Questions & Help

I couldn't find on the documentation any parameter that allow running a pipeline in FP16 mode. **Did I miss it or it's not a feature yet ?**


How can I track the performance of my GPT-2 model during finetuning?

Hi,

I am new in using Transformer HugginFace library.

I am using [Google Colab](https://colab.research.google.com/github/interactive-fiction-class/interactive-fiction-class.github.io/blob/master/homeworks/language-model/hw4_transformer.ipynb) to fine tune GPT-2 model. Google Colab only displays the output of last 5000 lines. So, I could not be able to figure out the performance of previous checkpoints, whose output vanishes.

I would like to track the performance of my training model during the whole period of time. Is it possible to track it by using **tensorboard**? or is there any other way exist? I have noticed a variable "logging_steps" in [run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) file, but I donot have an idea how can I use it to track performance of training model in Google Colab?


BioMed Roberta-Base (AllenAI)

Related to pretrained model cards

This PR includes the model card for Biomed-roberta base, which @kyleclo recently uploaded to allenai's huggingface model repository.


Fix roberta checkpoint conversion script

After #2521 and #2958, this script stopped working. We need to set the bias on the new `decoder` Linear directly.

cc @LysandreJik


# 🐛 Bug: Can't evaluate official TensorFlow NER model

## Information

Model I am using (Bert, XLNet ...): I am using bert-base-multilingual-cased

Language I am using the model on (English, Chinese ...): German

The problem arises when using:
* [X] the official example scripts: I was using the official script for the NER model training.
at this link https://github.com/huggingface/transformers/tree/master/examples/ner
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: I was training an NER model
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Follow the steps to download the data and run the model for TensorFlow
2. Training the model for 3 epochs
3. On evaluation, the model will fail
4. Then I tried to explicitly call the evaluation of the model. Using this:
python3 run_tf_ner.py --data_dir ~/data --model_type bert --labels ~/data/labels.txt --model_name_or_path $BERT_MODEL --output_dir $OUTPUT_DIR --max_seq_length  $MAX_LENGTH --num_train_epochs $NUM_EPOCHS --per_device_train_batch_size $BATCH_SIZE --save_steps $SAVE_STEPS --seed $SEED --do_eval

Here is what I see when calling the evaluation step:

```I0405 20:15:11.758301 140712645343040 modeling_tf_utils.py:388] loading weights file germeval-model/tf_model.h5
2020-04-05 20:15:12.024952: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-04-05 20:15:12.031397: I tensorflow/core/platform/profile_utils/cpu_utils.cc:102] CPU Frequency: 2200000000 Hz
2020-04-05 20:15:12.032399: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3ae8970 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-04-05 20:15:12.032438: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
I0405 20:15:15.697083 140712645343040 run_tf_ner.py:418] Loading features from cached file /home/taras/data/cached_dev_bert-base-multilingual-cased_128.tf_record
Traceback (most recent call last):
  File "run_tf_ner.py", line 641, in <module>
    app.run(main)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/usr/local/lib/python3.7/dist-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "run_tf_ner.py", line 576, in main
    args, strategy, model, tokenizer, labels, pad_token_label_id, mode="dev"
  File "run_tf_ner.py", line 314, in evaluate
    eval_iterator = progress_bar(eval_dataset, total=num_eval_steps, parent=master, display=args["n_device"] > 1)
  File "/home/taras/.local/lib/python3.7/site-packages/fastprogress/fastprogress.py", line 226, in __init__
    super().__init__(gen, total, display, leave, parent, master)
  File "/home/taras/.local/lib/python3.7/site-packages/fastprogress/fastprogress.py", line 24, in __init__
    parent.add_child(self)
  File "/home/taras/.local/lib/python3.7/site-packages/fastprogress/fastprogress.py", line 264, in add_child
    self.child.prefix = f'Epoch {self.main_bar.last_v+1}/{self.main_bar.total} :'
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'
```

## Expected behavior

Train an NER model and be able to evaluate and predict using the trained weights.

## Environment info
- `transformers` version: 2.7.0
- Platform: Linux-4.19.0-8-cloud-amd64-x86_64-with-debian-10.3
- Python version: 3.7.3
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.2.0-rc2 (False)
- Using GPU in script?: Yes, Tesla K80
- Using distributed or parallel set-up in script?: No


Wrong Mask LM prediction with BertForMaskedLM

# 📚 Migration

## Information

<!-- Important information -->

Model I am using (Bert, XLNet ...):
Bert, Electra

Language I am using the model on (English, Chinese ...):
English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## Details

### When I transfer to transformers=2.7.0, I find that the LM model failed to predict correct masked tokens.

I test the **transformer** model on the old LM example of **pytorch_pretrained_bert**:
"Who was Jim Henson ? Jim Henson was a puppeteer"

My test code goes like following:

```python
# coding: utf-8
import torch
from transformers import BertTokenizer, BertModel, BertForMaskedLM, AutoModel, AutoTokenizer, AutoModelWithLMHead, ElectraModel, ElectraForMaskedLM

MODEL_PATH = 'Resources/bert-base-uncased/uncased_L-12_H-768_A-12/'

VOCAB = MODEL_PATH

print('== tokenizing ===')
tokenizer = BertTokenizer.from_pretrained(VOCAB)

# Tokenized input
text = "Who was Jim Henson ? Jim Henson was a puppeteer"
tokenized_text = tokenizer.tokenize(text)

masked_index = 6
tokenized_text[masked_index] = '[MASK]'

# Convert token to vocabulary indices
indexed_tokens = tokenizer.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
input_mask = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]

# ======== Convert inputs to PyTorch tensors
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])

# ======== predict tokens ========
print('== LM predicting ===')
# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained(MODEL_PATH)
model.eval()

# Predict all tokens
predictions = model(tokens_tensor, segments_tensors)[0]

# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])
print('predicted_token', predicted_token)

```

## Other Details
(1) Such testing code works fine with the previous version of **pytorch_pretrained_bert**.
But now it seems that model predicts a random token.

(2) Random predicting also happened when I load electra model with ElectraForMaskedLM.

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.7.0
- Platform: linux
- Python version: 3.7
- PyTorch version (GPU?): 1.2-gpu
- Tensorflow version (GPU?): no
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):
pytorch-pretrained-bert

## Checklist

- [x] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [x] I checked if a related official extension example runs on my machine.


Summarization pipeline - Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/modelcard.json'

# 🐛 Bug

## Information
Loading the summarization pipeline will result in below assertion:

> Couldn't reach server at 'https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/modelcard.json' to download model card file.
> Creating an empty model card.

The problem arises when using:
* [V] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
`!pip install -q transformers --upgrade
from transformers import pipeline
summarizer = pipeline(task="summarization")`

1. Execute above code

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Translation pipeline bug after 398 characters

# 🐛 Bug

The translation pipeline with T5 does not seem to allow longer translations than 400~ character. It either automatically stops at 398, or if I play with the min/max_length parameters, it produces gibberish after 400~ characters.

## Information
I am using the translation pipeline (T5)
I tried both:
translator_de = pipeline(task='translation_en_to_de')
translator_fr = pipeline(task='translation_en_to_fr')

I tried the different suggestions in this short twitter discussion, but couldn't get it to work: https://twitter.com/PatrickPlaten/status/1244747294664200193

## To reproduce
Steps to reproduce the behavior:

```Python
translator_de = pipeline(task='translation_en_to_de')

text_en = "The 2019–20 coronavirus pandemic is an ongoing pandemic of coronavirus disease 2019 (COVID-19), caused by severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2).[6] The outbreak was first identified in Wuhan, Hubei, China, in December 2019. The World Health Organization (WHO) declared the outbreak to be a Public Health Emergency of International Concern on 30 January 2020 and recognized it as a pandemic on 11 March.[7][8] As of 30 March 2020, more than 745,000[4] cases of COVID-19 have been reported in over 190 countries and territories, resulting in approximately 35,000[4] deaths. More than 156,500[4] people have since recovered.[5]"

text_trans_de = translator_de(text_en, min_length=len(text_en), early_stopping=False)
text_trans_de[0]['translation_text']
```

Output:
'Zu den Bemühungen, die Ausbreitung des Virus zu verhindern, zählen Reisebeschränkungen, Quarantäne, Sperrzeiten, Arbeitsplatz-Gefahrkontrollen, Verschiebungen und Annullierungen von Veranstaltungen und Anlagenschließungen, darunter die Quarantäne in Hubei, nationale oder regionale Quarantäne in anderen Teilen der Welt, Sperrmaßnahmen in China und Südkorea, verschiedene Grenzschließungen oder Einreise\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad\xad'

## Expected behavior

Ideally, it would allow me to translate text of any length.

## Environment info
- `transformers` version: 2.7.0
- Platform: MacOS Catalania 10.15.3 (19D76)
- Python version: 7.3
- Using GPU in script?: No, CPU
- Using distributed or parallel set-up in script?: No.


[TransfoXL] fix argument order of update_mems fn in TF version

Wrong argument order of function. Thanks @dmytyar !


[Docs, T5] Fix TF T5 examples docstring

Update TF T5 docstring - since forgotten to do so in: #3547


Reinitializing layers in BERT

Hello,

I have a question regarding re-initialising the encoder layers in BERT. What happens if I call the __init__() method of a BERT layer, is the the layer re-initialised using the pre-trained BERT weights or does it get completely new weights ?

`model.bert.encoder.layer[0].__init__(config)`


Custom collate function that pads only to the longest sequence?

Currenty all input is padded to `max_seq_length` but in most cases the longest sequence in a batch is shorter than that, sometimes by a significant amount. If there is a custom collate function that pads only to the longest sequence in a batch, that will probably save quite some memory and time. Is that feasible?


Cased model + `--do_lower_case` in documentation?

The [examples README](https://github.com/huggingface/transformers/tree/master/examples/README.md) has a lot of examples using both a cased model and the `--do_lower_case` option. Is that an error?


[Bart] Replace config.output_past with use_cache kwarg

- Rename generation_mode -> `use_cache`

### Benefits
- Avoid confusion (see linked issues)
- allow unit tests to instantiate once, then test `forward` and `generate`. Avoiding extra 10 second init cost.
- Never accidentally have slow generation

### Costs
- If a developer is changing something and wants to turn caching off, they must edit `prepare_inputs_for_generation` and pass use_cache=False. This is documented. They are a developer by construction so this cost is low.
- inconsistency with other cachers like `CTRLModel`


Fix RoBERTa/XLNet Pad Token in run_multiple_choice.py

`convert_examples_to_fes atures` sets `pad_token=0` by default, which is correct for BERT but incorrect for RoBERTa (`pad_token=1`) and XLNet (`pad_token=5`). I think the other arguments to `convert_examples_to_features` are correct, but it might be helpful if someone checked who is more familiar with this part of the codebase.


How to get top 10 possible set of words to calculate Top-K accuracy and MRR?

# ❓ Questions & Help
Dear All,

I am a newcomer to build language model with GPT-2.

I have [fine-tuned](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) a  language model by using GPT-2.

Now, I would like to calculate Top-k accuracy and Mean Reciprocal Rank (MRR) of my model. For this purpose, I am using following strategy to get top 10 next predicted words:

1. Get a sub-sequence of text of length such as 40, contained in a test.txt, from the start of the file.
2. Next sub-sequence is created by moving a window of 40 words to next step 1. This process will go on untill, we get a list of all sub-sequences of test.txt file.
3. Pass the sub-sequences one by one to the generated model, which should give next 10 possible set of words.

For this purpose, I am using following segment of code to get top 10 words by adapting a code mentioned at this [link](https://github.com/huggingface/transformers/blob/master/examples/run_generation.py):

```
    prompt_text = 'hello world' #Its an example. In reality it will be a complete subsequence of 40 words
    encoded_prompt = tokenizer.encode(prompt_text, add_special_tokens=False, return_tensors="pt")
    encoded_prompt = encoded_prompt.to('cpu')
    output_sequences = model.generate(
        input_ids=encoded_prompt,
        max_length=1+len(encoded_prompt[0]),
        top_p=0.9,
        do_sample=True,
        num_return_sequences=10
    )
```

Is it a right way to get top 10 next words  on the basis of input string, which helps me in calculating Top-k accuracy and MRR of my model? Kindly let me know about your concerns.


Create README.md for ktrapeznikov/scibert_scivocab_uncased_squad_v2

Related to pretrained model cards


Create README.md for ktrapeznikov/albert-xlarge-v2-squad-v2

Related to pretrained model cards

adding readme for
ktrapeznikov/albert-xlarge-v2-squad-v2


Failing to load saved TFBertModel

TF version: 2.2.0-rc1
transformers version: 2.7.0

`import tensorflow as tf`
`import transformers`
`print(tf.__version__)`
`print(transformers.__version__)`
`MAX_LEN = 10`
`model_path = 'saved_model/temp_model'`
`ids = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)`
`mask = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)`
`token_type_ids = tf.keras.layers.Input((MAX_LEN,), dtype=tf.int32)`
`base_model = transformers.TFBertModel.from_pretrained("bert-base-cased"`
                `, output_hidden_states=False)`
`base_output = base_model([ids, mask, token_type_ids])`
`seq_out, _ = base_output[0], base_output[1]`
`base_model.trainable = False`
`model = tf.keras.models.Model(inputs=[ids, mask, token_type_ids], outputs=[seq_out])`
`model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])`
`print(model.summary())`
`model.save(model_path)`
`model = tf.keras.models.load_model(model_path)`

Model load fails with the following error:

Traceback (most recent call last):
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 378, in assert_same_structure
    expand_composites)
TypeError: The two structures don't have the same nested structure.

First structure: type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')}

Second structure: type=list str=[TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/2')]

More specifically: The two namedtuples don't have the same sequence type. First structure type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')} has type dict, while second structure type=list str=[TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/2')] has type list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "temp.py", line 29, in <module>
    model = tf.keras.models.load_model(model_path)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 190, in load_model
    return saved_model_load.load(filepath, compile)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py", line 116, in load
    model = tf_load.load_internal(path, loader_cls=KerasObjectLoader)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 604, in load_internal
    export_dir)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py", line 188, in __init__
    super(KerasObjectLoader, self).__init__(*args, **kwargs)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/saved_model/load.py", line 123, in __init__
    self._load_all()
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py", line 215, in _load_all
    self._finalize_objects()
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py", line 506, in _finalize_objects
    _finalize_saved_model_layers(layers_revived_from_saved_model)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py", line 677, in _finalize_saved_model_layers
    inputs = infer_inputs_from_restored_call_function(call_fn)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/load.py", line 921, in infer_inputs_from_restored_call_function
    spec = nest.map_structure(common_spec, spec, spec2)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 611, in map_structure
    expand_composites=expand_composites)
  File "/Users/sourabhmaity/anaconda3/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 385, in assert_same_structure
    % (str(e), str1, str2))
TypeError: The two structures don't have the same nested structure.

First structure: type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')}

Second structure: type=list str=[TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/2')]

More specifically: The two namedtuples don't have the same sequence type. First structure type=dict str={'input_ids': TensorSpec(shape=(None, 5), dtype=tf.int32, name='input_ids')} has type dict, while second structure type=list str=[TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/0'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/1'), TensorSpec(shape=(None, 10), dtype=tf.int32, name='inputs/2')] has type list
Entire first structure:
{'input_ids': .}
Entire second structure:
[., ., .]


ValueError: You have to specify either input_ids or inputs_embeds!

## Details

I'm quite new to NLP task. However, I was trying to train the T5-large model and set things as follows. But unfortunately, I've got an error.

```python
def build_model(transformer, max_len=512):
    input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
    sequence_output = transformer(input_word_ids)[0]
    cls_token = sequence_output[:, 0, :]
    out = Dense(1, activation='sigmoid')(cls_token)
    model = Model(inputs=input_word_ids, outputs=out)
    return model

model = build_model(transformer_layer, max_len=MAX_LEN)
```
It thorws

```
ValueError: in converted code:
ValueError                                Traceback (most recent call last)
<ipython-input-19-8ad6e68cd3f5> in <module>
----> 5     model = build_model(transformer_layer, max_len=MAX_LEN)
      6
      7 model.summary()

<ipython-input-17-e001ed832ed6> in build_model(transformer, max_len)
     31     """
     32     input_word_ids = Input(shape=(max_len,), dtype=tf.int32, name="input_word_ids")
---> 33     sequence_output = transformer(input_word_ids)[0]
     34     cls_token = sequence_output[:, 0, :]
     35     out = Dense(1, activation='sigmoid')(cls_token)
ValueError: You have to specify either input_ids or inputs_embeds
```


how can i run gpt2 model on tf serving ?


Add code to pretrain T5 model from scratch

# 🚀 Feature request

The T5 model can significantly improve NLP task accuracies. However, the existing pretrained models are all in English. I'd like to pretrain T5 model on different language datasets from scratch.

Can you add code on pretraining T5 model? Thanks.


Create model card

Related to pretrained model cards


default of weight_decay  for run_language_modeling.py

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


fix prepare_for_tokenization in tokenization_roberta.py

fix the corner case break run_glue.py with QQP task mentioned in #3608


Update notebooks

Update the notebooks in the documentation


Feature Request: Fill Mask more than 1 token

At the moment you can use hugging face's mask filling pipeline to predict 1 masked token in a sentence using the below:

```
!pip install -q transformers
from __future__ import print_function
import ipywidgets as widgets
from transformers import pipeline

nlp_fill = pipeline('fill-mask')
nlp_fill("I am going to guess <mask> in this sentence")
```

The request is that you also add the ability to predict N masked tokens rather than only 1 masked token.  So that for example if the sentence is `"I am going to make <mask> <mask> for breakfast"` then the model might predict "fried eggs" for the 2 masked tokens


Update German Bert model card

Related to pretrained model cards

We changed the vocab to work with run_split_on_punc tokenization.
Now there are much less [UNK] punctuation tokens.

For more details see deepset-ai/FARM/issues/60


Choosing between adding frequent out of vocab words and doing further pretraining.

I have a dutch medical dataset (for Namen Entity Recognition) which contains a lot of domain-specific words. The dutch BERT tokenizer therefor outputs a lot of [UNK] tokens when it tokenizes.
Given that I dispose over a corpus of 60k labelled tokens, and right now I have also a relatively small annotated corpus of 185k tokens, would it be best to:

- just add the most frequent out of vocab words to the vocab of the tokenizer
- start from a BERT checkpoint and do further pretraining on the unlabeled dataset (which is now of size 185k which is pretty small I assume..). There might be a possibility for me to obtain a much larger unannotated dataset of potentially millions of (unlabelled) tokens, but I was wondering if even millions of tokens is enough to do some meaningful further pretraining?
Thanks!


Out of memory error while training GPT2-large on 8x32GB Nvidia Volta

# 🐛 Bug

I'm getting an `out-of-memory error` while trianing `gpt2-large` using `batch_size=1`. I'm using the [examples/run_language_modeling.py](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) script. I'm using a custom dataset with varied length examples, maximum `block_size` is 1024.

This is the command I'm using:
```
python -m torch.distributed.launch --nproc_per_node 8 run_language_modeling.py --output_dir=./output_attention_mask_padding/ --model_type=gpt2 --model_name_or_path=gpt2-large --do_train --train_data_file=./data/training.txt --line_by_line --per_gpu_train_batch_size 1 --num_train_epochs 3 --fp16
```

I tried changing `args.gradient_accumulation_steps` but to no success.

Here's the traceback:

```python
                                                                                Traceback (most recent call last):                                                 | 9/213 [00:45<09:51,  2.90s/it]
  File "run_language_modeling.py", line 988, in <module>
    main()
  File "run_language_modeling.py", line 938, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 506, in train
    outputs = model(inputs, masked_lm_labels=labels, attention_mask=attention_mask) if args.mlm else model(inputs, labels=labels, attention_mask=attention_mask)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/distributed.py", line 442, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/deepspeed/.local/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 612, in forward
    loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/wrap.py", line 27, in wrapper
    kwargs)
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/utils.py", line 78, in casted_args
    new_args.append(cast_fn(x))
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/utils.py", line 71, in maybe_float
    return x.float()
RuntimeError: CUDA out of memory. Tried to allocate 190.00 MiB (GPU 2; 31.72 GiB total capacity; 28.71 GiB already allocated; 135.88 MiB free; 1.66 GiB cached)
Traceback (most recent call last):
  File "run_language_modeling.py", line 988, in <module>
    main()
  File "run_language_modeling.py", line 938, in main
    global_step, tr_loss = train(args, train_dataset, model, tokenizer)
  File "run_language_modeling.py", line 523, in train
    scaled_loss.backward()
  File "/usr/local/lib/python3.6/dist-packages/torch/tensor.py", line 118, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: CUDA out of memory. Tried to allocate 194.00 MiB (GPU 4; 31.72 GiB total capacity; 29.42 GiB already allocated; 155.88 MiB free; 951.73 MiB cached)

```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.6.0
- Platform: Linux
- Using distributed or parallel set-up in script?: Yes


Mismatch of loss shape in document with output of TransfoXL

# 🐛 Bug

## Information

Model: Transformer-XL

Language: English

The problem arises when using: a small demo of transformer-xl output

## To reproduce

Steps to reproduce the behavior:
```python3
model_name = 'transfo-xl-wt103'
model = AutoModelWithLMHead.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
sentence = 'Hello world'
tokenize_input = tokenizer.tokenize(sentence)
tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
outputs = model(tensor_input, labels=tensor_input)
print(outputs[0].size())
```
run the code above

## Expected behavior

The  output[0] is supposed to be a  tensor shaped as `(1,)`, as described in the [source code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_transfo_xl.py#L862), however, the actual shape is (1, 2)  (bsz, sequence_len).

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.7.0
- Platform: Ubuntu 18.04
- Python version: 3.6
- PyTorch version (GPU?): 1.4.0, with 1080Ti
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


The tensorflow implementation of T5ForConditionalGeneration runs much slower than the pytorch one. GPU utilization is 30%

# 🐛 Bug
When I am running the official example in examples/summarization/t5/example on PyTorch, I have much better performance than the Tensorflow one. When running on PyTorch it needs 4s per iteration and uses 100% of the GPU. When running the TensorFlow model it needs 30s per iteration and the GPU utilization is  15-20%.
## Information

Model I am using: t5-small

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [X] an official GLUE/SQUaD task: (give the name): CNN Dailymail
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
1. Follow the steps in the official examples/summarization/t5/example
2. Use the modified evaluate_cnn.py script provided below

```
import argparse
from pathlib import Path

from tqdm import tqdm

from rouge_score import rouge_scorer, scoring
from transformers import TFT5ForConditionalGeneration, T5Tokenizer

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i : i + n]

def generate_summaries(lns, output_file_path, model_size, batch_size):
    output_file = Path(output_file_path).open("w")

    model = TFT5ForConditionalGeneration.from_pretrained(model_size)

    tokenizer = T5Tokenizer.from_pretrained(model_size)

    # update config with summarization specific params
    task_specific_params = model.config.task_specific_params
    if task_specific_params is not None:
        model.config.update(task_specific_params.get("summarization", {}))

    for batch in tqdm(list(chunks(lns, batch_size))):
        batch = [model.config.prefix + text for text in batch]
        dct = tokenizer.batch_encode_plus(batch, max_length=512, return_tensors="tf", pad_to_max_length=True)
        input_ids = dct["input_ids"]#.to(device)
        attention_mask = dct["attention_mask"]#.to(device)

        summaries = model.generate(input_ids=input_ids, attention_mask=attention_mask)
        dec = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summaries]
        for hypothesis in dec:
            output_file.write(hypothesis + "\n")
            output_file.flush()

def calculate_rouge(output_lns, reference_lns, score_path):
    score_file = Path(score_path).open("w")
    scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)
    aggregator = scoring.BootstrapAggregator()

    for reference_ln, output_ln in zip(reference_lns, output_lns):
        scores = scorer.score(reference_ln, output_ln)
        aggregator.add_scores(scores)

    result = aggregator.aggregate()
    score_file.write(
        "ROUGE_1: \n{} \n\n ROUGE_2: \n{} \n\n ROUGE_L: \n{} \n\n".format(
            result["rouge1"], result["rouge2"], result["rougeL"]
        )
    )

def run_generate():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "model_size",
        type=str,
        help="T5 model size, either 't5-small', 't5-base', 't5-large', 't5-3b', 't5-11b'. Defaults to 't5-base'.",
        default="t5-base",
    )
    parser.add_argument(
        "input_path", type=str, help="like cnn_dm/test_articles_input.txt",
    )
    parser.add_argument(
        "output_path", type=str, help="where to save summaries",
    )
    parser.add_argument("reference_path", type=str, help="like cnn_dm/test_reference_summaries.txt")
    parser.add_argument(
        "score_path", type=str, help="where to save the rouge score",
    )
    parser.add_argument(
        "--batch_size", type=int, default=8, required=False, help="batch size: how many to summarize at a time",
    )
    parser.add_argument(
        "--no_cuda", default=False, type=bool, help="Whether to force the execution on CPU.",
    )

    args = parser.parse_args()
    # args.device = torch.device("cuda" if torch.cuda.is_available() and not args.no_cuda else "cpu")

    source_lns = [x.rstrip() for x in open(args.input_path).readlines()]

    generate_summaries(source_lns, args.output_path, args.model_size, args.batch_size)

    output_lns = [x.rstrip() for x in open(args.output_path).readlines()]
    reference_lns = [x.rstrip() for x in open(args.reference_path).readlines()]

    calculate_rouge(output_lns, reference_lns, args.score_path)

if __name__ == "__main__":
    run_generate()
```

## Expected behavior

The Tensorflow code   should work with similar  performance as the PyTorch one

## Environment info
absl-py                  0.9.0
astor                    0.7.1
attrs                    19.3.0
blinker                  1.4
boto3                    1.12.34
botocore                 1.15.34
cachetools               3.1.1
certifi                  2019.11.28
cffi                     1.14.0
chardet                  3.0.4
click                    7.1.1
cryptography             2.8
dill                     0.3.1.1
docutils                 0.15.2
filelock                 3.0.12
future                   0.18.2
gast                     0.2.2
google-auth              1.12.0
google-auth-oauthlib     0.4.1
google-pasta             0.2.0
googleapis-common-protos 1.51.0
grpcio                   1.27.2
h5py                     2.10.0
idna                     2.9
jmespath                 0.9.5
joblib                   0.14.1
Keras-Applications       1.0.8
Keras-Preprocessing      1.1.0
Markdown                 3.2.1
nltk                     3.4.5
numpy                    1.18.1
oauthlib                 3.0.1
opt-einsum               3.2.0
pip                      20.0.2
promise                  2.3
protobuf                 3.11.4
pyasn1                   0.4.8
pyasn1-modules           0.2.7
pycparser                2.20
PyJWT                    1.7.1
pyOpenSSL                19.1.0
PySocks                  1.7.1
python-dateutil          2.8.1
regex                    2020.2.20
requests                 2.23.0
requests-oauthlib        1.2.0
rouge-score              0.0.3
rsa                      4.0
s3transfer               0.3.3
sacremoses               0.0.38
scipy                    1.4.1
sentencepiece            0.1.85
setuptools               46.1.3.post20200325
six                      1.14.0
tensorboard              2.1.0
tensorflow               2.1.0
tensorflow-datasets      2.1.0
tensorflow-estimator     2.1.0
tensorflow-gpu           2.1.0
tensorflow-metadata      0.21.1
termcolor                1.1.0
tokenizers               0.5.2
torch                    1.4.0
tqdm                     4.45.0
transformers             2.7.0
urllib3                  1.25.7
Werkzeug                 1.0.1
wheel                    0.34.2
wrapt                    1.12.1

- `transformers` version: 2.70, 2.7.1, and builded from 81484b447b7d8504ff5e1cfff38ec35918383963
- Platform: Ubuntu Ubuntu 18.04.4 LTS
- Python version: 3.7.6
- PyTorch version (GPU?):
- Tensorflow version (GPU?):2.1.0
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:No


Added albert-base-bahasa-cased README and fixed tiny-bert-bahasa-cased README

Related to pretrained model cards


training GPT2 from scratch : implement causal attention mask?

# ❓ Questions & Help

I am trying to train a ```GPT2``` model from scratch but I noticed, by looking into the code here https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py that there doesn’t seem to be an implementation for a causal mask. Maybe it is in another repo and I missed it, I also couldn't find ressources on this in the docs.
I could write an ugly for loop and feed each of my sequences one token at a time to the network which would be super unefficient. I could also chop up each of my examples token by token, pad them and feed it like a batch, which is probably faster but doesn’t feel super satisfacting.

Do you know if there is a standard implementation of casal mask that I missed, or another way to do what I am describing ?

PS : I have already read huggingface’s blogpost on training from scratch, but unfortunately it doesn't say much about the implementation of said training :/.


corrected mistake in polish model cards


How can u make sure that my transformer model should only one GPU, though the serve has multiple GPU cards.

I have transformer BERT model and I am trying to train on lambda server which has 8 GPU cards, How can u make sure that this model should use only once GPU out of 8, by default , it is using all GPUs. even after setting
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


Filling more than 1 masked token at a time

I am able to use hugging face's mask filling pipeline to predict 1 masked token in a sentence using the below:

```
!pip install -q transformers
from __future__ import print_function
import ipywidgets as widgets
from transformers import pipeline

nlp_fill = pipeline('fill-mask')
nlp_fill("I am going to guess <mask> in this sentence")
```

But does anyone have an opinion on what is the best way to do this if I want to predict 2 masked tokens? e.g. if the sentence is instead `"I am going to <mask> <mask> in this sentence"`?

If i try and put this exact sentence into nlp_fill I get the error "ValueError: only one element tensors can be converted to Python scalars" so it doesn't work automatically.

Any help would be much appreciated!

Stack overflow question [link](https://stackoverflow.com/questions/60990897/best-way-of-using-hugging-faces-mask-filling-for-more-than-1-masked-token-at-a)


RobertaTokenizer corner case with empty string

Core: Tokenization

Internals of the library; Tokenization.

https://github.com/huggingface/transformers/blob/81484b447b7d8504ff5e1cfff38ec35918383963/src/transformers/tokenization_roberta.py#L239
this will introduce issue if `text == ""`, which will occur if anyone follows `run_glue.py` with QQP task as the `train.tsv` has two lines contains the empty column.
can be corrected to `if add_prefix_space and (not text or not text[0].isspace()):`


Allow the creation of "entity groups" for NerPipeline #3548

This pull request adds an `index` key to the dictionary returned by `NerPipeline`. The index will be necessary in order to identify **entity groups**, where an entity group is a contiguous series of tokens, having the same **entity type**.

Details of what I want to be able to do can be found in issue #3548.

If this PR gets merged, I would also like to ask if you guys would recommend that I implement the **entity group** transformation in `NerPipeline` itself.

Possibly, I can set the parameter `group` at initialization, where if `True`, the *grouped* version of the output will be returned.

E.g.

Instead of the following *ungrouped* output:

```
[{'entity': 'I-PER', 'score': 0.9983270168304443, 'word': 'En'},
 {'entity': 'I-PER', 'score': 0.9952995777130127, 'word': '##zo'},
 {'entity': 'I-ORG', 'score': 0.9984350204467773, 'word': 'Australian'},
 {'entity': 'I-ORG', 'score': 0.9967807531356812, 'word': 'National'},
 {'entity': 'I-ORG', 'score': 0.9959043264389038, 'word': 'University'},
 {'entity': 'I-ORG', 'score': 0.9900023937225342, 'word': 'AU'},
 {'entity': 'I-ORG', 'score': 0.9763911366462708, 'word': '##N'}]
```

We get something like the following *grouped* output:

```
[{'entity_group': 'I-PER', 'score': 0.9983270168304443, 'word': 'Enzo'},
 {'entity_group': 'I-ORG', 'score': 0.9984350204467773, 'word': 'Australian National University'},
 {'entity_group': 'I-ORG', 'score': 0.9900023937225342, 'word': 'AUN'}]
```


Fix typo in FeatureExtractionPipeline docstring

Fixed a typo in the docstring of `FeatureExtractionPipeline`


🐛 Summarization pipeline : T5-base much slower than BART-large

Core: Pipeline

Internals of the library; Pipeline.

# 🐛 Bug

## Information

Model : `bart-large-cnn` and `t5-base`

Language : English

The problem arises when using : [this colab notebook](https://colab.research.google.com/drive/1iAIFX1QQiFm1F01vMmnAgFh4oH1H-K8W), using both BART and T5 with pipeline for Summarization.

Dataset : CNN/DM

## To reproduce

Run the notebook and measure time for inference between the 2 models. On my run, I have :

```
BART = 73s
T5 = 369s
```

## Expected behavior

I expected T5 to be at least as fast as BART, since there is less parameters (for the base version at least). Instead it takes much longer with T5...

@patrickvonplaten Do you happen to know why T5 is so slow ?


Update README.md

model card

Related to pretrained model cards

Update AutoModel & AutoTokernizer loading.


Update README.md

Related to pretrained model cards

Update AutoModel & AutoTokenizer loading.


Multilingual BART -

This adds `mbart-en-ro` model, a BART variant finetuned on english-romanian translation.
### TODO
- [x] (docs) pretrained_model.rst
- [ ] (docs) README.md
- [ ] (docs) bart.rst

### Differences with Bart
`config.normalize_before`: all the `LayerNorm` calls happen before attention calls
`config.add_final_layer_norm`: There is one extra layer_norm in the decoder
`config.scale_embedding`: embeddings are multiplied by 32 (`sqrt(d_model=1024)`)

### Future PRs
- The model returns the same variables as fairseq, but the tokenizer is not yet at parity with fairseq. This is the next PR in the pipeline.
- the `mbart-large-cc25` (no finetuning) model has a very different state dict. Also WIP.

### Misc
- the link_tester got angry about me not typing out URLs in this PR. Unclear why it didn't happen earlier.
Needs documentation but unclear where to put it.


[Generate, Test] Split generate test function into beam search, no beam search

- Clean the generate testing functions
- Also should fix flaky behaviour of bad_word_tokens test (see #3367 and https://circleci.com/gh/huggingface/transformers/27997?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link)


Why isn't there a SequenceClassificationModel for GPT-2 (and some other models)?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

<!-- Description of your issue -->
Why isn't there a SequenceClassificationModel (like there is for BERT) for GPT-2? I was able to implement this pretty easily by adding a "[CLS]" token to the vocabulary (like in the GPT2DoubleHeadsModel), appending sequences with "[CLS]", and then adding a linear layer that maps from the embedding of "[CLS]" to a vector of logits corresponding to the classes. After training, this model worked comparably to BertSequenceClassificationModel for my use-case. It would be nice to have this model in the transformers library and not have to code it up from scratch.

If this sounds like a good idea, I can make a pull request with a GPT2SequenceClassificationModel added. If not, why is it not a good idea?


Why is there not a SequenceClassification model for GPT-2?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
Why isn't there a SequenceClassificationModel (like there is for BERT) for GPT-2? I was able to implement this pretty easily by adding a "[CLS]" token to the vocabulary (like in the GPT2DoubleHeadsModel), appending sequences with "[CLS]", and then adding a linear layer that maps from the embedding of "[CLS]" to a vector of logits corresponding to the classes. After training, this model worked comparably to BertSequenceClassificationModel for my use-case. It would be nice to have this model in the transformers library and not have to code it up from scratch.

If this sounds like a good idea, I can make a pull request with a GPT2SequenceClassificationModel added. If not, why is it not a good idea?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


After enable fp16, torch.save model has error

# 🐛 Bug

After complete training, the model cannot be saved.
## Information
![image](https://user-images.githubusercontent.com/37766299/78298025-7b5b1400-7563-11ea-8928-aa1dae6fb0eb.png)
![image](https://user-images.githubusercontent.com/37766299/78298046-8746d600-7563-11ea-955e-49b1e6673771.png)


CTRL generates French text when I want English texts

Ex: Generation

Natural Language Generation

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. Make sure to tag your question with the
     right deep learning framework as well as the huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers

     If your question wasn't answered after a period of time on Stack Overflow, you
     can always open a question on GitHub. You should then link to the SO question
     that you posted.
     -->

## Details
<!-- Description of your issue -->
CTRL generates French texts when I want English texts.
I run this command: **python examples/run_generation.py  --model_type ctrl --model_name_or_path ctrl --prompt "Looking well today" --length 500 --temperature 0.8 --repetition 1.2**

What do I need to add or change to generate English texts only?

<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->


batch_encode_plus with pad_to_max_length but no max_length is not padding the output

Core: Tokenization

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using BERT:

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
import torch
import numpy as np
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

seq1 = "This is a short sequence"
seq2 = "This will be a much longer sequence, so the short one requires padding"

input = [seq1, seq2]

# Explicitly specified padding length
max_len = 20
tck_temp = tokenizer.batch_encode_plus(input, max_length=max_len, pad_to_max_length=True)
inp_ids = tck_temp['input_ids']
assert len(inp_ids[0]) == len(inp_ids[1]) == max_len, "Both inputs should have length equal to 20"

# Implicit padding length set to models max length
model_max_len = tokenizer.max_len
tck_temp = tokenizer.batch_encode_plus(input, pad_to_max_length=True)inp_ids = tck_temp['input_ids']
assert len(inp_ids[0]) == len(inp_ids[1]) == model_max_len, "Both inputs should have length equal to %d" % model_max_len
```

## Expected behavior
According to the documentation, `batch_encode_plus` with `pad_to_max_length=True` should pad sequence to models maximal length, if the `max_length` is not explicitly specified.
The attached script should run without raising Exception.

From documentation
"If no max length is specified, the padding is done up to the model’s max length."

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.7.0
- Platform: Linux-4.15.0-74-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.4.0 (False)
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


[Generation] delete print statement

Somehow forgot to delete it from PR #3550.


Wrong tokenization for distilbert-base-multilingual-cased

Internals of the library; Tokenization.

# 🐛 Bug

## Information

Model I am using (DistillBert):

The problem arises when using:
* my own modified scripts: (give details below)

The tasks I am working on is:
*  my own task or dataset: (give details below)

## To reproduce
with transformers 2.3.0:

```python
import torch
from transformers import DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
result = torch.tensor(tokenizer.encode("Hello, my dog is cute"))
print (result)

itos = tokenizer.ids_to_tokens
print (itos[61694])
print (itos[10133])

# The original token for 'Hello' exits but for some reason it's not used?
print (itos[31178])
```

Output:
```bash
[101, 61694, 10133, 117, 15127, 17835, 10124, 21610, 10112, 102]
'hell'
'##o'
'Hello'
```

## Expected behavior
```python
import torch
from transformers import DistilBertTokenizer
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-multilingual-cased")
result = torch.tensor(tokenizer.encode("Hello, my dog is cute"))
print (result)
```

Output:
```bash
[101, 31178, 117, 15127, 17835, 10124, 21610, 10112, 102]
```

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.3.0
- Python version: >3.6


Transformers and BERT: dealing with possessives and apostrophes when encode

Let's consider two sentences:

    "why isn't Alex's text tokenizing? The house on the left is the Smiths' house"

Now let's tokenize and decode:

    from transformers import BertTokenizer
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
    tokenizer.decode(tokenizer.convert_tokens_to_ids(tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")))

We get:

    "why isn't alex's text tokenizing? the house on the left is the smiths'house"

**My question is how dealing with missing space in some possessives like *smiths'house*?**

For me, it seems that the process of tokenization in Transformers is done not right. Let's consider output of

    tokenizer.tokenize("why isn't Alex's text tokenizing? The house on the left is the Smiths' house")

we get:

    ['why', 'isn', "'", 't', 'alex', "'", 's', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "'", 'house']

So in this step, we already have lost important information about the last apostrophe. It would be much better if tokenization was done in the another way:

    ['why', 'isn', "##'", '##t', 'alex', "##'", '##s', 'text', 'token', '##izing', '?', 'the', 'house', 'on', 'the', 'left', 'is', 'the', 'smith', '##s', "##'", 'house']

In this way, tokenization keeps all information about apostrophes, and we will not have problems with possessives.


Issues with using SciBERT for Summarizer

Hi All, I am not sure if I am doing this right? But i want my summarizer to use SciBERT SciVocab instead of the traditional BERT vocab. Appreciate any help and advice! This is what I am currently using:

''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''

from transformers import BertTokenizer, BertModel

model_version = 'scibert_scivocab_uncased'
do_lower_case = True
model = BertModel.from_pretrained(model_version)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)

summarizer = pipeline(task="summarization", model = model, tokenizer = tokenizer)
summary = summarizer(readin_df['Text'][0])

I am facing this error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-39-83180e8b1c13> in <module>
      1 summarizer = pipeline(task="summarization", model = model, tokenizer = tokenizer)
      2
----> 3 summary = summarizer(readin_df['Text'][0])

/opt/conda/lib/python3.7/site-packages/transformers/pipelines.py in __call__(self, return_tensors, return_text, clean_up_tokenization_spaces, *documents, **generate_kwargs)
   1251
   1252             summaries = self.model.generate(
-> 1253                 inputs["input_ids"], attention_mask=inputs["attention_mask"], **generate_kwargs,
   1254             )
   1255

/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_no_grad(*args, **kwargs)
     47         def decorate_no_grad(*args, **kwargs):
     48             with self:
---> 49                 return func(*args, **kwargs)
     50         return decorate_no_grad
     51

/opt/conda/lib/python3.7/site-packages/transformers/modeling_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id)
    788         if self.get_output_embeddings() is None:
    789             raise AttributeError(
--> 790                 "You tried to generate sequences with a model that does not have a LM Head."
    791                 "Please use another model class (e.g. `OpenAIGPTLMHeadModel`, `XLNetLMHeadModel`, `GPT2LMHeadModel`, `CTRLLMHeadModel`, `T5WithLMHeadModel`, `TransfoXLLMHeadModel`, `XLMWithLMHeadModel`, `BartForConditionalGeneration` )"
    792             )

AttributeError: You tried to generate sequences with a model that does not have a LM Head.Please use another model class (e.g. `OpenAIGPTLMHeadModel`, `XLNetLMHeadModel`, `GPT2LMHeadModel`, `CTRLLMHeadModel`, `T5WithLMHeadModel`, `TransfoXLLMHeadModel`, `XLMWithLMHeadModel`, `BartForConditionalGeneration` )


Cannot load model in tranformers

Hello, I tried to upload your model and I get this error:

    tokenizer = AutoTokenizer.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")
    model = AutoModelWithLMHead.from_pretrained("nlpaueb/bert-base-greek-uncased-v1")

    ERROR:pytorch_transformers.modeling_utils:Model name 'nlpaueb/bert-base-greek-uncased-v1'
    was not found in model name list (bert-base-uncased, bert-large-uncased, bert-base-cased,
    bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese,
    bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking,
    bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc).
    We assumed 'nlpaueb/bert-base-greek-uncased-v1'
    was a path or url but couldn't find any file associated to this path or url.

If I download the model and upload it from a folder in the system I get this error:

    tokenizer = AutoTokenizer.from_pretrained("/home/transformers/huggingface/greekaueb")
    model = AutoModelWithLMHead.from_pretrained("/home/transformers/huggingface/greekaueb")

    ValueErrorValueError: : Unrecognized model identifier in /home/hatzimin/transformers
    /huggingface/greek_transfer_learning/greekaueb.
    Should contains one of 'bert', 'openai-gpt', 'gpt2', 'transfo-xl', 'xlnet', 'xlm', 'roberta'


min_length parameter in default pipeline summarization produces output smaller than min_length

Hi, I am using pipeline summarization. My code is below:

```
from transformers import pipeline, AutoTokenizer, AutoModel

summarizer = pipeline("summarization")
abstract_dictionary = {'Introduction':'','Methods':'','Results':'','Discussion':''}

for section in article_dictionary:
    if section == 'Introduction':
        min_length = 100
    elif section == 'Methods':
        min_length = 200
    elif section == 'Results':
        min_length = 250
    elif section == 'Discussion':
        min_length = 100

    summary = summarizer(article_dictionary[section], min_length=min_length)[0]['summary_text']
    abstract_dictionary[section] = abstract_dictionary[section]+' '+summary

for section in abstract_dictionary:
    print(section)
    print(abstract_dictionary[section])
    print(" ")
```

and I get the following summary. You will notice that each section is smaller than the min length specified.

Introduction
 Renal-cell carcinoma is characterized by susceptibility to both immunotherapeutic and antiangiogenic treatment approaches and resistance to cytotoxic chemotherapy. Agents such as sunitinib that target the vascular endothelial growth factor (VEGF) pathway are standard first-line therapy for advanced disease. We conducted the KEYNOTE-426 trial to determine whether pembrolizumab plus axit inib would result in better outcomes than sunit in patients with previously untreated advanced renal- cell carcinoma.

Methods
 Pembrolizumab (Keytruda, Merck Sharp & Dohme) plus axitinib (Inlyta, Pfizer) or sunitinIB (Sutent, Pfizers) was used in an open-label, phase 3 trial. Eligible patients were 18 years of age or older; had newly diagnosed or recurrent stage IV clear-cell renal-cell carcinoma; had received no previous systemic therapy for advanced disease; and had a Karnofsky performance-status score of 70 or more. Patients were excluded if they had symptomatic central nervous system metastases, active autoimmune disease, or poorly controlled hypertension. Data on adverse events were collected regularly,

Results
 A total of 1062 patients at 129 sites in 16 countries were screened for eligibility. Of these, 861 patients at 124 sites underwent randomization from October 24, 2016, to January 24, 2018. A total of 432 patients were assigned to the pembrolizumab–axitinib group, and 429 patients to the sunit inib group. The median duration of any treatment was 10.4 months in both groups. The estimated percentage of patients who were alive at 12 months was 89.9% (95% CI, 86.4 to 92.4) in the pembrozumab group and 78.3% (75.8 to 82.,

Discussion
 Treatment with pembrolizumab plus axitinib resulted in a 47% lower risk of death. The objective response rate was 23.6 percentage points higher in the pembrozumab–axit inib group than in the sunitin ib group. The benefit of pembrology plus ax itinib was observed across all subgroups tested. No deaths related to hepatic adverse events were reported in this trial. However, the overall frequency of toxic effects was similar in the two groups.


Evaluation - Output False Positive and False Negative Sentences

# ❓ Questions & Help
Hi, regarding the sequential classification task, after the evaluation on test data, how could I output the actual false positive and false negative sentences? Basically convert BERT embedding back to the actual sentences for error analysis?


added model_cards for polish squad models

model card

Related to pretrained model cards


How to fine tune T5 like for translation tasks?

# ❓ Questions & Help

## Details
<!-- Description of your issue -->
I want to pass 100000's of training instances to this syntax, but it says its limit is only 512. I am passing list of strings.
input_ids = tokenizer.encode('translate English to German: The house is wonderful. </s>', return_tensors='pt')
lm_labels = tokenizer.encode('Das Haus ist wunderbar. </s>', return_tensors='pt')
model(input_ids=input_ids, lm_labels=lm_labels)

And for 512 instances, it is not training, it is finishing in seconds without loading weights and training.

Could you explain how to fine-tune it in the correct way and use the fine-tuned model for the generation?
<!-- You should first ask your question on SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


when I run transformers in Docker container, it appeared this error

```
unknown exception: Model name 'bert-base-uncased' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1, bert-base-dutch-cased). We assumed 'bert-base-uncased' was a path, a model identifier, or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.
```
OS platform python 3.7.6 transformer 2.4 debian


Reason behind the layers taken for distilbert-multilingual

Hi,
I went across the training code for distilbert. I can see that the distillation process is used to train the distilbert model from bert model.

What is the reason only the **layers [0, 2, 4, 7, 9, 11]** have been taken to train distilbert model?Is there any idea about the layers behind this?

And,what is the **last two layers in distilbert layer** corresponds and are they equivalently similar to the **9 and 11** layer of the original bert model?

This was the training code i referred to:
https://github.com/huggingface/transformers/blob/master/examples/distillation/scripts/extract_distilbert.py


cased -> uncased in BERT GLUE example

similar to https://github.com/huggingface/transformers/issues/3183, the GLUE readme also have this issue, the MRPC example use `bert-base-cased` while at the same time `--do_lower_case`.


Dict in the first positional arguments

TensorFlow

Anything TensorFlow

Usage

General questions about the library

Could someone help me figure out what is wrong in the TFBertModel below?

```python
features = next(iter(dataset))
features
```
which prints:

```python
{'attention_mask': <tf.Tensor: shape=(16,), dtype=int32, numpy=array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)>,
 'input_ids': <tf.Tensor: shape=(16,), dtype=int32, numpy=
 array([  101, 13366,  2131,  1035,  6819,  2094,  1035,  2013,  1035,
        24471,  2140,  1006, 24471,  2140,  1007,   102], dtype=int32)>}
```
In turn, I loaded the `TFBertModel` and following the[ documentation page](https://huggingface.co/transformers/model_doc/bert.html#transformers.TFBertForPreTraining) I tried to use `features` as input:

```python
model = TFBertModel.from_pretrained('bert-base-uncased')
model(features)
```
But I'm getting the following error:

```python
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-19-8bcadb504daf> in <module>()
----> 1 model(features)

8 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    966           with base_layer_utils.autocast_context_manager(
    967               self._compute_dtype):
--> 968             outputs = self.call(cast_inputs, *args, **kwargs)
    969           self._handle_activity_regularization(inputs, outputs)
    970           self._set_mask_metadata(inputs, outputs, input_masks)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_bert.py in call(self, inputs, **kwargs)
    706         last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
    707         """
--> 708         outputs = self.bert(inputs, **kwargs)
    709         return outputs
    710

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    966           with base_layer_utils.autocast_context_manager(
    967               self._compute_dtype):
--> 968             outputs = self.call(cast_inputs, *args, **kwargs)
    969           self._handle_activity_regularization(inputs, outputs)
    970           self._set_mask_metadata(inputs, outputs, input_masks)

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_bert.py in call(self, inputs, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, training)
    545         # this attention mask is more simple than the triangular masking of causal attention
    546         # used in OpenAI GPT, we just need to prepare the broadcast dimension here.
--> 547         extended_attention_mask = attention_mask[:, tf.newaxis, tf.newaxis, :]
    548
    549         # Since attention_mask is 1.0 for positions we want to attend and 0.0 for

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py in _slice_helper(tensor, slice_spec, var)
    982         ellipsis_mask=ellipsis_mask,
    983         var=var,
--> 984         name=name)
    985
    986

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py in strided_slice(input_, begin, end, strides, begin_mask, end_mask, ellipsis_mask, new_axis_mask, shrink_axis_mask, var, name)
   1148       ellipsis_mask=ellipsis_mask,
   1149       new_axis_mask=new_axis_mask,
-> 1150       shrink_axis_mask=shrink_axis_mask)
   1151
   1152   parent_name = name

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py in strided_slice(input, begin, end, strides, begin_mask, end_mask, ellipsis_mask, new_axis_mask, shrink_axis_mask, name)
  10155         pass  # Add nodes to the TensorFlow graph.
  10156     except _core._NotOkStatusException as e:
> 10157       _ops.raise_from_not_ok_status(e, name)
  10158   # Add nodes to the TensorFlow graph.
  10159   if begin_mask is None:

/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py in raise_from_not_ok_status(e, name)
   6651   message = e.message + (" name: " + name if name is not None else "")
   6652   # pylint: disable=protected-access
-> 6653   six.raise_from(core._status_to_exception(e.code, message), None)
   6654   # pylint: enable=protected-access
   6655

/usr/local/lib/python3.6/dist-packages/six.py in raise_from(value, from_value)

InvalidArgumentError: Index out of range using input dim 1; input has only 1 dims [Op:StridedSlice] name: tf_bert_model/bert/strided_slice/
```
note: edited to correct typo.


Does the BART model support Chinese? Having the pre-trained Chinese model?

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->