tf BERT model produced by convert_graph_to_onnx has unclear or wrong input shapes

I tried to create a minimal notebook to reproduce [this example nb](https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb), but for TF.

when I run

```python
convert(framework="tf", model="bert-base-cased", output="onnx-test-tf/bert-base-cased.onnx", opset=11)
```
the output is
```
ONNX opset version set to: 11
Loading pipeline (model: bert-base-cased, tokenizer: bert-base-cased)
HBox(children=(FloatProgress(value=0.0, description='Downloading', max=433.0, style=ProgressStyle(description_…

HBox(children=(FloatProgress(value=0.0, description='Downloading', max=230.0, style=ProgressStyle(description_…

HBox(children=(FloatProgress(value=0.0, description='Downloading', max=526681800.0, style=ProgressStyle(descri…

Some weights of the model checkpoint at bert-base-cased were not used when initializing TFBertModel: ['mlm___cls', 'nsp___cls']
- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
All the weights of TFBertModel were initialized from the model checkpoint at bert-base-cased.
If your task is similar to the task the model of the ckeckpoint was trained on, you can already use TFBertModel for predictions without further training.
Creating folder onnx-test-tf
/!\ Please note TensorFlow doesn't support exporting model > 2Gb /!\
Using framework TensorFlow: 2.1.0, keras2onnx: 1.7.0
Found input input_ids with shape: {0: 'batch', 1: 'sequence'}
Found input token_type_ids with shape: {0: 'batch', 1: 'sequence'}
Found input attention_mask with shape: {0: 'batch', 1: 'sequence'}
Found output output_0 with shape: {0: 'batch', 1: 'sequence'}
Found output output_1 with shape: {0: 'batch'}
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertModel.call of <transformers.modeling_tf_bert.TFBertModel object at 0x7f3ddec46810>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertModel.call of <transformers.modeling_tf_bert.TFBertModel object at 0x7f3ddec46810>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertMainLayer.call of <transformers.modeling_tf_bert.TFBertMainLayer object at 0x7f3f0427cb50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module 'gast' has no attribute 'Num'
WARNING: AutoGraph could not transform <bound method TFBertMainLayer.call of <transformers.modeling_tf_bert.TFBertMainLayer object at 0x7f3f0427cb50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: module 'gast' has no attribute 'Num'
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3dddcc8390>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3dddcc8390>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3dddcc8a50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3dddcc8a50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3dddcd08d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3dddcd08d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd46f890>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd46f890>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd477510>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd477510>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd477a90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd477a90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd48c710>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd48c710>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd498390>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd498390>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd498910>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd498910>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddceea590>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddceea590>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddcef3210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddcef3210>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddcef3790>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddcef3790>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddcf08410>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddcf08410>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddcf08f90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddcf08f90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddcf12610>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddcf12610>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd42d2d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd42d2d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd42df10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd42df10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd4364d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd4364d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd449150>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd449150>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd449d90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd449d90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd452350>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd452350>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd45cf90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd45cf90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd3e6c10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd3e6c10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd3f21d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd3f21d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd3fde10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd3fde10>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd405a90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd405a90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd40e050>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd40e050>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd419c50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd419c50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd3a48d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd3a48d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd3a4e50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd3a4e50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd3b5ad0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd3b5ad0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd3be750>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd3be750>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd3becd0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd3becd0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd3cf950>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertSelfOutput.call of <transformers.modeling_tf_bert.TFBertSelfOutput object at 0x7f3ddd3cf950>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd35e5d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertIntermediate.call of <transformers.modeling_tf_bert.TFBertIntermediate object at 0x7f3ddd35e5d0>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd35eb50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertOutput.call of <transformers.modeling_tf_bert.TFBertOutput object at 0x7f3ddd35eb50>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING:tensorflow:AutoGraph could not transform <bound method TFBertPooler.call of <transformers.modeling_tf_bert.TFBertPooler object at 0x7f3ddcddda90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
WARNING: AutoGraph could not transform <bound method TFBertPooler.call of <transformers.modeling_tf_bert.TFBertPooler object at 0x7f3ddcddda90>> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.
Cause: Bad argument number for Name: 3, expecting 4
tf executing eager_mode: True
tf.keras model eager_mode: False
The ONNX operator number change on the optimization: 2575 -> 1670
```

when I run inference on input
```
{'input_ids': [array([ 101,  146, 1169, 1631, 1103, 3974,  117, 1169, 1128,  136,  102],
      dtype=int32)], 'token_type_ids': [array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32)], 'attention_mask': [array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)]}
```
I got the error
```python
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/xws61xnjc03fjiwfh7ci5cwgg1chmp3l-python3.7-onnxruntime-1.4.0/lib/python3.7/site-packages/onnxruntime/capi/session.py", line 110, in run
    return self._sess.run(output_names, input_feed, run_options)
onnxruntime.capi.onnxruntime_pybind11_state.InvalidArgument: [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got invalid dimensions for input: token_type_ids for the following indices
 index: 1 Got: 11 Expected: 7
 Please fix either the inputs or the model.
```

When I run it via the C API of onnxruntime, I got this error message:
```
Error: OrtError(Status { error_code: InvalidArgument, error_msg: "Got invalid dimensions for input: attention_mask for the following indices\n index: 1 Got: 418 Expected: 7\n Please fix either the inputs or the model." })
```

When I print out the model input shapes, I see
```
input 0: "attention_mask" ["N", 7] Int32
input 1: "input_ids" ["N", 7] Int32
input 2: "token_type_ids" ["N", 7] Int32
```
It is not clear to me how I should prepare the input to run it with onnxruntime.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- `transformers` version: 3.0.2
- Platform: Linux-5.4.57-x86_64-with-glibc2.2.5
- Python version: 3.7.7
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.1.0 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Information

Model I am using (Bert, XLNet ...): BERT

The problem arises when using:
* [x] the official example scripts: (give details below)
https://github.com/huggingface/transformers/blob/master/notebooks/04-onnx-export.ipynb

## To reproduce

```python
from transformers.convert_graph_to_onnx import convert
output_model_path = "onnx-test-tf/bert-base-cased.onnx"
convert(framework="tf", model="bert-base-cased", output=output_model_path, opset=11)

from transformers import BertTokenizerFast
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')

encoded = tokenizer("I can feel the magic, can you?", add_special_tokens=True)

import numpy
inputs_onnx = {k_: [numpy.array(v_, dtype=numpy.int32)] for k_, v_ in encoded.items()}
from onnxruntime import InferenceSession, SessionOptions, get_all_providers
sess_options = onnxruntime.SessionOptions()
session = onnxruntime.InferenceSession(output_model_path, sess_options, providers=['CPUExecutionProvider'])

results = session.run(None, inputs_onnx)
```


Truncated last sentence after bart finetuning on custom dataset.

- `transformers` version: 3.0.2
- Platform:
- Python version: 3.6
- PyTorch version (GPU?): 1.4
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Single GPU

### Who can help
@sshleifer

## Information

Model I am using (Bert, XLNet ...): BART

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Finetuning BART-large-xsum on my custom dataset with differing configs: min_length=590, max_length=620.
2. Doing inference on the trained model.
3. Sentences (specifically, last sentence) that BART produces oftentimes (~90% of the cases) is incomplete.

## Expected behavior

I would expect to have a complete and neat output, without having truncated outputs. Should mention that when I do the inference on the raw bart-large-cnn (or -xsum) checkpoint, I do not see this problem and all the outputs are complete. It sounds to me that finetuned-bart on custom dataset is not able to emit <EOS> token.

I also checked this thread: https://github.com/huggingface/transformers/issues/5674 which faces the same problem, but couldn't find the answer.


Longformer slow than Bert

When i set max_length = 2048, i found Longformer's speed is slower than commen bert, why?


Always got RuntimeError while converting ALBERT model to TorchScript (.pt file)

I am trying to convert ALBERT to a `.pt` file from the original albert model from transformers.(I am not very familiar with TorchScript so I want the `.pt` to be clean)

The code I ran (following the tutorial from [https://huggingface.co/transformers/torchscript.html](https://huggingface.co/transformers/torchscript.html)):
```

from transformers import AlbertModel, AlbertTokenizer, AlbertConfig
import torch

enc = AlbertTokenizer.from_pretrained("albert-xxlarge-v2")
text = "[CLS] Who was Jim Henson ? [SEP] Jim Henson was a puppeteer [SEP]"
tokenized_text = enc.tokenize(text)
masked_index = 8
tokenized_text[masked_index] = '[MASK]'
indexed_tokens = enc.convert_tokens_to_ids(tokenized_text)
segments_ids = [0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1]
tokens_tensor = torch.tensor([indexed_tokens])
segments_tensors = torch.tensor([segments_ids])
dummy_input = [tokens_tensor, segments_tensors]
config = AlbertConfig(vocab_size_or_config_json_file=73000, hidden_size=4096,
    num_hidden_layers=12, num_attention_heads=64, intermediate_size=16384, torchscript=True)
model = AlbertModel(config)
model.eval()
model = AlbertModel.from_pretrained("albert-xxlarge-v2", torchscript=True)
traced_model = torch.jit.trace(model, [tokens_tensor, segments_tensors])
torch.jit.save(traced_model, "albert-xxlarge-v2.pt")
```
But the second last line threw out a error:
`RuntimeError: The size of tensor a (15) must match the size of tensor b (14) at non-singleton dimension 3`

From the tutorial:
```
The trace is created relatively to the inputs’ dimensions. It is therefore constrained by the dimensions of the dummy input, and will not work for any other sequence length or batch size. When trying with a different size, an error such as:

The expanded size of the tensor (3) must match the existing size (7) at non-singleton dimension 2
```

So I tried changing `vocab_size_or_config_json_file` to a larger value, but still got the same error.
Am I doing something wrong? Thanks for any advice.


Add examples/bert-loses-patience who can help


Could not output hidden states using TFBertModel

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic (on Google Colab)
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
 @LysandreJik
 @jplu

## Information
1. When using TFBertModel, I tried to output the hidden states. Firstly, I tried to set config.output_hidden_states=True, but it gave the "tuple index out of range" error. The code is:

from transformers import TFBertModel, BertConfig
import tensorflow as tf
def single_bert():
    id = Input((128,), dtype=tf.int32)
    mask = Input((128,), dtype=tf.int32)
    atn = Input((128,), dtype=tf.int32)
    bert_config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
    bert_model = TFBertModel.from_pretrained('bert-base-uncased', config = bert_config)
    embedding = bert_model(id, attention_mask=mask, token_type_ids=atn)[2]
    model = tf.keras.Model(inputs=[id, mask, atn], outputs=embedding)
    return model
model = single_bert()
model.summary()

2. I have also tried to pass "output_hidden_states=True", but it still gave "tuple index out of range" error:

from transformers import TFBertModel, BertConfig
import tensorflow as tf
def single_bert():
    id = Input((128,), dtype=tf.int32)
    mask = Input((128,), dtype=tf.int32)
    atn = Input((128,), dtype=tf.int32)
    bert_model = TFBertModel.from_pretrained('bert-base-uncased')
    embedding = bert_model(id, attention_mask=mask, token_type_ids=atn, output_hidden_states=True)[2]
    model = tf.keras.Model(inputs=[id, mask, atn], outputs=embedding)
    return model
model = single_bert()
model.summary()

## To reproduce
Steps to reproduce the behavior:
1.
2.

## Expected behavior
I need to add some custom layers on top of the output hidden states and fine-tune the whole model, so firstly, I have to get the hidden states of Bert.


BERT and SpanBERT for Coreference Resolution

# 🌟 New model addition

## Model description

This is a recent approach for co-reference resolution using BERT, implemented from the papers [BERT for Coreference Resolution: Baselines and Analysis](https://arxiv.org/abs/1908.09091) and [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://arxiv.org/abs/1907.10529), which is the current state of the art on OntoNotes (79.6 F1). It uses tensorflow 1.14.0.

Reason why this is interesting is it achieves strong improvements on the OntoNotes (+3.9 F1) and GAP (+11.5 F1) benchmarks. Also, I think it would be a nice addition to huggingface library, as it has only the neuralcoref as the coreference resolution module.

## Open source status

* [x] the model implementation is available: (https://github.com/mandarjoshi90/coref)
* [x] the model weights are available: (https://github.com/facebookresearch/SpanBERT)
* [x] who are the authors: (@mandarjoshi90, @jkkummerfeld, @wenyudu)


Add Model Card for electra-base-german-uncased

model card

Related to pretrained model cards

This adds the model card for electra-base-german-uncased.

Could you please also have a look into #6495 because something went wrong with the upload.

Thanks
Philip


Model Upload does not show up `german-nlp-group/electra-base-german-uncased`

Hi,
yesterday I uploaded a new model to `german-nlp-group/electra-base-german-uncased`:

```bash
$ transformers-cli s3 ls --organization german-nlp-group
Neither PyTorch nor TensorFlow >= 2.0 have been found.Models won't be available and only tokenizers, configurationand file/data utilities can be used.
Filename                                          LastModified             ETag                               Size
------------------------------------------------- ------------------------ ---------------------------------- ---------
electra-base-german-uncased/config.json           2020-08-14T17:13:01.000Z "10c75064301189f269b4898d4265cd61"       467
electra-base-german-uncased/pytorch_model.bin     2020-08-14T17:13:37.000Z "a621e1cb07af0a08aaa643af52f9f189" 444881731
electra-base-german-uncased/tokenizer_config.json 2020-08-14T17:43:33.000Z "7f6d7cb22bc6342b9c942da874754264"        86
electra-base-german-uncased/vocab.txt             2020-08-14T17:43:31.000Z "e9fa1e40c556fc02c62ebaa214a52dc4"    275501
```

But it does not show up. See here: https://huggingface.co/german-nlp-group

What happened here? Could you fix that?

Thanks
Philip


[testing] a new TestCasePlus subclass + get_auto_remove_tmp_dir()

I present to you a new `TestCasePlus` class, which is an extension of `testutil.TestCase`. Currently it only has one extra feature, but I'm sure there will be more in the future, hence the more generic name.

So the intention was to provide:
- an easy way to create unique temp dirs in test modules and get them automatically removed at the end of the test, regardless of whether a test succeeded or not.
- an easy way not to remove the temp dir for debug purposes
- provide a hardcoded temp dir for debug purposes (and secure so that `rm -r /something` won't happen)
- optionally, clean up the temp dir right away if a hardcoded path is provided

Some ideas were discussed here: https://github.com/huggingface/transformers/issues/6471

So this PR implements this feature and uses it in 2 test modules that currently don't have a complete solution, and removing much much code on the way.

Usage:

Feature 1: Flexible auto-removable temp dirs which are guaranteed to get removed at the end of test.

In all the following scenarios the temp dir will be auto-removed at the end of test, unless `after=False`.

1. create a unique temp dir and delete it at the end, `tmp_dir` will contain the path to the created temp dir
```
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir()
```
2. create a temp dir of my choice and delete it at the end - useful for debug when you want to monitor a specific directory
```
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/run/test")
```
or just:
```
    tmp_dir = self.get_auto_remove_tmp_dir("./tmp/run/test")
```

3. create a temp dir of my choice and do not delete it at the end - useful for when you want to look at the temp results
```
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/run/test", after=False)
```
or just:
```
    tmp_dir = self.get_auto_remove_tmp_dir("./tmp/run/test", False)
```

4. create a temp dir of my choice and ensure to delete it right away - useful for when you disabled deletion in the previous test run and want to make sure the that tmp dir is empty before the new test is run
```
def test_whatever(self):
    tmp_dir = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/run/test", before=True)
```

Note 1: In order to run the equivalent of `rm -r` safely, only subdirs of the project repository checkout are allowed if an explicit `tmp_dir` is used, so that by mistake no `/tmp` or similar important part of the filesystem will get nuked. i.e. please always pass paths that start with `./`

Note 2: Each test can register multiple temp dirs and they all will get auto-removed, unless requested otherwise.

So you can see from the 4 main possible scenarios, during debug one needs to tweak only one line of code.

There is only one small remaining deficiency: Since the temp dir is pre-created, the tests will not be able to test things like `--output_dir` creation in examples - i.e. the dir will already be there. So if needed, the code can be extended to have a flag to not create the dir, but only register it for deletion. Though it'd be tricky for when `tmp_dir` is not passed explicitly and we rely on `tempfile`- I guess it can create and immediately delete the temp dir and save and reuse its path - I don't know whether there might be a race condition here. But chances are that this is not really needed.

Thank you for reading. Ideas and suggestions for improvements are welcome.

@JetRunner, @LysandreJik, @sshleifer, @sgugger


Fixes paths with spaces in seq2seq example

Fixes https://github.com/huggingface/transformers/issues/6477


Fixed label datatype for STS-B

The STS Benchmark has decimal labels instead of integers.
But inside the `glue_convert_examples_to_features` function, when you're using TensorFlow datasets it is casting the label as an integer in the returned TF data generator.
With this simple edit, the function changes its casting datatype according to the selected task.


Whole Word Masking Implementation

# 🚀 Feature request

Currently, training the models from scratch like Roberta do not support whole word masking (e.g., language modeling examples). Only pre-trained models are available. Is it possible to include whole word masking in the input layers?

## Motivation

Whole word masking leads to performance boosts. So, adding this feature would be useful if someone wants to train the models from scratch.


[Doc] add more MBart and other doc

This PR
1. adds example for MBart
2. adds MBart in pre_trained models list and readme (Pegasus was missing from readme, so also added that).

@sshleifer , @sgugger


GitHub Template: Tag @stefan-it for token classification related bug reports

Hi,

this PR adds myself as person to tag for all token classification related bug reports :)


Fix TPU Convergence bug introduced by PR#6151

Currently with the bug introduced we're taking two optimizer steps per
batch: one global one, where `xm.optimizer_step` injects a CRS between
all cores in training, and one without. This has been affecting training
accuracy (for example, XLNet GLUE on MNLI is not converging, etc.).


about encoder and decoder input when using seq2seq model

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
Hello, I'm trying to using seq2seq model (such as bart and EncoderDecoderModel(bert2bert))
And I'm little bit confused about input_ids, decoder_input_ids, tgt in model inputs.

As I know in seq2seq model, decoder_input should have special token(\<s> or something) before the sentence and target should have special token(\</s> or somethin) after the sentence. for example, `decoder_input = <s> A B C D E` , `target = A B C D E</s>`

so my question is
1. Should I put the these special tokens in decoder_inputs_ids and tgt_ids when using seq2seq model in this library?
or can i just pass the decoder_input_ids and tgt_ids without any special token ids?

2. Also, should I put `add_special_tokens=True` for encoder input_ids and put \</s> or \<eos> token after target ids?
for example, `input = a b c d e, decoder_input = <s>A B C D E, target = A B C D E</s>`
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


from_pretrained() never works

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux
- Python version: 3.6
- PyTorch version (GPU?): 1.5.1 (yes)
- Tensorflow version (GPU?):
- Using GPU in script?: not relevant
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik , @TevenLeScao , @mfuntowicz
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu
documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): any

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. `import transformers as pt`
2. `pt.AutoModelForSequenceClassification.from_pretrained(<any_valid_model_id>)`
3. Observe the error below

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

```python
>>> pt.AutoModelForSequenceClassification.from_pretrained('xlnet-base-cased')
I0814 15:00:47.832349 46912496391360 configuration_utils.py:264] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/xlnet-base-cased-config.json from cache at /xxx/torch/transformers/c9cc6e53904f7f3679a31ec4af244f4419e25ebc8e71ebf8c558a31cbcf07fc8.69e5e35e0b798cab5e473f253752f8bf4d280ee37682281a23eed80f6e2d09c6
I0814 15:00:47.832984 46912496391360 configuration_utils.py:300] Model config XLNetConfig {
  "architectures": [
    "XLNetLMHeadModel"
  ],
  "attn_type": "bi",
  "bi_data": false,
  "bos_token_id": 1,
  "clamp_len": -1,
  "d_head": 64,
  "d_inner": 3072,
  "d_model": 768,
  "dropout": 0.1,
  "end_n_top": 5,
  "eos_token_id": 2,
  "ff_activation": "gelu",
  "initializer_range": 0.02,
  "layer_norm_eps": 1e-12,
  "mem_len": null,
  "model_type": "xlnet",
  "n_head": 12,
  "n_layer": 12,
  "pad_token_id": 5,
  "reuse_len": null,
  "same_length": false,
  "start_n_top": 5,
  "summary_activation": "tanh",
  "summary_last_dropout": 0.1,
  "summary_type": "last",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 250
    }
  },
  "untie_r": true,
  "vocab_size": 32000
}

Traceback (most recent call last):
  File "/xxx/.conda/envs/xxx/lib/python3.6/site-packages/transformers/modeling_utils.py", line 655, in from_pretrained
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/xxx/.conda/envs/xxx/lib/python3.6/site-packages/transformers/modeling_auto.py", line 1363, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/xxx/.conda/envs/xxx/lib/python3.6/site-packages/transformers/modeling_utils.py", line 662, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load weights for 'xlnet-base-cased'. Make sure that:

- 'xlnet-base-cased' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'xlnet-base-cased' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
A pretrained model should be loaded. This worked (and still works) great in `pytorch_transformers`. I switched to `transformers` because XLNet-based models stopped working in `pytorch_transformers`. But surprise surprise in `transformers` no model whatsoever works for me.


Add tests/test_tokenization_reformer.py

As titled. Attends to issue [#6333](https://github.com/huggingface/transformers/issues/6333).


Assertion error when training a new RoBERTa from scratch

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-3.10.0-862.14.4.el7.x86_64-x86_64-with-centos-7.5.1804-Core
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: <No.

### Who can help

Maybe @LysandreJik  ? :-)

## Information

Model I am using RoBERTa:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

The dataset is a simple line-by-line dataset.

## To reproduce

Steps to reproduce the behavior:

1. Train a tokenizer according to [this](https://huggingface.co/blog/how-to-train#2-train-a-tokenizer)
2. Split line-by-line dataset into train and eval
3. Run below:

```sh
python run_language_modeling.py \
    --output_dir $MODEL_DIR/myBERT-small-v1 \
    --model_type roberta \
    --mlm \
    --config_name $MODEL_DIR/myBERT-small \
    --tokenizer_name $MODEL_DIR/myBERT-small \
    --do_train \
    --do_eval \
    --per_device_train_batch_size 8 \
    --learning_rate 1e-4 \
    --num_train_epochs 5 \
    --save_total_limit 2 \
    --save_steps 2000 \
    --per_gpu_train_batch_size 16 \
    --evaluate_during_training \
    --line_by_line \
    --train_data_file $HOME/myBERT/train.txt \
    --eval_data_file $HOME/myBERT/eval.txt \
    --seed 42
```

```log
08/13/2020 14:23:20 - INFO - transformers.configuration_utils -   loading configuration file /home/erippeth/myBERT/model/myBERT-small/config.json
08/13/2020 14:23:20 - INFO - transformers.configuration_utils -   Model config RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 52000
}

08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Model name '/home/erippeth/myBERT/model/myBERT-small' not found in model shortcut name list (roberta-base, roberta-large, roberta-large-mnli, distilroberta-base, roberta-base-openai-detector, roberta-large-openai-detector). Assuming '/home/erippeth/myBERT/model/myBERT-small' is a path, a model identifier, or url to a directory containing tokenizer files.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/added_tokens.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/special_tokens_map.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/tokenizer_config.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   Didn't find file /home/erippeth/myBERT/model/myBERT-small/tokenizer.json. We won't load it.
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file /home/erippeth/myBERT/model/myBERT-small/vocab.json
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file /home/erippeth/myBERT/model/myBERT-small/merges.txt
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:20 - INFO - transformers.tokenization_utils_base -   loading file None
08/13/2020 14:23:21 - INFO - __main__ -   Training new model from scratch
/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_auto.py:709: FutureWarning: The class `AutoModelWithLMHead` is deprecated and will be removed in a future version. Please use `AutoModelForCausalLM` for causal language models, `AutoModelForMaskedLM` for masked language models and `AutoModelForSeq2SeqLM` for encoder-decoder models.
  FutureWarning,
08/13/2020 14:23:27 - INFO - transformers.data.datasets.language_modeling -   Creating features from dataset file at /home/erippeth/myBERT/train.txt
08/13/2020 17:40:20 - INFO - transformers.data.datasets.language_modeling -   Creating features from dataset file at /home/erippeth/myBERT/eval.txt
08/13/2020 18:56:31 - WARNING - transformers.trainer -   You are instantiating a Trainer but Tensorboard is not installed. You should consider installing it.
08/13/2020 18:56:31 - INFO - transformers.trainer -   You are instantiating a Trainer but W&B is not installed. To use wandb logging, run `pip install wandb; wandb login` see https://docs.wandb.com/huggingface.
08/13/2020 18:56:31 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
08/13/2020 18:56:31 - WARNING - transformers.training_args -   Using deprecated `--per_gpu_train_batch_size` argument which will be removed in a future version. Using `--per_device_train_batch_size` is preferred.
08/13/2020 18:56:31 - INFO - transformers.trainer -   ***** Running training *****
08/13/2020 18:56:31 - INFO - transformers.trainer -     Num examples = 16661098
08/13/2020 18:56:31 - INFO - transformers.trainer -     Num Epochs = 5
08/13/2020 18:56:31 - INFO - transformers.trainer -     Instantaneous batch size per device = 8
08/13/2020 18:56:31 - INFO - transformers.trainer -     Total train batch size (w. parallel, distributed & accumulation) = 16
08/13/2020 18:56:31 - INFO - transformers.trainer -     Gradient Accumulation steps = 1
08/13/2020 18:56:31 - INFO - transformers.trainer -     Total optimization steps = 5206595
^MEpoch:   0%|          | 0/5 [00:00<?, ?it/s]
^MIteration:   0%|          | 0/1041319 [00:00<?, ?it/s]ESC[A
^MIteration:   0%|          | 1/1041319 [00:01<508:20:24,  1.76s/it]ESC[A
^MIteration:   0%|          | 2/1041319 [00:02<395:24:33,  1.37s/it]ESC[A
^MIteration:   0%|          | 3/1041319 [00:02<306:50:22,  1.06s/it]ESC[A/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [229,0,0], thread: [32,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [229,0,0], thread: [33,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
...
/opt/conda/conda-bld/pytorch_1595629416375/work/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [275,0,0], thread: [63,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Iteration:   0%|          | 3/1041319 [00:03<332:03:04,  1.15s/it]
Epoch:   0%|          | 0/5 [00:03<?, ?it/s]
Traceback (most recent call last):
  File "run_language_modeling.py", line 281, in <module>
    main()
  File "run_language_modeling.py", line 245, in main
    trainer.train(model_path=model_path)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/trainer.py", line 499, in train
    tr_loss += self._training_step(model, inputs, optimizer)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/trainer.py", line 622, in _training_step
    outputs = model(**inputs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_roberta.py", line 239, in forward
    output_hidden_states=output_hidden_states,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 762, in forward
    output_hidden_states=output_hidden_states,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 439, in forward
    output_attentions,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 371, in forward
    hidden_states, attention_mask, head_mask, output_attentions=output_attentions,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 315, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions,
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/erippeth/miniconda3/envs/myBERT/lib/python3.6/site-packages/transformers/modeling_bert.py", line 258, in forward
    context_layer = context_layer.permute(0, 2, 1, 3).contiguous()
RuntimeError: CUDA error: device-side assert triggered
```

## Expected behavior

Model should train without failure, but instead it fails with an assertion error. I believe this is related to an embedding dimension issue, but the config's `vocab_size` matches the length of the newly-trained tokenizer and this is the embedding dimension set in the training script.


Regarding GPU use for LM

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Hi,
I am running example given in README.md of language_modeling using following command:
export TRAIN_FILE=/path/to/dataset/wiki.train.raw
export TEST_FILE=/path/to/dataset/wiki.test.raw

python run_language_modeling.py \
    --output_dir=output \
    --model_type=gpt2 \
    --model_name_or_path=gpt2 \
    --do_train \
    --train_data_file=$TRAIN_FILE \
    --do_eval \
    --eval_data_file=$TEST_FILE

It has started training but it is not using GPU (TITAN X) at all, when I see throug nvidia-smi command
I am new to this So Can you please let me know if I'm missing anything here.

Thanks.-->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Longformer Memory Consumption query

Hello,
Apologies if I am misunderstanding it, but if I use roberta with a max sequence length of 256 and I can run, for example, say, a batch size of 64 on one gpu for a task, with the longformer, can I use the same batch size with a window length of 256 with the max sequence length being 4096?


what's the difference between TFBertOutput and TFBertSelfOutput?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->

`TFBertOutput` with `TFBertSelfOutput` seem same in their codes. why did you write two same layers? is there for some reasons?


Import accuracy_score


[TFTrainer] gradient accumulation error

## Environment info

- `transformers` version: master (#9a8c168)
- Tensorflow version: 2.3.0

### Who can help

 Trainer: @sgugger
 tensorflow: @jplu

## Information

When using >1 `gradient_accumulation_steps` with TFTrainer and model inputs which are *not* simple tensors (for example dicts) the trainer fails. Also, to me it looks like there are logic issues in the way the `reduced_features` are computed for the gradient accumulation (not sure though).

Issue is here: https://github.com/huggingface/transformers/blob/9a8c168f56fe3c0e21d554a577ac03beb004ef89/src/transformers/trainer_tf.py#L602

## To reproduce

```python
import tensorflow as tf
from transformers import TFT5ForConditionalGeneration, TFTrainer, TFTrainingArguments

input_ids = [[1, 2, 3], [1, 2, 3]]
labels = [1, 2, 3]

dataset = tf.data.Dataset.from_tensor_slices(({'input_ids': input_ids}, labels))
model = TFT5ForConditionalGeneration.from_pretrained("t5-base")

training_args = TFTrainingArguments(
    output_dir='./results',  # output directory
    logging_steps=100,
    max_steps=2,
    save_steps=2000,
    per_device_train_batch_size=2,  # batch size per device during training
    per_device_eval_batch_size=8,  # batch size for evaluation
    warmup_steps=0,  # number of warmup steps for learning rate scheduler
    weight_decay=0.0,  # strength of weight decay
    learning_rate=5e-5,
    gradient_accumulation_steps=2
)

with training_args.strategy.scope():
    model = TFT5ForConditionalGeneration.from_pretrained("t5-base")

trainer = TFTrainer(
    model=model,  # the instantiated 🤗 Transformers model to be trained
    args=training_args,  # training arguments, defined above
    train_dataset=dataset,  # training dataset
)

trainer.train()
```

## Issues

### Error
Produces error `TypeError: unhashable type: 'slice'`. Also `/` produce floats on python3, which I guess is not intended here.
Solution in same spirit could be conditional use of
```
reduced_features = {
 ft: features[ft][:self.args.train_batch_size // self.args.n_replicas]
 for ft in features
}
```

Already mentioned here https://github.com/huggingface/transformers/pull/6038#issuecomment-664706046

### Logic issue
I don't understand what the `n_replicas` has to do with the gradient accumulation here? Shouldn't the denominator rather be `gradient_accumulation_steps`? And shouldn't it actually use the different slices of the features, and not always the "first" slice?
Might be totally misunderstanding this.

Also this line doesn't seem to have any purpose: https://github.com/huggingface/transformers/blob/master/src/transformers/trainer_tf.py#L607

Happy to provide PR if someone can give me a hint on the logic issue.


Upladed model is not indexed

# ❓ Questions & Help

Hi guys, I uploaded a model several hours ago (t5-base-finetuned-boolq) and it is not indexed in the model hub search engine yet!

Thanks, Manu


### Who can help

 examples/distillation: @VictorSanh
 examples/seq2seq: @sshleifer

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Try running seq2seq/finetune.sh with data_dir or output_dir with escaped spaces in it
2. You'll get a `finetune.py: error: unrecognized arguments`

This is bad because Google Drive mounts at `/content/drive/My Drive/` in Colab and thus the example scripts won't work if saving or reading from Drive.

I've created a [Colab Notebook](https://colab.research.google.com/drive/1N-8m9FC9GbAywVJZAgSBkLqe24SPRfl8?usp=sharing) with repro.

The fix I've found is to change:
```
python finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.1 \
    $@
```
to
```
python finetune.py \
    --learning_rate=3e-5 \
    --fp16 \
    --gpus 1 \
    --do_train \
    --do_predict \
    --n_val 1000 \
    --val_check_interval 0.1 \
    "$@"
```


Question about loss computing in BartForConditionalGeneration

I notice that in [BartForConditionalGeneration](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bart.py#L1043), the labels and logits are not shifted when computing cross-entropy loss. Should I provide a pre-possessed shifted labels to the model for training?


Use hash to clean the test dirs

This one solves it once for all. What do you think? @sgugger @LysandreJik


Training Data of  xlm-roberta-large-finetuned-conll03-* models

Hi, I'm curious about the training data of xlm-r models finetuned on conll ner datasets (e.g.  xlm-roberta-large-finetuned-conll03-german,  xlm-roberta-large-finetuned-conll03-english), are the models trained on train+dev sets?


[sched] polynomial_decay_schedule use default power=1.0

As discussed in https://github.com/huggingface/transformers/pull/6361 we weren't sure why fairseq's `polynomial_decay_schedule` `power` default was `1.0`, and decided to go with `2.0` as the latter does something polynomial.

I got the devs at fairseq to answer this question: https://github.com/pytorch/fairseq/issues/2466#issuecomment-673146603

> myleott wrote:
> This is based on the original BERT code, which implemented a linear decay via a polynomial schedule with power=1.0: https://github.com/google-research/bert/blob/f39e881b169b9d53bea03d2d341b31707a6c052b/optimization.py#L37

So, perhaps we do the same or we don't.

If we don't - then the doc needs to be fixed that the default is `power=2.0` as currently it says `1.0` - my mistake. If we do (this PR), then the doc is already correct.

Thanks.


Hi I have trained a Bert token classification model for the Italian language using the "dbmdz/bert-base-italian-uncased". I have trained the model in a machine running Pytorch-1.4.0 and transformer 3.0.2, when I installed it few days back as it's the latest version.
I copied the saved best model to a server that runs Pytorch-1.4.0 & transformer version 2.3.0. I sent a request to the model to get the predictions, but I got the following warnings.

# Inference code
```
tokenizer = transformers.BertTokenizer.from_pretrained("dbmdz/bert-base-italian-uncased", do_lower_case=False)

Assuming I have tokenized the requested text into the variable "tokens"

indexed_tokens = tokenizer.convert_tokens_to_ids(tokens)
segments_ids = [0] * len(tokens)
tokens_tensor = torch.tensor([indexed_tokens]).to(device)
segments_tensors = torch.tensor([segments_ids]).to(device)
logit = model(tokens_tensor, token_type_ids=None, attention_mask=segments_tensors)
```

# Warnings
```
Model name 'dbmdz/bert-base-italian-uncased' not found in model shortcut name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased, bert-base-finnish-cased-v1, bert-base-finnish-uncased-v1). Assuming 'dbmdz/bert-base-italian-uncased' is a path or url to a directory containing tokenizer files.
Didn't find file dbmdz/bert-base-italian-uncased/added_tokens.json. We won't load it.
Didn't find file dbmdz/bert-base-italian-uncased/special_tokens_map.json. We won't load it.
Didn't find file dbmdz/bert-base-italian-uncased/tokenizer_config.json. We won't load it.
loading file https://s3.amazonaws.com/models.huggingface.co/bert/dbmdz/bert-base-italian-uncased/vocab.txt from cache at /root/.cache/torch/transformers/02b5ab8ef6a3a1d4af18c318bb4c53155a59a3893dd557b922d2467b269cd405.5cbaac66fdfadbe363aad01956dac0be9bf700f2c8c87012dc078b87e2fa4181
loading file None
loading file None
loading file None
```
```
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertForTokenClassification' has changed. Saved a reverse patch to BertForTokenClassification.patch. Run `patch -p0 < BertForTokenClassification.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertModel' has changed. Saved a reverse patch to BertModel.patch. Run `patch -p0 < BertModel.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertEmbeddings' has changed. Saved a reverse patch to BertEmbeddings.patch. Run `patch -p0 < BertEmbeddings.patch` to revert your changes.  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.normalization.LayerNorm' has changed. Saved a reverse patch to LayerNorm.patch. Run `patch -p0 < LayerNorm.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertEncoder' has changed. Saved a reverse patch to BertEncoder.patch. Run `patch -p0 < BertEncoder.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.container.ModuleList' has changed. Saved a reverse patch to ModuleList.patch. Run `patch -p0 < ModuleList.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertLayer' has changed. Saved a reverse patch to BertLayer.patch. Run `patch -p0 < BertLayer.patch` to revert your changes.  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertAttention' has changed. Saved a reverse patch to BertAttention.patch. Run `patch -p0 < BertAttention.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertSelfAttention' has changed. Saved a reverse patch to BertSelfAttention.patch. Run `patch -p0 < BertSelfAttention.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. Saved a reverse patch to Linear.patch. Run `patch -p0 < Linear.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertSelfOutput' has changed. Saved a reverse patch to BertSelfOutput.patch. Run `patch -p0 < BertSelfOutput.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertIntermediate' has changed. Saved a reverse patch to BertIntermediate.patch. Run `patch -p0 < BertIntermediate.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertOutput' has changed. Saved a reverse patch to BertOutput.patch. Run `patch -p0 < BertOutput.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'transformers.modeling_bert.BertPooler' has changed. Saved a reverse patch to BertPooler.patch. Run `patch -p0 < BertPooler.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
./torch/serialization.py:593: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Tanh' has changed. Saved a reverse patch to Tanh.patch. Run `patch -p0 < Tanh.patch` to revert your changes.
  warnings.warn(msg, SourceChangeWarning)
```
and finally it ended with the below error.

```
"BertEncoder' object has no attribute 'output_hidden_states".
```
Can someone help me understand Is it because of the Pytorch, transformer version mismatch between the trained model on a machine and the inference on the server? or if "dbmdz/bert-base-italian-uncased" is available in the 2.3.0 version or not? or is there any other way I can make this work instead of retraining the model at a lower version to match the version of the server?
Assuming that changing the versions in the server is not quite possible as of now.
Appreciate your help.


[testing] automatically clean up temp dirs during teardown

Recently, a crucial fix was applied to several tests https://github.com/huggingface/transformers/pull/6453/files. the temp dir wasn't getting cleaned and subsequent tests were unreliable as they were tapping into invalid old data. The remaining issue is that the added fix is not guaranteed to run. And it repeats itself many times.

I thought of several ways to ensure the removal of the temp dir and how to make it easier to use in the tests. Here are some ideas I came up with

## Idea 1

Using a simple `tempfile.TemporaryDirectory` context manager:

```
from tempfile import TemporaryDirectory
class ExamplesTests(unittest.TestCase):

    def test_run_pl_glue(self):
        with TemporaryDirectory() as tmp_dir:
            testargs = f"""
                run_pl_glue.py
                --output_dir {tmp_dir.name}
           [...]
```
Pros:
- generic code
- very localized
- can be done multiple times in the same test and having a fresh temp dir

Cons:
- can't pass a specific fixed dir so that it's easier to debug - could write a custom context manager that supports both random and fixed path.
- tricky to debug - if one wants the temp dir to not be removed while developing the test - could write a custom version that supports an argument that will not remove the temp dir
- have to reach into the object to get the actual path with `obj.name`

## Idea 1.5

This one solves the cons of idea 1

Write a  custom context manager that takes a hard-coded path and a flag to clean up or not to make it easy to debug, to be built on top of `tempfile.TemporaryDirectory`.

I haven't written it yet. But the core should be similar to `TestsWithTempDir` shown in the next idea, plus a context manager.

But here is how it would be used:

```
from transformers.test_utils import temp_dir_ctx
class ExamplesTests(unittest.TestCase):

    def test_run_pl_glue(self):
        with temp_dir_ctx(cleanup=True, path=None) as tmp_dir:
            testargs = f"""
                run_pl_glue.py
                --output_dir {tmp_dir.name}
           [...]
```
So that we could have:

Most of the time with minimal extra code, which will use a random path and auto-delete:
```
        with temp_dir_ctx() as tmp_dir:
            do_something_with(tmp_dir.name)
```
If we want a specific tmp path:
```
        with temp_dir_ctx(path="/use/my/path") as tmp_dir:
            do_something_with(tmp_dir.name)
```

if we are debugging and don't want the auto-deletion
```
        with temp_dir_ctx(cleanup=False) as tmp_dir:
            do_something_with(tmp_dir.name)
```

the only remaining cons:
- have to reach into the object to get the actual path with `obj.name` - can fix with `def __str__(self): return self.name`

## Idea 2

Solve the problem on the test class level, so that the tests don't need to do anything at all to clean up the temp dir. This solution uses `unittest.TestCase`'s `setUp`/`tearDown` fixtures.

```
from pathlib import Path
import tempfile
import shutil
class TestsWithTempDir(unittest.TestCase):
    """
    This class is for tests that need to automatically remove a temp dir at the end of the test
    regardless of its success or failure.

    if no `tmp_dir` is passed a unique temp dir is created. if it's passed that passed dir is used instead.

    In either case that path is created and `self.tmp_dir` is set to the path that was used.

    Example 1: Let the system choose the path

    class ExamplesTests(TestsWithTempDir):

        def test_run_something(self):
            print(f"{self.tmp_dir} will be removed at the end of the test")

    Example 2: Use the path I supply

    class ExamplesTests(TestsWithTempDir):

        def __init__():
            super.__init__(tmp_dir="./foo/bar")

        def test_run_something(self):
            print(f"{self.tmp_dir} will be removed at the end of the test")

    """
    def __init__(self, tmp_dir=None):
        self.tmp_dir = tmp_dir
        self.tmp_dir_obj = None

    def setUp(self):
        if self.tmp_dir:
            Path(self.tmp_dir).mkdir(parents=True, exist_ok=True)
        else:
            self.tmp_dir_obj = tempfile.TemporaryDirectory()
            self.tmp_dir = self.tmp_dir_obj.name

    def tearDown(self):
        if self.tmp_dir_obj:
            del self.tmp_dir_obj
        else:
            shutil.rmtree(self.tmp_dir, ignore_errors=True)
```

Pros:
- moves the cleaning up responsibility away from the test, leaving the test focused to just what it tests
- very flexible - can handle custom and random paths
- debug should be relatively easy - just need to add another option or a method to not tear-down (I haven't implemented it yet)

Cons:
- only supports one tmp dir per test - won't work if multiple executions happen in the same test
- the action is far removed from the code - could be hard to see - I'm especially concerned with running `shutil.rmtree` at a distance - it'd be easy to make a mistake of passing `/tmp/foo` instead of `./tmp/foo` or worse. I'd rather not use  `shutil.rmtree` at all unless it's right there when the developer can see what they are removing.

-----

After contemplating these different solutions, I feel that locality is more important than behind the scenes magic, so I feel the best solution would be Idea 1.5 - i.e. a custom context manager that makes it easy to debug, to be built on top of `tempfile.TemporaryDirectory`, and also supports a hardcoded tmp path.

Please, let me know if any of these resonate with you and then I can code a PR that can be seen in action.

Thank you!


Generation doc

Add documentation (and clean docstrings) of `GenerationMixin` and `TFGenerationMixin`.


Fix typo

Related to pretrained model cards