Wrong argument passed during TFRobertaClassificationHead initialization
Author: harkous
Pull request: no
Labels: name: TensorFlow

name: Core: Modeling

name: Should Fix

# 🐛 Bug

## Information

There is an issue preventing a RoBERTa classification model from being serialized. It is related to a problem in passing `config` as the first argument to `tf.keras.layers.Layer`. However, the [expected positional argument](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer) is `trainable`:

https://github.com/huggingface/transformers/blob/d6a677b14bcfd56b22fafeb212a27c6068886e07/src/transformers/modeling_tf_roberta.py#L327-L331

This is the root cause behind issue #3664 (about serialization).
A related fix for GPT2: #2738.

Model I am using (Bert, XLNet ...):
RoBERTa

Language I am using the model on (English, Chinese ...):
English
The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the code below:
```python
from transformers import (TFRobertaForSequenceClassification)
base_model = TFRobertaForSequenceClassification.from_pretrained("roberta-base")
print(base_model.classifier.trainable)
```

## Expected behavior
The output is:
`True`

The current output is
```
RobertaConfig {
  "architectures": [
    "RobertaForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "bos_token_id": 0,
  "eos_token_id": 2,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-05,
  "max_position_embeddings": 514,
  "model_type": "roberta",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 1,
  "type_vocab_size": 1,
  "vocab_size": 50265
}
```


## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.10.0
- Platform: Colab
- Python version: 3.6.9
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.2.0
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


Comments:
Hello,

The way the TF models behave has been recently updated. Can you please retry with the `master` version?

Hi Julien,

Thanks. It's the same issue in the `master` currently:

https://github.com/huggingface/transformers/blob/9f5d5a531d769d07403f59661884e254f8420afe/src/transformers/modeling_tf_roberta.py#L320-L324

`config` is still passed as the first parameter to the `__init__` of  parent class `tf.keras.layers.Layer` while the latter expects `trainable` as the first parameter. I fixed it on my side by simply removing that `config` parameter from the `super().__init__(` call. But I wasn't sure if this affects other parts of the repo. Otherwise, I would have submitted a PR.


Ok, thanks for the feedback. Indeed, the `config` parameter is important. I will take some time to review this. Sorry for the inconvenience.

You are totally right, `config` here is useless and the same appears in other models . Do you mind to do a PR? And I will help you to fix all this :)

Thanks for making the checks. I submitted a PR: https://github.com/huggingface/transformers/pull/4884