TFBertModel not working at all with any type of model
Author: brunopistone
Pull request: no
Labels: name: wontfix

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: macOS
- Python version: 3.7
- PyTorch version (GPU?): NO
- Tensorflow version (GPU?): 2.3.1
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): Bert "dbmdz/bert-base-italian-xxl-cased"

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

<---------------------------------------------->

Hi @LysandreJik , @stefan-it , @sgugger , I'm trying to use `dbmdz/bert-base-italian-xxl-cased`for creating a keras model for a classification task.
I've followed the documentation but I continue to receive the following error:

```
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[5,0] = 102 is not in [0, 2)
	 [[node functional_1/bert/embeddings/token_type_embeddings/embedding_lookup (defined at /anaconda3/envs/profanity-detector/lib/python3.7/site-packages/transformers/modeling_tf_bert.py:186) ]] [Op:__inference_train_function_29179]
```

This is the model:

```
from transformers import TFBertModel, BertTokenizer

bert_model = TFBertModel.from_pretrained("dbmdz/bert-base-italian-xxl-cased")
tokenizer = BertTokenizer.from_pretrained("dbmdz/bert-base-italian-xxl-cased")

input_ids = tf.keras.layers.Input(shape=(constants.MAX_SEQ_LENGTH,), dtype=tf.int32)
token_type_ids = tf.keras.layers.Input(shape=(constants.MAX_SEQ_LENGTH,), dtype=tf.int32)
attention_mask = tf.keras.layers.Input(shape=(constants.MAX_SEQ_LENGTH,), dtype=tf.int32)

seq_output, _ = bert_model({
        "input_ids": input_ids,
        "token_type_ids": token_type_ids,
        "attention_mask": attention_mask
})

pooling = tf.keras.layers.GlobalAveragePooling1D()(seq_output)
dropout = tf.keras.layers.Dropout(0.2)(pooling)
output = tf.keras.layers.Dense(constants.CLASSES, activation="softmax")(dropout)

model = tf.keras.Model(
        inputs=[input_ids, token_type_ids, attention_mask],
        outputs=[output]
)

model.compile(optimizer=tf.optimizers.Adam(lr=0.00001), loss='sparse_categorical_crossentropy', metrics=['accuracy'])
```

My dataset is tokenized by this method:
```
    def map_to_dict(self, input_ids, attention_masks, token_type_ids, labels):
        return {
            "input_ids": input_ids,
            "token_type_ids": token_type_ids,
            "attention_mask": attention_masks,
        }, labels

    def tokenize_sequences(self, tokenizer, max_sequence_length, data, labels):
        try:
            token_ids = []
            token_type_ids = []
            attention_mask = []

            for sentence in data:
                bert_input = tokenizer.encode_plus(
                    sentence,
                    add_special_tokens=True,  # add [CLS], [SEP]
                    max_length=max_sequence_length,  # max length of the text that can go to BERT
                    truncation=True,
                    pad_to_max_length=True,  # add [PAD] tokens
                    return_attention_mask=True  # add attention mask to not focus on pad tokens
                )

                token_ids.append(bert_input["input_ids"])
                token_type_ids.append(bert_input["token_type_ids"])
                attention_mask.append(bert_input["attention_mask"])

            return tf.data.Dataset.from_tensor_slices((token_ids, token_type_ids, attention_mask, labels)).map(self.map_to_dict)
        except Exception as e:
            stacktrace = traceback.format_exc()

            logger.error("{}".format(stacktrace))

            raise e

ds_train_encoded = tokenize_sequences(tokenizer, 512, X_train, y_train).shuffle(10000).batch(6)
```
X_train examples:
```
["Questo video è davvero bellissimo", "La qualità del video non è proprio il massimo"......]
```

y_train examples:
```
[[1], [0]...]
```

I continue to receive the error described before.

```
tensorflow.python.framework.errors_impl.InvalidArgumentError:  indices[5,0] = 102 is not in [0, 2)
	 [[node functional_1/bert/embeddings/token_type_embeddings/embedding_lookup (defined at /anaconda3/envs/profanity-detector/lib/python3.7/site-packages/transformers/modeling_tf_bert.py:186) ]] [Op:__inference_train_function_29179]
```

If I try to use TFBertForSequenceClassification everything works fine (for this reason I'm excluding tokenization problems).

Can you please provide a solution or a well documented guide for using TFBertModel class with Keras model (I cannot find it)?

Thank you


Comments:
Hi! Thanks for opening such a detailed issue!

Let me ping @jplu, the TensorFlow master so that he may help you.

Hello!

This is because you are not updating your embedding size for `token_type_embeddings`.  If you check carefully the config for the `dbmdz/bert-base-italian-xxl-cased` mode you will see:
```
BertConfig {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 32102
}
```
And `type_vocab_size` equals `2`. While in your example your are trying with a different value, that's why the error tells you that it cannot find any lookup in the range `[0:2)`.


Hi @jplu , thank you for your answer. Now it works!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.

I am trying this exact thing and i get an error on this line:
net = tf.keras.layers.Dense(512, activation='relu')(seq_output)
Inputs to a layer should be tensors. Got: last_hidden_state
Any ideas?