Could not output hidden states using TFBertModel
Author: YLi999
Pull request: no
Labels: name: wontfix

## Environment info
     
- `transformers` version: 3.0.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic (on Google Colab)
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


### Who can help
 @LysandreJik 
 @jplu 

## Information
1. When using TFBertModel, I tried to output the hidden states. Firstly, I tried to set config.output_hidden_states=True, but it gave the "tuple index out of range" error. The code is:

from transformers import TFBertModel, BertConfig
import tensorflow as tf
def single_bert():    
    id = Input((128,), dtype=tf.int32)
    mask = Input((128,), dtype=tf.int32)
    atn = Input((128,), dtype=tf.int32)   
    bert_config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
    bert_model = TFBertModel.from_pretrained('bert-base-uncased', config = bert_config)        
    embedding = bert_model(id, attention_mask=mask, token_type_ids=atn)[2]
    model = tf.keras.Model(inputs=[id, mask, atn], outputs=embedding)   
    return model
model = single_bert()
model.summary()

2. I have also tried to pass "output_hidden_states=True", but it still gave "tuple index out of range" error:

from transformers import TFBertModel, BertConfig
import tensorflow as tf
def single_bert():    
    id = Input((128,), dtype=tf.int32)
    mask = Input((128,), dtype=tf.int32)
    atn = Input((128,), dtype=tf.int32)   
    bert_model = TFBertModel.from_pretrained('bert-base-uncased')     
    embedding = bert_model(id, attention_mask=mask, token_type_ids=atn, output_hidden_states=True)[2]
    model = tf.keras.Model(inputs=[id, mask, atn], outputs=embedding)
    return model
model = single_bert()
model.summary()

## To reproduce
Steps to reproduce the behavior:
1.
2.

## Expected behavior
I need to add some custom layers on top of the output hidden states and fine-tune the whole model, so firstly, I have to get the hidden states of Bert.


Comments:
Sorry for the format of the two codes. I have modified them and posted them here:

1. 
```python
from transformers import TFBertModel, BertConfig
import tensorflow as tf
def single_bert():
    id = Input((128,), dtype=tf.int32)
    mask = Input((128,), dtype=tf.int32)
    atn = Input((128,), dtype=tf.int32)
    bert_config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
    bert_model = TFBertModel.from_pretrained('bert-base-uncased', config = bert_config)
    embedding = bert_model(id, attention_mask=mask, token_type_ids=atn)[2]
    model = tf.keras.Model(inputs=[id, mask, atn], outputs=embedding)
    return model
model = single_bert()
model.summary()
```
2. 
```python
from transformers import TFBertModel, BertConfig
import tensorflow as tf
def single_bert():
    id = Input((128,), dtype=tf.int32)
    mask = Input((128,), dtype=tf.int32)
    atn = Input((128,), dtype=tf.int32)
    bert_model = TFBertModel.from_pretrained('bert-base-uncased')
    embedding = bert_model(id, attention_mask=mask, token_type_ids=atn, output_hidden_states=True)[2]
    model = tf.keras.Model(inputs=[id, mask, atn], outputs=embedding)
    return model
model = single_bert()
model.summary()
```

I have ran your code with minor edits:

embedding = bert_model(id, attention_mask=mask, token_type_ids=atn, output_hidden_states=True)

For the variable embedding, it only output 2 element in a tuple
(<tf.Tensor 'tf_bert_model/Identity:0' shape=(None, 128, 768) dtype=float32>,
 <tf.Tensor 'tf_bert_model/Identity_1:0' shape=(None, 768) dtype=float32>)

So I think you would want to extract the last embedding layer index -1 or just 1 instead of index 2 (non-existent). 
I have seen other people with index 2 (such as : [https://github.com/huggingface/transformers/issues/4048](url) ), but of course their tuple has length more than 2, you should investigate more in your code. I even tried to have your code structured like theirs 

` embedding = bert_model(input = [id, atn])`

But it gives out the same output, may be it's just because of the different pre-trained model itself that give out different tuple length, so try to investigate more 



Hello!

There are three possibilities to use `TFBertModel` either with a list, a dic or positional argumentst:
1) With list: you have to explicitely give a list of size 10 corresponding to `[input_ids, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, output_attentions, output_hidden_states, return_dict, training]`
2) With a dict. This is the recommended way, as you can specify only the keys you need.
3) With positional arguments (as proposed by @vuhluu)

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.