TF T5-small with output hidden state and attention not owrking
Author: pathikchamaria
Pull request: no
Labels: name: wontfix

- `transformers` version: 2.11
- Platform: Multiple
- Python version: multiple

### Who can help
 T5: @patrickvonplaten
 tensorflow: @jplu
 
## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when t5-small is loaded from pretrain with output_hidden_states=True, output_attentions=True

sample script https://colab.research.google.com/drive/1oF8hMaQg1yl2fE6QPUYKSTZcer4Mlk6S?usp=sharing

if these parameter is removed script works 

I am getting following error. 

```python
/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache)
    780                 encoder_outputs=encoder_outputs,
    781                 attention_mask=attention_mask,
--> 782                 use_cache=use_cache,
    783             )
    784         else:

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py in _generate_beam_search(self, input_ids, cur_len, max_length, min_length, do_sample, early_stopping, temperature, top_k, top_p, repetition_penalty, no_repeat_ngram_size, bad_words_ids, bos_token_id, pad_token_id, decoder_start_token_id, eos_token_id, batch_size, num_return_sequences, length_penalty, num_beams, vocab_size, encoder_outputs, attention_mask, use_cache)
   1027                 input_ids, past=past, attention_mask=attention_mask, use_cache=use_cache
   1028             )
-> 1029             outputs = self(**model_inputs)  # (batch_size * num_beams, cur_len, vocab_size)
   1030             next_token_logits = outputs[0][:, -1, :]  # (batch_size * num_beams, vocab_size)
   1031 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    983 
    984         with ops.enable_auto_cast_variables(self._compute_dtype_object):
--> 985           outputs = call_fn(inputs, *args, **kwargs)
    986 
    987         if self._activity_regularizer:

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_t5.py in call(self, inputs, **kwargs)
   1061             encoder_attention_mask=attention_mask,
   1062             head_mask=head_mask,
-> 1063             use_cache=use_cache,
   1064         )
   1065 

/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py in __call__(self, *args, **kwargs)
    983 
    984         with ops.enable_auto_cast_variables(self._compute_dtype_object):
--> 985           outputs = call_fn(inputs, *args, **kwargs)
    986 
    987         if self._activity_regularizer:

/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_t5.py in call(self, inputs, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, past_key_value_states, use_cache, training)
    572             # required mask seq length can be calculated via length of past
    573             # key value states and seq_length = 1 for the last token
--> 574             mask_seq_length = shape_list(past_key_value_states[0][0])[2] + seq_length
    575         else:
    576             mask_seq_length = seq_length

IndexError: list index out of range
```

## Expected behavior

how to get the attention and hidden states as output?
Even if you can share a sample for pytorch I would be able to make it work for TF. 

Comments:
Hello!

Unfortunately this is a known bug we have with few of the TF models. We are currently reworking all the TF models to solve this issue among others.

@jplu I tried the same thing with Pytorch model also. It is also giving error.  Any idea if I can get the attentions with pytorch?

You get the same error with PyTorch? For PyTorch I will let @patrickvonplaten take the lead to help you, he knows better than me.

The error is the same instead of list it just says tuple

Hey @pathikchamaria - is it possible to update your version? 2.11 is very outdated by now. Could you try again with the current version of transformers (3.5) ? 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@jplu are there any updates on this on the tensorflow side?