export T5 model to onnx with past_key_values
Author: Ki6an
Pull request: no

## Environment info
- `transformers` version: 4.3.3
- torch version 1.7.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.7.0 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

Models:
-  t5: @patrickvonplaten, @patil-suraj

while exporting `T5 decoder` with `past_key_values`, I'm getting the following error,
````python

/usr/local/lib/python3.7/dist-packages/torch/onnx/utils.py:1109: UserWarning: Provided key encoder_hidden_states for dynamic axes is not a valid input/output name
  warnings.warn("Provided key {} for dynamic axes is not a valid input/output name".format(key))
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py:646: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if torch.isinf(hidden_states).any():
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py:684: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if torch.isinf(hidden_states).any():
/usr/local/lib/python3.7/dist-packages/transformers/modeling_utils.py:244: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if causal_mask.shape[1] < attention_mask.shape[1]:
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py:494: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  position_bias = position_bias[:, :, -seq_length:, :]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-7-baab8a36e37d> in <module>()
----> 1 generate_onnx_representation(model_to_be_converted, 't5')

25 frames
/usr/local/lib/python3.7/dist-packages/transformers/models/t5/modeling_t5.py in forward(self, hidden_states, mask, key_value_states, position_bias, past_key_value, layer_head_mask, query_length, use_cache, output_attentions)
    497                 position_bias = position_bias + mask  # (batch_size, n_heads, seq_length, key_length)
    498 
--> 499         scores += position_bias
    500         attn_weights = F.softmax(scores.float(), dim=-1).type_as(
    501             scores

RuntimeError: output with shape [5, 8, 1, 2] doesn't match the broadcast shape [5, 8, 2, 2]
```

Comments:
@LysandreJik @mfuntowicz - how do we deal with ONNX issues currently? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.