[T5] Fix Cross Attention position bias
Author: ZhuBaohe
Pull request: yes

This PR fixes the Cross Attention position bias assignment in Class T5Stack.

Comments:
# [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=h1) Report
> Merging [#4499](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=desc) into [master](https://codecov.io/gh/huggingface/transformers/commit/a08652772791fdaeed6f263b1a99926ca64be5dc&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/huggingface/transformers/pull/4499/graphs/tree.svg?width=650&height=150&src=pr&token=9qOlN6Hb1c)](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master    #4499      +/-   ##
==========================================
- Coverage   77.83%   77.82%   -0.02%     
==========================================
  Files         123      123              
  Lines       20514    20514              
==========================================
- Hits        15968    15964       -4     
- Misses       4546     4550       +4     
```


| [Impacted Files](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/transformers/modeling\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/4499/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190NS5weQ==) | `83.53% <100.00%> (ø)` | |
| [src/transformers/modeling\_tf\_t5.py](https://codecov.io/gh/huggingface/transformers/pull/4499/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9tb2RlbGluZ190Zl90NS5weQ==) | `95.16% <100.00%> (ø)` | |
| [src/transformers/hf\_api.py](https://codecov.io/gh/huggingface/transformers/pull/4499/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9oZl9hcGkucHk=) | `93.06% <0.00%> (-4.96%)` | :arrow_down: |
| [src/transformers/file\_utils.py](https://codecov.io/gh/huggingface/transformers/pull/4499/diff?src=pr&el=tree#diff-c3JjL3RyYW5zZm9ybWVycy9maWxlX3V0aWxzLnB5) | `73.85% <0.00%> (+0.41%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=footer). Last update [a086527...e9775b2](https://codecov.io/gh/huggingface/transformers/pull/4499?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).


Hi @ZhuBaohe, 

Thansk for your PR! Can you explain a bit more in-detail what the fix is doing here? :-) 

@patrickvonplaten 

I fixes a bug that the variable **encoder_decoder_position_bias** was  incorrectly assigned by cross-attention weights, not by cross-attention position bias.

See Line 745 of the file modeling_t5.py as follow:
```
# layer_outputs = hidden-states,                   -> 0
                  key-value-states,                -> 1
                  (self-attention weights),        -> 2                               
                  (self-attention position bias),  -> 3  
                  (cross-attention weights),       -> 4 
                  (cross-attention position bias)  -> 5 
```
**encoder_decoder_position_bias** should be assigned by layer_outputs[5]  instead of layer_outputs[4] .

Great, I agree with you. Previously the attention weights of the cross attention layer were taken instead of the bias. 

@LysandreJik @thomwolf I am quite surprised that we did not see an error earlier. I checked the slow tests and the summarization / translation results are equivalent as before. 

So good to merge for me!

Surprising indeed @patrickvonplaten , I did fix a similar bug when implementing T5.

We should switch to NamedTuples one day 😄