Add caching mechanism to BERT/RoBERTa/GPT2 for Seq2Seq accelerated generation
Author: patrickvonplaten
Pull request: no
Labels: name: Good Second Issue

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

All Seq2Seq models that make use of `generate()` usually allow `past_key_values` to be cached for both the cross-attention layer and the uni-directional decoder self-attention layer. For this feature request we should implement the feature for Bert2Bert, and, Roberta2Roberta.

We should implement this feature analogs to how it is implemented in Bart. This means that we should 

- 1) add the caching mechanism in the AttentionLayer as shown here for Bart: https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L234
- 2) pass the `past_key_values` as tuple through the layers, making sure that it's optional for the cross-attention layer: https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L433
- 3) Adapt the mask correspondingly. The easiest option is probably to just copy how it's done in Bart and remove the old attention_masking logic (making sure that all tests pass): https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L91 and https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/bart/modeling_bart.py#L76
- 4) Add a test for `BertLMHeadModel` and `RobertaForCausalLM` that verifies that the caching mechanism works as expected:
https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/tests/test_modeling_bart.py#L287 
- 5) "Turn on" caching for Encoder-Decoder (this should be the last step and this might cause some other problems - happy to help here!): https://github.com/huggingface/transformers/blob/b01ddc9577b87f057e163d49563ee3f74f4810cf/src/transformers/models/encoder_decoder/modeling_encoder_decoder.py#L427

This might be a good issue for you @patil-suraj  if interested :-) 

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
@patrickvonplaten Hi, I'd like to ask why the decoding only reuses previous key and values but no query. Since if the model parameter rests static, the query vector can be reused as well.

Appreciate for your reply.

Hi @liyucheng09 

Good question!
We don't need to reuse query states because when caching is enabled we just need the query states for the current last token since only the last query vector is needed to predict the next token.

Hope this makes it clear.

These blogs might actually help as well: 
- https://huggingface.co/blog/encoder-decoder
- https://jalammar.github.io/illustrated-gpt2/ 

to better understand the difference between query, key and value :-)