Clarification regarding past/layer_past in GPT-2
Author: zphang
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

I'm hoping to get some clarification regarding how past/layer_past are meant to work in GPT-2. My prior impression was that the query/key/value at every layer (other than the first) should be influenced by all tokens the model is able to see. As such, it shouldn't make much sense to use pre-computed key/value activations from prior steps. Could you clarify this issue?

(Hope my question makes sense.)

Comments:
the `past` variable in GTP-2 stores all previously computed key and value vectors. Because GPT-2 uses masked self-attention only the query vectors of previous tokens are updated at every step, but not the key and value vectors. Therefore the `past` variable can be used to speed up decoding. 

To better understand how GPT-2 works, I highly recommend reading [the Illustrated GPT-2](http://jalammar.github.io/illustrated-gpt2) especially the part: 

**The Illustrated Masked Self-Attention**

Thanks for the response! 

I'm still not sure this is intuitive for me. The linked article mentions: "Now in the next iteration, when the model processes the word robot, it does not need to generate query, key, and value queries for the a token. It just reuses the ones it saved from the first iteration:", which seems to imply Q, K and V are reused, whereas it seems we're only (optionally) reusing K and V. 

On the other hand, I'm not seeing where in the code that masked self-attention only affected the query vectors. It seems to be that attention masking is applied to the scoring vectors at each layer, and that should affect the generated Q/K/V for all downstream layers.

It feels like there may be some key part of the intuition I'm missing, so thanks for the help.

I read the article @patrickvonplaten pointed. I am still very confused about the usage of attention when there are "past" vectors. The dimension just doesn't match.

19 is the batch size here; I am using gpt-2 small. 40 is the encoding seq len; 23 is the decoding seq len.
Input document past[0] size: torch.Size([2, 19, 12, 40, 64])
to-decode-sequence input embedding: torch.Size([19, 23, 768])
mask of the input document: torch.Size([19, 40])
mask of the to-decode-sequence: torch.Size([19, 23])
concat of two masks: torch.Size([19, 63])    // 63=23+40

concat doesn't work          // _decoder_output = self.transformer(past=presents,attention_mask=concat_attention_mask,   inputs_embeds=gt_input_embedding) fails

to-decode-mask doesn't work      // _decoder_output = self.transformer(past=presents,attention_mask=partial_oracle_trajectory_mask,inputs_embeds=gt_input_embedding ) fails


Error message for concat:
`    attention_mask = attention_mask.view(-1, input_shape[-1])
RuntimeError: shape '[-1, 23]' is invalid for input of size 1197`

Error message for to-decode-mask only
`RuntimeError: The size of tensor a (63) must match the size of tensor b (23) at non-singleton dimension 3
`

@zphang any idea? not sure if this correlates with what you said tho. 

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi @jiacheng-xu - sorry for answering that late. I think the problem with the attention mask was recently fixed in PR #3033. The issue was also mentioned in #3031 I think. Let me know if you still have problems when using the current version of master.