Is there any way that I can extract the hidden output from the self-attention layer?
Author: h56cho
Pull request: no
Labels: name: Help wanted

name: wontfix

name: PyTorch

name: Usage

Hello,

From my understanding, for the ```GPT2LMHeadModel```, the output ```past``` allows me to retrieve the key and value vectors that are used in the self-attention block (which is prior to the feedforward block). 

Is there any way I can extract the output of the self-attention block **at a particular head of a single layer** of ```GPT2LMHeadModel``` (if I am understanding this correctly, the output ```hidden_states``` only returns the output after the input had gone into the feedforward block... but what I want is to extract the output from the self-attention block, which happens before the feedforward block).

Thank you,

Comments:
That would be the attentions, which you can output by specifying `output_attentions=True` in your configuration object.

Hello,

Thank you very much for your reply.

What I want to obtain though, is not the individual attention weights themselves but rather the final product of the self-attention layer at each head (the transformed embeddings that the self-attention layer produces, before they go into the feedforward layer for final processing).

Is there any way that I can get this final product of the self-attention layer at each head?

Thank you,

You mean you want to obtain the result after the softmax multiplied by the value vector?

Hello,

I would like to obtain the result that is obtained after the sum of (value) * (softmax) got multiplied by the matrix H (i.e. the final output embedding of the self-attention layer from a single head)

Thank you,

Then that is literally the attentions I mentioned earlier, see in the [source code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_gpt2.py#L163-L164).

Hello,

so the output of a single attention-head, which is the thing I want to extract, can be formulated as the following:

O = AE(W^V)H

where

O = output of a single attention-head
A = matrix that stores attention weights for all tokens in sequence
E = matrix that stores the embeddings of all tokens in a sequence
W^V = matrix that we multiply with E to generate the value vector of all tokens of the sequence
H = projection matrix that is used to generate the final product of a single attention-head

If I am not mistaken, ```attention``` gives out the matrix A...
but what I am looking to get is the output O.....

Is there anyway that I can get the output O? or does ```attention``` give out the output O, like you described before?

Thank you and sorry for the long question, your help is much appreciated.



To be even more clear, I just want the output of each head within the layers of transformer. Is there any way that I can get the output of each individual head?

Thank you,

Hello, if I can't get the output of individual attention-head explicitly, is there any way that I can retrieve the matrix H, where H is from the formula below: 

O = AE(W^V)H

O = output of a single attention-head
A = matrix that stores attention weights for all tokens in sequence
E = matrix that stores the embeddings of all tokens in a sequence
W^V = matrix that we multiply with E to generate the value vector of all tokens of the sequence
H = projection matrix that is used to generate the final product of a single attention-head

Thank you,

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@h56cho 

Hello

I also want to know if I can get such hidden outputs
Do you have any progress with it?

Thank you in advance