How to get cross attention weights of decoder when using 'encoderdecodermodel'
Author: kimmo1019
Pull request: no
Labels: name: wontfix

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag: 
     https://stackoverflow.com/questions/tagged/huggingface-transformers 
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
I'm recently building a encoder-decoder model (Bert2Bert) using `encoderdecodermodel`.
But I found that it is really hard to get cross attention weights of the decoder.
The document of this API said the return of the `forward` function will be the following


- loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) – Languaged modeling loss.

- logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) – Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).

- past_key_values (List[torch.FloatTensor], optional, returned when use_cache=True is passed or when config.use_cache=True) – List of torch.FloatTensor of length config.n_layers, with each tensor of shape (2, batch_size, num_heads, sequence_length, embed_size_per_head)).

- Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be used (see past_key_values input) to speed up sequential decoding.

- decoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) – Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

- Hidden-states of the decoder at the output of each layer plus the initial embedding outputs.

- **decoder_attentions** (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) – Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

- Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the self-attention heads.

- encoder_last_hidden_state (torch.FloatTensor of shape (batch_size, sequence_length, hidden_size), optional) – Sequence of hidden-states at the output of the last layer of the encoder of the model.

- encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) – Tuple of torch.FloatTensor (one for the output of the embeddings + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

- Hidden-states of the encoder at the output of each layer plus the initial embedding outputs.

- encoder_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) – Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, sequence_length).

- Attentions weights of the encoder, after the attention softmax, used to compute the weighted average in the self-attention heads.

The `decoder_attentions` in the above only can return the self-attention weight `(batch_size, num_heads, sequence_length, sequence_length)`. However I want the cross attention weight  `(batch_size, num_heads, input_sequence_length, target_sequence_length)`.  I checked the source code of `encoderdecodermodel` and found the decoder is instantiated from the `AutoModelForCausalLM ` class. I also checked the source code of `AutoModelForCausalLM` and found no way to get cross-attention weight.

Is there any method I can get this cross-attention weight? 

@patrickvonplaten Thanks very much! Would be really appreciated if you can help me. It bothers me for a while.

Comments:
There isn't a method yet, you need to modify the source code.
Here is how I did it for for bart https://github.com/huggingface/transformers/pull/6967

Hey @kimmo1019, yeah we should probably add all those cross-attention weights to the output as well. Might be a good idea to do this for all Seq2Seq models at once in one PR.

> Hey @kimmo1019, yeah we should probably add all those cross-attention weights to the output as well. Might be a good idea to do this for all Seq2Seq models at once in one PR.

That would be great it this can be added!! Thanks in advance.

Looking forward to this feature, too. How is it going now?

It's on a ToDo-List!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.