New config param for cross-attention dimensionality
Author: Squire-tomsk
Pull request: no
Labels: name: wontfix

# 🚀 Feature request

Add a new config param `n_cross` for each model that has cross-attention. This param will determine key and value matrices dimensionality in cross-attention.

## Motivation

I have pretrained encoder (hidden size 512) and want to combine it with GPT-2 medium (hidden size 1024) by cross-attention. Now I cant do it, because in according with this PR https://github.com/huggingface/transformers/commit/1d6e71e1167dea9e026391ec5a1a2d7ec33d22af query and key matrices of cross-attention have dimensionality same as self-attention.

In code it could looks like:
```
config.is_cross_attention = True
config.n_cross = 512
```
And in doc string:
```
n_inner (:obj:`int`, optional, defaults to None):
                Dimensionality of the inner feed-forward layers. :obj:`None` will set it to 4 times n_embd.
n_cross (:obj:`int`, optional, defaults to None):
                Dimensionality of the cross-attention input. :obj:`None` will set it same as n_embd.
```

## Your contribution

Actually I have already fixed this in my fork, but I didn't make any tests and docs. But I would be able to prepare PR within next two weeks.


Comments:
is this different than d_model/hidden_size?

Yes it is. As I undestand from T5 and TransformerXL config this param named `n_embd` in GPT2 config (embedding and hidden size dimensionality). It has impact on selt-attention dimentionality:
Q (n_embd x n_embd) 
K (n_embd x n_embd) 
V (n_embd x n_embd). 

For now cross-attention dimensionality same, but I want:
Q (n_embd x n_embd)
K (n_cross x n_embd) 
V (n_cross x n_embd) 
in cross-attention.

Hey @Squire-tomsk, thanks for your issue. This can easily be resolved by introducing a new config parameter for each model that can be used as a decoder in the `EncoderDecoder` Framework, so probably in `configuration_utils.py`. By default it can be set to `d_model` or `hidden_size`, if not further specifiied.

IMO, adding another config param for this case is OK. What do you think? @sshleifer, @sgugger, @LysandreJik ? 

As long as the parameter is set to the right default and doesn't break backward compatibility, I have no objection.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


Hi,

This feature has recently been implemented in #13874 . We called the attribute `cross_attention_hidden_size`.

In short, there are 2 options:
- either you set it to `None`, in which case the `encoder_hidden_states` will be projected using a single linear layer, to match the hidden size of the decoder (in case they don't match).
- either you set it to the size of the encoder, in which case the decoder will project the `encoder_hidden_states` to the same dimension as the decoder when creating `keys` and `values` in each cross-attention layer. This is the case for the recently added TrOCR model.