BART model parameters should not contain biases for Q, K and V
Author: ayaka14732
Pull request: no

## Environment info

- `transformers` version: 4.16.2
- Platform: Linux-5.11.0-1018-gcp-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.10.2+cu102 (False)
- Tensorflow version (GPU?): 2.7.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.4.0 (cpu)
- Jax version: 0.3.0
- JaxLib version: 0.3.0
- Using GPU in script?: False
- Using distributed or parallel set-up in script?: False

### Who can help

@patrickvonplaten

## Information

Model I am using: BART

## To reproduce

Steps to reproduce the behavior:

```python
>>> from transformers import FlaxBartForConditionalGeneration
>>> import jax.numpy as np
>>> model = FlaxBartForConditionalGeneration.from_pretrained('facebook/bart-base', dtype=np.bfloat16)
>>> model.params['model']['decoder']['layers']['0']['encoder_attn']['q_proj']['bias']
DeviceArray([-6.86645508e-02,  3.71704102e-02,  3.40820312e-01,
             -3.87451172e-01,  1.04980469e-01, -9.60693359e-02,
             -2.45361328e-01, -1.50985718e-02,  2.50244141e-01,
             -2.20581055e-01,  9.33074951e-03,  1.28295898e-01,
              ...
>>> model.params['model']['decoder']['layers']['0']['encoder_attn']['k_proj']['bias']
DeviceArray([-4.09317017e-03,  3.31687927e-03,  5.46646118e-03,
             -9.59777832e-03,  4.63104248e-03, -4.14276123e-03,
             -6.32858276e-03, -3.68309021e-03,  2.80380249e-03,
             -2.85339355e-03,  1.19304657e-03,  1.88636780e-03,
              ...
>>> model.params['model']['decoder']['layers']['0']['encoder_attn']['v_proj']['bias']
DeviceArray([-6.41479492e-02, -5.52749634e-03, -4.86145020e-02,
              5.64575195e-03, -6.95800781e-02, -2.50053406e-03,
              6.31332397e-03, -1.13952637e-01,  8.81195068e-03,
              4.99877930e-02,  1.15814209e-02,  8.41617584e-04,
              ...
```

## Expected behavior

The parameters should not contain biases for Q, K and V. However, it turns out that there exists non-zero biases for Q, K and V.

Reason:

> BART (https://arxiv.org/pdf/1910.13461.pdf): BART uses the standard sequence-to-sequence Transformer architecture from (Vaswani et al., 2017), except, following GPT, that we modify ReLU activation functions to GeLUs (Hendrycks & Gimpel, 2016) and initialise parameters from N (0, 0.02).
>
> Transformer (https://arxiv.org/pdf/1706.03762.pdf): In practice, we compute the attention function on a set of queries simultaneously, packed together into a matrix Q. The keys and values are also packed together into matrices K and V.

This proves that Q, K and V are **matrices** instead of **linear layers**. Therefore, they should not contain biases.

Comments:
Hey @ayaka14732 !

Note that the original BART checkpoints use `bias` in Q, K, V layers.

> This proves that Q, K and V are matrices instead of linear layers. Therefore, they should not contain biases.

The weights of linear layers are also just matrices or for that matter, every layer weights in transformers are just matrices.

Hey @patil-suraj !

This should be very clear in the Transformer paper. There is a formula for scaled dot-product attention:

![](https://user-images.githubusercontent.com/68557794/155732934-b9aadaba-937f-4398-b16b-19df83c479d3.png)

And for multi-head attention:

![](https://user-images.githubusercontent.com/68557794/155733277-77e4b0a5-f095-426e-bbe6-41ffe5500547.png)

You can see the difference clearly if you compare them with the formula for feed-forward network:

![](https://user-images.githubusercontent.com/68557794/155733571-f55d2b6c-5ea8-4307-839b-5284131bccb6.png)

Besides, if you read any of the Transformer tutorials online, they will say Q, K and V are three **matrices**. And there is a nice illustration for that (https://jalammar.github.io/illustrated-transformer/):

![](https://user-images.githubusercontent.com/68557794/155734828-59dc8f2c-1649-4915-818c-e76f9aebfa32.png)



Hey @ayaka14732, if you load the original BART checkpoints https://github.com/pytorch/fairseq/blob/main/examples/bart/README.md#pre-trained-models you will see that Q, K, and V have bias weights attached to them.

Thanks for pointing out!

I can confirm that the original BART checkpoints have bias weights attached to Q, K, and V.

```python
>>> import torch
>>> bart = torch.hub.load('pytorch/fairseq', 'bart.large')
>>> bart.model.encoder.layers[0].self_attn.k_proj.bias
Parameter containing:
tensor([-1.8740e-03, -2.8801e-03, -1.6394e-03,  ..., -1.9321e-03,
        -5.0664e-06, -3.0565e-04], requires_grad=True)
```

It should be somewhat concerned if any researcher wants to compare BART with their models, since BART would be in an advantageous position due to the extra parameters.