relativeattentionbias.weight in block 0 EncDecAttention of T5 Model not in original tf model. Where do we get it from?
Author: swapnull7
Pull request: no
Labels: name: wontfix

## ❓ Questions & Help

Hi, I was comparing the weights in original tf model and the pytorch t5 model and it looks like there is an extra embedding in the EncDecAttention layer (layer_1) in block_0 (relative_attention_bias.weight). I could find and compare the other embedding weights in the model but not this particular one. Was this new parameter randomly initialized using the initializer and stored as is for pre-trained model or was it fine-tuned somehow and stored?


Comments:
It should also be in the TF version, this is the shared relative attention bias (shared among layers).

Do you want to give more details on how you compared both lists of weights and what make you think it's missing?

Sure. By the way, when we say the TF version, I mean the weights released by Google. So for the TF weights, here's what I do for `T5-Small`:

TF:
``` 
import tensorflow as tf
import pprint  # to prettify prints 
var_list = tf.train.list_variables("/path/to/stored/T5/weights")  # basically replicated directory from google cloud
pprint.pprint(var_list)
```
Pytorch:
```
import transformers
import pprint
from transformers import T5Model
model = T5Model.from_pretrained('t5-small')
pytorch_var_list = [x[0] for x in model.named_parameters()]  # get names cause we only use them
pprint.pprint(pytorch_var_list)
```
TF output for the `small` version looks something like: 
```
[('decoder/block_000/layer_000/SelfAttention/k', [512, 512]),
 ('decoder/block_000/layer_000/SelfAttention/k_slot_vc', [512]),
 ('decoder/block_000/layer_000/SelfAttention/k_slot_vr', [512]),
 ('decoder/block_000/layer_000/SelfAttention/o', [512, 512]),
 ('decoder/block_000/layer_000/SelfAttention/o_slot_vc', [512]),
 ('decoder/block_000/layer_000/SelfAttention/o_slot_vr', [512]),
 ('decoder/block_000/layer_000/SelfAttention/q', [512, 512]),
 ('decoder/block_000/layer_000/SelfAttention/q_slot_vc', [512]),
 ('decoder/block_000/layer_000/SelfAttention/q_slot_vr', [512]),
 ('decoder/block_000/layer_000/SelfAttention/relative_attention_bias', [8, 32]), 
 ('decoder/block_000/layer_000/SelfAttention/relative_attention_bias_slot_v',
  [8, 32]),
 ('decoder/block_000/layer_000/SelfAttention/v', [512, 512]),
 ('decoder/block_000/layer_000/SelfAttention/v_slot_vc', [512]),
 ('decoder/block_000/layer_000/SelfAttention/v_slot_vr', [512]),
 ('decoder/block_000/layer_000/layer_norm/scale', [512]),
 ('decoder/block_000/layer_000/layer_norm/scale_slot_v', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/k', [512, 512]),
 ('decoder/block_000/layer_001/EncDecAttention/k_slot_vc', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/k_slot_vr', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/o', [512, 512]),
 ('decoder/block_000/layer_001/EncDecAttention/o_slot_vc', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/o_slot_vr', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/q', [512, 512]),
 ('decoder/block_000/layer_001/EncDecAttention/q_slot_vc', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/q_slot_vr', [512]), 
 # --------------------------------- Note: No relative_attention_bias in layer_001
 ('decoder/block_000/layer_001/EncDecAttention/v', [512, 512]),
 ('decoder/block_000/layer_001/EncDecAttention/v_slot_vc', [512]),
 ('decoder/block_000/layer_001/EncDecAttention/v_slot_vr', [512]),
 ('decoder/block_000/layer_001/layer_norm/scale', [512]),
 ('decoder/block_000/layer_001/layer_norm/scale_slot_v', [512]),
 ('decoder/block_000/layer_002/DenseReluDense/wi/kernel', [512, 2048]),
 ('decoder/block_000/layer_002/DenseReluDense/wi/kernel_slot_vc', [2048]),
 ('decoder/block_000/layer_002/DenseReluDense/wi/kernel_slot_vr', [512]),
 ('decoder/block_000/layer_002/DenseReluDense/wo/kernel', [2048, 512]),
 ('decoder/block_000/layer_002/DenseReluDense/wo/kernel_slot_vc', [2048]),
 ('decoder/block_000/layer_002/DenseReluDense/wo/kernel_slot_vr', [512]),
 ('decoder/block_000/layer_002/layer_norm/scale', [512]),
 ('decoder/block_000/layer_002/layer_norm/scale_slot_v', [512]),
 ('decoder/block_001/layer_000/SelfAttention/k', [512, 512]),
 ...
 ...    # Similar weights for all the other decoder blocks  
 ...
 ('decoder/block_005/layer_002/layer_norm/scale_slot_v', [512]),
 ('decoder/final_layer_norm/scale', [512]),
 ('decoder/final_layer_norm/scale_slot_v', [512]),
 ('encoder/block_000/layer_000/SelfAttention/k', [512, 512]),
 ('encoder/block_000/layer_000/SelfAttention/k_slot_vc', [512]),
 ('encoder/block_000/layer_000/SelfAttention/k_slot_vr', [512]),
 ('encoder/block_000/layer_000/SelfAttention/o', [512, 512]),
 ('encoder/block_000/layer_000/SelfAttention/o_slot_vc', [512]),
 ('encoder/block_000/layer_000/SelfAttention/o_slot_vr', [512]),
 ('encoder/block_000/layer_000/SelfAttention/q', [512, 512]),
 ('encoder/block_000/layer_000/SelfAttention/q_slot_vc', [512]),
 ('encoder/block_000/layer_000/SelfAttention/q_slot_vr', [512]),
 ('encoder/block_000/layer_000/SelfAttention/relative_attention_bias', [8, 32]),
 ('encoder/block_000/layer_000/SelfAttention/relative_attention_bias_slot_v',
  [8, 32]),
 ('encoder/block_000/layer_000/SelfAttention/v', [512, 512]),
 ('encoder/block_000/layer_000/SelfAttention/v_slot_vc', [512]),
 ('encoder/block_000/layer_000/SelfAttention/v_slot_vr', [512]),
 ('encoder/block_000/layer_000/layer_norm/scale', [512]),
 ('encoder/block_000/layer_000/layer_norm/scale_slot_v', [512]),
 ('encoder/block_000/layer_001/DenseReluDense/wi/kernel', [512, 2048]),
 ('encoder/block_000/layer_001/DenseReluDense/wi/kernel_slot_vc', [2048]),
 ('encoder/block_000/layer_001/DenseReluDense/wi/kernel_slot_vr', [512]),
 ('encoder/block_000/layer_001/DenseReluDense/wo/kernel', [2048, 512]),
 ('encoder/block_000/layer_001/DenseReluDense/wo/kernel_slot_vc', [2048]),
 ('encoder/block_000/layer_001/DenseReluDense/wo/kernel_slot_vr', [512]),
 ('encoder/block_000/layer_001/layer_norm/scale', [512]),
 ('encoder/block_000/layer_001/layer_norm/scale_slot_v', [512]),
 ...
 ...    # Similar weights for all the other encoder blocks  
 ...
 ('encoder/block_005/layer_001/layer_norm/scale_slot_v', [512]),
 ('encoder/final_layer_norm/scale', [512]),
 ('encoder/final_layer_norm/scale_slot_v', [512]),
 ('global_step', []),
 ('shared/embedding', [32128, 512]),
 ('shared/embedding_slot_vc', [32128]),
 ('shared/embedding_slot_vr', [512])]```
```
Pytorch output:
```
['shared.weight',
 'encoder.block.0.layer.0.SelfAttention.q.weight',
 'encoder.block.0.layer.0.SelfAttention.k.weight',
 'encoder.block.0.layer.0.SelfAttention.v.weight',
 'encoder.block.0.layer.0.SelfAttention.o.weight',
 'encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight',
 'encoder.block.0.layer.0.layer_norm.weight',
 'encoder.block.0.layer.1.DenseReluDense.wi.weight',
 'encoder.block.0.layer.1.DenseReluDense.wo.weight',
 'encoder.block.0.layer.1.layer_norm.weight',
 'encoder.block.1.layer.0.SelfAttention.q.weight',
 ...
 ...    # Similar weights for all the other encoder blocks  
 ...
 'encoder.block.5.layer.1.layer_norm.weight',
 'encoder.final_layer_norm.weight',
 'decoder.block.0.layer.0.SelfAttention.q.weight',
 'decoder.block.0.layer.0.SelfAttention.k.weight',
 'decoder.block.0.layer.0.SelfAttention.v.weight',
 'decoder.block.0.layer.0.SelfAttention.o.weight',
 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight',
 'decoder.block.0.layer.0.layer_norm.weight',
 'decoder.block.0.layer.1.EncDecAttention.q.weight',
 'decoder.block.0.layer.1.EncDecAttention.k.weight',
 'decoder.block.0.layer.1.EncDecAttention.v.weight',
 'decoder.block.0.layer.1.EncDecAttention.o.weight',
 'decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight',  -----> Where does this guy come from? am I missing something in the original weights?
 'decoder.block.0.layer.1.layer_norm.weight',
 'decoder.block.0.layer.2.DenseReluDense.wi.weight',
 'decoder.block.0.layer.2.DenseReluDense.wo.weight',
 'decoder.block.0.layer.2.layer_norm.weight',
 ...
 ...    # Similar weights for all the other encoder blocks  
 ...
 'decoder.block.5.layer.2.layer_norm.weight',
 'decoder.final_layer_norm.weight']
```

Sorry, now that I think about it, I should have provided this information in the original post itself. So I was wondering where does the weight `decoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight` come from cause I don't seem find it in the original tf weights file or am I missing something?

@thomwolf sorry for the push. Any update on this?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


@swapnull7  how did you solve this issue?

Hey @swapnull7 , it seems that that it was a mistake and T5 isn't supposed to have relative attention bias between encoder and decoder. 
It has been removed in the new version of transformers.
I don't know where the pretrained weights for it came from 🤔 

https://github.com/huggingface/transformers/issues/8933#issuecomment-739251827