[WIP] Add Fusion-in-Decoder
Author: bhavitvyamalik
Pull request: yes
Labels: name: WIP

# What does this PR do?

This PR adds the Fusion-in-Decoder model to the repository.

Paper: https://arxiv.org/abs/2007.01282
Code: https://github.com/facebookresearch/FiD

## Who can review?
Anyone in the community is free to review the PR once the tests have passed.
@patil-suraj, @patrickvonplaten, @qqaatw

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15902). All of your documentation changes will be reflected on that endpoint.

In their [model](https://github.com/facebookresearch/FiD/blob/main/src/model.py) code, they are using `EncoderWrapper` and `CheckpointWrapper` on top of `T5ForConditionalGeneration`. The model loads without adding them too but then it gives warnings like:

> Some weights of the model checkpoint at ../../../FiD/pretrained_models/nq_reader_base/ were not used when initializing T5ForConditionalGeneration: ['encoder.encoder.block.10.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.3.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.0.module.layer.0.layer_norm.weight', 'encoder.encoder.block.10.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.4.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.11.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.11.module.layer.0.layer_norm.weight', 'encoder.encoder.block.1.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.4.module.layer.1.layer_norm.weight', 'encoder.encoder.block.5.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.6.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.5.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.1.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.4.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.1.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.2.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.8.module.layer.1.layer_norm.weight', 'encoder.encoder.block.7.module.layer.0.layer_norm.weight', 'encoder.encoder.block.6.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.10.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.4.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.9.module.layer.1.layer_norm.weight', 'encoder.encoder.block.9.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.3.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.2.module.layer.0.layer_norm.weight', 'encoder.encoder.block.11.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.10.module.layer.0.layer_norm.weight', 'encoder.encoder.block.4.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.9.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.5.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.6.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.4.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.1.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.0.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.3.module.layer.0.layer_norm.weight', 'encoder.encoder.block.3.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.0.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.9.module.layer.0.layer_norm.weight', 'encoder.encoder.block.2.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.2.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.0.module.layer.1.layer_norm.weight', 'encoder.encoder.block.1.module.layer.0.layer_norm.weight', 'encoder.encoder.block.5.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.11.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.5.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.2.module.layer.1.layer_norm.weight', 'encoder.encoder.block.5.module.layer.0.layer_norm.weight', 'encoder.encoder.block.5.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.3.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.3.module.layer.1.layer_norm.weight', 'encoder.encoder.block.6.module.layer.0.layer_norm.weight', 'encoder.encoder.block.0.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.8.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.0.module.layer.0.SelfAttention.relative_attention_bias.weight', 'encoder.encoder.block.5.module.layer.1.layer_norm.weight', 'encoder.encoder.block.11.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.1.module.layer.1.layer_norm.weight', 'encoder.encoder.block.6.module.layer.1.layer_norm.weight', 'encoder.encoder.block.7.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.0.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.10.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.7.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.3.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.9.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.10.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.2.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.0.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.1.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.4.module.layer.0.layer_norm.weight', 'encoder.encoder.block.2.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.9.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.final_layer_norm.weight', 'encoder.encoder.block.6.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.6.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.10.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.11.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.7.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.7.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.10.module.layer.1.layer_norm.weight', 'encoder.encoder.block.9.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.3.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.11.module.layer.1.layer_norm.weight', 'encoder.encoder.block.8.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.4.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.6.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.7.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.7.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.7.module.layer.1.layer_norm.weight', 'encoder.encoder.block.0.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.8.module.layer.1.DenseReluDense.wi.weight', 'encoder.encoder.block.1.module.layer.0.SelfAttention.o.weight', 'encoder.encoder.block.9.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.8.module.layer.0.SelfAttention.v.weight', 'encoder.encoder.block.8.module.layer.0.SelfAttention.k.weight', 'encoder.encoder.block.8.module.layer.1.DenseReluDense.wo.weight', 'encoder.encoder.block.2.module.layer.0.SelfAttention.q.weight', 'encoder.encoder.block.8.module.layer.0.layer_norm.weight', 'encoder.encoder.embed_tokens.weight', 'encoder.encoder.block.11.module.layer.1.DenseReluDense.wi.weight']
This IS expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
This IS NOT expected if you are initializing T5ForConditionalGeneration from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

> Some weights of T5ForConditionalGeneration were not initialized from the model checkpoint at ../../../FiD/pretrained_models/nq_reader_base/ and are newly initialized: ['encoder.block.1.layer.0.SelfAttention.k.weight', 'encoder.block.11.layer.1.layer_norm.weight', 'encoder.block.11.layer.1.DenseReluDense.wo.weight', 'encoder.block.11.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.0.SelfAttention.v.weight', 'encoder.block.6.layer.1.DenseReluDense.wi.weight', 'encoder.block.2.layer.0.layer_norm.weight', 'encoder.block.5.layer.1.DenseReluDense.wo.weight', 'encoder.block.10.layer.0.SelfAttention.o.weight', 'encoder.block.11.layer.0.SelfAttention.q.weight', 'encoder.block.11.layer.0.layer_norm.weight', 'encoder.block.8.layer.0.layer_norm.weight', 'encoder.block.1.layer.0.SelfAttention.v.weight', 'encoder.final_layer_norm.weight', 'encoder.block.3.layer.1.layer_norm.weight', 'encoder.block.5.layer.0.layer_norm.weight', 'encoder.block.0.layer.1.DenseReluDense.wo.weight', 'encoder.block.2.layer.1.DenseReluDense.wi.weight', 'encoder.block.9.layer.1.layer_norm.weight', 'encoder.block.6.layer.1.layer_norm.weight', 'encoder.block.10.layer.1.DenseReluDense.wo.weight', 'encoder.block.2.layer.0.SelfAttention.o.weight', 'encoder.block.7.layer.1.DenseReluDense.wi.weight', 'encoder.block.5.layer.1.DenseReluDense.wi.weight', 'encoder.block.9.layer.0.SelfAttention.o.weight', 'encoder.block.10.layer.0.SelfAttention.v.weight', 'encoder.block.2.layer.0.SelfAttention.k.weight', 'encoder.block.7.layer.0.SelfAttention.q.weight', 'encoder.block.0.layer.1.layer_norm.weight', 'encoder.block.10.layer.0.SelfAttention.k.weight', 'encoder.block.7.layer.0.SelfAttention.v.weight', 'encoder.block.3.layer.0.SelfAttention.v.weight', 'encoder.block.4.layer.0.SelfAttention.k.weight', 'encoder.block.7.layer.0.SelfAttention.o.weight', 'encoder.block.1.layer.1.DenseReluDense.wi.weight', 'encoder.block.0.layer.1.DenseReluDense.wi.weight', 'encoder.block.0.layer.0.SelfAttention.v.weight', 'encoder.block.5.layer.1.layer_norm.weight', 'encoder.block.5.layer.0.SelfAttention.o.weight', 'encoder.block.11.layer.0.SelfAttention.k.weight', 'encoder.block.11.layer.0.SelfAttention.v.weight', 'encoder.block.9.layer.0.SelfAttention.q.weight', 'encoder.block.7.layer.1.DenseReluDense.wo.weight', 'encoder.block.5.layer.0.SelfAttention.q.weight', 'encoder.block.8.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.0.SelfAttention.o.weight', 'encoder.block.3.layer.0.SelfAttention.k.weight', 'encoder.block.9.layer.1.DenseReluDense.wo.weight', 'encoder.block.10.layer.0.layer_norm.weight', 'encoder.block.6.layer.1.DenseReluDense.wo.weight', 'encoder.block.8.layer.0.SelfAttention.v.weight', 'encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'encoder.block.7.layer.0.SelfAttention.k.weight', 'encoder.block.6.layer.0.SelfAttention.q.weight', 'encoder.block.7.layer.0.layer_norm.weight', 'encoder.block.5.layer.0.SelfAttention.k.weight', 'encoder.block.11.layer.0.SelfAttention.o.weight', 'encoder.block.4.layer.0.SelfAttention.q.weight', 'encoder.block.7.layer.1.layer_norm.weight', 'encoder.block.2.layer.1.layer_norm.weight', 'encoder.block.5.layer.0.SelfAttention.v.weight', 'encoder.block.3.layer.1.DenseReluDense.wi.weight', 'encoder.block.0.layer.0.layer_norm.weight', 'encoder.block.6.layer.0.SelfAttention.o.weight', 'encoder.block.3.layer.0.SelfAttention.o.weight', 'encoder.block.3.layer.0.layer_norm.weight', 'encoder.block.10.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.1.DenseReluDense.wo.weight', 'encoder.block.9.layer.0.SelfAttention.k.weight', 'encoder.block.2.layer.0.SelfAttention.v.weight', 'encoder.block.1.layer.0.layer_norm.weight', 'encoder.block.1.layer.0.SelfAttention.o.weight', 'encoder.block.2.layer.0.SelfAttention.q.weight', 'encoder.block.8.layer.1.DenseReluDense.wo.weight', 'encoder.block.2.layer.1.DenseReluDense.wo.weight', 'encoder.block.9.layer.1.DenseReluDense.wi.weight', 'encoder.block.6.layer.0.SelfAttention.v.weight', 'encoder.block.9.layer.0.layer_norm.weight', 'encoder.block.8.layer.0.SelfAttention.q.weight', 'encoder.block.1.layer.0.SelfAttention.q.weight', 'encoder.block.8.layer.0.SelfAttention.o.weight', 'encoder.block.10.layer.1.layer_norm.weight', 'encoder.block.0.layer.0.SelfAttention.o.weight', 'encoder.block.1.layer.1.layer_norm.weight', 'encoder.block.6.layer.0.layer_norm.weight', 'encoder.block.3.layer.1.DenseReluDense.wo.weight', 'encoder.block.8.layer.0.SelfAttention.k.weight', 'encoder.block.6.layer.0.SelfAttention.k.weight', 'encoder.block.1.layer.1.DenseReluDense.wo.weight', 'encoder.block.8.layer.1.layer_norm.weight', 'encoder.block.0.layer.0.SelfAttention.q.weight', 'encoder.block.9.layer.0.SelfAttention.v.weight', 'encoder.block.4.layer.0.layer_norm.weight', 'encoder.block.4.layer.1.DenseReluDense.wi.weight', 'encoder.block.4.layer.1.layer_norm.weight', 'encoder.block.10.layer.0.SelfAttention.q.weight', 'encoder.block.0.layer.0.SelfAttention.k.weight', 'encoder.block.3.layer.0.SelfAttention.q.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

Adding those modules on top of `T5ForConditionalGeneration` would be the right thing to do? If yes, then are there any existing examples I can look into to understand how to implement it?

Think we should rename the weigths here then -> @patil-suraj think you know best how to guide @bhavitvyamalik here