SwinTransformer as encoder and Bart as decoder
Author: RishabhMaheshwary
Pull request: no

## Environment info

- `transformers` version: 4.16.2
- Platform: Linux-5.4.0-81-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?:  yes
- Using distributed or parallel set-up in script?: yes

### Who can help
@NielsRogge

## Information
I wanted to use an encoder decoder model with ```SwinTransformer``` as an encoder and ```bart-large``` as a decoder.
I used ```VisionEncoderDecoderModel.from_encoder_decoder_pretrained("microsoft/swin-base-patch4-window12-384", "facebook/bart-large")``` command and it results in the following. 

```
  File "train_bart.py", line 158, in <module>
    model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained("microsoft/swin-base-patch4-window12-384", "facebook/bart-large")
  File "/private/home/rbh/anaconda3/lib/python3.8/site-packages/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py", line 399, in from_encoder_decoder_pretrained
    return cls(encoder=encoder, decoder=decoder, config=config)
  File "/private/home/rbh/anaconda3/lib/python3.8/site-packages/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py", line 213, in __init__
    self.encoder.config.hidden_size != self.decoder.config.hidden_size
  File "/private/home/rbh/anaconda3/lib/python3.8/site-packages/transformers/configuration_utils.py", line 250, in __getattribute__
    return super().__getattribute__(key)
AttributeError: 'SwinConfig' object has no attribute 'hidden_size'
```
Can the SwinTransformer with a Bart decoder be initialized using ```VisionEncoderDecoderModel``` or do I need to write my own model with SwinTransformer as enocder and Bart as decoder ?

Comments:
Hi @NielsRogge I did the changes mentioned in the PR above. But then I get the following error.
```
Traceback (most recent call last):
  File "train_donut.py", line 260, in <module>
    trainer.train()
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 1398, in train
    tr_loss_step = self.training_step(model, inputs)
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 1980, in training_step
    loss = self.compute_loss(model, inputs)
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 2012, in compute_loss
    outputs = model(**inputs)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/private/home/rbh/trocr/transformers/src/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py", line 478, in forward
    encoder_hidden_states = self.enc_to_dec_proj(encoder_hidden_states)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 103, in forward
    return F.linear(input, self.weight, self.bias)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/functional.py", line 1848, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (576x1024 and 128x1024)
```
Do we need to add another projection layer?

The way `VisionEncoderDecoderModel` works is as follows:

First, the image (`pixel_values`) is sent through the encoder to obtain some `last_hidden_state`. These are the features at the last layer of the model. In case of `microsoft/swin-base-patch4-window12-384`, this is a tensor of shape (1, 144, 1024) in case of a single image, as can be seen as follows:

```
import torch
from transformers import SwinModel

model = SwinModel.from_pretrained("microsoft/swin-base-patch4-window12-384")

pixel_values = torch.randn(1,3,384,384)

outputs = model(pixel_values)
print(outputs.last_hidden_state.shape)
```

Next, the decoder will use these hidden states to perform cross-attention. If you're using `facebook/bart-large` to initialize the weights of the decoder, the decoder will have a hidden size (embedding dimension) of 1024, as can be seen below:

```
from transformers import BartConfig

config = BartConfig.from_pretrained("facebook/bart-large")
print(config.hidden_size)
```

Hence, no projection is needed as both have 1024 as hidden dimension (making cross-attention possible).

For now, I'd advise you to set the `hidden_size` attribute of the encoder to 1024, in which case no projection will be added.

```
from transformers import SwinModel, BartForCausalLM, VisionEncoderDecoderModel

encoder = SwinModel.from_pretrained("microsoft/swin-base-patch4-window12-384")
decoder = BartForCausalLM.from_pretrained("facebook/bart-large")

# Set encoder config hidden size (which will make sure no projection layer is added)
setattr(encoder.config, "hidden_size", 1024)

# Initializing a model with a pretrained Swin as encoder & a pretrained BART-large as decoder
model = VisionEncoderDecoderModel(encoder=encoder, decoder=decoder)
```

Looking into it, the PR above will not solve the issue appropriately. Swin Transformer is a bit special in the sense that the shapes of the hidden states are not identical after every layer: the channel dimension is increased by a factor of 2 after each stage.

I'll update my PR accordingly to set the `hidden_size` attribute. 

If you install transformers from my PR, this now works:

```
from transformers import VisionEncoderDecoderModel

# Initializing a model with a pretrained Swin as encoder & a pretrained BART-large as decoder
model = VisionEncoderDecoderModel.from_encoder_decoder_pretrained("microsoft/swin-base-patch4-window12-384",
                                                                  "facebook/bart-large")
```

@RishabhMaheshwary let me know how it goes, then I'll merge the PR!

Hi, I am getting this error now (during validation).
```
***** Running Evaluation *****
  Num examples = 3124
  Batch size = 8
Traceback (most recent call last):
  File "train_donut.py", line 261, in <module>
    trainer.train()
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 1473, in train
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 1598, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer_seq2seq.py", line 70, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 2260, in evaluate
    metric_key_prefix=metric_key_prefix,
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer.py", line 2427, in evaluation_loop
    loss, logits, labels = self.prediction_step(model, inputs, prediction_loss_only, ignore_keys=ignore_keys)
  File "/private/home/rbh/trocr/transformers/src/transformers/trainer_seq2seq.py", line 175, in prediction_step
    **gen_kwargs,
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/private/home/rbh/trocr/transformers/src/transformers/generation_utils.py", line 1095, in generate
    inputs_tensor, model_kwargs, model_input_name
  File "/private/home/rbh/trocr/transformers/src/transformers/generation_utils.py", line 508, in _prepare_encoder_decoder_kwargs_for_generation
    model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs)
  File "/private/home/rbh/anaconda3/envs/donut/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
TypeError: forward() got an unexpected keyword argument 'attention_mask'
```

Is this similar to https://github.com/huggingface/transformers/issues/13812


Can you provide a notebook to reproduce? Swin Transformer (as any other vision encoder) doesn't take `attention_mask` as input. 

Here is the code I am using.
https://colab.research.google.com/drive/1x0ARdnAk0XlchxktcRHI7s21FvaCbCWC?usp=sharing
It is basically the TrOCR demo.


Here https://github.com/huggingface/transformers/blob/fcb4f11c9232cee2adce8140a3a7689578ea97de/src/transformers/trainer_seq2seq.py#L172
It passes ```attention_mask``` as model params.
When I remove the ```attention_mask ``` I am able to run the above [script](https://colab.research.google.com/drive/1x0ARdnAk0XlchxktcRHI7s21FvaCbCWC?usp=sharing) without any error.

@NielsRogge Is it possible to do the same thing using Convnext as encoder ?

thank you !

Hi,

The `VisionEncoderDecoderModel` class doesn't support it at the moment, but it's possible technically.

As Convnext is not Transformer-based, one would need a custom strategy to turn the final feature map of ConvNext into a sequence of "tokens", which can be used for cross-attention with the language decoder. A typical strategy here would be to flatten the 2D feature map into a sequence of tokens, and use a linear projection layer to make sure the dimensions between the encoder and decoder match. This is used in ViT hybrid for instance, see [here](https://github.com/huggingface/transformers/blob/9a6c6ef97fa5df4b1fb8dbc9e8c10ee3a9ed7e2a/src/transformers/models/vit_hybrid/modeling_vit_hybrid.py#L200-L201).

For models like ViT, BEiT, DeiT and Swin Transformer, this is straightforward as they output a sequence of "tokens" by default.