Vision Encoder-Decoder fails with LLaMA decoder due to missing cross-attention implementation
Author: amazingvince
Pull request: no
Labels: name: Core: Encoder-Decoder

name: bug

name: Vision

### System Info

- `transformers` version: 4.46.2
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.5.0+cu121 (True)
- Tensorflow version (GPU?): 2.17.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.5 (gpu)
- Jax version: 0.4.33
- JaxLib version: 0.4.33
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA L4



### Who can help?

Not sure for multi modal models:
text models: @ArthurZucker
vision models: @amyeroberts, @qubvel
generate: @zucchini-nlp 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

#### Description
When using a vision encoder-decoder model, there's an incompatibility when using LLaMA as the decoder. While both GPT-2 and LLaMA are decoder models, GPT-2 implements an output class that includes cross-attention outputs, whereas LLaMA's output class (`CausalLMOutputWithPast`) does not include this attribute. This causes the vision encoder-decoder forward pass to fail when attempting to access cross-attention outputs.

#### Current Behavior
The model raises an AttributeError because LLaMA's implementation of `CausalLMOutputWithPast` doesn't include cross-attention outputs, while the vision encoder-decoder expects this attribute to be present (as it exists in GPT-2's implementation).

Error message:
```python
AttributeError: 'CausalLMOutputWithPast' object has no attribute 'cross_attentions'
```

#### Technical Analysis
1. GPT-2's decoder implementation returns an output class that includes cross-attention information
2. LLaMA's decoder implementation returns `CausalLMOutputWithPast` which doesn't include cross-attention
3. The vision encoder-decoder architecture assumes the presence of cross-attention in the decoder outputs


#### Steps to Reproduce
1. Initialize a vision encoder-decoder model with LLaMA as the decoder
2. Attempt to run a forward pass or generate
3. The error occurs in `modeling_vision_encoder_decoder.py` when trying to access `decoder_outputs.cross_attentions`

The error occurs in modeling_vision_encoder_decoder.py around line 651:
```python
decoder_hidden_states=decoder_outputs.hidden_states,
decoder_attentions=decoder_outputs.attentions,
cross_attentions=decoder_outputs.cross_attentions,  # This line causes the error
encoder_last_hidden_state=encoder_outputs.last_hidden_state,
encoder_hidden_states=encoder_outputs.hidden_states,
```

#### Workaround
Setting cross_attentions to None allows the model to work, suggesting that the architecture doesn't strictly require this information for functioning.

#### Proposed Solutions
1. Short term: Modify the vision encoder-decoder implementation to handle decoders that don't provide cross-attention outputs:
```python
cross_attentions = getattr(decoder_outputs, 'cross_attentions', None)
```

Happy to submit a PR if this is an appropriate solution 

### Expected behavior

modeling_vision_encoder_decoder.py should support different decoder models without custom causal lm cross attention output classes.

Comments:
@amazingvince hey! Seems like the vision-encoder-decoder model is tuned to work only with LM decoder that support cross attention. Otherwise the pixel value hidden states will not be used in any way, as they are passed as `encoder-hidden-states` for cross attention. So instead of making the output `cross_attn` optional we can make the model docs more explicit on what LM backbones to use 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

  I am encountering th same issue on `transformers==4.52.4` and `torch==2.7.0` in the `VisionEncoderDecoderModel` class. The docs state that the module works with decoder only models and cross attention layers are inserted and randomly initialized if already not present and `gpt-2` is given as an example. But in modern decoder-only models, the  `CausalLMOutputWithPast` class does not have the attribute `cross_attentions`. I have this with Gemma3, Llama-3.2, etc. to no avail. How can this issue be addressed? 

@LysandreJik @zucchini-nlp  Can the docs be made clearer to reflect exactly what kind of language models can be used in this framework? And if Causal LM decoders (with no cross attention layers) will not work with this, then will it be possible at all to pair vision encoders with these decoder models through some other transformers class or API (perhaps `AutoModelForImageTextToText`)?

@SwapnanilMukherjee I agree that the docs could be clearer about that, we need to add a small tip on how to choose the backbone language model correctly

As per the second question, if you are planning to train a vaniall vision LLM, I suggest to use a [`LLaVa` model](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llava/modeling_llava.py) initialized with the necessary LM and ViT configs. The model won't do cross attention (because the LLM can't do cross-attn) but rather will append ViT embeddings with text embeddings

The vaniall architecture works with almost all backbones afaik, but lmk if there are any edge cases. Though it might not be the optimal choice when the compute is limited or higher-res images are used. In that case you can experiment and add layers on top (e.g avg pooling)

Would you like to submit a PR to make the documentation clear? :)

Sure, I will submit a PR to update the documentation. Just to be sure that I understand correctly, whatever LM model is chosen, it must output an instance of `--OutputWithCrossAttentions`. That can be BaseModelOuptut or CausalLMOutput or whatever, shouldn't matter, as long as it returns the cross attentions as a part of the output. And the documentation for choosing the LM must state it explicitly that LM family must be chosen keeping this requirement in mind. 

Regarding my own issue, are you suggesting that I use the Llava model class as the general architecture and load the pre-trained weights of any custom vision encoder and language decoder? But wouldn't that raise conflicts between architectures? For example, initializing a Llava Model with Gemma 3 weights will certainly not work, I think. Alternatively, if you're suggesting that I use pre-trained Llava models specifically for my task, that won't work because I am doing an exploratory study trying to research some specific properties of different vision and language model families. I can train the system with different vision encoders and language decoders end-to-end with my custom code, but the only reason I wanted to use the VisionEncoderDecoderClass is because it provides the `generate()` method built-in, and I wouldn't have to write my own decoding function which is a rather complex piece of work. 

> it must output an instance of --OutputWithCrossAttentions

Yeah, it has to be a generative model with cross attention support, and thus output a CausalLMOutput. I think there aren't many such models, and anything old-style pre-llama would work. As a general tip

>  I use the Llava model class as the general architecture and load the pre-trained weights of any custom vision encoder and language decoder

No, I meant using llava and initializing a dummy weight model, for further pre-training on your own data. It won't work with any pre-trained weights. The idea is similar to vision-encoder-decoder where we also have no ready checkpoints and the users need to train their own models. Vanilla llava arch allows to easily stack VIT+LM and get a working model with `generate()`  method

So smth like the below pseudo code:

```python
qwen3_config = Qwen3Config(**my_config_kwargs)
vit_config = VitConfig(**my_config_kwargs)
config = LlavaConfig(text_config=qwen3_config, vision_config=vit_config)
model = LlavaForConfditionalGeneration(config)
processor = LlavaProcessor(tokenizer=Qwen3Tokenizer, image_processor=VitImageProcessor, **kwargs)

model.eval() 
model.generate()

```

Yeah, that is true. Considering that there aren't many such decoder models that would satisfy this requirement, while the documentation can of course be updated to reflect this specificity, wouldn't it be a better move to update the implementation of this `VisionEncoderDecoderClass` to support modern decoder-only LMs without cross attention as well? Perhaps we can do something as simple as concatenating the image and text embedidngs and then down-projection (to not blow up the effective dimensionality) and the projection dimension/layer can be user-defined? This would allow for far greater utility of the class compared to its current state. I'll be happy to discuss the necessary changes in detail and submit a PR with the implementation.

Thanks for the code example for Llava. It makes sense now. However, it won't work for our use-case because we want to evaluate pre-trained models for our study. I guess we'll just implement a decoding function at this rate.

> Perhaps we can do something as simple as concatenating the image and text embedidngs

I think the current vanilla llava does that already, and if users want to use something more elaborate with pooling or video support, there are other architectures available. We try to support any type of LLM/VIT backbone when a new vision LLM is added so that it can be plugged-in for experiments.

Also, we usually add new classes when there are pre-trained checkpoints available. Creating building blocks for users to experiment is a bit divergent from transformers philosophy, I prefer to just document the "usage tip" in vision-encoder-decoder for now

 

> we want to evaluate pre-trained models for our study

Ah, I wonder then why we can't use `AutoModelForImageTextToText(pretrained_model_id)` to test all models? Or maybe you meant that not all models are transformers-supported?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.