Possibly a bug in Pix2Struct outputs
Author: artyomxyz
Pull request: no

I'm sorry if I'm wrong, as I don't have much experience with transformers internals.
I was playing with Pix2Struct and trying to visualise attention on input image. `output.cross_attentions` shape didn't make much sense as it didn't have `patch_count` as any of dimensions. After inspecting `modeling_pix2struct.py` I have notices the following
```
            # layer_outputs = hidden-states, key-value-states (self-attention position bias), (self-attention weights),
            # (cross-attention position bias), (cross-attention weights)
```
And then later
```
            if output_attentions:
                all_attentions = all_attentions + (layer_outputs[2],)
                all_cross_attentions = all_cross_attentions + (layer_outputs[3],)
```

As I understand `layer_outputs[3]` is `(self-attention weights)` and it should be replaces with `layer_outputs[5]` which is `(cross-attention weights)`. The same goes of `(layer_outputs[2],) => (layer_outputs[3],)`

Does it make sense, or am I getting something wrong?

I tried to patch it locally and output + visualisation make sense (highlight image patch with information in token)

https://github.com/huggingface/transformers/blob/fe861e578f50dc9c06de33cd361d2f625017e624/src/transformers/models/pix2struct/modeling_pix2struct.py#L1550C74-L1550C74

Comments:
Hi @artyomxyz, thanks for reporting this issue.

Yes, you're right, there is a current issue with indexing in the Pix2Struct model. Related PR: #23985

cc @younesbelkada 

> Hi @loveisp ! Again thanks for your contribution on this Can you share with us why this PR got closed? The PR should also fix #24717 so it would be great to merge it :D

I initially made the change like this, but it didn't pass all the tests. The len(layer_outputs) here seemed a bit strange, so I changed it to what came later. Even though it can pass all the tests, there are still issues with the underlying logic. Regarding this matter, you can take a look at my discussion with @amyeroberts . I realized that I cannot fix this bug in a short amount of time, so I closed it. Will it be sufficient to make this change as he suggested, so that it passes all the tests? If so, then go ahead and make the change.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

For reference, a later PR was merged which (should have) resolved this: #25200

Hello, I have a question, How does `Pix2StructForConditionalGeneration.decoder` works for `doc-vqa`? 
I have got output from `Pix2StructForConditionalGeneration.encoder` which includes `last_hidden_state=[some tensor], hidden_states=None, attentions=None`.

I'm trying to manually use Encoder and Decoder part

Hi @rish-hyun. 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

If you want to see examples of how to use a model, you can check out the docs and model code: 
* https://huggingface.co/docs/transformers/model_doc/pix2struct#transformers.Pix2StructForConditionalGeneration.forward.example
* For example, you can see the model output of the [encoder here](https://github.com/huggingface/transformers/blob/e7b001db4fbd33d77de95cf684d13d7605660d1b/src/transformers/models/pix2struct/modeling_pix2struct.py#L654). 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.