Keep `image_sizes` in output of `PixtralProcessor`
Author: ywang96
Pull request: yes

This PR keeps the `image_sizes` in the output of `PixtralProcessor.__call__`.

For the context, on vLLM we have been using[ this particular kwarg from the output of `PixtralImageProcessor` to differentiate if a model is Pixtral-hf or LLaVA](https://github.com/vllm-project/vllm/blob/db87eb6c67271eb61ba9fd8559ce811a1a398a4d/vllm/model_executor/models/llava.py#L412-L415) (since they're both defined as `LlavaForConditionalGeneration`).


We are currently in a process of migration to directly use `PixtralProcessor` (and similarly `AutoProcessor` for a lot of other multimodal models) to reduce the development overhead, but noticed this is dropped during the processor call. I hope it's not a big deal to keep this (since it's just a 1d tensor) in the output, but let me know if there's any issue or better suggestion.

cc @mgoin @DarkLight1337

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@zucchini-nlp let me know if I need to provide more information - thanks in advance!

> Hey @ywang96 !
> 
> If we are to keep this kwarg in outputs, it should be also acceptable in model's forward call which is not true. We usually don't output kwargs that are not in forward signature
> 
> I realize Pixtral is a bit different from LLaVA and that causes headache in processing. We've been discussing with @Rocketknight1 the possibility of returning one batched tensor for the pixel values instead of list of tensors. I believe it would be a cleaner solution + in line with other VLMs
> 
> WDYT?

Yea - I'm honestly not a huge fan of using LLaVA definition to support Pixtral.

Regarding `one batched tensor for the pixel values instead of list of tensors`, what does the shape look like in your mind? If two images have different number of patches, are you going to return a tensor of `(total_num_patches, 3, h, w)`? 

This design has a caveat that if it's the only output of the processor, then you can't tell which patches are for which image. This is totally fine if for inference only, but tricky if users want to cache the embeddings of each individual images (something we want to do on vLLM).

@ywang96 I was mostly thinking about padding the images before the vision backbone and then unpadding them back after, similar to what we do in llava-next and also did in Emu (which is not merged to main library yet). I am not 100% sure if this will work out as expected because the model is not trained that way and adding extra zeros on height/width might change the outputs from patch embedding. That needs to be tested if it's almost same as looping over, as it worked with some other models

> cache the embeddings of each individual images 
Can you share more on this? You mean caching image last hidden states? The last hidden states should not change is shape because what currently happens is that images are embedded by looping over and then stacked as batch by flattening on the unmatched size dimensions.

Made a PR as POC and verified that padding the inputs matches the output with looping over inputs, https://github.com/huggingface/transformers/pull/35122

Hi @zucchini-nlp! I'm closing this PR as we might just deal with it on the vLLM side for now. Thanks!