Add Idefics2ForSequenceClassification
Author: frasermince
Pull request: yes

# What does this PR do?
Adds a sequence classification model for Idefics2. I personally have found this useful for reward models and wanted to see if there was interest in adding this to transformers. This is largely based upon MistralForSequenceClassification.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@amyeroberts I assume you are the person to ping for this! Also if you prefer I reach out to see if such an addition is desired please let me know!

Hi @frasermince, thanks for opening! I am the person to ping, but if you're ever unsure whoever you ping from the HF team will be happy to point you to the right person :) 

Regarding specific task-heads, we only add these if there's specific checkpoints available or in some cases if it's a highly requested feature. Are there associated checkpoints or a related issue for this? 

> Hi @frasermince, thanks for opening! I am the person to ping, but if you're ever unsure whoever you ping from the HF team will be happy to point you to the right person :)
> 
> Regarding specific task-heads, we only add these if there's specific checkpoints available or in some cases if it's a highly requested feature. Are there associated checkpoints or a related issue for this?

No related issue for this. Can you clarify what you mean by associated checkpoints?

@frasermince Sure! What I meant was are there already published, pretrained checkpoints available for this task, which can be loaded into the model? It's OK if they need to be converted to have the state dict match. 

These should ideally have the same license as the model's other checkpoints. If the checkpoints aren't official, then we'd have to make a judgement call based on how popular they are. This is in part because tasks such as classification can be implemented in many ways e.g. depending on how one might choose to pool the output. 

> @frasermince Sure! What I meant was are there already published, pretrained checkpoints available for this task, which can be loaded into the model? It's OK if they need to be converted to have the state dict match.
> 
> These should ideally have the same license as the model's other checkpoints. If the checkpoints aren't official, then we'd have to make a judgement call based on how popular they are. This is in part because tasks such as classification can be implemented in many ways e.g. depending on how one might choose to pool the output.

Got it, I assumed that the classification heads would just be used for downstream tasks. I do not currently have a specific checkpoint. Thank you for the clarification on this! Does that mean other sequence classification heads do not merely rely upon the base models parameters?

> Got it, I assumed that the classification heads would just be used for downstream tasks.

They are! But there can be ambiguity in how to approach the task without official checkpoints. For example, there's some debate about how this was implemented for CLIP and SigLIP #30952, #30373. In this case, it makes most sense for users to implement their own task-specific heads for their task.

> Does that mean other sequence classification heads do not merely rely upon the base models parameters?

I'm not sure I fully got the question re the base parameters here. If understood correctly, you're asking whether the task-specific models are fully defined by the base model and its config? To a large degree, yes, however there will still be some additional parameters e.g. number of categories to be classified which are new. For more complex tasks such as object detection, there can be a lot of design decisions and additional params on top of the base model 

> > Got it, I assumed that the classification heads would just be used for downstream tasks.
> 
> They are! But there can be ambiguity in how to approach the task without official checkpoints. For example, there's some debate about how this was implemented for CLIP and SigLIP #30952, #30373. In this case, it makes most sense for users to implement their own task-specific heads for their task.
> 
> > Does that mean other sequence classification heads do not merely rely upon the base models parameters?
> 
> I'm not sure I fully got the question re the base parameters here. If understood correctly, you're asking whether the task-specific models are fully defined by the base model and its config? To a large degree, yes, however there will still be some additional parameters e.g. number of categories to be classified which are new. For more complex tasks such as object detection, there can be a lot of design decisions and additional params on top of the base model

That makes sense! I was thinking of this as a classifier head on top of a language model due to that being what is useful for reward modeling. Makes sense though that for a multimodal model there is some ambiguity of implementation based on use case. Thank you for answering my questions! If this is not a good fit for the project I can close the PR!

@frasermince Let's close the PR for now. If there's popular checkpoints in the future then we can always reopen. Thanks again for contributing to the library!