[WIP] Add FLMR model
Author: LinWeizheDragon
Pull request: yes
Labels: name: New model

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR aims to add a new model, Fine-grained Late-interaction Multi-modal Retriever (FLMR). The model was proposed in [here](https://openreview.net/forum?id=IWWWulAX7g) and [here](https://arxiv.org/abs/2402.08327). This PR contains the following content:
- Model class for both FLMR and PreFLMR.
- Example scripts to run indexing and inference with FLMR and PreFLMR.

The original code base of this work is at [here](https://github.com/linweizhedragon/retrieval-augmented-visual-question-answering). The pre-trained checkpoints are [here](https://huggingface.co/models?search=PreFLMR).

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@ArthurZucker
@younesbelkada
@amyeroberts

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
@ArthurZucker
@younesbelkada
@amyeroberts
Can anyone help me to finish this PR or assign someone knowledgeable? The whole process is a little complicated and I have no idea what to do next. Thanks!

Hello @LinWeizheDragon, could you update the readme to have links to the pretrained checkpoints, the original codebase etc? 
I would recommend you to first start with a [code on the hub ](https://huggingface.co/docs/transformers/en/model_sharing)+ an issue on `transformers` to add support for this model. If it picks up the interest of the community 😉 



> Hello @LinWeizheDragon, could you update the readme to have links to the pretrained checkpoints, the original codebase etc? I would recommend you to first start with a [code on the hub ](https://huggingface.co/docs/transformers/en/model_sharing)+ an issue on `transformers` to add support for this model. If it picks up the interest of the community 😉

@ArthurZucker Thanks for your reply. I have added the requested info to the description of this PR.
 We are the authors of this model. We plan to release the pre-trained models here directly to provide easy access to multi-modal late-interaction models. I followed the docs [here](https://huggingface.co/docs/transformers/add_new_model#how-to-add-a-model-to--transformers) to finish a runnable version. The checkpoints have been converted and uploaded to the hub. I think the remaining things are having knowledgeable people to review the changes and pass all the tests.
Could you please let me know what to do next?


Hi @LinWeizheDragon, thanks for all the work opening this PR! 

As Arthur mentioned, we suggest adding the model directly to the hub. This is the recommended way to add models and we try to have as much support as possible for enabling this. This way, the model will be available to use and findable on the hub immediately. Adding a model into the transformers repo introduces maintenance costs. As such, the bar for adding models is high and the review process can take a long time.   

> Hi @LinWeizheDragon, thanks for all the work opening this PR!
> 
> As Arthur mentioned, we suggest adding the model directly to the hub. This is the recommended way to add models and we try to have as much support as possible for enabling this. This way, the model will be available to use and findable on the hub immediately. Adding a model into the transformers repo introduces maintenance costs. As such, the bar for adding models is high and the review process can take a long time.

@amyeroberts Thanks for your reply. Given that the model weights are already on the Hub, what is the recommended way of sharing the model classes and definitions with users? Is there a way to allow users to do "AutoModel.from_pretrained" easily?

@LinWeizheDragon Yes! :D Here's a guide on custom models and making them available on the hub: https://huggingface.co/docs/transformers/custom_models

If you want to use `AutoModel.from_pretrained`, you'll need to make sure to [register your model](https://huggingface.co/docs/transformers/custom_models#registering-a-model-with-custom-code-to-the-auto-classes). Here's an example of modeling code on the hub: https://huggingface.co/baichuan-inc/Baichuan2-13B-Chat/blob/main/modeling_baichuan.py

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.