README.md

---
library_name: transformers
license: mit
language:
- en
tags:
- retrieval
- multi-modal
- knowledge-based visual question answering
- FLMR
- PreFLMR
---

# FLMR model card

FLMR is an open-source model for multimodal knowledge retrieval. It is a transformer-based model that uses a combination of text and image inputs to retrieve relevant documents from a large corpus.

## Model Details

### Model Description

- **Model type:** FLMRModelForRetrieval
- **Language(s) (NLP):** English
- **License:** MIT License

### Paper and resources for more detail

- **Blog Post for quick overview:** https://www.jinghong-chen.net/fined-grained-late-interaction-multimodal-retrieval-flmr/
- **Paper:** https://openreview.net/forum?id=IWWWulAX7g
- **Repository:** https://github.com/LinWeizheDragon/FLMR 

## Uses

### Direct Use

This model can be used directly to retrieve documents from a large corpus using a combination of text and image input queries. The retrieval usage can be found in the [official implementation](https://github.com/LinWeizheDragon/FLMR).

### Downstream Use 

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

This model can be used combined with language models to create a retrieval-augmented language model. The use for Knowledge-based VQA can be found in [RAVQA](https://github.com/linweizhedragon/retrieval-augmented-visual-question-answering) 

## How to Get Started with the Model

For details of training, indexing, and performing retrieval, please refer to [here](https://github.com/LinWeizheDragon/FLMR).

## Training datasets
The model is pre-trained on
1. Image to Text retrieval: WIT
3. Image & Question to Text retrieval: OKVQA

For details on the dataset split and conversion process, please refer to the paper [Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering](https://openreview.net/forum?id=IWWWulAX7g).

The processed datasets are:
- https://huggingface.co/datasets/BByrneLab/OKVQA_FLMR_preprocessed_data
- https://huggingface.co/datasets/BByrneLab/OKVQA_FLMR_preprocessed_GoogleSearch_passages


## Evaluation datasets

The model is evaluated on OKVQA, Infoseek, and FVQA.

Please find the evaluation results in [the paper](https://openreview.net/forum?id=IWWWulAX7g).

## Citation 

**BibTeX:**
```
@inproceedings{
  lin2023finegrained,
  title={Fine-grained Late-interaction Multi-modal Retrieval for Retrieval Augmented Visual Question Answering},
  author={Weizhe Lin and Jinghong Chen and Jingbiao Mei and Alexandru Coca and Bill Byrne},
  booktitle={Thirty-seventh Conference on Neural Information Processing Systems},
  year={2023},
  url={https://openreview.net/forum?id=IWWWulAX7g}
}
```