Mantis: Interleaved Multi-Image Instruction Tuning
Authors: Dongfu Jiang
Xuan He
Huaye Zeng
Cong Wei
Max Ku
Qian Liu
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: Large multimodal models (LMMs) have shown great results in single-image vision language tasks. However, their abilities to solve multi-image visual language tasks is yet to be improved.
The existing LMMs like OpenFlamingo, Emu2, and Idefics gain their multi-image ability through pre-training on hundreds of millions of noisy interleaved image-text data from the web, which is neither efficient nor effective.
In this paper, we aim to build strong multi-image LMMs via instruction tuning with academic-level resources. Therefore, we meticulously construct Mantis-Instruct containing 721K multi-image instruction data to train a family of Mantis models. The instruction tuning empowers Mantis with different multi-image skills like co-reference, comparison, reasoning, and temporal understanding. 
We evaluate Mantis on 8 multi-image benchmarks and 6 single-image benchmarks. Mantis-Idefics2 can achieve SoTA results on all the multi-image benchmarks and beat the strongest multi-image baseline, Idefics2-8B by an average of 13 absolute points. Notably, Idefics2-8B was pre-trained on 140M interleaved multi-image data, which is 200x larger than Mantis-Instruct. We observe that Mantis performs equivalently well on the held-in and held-out benchmarks, which shows its generalization ability.
We further evaluate Mantis on single-image benchmarks and demonstrate that Mantis also maintains a strong single-image performance on par with CogVLM and Emu2. 
Our results show that multi-image abilities are not necessarily gained through massive pre-training, instead, they can be gained by low-cost instruction tuning. The training and evaluation of Mantis has paved the road for future work to improve LMMs' multi-image abilities.

Published in Transactions on Machine Learning Research (11/2024)
Mantis: Interleaved Multi-Image Instruction Tuning
Dongfu Jiang Xuan He* Huaye Zeng*, Cong Wei Max Kut, Qian Liu", Wenhu Chen Univeerstyy of Waterloo, TTinghua University, Sea AI Lab
{dongfu. blongfu.jiang wenhuchen)@@uvaterlooo.
Reviewed on OpenReview: http:///pppoorocceee//////////ss
Abstract
Large multimodal models (LMMs) have shown great results in single-image vision language tasks. However, their abilitics to solve multi-image visual language tasks is yet to be improved. The existing LMMs like OpenFlamingo, Emu2, and Idefics gain their multi-image ability through pre-training on hundreds of millions of noisy interleaved image-text data from the web, which is neither efficient nor effective. In this paper, we aim to build strong multi-image LMMs via instruction luming with academic-level resources. Therefore, construct MANTIS-INSTRUCE containing 721K multi-image instruction data to train a family of MANTIS models. The instruction tuning empowers MANTIS with different multi-image skills like co-reference, comparison, reasoning, and temporal understanding. We evaluate MANTIS on multi-imaae benchmarks and 6 single-image benchmarks. MANrIS--deffcss can achieve SoTA results on all the multi-image benchmarks and beat the strongest multi-image bascline, Idefics2-8B by an average of 13 absolute points. Notably, Idefies2-BB was pre-trained on 140M interleaved multi multi-image data, which is 200x larger than that MANTIS performs equivalently well on the held-in and held-out benchmarks, which shows its generalization ability. We further evaluate MANTIS on single-image benchmarks and demonstrate that MANTIS also maintains a strong single-image performance on par with CogVLi VLM and Emu2. Our results show that multi-image abilities are not necessarily gained through massive pro-training, instcad, they can be gained by low-cost instruction tuning. The training and evaluation of MANTIS has paved the road for future work to improve LMMs' multi-iman multi-image abilities.
meticulously
We observe
1
Introduction
Large Multimodal Models (LMMs) have advanced significantly in recent years. Both closed-source models like
GPT-4V (Achiam et al. 2023 Gemini Team et al. 2023), Reka (Ormazabal et al. 2024
MM1 (McKinzie
et al.
2024) and open-source models like LLaVA Liu et al. 2023c b). LLaVA- NexTT
(Liu et al. 2024
BLIP Li et al.) 2023c), CogVLM (Wang et al. 2023). Idefics Laurençon et al.)
2023)
have shown strong
visual-language understanding and generation capabilities in single-image tasks like VQA (Antol et al. 2015).
Tex (Singh et al., 2019bb, GQA (Hudson Manning, 2019), etc. Despite the significant interest.
improving LMMs' performance on singlo-image vision-language tasks, relatively less work attempts to improve LMMs' capability to solve multi-image vision-language tasks. To the best of our knowledge, only few
open-source models (e.g. Idefics (Laurençon et al. 2023), OpenFlamingo (Awadalla et al. 2023 Emm2 (Sun
ct al. 2023) and VILA (Lin et al. 2023ch support multi-image inputs.
We argue that multi-inage visual ability also a crucial in real-world applications. Specifically, we categorize the ability into four major skills we want the LMM to encompass: (1) Co-reference: understanding the references like "sccond image" the natural language expression and grounding it on the referred image
generate a response. (2) Comparison: capturing the nuances and commonalities between several images.
(3) Reasoning: capturing the information across multiple images and reasoning over these multiple pieces

---

Published in Transactions on Machine Learning Research (11/2024)
65
Does the shop frame in (image 1: <IMAGE>
Single-Image (avg)
</IMAGE>) have
the same color as the vending machine in (image 2: <IMAGE>
MVBench
NLVR2
55
</IMAGE>}?
51
90
68
Co-reference Compare
Reasoning Temporal
15
InstructBLIP
TOOOOR
28
LLaVA-1.5
20
QwenVL-Chat
00O000
38
CogVLM
No, they have different colors. The shop frames in the first image is red but the vending machine in the second image is blue. So the answer no.
70
Idefics2
BLINK
40
Q BeBch
Emu2-Chat
Mantia-SigLIP
Mantis-Eval
Mantis-Idefics2
Figure 1: Mantis accept interleaved text-image to perform different multi-image tasks like coherence, comparison, logical reasoning, and temporal understanding. Mantis achieves state-of-the-art performance on the five multi-image benchmarks and preserves abilities for single-image tasks.
to derive the response. (4) Temporal understanding: observing multiple frames of a video or a scene to understand the temporal information like actions, behaviors, interactions, etc.
The existing multi-image LMMs like Idefics (Laurençon et al. 2023), OpenFlamingo (Awadalla et al. 2023)
Emu2 (Sun et al., 2023), MM1 (McKinzie al.) 2024), VILA (Lin et al. 2023c heavily rely on pre-training massive interleaved image-text web documents like MMC4 (Zhu et al. 2024). Obelics (Laurençon et al, 2023). For example, MMC4 contains 500 million examples, while Obelics contains 140 million examples. Such pro-training procedure would require massive amount of computation, which hampered its adoption in most open-source LMM training pipelines. In this paper, we al tempt. to build LMMs with academic-level resources to support interleaved multi-image inputs. Specifically, we made a few efforts along this line:
Datasct: We create the first multi-image Instruction-tuning dataset MANTIS-INSTRUCT. It has total of 721K instances, consisting of 14 subsets to cover all the multi-image skills. Among the 14 subsets, 10 subsets IconQA (Jhamtani & Berg- Kirkpatrick Birds-to-Nords [orbes et al.) 2019).
are from the existing datasets. For example, NLVR2 (Suhr et al.
2018
2018), etc are used to cover 'reasoning' skill; DreamSim (Fu et al.
2023
etc are used to cover 'comparison' skill; NEXT-QA (Xiao et al.) cover 'temporal understanding' skill. Additionally, 4 new subsets are newly curated, where LLaVA-665k-multi,, and LRV-multi are used to to cover the 'coref" skill, Contrast-Caption, and Multi- are used to broaden the 'reasoning' skill. The interleaving image placeholders are inserted into the text under various heuristics.
2021). STAR (Wu & Yu 2021). etc are used to
Architecture: Similar to LLaVA (Liu et al. 2024), we start from a pre-trained language model like LLaMA- (AI@Mcta 2024) or Fuyu (Adept AIl 2023) and vision transformer encoder from CLIP (Radford al., 2021] SigLIP (Zhai et al. 2023). We use the same multimodal projector LLaVA to map the vision embeddings to the text embeddings space to simply concatenate them with the text embeddings. We also define text-image interleaving format: ".. (image {i}: <BOI>image embeddingg<<BBDI>.... where <BOI> and</E0I> are the image delimiters. We explored multiple instantiations and found this optimal design.
Evaluation: We manually curate new challenging dataset Mantis-Eval to cover several multi- skills. This evaluation can help us better analyze LMMs' multi-image abilities.
We Irain MANTIS by combining MANTIS-INSTRUCY with another 268K single-image vision-language data to balance the multi-image and single-image abilities. To evaluate the multi-image skills, we adopt 8 benchmarks including NLVR2 (Suhr et al.] 2018), BLINK (Fu al. (024),, MVBench (Li al. 2023d). Mantis- Eval, etc. to cover all the mentioned skills. We include 15 competitive baseline models including single-image LMMs like LLa VA-Next (Liu et. al. 2024), Qwen-VT. (Bai al. 2023 and multi-image LMMs like GPT-4V,
2

---

Published in Transactions on Machine Learning Research (11/2024)
Idefics1/2, etc. For single-image LMMs, we merge multiple images horizontally as single-image input. By training on 16x x A100 for 36 hours, MANTIS achieves state-of-the-art performance on all the multi-image tasks, while preserving competitive performance in single-image tasks. Notably, MANTIS-Idefien outperforms the strongest bascline Idefics2 (Laurençon et al. 2023 by an average of 13 absolute points on the 7 avaialble benchmarks (excluding MileBench), where Idelics2 is nulti-image pre-trained, showeasing the effectiveness of multi-image instruction-tuning. The held-in and held-out results are oquivalently strong, which shows the generalization ability of MANTIS.
These results are encouraging because show that the low-cost instruction tuning 721K high-quality data can lead to much better generalization performance than other LMMs intensively pre-trained on 100x larger datasets. Mantis also preserves its strong performance of single-image tasks and is on par with CogVLM (Wang et al., 2023) and Emu2 (Sun et al., 2023). We also present insights on the architecture selection and inference approaches based on the results in section 3] We believe
can serve as an essential bascline for future studics on multi-image LMMs due to its simplicity and low cost. We will release all the code, dala, and models to help with the reproducibility of our results.
2 Mantis: Interleaved Multi-Image Instruction Tuning
2.1 Method
Model Architecture: Most existing LMMs do not support multi-image inputs, either due to the architecture
design Huang et al., 2016: Li et al.) 2023c), or the lack of associated code support (Peng et al. 2023 Adept 2023). To enable multi-image training and inference, we have modified the LLaVA architecture and added code support for it. Idefics2 (Laurenccon et al. 2024h already supports multi-image as inputs naturally, we thus inherit their structure.
Image Context Length: How many images can a iulti-image supported model accept? This is limited by both the LLM backbone's max token length and the number of image tokens the vision encoder generated. For the LLaVA architecture, cach image consumes a fixed (336/14)² = 576 image tokens. For Llama3 with a maximum window size of 8K, we can feed al most. 14 images. For Idefies2, the number of tokens per image is resampled to be 64 tokens only due to the existence of perceiver, which is pretty efficient. Given 8K context length, it can accept 128 images at most, even more than some video models.
Interleaving Text-Image: A proper text-image interleaving format can help acquire multi-image under- standing and reasoning ability. We contend that good text-image interleaving format should: (1) mark boundaries between images clearly, and (2) denote the serial number of images. Following this principle, we designed our interleaving format as follows: (image {i}: 300><<maae>><011
where <BOI> is the begin of image token and <EDI> is the end of image token. <image> is the placeholder for image patches. This format adds clear separators between images, and gives serialized information of the image through 'image {i}'. In practice, we set <BOI> and <EDI> be <Image> and </Image> respectively. Previous work also demonstrates its effectiveness through a comprehensive ablation study (Wu et al., 2024).
2.2 Mantis Instruct:: A large scale multiple image question answering dataset
We construct MANTIS-INSTRUCH from multiple publicly available datasets. Detailed statistios are shown in (Talle I with several newly curated subset by us. We use a similar dataset format with LLaVA's, As, where each data item contains multiple images and multiple turns of QA pairs are gathered. We demonstrate some examples of our newly curated subset in Figure 2 We report the number of examples, the number of images per item, the average conversation turns, and the average length. the 4 multi-image skills. We here describe the gathered subsets for each skill briefly. Construction details can be found in subsection A.I
is collected based on
Co-reference requires the model create mapping from natural language references, such as "second image,' to the actual images in the input. It. does not require the model to infer across multiple images.
3

---

Published in Transactions on Machine Learning Research (11/2024)
Subsets
Skill
# Instance
# Avg-I
# Max-l
# Turns
Length
Length?+/
Multi-Image Reasoning Datasets
LLaVA-665k-miltt
Coref
313K
2.0
21.4
558
1710
URV-multi"
Coref
8K
3.5
83.4
2234
4251
CoInstruct
Compare
151K
2.7
7.5
314
1620
Dreamsim
Compare
16K
3.0
2.0
103
1831
Spot-the-Diff
Compare
8K
2.0
4.0
121
1273
Birds-to-Words
Compare
3K
2.0
2
2.0
101
1253
NLVR2
Reason
2.0
1257
IconQA
Reason
64K
2.4
2.0
71
1454
Contrast- Contrast-Caption *
Reason
36K
3.8
7.6
871
3067
ImageCoDe
Reason
17K
10
10
1.0
126
5886
Multi-VQA*
Reason
5K
4.0
11.0
1102
3417
VIST
Temporal
7K
20.3
530
12238
NEXT-QA
Temporal
4K
8.0
17.6
572
5180
STAR
Temporal
3K
8.0
30.2
961
5569
Total
721K
4.7
14.4
555
3584
Single- Image Reasoning Datasets
DVQA
Dac
200K
1.0
40.0
304
880
DocVQA
Doc
39K
1.0
638
ChartQA
Chart
28K
1.0
2.0
66
642
Total
268K
1.0
14.7
144
720
Table 1: The statistics of MANTIS- INSTRUCT. We list the number of examples, average images per example, maximum image per example, average turns, average text token lengths, and average text-+image token lengths. o means the new subsets we constructed this paper. Here we use Llava's tokenizer and assume each image occupies 576 image tokens, which is the number of tokens for each image. Here Avg Length?' denotes the average number of tokens for the text part, while 'Avg Length means the average number of tokens of text and images in total. Heuristics about how each subset is curated can be found in subsection A.1
Please generate 10 independent QA pairs. Each question shall involve at least 2 images to answer. Try to cover different ability like reasoning, planning, common sense understanding, etc. Be creative with your questions and make sure the answers require integration information from multiple images** **, Use 'image to refer to the i-th image in your questions.
Output format:
Question: First question?
Answer: The answer to the first question.
Question: Second question?
Answer: The answer to the second question.
Table 2: Propmt template used to curate the Multi-VQA subset.
achieve this, we constructed LLaVA-665k-multt and LRV-multi by concatenating multiple single-image conversations into multi-image sequence. Deliberately, we included natural language references like 'For the second image" for each question.
Comparison requires the model to not only have good co-reference ability but also be able compare and understand differences across multiple images. We gat Co-Instruct, Dreamsim, tDot-the-Diff, and
4

---

Published in Transactions on Machine Learning Research (11/2024)
Contrast-Captioning
Multi-VQA
Images:
Images:
PHARMARY
Q: Here are images <image> <image> limage-iimgge--mmgee which image
Q: Here are 5 images <image> <image> <image> <image> <image>, What features are noticesble
social media campaign for charity event happening at the Walgreena Pharmacy, which element image could
Q: someone
shows the following content:
foremost image??
encourage online engagement and 25image><<image>>
(caption of image 2]
Answer: (caption of image 1)
Options:
Image (B)
Image
Q:
What features noticeable the fourth image in sequence?
image could to encourage online engagement because it common symbol positive interaction on social media platforms, suggesting approval support
Answer:
(D) Image4 4 (C) Image
Answer:
Answer: (caption of image
Figure 2: Illustaatoons of MANTIS-INSTRUCY datasets from Contrast- cettraas-Caationnng and Multi- Contrastivo- Captioning is constructed from existing image captioning datasets to enhance LMM's multi-image "co-reference" ability. Multi- Aitti--QAA GPT-4V synthesized multi-tury multi-turn multi-iman data used to enhance LMM's multi-image 'reason' ability.
Birds-to-Words together to enhance the model's comparison ability. They cover wide range of topics such image quality, visual similarity, and difference description.
Reasoning requires the model to further make inferences by combining its world knowledge with the information collected using the co-reference and comparison ability. It distinguishes from the comparison skill by introducing more complex human instruction associated with the images. We gather NLVR2, IconQA, Contrast-Caption by reformatting captioning datasets, ImageCoDe, and our self-collected Multi-VQA to further equip the model with reasoning ability. The topics include logical reasoning, counting, image matching, image retricval, and free-form multi-image QA. Multi-VQA is synthesized by prompting GPT-4V, where the prompt template is shown in Table? 2
Temporal Understanding requires the model to understand temporal image sequences such as videos, comics, etc. We include small number of video understanding datasets (14K), VIST, NExT-QA, and STAR, to activate the model's temporal understanding ability. We contend that model with the above 3 multi-image skills shall be easily tuned to understand image sequences.
2.3 Heuristics of data curation
Similar visual instruction tuning (Luu et al. 2023c), data curation is an important step ensure good performance. During the curation, found some heuristics that could improve performance. We list them below. Note that the first heuristics is simple way diversify QA format to avoid overfitting, and the rest two heurislics have already been proven effective by previous work Co-Instruct (Wu et al. 2024). We thus adopt them directly to avoid duplicate efforts to confirm their effectiveness.
1. Conflicts between short answers and long answers. Many datasets we collected are domain- specific, and some of them are classification tasks like NLVR2 (Shr et al. 2018 and DreamSim (Fu et al. 2023 For these datasets, we convert them into multiple-choice QA format. to avoid conflicts between repeated short-answers and some long answers from other datasets.
2. Addition of image denotation in the text for each question. We manually add image denotations like "the first image', "image 2", "the right image', etc. to each question in our dataset. to make sure the question involved with multiple images is clear to answer.
Positions of the "<image>" placeholder. For each item with multiple images, we write some rules to randomly put the image placeholders at the beginning of the first question or the end of the first. question. This technique is also used by the curation LLaVA-665k (Liu et al. 2023b
5

---

Published in Transactions on Machine Learning Research (11/2024)
3 Experiments
In this section, aim to evaluate MANTIS's ability on both multi-image and single-image vision-language tasks. For multi-image tasks, depending on whether the LMM can handle multiple images, we either concatenate multiple images ("merge") horizontally as a single image or feed multiple images as a 'sequence' to the model. We have trained four model variants listed in Table 3 where we list the architecture and dataset details. Mantis-CLIP and Mantis-SigLIP only adopt the CC3M 560K subset for feature alignment. Mantis-Flamingo and Mantis-Ideficst Idefics2 are initialized from OpenFlaminge and Idefics2, respectively, which have been intensively pre-trained the multi-image corpus. We also include Mantis-Fuyu to investigate the significance of vision encoder in the LMM architecture. After this pre-training and preparation, we fine-tune them on MANTIS-INSTRUCE to get Mantis.
Mantls-Models
V-Encoder
LLM-backbone
Pre-Training (PT)
PT size
Fine- Fine-Tuning
Tokens /
Mantis-Fuye
Persimmon-8B
Unknown
Unknown
Instruct
[1,2304]
Mantis-Flamings
Mantis- Flamingo
CLIP
MPT-7B
MMC4, LAION
2421M
Mantis-Instrus
576
Mantis-CLll
CLIP
LLaMA-3-8B
CC3M subset
0.56M
InstiisInstruct
Mantis-
SigLIP
LLaMA-3-8B
CC3M subset
0.56M
Mantis-Instrun
576
IdefissI
SiGLIP
Mistral-7B-v0.11
OBELICS, Cauldron
143M
Instruct
64
Table 3: Architecture and training details of mantis model family. 'I" for 'Image' in last column.
3.1 Training Details
For MANTIS-CT.IP and MANTIS-SigI.IP, we first pre-train the multimodal projector on LLaVA pre-train data, where the learning rate is set to 1e-3, the batch size is set to 256. After pre-training, we fine-tune the 2 models the MANTIS-INSTRUCE For MANTIS-Fuyu, MANTIS-Flamings and MANTIS-Idefics2, directly fine-tune them on MANTIS-INSTRUCT since they have already been pre-trained on millions of data.
During the fine-tuning, we train each model on the data for 1 epoch, with a batch size of 128. The maximum context length is set to 8192. The learning rate is all set to 1e-5, except for Idefics2, where the learning rate is set to 5e-6 to better preserve its original knowledge. We set the warmup ratio to be 0.03 and use a cosine learning rate scheduler. We speed up our training and inference with Flash-attention2 (Dao 2023). ) We use DeepSpeed Zero-3 (Aminabadi et al.) 2022) for full fine-tuning. We apply QLoRA (Dettmers al., 2023) along with DoRA (yang Liu et al., 2024) to more efficiently do comprehensive ablation studies under limited resources. All full fine-tuning
16 A100 GPUs while the ablation study ran on 8 A100 GPUs.
3.2
Baselines
For 'merge' evaluation, we include BLIP-2-13B (Li al. 2023c), InstructBLIP-13B DDa al. 2023); Qwen-
VI- Chat [Bai et al., 2023), LLaVA-1.5-7B (Liu al. 2023c], LLaVA-1.6-7B (Liu et al.) 2024), Kosmos2 (Peng
et al.
2023 and Fuyu-8B (Adept Al 2023). For 'sequence' evaluation, we include OpenFlamings (Awadalla
ct al.
2023). Otter (Li et al. 2023b), VideoLLaVA [Lin et al. 2023a Emu2-Chat-341
(Sun et al.
2023
VILA
Lin et al. 2023b) Ideficsl [Laurençon et al. 2023), Idefics2 [Laurençon et al. 2023
GPT-4V
Achham
et al.
2023). Additionally, for single-image tasks, we include InstructBLIPP-B-- Vicuna
Dai et al.
2023
Yi-VL-6B
(AI et al. 2024
3.3
Evaluation Benchmarks
For multi-image tasks, we use 2 held-in benchmarks: NLVR2 (Suhr ct al. 2018) and Qbench (Wu et al. 2023ah; and 3 held-out benchmarks: Mantis-Eval, BLINK (Fu et al. 2024h, and MVBench (Li et al. 2023d). We report detailed statistics in [Table 4
NLVR2 (Suhr et al., 2018) evaluates the model's ability to conduct logical reasoning across the contents of images. It asks the model compare the contents of the two given images and judge whether given statement is correct not. All questions are multiple-choice. We use the test-public split for evaluation.

---

Published in Transactions on Machine Learning Research (11/2024)
Benchmarks
Skill
Type
Instances
# Avg-1
# Max-1
Lenght T+7
NLVR2
Reason
MCQ
6,967
2.0
2
1,190
Q-Bench!
Comparison
MCQ
1,000
2.0
1,167
Mantis-Eva
Reason & Coref
MCQ & Short
217
2.5
5
1,455
BLINK
Reason
MCQ
1,901
1.9
4
1,129
MVBench
Temporal
MCQ
4,000
8.0
8
4,625
MileBench
Mixed
MCQ
6,440
15.2
109
9,294
MuirBench
Mixed
MCQ
2,600
4.3
9
2,502
MMTU
Mixed
MCQ
11,600
6.6
32
3,852
Table 4: Statistics of the evaluation dataset. Held-in benchmarks, NLVR2 and Q-Bench, are marked with i notation. The rest of them are held-out benchmarks. "Mixed" means
Qbench (Wu et al., 2023a) is a benchmark evaluating whether LMMs can properly judge and compare the quality of benchmark. Q-bench aims to evaluate the low-level visual abilities of LMMs, such as judging the image quality. All questions multiple-choice. In experiments, evaluate the Qbench2-A2-p dev set, where low-level visual question is asked based multiple image contents.
Mantis-Eval consists of 217 multiple-image reasoning examples that cover different topics, including size perceptions, weight comparisons, etc. This dataset is carefully curated by our annotators, where images are acquired via Google Search, and then come up with a proper question which requires undorstanding the contents the two images well to answer. Mantis-Ewee Eval contains both multiple-choice and short-answer questions. We report results in the test split.
BLINK (Fu et al., 2024) is benchmark most tasks "within blink" (e.g., relative depth estimation, visual correspondence, forensics detection, and multi-view reasoning). Some questions in the benchmark involved multiple images, such as image similarity comparison. We report results in the validation set of the benchmark.
visual perception abilitics, where humans solve
MVBench (Li et al., 2023d) a comprehensive multimodal video understanding benchmark covering 20 challenging video lasks. The questions camnot. be solved with a single frame, necessitating understanding an image sequence to solve the questions the benchmark. We extract frames per vidco for the evaluation. We report results in test split.
MileBench (Song et al., (024) is novel benchmark designed to test the multimodal long- long-context capabiliiiiss of MLLMs. It comprises realistic evaluation and diagnostic evaluation. The former involves tasks like temporal understanding and semantic multi-image comprchension. The later diagnostic evaluation involves TextNeedle and ImageNeedle retrieval tasks. We only reported the performance of the realistic evaluation. The results of the diagnostic evaluation can be found in MileBench's paper.
MuirBench (Wang al., 20244) comprchensive benchmark consisting of 12 diverse multi-image lasks, such as scene understanding, ordering, elc. It contains 2,600 multiple-choice questions with 11,264 images involved in total. We report the overall average performance across the 12 tasks.
MMIU (Meng et al., 2024) is multi-image evaluation benchmark encompassing types of multi-image relationships, 52 tasks, 77K images, and 11K multiple-choice questions. We report the overall average performance across all the tasks.
3.4 Results on Multi-Image Tasks
We report our results on the multi-image task in Table 5. We put the results of GPT-4V/o at the top of the table as it's closed-source LMM. We can see that best version Mantis-Idefics?? almost matches the overall performance of GPT-4V. We put comparison results between MANTIS and other open-source LMMs, along with the insights derived in the following paragraphs.
Mantis performs well on held-in evaluation: We found that Mantis-SigLIP without any multi-image pre-training can already achieve the best- known performance on these two benchmarks. Mantis- Mantis-Idefics2 has better performance of 89.71 and 75.20 on NLVR2 and Q-Bench respectively, marking it the SoTA for these two tasks. Notably, Mantis Idefics2 Mantis-Idefics2 also beats the Idefics2 by 2.84 points, which was already trained on NLVR2. It. shows the cross-task benefits of our instruction tuning. The remarkable performance these 2
7

---

Published in Transactions on Machine Learning Research (11/2024)
Models
Size
NLVR2
D-Bench
Mantis-Eval
BLINK
MVBench
MileBench
MuirBench
MMIU
GPT-4V/o
88.80
76.52
62.67
51.14
43.50
53.00
68.00
55.70
Open Source Models (merge)
Random
48.93
40.20
23.04
38.09
27.30
22.30
23.99
27.40
Kosmas2
1.6B
49.00
35.10
30.41
37.50
21.62
VA-v--1
7B
53.88
49.32
31.34
37.13
36.00
38.00
23.46
19.20
LLVVA-VI.6
7H
58.88
54.80
45.62
39.55
40.90
38.10
22.20
Qwen-VVi-Chat
7B
58.72
45.90
39.17
31.17
42.15
39.10
15.90
Fuyu
8B
51.10
49.15
27.19
36.59
30.20
BLIP-2
13B
59.42
61.20
49.77
39.45
31.40
InsIructBLIP
13B
60.26
44.30
45.62
42.24
32.50
CogVLM
17B
38.58
53.20
45.16
41.54
37.30
20.85
Open Source Models (sequence)
OpenFlamingo
9B
36.41
19.60
12.44
39.18
7.90
27.40
23.73
Otter-Image
9B
49.15
17.50
14.29
36.26
15.30
Ideficsl
9B
54.63
30.60
11
24.69
26.42
35.43
12.80
VideoLLaVA
7B
56.48
45.70
35.94
38.92
44.30
Emu2-Chat.
37B
58.16
50.05
37.79
36.20
39.72
VILA-1.5
8B
76.45
45.70
51.15
39.30
49.40
44.40
Idefics2
8B
86.87
57.00
48.85
45.18
29.68
26.08
27.80
Mantis-Fuyu
8B
72.41
54.10
37.79
39.87
39.12
Mantis-Flamingo
9B
52.96
46.80
32.72
40.83
Mantis-CLi
8B
84.66
66.00
55.76
47 17.06
48.30
37.38
Mantis-SIGLIP
8B
87.43
69.90
59.45
46.35
50.15
47.5
36.12
Mantis-Idefiss2
8B
89.71
75.20
57.14
49.05
51.38
44.50
45.60
A over SOTA
2.84
18.20
8.30
3.87
1.98
3.10
9.07
17.80
Table 5: Evaluation results on multi-image benchmarks. The highest score of open-source models for each task is bolded. The highest score of the open-source bascline model is underscored. A the performance gains of MANTIS compared to the best baseline model for each task. We report the GPT-4o's results MuirBench and MMIU, and GPT-4V's the rest of them.
Model
Size
Tex
VQA
MMB
MMMU
OKVQA
SQA
Avg
OpenFlamingo
9B
46.3
58.0
32.4
28.7
51.4
45.7
43.8
Idefics1
9B
39.3
68.8
45.3
32.5
50.4
51.6
48.0
InstructBBLPP
7B
33.6
75.2
38.3
30.6
45.2
70.6
48.9
Yi-VL
6B
44.8
72.5
68.4
39.1
51.3
71.7
58.0
Qwen-VI-CC VL-Chat
7B
63.8
78.2
61.8
35.9
56.6
68.2
60.8
LLaVA-1.5-Vieae Vieuna
7B
58.2
76.6
64.8
35.3
53.4
70.4
59.8
LLaVA-1.6-Viis
7B
64.9
81.8
67.4
35.8
44.0
68.5
60.4
Emu2-Chat
37B
66.6
84.9
63.6
36.3
64.8
65.3
63.6
CogVLM
17B
70.4
82.3
65.8
32.1
64.8
65.6
63.5
VILA-1.5
8B
64.4
80.9
72.3
36.9
0.0
79.9
55.7
Idefics2
8B
70.4
79.1
75.7
43.0
53.5
86.5
68.0
Mantis-Cle CLIP
8B
56.4
73.0
66.0
38.1
53.0
73.8
60.1
Mantis-SigLII
8B
59.2
74.9
68.7
40.1
55.4
74.9
62.2
Mantis-Idefies?
8B
63.5
77.6
75.7
41.1
52.6
81.3
65.3
Table 6: Evaluation results of MANTIS on various single-image tasks. The full name of the abbreviations in the table are listed here. Text Text VQA validation set; VQA: VQA-v2 validation set; MMB: MMBench-Engghn English test set; SQA: Science validation set. The highest score for each task is bolded, while the second highest is underscored.

---

Published in Transactions on Machine Learning Research (11/2024)
benchmarks demonstrates that Mantis has gained strong ability in multi-image logical reasoning and low-level vision understanding.
Mantis generalizes well on held-out evaluation: We show that our models can attain strong performance the 6 held-out benchmarks. Mantis-SigLIP achieves 59.45 on Mantis-Eval, and Mantis- Mantis-Idefics2 achieves 49.05, and 51.38 on BLINK and MVBench respectively, surpassing all other baselines on these held-out tasks. It's also worth mentioning the reported performance of Mantis on the 3 recently released mullt--imaee benchmarks with comprehensive evaluation tasks. Mantis models surpassed the VILA and Idefics2, which are pre-trained with millions of multi-image data, by a large margin, making them the best open-source models on the leaderboard. This is strong evidence of Mantis's good generalization ability in the unseen scenario.
Multi-Image Pre-training is not necessary: In these experiments, Mantis- and Mlantis-SgLLII are attaining much better performance than Mantis-Flande Flamingo, and similar performance as Mantis-Idefics, which are pre-trained on millions of text-image interleaved data. These results reveal that the multi- pre-training (on massive web corpus) is not a necessary path toward strong multi-image performance.
'Merge' vs 'Sequence' input: The 'merge' evaluation an important baseline to evaluate the multi-image understanding ability of those LMMs that can only accept one image at time. LLaVA-v1.6' dynamic image resolution feature can divide single image into multiple tiles, processing the 'merge' image like multiple images (iu et al. 2024). But it still performs poorly multi-image tasks, only getting 45.62 and 39.55 on the Mantis-Eval and BLINK benchmark, which are at least 10 absolute points lower than Mantis. This result indicates the advantages of building models to accept sequence image inpuls.
Unlike many other LMMs like Flamingo, LLaVA, and BLIP, Fuyu is a decoder- only LMM without a vision encoder. To investigate the importance of vision encoder, we also train an additional Mantis-Fuyu as a comparison. Results in [Table 5) show that the performance of Mantis-Fuyu is worse than Mantis-CLie and Mantis-SigLi where a vision encoder exists. While we acknowledge a possible factor brought by the different LLM backbones, we think whether vision encoder exists bigger difference and conclude that a good vision encoder can enhances the performance.
Vision encoder matters:
3.5 Results on Single-Image Tasks
We also evaluate Mantis-CLIP and Mantis-SgLLIP on various single-image tasks, including Text (Singh et al. 2019ap, VQA-v2 (Goyal et al. 2016), MMBench (Liu et al.] 2023d), MMMU (Yue el al., 2023 1, etc.
Results are shown in Table 6 All the evaluations are conducted with the help of LMMs-Eval (Li al, 2024 tool. Mlants-Fllamnnge
and Mantis-Fuyu's performance is omitted due to its bad performance.
Compared to set of popular LMMs, Mantis-SigL.IP gets significant improvements on MMBench-English, MMMU, and ScienceQA benchmarks though we did not specifically optimize the single-image abilitics. Solving problems in these tasks requires not only the visual perception ability but also the good reasoning ability of the LLM backbone. Considering that we are using the powerful LLaMA-3 as the backbone LLM, these improvements are expected. We also compare the results of Mantis-Tdefics2 against Idefics2 on single-image tasks. On several tasks like MMB, MMMU, and OKVQA, Mantis-Idefics maintains a similarly strong performance Idefics2. However, we do observe some drops in ScienceQA, which are mainly due to a lower mix ratio of OCR and Math data in our instruction tuning. Overall, the average drop of 4% is tolerable.
3.6 Ablation Studies
Is multi-image instruction tuning necessary? To investigate, we further train a Mantis-Flamine basod on OpenFlamingo (Awadalla et al. 2023), which has been trained on MMC4 (Zhu et al. 2023), text-image interleaved pre-training dataset. As shown in Table 5) the average performance of Mantis-F Flammngo increased by 19.2 points after further fine-tuning on MANTIS-INSTRUCE improvement space, even if a model is fully pre-trained on a multi-image dataset, which demonstrates the necessity of learning multi-image skills in the fine-tuning phase.
The results show that there is still huge
Is multi-image pre-training necessary? To investigate, we design an ablation study, where one model is pre-trained on the CC3M 560K subset, which contains only image-text pairs, while the other one is pre-trained additionally on OBELICS- subset, which contains interleaved multi-inage-text pairs. Due
9

---

Published in Transactions on Machine Learning Research (11/2024)
Pretrain
NLVR2
Q-Bench
Mantis-Eval
BLINK
MVBench
Avg
CC3M subset + Mantislnstruct subset
76.73
57.00
47.00
43.79
47.08
54.32
CC3M subset + Mantislnstruct subset
76.83
60.50
53.00
45.20
45.97
56.30
OBELICS-IOND
A (with OBBLICS-100K)
+0.10
+3.50
+6.00
+1.41
-1.11
+1.98
Table 7: Ablation on whether multi-image pre-training is nocessary. One model is pro-trained on the CC3M subset, while the other additionally pre-trained on the a OBELICS-100K multi-image dataset. They are both fine-tuned on a subset MANTIS-INSTRUCE for comparison. Results show minor improvement with additional multi-image pro-training. Each ablation model is trained with QLoRA with DoRA.
Models
NIVR2
Q-Bench
Mantis-Fxal Eval
BLINK
MVBench
Avg
A
Idefics2-81
86.87
57.00
48.85
45.18
29.68
53.52
+ Coref
84.48
63.20
57.14
48.58
49.40
60.56
+7.04
+ Compare
85.92
69.00
53.92
48.
48.33
61.03
+0.47
+ Reason
89.62
68.50
62.21
48.31
47.93
63.31
+2.28
+ Temporal
89.54
67.00
61.29
49.34
50.02
63.44
+0.12
Table 8: Data ablation study results of different subsets of MANTIS-INSTRUCT, where each skill corresponding to specific subsets defined in Table 1 A column of each row represents the performance improvements after adding new subsets to the last column results. Each ablation model is trained with QLoRA with DoRA.
to limited computation resources, we fino-tune both models on a downsampled MANTIS-INSRRUES dataset to compare their performance. Results in Table 7 show that there are little improvements by adding additional pre-training on OBELICS- 100K. We thus conclude that multi-image pre-training could be useful, but is not a necessary path for models to gain multi-image abilities. The poor performance on Mantis-Flaming also confirm our hypothesis.
components: We conduct data ablation study to analyze the effects of different subsets in MANTIS- to enhance model's multi-image ability. As shown in Table As new subsets of MANTIS- INSTRUCT are added continually, the overall performance is continually improving, proving that every subset is contributing positively.
Ablation study of Mantis-
3.7 Qualitative Results
To conduct a qualitative analysis of Mantis models on specific multi-image scenarios, we include some case studies in Figure 3 where we include Emu2, which can accept a sequence of images as inputs, and LLaVA-1.5, which can only accept one image a time, and the powerful GPT-4V, which perform well on both single-image and multi-image scenario.
The first requires the model to count the number of dice respectively in the two images. While previous LMMs might have learned good counting ability in the single-image scenario, most of them failed in the multi-image counting problem, including GPT-4V. The second case requires the model to involve their world knowledge to infer the mood the character in three images. However, Emm2, and LA-15- output similar contents, and both fall back into the simple captioning behavior. demonstrates that these models struggle distinguish two images as separate information. In contrast, Mantis can correctly do the task due to the existing of reasoning subset in Mantis-Instree Instruct.
10

---

Published in Transactions on Machine Learning Research (11/2024)
How many dices are there in image 1 and image 2 respectively?
Which image shows different mood ch character from the others?
Mantia: There are dioes image and dices in image (Correct)
Emu image 1, there are six dices, image 2, there are also (Wrong)
Mantis: Second Image. (Carreet)
Emn 2: The image that shows a different mood character from the others the where smiling and laughing. (Wromg)
LLaVA-1.5 Sc in image thon are six dices, and in imagee 2, there seven diees. (Wrons)
GPT-4V: the firsr imago, There five dice,, the second also Tive dice. (Wrons)
LLaVA-1,S: S:: The image that shows fterent mood of cha Pnaracter& others featuring woman laughing, ... (Wrong)
GPT-4V: The second limage shows difterent mood character from others.... (Correct)
Figure 3: Two case studics in multi-image scenario. MANTIS can understand multiple images and complete the given task, thanks to the 4 multi-image skills learned from MANTIS-INSTRUCT
4 Related Works
4.1 Large Multimodal Models
Large Multimodal Models (LMMs) are mostly denoted as large language models that can understand multiple modalities besides the human language. Some works aim fuse image, audio, video, more modalities with language (Wu et al. 2023b Zhan et al.] 2024), while more works mainly focus
better-combingt vision knowledge with the language (Alayrac et al. 2022 Awadalla et al. 2023 Dai et al. 2023). BLIP-2 has curated a large-scale image captioning dataset, bootstrapping a language model along with a vision encoder to be a powerful multimodal model (Li et al. 2023c). Following this, LLaVA further proposes a cheaper approach lo train a powerful LMM through visual instruction tuning (Huang et al. 2016). LLaVA-Next further optimizes single-image performance, with a cost of an incrcasing number of image tokens per token (Liu et al.] (024)), consuming more than 2k for each image, which is about 4 times of the original LLaVA model. Later LMMs like QwenVL (Bai et al. 2023 CogVLM (Wang et al. 2023), Yi-VL (Al et al., 2024h, etc. all follow similar architecture of LL. However, real-world visual problems way more complex and usually require a sequence of images to describe the situation. Although better performance is achieved in the single-image scenario, it's doubtful whether it's worth the cost.
4.2 Multi-Image Supported Large Multimodal Models
Previous works have also noticed the importance of multi-image ability. Deepmind's close-sourced Flamingo (Alayrac et al. 2022 focuses optimizing in-context learning for a sequence of Image/text pairs but lacking free-from image-text interleaved training examples. Kosmos2 (Peng et al. 2023). Fuyu (Adepi)
2023 both support text-image interleaved format given the model structure, but they did not optimize in multi-image reasoning, and the released inference codes also do not support multi-image as input. Emu2 (Sun) et al. 2023 is a generative multimodal model that supports interleaved text-image inputs, as well as gencrating both images and texts. However, it's mainly focused on in-context examples like Flamingo and auto-regressive image generation, instead of the skills mentioned in (section Another common variant of LMMs that can accept multiple images input is video understanding models, such as Video-LLat LLaMA (Zhang behaves
et al. 2023a). However, shown in the MVBench benchmark, it is also shown that worse than LMMs that can only accept one image as input (Li et al. 2023d), which raises doubts about its multi-image underslanding ability. Different from these works, Mantis aims to optimize the multi-image ability of LMMs guided by the defined skills, co-reference, reasoning, comparison, and temporal understanding, thus equipping the model with better multi-image ability. Our contemporary works on optimizing interleaved ----ext--nttrrleeee multi-image ability include LLaVA VA-Next Interleave (Li et al., (024b) and LLaVA-OneVision (Li et al., 2024a),
which also collects large scale of single-image, multi-image, and video understanding data to train a
11

---

Published in Transactions on Machine Learning Research (11/2024)
powerful vision language model. These two works appear after our work and incorporate our Mantis- Instruct as their training data, demonstrating our dataset's 's value.
4.3 Multi-Image Evaluation
While many benchmarks have been developed for single-image tasks, such as VQx (Singh et al. 2019a
VQA-V2 (Goyal et al. 2016), MMBench (Liu et al. 2023d), and so on, the multi-image evaluation did not capture much atlention until recently. Difference description is the first lask which large-scale data and good evaluation benchmarks are developed for, including works like Spot-the-Diff (Jhamtani & Berg-BBirraatrin Kirkpatrick 2018), Birds-to- Words (Forbos et al., 2019), etc. They are usually evaluated by n-gram metrics like BLEU, and ROUGE, and only involve two images. The later NLVR2 (Suhr et al., 2018) further includes logical reasoning across 2 images into evaluation, focusing on a higher level of cognition ability. Q-bench (Wu et al. 2023a and BLINK (Fu et al.) (024)) are both proposed for low-level multi-image vision tasks, where humans casily do but model struggles. Another set of multi-image evaluations comes from video or image sequence evaluation, including Mementos (Wang et al.) 2024b), MVBench (Li et al.) 2023d), etc. In this case, the model usually needs to take more than 2 images to answer. Recently, there are also attempts to prompt LMMs to evaluate vision-language task (Zhang et al.) 2023b), or image-generation tasks (Ku et al. 2023), which also demands good multi- multi-image reasoning ability. The accuracies of the these tasks of different LMMs can be seen as another form of benchmark for multi-image ability.
5 Conclusion
We propose MANTIS, a new large multimodal model family designed to be able to accept interleaved text-image inputs. MANTIS is fine-tuned on our MANTIS-INSTRUCT, a text-image interleaved dataset consisting of 721K examples, covering crucial skills, including co-reference, reasoning, comparison, and temporal understanding, for multi-image tasks. We also propose Mantis-Eval, a high-quality multi-image evaluation benchmark consisting of 217 examples with an average of 2.5 images per example. Comprehensive experiments show that, MANTIS models effectively acquire the 4 multi-image skills and achieves saate-of-the-art performance 5 multi-image evaluation benchmarks, and preserve decent single-image reasoning performance. We shared insights about the model architecture and inference approaches based on the experiment results. Extending the image sequence context length and compressing the mumber tokens per tokens efficiently are regarded as our future work.
Limitations
Our work mainly focuses on improving the multi-image ability of LMMs. Our collected dataset Mantis-Instruct reformat some existing multi-image datasets, which might still contain some noise due to the quality of the original collection. Besides, we found that most of the examples in the Iantis- tend to be multiple- choice QA questions, where the response tends to be short, resulting in the current version of Mantis models tending to output shorter responses instead of long reasoning texts and falling short in cnstruction-folowiiii ability in the multi-image scenario. Although this problem still exists, and we will leave it to future work by introducing more long-form multi-image data. While enhancing the multi-image ability well, we find out that there is a trend of performance degeneration in the single-image performance compared to the original model Table 6 Some single-image QA datasets have been introduced to alleviate this issue, but the conflicts between multi--maae and single-image ability seem to exist continually. How balance these two abilities will also be our future direction.
have introduced Multi-VQA subset to alleviate this issue,
Societal Impacts
Our proposed model, Mantis, and released dataset, Mantis-Instruct can have substantial impacts on society through the application of Large Multimodal Models. LMMs with multi-image ability can analyze, and reason in real-world scenarios, helping machines, and robots, make decisions automatically, assisting people in website browsing, travel planning based on multiple maps and pictures, etc. We believe Mantis serve
12

---

Published in Transactions on Machine Learning Research (11/2024)
one of the first useful open-source LMMs that make progress in these applications to make these scenarios possible and efficient.
However, there is a potential that our Mantis can generate hallucinations. make the wrong decisions, and fail to reason across complex real-world scenarios. While we have tried to improve Mantis's accuracy and performance in instruction-following and multi-image understanding by carefully training on high-quality Mantis-Instruct, Mantis can still sometimes make mistakes, thus harming its faithfulness a useful LMM tool. There is also the potential for misuse of our model and dataset after make them open source. Though have added proper licenses and terms of usage to our assets, misuse cannot be safely eliminated.
13

---

Published in Transactions on Machine Learning Research (11/2024)
References
OpenAI Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum. Irwan Bello, Jake Berdine, Gabriel Sernadett-Shaping Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luiss Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim 'n Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Hcaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantamu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jomn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David M'ely, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neclakantan, Richard Ngo, Hyconwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipee de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rac, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schmurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shich, Sarah Shoker, Pranav Shyam, Szymon Sidor, Fric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Tan Sohl, Benjamin D. Sokolowsky, Yang Song, Natalic Staudacher, Felipe Petroski Such, Natalie Summers, Tlya Sutskever, Jie Tang, Nikolas A. Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023. URL https://app httppp///.ppp.eemmmcccccccoooo.. Dogpaaattcccccooooooooooo/oo//aa 2577532811
semanticacholar
Adept AI. Fuyn-8B: A Multimodal Architecture for AT Agents. 2023.
01. AI, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xic, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nic, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by Oi.ii, 2024.
14

---

Published in Transactions on Machine Learning Research (11/2024)
AI@Meta. Llama 3 model 2024. http:///t/gggggt/oo/////////////
CARD.md
Yean-Baptist Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza. Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning- ArXiu, abs/2204.144 2022. URL http:///aap..eemmiiscccooooooooe Irg.CorpssIDICDCC........ 248476411
Reza Yazdani Aminabadi, Samyam Rajbhandari, Minjia Zhang, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Jeff Rasley, Shaden Smith, Olatunji Ruwase, and Yuxiong He. Deepspeed- inference: Enabling efficient inference of transformer models at umprecedented scale. SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15, 2022. URL https: ; 7/api ,
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on compuler vision, pp. 2425 2433, 2015.
Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Yitzhak Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openftamingo: An open-source framework for training large autoregressive vision-language models. ArXiv, abs/2308.011 00000, 2023. URL https: //api. sww.nticschohooarrrccooooo.. http://rr...roooopuaaa2221100000
Jinze Bai, Shuai Bai, Shusheng Yang, Shijic Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. ArXin, abss/330..19060 2023. URL http:///p.ssmaaniccchllaaaaa
org/CorpusID :ttp//oppus1:::287557777
Lin Chen, Jinsong Li, Xiao wen Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4y: Improving large multi-modal models with better captions. ArXiv, abs/ abs/2311 /ts/2311..2299, 2023. URL
https://api. sem.amiicsmatiichlllcrrrr wttp:eesscoaaarrrrrrrroeeeee wrwp/oorrsIID 265308687
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ArXio, abs/ Abs/1504.00325, 2015. URL
htttps://apl www.stiiimcoelateeeeeeeeeee httppp/ae.............oooooorrrr 22104555
Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Arxi abss/3305.0655 2023. URL htt/hhttpp//// wwwpppssaaaaniicccaaaaa..oo orw.aaapuorCCCCCCCD 258616266
Tri Dao. FlashAttention-2:: Faster attention with better parallelism and work partitioning. ArXiv, 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finctuning of quantized llns. ArXie, /////2355.14314 2023. URL
htwp:appppeeeannncccccoorrrrrrre org/CorpusID : 258841328
Maxwell Forbes, Christine Kaeser-Chen, Chen, Piyush Sharma, and Serge J. Belongie. Neural naturalist: Generating fine-grained image comparisons. In Conference on Empirical Methods in Natural Language Processing, 2019. URL https://api. www.ssiiscmooonaaa.aaaaaa http://aaa...ee//ooorrpaaa/aaaa0
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. ArXiv, abs obs/2230.099444 2023. URL http:////pp..ssnnnnssccchhhaaaa wttph/oaaarrrrerooosssi:2:///// :t0
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A. Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive, 2024.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. International Journal of Computer Vision, 127:398 414, 2016. URL httpps///ap..sennnnicccccchhrr ottp://tte..acaioooooooo/o/oo/oa pxgg/CorpusDD:80811
:8081284
Ting-Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Girshick, Xiaodong Hc, Pushmeet Kohli, Dhruv Batra, al. Visual storytelling. In 15th Annual Conference of the North American Chapler of the Associalion for Compulational Linguistics (NAACL 2016), 2016.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset. for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6700 6709, 2019.
Harsh Jhamtani and Taylor Kirg-Kirkaatrick Learning describe differences between pairs of similar images. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4024-4034, 2018.
Kushal Kafle, Scott D. Cohen, Brian L. Price, and Christopher Kanan. Dvqa: Understanding data visualiza- tions via question answering. 2018 IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5648-5656, 2018. URL https:://pp. seman wtt.://cchoooooooooooorrsrssaooa
Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo J Kim. Large language models are temporal and causal reasoners for video question answering. In EMNLP, 2023.
Benno Krojer, Vaibhav Adlakha, Vibhav Vincet, Yash Goyal, E. Ponti, and Siva Reddy. Image retrieval from contextual descriptions. In Annual Meeling of the Associalion for Compulational Linguislies, 2022. URL
https://api. www.eticscooolseee..ee orgooog///ooo.ooooooooooo Satg/CorpusDDD ID: 247793003
Max W.F. Ku, Dongfu Jiang, Cong Wei, Xiang Yue, and Wenhu Chen. Viescore: Towards explainable
metrics for conditional image synthesis evaluation. ArXiu, abs/2312.14867, 2023. URL https://apl
semanticscholar. org/ ID: 2665211111
Alina Kuznetsova, Hassan Rom, Neil Gordon Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont- Pont-Tusct, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4. International Journal of Computer Vision, 128:1956 1981, 2018. URL
https://api. wwmanticscoooart......ee.aaaaaaa wet.aaaa..ccccco.....o.oooooooaa :t:p///oppu111:3326666666 53296866
Hugo Laurenccon, Léo Tronchon, Matthicu Cord, and Victor Sanh. What matters when building vision- language models? 2024. URL https: http::::/app..smmantieeeeee sww.ntccscchooa.r...mmmo. org/CorpusID:2295878885
Hugo Laurengon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang, C. Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. ArXiv, abs/2306 03405.0522, 2023a. URL https://api semanticscholar
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi- modal model with in-context instruction tuning, ArXiv, Abs/2305.033725 2023b. URL https://api
semanticscholar ceteriissccolaa..r//rrooooo 258547300
Bo Li*, Peiyuan Zhang*, Kaicheng Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024. URL http:://tttttoooooooo/////////aa
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chumyuan Li. Llava-onevision: Easy visual task transfer. ArXiv, 0er//2488.0333 2024a. URL
/ttps://api.... semanticacholar http://ccchca.t.ooo.oooooaoa/a/a
16

---

Published in Transactions on Machine Learning Research (11/2024)
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next- interleave: Tackling multi-image, video, and 3d in large multimodal models. ArXiu, abs/2407.07888 2024b. URL http://app...ennaticcccch..o.ose httpaoooiia..rrrroouuoiDi:: 27 271088459
Jumnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023c. URL https://aph: www.seeeeacnnnoai.e.o.. https://ppi..emmttiiccho....
org/CorpusID :2560r000 256390509
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. Mybench: A comprehensive multi-modal video understanding benchmark. ArXie, 5bs/2311.17005, 2023d. URL https://api......
wwtp:aaooo//ooo../eeee.e... 265466214
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united vi- sual representation by alignment before projection. ArXiv, abs/2311.10122, 2023a.
URL https
//api. sepiiismmnticscholarr http:///cc..oooooooooooo/roaaaaa hrgp/oorpss1::2222815444
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shocybi, and Song Han. Vila: On pre-training for visual language models. ArXiu, (tss/2211.07533 2023b.
URL https://apt: www.steimmmooollla.....ooooo htttass//ap...eeaatttccchholl set.sttticcttaaa...oooooopasaaaa
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. ar preprint ar Xrx:v:331 075333, 2023c.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint 2rXiv:2306.1455 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.
ArXie, abs/2310.03744, 2023b. URL https://api. www.seiiscoooooa.aao.oeoo http:aaaw....caaa..ooooooo/aaaaa :tt.:oorppp..:::222725000
Haotian Liu, Chunyuan Li, Qingvang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs (bs/2304.08485, 2023c. URL https://app. semanticscholast ottpa/rro/oorrrs11:211111777
Haotian Liu, Chunyuan Lá, Yuheng Lú, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Leo. Llava-next: Improved reasoning, ocr, and world knowledge, Jamuary 2024. URL httpps://llaaa-vl.l.c wwwpaaa.www..ttuu..i.....
2024-01-30-1lav--nes
Yuanzhan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: your multi-modal model an all-around player? Arxi abs/2307. 03388, 2023d. URL
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq R. Joly, and Enamul Hoque. Charlqa: A benchmark for question answering about charts with visual and logical reasoning. ArXiv, abs/2203.10244, 2022. URL
https://api. wtt.appppssaaaacccccooaaa.aaa..a wwt.:/o/ar..rrrrroro:i:::::22:0
Minesh Mathew, Dimosthenis Karatzas, R. Manmatha, and C. V. Jawahar. Docvqa: A dataset for vqa on document images. 2021 JEEE Winter Conference on Applications of Computer Vision (WACV), pp. 2208, 2020. URL httt:://appp.eeaaatccccccolarrrt
2200880200
Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXix:2403.000 1, 2024.
Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. 2024. URL https://api. htt.lattss:/pp...maaniii.. wttpa/accccaaaa.oorrroooooooaaaa
Aitor Ormazabal, Che Zheng, Cyprien de Masson d' d'Autmme, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, et al. Reka core, flash, and edge: A series of powerful multimodal language models. Xrxie preprint arx: 12387, 2024.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos- 2: Grounding multimodal large language models to the world. ArXiv, abs/2306. 14824, 2023. URL
httee///pp..semaaniccccccoooo.e
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021. semanticscholaar
URL httppp///app...tttttcccc.tt..... hagaaarrrooggooopuuuui2/:211111.
Christoph Schuhmann and Peter Bevan. 20l-spptaviisoonccaattoss www.aaapppooooo--ll..aaaaa https://nugginggace.coo http://oooooooooo////ooogoooooor 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8309 8318, 2019a. http::ttt///apaa semameicccaantnccoeeoo..ooo tttpppppp.//pppp..eeaaatttttaaaa org/Corpus 85553602
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8317-8326, 2019b.
Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. Milebench: Benchmarking milms in long context. ArXau, 18532404.18532, 2024. URL http::://pp...mmaatiiccccm
II:
Alanc Suhr, Stephanic Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. ArXio, abs/1811.00491, 2018. URL http://t///ppp..mmaatcccc/c//// sewanticcchhihhhrr
org/CoorpusDD:53178555
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. ArXie, hbss/232.112886 2023. URL httpss//app. http//pp...smanicsschhollllr org/CorppsID::eeer 266374640
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint or 211: 221122211110 11805, 2023.
Fei Wang, Xingyu Fu, James Y. Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, Tianyi Yan, Wenjie Jacky Mo, Hsiang-Hui Liu, Pan Lu, Chunyuan Li, Chaowei Xiao, Kai-Wei Wei Chang, Dan Roth, Sheng Zhang, Hoifung Poon, and Muhao Chen. Muirbench: A comprehensive benchmark for robust multi- image understanding. ArXiv, abs/2446.09411, 2024a. URL /ttps://apii.smmnticcese
Mwgg/CorpusID:22040000
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models. ArXiu, abs/2311.0300 2023. URL https:
//api Stapiissmanticscholla. careollaarrrggg///ee wwwo/////ppeeee Corpus Ittp/Corpps.D::265000 26503422288
Xin Eric Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan fang Wang, and William Yang Wang, Vatex: A large- scale, high-quality multilingual dataset for video-and-language research. 2019 IEEE/CVI International
Conference on Computer Vision (ICCV), pp. 4580 4590, 2019. URL ttttp:::/app..smmattiie
Btc//oorpSID::023521111
Xiyao Wang, Yuhang Zhou, Xiaoyu Liu, Hongjin Lu, Yuancheng Xu, Feihong He, Jaehong Yoon, Taixi Lu, Gedas Bertasius, Mohit Bansal, et al. Mementos: A comprchensive benchmark for multimodal large language model reasoning over image sequences. Xie preprinl arXin:2401. 2024b.
Bo Wu and Shoubin Yu. Star: A benchmark for situated reasoning roal-world videos. In NeurIPS Datasets and Benchmarks, 2021. URL wtt.tt/tt.////pp...eaaaacaaaa.aa wtt.a//ccc.aa....oooooor//aaaaaa
18

---

Published in Transactions on Machine Learning Research (11/2024)
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, and Weisi Lin. Q-bench: A benchmark for general-purpose foundation models on low-level vision. ArXiv, abs/2309. 2023a. URL
hwwpaaatiicccoolaoooo//oooorrrpo Drg/CorpusID:
262824606
Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guanglao Zhai, Shiqi Wang, and Weisi Lin. Towards open-ended visual quality comparison. ArXiv, abs/2402.166411 2024. URL I
.rrgc//ooaarrrg/////oooooo org/
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-ooany multimodal 11m. ArXiv, 205//339.05519, 2023b. URL https://api. aammmaaticcccolaa..or///ooaalppp
Junbin Xiao, Xindi Shang, Angela Yao, and Tat seng Chua. Next-qa: Next phase of question-ansswrring
explaining temporal actions. 2021 IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CVPR), pp.
2021. URL https://app http:///app.sseammccccchoolaaraa wtt.://o......ooooppopaaaa////00
Shih yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. ArXiv, abs/ abs/2402.00933,, 2024. URL
https://ap1s seapp.ssemanniscchla.a..
http:/oorpss:::276667777
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. ArXic, abs/2311.16502, 2023. URL
http:/pps..emmttccccchhhhholaa wwgp/orppssI. 2654666555
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. 2023 IEEE/CVE International Conference on Compuler Vision (ICCV), pp. 11941 11952, 2023. URL https://api. www.ntccccnooaaa......ooooomaaaa wwtp://oooo....25577777..
Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, Hang Yan, Jie Fu, Tao Gui, Tianxiang Sun, Yugang Jiang, and Xipeng Qin. Anygpt: Unified multimodal Ilm with discrete sequence modeling. ArXiv, abs/2402.1222 2024. URL https://api. www.atiissooollate......ei wwwpacholaar.rg/ooruussssslleeee
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tumed audio-visual language model for video understanding. ArXiv, Abs/2306.002855 2023a. URL TTR.tttps:://pp/
259075356
Xinlu Zhang, Yujie Lu, Weizhi Wang, An Yan, Jun Yan, Lianke Qin, Heng Wang, Xifeng Yan, William Yang Wang, and Linda Ruth Petzold. Gpt-4v(ision) as generalist evaluator for lision-language tasks. ArXio, abs/2311.0136) 2023b. URL
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal e4: An open, billion-scale corpus of images interleaved with text. ArXiv, abs/ abs//2304.06939 2023. URL https://api... http:/ppp...mmatiicccchhoolll org/
25817467
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjac Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interleaved with text. Advonces in Neural Information Processing Systems, 36, 2024.
19

---

Published in Transactions on Machine Learning Research (11/2024)
A Appendix / supplemental material
A.1 Details of Mantis-Instruct subsets
We construct MANTIS- INSTRUCT from multiple publicly available datasets. Detailed statistics are shown in Tabll 1| We use a similar dataset format with LLaVA's, where each data item contains multiple images and multiple turns of QA pairs are gathered. We report the number of examples, the number of images per item, the average conversation turns, and the average length. We describe the task type of each dataset and the processing techniques used in the following:
LLaVA-665k-multh Besides the mentiomen multi-image reasoning tasks, we also manually reformat the ALaVA-655k dataset from LLa into 'synthetic' multi-image dataset. Since each original data item consists of multiple QA pairs about a single image, we randomly merge 2 to 4 data items into a multi-image data item. Then we add proper image denotation like 'For the second image, ... for each question, and shuflle all the QA pairs to form the final multi-image data item. Answering each question in this "synthetic' dataset does not require reasoning across images, but demands high image co-reference ability. Besides, it also helps avoid forgetting dingle-image ability in multi-image scenarios and increases instruction-folloiigg ability.
Co-Instruct (Wu et al., 2024): Co-Instruct is a dataset curated to compare the quality of 2-4 images in open-ended answer or multi-choice QA format. It's similar to the tasks in the Qbench (Wu et al.) 2023a) NLVR2 (Suhr et al., 2018): NLVR2 is a natural language reasoning datasct grounded in images. The lask is to determine whether a statement. is true or false given the visual contents of 2 images. NLVR2 covers many linguistic phenomena, such as cardinality, existential quantifiers, universal quantifiers, etc.
Drcamsim (Fu et al.) 2023): DrcamSim proposes a dataset to train model to judge image similarity. We reuse their datascts, and the task to judge which candidate image is more similar to a reference image. ImageCoDe (Krojer et al.) 2022):: ImageCoDe is multimodal challenge called image retrieval from contextual descriptions, where a multimodal model is required to retrieve/select the correct image from a set of 10 minimally contrastive images. For each image, 9 similar images are retrieved from Open Images Datasel V6 (Kuznetsova et al.] 2018) using CLIP encodings.
Cattrass-Captinn Inspired by the paradigm of contrastive learning, we reformat existing image captioning tasks into a multi-image version, where 2 tasks are defined. Given multiple images, the first task we defined is to judge which image matches the provided caption, and the second task to generate a caption for a denoted image. We randomly select 2 to 8 images along with their captions to form single item, then apply the template of the 2 tasks defined. We used the images and captions provided by ShareGPT-4V (Chen et al. 2023) and LAION GPT-4V captions from LVIS (Schuhmann & Bevan 2023).
Spot-the-diff (Jhamtani & Berg-Kirkpatrick 2018): This dataset requires the model to generate the difference dcscription given 2 images. We use the procossed version from MIMICIIT (Li ct al. 2023a).. LRV-multi (Liu et al., 2023a): This a single-image dataset used to mitigate hallucinations of LMMs. We process similarly to L.L. V.VV-6655--ullt to get a "synthetic' multi-image dataset.
Bords-to-Words (Forbes et al., 2019): The task of this dataset is to generate different descriptions and give 2 images of different birds. We directly take the training split for training. Besides, we prompt CPT-3.5-turbo to generate a possible question given the reference answer, so we can get more diverse question pool instead of always asking the model to generate the difference description. Multi-VQA: This dataset contains multi-image reasoning QA pairs, where each question is generated by GPT-4 based on the captions of the images. Each question is required to involve at least two images. Images and captions are sampled from ShareGPT-4V (Chen et al. 2023). The prompt template in Table 2
Video QA: We also include some video question answering datasets, including NExTQA (Xiao et al., 2021), STAR (Wu & Yu 2021], I, and visual-story-telling (Huang et al.. 2016). nextqa and star are the processed version from F Flipped-VOL VQA (Ko et al.) 2023). The sisual-gtory--elling are the processed version
from MIMIC-IT (Li et al.) (023a).
Single-image- VQA: Instead of focusing only on multi-image lasks, we claim that it's necessary to also include some lingle-image VQA dataset to avoid catastrophic forgetting on single-imas image tasks, We included some of the single image data from LLL as well as DocVQA (Mathew et al.] (200)), DVQA (Kafle ct al. 2 and ChartQA (Masry et al.) (022) to enhance its ability on diagrams and OCR. In practice, since the size of DVQA is too large, we only use 30k out of 200k examples for the training.
20

---

Published in Transactions on Machine Learning Research (11/2024)
A.2 Investigation of Mantis's capability open-ended tasks
To assess Mantis's performance open-ended generation tasks, evaluated it image captioning (using the CoCo-e017-Lite dataset) (Chen et al. 2015) and video captioning (using the Vatex dataset) (Wang et al. 2019 For evaluation, we utilized Imms-eval Li* et al. (2024). The results are presented in Table 9 and Table 10 As shown in these tables, Mantis- Idefics2's captioning performance improved after training on Mantis-Instruct compared to its previous checkpoint. Idefics2, though still fall behind of the expert captioning models in these arcas.
Mantis-Idefics2's
We attribute this improvement to the design of our dataset, which encourages the model to differentiate between images and better understand semantic information within visual content. Notably, contrastive learning of this kind is often limited in vision-language pre-training datascts, so fine-tuning on Mantis- appears to enhance the model's comprehension of visual content further.
Model
BLEU-4
BLEU-3
BLEU-2
BLEU-1
METEOR
ROUGE-L
CIDEr
Idefics2
0.1890
0.2810
0.4128
0.5896
0.2453
0.4545
0.7453
Mantis-Idefics2
0.2288
0.3266
0.4570
0.6155
0.2670
0.4906
0.8210
Manis-SIGLIP-LLLMMA
0.2474
0.3479
0.4824
0.6500
0.2692
0.5045
0.9042
Table 9: Results of Mantis on the image captioning task (CoCo-2017-Litg)
Model
BLEU-4
METEOR
ROUGE-L
CIDEr
Idefics2
0.1454
0.1448
0.3183
0.2307
Idefics2
0.1702
0.1573
0.3724
0.2667
Manti-SICLIP-ILLMMM
0.1545
0.1431
0.3607
0.2263
Table 10: Results Mantis on the video captioning task (Vatex)
A.3 Investigation of Mantis's capability on long- long-context scenarios
While claiming Mantis to be a model that can accept text-image interleaved inputs, it's interesting to investigate how Mantis will perform in long-context scenarios.
We first conduct experiments see how Mantis's performance will change as the number of maaage/fframee increases. We Mantis- on MVBench using 2/4/8/16 frames and report the performance in Table 11 It turns out that Mantis's performance is stable across various numbers of sampled frames. What's worth noting is that Mantis can use only 2 frames to achieve 48.38 accuracy, which means that most of the questions in the MVBench can be answered by keyframes. As the number of frames goes up from 2 to 8, the accuracy also goes up to 51.38 accuracy. The performance under 16 samples is 1 point lower than the performance under 8 frames, we hypothesize that the redundant information might distract Mantes from giving the current answer.
Number of frames
2
4
8
16
MVBench Acc
48.38
50.02
51.38
50.2
Table 11: Mantis-Idefics2's performance MVBench with different number of uniformly sampled frames.
As shown in Table 4 MileBench consists of examples with an average of 15 images per example and a sequence length ranging from 1,000 to 9,000 tokens, making it well-suited for evaluating Mantis's long-context ability. As shown in [Tblle 5 Mantis-SIGLIP achieves 47.5 on MileBench, only 5.5 points behind GPT-4V. We do believe this is good evidence that Mantis maintains competitive multimodal long-context. ability.
21

---

Published in Transactions on Machine Learning Research (11/2024)
A.4 Comparison with MIMIC-IT
MIMIC-IT is also a multi-modal dataset that aims to boost the model's ability for in-context learning, which also contains multiple images for most of the QA pair (Li et al., 2023a). To investigate how well Mantis- Instruct outperforms MIMIC-IT, we conduct a fair comparison between the Otter and Mantis-Flaming As shown in Table 12, Otter is tuned from the Open-Flamingo using MIMIC-IT as the instruction-tuning dataset, and one of our Mantis's variants, Flamii,,,,,,n,,, also tuned from the Open-Flamingo but instead using our
as the instruction-tuning dataset. Therefore, Mantis- Flamingo and Otter are a fair comparison where they share the same pre-training dataset, MMC4+LAION, and trained different instruction-follovin datasets, MIMIC-IT and Mantis-Instruct, respectively.
We report the performance in Table 13 It turns out that Mantis- Mantis-Fllaminng Flamingo is better than Otter in all these benchmarks, based on the same pre-trained model but different instruction-tuning datasets. This proves the superior quality of Mantis-Instruct compared to the MIMIC-IT. This mainly because Mlantis-Intrrun is designed to equip models with the essential abilitics for multi-image reasoning, while MIMICIIT mainly designed for multimodal in-context. reasoning.
Model
Pre-training MMC4, LAION MMC4, LAION
Instruction-Tuninne
Size
Otter
MIMIC-IT Mantis-Instruct
9B
Mantis-Plamingo
9B
Table 12: Comparison of Mantis-Flamingo and Otter's training datasets. Mantis-Flaminge and Otter are a fair comparison where they share the same pre-training dataset, MMC4+LAION, and trained on different. instruction-follwwer datasets, MIMIC-IT et al. 2023a and Mantis-Instruct, respectively
Model
NVLR2
Q-Bench
Mantis-Eval
BLINK
MVBench
Average
Otter
49.15
17.50
13.29
36.26
15.30
26.30
Mantis Mantis-Flamingo Flamingo
52.96
46.80
32.72
38.00
40.83
42.26
A over Otter
+3.81
+29.30
+19.43
+1.74
+25.53
+15.96
Table 13: Comparison of Mantis-
Mantis-Flamingg Flamingo and Otter's (Li et al. 2023b) performance in multi-image
benchmarks. The result numbers are sourced from Table
22


Reviews:
[{"review_id": "VsTT0u5Qic", "reviewer": "Reviewer_8Nmm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is backed by a large number of carefully controlled finetuning experiments and comparisons across 8 different evaluation tasks (both “held-in” and “held-out” with respect to the Mantis-Instruct data). I specifically want to call attention to the following experiments that I thought were particularly well-executed:\n\n1. “Merge” vs. “Sequence” Input (Sec. 3.4, Table 5) — This is a necessary and often overlooked experiment in the MLLM space that explicitly compares “merging” multi-image contexts into a single image (i.e., as a large resolution mosaic) against explicitly modeling individual images as a sequence. I am very convinced by the results that demonstrate the superior performance of Mantis-tuned MLLMs over strong “single-image” SOTA MLLMs like LLaVa 1.6. Nice work!\n\n2. “Is Multi-Image Pretraining Necessary (Sec 3.6, Table 7) — Like many others in this space, I’ve questioned the value of pretraining on massive datasets like OBELICS or MMC4 for inducing multi-image reasoning in MLLMs; I really appreciate how this set of experiments not only compares Mantis-tuned MLLMs to existing interleaved models like OpenFlamingo and Idefics2, but also systematically ablates multi-image pretraining on OBELICS-100K, and shows no meaningful improvements for downstream tasks. This is a very nice and useful result.\n\n---\n\nHowever, there are a few areas that I found that paper lacking. First, beyond the construction of the Mantis-Eval benchmark (just 217 examples), there is very little that’s new on the artifact side: all of Mantis-Instruct was compiled from existing datasets (albeit gently reformatted / transformed to suit the instruct tuning format). Similarly, the remaining 7 evaluation benchmarks are similarly compiled from existing work. \n\nNote that is not a problem! I specifically don’t want the authors or other reviewers to think I am penalizing this work for a lack of novelty; rather, I want to call this out explicitly to ask the authors to refocus the framing and details in the paper around the **really strong and interesting experimental results** as well as providing more detail about the individual dataset curation process. Specifically, Section 2.3 mentions a series of heuristics that were seemingly important during dataset curation (e.g., importance of image denotations, positions of the <image> placeholder token) — I would *love* to understand these heuristics and the actual impact of these a lot more. \n\nI also think there’s a lot of space for deeper analysis into the fidelity of multi-image reasoning — how do Mantis-tuned models generalize as the image context grows larger? Is that tied to the relative proportions of (# images / dataset)? When processing redundant image contexts (i.e., frames from a video), do Mantis models need all 8 / 16 / 50 frames, or are the models attending to just a few “keyframes”? I think these analyses are pretty straightforward to do with the existing suite of models, and would really help a lot in solidifying the experimental/empirical contributions of this work.\n\n---\n\nBeyond the above, I found the taxonomy (coreference, comparison, reasoning, temporal understanding) of different “multi-image” reasoning capabilities a bit forced/unmotivated. I understand the desire to categorize each of the different component datasets, but in a lot of cases, it’s not immediately clear that the distinction is meaningful. I think minimally, just splitting into three categories (single-image reasoning, intra-image reasoning, temporal reasoning) might fit a little bit better?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This is a clean, straightforward paper that introduces a new dataset, evaluation benchmark, and pretrained models for the task of visually-conditioned language modeling from *interleaved, multi-image* contexts. The work identifies that existing multimodal large language models (MLLMs) are deficient in their ability to reason over multiple image contexts (e.g., paired images, frames sampled from a video), and identifies four target “capabilities” that a multi-image/interleaved MLLM should have: \n\n1. Coreference: Ability to localize natural language references to the appropriate image in a multi-image context (e.g., “in the second image”). *Nit: I would rename this to be “instance grounding” or “image grounding” — coreference means something very different in NLP)*\n\n2. Comparison: Ability to compare/contrast features *across* multiple images (e.g., spotting the difference between image pairs).\n\n3. Reasoning: Ability to perform “deeper” types of multi-image reasoning across multiple image contexts, such as logical/compositional reasoning, counting, free-form QA.\n\n4. Temporal Understanding: Ability to model the arrow of time, answering questions about ordered frames sampled from videos, still images sampled from comic strips, etc.\n\nTo tackle the above, the authors curate and/or gently transform existing datasets into a format amenable to multi-image (interleaved) language modeling. The aggregated dataset — labeled Mantis-Instruct — consists of 721K (text prompt, multi-image context, text response) triples, with a mix of examples that consist of ~5 images on average. \n\nIn parallel to the curated Mantis-Instruct data, the authors assemble a multi-image (interleaved) evaluation benchmark consisting of 7 existing benchmarks, along with an 8th newly proposed benchmark — Mantis-Eval — consisting of 217 multi-image examples provided by crowdworkers (using images from Google Search).\n\nTo evaluate the impact of the Mantis-Instruct dataset, the authors finetune several pretrained MLLMs (including Fuyu, OpenFlamingo, Idefics2, and two LLaVa variants), in both the full-finetuning and LoRA regime. Through carefully constructed ablation experiments and comparisons to existing (multi-image) MLLMs, the work clearly demonstrates that Mantis-Instruct is a compact and useful dataset for inducing multi-image reasoning abilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "As it stands the work presents strong empirical contributions, a nice new instruct tuning dataset, and a multi-image evaluation benchmark that has already been seeing wide adoption in recent papers introducing new MLLMs (e.g., LLaVa-NEXT Interleave: https://arxiv.org/abs/2407.07895). \n\nI definitely advocate for accepting this work, but would love for authors to consider the reframing / additional analyses above. Specifically:\n\n1. Understanding the impact of the curation heuristics (importance of image denotations, positioning of <image> token) on downstream performance, as alluded to in Section 2.3.\n\n2. Deeper analysis of how Mantis models use longer multi-image contexts (generalization as a function of K images, relationship to component datasets in Mantis-Instruct)?\n\n3. Analysis of how Mantis models use redundant/video contexts (e.g., what frames do they attend to, what types of MLLM inductive bias is better for longer-context videos).\n\nI would also suggest adding significantly more detail about how Mantis-Eval was collected/constructed; if the authors could provide the interface/process crowdworkers used to produce the 217 examples, that’d be great.", "broader_impact_concerns": "The existing Societal Impacts statement does a good job of capturing the general potential for misuse of MLLMs in general. Given that most of the data/evaluations in this work use existing image/text datasets (with proper licenses and attribution), I think most of the ethical concerns I have are addressed.\n\nOne thing — the Mantis Eval benchmark created in this work mentions using Google Search to source images. I assume that the authors limited the images to publicly available/creative-commons images — if so, I think adding a sentence about this would be great. If not, then I would want to flag the Mantis-Eval data for additional ethics/licensing review.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C2235bheIA", "reviewer": "Reviewer_berU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* *Crucial Research Problem*: The paper addresses an important gap in the field by shifting the focus from single-image instruction tuning to multi-image settings, which better reflects real-world applications that require reasoning over multiple images.\n* *Novel Dataset*: The authors introduce Mantis-Instruct, a novel dataset specifically tailored for multi-image instruction tuning. This contribution is significant, as it provides a valuable resource for training and evaluating vision-language models in a multi-image context.\n* *Open-weight Models*: The paper releases a family of LVLMs fine-tuned with the proposed dataset, demonstrating the effectiveness of their approach across multiple models. Additionally, the released models provide valuable resources for supporting future research.\n* *Strong Empirical Results*: The paper conducts an extensive evaluation across multiple multi-image datasets and tasks, including newly proposed benchmarks, offering strong empirical evidence to validate the effectiveness of their method.\n\n**Weaknesses**\n* *Comparison with Concurrent Work*: The paper lacks a comparison or at least a brief discussion of recent concurrent work on multi-image instruction tuning, such as LLAVA-OneVision [1] and LLaVA-NeXT-Interleave [2]. \n* *Open-ended Evaluation*: The evaluation primarily focuses on multiple-choice question answering (MCQA) tasks, which may not fully capture the model's capability in diverse real-world multi-image scenarios. Broader evaluations on generation tasks like image captioning or open-ended visual question answering could strengthen the findings.\n* *Influence of Interleaving Strategy*: The paper emphasize the interleaving format of image and text during training, but a more thorough analysis of its impact on model performance is missing. Exploring how different input formats influence outcomes would provide valuable insights into optimizing training processes.\n* *Data Statistics*: The paper mainly provides the detailed data statistics by their source datasets. A more comprehensive breakdown of the Mantis-Instruct dataset, including statistics on the types of images, distribution of questions, complexity, and diversity, would give a better insight into the dataset's representativeness and utility.\n\n[1] Li, Bo, et al. \"Llava-onevision: Easy visual task transfer.\" arXiv preprint arXiv:2408.03326 (2024).\n\n[2] Li, Feng, et al. \"Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models.\" arXiv preprint arXiv:2407.07895 (2024).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper addresses the instruction tuning of large vision-language models (LVLMs) in a multi-image setting, bridging the gap between traditional single-image instruction tuning and real-world applications that require reasoning over multiple images. The authors introduce a new dataset, Mantis-Instruct, specifically designed for this purpose, along with details of their training process. Using this dataset, they fine-tune a family of LVLMs, demonstrating improved performance across several multi-image datasets and benchmarks, including a novel dataset introduced in the paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, this paper makes a significant contribution and is worthy of acceptance. The following points, if addressed, could further enhance its impact.\n* A detailed discussion of previous and concurrent work on multi-image instruction tuning, such as VILA and LLaVA-OneVision. It would contextualize the contribution and ensure the novelty of the approach is clearly established.\n* Expanding the evaluation beyond MCQA to include open-ended tasks would strengthen the work. While not critical for acceptance, this would significantly enhance the generalizability and impact of the paper’s findings.\n* Providing more detailed statistics on the Mantis-Instruct dataset would strengthen the paper. While not critical for acceptance, these details would improve the transparency and utility of the dataset for future research.\n* A deeper analysis of the influence of interleaving images and text during training could strengthen the work. Although not critical for acceptance, understanding the impact of different interleaving strategies could offer valuable insights for future data curation.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uf7BZQXWqS", "reviewer": "Reviewer_Eh2y", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n\nClear Motivation: The paper identifies a relevant gap—multi-image capabilities in LMMs—and aims to address it through instruction tuning, rather than relying on extensive pre-training.\n\nComprehensive Evaluation: Mantis is tested on both multi-image and single-image benchmarks, showing competitive or better performance than other models in the same category. The results are backed by strong empirical data from diverse benchmarks.\n\nDataset Construction: The creation of Mantis-Instruct, a dataset that systematically targets multi-image skills (comparison, co-reference, reasoning, and temporal understanding), is a well-thought-out contribution. It enhances the ability to test models on specific visual reasoning tasks.\n\nEfficiency Claims: Mantis reportedly achieves superior performance with significantly fewer data and computational resources than Idefics and OpenFlamingo, which is a notable achievement in an era where scaling data is a challenge.\n\n- Weaknesses\n\nBaselines of the Dataset\n\nThe idea of multi-image instruction tuning is not necessarily novel.  Though the previous works focus on in-context learning capabilities unlocked by those multi-image instructions, these previous works should be compared as baselines, given the paper claims the dataset as a contribution. An obvious baseline is MIMIC-IT, which the paper borrows data from.\n\nLimited Analysis of Failure Cases\n\nWhile the paper emphasizes Mantis’s strengths across various benchmarks, it does not provide sufficient analysis of its failure modes. In particular, it would be useful to understand how and why Mantis underperforms in specific tasks. For example, is the degradation in performance on certain single-image benchmarks due to its focus on multi-image tuning? Providing qualitative or failure case analysis could help guide future improvements. Generally, more visual analysis will be appreciated.\n\nPotential Over-Reliance on Specific Skills\n\nThe focus on four core skills (co-reference, comparison, reasoning, and temporal understanding) might overlook other important multimodal reasoning abilities. Additional exploration of tasks involving spatial reasoning or higher-level semantics (e.g., abstract reasoning across images) could help further generalize the model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Mantis, a model family designed to improve large multimodal model (LMM) performance on multi-image visual language tasks via instruction tuning. It claims that Mantis achieves state-of-the-art (SoTA) performance on multiple benchmarks while using significantly smaller datasets compared to models pre-trained on massive datasets, like Idefics. The authors propose Mantis-Instruct, a dataset focused on multi-image tasks, and perform extensive evaluations across both multi-image and single-image benchmarks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Dataset Baselines: Compare the proposed Mantis dataset with existing multi-image instruction tuning datasets on a fixed base model.\n\nFailure Mode Analysis: The addition of qualitative analysis—particularly on tasks where Mantis underperforms—would provide valuable insights for further tuning and refinement of the model.\n\nLonger Output Tasks: Including tasks that require long-form reasoning (rather than shorter multiple-choice answers) would better evaluate the model’s ability to handle complex multi-image scenarios.\n\nScalability and Efficiency Metrics: Provide more concrete details on how the model performance scales with task complexity (e.g., the number of tasks, the number of images, or sequence length).\n\nChallenges of Dataset Construction: Describe the challenges when making the dataset, and how they were addressed in the study or how they may be addressed in the future.", "broader_impact_concerns": "The broader impacts are well stated and addressed in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers recommended acceptance, based on the reasons summarized under "Claims and Evidence" above. I agree with this consensus.

I thank the authors for already revising their manuscript to address the reviewers' comments, especially for the additional experiments in Appendix A. One minor comment: In the sentence mentioning LLaVA-Next-Interleave and LLaVA-OneVision in Section 4.2, it would be more precise to note that these works appeared after Mantis, which would also explain why these works incorporate Mantis-Instruct.
Decision: Accept (poster)