Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models
Authors: Byung-Kwan Lee
Chae Won Kim
Beomchan Park
Yong Man Ro
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language and Vision Model
Primary area: other

Abstract: The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models.

Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models
Byung-Kwan Lee KAIST leebk@kaist.c ac.
Chae Won Kim KAIST chaewonkim@kaist.ccm ac.
Beomchan Park KAIST bpark0810@kaist.com ac.
Yong Man Ro KAIST
ymro@kaist ac
Abstract
The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the per- formance gap with powerful closed-source LLVMS. These advancements are attributed multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math prob- lems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present new efficient LLVM, Mamba-based
traversal rationales Meteor), which leverages multifaceted rationale to en- hance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce new concept traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained gen- erate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple eval- uation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models. Code is available https://gihuuh.co/BByngggK weane////eeaeeeeeeeeeeeeee
Introduction
Following the successful zero-shot achievements of instruction-tuned large language models (LLMs) T 22 visual instruction tuning (3) has spurred the rapid development large language and vi-
sion models (LLVMs). The emergence of closed-source LLVMs, such as GPT-4V [4], Gemini-Pro (5].
and Qwen- VL-Plus [6), has prompted several studies create high-quality question- -nnwer visual instruction tuning datasets [I and to scale up the model sizes of open-sooure LLVMS [OO 12 15)
aiming compete with their closed-source counterparts by leveraging the scaling law [66
2
19 20 and dividing images into smaller sections [4 10 21 to improve image perception capabilities. Additionally, some studies have utilized additional vision encoders (22-25) such as EVA-CLIP (26). DINOv2 [27). SAM 28 and LIP 29] Various computer vision models 30 34 have also been employed
Recent research trends focus on enhancing image resolution 18
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Meteor
Meteor
2330
2253
2123
2033
1630
MME
MMB
DIED
Metear
Meteor
50-
Levertent (TA
35-
MathVista
A12D
25
Figure Across 7B to over 110B parameters, comparing lots of open- and closed-source LLVMs with
Meteor on MME 35 MMB (36), Math Vista [37. and AI2D [38] requiring diverse capabilities for image understanding, common-sense knowledge, non-object concept understanding, etc.
for tasks such segmentation, detection, scene graph generation, and optical character recognition (OCR) to enhance LLVMs' answering capabilities with the help of external perception information.
These efforts, along with the curation of high-quality visual instruction datasets, have significantly reduced the performance gap between open- and closed-source LLVMs across numerous evaluation benchmarks and have even led to superior performances developments are credited to the multifaceted information necessary for a wide range capabilities. This encompasses fundamental image understanding, real-world knowledge of common- and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions.
benchmarks. These successful
Inspired by the key importance of the multifaceted information, we explore possibility of designing efficient LLVMs that implicitly embed it as a form of multifaceted rationale (See Appendix Aoro more details), without significantly increasing model size and without using additional explicit vision encoders and computer vision models during the inference phase. Hence, we present
efficient LLVM, Mamba-base traversal of rationale Meteor), comprising two core components: the Mamba architecture [39] and a multimodal language model (MLM) based on a pretrained large language model (LLM). The multifaceted rationale has rich information for achieving diverse capabilities, so its length is inherently long. This is why we employ the Mamba architecture. hereinafter referred to as Meteor-Mamba, which takes advantage of embedding lengthy input. serves embedding module for the rationale, enabling Meteor-MIMM the MLM component, to address questions with the help of these embedded rationales. When conveying the knowlede ledgledge of embedded rationales from Meteor-Mamte to Meteor-MLM, we introduce new concept of traversal of rationale, which spurs embedding of long sequential rationales.
To ensure that Meteor encompasses diverse capabilities for lision-language tasks (c.g., image understanding, common-senss knowledge, charts, diagrams, documents, signs, symbols, and math problems), we gather M question-answe pairs from existing visual instruction tuning datasets: ShareGPTAV-Caption/Inniit
(20). GLLaVA-Aliga/Inttnne [40]. and Math- Vision//nstruct/Plus [11 43 Subsequently, we utilize
(7). MiniGemini-IIstruucl [TO], Pio-Dowwnstremm/Reasss
the light and fast Claude Haiku API 4 to generate detailed and comprehensive rationales tailored for the collected M question-anm answer pairs. These rationales are carefully filtered by human reviewers with the aid of GPT-4V, resulting in 1. cnestioonrationaal-anss
triples (Appendix A).
Using the question-rationale rationale pairs the curated M. triples, the first training step involves training Meteor- Meteor-Mamba and miscellancous projectors, i.e., vision projector and tor projector. During

---

SQA-IMG
SQA-IMG
08.3
60.3
MM--Vet
$7.3
MathVista
MMStar $2.8
06.6
MathVista
70.1
39,7
87.1
LLaVA-W
185
MME
ChartQA 78.5
34.3
1858
1033
1015
2229
2103
2229
6412
68.6
20
72.3
78.2
68
77.0
70.9
MMB
AI2D
77.9
$2.9
MMB
75.0
SEED-IMG
SEED-IMG
Somini
(a) Open-source LLVMs
(b) Closed-source LLVMs
Figure 2: Overall comparison of Meteor compared with other open- and closed-source LLVMs.
this step, Meteor-Mamba trained to embed long sequential rationales. the second training step, all components of Meteor are trained using the question-answer pairs in the curated 1.1M triples. Through these steps, we demonstrate that Meteor significantly improves vision- langual performance, shown in Figure II compared with other open- and dlosed-source LLVMs, on numerous benchmarks requiring diverse capabilities. As illustrated in Figure these results advocate for the possibility of building efficient L.VVMs with a multifaceted rationale, beyond the scope of scaling model size, additional vision encoders, and multiple computer vision models.
Our contribution can be summarized into main aspects:
We introduce new efficient large language and vision model (LLVM), Mamba-hased traversal of rationale @ Meteor), which comprehends long sequential rationales under new concept of traversal of rationale and predict answers with the help of the rationale.
Despite its efficient model size. Meteor showcases significant advancements across various evaluation benchmarks requiring diverse capabilities for image understanding. common-sense, non-object concepts, and more.
Related Works
Rationale-Guided Prediction. Behind the answers of large language models (LLMs), rationale
has played significant role in enbancing answering capabilities across myriad of natural language processing and vision language tasks various forms such as (a) human annotation (45) 46), (b)
knowledge distillation [47-5T] and (c) chain-of-thougl! (CoT) [52+55]. Rationale mimics the human thought process, providing explanations or justifications before answering questions. (a) Strout al. [45] and Lu et al. [46] integrate human-ane annotated rationale with LLMs to fortify their performance, thereby enhancing model robustness against out-of distribuutoon scenarios with human annotations. (b) In knowledge distillation, rationale is used to effectively distill LLMs 51 into smaller language models. They first extract rationale from LLMS and then fine-tune smaller language models with the extracted rationale, demonstrating their efficacy through various evaluation benchmarks. (c) To directly apply rationale to LLMs, researchers have used a 'think sstpbby-tepp' prompt called Chain of Thought (CoT) [52]. When combined with few-shot learning, CoT clicits step-by-step rationale directly from LLMs for an input question followed by series questions, rationales, and answers used few-shot examples. This has been further streamlined by automating rationale generation in CoT prompts, as explored by [53] and [54] eliminating the need for human-annotated rationale for few-shot examples.
Similar to (b), we also leverage the power closed-source LLVMSs but, contrast, employ another model, Meteor-Mamba, to embed the rationale, instead of directly training Meteor-MI.M generate the created rationales. other words, we separate the roles of the models where Meteor-MI.M generates the answer, while Meteor-Mambe embeds the rationale, which encompasses diverse

---

capabilities: fundamental image understanding, incorporation of real-world knowledge of common- sense, understanding of non-object concepts (e.g., charts, diagrams, symbols, signs, and math), and following systematic step-by-step procedures for solving complex questions. Once accompanied by LLVMs, their answer capabilities are expected to improve with the help of the rationale.
Large Language and Vision Models. Following the emergence of visual instruction tuning datasets created by LLaVA 3
g 12] and InstructBLIP (9]. there has been rapid development of large lan-
guage and vision models (LLVMs): Shikra [36]. IDEFICS [57]. Qwen-VI VL [6] MiniGPT-4 [58).
Otter [59], mPLUG-OW [60] 19]. ShareGPT4V 77. LLaVA-XTuume
[6]] Intern- XC 62] MM1 14
MiniGemini 10, Intern Families [5]
63 along with efforts to gather or curate high-quality visual
instruction tuning datasets for various purposes: ShareGPT4V 77. ALLa VA [64], MiniGemini 10]
mPLUG-DocOw/ [20], GLLaVA [40], Math Vision [11. MathInstruct 44). and MathPlus [43 Re-
cently, Otter-HD [8), Qwen-VL VL (6) Cog VLM [ and mPLUG Families 19 20 have increased
image resolution. Additionally, LLaVA-NeXT 12]. MM1 [14. and MiniGemini [10 divide images
into smaller sections, while VL-VA-UHD 21 and Intern 1L.1.5 [63] employ dynamically split image partitions depending on their sizes. These research trends aim to improve image perception capabil- ities. thereby enhancing understanding of images and natural language instructions. Furthermore, BRAVE [22]. DeepSeek-Vi VL (23). OmniFusion (24). MoVA [65] and AM-RADIO (25) have utilized
additional vision encoders such as EVA-CLIP 26 DINOv2 27 SAM
28
and SigLIP 29 Apart
from those, SpatialVLM [30), ASMv2 BI, LAR/LAF 32, CoLLaVO
33 and MoAI 34 employ
multiple computer vision models for tasks such as segmentation, detection, scene graph generation,
and optical character recognition (OCR).
We view this series of efforts as procedures for enlarging LLVM's knowledge space regarding multifaceted information. From a fresh perspective, we believe that embedding it in the form of a multifaceted rationale serves as a key in developing efficient LLVMs, where they include in-depth explanations required for acquiring diverse capabilities. From this perspective, Meteor is expected to inherently embed the multifaceted rationale and improve answering capabilities with the help of the embedded rationale, even without significantly increasing model size or relying on additional explicit vision encoders and computer vision models.
3 Meteor: Mamba-based traversal of rationale
Model Architecture. As illustrated in Figure
Meteor comprises a vision encoder, vision projector, Mamba architecture (39], tor projector, and backbone multimodal language model (MLM) based on pretrained large language model (LLM). For the vision encoder, we use CLIPP-L114 [66].
which is text-aligned vision module that takes advantage of image understanding adeptness powered by text descriptions. For the vision projector and tor projector, we employ MLP modules containing two fully- fully-connected layers with the GELU activation function [67]. Next, we use the Mamba- 130M
architecture for computational efficiency and adopt InternLM2-78
[68 69, learned with 2T tokens of multilingual text data in RLHF [70], as the backbone large language model (LLM).
Configuration of Visual Instruction Tuning Dataset. To build a visual instruction tuning set, we cover not only fundamental image understanding but also a wide range of diverse capabili- ties: common-sense knowledge, non-object concepts (e.g., charts, diagrams, documents, signs, symbols, and math problems), cognitive reasoning, multi-discipline tasks, and integrated abilities. For the question-answer visual instruction tuning dataset, we choose 664K question-answer pairs in ShareGPT4V-Innstn Caption 77. we select 91K image descriptions for images from LAION (71). CC (72), SBU [73], MS-COCO 7 TexICaps (75), and web images (76778) that depict landmarks, animals, celebrities, art, text, and nature. The selected question-answe answer pairs primarily focus on fundamental image understanding and common- sense knowledge, with fewer data samples covering non-object concepts, cognitive reasoning, multi- multi-discipline tasks, and integrated abilities. To strengthen these areas, we selectively gather 27K question-
(7]. including
IA-nnstructt-665K (8. Additionally, in ShareGPT4V.
pairs of VQA (79), ChartQA [0), DVQA [8], and A12D 38 from MiniGemini-Instruu [10]. Moreover, we use 574K/27K question-answer pairs of DeepForm 82 Info [3 DOc [79], KleisterCharity [4 TabFact [35], TextVQA [86).
WikiTable
87 TextCaps 75 and VisualMRC (88) from Doc-Dowwsttream/Reassn
(20) To achieve

---

lage sprr saduus
Dater
Pretrained Large Language Model (LLM)
Metear-Multimodal Language (MLM)
DMYO
D
BB
+
Wherewae ploto takin?
Powaaar
Meteor-Mamba
Meteor- -amba
DMM
DD0
+
+
where whe plona takes?
Where warthe photo taben?
Pripeter
isamligg spots stadum filidd vir dheeing tuants stention etamatter
najer saumemaile, pou dtandi22 IIA
Skann
Image
Question
Rationale
Question
(a) First Training Step
(b) Second Training Step
Figure 3: Overview of Meteor architecture and its training steps. Note that, 'Meteor Muttoor-uulimmdal Language Model (MLM)' indicates that as training progresses, the pretrained language model evolves into a multimodal language model.
broad coverage of math knowledge, we also include 177K GLLaVA-Align/Inttue 40] 3K Math- Vision [4]] and 566K text-only samples from Math- Saath-Inttrut/Plus 42 43 summary, we collect 755K real-world images, 627K images for documents, charts, diagrams, signs, and symbols, and 747K math samples (180.5K with images and 566.8K text-only). Overall, the question-ansee answer visual instruction tuning samples sum up 2.1M.
Curating Rationale. Using the gathered 2. I. question- pairs, utilize the light and fast Claude Haiku API [44) to generate detailed and comprehensive rationales. We use the prompt
template: "Question: (). Answer: {}. Based the question and answer, carefully provide
explanation about how to answer the question in detail." Here, (} represents the placeholder for the corresponding language description. Afterward,
assess the rationale score from GPT-4V 4
using the following template: "Question: {)- Rationale: [). Answer: []. Based on the question, rationale, and answer, provide score from 0 to 10, evaluating how well the rationale is described to solve the question. If the given rationale is insufficient, you should rigorously give a score below 5." Subsequently, filter out the generated rationales with automated evaluation are then subjected to human review to determine Yes or No on whether they provide a proper description to address the question. Finally, this series of processes yields 1. ceastion-ratinnalc-ee
below 5. The rationales that pass this
triples, which include 338K real-world images covering common-sense knowledge and few samples for diverse capabilities, 379K images for documents, charts, diagrams, signs, and symbols, and 342K math samples (165K with images and 1 177K text-only).
Mamba Architecture. To make LLVMs inherently possess rationale when addressing complex questions, we generate comprehensive rationales based on question-ansee answer pairs. Subsequently we employ the Mamba architecture [39]. leveraging its capability to handle lengthy rationales while maintaining computational efficiency. This approach allows us effectively incorporate the rationale in an environment where the curated 1. M diuestion-rationss
pairs have average length of 213 tokens, which is approximately ten times longer than the average length of 22 tokens ground truth answers in typical visual instruction tuning datasets 21 7
Traversal of Rationale. However, it is crucial to note that we cannot acquire and utilize rationales during the inference phase without API-based models, since only user questions are given. Therefore, we propose new concept called traversal of rationale to effectively provide the rationale to Meteor- ML.M without any help from external APIs the inference phase. Inspired by retrieval-based knowledge [89), we introduce special token, <tor> (stands for traversal of rationale), and evenly distribute fixed number of <tor> tokens, as described Figure 3 The rationale planted with <tor> is propagated into Meteor-Mamba along with image and question tokens, and then the output
5

---

LLVMs
Q-Bench SQA' AL2D ChartQA SEED'
POPE
HallB MME
MathVissa MMB MMB
MM-Vet LLaVA
BLIP2-13B (0)
61.0
46.4
85.3
1584
22.4
Instrcc18LIP
25.3
26.2
InstructB1IIP 1.-eiBBIIPPP.11 13B
63.1
78.9
33.9
25.6
IDEFICS-9B
Qwen-VL-7B [6]
59.4
67.1
38.2
7.4
Owsn-VI-Chat-78
33.8
68.2
58.2
56.4
1849
60.6
56.7
47.3
MiniGGPT-477 [58
51.8
23.1
23.0
11.9
22.1
Otter-7B [59
47.2
72.5
1599
19.7
48.3
24.7
[lavA-7B
38.5
80.2
44.1
1055
34.1
14.1
26.7
LLaVA1.5-7B 8
60.1
66.8
58.6
85,9
1805
64.3
58.3
30.5
63.4
LLaVA1.5-13B 8
61.4
71.6
54.8
18.2
61.6
85.9
46.7
1826
27.6
67.7
63.6
35.4
mPLLUG-Ow)-7B 60
58.9
22.2
46.6
m=PLUG-OWI1277
62.9
68.7
64.5
60.3
36.2
SharoGPT4V-78 y
63.4
68.4
69.7
49.8
1944
25.8
68.8
62.2
37.6
ImtemI.M.XC-77
1919
29.5
Monkey: 108 [9]]
69.4
68.9
58.4
1924
34.8
72.4
67.5
33.0
VIIAA-BR (2]
VILA-133B 92
73.7
62.8
84.2
70.3
64.3
38.8
SPHINX-7B 93
71.6
SPHINX-MoE-7Bx48 94
66.2
70.6
73.0
89.6
1852
42.7
71.3
40,9
SPHINX-Plus- 00
66.2
70.6
74.8
89.1
52.1
1741
36.8
71.0
47.9
LLaVA-NcXT-7B 12
70.1
70.2
86.5
1851
34.6
69.6
63.3
43.9
72.3
LLAVA-NeXT-88
71.6
69.5
1972
37.5
80.1
LLaVA NeXT-133
12
73.6
70.0
62.2
72.2
86.7
1892
35.1
70.0
68.5
47.3
72.3
MM1-7B [4
72.6
69.9
86.6
1858
35.9
72.3
42.1
MMI-Mob-7Bx331
14
74.4
70.9
87.8
1992
40.9
72.7
45.2
MinnGemiii-HDD78
10
1865
32.2
65.8
41.3
MiniGemini-Hee HD-13B
10
1917
37.0
68.6
50.5
Meteor 7B
69.2
88.3
77.9
74.9
75.0
88.7
60.4
2229
53.4
82.9
82.1
57.3
87.1
Table 1: Comparison with the current existing spen-source LLVMs, evaluating vision language
performances of Meteor numerous evaluation benchmarks requiring diverse capabilities: Q- ChartQA [80), SEED 197). POPE 198) HallB [99], MME [35], Math Vista (37], MMB [6) MMB CN [6), MM- Vet (100), and LLaVA w B. Note that, AI2D and ChartQA performances for LLaVA family models are evaluated under zero-shot conditions, while
Bench 195]. SQA! 196. A12D 38
Meteor uses training dataset for them.
features in Meteor-Mamba are directly propagated into Metcor-MLM. Here, we autoregressively train Meteor to generate the part of the rationale between <tor>, whenever Meteor sees the special token <tor>. This procedure ensures that each <tor> represents the following rationale part until the next <tor> encountered. Using single <tor> token encompass the rationale may not work well when embedding lengthy rationales, and if we do not consider distributing <tor> tokens in the rationale, then the later token does not refer to the earlier ones well due to the common problem of the autoregressive mechanism's 's forgetting nature (39] This is why we place multiple <tor> tokens in the rationale instead of one.
Training Strategy. the first training step, we leverage the question-rationale pairs the cu- rated 1.1M triples to train Meteor-Mambe and miscellaneous projectors. Throughout this step, the long sequential rationale is embedded into Meteor-Mamba through traversal rationale by autore- gressively generating rationale parts between the special tokens <tor>. By freezing Meteor-MLM Meteor-Mamba seamlessly incorporates the rationale. the second training step, we utilize the question- andwerr pairs in the curated 1.1M triples to jointly train Meteor- Patteor-aamba, Meteor-MLM. and the miscellaneous projectors. Here, multiple <tor> special tokens are only propagated to Meteor- Mamba. Then, the rationale-embedde features in Meteor-Mamba corresponding to the special tokens <tor> are only fed into Meteor-MLM. enabling it to adeptly answer complex questions, even in the absence of explicit rationale descriptions. In essence, these steps equip * Meteor with the capability to effectively address complex questions with the aid of the rationale.
4 Experiment
Implementation Detail. To ensure successful reproducibility, we outline three crucial technical
details of Meteor: the structure of (a) Meteor- and Meteor-MLM, (b) vision encoder and miscellaneous projectors, and (c) training and inference details.
(a) To build Meteor-Mamba, we use the Mamba architecture with 24 layers and 768 hidden dimension, resulting in total of 130M parameters, which is relatively trivial compared to the approximately 7B parameters of the pretrained InternLM2-7B 668 69]. It is executed under the

---

LLVMs
CP
FP
1R
LR
ST
MA
Avg
LLVMs
TD
TL
10
V1
VD
VO
VL
53.2
312
520
12
12.4
35.2
36.1
G-M.VA--T 40
20.9
20.7
21.
172
164
94
16.5
CogVLM Chat-----a-17 [
66.8
36.8
49
31.2
23.6
11.6
36.5
LLaVA-NeXT-133 122
12.8
12.0
9.9
10.7
4.
6.3
10.3
SPHINX Mox-7Bx8
54
58.4
40.8
47.6
35.2
1922
32.0
38.9
ShaneCiPTiV
16.2
16.2
6.6
15.5
13.8
3.7
13.1
IntemVL1.2-408
15
67.6
43.2
61.2
47.2
24.0
19.2
43.7
OPHINX-Plus-155 2
13.9
11.6
1499
116
13.5
10.4
122
MaVA-NeNT.34B
12
536
52.1
SPHINX MoTINNXMMoF-7PB MoF-7B
[9
17.4
26.7
16.7
Meteor-7B
69.6
45.6
63.6
53.2
42.0
42.8
52.8
Metcor-B8
25.5
21.7
27.4
21.7
19.2
14.7
21.7
(a) MMStar 101
(b) Varhe Verse
102
Benchmarks OmniFusion 24
DeepSeek-VL. VL 23
Mo (2
ASMv2 [3]
LAF
CoLLaVO
MoAl [M
Meteor
POPE
87.2
88.1
88.6
86.3
88.8
87.2
87.1
88.7
SQA-IMG
69.2
57.7
74,4
87.1
80.7
83.5
88.3
LLaVA-W
78.9
69.5
71.9
87.1
MM- Vet
39.4
41.5
41.3
38,9
40.3
43.7
57.3
MMStar
42.1
48.7
52.8
(c) Comparison with LLVMs using additional vision encoders and computer vision models
LLVMs
Recognition
OCR
Knowledge
Language Generation
Spatial Awareness
Math Problems
Avg.
LoLLaVO-7B 53
45.6
31.1
29.8
30.2
37,9
5.8
41.0
MoAL-7B
48.3
34.8
33,5
33.0
39.7
7.7
Metcor-78 78 w.o. Meteor-Mamba
44,5
33,5
41.8
31.3
38.6
29.2
44,8
Metcor-BB
45.0
59.3
(d) Evaluating sub-benchmarks in MM-Vet [100] with LLVMs utilizing computer vision models
Table 2: Detailed comparison of Meteor across more challenging evaluation benchmarks.
efficient computation of hardware-awarc state expansion 33 where we borrow the tokenizer 103 from the backbone MLM to fit the language expression space in the backbone MLM. Meteor-MIMM is based on InternL.M2-BB [68 69] with 32 layers and a 4096 hidden dimension.
(b) We use vision encoder with 428M CLIP-L/14 [66]. which has 24 layers and 1024 hidden dimension. The resolution of the positional embedding is interpolated from 24 x 24 to 35 x 35 to accommodate a 490 x 490 image resolution. The vision projector involves an MLP that adapts the hidden dimension from 1024 to 4096 to fit that of the backbone MLM. Similarly, we build the tor projector to convey embedded rationales from Metcor-Mamba into Meteor-MLMM employing the same structure as the vision projector but transferring the hidden dimension from 768 to 4096.
(c) We train and evaluate Meteor in the following computing environment: Intel(R) Xeon(R) Gold 6230, 256 GB RAM, and 8 XXNVIDIA RTX A6000 48GB VRAM. To efficiently train it, we use one epoch of training for each training step under 4-bit quantization and bfloat 1 data type [044 for Meteor-MIMM where double quantization and normalized float 4-bit (nf4) [105] are used. Meteor- Mamba uses 3 data type because training it with b or float16 has been reported to produce an unstable learning process. In addition, QLoRA [106. 105 is used train Meteor-MI.M with 64 rank and 64 alpha parameters. We use the Adam W 107 optimizer and schedule the learning rate by cosine annealing (188 from le-4 le-6 in each training step, with gradient checkpointing [109] applied to Meteor MLLM for efficient memory management. With a gradient accumulation of 6, we set batch sizes of 192 and 576 for each training step, and each step takes approximately three days. For efficient inference, Metcor is validated in 4-bit quantization, and we use deterministic beam search (n 10 for text generation. Note that we implement not only Metcor-MLM, but also numerous baselines under the efficient propagation from Atteninttn 1
Evaluation. We have evaluated Meteor on numerous vision- lasoun-language benchmarks, the details of which are described in Appendix B These benchmarks require multifaceted information for diverse capabilities, including fundamental image understanding, real-world knowledge of common-sente knowledge, charts, diagrams, documents, signs, symbols, math problems, and more. Figure 1 and Table [ illluttrates vision language performances various LLVMs, including Meteor-7B, open- and closed-source L.VMMs with various sizes. It is noteworthy that Meteor-7B noticeably outperforms the other models, demonstrating its efficacy and efficiency in using embedded multifaceted rationales from Meteor-Mant Mamba. The detailed generation quality of Meteor is described in Appendix C Apart from the results in Table those in Table 2 benchmarks, which require multifaceted information simultancously. Meteor-77 7B has outperformed
signify that Meteor-7B also excels at more challenging
7

---

Arch
Param
BPS
MMB Me-Vet
LLMS
Param
MMB
MM- Vet
Num
MMB
MM-Vel Vet
BERT-B
110M
80.6
53.6
Vicunal.5
7B
80.1
53.8
#2
76.1
47.9
GPT2-S
117M
80.9
53.5
LLaMA2
7B
78.8
51.6
#5
82.2
55.8
XLNet-B
110M
56
81.6
53,9
LLaMA3
8B
81.7
56.0
#10
82.9
57.3
Mamba
130M
118
82,9
57.3
InternLM2
7B
82.9
57.3
11
82.8
$7.3
(a) Metcor-Mamba
(b) Metcor-MLM
(c) Number <tor> tokens
Position
MMB
MM-Vet
Mamba
Rationale
MMB
MM-Vet
Q-R
MME
MMB
MM-Vet
Start
79.8
48.2
x
X
73.2
44.8
0%
1989
74.0
45.9
End
74.2
45.1
x
77.0
48.2
30%
2105
77.5
51.7
Random
74.3
45.4
/
74.0
45.9
60%
2218
82.5
56.8
Even
82.9
57.3
/
82.9
57.3
90%
2229
82.9
57.3
(d) Position of <tor> tokens
(e) Mamba & Rationale
(f) Ratio Q-R in training
Table 3: Ablation studies to identify the effectiveness of Meteor-Mamba and rationale through traversal of rationale by controlling the six main factors.
other existing models by a large margin, some of which are equipped with additional vision encoders computer vision models, demonstrating that rationales can provide multifaceted information more effectively than enhanced visual perception.
Ablation Studies. Furthermore, we have conducted several ablation studies securely corroborate the effectiveness of our proposed method in light of six factors: (a) Meteor-Mamba, (b) Meteor-MIMM (c) the number of <tor> special tokens, (d) the distribution of <tor> special tokens, (e) rationale, and (f) 1.IM question-rationall-eee answer triples. Table 3 shows the following findings. Note that, Appendix D represents further ablation studies.
(a) Once Metcor-Mamba is replaced with other transformer-bssed models: BERT
GPT2 M4.
and XLNet [155, we have discovered that the Mamba architecture takes advantage of its efficiency for embedding multifaceted rationales terms of both computational complexity and model size. As the results suggest, Mamba demonstrates the highest batches- batches-per-secone per-second (BPS) value and zero-shot performances on MME and MMB benchmarks among architectures of similar sizes, enabled by its inherent computational efficiency based on linear complexity and strong long sequence modeling capability 39, which other Transformer-basel model architectures lack.
(b) We have tried using various pretrained LLMs of comparable sizes for Meteor-MLM, in order to identify the effectiveness of embedding multifaceted rationale together with traversal of rationale. We have observed that InternLM2
[69 has shown the best performances.
(c) Varying the number of <tor> special tokens from 2 to 15, have optimized it based on the vision language performances. The results suggest that using 10 <tor> special tokens shows the best performances for embedding abundant multifaceted rationales, balaneing between compression and information preservation.
(d) The performances of Meteor depend on the distribution of <tor> special tokens when Meteor- Mamba is trained to embed multifaceted rationales. Given the observations, evenly distributing the tokens across lengthy rationales has shown the best performances. Prepending them to lengthy rationales may hinder effective embedding due forgetting nature, and appending them to rationales in the end may not be guaranteed to understand the rationale. Randomly distributing the tokens across rationales may disrupt Meteor-Mamba 's ability to stably learn the pattern of embedding rationales. Conversely, evenly distributed <tor> special tokens can segment lengthy rationales into shorter chunks and progressively embed them in consistent manner, avoiding the issues of other distributions.
(e) In order to prove the effectiveness of multifaceted rationales through Meteor Mamba, we ablated the use of the Mamba architecture and the rationales. The first row represents baseline performances where backbone MLM is only trained. For the second row, we only train backbone MLM with the curated multifaceted rationales without Meteor-Mamba, and for the third row, Mamba has been trained to embed answer instead of the rationales. Compared to the last row where Meteor is evaluated, the second and third rows fall short of performances, clearly showing that using multifaceted rationales through the Mamba architecture has contributed to performance improvement.
(f) As another way of showing the significance of multifaceted rationales, we have trained Meteor with different amounts of question-rationall pairs and evaluated Meteor trained with each of them.
8

---

5.1
0.00
0:12
<tor> W.O. rationale
0.15
0
4.00
4.11
0.000.00
0.05.0.00
0.03
0.62
3.2
0.20
0.65
0:03
218
0.51
0:15
0:20
Netterkaaba
Midhor Marbe
0.00
6
5.02 0.50
0.3
4.1 0.04
4.00
4.3
0.73
0:22
015
0.02
0.59
3.14
0.24
0.00
-02
<tor> w. rationale
Figure 4: Illuminating how the feature correspondences of cosine similarity are computed under the trained Meteor-Mamba, and showing the feature disparity for <tor> with/without rationale.
As expected, the rattonal pairs used in first training step, the better performances achieves, demonstrating the significance utilizing multifaceted rationales for diverse capabilities.
Meteor-Mamba's Ability to Embed Rationales. We conduct thorough analysis to confirm that Meteor-Mamba effectively embeds the rationales. To do this, we perform a retrieval task for multifaceted rationales, where we prepare ten different question-rationale pairs (2i, R.)) where i= 0, 1, 9. These pairs are propagated through Meteor-Mamha with or without rationales under <tor> special tokens. This results in two sets of output features: one with rationale z and one without rationales 2,,0, with j 0,1, 9. We extract features corresponding to the placement of <tor> tokens, resulting in z E 10 and 2,:00 E R0007668 where the dimension 10 corresponds to the number of <tor> tokens. We then compute the cosine similarity between " and x to measure the similarity of their representations. As illustrated in Figure the diagonal values in the cosine similarity matrix are much higher than the off-diagonal values. This result indicates that Meteor-Mambe successfully embeds the rationale, and its output features contain multifaceted information even without explicit rationales in natural language. This explains how Meteor-Mambe operates effectively during the inference phase without explicit rationales.
Discussion and Limitation. From the experimental results observed, we gain the insight that equipping LLVMS with multifaceted rationale is a key factor in building efficient LLVMs that demonstrate impressive vision language performances across numerous evaluation benchmarks requiring diverse capabilities. This rationale, furthermore, naturally reduces hallucination effects in POPE (98 and HallusionBench [99] in Table Additionally, Table ([c) shows that the need for additional vision encoders and computer vision models can be mitigated by incorporating multifaceted rationale. However, Meteor might still considered inefficient in terms of model size by users without high-end GPU resources, as it requires at least multiple GPUs with 48GB and 32GB VRAM for normal training and inference without (Q)LoRA [I06] [05 and 4/8- quantization. Although many closed-source LLVMs have demonstrated superior performances following the scaling law [6), our goal is to reduce the model size while maintaining vision language performances as much possible. We strongly believe that small language and vision models, I~BB parameters, can effectively narrow the performance gap with the source LLVMs by using layer-anale analyzing approaches such mixture of depths [1 and others [23), despite their inherent limitation in layer number and hidden dimension.
those with about
Conclusion
To build efficient LLVMs. we incorporate multifaceted rationale encompassing various aspects such as image understanding, incorporating external common-sense knowledge, understanding non-object concepts (e.g., charts, diagrams, symbols, signs, and math), and following systematic step-by-step procedures for how to solve complex questions. Meteor demonstrates significantly enhanced vision language performances across various evaluation benchmarks without the need scale up LLVMs, use additional vision encoders, or employ multiple computer vision models. In designing Meteor, the traversal of rationale combined with Mamba architecture proves highly effective in embedding lengthy rationales. We believe this rationale, facilitated by the traversal of rationale, can pave the way for more efficient models, representing a promising step towards achieving more efficient LLVMs.

---

Acknowledgments and Disclosure of Funding
This work was partially supported by two funds: Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD (UD230017TD) and IITP grant funded by the Korea government (MSIT) (RS-2022-11220884).
References
[1] J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le, "Finetuned language models are zero-shot learners," in International Conference on Learning Representations, 2022.
[2] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. De- hghani, S. Brahma, et al., "Scaling instruction-finetunee language models," arXiv preprint arXiv:2210.1 1416, 2022.
[3] H. Liu, Li. Q. Wu, and J. Lee, "Visual instruction tuning," in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[4] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida,
J. Altenschmidt, Altman, S. Anadkat, et al., "Gpt-4 technical report," arXiv preprint arXiv:2303.087727 2023.
[5] Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrae, Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, al., "Gemini: family of highly capable multimodal models," arXiv preprint arXiv:2312. 1800, 2023.
[6] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou, "Qwen-vl: A frontier large vision-language model with versatile abilities," arXiv preprint Xiv:2308811266 2023.
[7] L. Chen, J. Li, X. Dong, P. Zhang, C. He, Wang. F. Zhao, and Lin, "Sharegpt4v: Improving large multi-modal models with better captions," arXiv preprint drXiv:2311.12799 2023.
[8] H. Liu, C. Li, Li, and Y. J. Lee, "Improved baselines with visual instruction tuning." arXiv preprint arXiv:2310.03744, 2023.
[9] W. Dai, J. Li, D. Li, A. Tiong. J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, "InstructBLIP: Towards seneral-purpost languala models with instruction tuning," in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[10] Y. Li, Y. Zhang, C. Wang. Z. Zhong, Y. Chen, R. Chu, S. Liu, and J. Jia, "Mini-gemin:: Mining the potential of multi-modality vision language models," arXiv preprint arXiv:2403. 18814, 2024.
[1I] W. Wang, Q. Lv, W. Yu, W. Hong. J. Qi. Y. Wang. J. Ji, Z. Yang. L. Zhao, X. Song, et al., "Cogvlm: Visual expert for pretrained language models," arXiv preprint 2rX0v:2311.03079 2023.
[12] H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and J. Lee, "Llava-next: Improved reasoning, ocr, and world knowledge," January 2024.
[13] A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, H. Li, J. Zhu, J. Chen, J. Chang, et al., "Yi: Open foundation models by 01. ai." arXiv preprint arXiv:2403.04652 2024.
[14] B. McKinzie, Z. Gan, J.-P. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng. F. Weers, al., "Mml: Methods, analysis & insights from multimodal Ilm pre-training," arXiv preprint arXiv:2403.094
2024.
[15] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, el al., "Internvl: Scaling up vision foundation models and aligning for generic linguisic tasks." arXiv preprint 2rXiv:2312.142255 2023.
10

---

[16] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Rad- ford, Wu, and D. Amodei, "Scaling laws for neural language models." " arXiv preprint arXiv:2001.0828 2020.
[17] Y. Tay, M. Dehghani, J. Rao. W. Fedus, S. Abnar, H. W. Chung, S. Narang, D. Yogatama, A. Vaswani, and D. Metzler, "Scale efficiently: Insights from pre-training and fine-tuning transformers," arXiv preprini 1rX.v:2109. 10686, 2021.
[18] B. Li, P. Zhang, J. Yang, Y. Zhang, F. Pu, and Z. Liu, "Otterhd: A high-resolution multi- modality model," arXiv preprint arXiv:2311.042. 2023.
[19] Q. Ye, H. Xu, J. Ye, M. Yan, H. Liu, Q. Qian, J. Zhang, F. Huang, and J. Zhou, "mplug- owl2: Revolutionizing multi-modal large language model with modality collaboration," arXiv preprint arXiv:231 1.04257, 2023.
[20] A. Hu, H. Xu, J. Ye, M. Yan, L. Zhang. B. Zhang, C. Li, J. Zhang, Q. Jin, F. Huang, et al., "mplug-docow docowl 1.5: Unified structure learning for ocr-free document understanding," arXiv preprint 2nxi: 12895, 2024.
[21] R. Xu, Y. Yao, Z. Guo, J. Cui, Z. Ni, C. Ge, T.-S. Chua, Z. Liu, M. Sun, and G. Huang, "Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images." arXiv preprint arXiv:2403. 10.8.117000 1703, 2024.
[22] O. F. Kar. A. Tonioni, P. Poklukar, A. Kulshrestha, A. Zamir, and F. Tombari, "Brave: Broad- ening the visual encoding of vision-language models," " arXiv preprint urXiv:2404.07204 2024.
[23] H. Lu, W. Liu, B. Zhang, B. Wang. K. Dong, B. Liu, J. Sun, T. Ren, Z. Li, Y. Sun, et al., "Deepseek-vl: towards real-world vision-language understanding," arXiv preprint WrXiv:2403005522 2024.
[24] E. Goncharova, A. Razzhigaev, M. Mikhalchuk, M. Kurkin, I. Abdullaeva, M. Skripkin, I. Oseledets, D. Dimitrov, and A. Kuznetsov, "Omnifusion technical report," arXiv preprins arXiv:2404. AxXiv:2044062122, 2024.
[25] M. Ranzinger, G. Heinrich, Kautz, and Molchanov, "Am-radio: Agglomerative model- reduce all domains into one," arXiv preprint arXiv:2312.06709 2023.
[26] Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang. T. Huang, X. Wang, and Y. Cao, "Eva: Exploring the limits of masked visual representation learning at scale," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19358-19369) 2023.
[27] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, Haziza, F. Massa, A. El-Nouby, el al., "Dinov2: Learning robust visual features without supervision," arXiv preprint 0047:2230..07193, 2023.
[28] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al., "Segment anything," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
[29] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, "Sigmoid loss for language image pre- training." in Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11975-11986 2023.
[30] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and Xia, "SpatialvIm: Endowing vasion-language models with spatial reasoning capabilities," arXiv preprint wxXiv:2401.12166 2024.
[31] W. Wang, Ren, H. Luo, Li, C. Yan, Z. Chen, W. Wang, Q. Li, L. Lu, Zhu, es al., "The all-seeing project v2: Towards general relation comprehension of the open world," arXiv preprint arXiv:2402. 19474, 2024.
[32] Q Jiao, D. Chen, Y. Huang, Y. Li, and Y. Shen, "Enhancing multimodal large language models with vision detection models: An empirical study," arXiv preprint 178777, 2024.
11

---

[33] B.-K. Lee, B. Park, C. W. Kim, and Y. M. Ro, "Collavo: Crayon large language and vision model," arXiv preprint arXiv:2402. 12248 2024.
[34] B.-K. Lee, B. Park, C. W. Kim, and Y. M. Ro, "Moai: Mixture of all intelligence for large language and vision models," arXiv preprint arXiv:2403.07508 2024.
[35] C. Fu, P. Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, Yang, X. Zheng, K. Li, X. Sun, et al., "Mme: A comprehensive evaluation benchmark for multimodal large language models," arXiv preprint arXiv:2306. 13394, 2023.
[36] Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al., "Mmbench: Is your multi-modal model an all-around player?," arXiv preprint drXiv:2307.06288) 2023.
[37] P. Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and Gao, "Mathvista: Evaluating mathematical reasoning of foundation models visual contexts," arXiv preprint arXiv:2310.02255 2023.
[38] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishirzi, and A. Farhadi, "A diagram is worth dozen images." in Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp. 235-251, Springer, 2016.
[39] A. Gu and T. Dao, "Mamba: Linear-time sequence modeling with selective state spaces," arXiv preprint arXiv:2312.007522 2023.
[40] J. Gao, R. Pi, J. Zhang, J. Ye, W. Zhong, Y. Wang, L. Hong. J. Han, H. Xu, Z. Li, et al., "G-llava: Solving geometric problem with multi-modal large language model," arXiv preprint arXiv:2312. 2023.
[41] K. Wang, J. Pan, W. Shi, Z. Lu, M. Zhan, and H. Li, "Measuring multimodal mathematical reasoning with math-vision dataset," arXiv preprint arXiv:2402. 2024.
[42] X. Yue, X. Qu, G. Zhang, Y. Fu, W. Huang, H. Sun, Y. Su, and W. Chen, "Mammoth: Building math generalist models through hybrid instruction tuning." arXiv preprint 2309:23009.05533 2023.
[43] X. Yue, T. Zheng, G. Zhang, and W. Chen, "Mammoth2: Scaling instructions from the web." 2024.
[44] Anthropic, "The claude 3 model family: Opus, sonnet, haiku." https:// http::://wwwaathropi..
com 2024.
[45] J. Strout, Y. Zhang, and R. Mooney, "Do human rationales improve machine explanations?," in Proceedings of the 2019 ACI. Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 56 62, 2019.
[46] J. Lu, L. Yang, B. Namee, and Y. Zhang, "A rationale-centric framework for human-in-the- loop machine learning." in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6986-6996, 2022.
[47] C.-Y. Hsieh, C.-L. Li, C.-K. Yeh, H. Nakhost, Y. Fujii, Ratner, Krishna, C.-Y. Lee, and T. Pfister, "Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes," arXiv preprint (.X.::3335002000 2023.
[48] P. Wang. Z. Wang, Z. Li, Y. Gao, B. Yin, and X. Ren, "Scott: Self-consistent chain-of-thought distillation," in Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5546-5558, 2023.
2023.
[49] W. Xiong, Y. Song, P. Wang, and S. Li, "Rationale-enhanced language models are better continual relation learners," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 15489-15497, 2023.
12

---

[50] L. Li, J. Hessel, Y. Yu, X. Ren, K.-W. Chang, and Y. Choi, "Symbolic chain- chain-of- offtthuughtt distillation: Small models can also "think" step-by-step," in Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume I: Long Pupers), pp. 2665- 2679, 2023.
[51] M. Kang. S. Lee, Back, K. Kawaguchi, and S. J. Hwang, "Knowledge-augmenned reasoning distillation for small language models in knowledge-intensivy tasks," Advances in Neural Information Processing Systems, vol. 36, 2024.
[52] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., "Chain-of-thouue thought prompting elicits reasoning in large language models," Advances in neural information processing systems, vol. 35, pp. 24824 20224,2437,,. 2022.
[53] Z. Zhang, A. Zhang. M. Li, and A. Smola, "Automatic chain of thought prompting in large language models," arXiv preprint arXiv:2210.03499 2022.
[54] K. Shum, S. Diao, and T. Zhang, "Automatic prompt augmentation and selection with chain- of-thought from labeled data," " in Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 12113-12139, 2023.
[55] S. Krishna, J. Ma, D. Slack, A. Ghandeharioun, S. Singh. and H. Lakkaraju, "Post hoc expla- nations of language models can improve language models," Advances in Neural Information Processing Systems, vol. 36, 2024.
[56] K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, "Shikra: Unleashing multimodal I1m's referential dialogue magic," arXiv preprint 233.:2300.119955 2023.
[57] H. Laurençon, L. Saulnier, L. Tronchon, S. Bekman, A. Singh, A. Lozhkov, T. Wang, S. Karam- cheti, A. M. Rush, D. Kiela, et al., "Obelisc: An open web-scale filtered dataset of interleaved image-text documents," arXiv preprint arXiv:2306.16527 2023.
[58] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, "Minigpt-4: Enhancing vision-language understanding with advanced large language models," arXiv preprint arXiv:2304. 10592, 2023.
[59] B. Li, Y. Zhang, L. Chen, J. Wang. J. Yang, and Z. Liu, "Otter: A multi-modal model with in-context instruction tuning," arXiv preprint 23X5v:2305.03726 2023.
[60] Q. Ye, H. Xu, Xu, Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi. Y. Shi, et al., "mplug- owl: Modularization empowers large language models with multimodality," arXiv preprint arXiv:2304. 14178, 2023.
[61] X. Contributors, "Xtuner: toolkit for efficiently fine-tuning Ilm." httpp:://itthub. com/ InternLM/ztuner 2023.
[62] P. Zhang, X. D. B. Wang, Y. Cao, C. Xu, L. Ouyang, Z. Zhao, S. Ding, S. Zhang, H. Duan, H. Yan, et al., "Intemllnxxcopposer A vision-language large model for advanced text-image comprehension and composition," arXiv preprint 21X1v:2309.15112, 2023.
[63] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, al., "How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites," arXiv preprint arXiv:2404.1682 1682/, 2024.
[64] G. H. Chen, S. Chen, R. Zhang, J. Chen, X. Wu, 7. Zhang, 7. Chen, J. Li, X. Wan, and B. Wang, "Allava: Harnessing gpt4v-synthesized data for a lite vision-language model," arXiv preprint 2rXiv:2402. 1682, 2024.
[65] Z. Zong, B. Ma, D. Shen, G. Song. H. Shao, D. Jiang, H. Li, and Y. Liu, "Mova: Adapting mixture of vision experts to multimodal context," arXiv preprint 240.:24444.13466 2024.
[66] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, G. Krueger, and I. Sutskever, "Learning transferable visual models from natural language supervision," in Proceedings of the 38th International Conference on Machine Learning (M. Meila and T. Zhang, eds.), vol. 139 of Proceedings of Machine Learning Research, pp. 8748 8763, PMLR, 18-24 Jul 2021.
13

---

[67] D. Hendrycks and K. Gimpel, "Gaussian enror linear units (gelus)," " arXiv preprint arXiv:1606.00 08415, 2016.
[68] I. Team, "Internlm: A multilingual language model with progressively enhanced capabilities." https://giitin htwptttttbooo.ooaaaooooootooottt IntteralA/Inerrran-eeeeeeee
2023.
[69] Z. Cai, M. Cao. H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chu, et al., "Internlm2 technical report," arXiv preprint arXiv:2403. 17297, 2024.
[70] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, al., "Training language models follow instructions with human feedback," Advances in Neural Information Processing Systems, vol. 35, pp. 27730-27744, 2022.
[71] C. Schuhmann, Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al., "Laion-Sb: An open large-scale dataset for training next generation image-text models," Advances in Neural Information Processing Systems, 2022.
vol. 35, pp.
[72] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, "Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts," " in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 3558-3568, 2021.
[73] B. Saleh and A. Elgammal, "Large-scale classification of fine-art paintings: Learning the right metric on the right feature," arXiv preprint arXiv: 505.00 2015.
[74] T.-Y. Lin, M. Maire, Belongie, J. Hays, Perona, D. Ramanan, P. Dollár, and C. Zitnick, "Microsoft coco: Common objects in context," in Computer Vision- 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755, Springer, 2014.
[75] O. Sidorov, R. Hu, M. Rohrbach, and A. Singh, "Textcaps: dataset for image captioning with reading comprehension." in Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pp. 742-758, Springer, 2020.
[76] V. Ordonez, G. Kulkarni, and T. Berg, "Im2text: Describing images using million captioned photographs," Advances in neural information processing systems, vol. 24, 2011.
[77] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and Komatsuzaki, "Laioo-4000: Open dataset clip-filtered 400 million image- text pairs," arXiv preprint arXiv:2 / 000 14, 2021.
[78] P. Sharma, N. Ding, S. Goodman, and R. Soricut, "Conceptual captions: A cleaned, hyper- nymed, image alt-text dataset for automatic image captioning," in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pp. 2556-2565. 2018.
[79] M. Mathew, D. Karatzas, and C. Jawahar, "Docvqa: A dataset for vqa on document images," in Proceedings of the IEEE/CVF winter conference applications of computer vision, pp. 2200- 2209, 2021.
[80] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, "Chartqa: A benchmark for question answering about charts with visual and logical reasoning." arXiv preprint 2XXv:2203.10244, 2022.
[81] K. Kafle, B. Price, S. Cohen, and C. Kanan, "Dvqa: Understanding data visualizations via question answering," " in Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3648-5654 2018
[82] S. Svetlichnaya, "Deepform: Understand structured documents at scale," " 2020.
[83] M. Mathew, V. Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawahar, "Infographicvqa." in Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
14

---

[84] T. Stanistawek, F. Graliñski, A. Wróblewska, D. Lipiński. A. Kaliska, P. Rosalska, B. Topolski, and P. Biecek, "Kleister: key information extraction datasets involving long documents with complex layouts," in International Conference on Document Analysis and Recognition, pp. 564-579, Springer, 2021.
[85] W. Chen, H. Wang, J. Chen, Y. Zhang, H. Wang, S. Li, X. Zhou, and W. Y. Wang, "Tabfact: A large-scale dataset for table-based fact verification," arXiv preprint arXiv: (90:::9990000 02 164, 2019.
[86] Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, "Towards vqa models that can read," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
[87] P. Pasupat and P. Liang, "Compositional semantic parsing on semi-structured tables," in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (C. Zong and M. Strube, eds.), (Beijing, China), pp. 1470-1480, Association for Computational Linguistics. July 2015.
[88] R. Tanaka, K. Nishida, and S. Yoshida, "Visualmre: Machine reading comprehension on document images," " in Proceedings of the AAAI Conference on Artificial Intelligence. vol. 35, pp. 13878-13888 2021.
[89] M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettle- moyer, and W.-t. Yih, "Retrieval-augmenn multimodal language modeling," arXiv preprint 2022.
arXiv:2211.12560
[90] J. Li, D. Li, S. Savarese, and S. Hoi, 2Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models," arXiv preprint arXiv:2301.12597, 2023.
[91] Z. Li, B. Yang, Q. Liu, Z. Ma, S. Zhang, J. Yang, Y. Sun, Y. Liu, and X. Bai, "Monkey: Image resolution and text label are important things for large multi modal models," arXiv preprint NKKV:2311.06600) 2023.
[92] J. Lin, H. Yin, W. Ping, Y. Lu, P. Molchanov, A. Tao, H. Mao, J. Kautz, M. Shoeybi, and S. Han, "Vila: On pre-training for visual language models," arXiv preprint arXiv:2312.07533, 2023.
[93] Z. Lin, C. Liu, R. Zhang, P. Gao, L. Qiu, H. Xiao, H. Qiu, C. Lin, W. Shao, K. Chen, et al., "Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-m multi-modal large language models," arXiv preprint arXiv:2311.07575 2023.
[94] P. Gao, R. Zhang, C. "Sphinx-x: Scaling data and parameters for a family of multi-modal large language models," arXiv preprint arXiv:2402.0mm 009355955 2024.
Liu, L. Qiu, S. Huang, W. Lin, S. Zhao, S. Geng, Z. Lin, P. Jin, al.,
[95] H. Wu, Z. Zhang, Zhang, C. Chen, L. Liao, A. Wang, Li, W. Sun, Q. Yan, G. Zhai, et al., foundation models on low-level vision," arXiv
"Q-bench: A benchmark for preprint arXiv:2309. 14181, 2023.
[96] P. Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P. Clark, and A. Kalyan, "Learn explain: Multimodal reasoning via thought chains for science question answering," Advances in Neural Information Processing Systems, vol. 35, pp. 2507-2521, 2022.
[97] B. Li, R. Wang, G. Wang, Y. Ge, Y. Ge, and Y. Shan, "Seed-bench: Benchmarking multimodal Ilms with generative comprehension," arXiv preprint arXiv:2307 2023.
[98] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.-R. Wen, "Evaluating object hallucination in large vision-language models," arXiv preprint arXiv:2305. 10355, 2023.
[99] F. Liu, T. Guan, Z. Li, L. Chen, Yacoob, D. Manocha, and T. Zhou, "Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), 1.5, and other multi-modality models," arXiv preprint arXiv:2310.1456,, 2023.
15

---

[100] W. Yu. Z. Yang. L. Li, J. Wang, K. Lin, Z. Liu, X. Wang, and L. Wang, "et: Evaluating large multimodal models for integrated capabilities," arXiv preprint arXiv:2308.00 02490, 2023.
[101] L. Chen, Li, X. Dong, P. Zhang, Y. Zang. Z. Chen, H. Duan, J. Wang, Y. Qiao, D. Lin, et al., "Are we on the right way for evaluating large vision-language models?," arXiv preprint S.ti::2403.2033333 2024.
[102] Zhang. D. Jiang, Y. Zhang, H. Lin, Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Gao, et al., "Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems?." arXiv preprint arXiv:2403. 2024.
[103] T. Kudo and Richardson, "Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing," arXiv preprint arXiv: 8118808800226 2018
[104] D. Kalamkar, D. Mudigere, N. Mellempudi, D. Das, K. Banerjee, S. Avancha, D. T. Vooturi, Jammalamadaka, J. Huang, H. Yuen, al., "A study of 6 for deep learning training," " arXiv preprint arXiv: /905. 12322, 2019.
[105] T. Dettmers, A. Pagnoni, A. Holtzman, and L. Zettlemoyer, "Qlora: Efficient finetuning of quantized Ilms," arXiv preprint arXiv:2305. 14314, 2023.
[106] E. J. Hu, Y. Shen, P. Wallis, Allen-Zhu, Y. Li, S. Wang. L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models," arXiv preprint arXiv:2106.099: 09685, 2021.
[107] I. Loshchilov and F. Hutter, "Decoupled weight decay regularization,"" in International Confer- ence on Learning Representations, 2019.
[108] I. Loshchilov and F. Hutter, "Sgdr: Stochastic gradient descent with warm restarts," arXiv preprint
2016.
[109] N. S. Sohoni, C. R. Aberger, M. Leszczynski, J. Zhang, and C. Ré, "Low-memory neural network training: technical report," arXiv preprint arXiv:/904 10631, 2019.
[110] M. Freitag and Y. Al-Onaizan, "Beam search strategies for neural machine translation," in Proceedings of the First Workshop Neural Machine Translation (T. Luong, A. Birch, G. Neubig, and A. Finch, eds.), (Vancouver), pp. 56 56-60, Association for Computational Linguistics, Aug. 2017.
[111] T. Dao, D. Fu, S. Ermon, A. Rudra, and C. Ré, "Flashattention: Fast and memory-efficient exact attention with io-awareness," Advances in Neural Information Processing Systems, vol. 35, pp. 16344-16359, 2022.
[112] T. Dao, "Flashattention-2:: Faster attention with better parallelism and work partitioning," arXiv preprint arXiv:2307.08691, 2023.
[113] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv: 2018.
[114] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I, Sutskever, et al., "Language models are unsupervised multitask learners,"
[115] Z. Yang. Z. Dai, Y. Yang. J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, "XInet: General- ized autoregressive pretraining for language understanding." Advances in neural information processing systems, vol. 32, 2019.
[116] D. Raposo, S. Ritter, B. Richards, T. Lillicrap, C. Humphreys, and Santoro, "Mixture- of-depths: Dynamically allocating compute in transformer-based language models," arXiv preprint arXiv:2404.0222 022558, 2024.
[117] B.-K. Lee, Y. Yu, and Y. M. Ro, "Towards adversarial robustness of bayesian neural network through hierarchical variational inference," 2021.
[118] J. Kim, B.-K. Lee, and Y. M. Ro, "Distilling robust and non-robust features in adversarial examples by information bottleneck," Advances in Neural Information Processing Systems, vol. 34, pp. 17148-1711 17159, 2021.
16

---

[119] B.-K. Lee, J. Kim, and Y. M. Ro. "Masking adversarial damage: Finding adversarial salieney for robust and sparse network," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15126-15136, 2022.
[120] J. Kim, B.-K. Lee, and Y. M. Ro, "Demystifying causal features on adversarial examples and causal inoculation for robust network by adversarial instrumental variable regression," " in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12302-12312, 2023.
[121] B.-K. Lee, J. Kim, and Y. M. Ro, "Mitigating adversarial vulnerability through causal param- estimation by adversarial double machine learning," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4499-4509, 2023.
[122] J. Kim, B.-K. Lee, and Y. M. Ro, "Causal unsupervised semantic segmentation," arXiv preprint arXiv:2310.07379, 2023.
[123] Kim, Kim, B.-K. Lee, S. Shin, and Y. M. Ro, "Mitigating dataset bias image captioning through clip confounder-free captioning network," in 2023 IEEE International Conference Image Processing (ICIP), pp. 1720-1724, IEEE, 2023.
[124] G. Luo, Y. Zhou, Y. Zhang, X. Zheng, X. Sun, and R. Ji, "Feast your eyes: Mixture-of- resolution adaptation for multimodal large language models," arXiv preprint arXiv:2403.0300 030000 2024.
[125] H. Zhao, M. Zhang, W. Zhao, P. Ding, S. Huang, and D. Wang, "Cobra: Extending mamba to multi-modal large language model for efficient inference," arXiv preprint 14X5: 2024.
[126] Y. Qiao, Z. Yu, L. Guo, S. Chen, Z. Zhao, M. Sun, Q Wu, and J. Liu, "VI-mamba: Exploring state space models for multimodal learning," arXiv preprint arXiv:2403. 13600, 2024.
[127] J. Liu, M. Liu, Z. Wang, L. Lee, K. Zhou, P. An, S. Yang, R. Zhang, Y. Guo, and S. Zhang, "Robomamba: Multimodal state space model for efficient robot reasoning and manipulation." arXiv preprint arXiv:2406.043339 2024.
[128] W. Huang and Hu, "MI-mamba: Efficient multi-modal large language model utilizing mamba-2," arXiv preprint 1988 2024.
17

---

A Examples of Curated 1.1M Question-Ratonall-Ansse Triples
setiigg aegg conp-
2. "Minninng the work rass the murch
quezizn that
mutor day..
this givnsss werk new of E-s0Sa muntnn por
day. Sina the queatinn anl aboun mann mulipphy the wont nou by 35 to the tosal wort done per day 00023 mover per man par day mune mutars day. piy cailly work rake
we brow that wark 5533 metaro
of sayo 15 loaa ln
days etar m-mmm-mmmmt
The ane orppiis chklin pisblnes ane
indeerradd the intorrattion and the
nwn.
the
angihval wull munin which miteas
Answer:
Firs, out wark 2irmar..
ef mhre con wallt meter soil digi then the work done by now man auon dey kitee
Now want fid how much work man days.
15
mantees nai per mew mattrn
would
Thenetore 15 build 42 meter lingwol daw..
D
00
Questiin:
Fistt rossoning finals select from the the following femmat Anower extion we lave repular hevagen AR0000 and vade i,o suaare B0OH conson cood what the sie angl Hil wher segnent AM boreooooeoeo coreected
62
Rationale:
ofgegu go
owan enfomatior:
i contrrcted indde
tind thorntoeer nightt regalar
relonar andlo- at a8o" For regila broage,
interior angll 21 s007
the trsarion the voceoor angle of sopare 90" regular arr vore sales
Step Find the angle 14A6.
since BCSH la squers angle HBc 2007
The angla the ragular hapagon
Tharefors you anaver A.A
Answer:
The tesator angle regaler hasager intarior angle of aquare Thunddss, ABC angle H0C Thersfon, angla 4BH Additionll sinca EIl anglo
AnawrrA
18

---

D
E
Question:
feline what obverve whe
Rationale:
The image thows gromante digran with trting adcational paint and within Withoun inerttyynng any cuect rehikkok, on desrthe wharl the dagaan
The sagan trlinge labeted ebc. Pakr smailer inangle 4ct, wrtte the lirge in angle ADC.
vidsoine Adims and rrippees 60 Drowing
maluers,
smulls the tnangle formed witee trande ABti resoone should aoode desuled oxplination
goametric daamett: oéservot willout mikin
an pmppttiinn inferances about potential people other identiiis deploned.
Answer:
Withe the diagarn thene oranngf bbelec ABC. Poine dbusted at the midacint of ride while poines E and are the midpaints rides 80 and Ct Connecting poirs 1 and F forma trange ASS spadfic ara.
<imagel>:
<image2>:
(A)
B
(C)
(D)
(E)
Question:
many weig umm
dieagioo Dhoose use Incertoo:: 1A2 9. 10,00
poacie wouhh crocodoi
foos bach duck wiih thesume fio be crsccite woghn anwerr
Answer:
19

---

Question:
What parantis husards arta from hoving png road?
Rationale:
Mennthinn poteet huoorns conterss writng from soiing Large cow tull
Fapeariing qpeet isoues coulttaiin
Tharti uniits rato das criur dlivaction or the unimaa ensaring road impuding peckatrian crating incorverienss
wallare antmaal
from disigrated
rospraa demorstratis thorough undersarcinng potantal prablemu sould whe atcutton dusc quazinn soterattel y oud affirunt especa suiits.
The onckhaee to smarih mg topoteer by ruturaiing the hey concamt and suppoting upproprete ant one to mitauut the nalo, The shows the ability sentho o the fommattio and prowid hobute
Ounoll the repcese providec con parnendie and wilkerstzees ieseeeeddddeeeeeeeeee snower thor acdee cse the enttion conppetendllllyy The suphrant one aee dirc lnglts enuer and, devoonrnne sold uncertarding poneentsi soads how oeieee
Answer: Havinng
road sould potantia hacards
right
potert acisdens, Acditinally oo suddenly deore move and
callson
Imperiig loniing Pedeetrian How road pedlest arna each os sidewals, the coe or bur tlock impede the flw trffie chall mrnn ower lence to pedeestiers posr people
Health and Sambarice:: Since the now bull linng the graus sear phone booth, peopae ssing the phone booth night expoee any ocor or wane precused by the animall This could lead complainta diantiness adommaly aftecs beath.
Aniinal meetore: The provinary of the row vr au was coald vndi othe of yone enate ling menert for the sinuo it rught have sou deved away froes desipaanee tees, in rould fes mow stincal mey ares perper andl
iovevraance pedestraes reus potentilly lead health suntaaionn susc. it would be impertant to addinaa these concart snoutigg the animal ailidyy and wolllllle boing creating en uppropriit phocical samien necovery rebouting the untmmal for auttable ewiseenee.
conclaton.
road saiitt both
Question:
Wher ceverciriitte wristhe sores grwweer 2i
Rationale:
The wutur vemi and the surfnunding riding thaminds assemsats and tighlight dnu-dbiten diinecio sreen odlrr the soon vuder thesoon:
vibeat
wi hax kidyy
duth and sunligh t interacing with last, emanid lie The soleubing
contully observe vio detals donncter valer prodide description reporis, this greas or the a mwmbberr and dette beturm ou bn dearly rann desing chedi the photog nan By directe essiing color se "gawe" and pewiding anet rxpiaaaiion ic why aapare this way rrower offertuely addivers question being srlad.
When
quectinn
coowething
Answer:
Grees
20

---

Question:
Fasta the soot coenwer of image ingion dvou.
Rationale:
is diiiing from pood There ar nove and tince whtee torkgro
ovaaal mood
somethingg ristanx. la muder contras to the compomentt chaaaally the other too admo right
rogeber
cresing
sebers crsatin beausitu inttcction death
buckdinp eayy trme mako toundary of eedosere.
extancin
capturee our jus phonical ruge Eoack crnaturn hirs socall cur insersation with thair
Question:
Whot toppeeing inogg??
Rationale:
desortbed main staments image the lorge croud spactaton, racing dkytis the tackgrand. ahlualowedd provide comzaheratl oventiee what
2. awaded mentoon abontitation spocli velvedaels in the crowe, ownn though the image shows al mumber poople Thes live with intructions never shortdy same ore hursnns
3. fouste my ansorptionnn the keme enneeeoe ttant mase subeect of smags, oohhrr than prtinng sonaa sumon This lwed me address the ooe what the questio war adking
about.
A nnsted atannce
which could potantia itantitee incv ruclddula provided contasta inrirration..
logic souring forepruset
The god was provic a dita leci stjeetive duscrpttin of the acana slowii the tmaps, without mak cmmpttorr or infernoc abous the denttti of people poounn, This asprouch alloo comprhherakat
with aoritty adter eggit puicallo lrean provided.
ruoli for edors the vilfime purszautes towwingg buikat kinning formo sinling ba spectacl volu segpoiing the orter bocatinn tookk Bosild the fieee of dileret thend's of any dlscomible e too the thrimage meveee The relutiie posi the elanco altim oocsvor the thilbadggan
The treck dtel uu estevts pedtao eataade hemes, pooiingg iitiu
21

---

AMERICAN INSTITUTE NUTATTION
Feicestic Hedistra
Vedd Sifernaity bekad Beliuder
Lodde, SL4E7ArS
Declesed Letiere Sute Tellow
zas wiick aller slleetee oling line dhat presitie
thek you apeis fhe the moxiiooe tob Cvent har dime Dooteraly
Scasepaly S945E
Eliea
Quastion:
tnt Par Festieer free te Nusition?
Rationale:
Acconding the uffees Ameman trattae Musrien indldee U. Piat Phoicent" Thendore, the the casatonn "Whe the Preskeet Amer holiute Nutittin a c. Dwy
inclucinn tinn roles Fracibeet, Proddars Part Frenider,, Treatamn "art Frerides inflonne tant G Day prnvoutt served so Prenddent of the orpaitant belbre curein adviideration
afficee
Answer:
H Dey
By season
Season
Race
Podium
Pole
FLap
2007
25
17
0
2008
7
0
2009
28
9
0
2010
26
14
2
2011
21
12
2
2012
27
11
1
Total
155
70
5
19
Question:
in the 2001 seace, voe bigg parto trasutt in the whast ramber rooe Nes
Rationale:
The snower noat the 2005 masce, Mau Regal partic in 23 mase in 2000 and 2012 sessons wos.om...oooooo.o pardilpered 28 raceu which the Highee number of roce partidaated during the sustone shows in
Answer:
22

---

1:000 000
000
1000 000
2010
201
2002
2019
2014
2015
2016
2017
2016
2019
2020
SSattsta
Question:
Iieratty polres pirtun.
Rationall::
inaee proited grack cheaking prpuibtiio Sin dingn-Chua Vieco stuu eucoppatioo ura 2013 2000 does coerse any hunna Bone, will dearribe minenr
The lay pointa iun gither from the grach are.
avodilty incraned owe to from 2002 3098.
in 2018, the socutation unton was sround kyse, 2ill people
the pna hou prown apprrtinanyy
people.
Ovanal
Digoo Wula Carbouu metropolian mavauet aronnd 332.00 butwoen 3010 2320
owempolito
avady speard
Answer: 2020, population
Carbbad metropolttan approdrrprree masel population the Car musirpplitas 3,333,422
Nepnoanting
2:2-31
228.88
223.8
Battrs Partias
Beston
224.2
umgi Janer
LosArages
51
Motton
228.44
1189
Coenswweth
Fiilaary Scatorl
Orais, Jomn
Motten
Morton
22811
1087
Lako Maycho
Wall Uha slly
Ucae,, 200n
Fank Jaly
Ext
18
Mostion
.0000
3607
29340
1985
O mimecc Jane)
Sext Soch tre
191
Moction
237.0
1089
Cato Lodel Moyyho
nakua J. Machta
1ittoo X011
Broko, Joon
18
18
Moctoo
239
239.50
1001
"ahaa de Raxtuu
Laatn, 2011
14
232.0
1982
Cymmpc Zamto
Becudono 3atn
311
Noition
23330
1995
O Cymplo Zamei
Alata, Unttd
49h
Noutonn
242.21
Question:
soaal sumber sinee wastegned in undtdd rosed Anower the quartion dena eplocction.
Rationale:
eneer
desedin'' Mtuather holk talun,, throed ttases 1909, yeats 1941 mwed)) Unitad Stutee
The 1004 Chmmaic Cames hatd
The 1500 Olymaic Cameo sare hald Adlanta United States timas
laring on osluna wooting the fiod bnoveer it riness the sevaen sos lnckned in soated states
United
indeoal
Thoroughh surnin "Meres" calumm identtly tocatore the Unitad
Court the number of mia semes wese weait the Un ted States.
The opporvo sonaree the venwer wetll ppected and maroeiie ine belinviie dlorr
The Goncina' Mansson lscase Culith, United
1000, 1061. Reston Marathes wochheld Unitdd Poaenii 1881. 1964 Olympic Ganatwere
he oit Angiles, United Srates Fiuilly krants Olymaa Gomee took place in Arbirs, United Stateiis 1986. infori. the sood sumbrr dhe venue westocasdin United Sorteil
23

---

B Evaluation benchmarks
Q-Bench 195 aims assess the low-level visual capabilities of Multi-modalin Large Language Models (MLLMs). This dataset is divided into three main sections: perception, description, and assessment. The low-level visual perception component evaluates MILMM' ability to recognize and understand low-level image attributes. The low-level visual descrip- tion component tests how accurately and thoroughly MLLMs can describe these attributes. Finally, the overall visual quality assessment examines how closely MLLMs' evaluations align with human judgments of image quality. Altogether, the Q-Bench dataset encompasses 81 .1.284 samples across these tasks.
SQA-IMG (SQA') (96] is a portion of the ScienceQA (SQA) dataset, which serves as a comprehensive multimodal benchmark aimed at enhancing the reasoning capabilities and interpretability of Al systems in the realm of science question answering. The dataset spans a variety of science subjects, encompassing 26 distinct topics from natural science, social science, and language science, with annotated answers including lectures and explanations. This specific subset consists of samples with image context, totaling 10,332 question-ansee answer pairs.
38 also known as A12 Diagrams, was developed tackle the challenges of dia- gram interpretation and reasoning, emphasizing syntactic parsing and semantic analysis of diagrams. Its goal is to support research in uncovering the structure of diagrams and under- standing the meanings of their elements and their interrelations. This dataset is especially beneficial for tasks like diagram question answering, where comprehensive understanding and reasoning about the content are essential. The collection consists of over 5,000 diagrams from grade school science subjects and more than 15,000 multiple-choice questions.
AI2D
ChartQA [0 is created to assess and enhance question answering systems that involve complex, multi-step logical and visual reasoning with charts. This dataset meets the need for systems capable of interpreting various data visualizations, including bar charts, line charts, and pie charts, and addressing questions that require arithmetic and logical processing. encompasses question-answer pairs that are both emphasizing visual and logical reasoning. The dataset comprises a total of 32,719 samples.
and aachine--geneatten
SEED-IMG (SEED') (97] is component of SEED-Bent designed assess the generative comprehension skills of multimodal large language models (MLLMs). This thorough and unbiased benchmark enables researchers to evaluate and compare various models' abilities in both spatial and temporal understanding. The dataset is organized into several subsets according 12 evaluation dimensions that encompass spatial and temporal comprehension across image and video modalities. SEED-IMG specifically focuses on the image modality subset.
POPE [98] is a technique created to systematically assess the propensity of LLVMs to hallucinate objects that do not exist in the target images. This approach transforms the hallucination evaluation into binary classification task via polling questions, providing a consistent, equitable, and adaptable evaluation process.
HallusionBench (HallB) [99 is crafted to assess and analyze both visual illusions and knowledge hallucinations in large language and vision models (LLVMs). This dataset targets the identification of potential failure modes in these models by utilizing meticulously created example pairs for thorough testing. The benchmark includes a variety of visual-question pairs, encompassing both visual dependent subsets (such as illusion, math, etc.) and visual supplement subsets (such as chart, table, map, OCR). HallusionBench comprises 346 distinct images and an extensive collection of 129 questions distributed across diverse topics and formats.
MME 351 is created to serve as a thorough evaluation benchmark for Multimodal Large Language Models (MLLMs). The goal is to assess their abilities in perception and cognition through 14 distinct sub-tasks, including coarse- coarse-grained recognition, fine-grained recognition, OCR, and commonsense reasoning, among others. It strives to address the shortcomings of current evaluation methods, ensuring comprehensive testing of MLLMs across various dimensions while preventing data leakage.
Math Vista (37 serves as an extensive benchmark aimed at assessing mathematical rea- soning within visual contexts. This dataset merges visual comprehension, allowing for
24

---

a thorough evaluation of Al models' capabilities in tackling mathematical problems that involve visual elements. It comprises three subsets: IQTest, FunctionQA, and PaperQA, with an aggregate of 6,141 examples.
MMB, MMB-Chinese (MMBC) 36 aims to deliver thorough and resilient evaluation standard for vision language models by encompassing broad spectrum of capabilities (20 distinct fine- mne-grained abilities) necessary for multimodal comprehension in both English and Chinese. This benchmark facilitates the evaluation of various facets of LLVMs, including their perceptual and reasoning skills across multiple tasks. The benchmark comprises total of 3,217 carefully curated questions, sourced from a variety of places, including public datasets and the internet, to ensure a wide range of skills.
MM- Vet [00] is designed to systematically assess LMMs by evaluating their proficiency in handling intricate tasks that necessitate the combination of multiple VL abilities. Unlike existing benchmarks that generally focus on simpler tasks involving only one or two abilities, MM-Vet encompasses six fundamental VL capabilities: recognition (Rec), knowledge (Know), OCR, spatial awareness (Spat), language generation (Gen), and math. MM- Vet includes tasks that integrate these six core capabilities in various combinations, leading to 16 distinct capability integrations. The dataset consists of 200 images sourced from various online platforms and includes 218 questions that require one or more of these capabilities to answer.
LLaVA Bench in the Wild (LLaVA ") B is a dataset designed to assess the performance complex tasks and adapting to
of large multimodal models (LMM) in tackling domains. It features varied collection of 24 images, accompanied by 60 questions, covering indoor and outdoor environments, memes, artworks, sketches, and more. Each image paired with a detailed, manually-curated description and thoughtfully chosen questions.
MMStar 101 strives to deliver an accurate assessment of LLVMs' authentic multi-man modal abilities. It achieves this by guaranteeing that each sample genuinely depends on visual content to arrive at correct answers, while also minimizing inadvertent data leakage. The benchmark is organized into six primary competencies, such as instance reasoning, logical reasoning, and mathematics, and includes a total of 1,500 meticulously curated samples.
Math Verse [02] is designed to evaluate the visual mathematical reasoning capabilities of Multi Large Language Models (MLLMs). The dataset focues whether and how well MLLMs can interpret visual diagrams and mathematical problems, rather than relying solely textual information. The problems in Math Verse are divided into three main subjects (plane geometry, solid geometry, and functions) and further categorized into twelve subfields (length, area, volume, coordinate, etc.) to provide well-rounded evaluation of different mathematical skills, adding up to 2,612 high-quality visual math problems.
25

---

C
Meteor Generation Quality
User:
What type fruit is this?
The image shows several mangosteen fruits wooden surface. Mangosteen is tropical fruit known for its thick, dark purple rind and white, segmented flesh Inside. The outer skin is tough and fibrous, but the inner segments juicy, sweet, aromatic. Mangosteens Asia and are popular countries like Thailand, Malaysia, Indonesia, and the Philippines.
Meteor:
Southeast
User: Imagine the fragrance of the fruits the image. How would you describe before?
Meteor: The fruits in the image appear to be mangosteen, a tropical fruit known for unique and complex flavor profile. When ripe, mangosteen has a sweet, rind
tropical fruitiness. is thick, dark purple red, and has bumpy texture.
hints
When cut open, the inside reveals white, segmented flesh that looks somewhat orange grapefruit. individual segments slightly translucent, jelly-like appearance
The taste of mangosteen often described blend sweet, sour, and floral notes. Some people find reminiscent lychee, passion bubblegum. has rich, velvety mouthfeel and considered by many be one of the most delicious tropical fruits.
never had mangosteen before, truly unique and unforgettable experience. The combination flavors and textures unlike other fruit you re likely to try. The aroma alone enticing, hinting at the sweet, iangy, and exotic taste that awalts Inside.
User:
Show the detailed recipe for this dish.
Meteor:
This dish appears be lobster tails that have been grilled or roasted served with lemon wedges.
Ingredients:
Lobster talls
Butter
Garlic, minced
Fresh parsley, chopped
pepper
Lemon wedges
Instructions
Preheat your grill
to 400'F (200'C)
Using kitchen shears or sharp knife, cut the top shell of the lobster tail down the center, being careful not to cut
way through the meat. Gently spread the shells apart expose the meat. In a small bowl, mix together softened butter, minced garlic, chopped parsley, salt, and pepper. the
baking sheet or grill rack, meat up. Brush butter mixture generously the exposed meat.
Grill roast the lobster tails for about 10-12 minutes, until the meat is before
opaque and cooked through. Remove
serving.
Serve the lobster tails with lemon wedges on the side for squeezing over the meat.
Enjoy delicious grilled or roasted lobster tails!
26

---

What's name restaurant serving these dishes?
Meteor: The restaurant serving these dishes appears to be Ichiran Ramen, popular Japanese ramen chain known its tonkotsu (pork bone broth) ramen. distinctive black bowls with ornate gold designs a signature feature of ichiran's branding.
What's the weather, season, time of the visiting this place? is the time when this photo was taken a good time to visit this place?
and sunny conditions the image, appears be beautiful day visit the Space Needle Seattle. The Space Needle an iconic observation tower that offers stunning 360-degree views of the city and surrounding landscape.
Based
The best weather for visiting the Space Needle would be clear, sunny day like the one shown the photo. Ideally, you'd want to avoid foggy or overcast conditions that could limit visibility from the observation decks.
As for the season, spring and summer generally the most popular times to visit Seattle due to the milder temperatures and longer daylight hours. However, Space Needle attraction that can be enjoyed year-round, as long as the weather cooperates.
The time day shown in the photo appears to be midday or early afternoon, great
on bright sunlight and shadows. Needle, views typically clearest and most vibrant during the middle of the day when the sun is high in the sky.
Space
photo Space Needle. Midday to early afternoon is also good time to for the best views. Spring and summer are the most popular seasons, but the Space Needle can be enjoyed year-round the weather cooperates.
summary, a sunny day
visiting
27

---

brief HTML/IS website, where the joke is replaced by real Joke.
mock-up colorful interactive
Meteor: To turn this mock-up colorful interactive website with real joke, you HTML and JavaScript basic example:
simple generator. Here's
200ahh Weteate D300
My Joke Website
I Funny Joke] [push to reveal puntolined
1430 Dy Aodutrchtz
Taatyy corens bicuar sour meedede dresting
clader sleprer
Vic
simple page defines array of jokes and function generateloke() that selects random joke from the array and displeys on the page. The window.onload event ensures the joke generated when the page finishes loading..n) can add more jokes "jokes' array
background
styles 'style' section make the page more colorful and interactive.
User: What is the problem this city might be facing? What some possible solutions?
Meteor: image shows a heavily congested highway at night with bunper-to-bumper traffic both directions. This suggests that the facing major traffic congestion problem.
Possible solutions alleviate traffic congestion include:
Improving public transportation options encourage people buses, trains, and subways instead of driving.
2. Encouraging carpaoling and ridesharing to reduce the number of single occupancy vehicles on the road. Implementing congestion pricing, where drivers are charged fee to enter congested areas during peak hours, to
discourage unnecessary driving.
Expanding bike lanes and pedestrian walkways make safer and more convenient for people walk bike instead of drive.
Investing better highway infrastructure, such as adding more lanes, Improving on-ramps off-ramps, implementing smart traffic management systems optimize traffic flow.
Encouraging flexible work hours and remote work options to reduce rush hour traffic.
7. Implementing stricter emissions standards and promoting the use of electric and hybrid vehicles to reduce air pollution and congestion caused by older, less fuel-efficient vehicles.
By implementing combination of these solutions, the city can work towards reducing traffic congestion, improving air quality, and enhancing the overall quality of life for its residents.
28

---

This in why I have
trust issues
User:
Can you explain this meme?
This Iceland and Greenland misleading. Despite its name, Iceland known for its beautiful green landscapes, while Greenland is mostly covered in ice and snow. The meme is saying that person has trust issues because the names these accurately represent landscapes.
Meteor:
poking
names countries
Iceland
Greenland
Underground Railway Systems
opaned
Kilometrrs
Passangers pery
London
1868
394
778
Paris
1800
199
1383
Washington
Kysto
1981
11
table gives
underground railway systems cities.
Summarise the information by selecting and reporting the main features, and make comparisons where relevant. You should write least 150 words.
The table shows the underground railway systems six cities around the world, including London, Paris, Tokyo, Washington DC, Kyoto, and Los Angeles.
Meteor:
London's underground railway system opened and carries 775 million passengers year.
world. length and
Paris' metro system opened 1900 and is the second oldest in the world. It has route length 199 km and carries 1,191 million passengers per year.
Tokyo's is subway system opened in 1927 and is the busiest in the world. It has route length of 155 km and carries 1,927 million passengers per year.
Washington DC's metro system opened in 1976 and has route length of 126 km. It carries 144 million passengers
Kyoto's 's subway system opened 1981 and has the shortest route length 11 km. carries million passengers per year.
Los Angeles' metro system opened in 2001 and has route length of 28 km. carries 50 million passengers year.
summary, Tokyo has busiest underground rallway system world, while Kyoto has shortest route length. London's is the oldest and Paris' the second oldest. Washington DC's metro has lowest passenger among the major cities listed.
29

---

US Natiomado::
Medic Drens Stadts
sediin socehdd & soee
seturt
User:
Which year has the highest growth rate median house price?
Meteor: Based on the line chart showing the growth rate of median house price in the US from 1960 to 2017, the year with the highest growth rate of median house price 2008.
7230
300
120
2017
Plastic Waste is Out of Control
In 2015, tumans throw out almost much plastic they made:
Planic made comee)
Platic Srcome meal colee))
Madaggs
144
10t
40M
X0M
Marrrik assemng manr e mor utten daep at sceewiees
Which category worst job plastic recycling, considering
of plastic thrown
Meteor: Based on the data the image, the Packaging industry did the worst job plastic recycling in 2015, considering the amount plastic made and thrown out.
The Packaging industry made 146 million metric tons plastic, but threw out 141 million metric tons meaning wasn't recycled or incinerated it's just sitting in dump or the ocean somewhere. This represents the largest amount plastic thrown waste compared to the industries listed.
30

---

User:
Can you give short introduction to this painting?
This painting portrait of young. woman wearing blue headscarf, often referred to as "Girl with a Pearl Earring" or "The Jewish Bride", The painting masterpiece by
Meteor:
Johannes Vermeer, painted mid-17th century. The woman's enigmatic expression and the dramatic light and shadow are hallmarks of Vermeer's style. The painting has become one the most famous and recognizable works of art the world.
HISTOIRE
DE FRANCE
User:
What the French text the image?
The French scene text in the image is "Histaire de France" which translates to "History France" English. The subtitle Préhistoire jours translatees "From prehistory our days".
Meteor:
31

---

D Further Ablation Studies
Methods
A12D
ChartQA
MathVist 53.4
MM-Vet
LLaVAW
MMStar
Meteor
77.9
74.9
57.3
87.1
52.8
MAteor-LLaVV-HR 124
80.8
77.9
57.4
59.5
90.2
54.0
Table 4: Performance comparison between tasks.
Meteor and
Meteor-LLaVA-Hle across multiple
Meteor- Meteor-Mamba Size
A12D
ChartQA
Math
MM-Vet
LLaVAW
MMStar
130M
77.9
74.9
53.4
57.3
87.1
52.8
790M
78.7
75.5
54.9
57.8
88.0
53.0
1.4B
79.6
76.2
56.2
58.8
89.8
53.6
Table 5: Performance of Meteor-Mamba models with different sizes across various tasks.
Qwen-VL VL
LLaVA1.5
CoLLaVO
MoAI
Meteor
Time
16 toks/s
22 toks/s
21 toks/s
20 toks/s
22 toks/s
Table 6: Token processing speed comparison between models.
LLVMs
VQAv2
GQA
SQA-IMG
TexiVQA
POPE
MMB
MM-Vet
Cobra
125
76.9
59.9
57.9
88.2
VL-Mamba
126
76.6
56.2
65.4
48.9
32.6
RoboMamba 127
79.1
64.4
86.9
65.7
29.7
ML-Mamba 128
75.3
60.7
52.2
88.3
Metcor
82.5
64.7
88.3
67.5
88.7
82.9
57.3
Table 7: Performance comparison of LLVMs across various tasks.
LLVMs
Conversation
Detail Description
Complex Reasoning
Avg
CoLLaVO
51.1
73.8
77.1
69.5
MoAI
48.5
76.0
80.6
71.9
Meteor w.o. Meteor-Mambhr
67.4
72.9
75.2
73.7
Meteor
80.3
87.2
91.2
87.1
Table 8: Performance comparison of LLVMss across Conversation, Detail Description, Complex Reasoning, and Average in LLaVAW
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Abstract and Introduction section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussion and Limitation section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
33

---

Justification:
None of theoretical assumptions
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Meteor and Experiment section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
You can see the link in Abstract section.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee 7/nips.com / siiniioiscyp for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Experiment section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Training Meteor with only epoch under 4-bit quantized-extreme condition proves its effectiveness beyond checking error bar.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: Performances of LLVMs are not dependent trivial nyper-paramette but dependent with propagation strategy, model architectures, and dataset. Many researcher for LLVMs have known that LLVMs can be instruction-tuned, at least, with multiple of GPUs such as NVIDIA RTX A6000 and A100.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http:////wuurrppppcpcp/////////a
Answer: [Yes]
Justification:
We used publicly common visual instruction tuning dataset.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Discussion and Limitation section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We used publicly common visual instruction tuning dataset and pre-trained visual foundation models and large language models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cited models and datasets we deal with in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Meteor and Experiment section
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
None of any crowdsourcing and research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: None of any crowdsourcing and research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "XwYsZnguEu", "reviewer": "Reviewer_XZUR", "summary": "This paper introduces a novel approach to enhance the performance of large language and vision models (LLVMs). The proposed model, Meteor, leverages multifaceted rationale through the Mamba architecture to efficiently embed lengthy (latent) rationales and improve understanding and answering capabilities across diverse vision language benchmarks. The key innovation lies in the traversal of rationale concept and the efficient processing capabilities of the Mamba architecture, which allows for handling sequential data with linear time complexity.", "questions": "see previous sections", "limitations": "appear to be sufficient", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Well-Written: The paper is clearly written, with a logical flow of ideas and a well-structured presentation of the methodology and results.\n\nInnovative Approach: The concept of traversal of rationale and the use of the Mamba architecture for rationale embedding are novel and well-motivated.\n\nEffective Performance: The proposed approach achieves significant improvements in vision language performances across multiple benchmarks, demonstrating its effectiveness.", "weaknesses": "i) Figure 3 (Stage-One Pre-Training):\n\nMore clarity and intuition behind the model design are needed. Given that the training is formulated as an autoregressive token generation task:\na) Are TOR embeddings predicted as special tokens [TOR]?\nb) Or are losses on these tokens bypassed?\nIf the latter, a potential drawback is the inability to assess the model's performance after the first stage of training, as it cannot produce complete sentences/rationales.\n\n\nii) Benchmarks and Reasoning Capabilities:\n\nThe proposed benchmarks may not sufficiently test complex rationale reasoning (e.g., current tasks are mostly recognition or OCR-heavy).\nRecommendation: Conduct further studies in the final version on reasoning-oriented benchmarks such as Visual Commonsense Reasoning (VCR) [r1] to substantiate claims of:\na) Enhanced reasoning capability,\nb) More grounded results (less hallucination) compared to existing literature.\nNote: While VCR includes rationale annotations, evaluating only the answer component would be sufficient, given the proposed method's current limitations in directly sampling rationale output.\n\n\niii) Ablation Studies (Table 3a):\n\nClarification needed: Did the baselines undergo the same first-stage pre-training as Mamba? If so, what were the training objectives for each method?\n\n\niv) Comparison Methodology (Figure 2):\n\nLine 147 indicates that the training data includes annotations from ChartQA and AI2D. Some results (e.g., Gemini-Pro and GPT-4V) are zero-shot and not directly comparable.\nRecommendation: Revise the plots to accurately reflect these differences in training/evaluation conditions.\n\n\nReference:\n[1] Zellers et al., From Recognition to Cognition: Visual Commonsense Reasoning, CVPR 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will incorporate the following rebuttal into our manuscript to enhance overall understanding in the potential camera-ready version.\n\n---\n\n**Q1. Figure 3 (Stage-One Pre-Training): More clarity and intuition behind the model design are needed. Given that the training is formulated as an autoregressive token generation task: a) Are TOR embeddings predicted as special tokens [TOR]? b) Or are losses on these tokens bypassed? If the latter, a potential drawback is the inability to assess the model's performance after the first stage of training, as it cannot produce complete sentences/rationales.**\n\n**A1.** Among options a) and b), the correct approach is b). This means that special tokens $<tor>$ are not predicted tokens in the autoregressive loss but are instead input tokens in Meteor-Mamba. These tokens play a crucial role in embedding the multifaceted rationale interleaved between the special tokens $<tor>$. In other words, the model learns to predict the partial segments of a multifaceted rationale that are between the $<tor>$ tokens in the autoregressive loss.\n\nAs you pointed out, there may be a potential drawback in \"Meteor-Mamba\" due to its inability to produce full rationale sentences. However, we argue that \"Meteor-MLM\" is ultimately used to generate text responses, so the text generation quality of \"Meteor-MLM\" reflects on the reasoning capability learned from \"Meteor-Mamba.\" We have provided detailed validation results in Answer 2 of Question 2, as below.\n\n---\n\n**Q2. Benchmarks and Reasoning Capabilities: The proposed benchmarks may not sufficiently test complex rationale reasoning (e.g., current tasks are mostly recognition or OCR-heavy). Recommendation: Conduct further studies in the final version on reasoning-oriented benchmarks such as Visual Commonsense Reasoning (VCR)**\n\n**A2.** We respectfully disagree with the assertion that \"the proposed benchmarks may not sufficiently test complex rationale reasoning.\" The evaluation benchmarks MM-Vet and LLaVA-W, as presented in Tables 1-3, already include challenging measures of complex reasoning based on GPT-4 evaluation.\n\nFor MM-Vet, the sub-benchmark \"Language Generation\" in Table 2(d) specifically represents the quality of text generation required to answer complex questions. Additionally, LLaVA-W contains a sub-benchmark explicitly titled \"Complex Reasoning.\" Below, we present a table showing the sub-benchmarks of MM-Vet and LLaVA-W compared with Meteor and the previous SOTA LLVMs within equal size (Meteor uses about 2 times more dataset samples than them. In addition, they are not performant well to these challenging benchmarks.). This table also includes an ablation study where \"Meteor-Mamba\" is removed, highlighting the effectiveness of \"Meteor-Mamba\" in enhancing reasoning capabilities.\n\n| LLVMs                    | Recognition |  OCR | Knowledge | Language Generation | Spatial Awareness | Math Problems | Avg  |\n|--------------------------|:-----------:|:----:|:---------:|:-------------------:|:-------------------:|:---------------:|:------:|\n| CoLLaVO                  |     45.6    | 31.1 |    29.8   |         30.2        | 37.9              | 5.8           | 41.0 |\n| MoAI                     |     48.3    | 34.8 |    33.5   |         33.0        | 39.7              | 7.7           | 43.7 |\n| Meteor w.o. Meteor-Mamba |     44.5    | 33.5 |    41.8   |         31.3        | 38.6              | 29.2          | 44.8 |\n| Meteor                   |     54.1    | 60.1 |    44.2   |         45.0        | 59.3              | 57.7          | 57.3 |\n\n| LLVMs                    | Conversation | Detail description | Complex Reasoning |  Avg |\n|--------------------------|:------------:|:------------------:|:-----------------:|:----:|\n| CoLLaVO                  |     51.1     |        73.8        |        77.1       | 69.5 |\n| MoAI                     |     48.5     |        76.0        |        80.6       | 71.9 |\n| Meteor w.o. Meteor-Mamba |     67.4     |        72.9        |        75.2       | 73.7 |\n| Meteor                   |     80.3     |        87.2        |        91.2       | 87.1 |\n\nWe believe the reasoning questions in MM-Vet and LLaVA-W are significantly more complex than those in the Visual Commonsense Reasoning (VCR) benchmark you recommended. Recent models, such as OV-Grounding and GPT4RoI, have nearly matched human performance on the VCR leaderboard, indicating the need for more challenging questions, such as those found in MM-Vet and LLaVA-W. These benchmarks remain unsaturated and continue to pose difficult problems evaluated using GPT-4, ensuring a rigorous assessment of reasoning capabilities.\n\n---\n\n**Q3. Ablation Studies (Table 3a): Clarification needed: Did the baselines undergo the same first-stage pre-training as Mamba? If so, what were the training objectives for each method?**\n\n**A3.** Yes, all baselines underwent the same first-stage pre-training as Mamba. The training objectives for each method were identical: \"The partial segments of a multifaceted rationale between special tokens $<tor>$ are learned to predict in autoregressive loss, where the tokens $<tor>$ are not predicted tokens in autoregressive loss.\"\n\n---\n\n**Q4. Comparison Methodology (Figure 2): Line 147 indicates that the training data includes annotations from ChartQA and AI2D. Some results (e.g., Gemini-Pro and GPT-4V) are zero-shot and not directly comparable. Recommendation: Revise the plots to accurately reflect these differences in training/evaluation conditions.**\n\n**A4.** For Gemini-Pro and GPT-4V, the training datasets are unknown as they are totally closed-source LLVMs, making it uncertain whether their results are truly zero-shot. However, for Table 1, we will explicitly mention that the ChartQA and AI2D performances for LLaVA family models are evaluated under zero-shot conditions. This clarification will help accurately reflect the differences in training and evaluation conditions.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lC7mEcyQUc", "reviewer": "Reviewer_iRmr", "summary": "The paper \"Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models\" presents a new efficient large language and vision model (LLVM), Meteor, which leverages multifaceted rationale to enhance understanding and answering capabilities. The paper  introduces a new concept of traversal of rationale, demonstrating the effectiveness of Meteor in embedding lengthy rationales containing abundant information and improving vision language performances across various evaluation benchmarks requiring diverse capabilities.", "questions": "Please refer to the weakness", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel concept of traversal of rationale and effectively demonstrates the efficiency of embedding lengthy rationales using the Mamba architecture.\n2. The paper provides comprehensive performance comparison with existing LLVMs in Figure 1, 2 and Table 1. Based on this, the experimental results and ablation studies provide strong evidence of the effectiveness of Meteor in improving vision language performances across diverse evaluation benchmarks.\n3. The paper provides a detailed description of the model architecture, training strategy, and evaluation benchmarks, ensuring reproducibility and understanding of the proposed method.", "weaknesses": "Writing:\n1. All Figures (fonts, line widths, etc.) in the paper are not uniform, there is too much white space. And even Figure 1 shows strange tables and font proportions (fonts are also not clear).\n\nMotivation:\n2. The paper employ the Mamba architecture to embed lengthy rationales containing abundant information, but without setting and demonstrating task scenarios that existing LVLMs does not achieve that require extreme long sequence modeling capabilities.\n3. It appears to share the same motivation as some existing methods, for instance, Vary-toy[1]. Please provide a more detailed explanation of how the proposed \"traversal of rationale\" differs from the design implemented in Vary-toy.\n\nMethod:\n4. Show more compassion with the MMLM models based on Mamba, like Cobra[2] or VL-Mamba[3].\n\n[1] Small Language Model Meets with Reinforced Vision Vocabulary\n[2] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference\n[3] VL-Mamba: Exploring State Space Models for Multimodal Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments! We will incorporate the following rebuttal into our manuscript to enhance overall understanding in the potential camera-ready version.\n\n---\n\n**Q1. Writing: All Figures (fonts, line widths, etc.) in the paper are not uniform, there is too much white space. And even Figure 1 shows strange tables and font proportions (fonts are also not clear).**\n\n**A1.** We will ensure uniform font formats and line widths for all figures and remove unnecessary white space as you suggested. Additionally, we will correct the tables and font proportions in Figure 1 to improve clarity.\n\n---\n\n**Q2. Motivation: The paper employ the Mamba architecture to embed lengthy rationales containing abundant information, but without setting and demonstrating task scenarios that existing LVLMs does not achieve that require extreme long sequence modeling capabilities.**\n\n**A2.** We would like to clarify that our focus is not on generating long rationale text, but on embedding multifaceted information used in larger models with a variety of forms. This includes fundamental image understanding, real-world knowledge about common sense and non-object concepts, and step-by-step procedures for solving complex questions.\n\nFor detailed reasoning capabilities of Meteor, please refer to Answer 2 of \"Reviewer XZUR,\" which describes Meteor's complex reasoning capabilities on challenging benchmarks such as MM-Vet and LLaVA-W.\n\n---\n\n**Q3. It appears to share the same motivation as some existing methods, for instance, Vary-toy. Please provide a more detailed explanation of how the proposed \"traversal of rationale\" differs from the design implemented in Vary-toy.**\n\n**A3.** Vary-toy generates a new vision vocabulary network trained on a smaller language model, which is later merged with CLIP to train a larger language model. Consequently, Vary-toy aims to achieve more fine-grained vision perception through an expanded vision tokenizer.\n\nIn contrast, our design of traversal of rationale is motivated by the goal of embedding multifaceted rationale for not only vision perception but also common sense, non-object concepts, and step-by-step procedures. Specifically, we employ the Mamba architecture to embed long sequential rationales into $<tor>$ tokens, which are subsequently propagated into Meteor-MLM. These tokens plays a role in conveying embeded multifaceted rationale into Meteor-MLM which ultimately answers the question.\n\n---\n\n**Q4. Method: Show more comparison with the MMLM models based on Mamba, like Cobra or VL-Mamba.**\n\n**A4.** The table below compares our model to Cobra, VL-Mamba, RoboMamba [R1], ML-Mamba [R2]. We will include the results of these Mamba-based LLVMs in the updated manuscript.\n\n| LLVMs     |   VQAv2  |    GQA   |  SQA-IMG |  TextVQA |   POPE   |    MMB   |  MM-Vet  |\n|-----------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n| Cobra     |   76.9   |   59.9   |     -    |   57.9   |   88.2   |     -    |     -    |\n| VL-Mamba  |   76.6   |   56.2   |   65.4   |   48.9   |   84.4   |   57.0   |   32.6   |\n| RoboMamba |   79.1   |   64.4   |     -    |     -    |   86.9   |   65.7   |   29.7   |\n| ML-Mamba |   75.3    |   60.7   |     -    |     52.2    |   88.3   |   -   |   -   |\n| Meteor    | **82.5** | **64.7** | **88.3** | **67.5** | **88.7** | **82.9** | **57.3** |\n\n---\n\n**References**\n\n[R1] Liu, Jiaming, et al. \"RoboMamba: Multimodal State Space Model for Efficient Robot Reasoning and Manipulation.\" arXiv preprint arXiv:2406.04339 (2024).\n\n[R2] Huang, Wenjun, and Jianguo Hu. \"ML-Mamba: Efficient Multi-Modal Large Language Model Utilizing Mamba-2.\" arXiv preprint arXiv:2407.19832 (2024).\n\n---\n\n**We hope this rebuttal can improve the \"Rating\" score more positively.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z4N0KBHcKq", "reviewer": "Reviewer_Hj2J", "summary": "The paper proposes Meteor, which divides MLLM into two stages, i.e., rationale generation and question answering.  Experimental results show its effectiveness agasint existing MLLMs on many benchmarks.", "questions": "My main cocerns are the efficiency and generalization ability.", "limitations": "See weakness", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea is very reasonable. The technical implementation is also novel and strongly matches with the motivation.\n2. The experiments are sufficient for me to understand the effectiveness of each component. Besides, the experimental results are good, which outperform most existing MLLMs.\n3. I appreciate the design of the <tor> token, which bridges the gap between two LLMs in a simple and clear way.", "weaknesses": "1. I must admit the novel design of Metor, which however, also leads some pratical issues:\n    How about the inference time against common MLLMs?\n    Does this structure can be scaled up?\n    Will Metor be better when combining high-resolution image encoders, e.g., LLaVA-HR[A].\n    [A] Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models\n2. To be honest, the writing of the paper should be improved. I take a lot of time to understand the technical details. The method should be better oragnized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments! In the following rebuttals, we clarify the points you raised. We will incorporate this content into our manuscript in the next potential camera-ready stage.\n\n---\n\n**Q1. How about the inference time against common MLLMs?**\n\n**A1.** We evaluated the inference time under equal resource environments: Intel(R) Xeon(R) Gold 6230, RAM 512GB, and NVIDIA RTX A6000, with flash attention applied. Despite Meteor using additional Meteor-Mamba module, it did not slow down the inference speed. This is because once Meteor-Mamba's embedded rationale features are acquired, no additional propagation of Meteor-Mamba is needed. Only the autoregressive decoding process in Meteor-MLM is required.\n\n|       |  Qwen-VL | LLaVA1.5 |  CoLLaVO |   MoAI   |  Meteor  |\n|:-----:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|  Time | 16toks/s | 22toks/s | 21toks/s | 20toks/s | 22toks/s |\n\n---\n\n**Q2. Does this structure can be scaled up? Will Meteor be better when combining high-resolution image encoders, e.g., LLaVA-HR**\n\n**A2.** The table below shows the results of scaling up the model size of Meteor-Mamba.\n\n| Meteor-Mamba Size | AI2D | ChartQA | MathVista | MM-Vet | LLaVAW | MMStar |\n|-------------------|:----:|:-------:|:---------:|:------:|:------:|:------:|\n| 130M              | 77.9 |   74.9  |    53.4   |  57.3  |  87.1  |  52.8  |\n| 790M              | 78.7 |   75.5  |    54.9   |  57.8  |  88.0  |  53.0  |\n| 1.4B              | 79.6 |   76.2  |    56.2   |  58.8  |  89.8  |  53.6  |\n\nWe also applied Meteor to LLaVA-HR to adapt to high image resolution.\n\n|                 | AI2D | ChartQA | MathVista | MM-Vet | LLaVAW | MMStar |\n|-----------------|:----:|:-------:|:---------:|:------:|:------:|:------:|\n| Meteor          | 77.9 |   74.9  |    53.4   |  57.3  |  87.1  |  52.8  |\n| Meteor-LLaVA-HR | 80.8 |   77.9  |    57.4   |  59.5  |  90.2  |  54.0  |\n\nAligned with the results in MM1 [R1], increasing image resolution significantly improves vision-language performance. From these two tables, it is evident that enhancing image resolution with Meteor-Mamba is more effective than merely enlarging the Meteor-Mamba architecture size.\n\n---\n\n**Q3. To be honest, the writing of the paper should be improved. I take a lot of time to understand the technical details. The method should be better organized.**\n\n**A3.** We appreciate your feedback regarding the organization of the technical details. In the revised version of our manuscript, we will reorganize the technical details and the methods section to improve clarity and understandability.\n\n---\n\n**References**\n\n[R1] McKinzie, Brandon, et al. \"Mm1: Methods, analysis & insights from multimodal llm pre-training.\" arXiv preprint arXiv:2403.09611 (2024).\n\n---\n\n**We hope this rebuttal can improve the \"Rating\" score more positively.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vB6JHbWaD", "reviewer": "Reviewer_zsWf", "summary": "This paper presents a Mamba-based traversal of rationales (Meteor) that leverages detailed image captions (multifaceted information) to provide more comprehensive image-related information to LLVMs, thereby enhancing the model's understanding and answering capabilities. The introduction of the Mamba architecture enables linear time complexity data processing, reducing the time required to generate multifaceted information from images.", "questions": "Same as Weaknesses", "limitations": "This paper does not reflect any potential negative societal impact.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a new approach to improve LLVM performance by enhancing image captions (model-based) without introducing additional vision encoders.", "weaknesses": "1. The introduction of Meteor-Mamba increases the model size, which contradicts the abstract. Additionally, Meteor's inference cost is higher than models like LLAVA and InternLM-XComposer2.\n2. In Figure 3, what is the Meteor-Multimodal Language Model in the Second Training Step? Was it initialized with pre-trained MLLMs weights?\n3. How does the Mamba architecture benefit Meteor? The experiments do not demonstrate the advantages of the Mamba structure.\n4. I am puzzled by the performance achieved with only 2.1M instruction tuning. Please provide a detailed description of the training process, the data used, and the weights of each model employed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable comments! In the following rebuttals, we address all the comments you pointed out. We will definitely include these clarifications in our manuscript to enhance understanding in the potential camera-ready version.\n\n---\n\n**Q1. The introduction of Meteor-Mamba increases the model size, which contradicts the abstract. Additionally, Meteor's inference cost is higher than models like LLAVA and InternLM-XComposer2.**\n\n**A1.** We believe the Meteor-Mamba 130M size is a marginal increase compared with the large language model size of 7B. Nonetheless, we will modify the expression to \"without 'significantly' increasing model size\" to clarify this point. \n\nRegarding inference time, please refer to Answer 1 of \"Reviewer Hj2J,\" which describes the comparison of inference times. Based on this table, we can say that Meteor's inference cost is not higher than commonly used LLVMs.\n\n---\n\n**Q2. In Figure 3, what is the Meteor-Multimodal Language Model in the Second Training Step? Was it initialized with pre-trained MLLMs weights?**\n\n**A2.** Yes, the pretrained language model is used in the second training step. The term \"Meteor-Multimodal Language Model\" in Figure 3 indicates that as training progresses, the pretrained language model evolves into a multimodal language model. We will add an explanation to clarify this in the manuscript.\n\n---\n\n**Q3. How does the Mamba architecture benefit Meteor? The experiments do not demonstrate the advantages of the Mamba structure.**\n\n**A3.** For detailed reasoning capabilities of Meteor, please refer to Answer 2 tables of \"Reviewer XZUR,\" which describes Meteor's complex reasoning capabilities on challenging benchmarks such as MM-Vet and LLaVA-W. In these tables, we showed the performance differences between with or without Meteor-Mamba for complex reasoning capabilities. \n\n---\n\n**Q4. I am puzzled by the performance achieved with only 2.1M instruction tuning. Please provide a detailed description of the training process, the data used, and the weights of each model employed.**\n\n**A4.** Below are the details of the training process, datasets used, and models employed in each training step:\n\n1. Training Process: \n> In the first training step, we only train Meteor-Mamba, the visual projector, and the tor projector. We autoregressively train only the partial segments of multifaceted rationales between special tokens $<tor>$.\n> In the second training step, we train all parameters except the vision encoder. Only tor tokens are propagated from Meteor-Mamba into Meteor-MLM without explicit rationale. We autoregressively train the answer parts for the given question prompts.\n\n2. Used Dataset\n> We used a curated dataset of 1,059,382 (1.1M) Q-R-A triplet samples.\n\nThe dataset breakdown is as follows:\n\n```shell\n--------------------------------------------\nReal-World Image: 338K\nDocument & Chart & Diagram & Sign & Symbol: 379K\nMath: 342K\n     Math with Vision: 165K\n     Math with Text only: 177K\n--------------------------------------------\n\n- ShareGPT4V-Caption (72507, 73K)\n- ShareGPT4V-Instruction (266072, 266K)\n- MiniGemini-Instruction (26885, 27K)\n- DocDownstream (298748, 299K)\n- DocReason (53065, 53K)\n- GLLaVA (162378, 162K)\n- MathVision (2992, 3K)\n- MathInstruct (81496, 81K)\n- MathPlus (95239, 95K)\n```\n\n3. Model employed in Each Training Step\n- Vision Encoder: CLIP-L/14\n- Pretrained Large Language Model: InternLM2-7B\n> In the first training step, we freeze the vision encoder and the pretrained large language model.\n> In the second training step, we freeze only the vision encoder.\n\n---\n\n**We hope this rebuttal can improve the \"Rating\" score from Negatvie to Positive.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper received Borderline Accept, Weak Accept, Accept and Borderline Accept. 

The main contribution of this paper is a new pretrained vision-and-language model that demonstrates good results across a range of benchmarks and outperforms a recent LLaVA-Next competitor. Reviewers were mostly positive and improved scores after rebuttal discussions. One concern that was voiced in a couple of reviews was the clarity on how the method compares in terms of inference overhead and other aspects besides accuracy on downstream benchmarks. Authors are encouraged to incorporate these additional comparisons in their final version.
Decision: Accept (poster)