Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models

Authors: byung-kwan  lee
chae won kim
beomchan  park
yong man ro
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 20:55:50

Abstract: The rapid development of large language and vision models (LLVMs) has been driven by advances in visual instruction tuning. Recently, open-source LLVMs have curated high-quality visual instruction tuning datasets and utilized additional vision encoders or multiple computer vision models in order to narrow the performance gap with powerful closed-source LLVMs. These advancements are attributed to multifaceted information required for diverse capabilities, including fundamental image understanding, real-world knowledge about common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions. Drawing from the multifaceted information, we present a new efficient LLVM, Mamba-based traversal of rationales (Meteor), which leverages multifaceted rationale to enhance understanding and answering capabilities. To embed lengthy rationales containing abundant information, we employ the Mamba architecture, capable of processing sequential data with linear time complexity. We introduce a new concept of traversal of rationale that facilitates efficient embedding of rationale. Subsequently, the backbone multimodal language model (MLM) is trained to generate answers with the aid of rationale. Through these steps, Meteor achieves significant improvements in vision language performances across multiple evaluation benchmarks requiring diverse capabilities, without scaling up the model size or employing additional vision encoders and computer vision models.

Main paper: 
 introduction: 
 Following the successful zero-shot achievements of instruction-tuned large language models (LLMs) [1, 2] , visual instruction tuning [3] has spurred the rapid development of large language and vision models (LLVMs). The emergence of closed-source LLVMs, such as GPT-4V [4] , Gemini-Pro [5] , and Qwen-VL-Plus [6] , has prompted several studies to create high-quality question-answer visual instruction tuning datasets [6] [7] [8] [9] [10] [11] and to scale up the model sizes of open-source LLVMs [10, [12] [13] [14] [15] , aiming to compete with their closed-source counterparts by leveraging the scaling law [16, 17, 2] . Meteor on MME [35] , MMB [36] , MathVista [37] , and AI2D [38] requiring diverse capabilities for image understanding, common-sense knowledge, non-object concept understanding, etc.for tasks such as segmentation, detection, scene graph generation, and optical character recognition (OCR) to enhance LLVMs' answering capabilities with the help of external perception information.These efforts, along with the curation of high-quality visual instruction datasets, have significantly reduced the performance gap between open-and closed-source LLVMs across numerous evaluation benchmarks and have even led to superior performances on some benchmarks. These successful developments are credited to the multifaceted information necessary for a wide range of capabilities. This encompasses fundamental image understanding, real-world knowledge of common-sense and non-object concepts (e.g., charts, diagrams, symbols, signs, and math problems), and step-by-step procedures for solving complex questions.Inspired by the key importance of the multifaceted information, we explore the possibility of designing efficient LLVMs that implicitly embed it as a form of multifaceted rationale (See Appendix A for more details), without significantly increasing model size and without using additional explicit vision encoders and computer vision models during the inference phase. Hence, we present a new efficient LLVM, Mamba-based traversal of rationale ( Meteor), comprising two core components: the Mamba architecture [39] and a multimodal language model (MLM) based on a pretrained large language model (LLM). The multifaceted rationale has rich information for achieving diverse capabilities, so its length is inherently long. This is why we employ the Mamba architecture, hereinafter referred to as Meteor-Mamba, which takes advantage of embedding lengthy input. It serves as an embedding module for the rationale, enabling Meteor-MLM, the MLM component, to address questions with the help of these embedded rationales. When conveying the knowledge of embedded rationales from Meteor-Mamba to Meteor-MLM, we introduce a new concept of traversal of rationale, which spurs embedding of long sequential rationales.To ensure that Meteor encompasses diverse capabilities for vision-language tasks (e.g., image understanding, common-sense knowledge, charts, diagrams, documents, signs, symbols, and math problems), we gather 2.1M question-answer pairs from existing visual instruction tuning datasets: ShareGPT4V-Caption/Instruct [7] , MiniGemini-Instruct [10] , Doc-Downstream/Reason [20] , GLLaVA-Align/Instruct [40] , and Math-Vision/Instruct/Plus [41] [42] [43] . Subsequently, we utilize the light and fast Claude Haiku API [44] to generate detailed and comprehensive rationales tailored for the collected 2.1M question-answer pairs. These rationales are carefully filtered by human reviewers with the aid of GPT-4V, resulting in 1.1M question-rationale-answer triples (Appendix A).Using the question-rationale pairs in the curated 1.1M triples, the first training step involves training Meteor-Mamba and miscellaneous projectors, i.e., a vision projector and tor projector. During this step, Meteor-Mamba is trained to embed long sequential rationales. In the second training step, all components of Meteor are trained using the question-answer pairs in the curated 1.1M triples. Through these steps, we demonstrate that Meteor significantly improves vision-language performance, as shown in Figure 1 , compared with other open-and closed-source LLVMs, on numerous benchmarks requiring diverse capabilities. As illustrated in Figure 2 , these results advocate for the possibility of building efficient LLVMs with a multifaceted rationale, beyond the scope of scaling model size, additional vision encoders, and multiple computer vision models.Our contribution can be summarized into two main aspects:• We introduce a new efficient large language and vision model (LLVM), Mamba-based traversal of rationale ( Meteor), which comprehends long sequential rationales under a new concept of traversal of rationale and predict answers with the help of the rationale.• Despite its efficient model size, Meteor showcases significant advancements across various evaluation benchmarks requiring diverse capabilities for image understanding, common-sense, non-object concepts, and more. 
related works: 
 Rationale-Guided Prediction. Behind the answers of large language models (LLMs), rationale has played a significant role in enhancing answering capabilities across a myriad of natural language processing and vision language tasks in various forms such as (a) human annotation [45, 46] , (b) knowledge distillation [47] [48] [49] [50] [51] , and (c) chain-of-thought (CoT) [52] [53] [54] [55] . Rationale mimics the human thought process, providing explanations or justifications before answering questions. (a) Strout et al. [45] and Lu et al. [46] integrate human-annotated rationale with LLMs to fortify their performance, thereby enhancing model robustness against out-of-distribution scenarios with human annotations. (b) In knowledge distillation, rationale is used to effectively distill LLMs [47] [48] [49] [50] [51] into smaller language models. They first extract rationale from LLMs and then fine-tune smaller language models with the extracted rationale, demonstrating their efficacy through various evaluation benchmarks. (c) To directly apply rationale to LLMs, researchers have used a 'think step-by-step' prompt called Chain of Thought (CoT) [52] . When combined with few-shot learning, CoT elicits step-by-step rationale directly from LLMs for an input question followed by a series of questions, rationales, and answers used as few-shot examples. This has been further streamlined by automating rationale generation in CoT prompts, as explored by [53] and [54] , eliminating the need for human-annotated rationale for few-shot examples.Similar to (b), we also leverage the power of closed-source LLVMs but, in contrast, employ another model, Meteor-Mamba, to embed the rationale, instead of directly training Meteor-MLM to generate the created rationales. In other words, we separate the roles of the models where Meteor-MLM generates the answer, while Meteor-Mamba embeds the rationale, which encompasses diverse capabilities: fundamental image understanding, incorporation of real-world knowledge of commonsense, understanding of non-object concepts (e.g., charts, diagrams, symbols, signs, and math), and following systematic step-by-step procedures for solving complex questions. Once accompanied by LLVMs, their answer capabilities are expected to improve with the help of the rationale.Large Language and Vision Models. Following the emergence of visual instruction tuning datasets created by LLaVA [3, 8, 12] and InstructBLIP [9] , there has been rapid development of large language and vision models (LLVMs): Shikra [56] , IDEFICS [57] , Qwen-VL [6] , MiniGPT-4 [58] ,Otter [59] , mPLUG-Owl [60, 19] , ShareGPT4V [7] , LLaVA-XTuner [61] , Intern-XC [62] , MM1 [14] , MiniGemini [10] , InternVL Families [15, 63] , along with efforts to gather or curate high-quality visual instruction tuning datasets for various purposes: ShareGPT4V [7] , ALLaVA [64] , MiniGemini [10] , mPLUG-DocOwl [20] , GLLaVA [40] , MathVision [41] , MathInstruct [42] , and MathPlus [43] . Recently, Otter-HD [18] , Qwen-VL [6] , CogVLM [11] , and mPLUG Families [19, 20] have increased image resolution. Additionally, LLaVA-NeXT [12] , MM1 [14] , and MiniGemini [10] divide images into smaller sections, while LLaVA-UHD [21] and InternVL1.5 [63] employ dynamically split image partitions depending on their sizes. These research trends aim to improve image perception capabilities, thereby enhancing understanding of images and natural language instructions. Furthermore, BRAVE [22] , DeepSeek-VL [23] , OmniFusion [24] , MoVA [65] , and AM-RADIO [25] have utilized additional vision encoders such as EVA-CLIP [26] , DINOv2 [27] , SAM [28] , and SigLIP [29] . Apart from those, SpatialVLM [30] , ASMv2 [31] , LAR/LAF [32] , CoLLaVO [33] , and MoAI [34] employ multiple computer vision models for tasks such as segmentation, detection, scene graph generation, and optical character recognition (OCR).We view this series of efforts as procedures for enlarging LLVM's knowledge space regarding multifaceted information. From a fresh perspective, we believe that embedding it in the form of a multifaceted rationale serves as a key in developing efficient LLVMs, where they include in-depth explanations required for acquiring diverse capabilities. From this perspective, Meteor is expected to inherently embed the multifaceted rationale and improve answering capabilities with the help of the embedded rationale, even without significantly increasing model size or relying on additional explicit vision encoders and computer vision models.3 Meteor: Mamba-based traversal of rationale Model Architecture. As illustrated in Figure 3 , Meteor comprises a vision encoder, vision projector, Mamba architecture [39] , tor projector, and backbone multimodal language model (MLM) based on a pretrained large language model (LLM). For the vision encoder, we use CLIP-L/14 [66] , which is a text-aligned vision module that takes advantage of image understanding adeptness powered by text descriptions. For the vision projector and tor projector, we employ MLP modules containing two fully-connected layers with the GELU activation function [67] . Next, we use the Mamba-130M architecture for computational efficiency and adopt InternLM2-7B [68, 69] , learned with 2T tokens of multilingual text data in RLHF [70] , as the backbone large language model (LLM).To build a visual instruction tuning set, we cover not only fundamental image understanding but also a wide range of diverse capabilities: common-sense knowledge, non-object concepts (e.g., charts, diagrams, documents, signs, symbols, and math problems), cognitive reasoning, multi-discipline tasks, and integrated abilities.For the question-answer visual instruction tuning dataset, we choose 664K question-answer pairs in ShareGPT4V-Instruct [7] , including LLaVA-Instruct-665K [8] . Additionally, in ShareGPT4V-Caption [7] , we select 91K image descriptions for images from LAION [71] , CC [72] , SBU [73] , MS-COCO [74] , TextCaps [75] , and web images [76] [77] [78] that depict landmarks, animals, celebrities, art, text, and nature. The selected question-answer pairs primarily focus on fundamental image understanding and common-sense knowledge, with fewer data samples covering non-object concepts, cognitive reasoning, multi-discipline tasks, and integrated abilities. To strengthen these areas, we selectively gather 27K question-answer pairs of DocVQA [79] , ChartQA [80] , DVQA [81] , and AI2D [38] from MiniGemini-Instruct [10] . Moreover, we use 574K/27K question-answer pairs of DeepForm [82] , InfoVQA [83] , DocVQA [79] , KleisterCharity [84] , TabFact [85] , TextVQA [86] , WikiTable [87] , TextCaps [75] , and VisualMRC [88] from Doc-Downstream/Reason [20] . To achieve Where was the photo taken? broad coverage of math knowledge, we also include 177K GLLaVA-Align/Instruct [40] , 3K Math-Vision [41] , and 566K text-only samples from Math-Instruct/Plus [42, 43] . In summary, we collect 755K real-world images, 627K images for documents, charts, diagrams, signs, and symbols, and 747K math samples (180.5K with images and 566.8K text-only). Overall, the question-answer visual instruction tuning samples sum up to 2.1M.Curating Rationale. Using the gathered 2.1M question-answer pairs, we utilize the light and fast Claude Haiku API [44] to generate detailed and comprehensive rationales. We use the prompt template: "Question: {}. Answer: {}. Based on the question and answer, carefully provide an explanation about how to answer the question in detail." Here, {} represents the placeholder for the corresponding language description. Afterward, we assess the rationale score from GPT-4V [4] using the following template: "Question: {}. Rationale: {}. Answer: {}. Based on the question, rationale, and answer, provide a score from 0 to 10, evaluating how well the rationale is described to solve the question. If the given rationale is insufficient, you should rigorously give a score below 5." Subsequently, we filter out the generated rationales with a score below 5. The rationales that pass this automated evaluation are then subjected to human review to determine Yes or No on whether they provide a proper description to address the question. Finally, this series of processes yields 1.1M question-rationale-answer triples, which include 338K real-world images covering common-sense knowledge and a few samples for diverse capabilities, 379K images for documents, charts, diagrams, signs, and symbols, and 342K math samples (165K with images and 177K text-only).Mamba Architecture. To make LLVMs inherently possess rationale when addressing complex questions, we generate comprehensive rationales based on question-answer pairs. Subsequently, we employ the Mamba architecture [39] , leveraging its capability to handle lengthy rationales while maintaining computational efficiency. This approach allows us to effectively incorporate the rationale in an environment where the curated 1.1M question-rationale pairs have an average length of 213 tokens, which is approximately ten times longer than the average length of 22 tokens of ground truth answers in typical visual instruction tuning datasets [21, 7] .Traversal of Rationale. However, it is crucial to note that we cannot acquire and utilize rationales during the inference phase without API-based models, since only user questions are given. Therefore, we propose a new concept called traversal of rationale to effectively provide the rationale to Meteor-MLM without any help from external APIs in the inference phase. Inspired by retrieval-based knowledge [89] , we introduce a special token, <tor> (stands for traversal of rationale), and evenly distribute 10 fixed number of <tor> tokens, as described in Figure 3 . The rationale planted with <tor> is propagated into Meteor-Mamba along with image and question tokens, and then the output , AI2D [38] , ChartQA [80] , SEED I [97] , POPE [98] , HallB [99] , MME [35] , MathVista [37] , MMB [36] , MMB CN [36] , MM-Vet [100] , and LLaVA W [3] . Note that, AI2D and ChartQA performances for LLaVA family models are evaluated under zero-shot conditions, while Meteor uses training dataset for them.features in Meteor-Mamba are directly propagated into Meteor-MLM. Here, we autoregressively train Meteor to generate the part of the rationale between <tor>, whenever Meteor sees the special token <tor>. This procedure ensures that each <tor> represents the following rationale part until the next <tor> is encountered. Using a single <tor> token to encompass the rationale may not work well when embedding lengthy rationales, and if we do not consider distributing <tor> tokens in the rationale, then the later token does not refer to the earlier ones well due to the common problem of the autoregressive mechanism's forgetting nature [39] . This is why we place multiple <tor> tokens in the rationale instead of one.Training Strategy. In the first training step, we leverage the question-rationale pairs in the curated 1.1M triples to train Meteor-Mamba and miscellaneous projectors. Throughout this step, the long sequential rationale is embedded into Meteor-Mamba through traversal of rationale by autoregressively generating rationale parts between the special tokens <tor>. By freezing Meteor-MLM, Meteor-Mamba seamlessly incorporates the rationale. In the second training step, we utilize the question-answer pairs in the curated 1.1M triples to jointly train Meteor-Mamba, Meteor-MLM, and the miscellaneous projectors. Here, multiple <tor> special tokens are only propagated to Meteor-Mamba. Then, the rationale-embedded features in Meteor-Mamba corresponding to the special tokens <tor> are only fed into Meteor-MLM, enabling it to adeptly answer complex questions, even in the absence of explicit rationale descriptions. In essence, these steps equip Meteor with the capability to effectively address complex questions with the aid of the rationale. (a) To build Meteor-Mamba, we use the Mamba architecture with 24 layers and a 768 hidden dimension, resulting in a total of 130M parameters, which is relatively trivial compared to the approximately 7B parameters of the pretrained InternLM2-7B [68, 69] . It is executed under the (b) MathVerse [102] Benchmarks OmniFusion [24] DeepSeek-VL [23] MoVA [22] ASMv2 [31] LAF [32] CoLLaVO [33] (d) Evaluating sub-benchmarks in MM-Vet [100] with LLVMs utilizing computer vision models Table 2 : Detailed comparison of Meteor across more challenging evaluation benchmarks.efficient computation of hardware-aware state expansion [39] , where we borrow the tokenizer [103] from the backbone MLM to fit the language expression space in the backbone MLM. Meteor-MLM is based on InternLM2-7B [68, 69] with 32 layers and a 4096 hidden dimension. Evaluation. We have evaluated Meteor on numerous vision-language benchmarks, the details of which are described in Appendix B. These benchmarks require multifaceted information for diverse capabilities, including fundamental image understanding, real-world knowledge of common-sense knowledge, charts, diagrams, documents, signs, symbols, math problems, and more. Figure 1 2 (f) Ratio of Q-R in trainingTable 3 : Ablation studies to identify the effectiveness of Meteor-Mamba and rationale through traversal of rationale by controlling the six main factors.other existing models by a large margin, some of which are equipped with additional vision encoders or computer vision models, demonstrating that rationales can provide multifaceted information more effectively than enhanced visual perception.Ablation Studies. Furthermore, we have conducted several ablation studies to securely corroborate the effectiveness of our proposed method in light of six factors: (a) Meteor-Mamba, (b) Meteor-MLM, (c) the number of <tor> special tokens, (d) the distribution of <tor> special tokens, (e) rationale, and (f) 1.1M question-rationale-answer triples. Table 3 shows the following findings. Note that, Appendix D represents further ablation studies.(a) Once Meteor-Mamba is replaced with other transformer-based models: BERT [113] , GPT2 [114] ,and XLNet [115] , we have discovered that the Mamba architecture takes advantage of its efficiency for embedding multifaceted rationales in terms of both computational complexity and model size. As the results suggest, Mamba demonstrates the highest batches-per-second (BPS) value and zero-shot performances on MME and MMB benchmarks among architectures of similar sizes, enabled by its inherent computational efficiency based on linear complexity and strong long sequence modeling capability [39] , which other Transformer-based model architectures lack.(b) We have tried using various pretrained LLMs of comparable sizes for Meteor-MLM, in order to identify the effectiveness of embedding multifaceted rationale together with traversal of rationale.We have observed that InternLM2 [69] has shown the best performances.(c) Varying the number of <tor> special tokens from 2 to 15, we have optimized it based on the vision language performances. The results suggest that using 10 <tor> special tokens shows the best performances for embedding abundant multifaceted rationales, balancing between compression and information preservation.(d) The performances of Meteor depend on the distribution of <tor> special tokens when Meteor-Mamba is trained to embed multifaceted rationales. Given the observations, evenly distributing the tokens across lengthy rationales has shown the best performances. Prepending them to lengthy rationales may hinder effective embedding due to forgetting nature, and appending them to rationales in the end may not be guaranteed to understand the rationale. Randomly distributing the tokens across rationales may disrupt Meteor-Mamba's ability to stably learn the pattern of embedding rationales. Conversely, evenly distributed <tor> special tokens can segment lengthy rationales into shorter chunks and progressively embed them in a consistent manner, avoiding the issues of other distributions.(e) In order to prove the effectiveness of multifaceted rationales through Meteor-Mamba, we ablated the use of the Mamba architecture and the rationales. The first row represents baseline performances where backbone MLM is only trained. For the second row, we only train backbone MLM with the curated multifaceted rationales without Meteor-Mamba, and for the third row, Mamba has been trained to embed answer instead of the rationales. Compared to the last row where Meteor is evaluated, the second and third rows fall short of performances, clearly showing that using multifaceted rationales through the Mamba architecture has contributed to performance improvement.(f) As another way of showing the significance of multifaceted rationales, we have trained Meteor with different amounts of question-rationale pairs and evaluated Meteor trained with each of them.Rationale Image QuestionWhere was the photo taken? Where was the photo taken? As expected, the more question-rationale pairs used in first training step, the better performances achieves, demonstrating the significance of utilizing multifaceted rationales for diverse capabilities.Meteor-Mamba's Ability to Embed Rationales. We conduct a thorough analysis to confirm that Meteor-Mamba effectively embeds the rationales. To do this, we perform a retrieval task for multifaceted rationales, where we prepare ten different question-rationale pairs(Q i , R i ) where i = 0, 1, • • • , 9.These pairs are propagated through Meteor-Mamba with or without rationales under <tor> special tokens. This results in two sets of output features: one with rationale z w. i and one without rationales z w.o. j , with j = 0, 1, • • • , 9. We extract features corresponding to the placement of <tor> tokens, resulting in z w.i ∈ R 10×768 and z w.o. j ∈ R 10×768 , where the dimension 10 corresponds to the number of <tor> tokens. We then compute the cosine similarity between z w.i and z w.o. j to measure the similarity of their representations. As illustrated in Figure 4 , the diagonal values in the cosine similarity matrix are much higher than the off-diagonal values. This result indicates that Meteor-Mamba successfully embeds the rationale, and its output features contain multifaceted information even without explicit rationales in natural language. This explains how Meteor-Mamba operates effectively during the inference phase without explicit rationales.Discussion and Limitation. From the experimental results observed, we gain the insight that equipping LLVMs with a multifaceted rationale is a key factor in building efficient LLVMs that demonstrate impressive vision language performances across numerous evaluation benchmarks requiring diverse capabilities. This rationale, furthermore, naturally reduces hallucination effects in POPE [98] and HallusionBench [99] in Table 1 . Additionally, Table 2(c)-(d) shows that the need for additional vision encoders and computer vision models can be mitigated by incorporating a multifaceted rationale. However, Meteor might still be considered inefficient in terms of model size by users without high-end GPU resources, as it requires at least multiple GPUs with 48GB and 32GB VRAM for normal training and inference without (Q)LoRA [106, 105] and 4/8-bit quantization. Although many closed-source LLVMs have demonstrated superior performances following the scaling law [16] , our goal is to reduce the model size while maintaining vision language performances as much as possible. We strongly believe that small language and vision models, even those with about 1∼3B parameters, can effectively narrow the performance gap with the closed-source LLVMs by using layer-analyzing approaches such as mixture of depths [116] and others [117] [118] [119] [120] [121] [122] [123] , despite their inherent limitation in layer number and hidden dimension.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Discussion and Limitation section Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
theory assumptions and proofs: 
 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [Yes] Justification: You can see the link in Abstract section.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 To build efficient LLVMs, we incorporate a multifaceted rationale encompassing various aspects such as image understanding, incorporating external common-sense knowledge, understanding non-object concepts (e.g., charts, diagrams, symbols, signs, and math), and following systematic step-by-step procedures for how to solve complex questions. Meteor demonstrates significantly enhanced vision language performances across various evaluation benchmarks without the need to scale up LLVMs, use additional vision encoders, or employ multiple computer vision models. In designing Meteor, the traversal of rationale combined with Mamba architecture proves highly effective in embedding lengthy rationales. We believe this rationale, facilitated by the traversal of rationale, can pave the way for more efficient models, representing a promising step towards achieving more efficient LLVMs. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Experiment section Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Training Meteor with only one epoch under 4-bit quantized-extreme condition proves its effectiveness beyond checking error bar.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: We used publicly common visual instruction tuning dataset and pre-trained visual foundation models and large language models.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited models and datasets we deal with in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Question:If 20 men can build a wall 112 metres long in 6 days, what length of a similar wall can be built by 15 men in 3 days?The explanation provided in the answer is very clear and comprehensive. Let's go through it step-by-step: The key steps in solving this problem are:1. Understand the given information and find the work rate of the 20 men.2. Use the work rate to calculate the work done by 15 men in 3 days. 3 . The final answer is the length of the wall built by 15 men in 3 days, which is 42 meters.The explanation provided in the answer covers all these steps clearly and logically, making it easy to understand the thought process behind the solution.First, let's find out the work rate of the 20 men. First perform reasoning, then finally select the question from the choices in the following format: Answer: xxx. Question: W e have a regular hexagon ABCDEF and inside it, a square BCGH is constructed. What is the size of angle HAB when line segment AH is connected? Choices: A: 75°B : 60°C : 55°D : 45°RT o solve this problem, we need to understand the properties of regular hexagons and squares.-ABCDEF is a regular hexagon.-BCGH is a square constructed inside the hexagon.Step 1: Find the interior angle of a regular hexagon.The interior angle of a regular polygon with n sides is given by the formula: Interior angle = (n-2) × 180°/ n For a regular hexagon, n = 6, so the interior angle = (6-2) × 180°/ 6 = 120°.Step 2: Find the interior angle of a square.The interior angle of a square is 90°, as it is a regular polygon with 4 sides.Step 3: Find the angle HAB.Since BCGH is a square, angle HBC is 90°. The angle ABC in the regular hexagon is 120°. Therefore, the angle ABH is 120°-90°= 30°.Step 4: Find the angle HAB.Since BA = BH, the angle HAB is equal to (180°-30°) / 2 = 75°.Therefore, the answer is A: 75°.The interior angle of a regular hexagon is 120°, and the interior angle of a square is 90°. Therefore, angle ABC = 120°and angle HBC = 90°. Therefore, angle ABH = 30°. Additionally, since BA = BH, we have angle HAB = (180°-30°) / 2 = 75°. Therefore, the answer is A. Answer:ATell me what you observe in the image.The T o answer this question, I should focus on analyzing the geometric properties and relationships within the diagram, rather than identifying any individuals. The key aspects to describe are the triangles, midpoints, and how the smaller triangle AEF is formed within the larger triangle ABC. My response should aim to provide a detailed explanation of the geometric elements observed, without making any assumptions or inferences about potential people or other identities depicted.Within the diagram, there is a triangle labeled ABC. Point D is situated at the midpoint of side BC, while points E and F are the midpoints of sides BD and CE. Connecting points A, E, and F forms triangle AEF, which has a specific area.How many ducks weigh the same as a crocodile? This puzzle is based on weight. The crocodile weighs the same as two lions. Each duck weighs the same as a lion cub. So the crocodile weighs the same as four ducklings. The answer is B.B Question:Explain the visual content of the image in great detail.The image contains three zebras in a zoo enclosure. The zebras are all black and white stripes. The zebra on the left is standing alone, while the other two are standing close together. The zebra closest to the right is drinking from a pond. There are trees and a fence in the background.T o answer the question in detail, you could describe the zebras' physical appearance, their behavior, and their interaction with their environment. You could also discuss the symbolism of the zebras and the overall mood of the image.In the tranquil setting of a zoo enclosure, three zebras, their coats a striking contrast of black and white stripes, command the viewer's attention. The zebra on the left stands alone, its gaze directed towards something unseen in the distance. Its solitude is a stark contrast to the companionship shared by the other two zebras on the right.These two are standing close together, their bodies almost touching, suggesting a bond between them. The zebra closest to the right is captured in the act of drinking from a small pond. The pond, encircled by rocks, mirrors the zebra's image, creating a beautiful reflection that adds depth to the scene.The backdrop of this tableau is a lush array of trees and a fence that marks the boundary of their enclosure. The trees provide a naturalistic element to the enclosure, enhancing the overall aesthetic of the image. The fence, while unobtrusive, is a subtle reminder of the man-made nature of this environment.This image captures not just the physical attributes of these magnificent creatures but also hints at their social dynamics and their interaction with their environment.What do you see happening in this image?The key points in how I answered the question are:1. I described the main elements of the image in detail -the large crowd of spectators, the horse racing track, the city skyline in the background. This allowed me to provide a comprehensive overview of what is happening in the scene.2. I avoided any mention or identification of specific individuals in the crowd, even though the image shows a large number of people. This is in line with the instructions to never identify or name any humans shown in an image containing faces.3. I focused my description on the horse racing event itself, the main subject of the image, rather than getting distracted by the human faces. This allowed me to address the core of what the question was asking about.4. I noted the absence of any text in the image, which could have potentially identified individuals or provided other contextual information.5. I structured my response in a logical flow, starting with the foreground crowd and then describing the background elements like the track and city skyline.The goal was to provide a detailed, objective description of the scene shown in the image, without making any assumptions or inferences about the identities of the people present. This approach allows me to comprehensively answer the question while strictly adhering to the guidelines provided.The image captures a vibrant scene at a city-based horse racing track. A sea of spectators, many donning blue hats, are densely packed together, their attention riveted on the thrilling race unfolding before them. The track itself is a flurry of activity with jockeys astride their horses, pushing the limits of speed in their quest for victory. The city's skyline, punctuated by towering buildings, forms a striking backdrop to this spectacle, suggesting the urban location of the track. Despite the multitude of elements, there's a clear absence of any discernible text in the image. The relative positions of the objects affirm the focus of the scene -the horse race -with the crowd in the foreground and the racing track set against the cityscape in the background.Identify some key points in this picture.The image provided is a graph showing the population of the San Diego-Chula Vista-Carlsbad metropolitan area from 2010 to 2020. It does not contain any human faces, so I will describe the relevant information in the graph.The key points I can gather from the graph are:1. The population has steadily increased over the 10-year period from 2010 to 2020. 2. In 2010, the population was around 3,095,313 people. In 2020, the population of the San Diego-Chula Vista-Carlsbad metropolitan area was approximately 3,305,462. In 2020, the population of the San Diego-Chula Vista-Carlsbad metropolitan area was 3,305,462.what were the total number of times the venue was located in the united states? Answer the question with detailed explanation.The total number of times the venue was located in the United States is 6, based on the following details from the provided information: By carefully analyzing the "Venue" column in the table, we can identify all the instances where the venue was located in the United States. The explanation provided covers each of these occurrences in detail, leading to the final answer of 6 times the venue was located in the United States.The key steps in answering this question are: 1. Thoroughly examine the "Venue" column to identify all the locations in the United States.2. Count the number of times the venue was in the United States.3. Provide a detailed explanation citing the specific events and years where the venue was in the United States.This approach ensures that the answer is well-supported and the reasoning behind it is clear.Answer: 6• Q-Bench [95] aims to assess the low-level visual capabilities of Multi-modality Large Language Models (MLLMs). This dataset is divided into three main sections: perception, description, and assessment. The low-level visual perception component evaluates MLLMs' ability to recognize and understand low-level image attributes. The low-level visual description component tests how accurately and thoroughly MLLMs can describe these attributes. Finally, the overall visual quality assessment examines how closely MLLMs' evaluations align with human judgments of image quality. Altogether, the Q-Bench dataset encompasses 81,284 samples across these tasks. • SQA-IMG (SQA I ) [96] is a portion of the ScienceQA (SQA) dataset, which serves as a comprehensive multimodal benchmark aimed at enhancing the reasoning capabilities and interpretability of AI systems in the realm of science question answering. The dataset spans a variety of science subjects, encompassing 26 distinct topics from natural science, social science, and language science, with annotated answers including lectures and explanations. This specific subset consists of samples with image context, totaling 10,332 question-answer pairs. • AI2D [38] , also known as AI2 Diagrams, was developed to tackle the challenges of diagram interpretation and reasoning, emphasizing syntactic parsing and semantic analysis of diagrams. Its goal is to support research in uncovering the structure of diagrams and understanding the meanings of their elements and their interrelations. This dataset is especially beneficial for tasks like diagram question answering, where comprehensive understanding and reasoning about the content are essential. The collection consists of over 5,000 diagrams from grade school science subjects and more than 15,000 multiple-choice questions. • ChartQA [80] is created to assess and enhance question answering systems that involve complex, multi-step logical and visual reasoning with charts. This dataset meets the need for systems capable of interpreting various data visualizations, including bar charts, line charts, and pie charts, and addressing questions that require arithmetic and logical processing. It encompasses question-answer pairs that are both human-authored and machine-generated, emphasizing visual and logical reasoning. The dataset comprises a total of 32,719 samples. • SEED-IMG (SEED I ) [97] is a component of SEED-Bench designed to assess the generative comprehension skills of multimodal large language models (MLLMs). This thorough and unbiased benchmark enables researchers to evaluate and compare various models' abilities in both spatial and temporal understanding. The dataset is organized into several subsets according to 12 evaluation dimensions that encompass spatial and temporal comprehension across image and video modalities. SEED-IMG specifically focuses on the image modality subset. • POPE [98] is a technique created to systematically assess the propensity of LLVMs to hallucinate objects that do not exist in the target images. This approach transforms the hallucination evaluation into a binary classification task via polling questions, providing a consistent, equitable, and adaptable evaluation process. • HallusionBench (HallB) [99] is crafted to assess and analyze both visual illusions and knowledge hallucinations in large language and vision models (LLVMs). This dataset targets the identification of potential failure modes in these models by utilizing meticulously created example pairs for thorough testing. The benchmark includes a variety of visual-question pairs, encompassing both visual dependent subsets (such as illusion, math, etc.) and visual supplement subsets (such as chart, table, map, OCR). HallusionBench comprises 346 distinct images and an extensive collection of 1129 questions distributed across diverse topics and formats. • MME [35] is created to serve as a thorough evaluation benchmark for Multimodal Large Language Models (MLLMs). The goal is to assess their abilities in perception and cognition through 14 distinct sub-tasks, including coarse-grained recognition, fine-grained recognition, OCR, and commonsense reasoning, among others. It strives to address the shortcomings of current evaluation methods, ensuring comprehensive testing of MLLMs across various dimensions while preventing data leakage. • MathVista [37] serves as an extensive benchmark aimed at assessing mathematical reasoning within visual contexts. This dataset merges visual comprehension, allowing for a thorough evaluation of AI models' capabilities in tackling mathematical problems that involve visual elements. It comprises three subsets: IQTest, FunctionQA, and PaperQA, with an aggregate of 6,141 examples. • MMB, MMB-Chinese (MMB CN ) [36] aims to deliver a thorough and resilient evaluation standard for vision language models by encompassing a broad spectrum of capabilities (20 distinct fine-grained abilities) necessary for multimodal comprehension in both English and Chinese. This benchmark facilitates the evaluation of various facets of LLVMs, including their perceptual and reasoning skills across multiple tasks. The benchmark comprises a total of 3,217 carefully curated questions, sourced from a variety of places, including public datasets and the internet, to ensure a wide range of skills. • MM-Vet [100] is designed to systematically assess LMMs by evaluating their proficiency in handling intricate tasks that necessitate the combination of multiple VL abilities. Unlike existing benchmarks that generally focus on simpler tasks involving only one or two abilities, MM-Vet encompasses six fundamental VL capabilities: recognition (Rec), knowledge (Know), OCR, spatial awareness (Spat), language generation (Gen), and math. MM-Vet includes tasks that integrate these six core capabilities in various combinations, leading to 16 distinct capability integrations. The dataset consists of 200 images sourced from various online platforms and includes 218 questions that require one or more of these capabilities to answer. • LLaVA Bench in the Wild (LLaVA W ) [3] is a dataset designed to assess the performance of large multimodal models (LMM) in tackling more complex tasks and adapting to new domains. It features a varied collection of 24 images, accompanied by 60 questions, covering indoor and outdoor environments, memes, artworks, sketches, and more. Each image is paired with a detailed, manually-curated description and thoughtfully chosen questions. • MMStar [101] strives to deliver an accurate assessment of LLVMs' authentic multi-modal abilities. It achieves this by guaranteeing that each sample genuinely depends on visual content to arrive at correct answers, while also minimizing inadvertent data leakage. The benchmark is organized into six primary competencies, such as instance reasoning, logical reasoning, and mathematics, and includes a total of 1,500 meticulously curated samples. • MathVerse [102] is designed to evaluate the visual mathematical reasoning capabilities of Multi-modal Large Language Models (MLLMs). The dataset focues on whether and how well MLLMs can interpret visual diagrams and mathematical problems, rather than relying solely on textual information. The problems in MathVerse are divided into three main subjects (plane geometry, solid geometry, and functions) and further categorized into twelve subfields (length, area, volume, coordinate, etc.) to provide a well-rounded evaluation of different mathematical skills, adding up to 2,612 high-quality visual math problems.User: What type of fruit is this?Meteor: The image shows several mangosteen fruits on a wooden surface.Mangosteen is a tropical fruit known for its thick, dark purple rind and white, segmented flesh inside. The outer skin is tough and fibrous, but the inner segments are juicy, sweet, and aromatic. Mangosteens are native to Southeast Asia and are popular in countries like Thailand, Malaysia, Indonesia, and the Philippines.User: Imagine the fragrance of the fruits in the image. How would you describe this to someone who has never had this fruit before?Meteor: The fruits in the image appear to be mangosteen, a tropical fruit known for its unique and complex flavor profile. When ripe, mangosteen has a sweet, floral aroma with hints of citrus and tropical fruitiness. The outer rind of the fruit is thick, dark purple or red, and has a bumpy texture.When cut open, the inside reveals a white, segmented flesh that looks somewhat like an orange or grapefruit. The individual segments have a slightly translucent, jelly-like appearanceThe taste of mangosteen is often described as a blend of sweet, sour, and floral notes. Some people find it reminiscent of lychee, passion fruit, or even bubblegum. It has a rich, velvety mouthfeel and is considered by many to be one of the most delicious tropical fruits.If you've never had mangosteen before, it's a truly unique and unforgettable experience. The combination of flavors and textures is unlike any other fruit you're likely to try. The aroma alone is enticing, hinting at the sweet, tangy, and exotic taste that awaits inside.User: Show the detailed recipe for this dish.Meteor: This dish appears to be lobster tails that have been grilled or roasted and served with lemon wedges.Ingredients:-Lobster tails -Butter -Garlic, minced -Fresh parsley, chopped -Salt and pepper -Lemon wedges Instructions 1. Preheat your grill or oven to 400°F (200°C) 2. Using kitchen shears or a sharp knife, cut the top shell of the lobster tail down the center, being careful not to cut all the way through the meat.Gently spread the shells apart to expose the meat. 3. In a small bowl, mix together softened butter, minced garlic, chopped parsley, salt, and pepper. 4. Place the lobster tails on a baking sheet or grill rack, meat side up. Brush the butter mixture generously over the exposed meat. 5. Grill or roast the lobster tails for about 10-12 minutes, or until the meat is opaque and cooked through. 6. Remove from the heat and let the lobster tails rest for a few minutes before serving. 7. Serve the lobster tails with lemon wedges on the side for squeezing over the meat.Enjoy your delicious grilled or roasted lobster tails! User: What's the name of the restaurant serving these dishes?Meteor: The restaurant serving these dishes appears to be Ichiran Ramen, a popular Japanese ramen chain known for its tonkotsu (pork bone broth) ramen.The distinctive black bowls with ornate gold and red designs are a signature feature of Ichiran's branding.User: What's the best weather, season, time of the day of visiting this place? Is the time when this photo was taken a good time to visit this place?Meteor: Based on the clear blue skies and sunny conditions in the image, it appears to be a beautiful day to visit the Space Needle in Seattle. The Space Needle is an iconic observation tower that offers stunning 360-degree views of the city and surrounding landscape.The best weather for visiting the Space Needle would be on a clear, sunny day like the one shown in the photo. Ideally, you'd want to avoid foggy or overcast conditions that could limit visibility from the observation decks.As for the season, spring and summer are generally the most popular times to visit Seattle due to the milder temperatures and longer daylight hours. However, the Space Needle is an attraction that can be enjoyed year-round, as long as the weather cooperates.The time of day shown in the photo appears to be midday or early afternoon, based on the bright sunlight and shadows. This is a great time to visit the Space Needle, as the views are typically clearest and most vibrant during the middle of the day when the sun is high in the sky.So in summary, a clear, sunny day like the one in the photo is ideal for visiting the Space Needle. Midday to early afternoon is also a good time to go for the best views. Spring and summer are the most popular seasons, but the Space Needle can be enjoyed year-round if the weather cooperates.User: Can you give a short introduction to this painting?Meteor: This painting is a portrait of a young woman wearing a blue headscarf, often referred to as "Girl with a Pearl Earring" or "The Jewish Bride". The painting is a masterpiece by the Dutch artist Johannes Vermeer, painted in the mid-17th century. The woman's enigmatic expression and the dramatic use of light and shadow are hallmarks of Vermeer's style. The painting has become one of the most famous and recognizable works of art in the world.User: What are all the French scene text in the image?Meteor: The French scene text in the image is "Histoire de France" which translates to "History of France" in English. The subtitle "De la Préhistoire à nos jours " translates to "From prehistory to our days". Justification: Abstract and Introduction section Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper received Borderline Accept, Weak Accept, Accept and Borderline Accept. 

The main contribution of this paper is a new pretrained vision-and-language model that demonstrates good results across a range of benchmarks and outperforms a recent LLaVA-Next competitor. Reviewers were mostly positive and improved scores after rebuttal discussions. One concern that was voiced in a couple of reviews was the clarity on how the method compares in terms of inference overhead and other aspects besides accuracy on downstream benchmarks. Authors are encouraged to incorporate these additional comparisons in their final version.

Reviews: review: paper_summary: paper_summary: This paper introduces a novel approach to enhance the performance of large language and vision models (LLVMs). The proposed model, Meteor, leverages multifaceted rationale through the Mamba architecture to efficiently embed lengthy (latent) rationales and improve understanding and answering capabilities across diverse vision language benchmarks. The key innovation lies in the traversal of rationale concept and the efficient processing capabilities of the Mamba architecture, which allows for handling sequential data with linear time complexity.
strength_weakness: strength_weakness: i) Figure 3 (Stage-One Pre-Training):

More clarity and intuition behind the model design are needed. Given that the training is formulated as an autoregressive token generation task:
a) Are TOR embeddings predicted as special tokens [TOR]?
b) Or are losses on these tokens bypassed?
If the latter, a potential drawback is the inability to assess the model's performance after the first stage of training, as it cannot produce complete sentences/rationales.


ii) Benchmarks and Reasoning Capabilities:

The proposed benchmarks may not sufficiently test complex rationale reasoning (e.g., current tasks are mostly recognition or OCR-heavy).
Recommendation: Conduct further studies in the final version on reasoning-oriented benchmarks such as Visual Commonsense Reasoning (VCR) [r1] to substantiate claims of:
a) Enhanced reasoning capability,
b) More grounded results (less hallucination) compared to existing literature.
Note: While VCR includes rationale annotations, evaluating only the answer component would be sufficient, given the proposed method's current limitations in directly sampling rationale output.


iii) Ablation Studies (Table 3a):

Clarification needed: Did the baselines undergo the same first-stage pre-training as Mamba? If so, what were the training objectives for each method?


iv) Comparison Methodology (Figure 2):

Line 147 indicates that the training data includes annotations from ChartQA and AI2D. Some results (e.g., Gemini-Pro and GPT-4V) are zero-shot and not directly comparable.
Recommendation: Revise the plots to accurately reflect these differences in training/evaluation conditions.


Reference:
[1] Zellers et al., From Recognition to Cognition: Visual Commonsense Reasoning, CVPR 2019.
questions: questions: see previous sections
limitations: limitations: appear to be sufficient
paper_summary: paper_summary: The paper "Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models" presents a new efficient large language and vision model (LLVM), Meteor, which leverages multifaceted rationale to enhance understanding and answering capabilities. The paper  introduces a new concept of traversal of rationale, demonstrating the effectiveness of Meteor in embedding lengthy rationales containing abundant information and improving vision language performances across various evaluation benchmarks requiring diverse capabilities.
strength_weakness: strength_weakness: Writing:
1. All Figures (fonts, line widths, etc.) in the paper are not uniform, there is too much white space. And even Figure 1 shows strange tables and font proportions (fonts are also not clear).

Motivation:
2. The paper employ the Mamba architecture to embed lengthy rationales containing abundant information, but without setting and demonstrating task scenarios that existing LVLMs does not achieve that require extreme long sequence modeling capabilities.
3. It appears to share the same motivation as some existing methods, for instance, Vary-toy[1]. Please provide a more detailed explanation of how the proposed "traversal of rationale" differs from the design implemented in Vary-toy.

Method:
4. Show more compassion with the MMLM models based on Mamba, like Cobra[2] or VL-Mamba[3].

[1] Small Language Model Meets with Reinforced Vision Vocabulary
[2] Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference
[3] VL-Mamba: Exploring State Space Models for Multimodal Learning
questions: questions: Please refer to the weakness
limitations: limitations: None
paper_summary: paper_summary: The paper proposes Meteor, which divides MLLM into two stages, i.e., rationale generation and question answering.  Experimental results show its effectiveness agasint existing MLLMs on many benchmarks.
strength_weakness: strength_weakness: 1. I must admit the novel design of Metor, which however, also leads some pratical issues:
    How about the inference time against common MLLMs?
    Does this structure can be scaled up?
    Will Metor be better when combining high-resolution image encoders, e.g., LLaVA-HR[A].
    [A] Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models
2. To be honest, the writing of the paper should be improved. I take a lot of time to understand the technical details. The method should be better oragnized.
questions: questions: My main cocerns are the efficiency and generalization ability.
limitations: limitations: See weakness
paper_summary: paper_summary: This paper presents a Mamba-based traversal of rationales (Meteor) that leverages detailed image captions (multifaceted information) to provide more comprehensive image-related information to LLVMs, thereby enhancing the model's understanding and answering capabilities. The introduction of the Mamba architecture enables linear time complexity data processing, reducing the time required to generate multifaceted information from images.
strength_weakness: strength_weakness: 1. The introduction of Meteor-Mamba increases the model size, which contradicts the abstract. Additionally, Meteor's inference cost is higher than models like LLAVA and InternLM-XComposer2.
2. In Figure 3, what is the Meteor-Multimodal Language Model in the Second Training Step? Was it initialized with pre-trained MLLMs weights?
3. How does the Mamba architecture benefit Meteor? The experiments do not demonstrate the advantages of the Mamba structure.
4. I am puzzled by the performance achieved with only 2.1M instruction tuning. Please provide a detailed description of the training process, the data used, and the weights of each model employed.
questions: questions: Same as Weaknesses
limitations: limitations: This paper does not reflect any potential negative societal impact.

Comments: title: Response
Official Comments by Reviewer iRmr
Questions have been addressed in the rebuttal
Wait for Response
comment: We agree with the pre-encoded system is a common structure, but we disagree with that our work seems limited contribution. We newly presented the pre-embedded concept of "multifaceted rationale". The key innovation lies not in the structure itself—whose significance is widely recognized—but in the nature of what is embedded. We think embedding new reasonable thing is really novel where we introduce new technique of "traversal of rationale".
Thanks the author for addressing my concerns. I would like to improve my score to borderline accept.
The rebuttal has addressed part of my questions. About Q3, many existing methods apply the pre-encoding method to achieve united representation.  Vary-toy is just one example of this idea. So i still think the contribution is a little bit limited. Considering the impressive results, i will still maintain my score.
The rebuttal has addressed the questions from the initial review and the authors indicated corresponding revisions in the final version. Therefore, I maintain my original positive review.
The three reviewers: XZUR, iRmr, and Hj2J scored our paper to positively "5", "5", "7", while the one reviewer zsWf scored negatively "4". We hope our rebuttals make the reviewers who do not respond yet possibly improve their scores. If the discussion should be needed more, then we will be sincerely and rapidly participating in the future discussion. Kindly note that, we would like to respectfully thank the reviewer Hj2J for rapid response.
Thanks for authors'rebuttal. The provided results further address my concerns. The newly added comparisons should be included in the final version.  Therefore, I would like to improve my score to accpet.