MMSearch: Unveiling the Potential of Large Models as Multi-modal Search Engines
Authors: Dongzhi Jiang
Renrui Zhang
Ziyu Guo
Yanmin Wu
jiayi lei
Pengshuo Qiu
Pan Lu
Zehui Chen
Guanglu Song
Peng Gao
Yu Liu
Chunyuan Li
Hongsheng Li
Venue: ICLR 2025 Poster (2025)
Keywords: Large Multimodal Model
AI Search Engine
Benchmark
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The advent of Large Language Models (LLMs) has paved the way for AI search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet interaction. However, most current AI search engines are limited to text-only settings, neglecting the multimodal user queries and the text-image interleaved nature of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they can function as AI search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design a delicate pipeline, MMSearch-Engine, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no overlap with the current LMMs' training data, ensuring the correct answer can only be obtained within searching. By using MMSearch-Engine, the LMMs are evaluated by performing three individual tasks (requery, rerank, and summarization), and one challenging end-to-end task with a complete searching process. We conduct extensive experiments on closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSearch-Engine achieves the best results, which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness of our proposed pipeline. We further present error analysis to unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time computation for AI search engine. We hope MMSearch may provide unique insights to guide the future development of multimodal AI search engine.

Published as a conference paper at ICLR 2025
MMSEARCH:
UNVEILING THE POTENTIAL OF LARGE MOD- ELS AS MULTI-MODAL SEARCH ENGINES
Dongzhi Jiang", Renrui Zhang".3.1, Ziyu Guo", Yanmin Wa', Jiayi Lei* Pengshuo Qiu', Pan Lu Zehui Chen², Guanglu Song'
Peng Gao", Yu Liu?, Chunyuan Li", Hongsheng Li
CCUK MMLab & MiiuLar Lab *ByteDance *Shanghai AI Laboratory Peking University Stanford University
"Sensetime Research 8 CPII under InnoHK
[dzjiang, renruizhang, zi yuguo el @link. cuhk.
Equal contribution
Project lead
Corresponding author
ABSTRACT
The advent of Large Language Models (LLMs) has paved the way for Al search engines, e.g., SearchGPT, showcasing a new paradigm in human-internet inter- action. However, most current AI search engines are limited to text-only set- tings, neglecting the multimodal user queries and the text-image interleaved na- ture of website information. Recently, Large Multimodal Models (LMMs) have made impressive strides. Yet, whether they function Al search engines remains under-explored, leaving the potential of LMMs in multimodal search an open question. To this end, we first design delicate pipeline, MMSEAN RCCH- ENGINE, to empower any LMMs with multimodal search capabilities. On top of this, we introduce MMSEARCH, comprehensive evaluation benchmark as- sess the multimodal search performance of LMMs. The curated dataset contains 300 manually collected instances spanning 14 subfields, which involves no over- lap with the current L.MS' training data, ensuring the correct answer can only obtained within searching. By using MMSEARCCH-ENGINE. the LMMs are evaluated by performing three individual tasks (requery, rerank, and summariza- tion), and one challenging end-to and-to-end task with a complete searching process. We conduct extensive experiments closed-source and open-source LMMs. Among all tested models, GPT-4o with MMSSARCH-ENNGH which surpasses the commercial product, Perplexity Pro, in the end-to-end task, demonstrating the effectiveness our proposed pipeline. We further present error analysis unveil current LMMs still struggle to fully grasp the multimodal search tasks, and conduct ablation study to indicate the potential of scaling test-time com- putation for search engine. We hope MMSEARGS may provide unique insights to guide the future development of multimodal AI search engine. Project Page is at ht https:/// /wmssaachhccccccccccco/ g thub.
achieves the best results,
INTRODUCTION
Search engines (Brin & Page, 1998) have been the main tools for humans to navigate through the overwhelming quantity of online resources. Recently, Large Language Models (LLMs) (OpenAI, 2023a;b; Touvron et al., 2023a) have demonstrated impressive performance on various zero-shot downstream upplications. On top of this, AI search engine (OpenAI, 2024c), which integrates LLMs with traditional search engines, stands among one of the most promising ones. It points the direc- tion of the next-generation interaction paradigm of human and Internet. Combining the language

---

Published as a conference paper at ICLR 2025
News Finance
News - Sports
News Science
O
News Fntertainment
WukONG
Question: the closing price (at shown image August higher than the closing price on the day shown the image?
4:00 PM) of the
Question: How times did this player play 2024 spring scason?
Question: What formar of data this convert 3D point clouds to for sogmentation?
How the number of players the finst day this game on Steam hiscory?
Question:
Answer: No
Answer:
News General
News False Premise
How many cabins on caught music festival in Germany in August 2024?
Question:
Question: When did Roger Federer his first Olympic gold medial in singles tennis?
Answer:
Answer: Invalid question
C Knowledge Astronomy
Knowledge Auto
MMSearch
Question: What is the radius of the nebula shown in image?
Question: What is the drag cnaffic. ient this car?
@
light-years
Answer: 0,198CU
Knowledge
Fashion
Knowledge Art
H Knowledge Architecture
Knowledge Anime
Question: Which library houses the book collection includes this painting?
Question: What kind of people can go this huilding withour buying ticket?
When did this clothes relouse?
Question:
Question: old
this stand?
Answer: 2024-08-31
enjty Library
Answer:
Heidelberg Univ-
Under)
Answer:
Children (18 and
T
Answer: 19
Figure 1: Overview of the MMSEARCH Benchmark. MMSEARCH aims evaluate any LMM's potential to be a multimodal Al search engine. The benchmark contains two primary areas: latest news and rare knowledge ensure no overlap with LMM's inherent knowledge.
understanding ability of LLMs and up-to-date information from the Internet. Al search engines could better grasp the user's intention and summarize contextual-alignet answers from the raw web information. These systems only process textual queries and interpret textual web content, sig- nificantly constraining user query scenarios and information-seeking methods (Barbany et al., 2024; Xie al., 2024). This limitation impacts both the range of input queries and the accuracy of sults (Jiang et al., 2024a; Chen et al., 2021; La et al., 2024), particularly given the complexity and interleaved nature of modern websites (Liu al., 2024c). For example, consider scenario where you possess numerous medals belonging to your grandfather but are unaware of their specific names. multimodal AI search engine could match photographs of these medals with an interleaved table of images and text retrieved from the Internet, thereby identifying each medal. In contrast, text- only search engines can neither take photographs for searching nor understand the interleaved table. Hence, a multimodal AI search engine is crucial for advancing information retrieval and analysis.
the other hand, with the recent rapid advancements, Large Multimodal Models (LMMs) (Liu et al., 2023a; Lin et al., 2023; OpenAl, 2023c; Gao et al., 2024; Zhang al., 2024b) have showcased significant abilities across diverse scenarios, including general image understanding (Fu et al., 2023; Liu et al., 2023b; Yu al., 2023), expert image reasoning (Zhang al., 2024d; Gao et al., 2023a; Zhang et al., 2024c; Guo et al., 2025b), multi-iman multi-image perception (Li et al., 2024a; Wang et al., 2024; Jiang al., 2024b; Li et al., 2024c), and spatial environment perception (Guo et al., 2023; Yang
2

---

Published as a conference paper at ICLR 2025
et al., 2023; Han et al., 2023). Despite these developments, framework for LMMs to function as multimodal AI search engines remains largely unexplored. Consequently, the potential of LMMs in multimodal searching also remains a significant open question.
a multimodal Al search engine pipeline, empowering any LMMs with advanced search capabilities. MMSEARCH- utilization of LMMs' multimodal information comprehension abilities, incorporating both visual and textual website content as information sources. On top of this, we introduce MMSEARCII, a multimodal Al search engine benchmark to comprehensively evaluate LMMs' searching perfor- facilitates the zero-shot evaluation of any LMMs
To bridge this gap, we first present MMSEARCH-
maximizes the
mance. The design of MMMSEARCH-ENGIAN within the context of AI search engine. Our experiment covers state-of-the-ar closed-source (Ope- nAI, 2023c; Anthropic, 2024; Gemini Team, 2023) and open-source L.MMs (Li et al., 2024b; Qwen Team, 2024; Chen et al., 2024d; Ye et al., 2024). Our efforts are summarized as follows:
MMMSEARCH-ENGIN a multimodal Al search engine pipeline for LMMs, empower- ing large models for multimodal searching. In contrast with the conventional text-only Al search engines, MMSEARCHI-ENGINN fully integrates multimodal information in two ways: (i) for queries containing images, we conduct web searches across both textual and visual modalities. We utilize Google Lens (len) to identify critical visual information from the input image; (ii) all search results are presented in both textual and visual formats, ensuring a comprehensive understanding of the interleaved website content. The working flow of AMSEARCH-ENTE ternet. The LMM needs to first requery the user question into a search-engine-friendde format. Then, the LMM reranks the retrieved websites based on its helpfulness. Finally, the LMM is required to summarize the answer based on the most informative webpage con- tent selected from the rerank. Thanks to the design of the pipeline, we propose step-wiss wise evaluation strategy on the three core tasks within the searching process: requery, rerank, and summarization. The final score is weighted by the end-to-end evaluation results and scores of the three core tasks.
contains multi-round interaction between LMM and the In-
ii MMSEA RCH,, a comprehensive benchmark for multimodal Al search engines, which, to our best knowledge, serves as the first evaluation dataset to measure LMMs' multimodal searching capabilities. Our benchmark categorizes searching queries into two primary ar- eas: News and Knowledge, as shown in Fig. 1. We employed different strategies for these two areas to ensure the challenging nature of the benchmark. News area covers the latest news at the time of data collection (August, 2024). This is to guarantee the answers to the queries will not be present in the training data of LMMs. As for the area of Knowl- edge, we collect queries requiring rare knowledge and then select the queries unable to be answered by current SoTA LMMs such as GPT-4o (OpenAI, 2024b) or Claude-3.5 (An- thropic, 2024). The two areas sum up to 14 subfields. In total, MMSEARCH encompasses 300 meticulously collected queries, with 2901 unique images.
ili Extensive experiments and error analysis for future development direction. We evaluate popular closed-source models and open-source LMMs on MMSEARCH. GPT-4o achieves the best overall performance across different tasks. Surprisingly, our MMSEARCH- ENGINE equipeed with SoTA LMMs, such as GPT-4o and Claude 3.5 Sonnet, even sur- passes the prominent commercial AI search engine Perplexity Pro (Perplexity) in the end- to-end task. Our thorough eтror analysis reveals that current LMMs still struggle gen- eralize to multimodal search-specific tasks. Their poor requery and rerank capabilities significantly limit their ability to correctly identify useful websites and extract relevant answers. Additionally, we identify five error types for requery and summarization tasks, respectively. We find that current LMMs cannot fully understand the requery task and do not know how to query the search engine. As for the summarization task, LMMs often have difficulty in extracting useful information, either from text or images. These capabil- ities essential for LMMs to function robust multimodal search engines and require further development. We also conduct a preliminary ablation study to explore the potential of scaling test-time computation versus scaling model size (OpenAI, 2024a). Initial results indicate that scaling test-time computation demonstrates superior performance in this task.
3

---

Published as a conference paper at ICLR 2025
Round Requery
Infiraitiio Query: Which move coused this team change their lvery the provated imagr for the 2014 lidginn Gradd Pric?
Image:
Image Search Sereenhoot:
Morie Alpine their black,, yellow for the 2004 Belgian Grand Prix
LMM
Google Leve
Round Rerank
End- -ond
Vale aeu cnek teadcood & oo feor le Reque Circ Fils
Renult
Tale pou deadood foler bvery sue
Websitt
Soothh Engine
Tike move bodker avyy ti credd
Round Summarization
This Alpnn sorec traeport& Webeere tieey ta StivE Soep
Raw
Servenshot
Conten
Couaat Sass en Likk didails than Phacns Suax Yil audtapa Agliai dhe lave Fioaala tai soland was del vid cacef Teey wothodd y Edlass Grand hax Alpec's Algan Garad workadd
Retrieve Relevant Content
Crawter
Deak od aheen Budhin
Summarize Desipaol
Figure 2: The Pipeline of MMSSAACH-ENGGHA
The process comprises three sequential stages executed by LMM: (i) requery, (ii) rerank, and (iii) summarization. In the end-to-end evaluation task, the LMM completes these three stages sequentially to generate the final output.
2 MMSEARCH
In Section 2.1, we first detail the design of our multimodal Al search engine pipeline, which serves as both data collection and evaluation tools. Then, in Section 2.2, we detail the data composition and collection we elaborate on our step-wise evaluation strategy. Finally, we detail the dynamic nature of our benchmark in Section 2.4.
the curated multimodal search benchmark MMSEARCII Then, Section 2.3,
2.1 MMSEARCU-ENNGING A MULTIMODAL AI SEARCH ENGINE PIPELINE
The searching process is complex action including multi-round interactions between LMMs and conventional search engines. We develop a delicate pipeline that queries LMMs multiple times to accomplish this task. Leveraging the image comprehension capabilities of LMMs, we incorporate two types of visual data. First, we incorporate Google Lens (len) to search for information from the image. The second type of visual data is the screenshot of the retrieved websites, the purpose of preserving the original format of website content. Our framework is shown in Fig. 2. Below we detail how an LMM works with this pipeline, which comprises three sequential phases:
i. Requery. The query direct from users may contain references certain information in the image, e.g., the News- Finance example shown in Fig. 1. Since conventional search engine only accept text-only input, it is necessary for LMM combine it with the query to ask a valid question to it. In addition, the raw user query may be ambiguous inefficient sometimes (Chan et al., 2024; Ma al., 2023), reformulating the query to be more clear is also a must for LMM. If the user query contains an image, we incorporate the screenshot of the image search result from the google lens (len). We treat the user query, user image, and the image search screenshot basic information of the query. This information will be input LMM in every round the pipeline. For the requery round, we prompt LMM to output a requery to conventional search engine.
translate the image content and
ii Rerank. The requery is sent to a search engine API, e.g., DuckDuckGo, to retrieve top K relevant websites. Depending on the requery quality, not all retrieved websites are necessar- ily relevant for query answering. Hence, we prompt LMM to select one most informative website for answer summarization. Due the LMM's context length limitations and the
4

---

Published as a conference paper at ICLR 2025
Figure 3: Area and Subfield Distribu- tion of MMSEARCH.
Table I: Key Statisties of MMSEARCH.
Statistic
Number
Total questions
300
Anioe 3.39
Net 39
Tradttionl
Questions with images
171 (57.0%)
Ammouny 1.9%
Sports
Questions without images
129 (43.0%)
Fashion 398
9,3%
Total Wehsites
2,280
Total Areas/Subficlds
2/14
Sports 14.7%
Arts 8.3%
Tuchooogy 4.9%
Number of unique images
2,901
Query images
163
Google search images
Top section serecnshot images
2,280
163
Architecture 4,7%
Knowlddge
26, 3%
Science 14.0%
Paper 8,0%
Number of unique questions
Full-page screenshot images
289
295
Prensied
News 73.7%
4.3%
Number of unique requeries
289
Enerrtiiment 13,7%
Number of unique reranked websites
2,400
General News 16.7%
Numher unique answers
264
Maximum question length
41
Finance
Maximum answer length
12
3%
Average question length
14.0
Average answer length
1.9
extensive content of websites, provide only essential information of each website, which we term brief results. These brief results include the title, the snippet, and screenshot of the webpage's s top section, which serves as the input for LMM's reranking. The inclusion of the screenshot serves two purposes. First, the screenshot offers a visual cue to assess the web's credibility, as a wrll-orga website often appears more trustworthy than one cluttered with advertisements (Fogg et al., 2001; Sillence et al., 2004). Additionally, the screenshot may contain essential visual information. For instance, it might include images similar or identical to query images, as shown in the Website 2 in Fig. 2.
ii.. Summarization. We start by crawling the selected website to gather all the available in- formation. We parse the HTML to obtain the raw textual content and capture a full-page sereenshot of the website. However, there are two issues: the raw content tends to be ex- tensively lengthy and disorganized, while substantial areas in the full aull-page screenshot blank due to the ad blocks on the website. These two issues lead to a large number of input tokens filled with irrelevant information. To enhance data efficiency, and retrieve the relevant content before inputting them to LMM. For the full-page screen- shot, we identify the blank areas and remove them iteratively, detailed Appendix As for the text content, we apply text embedding model (Chen et al., 2024a) to retrieve max- imum of 2K tokens relevant to the requery from the raw content. We define the slimmed screenshot and the retrieved content as full website content. Finally, we input the full web- site content, website title, and website snippet, along with the query information, to LMM for summarizing the answer.
slim the screenshot
2.2 DATA COMPOSITION AND COLLECTION
To thoroughly assess multimodal search proficiency, we compile a comprehensive problem set cov- ering broad spectrum of news topics, specialized knowledge domains, and query image patterns. This widespread collection for MMSEARCH aims to simulate diverse user searching scenarios, en- suring a robust evaluation of LMMs' capabilities in multimodal search.
Data Composition and Categorization. Our benchmark aims to isolate LMMs' inherent knowl- edge and assess their actual search capabilities. We focus on two primary areas: News and Knowl- edge. For the News area, the queries are related to the latest news at the time of data collection (August 2024). This guarantees no overlap between the current LMMs' training data and questions in our benchmark. All questions in this area are recorded with their occurrence time. For faimesss LMMs with recently updated knowledge should be tested on queries that occurred after their lat- est data update. Due to its time-sensitive nature, the News benchmark. Please refer to Section 2.4 for details. As for the Knowledege area, we focus on rare knowledge in targeted domains. Each question proposed by an annotator is verified be beyond the
serves dynamic part of our
5

---

Published as a conference paper at ICLR 2025
Task
Input
LMM Output
Ground Truth
Requery
Query Information
LMM Requery
Requery Annotation
L301
Rerank
Information
Brief Result
LMM Rerank
Rerank Annotation
Summarization
Query Information
Full Websstte
LMM Answer
Answer Annolation
End-to-end
Query
LMM Answer
Answer Annotation
Information
Requery
Figure 4: Outline of Evaluation Tasks, Inputs, and Outputs. Our evaluation contains four tasks. The requery, rerank, and summarization tasks assess the LMM's proficiency in individual pipeline rounds. The end-to-end task simulates real- worldworld search scenario by sequentially executing all three stages. An example of the input and output is shown in Fig. 10 in the supplementary material.
capabilities of state-of-the-art Large Language Models (LLMs) such as GPT-4o (OpenAI, 2024b) or Claude 3.5 Sonnet (Anthropie, 2024). The Knowledge area serves as a static component of our benchmark and remains constant over time. We collect a total of 300 queries across the 2 primary areas and 14 subfields. This dataset size balances comprehensive evaluation with efficiency, con- sidering the multi-round interactions with the search engine and Internet for each query. Detailed statistics for data composition and categorization are presented Table and Fig. 3. Definitions of each subfield are in Appendix G.2.
Data Collection and Review Process. Thanks to the design of our pipeline, the data collection pro- cess follows similar procedure introduced in the pipeline. An annotator is first required to propose query and provide its answer, either sourced from the latest news or rare knowledge. The annotator then formulates requery based the query information. After K websites are retrieved from the search engine, the annotator is required to divide all K websites into three sets based on the brief results: valid (likely to contain the answer), unsure (relevance is difficult determine). and invalid (entirely irrelevant to the question). We mandate that at least one website must be classified as valid; if this criterion is not met, the annotator is required to adjust the requery obtain new search results. Finally, we randomly pick one website from the valid set and obtain its full content. To ensure the question is answerable, another annotator is employed to give an answer to the query based on the full content. If the answer is incorrect, the question needs to be revised until it is answerable.
2.3 EVALUATION PROTOCOL
In contrast with previous LMM benchmarks, the multimodal search process of LMM contains mul- tiple rounds. Only the end-to-end evaluation of the final answer inadequate reveal the models' deficiency in each core searching step. For example, the errors made by the model may occur during the summarization process, but might also stem from choosing an incorrect website during the reranking stage. To this end, we propose a step- wise strategy to evaluate the LMMs' capability on the three core searching steps, in addition to the end-to-end evaluation.
End-to-end score (S.2>): We compute the Fl score between the predicted answer and the ground truth to judge if the answer is correct.
Requery score (Sreg): We apply the average of ROUGE-L and BLEU-1 scores measure the similarity between the model's requery and human-annotated requery.
Rerank score (Sper): The rerank score is derived from the LMM's selection among K pre-defined websites. The score values is 1.0 for valid set, 0.5 for unsure set, and 0 for invalid set or incorrect format.
Summarization score (Ssum): Again, we compute the F1 score of LMM's answer based a pre-defined website content against ground truth.
The input, output, and ground truth of the four tasks are visualized in Fig. 4. The final score is weighted by these four scores. We assign the highest weight (75%) to the end-to-end task, as it reflects the real-world multimodal search capability. The remaining 25% is distributed among the intermediate steps: 10% each for the rerank and summarization tasks, and 5% for the requery task. The lower weight for the requery task accounts for the inherent uncertainty in this process. The scoring process be formulated as:

---

Published as a conference paper at ICLR 2025
S Sinal = 0.75 Seze + 0.05 Srey +.1 Sver + 0.1 Seum
(I)
Histogram Timestamps by Half Month
73
74
Frequency
60
30
18
20
1H, 24-05
2H, 24-05
IH, 24-06
2H, 24-06
1H, 24-07
2H, 24-07
1H, 24-08
24-08
Time
Figure 5: Timestamps Distribution of Questions in the News Area. All events of collected data occurred after May 2024. The majority of data concentrates on August. This ensures the data captures only recent events, falling beyond the knowledge cutoff dates of LMMs. The False Premise subfield is not included, since it is infeasible to determine the timestamp for an event that never occurred.
2.4 BENCHMARK EVOLUTION
In Fig. 5, we showcase the statistics of data timestamp distribution the News area. Our dataset spans from 1st May 2024 to 31th August 2024. By the time of evaluation, we inspect the knowledge cutoff dates of the closed-source models. Claude 3.5 Sonnet reports knowledge cutoff of April 2024, while both GPT-4V and GPT-4o state they lack information from 2024. For open-source models, we examine their release dates and training data, confirming that none possess knowledge beyond May 2024. This temporal gap ensures the fairness of our evaluation, the models' perfor- mance solely reflects their multimodal search capabilities rather than pre- are-existing knowledge. We will update the News area if new LMM's training data may overlap with collection period.
3 EXPERIMENT
In this section, we conduct a systematic evaluation of existing LMMs on MMSEARCH. We first introduce the experimental setup in Section 3.1. Then, detail the quantitative results in Sec- tion 3.2 and narrate the eтror analysis in Section 3.3. Finally, we conduct an in-depth ablation study of scaling test-time compute versus scaling model size in Section E. I. in the supplementary material.
3.1 EXPERIMENT SETUP
Evaluation Models We examine the performance of foundation models across three distinct cate- gories MMSEARCH: (a) Commercial AI Search Engines, represented by Perplexity (Perplexity). We test the pro version of Perplexity, which takes only the user query and image as input. Since SearchGPT (OpenAI, 2024c) has not been public yet, do not test it. (b) Closed-source LMMss, represented by models like GPT-4V (OpenAI, 2023c), GPT-4o (OpenAI, 2024b), and Claude 3.5 Sonnet (Anthropic, 2024). and (c) Open-source LMMs, featuring models such as LLa Vision- 7B (Li et al., 2024b) (Qwen2-7B (Yang et al., 2024a)), LLaVA-One Vision-72 (Li et al., 2024b) (Qwen2-72B (Yang et al., 2024a)). LLaVA-NeXT--mterlawww (Li al., 2024c) (Qwen1 1.5-BB (Yang al., 2024a)), Intern VL2 (Chen et al., 2024d) (IntemLMM..5-7B-CCL)
(Cai et al., 2024))24 InternLM- XC2.5 (Zhang et al., 2024a) (InternLM2-7B (Cai al., 2024)), Qwen2-VL VL-7B (Qwen Team, 2024) (Qwen2-7B (Yang et al., 2024a)), Qwen2-VL-721 (Qwen Team, 2024) (Qwen2-72B (Yang et al., 2024a)). mPlug-Owe Ow13 (Ye et al., 2024) (Qwen2-7B (Yang et al., 2024a)), Idefics3 (Laurençon et al., (AI@ Meta, 2024)), and Mantis (Jiang et al., 2024b) (LLaMA3- 7B (AI@ Meta, 2024)). Note that the open-source LMMs' sizes are 7B unless otherwise specified.
2024) [LLaMA3.1-7B-Imgg
Implementation Details. We set the number of retrieved websites K as 8. We include two image input resolution settings. For the default settings, the longest edge of the input image is resized to match the largest resolution of the vision encoder of LMM. This ensures the image not to be cropped to multiple images and will only take up the minimum of tokens for image input. For any resolution settings, we input the image without resizing. More details available in Section F.
7

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation Results of Four Tasks in MMSEARCH. We report the scores of news and knowledge areas and their average score in each task. Subscript AnyRes indicates original resolution image input; otherwise, rewoluttttion images and open-source LMMs are marked in red and blue. For spen-source LMMs, we adopt the models with 7B parameters unless otherwise specified.
used. The highest scores for closed
Model
All
End-go-end
Requery
Rerank
Summarize
Avg
News
Know
Avg
News
Know.
Avg
News
Know.
Aug
News
Know.
Arg
News
Know.
Boseliwx
Human
09.2
69.6
68.1
68.2
68.6
67.1
43.7
45.0
40.1
85.7
87.3
81.2
72.8
71.4
76.7
Connercial Al Search Engines
Perplesity Pro (Porplexity)
47.8
52.7
34.1
Cloved-ourrer LMMs with MMSPARCH-
Claude 3.5 Sonner (Anthropic, 2024)
$3.5
53.1
54.7
49.3
42.0
43.6
37.7
80.2
78.7
84.2
59.4
60.3
GPT-4V (OpenAl, 2023c)
55.0
55.0
55.3
52.1
52.2
51.9
45.7
49.2
35.8
79.3
76.9
86.1
57.4
56.7
59.4
GPT-4o (OpenAl, 2024b)
62.3
61.2
65.3
60.4
39.0
64.5
46.8
49.9
38.0
83.0
82.4
84.8
63.1
62.2
65.6
Open-vonnce 7.MMyy
with
MMSEARCH-ENNE
Mantis (Jlang ol., 2024b)
18.7
19.8
15.9
7.4
39.7
41.0
36.1
22.0
11.5
M-XMLM (Zhing all, 2024g)
22.2
22.8
20.5
22.9
23.6
20.8
25.0
24.3
27.0
0.0
0.0
0.0
37.7
38.6
35.0
Interal M-XC2 Saotee
22.3
23.9
17.5
23.2
25.4
16.9
21.7
19.8
26.9
0.0
0.0
0.0
37.7
38.6
35.1
AlVV-NNXXT-mmeeooee el al., 2024c)
28.3
29.2
25.6
23.8
20.5
13.5
55.3
58.6
46.2
mPPlgg-O13 (Yo al., 2024)
52.1
24.4
28.1
74.3
45.6
45.6
mPllg- Onede
33.9
35.5
29.3
27.3
29.1
21.2
31.8
36.1
19.9
74.5
72.9
79,1
13.9
43.6
41.6
InteraVL2 (Chen at., 20244)
34.3
35.7
30.2
30.9
32.5
26.2
32.3
36.1
21.6
46.5
49.3
38.6
48.5
45.9
55.8
34.1
34.2
33.7
30.0
30.3
29.0
31.4
35.5
19.7
53.2
52.3
55.7
46.9
44.4
53.7
Idefies3 (Laurençom 2021)
36.2
38.5
29.6
32.3
73.3
Idefis3teote
35.7
38.2
28.7
30.1
32,9
22.3
27.2
32.2
13.2
72.7
73.1
71.5
45.2
46.3
42.1
LLaVA-One OneVisnn (Li et all, 2024b)
36.6
39.4
28.9
29.6
33.1
19.7
35.8
40.3
23.2
72.8
73.5
70.9
53.5
51.8
58.5
Qwen2- Qwen2-VLausa VLuusu (Qwen Team, 202()
45.3
44.1
48.7
40.3
39.2
13.5
39.0
41.9
30.8
76.7
73.8
84.8
51.7
52.7
60.1
AVA-Oneeisionn (72R)
50.1
50.1
50.2
44,9
45.1
42.9
34.3
82.2
80.1
86.7
61.44
59.1
67.7
QwenZ-VLLako, (72B)
52.7
52.0
54.5
76.7
72.9
3.2 EXPERIMENTAL ANALYSIS
To thoroughly investigate the multimodal searching capabilities, we present the evaluation results of different models on MMSEARCH following the proposed step-wise evaluation strategy in Table 2 and fourteen subfields Table We now provide a detailed discussion of notable findings and their implications for multimodal search capabilities.
Any-resolution input only provides slight or no improvement. Of the tested LMMs, four models, which are InternL.M-XC2.55, (LowRes) and any-resolution input (AnyRes). As one would expect, AnyRes input enables better OCR and perception of the image. However, we only observe slight even enhancement com- paring the difference between the LowRes performance and its AnyRes counterpart. Take mPlug- Ow13 as example, AnyRes input surpasses LowRes input overall score by 1.8%, end-to-end score by 2.7%, and rerank on 0.2%. While it falls behind LowRes on requery by 0.8% and summa- rization by 1.7%. This suggests that the OCR and perception quality do not bottleneck the search performance, Rather, the suboptimal performance appears to stem from the LMMs' inherent lack of robust search capabilities.
Vite mPlug-Ow13, and Idefic3, all support both low-resolution
Current LMMs still have significant shortcomings in requery and rerank. Comparing the aver- age score of the end-to-end task with that of the summarization task, we find that the summarization score consistently surpasses the end-to-end task by a large margin, both in the closed-source and open-source models. The minimum margin is 2.7% for GPT-40, while the maximum is 23.9% for LLLVA- VLLaVA--neiiiioon7... quality. While the summarization task input always contains the answer, the end-to-end task's third- round input quality depends on the model's requery and rerank quality in previous rounds. The magnitude of this performance gap reflects the disparity between a model's summarization ability and its capacity for requery and rerank tasks. The larger the difference, the larger the capability gap. Observing the result, we find that this gap of most open-sourced models exceeds 14%, while the
Vision-7B. This discrepancy can be attributed to the differences in the tasks' input
models all below 10%. This suggests all current LMMs needs improvement
8

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation Results on Difffernt Subfields in MMSEARCH. SPO: Traditional Sports;
ESP: E-Sports; ENT: Entertainment: GEN: General News; PAP: Paper; TEC: Technology; FIN:
Finance; FAL: False Premise; ART: Arts; ARC: Architecture; AST: Astronomy: ANI: Anime; AUT:
Auto: FAS: Fashion. The highest scores for closed-source and open- sounes LMMs are marked in red and blue. For open-source LMMs, we adopt the models with 7B sizes unless otherwise specified.
Knowledge
Model
Avg
SPO
ESP
Avg ART
FAS
Clovedfoouree LMMs with MMSEARCH- ENGINE
Clande 3.5 Scanet (Aniaropic, 202()
53.5
53.0
37.4
50.2
63.6 49.2
52.8 67.7 43.4
63.1
51.7
59.0
42.9
70.9
56.5
60.6
36.4
GPT-4V (OpenAI, 2023c)
55.0
54.9
49.3
48.5
67.4
43.5
37.3 64.2 59.1
90.2
55.3
54.6
45.6
63.1
63.0
52.2
56.2
GPT-4o (OpenAI, 2024b)
62.3
61.2
63.7
61.2
72.3 51.3 48.6 68.6 6000
76.8
65.3
73.8
52.0
57.6
76.8
68.4
55.8
LMMs
wath MMSEARCH ENGINE
InternL (Zhang al., 2024a)
22.2
22.8
22.6
13.6
28.9
23.5
15.5
27.8
32.9
3.1
20.5
29.2
24.0
20.8
2.0
20.4
11.8
InteraLM-XC2.55
22.2
23.9
25.2
12.8
30.6
21.0
18.7
3144
32.8
13.9
17.6
22.9
25.6
10.8
2.0
28.1
4.7
Mantis (Jing et 2024b)
18.8
19.8
17.4
9.7
25.3
19.6
22.1
35.3
18.2
6.2
15.9
23.0
17.5
12.5
14.8
12.7
3.3
LLaVA-NeXT-bntereewe (li et nl,, 2024c)
28.3
29.3
23.3
18.5
41.9
33.6
26.2
27.3
28.0
14.7
25.6
31.1
22.7
49.7
16.5
19.7
7.1
InternVL22 (Chen et al., 2024d)
34.3
35.7
38.7
20.7
46.8
33.2
24.6
44.1
44.5
27.2
30.1
41.7
33.5
26.9
15.4
36.8
7.9
LntenVL2avvee
34.0
34.2
32.0
26.6
45.4
35.0
18.7
23.7
43.1
36.3
33.7
49.5
26.4
36.5
18.3
35.2
15.5
mPlug-Owl3 (Ye al., 2024)
32.1
34.8
80.0
20.6
46.7
33.7
22.3
30.4
45.7
41.5
24.4
28.7
24.6
33.5
18.3
24.2
10.5
mPllg--OlBsseee
33.9
35.3
42.0
27.1
49.7
34.4
24.5
304
41.3
18.9
29.3
94.4
20.3
40.9
18.8
32.8
24.4
Idefics3 (Laurençon et al,, 2024)
36.2
38.5
43.5
24.2
50.1
41.7
27.5
36.2
38.8
37.4
29.6
34.2
32.1
32.1
29.6
32.7
9.1
Idelie33aote
35.7
38.2
40.5
27.3
48.0
414
33.1
36.9
42.1
17.4
28.8
40.5
26.2
27.1
14.2
40.8
7.3
anne-Onniisioo (I at., 2024h)
36.6
39.4
27.1
50.0
31.4
52.1
47.4
23.5
28.9
38.0
34.3
21.0
29.6
11.2
Qwen2 VLaatu (Qwen Team, 2024)
45.6
38.1
49.8
70.0
54.4
23.3
LLaVA-One (72B)
45.4
62.1
70,0
31.7
Swen2-VLoesee (72B)
52.9
50.0
46.3
57.2
58.0
41.9
of their requery and rerank ability, especially for open-source models. Mantis is one exception of open-source models with a margin of only 3.4%. This means its poor summarization capabil- ity bottlenecks its end-to-end performance. Qwen2-VL-72B's 10.5% gap. also falling below 14%, highlights its superiority among other open-source LMMs.
Closed-source LMMs are better-performed than open-sourced LMMs on overall performance. For the final score, closed-souree source LMMs consistently outperform the open-source LMMs. GPT-4o achieves the highest overall score of 62.3%, demonstrating superior zero-shot multimodal search capabilities. While Qwen2-VL-7D7 leads among open-source models, it still lags behind GPT-4o by 9.6%. The performance gap widens to 11.3% on the most challenging end-to-end task and further expands to 20.1% for 7B open- LMMs. These significant disparities highlight substantial room for improvement in open-source models.
SoTA LMMS with MMSSARCH-ENNGAN surpass commercial Al search engines in the end-to-end task. We also evaluate the pro version of Perplexity (Perplexity), a prominent dataset. Per-
commercial AI search engine that accepts both image and text queries, plexity pro can accept both image and text in the user query. Surprisingly, although Perplex- ity also leverages SoTA LMMs like GPT-4o and Claude 3.5 Sonnet, it largely underperforms MMSEERCCH-ENGINE equipped with the same model in the end-to-end task. Even more remark- ably, MMSEARCH-EENINNE can even surpass Perplexity with Qwen2-VL-72B, an open-source LMM. This suggests that our AMMSEARCH-ENGINE provides a better open-source plan for mul- timodal Al search engine. The performance gap validates AMMSEARCH-ENGINEE tiveness and highlights the value of testing various LMMs within our pipeline, since the pipeline can indeed achieve remarkable performance when using powerful LMMs. Upon investigating Per- plexity's 's sub-optimal performance, we discovered that it appears to utilize only rudimentary im- age search algorithm, if any. This limitation leads to poor identification of the key objects in the image and failure to retrieve relevant information. Our findings underscore the effectiveness of MMSEARCH-ENT -NGHNNE'''''SS design, particularly the incorporation of a robust image search step, which plays crucial role in accurately recognizing important information from the input image.
design effec-

---

Published as a conference paper at ICLR 2025
GPT-4o
Qwen2-V1-78
Requery Eccor Type
Sumamarize Error Type
infremal 17.1%
informal
Charge
summmatio
Lockin 3414
louge AB%
14.%%
Rearnserr
sequery
Imagoonc Agpropita Sore
Halkeraaia A0%
Figure 6: Distribution of Error types of
Figure 7: Distribution of Error types of Qwen2-VL-7B (Qwen Team, 2024) in the reuqery and summarization task respectively.
GPT-4o (OpenAI, 2024b) and Qwen2-VL-
7B (Qwen Team, 2024) in the end-to-end task.
3.3 ERROR ANALYSIS
To investigate the limitations of current LMM search capabilities, we conducted comprehensive analysis of error types observed in our evaluation. Our proposed step-wise evaluation strategy en- ables analysis of failure modes for each core search step, complementing the end-to-end assessment. This analysis encompasses the entire benchmark. We first examine the end-to-end error types for both the best-performing closed-source model (GPT-4o) and open-source model (Qwen2-VL-7B), To better understand the failure cases, we then identify distinct error types in the requery and summa- rization task, which requires open-ended generation. We quantify these error types for systematic understanding of current LMM limitations and point out critical areas for improvement.
Error Analysis of End-to-end Task In this paragraph, we are trying to answer the question: Which step does LMM make mistake in the end-to-end evaluation? In Fig. 6, we showcase the statistics of different error types occurring in GPT-40 and Awen2-VLL-7B. We define the following four error categories: (i) requery, where the model requery is incorrect, and leads to all retrieved websites being invalid; (ii) rerank, where the model selects a website without a correct answer; (iii) summarization, where the full website content contains the information of correct answer, but the model fails to extract it; (iv) informal, the output format deviates from the prompt specifications. As shown in the figure, GPT-4o's primary error sources are rerank and summarization errors, while requery and informal errors account for approximately half the frequency of the main error causes. This suggests that GPT--4o's 's limitations lie primarily in information source ranking and multimodal information integration. As for Qwen2-VL, all four error types occur with similar frequeency The rise of the informal error portion may be attributed the model's inferior instruction-follle following ability. Besides, it should be noted that the requery task demands advanced comprehension and key image information extraction ability. This task seldom appears in the training data of current LMMs. The prevalence of this error type in LL may indicate that it fails to generalize to adequately address this complex task.
Error Analysis of Reugery and Summarization Task To better understand how open-source LMM makes the mistake, we dive into the requery and summarization task to find out the error patterns of Qwen2-VL-7B. We particularly select the two tasks requiring open-ended generation, which provides more information to identify the error. We define five types of eгror for both the requery and summarization tasks in Section E.2 and showease the statistics Fig. 7. The requery emror suggest that LMM often fails to fully understand the requery task and aggregate all available information. Besides, inefficient query error indicates that LMM has no clue about the real working scenario and query principles of search engines.
4 CONCLUSION
In this paper. we investigate the potential of LMMs as multimodal AI search engines. We first design MMSFARCH-ENGING a streamlined pipeline, enabling zero-shot LMMs perform mul- timodal searches. To comprehensively assess the search capabilities, we introduce MMSEARCH, a benchmark comprising 300 queries across 14 subfields. Our evaluation methodology analyzes LMM search abilities step-by-step, facilitating a deeper understanding of their limitations. Using MMSEARCH- ENNGNE, we evaluate various closed-source and open-source LMMs, revealing that current models still fall short of human-leve search proficiency. Through thorough error analysis, we identify specific patterns of failure in key search process steps, providing valuable insights for future improvements in LMM search ability.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This project is funded part by National Key R&D Program of China Project 20222D0161 100, by the Centre for Perceptual and Interactive Intelligence (CPI)) Ltd under the Innovation and Technol- ogy Commission (ITC)'s s InnoHK, by NSFC-RGC Project N.CUH4498/24. Hongsheng Li is a PI of CPII under the InnoHK.
REPRODUCIBILITY STATEMENT
We provide the demo code of SMISEARCH-ENGING which be also used for inference, in the code directory in the supplementary material. We also include the end-to-end data the data directory in the supplementary material, along with the loading script. Implementation details of inference settings are demonstrated in Section 3.1 and Section
REFERENCES
Google lens. URL
https://lens.googge com.
Web interface available at
https:/immages.goglle.oo
AI@Meta. Llama model card, 2024. URL
plob/main/MOBn CARD md.
Anthropic. Claude-3.5. httppp::www.aattarppp.....oooees 2024.
sonnet,
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannanch Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflcction. arXiv preprint arXiv:2310.11511, 2023
Oriol Barbany, Michael Huang, Xinliang Zhu, and Arnab Dhua. Leveraging large language models for multimodal search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1201-1210, 2024.
Patrice Béchard and Orlando Marquez Ayala. Reducing hallucination in structured outputs via vetrieval-augmented generation. arXiv preprint arXiv:2404.08/ 2024.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann,, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International conference machine learning, pp. 2206-2240. PMLR, 2022.
Sergey Brin and Lawrence Page. The anatomy of a large-scale hypertextual web search engine. Computer networks and ISDN systems, 30(1-7):107-11 1998.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li. Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong. Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma. Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang. Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang. Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang. Ziyi Wang, Xingjian Wei, Qizhen Weng. Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang. Haochen Ye, Huaiyuan Ying. Jia Yu, Jing Yu, Yuhang Zang. Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijic Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang, Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin. Internim2 technical report, 2024.
Chi-Min Chan, Chumpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. Rq-rag: Learning to refine queries for retrieval augmented generation. arXiv preprint arXiv:2404.00610 2024.
11

---

Published as a conference paper at ICLR 2025
Guo Chen, Yin-Dong Zheng, Jiahao Wang. Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2335.13322. 2023a.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi-lingual, fultiifuuctionnalty. multi-gra uulti-granularity text embeddings through self- sofflknoweedge dis- tillation, 2024a.
Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17754-17762 2024b.
Jun Chen, Deyao Zhul Xiaoqian Shenl Xiang Li, Zechun Liu2 Pengchuan Zhang, Raghuraman Krishnamoorthi2 Vikas Chandra2 Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: Large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478. 2023b.
Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websre: A dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465, 2021.
Zehui Chen, Kuikun Liu, Qiuchen Wang, Jiangning Liu, Wenwei Zhang, Kai Chen, and Feng Zhao. Mindsearch: Mimicking human minds elicits deep ai searcher. arXiv preprint arXiv:2407.20118 2024c.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are we to gpt-4v? closing the gap to com- mercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024d.
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Intermm- xxcopossrx:m Mastering free- form text-image composition and comprehension in vision-language large model. arXiv preprint arXiv:2401. 16420, 2024.
Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. A survey on rag meeting 1lms: Towards retrieval-auggmente large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6491 6501, 2024.
Brian Fogg, Jonathan Marshall, Othman Laraki, Alex Osipovich, Chris Varma, Nicholas Fang, Jyoti Paul, Akshay Rangnekar, John Shon, Precti Swani, et al. What makes web sites credible? report on a large quantitative study. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 61 61-68, 2001.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. /3394, 2023.
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evalua- tion benchmark of multi-modal 11ms in video analysis. arXiv preprint arXiv:2405.21075, 2024.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, al. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:2312.11370 2023a.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint ar3iv:2304. 20.22344400010 15010, 2023b.
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family multi-modal large language models. ICMI. 2024, 2024.
12

---

Published as a conference paper at ICLR 2025
Google Gemini Team. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:23/2.11805. 2023.
Zilu Guo, Hongbin Lin, Zhihao Yuan, Chaoda Zheng, Pengshuo Qiu, Dongzhi Jiang, Renrui Zhang, Chun-Mei Feng, and Zhen Li. Pisa: A self-augmented data engine and training strategy for 3d understanding with large models. arXiv preprint arXiv:2503. 10529, 2025a.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang. Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00 00615, 2023.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Chengzhuo Tong, Peng Gao, Chunyuan Li, and Pheng- Ann Heng. Sam2point: Segment any 3d as videos in zero-shot and promptable manners. arXiv preprint arXiv:2408. 16768, 2024.
Ziyu Guo, Ray Zhang, Hao Chen, Jialin Gao, Dongzhi Jiang, Jiaze Wang, and Pheng-Ann Heng- Sciverse: Unveiling the knowledge comprehension and visual reasoning of Imms on multi-modal scientific problems. arXiv preprint arXiv:2503.10627, 2025b.
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng- Ann Heng. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint vrXiv22501.39226 2025c.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pp. 3929-3938.
PMLR, 2020.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint Stti:2300.03900. 2023.
Qiuxiang He, Guoping Huang, Qu Cui, Li Li. and Lemao Liu. Fast and accurate neural machine translation with translation memory. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( VVoluee 1: Long Papers), pp. 3170-3180, 2021.
Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025.
Lei Huang, Weijiang Yu, Weitao Ma. Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng. Xiaocheng Feng, Bing Qin, et A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05232, 2023.
Yueru Jia, Jiaming Liu, Sixiang Chen, Chenyang Gu, Zhilue Wang, Longzan Luo, Lily Lee, Pengwei Wang, Zhongyuan Wang, Renrui Zhang, et al. Lift3d foundation policy: Lifting 2d large-scale pretrained models for robust 3d robotic manipulation. arXiv preprint arXiv:2411.1886 2024.
Bowen Jiang, Yangxinyu Xie, Xiaomeng Wang, Weijie Su, Camillo J Taylor, and Tanwi Mallick. Multi-modal and multi-agent systems meet rationality: A survey. arXiv preprint arXiv:2406.0025: 2024a.
Dongfu Jiang, Xuan He, Huaye Zeng, Con Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint arXiv:2405.01483 2024b.
Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXiv preprint arXiv:2404.0336 2024c.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi. Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking th-thugt in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.0661 09621, 2025.
13

---

Published as a conference paper at ICLR 2025
Nick Kanopoulos, Nagesh Vasanthavada, and Robert L Baker. Design of image edge detection filter using the sobel operator. IEEE Journal of solid-state circuits, 23(2):358-30 1988.
Hugo Laurençon, Andrés Marafioti, Victor Sanh, and Léo Tronchon. Building and better under- standing vision-language models: insights and future directions., 2024.
Bo Li, Kaichen Zhang. Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Iims supercharge multimodal capabilities in the wild. 2024a.
https//llava-vll.iiheee
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint OrXiv:2408.03323 2024b.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-imterleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.00 00795, 2024c.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900 PMLR, 2022.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.0635: 2023.
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. arXiv preprint urXiv:2402.12877 2024d.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. ECCV 2024, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:/ 1lava-vl. g githu......
1-0--00--11----- 1lava-nex//.
Jiaming Liu, Mengzhen Liu, Zhenyu Wang, Lily Lee, Kaichen Zhou, Pengju An, Senqiao Yang, Renrui Zhang, Yandong Guo, and Shanghang Zhang. Robomamba: Multimodal state space model for efficient robot reasoning and manipulation. NeurIPS 2024, 2024b.
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal 1lms evolved in web page understanding and grounding? arXiv preprint arXiv:2404. 05955, 2024c.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmhench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.0628/1 2023b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chun yue Li, Hannanch Hajishirzi, Hao Cheng, Kai-
Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with 4v,-4v, bard, and other large multimodal models. ArXiv, abs/2310.02255. 2023.
Xing Han Lü, Zdenčk Kasner, and Siva Reddy. Weblinx: Roal-world website navigation with multi- turn dialogue. arXiv preprint ar 055303 2024.
14

---

Published as a conference paper at ICLR 2025
Feipeng Ma, Hongwei Xue, Yizhou Zhou, Guangting Wang, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Visual perception by large language model's weights, Advances in Neural Information Processing Systems, 37:28615-222 28633,,, 2025.
Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, and Nan Duan. Query rewriting for retrieval- augmented large language models. arXiv preprint mrXiv:2305.14283. 2023.
OpenAl. Chatgpt. https: //chtt///httt/tteeett www.ooeen....... com, 2023a.
OpenAl. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023b.
OpenAl. GPT-4V(ision) system card, 2023c. URL http:::/openaii.cmm/rrees
eot-4v-systen-card.
OpenAL. Openai ol. [Online], 2024a.
htt:s://ogeeall.com/neeeee
learning-tor reason- with withwwwwthhhh........
OpenAl. Hello gpt-4o. http:///pggnii.cco//n/eeeeeeeele
OpenAl. Searchgpt.
Searchppt-prototype [Online], 2024c.
https://openai: http://aeiccmm///e/////o
Perplexity. Perplexity.ai. [Online]. https::/ww..prppleettt
Qwen Team. Qwen2-vl. 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Interna- tional Conference on Machine Learning, 2021. URL https: api semanticscholas Tzg//Crpussl 23159144 145.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp.
2022.
Elizabeth Sillence, Pam Briggs, Lesley Fishwick, and Peter Harris. Trust and mistrust of online health sites. In Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 663-670, 2004.
Yiwen Tang, Zoey Guo, Zhuhao Wang, Ray Zhang, Qizhi Chen, Junli Liu, Delin Qu, Zhigang Wang, Dong Wang, Xuelong Li. et al. Exploring the potential of encoder-free architectures in 3d Imms. arXiv preprint wwXXi:2502209622 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. /3971,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023b.
Fei Wang. Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:24066 09000009944111 2024.
Peng Xia, Ze Chen, Juanxi Tian, Gong Yangrui, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, et al. Cares: comprehensive benchmark of trustworthiness in med- ical vision language models. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks' Track.
15

---

Published as a conference paper at ICLR 2025
Peng Xia, Siwei Han, Shi Qiu, Yiyang Zhou, Zhaoyang Wang. Wenhao Zheng, Zhaorun Chen, Chenhang Cui, Mingyu Ding, Linjie Li, et al. Mmie: Massive multimodal interleaved compre- hension benchmark for large vision-language models. arXiv preprint 1rXiv:2410.10139 2024a.
Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. Mmed-rag: Versatile multimodal rag system for medical vision language models. arXiv preprint arXiv:2410.1308 2024b.
Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 1081-1093, 2024c.
Junlin Xie, Zhihong Chen, Ruifei Zhang, Xiang Wan, and Guanbin Li. Large multimodal agents: A survey, arXiv preprint 15111:240..15166, 2024.
Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models understand point clouds. arXiv preprint arXiv:2308. 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang. Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc. Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint 2rXiv:2407.1061 2024a.
Senqiao Yang. Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, and Shanghang Zhang. Lidar-lim: Exploring the potential of large language models for 3d lidar understanding. arXiv preprint 1rXiv:2312. 14074, 2023.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. Gtag-commrehensiing rag benchmark. arXiv preprint arXiv:2406.044 0074444444444444 2024b.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint arXiv:2408. 048400 2024.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qi Qian, Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023a.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration, 2023b.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. ArXiv, abs/2308.00 02490, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instraction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.00 02858, 2023.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. deemmlm-xcoposeers A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03322 2024a.
16

---

Published as a conference paper at ICLR 2025
Renrui Zhang. Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. LLaMA-adapters Efficient fine-tuning of large language models with zero- initialized attention. In The Twelfth International Conference Learning Representations, 2024b. URL https::////. /ttps///pppeeeeeeeeeeeeeeee ew. net forum? A
Renrui Zhang, Dongzhi Jiang. Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? ECCV 2024, 2024c.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv preprint (xX::2447000839, 2024d.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Zhuofan Zong, Dongzhi Jiang, Bingqi Ma, Guanglu Song, Hao Shao, Dazhong Shen, Yu Liu, and Hongsheng Li. Easyref: Omni-generalize group image reference for diffusion models via mul- timodal 1lm. arXiv preprint arXiv:2412.0966/8 2024a.
Zhuofan Zong Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404. 13046, 2024b.
17

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL OVERVIEW
Section A: Related work.
Section B: Automated data curation pipeline.
Section C: Additional dataset details.
Section D: Future direction.
Section E: Additional experiments and analysis.
Section F: Additional experimental details.
Section G: More dataset details.
Section H: Qualitative examples.
A
RELATED WORK
Large Multimodal Models. Recently, multimodal models (Radford ct al., 2021: Li et al., 2022; OpenAI, 2023c; Rombach et al., 2022; Jiang et al., 2024c; Zong et al., 2024a) has gained unparal- leled attention. Building on the success of Large Language Models (LLMs) (Touvron et al., 2023a;b) and large-scale vision models (Radford al., 2021), Large Multimodal Models (LMMs) are gaining prominence across diverse domains. These models extend LLMS to handle tasks involving various modalities, including mainstream 2D image processing (Liu et al., 2023a; Zhu et al., 2023; Lin et al., 2023; Gao et al., 2023b; Ma et al., 2025), as well as 3D point clouds (Xu et al., 2023; Guo et al., 2023: 2024: Tang al., 2025; Guo et al., 2025a), medical images (Xia et al., 2024c;b), videos (Li et al., 2023; Chen et al., 2023a; Zhang et al., 2023; Fu et al., 2024), reasoning Guo et al. (2025c); Jiang al. (2025), and robotics Liu al. (2024b); Jia et al. (2024). Among these LMMs, Ope- nAI's GPT-4o (OpenAI, 2024b) and Anthropic's 's Claude 3.5 Sonnet (Anthropic, 2024) demonstrate outstanding visual reasoning and comprehension capability, setting new standards in multi-modal performance. However, their closed-source nature limits broader adoption and development. In contrast, another research trajectory focuses on open-source LMMs for the community. Pioneer- ing works like LLaVA (Liu et al., 2023a; 2024a; Li et al., 2024c;b), ALaMA-Adapter (Zhang al., 2024b; Gao et al., 2023b), and MiniGPT-4 (Zhu et al., 2023; Chen et al., 2023b) incorporate a frozen CLIP (Radford et al., 2021) model for image encoding and integrate visual information into LLM for multi-modal instruction tuning. Later, works such as mPLUG-OWl (Ye et al., 2023a;b; 2024), (Dong al., 2024) further
SPHINX (Gao al., 2024; Lin et al., 2023), and DeormLM-XCooponn advanced the field by incorporating diverse visual instruction tuning data and generalizing to more scenarios. More recent developments in the field have taken diverse directions. For example, several studies (Zong et al., 2024b; Tong et al., 2024) explore multiple vision encoders design. Meanwhile, other works (Liu al., 2024a; Chen et al., 2024d; Qwen Team, 2024) incorporate high-resolution image input. Multi-image instruction data (Li et al., 2024c; Jiang et al., 2024b) is also integrated to enable perception across multiple images. While various benchmarks, both in the general (Fu et al., 2023; Jiang al., 2025; Liu et al., 2023b; Yu et al., 2023; Xia et al., 2024a; Hong al., 2025) and expert (Zhang et al., 2024c; Lu et al., 2023; 2022; Xia et al.) domain, has been proposed, the poten- tial of LMM to function introduce the MMSEARCH benchmark, which evaluates LMMs' zero-shot abilities of multimodal search, offering valuable insights for future research.
multimodal search engine remains largely unexplored. To this end, we
Large models with Retrieval Augmented Generation (RAG). RAG [Retrieval-Augmentee Gen- eration) an effective strategy for enhancing model knowledge by retrieving relevant information from external sources (Fan et al., 2024). RAG has been leveraged in various scenarios including knowledge-intenee intensive question answering (Borgeaud et al., 2022; Guu et al., 2020), machine trans- lation (He et al., 2021), and hallucination elimination (Béchard & Ayala, 2024). Current works has focused on improving specific aspects of RAG. RG-RAG (Chan et al., 2024) proposes to refine the query for retrieval by decomposition and disambiguation. Self-RAG (Asai et al., 2023) incorpo- rates the self-reflection of LLM to enhance the generation quality. The AI search engine could be viewed as form of RAG with the Internet serving as the external knowledge source. Recently, MindSearch (Chen et al., 2024c) proposes an AI search engine framework to simulate the human minds in web information seeking. Meanwhile, multiple benchmarks of RAG (Yang al., 2024b;
18

---

Published as a conference paper at ICLR 2025
Chen al., 2024b) have been introduced to comprehensively evaluate RAG system. However, both the current AI search engine and RAG benchmark are limited to the text-only setting. leaving the multimodal search engine and evaluation largely unexplored. To bridge this gap, introduce CAISFARCH-ENGINS and MMSFARCH, a multimodal Al search engine pipeline and dataset de- signed to evaluate various multimodal scenarios.
B AUTOMATED DATA CURATION PIPELINE
Website
Question Chock
Requery
Summarization
Queskin hrage Cunceel
httpy:://wwwwww hwtp://wwwwwee//e/e thegrandd
isferratin lollage?
Full Webate Content
https///cchoooooggmaaaaaaee
Valid image??
Crawling
Requery
Sports Finance Science Emerraimennt
Randonly
WWW
choose valid website
News Filter
Question Proposal
Human Check
Rerank
Sazabke Laior
Valid
Abstract
Any Value
Unsef Zeaate
Baief Resuls
Unsure
Imvlidd
(a) End-to-end data generation
(b) Other three tasks data generation
Figure 8: Illustration of the Data Curation Pipeline. We first collect the end-to-end data by crawling inside the website pool and prompting LMMs to raise questions based on the content. Then we further prompt the LMM to generate annotation for other three tasks. The human check is optional for the end-to-end data generation but compulsory for other three tasks data generation.
data curation pipeline. The figure is shown in Fig, 8. We first define website pool and model pool. The website pool contains general news websites like CNN and expertise websites like Arxiv. The model pool contains state-of-art models for the data curation pipeline to guarantee diversity and fairness.
Now we introduce our automated//emi-attooial
1. End-to-end data curation. We employ web crawler to obtain all the sub-websites pub- lished later than a specific date. However, not all the websites are suitable for raising question to test the LMMs' searching capability. For example, some websites do not con- tain any recent news, while some websites' contents are difficult to convert into a question with definite answer. Therefore, we randomly choose a model from the model pool to serve as a news filter, prompting to filter valid websites by providing some few-shot ex- amples. Next, from the model pool. The model is asked to raise several questions based on the website content. It is encouraged to raise questions unable to be answered only by text. As for the question with an image, the model is asked to briefly describe the image content, and we will later the description search in Bing and obtain the first result image. Finally, apply the quality check of the generated questions and their corresponding images either by human or model from the model pool.
provide the text contents and screenshots of the valid websites to model
2. Rerank data generation. We provide the generated requery to the search engine and retrieve websites for rerank. Again, we provide the question, the question image content, and the answer to model and ask to categorize each website into valid, unsure or invalid.
3. Summarization data generation. We randomly choose one website from the websites marked as valid in the last step and obtain its full content.
19

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation Results of Different Complexity in MMSEARCH. We categorize all the
questions into three levels of complexity.
Model
All
End-to-ent
Requery
Rerank
Summarization
Easy
Midklle
Hard
Easy
Middle
Hand
Hesy
Middle
Hard
Hasy
Middle
Hard Pasy
Middle
Hard
slorre--ource LMMs
with MMSEARCH-
BEGGII ENGINE
Clande 3.5 Sonnet (Anthropic, 2024)
56.2
53.6
48.9
53.0
49.8
45.0
49.3
37.6
35.0
80.3
84.5
75.3
59.8
59.9
58.3
GPT-4V (OpenAI, 2023c)
55.3
56.4
52.7
52.9
41.4
76.5
(OpenAt,
63.9
62.3
59.0
Open-sountee L.Ms with
MMSEARCI-
Muntis (liang et al., 2024b)
22.8
18.0
13.0
19.4
15,9
10.0
28.5
12.6
14.5
40.5
43.1
34.6
28.2
11.5
12.9
IntenLLMXXC2 (Zhang al., 202
26.5
19.3
18.1
28.5
20.0
16.7
21.3
26.0
25.0
0.0
0.0
0.0
39.5
30.3
42.7
M-
25.6
18.1
21.2
27.5
18.3
21.4
20.2
24.6
20.9
0.0
0.0
0.0
39.5
31.5
41.4
BLaVA-NeXT--jttrllee (Li al., 2024g)
33.2
25.6
23.3
28.5
20.2
16.9
31.0
22.1
22.7
61.4
51.7
49.4
41.5
41.6
45.0
mPPlu--wiB al.,
18.4
mPlag- Ow
37.5
32.8
29.0
31.4
26.4
21.5
38.6
26.9
26.0
76.5
77.6
67.9
44.0
39.6
48.3
IntemVL2 (Chon et 20246)
38.7
30.9
30.5
35,0
28.3
26.8
39.5
25.3
27.9
57.6
38.5
37.0
47.4
46.2
52.8
IntemVLLunee
38.0
30,0
32.0
34.6
24.5
28.2
38.8
24.5
26.5
53.8
59.8
45.1
16.7
43.9
50.2
Idelics3 (1 aurrengoo et al., 2024)
43.1
30.5
31.0
38.0
21.3
23.5
40.9
23.3
23.3
76.5
83.3
690
48.9
50.0
53.1
Idelics3anti
43.2
31.2
28.4
38,4
24.1
23.0
37.6
20.3
17.6
76.5
74.7
64.2
48.4
46.6
38.4
LLaVA-Onc (Li al., 2024b)
28.9
53.6
Qwen2- Vieen-VVLusse. (Qwen Team, 2024)
44.9
41.2
38.9
46.5
34,0
73.5
74.7
56.8
59.6
LLaVA-OneVision (728)
52.2
48.0
49.0
47.6
41.6
44.0
51.0
39.6
33.2
79.5
85.1
83.3
60.3
63.7
60.6
Qwen2-Vi Luuss (728)
53.7
534
50.2
50.7
48.6
46.9
52.3
40.0
37.1
72.3
82.2
77.8
58.5
67.0
53.6
Notably, human check is must for the requery data generation process. There should be at least one valid website to guarantee the effectiveness of the generated requery. Only after human check of this step, the quality of rerank and summarization data generation is assured.
C ADDITIONAL DATASET DETAILS
We manually annotate the complexity of the data based on the difficulty of the three steps in MMSSearch-Engine:
1. Requery difficulty. This concerns the complexity of transforming the original question into an effective search query. Complex cases arise when the question references image content, requiring the LMM to first analyze the visual information and then synthesize with the text question into a coherent search query. For instance, when user asks about landmark shown in an image, the LMM must first identify the landmark through image search and then incorporate this information into text query about the landmark 's specific attributes.
2. Rerank difficulty. This dimension evaluates the challenge of identifying and prioritizing relevant search results. The difficulty primarily scales with the information scarcity. If there are only very limited websites containing useful information, it is more difficult to successfully retrieve and choose the website.
3. Summarization difficulty. This aspect involves both information synthesis and multi- modal reasoning challenges. In cases requiring synthesis, answers cannot be derived from single source sentence the LMM must integrate information scattered across different parts the website. For example, comparing event frequencies across locations (like con- counts between cities) requires gathering and analyzing distributed data. Additionally, some questions demand analysis of both textual and visual website content, sometimes necessitating comparison with input images with images in the website.
Based on these criteria, we have categorized all questions into three difficulty levels, with the fol- lowing distribution: hard (28%), medium (27.7%), and easy (44.3%). We provide the evaluation results grouped by the difficulty levels in Table 4.
20

---

Published as a conference paper at ICLR 2025
D FUTURE DIRECTION
Our proposed MMMSFARCH-ENGING can be enhanced through interactive user feedback loops. When the model produces an incorrect answer, users can identify the specific step where the er- ror occurred and prompt the model to reconsider its reasoning. This iterative process allows for guided model refinement until accurate results are achieved. We conducted preliminary experiments on GPT-4o with this approach on three test cases. The results demonstrated that the LMM suc- cessfully interpreted user feedback and appropriately adjusted its responses based the provided guidance. The model's ability to understand the user input and revise its reasoning suggests signifi- cant potential for improving task accuracy.
E ADDITIONAL EXPERIMENTS AND ANALYSIS
E. SCALING TEST-TIME COMPUTE VS SCALING MODEL SIZE
Recent works such OpenAl ol (OpenAI, 2024a) and Li et al. (2024d) have highlighted the critical role of scaling test-time computation in enhancing model performance. Our end-to-ens task, which requires multiple Internet interactions, presents an opportunity investigate the potential of scaling test-time computation compared to scaling model size. To explore this, we conduct experiments using LLaVA- One Vision- (Li et al., 2024b), focusing on scaling test-time computation, and com- pare against LLaVA-One VAn--Aeeiiiion-72 scaling in model size, which aims to provide insights into the relative benefits of increased inference computation versus increased model parameters.
For scaling up the test-time computation, we adopt multi-modal search strategy similar to best-of-N solu- tion, where 'N' denotes 25 in our settings. Specifically, for LLaVA-Onee Vision-n- we first prompt the model to generate requery 5 times, from which we selected the one with the highest requery score Sreq. This requery is then used retrieve brief results from 8 websites from a search engine. The model is again prompted 5 times select the most informative website. After removing duplicates from the selected websites. we extract the full website content from the remaining ones and prompt the model to answer 5 times, obtaining 25 end-to-end outputs in total. We compute the Fl score for each answer against the ground truth and take the maximum as the model's end-to-end score for the query. Table 5 shows that LLa Vis (TTC) achieves the score of 55.2% in the end-to- end task, significantly enhancing the original score of 29.6%, which surpasses LLaVA- VA-OOee Vision- 72B's 44.9% and GPT-4V's 52.1%. 1%. This result reveals the substantial potential of scaling test-time computation, validating the effectiveness of this technique as introduced by OpenAl ol. Our find- ings provide valuable insights for future research in this domain, suggesting that increased inference computation may offer comparable or superior performance improvements to increased model size not only in math and code tasks, but also in multimodal search tasks.
Table 5: Scaling Test-Time Compute vs Scaling Model Size. 'TTC' and S.2e denote Test- Time Computation and the score of end-to-end task.
Model
Inference Cost
S.50
LLAVA OV:7B
29.6
LLaVA-OV-78 (TTC)
25
55.2
LLaVA-OV-72B
44.9
E.2 DEFINITION OF ERRORS IN THE REQUERY AND SUMMARIZATION TASKS
Five types of requery error:
Lacking Specifility, where the model fails to include all the specific information in the requery and therefore leads to sub-optimal search results. For example, the query is asking the release date of Vision Pro in China. However, the model omits the condition of China and directly asks about the release date of Vision Pro.
Inefficient Query, where the model does not consider the real scenario and the requery is inefficient for the search engine to find the answer. For example, the query is asking whether the Van Gogh's Sunflowers and Antoni Clavé's Grand Collage are both oil paint- ings. Clearly, it is a commonsense that Van Gogh's 's Sunflowers is an oil painting and Antoni Clavé's s Grand Collage is much less well-known. An efficient query should be asking about the images of Antoni Clavé's Grand Collage and further determine it is also an oil paint- ing by directly looking it. However, the model directly asks the original query to the
21

---

Published as a conference paper at ICLR 2025
search engine. There is very little chance that exact question has ever been raised so probably this requery will bring very little helpful information.
Excluding Image Seurch Results, where the model totally ignores the information in the screenshot of the image search results and therefore lacks important specific information in the requery. For example, the query is 'When did this football player obtain the gold medal?" and provides an image of the player. The model is supposed to find out the player's name by viewing the image search result and raise a requery like '[PLAYER NAMEJ ob- tained the gold medal time'. However, the model fails incorporate the player's name in the requery and definitely the retrieved websites will not include any helpful information.
No Change, where the model just uses the question as the query input to the search engine.
Irrelevant where the model either matches wrong information from the image search result mistakenly understands the query and outputs irrelevant requery.
Five types of summarization error:
Text Reasoning Error, where the model fails to extract the answer from the website textual information.
Image-text Aggregation Error, where obtaining the answer needs combining the informa- tion from both images and texts. The model fails to do so.
Image reasoning Error, where the model fails to extract the answer from the image, and the answer can only be obtained from the image.
Hallucination (Huang et al., 2023), where the model provides an unfaithful answer that cannot be grounded in the given content.
Informal, the output format does not follow the prompt specifications, the same error type in the end-to-end task.
F ADDITIONAL EXPERIMENTAL DETAILS
Rationality of the weight in final score. Considering the sequential nature of the tasks, the reason for our weighting scheme includes two complementary perspectives:
1. The importance of each task due to the cascaded nature is already reflected in the end-to-end score. Detailedly, although it is easy discern that the upstreamed task is more important, it is difficult to assign precise weight to each of them. So we do not manually assign the weights but directly focus mainly on the end-to-end score, which implicitly considers their cascaded nature.
2.
The individual task weights serve as complementary metrics rather than indicators of rel- ative importance. Relying solely on end-to-end evaluation, while comprehensive, may obscure the performance characteristics of individual components and hinder targeted im- provements. We therefore maintain independent evaluation of each task, with the weight distribution designed to balance the prominence of the end-to-end metric against the component-level assessments. This dual evaluation strategy enables both system-level op- timization and component-specific refinements.
More Implementation Details All our experiments of open-source models are conducted without any fine-tume tine-tuning on search data or tasks. As for the prompts, the requery prompt contains 3 examples to better guide LMMs to output valid requery. While prompts for other tasks are all in a zero- shot setting. We prompt the LMM to output as few words as possible for a better match with the ground truth. We employ the metric introduced in Section 2.3. Besides, recruit eight qualified college students and ask them to solve the problems in MMSEARCH independently, following the same pipeline of MMSEARCH -ENGINE. This score serves baseline for human performance. We conduct all experiments on NVIDIA 1 GPUs.
The input image dimensions for the webpage's 's top section screenshot set to x 1024 pixels. For the full-page screenshot, we set the initial webpage width to 512 pixels, although the actual width of small portion of webpages may vary due to its layout settings. Furthermore, considering that
22

---

Published as a conference paper at ICLR 2025
full- page screenshot can be extremely lengthy, directly inputting it single image into an LLM would result in excessive downsizing, making the content too vague for accurate identification. To address this, we segmented the full-page screenshot into multiple images, starting from the top, with each segment measuring 512 pixels in height. Because of the context length limitations of LMMs, the maximum number of full-page screenshot segments is therefore restricted to 10.
Full-page Screenshot Slimming. For the full-page screenshot, we compute the Sobel gradi- ents (Kanopoulos al., 1988) detect the edges and generate gradient magnitude image. We iteratively remove the areas with gradients below a threshold, which represent the blank areas. This approach, shown in Fig. 9, effectively reduces image size while maintaining the document content.
Origiaa gugg
Viayltaal
Skaand Scvenakat
Figure 9: Illustration of the Screenshot Slim Process. We leverage Sobel gradients (Kanopoulos et al., 1988) to identify blank areas and remove them. After slimming, the screenshot size is largely reduced without any information loss.
Model Sources. For different LMMs, we select their latest models with size around 7B for evalua- tion to fully reveal their multimodal search proficiency. Table 6 presents the release time and model sources of LMMs used in MMSEARCH.
Table 6: The Release Time and Model Source of LMMs Used in MMSEARCH.
Release Time
Model
Source
GPT-4V (OpenAI, 2023c)
2023-09
https://plat apenai..om
GPT-4o (OpenAl, 2024b)
2024-05
httpps:/pplatffmmoooeee www.:/oooommm.ee.....mmmm con/
https://www.atthee com/news/
Claude 3.5 Sonnet (Anthropic. 2024)
2024-06
claude-3-S-sonne:
httpe://gttaub.ccm//meeet
Internl ML..M-XXCS M-X22.5 (Zhang al., 2024a)
2024-07
Intern1Mt-XCoposee
tttps://ilger-aallab.ooo Mantis/
Mantis (liang et al., 2024b)
2024-05
sttpet//glthub..cmm/LLaa// LLaVA-NeXI
LaaVANeeXTHmnerleeee (li et al., 2024c)
2024-06
InternVI.2 (Chen et al., 2024d)
2024-07
InteraVL
http:///itthb..ccm//X11110g mP1.06-0w
mPlug-Ow13 (Ye al., 2024)
2024-08
Idefics3 (Laurençon et al., 2024)
2024-08
fiagringF2cceM///deeficccss ana3
LLaVA-OneVisionn (Li et al., 2024b)
2024-08
2024-08-05-11avv--omoi
Qwen2-VL (Qwen Team, 2024)
2024-08
Qwen2-V1.
Input Prompts of LMM for Response Generation. We showcase the input prompts of LMM for the three tasks respectively in Table 7-9, We adopt two types of prompts for queries with an image and without images. For query with image, we specifically require the LMM to leverage the image search result to solve the task.
23

---

Published as a conference paper at ICLR 2025
Table Input Prompt of LMMs for Requery. We adopt two different prompts for the query with image input and without image input. We leverage a 3-shot prompt to guide the LMM to generate reasonable requery.
Question
Prompt
You are helpful assistant. am giving you a question, which cannot be solved without external knowledge. Assume you have access to a text-only search engine (e.g.. google). Please raise a query to the search engine search for what is useful for you answer the ques- tion correctly. Your query needs to consider the attribute of the query search engine. Here are examples:
Question: Did Zheng Xiuwen wear knee pad in the women's 's sin- gles tennis final in 2024 Paris Olympies? Query to the search engine: Images of Zheng Xiuwen in the women's 's singles tennis final in 2024 Paris Olympics
Query without image
Question: When will Apple release iPhone16? Query to the search engine: iPhone 16 release date
Question: Who will sing a French song at the Olympic Games closing ceremony? Query to the search engine: Singers at the Olympic Games closing ceremony, French song.
Question: {question). Query to the search engine (do not involve any explanation):
You are a helpful assistant. am giving you question including an image, which cannot be solved without external knowledge. Assume you have access search engine (e-g-, google). Please raise a query to the search engine to search for what is useful for you answer the question correctly. You need to consider the characteristics of asking questions to search engines when formulating your questions. You are also provided with the search result of the image in the question. You should leverage the image search result raise the text query. Here are examples:
Question: Did Zheng Xiuwen wear knee pad in the women's 's sin- gles tennis final in 2024 Paris Olympics? Query to the search engine: Images of Zheng Xiuwen in the women's singles tennis final in 2024 Paris Olympics
Query with image
Question: When will Apple release iPhone16? Query to the search engine: iPhone 16 release date
Question: Who will sing a French song at the Olympic Games closing ceremony? Query to the search engine: Singers at the Olympic Games closing ceremony, French song Question: {query image) (question). The image search result is: {image search result) Query to the search engine (do not involve any explanation):
24

---

Published as a conference paper at ICLR 2025
Table Input Prompt of LMMs for Rerank. We adopt two different prompts for the query with image input and without image input.
Question
Prompt
You are helpful assistant. am giving you question and 8 website information related to the question (including the screenshot, snippet and title). You should now read the screenshots, snippets and titles. Select website that is the most helpful for you to answer the question. Once you select it, the detailed content of them will be provided to help you correctly answer the question. The question is {question]. The website informations is cerebsite_infommaate
Query without image
You should directly output website' index that can help you most, and enclose the website in angle brackets. The output format should be: <Website Index >. An example of the output is: <Website >. Your answer:
You are a helpful assistant. am giving you a question includ- ing image. You provided with the search result of the im- age in the question. And you are provided with 8 website infor- mation related to the question (including the screenshot, snippet, and title). You should now read the screenshots, snippets and ti- tles of these websites. Select website that the most helpful for to answer the question. Once you select it, the detailed content of them will be provided to help you correctly answer the question. The question is (query siragee (guestion). The image search result is sanageesearch.resuull The website information [website_in formation).
Query with image
You should directly output website's index that can help you most, and enclose the website in angle brackets. The output format should be: <Website Index >. An example the output is: <Website >. Your answer:
25

---

Published as a conference paper at ICLR 2025
Table 9: Input Prompt of LMMs for Summarization. We adopt two different prompts for the query with image input and without image input.
Question
Prompt
You are helpful assistant. am giving you question and 1 website information related to the question. Please follow these guidelines when formulating your answer: 1. the question contains false premise assumption, answer "invalid question". 2. When answer- ing questions about dates, use the yyyy-mm-dd format. Answer the question with few words you can.
Query without image
You should now read the information the website and answer the question. The website information is twebsite in formation). The question (queslion). Please directly output the answer without any explanation:
You are helpful assistant. am giving you question including an image. You are provided with the search result of the image in the question. And you are provided with website information related to the question. Please follow these guidelines when formulating your answer: 1. the question contains false premise or assumption, answer "invalid question". 2. When answering questions about dates, use the yyyy-mm-de format. 3. Answer the question with as few words as you can.
Query with image
You should now read the information of the website and answer the question. The website information is (website_inf (ornciiioo ) The image search result is simage.search_rsutle {query somage {question). Please directly output the answer with- out any explanation:
The question is
26

---

Published as a conference paper at ICLR 2025
G MORE DATA DETAILS
G.1 DATA EXAMPLE OF 4 EVALUATION TASKS
Search Resalt
Query Information
Imoge
Question:
which day of 2014 the laner project ansouseed cinceled?
Task1 Requery
Query Information
LMM Requery:
Requery Annotation: NASA Ends VIPER Project
VIPER was announced to be cancelled
Task2 Rerank
Query Information
Brief Result
<Wibidle
Thile KASA Pegist Coed
ixpleaiion Temmnn boyl- wes BoEE ta (Vodetto tevangutia Englzrake mata Eaas goodi scaee
NASA
Sutppet
"Thboo haaaer
thegee, Cvemacchiov
ras. Lee Copac Wastina TON Juy 21,2
Tale hiddfcaneda VIPER sover teanchews
<Widuite Io
<Wobbltee to:
Tale Nass relooe wn ower projet Vinci olen
selppue:
rows Cradle
Selppet NASA anded ViSGi prajes, vilck
in en tur would lave canceed antiic of
to loam di spmens whele her revert dormes due thisss
Rerank Annotation:
Valid: [< Website 1>, <Website 2>, <Website 3>]
Website 2>
Unsure: [<Website 4>, <Website
Invalid: [=Website 62, <Website 7>,
<Website 8>]]
Task3 Summarization
Query Information
Full Website Content
Content:
Full-page Sereenshot
wodld wwnchtrncccooooo soanbod the maspell or mom. erngerddpenr dotes ane ovn everen NASA aood Jide oent werd devieance of
Auseledu In nos Aillwene eoplen lnso
kevil
and fomie anll leoea wit fime Ale healing
cacac otfecat and houdkay 304kk/ Cluck Opeeric
(Dye cancd
Clic Fuscheds
Dyen wind
"s Ladee dai wesl folvvc de none to mase cader CLPS onter natt 522 ailiie... MADA and Gorlls eut epocialia mady fe
planre the now Direle akilit deiiiyyi prolisd. Kerm
dang and arove
NASA caniinad VIPER cancal' calliaa Jod
atremm modimase KiSs scmee Nove Denockente, ieetee..ai and lakes
LMM Answer: July 17
Answer Annotation:
07-17
Task4 End-to-ent to-enn-
Query Information
LMM Answer: Sep
Answer Annotation:
07-17
Figure 10: Example Input, LMM Output, and Ground Truth for Four Evaluation Tasks. The color-coding of each module corresponds to Fig. 4. Task1 Requery (green), Task2 Rerank (purple), Task3 Summarization (blue), and Task4 End-to-end (yellow) are shown. Image best viewed in color.
27

---

Published as a conference paper at ICLR 2025
G.2
SUBFIELD DEFINITION
News area encompasses a vast spectrum of information, ranging from everyday events to engag- ing entertainment content and specialized fields such scientific discoveries and financial analysis. This comprehensive coverage serves as a rigorous assessment of the model's 's ability to process in- formation in diverse domains. We divide this expansive area into eight distinct subfields:
Data concerning traditional athletic competitions, team performances, player statistics, and sporting events. This includes scores, league standings, player trans- fers, and analysis various professional sports across different leagues and countries.
Traditional Sports:
e-Sports:
Information about competitive video gaming, including tournament results, player rankings, and league information. This covers various game titles, team formations, streaming viewership statisties, and tournament information.
Technology:
Information about technological innovations, gadgets, software develop- ments, and tech industry news. This includes product launches, software updates, cyberse- curity issues, and artificial intelligence advancements.
Paper:
Content related to academic papers, research publications, and scholarly articles in various artificial intelligence fields. The queries include method explanation, figure under- standing, and experiment settings.
Entertainment: Data about movies, music, television, celebrities, and other forms of pop- ular entertainment. also includes data concerning video games.
Information on financial markets, economic indicators, business news, and mon- etary policies. This covers stock prices, company earnings reports, company financial state- ments, and regulatory news regarding finance.
Finance:
General News: Broad coverage of various news topics not specific to any particular sub- field. This includes a mix of local and global events, human interest stories, lifestyle arti- cles, climate news, and general interest content that doesn't fit neatly into other specialized news subfields.
Data related to misinformation or incorrect assumptions in the query. This subfield focuses on fact-checking capabilities. All the answers to the queries of this subfield are 'invalid question'.
False Premise:
Knowledge area represents broad subfields of information and data related to general knowledge across various disciplines. This area concentrates on rare knowledge that most LMMs fail to answer. We categorize this area into five subfields:
Architecture:
Information about building design, architectural styles, building informa- tion, and construction projects. This includes city landmarks, the comparison of architec- tural styles, and vulti-view architecture matchings.
Data concerning visual arts, drawings, sculptures, badges, and other forms of creative expression. This covers artwork details, artist profiles, artwork history, and artwork style comparisons.
Arts:
Fashion:
Content related to clothing trends, fashion brands, and designer collections. This includes retail price, clothing style, release date, and brand information.
Information about celestial objects, space exploration, astronomical phenom- ena, and related research. This covers observational data from telescopes and image results from space missions. The questions focus on the background information of these celestial objects presented in the query image.
Astronomy:
Anime: Data about Japanese animation, including series storylines and character informa- tion. This encompasses character background, character appearance, voice actor informa- tion, and chapter information.
Auto: Content related to automobiles, including vehicle specifications, industry trends, and automotive technology. This covers new models, performance test results, coefficients of cars, and release date.
28

---

Published as a conference paper at ICLR 2025
H QUALITATIVE EXAMPLES
Question:
the LPL 2024 Summer season, how many teams were Group Ascend?
Answer:
Area: News
Subfield:
X Sports
Round1 Requery
GPT-4o: LPL 2024 Summer season Group Ascend teams count
Qwen2-VI:: LPL 2024 Summer season Group Ascend teams
L.LVA--OneVislon: LPL 2024 Summer season Group Ascend teams
Round2 Rerank
GPT-4o Brief Results:
-Websth Es-
Webblte &
Tilk: U1% 2014 Samner- Lanpapaliaa Logus Legancs
Tille crep Stare LLS. 2024 Saneer achakl rouda Soreaess Snipper
Fxpore
helppet
tepeedd targns
Lageak logge The can uoue oas rujer change of ereetus intoductionn Miaaras butth
whollh le Lat Nows ths and padictuns Manche Townn Touaa woro3
sayen
2000
fid for delli read rolaes and then is the diggl mound
Stage Plyil Regiona Fiulk Greap Singi Plyoff Placa 3
Websitee 20
<Website 6
like Ltt. Summur 2004 da Leagu Lepance Wild Suippet: The L.P, Sanne 2004 apto he soodd ipta the
Tiile 2014 I- vaucs Wilipoda
Salppet: The 2004 Ln0 scave in 20 sageing saves Legas Logand League...
acaues laua raatman
Si wegghi Spotoon mhove Siee and linjing Bilkel Carning
connd oilliieonn minhh [s] Auper soad the shan pic 32h0 guilic
slag duuppio
Wolld chaepoocclg lpus
Lagesk:
Tälk: U1% Sanner ilaecmnto Lappapataal of
WWdblt I-
Tile: I0 caniig DDvadd Garin LLF Sunmeer Pacurate-
Wilbllt To.
Sulppet:
Leponds Lmgao
Snipper Wook
Lagnaki kogge The nanine wose sut rupe change if
emeeuaa tacluding intoductionn Miaddas Duthh odee
2004 Sthovik Was darvea Joos 10% adrase trisp Aacaad alqgdd
Suamet by spela Then Wi nas Gorning (19) (191.2000
thus Diuresce
dubble sumns then single
Weehllte
Thtee Rummer Conrage Conttaness
Takee largan Lagaak Semmar Spurndadss Suappet 2034 Semoer Sela thear varty Samver will Haemann teams divitdd imo der grage
Snippet: Laggs L'gaeed Largas
chanoo emseus Ficorann sumaree cinemaie grocsigs
Loaah
poup
GPT-4o Rerank: <Website 6>
Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VI. LL- 7B (Qwen Team, 2024), and LLaVA-One (Li et al., 2024b) the end-to-end task.
Figure 11:
29

---

Published as a conference paper at ICLR 2025
Round2 Rerank
Qwen2-VL. LL. Brief Results:
Losperada Laague
<Webslte &
Eapora Wal Ttadoe
Solypet The 1850 torgooa legeak the oage et th
Tiile Lit Suamnce Saaves 3304 uad Canco Leganaa Snigpet. View the sate ber Lnt Sameet Scave roull, snkinngg.. hod player playa
uncnd neolth your harl per fisiined Legge
aavdactioo "Toatan Dradl" rulooadd
intan of prepp siinn meseares poaa apieet eack
Website
Webbitte
Tikle Sa 2804 Greay Sugii La
Salppet: The LP,. Sanceo 3824
Snlppet: Liqu raje Revarped player nan pagas wit wholsk, cou cus, ackivenn es rwor. Unvellied taclano anti vevsone Andrral
Leage of Lapeedd Tho cagas sasss Say upis dan with nusch rand schdilos booadaats hand
Titee LPC.2804 Sance Summne Thoonane Leaguz Legendi Experts Wil
<Wbsitee
Thkee 1PL.20 Suru Samese PacerensChaeplee
Webbitt
Salgpet: The hate Leganna Leyrak Pee oagu iis
Seric
you per
Snlgpet Inmmanes: L.t L1542004 beuen Summee Phoceett-
Open shannie
situiaas inclddog rnvdactton "haata Doal" rilo and octan proap dagp etace luatu cach otae whew Thetay teens.
Thel Played Clange Ramka By Rod (Quary Parulleka) Tip.
<ebbitte
<Webite d-
2014
Tilk:
Conaagel Gostlaanes
Legesh k Will
Suigpet Lega cgrab
Sulppet: The Greap Stage Daw Saoe will beld on 21, Reall
Taan guooo zau sad pasta cauas
disggo Lagah logas Tax
Jokcs dun, ID0L UP One tain Boes Faul 2 and will
evereen Pooeees seruine will detern as g grop
muth group preap will competie groes
Qwen2-VL Rerank: Website
LLava-OneVision Brief Results:
Webbite
<Webiite So
LPt, Sumrec 2004 squipdda la Lergee of legeads Wis Suigpet:
Title LiI Surnee Sesees 5014 Can Legiess
Suigpet: the
Sanme Scare
Spik will buld Chisa
changean iPt sannet 2024 Taammiinn Top
sharge suton, Skoodie Rej claning witk afficm
polkiie and how Sigu 1log (e). mane Senoon
dandica 3257 Avcrag kill: ganso Steeiat
Title LrS 3024 Summer- Louperncia League Legands Espora
<Webhitee 2n:
Tille Stage/ Ln,, 2014 Surencr vched de ros bo nonoss
-Website fo.
Salppet oeggn Layuak fisiicad
rwitth
Salppel:
Masch roula, calandas, VDDss
Lepande hogae. bhe summer muace aa expor chumges LPU. stuuaans bainn alwel shoe thrertie
rilooadd
cheiieee for New thes yoeki fies Myyies howes Yours
Thoers 2024 Summer
3534-0-00 2000-00
for in four dubki aubits bubiis and tick in teooingg roed
Sugo Hayuf feg Forks Grang Sloge Poyulf Phaie
<Webiite
Lit. 3044 Summer Creap slage- Liquppak Leugac
Tiik:
<Webelte To
Willadda
sici cSsT Twrn qplned asiil park eataa quelty grops Faul comoii bakes camm camdy: ohe
Nalgpet:
The Ganap Stage now i
huldion May
Snlapet 2024 Las, grepp wowe ont the form ceroiiooo greep bars will limeers camoi in additend mass (6)Aa usall cuurlius China camberr soo Spring, Scace
Pve The
ongaing scaues
Fool and soedid with grop the gaaa compria tno Paul sam, visk
groegy tavevvee...
<Website 4-
Thile 1Mt 2014 Sucoeer Ramble Stage Aeyiped laggle
Legead
Tikle N0ht Conrage Greadoeees
Suigpel Formi Kanbie Stage Stagle mand subiis wahi
coood op noetth you of Cheals pertodinal langano Logana Poore vaves deages ter the giay hus hi01 ot.
""ucrrch (Tligs Grosp Tap taans get undid invite
canatt add place soow pat saaded
heddooed plants.
I.LaVAOOneVssion Rerank: Website 4>
Figure 12: Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL. 7B (Qwen Team, 2024), and LLaVA-One Vision--B (Li et al., 2024b) in the end-to-end task.
30

---

Published as a conference paper at ICLR 2025
Round3 Summarization
GPT-4o
Full-page Screenshot:
Content:
for Spring 2024 will qualify for the 2024 Mid-Scason Invitational 's group runner-up will qualify for the 2024 Mid-Season Invitational play-in stage I [5] Summer split format is split into three slages The first stage will consist of double round-robin group stage where all seventeen ( 17 teams are split into three groups of 4 and a group of 5 This first stage will implement the Fearless Drall drafting format seen China LDL developmental league where Champions from previous games in the top four nirvana group teams advance the sigjeelliinaatonn. kinng-o-thhe hill tournament The bottom two ascend group teams and the top four nirvana group teams will have to contest additional match I As per usual the champion for Summer 2024 will qualify for the 2024 World Championship as China's number one sead Spring I edit Regular Senson edit ] Spring 2024 is the recond-conrecutive season where Top seed only lost game during the entirety of the regular season. Morcower, Hilibiii Garinng the first organization
while the
Wikipedia fice encyclopedia Sports scason 2024 scason League LPL Sport League of Legends Duration 22 January 20 April Spring TBD Summer Number
2024 LPL scason Add links
17 Spring Split Champions Biibbii Gaming Rumner-up Top Esports Season MVP Zhuo knight Ding Bilibili Gaming Summer Split LPL scasons - 2023 2025
ongoing season Pro League LPL ) Chinese professional esports league for the video game League of Legends. :
2024 seuson is
League of Legends
Similar to its previous yearly splits, the 2024 I.Pl. season will he divided into two splits : Spring Summer The Spring Split began
on April 2024 for the Grand Finals. Meanwwile, the Summer Split will hegin in the latter part of 2024 following the conclusion of the Mid-Scnson Invitational MSI 2024. Biliblii Gaming was crowned 2024 Spring Champions after defeating Top Esports 3-1 in the Grand Finals rematch the Upper Bracket Finals This was Bilibili Garinn first organizational tille under the
Jamuary
Bilibili Gaming in the LPL. ] The previously defeated Top Esports in the Upper Bracket Finals 3-2 [2] and Top Esports defeating defending MSI champions JD Gaming 3-1 in the Lower Bracket Finals [3] Bilbbli Gaming
qualify for MSI 2024 Group Stage alongside the champions the LCS, LLK and LEC, while Top Esports will be pooled with the nunner-up
qualify for the play-in toumament per the MSI miles Formar edir
The Spring Split will have seventeen
thus far to back-to-back 15-1 records
different splits. BLG went in the Summer Split and finished Spring 2024 with another 15-1 record. 7 [7]18 Pos Team Pid W L. PCT GW +/- Qualification Bilibili Gaming 15 0,935
Upper Semifinals
Top Esports 16 13 0.813 29 8 +21 JD Gaming 16 13 0.813 27 12 +15 Advance to Round 3 Ninjas in Pyjamas
FunPlus Phocnix
0.688
0.625
Gamiing will return to MSI 2024 as one of the four group-stage qualifying teams Top Esports through the Play-In stage where they are alongside the 2023 League of Legends World Champions Tl 10 Bracket edit Round Round Round Upper somifinals Upper final Final Bilibili Gaming FunPlus Phoenix 5 Ninjas in Pyjumas 1. Ninjas in Pyjamas Ninjas in Pyjamas 3 Bilibili Gaming 3 My God
defcal means their qualification
2024
WE
April 2024
All teams qualified for MSI 2024 League of Legends ONE Esports Retrieved April 2024, v League of Legends Pro League Teams Anyone Legend Bilibili Gaming Edward Gaming FumPlus Phoenix Invictus Gaming JD Gaming LGD Gaming
www.oneespons.g2 88 , 20 April
My Rare Atom Royal Never Give Up Team WE Top Esports TT Gaming Ultra Prime Weibo Gaming Seasons 2020 2021 2022 2023 2024 c 2024 professional League of Legends competition Interaatoonal Mid-Scuson Invitational Esports World Cup World Championstip Regional LCS LEC LCK LPL PCS Retrioved from htlps /lnn wikipeeiiiia.ooo org/liiorg/iiiin x.php ? titl--2024 sPasson & 0l370-123745000
Esports Ninjas in Pyjamas
Categories League of Legends Pro League sensons 2024 in esports Hidden categories : Use dmy dates from April 2024 Articles with short description Short description matches Wikidata.
GPT--0o Summarize: 2
Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL- 7B (Qwen Team, 2024), and LLaVA-OneVisiion77 (Li et al., 2024b) in the end-to-end task.
Figure 13:
31

---

Published as a conference paper at ICLR 2025
Round3 Summarization
Qwen2-VI.
Full-page Screenshot:
Content:
twelth year of China 's professional League Legends league The summer changes of LPL stracture including introduction of Fearless Draft rules and return of group stages Seventeen
major
play against each other first in four double round rohins, and then two single round robins The lop ton teams advanec to the playofl Overview Format Two groups Group Ascead having teams Group Nirvana having Single Round Robin within group Matches ure best of three The Top 7 teams in Group
Chinese Toumaments, Comperitions, Premier Events 2024 Summer LPI.. | 2202 Season Sign in odit History Talk ( European Pro League Scason vs. Septeeberr 2024 12:00:00
10000 LIVE PRM 2nd Div 2025 Spring Promoction vs. September 2024 16:00:00 +0000 LIVE
arrYY Summer League 2024 vs September 2024 16:00:00 +0000 LIVE + arMY Summer League 2024 vs September 2024 17:00:00 +0000 LIVE arMY Summer League 2024
Google Calendar Social Media Links Contents 1 Overview 1.1 Format Participants 2.1
Group Ascend 2.2 Group Nirvana Results 4 Match Schedule 5 VODs & Match Links 6 Individual
Awards 6.1 Man of Match Standings 6.2 Weekly Award 6.3 Scasen Awards Media
Strcams 7.2 Broadcast Taleut 7.2.1 English 7.2.2 Mandarin 7.2.2.1 Stage Studio Hosts 7.2.2.2
Casters 7.3 Additional Content 7.4 Vieworship Statisties 7.5 Amnouscement 8 Home Venues 9 References The 2024 League of Legends Pto League ( LPL) the second split the Ascend qualify for Playofiss The Bottom 2 teams in Group Ascend and Top teams in Group Nirvana qualify for Play-in Stage The Bottom teams Group Nirvana are qualified playoffs Show Tichreaker Rules Hide Ticbreaker Rukes two Ieams have the number of series won. ties will be broken by Game Differential Head to Head record Participants [] Show Rosters Hide Rosters Rosters By Game Player Chart Group Ascend
Studio Hosts 7.2.2.3 Casters 7.2.2.4
Anyone Legend Ale Croco Shanks Hope Kael Tabe Qingsi Hilibili Gaming Bin Xun Wei Knight Elk
Invictus Gaming YSKM glfs neny Ahn Vampire Rashomon Oh My God.
Hery Tianzhnn Angel
Starry poged noname Geitang Kare Atom Xiaoxu Xiaohao Vial a Asam Jwei Deceit JMZ RNG Juice Geju XBY Tangyuan Xuz huandeng Iwandy Ming Teacherna May
Wayward
Yanxiang FoFo Able Mark chengz Zoem TT Gaming HOYA Beichuan ucal Ixn Feather AFei Ulma Qingtian Häcker Yuckai Doggo Niket Xinobai Yuzhang Results []] Group Legend Round 4 Seed Round Seed Round 2 Seed Round Seed Play-In Seed Team Series September 2024 18: 18:10:00 +4000 LIVE arrYY Summer Leggue 2024 vs 2 September 2024 19:00:00 +0000 LIVE NACL 2025 Spring Promotion
18:15:00
Summer League
September
September 2024 22:00:00 -0000 LIVE Emerald September 2024 23:00:00 -0000 LIVE NACL 2025 Spring Promation vs September 2024 00:00:00 +0000 LIVE LCK CL 2024 Summer Playoffs vs Playoffs Split 2 Replacements Split Playoffs Split 3 Grand Finals 2024 Season Overview Placements Split Replacements Split Playoffs Split Replacements 2 Split 2 Playoffs Split Finals Overview Spring Scason Spring Playolfs Summer Placements Summer Season Summer Playoffs Regional Finals Championship Points Overview Team Rosters Picks & Bans Player
League Special Edition
Scorcboards Runes Match History Champion
LPL 2024 Summer Scason Toumament Information Organizer TJ Spocts Location & Dates Region CN China Event Type Offline Country China Surt Date 2024-07-05
Date 2024-07-31 Broadusst Streams Twitch Tencent Full List Schedule Spoiler-Free Schedale Export
Emerald League Special Edition vs September 2024 23:00:00 +0000 LIVE NACL 2024
Spring Prornotion TBD TBD September 2024 00:00:00 +0000 LIVE + LCK CL 2024 Summer Playol's vs TBD September 2024 08:00:00 -0000 LIVE European Pto League Season vs September 2024 10:00:00 +0000 LIVE European Pro League Season 3
September 2024 12:00:00 +0000 LIVE PRM 2nd Div 2025 Spring Promotion TBD vs TBD September 2024 16:00:00 +0000 LIVE L.FL. 2025 Promotion
Qwen2-VL. Summarize: 9
Figure 14: Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL- 7B (Qwen Team, 2024), and LLaVA-One Visionn-- (Li et al., 2024b) in the end-to-end task.
32

---

Published as a conference paper at ICLR 2025
Round3 Summarization
LLaVA-OneVistee
Full-page Screenshot:
Content:
register and log to pages You can also join our leagueoflegends contributor community our Discord Liquipedia app match pages updated Liquipedia app match pages overhauled Download Liguipedia app match pages got completely rovamped with game data slandings VODs and more Download the the latest version on iOS or Android and read our updatic blog here LPL 2024 Summer Rumble Stage From Liquipedia League Legends Wiki LPL 20245ummer Group "Ascend" (High Group
Android
2.2 Rumble Stage Group Nirvana "( (Lw Group 2.3 Detuiled Results 2.3.1 Week 2.3.2 Week 2 2.3.3 Week 2.3.4 Week 2,3.5 Week 5 Referenees Format [ odit Rumble Stage Single round rohin within ench group All matches Bo3 using Standard Draft. For the Group Ascend High Group J: Top teams get seoded the semifinals
get seaded into the fourth round playoffs 5th and 6th place teams seeded into the d LPL & LDL/LSPL 2024 LPL Group Ascend ( High Group odit Group Ascand Woek Week Week Week 3 Week 4 Week 5 1. Anyone's 's Logead 1-0 2-0 -2 LNG Esports 1-0 2-0 +2 Weiho Gaming
and 4th place
Top Esports 0-0 0-0 7. FunPlas Phoenix 0-1 0-2 -2 JD Gaming 0-1 0-2 NiNjsa
Bilbbili Gaming
LGD Gaming
Pyjamas 0-2
Esports 6-1 4. A
Page Legend
2024 17:00 CST Match Page 36:33 22:46 MVPs Bin knight Bans Game 1 Game 2 JDG TES Gaming 0.2 Top Esports July 31 2024
41 08 26:12 MVPs : 369 369 Rans Game Game 2. Match was held in Reijing. References legendsindcox.hp tubb-LPL/2224SummmerrF
| Retrieved from litips :
//liquipedia netrleguuof Stage & oldid-S26570
Hidden categories Pages reading from original match table Pages storing into original table Pages
original match table Do you want help Just
LNG LGD Gaming 2 LNG Esports July 27. 2024 15:10 CST Match Puge 32:40 30:05 30:02 MVPs: haichao Hang. Weiwei Bans Game Game 2 Game Match
was held in Suzhou EDG UP EDward Gaming 2 Ultra Prime July 27, 2024
17:10 CST Match Page 29:19 34:56 MVPs: Cryin. Jieji Bans Game I Game FPX
0.2 TES FunPPuus Phoenix 0 2 Top Esports July 27, 2024 19:30 CST Match Page
33:43 24:02 MVPs:
19:10 CST Motch Page 26:56 27:10 MVPs: : Zika Weiwei Hons Game 1 Game
Match
0 EDward Gaming 0 2 Team WE July 8 2024 17:10 CST Match Page 27:19 29:46 Yanxiang...
Stunuzhin. Week edit Week
8,
Game
Match was hold Xi'an LGD
LGD Gaming 0 Anyone 's Legend July 8 2024 19:10 CST Match Page 28:09
28:36 MVPs 2024 17:10 CST Match Page Spring Summer Regional Finals Championship Points LDL Seeding Stage Stuge Stage Season
Show the full list 2023 LPL. Spring Summer Regional Finals
Demacia Cup Legend Cup Click
link on the right Championship Points Stage Stage 2 Stage Others LPI. Spring Summer Regional Finals Championship Points LDL Spring Summer Others Demacia Cup LCC 202] LPL Spring Summer Regional Finals Championslipp Points LDL Spring Summer Others Demacia LCC 2020 LPL Spring Summer Regional Finals Championship LNG Esports July 2024
Cup LCC
33:22 41:37 MVPs Xiaohao Xisoxu Buns Game Game 2 JDG WBG JD Gaming 0.2 Weibo Gaming July 6 2024 19:10 CST Match Page 43:33 33:25 MVPs Tazzan Breathe Bans Game Game 2 Match
hold in Beijing-
7
2024 UP 0 2 IG Ultra Prime 0 Invictus Gaming July 7, 2024 17:10 CST Match
31:35 39:02 MVPs: Giff,, Hans Game Game
Ninjas
Pyjamas 0 July 27. 2024 LGD 1.2
LLaVA-OneVisia Summarize: 8
Figure 15:
Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VLL VL- 7B (Qwen Team, 2024), and LLaVA-One Vision--7 (Li et al., 2024b) in the end-to-end task.
33

---

Published as a conference paper at ICLR 2025
Image:
Image Search Result:
Question:
As college student, how much should I pay for going this building?
111
Answer: 210 yen
Area: Knowlodge
Subfield:
Architecture
Round1 Requery
GPT-4o: Akita Museum of Modern Art college student admission fee
Qwen2-VL: Akita Museum of Modern Art ticket price
I.LaA--Oneisiion Akiha Museum of Modern Art
Round2 Rerank
GPT-4o Brief Results:
<Welolte
Webslte
Tiste: Lacaloon adiiaaioo MoMas
Tike Masses Arn Alra Alia Tooal
Snippet Mavar Moden Art beations, born, and abntnse Capker The Muscm Madenn An Acawvaii
Snippet The Akit Mascn al inied de
Akia tepattnoun As 10 ceimic be with panto visk Thasoaanuss
Expemiie Usstped
collage
Gurder Staden Full
axilding
coruidy aho wit fire arand world heuus lars callectin ofecct Fijit...
incraeeall orders 517 Childrra under Marcha Free... Adadedoo Bus MaldA supporad MoMA
Title:
Thile
of Mador
the filliiing puoses for reteeption bervition rloss plea volo the pau vichis
Disces and Melki
poin by Look.
Suigpel: Locatd in Akia Caty the Akia Nascas Modan An mesdeetd muccm chorcaing maker hitrry
cary
NYC
flom dhre Recludibe Caner Afler viiing sight, nake you way Maldi explaos cu flom mooeemoed
call ant sereeil in the memes furtiing whwaris of variaa rediines and tio poikss ar anchodogical wribuus mosee
5,00
<Weloite
<Webiite
Tatle: diru Mucrar of Medien WWkww City Trerics Promutio Organtatiin
Tikee MokdA Seenberkip Accree Moaran dedea Snippet: Addilcaa munnhand
dissern
Seippet 1994, Akita Maucur Mador Art oponad De Firaaado
cites and maaeem on availble Call; (i0 990 8861 lasn rase Manbbrrkie
piples cocalles
tonatteblle Manthr hewffis and pro to srage. Unviniied fire zdmaiakoo Teack uotued worting
Akit inche by the Auck and vesat's partod Advisioo frah Fulibekks (Ditttnnes
Tille
Tilk:
Biqe Nbwchake...
Snigpet the Alco Moscen Mtades Art Aeropene doos Ayei Akis Pamaeas Vellgo. Qpeiin
Sndpptt La Unliettd free derii MuNtA vitwor warting ikke fiee. Mantemserty vy gallng ralis, eubist prosknus Aatibbal manaandka criass end sibes ave dille Call pood s961 lerm Menisoolie
New You bolidoyeed mukiamane days tawarys Adbrswen en orecodeesii Enrance for rngincel specil
by
GPT-4o Rerank:
Website 5
Figure 16: Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL- 7B (Qwen Team, 2024), and LLaVA-OOnVViiion- (Li et al., 2024b) the end-to-end task.
34

---

Published as a conference paper at ICLR 2025
Round2 Rerank
Qwen2-VL Lee Brief Results:
Webblit P.:
<Weboite Sr:
Titl: Akia Mescas of- Musdie Aecon, thop & Prioe Sulppet This mxssen was
Tille Alc Stacamool An Adka Traech jap.rgaaicccc... oves Suapper situ
Tada linelar repaaal recaallinng in and mmgaas omaade voll crcaks A4b- end criste space.
rodam
rusea oir Alin City dody racsenad
exciad Ando tadla The munee hhit large adfictie
by thgar bougdaan Fogjit augullyy perkct
is Leamnd
Wooter
Webbite Zh:
<Webelts &
Title akia Mescca of Moder Art hhaooee City louaio Orpasiation
Tilke Alll Stocee of Mooees fot kkgele sulgpet Akka stavwer Madkes
vilage Aiaa raaando oillct and dielige cuadiza worlo of an lves bi madeen incading Horenvesk parntig
Aua Seenioa Kinda jajanaat opend Voluas, Pedactur Apas 1994 anchecss as inpurtus oflectiooo Alstireage
painel
-Webelte
Maoras of Meders Aun Opening loors spocts
Tiite Alro Nourau Modeen Places Eipe lnooe Snippet admoosmtt lrce Whe aud
Indlddn Auau Mhscam Mader lacks tnkirn and recevatioon, openiag louns trat mation rupi.
estuilia csea vitte sauses ivcalien optio some (ust (IA) Immokkaee slAl)
egaifiiance
and vine praciial infermaors tros Alci Masccn of Nodan Art tucc at supewa kabli price thoogh top oue
< <Weboite 4h:
<Weboite Rh:
Maonee Medees (Papkane lapacs
Tatle Akru Vouua Trase Eiqux
Suappet Moura Au Alcs City, Japan, poo0 tsdak werh Taggs furn tngit ober resernl atte booggal andricn ladeo Anda thc exiiag balliing
Akia Prcledenc lipemmnly
eatric er roteod artous adkiirs the Hirracee Un bny lokk
officin
Qwen2-VL. Rerank: Websiit 3>
LLaVA-One Visiion Brief Results:
WWebolte
Tale
Seipget
Beroue Ireedwase
Seippet Alts Miseen MfMieee Art See at ts do. See dlkag de Alias Masen Mader 19 revow full viw. photew (2l)) biga edlust imprree wher. ve shon anproed dhe leiiiinng fop weys expeiine scarlyy
Unevrd inkerntion Autios Time Akankk
town cit
oretivi viick. and poo sie te maiio exooniinar Bodio and villarioo modess comtopppoo soows an
Salpper hoakk
sovder Colletionn Wailin Moeemage
Selppet: National Musca Medaam, Ant 2000 yeu instratiis ctncsira mexty as idca. That in ccansad al ligely....
The Muscau Nodtzn Ramooatto and goodd Scclidi Reaflo lihesrnc Cundder.
oWebsite
Weebolte To:
Title: The Campleas Grida the Mussen Moders Now York City TipSany
Timle Mocic Viats The Advik Triddway Snappet Tw Mucra coddddm MuldA ces Nite York City Clty raan rxpeediv raavirs Darignadlty Anp urenict rasinn i bora svirary merting edittiniad parmaner art calectoon maio maz plas shoad eussube thee.
Seipget New York Citys Maxoue of Moder Art a NokiA, was diood
teega bei
expaaded by 890 sqaae fien 10 paroet incuasd SUMAA bases aaadlegiall adaacmmrnas emepgeg kippoayy personent oxllation...
nowe the 200
Solypet tocance MaMt
Shore
Thaninge and Chriseaa bilen oilonioo gelleii ciew apees oaildy wexdes prac Muusee osing Our
takal beewth deren Rindrol pasko glos in Midaoon Me the close oegler plans wakk Yokko Treguetes
LLaVA-One Vision Rerank: <ebssite 1>
Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-Va VL- 7B (Qwen Team, 2024), and LLaVA-One Visionnnn (Li et al., 2024b) in the end-to-end task.
Figure 17:
35

---

Published as a conference paper at ICLR 2025
Round3 Summarization
GPT-40
Full-page Sereenshot:
Content:
Akita Akita City Culture Akita Akila City Culture May 9, 2016 2 min read Akita Muscum of Home to one largstt Fujiza Tsaguharu collections By Victoria Kamila Community writer The Akita Museum of Art is listed as the # thing to do in Akita by Trip/ddvisor, At just 310 yen entrance ( 2100 college students ), and with free public gallery on first floor, the art display here certainly worth visit. This muscum also one the fow around the world
Art" comes with result indicating the muscum penuancally closed. . This is in reference to the museum 's old location. , which was closed in 2013. The new museum is in fact, still open. Things to Do in Akitu 6 Share Facebnols Share on ( Twitter Copy link shire Was this article helpful Help us improve the site Give Feedback By Vietoria Kamila Community writer Following Follow +6 Infermation 1---2 Nakadori, Akita City- Akita, Japan (Directions) ) +81 18 853
8686 satt---mccmm-o-ariii Book your trip Hotels Trip Find neaby hotel Explore some of the best nearby hotels and deals for the perfect place to stay in Japan. Your stay dates: Search with Tripadvisor your trip now When do you want travel My dutes are flexible Get started Explore nearhy Hotel Richmond Akita Eki-mae By Rosson Lam Kagum Restaurant Hotel Richmond By Bonson Lam Akita Design Hub and Craft Center By Bonson Lam Kubota Castlc By Hitoshi Kawai Discover Akila City Top Articles Recommended Popular I Freshness Burger Okinawa Fair Food 2 Guided Tour
Foujita 's he became a patron of his. The free gallery located on the first floor displays temporary exhibitions of art and other projects or collaborations by local artists. Photography is perminted any the galleries Outside of the museum there also public market selling kitchenwares, teapels, those lucky waving cats, and those weird bear looking forest spirit things you outside houses and temples all the time in Japan. 's also important mention that quick Google search for Akila Muscum of known as Leonard Foujita- name his life in France and eventually acquirin French citizenship. He did exhibit foodness for in particular, amtisicallly expressing
adapted spending considrrable amount
twentyy meter long mural titled Annual Events in Akita ". The mural depicts life in Akita throughout the four sensons and is given permannet status this gallety Also on display separate galleries is the work of Hirano Masakichi, who was bom in Akita, Having developed an affection for art, particularly regards
claim about Google saying it closed. Wikipedia, Google maps, and the official website all say it open ( in English and in Jupanese) ). Pertaps there few outdated websites that list as closed. I used go the old cafe looks nice. Reply Show all 0 replics I comment in total
when
Akita 2011-2012. The new one with poolside
houses largo collection work by Fujita Tsaguharu. arguably one the most famous Western pointers of Japan. The Tsugubaru gallery is located up the frce-standing staincase which illuminated by a large triangular skylight just past cafe the second floor You do need to pay an entrance fee view the gallery (300 yen though the cafe and the museum shop are free enter. The cafe itself features great view accented by an infinity pool nanning along the glass-walled windows. Internationally Fujita is also for Foreign Visitors to Experience Traditional Japanese Performing Arts and Learn About the History of Ginza News Sponsored Introducing Nomad cSIM Planning Sakura-tei The World's 's Largest Okenomiyaki Restaurant Tokyo Kamakura :A Memorable Summer Getaway Karagawa 6 A Foot Bath Puth In Kaminoyuma Yamagata Fairytale Experiences Nukumori- no-mori Shiznoka A Comprehensive Guide Nomad 's Japan eSIM Planning 9 Tokyo Ginza Noh Week Tokyo Sponsored 10 Fürin, the Symbol of Japanese Summer Culture A Guide Japanese Visas Planning 2 Guide to Bringing Medicines Into Japan Planning
GPT--4o Summarize: 210 yen
Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL. VL- Figure 18:
7B (Qwen Team, 2024), and LLaVA-OneVisioon-- (Li et al., 2024b) in the end-to-end task.
36

---

Hotel Published as a conference paper at ICLR 2025
Round3 Summarization
Qwen2-VL
Full-page Sereenshot:
Content:
Reviews US S 66.00 View Top Restaurant Picks Near Akita Museum of Modern Art 1. Bar Pasaporte
Thrvef Guile Akite
Address 204-1 Aza Takuboshita Fuke Otsutsumi Distance 445m Bar Pasaporte No reviews yet Other
Cuisine View 2. Kuidoraku Price : 8.00 Address 7-2 Ekimaecho, Yokote Akita 013-0036 Distance: 2.28km Kuidoraku No reviews yet Bans Bistros US $ 8.00 View 3 Komauen Yokoteten Price S 5.00 Address: It is 28-1, Sanmaibashi in Vlego, Yokote-shi. Akita character Distance: 2.03kmm Korakuen Yokotcton No reviews yet Other Chinesc Cuisine US S
5.00 View Ganso Kamiya Yakisoba Restaurant Address Nakano-117-66 Oyashinmachi. Yokote
Akila 013-0051, Japan Distance: I 17km Ganso Kamiya Yakisoba Restaurant No reviows yet Fast Food
View Verified Reviews Akita Museum Modem Art 38 5 SEAA 18 F B SE-AEEAA AA
(H Ht 16 1 A 15 di Se 18 E In 1915,
fi &8
fo M Jody Tan +
Hh. ME RE
1 J. *
36 BABEE
Aoh ti E
T E (t HE 18814 8.5
K 67 3 th. E Emma FRI. HI A M.
xinomouti 12121. 13
E ARATORAT
RR 130
# B9 Also Popular With
Visitors to Akita Muscum of Modern Ast 1. Sendai Umino-Mori Aquariam Price : 14.30 Discount::
2.04 Recommended sightsceing time: :: 4-5 bours Address: Japan, T983-0013
and free everything related to the ninjas were very fun Edo Wonderland Nikko Edooann 4.5 43 Reviews No.3 Best Things Muscum of Modern Art Opening Times Akita Muscum of Modern Art Address Suggested Visit Duration for Akita Museum Modern Featured Accommndation Near Akita Museum of Modem Plaza Annex Yokote 2. Yokote Central Hotel 3. Quad Inn Yokoto 4. Yokote Plaza Hotel Top Restsurant Museum Modern
Nikko Theme Parks From US 37.43 View Contents Akita
Picks
Bar Pasaporte
Hotels Flights Trins Curs Car Rentals Airport Trunsfers App Customer Support USD Search Bookings Sign in Register Travel the World With Trip, com Travel Guide for Akita Museum of Modemn September Updated 2024) Akita Museum of Modem Art Opening Times Year round : 9:30-17:00
Akita Museum Modern Ant Address 62-46, , Akasaka Tomigazawa Inside Akita Furusato Mura
Yokotc, Akila Prefecture Sugaosted Visit Duration for Akiia Muscum of Modern Art 1-2 hours
Featured Accommodation Near Akita Museum of Moder Art 1. Hotel Plaza Annex Yokate Address
2. Kuidoraku Korakuen Yokoteten 4. Ganso Kamiya Yakisohn Restaurant Verified Reviews Alcit
Museum of Modern Ari Also Popular With Visitoes to Akita Museum of Modemn Art Sendai Umino- Mori Aquarium 2 Tugann- han Neputa
Village 3
Suntopia World Edo Wonderland Nikko
Edomun Coneents Akina Museum of Modern Art Opening Times Akitn Museum of Modem Art Address Suggested Visit Duration for Akita Muscum Moden Featured Accommodation Near Akila Museum of Modern Art 1. Hotel Plaza Annex Yokote 2. Yokote Central Hotel 3. Quad Inn Yokote 4 Pasapocte Kuidoraku 3. Korakuen Yokoteten 4. Ganso Kamiya Yakisoba Restaurant Verified Reviews of Akita Museum Modem
Yokoli Pluza Hotel Top Restaurant Picks Near Akita Muscum Modom Art
Popular Visitoes to Akita Museum Modern
Sendai Umino-
Mori Aquarium 2, TTggan-han Nepota mara Village 3 Suntopia World 4 Edo Wonderfand Nikko Edomura Popular Travelogues Bangkok Travelogue Manils Travelogue Tokyo Travelogue Taipei Travelogue Hong Kong Travelogue Scoul Trivelogue Kuala Lumpur Tnveloguee oss Angeles Travelogue| Shanphai Traellguee
S 14.33 View 2. Tsugau- han Neputa mura Village Price: $3.38 Address: Japan. T336-8332 Aomori, Hirosaki, Kamenokonnahhi 6 Distance : 0.63 mi Tsugaru-han Neputa mura Village No reviews yet From US 3.38 View Suntopin World Price : 8.85 Recommended sightseeing time 0.5-1 day Address: 1-1 Kubo Agano Nigata 959-2212 Japan Distance 5.06 mi What travelers say M5h Very suitable for fumily outings playground. Femrs wheel, pirate ship fun. Suntopia World 5 Reviews Amusement
Reviews -5 $3.00 View 2. Yokote Central Hotel Address: Heiwacho 9-10 Price: 5 50.00 Distance: 2.91km Yokote 50.00 View 3. Quad Inn Yokote Address: Sekihatn-522 52-1 Yanıda Price: 5 58.00 Distunce: : 1.9km Quad Inn Yokote 4.1/S /5 14 Reviews US 58.00 View 4, Yokote
Ekimacchou Price: 53.00 Distance: 2.23km Hotel
Anxx Yokotc
Central Hotel 3.7 Reviews 45
Plaza Hotel Address: Ekimaecho Price: S 66.00 Distunce 2.27km Yokote Plaza Hotel
Qwen2-VL. Summarize: US $ 66.00
Figure 19:
Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL- 7B (Qwen Team, 2024), and LLaVA-OneVision-11 (Li et al., 2024b) in the end-to-end task.
37

---

Published as a conference paper at ICLR 2025
Round3 Summarization
LLaVA-OneVision
Full-page Sereenshot:
Content:
Museum of Modern 23 Edit links Coordinates 39"17'33' 34'51' / 39,19250'N 140,54750'E/ 39,29250 140.54750 From Wikipediaa the free encyclopedia Building in Prefecture Japan Museum Modem 8 @ ft WhiR Wikimedia OpenStrectMap General information Address Tomigusawa 62- Akin
Alita
Country Japan Coordinates 39 7"33 140'32511" / N9,29250'N 140,54750"E 39.292550 140.54750 Opened 20 April 1994 Technical details Floor count Floor arca 11,166.5 m Design and construction Architcet Yamashita Sokkei Architecture fimm
Town or city
www.yaamssiiusekksi...
hee//orks/is//mmmalll
opened
a.tml Website homepage
Akita Museum of Modem Art BLR SE 9F Akita Kcaritsu Kindai Bijutsukan ) opened in Yokote Akita Prefecture Japan in 1994 and important collection Akiu ranga [1][2 [
Prederdimm
[
Commons has media related to Alita Museum of Moder Art Akita Prefectural Museum of Cultural Propertios Japan paintings Akita Yöga art References edit 1 th. 2 (Summary [ in Japanese Akita Prefecture Archived from the original
inporian:
Aero range
October 2013 Retrieved October
AE #F12 (HFF) Alii
Soc 3.50 (ect
Museum of Modem
( in Japanese Yokote City. Retrieved 19 October 2013
Akiia Prefectant
External links edit [ in Japanese ) Akita Museum Moden Art hide Authority control databases International VIAF National United States Japan Academics CINii People ISIL JP-2000539 This amtille related to museum in Japan is stub. You can help Wikipedia by expanding
paimting (Ver))
Bil thur phurmon or wore
Retrieved from hipps /ion wikippiiipdia.oo orp/ winde p.p
tite-Akiia Muscum of Mod em Art & oldid-1199347400 Categories:
Akita museums galleries in
Prefecture Prefectural museums
Mupeamn Uodenn Ard in cepeeeet
miseums and galleries estublished in 1994 1994 establishments in Japan Important Cultural Properties Akita
Japanese museum stubs Hiddeen categories Pages using gadget WikiMimiArlas CSI uses Japanese- Japanese-tangung langung scripe ja CSI Japanese- langage sourees ( ja ) dmy dates from November 2019 Articles
Bn Aspesesd Avdi Numnen
short description Short description different from Wikidata Infobox mapframe without OSM relation ID Wikidata Courdinates on Wikidata Articles containing Japanese-lange language text Commons category link is Wikiddata Articles with lapanese-lmmgunee sources All stub articles Pages using the Kartographer extension
L.LVA-OOeVian Vision Summarize: free
Figure 20: Response and middle results comparison of GPT-4o (OpenAI, 2024b), Qwen2-VL VL- 7B (Qwen Team, 2024), and LLaVA-One OneVision-7B (Li et al., 2024b) in the end-to-end task.
38


Reviews:
[{"review_id": "3IVoWL9Gas", "reviewer": "Reviewer_7DLE", "summary": "This work describes a new pipeline to empower LMMs with multimodal search capabilities called MMSearch-Engine, and introduce MMSearch, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs which includes 300 high-quality instances across 14 subfields. The study also points out LMMs' challenges in multimodal search and suggests scaling test-time computation could improve AI search engines.", "questions": "* Due to the sequential nature of requery, rerank, and summarization, errors in each step can lead to subsequent errors. How to ensure the reasonableness of the weight of each score in formula 1?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* MMSearch-Engine is a well-designed three steps pipeline that empower LLMs with multimodal search capabilities. \n* MMSearch is a fair evaluation benchmark because its data is all dated later than the model's knowledge base update time.\n* Case analysis is comprehensive.", "weaknesses": "* The small amount of data may lead to a certain degree of randomness.\n* When constructing the data pipeline, human annotators perform multiple requery operations in cases where no website is classified as valid. Only one requery operation may not fully demonstrate the LMMs' capabilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VIw56uTct4", "reviewer": "Reviewer_3MNQ", "summary": "This paper is about benchmarking LLMs as multimodal search engines. The authors propose:\n1) A pipeline that enables this type of multimodal search, where images are translated into textual queries before querying a search engine and summarizing the results.\n2) A small, but high-quality dataset as benchmark. They do their best to ensure that there is no information leakage between the LLM's training data and the benchmark, by collecting long-tail knowledge data, and very recent news.\nThey present results using commercial search engines, closed-source, and open-source LLMS, and show that there is much room for improvement, analysing the sources of error along the way.", "questions": "Please, refer to the \"Weaknesses\" section.\n\nI have another question for the authors that I do not consider a weakness, just interesting: In line 409 you state \"Any-resolution input only provides slight or no improvement.\". Have you tried to corrupt a clean image with differend kinds of corruptions (e.g., occlusions, blur, gaussian noise, etc.), and see how it impacts the search results?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1) The paper is very well-written, with excellent presentation and clarity. In my view, the scope is very well-defined, and there is no redundant information.\n2) I particularly like that the authors try to create a dataset that LLMs have not seen before.\n3) There is a very large appendix with interesting additional experiments and qualitative analysis.", "weaknesses": "1) The way the authors construct the dataset is not future-proof. We have to assume that every piece of news that comes out is immediately ingested, at the very least by large corporations with commercial search engines. In that sense, the benchmark will be obsolete in a couple of months. I would love to see a piece of discussion that proposes a way to make the data collection and annotation pipeline future-proof.\n2) The benchmark dataset, even though high quality, is rather small. Maybe I am missing something, but are there any indications that its coverage over the problem space is sufficient?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ttbB5vPTao", "reviewer": "Reviewer_mwVo", "summary": "This paper introduces MMSearch, a benchmark that evaluates the capacity of large multimodal models (LMMs) as AI-driven search engines for handling complex multimodal queries (text and image). The study proposes MMSearch-Engine, a tailored pipeline enabling LMMs to address multimodal search tasks, divided into stages: requery, rerank, and summarization. Experimental results show that models like GPT-4o, integrated into this pipeline, perform better than current commercial systems, such as Perplexity Pro, in end-to-end multimodal search tasks. The paper presents detailed error analyses, offering insights into the limitations of LMMs in search-oriented subtasks, especially in requery and rerank.", "questions": "(1) In future versions, could the pipeline include user feedback to improve task accuracy iteratively?\n\n(2) How general and adaptable is the approach to data from specific or specialized fields, such as medicine or law?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) This paper constructs a novel benchmark MMSearch. It fills an important gap in multimodal AI evaluation by focusing on search capabilities and interaction with multimodal data. It offers a unique benchmark that pushes beyond standard image-text alignment tasks.\n\n(2) This paper proposes an effective multimodal retrieval pipeline MMSearch-Engine, which consists of requery, rerank, and summarization. Experiments have shown that this pipeline can effectively improve the performance of the model.\n\n(3) The experimental analysis is comprehensive and provides good reference conclusions. The error analysis and comparison between commercial and open-source models are valuable for understanding current model limitations, providing useful insights for improving multimodal search models.\n\n(4) This paper uses a weighted score of four scores to evaluate the effect of the model, which not only focuses on the correctness of the model's results, but also focuses on the correctness of its process, which can provide a more comprehensive and effective understanding of the model's performance.", "weaknesses": "(1) Although MMSearch covers a wide range of news and knowledge domains, the total number is only 300 instances. Such a number and scale are not enough to fully reflect the generality of the model, etc. The author may need to further expand the scale of the dataset in the future.\n\n(2) Lack of task complexity hierarchy. The complexity of current MMSearch tasks is relatively consistent, lacking a hierarchy of tasks from simple to complex. In the future, tasks of different difficulty levels can be designed to better measure the performance of the model when dealing with tasks of increasing complexity.\n\n(3) There are still limitations in the validation of the model's adaptability. The adaptability of this method in different fields and application scenarios is still unclear, especially on data from special or professional fields (such as medicine or law). Introducing datasets from these fields can more comprehensively evaluate the versatility and adaptability of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dD8cpXWGpc", "reviewer": "Reviewer_gUra", "summary": "The authors present MMSEARCH-ENGINE, a multimodal AI search engine pipeline, and MMSEARCH, a comprehensive evaluation benchmark to assess the multimodal search performance of LMMs.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper introduces the first mm RAG pipeline (at least based on the related work introduced) and the first mm search evaluation.\n2. The methods for building the MM RAG pipeline and evaluation are solid and with comprehensive statistics.\n3. The methodology statement is easy to follow.\n4. Comprehensive experiments.\n5. Step-wise evaluation of the RAG pipeline is cool.", "weaknesses": "1. The evaluation data creation pipeline is not efficient, which may result in difficulties to keep the evaluation dynamic.\n2. Evaluation data only comprises 300 entries, which is far from comprehensive. Generally, a benchmark should have 1k+ samples to be robust.\n3. The authors didn't exhibit the step-wise scores for the proposed RAG pipeline, which harms the result soundness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)