ZooProbe: A Data Engine for Evaluating, Exploring, and Evolving Large-scale Training Data for Multimodal LLMs
Authors: Yi-Kai Zhang
Shiyin Lu
Qing-Guo Chen
De-Chuan Zhan
Han-Jia Ye
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
Training Data Engine
Deep Learning
Primary area: foundation or frontier models, including LLMs
TL;DR: ZooProbe: a new training data engine for Multimodal LLMs that features the Evaluating-Exploring-Evolving (E3) loop.

Abstract: Multimodal Large Language Models (MLLMs) are thriving through continuous fine-tuning by LLMs. Driven by the law that "scale is everything", MLLMs expand their training sets during version iterations. In this paper, we propose a large-scale training data engine built around an evaluating-exploring-evolving (E3) loop. Evaluating the data provides insights into its characteristics. Exploring quality rules helps identify which data enhances training. Together, these processes facilitate the systematic evolution of new, high-quality data. With the E3 loop, we introduce ZooProbe, an efficient data engine for MLLMs. First, the problem of data expansion is formalized as a tree of sampling and growth. ZooProbe introduces a small-scale model *zoo* to obtain comprehensive evaluations for child datasets. From multiple perspectives, visual, textual, and multimodal models cover over 50 dimensions of intrinsic and meta attributes, such as object and topic distribution, and higher-level properties, like annotation quality and scene complexity. ZooProbe constructs based on A$^\star$ search, modeling the heuristic function as a quality estimate from data evaluation results. It dynamically explores the rule of data quality based on the model state of the *probe* datasets. Additionally, it evolves new targeted data with identified high-quality rules. We also develop an extra heuristic quality ranker with the data utilized and discarded during the expansion. Our experiments show that ZooProbe significantly breaks the scaling law in multimodal instruction fine-tuning at scales of 260$k$ and below.
ZooProbe generates high-quality data that accelerates MLLM training and enhances performance, automating the evolution of large-scale training data.

Published as a conference paper at ICLR 2025
ZoOPROBE: A DATA ENGINE FOR EVALUATING, EXPLLRING, AND EVOLVING LARGE-SCALE TRAINING DATA FOR MULTIMODAL LLMS
Yi-Kai Zhang 1.233+ Shiyin Lu
Qing-Guo Chen De-Chuan Zhan 1.2
Han-Jia Ye1.21
1 School of Artificial Intelligence, Nanjing University
National Key Laboratory for Novel Software Technology, Nanjing University
3 Alibaba International Digital Commerce
ABSTRACT
Multimodal Large Language Models (MLLMS) are thriving through continuous fine-tuning by LLMs. Driven by the law that "scale is everything". MLLMs expand their training sets during version iterations. In this paper, we propose a large-scale training data engine built around an exppttatinn-ellloinggeeee (E3) loop. Evaluating the data provides insights into its characteristics. Explor- ing quality rules helps identify which data enhances training. Together, these processes facilitate the systematic evolution of new, high-quality data. With the E3 loop, we introduce ZooPROBE, an efficient data engine for MLLMs. First, the problem of data expansion is formalized as tree of sampling and growth. ZooPROBE introduces a small-scale model zoo to obtain comprehensive evalu- ations for child datasets. From multiple perspectives, visual, textual, and multi- modal models cover over 50 dimensions of intrinsic and meta attributes, such as object and topic distribution, and higher-level properties, like annotation quality and scene complexity. ZoOPROBE constructs based on A search, modeling the heuristic function as a quality estimate from data evaluation results. It dynamically explores the rule of data quality based on the model state of the probe datasets. Additionally, it evolves new targeted data with identified high-quality rules. We also develop extra heuristic quality ranker with the data utilized and discarded during the expansion. Our experiments show that ZooPROBE significantly breaks the scaling law in multimodal instruction fine-tumi fine-tuning at scales of 260k and below. ZOOPROBE generates high-quality data that accelerates MLLM training and en- hances performance, automating the evolution of large-scale training data.
INTRODUCTION
The rapid advancements in Large Language Models (LLMs) (Du et al. 2022
Chiang et al. 2023
Diang al.] 2023) OpenAl| 2022 Touvron et al. 2023a b) have given rise Multimodal LLMs
(ML.MMs) which build upon the foundations of LLMs and are continuously trained to process
visual instructions Chen et al. 2023: Zhu et al. 2024 Liu et al., 2023] 2024 Yao et al., 2024
b). This growth often aligns with the "scale is everything" principle (Kaplan ct al. 2020 Hoffmann et al. 2022), suggesting that generalization ability may follow a power-law relationship with the size of training data. Ongoing efforts focus expanding visual fine-tuning
OpenAl 2023 2024a
datasets into developing the next generation of MLLMS (Bai et al. 2023 Wang et al. 2024).
Rich multimodal information in training data, especially the knowledge hierarchy and interaction between visual prompts and text, significantly boosts learning efficiency and performance. The data quality - reflecting the "garbage in, garbage out" greatly influences learning speed and the depth of acquired knowledge (Power et al. [2022] Xiie et al., 2023bb Lee et al.] [2024] McKinzie et al. 2024 [L] ct al. 2024c). In the context of MLLM training data, researchers frequently introduce supplementary MLLM to label answers and even to generate visual instructions from images freely (Chen et al.
WWrk done during the internship at Alibaba International Digital Commerce. Corresponding author, email: yehj @ laada.nju.edu..

---

Published as a conference paper at ICLR 2025
2024a: Adler et al., 2024). Such a pipeline can limit knowledge guidance and result in biased training set that relies on the preferences of data- Mata-generating models, such as most are captioning tasks with shallow understanding. Moreover, the multimodal QAs significantly large as the Cambrian (Tong et al. 2024) consists of 10m entries. Therefore, effectively sampling from diverse existing data and generating new, targeted data is essential for developing next-generation datasets.
Firstly, a thorough evaluation can reveal the characteristics of the new training data and eliminate poor-quality entries. Existing data evaluation methods often focus on natural characteristies, such as text length and perplexity (Cao et al.) 2023] Xu ct al. 2024), or rely on fixed and uniform difficulty metries from the additional scorer (Lian et al.) 2023; Chen al.) 2024c). After obtaining the evalu- ation results of the data, it is crucial to explore a rule to determine what distribution, when added to the existing dataset, can contribute to forming a high-quality training set.
One of the high-qualite selection rule expressions is the distribution ratio of dimensions (Wen et al., 2023 Fan et al. [024] [u et al., 2024a] Tong et 2004). Previous methods (Xue et al., 2021 Rae et al. 2021) relied on heuristic functions for threshold control, often using random or manually set values. In real-world applications. the definition of data quality changes with existing distributions and the model status on the available dataset (Zhou et al. 2023 Chen et al., 2024d). In addition to sampling from real-world sources, we can generate and evolve visual instructions based on identi- fied rules. This data pipeline yields more efficient outputs, enhancing MLLMs to discover visual information, particularly regarding spatial, temporal, and deeper contextual reasoning.
These key elements construct cyclical engine of comprehensive data evaluation, exploratory rule discovery, and targeted iterative evolution. Such an efficient data engine can fundamentally lower computing expenses and minimize the carbon footprint. In this paper, we introduce ZooPROBE, (E3) loop.
a data engine for MLLM training that implements an evaluating-exploring--ooo ZooPROBE utilizes a small-scale model zoo evaluate data, explore for high-quality data rules, and heuristically expand the training dataset. It also continuously evolves new data through rule- based guidance, constructing a seamless cycle from assessment to search to generation. ZoOPROBE produces high-quality data for MILM version iterations, beating the limitations of scaling laws. Ini- tially, ZooPROBE derives data description vectors from a compact collection of visual, textual, and multimodal models, tackling over 50 intrinsic and meta dimensions. The intrinsic factors include attributes such as resolution, language, instruction length, visual objects, and topics. On the other hand, the meta factors examined from a higher perspective, including annotation reliability, con- textual relevance, external knowledge dependency, and multimodal interactivity. These dimensions offer valuable insights into the dataset's nature and properties without incurring feature preservation costs or requiring excessive backward computation (Gao et al. 202T
ZooPRoBE then develops an ba-ba search framework that includes the reward-so-far function and the based on model states, ZooPROBE explores the optimal distribution through perturbations of the initial diverse one. It then establishes a connection between data descriptions and training quality, iterating to estimate quality scores of the data expansion. Besides sampling labeled visual instruc- tions from diverse wild collections, ZoOPROBE integrates the identified high-quality rules and rep- resentative examples into an additional MILLM, enabling valuable data's efficient generation and evolution. Based on the proposed E3 loop. ZooPROBE eliminates the cost of manual prompt engi- neering. Its meta-description vectors offer inherent interpretability. facilitating the discovery of new dimensions in scaling laws. Moreover, we specifically learn an MLLM to assess the data quality and validate its effectiveness. The experimental results show that ZooPROBE can surpass the MLLM scaling law from 10k to 260k. In summary, our contributions are as follows:
training-qualinn estimates. Starting from a dynamic definition of high-data-quality
We develop the E3 loop of the data engine using a heuristic search framework to expand datasets, evaluate diverse data dimensions, and adaptively explore quality while evolving new data.
We present ZOOPROBE, which features a diverse model zoo for a thorough evaluation, establishes probe set for targeted rule selection, and facilitates the evolution of missing distribution.
During ZooPRoBE's expansion, we develop MLLM-Scorer to assess data quality. Experiments demonstrate that ZOOPROBE significantly beats scaling laws across scales from 10k to 260k.
2

---

Published as a conference paper at ICLR 2025
version model's training set.
D
Sample or generate expanded set as d] d].
d
Analyze ( M, to select the optimal expanded set. Iterative search to derive the training of current version:
4
4
d5
dF
Deu D dā dē dž.
d;
Timely update Q and then initiate the search loop.
Figure 1: Illustration of dataset expansion tree. As described in secciinn based on the current dataset, we perform heuristic search for each expansion from multiple candidate batches, calculat- ing the Q values to select the optimal one. The solid black line represents the expansion path.
PRELIMINARY: TRAINING DATA EXPANSION IN MLLM ITERATIONS
The training of MLLMs involves iterative enhancements, and the critical step is expanding the train- ing set. Starting with base pre-train pre-trained MLLM M the current version, denoted as M initial- ized from the M and tuned on instruction dataset D1_ For the next version M the dataset is expanded to D2 While it is expected to fine-tune M on D and obtain the M incrementally fine-tuning 2 on D may risk catastrophic forgetting (Zhou et al. 2024 Zhang ct al. 2024).
Consider the cumulative expansion process, e.g., when 10k new instruction data is added, denoted as d we define dataset expansion as D2 being D plus d'' Thus, D D + d' + + In summary, our goal is to fine-tune M Deur to achieve the optimal performance, where Dour expanded from the initial DD In the context of expanding our dataset at each step. we consider the exploration efficiency and sample the next one from n potential candidate batches, denoted as d dd....,,d, d! The most valuable batch (selected) improves the overall quality of the existing training set. These candidate batches are derived from existing labeled instruction dataset Diaheled, or be generated from unlabeled visual cues or other in- the-wild dataset Dunlabeled-
As illustrated in Figure 1. the expansion process from Dí Di+1 be visualized as growing tree of the dataset. Acquiring high-quality training data can thus be framed as the search for the optimal path. We introduce heuristic strategy, where the tree structure step generates n candidate batches: di, .d....,dd d' Motivated by the A search framework, we propose quality function:
Qd,; M) (dy; M) + 7 (dj; " (s))
For simplicity, we omit the step-index i. Here, M and represent the training model and its current state, respectively. r(-) is the reward-so-Fr far function that indicates the quality score of adding dy to the existing dataset; π² (s) is rule for high-qualis data based on the model state s, and h( a heuristic estimate the future training quality of dy. The ratio is the discount factor.
In Equation it is adequate calculate the component r(-). it represents the existing performance score or training cost. Meanwhile, the heuristic h1-) requires estimating the relationship between the data characteristic and the future quality of training. The inputs for h(-) categorized into high-level meta-information as IT meta- and an intrinsic representation, denoted as inminsic-- In this paper, we examine the components of wvallatin--epllornng-vo provided. Specifically, we focus on evaluating the data to derive Iitrinsi and Imsaa exploring for the rule "t estimate the high-quality distribution based on the model state s; and systematically generating and evolving more targeted data from D Dunlabeled-
loop based on the formalization
2.1 COMPREHENSIVE INTERPRETABLE DATA DESCRIPTION
Some in-context Yang et al. 2023 2024b) or related data pruning works (Cao et al. 2023 Lu
ett al. 2024b) involve intrinsic descriptions, such as the visual attributes, resolution, clarity, color distribution, text-related metries, such as instruction/answer length, perplexity, and vocabulary. With the advanced zero-shot capabilities of LLMs and their multimodal extensions, many stud-
ies (Epasto et al. 2017 Lian al. 2023 Chen al. 2024c) propose using additional models to

---

Published as a conference paper at ICLR 2025
replace manual evaluation. For example, Chen al. (002bb) introduce ChatGPT, which improved performance efficiency compared to the Alpaca instruction tuning set, while Li et al. (2024a b) ana- lyze the crrrructionfolllwwii difficulty metrics. Due to storage constraints, some embedding-relatte methods are challenging to deploy in MILM training. In this paper, we examine intrinsic and meta levels, presenting multi-dimensional model zoo comprehensively describe the dataset's nature, properties, and distribution.
2.2 DYNAMIC HIGH- HIGH-QUALE QUALITY DATA SELECTION RULES
The data description serves as an input for the future quality estimation function h )) At this point, designing high-quality data evaluation rule π based on the model state becomes crucial. Previous methods (Brown et al. 2020 Rac et al. 2021) are constrained by narrow data descriptions, resulting in unstable strategies influenced by random variables. They rely on costly empirical rules or deter- ministic mechanisms with manually set hyper-thresholds. For instance, some works (Raffel et al.) 2020 Laurençon et al. 2022) consider word counts, statistical ratios, and multiple weight-based traversals, In recent years, increased interest has been in optimizing high-quality data rules based on model states and training environments (Xie al., 2023c MMingiaan al. 2022). Similar to the "no free lunch" theorem. no single rule is optimal for all situations. Approaches like DoReMi (Xie et al.) 2023a) and DoGE (Fan et al. 2024) focus on optimizing the alignment of transferred distributions with the surrogate models. Some works (Biderman et al. [2023] Swayamdipta et al. (020) start to consider searching for dynamic, high-quality data rules. In this paper, exploring is critical step in ZooPROBE for determining the form of the heuristic function h (-).
2.3 EFFICIENT TARGETED TRAINING DATA EVOLUTION
During the dataset expansion process, the candidate dataset can be sourced from collections or generated by additional MILMs. Recently, the multimodal community has significant ad- vancements in data engineering, starting with large image-text pair datasets such as Conceptual
Captions (Sharma et al. 2018 Changpinyo et al., 2021 LAION (Schuhmann et al. 2021). D), and
SBU (Ordneez et al. (011)) One type of approaches (Dai et al. 2023; Wang et al.] 2023 Zhang
al. 2023a Li et al., 2023b Moon al. 2023) tailor annotation information to create instruction
candidates based on multimodal QA datasets, using strategies like template transformation, filter-
ing, and deduplication to refine the data. Other methods (Li al. 2023a Yin et al. 2023 Wu al.)
2024 Zhao et al.) 20236] Zhang et al., 2023b Wei et al.) 2023) leverage the powerful zero-shot capabilities of LLMS and their extensions. For example, methods like LLaVA (Liu et al., 2023 2024) convert annotated content into diverse task prompts to prompt ChatGPT, GPT-4, GPT-4V for constructing visual instruction supervisions. However, such generative approaches, represented by methods like ALLaVA (Chen et al.
2024a). may lack targeted control over distributions, poten-
tially harming diversity (Zhao et al. generative MI.LM (Xu et al. 2024). In contrast, LLMs such as Nemotron-4 (Adler et al., (024) and SmolLM (Allal et al. 2024) pre-define detailed task guidelines, addressing these concerns more ef- fectively. In this paper, ZooPRoBE investigates the relationship between more comprehensive data evaluation metrics and the high-quality rules to evolve accurately targeted new data.
2023a) and introducing bias based on the preferences of the
3 METHOD: EVALUATING, EXPLORING, AND EVOLVING LOOP
In ZOOPROBE, we implement a dataset expansion framework using a heuristic search to enhance the training set for the next MLLM version. This process creates a dataset expansion tree, where each step generates multiple child-expanded dataset batches as leaf nodes from the existing training set. We evaluate and derive batch data descriptions for each child dataset, and explore selection rules for high-quality data based on these descriptions. Concurrently, we utilize these high-quality rules to systematically generate and evolve additional instructions and answers. ZooPROBE with E3 loop facilitates the growth of child datasets and promotes new data generation.
3.1 EVALUATING DATA THROUGH EXTRA SMALL-SCALE MODEL. Zo0
To comprehensively evaluate the data, we create smaller-scale diversity model zoo that includes various models from different perspectives, such topic classification (Li et al. 2023c). object
4

---

Published as a conference paper at ICLR 2025
detection (Kirillov al., 2023), ) depth estimation Yangg al. 2024a), visual semantic representa- tion (Dosovitskiy et al., 2021] Radford et al. 2021). and concept recall models (Jiang et al. 2024). We categorize description into meta properties and intrinsic attributes, where meta ones involve the interactions of high-level-evaluatte
information, and intrinsic ones refer to identifying visual ob- jects, textual themes, and the detailed distribution of subjects. In ZoOPROBE, we conduct thousands of distribution assessments and evaluate over 50 dimensions to construct data descriptions.
Extraction of meta descriptions. Considering the efficiency of training set evaluation, ZoOPROBE adopts the textual and multimodal property ranker (Li et al. 2023c Diang et al. 2024) for batch of data, denoted as M Manker for the critical dimensions, similar to the recommendation systems, where the ranker retrieves relevant meta information for each data. Specifically, we extract prototypes P as the core representation that encodes the subjects and properties, then ZooPROBE organizes and measures them according to their groups. At this stage, each batch of data retrieves corresponding alignment degree from multiple meta prototypes. For instance, we offer keywords for over 5k topics, with a 5-level prototype for each property to annotate concepts from poor to excellent. Different rankers compute the alignment between the data and the meta prototypes. We formalize the meta- description as:
II meta (d: P) concat (Exed | (x. P) pEPEP] }
where II represents the concatenated distribution of dimensional evaluations for various prototypes, as denoted by concat {.}. The expectation vector E is computed based on the current expanded data batch d for each prototype E P.
Considering that the textual instructions in the training data often generated based on visual content, we extract dominant fine- aine-graied visual features. In detail, ZooPROBE introduces specialized computer visual models (Dosovitskiy al., [021] Radford al.)
Extraction of intrinsic attribute.
2021 Kirillov et al., [203]] Yang et al. 2024a) for object recognition, target detection, localization, depth estimation, and so on. Then it gathers expectation distribution on batch data, such as the count, location, area, depth, and other recognition information of objects in the foreground and background. Similarly, ZOoPROBE also collects some intrinsic attributes, such as the resolution and color style of the image, as well as the language and length of the text. Together, these distributions Iimiinsicc along with the meta- dieeeddimessooss meta mentioned above, form the description vector II = concat mImma I of the data batch d. We have:
Inrinsic (d) concat (Exed [MEV()) ..... Exed [Attr" (x) (x)) ....)
(3)
where Mev represents a fine-grained visual model, and Attr represents the intrinsic attributes the data itself. In ZOOPROBE, during each step of selecting the next batch in the expansion tree, the evaluation function measures the current nodes' quality based on the data description distribution. Experiments show that ZooPROBE allows for dynamic adjustment of the missing dimensions in II based on available resources, enabling partial evaluation within the model zoo.
For evaluation dimension taxonomy, some in data description II belong to allocation, manifesting as the distribution of themes or objects. Others include information about properties, expressed as high-level meta-descriptions, such as diversity, dependency, guidance, stability, ambiguity, and other cognitive interactions. On the other hand, high-quality data maintains specific ratios across some scattered dimensions. For example, the optimal balance between text-only and image-text instruc- tions varies by context. Additionally, the language styles differ based on the situation. Mcanwhile, low-quality data is detected and pruned in some other filtered dimensions, such as in dimensions related to data annotation quality, social biases, violence, and pornography.
3.2 EXPLORING HIGH- QUGH-QUAIITY SELECTION RULES BASED ON MODEL STATE
In in|section2) 2] we formalize the setting for high-quality data expansion and frame it as search process on the expansion tree. [Equation describes the quality function which estimates the candidates of the next step. This function consists of the reward-so-fai and heuristic estimate of future training quality, denoted as r(.; .M) and h(:; T"), where π" is a high-quality rule based on model state This subsection begins by defining the model state within the current training set Deur, fol- lowed by an explanation of the reward-so-fan function. We then elaborate on the concept of high quality, which introduces the construction of the heuristic estimate h( *²). Next, we detail how
5

---

Published as a conference paper at ICLR 2025
E3 Loop of ZooPROBE
Visual Perception
Step 1: Evaluating }
CLIP, SAM Depth Anything v2
Objects the loreground
Model Zoo
bodiground, and their count.
location, depth...
EL
Multimodal Analysis
Phi-3-mini, ES-V
d
Mera properties, e. mocality
complemenie by, Intoractiviiy, cons stency...
Instruction Answer Examination
Question:
How raas unleelks langing up in the loaddy supes supes-llove dlove Answer: ftor.
GTE Qwen2
divnng stoi?
D
such theme. length, langunge
Then obtain Description intrinsid)
Amea Plmiiinsic distribution each batch
Step Evolving Data
22
TT
M
derend
Step 2: Exploring Quality Rule y
T
Prepare probe set daoan meet:
of dpsts "metches" nt
Wings Targeted
Sample the next rule at' with "probe score" dipete.
incorpo ote the mensions requiting
untuncement quality rulls prompi Use
generation necessary
Then obtain the Form of the Heuristic Function with st'
New Data Expansion
Randomly sample some batches dy. use evaluated data descriptions calculate data quality Q with quality rule rt' expand batch.
Figure 2: Visualization of
(E3) Loop.
ZooPROBE leverages data descriptions to identify πτ. Finally, we integrate this with the reward- so-far element and summarize the data qualities captured by the Equution 1
Reward-so-far data reflected in model state. We consider the performance of the current trained MLLM and the ratio of the learning speed as metries assess the reward so far. Formally:
on m) + StePecnvergence
Step convergence
(d: .) = Perf
(4)
StePiriiiing
We sample a subset from the classic benchmarks to evaluate performance and use the percentage of convergence steps to represent the learning speed, where a is the coordination factor.
High-quality conditional on the model state with the existing training data. Considering that evaluation results Equation 2) and [Equatio are expressed in the form of a distribution of expectation, ZOOPROBE naturally tends to describe high-quality selection rule π pretable distribution. For the newly added batch d, apart from some dimensions that are filtered as previously mentioned in folsubecttion 3.1. most scattered dimensions exhibit favorable distribution. We clarify that these scatfered dimensions do not maintain a fixed ratio during dataset expansion. At each step. they influenced by the current data Deur D d' + d2 + d + Some efforts (Tong et al. 2024) aim to avoid long-tailed, biased distributions, attempting to disperse the expectation on dimensions achieve sufficiently diverse dataset. We denote batch daiverse that makes the current dataset Deur more diverse. Using Kullback-Leible divergence to measure the difference, we have:
similar inter-
(5) Achieving greater diversity, namely the distribution of batch d. is the sampling trend for high- quality data. However, the optimal distribution may not always be uniform. For instance, certain challenging concepts, such as "person", which interact intricately with their context in multimodal environments, should have more coverage in the training set. The primary objective of the rule exploration is to determine the optimal distribution "¹ of the upcoming batch through the model state 8, it reflects the reward and state of current set Deur
h (daiverse: U, De DKL. III (DF")) 4) DKL (II (Dour ddiverse) 41) 0

---

Published as a conference paper at ICLR 2025
Exploration of heuristic estimate (high-quality rule). Firstly, we conduct random perturbations based on Ul to obtain 11; T2, ... T., Simultaneously, we sample corresponding batches, which we refer to the probe set, Le., dgrohe, 1, dgmbe, 2. dgmoeee r. subject to h (d T, D=")) > 0. Next, for the exploration step, the model is trained on the union of all probe sets dgmhe, and the current dataset Deur explore the reward. The adopted optimal distribution TT' prioritizes sampling based on b-based softmax of so-far- so-far-rewards. The sampling weight for T is:
k softmax" (r ji M)) s.t. h (dprobe, ji 3 D > 0.
The distribution т, associated with the higher so-far-reward of plotee suggests an increased likeli- hood of exploitation. The weighted sampling also maintains the exploration of other distributions.
We provide cost trade-off solutions for probe-based fine-tuning by: 1) fine-tuning with parameter- efficient methods, and 2) employing the method for pre-assess the performance of the model after fine-tuning on the probe set. We follow Ngguyen et al. (2020), introducing log expected empirical prediction a proxy metric for efficiently estimating the transferability between the base MLLM and the probe set. It only requires inferring the target set (D + dprobe, ;} once through the MLLM.
Next, the data tree is expanded within the A framework. After sampling π from Equation 6 refine the form of the heuristic function in Eqqation 5] Meanwhile, the reward-so-fal function, as given in Equation 4, is expressed in terms of the parent nodes. Subsequently, ZooPROBE randomly samples some expansion batches and calculates the quality with the Equation
3.3 EVOLVING NEW DATA FROM HIGH-QUALITY GUIDELINES
In the data expansion tree, ZooPROBE can sample the child datasets from existing labeled collec- tions of visual instructions. However, the wild scale can be thousands of times larger than the sample size. Like related works, ZooPROBE enhances sampling efficiency and introduces additional pre- trained MLLMs to generate new data. Furthermore, ZOOPROBE creates more high-quality data by considering the high-quality rules identified in Equaation 6 targeting the dimension gaps in the exist- ing training set. It first identifies missing features in the current distribution and uses a representative set with based prompts to generate detailed macro task information. This output then serves as subsequent prompt for generating
instructions and responses.
Macro task prompt as detailed data generation guidelines based on high-quality rules. As illustrated in Figure 2, ZooPROBE begins by integrating additional MILMM to outline the high- quality components missing in the current training distribution. For instance, if specific themes, such as mathematics or medicine, lack sufficient instructions or properties like inadequate spatial reason- ing or interaction with visual information, the additional MLLM generates relevant sub-themes and detailed descriptions.
New data generation with macro task prompts and representative examples. ZooPROBE re- trieves images based on specified rules and generates new instructions and responses using macro task prompts and representative examples. This data is then added to an iteration of the expan- sion tree and evaluated in the E3 loop. If it meets quality standards, the training set is enhanced. ZooProBe maintains a dynamic balance between sampling from wild collections and generating new data, ensuring a rich and diverse training data evolution.
3.4 LEARNING A DATA QUALITY SCORER ON EXPANSION TREE
In the E3 loop of data expansion, many filtered and discarded training sets are marked. We deploy an MLLM ranker to learn a quality score for the training data. Specifically, we construct both positive and negative samples, adding the prompt: "Please consider the and answer whether it suitable for training with just 'yes" or 'no"." We use the cross-entropy loss to ensure the probability of the model outputting "Yes" is high for positive samples. In subsequent heuristic expansions, the MLLM scorer can replace the evaluating and exploring processes, directly assessing the data quality.
4 EXPERIMENTS
In this section, we first outline the setting, architecture, and details of the E3 loop. Following that, analyze how ZooPROBE, improves performance and efficiency, along with the effectiveness of

---

Published as a conference paper at ICLR 2025
Perforanns
A12D
ScienceAA
MMStar
RealWorldQQ
SEEDBench
MMMU
40
Enginee
Random
LLaVA
Sample
61
88
58
Scale
70
36
48
64
.33
1:.0
260k
20k
260k:
110k
260k
20k
110k
260k
20k
11.K
260k
20k
110k
260k
Performacce
A12D
SciencoQA
MMStta
RoalWorid
SEEDBench
MMMU
Engine
Wings
Sample
Roncom
41
53
68
61
Scale
73
38
50
85
36
20k
110k
260k
20k
110k
260k
20k
110k
260k
20k
110R
260k
20k
110kk
260k
20k
110k
260k
Figure 3: Comparison of scaling-laws performance between the ZooPROBE data engine and random sampling across major benchmarks. The horizontal axis indicates the scale of training data in increments of 20k to 260k; i.e., from 20k, 60k, 110k, 160k, and 210k, 260k. The vertical axes are calibrated to maintain a consistent range of 8, facilitating relative comparison. The first row and second row are the results for LLaVA (Liu et al. (023) and Wings ZZhan al. (2024). respectively, where the yellow and pink lines indicate training with ZooPROBE.
the MLLMM-data-coorer in assessing data quality in subssection 3.4 the acceptable levels of zoo evaluate time, incomplete descriptions, or the absence of rule search.
In ablation studies, we evaluate
Details of the dataset expansion setting: To simulate the original dataset expansion, randomly sample 10k from the LLaVA-NeXT (Liu et al., 2024) fine-tuming tine-tuning instructions as the base dataset. Each expansion batch consists of 1k samples. After 50 heuristic updates, we add 50k samples and updated the existing training set. Subsequently, we perform quality rule sampling and updates based on Equation tjt refine the heuristic function for the next 50 iterations.
Details of MILM architecture, training, and benchmark Evaluation: In our experiments, we utilize the LLaVA (Liu al. 2023) architecture and its extended version, Wings (Zhang et al.] 2024). LLaVA employs a visual encoder with a connector to extract image features, which are then combined with instruction features for input into the LLM. Compared to LLaVA, Wings adds an attention compensation module. We utilize Llama3. 1 Instruct (Dubey et al. 2024) for all LLM parts and SigLIP (Zhai al. 2023) as the visual encoder. Starting from pretrained MILM VA- 1.5 (Liu et al.] 2023)) ShareGPT4V (Chen et al. 2023). and VALLVA-Caation (Chen et al. 2024a), we fix the visual encoder and fine-tuning the LLM and connector with a learning rate of 2e and le-5, respectively. All training is on 8x A100 GPUs. We evaluate MMMU (Yue et al. 2023), MMStar (Chen al. 2024c), RealWorIdQA (x.ai) 2024), A12D (Kembhavi et al. 2016) for test,
ScienceQA (Lu al. 2022 for test, and SEEDBench (Li et al. 2023d).
Details evaluating:
model zoo and dimensions: We design both the meta and intrinsic data characteristic dimensions, introducing multi-faceted textual, visual, and multimodal models to eval- uate the data comprehensively. As shown in Figure 2 GTE-Qwen2nareee (Li et al. 2023c) performs evaluations based instructions and answers, covering dimensions such as linguistic complexity, potential ambiguity, conceptual depth, knowledge coverage, logical structure, inferred established information, emotional subjectivity, temporal stability, and practical applicability. E5-V (Jiang al.) 2024) encompasses multimodal characteristics including annotation quality, image quality, the dir- ficulty of object recognition images, text comprehension levels. visual content diversity, scene complexity, dependency on visual and textual content, multimodal interactivity, requirements for ex- ternal knowledge, cognitive load, context interpretability, detail orientation, novelty and unseen con- cepts, as well as specific time series, action dynamics, emotion recognition, mathematical concept coverage, interpretation postures and gestures, spatial reasoning, and interpretability elements. Additionally, CLIP (Radford et al., 2021 and SAM (Kirillov et al. 2023) work together to complete visual object recognition, including counting. location, and size identification, while Depth-Anything V2 (Yang al., (024a) estimates the depth of objects. We also extract the lan- guage and token sequence length of the instructions and answers, as well as the images' brightness, color palette, resolution, and clarity.
graphic
8

---

Published as a conference paper at ICLR 2025
Paa Performa mene
RealWorldQQ
MLIM Sconor
(a) Introducing MLLM-Scorer as h() After 110k
ZooProoe:
A12D for Wings
A12D
LLaVA
68
Performante
Food
Performance
Samplc
Oppesite
ZooPauzz Engine
48
Scale
Randiom Sumpln
20k
110k
260k
SEEDBench
63
Half Dimension
Quareer of Dimereion
54
ZooPoow Engine
ULVA
69
Random Sample
68
59
Scale
61
Sca e
20k 60k 110k 160k 210k 260k
20k 60k 110k 160k 210k 260k
20k
110k
290k
(b) Training w/ Fixed Diverse Rule or Opposite Quality Rules (c) Training Partial Eval. Dimensions
Figure 4: Ablation studies of the heuristic function determined by the quality rule. (a) We employ an additional learned data quality, MLLM-Scorer, as the assess function for training LLaVA, introduced after the 110k scale and is highlighted in red. (b) We examine a fixed diverse strategy (a red dashed line) with a strategy that is opposite to ZooPROBE's (a gray dashed line). (c) We analyze the effects of partial dimensional in the data description, considering half-dimensional and quarter-dimensional setups marked with red and gray dashed lines, respectively.
Step Training Time k.s. Evaluating metar Matriasie
Step Training Time Exploring w4 T2g. "", Hr
GPU Memory
Time Cost
Time Cost
Overhead
On Expandon Dataset:
On Expansion Dotaset:
Step Training
-72x
Step Training
4500
40x
Eralusting Oa Dotaset:
Poobe Set:
Fine-grained Amett
-6x
Fine-tuning (Pull Pammetery
~15x
40x
Retrieving n meta
-Ix
Fine-tuning (LoRA)
~2.5x
-10x
Extracting Inntristi
-0.1x
LEEP Transferability
1x
(a) Inluitive Comparison of Training v.s. Evalusting
(b) Intuitive Comparison Training v.s. Exploring Rules
Figure 5: Comparison between the running overhead of ZooProBE and model training per step. (a) We compare the costs across three types of dimensions during the evaluation step. The average evaluation cost per iteration is approximately 1/50 of the step training cost. (b) We compare the overhead of rule exploring on the probe set every 50 steps of dataset expansion. The average exploring overhead is about 1/20 of the training overhead. We extend and probe five branches.
Details of exploring: rule sampling with reward-so-far evaluation. In Equation 6 we investi- gate and explore rules after conducting 50 iterations of A search. During this process. we refine the heuristic by perturbing the uniform distribution 2. This perturbation, utilizing the Dase- heuristic, disrupts certain dimensions that normally required maintain direction towards 2/ during optimization. To evaluate the effectiveness of different rules, we sample a probe set of 10k and calculate the cumulative reward. As shown in |subsection 3.2 for datasets with fewer than 200k parameters, we perform full parameter fine-tuning, and for larger datasets, we adopt parameter- efficient LoRA-tuning (Hu et al. 2022).
Details of evoluting: data sampling collections, data generation prompts, and conditions: As discussed in /subsection 3.2] most of the expanded data is sampled from existing data collections.
We consider massive fine-tuning pool, including ShareGPT44 (Chen et al. 2023). LLaVA- 5-5- finetune (Liu et al.) 2023). I.L.VA-NeX" (Liu et al. 2024), ALLaVA-linetune (Chen et al., 2024a), and Cambrian 10M (Tong et al. 2024), which contains subclasses such as language, OCR, counting, code, math, and science, all under 300k threshold. We filter the general, language, and OCR sub- classes and obtain about 3.5m samples. When sampling batch exceeds 1,000 but fails meet the quality standards, we initiate a data generation process. This involves generating macro task prompts to address the identified gaps, facilitating more accurate new instruction and response generation.
Comparative analysis of different architectures and scaling, As detailed in seccionn 4 the first and second rows contrast the scaling performance of L.LVA and Wings against the benchmarks

---

Published as a conference paper at ICLR 2025
set by random data sampling and the expanded dataset used by ZoOPROBE. Overall, ZOoPROBE significantly outperforms the random sampling strategy across all benchmarks for both MLLM ar- chitectures. One key advantage of ZooPROBE is its ability to tailor exploration rules based on model state and training situations. For instance, between the 60k to 260k scale range, LLavV and Wings apply different heuristies, resulting in varied data expansion strategies. Our experiments reveal that a dynamic quality rule derived from the probe set-based search, defined in [Equation 6 consis- tentl delivers superior performance across different architectures and model states. As we progress to larger scales, particularly beyond the 110k mark, ZooPROBE exhibits continuous performance enhancements, further amplifying its advantage over random sampling. This trend indicates that strategically exploring rules based on existing data not only accumulates training quality but also boosts performance as dataset sizes increase.
Analysis of Detailed Benchmarks. ScienceQA, MMStar, and RealWoridQA, ZooPROBE shows marked improvements across various domains. Remarkably, despite having only about one-third of LLVVA-1.5's training volume (260k), its performance on the A12D datasets is nearly equivalent. This indicates that our data engine en- hances MLLMs' capabilities and serves as automated pipeline for high-quality data adaptation.
When comparing across several benchmarks, including AI2D,
Training of an additional MLLM-Scorer to serve as the heuristic estimation. In In/Figure (a)) we present the results achieved after integrating MLLM-Scorer with over 110k data points. Compared to ZooProbe, MI.LM-Scorer performs at a similar level, highlighting its effectiveness in evaluating data quality and showcasing its robust performance.
A Ablation studies on the heuristic functions, evaluated dimensions, and efficiency of ZooPROBE.
In Figure 4] we validate the effectiveness of ZooPROBE's dynamic quality rule, demonstrating its ability to identify data that truly benefits model training. Next, we focus on the efficiency of ZooPROBE, controlling the number of dimensions during evaluation to see if we can maintain performance while only considering the partial dimensions. In Table we provide a detailed com- parison of the overhead of ZooPROBE relative to the training process.
Why is the dynamic exploring rule in ZOoPROBE effective? In Figure 4 (b), we first establish that the opposite rule leads to performance degradation. Although fixed diverse rules emphasize data variety and outperform random sampling, they are still weaker than ZooPROBE. This suggests that the data requirements for the model change dynamically; certain less utilized dimensions of data may become less important as the iteration progresses.
In resource- limitee situations, can ZooPROBE maintain performance by using only a subset of evaluation dimensions effectively? As shown in Figure 4 (c), we randomly sample half quarter of the evaluation dimensions and find that high-level evaluation information provided crucial insights for exploring high-quality rules and optimizing performance. While using partial dimensions results in some performance decline, it still outperforms random sampling.
We compare the costs of ZOOPROBE's evaluation and exploration part to step training. In Table (a), the fine-grained data description refers to those with derivational relationships, such as the high-level meta relationship concerning "quality". This part requires more time for inference outputs but constitutes only about one-sixth of all the dimensions. Additionally, in Table 5 (b), we present the costs of three different methods for estimating the quality of the probe set.
5 DISCUSSION AND CONCLUSION
In this paper, ZooPROBE continuously generates high-quality data through the proposed evaluating- exploring-evolvee loop, beating the performance barrier of scaling laws under equivalent training costs. The ZooPROBE's model zoo comprehensively evaluates data characteristics and, combined with heuristic exploration strategies, explores dynamic quality rules based on varying model states on probe set. Furthermore, ZoOPROBE can specifically generate high-quality data, address distri- bution gaps, and create an adaptive, universal data engineering framework. Data deconstruction in ZOOPROBE is similar to combinatorial dimensions in zero-shot learning, but ZooPROBE explores properties from a higher-level perspective. Future directions may include efficiently establishing the relationship between latent space and training quality while introducing learnable factors to acceler- ate the search process. We will also investigate the safety of data generation, ensuring quality while avoiding bias and considering transferability across different model scales.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work partially supported by NSFC (62376118, 62250069), Key Program of Jiangsu Science Foundation (BK20243012), Collaborative Innovation Center of Novel Software Technology and Industrialization.
REFERENCES
Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H. Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, al. Nemotron-4 340b technical report. CoRR, abs/2406. 11704, 2024.
Loubna Ben Allal, Anton Lozhkov, and Elie Bakouch. Smollm blazingly fast and remarkably powerful, July 2024. URI/https::/hugginggfaa.. htt.icee..ccb/g//////a//aa lms.blloo//mmmmmmm
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. (tt.::233../29000 /2966, 2023.
Stella Biderman, Hailey Schoelkopf. Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, Usvsn Sai Prashanth, et al. Pythia: A suite for analyzing large language models across training and scaling. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 2397-2430, 23-29 Jul 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, et al. Language models are few-shot learners. In NeurlPS, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020.
Yihan Cao, Yanbin Kang, and Lichao Sun. Instruction mining: High-quality instruction data selec- tion for large language models. CoRR, abs/2307 Abs/22307.0629, 2023.
Soravit Changpinyo, Piyush Sharma, Nan Ding. and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. 3558-3568, 2021.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamesssing sptte-sytteesized data for a lite vision-language model, 2024a.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training better alpaca with fewer data. ICLR, 2024b.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint 1.Xiv:2311.12793. 2023.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, and Feng Zhao. Are we the right way for evaluating large vision- language models? CoRR, abs/22403220333 2024c.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, al. How far are to gpt-4v? closing the gap to commercial multimodal models with open-source suites. CoRR, abs/2404.16821.. 2024d.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuma: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https:
//1msys. wrg.s..rrrbbbe //lg///loe og/ 222 - 30-vicuna/
Wenliang Dai. Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NewrIPS, 2023.

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Zhengxiao Du Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In ACL, pp. 320-335, 2022.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, et al. The llama 3 herd of models. CoRR, abs/2407.21783, 2024.
Alessandro Epasto, Silvio Lattanzi, and Renato Paes Leme. Ego-splitting framework: from non- overlapping to overlapping clusters. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 17, 2017. pp. 145-154, ACM, 2017.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. DOGE: domain reweighting with generalization estimation. In ICML, 2024.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In EMNLP, pp. 6894-6910, 2021.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, et al. Train- ing compute-optimal large language models. CoRR, abs/2203. 15556, 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023.
Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang. and Fuzhen Zhuang. E5-V: universal embeddings with multimodal large language models. CoRR, foss/2407.12880 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. CoRR, fass/2001.88366 2020.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Min Joon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram worth dozen images. In ECCV, volume 9908 of Lecture Notes in Com- puter Science, pp. 235-251, 2016.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yen Yen Lo, Piotr Dollár, and Ross B. Girshick. IEEE, 2023.
Segment anything. In ICCV, pp.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, et al. The bigscience roots corpus: A 1.6tb composite multilingual dataset. In NeurIPS, volume 35, pp. 31809-31826, 2022.
Nayoung Lec, Kartik Sreenivasan, Jason D. Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. In ICLR, 2024.
Chen Li, Yixiao Ge, Dian Li, and Ying Shan. Vision languaee instruction tuning: A review and analysis. Trans. Mach. Learn. Res., 2024, 2024a.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- aanuuag-and--o andeee-nddviieie assis- tant for biomedicine in one day. In NewrIPS, 2023a.
12

---

Published as a conference paper at ICLR 2025
Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, and Qi Liu. Mt A large-scale dataset towards multi- modal multilingual instruction tuning. CoRR, 0oss/2306003887, 2023b.
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In NAACL, pp- 7602-7635. Association for Computational Linguisties, 2024b.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-geminis Mining the potential of multi-modality vision language models. CoRR, abs/2403. 188144 2024c.
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkum Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281, 2023c.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan L. Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In CVPR, pp. 14963-14973, 2023d.
Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verifica- tion, 2023. URL
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https://
llavavvl.o www.vll.g............ io httbl//////////01111.11//1 30- 1lava nexaannex///
Haoyu Lu, Wen Liu, Bo Zhang. Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. Yaofeng Sun, Chengqi Deng, Hanwei Xu, Zhenda Xie, and Chong Ruan. Deepseek-vl: Towards real-world vision-language understanding. CoRR, abs/2403.05525. 2024a.
Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In ICLR, 2024b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurIPS, 35:2507-2524 2022.
Brandon McKinzie, Zhe Gan, Jwwa--caiiipppaae Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, et MMI: methods, analysis & insights from multi- modal LLM pre-training. CoRR, abs/2403.09611 2024.
Guang Mingjian, Yan Chungang. Liu Guanjun, Wang Junli, and Jiang Changjun. A novel saabborrooddweiighhe sampling method for imbalanced datasets. Chinese Journal of Electron- ics, 31(5):969-97) 2022. doi: 10. 1049/cee.22021 1.00..121...
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin, Tushar Nagarajan, Matt Smith, Shashank Jain, Chun-Fu Yeh, Prakash Murugesan, Peyman Heidari, Yue Liu, Kavya Srinet, Babak Damavandi, and Anuj Kumar. Anymal: An efficient and scalable any-modality augmented language model. CoRR, abs/2309. 16058, 2023.
Cuong V. Nguyen, Tal Hassner, Matthias W. Seeger, and Cédric Archambeau. LEEP: A new measure to evaluate transferability of learned representations. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 7294-7305. PMLR, 2020.
OpenAl. Chatgpt. https: htttps://opeaa http:///pppaaa.ccmoooooooogo///
com wtt.c/ooo/ooccca/ecooooooo chatgpt. 2022.
13

---

Published as a conference paper at ICLR 2025
OpenAL. Gpt-ov(ision) sysssmm card.
OpenAl. Hello http://wwwwaaaae.........././//a
Card. 2023. Accessed: 2024-05-26.
40/ 2024a. Accessed:
2024-05-26.
OpenAl. Introducing gpt-4o: our fastest and most affordable flagship model. https://
platform www.oor...pp.......mmmmoom...... com/
2024b, Accessed: 2024-05-26.
Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg, Im2text: Describing images using million captioned photographs. In John Shawe-Tayke Taylor. Richard S. Zemel, Peter L. Bartlett, Fernando C. N. Pereira, and Kilian Q Weinberger (eds.), NIPS, pp. 1143-1151, 2011.
Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, and Vedant Misra. Grokking: Gener- alization beyond overfitting small algorithmic datasets. CoRR, abs/2201.02177) 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 9748-8763 2021.
Jack W. Rac, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, et al. Scaling language models: Methods, anal- ysis & insights from training gopher. CoRR, abs/21 12.11.11444 11446, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., (1) 2020.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: open dataset of clip-filtered 400 million image-text pairs. CoRR, abs/2111 1100114,, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Iryna Gurevych and Yusuke Miyao (eds.), ACI., pp. 2556-2565, 2018.
Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In EMNLP, pp. 9275-9293, Online, 2020.
Shengbang Tong. Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. Cambrian-l: fully open, vision-centric exploration of multimodal Ilms, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 37X11 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajiwal Bhargava, et al. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09228 2023b. doi: 10.48550/ARXIV.2007 https::/doi... or1/000rrg10.... 00111..855500 48550/ arxiv www.v..wv.................... 2307 09288
URL
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 2rXiv:2409.12/11 2024.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou. Yu Qiao, and Jifeng Dai. Visionllm: Large language model also an open-ended decoder for vision-centric tasks. In NeurlPS, 2023.
14

---

Published as a conference paper at ICLR 2025
Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. CoRR, 0e0022338..207,,, 2023.
Cheng Wen, Xianghui Sun, Shuaijiang Zhao, Xiaoquan Fang, Liangyu Chen, and Wei Zou. Chath- ome: Development and evaluation of a domain-specific language model for home renovation. CoRR, 0oos/237..15290 2023.
Shenggiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multi- modal LLM. In JCML, 2024.
x.ai. Grok-1.5 vision preview, April 2024. URL http::t/s///
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Perey Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In NeurIPS, 2023a.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V Le, Tengyu Ma. and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In NeurIPS, 2023b.
Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In NeurlPS, 2023c.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpic: Alignment data synthesis from scratch by prompting aligned Ilms with nothing, CoRR, abs/2406.08465 2024.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: A massively multilingual pre-trained text-to-text transformer. In NAACL-HLT, pp. 483 498, 2021.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything V2. CoRR, abs/2406.09412. 2024a.
Xu Yang. Yongliang Wu, Mingzhuo Yang. Haokun Chen, and Xin Geng. Exploring diverse in- context configurations for image captioning. In NeurIPS, 2023.
Xu Yang. Yingzhe Peng, Haoxuan Ma. Shuo Xu, Chi Zhang, Yucheng Han, and Hanwang Zhang. Lever LM: configuring in-context sequence to lever large vision language models. In NeurIPS, 2024b.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0188 01800, 2024.
Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, Jing Shao, and Wanli Ouyang. LAMM: language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. In NeurlPS, 2023.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, et al. MMMU: A massive multi-discipline multimodal understand- ing and reasoning benchmark for expert AGI. CoRR, abs/231 1.11502,, 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In ICCV, pp. 11975-11986, 2023.
Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. PMC-VQA: visual instruction tuning for medical visual question answering. CoRR, abs/2305.10415, 2023a.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. CoRR, abs/2306.17100 2023b.
15

---

Published as a conference paper at ICLR 2025
Yi-Kai Zhang, Shiyin Lu, Yang Li, Yanqing Ma, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, De-Chuan Zhan, and Han-Jia Ye. Wings: Learning multimodal Ilms without text-only forgetting. CoRR, abs/2406.03396, 2024.
Bo Zhao, Boya Wu, and Tiejun Huang. SVTT: scaling up visual instruction tuning. CoRR, abs/2307.04087, 2023a.
Zhiyuan Zhao, Linke Ouyang, Bin Wang, Siyuan Huang, Pan Zhang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Milm-dataengine: An iterative refinement approach for MLLM. CoRR, abs/2308. 13566, 2023b.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In NeurlPS, 2023.
Re- Zhou, Hai-Long Sun, Jingyi Ning, Han-Jia Ye, and De-Chuan Zhan. Continual learning with pre-trained models: A survey, CoRR, abs/2401.16386, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ICLR, 2024.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
We sample some of the top-performing data on some evaluation dimensions, as shown below:
Action Dynamics
Question: <image>inWhat the differeut initializations?
The different initializations related mobility, competition and cooperation.
Answer:
Question: <image>/nAnalyzs the image comprchensive and detailed manner.
The image captures thrilling mement of kayaker navigating through turbulent waters. The kayaker, clad in green wetsuit and a black helmet, seen skillfilll maneuvering yellow logo its side. The water around the kayak is white and frothy. indicating the intensity of the rapids. The kayaker uses paddle to steer through tumultuous water, demonstrating high
Answer:
black kayak, which sports
skill and control. of depth and
The image taken from side perspective, adding dynamism to the scene. In the background, large rock formation looms, providing a stark contrast to the churning waters in the foroground. The image does provide identifiable information about landmark referred to 'sa 108'.
Question:
Is the man performing tricks on skateboard in the image?n-simage>
Answer: Yes, the man in the image performing tricks on skateboard.
Question: What type of image is used to capture the man's skateboard tricks?
Answer: time-lapse or a series of multiple pictures has been used to capture
skatehourd tricks, effectively showing movements and progression the trick he performing.
Question: Is there only one image, or are there multiple images combined to show the skatebourder's movernenss?
Answer: There are multiple images combined to show the skateboarder's movements while he performs trick.
Question: <imago>'u What are the key takcaways from the graph?
The key takeaways from the graph are that the proposed tracker outperforms the state-of-the-art trackers on almost all challenge attributes. In particular, the proposed tracker performs significantly better than all trackers on the occlusion attribute since it includes re- re-detection module which can re-acquire target case tracker fails in cluttered cavirenments
Answer:
removing clutter using GM- PHD filter. Similarly, the proposed tracker also outperforms other trackers on scale variation attribute since our tracker elegantly estimates the scale of the tracker the newly estimated target positions.
Question: <image>'n What pattern observed in the movements of the individuals across the images? A) individuals progressively moving fister. B) The individuals are standing still. C) individuals
enguging in coordinated activity. D) The individuals are randomly seattered without any clear pattem..
Answer: The images show a sequence of movements where the position and stance of individuals change systematically. suggesting coordinated activity like a dance or exereise routine. Therefore, conecc answer is C) The individuals are engaging in coordinated activity..
17

---

Published as a conference paper at ICLR 2025
Accuracy of OCR Data
Accurccy of #igerinoon athrret
Question: <inage>la What is the accuracy of the algorithm data in the datasel cap?inAnswer the question using single word or phrase.
Answer:
4116
Question: <image> Cann you accurately transcribe the from provided image me?
Answer: essential rol the treble tone for acousti guitars. Th e quality of vibra tion of the finger board the p rincipal charact cristic for gene rating the tone. reas on, ebony iter, becaus high use, ebeny
Question: <image>ln in What magnetic field strength depicted the energy diagram of the figure? I Tosla B) Tesla C) Tesla 10 Tosla
Answer: figure label VB TV which indicates the magnetic field strength used in the energy diagram Tesla. Therefore, the correct answer C) Tosla.
Question: <image>>nlHow many star symbols are there near the text m"nn% directly.
Answer:
52.000
Question: simage> are number in the image?
18

---

Published as a conference paper at ICLR 2025
Complexity of Visual Scenes
Question: <image>'n 'nAnalyee image in a comprehensive detailed manner.
Answer: The image caplures vibrant scene of large parade, leerning of individuals, marching with and pupose. They are holding up various flags and banners, the Russian flag and the flag of the Soviet Union being particularly prominent, adding a splash of color to the seene. The people are also holding up photographs of soldiers and other historical figures, perhaps remembrance or celebration. perspective of the photo from high angle, providing bird's eye view of the crowd, the flags they carry, and the photographs they hold up. This perspective allows number of participants. The background of the image is mix of trees and buildings, providing a contrast to the crowd and highlighting the urban setting the parade. The landmark identifier Vsaa_ 15895'" does not provide additional information about the specific location event.
people. The crowd,
of unity
appreciate the scale of the event and the
Characterzatiion using Jam
Question: simage>inBused on the molecular structures and data charts displayed in the background, what type of research or experiment might the scientist be conducting?
Laading
Lasc cooking
Answer: The presence molecular structures and detailed data charts the digital display suggests that the scientist likely involved in molecular biology or binchemistry rescarch. The use of pipette and petri dish indicates he might be conducting experiments reluted to cell culture, genetic analysis, or drug development. The complexity of the displayed data implies advanced research, possibly areas like gene editing, protein synthesis, pharmaceutical testing.
Question: Bamaage>inAssess the quality of the image by describing its visual complexity.
The image has high visual complexity. with multiple objects, overlapping boundaries, and various sizes and orientations of the items.
Answer:
Question: <image>InC Considering variety of floral patterss presented, which specific flower lcaf design appears the frequently throughout the composition, and what could be the significance of this repetition within the context of a coloring page?
The five-petal blossoms composition. The repetition of this specific flower could serve several purposes: it provides consistent element that ties the diverse patterns together, creating visual harmony: it offers simpler shape for coloring. which could be appealing for those who want repetitive and meditative coloring tasks; and aesthetically. it could represent theme of unity or abundance, which is oflen associated with nature and floral motifss
Answer:
be the frequently recurring floral design the
Question: <image>inhow many materials of e-waste given in the chapondd concisely.
Answer: 12
Question: how many substances from chart affect liver?
Answer: 5
Question: what the substances inside PC that causes liver cancer?
Answer: arsemic, plastics/PVV
19

---

Published as a conference paper at ICLR 2025
Contextual Interpretation
Question: <image>'n some skills required H-IB applications?
Answer: Most H-IB applications ( 75 % require high level computer knowledge and roughly hall require significant engineering and math skills. High level scientific knowledge is less commonty requested (11%)
NONEICTION FIO REACING
Question: <inuge>'nWhat is the title or main subject of the book mentioned in the image"
COMPEDMENSION
Answer: The book focuses Nonfiction Rending Comprehension in social studies, ideal for test practice. It targets Grade students and includes standards benehmarks. The book is created by Teacher Created Resources.
Question: Does the image indicate the content of the book?
Yes, the image suggests that the book contains nonfiction reading comprehension materials relating to social studies topics. This could potentially include information on cities, bridges, and other aspects of human society and infrastructure.
Answer:
Question: <image> you read and provide the displayed the image? This photo to have some interesting texl. Can you extract and type it for me?
Answer: S-al pha tetr ahydr ocorti sol a nd 5 -beta tetrahyd rocortiso reactio us which S-alph a edu et ase an 5- be ta- re ductase are the rate-] imit ing
Question: <image>'uWhatt What the main topic discussed the text?
Answer: he main topic discussed in the text is RDNG 4240 Literncy in K-4 and guidelines on how to write a critique of a journal article for the class.
Question: How many points are each article response worth?
Answer: Each article response is worth 5 points.
Question: How many article responses are required for the class?
Answer: Two article responses are required the class.
Question: simage-lutWla What does the \"Score Mapl" in the figures represent in the context of the scientific analysis being performed? A, The likelihood of the presence of an object various locations within the image. B. The color intensity variations across the image. The temperature distribution captured by thermal camera. D. The altitude variations on a topographic map.
The Score Map images represent heatmaps where higher intensities indicate higher likelihood of the presence of object at that location. This is indicated by the concentration of intense colors at certain points corresponding to the object's location in the ground-truth images. Therefore, the correct answer is A.
Answer:
20

---

Published as a conference paper at ICLR 2025
Dependence on Textual Information
Question: <image>laWhy do you think the title chart simply Titlee? What could imply?
Answer: The generie 'Title' could suggest that this chart template
example and has been customized for specific presentation. It likely serves as placeholder, indicating where descriptive title relevant to the data should be added.
Question: How importanl is to have descriptive title for a chart?
Answer: Having descriptive title crucial because provides context for the data and assists viewers in quickly understanding the chart's purpose. can convey subject scope data, making chart more effective accessible for interpretation.
Question:
<image>in What percentage global population depends on ground water for drinking'aAnsswe directly.
Answer: 50%
Tha FLIM
Question: Simage=inWhalt the perception of people about their lifestyle over the years and now?
Answer: in the 2006 Pew survey laken this summer ). Americans on average gave score to their lives five years ago ; a 6.8 score to their present quality of life and a 7.8 score the lives they expect to leading five years rating points worth
from now That adds up au aggregate average forward progress between years ago and five years hence
Question: <image=IaWhai do variational methods allow the estimation of? Directly refer to the text in the image to answer this question.
Answer: Variational methods allow estimation of hieracchical discrete choice models a fraction time roquired MCMC.
Question: What is the value proposition of variational methods? Directly refer the text in the image to answer this question.
Answer: The value of the variational approach is greatest when sub sampling of data ill-advised.
Question: According to the authors, should MCMC be abandoned favor variotional metheds? Provide image.
this question by directly using the provided the
Answer: We emphasize that we do not advocate abandoning MCMC in favor of variational methods.
21

---

Published as a conference paper at ICLR 2025
Detail Orientation
Question: simage>'nWhen patterns can inferred about interaction among balls from the input frame The balls not interacting with each other all. B. There interaction from one hall to all others. C. The interactions are complex and multi- directional among the balls. D. The balls interact only in a linear fashion.
inference module and the causal summory graph? A.
one-directional
The correct answer is because the causal summary graph for the balls shows multiple lines connecting each ball, indicating complex multi directiooal interactions. The lines represent interactions, and since they are not unidirectional or absent, options A, B. and incorrect. Therefore, the correct answer is C.
Answer:
What can be deduced about the precision of measurements for 1V c 5 5 graph? A. High precision, as indicated by the tightly packed points Moderate precision, indicated by spread data points precision, as indicated by the widely spread data points Precision cannot determined from the graphin-image=-nn
Question:
Answer: The first graph shows the data points for VV N_ = 5 W)) closely following the trend line with little deviation, indicating that the mensurements were consistently close to the expected values. This suggests high level precision in the measurements, making option the correct answer. Therefore, the correct
Question: <image='n What are the different difficulty levels?
Answer: The difficulty levels are based on the number of actions in the The casiest quests have only few actions, while the hardest quests have many actions.
Question: simage=/n What the total number of medals between south korea, north korea, sweden, and ansserr the question using a single word or phrase.
Answer: 150.
Question: <image>>nn are ways stay focused and productive mentioned the image?
SWAYSTI STAT FOCTSED
The ways to stay focused and productive mentioned in the image are/nl. Have priorities per year. If you have more than that, they 'r not prioritics, in2. Each day, focus closer to your goals.nn3. Don't get overwhelmed by all the things you need do. Focus putting foot in front of the other 'n4. Remember your top personal priority is energy management, If your energy is high, your output will be high. Protect your energy at all costs inš, Set specific times to check your emails. DMs, and texts. Stay in your own lane.
Answer:
needle-moving task that helps you get
Question: How would you describe the aesthetic of the image?
The image has pink and gold abstract background with white frame, which gives off modem and stylish appearance. The text well out, easy to read, complements the overall design while conveying the message effectively.
Auswer:
22

---

Published as a conference paper at ICLR 2025
Diversity of Visual Content
Question: <image>InBased about the datuset? High dimensionality with clear seperation Low variance among the data points C) Uniform distribution different dimensions Data points concentrated in specific region
scatter plot for 'satellite image, what be inferred
Answer: The scatter plot for 'satellite image' shows data points spread out distinct, multi durtt--dieetional pattern, indicating high dimensionality with some degree of separation between data points. Therefore, the correct answer A.
Question: <image='a What might be the intended application of these vector special effects based on their design and composition?
design composition, these vector special effects could intended use in digital media, such graphic design, web design, video production. They might used enhance visual appeal, simulate natural phenomena like fireworks or stars, create dynamic backgrounds and overlays for various types of visual content. The bright colors and glowing effects would be particularly useful for projects that aim to grab attention or convey sense energy vibrancy.
Answer: Based
Question: <image=in What the main subject of the image?
Answer: The main subject of the image is a collage of photos.
Question: What types of photos included the collage?
Answer: The collage contains photos of persons, animals, and lockets.
Question: What lype of question being asked in the image?
Answer: There is question present in the image; is collage of photos
featuring persons, animals, and lockets.
Question: <image Ceaegge--nnCooeee InCossidering assortment fabrics, could inferred about the potential origin these scraps?
Answer: The mix and disorder of the fabric scrups suggest they be remmants from various sowing or quilting projects. The diverse patterns and colors could imply that they are offeuts from a place where multiple types of garments textile products are made, studio, crafts workshop. Their varied nature indicates they might be used for patchwork, crafting. fillers for projects. given their vibrant and contrasting aesthetics.
failor's shop, fashion design
Question: Simage=/nDesrrioe the diversity of colors and patterns the vases in the image.
Answer: The have varied colors and patterns, such green and pink, lime and black, black and teal, terrncotta and green, and yellow and purple, making each vase unique and attractive.
23

---

Published as a conference paper at ICLR 2025
Emotion Recognition
Question: <image>'nPlenee describe process counting number of people distinctly present the provided image.
Answer: From the bounding box information, can identify that there least five distinct individunls: man sitting on the borse, a lady in a blue outfit, three women standing by man on horse, and people the background.
Question: <imagee-inCosiiderin the objects the man is interacting with and the number on his forehcad, what could be a possible scenario that explains the combination of these elements?
500
Answer: The scenario could that the man has participated challenge or bet, which is commonly marked by writing one's skin marker. number 1"500'" might indicate milestone, score, the challenge. The presence of the cigarette and the bottle could suggest that this activity is taking place in social setting where smoking and drinking are involved, perhaps a private gathering or a celebration. The man's serious expression and formal attire might indicate this event significant him, that
associated with
of contemplaation about the activity is engaged in.
Question: are there any other people besides boy."oriimgge//neespppeo concisely.
Answer: no
Question: how old does boy look?
Answer:
Question: picture indoors?
Answer: hard to toll
Question: <image>inis child male female?/nRespond conciscly.
Answer: fermal
Question: old girl?
Answer: 5
Question: what color her
Answer: brow
Question: <image>IuC heteeeaaCooooiiiiiie the theme of the image and the arrangement of the photographs, what might be the significance the partially obscured photo in the top right comer, narrative suggested by the image?
contribute the overall
STRANGE MYSTERIEST FROM OHIO
Answer: The partially obscured photo in the top right corner could represent an element of mystery or an unresolved aspect story. Its partial concealment behind other photographs suggests it may hold information that the theme of mystery. In narratives, especially those dealing with unsolved mysteries or cases, incomplete information or partially revealed clues can be deliberate device to engage the audience, prompting them to seek more information and become more involved the
fully revealed understood, contributing
the unraveling enigmatic quality of the image and invites further inquiry the mysteries presented.
story. Thus, the obscured photo
24

---

Published as a conference paper at ICLR 2025
Mathematical Concept Coverage
ORango q's 1ea re
<image>InCrtique image's quality and provide suggestions for improvement.
Question:
Answer: The image quality is detailed and high, capturing various elements such as the oranges' textures, chalkboard sign with white lettering, and even a watermark, However. could improved by minimizing the glare the light on the orange skin.
Question; <imuge>>nConsidering the objects the man is interacting with and the number on his foreheud, what could be a possible scenario that explains the combination of these elements?
500
Answer: The scenario could that the man has participated in challenge or bet, which is commonly marked by writing on one's skin with marker. number 1'500"" might indicate milestone, score, the challenge. The presence of the cigarette and the bottle could suggest that this activity is taking place in social setting where smoking and drinking are involved, perhaps a private gathering or a celebration. The man's serious expression and formal attire might indicate that this event is significant him, that moment of contemplation about activity is engaged in.
target associated with
X
y
Question: <amage='n The table shows function. the function lincar or nonlincar?
5
2
Answer: To determine whether function is linear or nonlinear, see whether has constant rate change /niick the points in any rows the table and calculate the of change between them. The first place to start 'nCall the values in the first row xl and yl. Call the values in the second row x2 and ynlnRate of change Vifaac (y2 yl] [x2 dijnn- - Wfrac (9 2) (6 5j'n Wfrte [7] [1 NowNow pick any other two rows and calculate rate of change between them Chem'nCaal and yalmnate change xl and yl. Call the values the third
good
6
9
7
16
values
second row
Wfrcc ly2 y1; [x2 xmm Vimee [16 9] 17 6)'m Wrrac (1jin 7hnThe two rates of change are the same inlf you checked the rate of chinge between rows and 3, you would find that it is also 7hinThis means the rate of change the same for each pair of points, So, the function has constant rute change. The function linear.
5cm
Question: Simagee>/EElbbnn on what shown illustration.
Answer: The diagram illustrates the lateral surface of a cone-shaped paper cup. The cone has bottom surface, and the generatrix shows the slant height of the cone.
15cm
Question: <image>'n CinnieeiuCnnsideriin the detailed information provided about the event, what could he the potential benefits for a student to participate in this Mathlathon, based on features advertised on flyer?
MANTA AATLAAM AEAR
The potential benefits for student participating in the Mathlathor, advertised, would include the opportunity to compete in an online setting with peers globally, which could enhance their competitive spirit and global awareness. The detailed reports could help students identify their strengths and weaknesses math concepts. The excitement of winning trophics and medals could serve motivation published in the Mathathon Hall of Fame could provide recognition for their achievements and could be si significant accolade for their acndemic portfolio.
Answer:
excel. Moreover, the possibility of being
25

---

Published as a conference paper at ICLR 2025
Novelty and Unseen Concepts
Question: <image>InHow would you rate the quality the photograph based on the provided information?
Answer: The photo looks very great, according to the image information.
THE WELL
Question; What be the symbolism of the man's silhouette with heart hanging from a rope?'n-image=-n
The symbolism of man's silhouette with heart hanging from a rope could represent the concept of vulnerability and emotional weight one might carry. The heart, often used as symbol love and emotions, dangling from rope, suggesting that it exposed and perhaps fragile. The silhouette of the man might indicate that this is universal experience, not limited to specific individual. Overall, this image could symbolize the emotional struggles, vulnerability, and the challenges that people face their inner lives.
Answer:
Question: <image>'nCoolle you tell me more about the artistic style used?
Answer: artistic style is reminiscent of digital painting with vihrant colors and dynamic strokes, giving sense of motion. The details both the mechanical and organic elements indicate careful balance between realism and fantastical design.
Question: How might this image interpreted symbolically?
Answer: Symbolically, the image could represent the intersection of humanity's natural essemce with the advancement of technology. It might be expressing perspoctive on the potential harmony or conflict between our biological origins artificial enhancements.
variety of activities and the presence of diverse architecture and signage, what can be inferred about the cultural and economic characteristies of this city?
Question:
bustling economy. The diverse architectural styles indicate blend of traditional and modern influences, possibly suggesting city with long history and willingness to embrace contemporary trends. The presence of street vendors and variety of shops signifies thriving local economy with a strong focus on commerce and trade. The different activities of the people, from leisurely walks engaging in commerce, imply city with diverse socio-economic activities, catering to both locals and visitors. The intricate signage may also indicate city that values artistic expression and attention to detail, further emphasizing its cultural richness.
Answer: image suggests city with rich cultural tapestry
TIG TOUBLE Poite
Question: <image>InConsiderin the stylistic elements and depicted charncters, what geare does this poster suggest the movie belongs to, and what themes or motifs are suggested by the imagery used?
The poster suggests that the movie belongs to the action-adventure genre with elements of fantasy. The central figure with the gun, the action poses of the characters, and the ominous overlord figure with supernatural glowing eyes indicate story that involves conflict, heroism, and otherworldly forces. The neon text, lightning effect, and the mystical object held by the seated older man imply themes of magic or supematural power. The imagery of the street scene with varied characters in dynamic poses suggests motifs of a journey or quest, possibly with group of diverse companions.
Answer:
26


Reviews:
[{"review_id": "TG7uQcfLeG", "reviewer": "Reviewer_Xi4n", "summary": "This paper presents a novel method to efficiently and effectively grow fine-tuning datasets of MLLMs. This method is built upon an evaluate-explore-evolve loop. In this loop, it utilizes other LLMs and smaller models to extract meta and intrinsic properties of additions to existing datasets, in order to estimate the performance of the base MLLM when trained on newer datasets and perform A* search based on this estimation. This method allows MLLMs to break scaling laws in multimodal instruction fine-tuning at scales of 260k, enabling MLLMs to systematically and automatically evolve training data in data/resource-constraint situations.", "questions": "Directly related to my concerns above:\n\n- What are the benchmarks used for performance computation in the reward / heuristics?\n- Are there duplication in the images used for the benchmarks used for reward computation and the evaluation benchmarks?\n- Can you provide more details on the dimensions used (i.e., a detailed table / number of dimensions for meta/intrinsic properties of data)?\n- Which dimensions/properties contributes most/least to high performance of the method?\n- Can you provide more details about the perturbed distributions, such as which of them are commonly observed, and how much does the perturbation matter (compared to targetting uniform distribution)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "This paper presents a systematic approach towards a challenging problem–selecting and growing datasets to maximize downstream performance during fine-tuning. While this problem is often open-ended, and ill-formed, the proposed solution effectively utilizes and combines existing methods, by gathering signals using a zoo of powerful models and a well-defined search space for A* search. The implementation of such a system is also highly complex, incoporating many models and many dimensions of the data of interest. The significant primary and ablation results also represent that an impressive technical effort by the author(s) on the proposed method.", "weaknesses": "While this paper presents some impressive results on breaking the scaling law, I have a few concerns / questions about the paper. My first concern is around the possibility of overfitting to the evaluation benchmarks of interest. While the proposed method did not explicitly and directly optimize towards the evaluation benchmarks, I wonder if the benchmarks used to compute the reward overlaps with the evaluation. Even if they do not directly overlap, the set of benchmark would affect the generalizability of the proposed method to the general capabilities of the model. I believe this weakness can be resolved by having the author(s) further clarify the benchmarks used in reward/performance computation process in the loop. Another aspect of this weakness is whether duplication was observed between the candidate dataset and the ones used by the performance computation / evaluation benchmarks, of which an overlap of these images could indicate overfitting to particular images common in the benchmarks.\n\nThe second concern is the sparseness of details in the dimensions considered as meta and intrinsic properties of the data. While the author(s) have listed a number of examples of dimensions being considered, no specific/quantitative details were provided about these dimensions, and there is only a brief ablation on the number of dimensions in the results (which I appreciate). The paper can be improved with further exploration on the usefulness of various dimensions, and/or report failure cases on the impact of specific/specific types of dimensions on the effectiveness of the proposed method.\n\nMy final concern would be the strong assumption of targetting the uniform distribution (or random perturbations that starts from the uniform distribution) for the high-quality rules. The paper can be improved by further explaining the most common perturbations that the methods have converged to, and how future method(s) might consider a different distribution to begin with.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vjonBM0X9U", "reviewer": "Reviewer_9FbC", "summary": "This paper presents ZooProbe, a system for synthetic data generation and selection for multimodal tasks. ZooProbe is characterised by the evaluate-explore-evolve (E3) loop. Starting from some seed data, ZooProbe first assesses the quality of the data, and then employs A* search to iteratively grow the data. It also evolves the target data with rules. \n\nEvaluation is conducted on a number of QA datasets including AI2D, ScienceQA, MMStar, RealWorldQA, SeedBench and MMMU. Results show that, in general, ZooProbe can elicit stronger performance than a random sample baseline, sometimes substantially.", "questions": "* The paper repeatedly claims that ZooProbe can \"break the scaling law\". It seems to mean ZooProbe is able to elicit strong performance from VLM. However, I'm not sure what that means exactly. Please clarify. \n\n* Furthermore, this phrase (\"breaking the scaling law\") is hyperbolical, as it is only empirically validated on one single VLM (LLaVA) and six datasets. I suggest the authors tone it down. \n\n* On line 107, the author mentions catastrophic forgetting. I agree that this is an important problem to be concerned with, and I'd like to see some discussions on how ZooProbe mitigates this forgetting problem.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Synthetic data generation has become a key technique to continue to push VLM/LLM performance. Thus, investigations in this area are timely and important. \n\n* The results show strong performance of the proposed technique.", "weaknesses": "* The clarity of the paper could be improved. Throughout the paper, there are phrases and sentences that are ambiguous (I will give some examples below). These ambiguities have an impact on the paper's readability. \n\n* The impressive evaluation results are compared on a \"random sample\" baseline. Stronger data synthesis baselines should be compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t1r6cQTtcJ", "reviewer": "Reviewer_tve5", "summary": "ZOOPROBE is a data engine designed to enhance the training of Multimodal Large Language Models (MLLMs) by evaluating, exploring, and evolving large-scale training data. The engine is built around an evaluating-exploring-evolving (E3) loop, which provides insights into data characteristics, identifies quality rules for data enhancement, and establishes a probe set for targeted rule selection tot facilitates the systematic evolution of new, high-quality data.\n\nExperiments demonstrate that ZOOPROBE significantly beats scaling laws across scales from 10k to 260k", "questions": "1. Could the authors elaborate on the relative importance of the various intrinsic and meta dimensions in the data evaluation process? Are there any dimensions that consistently have a higher impact on model performance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. ZOOPROBE uses a small-scale model zoo consisting of various models to evaluate data comprehensively. These models assess over 50 dimensions of intrinsic and meta attributes, such as object and topic distribution, annotation quality, and scene complexity.\n2. ZOOPROBE describes high-quality selection rules as an interpretable distribution based on the current data and model state. It uses Kullback-Leibler divergence to measure the difference in distribution and aims to achieve greater diversity in the dataset.\n3. The authors have demonstrated through experiments and analysis of the results that ZOOPROBE is more effective and has lower overhead compared to other data augmentation strategies.", "weaknesses": "1. The paper presents ablation studies on the heuristic functions and evaluated dimensions, but it could benefit from more detailed analyses. For instance, it would be insightful to see how different combinations of intrinsic and meta dimensions impact the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uHdIGFivlj", "reviewer": "Reviewer_p5bJ", "summary": "This paper introduces ZooProbe - an innovative multimodal large language model (MLLM) training data engine. ZooProbe implements an Evaluate-Explore-Evolve (E3) cycle system that leverages a small-scale model zoo to evaluate data quality, explore high-quality data patterns, and strategically expand training datasets. The key innovations include: 1) proposing a 50+ dimensional data description vector system covering both intrinsic (e.g., resolution, language) and meta (e.g., annotation reliability, knowledge dependency) features; 2) developing an A*-search-based framework to optimize data distribution; 3) generating new data through rule-guided approaches. Experiments demonstrate that ZooProbe significantly outperforms traditional data scaling rules across scales from 10K to 260K examples.", "questions": "1. Regarding the scalability of ZooProbe:\n\n    a. Have you tested or analyzed the approach's feasibility on datasets larger than 260K examples?\n\n    b. Does the A* search optimization remain tractable for much larger datasets?\n\n    c. Are there any theoretical bounds on the computational complexity as the dataset size grows?\n\n2. Implementation details:\n\n    a. Could you provide more details about the stopping criteria for the E3 cycle?\n\n    b. How is the trade-off between exploration and exploitation managed in practice?\n\n    c. What are the key hyperparameters that practitioners need to tune when applying ZooProbe?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper makes a clear technical contribution through its E3 cycle framework and model zoo approach. The innovation lies in systematically addressing data quality through a small-scale evaluation system, rather than relying on traditional scaling methods. The integration of A*-search for distribution optimization is technically sound and well-motivated.\n\n2. The empirical evaluation is thorough and convincing. The authors provide comprehensive ablation studies across different scales, with the MLLM-Scorer showing consistent improvements over baselines. The experimental design effectively validates each component of the proposed system.\n\n3. The work addresses a significant problem in MLLM training with practical implications. By reducing computational costs and carbon emissions while improving model performance, the approach offers valuable insights for the field. The methodology generalizes well to other domains, suggesting broader impact beyond the immediate application.", "weaknesses": "1. While the paper demonstrates improvements in data efficiency, the experimental evaluation lacks comparison with recent data-centric approaches such as ActiveLLM and DataSpell. These methods also address data quality optimization but through different mechanisms. Including these comparisons would better position the work's contributions within the current landscape.\n\n2. The scalability of the approach beyond 260K examples remains unclear. While the results are promising at the demonstrated scales, larger industrial applications often deal with datasets in the millions. The authors should either provide theoretical guarantees for scalability or acknowledge this limitation explicitly. Additionally, the sensitivity of the method to the initial model zoo composition deserves more investigation - how much does performance vary with different zoo configurations?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)