Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development
Authors: Daoyuan Chen
Haibin Wang
Yilun Huang
Ce Ge
Yaliang Li
Bolin Ding
Jingren Zhou
Venue: ICML 2025 spotlightposter (2025)
Keywords: Data Processing
Multimodal Large Models
Sandbox
Primary area: deep_learning->foundation_models
TL;DR: A new middleware links data and model feedback, enabling high performance and low cost verified in broad tasks.

Abstract: The emergence of multimodal large models has advanced artificial intelligence, introducing unprecedented levels of performance and functionality. However, optimizing these models remains challenging due to historically isolated paths of model-centric and data-centric developments, leading to suboptimal outcomes and inefficient resource utilization. In response, we present a new sandbox suite tailored for integrated data-model co-development. This sandbox provides a feedback-driven experimental platform, enabling cost-effective iteration and guided refinement of both data and models. Our proposed ``Probe-Analyze-Refine'' workflow, validated through practical use cases on multimodal tasks such as image-text pre-training with CLIP, image-to-text generation with LLaVA-like models, and text-to-video generation with DiT-based models, yields transferable and notable performance boosts, such as topping the VBench leaderboard. A comprehensive set of over 100 experiments demonstrated the suite's usability and extensibility, while also uncovering insights into the interplay between data quality, diversity, model behavior, and computational costs. All codes, datasets, and models are open-sourced to foster future research and applications that would otherwise be infeasible due to the lack of a dedicated co-development infrastructure.

Data-Juicer Sandbox:
A Feedback-Driven Suite for
Multimodal Data-Model Co-development
Daoyuan Chen 1 Haibin Wang !
Yilun Huang" !
Ce Ge Yaliang Li" Bolin Ding' Jingren Zhou'
Abstract
shaping model performance, there are fast- fast-growing efforts to curate datasets of larger scales and higher quality (Jakubik]
The emergence of multimodal large models has advanced artificial intelligence, introducing un- precedented levels of performance and function- ality. However, optimizing these models re- mains challenging due to historically isolated paths of model-centric and data-centric develop- ments, leading to suboptimal outcomes and in- efficient resource utilization. In response, we present new sandbox suite tailored for inte- grated data-model co-development. This sand- box provides feedback driven experimental plat- form, enabling cost-effective iteration and guided refinement of both data and models. Our pro- posed "Probe-Anai Aeee--Annllllziieene dated through practical use cases on multimodal tasks such as image-text pre-training with CLIP, image-to-text generation with LLaVA-like mod- els, and text-to-video generation with DiT-based models, yields transferable and notable perfor- mance boosts, such as topping the VBench leader- board. A comprehensive set of over 100 experi- ments demonstrated the suite's s usability and ex- tensibility, while also uncovering insights into the interplay between data quality, diversity, model behavior, and computational costs. All codes, datasets, and models are open-sourced to foster future research and applications that would other- wise be infeasible due to the lack of a dedicated co-development infrastructure.
et 2024
Gadre et al. 2023 Xu et al. 2024a).
However, the development trajectories of these models and datasets have historically diverged, guided more by intuition than by systematic co-development methodologies. Recent advances in enhancing multimodal large models tend be either model-centric or data-centric, rarely bridging these two aspects closely. For example, model-centric methods focus on algorithmic enhancements and architectural inno- vations under fixed data priors, while data-centric strategies usually focus on processing datasets independently of spe- cific model training contexts (Qin et al., 2025). Both ap- proaches usually suffer from lack of systematic principles and cooperative synergy. relying heavily on heuristic explo- ration and single-perspectin expertise. This fragmented landscape presents a notable barrier to achieving optimal performance, as the interplay between data characteristics and model capabilities remains largely under-exploited.
workflow, vali-
Moreover, putting multimodal large models into practice is further complicated by infrastructure limitations, increas- ing computational costs, and the faster pace of the project development (Xu et al.) 2024c). In the age of large models with rapidly growing model parameters and dataset sizes, the processes of data processing and model training be- come increasingly issurce-intensive, demanding substan- tial time and computations. Due the absence of cost- effective platforms that simplify and speed up data-model co-development, researchers and developers often face the dilemma of prioritizing result-driven development at the ex- pense of thorough, insight-led exploration. This deficiency hinders the iterative refinement for both domains, leading to sub-optimal outcomes as improvements one domain are hard inform, apply and enhance each other directly.
1. Introduction
The advent of multimodal large models has revolutionized artificial intelligence, pushing the boundaries of functional- ity and creativity across various domains (OpenAT 2024a Wang et al. 2024a). Recognizing the key role of data in
To fill this gap, we introduce the Data-Juicer Sandbox, feedback-driven suite for facilitating the co-development of multimodal data and models. Building upon an open-source data processing system tailored for multimodal large models, Data-Juicer (Chen et al., 2024a ()., our sandbox suite inte- grates a wealth of off-the thf-tf components optimized for usability and compatibility with open-source model-centric infrastructures. Collectively, it offers eustomizable orches-
'Equal contribution 'Alibaba Group. Correspondence to: Yaliang Li eyaliang. li@ allibab-innccom>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

A Feedback-Driver Suite for Multimodal Data-Model Co-devclopment
tration from different levels including end-to-end workflows, specific development behaviors, and underlying data-model development capabilities. Within the sandbox laboratory, users are empowered to optimize data and models based on their fruitful feedback in a cost-controleed environment. This accelerates insight discovery and informed decision- making, paving the way for transferable, resource-efficint data-model optimization.
training algorithms (Caffagni et al. 2024 Li et al.
2024a
and model architectures (He al. 2024a Yin al. 2024
have fueled this interest. Transformer scaling remains prevalent approach (Xu et al., 2023), though high compu- tational demands and optimization challenges often restrict insights to specific datasets and create gap in understand- ing how implicit data biases affect model performance.
Trends in Data-Centric Development. Recently, shift to- wards data-centric development has emerged (Jakubik et al., 2024), emphasizing data handling as key to the efficacy of large models such as CLIP (Gadre et al.) 2023). Despite the increasing recognition of data processing, the heterogeneous nature of multimodal data leads to predominantly heuristic approaches (Long et al. 2024), underscoring the pressing need for more systematic methodologies for data-model co-development.
To exemplify the efficacy of the sandbox, we propose "Probe-Anin Troobe-nallye-Refine workflow, crafted to explore the in- terplay between data processing operators (OPs), model per- formance feedback, and the transferability of these enhance- ments. We apply this workflow to five cutting- edge models: Mini-Gemini (L.i et al. 2024b) and InterVVL.22. (Chen ct al. 2024T), two LLa Vi-inspired models for image-to-text generation, EasyAnimate (Xu et al. 2024b) and T2V-Turbo (Li et al. 2024a), two Diffusion Transformer based mod- els for text-to- generation, and CLIP model (Gadre ce al. 2023) for image-text pre-training. Through over 100 experimental runs and comparative analysis across different tasks, scales and models, we show notable evidence of the Sandbox's usability, including topping the VBench (Huang eet al., 2024 leaderboard with superior performance over competitors such as Gen-3 (Runway.ML 2024) and VEn- hancer (He et al.] 2024a). The improvement is underpinned by a series of insights linking more than 40 data processing OPs and 70 benchmark metries, with fine-graine -ne-grained analysis on the balance between data quality, diversity, compute cost, and model performance, such as scaling behaviors derived from IB to 26B mode scales.
Open-Source Infrastructure. 'he ecosystem for multi- modal model development has expanded with fruitful open- sourced frameworks (Wolf et al., 2020; Liu et al. 2024b). However, contributions to multimodal data infrastructure often consist of raw datasets and preprocessing scripts, lack- ing standardized practices. Most existing data processing frameworks are tailored for single-modal data (Weber et al.) 2024), highlighting the early stage of multimodal data de- velopment. To address these gaps, our work presents a new intermediary layer that connects advanced model in- frastructures with the Data-Juicer system, facilitating better co-development between models and data.
We present more detailed comparisons of more related works in Appendix A
Our contributions can be summarized as follows:
3. The Proposed Sandbox Laboratory
To the best of our knowledge, this is the first open-source [ sandbox suite tailored for co-development between mul- timodal data and models, rendering experimental explo- ration in this field more insightful and systematic.
3.1. Motivation and Overview
Why do we need data-model co-development? In the era of large models, the development of both data and models necessitates collaboration involving numerous algorithm researchers and system engineers. Training data for large models is often highly heterogeneous in terms of quality, context, type, and timeliness. The processing and mixing of these datasets, known as data recipes, are complex and var- ied (Ge et al. 2024). The scale of data amplifies the stakes for refinement attempts on both data and models, imposing substantial computational and time burdens. Traditional data-centric model-centric strategies fall short by opti- mizing in isolation, leading to diminished overall efficiency and resource misallocation (Qin et al. (025). When either component requires adjustment, the dual optimization chal- lenge inflates costs, as one part may have already reached near-optimal status.
We present new effect-proven workflow for data-model co-development and substantiate its impact through em- pirical evidence among image understanding, video gen- eration, and image-text pretraining tasks.
We conduct extensive experiments on benchmarking the effects of dozens of data processing operators and model metrics, providing practical guidance toward further ad- vancements in multimodal large models.
2. Related Works
Model-Centric Progress in Multimodal Large Models. Multimodal large models have gained prominence for their remarkable capabilities (OpenAl 2024a b). Advances in
Why do we need a sandbox laboratory? Given the high
2

---

A Feedback-Driver Suite for Multimodal Data-Model Co-devclopment
Cost-controlled Data Pools
Ranking GPs by model metrics
OP1
OP1
OP,
Importance Analysis
OP₁ [0, 33%
OP3
[o, 33%]
[33, 67%
[67, 100%)
2
[67, 100% 2
Drmdoo
Single-OP Processing
OP2
OP2
OP2
[0, 33%)
B3,, 67%
[67, 100%)
Drandom
Train 3N+1
Compare Model Metrics
Low-cost Models
Which OPs useful which metrics?
OP 33%
OP
OP
33, 67%
167. 100%)
Correlation Analysis
Ranking Pools by data metrics
Duplication Analysis
Multi-OP OP Processing
OP 122,
Higher-Cost Data Scaling
OP OP OF OP
OP
OP
OP23
Q2: Can "top" OPs be effectively combined and scaled up?
Better Data Recipes Datasets
Better Madels
OP;
OP3
OP;
Ranking Pools by model metrics
Diversity Analysis
Figure 1: A probe-analyze-refine workflow within Data-Juicer sandbox for systematic data-model co-development.
cost associated with iterative development, existing methods often resort to heuristic approaches for improving data or models. For example, scaling up "cleaned" datasets can be problematic because determining what constitutes "clean" dataset and measuring its quality qualitatively remains challenge. Further, the impact of iterations on data and models is difficult to attribute due to numerous influencing factors and considerable engineering effort required. A unified sandbox environment enables experimentation with guided optimization. permits users to iterate both data and models based on comparative feedback from small-scale data processing and model training trials, thus helping to mine insights that can be transferred to full-scale production environments and increasing the return investment in computation and time costs.
vide measurable feedback. Specifically, we simplify the interfaces offered by Data-Juicer, which provides over 100 OPs and tools for multimodal data analysis, filtering, and synthesis, with industry-level optimization for large-scale processing (Chen al., 2024b). Built upon it, we inte- grate many SOTA open-source libraries for model training and evaluation such as Mini- Gemin,, Easy Animate, Mod
elScope) VBench, MMBench (Liu et al. 2024b). TextVQA (Singh et al. 2019) and MME (Fu et al.
2023).
Note that the underlying libraries are still evolving, and the sandbox is continuously maintained to incorporate more community efforts in other libraries, optimized for usability and reduced cognitive load. Dedicated scripts combine the classes and hooks into specific workflows, with code and parameters adjusted to ensure cost-controlled environment. This suite serves as groundwork for the deriving following experimental insights. More details from the infrastructure perspective are in Appendix
Layered orchestration of Data-Juicer sandbox labora- tory. Due to the current absence of ready-made open- source middleware infrastructure, we first propose the nec- essary codebase and suite, designed with decoupled com- ponents. They serve for data analysis, processing, recipe optimization, model training, and evaluation, all managed through configuration files and divided into three levels: (1) In the top level, the specific workflows are formed as ordered job list for co-development, organized into four phases-probing data/models, refining data recipes, execut- ing data/model operations, and evaluation. One can flexibly adjust task sequences these phases, reuse built-in work- flows, create their own easily. (2) In the middle level, the common behaviors are formed as hook functions; and (3) in the bottom layer, the necessary capabilities are formed as factory classes. The lower two layers work together to facilitate actionable development capabilities and pro-
3.2. A Probe-Anallye-Refin Workflow
To demonstrate the usage of the sandbox, we introduce a structured workflow illustrated in Fig. [ is designed to answer several key questions for informed decision-making and cost-effective data-model co-development:
(I) Given the variability in data, models, and application scenarios, we initially seek to find out which data process- ing OPs contribute most effectively to enhancing model performance (Sec. 3.2.I). This involves creating equal- size data pools, each processed uniquely by a single OP and subsequently sorted by interested data feedback signals. Reference models are trained these data pools low
3

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
and controlled costs. This step enables to analyze and understand OP impact and correlations with models.
sequential combination.
The number of possible S grows exponentially with ntg ne- cessitating non-trivial selections of combinations to explore. Building on insights from single-OP experiments, where the "Top" OPs are represented by their ranks, we propose two practical strategies: (1) combining "Top" OPs based on their progressively diminishing impacts on model performance, and (2) clustering OPs based on their Pearson correlation coefficients and then combining the "Top" OPs within each category. Similar to the single-OP scenario, we then con- sistently train reference models on {Ps} and Drand to gain comparative outcomes. More implementation details and empirical results on these two strategies can be found in Appendix D.T and Sec. 4.3 respectively.
(2) Guided by insights derived from the most impactful OPs that ranked highest by model feedback signals, we proceed to study whether these OPs can be effectively combined into data recipes and scaled up (Sec. (.2.2). We establish a hierarchical data pyramid, wherein data pools are catego- rized across different tiers based on OP ranks. This step also examines the viability of OP combinations when scaled with increased data volumes.
(3) Built upon found OP combinations, we delve into a dual analysis focusing on data duplication and diversity (Sec.
3.2.3p. We assess whether the model training would benefit more compute from the repeated use of high-quality data pools or from the inclusion of lower-quality data to expand the overall data pool. As result, we obtain data recipes optimized from numerous small-scale comparative experi- ments, which can then be applied to larger-scale scenarios with reduced amortized cost (Sec. 3.2.4..
3.2.3. PYRAMID-SHAN
DATA POOLS
Adopting a larger n, may lead to enhanced data quality while less available data volume. This phenomenon prompts an investigation: should we prioritize reusing high-qualig data or incorporate lower-quality yet more abundant data to escalate training compute scales?
3.2.1. SINGLE OPPERAORRR DATA POOLS
Starting with an initial dataset D, we define a single-OP data pool P. the dataset processed exclusively by the i-th OP (OP,) available in Data-Juicer as P, DJ DJOOP; (p.)(,))) where DJ denotes the data-processing function imple- mented by Data-Juicer, and pj is the syper-parametert gov- erning the operation of OP;, For example, filter OPs of Data-Juicer compute specific statistics and then apply thresh- old criteria to select data samples. Within this workflow, DD is processed by N interested filter OPs, the resulted pools {P.} are sorted by different data statistics and segmented into three equal-sized pools Pi.oow Pi Pi.middee and PP: nigh rep- resenting data with low, middle and high stats, respectively. Besides, D is randomly sampled to serve as a control group Drand such that all the 3N + 1 data pools have the same data size. This design fosters discriminative insights across varying degrees of data processing intensity.
To embed this inherent trade-off between data quality and di- versity, devise hierarchical pyramid architecture, where ny "top" OPs identified in OP experiments combine into - data pools. For example, the three-OP combi- nation OP 1,2.3 resides at the highest level of the hierarchy but has the smallest data volume after filtering The two-OP combinations, such as OP1,2, are placed at a lower level and may have volumes several times larger than 1 Pro- gressing downward through the pyramid, data pools exhibit descending average OP ranking (which potentially acts as feedback of reduced data quality) alongside an increase in data volume.
To strike the balance between data quality, diversity and compute scale, we consider two settings built upon this data pyramid: (1) repetitive training with data repetition from the top-layer data pools, and (2) non-repetitive training via progressively adding lower-layer data pools and applying deduplication. Specifically, we explore variable repetition rates for the first setting and make the number of trained data samples of the second setting consistently match the former. It allows us to qualitatively assess the efficacy of data reuse compared to the inclusion of suboptimal data, within the same compute costs at varying scales.
Subsequently, models are trained independently on } and Drand with consistent hyper-parameters data sample size and compute resources, and finally evaluated across interested performance metrics. This linking of feedback signals between data stats and model metrics enables insight mining and top OP identification.
3.2.2. MULTI-OPERATORS DATA POOLS
3.2.4. DISCUSSION ON SCALING AND COST
We then apply multiple OPs sequentially (denoted S) to examine whether these OPs complement or counter- act each other's effects. To achieve this, we extend the previous construction of (P,} to multi-OPs case: Ps- = (P.:)P (OP)(,,) oJ.... (OP(()))))())) where i, i.j...,, E S, indicating n, candidate OPs used for
All data pools are uniformly sampled and consistently de- rived from D, enabling insights gained from small-pool experiments to be extrapolated to larger-scale scenarios, and making the method capable of overall cost reduction.
To clarify, denote the model training time using the full
4

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
dataset D as T full- Let M represent the iterations required to achieve desirable performance without our sandbox. Let Tpoot represent the training time with a small pool, which is reduced by a ratio of compared to Truu- If the total number of planned spall-pool experiments is m, then the total time without and with our workflow is M x Trul and (Tittt + m x T poot) R (1 + mr)) x Trut, respectively, Our core idea is construct m lightweight experiments and utilize -ne-giii data-model feedback extract insights for scaling and reducing overall costs, making it preferable to disjoint model- or data-centric development that involves M costly (and usually heuristic) experiments.
imate model can lead to VBench Top-1 performance with another architecture-diffrrent model.
For image-text pre-training (TTP), we will show that the optimal recipe identified with a CLIP model hav- ing fewer FLOPS maintains consistent performance ad- vantages when model FLOPS and compute resources in- crease, similar to scaling law behaviors.
For image-captioning (IC) oriented fine-tuning, we will show that the optimal recipe identified with InternVL-22.0 model (Chen et al. 2024f), leads consistent and steady performance advantages as the model scale increases from 1B to 26B
Achieving (1 + mr) 2 .M is feasible. Typically, M > 2 accounts for multiple iterations over different model and dataset versions. The m positively corresponds to the in- terested OPs, often numbering in the dozens. The can be much smaller than 0.01, since there is no need for small-pool experiments to "converge" as long as they reveal informative phaiiee-positive or negative-in model performance after controlled data interventions with (Pt, }s} versus baselines using D Drand- Early stopping of unpromising experiment trials can also help expedite this process.
To demonstrate the flexibility, extensibility, and potential of the proposed sandbox, we further conduct experiments on "iterative workflow", "OPs beyond filters", and "model development of automated prompt adjustments". More details will be introduced later in Sec. 4.6
These examples instantiate different behavior hooks and ca- pability classes introduced in Sec. B.I] following the work- flow proposed in Sec. 3.2 A more structural summarization of these use cases is provided in Table 3 in Appendix E.
Furthermore, m reflects the balance between the intensity of data interventions and the informativeness of data-model feedback, offering flexibility to adjust the experimental plan based on available resources. As for the factor r, we have
In the next subsections, we delve into the primary insights, with complete experimental results provided in Appendix E The feedback utilized in these experiments comes from over 70 widely-used model benchmark measurements and over 40 Data-Juicer's data filtering OPs for text, image, video, and cross-mddalities. For brevity and comprehensive in- vestigation, on the main page, we report the performance changes relative to baselines on Drand, averaging from all evaluated benchmark scores. For a fair comparison, note that all experiments are maintained to ensure that the refer- ence models trained on P. and Ps have the same compute costs to baselines trained on Drand. Detailed task-specific settings can be found in Appendix D including the bench- mark measurements (in Appendix D.2). the functionalities of studied OPs (in Appendix (D3)). and the specific sources and sizes of (D, Drand, P., Ps), in Appendix D.4 to D.6
P|Apoot poot E[Aguu] 2 c| seseet -222/(b-aa)
where Apeol E [a, b] and fuull denote the model perfor- mance change with our small and full data pools respectively. This indicates that the error e introduced by our workflow de- creases exponentially as increases. More detailed analysis is provided in Appendix C
4. Practical Applications and Main Results
4.1. Use-Cases and Experiment Overview
To demonstrate the proposed suite's usability and effective- ness, aiding understanding and boosting confidence in its applicability, in this section, we conduct extensive experi- ments in various practical use cases. They vary in models and data OPs (Sec. 4.2 and Sec. (43). data samples and com- pute cost (Sec. 4.4). leading to insights and recipes useful for larger-scale scenarios (Sec. 4.5).
4.2. Ranking Single-Operator Data Pools
Table I summarizes the results of reference models trained on top- oop-performing data pools. The full tables of all studied OPs can be found in Appendix E.2
Observation 1 (Data vs. Model)
For image-to-text (I2T) generation, we will show that the optimal recipe derived from small pools achieves supe- rior model performance and higher data efficiency on the MGM model when applied to larger datasets.
Multimodal models' efficacy is closely tied to the fi- delity of their output modalities, which can be explic- itly reflected in the filtering of input training data.
For general text-to-vides (T2V) generation, we will show that the insights derived from small pools and the EasyAn. An-
For the text-to-vided task, all top-3 OPs are video-only OPs. For image-to text and image-text pre-training tasks, two of
5

---

their top-3 OPs are text-only, and image-text OPs respec- tively. This trend of influential OPs holds beyond top-3 ranks (detailed in Appendix E.2), suggesting that more at- tention and resources should be allocated data processing A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
Table I: The average performance changes relative to base- line models on Dand for t performing OPs, with mod- els trained on different single-OP data pools split by sorting OP-generated statistics: Pi.oow Pi,mid: Piloow- The training data pool for baseline is randomly sampled, and all com- pared pools are with equal data volume. Full ranking table can be found in Appendix E.2
Observation 3
(Spatiotemporal Dynamics)
Dynamic information in data is harder to learn for image-to-text and image-text pre-training models than for text-to-video models.
Due to the static nature of images, image-to-text and image- text pre-training models often struggle with dynamic con- tent, requiring in-depth semantic understanding. This is evident by their better performance on data pools with fewer text action numbers. In contrast, text-to-videe models per- form differently, showing opposed trends preferring higher text action numbers and video motion scores.
Avg. Perf. Changes (%)
Task OP Statistics
Ps,oow
Pi,miid
Psgiigh
Image NSFW Filter
7.13
18.44
66.38
12T
Text Action Number
59.90
0.29
-2.05
Language Score
49.90
0.85
-1.43
Observation 4
(Modality Alignment)
Video Aesthetics Score
-0.98
0.13
0.96
High data alignment between modalities is crucial for the multimodal tasks image-to-text, text-to-video, and image-text pre-training.
T2V
Video NSFW Score
0.82
-0.05
-0.57
Frames-Ter Text Similarity
-1.45
0.23
0.79
CLIP Image-Text Similarity
-32.57
-6.39
39.53
ITP
BLIP Image-Text Similarity
-24.28
1.82
25.39
Image-to-text and text-to-video models prefer modality- aligned data, as indicated by their good performance with pools featuring high image-texi similarity, phrase grounding recall, and frames-text similarity. This preference is most pronounced in image-texi negee-text pre- pre-training models, whose top-2 OPs are tied to image-text alignment.
Image NSFW Score
12.18
1.28
-18.38
Text Length
0.76
-3.13
-11.36
IC
Image Watermark Score
-0.64
-0.13
0.48
Character Repetition Ratio
0.45
-0.46
-0.63
4.3. Shaping Data Recipes of Top OP Combinations
Based on top-3 OPs in Table I and gained insights, we then study the model performance changes trained on multi-OP recipes Ps over baselines Drand-
related to the output modalities of the studied models.
Observation 5 (Effect of Sequential Combination)
Observation 2 (Diversity Quality)
The optimal data recipe isn't 't always achieved by com- bining the best individual OPs, nor does adding more high nigh-performing OPs guarantee better outcomes.
In contrast, data diversity is more crucial for image-to- text models, while data quality is key for ext-tto-vidoo and image-text pre-training models.
As shown in Fig. 2 combining higher-performing OPs does not always yield better results. For instance, in the image- to-text experiment depicted in Fig. [[a] the data pool with a high image NSFW score, despite performing best in single- OP experiments, generally diminishes performance when combined with others. Similarly, in Fig. 2(b) while pair- wise combinations of OPs show positive gains, integrating the top-1 OP into the combination of the other two OPs reduces the relative improvement from 2.48% to 1.88%. As for the image-text pre-training tasks (in Fig. 2 us- ing the image_ peet_eeeiiii___eeeeeeeeeeeeettee outperforms other combinations. This observation may be attributed to the filtering process relying on a high-quality CLIP model to train another CI.IP model, implicitly and effectively acting as a form of model distillation.
We find that some top OPs share similar functionalities, while their best persforrrrn pools exist in different statistical ranges. For example, Pi.high in image-to-text, and Pe,tow in text-to-video and image-text pre-training. A deeper analysis of OP ranks in terms of NSFW scores and language scores reveals that the studied image-to-text model prefer more diverse data compared to the text-to-video and image-text pre-training models. Intuitively, high-scoring images or videos in NSFW (Not Safe For Work) content are generally rare and occupy the long tail of the data distribution. Con- sequently, pools with high NSFW scores tend to be more diverse. Similarly, texts with low language scores indicate that the language of the text is more difficult to identify and usually less common, corresponding to more diverse pools. Appendix E.3 presents more quantitative evidence in terms of word entropy on these data pools' captions.
OP alone
Besides, we explore categorizing OPs based on analysis of their stats correlations (Appendix E.4) and select the
6

---

A Seedback-Driver Suite for Multimodal Data-Model Co-devclopment
10.29
-122
1.02
10.05
10.62
116,00
15,00
17.41
10.664
10.785
12.45
Image bSPWSSomm Score
61
$3.90
Toxt Action Number [19]
ArethettcSScomm
11.85
T2.48
Video NSNW Score
Enagee Simliarity (hiph)
1..2.0
113.32
Image: Siniar ghigh:
10.272
+0.00
48.203
10.34
11.10
TO 14.041
Langagee Stare (M))
Framee Foat 5 Emilanty (high)
Image NSF Soore (los)
Character Repatiton on Farts (880
(a) Image-to-text
(b) Text-lo-video
(c) CLIP
(d) Image Captioning
Figure 2: The model performance changes (%) from recipes combined with the top-3 OPs listed in Table
optimal OP from each category to form recipes. However, detailed results of Observation 8 in Appendix E.33 show that it also yields non posittoie outcomes. These two observations challenge the common assumption in many existing SOTA works that stacking multiple intuitively useful data-cleaning actions can enhance overall performance.
lines, including those incorporating data from suboptimal recipes. Specifically, for the image-to-text (see Fig. 3(a) model, duplicating top-quality data by a factor of 6 results in higher performance than both using 8x compute with subop- timal data (blue line) and using 8x compute with the original full-size data (green line). For the text-to-video model (see Fig. 3(b) the benefits of using top-quality data outweigh detrimental effects until 10x data repetitions, showing near- linear and significantly higher performance improvements compared to the baseline (blue line).
Observation 6
(Effect of Seed OPs)
Single OP performance positively correlates with the performance of multi-OP recipes containing it. Start- ing with tigh-perfrriig OPs is a good initial step to optimize higher-order data recipes.
For the image-text pre-training task, Fig. 3 illustrates improvements as both model size and compute scale in- crease. Notably, the linear growth in relative improvement with the exponential increase in computation aligns with known scaling laws (Cherti et al. 2023), suggesting that expanding model size and compute resources, alongside pro- cessing data via our high-quality recipes, yields consistent scaling of performance gains.
Although the Top-3 OP recipes exhibit suboptimal perfor- mance, we observe positive gains when combining some pairs of OPs in them can outperform both single top-1 and t combinations, such as in the case of the image- to-text task when combining TextAct onF P te and LanguageIDScoreFills
Moreover, we conduct experiments using Intern VL2.0 across various scales (IB, 2B, 4B, 26B parameters), with the results summarized in Fig. (3d]] (x-axis displayed on log scale). We initially experimented with the smallest model (IB parameters) on 23 single-OP and 6 multi-OP data pools to identify the optimal data recipe. The top-3 combination recipes derived from this initial phase were subsequently applied all selected model scales. Our findings indicate that all three recipes consistently maintain significant perfor- mance advantages the model scale increases from IB to 26B parameters, demonstrating clear scaling law behaviors.
4.4. Scaling Data Samples, Computes & Model Size
Recall that in Sec. 3.2.3 we proposed constructing pyramid- shaped data pools by combining different "top" OPs and explored how to scale up the data samples and computes by either reusing high-quality top pools or by progressively adding lower-level pools to enhance diversity with dedupli- cation. Here we select top-1 and top-2 recipes identified in Fig. as candidate data pools within the pyramid structure.
The results are summarized in Fig. 3 where the expansion rate indicates the compute scale in terms of the number of training data samples.
Observation 7 (Effect of Compute Scaling)
Table 2: Average performance of MGM-2B with different pretraining datasets on 4 benchmarks. The "*" indicates our reproduced version that is comparable with the official version. Detailed results are in Appendix E.6 Table TO
Scaling up compute resources with high-quality data benefits both the image-to-text, text-to-video, image-text pre-training, and image-captioning tasks. Among these, the image-text pre- pre-training and image- captioning tasks demonstrate clear scaling law behav- iors in exponential form across different dataset and model sizes.
MGM-2B
Num. of Training Instances
Avg. Perf. Changes (%)
Baseline
1226k
Ours
159k (x4)
+2.13 -2.12
Generally, the models trained with data derived from top recipes (red lines) achieve superior performance over base-
7

---

A Feedback-Dring Suite for Multimodal Data-Model Co-devclopment
148+
Tey Bayne Rooee
Delaa
Busl with ordinuarpirg
100
Adding Jubotiie Lea rook
Aediing Tareshir Parehhn
1.5
(a) Image-to-text
(b) Text-to-video
(c) Image-texi Pretraining
(d) Image Captioning Finetuning
Figure 3: Relative improvement over baselines when varying computation scales with and without data duplicates, In the latter sub-figures, "Topl" and "Top2" refer to applying the Top-1 I and Top2 recipes identified in Sec 4.3 respectively.
4.5. Transferring Recipes to Larger-Scale Heterogeneous Scenarios
with 40k data samples used in Sec. 3.2.2 to a 5.7x larger pool with 228k data samples. From a model development view, we undertake a challenge to assess the transferability of our findings across different model architectures, by re- placing the training model from the previous EasyAnimate with another SOTA model, T2V-Tuboo, which is improved from VideoCrafter-2.0 (VC2).
The previous image-text pre-training and image-caption fine-training experiments, conducted on CLIP and In- VL models of varying sizes (Fig. and Fig. } al- ready shown that top-performing recipes identified at small scales can be effectively utilized when compute resources are scaled up. In this section, we further apply the gained insights and top data recipes to larger-scale heterogeneous models (i.e., across different model architectures) and het- erogeneous datasets (i.e., from diverse data sources) for the other two tasks (image-to-text and textelo-video).
Table 3 showcases our notable performance on the VBench leaderboard, reporting average scores of 16 metrics from both quality and semantic dimensions. We first enhance T2V-Turbo (the last row) with data-enhanced distillation training 147k instances (the second row), and then self- distill it with other 228k instances (the first row). Com- pared to the base model in the last row, our method yields notable uplifts of 1.53% and 2.59% on the Board and Uni- form Average scores respectively. This verifies the effec- tiveness of our sandbox and insights again, which lever- ages SeteooFrmesTextte wttsestttmmaaaiiyyiiiimi to enhance the video-prompe alignment, and the VideoNsFWFilter OP to ensure the high quality of the generated video.
The case for image-to-text task. We follow Observations 6 and 7, and utilize the top data pool from Fig. 2(a with a 4x repetition, resulting in a pre-training dataset of 637k samples, which constitutes only half of its original pre- training dataset size. Then we train the MGM-2B model using this new pre-training dataset and its original full fine- tuning dataset for comparison.
OP
Table summarizes the results. Since the official MGM-2B model has not been evaluated on MMBench-CN, we trained a reproduced version using the official training scripts, and its performance is closely aligned with the benchmarks re- ported for the official model. We can see that compared to the baseline trained on the full dataset, our model achieves superior performance while trained on only 1/10 of the dis- tinct instances and 1/2 of its original total instances (i.e., using half of the baseline's compute FLOPs). Notably, they differ only in the processing of the pre-training data, with the model and training configurations being consistent, un- derscoring the effectiveness of our insights.
Note that these results highlight the data-efficiency of our method to reduce overall cost with dedicated data-model co-development. Taking the third-place model. VEhancer, as an example, it mainly focuses on architectural refine- ments building upon the foundation of VC2. While our model also originated from VC2, it gains a higher per- formance boost but with at least 22x less compute costs than VEhancer. Detailed calculation of FLOPs is presented in Appendix E.9 Besides, in Appendix E.7 we further conduct an ablation study to verify the effectiveness of our data-model co-improvement over the base model T2V- Turbo, including aspects from data (high-quality dataset with our derived recipes), model (LoRA initialization and self-distillationn and data- model co-design (real-data loss and self-evolution). These results demonstrate the potential cost-effectiveness of the proposed sandbox when utilizing fine-grained feedback from both the data and the model simultaneously.
The case for text-to-video task. In line with the best recipe from Fig. and Observation 6, we scale up the data pool on the full-size candidate video datasets introduced in Sec. D.5. with low video NSFW score and high frume-text similari- ties. Additionall following Observation 7, we conduct six model training passes through this dataset. From a data development view, this transition allows us to probe the scal- ability of our methodologies, advancing from the small pool
8

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
Table 3: Models on the VBench leaderboard as of Sep 23, 2024. "Board Avg." denotes the weighted average scores across 16 metrics defined by VBench and "Uniform Avg." denotes the arithmetic average. "_f" and "_1" indicates proprietary model, and acting as our base teacher model respectively. a indicates constant for specific gradient update implementation. Detailed ablation study, full ranking results, and FLOPs estimation are in Appendix E.
E.8 and E.9 respectively.
Models (Ranked by leaderhoard)
Board Avg. (%)
Uniform Avg. (%)
Dataset Size
Training Samples
Training Cost (EFLOPs
1. Data-Juicer (DJ, 228k)
82.53
81.26
228k
640k
7.68a
Data-Juicer (T2V, 147k)
82.10
80.54
147k
640k
7.68a
2. Gen-3 (RunwayML 2024)
82.32
79.64
_1
3. VEnhancer (VC2) He al.
2024a
81.97
80.00
350k
350k
167.3a
4. Kling (2024-07) Kuaishou
2024
81.85
79.54
_1
CogVideoX-SB Yang et al.
2024)
81.61
79.90
35M
35M
211841a
8. TuroTuroo (VC2) (Li et al.
2024a
81.01
78.67
4.6. Additional Experiment Results and Details
Implementation Details. We provide a summary of the varying scope of our sandbox experiments in Appendix [ detailing aspects such as main effectiveness evidence, model types, dataset size, and compute scale. Besides, we present discussion in Appendix D.I on the two strategies employed to combine multiple operators into recipes based on their correlations. Appendix outlines the model evaluation metrics adopted. The functionalities and corresponding statistics of the studied Data-Juicer operators are detailed in Appendix [D.]] Furthermore, Appendix [D4 claborates the detailed settings for the image-to-text use case, Ap- pendix for the text-to-vides use case, and Appendix D.6 for the image-text pre-training use case.
Further Analysis and Ablation Studies. Appendix E.3 AndFF.4 detail the data diversity and correlation analyses on our multiple-OP experiments, respectively. They showcase the analysis ability provided by the suite with word entropy, word cloud visualization, and visualizations of Pearson cor- relation coefficients among OP statistics and benchmark performance. Appendix E.5 evaluates the performance of models trained using recipes derived from the correlation analysis. Appendix E.7 provides a detailed ablation study of the Data-Juicer T2V models, focusing on data-model co-improvements beyond the base teacher T2V model. This includes analyses from the perspective of data (high-quality datasets with derived recipes), model (LoRA initialization and self- distiisiillatinn.. and data-model co-design (real-dae data loss and self-evolution).
5. Conclusion and Future Works
In this paper, we introduced the Data-Juicer Sandbox, a new open-source suite designed facilitate the co-development of multimodal data and models. By integrating flexible and customizable components at different levels, the sand- box enables systematic, cost-effective exploration and op- timization, bridging often-disjointed domains of data and model development. Through applying the proposed "Probe- maalyze-Refine" workflow in extensive scenarios includ- ing mmage-to-texto generation, text-to-video generation and image-text pretrianing, we showcased how our sandbox can yield not only improvements in both dataset and models, but also valuable insights into the complex interplay between data processing and model performance.
Extensibility of the Proposed Sandbox. Appendix E.10 10 verifies the capability of iterative workflows, illus- trated by the progression: ckpt, refine, recipe, tiio ckpt, refine, recipe, train, ckpt.- Notably, ckpt, demonstrated improved performance despite the evoly- ing recipes. Appendix 1 demonstrates the potential of operators beyond Filters, showcasing two represen- tative Mappers, the
httpeddffffioooonooo _er_puii___mppperrreee__e mapper and i i mmage_capt: on oniiinnnngno ing_mapprr and their integration with the MGM model. Finally, Appendix E.12 demonstrates how the sandbox's 's pre-built infrastructure can be leveraged to explore and auto-optimize prompts within the context of data-model co-development.
To facilitate reuse and expedite innovation, we will continu- ously extend the sandbox's 's compatibility to encompass more model-centee infrastructures and develop more off-the-shelf and effect- effect-proven workflows, such as for reinforced fine- tuning (Pan et al., 2025). Besides, we will theoretically investigating the trade-off between sandbox cost and feed- back transfcrability, such as taking the sampling ratio of data pools as a tunable knob.
Complete Results. As for the complete results of other previously mentioned experiments. Appendix E.2 presents the complete operator ranking from single-operator experi- ments. AppendixE.6 provides comprehensive results for the Image-to- Text task. The full V-Bench leaderboard results are listed in Appendix E.8 Finally, Appendix E.9 details the calculation of reported FLOPs.
9

---

All codes, datasets, and models of our work are openly accessible and actively main- tained at juierbllbmmanndoo//salloo... A Sredback-Driven Suite for Multimodal Data-Model Co-devclopment
Impact Statement
Lin, W., Li, Y., Ding, B., and Zhou, J. Data-Juicer 2.0: Cloud-scale adaptive data processing for foundation mod- els, 2024b. URLhttps: /ttpaaaxivvvv../../ org/ abs/ 2501 1 14755
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Chen, D., Li, Y., and Ding, B. Multi-modal data processing for foundation models: Practical guidances and use cases. Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, pp. 64155, 2024c.
Reproducibility Statement
Reproducibility is essential for validating research outcomes. To facilitate this, we have organized detailed descriptions within the appendix of our paper. Key components of our experimental setup, including implementation details such as datasets and training configurations for the image-to-tess text-to-vided and image-text pre-training use cases can be found in Appendix D.4 Appendix D.5 and Appendix D.6 respectively. We also provide details into the methodologies for combining multiple operators based on their correlations in Appendix D.I, as well as descriptions of performance metrics (Appendix D.2) Furthermore, Appendix D.3 out- lines the functionalities and statistics of the Data-Juicer OPs utilized in our experiments.
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang. X., Weng, C., and Shan, Y. Videocrafter2: Overcoming data limitations for high- video diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7310-7320, 2024d.
Chen, T.-.,, Siarohin, A., Menapace, W., Deyneka, E., Chao, H.-w., Jeon, E., Fang, Y., Lee, H.-Y., Ren, J., Yang. M.-H., et al. Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13331, 2024c.
Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q.. Zhu, X., Lu, L., et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24185-24198, 2024f.
References
Bai. T., Liang, H., Wan, B., Yang, L., Li, B., Wang, Y.,
Cui, B., He, C., Yuan, B., and Zhang, W. survey of multimodal large language model from data-centric perspective. arXiv preprint arXiv:2405. 16640, 2024.
Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., and Jitsev, Reproducible scaling laws for con- trastive language-iman image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2829, 2023.
Bain, M., Nagrani, A., Varol, G., and Zisserman, Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1728-1738, 2021.
Bradski, G.
The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
Delétang, G., Ruoss, A., Duquenne, P., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchi- son, M., Orseau, L., Hutter, M., and Veness, J. Language modeling is compression. In ICLR, 2024.
Caffagni, D., Cocchi, F., Barsellotti, L., Moratelli, N., Sarto, S., Baraldi, L., Cornia, M., Cucchiara, R., et al. The rev- olution of multimodal large language models: survey. Findings of the Association for Computational Linguis- tics: ACI 2024, 2024.
Du, R., Li, N., Jin, J., Carney, M., Miles, S., Kleiner, M., Yuan, X., Zhang, Y., Kulkarni, A., Liu, X., et al. Rapsai: Accelerating machine learning prototyping of multimedia applications through visual programming. In Proceed- ings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-23, 2023.
Chen, D., Huang, Y., Ma, Z., Chen, H., Pan, X., Ge, C., Gao, D., Xie, Y., Liu, Z.,, Gao, J., Li, Y., Ding, B., and Zhou, J. Data-Juicer: A one-stop data processing system for large language models. In International Conference Management of Data (SIGMOD), 2024a.
Fan, W., Si, C., Song. J., Yang, Z., He. Y., Zhuo, L., Huang, Z., Dong, Z., He, J., Pan, D., et al. Vchitect-2.0: Parallel transformer for scaling up video diffusion models. arXiv preprint arXiv:2501.0844 2025.
Chen, D., Huang, Y., Pan, X., Jiang, N., Wang, H., Ge, C., Chen, Y., Zhang, W., Ma, Z., Zhang, Y., Huang, J.,
10

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
Fu, C., Chen, P., Shen, Y., Qin, Y., Zhang, M., Lin, X., Yang, J., Zheng, X., Li, K., Sun, X., et al. MME: A comprehen- sive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.1333 13394, 2023.
Jiao, Q., Chen, D., Huang, Y., Li, Y., and Shen, Y. En hancing multimodal large language models with vision detection models: An empirical study. arXiv preprint arXiv:2401. 2024.
Gadre, S. Y., Ilharco, G., Fang, A., Hayase, J., Smyrnis, G., Nguyen, T., Marten, R., Wortsman, M., Ghosh, D., Zhang. J., Orgad, E., Entezari, R., Daras, G., Pratt, S. M., Ramanujan, V., Bitton, Y., Marathe, K., Mussmann, S., Vencu, R., Cherti, M., Krishma, R., Koh, P. W. Saukh, O., Ratner, A., Song, S., Hajishirzi, H., Farhadi, A., Beau- mont, R., Oh, S., Dimakis, A., Jitsev, J., Carmon, Y., Shankar, V., and Schmidt, L. DataComp: In search of the next generation of multimodal datasets. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Jiao, Q. Chen, D., Huang, Y., Ding, B., Li, Y., and Shen, Y. Img-diff: Contrastive data synthesis for multimodal large language models. In CVPR, 2025.
Kuaishou. Kling ai: Next-generation ai creative studio. com/ 2024.
Lhoest, Q. Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, I.., Davison, J., Saško, M., Chhablani, G., Malik, B., Brandeis, S., Le Scao, T., Sanh, V., Xu, C., Patry, N., McMillan-MMaor,, A., Schmid, P., Gugger, S., Delangue, C., Matussière, T., Debut, L.,, Bekman, S., Cistac, P., Goehringer, T., Mustar, V., Lagunas, F., Rush, A., and Wolf, T. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
Ge, C., Ma. Z., Chen, D., Li, Y., and Ding, B. Data mixing made efficient: A bivariate scaling law for lan- guage model pretraining. 2024. httl/httts//arttt/
org/abs/24C0 14908
Harper, E., Majumdar, S., Kuchaiev, O., Jason, L., Zhang, Y., Bakhturina, E., Noroozi, V., Subramanian, S., Nithin, K., Jocelyn, H., Jia, F., Balam, J., Yang, X., Livne, M., Dong, Y., Naren, S., and Ginsburg, B. NeMo: toolkit for Conversational AI and Large Language Models. URL https://githui
Li, J., Feng, W., Chen, W., and Wang, W. Y. Reward guided latent consistency distillation. Transactions on Machine Learning Research, 2024a.
com WWI.//VIIIANNNM
He, J., Xue, T., Liu, D., Lin, X., Gao, P., Lin, D., Qiao, Y., Ouyang, W., and Liu, Z. VEnhancer: Generative space- time enhancement for video generation. arXiv preprint arXiv:2407.076hh 2024a.
Li, Y., Zhang, Y., Wang. C., Zhong, Z., Chen, Y., Chu, R., Liu, S., and Jia, J. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv:2403 240..2884..81 2024b.
He, M., Liu, Y., Wu, B., Yuan, J., Wang, Y., Huang, T., and Zhao, B. Efficient multimodal learning from data-centric perspective. arXiv preprint 153. 2024b.
Liu, M., Chen, D., Li, Y., Fang, G., and Shen, Y. Chart- thinker: A contextual chain-of-thought approach to opti- mized chart summarization. Proceedings of the Inter- national Conference on Computational Linguistics (COL- ING), 2024a.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, 7., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1,2)33 2022.
Huang, Z., He, Y., Yu, J., Zhang, F., Si, C., Jiang, Y., Zhang, Y., Wu, T., Jin, Q.. Chanpaisit, N., et al. Vbench: Compre- hensive benchmark suite for video generative models. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., Hc, C., Liu, Z., et al. Mmbench: Is your multi modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024b.
Hwang, J., Hira, M., Chen, C., Zhang, X., Ni, Z., Sun, G., Ma, P., Huang, R., Pratap, V., Zhang, Y., et al. Torchau- dio 2.1: Advancing speech recognition, self- learning, and audio processing components for pytorch. In 2023 IEEE Automatic Speech Recognition and Under- standing Workshop (ASRU), pp. 1-9. IEEE, 2023.
Long, L., Wang, R., Xiao, R., Zhao, J., Ding, X., Chen, G., and Wang, H. On L.LMs-rrvee synthetic data generation, curation, and evaluation: survey. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 11065-11082, 2024.
Manduchi, L., Pandey, K., Bamler, R., Cotterell, R., Däubener, S., Fellenz, S., Fischer, A., Gärtner, T., Kirch- ler, M., Kloft, M., Li, Y., Lippert, C., de Melo, G., Nalis- nick, E.,, Ommer, B., Ranganath, R., Rudolph, M., Ullrich,
Jakubik, J., Vössing, M., Kühl, N., Walk, J., and Satzger, G. Data-centric artificial intelligence. Business & Informa- tion Systems Engineering, pp. 1-9, 2024.

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
K., den Broeck, G. V., Vogt, J. E., Wang, Y., Wenzel, F., Wood, F., Mandt, S., and Fortuin, V. On the challenges and opportunities in generative ai, 2024.
video generation. In Deep Generative Models for Highly Structured Data, ICLR 2019 Workshop. 2019.
von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lam- bert, N., Rasul, K., Davaadorj. M., Nair, D., Paul, S., Berman, W., Xu, Y., Liu, S., and Wolf, T. Diffusers: oftte-of-hhear diffusion models. https://githhbb com/hugg ng fa ce di fusers 2022.
MMagic Contributors. MMagic: OpenMMLaa multi- modal advanced, generative, and intelligent creation
toolbox. https:///ithhbe wtt. caam//ppen-eee mmmmmmmmmmm alo mmagic 2023.
OpenAl. Hello gpt-4o. https://openai.com com/
Wang, Y., He, Y., Li, Y., Li, K., Yu, J., Ma, X., Li, X., Chen, G., Chen, X., Wang, Y., et al. Internvid: A large- scale video-text dataset for multimodal understanding and generation. In The Twelfth International Conference on Learning Representations, 2023.
index/hello-gpt-Be 40/ 2024a.
OpenAl. Sora: Creating video from text. https://
openai wwm/soraaaaooooooooooooooo..oooe 2024b.
Pan, X., Chen, Y., Chen, Y., Sun, Y., Chen, D., Zhang, W., Xie, Y., Huang, Y., Zhang, Y., Gao, D., Li, Y., Ding, B., and Zhou, J. Trinity-rft: A general-purposs and uni- fied framework for reinforcement fine-tuning of large language models, 2025. URL hattps://arxiv.
Wang, Y., Chen, W., Han, X., Lin, X., Zhao, H., Liu, Y., Zhai, B., Yuan, J., You, Q., and Yang, H. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal reasoning. 2024a.
abs/ 2205/250..0 17826
Wang. Y.. Li, K., Li, X., Yu, J., He, Y., Chen, G., Pei, B., Zheng, R., Wang, Z., Shi, Y., et al. Intervideo2:: Scaling foundation models for multimodal video understanding. In European Conference on Computer Vision, pp. 396- 416. Springer, 2024b.
Peng, Y., Tian, J., Yan, B., Berrebbi, D., Chang. X., Li, X., Shi, J., Arora, S., Chen, W., Sharma, R., et al. Re- producing whisper-style training using an open-source toolkit and publicly available data. In 2023 IEEE Auto- matic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8. IEEE, 2023.
Wang. Y., Chen, X., Ma, X., Zhou, S., Huang. Z., Wang, Y., Yang, C., He, Y., Yu, J., Yang, P., et al. Lavie: High- quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, 133 (5)309993 2025.
Qin, Z., Chen, D., Zhang, W., Yao, L., Huang, Y., Ding, B., Li, Y., and Deng. S. The synergy between data and multi-modal large language models: A survey from co- development perspective. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2025.
Ward Jr. H. Hierarchical grouping to optimize an objective function. Journal of the American statistical association, 58(301):236-22) 2244224,, 1963.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Weber, M., Fu, D., Anthony, Q., Oren, Y., Adams, S., Alexandrov, A., Lyu, X., Nguyen, H., Yao, X., Adams, V., et al. Redpajama: open dataset for training large language models. Advances in neural information pro- cessing systems, 17:116462-11699 2024.
ML. Introducing gen-3 alpha: A new frontier for video generation, October 2024.
URL https: /ttp:s/y///uwwymmmm. com/ research/ rnducing gen 3 alpha
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Transformers: State-of -hce-of-thhearrt art natural language processing. In Proceedings of the 2020 conference on em- pirical methods in natural language processing: system demonstrations, pp. 38 38-44,115 2020.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 8317-8326, 2019.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Song, Y., Dhariwal, P., Chen, M., and Sutskever, Consis- tency models. In International Conference on Machine Learning, pp.
PMLR, 2023.
Xu, H., Xie, S., Tan, X., Huang, P.-Y,, Howes, R., Sharma, V., Li, S.WW., Ghosh, G., Zettlemoyer, L., and Feicht- enhofer, C. Demystifying CLIP data. In The Twelfth
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. FVD: A new metric for
12

---

A Feedback-Drive -Driven Suite for Multimodal Data-Model Co-devclopment
International Conference Learning Representations, 2024a.
Xu, J., Mei, T., Yao, T., and Rui, Msr-vtt: A large video description dataset for bridging video and language. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 2288-5296, 2016.
Xu, J., Zou, X., Huang, K., Chen, Y., Liu, B., Cheng, M., Shi, X., and Huang, J. Easyanimate: high-performance long video generation method based on transformer ar- chitecture. arXiv:2405. 18991, 2024b.
Xu, M., Yin, W., Cai, D., Yi, R., Xu, D., Wang, Q.. Wu, B., Zhao, Y., Yang, C., Wang, S., Zhang, Q. Lu, Z., Zhang, L., Wang. S., Li, Y., Liu, Y., Jin, X., and Liu, X. A survey of resource-efficient IIm and multimodal foundation models, 2024c.
Xu, P., Zhu, X., and Clifton, D. Multimodal learning with transformers: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(10):12113-12133 2023. doi: 00.109/TPAML.20333355111
Yang. Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J.. Yang. Y., Hong, W. Zhang, X., Feng, G., et al. Cogvideox: Text-to-video diffusion models with an ex- pert transformer. arXiv preprint arXiv:2408.06072 2024.
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., and Chen, E. survey on multimodal large language models. National Science Review, 11(12), 2024.
Zhang, D., Yu, Y.. Dong, J., Li, C., Su, D., Chu, C., and Yu, D. MM-LLMs: Recent advances in multimodal large language models. In Findings of the Association for Computational Linguistics ACL 2024, pp. 12401-12430, 2024.
Zhou, P., Wang, L., Liu, Z., Hao, Y., Hui, P., Tarkoma, S., and Kangasharju, J. survey on generative Al and LLM for video generation, understanding, and streaming, 2024a.
Zhou, T., Chen, D., Jiao, Q. Ding, B., Li, Y., and Shen, Y. Humanvbench: Exploring human-centric video un- derstanding capabilities of milms with synthetic bench- mark data, 2024b. URL/https:/ //tp:i////ivv./// org/abs/
2412 17574
13

---

A Feedback-Dring Suite for Multimodal Data-Model Co-devclopment
Appendix
A. Discussion on More Related Works
Table of Contents
Model-Centric Progress in Multimodal Large Models. Multimodal large models have captivated researchers with their formidable capabilities (OpenAI 2024a b). leading to a surge in model-centric development efforts. These fo-
Discussion on More Related Works
14
cuses mainly lie in refining training algorithms (Caffagni
Sandbox Suite from Infrastructure Perspective
15
ct al. [224:] Li ct al., 2024a Zhang et al. 2024), advancing
B.
1
Overview Architecture
15 model architectures and components (He et al.
2024a Yin
ct al.
2024 Tiao et al.
2024). and harnessing the models'
B.2 Flexible Capability Factory and Behavior
potential for various applications (Wang et al.
2024a Liu
Hooks
16
ct a
2024a Zhou al. 2024a). There is a growing con-
B.3
Extensibility of the Sandbox Suite
16
sensus that transformer- based scaling is predominant (Xu)
ct al. 2023). However, the high computational requirements imposed by scaling laws (Xu et al., (024c) and the optimiza- tion challenges inherent to large models (Manduchi et al., 2024) often confine insights to specific datasets or vague data characteristics. This situation leaves a significant gap in comprehending the extent to which models' performance and behavior hinge upon implicit assumptions and inductive biases embedded within the underlying data distributions. In contrast, our work demonstrates feasible and promising path to fill in this gap by explicitly linking data processing effects with the downstream performance of trained models through numerous contrastive sandbox experiments.
Analysis on the Costs of the Proposed Workflow
16
Impact of Sampling Ratio r for Data Pools
16
C.2 Impact of the Number m of Planned Small-
Pool Experiments
17
D Implementation Details of Sandbox Experiments
17
D.I Strategies to Combine OPs According to Correlations
17
D.2 Evaluation Metries
17
D.3 Descriptions of Studied Operators
18
Trends in Data-Centric Development for Multimodal Large Models. An emerging trend shifts the focal point from models to data (Jakubik et al. 2024] Bai et al.) 2024
D.4
Image-to- Generation
18
D.5
Text-to-Video Generation
18
Chen et al., 2024c Tiao et al. 2025). ) underscored by the notion that large models function akin to data compressors Delétang et al. 2024). Echoing the principle of "garbage in, garbage out", meticulous data processing is recognized as pivotal. Efforts now isolate data manipulation as a primary experimental variable multimodal generative modeling (He et al.] (024b). Nonetheless, multimodal data processing involves highly heterogeneous processing workflows, vast quantities, diverse types, and the high cost of training down- stream models. This complexity results in predominantly heuristic approaches, such as data filtering and synthesis guided by human intuition (Long al. 2024 2024b).
D.6
Image-Text Pre-training
20
E
Additional Experimental Results
20
E.1
Summarization of Sandbox Experiments
varying Multiple Scales).
20
Complete Opprrtor Ranking
20
Diversity Analysis
23
E,4
Correlation Analysis
23
Zhou et al.)
E.5
Recipes Based on Correlation Analysis
25
For example, one well-studied model type is CLIP. Data- Comp (Gadre et al. 2023) introduces benchmark to filter out high-quality data from 12.8 billion image-text pairs in Common Crawl to train better CLIP models, considering Filter Operators such as CLIP score, image size, and cap- tion length. MetaClip (Xu et al., 2024a) aims reproduce CLIP's data by introducing raw data pool and finding balanced subset based on CLIP's concepts. Unlike these data-centric approaches that isolate the model and training settings, focusing solely on the quality and seale of training datasets, our work emphasizes systematic methodologies for data-model co-development, considering both data and
E,6
6
Detailed Evaluation Results on MGM
26
E.7
7
Ablation Study Based on T2V-Turbo
26
Full Results on VBench leaderboard
27
E,9
Cost Comparison of all Sandbox Experiments
28
E.
10
Iteaative Workflows in Sandbox
30
E.
Operators beyond Filters
30
12
Potential Model Development: Automated
Prompt Adjustments
31
14

---

A Seedback-Driver Suite for Multimodal Data-Model Co-devclopment
End-to-end Workflows (Ordered Job List) for Data-Model Co-development
Data
Model
Data/Model Probe
Data Recipe Refinenment
Data/Model Execution
Evaluation
Model
Genetic Behaviors (Hook Functions)
Hook Process Data via Filters
Haok
Hook Predict Data Quality Scores
Hook
Hook
Hook Train Model
Hook Evaluate Model
Hook
Refine Recipe via rule
Analyze
Analyze Metric
Correlation
Gradients
Foundational Capability (Factory Classes)
DataProcessorfactory
DataEvaluatorFactory
MccelTTainxxecutoraaccom
ModelEvaluatorFactory
Fitteing Oparators
Data-jurcer k-o Tool
Cata luicer Analyzer
GPT 4V Evalustor
LLaVA-like
Sora- like Traintxtoutoor
MM-Benc:
V-Bencch Evallusnrr
Figure 4: Overview of the Data-Juicer Sandbox Laboratory. The workflow involves four stages, each allowing flexible customization different levels within the data-model co-development lifecycle. More details can be found in Sec.
models equally important. Specifically, incorporate per- formance signals from sandbox reference models many downstream tasks, conducting importance and correlation analysis to link data pools and these model metrics. Addi- tionally, we explored more model types beyond CLIP, such as LLaVA-lit Wike and DiT-based models for image-to- and text-to-vides tasks, and identified better training datasets for these models using workflow.
out by introducing innovative intermediary layer. We seamlessly integrate cutting-edge model-centric multimodal infrastructure with the Data-Juicer system. This integration fosters a streamlined and insightful co-development envi- ronment for both models and data, bridging the current gap and setting a standard for future efforts in the multimodal domain.
B. Sandbox Suite from Infrastructure Perspective
Open-Source Infrastructure for Multimodal Large Model Development. The landscape for multimodal model development has advanced significantly, boasting variety of strong infrastructures and frameworks for training and evaluation. Prominent examples include Transformers (Wolf et a 2020)) Diffusers (von Platen et al. 2002)) NeMo (Harper et al.) MMagic (MMagic Contributors] 2023), ES-
B.1. Overview Architecture
To support co-development based on feedback from data and model, we design the layered sandbox laboratory as illustrated in Fig. 4 The laboratory incorporates spec- trum of components for activities such as data analysis, filtering, recipe optimization, model training, inference, and evaluation, all orchestratable via configuration files. The architecture stratified into three tiers: bespoke end-to- end workflows, generic development behaviors. and founda- tional data-moda development capabilities.
PNet (Peng et al. 2023), and MMBench (Liu et al. 2024b).
However, when comes to multimodal data infrastructure, the primary contributions have been datasets and dataset- specific preprocessing tools, such DatasetsHub from Hug- gingFace (Lhoest et al. 202 The standardization and effi- cient utilization of practical expertise and foundational data processing capabilities remain unaddressed. Existing frame- works for data processing predominantly focus on single- modal data (Weber et al., 2024| Bradskii 20000) Hwang et al.] 2023), underlining the early stages of development in sys- tematic platforms for multimodal data (Chen et al.) 2024a
The top tier delineates co-development workflows executed sequentially across four phases: probing data/models, refining data recipes, executing data/model operations, and evaluation. The sequence of tasks within each phase is adjustable through an input configuration file, permitting users to leverage pre-established and effect-proven workflows customize their own with ease.
Du et al. 2023 Chen et al. 2024b).
Recognizing the critical interplay between datasets and mod- els where comprehensive, high-quality datasets enhance model performance, and advanced models contribute to the generation of even more refined datasets- our work stands
Moreover, users can flexibly introduce or innovate classes at the capabilities level (such as novel models, metrics, or data processing algorithms) and behavior hooks (like mul-
15

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
tidimensional data quality assessments and adaptive ad- justments based multiple probe outcomes) interchange- ably.
pythen tools/sanddox_steee brrbbox_ttatterrre
--config configs htt.f/gaa//////////o//////////ae demo/ http://annddoxoooooo yami
This design allows for streamlined configuration and reuse of established infrastructure. Importantly, it expedites the
B.3. Extensibility of the Sandbox Suite
prototyping of data and model development solutions, in- tegrating actionable and measurable capabilities for swift feedback derivation and informed decision- decision-making. Con- crete explorations of the lower tiers are provided in Sec. [..] while Sec. 3.2 showcases the sandbox's utility in estab- lishing a probe-analyze-refine workflow for data-model co- development.
As a middleware, the sandbox itself does not impose any additional specific hardware dependencies. Instead, it in- herits the dependencies of the integrated underlying li- braries/frameworks and is continuously maintained to in- corporate more community efforts in model training and evaluation. To simplify dependencies and avoid redundancy in an "all-in-one" environment, we have introduced and employed lazy-loader mechanism at the Python package level. The code and parameters are provided and adjusted to ensure stable and cost-controleed experiment environment, where, for example, a single GPU card can complete an end- to-end workflow within one day, enabling rapid feedback.
B.2. Flexible Capability Factory and Behavior Hooks
Actionable Perspective. Within the sandbox, we have created various factory classes and behavior hooks for data processing. These simplify and unify the interfaces offered by the open-source system, Data-Juicer. Users can utilize over 100 OPs and tools for data analysis, filtering, and synthesis. The toolkit automatically speeds up and scales up the processing through system optimization and parallel support. Classes for refning data recipes leverage tools like adjusting percentile distributions to distinguish data subsets or applying k-s k-sigma rules remove outliers.
Besides, the capabilities of its underlying libraries are still evolving. For example, the utilized Data-Juicer's opera- tors not limited to data filters investigated on the main page; they now encompass wide range of functionalities, including 100+ Mappers, Filters, Deduplicators, and Selec- tors. This diversity allows for research on various types of data processing utilities within the Sandbox. Since Mappers can be employed to examine the effects of data augmenta- tion and editing on downstream model task performance, which reserved further exploration beyond filters as fu- ture work.
Besides, model development classes integrate SOTA open- source libraries, streamlining interfaces and enhancing us- ability for rapid and user-friendly development experiences. For example, models can be easily trained with diverse func- tionalities, such as Mini-Geming for image-to-text, EasyAni- mate and T2V-Turbo for text-to-video, and ModelScope for general generative models.
C. Analysis on the Costs of the Proposed Workflow
In this section, we give a more detailed discussion on the factors presented in the main page Sec. 3.2.4
Measurable Perspective. The sandbox also provides many classes to encapsulate observational capabilities, abling subsequent optimization. For example, Data-Juicer- based classes can efficiently compute metrics such as text perplexity, video aesthetic value, and image quality scores with GPT API calls, along with statistical values like mean, variance, and percentiles. For models, various evaluation benchmarks are supported, like VBench and FVD (Un- terthiner et al., 2019) for synthesized video assessment, and TextVQA (Singh et al. 2019). MMBench (Liu et al.) 2024b), and MME (Fu et al. 2023) for image-to-text evaluation.
C.1. Impact of Sampling Ratio r for Data Pools
In this section, we give some theoretical discussion to demonstrate the rationality of why scaling data pools can work in a cost-effective way.
Following the notations defined in Sec. 3.2.4 the probability that the error introduced by our experiments on the small- scale data pools is greater than e can be denoted as:
More detailed developmental information can be found with the source code and accompanying documentation our open-soureed link, which are also continuously maintained online and evolved. For example, users can conveniently run the following one-line entry script to easily run the sandbox and flexibly adjust configurations to switch between many built-in hooks and capabilities.
PIA Pooo B|A 2 c].
It is worth noting that our sandbox aims to obtain effective feedback with minimal expenditure. This feedback is often less apparent than what is obtained from the full dataset and larger model, meaning that EiA Full 2 Ep]Apoo For
operators that have positive effect, we have R E f 2
16

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
p|Apoot] In this case, we can conclude that:
accommodates two strategies, with specific applications detailed in Sec. E.4
P A Ppot - EiAsurl 2 cl S P(A pool - D[Apood 2 c
S e e ((baa)
The first method involves computing Pearson correlation coefficients between the statistics generated by these OPs. Using a hierarchical clustering algorithm (Ward Jr 1963). we group the OPs into clusters. From each cluster, we select the OP whose data pool yields the strongest model performance to form potential combinations.
Here we use Hoeffding's s inequality with the assumption that Apoot E ba, As the sample rate increases, the variance of the improvement across different training trials decreases, which is positively related to (b a)². and thus the right term decreases. In conclusion, the probability of the enror exceeding e decreases exponentially as r increases.
Alternatively, leveraging the outcomes of single- -Pte--OP tests, we calculate Pearson correlation coefficients for each pair of dimensions within the evaluation metrics. Hierarchical clustering is again employed to categorize the metrics into k classes. The top- pop-performing OP from each class is cho- sen to create the combinations. This approach allows us to investigate whether these combinations lead to concurrent improvements or mutual inhibition across the evaluative metrics.
C.2. Impact of the Number m of Planned Small-Pool Experiments
In the proposed sandbox workflow as illustrated in Sec. 3.2 we choose to split each data pool after applying our studied Data-Juicer OPs into three buckets according to the ranks of their generated statistics. The number The number of buck- ets acts multiplying factor determine the total number of the planned small-pool experiments m, and finally the total cost of our experiments. It reflects the trade-off be- tween total cost, data intervention intensity (via operator stats), and the informativeness of model feedback (the met- ric changes A on interested tasks compared to the models trained on random sampled data pools). More buckets lead to greater statistical differences between buckets with differ- ent ranks (especially the first and last ones), strengthening attribution to data processing effectiveness. However, more buckets also reduce per-bucket data, increasing the risk of inadequate data for models to exhibit reasonable A.
D.2. Evaluation Metrics
In the paper, we mainly report overall performance as the relative changes over the baseline in terms of the average across all model metrics with normalization as follows:
MN s S EE A/N Σ S/N
Σ. (st sf) Σ 1
where N is the number of involved metrics, s; is the score of i-th model measurement metric, of is the corresponding score gained by the baseline model trained on randomly sampled data. Below are the specific evaluation metrics involved in this study.
As result, we do not aim for models to be "training done" or "converged" in the sandbox workflow experiments. In- stead, we want to observe enlightening changes- Mosiiis-positive or negative-in models after targeted data intervention ver- sus random data sampling. In our early experiments, we tested bucket counts of 5,5) to evaluate whether a model trained on randomly sampled data could reasonably de- crease loss after one epoch and show statistically significant changes on downstream tasks. Our findings indicate that three buckets are empirically good for the studied scenarios. Once determined, all controlled experiments aligned to one complete epoch and matched to the random pool data size.
Vex,VQA, MMBench, MME. These benchmarks serve as critical evaluators of MLLM's proficiency in understanding images. TextVQA (Singh et al. 2019) specifically targets the assessment of MLLMMs abilities to read and reason about textual content embedded within images. MMBench (Liu) et al. 2024b). a vast multimodal benchmark, encompasses perception and reasoning skills through a plethora of multi- choice questions, numbering in the thousands. Addition- ally, Chinese translation, MMBench-CN, is integrated for broader accessibility. MME (Fu et al., 2023) focuses on the perceptual and cognitive competencies of MI.LMs, incorporating 14 finely categorized subtasks, each address- ing Yes/No inquiries underpinned by meticulously crafted guidelines.
D. Implementation Details of Sandbox Experiments
VBench. We engage with VBench (Huang et al. 2024). )
D.I. Strategies to Combine OPs According to Correlations
holistic benchmark suite tailored for the rigorous evalua- tion of video generative models. It facilitates granular and objective assessment across a spectrum of dimensions, de- constructing the concept of "video generation quality" into 16 discrete metrics. Each metric is assessed using a carefully
In addition to assembling the OPs with the overall best per- formance, we also incorporate an analysis of inter-OP rela- tionships into our recipe formulation process. Our workflow
17

---

A Feedback-Driver Suite for Multimodal Data-Model Co-devclopment
curated suite of prompts, comprising 946 unique prompts, with the requirement to produce 5 videos per prompt.
For multi-OP data pools (Sec. we identify the top-3 highest-perfone perffoming OPs from single-OP experiments and study their possible combinations. Additionally, we analyze the correlations among the 23 data statistics produced by 22 OPs capable of generating instance-level stats 2 Fm- ploying a hierarchical clustering algorithm (Ward Jr 1963), these OPs are grouped into three clusters based on correla- tion coefficients, with the Highest-performing OP from each cluster selected for combination testing. To ensure a robust and fair comparison, we must acknowledge the constraints imposed by the limited data volume within the highest-tier data pool. As the number of combinations increases, the available dataset size diminishes. In particular, the size of Ps was reduced from 200k samples to 159k samples during the Top-3 combination experiments. Similarly, in the cluster-wise combination experiments, the dataset size decreased from 200k samples to 126k samples.
Owing to the disparity in evaluation criteria and the inherent variability across different modalities, we discern that the magnitude of performance fluctuation in the image-to-text generation substantially exceeds that observed in the text-to- video generation. This discrepancy underscores again the need for nuanced data-model co-development in addressing the complexities inherent in each modality.
Across all experiments, results are reported as averages with standard deviations from 2 to 5 repetitions for the image- to-text and text-to-video tasks, respectively, due to their differing levels of variance.
Metries for Image-Texx Text Pre-training task. For the CLIP experiments, we adopt 40 distinct benchmark scores used in DataComp (Gadre et al., 2023), e.g., the image classifica- tion on 30 data subsets for diverse scenarios like ImageNet derivatives, the image-text retrieval on 3 data subsets like Flicker and MSCOCO, the fairness related classification task like Dollar Street and GeoDE. For more details about these tasks, please refer to the DataComp and their original papers.
Next, in Sec. 4.4 we explore the optimal OP combination based on previous experiments and adopt the methodology from Sec. 3.2.3 for comparative experiments on training with repeated data versus non-repeated data. Note that due to filtering, the final instance count decreases from 200k to approximately 159k after the OP combination. These data are then repeated in increments from double to eightfold, mirroring the size of the original pretraining set.
D.3. Descriptions of Studied Operators
The study involves 31 OPs from Data-Juicer (Chen et al.] 2024a). Their corresponding statistics and detailed descrip- tions provided in Table 4
Collectively, all these experiments yield profound insights into image-to-text model training, data processing, and iter- ation strategies from data-mode co-development perspec- tive, further verified in the larger-scale scenario in Sec. 4.5
D.4. Image-to-Text Generation
In terms of the model training details, we train the MGM- 2B model from scratch with less training data (about 1/6 of the original training datasets) in baseline experiments to make sure each experiment can be finished within one day. We keep every training setting (e.g. learning rate scheduler, global batch size) the same as the original model except for training datasets and training devices. For single-OP and OP combination experiments are trained only A100 GPU for each experiment so we increase the number of gradient accumulation steps from 4 to 32 to keep the same global batch size. For experiments of duplicating high-quality datasets, A100 GPUs are involved to train the model, and the number of gradient accumulation steps is restored to 4. Each experiment repeated 3 times with different random seeds to make the final results more comprehensive.
Our first task focuses foundational image understand- (MGM-2B), state-of-the ahrt----ttt--tttt- (SOTA) 2 billion parameter multimodal LLM (Li et al. 2024b). The training protocol for MGM-2B involves two stages: pretraining and fine-tuning. Our ex- perimental focus lies the pretraining phase, which seeks to harmonize visual and textual representations. We uti- lize the original pretraining dataset as our original dataset D, consisting of approximately 1.2M instances. We set the size of D sample as 200k. The single-OP data pools D; and multi-OP data pools Ds are capped at maximum of 200k instances, ensuring consistency of data pool size. To mathh the down-sampling rate used during pretraining, the fine-tuning dataset is sampled into a 240k instance subset.
ing ability, by experimenting
We first conduct single-OP experiments (Sec. 4.2 that en- compasses 22 text-image relevant OPs from Data-Juicer, split evenly between text-only and image-related multi- modal OPs. After the two-stage training, model evaluation is conducted on established benchmarks including TeXtVQA (Singh et al., 2019). MMBench (Liu et al. 2024b), and
D.5. Vixt-oo-Video Generation
For the second task, text-to-video generation, we adopt the advanced DiT-based models, Easyanimate (Xu et al., 2024b), which originally integrates diverse datasets totaling
1.2M instances from Intern Vid (Wang et al. 2023) (606k),
MME (Fu et al. 2023).
Thhe image height and width are produced by one OP.
18

---

A Feedback-Driven Suite for Multimodal Data-Model Co-devclopment
Table Overview of involved OPs in the study, including the modality they pertain to, along with their statistical data and detailed descriptions of these statistics.
OP Name
Modality Statisties
Description
alphanumeric_filee character
text
Alphanameric Ratio
Alphanumeric ratio the text. Char-level n-gram repetition ratio in text.
text
Character
Repeti-
cepetitiio_fiiter
Rion Ratio
waraged_wordd___e filter
text
Flagged Word Ratio
Flagged-word ratio in the text
mmage_assthetics
image
Image
Aestheticy
Aesthetics score of the image
filteer
Score
image
Image Aspect Ratio
Aspect ratio of the image
_filter
image_nsf__filter
image
Image NSFW Score
NSFW score the image
image_shape_fllee filtee
image
Image Width/Height
Width and height of the image
image_size_illter
image
Image Size
Size in bytes of the image
image_text_matching
text,
BLIP Image-Text
Image-text classification matching score based BLIP model
_filter
image
Similarity
image_tes text
text.
CLIP
Image- Text
Image-texi feature cosine similarity based on CLIP model
similarityy
image
Similarity
image watermark
image
Image
Watermark
Predicted watermark score of the image based on an image classification model
_illter
Score
Language_id_se
text
Language Score
Predicted confidence score of the speci- fied language
_filter
perplexity_filter text
Text Perplexity
Perplexity score the text
phrase_grrunncing text,
Phrase Grounding
Locating recall of phrases extracted
_recall_filter
image
Recall
from text in the image
special_characters
text
Special Character
Special character ratio in the text
filter
Ratio
stopwords filter text arti_action___e filter
text
Stepword Ratio
Stopword ratio in the text
text
Text Action Number
Number of actions in the text
text_entity _dependency filtter filter
text
Entity Dependency
Number of dependency edges for an
Number
entity in the dependency tree of the text Length of the text
text_length_fill ffltee
text
Text Length
token_num_filtes
text
Token Number
Token number the text
videooaesthettics
video
Video
Aesthetics
Aesthetics score of sampled frames in the video
_filter
Score
Lideo_aspect_rart
video
Video Aspect Ratio
Aspect ratio of the video
_iitter
video_duration
video
Video Duration
Duration of the video
_filter
video_frames__ext
text,
Frames- -Txt Similar-
Similarities between sampled frames and text based CLIP/BLIP model Motion score of the video
_emilariity_fitter
video
ity
video_motion_score
video
Video Motion Score
filter
video_nsfw_filter
video
Video NSFW Score
NSFW score the video
videoo_orr_area_ee ratio
video
Video OCR-Area Ra-
Detected text area ratio for sampled frames in the video
filter
video_resolution
video
Video Width/Heightt
Width and height of the video
_filter
[ildco_watermar! video
Video
Watermark
Predicted watermark score of the sam- pled frames in the video based on an image classification model Number of words in the text
_filter
Score
worde_nun_filtee
Word Number
word_repetition
filter text
Word Repetition Ra-
Word-level n-gram repetition ratio in the text
tio
19

---

A Feedback-Driven Suite for Multimodal Data-Model Co-devclopment
Panda-70M (Chen al. 2024e) (605k), and MSR-VTT (Xu
cally, we utilize data from the small track of the DataComp competition (Gadre al. 2023) and adhere to its evaluation metries, which include 40 distinct evaluation subsets. Due to some broken links, we successfully downloaded 85.2% of the dataset, resulting in a total of 10.9 million samples as our D. All baseline models were trained on an equiva- lent volume of data as used the contrastive experiments, sampled randomly from this dataset.
al. 2016) (6k). The studied baseline model is trained
subset of 40k instances, employing LoRA (Hu et al. 2022)
for efficiency. As result, the size of D is 1.2M, and the size of 'D samples the single-OP data pools 2; and multi-OP data pools Ds are all 40k. Model outputs are assessed using VBench (Huang et al. 2024) across 16 metrics on video quality and video-text matchness.
Our investigation covered 21 OPs, including 13 text-only OPs and 10 video-related multimodal OPs. Analogous to the image-to-texx generation, we conduct single-OP and multi-OP combination experiments, in Sec. S.2. respectively. However, given the reduced relevance of data statistics in video-related OPs, our analysis centers on the correlations among the 16 VBench evaluation met- rics. These metrics are clustered into three groups, with the best-performing OP selected from each group.
In the multi-OP experiments (see Fig. 2() due to data limitations in the top-level data pool, the training dataset size is reduced to 880k for the top recipes. Correspondingly, the default computation scale of DataComp is adjusted to 0.25 expansion rate for the subsequent compute scaling experiments (see Fig. 3())
and and Sec. 4.3
E. Additional Experimental Results
E.1. Summarization of Sandbox Experiments varying Multiple Scales
Through OP combination experiments, we pinpoint the most effective set of OPs. In Sec. 4.4. we then sample 40k in- stances from the filtered data pool and repeat the training process for up to 10 epochs. For comparative analysis, we adhere to the method outlined in Sec. 3.2.3] selecting larger data volumes (80k, 120k,... .... up to 400k instances) for single- epoch training. To examine the effectiveness of the de- rived insights for text-to-vided data-model co- co-development, we finally incorporate them into larger-scale scenarios in Sec.
The Table provides an overview of our sandbox experi- ments on image-to-tex generation, text-to-video generation, and image-text pre-training tasks.
E.2. Complete Single-Operator Ranking
In Table 6 Table 7 and Table g we present complete numeric results conducted on individual OP experiments (Sec. 4.2), from which we can discern some more detailed observations.
In terms of the model training details, we adopt the advanced DiT-based FasyAnimate (Xu et al., 2024b) model, which integrates diverse datasets totaling 1.2M instances from In- tern Vid (Wang et al. 2023) (606k). Panda-70M (Chen et al.) 2024c) (605k), and MSR-VTT (Xu et al.) 2016) (6k). Base- line experiments are executed on a subset of 40k instances, employing LoRA (Hu et al. 2022) for efficiency. During training, we maintain a video resolution of 256x256, sample every other frame, and randomly select sequences of 16 consecutive frames. The training process involves perform- ing backward pass for the loss of every 8 samples, with single-OP and OP combination experiments trained on a single GPU with a batch size of 8 for 5k steps, amounting to approximately 16 GPU hours per training run. Experiments for duplicating high-quality data, as well as larger-scale training, are conducted with a batch size of I across 8 GPUs The models employ the Adam optimizer for training, with learning rate set to 2 x 10-5, weight decay parameter at 3 x 10-2, and epsilon configured to 10-10. Each exper- iment is repeated twice with random seeds of 42 and 45, respectively.
In image-to-text generation, it is preferable for the input of training images to align as closely as possible with the original configuration of the vision tower, such as training dimensions (height, width, and sizes). Additionally, CLIP similarity scores tend to be more influential than BLIP sim- ilarity scores. The BLIP similarity does not show much distinction and paradoxically, a lower similarity often re- sults in better performance, which defies common sense. Images with excessively high aesthetic quality may offer limited assistance in feature alignment, while watermarks might have certain impacts on the OCR performance of the model.
In text-to-viee -oieeoo generation, having a consistent aspect ratio for the training data is better than having ratios that are inconsistent but close to the 1:1 ratio used during training. For instance, a data pool with a 'middle' video aspect ratio consistently at 16:9 performs optimally. Videos with high video aesthetics scores and low video NSFW scores, as well as those with low video OCR-area ratios and high video motion scores, tend be of higher quality. While single-text-related operators might not be as critical in text- to-video generation, they can still effectively filter out some dirty data.
D.6. Image-Text Pre-training
For the third task, image-text pre-training, we adopt the sell-studied CLIP model (Radford et al. 2021). Specifi-
20

---

A Sredback-Driver Suite for Multimodal Data-Model Co-devclopment
Table 5: Overview of sandbox scenarios and their effective evidence across different scales.
Sandbox
Image-to-Teds Genera- tion
Text-to-Vide
Genera-
Image- Text Pretraining
Image Captioning Fine- tuning
Scenario
tion
Main Effec-
Optimal recipe derived from small data pools achieves superior model performance in larger data pools, using only half of baseline's 's compute cost.
Insights from small data pools results in VBench- Top1 model, transferring across data size, model scales, and model archi- tectures.
Best recipe identified in the model with fewer maintains
Optimal recipe identi- fied with Vntem0V--2.0 model, leads to con- sistent and steady per- formance advantages as the model scale in- creases from 1B 26B.
tiveness Ev-
idence
FLOPS optimal performance with increased model FLOPS and compute resources, showing clear scaling behaviors in power-law form.
Model
MGM-2B in all experi- ments
EasyAnimate to T2V- Turbo (Heterogeneous architec- tures)
CLIP: ViT-B-32 to ViT- B-16
Intern VL-2 with IB, 2B, 4B, 26B parameters
Scale
Range
(Different FLOPS)
Data Scale
126k 200k
40k 147k and 228k
880k 2,683k
24k to 189k
Range
(w.r.t
Distinct
Dataset
Size)
Compute
1 to Epochs
1 to 10 Epochs
to 14 Epochs
epoch in all experi- ments
Scale
Range
(w.nt Num- of Trained
ber
Sample)
21

---

A Feedback-Drive Suite for Multimodal Data-Model Co-devclopment
Table 6: The complete OP ranking, including their statistical dimensions and the improvements relative to the baseline. We consider three splits with low, middle, and high statistical values for each OP. The baseline used is based on random sampling with equal data volume.
Average Performance Changes (%)
Task
OP- OP-GGeerrtee Statistics
Data Pool (Low)
Data Pool (Mid)
Data Pool (High)
Image NSFW Score
7.13
18.44 + 18:45
66.38
Text Action Number
59.90 + 46.49
0.29 2.16
-2,05 = 2.48
Language Score
49.90 53.82
0.85 2.87
-1.43 + 2.40
CLIP Image-Tett Text Similarity
1,20 + 4.86
-1.81 2.88
49.81 44.72
Phrase Grounding Recall
-0.49 +: 3.87
-0.58 = 6.12
49.39 29.83
Image Width
42.04 57.27
10.31 + 12.59
1.35 4.36
Special Character Ratio
-3.08 + 0.63
-0.75 = 1.61
39.67 58.82
Flagged Word Ratio
38.48 27.76
-0.39 0.43
22.49
Image Height
35.66 1 48.62
12.91 1 1044
18.73 27.32
Word Repetition Ratio
33.14
2.59 = 5.31
-0.55
Text Length
30.67 + 28.54
-0.44 = 0.73
-3.71 1 0.39
Image-to-Ter
Stopword Ratio
3.34 +
24.62 +: 36.73
-1.56
Image Size
0.76 + 0.55
19.16 + 27.29
1.58 2.20
Text Perplexity
-1,69 1.30
16.70 + 24.49
18.26 23,02
Image Aesthetics Score
11.94 + 12:21
16.58 + 25.70
0.16 3.67
Word Number
15.96 29.01
-2.48 1: 0.26
-1.97 1. 2.05
BLIP Image-Text Similarity
11.76 22.83
1.74 2.49
1.34 1. 2.21
Image Watermark Score
-1.50 2.41
7,51 1: 12.82
11.54 13.14
Alphamumeric Ratio
2.35 7.63
-0.66 0.69
8.71 12.87
Character Repetition Ratio
0.00 + 1.13
-1.42 = 0.60
7.94 = 14.63
Entity Dependency Number
1.35
-0.87
6.67
Token Number
6,31 + 7,86
0,80 0.92
0.33 6,45
Image Aspect Ratio
0.00
1.89
-0.02
Video Aesthetics Score
-0.98 0.08
0.13 0.09
0,96 0.13
Video NSFW Score
0.82 1. 0.36
-0.05 0.07
-0.57 1: 0.07
Frames-Text Similarity
-1.45 0.69
0.23 : 0.21
0.79 0.15
Ratio
0,54 0.36
-0.13 0.70
-0. 14 0.10
Token Number
0.53 + 0.04
0.18 = 0.32
0,41 + 0.25
Character Repetition Ratio
-0.29 0.27
0.47
0.18 + -0.52
Video Height
-0.10 +: 0.21
0.12 0.13
0.46 + 0.44
Video OCR-Area Ratio
0.44 +
0.02
-0.66
Word Number
-0.49 1: 0.07
-0.41 0.72
0.44 1: 0.45
Entity Dependency Number
0.40 + 0.01
0.28 = 0.48
00.88 + 0.44
Text-to- Text-to-Video
Text Action Number
0.18 + 0.56
-0.71 0.28
0.37 + 0.28
Alphanumeric Ratio
-0.10
0.20 0.19
0.33 0.17
Video Motion Score
-0.55 1: 0.40
0.33 0.1
0.32 4: 0.15
Video Watermark Score
-0,27 0.27
-0,25 = 0.25
0.29 0.16
Text Perplexiry
0.15 0.69
-0.13 0.27
0.09 0.56
Stopword Ratio
-0.01 1: 0.05
-0.48 0.22
0.12 + 0.07
Video Aspect Ratio
00.32 0.14
0.11
-0.02
Language Score
-0.11 + 0.01
-0.03 0.38
0.09 + 0.03
Word Repetition Ratio
0.00 0.17
0.06
-0.43
Video Duration
-0.58 1 0.05
-0.16 0.09
0.04 1 0.84
Text Length
-0.09 0.63
-0.66
0.03 0.22
22

---

A Feedback-Driver Suite for Multimodal Data-Model Co-devclopment
Table 7: The complete OP ranking, including their statistical dimensions and the improvements relative to the baseline for image-text mmillarty/cllssifictton task. We consider three splits with low, middle, and high statistical values for each OP. The baseline used based on random sampling with equal data volume.
Average Performance Changes (%)
OP-Generated Statistics
Data Pool (Low)
Data Pool (Mid)
Data Pool (High)
CLIP Tmage-Text Similarity
-32.57
-6.39
39.53
BLIP Image-Text Similarity
-24.28
1.82
25.39
Image NSFW Score
12.18
1,28
18.38
Word Number
-18.65
0,74
9.78
Stopword Ratio
-4.28
-3.33
8.97
Special Character Ratio
8.86
4.15
-16.03
Phrase Grounding Recall
7.79
1.85
-10.60
Text Length
-8.31
1.81
7.29
Character Repetition Ratio
1.99
0.04
6.63
Image Aspect Ratio
4.93
4.55
5,87
Text Perplexity
5.27
2.46
-956
Image Width
-6.66
4.97
5,23
Image Height
4.03
5.02
0.89
Image Size
-12.11
5.00
2.87
Image Aesthetics Score
-9.61
-8.13
4.64
Image Watermark Score
3.84
-3.74
-4.22
Flagged Word Ratio
3.66
3,47
1.59
Entity Dependency Number
-5.53
-0.39
2.50
Word Repetition Ratio
-3.16
-0.91
1.84
Alphanumeric Ratio
-2.55
1.65
0.63
Token Number
-6.35
1.44
0.27
In image-text pre-training, we can see that the top three performing operations the ones generating stats of CLIP Image- Text Similarity, BLIP Text Similarity, and Im- age NSFW Score. The first two OPs generate statistics based on auxiliary CLIP and BLIP models, enlightening that the modality alignment degree is critical to the studied pair-wise learning task.
data pools, which can be normalized as follows:
Σ -P((w) <P(w)log log P(w),
where w is a word and P is the distribution words in a data pool. As we can see, data pools with higher NSFW scores and lower Language Score have higher word entropy, suggesting greater diversity within these data pools.
In image captioning fine-tuning, models trained shorter captions (low range of Text Length) achieve the best perfor- mance. Captions with less repetition on both character and word levels help to train models with better performance. It suggests that in the image captioning task, the quality of captions is the key to better performance.
We also introduce more direct way to show the data di- versity to users. We use tagging model to extract core objects, concepts, and domains from the contents of the datasets, including both texts and multimodal content. Then we analyze the domain-specific distribution via word clouds of these tags. We show an example of analyzed word clouds from videos on different data pools of Video NSFW Score in Fig. 5 As we can see, although the data pool with high video NSFW scores is the most diverse one, our model achieves the best performance on the data pool with low video NSFW scores, which is consistent with the argument
E.3. Diversity Analysis
In this subsection, we delve into the diversity of the data in interested data pools. From the perspective of quantitative indicators, we confine our focus to statistical analysis of words within text data and compute their entropy. We oper- under the assumption that the texts provide an accurate description of the images and videos. Consequently, the diversity inferred from the texts also serves as proxy for the diversity of the associated images and videos.
presented in Sec. 4.2
E.4. Correlation Analysis
To investigate the intrinsic relationships between OPs and to support our recipe formulation, we explore relevance from the following two perspectives.
The Table shows the entropy of text words for different
23

---

A Feedback-Driver Suite for Multimodal Data-Model Co-devclopment
Table 8: The complete OP ranking, including their statistical dimensions and the improvements relative the baseline for the image captioning task. We consider three splits with low, middle, and high statistical values for each OP. The baseline used is based on random sampling with equal data volume.
Average Performance Changes (%)
OP-Generated Statistics
Data Pool (Low)
Data Pool (Mid)
Data Pool (High)
Text Length
0.760
-3.125
11.364
Image Watermark Score
-0,637
-0.132
0.477
Character Repetition Ratio
0.449
-0.461
-0.634
Text Perplexity
-1.505
0.331
-3,220
Word Reperition Ratio
0,300
-3.125
-11,349
Image Width
0.265
-0.230
-0.112
CLIP Image-Text Similarity
-0,920
0.225
2.101
Phrase Grounding Recall
-1.318
0.166
0.188
Image Aesthetics Score
+0.300
0.186
-0.243
Image Aspect Ratio
0.171
-0.490
-0.118
Image NSFW Score
+0.030
0.062
0.113
Image Height
+0.80
-0.030
0.088
Stopword Ratio
-3.795
0.084
-4,930
Image Size
-0.122
-0.071
0.028
Flagged Word Ratio
-0,012
-0.663
-0,429
Language Score
-0.154
-1.534
-7.823
Token Number
+0,610
-0.987
-9.017
Alphanumeric Ratio
00795
-4.300
11.592
Text Action Number
-2,333
-0.931
88565
Word Number
-1.125
-2078
99,33
Special Character Ratio
-11.194
-4.722
-1.145
Entity Dependency Number
-1.43
-1.660
-7002
BLIP Image-Text Similarity
2.254
-1.446
-6.221
Table 9: The entropy of text words for data pools with different levels of Image NSFW score and Language score.
Word Entropy
Task
OP-Generate Generated Statistics
Data Pool (Low)
Pool (Mid)
Data Pool (High)
Image NSFW Score Language Score
6.97
7.35
7.29
Image-to- -ext
7.47
7.32
6.98
Video NSFW Score Language Score
5.84
6.03
6.01
Text-to-Video
6.30
5.85
5.73
air field
parking
traller
apricot
act
website
Searing
eee
massage
d
ledty kness
Geveball
pastuure
turn
takeoff
fenball
baseball
pitcher
tackle
super mar
Markett
debate
ket
Shelf
cliptoe
mascara
eye
ssin
rtistt
hairstyl
brighs
time
det
tras
ender
Fractrr Efarnland
er
tog?
seed
government
ee
marr
disaree
slot
siprecipe
anall polish lish.....
tattoo
s
heye
makeup too
shadow
document
manicure
home base
cennia court
goalkeeper
catcher
lot machine
president
makeup
Sranese
a
(a) Vidco NSFW Score (low)
(b) Video NSFW Score (mid)
(c) Vidco NSFW Score (high)
Figure 5: Domain-specific word clouds tagged from videos for data pools with different levels of and Video NSFW score.
24

---

A Baabback-Driver Suite for Multimodal Data-Model Co-devclopment
Figure 6: Pearson correlation coefficients for OP statistics in image-to-text generation.
Figure 7: Pearson correlation coefficients for OP statistics in text-to-video generation.
First, we adopt the most direct approach by examining the Pearson correlation coefficients between the statistics of OPs, as illustrated in Figures 6 g and g associations between the statistics of OPs utilized in image- to-text generation appear be significantly stronger than those in text-to-video generation. For instance, in image- to-text generation, phrase grounding recall shows strong positive correlation with text perplexity and special char- acter ratio, while exhibiting a strong negative correlation with the alphanumeric ratio, language score, number of text actions, stopword ratio, and text length. contrast, in text-to-video generation, we observe relationships primarily among the purely textual OPs, while video-related operators largely orthogonal to others. Therefore, for image-to- text generation, we categorize OPs into three groups based on the correlation of their statistics and select the optimal OP from each category to create combinations. However, this method does not appear appropriate for text-to-video generation due to the sparser correlations observed.
thus preventing videos without movement from being rated optimal. Based on these observations, for text-to- video generation, we apply hierarchical clustering algo- rithm (Ward Jr 1963) to classify the VBench metrics into three categories based on their correlations: static video met- rics, dynamic video metrics, and video quality along with video-text matching. We then select the best pest-performing OP for each of these three metric categories, where each excels in different aspects. These selected OPs are subsequently combined and experimentally evaluated together.
Intuitively, the
E.5. Recipes Based on Correlation Analysis
In addition to selecting the overall best-performing OPs, we aim to identify operators with distinct advantages to ex- plore whether combining these operators can synergistically leverage their strengths for improved outcomes. We se- lected operators with unique strengths based on correlation information obtained from single-operator experiments. Detailed correlation analyses be found in Appendix E.4
In the second approach, categorize the evaluation met- rics based their correlations, illustrated in Fig. 10 for image-to-text generation and Fig. [T for text-to-video generation. In image-to-tex generation, several evaluation metrics are highly specific and possess unique characteris- tics, such as those for OCR and coding tasks, resulting in a greater number of categories upon classification. contrast, VBench's 's evaluation metries can be broadly divided into several classes, including static video metrics (e.g., subject and background consistency). dynamic metrics (e.g., dy- namic degree), and video quality indicators (e.g., aesthetic quality and imaging quality).
Specifically, for the image-to-text generation, we categorize the OPs by calculating correlations between their statistics. We then select representative operators within each category: FextActioons onF llee for text, ImageNSFWE WiiietFrrereerrrrrrrrrr ee for images, and PhraseGroundiingeecllllE.i for combined text-image relevance. For the text-to-video generation, based on single-operator experiments, we categorize the metrics of VBench (Huang al. (024] into three classes according to their relevance and select the best-performing operator from each category. The chosen operators are VideoMoti excels video static features; Video:s VidedDurationiin which is superior in dynamic features; and
which
Notably, the dynamic degree negatively correlates with many other metrics, particularly those favoring static videos,
25

---

A Feedback- Driven Suite for Multimodal Data-Model Co-devclopment
Figure 8: Pearson correlation coefficients for OP statistics in image-text pre-training.
Figure 9: Pearson correlation coefficients for OP statistics in image-captioning oriented fine-tuning.
is defined as:
Vi WidcoAestheticsPPitee. which exhibits the best composite performance in video quality and video-text matching. We summarize the experiment results in
L Ring 2 X Ryid.
(3)
where Len the loss of consistency distillation (Song et al. 2023), Ring is the reward of image quality of generated video frames from HPSv2.1 (Wu et al. 2023 and Rvid is the reward of video quality from Intern Video2 (Intern- Vid2 S2) (Wang et al. 2024b). In the paper, we make the following cumulative modifications to T2V-Turbo:
Figures 12(a) and 12(b)
Observation 8 (Effect of Orthogonal Combination)
Combining OPs that excel in orthogonal dimensions on model or data does not guarantee complementary effects; rather, is more likely that they will impede each other's performance.
1. Proposed Data Pool (Enhancement from Data View). Utilizing the optimal data pool identified in Sec. 4.4 which consists of 147k instances with low video NSFW scores and high frame-text similarities, we replace the WebVid data and train for epochs based on the insight from Sec. 4.4
As depicted in Figures (2 and 12bb) regardless of how these top-pes hoppeerforming OPs are combined, they ultimately re- duce the model's 's performance in both image-to-text and text- to-video generation. This observation challenges the naive assumption widely used existing SOTA works, that vari- ous intuitively useful data cleansing actions, when stacked serially, synergistically enhance performance.
2. Initialization for LoRA (Enhancement from Model View). We use the T2V-Turbo model to initialize the parameters for training with LoRA.
3. Deff-Distillation. (Enhancement from Model View). When we initialize with the T2V-Turbo model, the stu- dent model is already outperforming the teacher model, VideoCrafter-2.0. potentially leading unstable training. To mitigate this, we use the T2V-Turbo model itself as the teacher to ensure that the reinforcement learning does not diverge excessively.
E.6. Detailed Evaluation Results MGM
The detailed experimental results in terms of different eval- uation benchmarks shown in Table 10
E.7. A blation Study Based on T2V-Turbo
The results shown in Table representing the enhance- ments we achieved based on Turbo (Li et al. 2024a) T2V-Turbo applies LoRA (Hu et al. 2022) to VideoCrafter- 2.0 (Chen et al. 2024d) and is trained on the Web Vid (Bain et al. 2021) dataset, using VideoCrafter-22. 2. as a teacher for distillation and incorporating reinforcement learning with rewards for the generated videos. The loss of Tur-o
4. Real-Data Loss (Data-Model Co-Enhancement), To enhance the role of the proposed data pool during train- ing, we add a real-data loss between the generated videos and the input videos to the distillation loss and reward loss. Furthermore, we set the weights of both the real- data loss and the distillation loss to 0.5. The modified
26

---

A Feedback-Driver Suite for Multimodal Data-Model Co-devclopment
Table 10: Performance of MGM-2B with different pretraining datasets. MGM-2B pretrained with only 1/10 distinct instances and 1/2 of the total instances performs better. than the baseline trained on the full dataset. The "*" indicates our reproduced version that comparable with the official version.
MGM-2B
Num. of Instances
Avg. Perf. Changes (%)
MMBench
MMBench-CN
MME- Perception
MME- Cognition
Baseline*
1226k 159k (x4)
59.2
51.6
1334 1323.37
302 311.07
Data-Juicer
+2.12
62.08
52.32
311
Figure 10: Pearson correlation coefficients for each dimen- sion in TexiVQA, MMBench, and MME.
Figure 11: Pearson correlation coefficients for evaluation metrics in VBench.
loss from Equation can be specified as:
we replace the WebVid data with our proposed data pool, the model experiences a notable improvement, with the total score increasing from 81.01% % to 81.84%. Subse- quently, initializing the LoRA training parameters with the T2V-Turbo model does not lead further enhancements in model performance. We suspect this might be because the teacher model is less effective than the T2V-Turbo model. Therefore, we use the T2V-Turbo model for self-distillation. While this method effectively raises the semantic score, it results in unstable video generation characterized by signif- icant temporal flickering, which severely lowers the video quality. To counteract this, add a real-data loss with the input data secure the quality of the generated videos. Moreover, evolve the model by using our trained model as the teacher model for continuous training on additional data, rather than T2V-Turbo. Ultimately, as we continue to enhance both the quality and semantic scores, we establish new ofatc-of-the-art.
L 0.5 +.5x Lreat- 1x Rang- 2x Rvid, (4)
where x Lyeat is the real-data loss. Training on 147k instances with 6 epochs, obtain the model, Data- Juicer (T2V, 147k).
5. Self-Evolution (Data-Model Co-Enhancement), To validate the scalability of our methodologies, we expanded the original data pool and, by ap- plying the video_motion wtt.smctooosscrreffiiie
aeteo_asstheticcs fiiteer and to the existing recipes, we select an additional 228k instances to evolutionary self-distill Data-Juicer (T2V, 147k). Mcanwhile, in order to further amplify the impact of data, we have reduced the emphasis on reinforcement learning, thereby:
L = 0.5 L.5 Lreat - 0 Ramg 0.4 x Rvid-
(5)
We finally get Data-Juicer (DJ, 228k) in 4 epochs train- ing.
E.8. Full Results on VBench leaderboard
Backed by our proposed methodology and experiments, our Data-Juicer (DJ, with 228k) model refreshes the state-of- the-art on VBench (Huang et al. 2024). surpassing models
Table [I presents the ablation experiments of our modi- fications in the VBench evaluation. clear that when
27

---

A Feedback-Driven Suite for Multimodal Data-Model Co-devclopment
Text Action Number (low)
Phrase Grounding Recall (high)
Video Motion Score (low)
Aesthetics Score (high)
15.21 0.43)
14.74 (# 1.35)
16.09 (# 2.34)
40.55 (# 0.40)
10.06 (# 0.76)
10.96 (# 0.13)
15.54
10.43
2.94)
(# 0.77)
(# 3.84)
12.72
3.66
41.14
0.04)
(#: 0.20)
10.33
(#t 2.55)
15.40 (# 2.83)
10.04 (= 0.84)
Image NSFW Score (high)
Video Duration (high)
(a) Cluster-recipes for image-to-text model
(b) Cluster-recipes for text-to-vidco model
Figure 12: The improvements (%) of recipes with different OP combinations. The cluster recipes stem from the combinations of the best OPs in categories. In the image-to-text generation, due to the limited data volume in the top-level data pool, the size of Ps decreased from 200k to 126k in the cluster recipes.
Table 11:
Our model undergoes ablation experiments on the VBench leaderboard evaluation. Each '+' sign in the row indicates that the modification is added top of the previous row's configuration.
of
Model
Total Score (%)
Quality Score (%)
Semantic Score (%)
T2V-Turbo (VC2)
81.01
82.57
74.76
+ Enhanced Data Pool
81.84
83.40
75.60
+ Initialization for LoRA
81.82
83.47
75.19
+ Self-Distillation
79.16
79.48
77.92
+ Real-Data Loss
82.10
83.14 14
77.93
+ Self-Evolution
82.53
83.38
79.13
like Gen-3 (RunwayML. 2024) and Kling (Kuaishou 2024).
51.8% of the FLOOS.
as illustrated in Fig. 13 and detailed in Table 3
For the CLIP experiments, refer to Figure 12; the FLOPs with full expansion rate are 189 a PFLOPS (14.78G in- ference FLOPS per sample) and 526 a PFLOPS (41.09G inference FLOPS per sample) for B-32 and B-16, respec- tively.
E.9. Cost Comparison of all Sandbox Experiments
Besides, we present the compute details for our experiments and compare them to other baselines. Our experiment on a single data pool is designed to be completed within one day using a GPU. Consequently, with A100 GPUs, we can run our experiments within approximately days, days, and 10 days for the CLIP, image-to-text, and text-to- video experiments, respectively. The training sample size is in the order of (11) million video-text pairs and O(1) million image-text pairs in our experiments. As reference, the pre-training cost for cutting-edge text-to-video model like Meta's s MovieGen involves 6, 1.44 H100 GPUs, O(100) million video-text pairs, and ((1) billion image-text pairs.
For the text-to-video experiments, Noting that estimating the FLOPS for some models in TableBaand and Table 12 poses several challenges due to the lack of publicly available infor- mation. For example, some of their training codes, and train- ing datasets have not been open- sourced, and specific details such as the resolution and frame rate training videos are unavailable. Thus, report the estimated FLOPPS for the models with sufficient details about the training samples and open-sourced models below. We utilized the inference code their released open-source models and measured the FLOPS during inference using the calflops tool. We
More specifically, for the image-to-text experiments, in Ta- ble 2, MGM model shows +2.12% average perfor- mance improvement over baseline, using approximately
28

---

A Feedback-Drivee Suite for Multimodal Data-Model Co-devclopment
Table 12: Leading models the VBench leaderboard of Sep 23, 2024. "Board Avg." denotes the weighted average of normalized scores across 16 metrics defined by VBench. "Quality Avg." represents the aggregated scores from video quality metrics, whereas "Semantic Avg" aggregates scores from metrics evaluating the consistency between prompts and generated videos. "Uniform Avg." indicates the simple average of scores related to quality and semantic metrics. Data-Juicer (72V, 147k) based on T2V-Turbo distillation training on 147k instances. In order to demonstrate the scalability of our methodologies, we then further self-distilled our model to obtain Data-Juicer (DJ, 228k) on other 228k instances.
Models (Ranked by leaderhoard)
Board Avg. (%)
Uniform Avg. (%)
Quality Avg. (%)
Semantic Avg. (%)
1. Data-Juicer (DJ, 228k)
82.53
81.26
83.38
79.13
Data-Juicer (T2V, 147k)
82.10
80.54
83.14
77.93
2. Gen-3 MRLnwayML VEnhancer (VC2) le
2024)
82.32
79.64
84.11
75.17
2024a
81.97
80.00
83.27
76.73
4. Kling (2024-07) Kuaishou
2024
81.85
79.54
83.39
75.68
Vaie22 (Wang 2025
81.75
79.50
83.24
75.76
6. Cog XBBeoX-5B (Yng et
2024)
81.61
79.90
82.75
77.04
Vchitect 2.0-2B
2025)
81.57
80.15
82.51
77.79
8. T2V-Turbo (VC2) (Li al.
2024a)
81.01
78.67
82.57
74.76
Table 13: Detailed FLOPs comparison for representative T2V models. The factor indicates constant taking into account the disclosed quantity of training data.
Models
Performance
Full size of used Dataset
Inference FLOPS
(Estimated) per
(Estimated)
of Training Sam- ples
Number
Training Cost in EFLOPS
Sample
Ours
(stems from VC2)
82.53
228k
12T/sample, 320p * 8fps
640k
7.68a
Ours
(stems from VC2)
82.10
147k
12T/sample, 320p * 8fps
640k
7.68a
VEnhancer
(stems from VC2)
81.97
350k
478T/sample, 720p 24fps
350k
167.3a
CogVideoX-5B
81.61
1,051M
338T/sample, 720p 8/16fps
35M
> 11841a
Vchitect 2.0-2B
81.57
252T/sample, 768p * 8fps
29

---

A Feedback- -riienn Suite for Multimodal Data-Model Co-devclopment
Specs Weradd Leccceeee
VBench Leaderboard
voscobbeed Fies sctuiionae coberas crnieess the wpccianniiiio VLUTC stap 7h, Ind OC pooeed Sen 1. fo dioog Nox frra diode aitiss cutcs
Solect Quality Oiverdere
Select Semurdc deseniion
Deotlct #1
Starce
Tetal Crars
Seore
Saturied Graws
vajjott revistorry
15 50
CHaAstooA C3. SSAT asclntad ICerch Tacr
07.75
77
00 3M
0:
4.50
Figure 13: The VBench Leaderboard as of September 23, 2024, illustrating the rank achievement empowered by our data-model co-development workflow.
then estimated the training FLOPS by applying factor of and taking into account the disclosed quantity of train- ing data. From this, we can see that our method achieves strong performance while utilizing fewer FLOPS, especially in comparison to the third-place model, VEnhancer, which also originates from VC2, the same upstream model as ours. The second-place Gen-3 and the fourth-place Kling are omitted as they are proprietary models within commercial products.
develpment on the checkpoint
Results. As the Table [4 shows, the second iteration contin- ues to improve the model performance. Notably, while the ranking of OP effects changes slightly between iterations, most effective OPs remain consistent: out of the top- OPs in the second iteration overlap with those from the first iteration; 7 of top-10 10 OPs in the first iteration increase their rankings by average of 6. 6.14 positions and 3 of them decrease by an average of 3.33 positions. Using the same OPs, data pool performance changes, providing actionable insights for dynamic environments.
E.10. Iterative Workflows in Sandbox
Data-Juicer Sandbox allows iterative workflows, where data- model co-development is activated iteratively to enable data and model improvements for each other. We conduct fur- ther iterative experiment based on the basic single-operator experiments for the image captioning fine-tuning task in Sec. 4.2 where the full experimental results are shown in Table
This also highlights promising future directions and un- derscores the distinction between our work and DataComp. While DataComp primarily focuses CLIP pretraining and scaling compute with duplicated data, our data recipes are validated across a broader range of tasks, including text-to- video generation Animate, T2V-Turbo), image-to-text pretraining (MGM), post-training (IntemVV--2.0) and the iterative co-development.
g
Experiment Settings, We regard the single-operator recipes in the basic experiments as the recipe,. and their trained models as checkpoint Based on these first-iteration re- sults, we select the best checkpoint from the best single operator Text Length to conduct a second iteration of single-operator continuous training for all singlc-operator recipes recipe2 and obtain the second-iteration trained mod- els checkpointy. Finally, evaluate the checkpoints to find out the impacts of the second-iteration data-model co-
E.11. Operators beyond Filters
There are diverse types of OPs in Data-Juicer in addition to Filters. In this section, we conduct several experiments on two representative Mappers using MGM-2B on the image- to-text task to verify the effectiveness of operators beyond Filters of the sandbox.
30

---

A Feedback-Driven Suite for Multimodal Data-Model Co-devclopment
Experiment Settings. We the same experiment ting of single-operator experiments for the image-to-text task in Sec. 4.2 except that we replace the single-operator recipes of Filters with two new recipes of two Map- pers: image_di ffusffuiiin om_mapper: it regenerates a new image for each sample based on the given caption; ingiiinnnnnnneeeennneee it recaptions the exist- ing image each sample.
Results. As the Table 15 shows, images generated by image_di wfffffsiooooooo onmmapeer significantly improve the model performance compared to the original images. In Fig. 14 it highlights a visual example before and after pro- cessing with these two Mappers, with the core objects in the captions marked in red. We can observe that diffu- sion models used in the image_di di
mapper effectively locate and better present the key objects (e.g., "setting sun") in the newly generated image, and it also removes redundant information (e.g., watermarks, link texts). We think this kind of "denoising" capability of image_da ffffionnon Eaappppaanppeerrrrttt_eeeaeaeeet mapper contributes to perfor- mance gains. Experiments on other kinds of OPs are left to future work.
E.12. Potential Model Development: Automated Prompt Adjustments
Most parts of this paper focus on the data improvements in data-model co-development. In this section, we present several experiments for automated prompt adjustment to show the potential model development part of the Data- Juicer Sandbox.
Experiment Settings. We select the image captioning fine- tuning task, extend new model inference hook with about 40 lines of code changes, and update its prompt config- uration file. Based the iterative experiment results in Sec. 10 we use the top-1 recipes found in two iterations. Then we select 10 different prompts generated by Qwen2.5- max and study the effects of them on trained models in different iterations.
Results. As the Table 16 summarizes, the ranking and performance of these prompts vary significantly across it- erations, ranging from -1.673 +0.944. Some of them beat the baseline prompt, and some of them can enhance or degrade performance in both iterations. It seems that the most suitable prompt is changing during the iterative data-model co-development, which indicates the potential for auto-optimizing model configurations in the context of the co-development of the Data-Juicer Sandbox.
31

---

A Feedback-Drivee Driven Suite for Multimodal Data-Model Co-devclopment
Original Image- caption pair
Regenerated contents
Image diffusion mapper
shutterstsck
chees
www.shuaterssock.com .com 407617375
Image
captioning mapper
lone tree in the middle of nowhere, serengeti nat'lpark rwanda ndakarara, asia kenya afric
the setting sun in africa on cloudy day stock photo monkeypix
Figure 14: An example of image and caption pair before and after being transformed by amage_diffusion and mmagg_ccaptioning.... mapper.
Table 14: Top-10 10 OP performance ranking with 2 iterations, including their statistical dimensions and the improvements relative the baselines for the image captioning task. The 2nd iteration is continuously trained the top-1 OP (Text Length) checkpoint in the 1st iteration and the improvements relative to it well. The OPs in bold in the 2nd iteration also appear in top-10 10 OPs of the 1st iteration.
1st iter.
2nd iter.
OP- Pe-Greerteed Statistics
Average Performance
OP-Gen Generated Statistics
Average Performance Changes (%)
Changes (%)
Text Length
0.760
Flagged Word Ratio
0.607
Image Watermark Score
0.477
CLIP Image-Text Similarity
0.599
Character Repetition Ratio
0.449
Text Perplexity
0.521
Text Perplexity
0.331
Character Repetition Ratio
0.434
Word Repetition Ratio
0.300
Image Aesthetics Score
0,420
Image Width
0.2665
Image Size
0.376
CLIP Image-Text Similarity
0.225
Phrase Grounding Recall
0.358
Phrase Grounding Recall
0.188
Image Width
0.320
Image Aesthetics Score
0.186
Image Height
0.285
Image Aspect Ratio
0.171
Image Watermark Score
0.264
32

---

A Feedback-Drivee Suite for Multimodal Data-Model Co-devclopment
Table 13: The performance improvements of two mapper OP, image gagge_ddifsoonmmaite _appeerrppprrreee and imuge_captioning_maee mapper, relative to the baseline, which regenerates a new image/caption for each sample using diffusion/BLIP models respectively. The baseline is based on the original raw images.
Mapper OP
Average Performance Changes (%)
image_ dibbseeddffiion___e diffusion_mapper
mapper
13.45 i 11.33
-1.07 = 0.65
Table 16: The experimental results of automated prompt adjustment on the top-1 recipes found in iterations with 10 different prompts for the image captioning task. The baseline prompt is the default prompt of InternVL2.5. The values that are in bold or underlined are the best or the 2nd best ones in each column.
Average Performance Changes (%)
Prompt
1st vs. 2nd
1st iter.
2nd iter.
(Baseline) Provide a one-sentence caption for the provided image.
0.760
0.607
>
Provide a brief yet comprehensive caption that highlights the primary elements and overall mood of the image.
0.944
-0.705
>
Describe the image in one sentence, focusing on the main subject and its surrounding environment.
0.832
-0.241
>
Summarize the scene depicted in the image with a concise and descriptive sentence.
0.734
0.148
Summarize the content of the image in one clear and accurate sentence without omitting important details.
0.542
0.186
Describe the most important details of the image in one sentence.
0.400
0.130
Craft concise image caption highlighting the main activity alongside coexisting static elements in the image.
0.083
-1.622
What is the main subject of the image, and how would you describe it in one sentence?
-0.026
-0,463
If there is movement or activity in the image, summarize it in single descriptive sentence.
-0.594
0.433
Please provide a one-sentence caption that describes the primary subject's 's action and its relationship with the surrounding environment.
-1.578
-1.510
What stands out most in the image? Capture it in concise and striking one-sentence caption.
-1.673
-1.300
33


Reviews:
[{"review_id": "S8zhLzGxIj", "reviewer": "Reviewer_cLJN", "summary": "This paper introduces Data-Juicer Sandbox, a feedback-driven suite for multimodal data-model co-development. The system integrates the data processing system with model-centric infrastructure, and designs a \"Probe-Analyze-Refine\" workflow to systematically explore the relationship between data processing operators and model performance. The main contributions of the paper include: (1) the first open-source sandbox platform supporting joint optimization of multimodal data and models; (2) empirical validation demonstrating significant performance improvements in image-to-text generation, text-to-video generation, and image-text pretraining tasks ; (3) scalability verification showing that optimization strategies obtained from small-scale experiments can be transferred to large-scale scenarios, thereby reducing computational costs.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your acknowledgment and constructive feedback on our work! We respond to the raised only concern with the following new experiments.\n\n> The mentioned new results: https://anonymous.4open.science/r/icml_submission8414-7C89/rebuttal.pdf\n\n---\n### [Exp Design, Weakness & Suggestion] \"additional experiments on other models,\" \"... with models of different scales (from small to large), ...\"\nThank you for your valuable suggestion! Per your comments, we have conducted new experiments on an additional model, *InternVL2.0*, across different scales (1B, 2B, 4B, 26B), covering a new task (*image captioning*) and new stage (*fine-tuning*).\n\n\n`Settings`\n\nTo ensure consistency with our proposed workflow:\n- We first experiment with the smallest model size (1B parameters) on *23 single-op* and *6 multi-op data pools*, deriving the optimal data recipe.\n- The identified recipe is then applied to all selected model scales.\n  \nSpecifically:\n- We fine-tune InternVL2.0 for image captioning using the *COCO Caption training dataset* (567k image-caption pairs). \n- Each single-op data pool contains ~189k samples, while multi-op pools are reduced to ~24k samples due to intersections.\n- Evaluation metrics include *Bleu-1/2/3/4, METEOR, ROUGE_L, and CIDEr*, along with the average performance change relative to baseline models trained on randomly sampled data pools.\n\nFor fine-tuning implementations, we set the global batch size to 512. We only take 1 epoch for each data pool with 1B-parameter model. For all experiments, we use H20 GPUs to fine-tune and evaluate the models. Training on one single-op, multi-op data pool with 1B-parameter model takes about 8 and 1.5 GPU hours. Training the optimal recipe including about 24k samples on models with 1B/2B/4B/26B parameters takes about 1.3/2/4/18 GPU hours respectively. Due to time constraints, experiments on various scales using the optimal recipe were repeated twice (random seeds: 42, 1024). Other experiments were conducted once.\n\n  \n`Key Results`\n\n1. *Single-OP Results*:  \n   - Full results are ranked in *Table 1* of the updated PDF.  \n   - Models trained on shorter captions (*Text Length Low*) achieve the best performance.  \n   - Most OPs show positive improvements.\n\n2. *Multi-OP Results*:\n   - Top-3 OP combinations do not yield further gains, likely due to low correlation between these OPs (plz see *Figure 4* in the updated PDF for Pearson correlation coefficients).\n\n4. *Scaling Behavior*:\n   - Using the top-3 combination recipes, we fine-tuned InternVL models across all scales.  \n   - Results are shown in *Fig.1(b)* of the updated PDF (x-axis in log scale).  \n   - *Highlight*: All three recipes maintain consistent and steady performance advantages as the model scale increases from 1B to 26B, demonstrating clear *scaling law behaviors*. In conclusion, our sandbox can consistently achieve performance improvements across these studied scales.\n\n---\nThank you again for your time and support! We hope these new results and revisions can address your concern and strengthen the work's applicability. If you have further questions or suggestions, we warmly invite your input.", "claims_and_evidence": "The paper's main claims are supported by experimental results. The authors assert that Data-Juicer Sandbox can optimize data processing strategies through feedback-driven approaches, thereby enhancing model performance, which has been validated through experimental results across multiple tasks.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally reasonable.", "theoretical_claims": "The paper does not present theoretical claims requiring rigorous mathematical proof. Its core contribution is a practical system framework and methodology, primarily validated through experimental results.", "experimental_designs_or_analyses": "Experimental results in table 3 showcase the significant performance gain by distillation on T2V-Turbo. There could be additional experiments on other models to explore the broader applicability and benefits of Data-Juicer", "relation_to_broader_scientific_literature": "This work is related to the broader research field of multimodal data processing and model training.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths:\n- The system design has practical value, especially for researchers and engineers who need to rapidly iterate on multimodal model development.\n- Experimental results indicate that the method is effective across multiple tasks, demonstrating its versatility.\n\nWeaknesses:\n- Baseline comparisons are not comprehensive enough.", "other_comments_or_suggestions": "I suggest adding experiments with models of different scales (from small to large) to better demonstrate the scalability of the method.", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jwXthZ4qj2", "reviewer": "Reviewer_3zp6", "summary": "In their work, authors describe and implement a method and procedures for data-model co-development, aiming on improving pretraining of various foundation model types (language-vision CLIP, diffusion based text-to-video generative models, Llava based image-text generative model). The framework authors introduce is Data Juicer Sandbox. It sets up a pipeline for data composition, making use of available data pools to apply various data processing operators (OP) aiming for data composition that will improve model training. Authors define a probe-analyse-refine workflow, which applies OPs to data pools, employing low cost model training and evaluating those, which provides dataset quality scores. Based on those, OPs leading to top performing evals can be combined to processing pipelines. Those are used to create datasets at larger scales, which should further improve model pre-training. Authors test their approach by performing dataset composition for the aforementioned model types and showing that it leads to strong scores on standardized benchmarks like MMBench, VBench, ImageNet zero-shot classification, such that obtained models compare well to the known strong reference baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition of the *method, evaluation, theoretical argument & exp design* of our work. Regarding your raised concerns, we address all of them point by point as follows:\n\n> New results added: https://anonymous.4open.science/r/icml_submission8414-7C89/rebuttal.pdf\n\n---\n\n### [In Evidence, Weakness & Question parts] \"experiments conducted on *per model* with *various scales*; consistent trend improving across scales?\"\nFollowing your suggestion, we conducted additional experiments on a larger model suite (*1B, 2B, 4B, 26B InternVL*) and a new task (*fine-tuning for captioning*). These results further validate the generalizability of our work. For details, please refer to our response to Reviewer cLJN.\n\n---\n\n### [In Evidence & Literature parts] \"iterative co-development, ... push further than DataComp\"\nThank you for your insightful comments! To address this, we performed new experiments to demonstrate iterative training:\n\n- *Setting*: Starting with `ckpt0`, we refined `recipe_1` → trained `ckpt1` → refined `recipe_2` → trained `ckpt2`. The first iteration aligns with the single-operation (OP) experiments from previous tasks. Based on the first-round results, we selected the best checkpoint and applied the found top OP, *text_length_filter*, to the original training dataset as our base model. We then conducted a second iteration of single-OP continuous training to evaluate its impact.\n\n- *Results*: Summarized in Table 2 of the uploaded PDF, the second iteration continues to improve performance. Notably, while the ranking of OP effects changes slightly between iterations, most effective OPs remain consistent: 8 out of the top-10 OPs in the second iteration overlap with those from the first iteration.\n\n- *Remark*: This highlights promising future directions and underscores the distinction between our work and DataComp. While DataComp primarily focuses on CLIP pretraining and scaling compute with duplicated data, our data recipes are validated across a broader range of tasks, including text-to-video generation (*EasyAnimate, T2V-Turbo*), image-to-text pretraining (*MGM*), post-training (*newly added InternVL in the rebuttal*) and this iterative co-development. For a more detailed comparison in the scaling perspective (including scaling across compute and number of distinct data samples), plz see Fig. 3, Table 5 in the original paper, and the newly added InternVL experiments mentioned above.\n\n\n---\n### [Writing Suggestion & Appendix] \"have CLIP info more visible in the main part\", \"one important figure 12 is missing\"\nWe appreciate these constructive suggestions! Regarding \"Figure 12\" mentioned in L1424, it should indeed be \"Table 12.\"  \nFor CLIP-related information, we will move more detailed settings and results analysis from the Appendix into Section 4.1 and the first paragraph of Section 4.5.\n\n---\n\n**Empirical Evidence Summarization** Collectively, extensive experiments demonstrate the effectiveness and usability of the sandbox, covering different  \n- *Scenarios*: Text-to-video generation, image-text pre-training, image-to-text generation for general image understanding, and image captioning, iterative enhancement (see details above).\n- *Stages*: Both pre-training and fine-tuning.  \n- *Model Architectures*: EasyAnimate, T2V-Turbo, MGM, InternVL2, and CLIP.  \n- *Scales*: Model parameters, deduplicated data sizes, and compute FLOPs.  \n- *Data Processing Solutions*: Over 40 Data-Juicer Filter OPs and Mappers (see response to Reviewer 5ZDF)\n- *Model Feedback*: 78 metrics.\n\n---\n\nThank you again for your helpful feedback! We will incorporate these improvements into the final version, and kindly request you to review these responses and re-evaluate the merits of this work. Your feedback is highly anticipated and greatly valued.", "claims_and_evidence": "Main claim of the study is to be able to perform model-dataset codevelopment. Authors argue that dataset composition performed by their method leads to stronger models by comparing models they obtain to well known references. The comparison is done on fixed compute scales for each model (for video models) or on few reference scales (for CLIP). For full comparison, I think it would require a proper scaling law per each model, as taking fixed reference compute scales might be misleading (eg differences models show for certain fixed scales can change across scales, so trend across scales for each tested model is the actual interesting measure for comparison). Another problematic issue I think is that to claim codevelopment, authors would have to show at least two iteration cycles of dataset composition and model improvement. In current form it seems to me that while the evidence might enough to claim dataset composition via the described procedure leads to training a strong model, it is not clear whether then the obtained model can be used to further improved dataset composition, or at least that the dataset can be further improved from insights of conducted experiments, and so on (which would be the case of co-development).", "methods_and_evaluation_criteria": "Authors make use of a number of standardized benchmarks to measure the model quality, which makes sense.", "theoretical_claims": "Authors make a theoretical argument to back up the approach for carrying over observations done on small scale pool experiments to larger dataset composition. It seems to me the derivation makes sense.", "experimental_designs_or_analyses": "The experimental design that describes the dataset composition testing pipeline seems sound to me.", "relation_to_broader_scientific_literature": "The work continues tracks set by important previous works like DataComp, emphasizing its difference in attempt to incorporate model signals during dataset composition. It is however I think not correct that previous works were not doing so, eg DataComp was incorporating CLIP models to filter the data, which then can be also considered as a case of model-data codevelopment. As authors do not show many iterations of such a cycle, they do not really push further in approach compared to DataComp, in my opinion (which would be different if authors have demonstrated iterative improvement of model over some cycles of their co-development)", "essential_references_not_discussed": "Relevant scientific literature is well covered.", "other_strengths_and_weaknesses": "Strength of the work is variety of different models authors consider and in attempt to define a generic pipeline for dataset composition improvement. Weakness are missing scaling laws that would be necessary to see whether improvements are happening indeed across various scales and not on some fixed reference scale authors happen to choose.", "other_comments_or_suggestions": "A lot of important information on CLIP experiments is in the Appendix. It would be good to have those more visible in the main part, as this is important reference model class and showing improvements there provides strong evidence for the validity of the dataset composition approach.\n\nUPDATE: increasing score to 3 after clarifications on model-data co-development.", "questions_for_authors": "Scaling law derivation is performed on smaller scales. Would it be possible to conduct dataset composition at smaller scales and show that compositions made by the approach result in consistent trend improving across scales, without investing much compute?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IDlUEKRuH9", "reviewer": "Reviewer_s5fL", "summary": "This paper introduces a sandbox suite with a feedback-driven experimental platform, which supports cost-effective iteration and guided refinement of both data and models. The authors conduct experiment on image-to-text generation, text-to-video generation and image-text pretrianing. The results demonstrate the proposed sandbox can effectively improve dataset and models, and also give valuable insights into the relationship data processing and model performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our work as \"valuable for improving research efficiency\" and providing \"valuable insights\"! Below, we address the raised weaknesses (W) and questions (Q) with point-to-point clarifications.\n\n> The mentioned new results: https://anonymous.4open.science/r/icml_submission8414-7C89/rebuttal.pdf\n\n\n---\n\n## [W1 & Q2] \"how the sandbox improves model development compared to other platforms remains unclear.\" \"What are the advantages of the sandbox in model refinement?\"\n\n- In the perspective of **development process**: The sandbox is designed for *cost-sensitive* model optimization, enabling broader and more quantitative experiments within the same resource constraints. Compared to naive trial-and-error, this allows *more possibilities* for model improvement with less computational cost.\n\n- In the perspective of **transferable mechanism**: Our sandbox is insight-driven and emphasizes *analysis and scaling laws* derived from extensive experimental feedback. This approach has been validated across 40+ data processing operations and 70+ performance metrics, with empirical success on 5 cross-task, cross-scale models (e.g., EasyAnimate, T2V-Turbo, MGM, CLIP, InternVL).\n\n- **Rapid Expansion & New Experiments**: The framework is highly flexible and extensible:\n  - We added experiments on a new model series, *InternVL (1B, 2B, 4B, 26B)*, to verify generalization in captioning tasks. Details are in our reply to Reviewer CLJN.\n  - We demonstrated iterative training: starting with ckpt0 → refining recipe_1 → training ckpt1 → refining recipe_2 → training ckpt2. Notably, ckpt2 improved performance despite evolving recipes. See our reply to Reviewer 3zp6.\n\n- **Evidence Summarization**: \nCollectively, extensive experiments demonstrate the effectiveness and usability of the sandbox, covering different  \n   - *Scenarios*: Text-to-video generation, image-text pre-training, image-to-text generation for general image understanding, and image captioning, iterative enhancement.\n   - *Stages*: Both pre-training and fine-tuning.  \n   - *Model Architectures*: EasyAnimate, T2V-Turbo, MGM, InternVL2, and CLIP.  \n   - *Scales*: Model parameters, deduplicated data sizes, and compute FLOPs.  \n   - *Data Processing Solutions*: Over 40 Data-Juicer Filter OPs and Mappers (see response to Reviewer 5ZDF)\n   - *Model Feedback*: 78 metrics.\n\n\n---\n\n## [W2 & Q1] \"Data recipes obtained from small-scale experiments may not generalize to large-scale\" \"I am curious to learn about how the data recipes obtained from this sandbox compare to the optimal data recipes in large-scale experiments.\"\n\n- **Clarification on Workflow**: Recall that our workflow begins with small-scale experiments (Sec 4.2–4.3) but rigorously validates findings on larger scales (Sec 4.3–4.4). Empirically:\n  - Small-scale recipes benefit larger scales across architectures, data, and compute (see structural summarization in Table 5).\n  - Scaling behavior (Fig.3 & new InternVL exps), high data efficiency (Table 2), and SOTA performance (Table 3, with many *SOTA* large-scale recipes used by baselines) are also validated even under challenging distribution shifts.\n\n- **Methodological Flexibility**: The open-source framework supports custom workflows while reusing data/model infra (Sec B.1–B.3). Advanced strategies like dataset distillation [1] and tensor shaping [2] can be integrated via new hooks. We leave theoretically addressing distribution shifts as future work, which requires substantial new effort.\n   \n[1] Large scale dataset distillation with domain shift, ICML'24\n\n[2] Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement, ICLR'24\n\n---\n\n## [W3 & Appendix] \"There is no supplementary material of this submission\", \"I think the sandbox's properties should be the core focus, but there are many paragraphs talking about observations from data experiment\"\n\nWe agree that the sandbox's properties are critical and clarify potential misunderstandings:\n- **Extensive Appendix**: Contrary to the comment (you may not have noticed our details), we provide a dedicated appendix (~1.5 pages) focusing on infrastructure:\n  - Overview architecture (Sec B.1),\n  - Capability factory and behavior hooks (Sec B.2),\n  - Extensibility (Sec B.3).\n\n- **Main Paper Focus**: Due to space constraints, we prioritize showcasing practical results and insights in the main text. Without concrete demonstrations, readers may struggle to appreciate the sandbox’s utility and understand its usability. System details are thus deferred to the appendix and source code for a smooth and progressive reading experience.\n\n---\nThanks again for your time and valuable comments! We hope that you can consider an increase in the rating if these responses and clarifications address your comments. If you have any additional comments, we warmly invite you to share them with us.", "claims_and_evidence": "The paper claims that the proposed sandbox effectively achieves data-model co-development. However, only the contribution to data refinement has been proved, while how the sandbox improves model development compared to other platforms remains unclear.", "methods_and_evaluation_criteria": "The proposed method for data and model development is reasonable. However, the evaluation mainly focuses on the sandbox's application in data refinement, which may not be enough to demonstrate its superiority over other platforms.", "theoretical_claims": "I have checked the correctness of theoretical claims in the submission.", "experimental_designs_or_analyses": "The evaluation mainly focuses on the sandbox's application in data refinement, which may not be enough to demonstrate its superiority over other platforms.", "relation_to_broader_scientific_literature": "The authors introduce a new sandbox for cost-effective data and model refinement, which can enhance the efficiency of model development and is valuable to the research community.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths:\n   1. The paper proposes a platform for efficient data and model development, which is valuable for improving research efficiency.\n   2. The paper analyzes the impact of different data recipes on model performance and provides several valuable insights.\n\nWeaknesses:\n   1. Although the paper claims the proposed sandbox effectively enables data-model co-development, only the contribution to data refinement has been proved, how the sandbox improves model development compared to other platforms remains unclear.\n   2. Data recipes obtained from small-scale experiments may not generalize to large-scale experiment, thus its challenging to determine the effectiveness of the sandbox on large-scale training. \n   3. The organization of the paper is somewhat confusing. I think the sandbox's properties should be the core focus, but there are many paragraphes talking about observations from data experiment. This makes it hard for readers to focus on the key advantages of the sandbox.", "other_comments_or_suggestions": "It would be better to improve the organization of the paper to enhance readability.", "questions_for_authors": "1. I am curious to learn about how the data recipes obtained from this sandbox compare to the optimal data recipes in large-scale experiments.\n2. What are the advantages of the sandbox in model refinement?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2AqvVtDGOo", "reviewer": "Reviewer_Dvoe", "summary": "This paper introduces the data juicer sandbox, an open-source suite that analyzes various metrics and make use of heuristics to facilitate the integrated development of multimodal data and models. The proposed \"Probe-Analyze-Refine\" workflow was validated through image-text pre-training with CLIP, MLLMs, and text-to-video generation. The `feedback-driven` approach, manually executed rather than automated, yields performance improvements and provides insights into the interplay between data quality, diversity, model behavior, and computational costs.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time, insightful feedback, and recognition of the work's significance! Below, we address your comments point by point.\n\n> The new results: https://anonymous.4open.science/r/icml_submission8414-7C89/rebuttal.pdf\n\n### [Para 1 in Claims part] \"generalizability is questionable, ..., particularly during specific model and task stages (eg the pretraining of a 2B MLLM)\"\nPer your suggestion, we conducted new experiments on a larger model suite (*1B, 2B, 4B, 26B InternVL*) and a different task (*fine-tuning for captioning*). For details, plz see our response to Reviewer cLJN.\n\nCollectively, extensive experiments demonstrate the generalizability, covering different  \n- *Scenarios*: Text-to-video generation, image-text pre-training, image-to-text generation for general image understanding & image captioning, iterative enhancement (see response to Reviewer 3zp6).\n- *Stages*: both pre-training and fine-tuning.  \n- *Model Architectures*: EasyAnimate, T2V-Turbo, MGM, InternVL2, and CLIP.  \n- *Scales*: Model parameters, data sizes, and compute FLOPs.  \n- *Data Processing Solutions*: over 40 Data-Juicer Filters and Mappers (see response to Reviewer 5ZDF)\n- *Model Feedback*: 78 metrics.\n\n---\n\n### [Para 2 In Claims part] \"... measures such as embedding or domain specification ... \" \"differ from previous findings ... CVPR24\"\n\n`More acknowledged measures`\n\nFollowing your suggestion, we conduct new experiments to dive into the training data in the text-to-video task with a tagging model, extracting core objects, concepts, and domains. We analyzed the domain-specific distribution of selected examples (*video_nsfw_filter*) via word clouds of these tags. Results are in Fig. 2 of the uploaded PDF.\n\nAs we can see, although the data pool with high video NSFW scores is the most diverse one, our model achieves the best performance on the data pool with low video NSFW scores, which is consistent with the argument presented by the left column of lines 296-298 in our submission.\n  \n`The CVPR24 work`\n\nThanks to bring this work into our attention. We respectfully highlight that the differences in conclusions stem from distinct experimental settings, which we believe lead to complementary insights:\n  1. **Data Curation Setup:**  \n   In their most relevant scaling experiments (Figs 5 & 11), the authors examine data filtering strategies by varying the *\"quantile thresholds\"* while keeping the filtering operator fixed. In contrast, we explore *different filtering operators* while maintaining the same \"quantile threshold.\"\n  2. **Task and Compute Setup:**  \n   Their work mainly focuses on DataComp and CLIP pretraining task, scaling compute with duplicated data. Beyond CLIP, we validate our data recipes across more tasks, including text-to-video generation, image-to-text pretraining, and post-training. For a detailed comparison, please refer to Fig.3 & Table 5 in our original paper, and the newly added InternVL experiments mentioned above.\n\n---\n\n### [Methods part] \"feasible, ... yet it appears impractical.\" \"This issue is common in data selection work, such as ...\"\n\n`HPO considerations`\n\nWe acknowledge that the cost analysis in our claim omits the coefficient introduced by HPO, under the practical assumption that *HPO is equally required for fair comparisons in large-scale scenarios*. In fact:\n- Large-scale systems often introduce additional complexities (e.g., debugging in distributed clusters, hardware/software errors), which can inflate the HPO coefficient significantly.\n- Industry practices typically rely on small-scale experiments to predict or generalize HPO for larger settings, as discussed in [1].\n\n[1] *Predictable Scale: Part I — Optimal Hyperparameter Scaling Law in Large Language Model Pretraining.*\n\n`The work \"Random Selection is Almost All You Need\"`\n\nWe thank the reviewer for pointing out it. However, our work differs from it notably in both scope and baseline comparisons:\n1. **Different Scenarios:**  \n   - Our work focuses on *four diverse multimodal tasks* (text, image, video), whereas the cited works primarily study *pure text post-training*.\n   - Our data selection strategy is derived from *40+ operators and 70+ performance metrics*, offering a broader and more adaptive framework compared to specific strategies studied in this work, like GPT-based scoring or heuristic methods (e.g., longest instruction).\n\n2. **Empirical Evidence of Feasibility:**  \n   - All numerical results in our small-scale experiments are benchmarked against a *Random baseline pool of equivalent data size* (lines 216 & 227).  \n   - The positive values reported in Table 3 and the Flops comparison across different scales demonstrate the effectiveness of our method in identifying solutions that consistently outperform random selection across various scenarios.\n\n--- \n\nWe appreciate the  valuable feedback and hope that these responses can address your comments, leading you consider an increase in the rating. Thank you once again!", "claims_and_evidence": "The primary contribution of the paper lies in its observations derived from comprehensive small-scale exploration. However, the generalizability of these conclusions, derived in  low data regimes, is questionable, particularly during specific model and task stages (eg the pretraining of a 2B MLLM). It remains uncertain whether these insights can effectively scale up with increased model parameters and data volume.\n\nFurthermore, the conclusions are not expressed with precision or sufficient validation . For example, the definitions of quality and diversity are based on the performance/feedback of minitest on the target tasks, with the number of combinations of different quality levels serving as the diversity metric, rather than widely acknowledged measures such as embedding or domain specification. The definition of diversity, based on varying OP scores or the entropy used in Sec.E3, works like simple bin sampling in traditional AL, may work but not effective, which makes the claim less persuasive. \nBesides, the conclusion differ from previous findings, such as Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic, CVPR24.", "methods_and_evaluation_criteria": "The cost of using the sandbox to identify high-quality training data is described as (1 + mr) ≤ M, where M represents the epochs required for training on the original data, m is the number of epochs times the number of OPs on the subset, and r is the subset ratio. The authors claim this cost is \"feasible,\" yet it appears impractical. In fact, the cost of searching for optimal hyper-OPs might exceed that of simply training on the full dataset. This issue is common in data selection work, such as \nDiddee etal. Chasing Random: Instruction Selection Strategies Fail to Generalize, 2024\nXiaetal. Rethinking Data Selection at Scale: Random Selection is Almost All You Need, 2024.", "theoretical_claims": "There is no theoretical claims in the manuscript.", "experimental_designs_or_analyses": "Yes I've checked the validity of the proposed experimental designs. The experiments conducted are comprehensive. My concerns are listed above.", "relation_to_broader_scientific_literature": "The significance of the task and the motivation behind the work are both significant. However, there are doubts regarding the practical applicability of the experimental design and the generalizability of the conclusions drawn from the study, as discussed above.", "essential_references_not_discussed": "Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic, CVPR24.\nActive learning for convolutional neural networks: A core-set approach. ICLR 2018.", "other_strengths_and_weaknesses": "The authors' engineering implementation and comprehensive experiments covering three different tasks should be acknowledged.", "other_comments_or_suggestions": "no further comments", "questions_for_authors": "no further questions", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IJkfnWunYJ", "reviewer": "Reviewer_5ZDF", "summary": "The paper introduces Data-Juicer Sandbox, an open-source suite that supports co-development of multimodal data and models. It proposes a feedback-driven approach in which data processing and model training are iterated together, rather than in isolation.\n\nA central component is the “Probe-Analyze-Refine” workflow, where data are filtered or enriched using various operators, then tested with smaller reference models. Feedback is collected on both data characteristics (such as quality and diversity) and model metrics (across multiple benchmarks), leading to insights on how specific data operations affect performance. These insights guide incremental recipe-building steps, combining or scaling up promising data operations to improve model outcomes at a lower computational cost.\n\nThrough experiments on three multimodal tasks (I2T generation, T2V generation, and I-T pre-training), the paper reports consistent gains in performance (including a top entry on the VBench leaderboard). The work further highlights how these data-driven refinements can transfer to larger-scale models and training runs, showing data efficiency gains and reinforcing the synergy between well-designed data curation and model training.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time, thorough evaluation and valuable feedback! Below, we address all your raised Weaknesses (W), Comments (C), Questions (Q) and Suggestions (S).\n\n> New results: https://anonymous.4open.science/r/icml_submission8414-7C89/rebuttal.pdf\n\n---\n\n## [C on Claims & Methods] \"Potential Gaps in Generality ... are within three tasks\"\nWe have added new experiments to enhance generalization verification, including scaling on the InternVL series (1B, 2B, 4B, 26B) and a new captioning task. For details, see reply to Reviewer CLJN.\n\n---\n\n## [C on Variance, W2 & Q3] \"... Without repeated trials ... \"\n\nYou may have missed this detail (e.g., error bars in Fig. 3). We have ran experiments 2–5 times, but included std in the appendix (lines 901, 991, Table 6) due to space limits. Note the reported performance reflects changes relative to *random pools*. Consistent *positive values* across trials demonstrate reproducibility and generalizability.\n\n\n---\n\n## [C on Literature; S1 & Q5] DaaR & JEST; \"Iterative routines...\"\nThank you for pointing out these works! DaaR was released (Feb 5) after the ICML submission ddl (Jan 9) and already cites and builds upon our work (see its Sec B.4). JEST's dynamics and your mentioned iterative sandbox align with future directions we aim to explore. We will cite both works in the final version and expand discussions accordingly.\n\nBesides, during the rebuttal, we added new experiments supporting iterative training: starting with ckpt0 → identifying recipe1 → training ckpt1 → refining recipe2 → training ckpt2. Interestingly, while recipes evolved, ckpt2 still improved performance. For details, see reply to Reviewer 3zp6.\n\n---\n\n## [W1 & Q2] OP Combination: Synergy vs. Conflict, Failure vs. Detection\nGreat suggestion! In Appendix E.3–E.5 (Table 8, Figs. 5–8), we provide quantitative analyses using Pearson correlations and entropy. We plan to extend this suite with automated and visualized lineage analysis tools. For example, t-tests can detect significant changes in filter outputs' stats dimensions after adjacent OPs. A practical use case: if an image-text matching filter notably reduces text length stats, it may indicate noisy captions affecting accuracy.\n\n---\n\n## [W3 & Q4] Impact of Distribution Shifts\nThank you for this important comment! While addressing distribution shifts robustly requires substantial effort (left for future work), we highlight:\n\n1. Empirically, our results already cover challenging distribution shifts (Table 5), where small-scale recipes benefit larger scales across model architectures, data, & compute.\n\n3. Methodologically, the framework’s open-source extensibility allows flexible customization of workflows while reusing data/model infra (see Sec. B.1–B.3). New hooks can integrate advanced strategies like dataset distillation [1] and intermediate tensor shaping [2]. Tools mentioned in [W1, Q2] also offer intuitive signals.\n   \n[1] Large scale dataset distillation with domain shift, ICML'24\n\n[2] Scaling for Training Time and Post-hoc Out-of-distribution Detection Enhancement, ICLR'24\n\n---\n\n## [S2~S4] Writing Improvements\nThank you! We have provided hardware details (lines 927–992, 1037–1041) and will expand the \"best practices\" subsection with summarized OP-specific labels from Tables 6–7 and Figures 2, 9. We’ll also unify reference formatting as suggested.\n\n---\n\n## [Q1] OPs Beyond Filters (e.g., Paraphrasing Q&As, Image Transformations)\nInsightful question! Per your request, we conducted new experiments on two representative Mappers using MGM:  \n- image_diffusion_mapper: Regenerates a new image for each sample based on the caption.  \n- image_captioning_mapper: Recaptions the existing image in each sample.  \n\n`Results`\n - As shown in *Table 3* and *Fig. 3* of the new PDF, *image_diffusion_mapper* significantly improves performance compared to the original images.  \n - Fig. 3 highlights a visual example before and after processing with these mappers. Core objects in captions are marked in red. We observe: (1) Diffusion models effectively locate and better present key objects (e.g., \"setting sun\") in generated images. (2) They also remove redundant information (e.g., watermarks, link text), likely contributing to performance gains.\n    \n---\n\n## [Q6 & Q7] Deployment & Sys Details\n- Appendix B discusses the architecture, capability factory, behavior hooks, extensibility. This decoupling enables rapid adaptation to new tasks while reusing workflows, helping us do quick new exps during rebuttal. The work is also already deployed in several industrial scenarios (specific entities undisclosed due to anonymity).\n- To simplify complexity, we are service-ifying it: RESTful APIs & MCP servers for OPs, unified & transparent environment switch for third-party modules. These efforts reduce programming complexity.\n\n---\n\n## [Q8] Reusability & Rating \nWe will incorporate these improvements into final version and hope they strengthen your confidence. Thanks again!", "claims_and_evidence": "The paper’s core claims—that a unified sandbox improves multimodal data and model co-development by (1) revealing useful data operations via small-scale experiments, (2) reducing trial-and-error costs through incremental “probe-analyze-refine,” and (3) boosting performance on tasks such as image-to-text, text-to-video, and image-text pre-training—are mostly backed by clear experimental outcomes. For example, the text-to-video experiments include leaderboard results on VBench, which supports the performance gains they claim. The authors also provide cost comparisons and scaling analyses, lending some credibility to the claim of more efficient and systematic co-development.\n\nPotential gaps lie in: 1) Generality: Most evidence comes from three tasks; more diverse tasks or larger sets of models could further strengthen or limit the conclusions. 2) Comparative baselines: Although they compare with existing methods (e.g., Runway’s Gen-3 or other pretraining strategies), it is not always clear that the baseline setups or alternative data-processing methods were exhaustively explored.\n\nOverall, while the main claims are supported by strong empirical findings on chosen tasks, further evidence across broader data types or tasks could make the presented framework’s benefits even more convincing.", "methods_and_evaluation_criteria": "The paper’s proposed probe–analyze–refine workflow aligns well with the intended goal of iteratively improving data and model quality. The authors use task-relevant metrics (e.g., MMBench for image-to-text, VBench for text-to-video, and standard contrastive-learning metrics for image-text pretraining), which are fairly standard and reflect the models’ target abilities. These evaluation criteria effectively show how data interventions translate into performance gains.\n\nA minor limitation is that most experiments are within three canonical tasks, so the broad generalizability of the chosen metrics or methods could be explored further. Even so, the approach is well-reasoned and uses suitable benchmarks to highlight measurable improvements.", "theoretical_claims": "The authors do provide a concise statistical bound (inAppendix C) that links performance changes observed in small-pool experiments to those expected at full scale, using a standard concentration inequality. Their derivation assumes (1) a simple i.i.d. sampling scheme from the original dataset and (2) that the relevant performance changes can be bounded within a fixed interval [a, b]. Under these assumptions, their stated probability bound,\n\n$\nP[\\Delta_{\\text{pool}} - \\mathbb{E}[\\Delta_{\\text{full}}]\\ge\\epsilon] \\le\n\\exp\\bigl(\\tfrac{-2\\epsilon^2}{(b-a)^2}\\bigr),\n$\n\nis a straightforward application of Hoeffding’s inequality. The argument appears correct on its face, though it is relatively high-level and does not deeply address issues like data distribution shifts or non-i.i.d. data. In other words:\n\n- If the dataset and performance metrics meet the i.i.d. assumptions, the bound is formally correct.  \n- If there is strong distribution mismatch or outlier-heavy data, the theoretical guarantee would require additional considerations.  \n\nThe paper does not provide deeper or more elaborate proof, and no immediate errors stand out. The result is intended mainly as a heuristic justification for why small-pool insights might translate to the full dataset.", "experimental_designs_or_analyses": "The paper’s “probe–analyze–refine” workflow is generally well-conceived: they start with single-operator data pools to observe isolated effects, combine top-performing operators, and then scale data. This incremental approach reduces trial-and-error costs while producing comparative feedback. They also use a consistent set of baseline models and hyperparameters so that only the effect of data changes is measured.\n\nOne caution is that they rely on single-pass experiments (with no mention of repeated runs) to quantify the impact of each operator or recipe. While this is typical for many large-scale training studies (due to resource limits), it means we don’t see variance estimates that might highlight sensitivity to random seeds or sampling. Additionally, there is an assumption that small-scale subset behavior extrapolates to larger scales. They do provide a brief theoretical justification and some empirical scaling analyses, but distribution shifts or non-i.i.d. sampling could reduce perfect transfer.\n\nIn short, the design is sensible and largely valid for controlled comparisons, but it would be stronger with additional runs or variance reporting to confirm each operator’s reliability in improving results.", "relation_to_broader_scientific_literature": "### Multimodal Model Training: Traditional vs. Co-Development  \n\n- **Traditional Training:** Treats data and model design separately; data is fixed before training, limiting feedback.  \n- **Co-Development:** Data-Juicer integrates data and model training through an iterative “Probe-Analyze-Refine” workflow, improving both simultaneously.  \n\n---\n\n### Data-Centric AI vs. Data-Juicer  \n\n- **Traditional Data-Centric AI:** Focuses on dataset curation and filtering before training, lacking real-time feedback.  \n- **Data-Juicer:** Iteratively refines datasets using model performance, optimizing data selection dynamically.  \n\n---\n\n### Cost-Efficient Co-Development  \n\n- **Challenges:** Large models require costly computation; efficient iteration is crucial.  \n- **Data-Juicer’s Solution:** Uses small-scale experiments before full training, reducing unnecessary compute.  \n\n---\n\n### Optimizing Data Selection  \n\n- **Data-Juicer’s Insight:** Task-specific data strategies—image-text models benefit from diversity, video models need high-quality samples.  \n- **Compared to Other Methods:** Unlike RL and gradient-based selection, Data-Juicer enables empirical testing of data modifications.  \n\nBoth reinforce feedback-driven data selection, optimizing dataset quality alongside model training.", "essential_references_not_discussed": "The paper does not cite Ling et al. (2025), \"Diversity as a Reward\" (DAAR), which introduces a structured framework for optimizing data diversity during fine-tuning. DAAR provides an automated approach to balance inter-domain and intra-domain diversity, using model-driven feedback to guide data selection. This contrasts with Data-Juicer's human-guided, iterative approach to dataset refinement.\n\nAdditionally, JEST (Evans et al., NeurIPS 2024) formulates data selection as a dynamic optimization problem, significantly reducing training iterations by prioritizing high-value samples. This aligns with Data-Juicer’s goal of efficient data-model co-development but takes a different approach by embedding selection directly into the training loop.\n\nLing, Zhenqing, Daoyuan Chen, Liuyi Yao, Yaliang Li, and Ying Shen. \"Diversity as a Reward: Fine-Tuning LLMs on a Mixture of Domain-Undetermined Data.\" arXiv preprint arXiv:2502.04380 (2025).\nEvans, Talfan, Nikhil Parthasarathy, Hamza Merzic, and Olivier Henaff. \"Data curation via joint example selection further accelerates multimodal learning.\" Advances in Neural Information Processing Systems 37 (2024): 141240-141260.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a tangible, open-source sandbox framework that addresses data-model co-development – a recognized but underexplored area in multimodal research.\n2. The “Probe–Analyze–Refine” strategy is clear, methodical, and well-presented. It extends beyond high-level theory by offering a practical pipeline (with code) that other researchers can use.\n3. Empirical results underscore the potential cost savings of systematically iterating on small data subsets before scaling up – valuable in a field where full-model training costs can be huge.\n4. Demonstrations in image-to-text, text-to-video, and image-text pretraining are reasonably diverse, strengthening the paper’s impact and showing general applicability.\n\nWeaknesses:\n\n1. No Formal Validation of Combining Operators:\nWhile the paper clearly shows individual operators’ effects, the process of stacking multiple top operators sometimes degrades performance. It remains unclear why certain combinations interact negatively. For instance, a multi-operator strategy could preserve only a narrow slice of data distribution or inadvertently filter out complementary samples. A deeper exploration of these “operator interaction” phenomena (e.g., synergy vs. conflict) would strengthen the paper’s argument that combining top operators is not trivially beneficial.\n2. Limited Empirical Repetitions:\nThe paper often uses single-run experiments for each operator/recipe, presumably to control computing. However, large-scale training can show variance in results (random seeds, sampling). Without repeated trials or uncertainty estimates, it is difficult to assess how reproducible (or fragile) these improvements are. The text occasionally notes that cost constraints make multiple runs challenging, but even small-scale tests repeated 2–3 times would add confidence that results do not hinge on luck.\n3. Unclear Impact of Distribution Shifts:\nThe framework assumes that small-pool “probe” experiments reliably anticipate behavior at full scale. Yet in practice, distribution shifts or domain mismatches could exist between the reduced sampling and the real-world training set. In particular, data rebalancing or filtering might produce biases that only manifest when the entire large dataset is used. More discussion or case studies on how best to mitigate such shifts would be valuable.", "other_comments_or_suggestions": "1. While the blueprint for combining data-centric methods (e.g., filtering, annotation generation) with model-centric improvements is very appealing, the Data-Juicer Sandbox as presented does not clearly include many mechanisms for the model’s evolving needs. In other words, there could be additional iterative routines where model improvements inform new data requests or automated prompt adjustments. Explaining how the pipeline might be extended to facilitate continuous model-data evolution would strengthen the paper’s vision.\n2. Consider adding a “best practices” subsection in the appendix, summarizing lessons about which operators and metrics are especially critical under different conditions. This might serve as a quick reference for new sandbox users.\n3. The text occasionally references “industry-level” or “production-ready” usage; clarifying any special hardware or cluster requirements would make it clearer how to adopt this in large-scale settings.\n4. Minor editorial note: Some references in the bibliography appear slightly inconsistent in formatting (e.g., capitalization in titles). It may help to standardize them.", "questions_for_authors": "1. Why All Operators Are Filters? The paper positions several “operators” (OPs) as different filtering or perplexity-based selection procedures. It is not fully clear whether the pipeline includes other augmentation methods or more diverse operators (e.g., paraphrasing Q&As, image transformations). Highlighting (or adding) such diversity might boost the effectiveness and generality of this work.\n2. Could you elaborate on why the highest-performing individual operators sometimes fail to improve results when combined? Are certain filters redundant, or do some systematically remove data that others rely on? How might future versions of the sandbox help diagnose when operators conflict?\n3. Do you have preliminary variance estimates (e.g., dev set performance standard deviations) for a small subset of your probe experiments? If so, how large are the fluctuations? Knowing approximate variability would help judge whether a single-run improvement of, say, +1% is robust or possibly random.\n4. Have you found any real-world cases where an operator looks beneficial in small-pool settings but fails to extrapolate at full scale (or vice versa)? If so, how do you suggest researchers detect and mitigate such mismatches early on?\n5. One appealing application would be continuously refining data as new samples appear (e.g., updated data streams). Would your sandbox accommodate dynamic additions to the dataset, or is it currently designed only for static snapshots?\n6. You mention “industry-level” usage in passing. Do you envision a canonical workflow for large organizations to adopt Data-Juicer Sandbox, or do you foresee each group creating highly customized pipelines? A brief discussion of how you see this platform evolving in real deployments would be illuminating.\n7. Because reproducibility and generalization are important for future work, could you share more implementation details or guidelines to ensure that this framework can be adapted to different tasks? How to ensure the complex codebase can be easily adapted and reused.\n8. I intend to give a rating of \"accept\" due to the potential impact of this work. However, I am still concerned about the reusability of this work. I will consider raising my rating further if all concerns are addressed properly.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a framework for multimodal data-model co-development. While reviewers initially raised concerns about scalability and baselines, the authors addressed these through additional experiments. The work demonstrates practical value across multiple architectures and tasks. Reviewers acknowledged the improvements, and the paper's contributions outweigh remaining concerns. ​The AC recommends acceptance of the paper. Congratulations to the authors!
Decision: Accept (spotlight poster)