Counterfactual Evolution of Multimodal Datasets via Visual Programming
Authors: Minghe Gao
Zhongqi Yue
Wenjie Yan
Yihao Hu
Wei Ji
Siliang Tang
Jun Xiao
Tat-Seng Chua
Yueting Zhuang
Juncheng Li
Venue: NeurIPS 2025 poster ()
Keywords: Dataset Evolution
Visual Programming
Counterfactual Evolution
Multimodal Datasets
Primary area: deep_learning

Abstract: The rapid development of Multimodal Large Language Models (MLLMs) poses increasing demands on the diversity and complexity of multimodal datasets. Yet manual annotation pipelines can no longer keep pace. Existing augmentation methods often follow fixed rules and lack verifiable control over sample diversity and reasoning complexity. To address this, we introduce Scalable COunterfactual Program Evolution (SCOPE), a framework that uses symbolic Visual Programming to guide program evolution via counterfactual reasoning. SCOPE performs the three steps of counterfactual inference: (1) Abduction, by generating verifiable programs to model reasoning associations; (2) Action, by intervening on program structure along three axes—reasoning path, visual context, and cross-instance composition; and (3) Prediction, by categorizing evolved instances by difficulty, structure, and input multiplicity. Based on this process, we build SCOPE-Train and SCOPE-Test, evolving benchmarks with expert validation. To support training, we propose MAP, a curriculum learning strategy that aligns model capacity with sample difficulty. Experiments show that SCOPE improves reasoning performance, exposes model blind spots, and enhances visual dialog capabilities.

have been foundational to the development of powerful Counterfactual Evolution of Multimodal Datasets via Visual Programming
Zhongqi Yue'; Wenjie Yan', Yihao Hut, Wei Ji' Siliang Tang', Jun Xiao', Tat-Seng Chua², Yueting Zhuang', Juncheng Li
Minghe Gao
Zhejiang University
Nattiona University Singapore
NNnyynng Technological University *Hainan University 'Nanjing University (minghegao, 22551068, siliang, junx, yzhuang, junchengli j.zju. edu. on saneyuuezhonngi@@mmaii com, 20223003513thainan edu on wei com, chuats@ nus. nuw..uuuuuuuu.....
Abstract
The rapid development of Multimodal Large Language Models (MLLMs) poses increasing demands on the diversity and complexity of multimodal datasets. Yet manual annotation pipelines methods often follow fixed rules and lack verifiable control over sample diversity and reasoning complexity. To address this, introduce Scalable COunterfactual Program Evolution (SCOPE), a framework that uses symbolic Visual Programming guide program evolution via counterfactual reasoning. SCOPE performs the three steps of counterfactual inference: (I) Abduction, by generating verifiable programs model reasoning associations; (2) Action, by intervening program structure along three axes-reasoning path, visual context, and cross-instance composition; and (3) Prediction, by categorizing evolved instances by difficulty, structure, and input multiplicity, Based on this process, we build SCOPE- Train and SCOPE-Test evolving benchmarks with expert validation. To support training, we propose MAP, a curriculum learning strategy that aligns model capacity with sample difficulty. Experiments show that SCOPE improves reasoning performance, exposes model blind spots, and enhances visual dialog capabilities.
no longer keep pace. Existing augmentation
Introduction
Multimodal datasets 37
21 5 28
36 have devoted substantial effort to collecting and curating data for both the pretraining and evaluation, aiming to enhance the quality of raw datasets. An idcal dataset should be sufficiently diverse and scalable, with controllable attributes such difficulty, input modality, and the ability to explicitly reflect reasoning processes. However, existing datasets mostly static, lack diversity, rely heavily manual annotation that no longer keep pace with model development. The growing mismatch between the speed of dataset evolution and model advancement has become critical bottleneck in developing more generalizable and robust MLLMs. systematic dataset evolution framework is urgently needed to address the limitations posed by low-quality data resources.
MLLMs. Recent works [10] 22
35
Several recent efforts have attempted to modify existing datasets, which can be broadly categorized into three approaches. A: Prompt-based transformations using large models, such MAmmo TH- VL 16 CoSyn 411, and OmniSearch 20 These methods feed original VQA or image-question
"Equal contribution. Conresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A: Prompt-based Transformation Lack of verifiability & Model dependency
Verifiable Abduction
SCOPE
Generate more challenging somple based the QA poir or coption.
Prompt
Image
QA
Caption
VQA Pairs
Code-Pretrained Model
Program Representation
B: Template-based Augmentation
No difficulty increase Neglect logic
Controllable Action
Edit
Only edit the image or only modify the query.
Image
Query
Image
Query
more variations
-======
C: Decomposition basses Extraction
Limited diversity & Globe context loss
Evolving Prediction
Extract new QA pairs
Difficulty
Image
Extract
Scene Graph
based on subgraph or decompose into sub-query.
Evalution Direction
Structure
>>
Query
Decompose
Sub-query
input
VQA Pairs
Evoluted
Figure 1: Comparison between current data evolution methods and SCOPE.
pairs into large model and prompt it to generate more challenging examples. However, this approach lacks verifiability, heavily depends on the model's capabilities, and contradicts the original intention of improving models through data rather than relying on the model to generate data. B: Template- based augmentation, as seen in VLB [42 and SK-VQA (38), where predefined operations are applied to edit questions, modify images, incorporate external knowledge to produce more diverse samples. Nonetheless, these methods do not increase sample difficulty and often overlook the underlying reasoning logic of the original problems, potentially rendering the augmented data ineffeettvee C: Decomposition- based local extraction, exemplified by AGQA-Decomp 13] and Provision [45).
These methods split complex questions into sub-questions or represent images scene graphs to generate fine-ge -me-griined samples from a local perspective. Yet, this leads to the loss of global context, accumulation of local biases through multiple iterations, and limited diversity. We summarize these limitations in Figure [ Overall, they remain far from the ideal of dataset evolution-namely, verifiable transformations, controllable difficulty enhancement, and diverse sample generation.
To address these limitations, we introduce a counterfactual approach and adopt Visual Program- ming [39 as the core tool. Visual Programming leverages code- pretrained models to synthesize exccutable programs for vision tasks, where vision- saiiun-a reasoning is represented through sym- bolic code. This allows to explicitly construct and trace a reasoning chain in fully interpretable and verifiable manner throughout the entire process, laying the foundation for structured counterfac- tual inference through abduction, action, and prediction.
Specifically, we propose SCOPE, a three-stage dataset evolution framework: In the verifiable abduction stage, we generate Python programs for each vision-language task and link the verifiable code statements explicitly to the underlying reasoning paths. In the controllable action stage, enable targeted extensions along three dimensions: reasoning path expansion, visual context editing, and tross-instance composition, thereby allowing for systematic and proactive intervention in the reasoning process. In the evolving prediction stage, instances categorized based on reasoning difficulty, structural variation, and input multiplicity relative to the original samples, enabling continuous and diverse evolution from the source data. Finally, propose MAP, curriculum strategy that enables fine-gr greined scheduling, targeted module activation, and adaptive difficulty weighting, forming training paradigm coupled with SCOPE's 's program-centric design.
Building on SCOPE, we construct the benchmark sets SCOPE-Train and SCOPE- Unlike static benchmarks, they support continuous evolution toward increased diversity and complexity. The test set was curated and verified by domain experts, achieving high approval rates during manual review. Static and comparative results against prior benchmarks are shown in Figure 2a and 2b
Extensive experiments validate the effectiveness of SCOPE across multiple dimensions: (1) SCOPE- Test reveals systematic blind spots in current models that remain undetected under conventional benchmarks, with expert review confirming the high quality of generated samples. (2) Models of varying scales benefit from SCOPE, exhibiting a clear scaling trend in reasoning performance. (3) Models trained with SCOPE demonstrate strong easy-to-hard generalization and achieve notable gains on visual dialog tasks. (4) The MAP curriculum learning framework further enhances training efficiency by aligning sample difficulty with model capacity across iterative expansion rounds. In summary, our contributions are threefold:
2

---

16967
7286
Interrestable
Controllable Expansion
Explicit
Scalable
$719
5745
2449
25759
Benchmark
SOCPE
SCOPE
MME
X
X
X
Train
2393
Test
1043
[2]
MMBench
X
X
3081
1391
31
MME-CoT
12003
5195
3190
6954
2913
Provision
X
X
X
Easy Dusth Espard
Meduum Balanced
Hard Width Bepard
SCOPE-Test
Singi rist
Duskrpuu
Wultinppt
(b)
Figure 2: (a) Data distribution of SCOPE-Train and SCOPE-Test. (b) Benchmark comparison.
We propose SCOPE, which constructs verifiable associations, enables controllable interventions,
and supports counterfactual evolution through symbolic program.
We construct SCOPE-Train and SCOPE-Test, a continuously evolving benchmark supporting scalable training and fine-grained diagnostic evaluation.
We introduce the Memory and Attention Path (MAP) Learning, which aligns training with SCOPE's
structured difficulty progression, improves generalization easy-to-hard samples.
Related Work
2.1
Visual Programming Edition.
has recently emerged as a promising paradigm for generating in- terpretable and modular reasoning pipelines in multimodal tasks by utilizing neural symbols Python modules for task synthesis and execution. Unlike black-box end-to-end models or purely textual Chain-of-Thought (CoT) generation, it enables fine-grained control over the reasoning process through code editing. Prior work has explored its potential from various angles: De-fine [5] inte-
Visual Programming 39 17
grates feedback loops improve code generation quality, while VPD [9] and Fact [4] transform execution traces into explicit reasoning paths for distillation learning. Although Provision 45 also
uses Python code for data generation, remains restricted by predefined scene graphs and lacks broader data-space adaptability. In contrast, our SCOPE framework advances Visual Programming from generation tool to a controllable benchmark construction mechanism. By supporting both program-leve augmentation and iterative evolution, SCOPE enables structured dataset expansion.
2.2 Counterfactual Data Generation
33 4 has emerged promising strategy for improving model robustness and mitigating language biases in multimodal learning. Approaches such as COCO- Counterfactuals (25 and Counterfactual Prompt Learning (CPL) [T8] generate hypothetical variants of image-text pairs or prompts to challenge spurious correlations. Notably, Counterfactual Samples Synthesizing (CSS) (6) introduces a model-agnostic training scheme that creates counterfactual examples by masking critical objects in images or keywords in questions, enhancing both visual- explainable and question- question-sensitive capabilities of VQA models. However, most existing methods rely on heuristic rules and lack fine-ge mme-ggrined control over reasoning complexity. In contrast, our SCOPE framework integrates counterfactual generation into structured causal learning paradigm via symbolic Visual Programming, enabling interpretable and scalable dataset evolution.
Counterfactual data generation
3 Method
To enable controllable dataset evolution, we introduce SCOPE, as illustrated Figure 3 Starting from an maage-question-nnsweee triplet, we employ Visual Programming to generate Python program, retaining only those that execute correctly (Section 3.T). This executable representation serves as the foundation for structured expansion: reasoning path expansion, visual context editing, and

---

Uncontrollable
OF
A New QA Polrs
Lock diversity
images
Pairs
Coptions
Template
Blased expansion
FSTE
Berative
Original Samples
Program-based Proactive Intervention
Counterfactual Cotegorlaation
Pre-train Model
Train Test
Training
def execute_ execute_commante command(
def execute command(
function()
= function()
function()
Controliable
Exponsion
@
Evolution
Unblosed
function()
A
3
function()
return
Figure 3: The pipeline of SCOPE: generate executable visual programs for the given instances, enabling iterative controllable expansion and categorization of their evolutionary progression.
cross-instance composition (Section 3.2) The expanded instances are then categorized by reasoning difficulty, structural variation, and imput multiplicity (Section (.3). Finally, we introduce MAP that aligns model curriculum training with the structured evolution of the dataset (Section 3.4.
3.1 Verifiable Abduction through Visual Programming Generation
Programs, with their explicit logic and variable dependencies, offer greater verifiability and inter- pretability than natural language. Leveraging Visual Programming, we translate image-question pairs into executable Python code, enabling fine-grained manipulation of inputs and functions. This symbolic form serves as verifiable abduction of the reasoning path, ensuring structural coherence and semantic grounding. Moreover, code-based reasoning aligns with the strengths of pretrained models program synthesis, supporting efficient knowledge integration and streamlined expansion.
More formally, given an image æ and its corresponding query 4. we use program generator л (e.g.. GPT-4o) to produce executable code z = (2, q). This program is then evaluated by an execution engine 6(2, x) that applies to the input. If the program output matches the ground-truth a, the instance is retained. Each valid instance represented as four-tuple (g, I, 2, a) comppriis the question, image, program, and answer answer-which serves as the foundation for all subsequent expansions. Additional implementation details, including API calls and prompts, are provided in the Appendix.
3.2 Controllable Action by Proactive Intervention
To address the limitations heuristic or template-bas based augmentation. SCOPE introduces structured expansion paradigm grounded in symbolic reasoning. Through program-level control, it supports systematic intervention from simple to complex instances via logic composition, visual variation, and cross-source integration. Specifically, we propose three expansion strategies:
Reasoning Path Expansion. We use modular API library of atomic functions (e.g., find, exist) that encapsulate distinct reasoning steps. These functions can be composed or nested within program structures increase reasoning depth width. For example, extending "How many apples there?" to "How many red apples are there?" involves inserting a veri "y_properrt call, This compositional scheme ensures extensibility and preserves logical traceability for analysis.
Visual Context Editing. To diversify visual contexts, we perform localized edits on segmented image entities using tools like SAM 224] By replacing, altering, removing objects, generate fine-grained visual variations while holding the program structure constant. This method supports incremental complexity increases in visual reasoning and mitigates biases arising from overexposure to image, promoting controllable generalization.
Insst-Inttance Composition. To enhance cross-domain diversity, SCOPE expands single-instance tasks into multi-image or multi-source settings by combining examples from different datasets with semantically aligned questions. This approach maximizes code reuse through shared program

---

Counterfactual Categorization
Structurol Variation
Reasoning Difficulty
t def evecute commondi imagop
image_ potch = mnogeRatch(mogee)
Image
Halstead
x
aaplc_p image_patte fadthh fodd"apple"
image potch
Depth
tog.
lenfopple patches)
return answer
apple opple_patches potcches
|++ Width
DXV
def commondy magel. imagel]:
E 323
mageaatenfmagee))
image. potch2 a imageRotch//man apple potches lmoge potch1 fiedd apple""
Imagez
Image2
E 2146
Easy to Hord Generolization
opple_patches apple_pppeeee poplc_potches append
Image ppttoh
poten2
(imgge patch2 "ppll")
red apple potches =]
Input Multiplicity
for potch
appple patch.veniyy property" oppple" 'ted") Depth
petches:
upple_patch
Single-Input
resl spple_ http:p/ptteeee.ppe/ooyoo potcn)
Je-+WWddhh
red hapll_pptchee
lenfred apple. patches)
answer
Dual-input
Figure 4: SCOPE categorizes the data based on changes in task difficulty, program structure, and input modalities before and after code expansion, involved images can be found in Appendix.
structures and reduces visual redundancy. It enables composition across hetcrogeneous domains, facilitating richer task representations while avoiding contamination typical in monolithic datasets.
3.3 Evolving Prediction through Counterfactual Categorization
Following the intervention, each instance is accompanied by a structured program representation. Unlike natural language, code offers explicit syntactic and semantic regularities, enabling systematic comparison with the original program and categorization: reasoning difficulty, dependency structure, and input multiplicity. The figure illustrates the analysis of these three dimensions.
Reasoning Difficulty. The complexity of program reflects the cognitive effort required to solve the corresponding task. We adopt the Halstead effort from software engineering to assess difficulty:
N₂
= DxV where V N Nlogn,
12 + 122
D
2
112
Here, n,, n2 are the counts of unique operators and operands, and N₁. N2 are their total occurrences, with + 122, N N1 N2. A higher E reflects increased reasoning complexity. We use this metric in different threshold to classify the instance into three levels: easy, medium, and hard.
Structural Variation. We represent each program as directed acyelic graph (DAG). where nodes denote variables and edges capture dependency relations. Depth (longest path) and width (maximum fan-in) characterize the structure. We define depth and width expansion factors as relative increases in these values post-expansion. An expansion is classified as depth, width, or balanced, depending on which factor dominates or whether both expand comparably. This taxonomy ensures balanced representation of program complexities, avoiding overrepresentation of singular structural patterns.
Input Multiplicity. Expansions may also vary the input modality, transitioning from single-image to multi-image VQÀ tasks. We track the number of distinct input sources per instance proxy for input complexity. This metric enables separation of reasoning complexity from input diversity and supports nuanced analysis of how cross-modal integration contributes to generalization.
3.4 Memory and Attention Path Curriculum Learning
SCOPE offers a natural foundation for curriculum learning by supporting the generation of samples with progressively increasing difficulty. To fully leverage this property, design progressive curriculum learning framework that aligns with SCOPE's controllable evolution paradigm.
We first define a curriculum scheduler that ranks training samples by increasing difficulty, prioritizing (1) input multiplicity, (2) structural variation, and (3) reasoning complexity. For example, single- image, depth-oriented, low-complexity samples are treated as easiest, while multi-image, width- oriented, high-effort samples are considered hardest. structural variation, we heuristically regard width expansions more challenging than depth ones, rationale that will be further analyzed in the
5

---

Query:
Query:
fx
How mony donuts
types of donuts image?
Image:
Program:
Image:
Progrom:
dej execute_ commandjjmugee:
def exccute, cammanddimage image, sotch-linogoPetchiiii
imoge potch imagePatch/mage)
donutpootches imupe_potch, potch.fied" [ind["Vonut" "Vonut")
donut potches-imoge patch,find" "onuu""""""")
countsienjdonut patches) return donut, count
for donut donut potches:
types- seti)
donut_ Lonk__yypeooodd simple query
("What type of donut is this?")
donut, types, adees.dd [ype)
return donul_
Answer:
22
Answer: l'shocolate "plazed
Query:
Query:
What types desserts there image?
What of desserts there both images?
Image:
Program:
Image:
Progrom:
def exccute commandfimagel:
+
enecute commanddjiagee
image_ petteeinogeeonccmmmpr
dessert (ypesmeet))
dessert_ potchesewnogge __oo___pphhhhh findl findd"desssr"" dessort
image image,
dessert, [ypees seti)
imoge potth-ilnogePotchimmgee
for dessert dessert pootche:
dessert putches timage patch find "dessert")
dessert type:dessert simple
for dessert dessert potches:
('What type
this?")
dessert type= dessert simple query
types odd[dessortt type)
("What type dessert is this?")
return dessert types
dessert addjdessert (ypes
Answer:
("boke", "cupcoke")
("coke", "oupcolie" "macaroon""
crnam",)
Figure 5: An example of SCOPE evolution and the annotation format is provided in the Appendix.
experimental section. Next, we introduce two adaptive modules: a memory-augmented branch M for depth-oriented tasks, and a parallel attention branch .A for width-oriented tasks. A gating function JF selects the appropriate path based on the sample type.
M(x;),
if r; depth,
A(x;), F(x)) =
r; width,
{M(x) + A(x)) if r, balanced.
Third, we incorporate difficulty-aware loss function. In addition to the base training objective, apply a normalized Halstead effort score (from Section 3.3) as a dynamic weight to modulate each sample's contribution to the loss.
E, ΣØ=1
Li Fo L.me
Eg
1
Lootal
Es Liase
Y1
This encourages the model to adaptively allocate learning focus according to reasoning complexity. Together, these components enable effective curriculum-style training over SCOPE's structured expansions, yielding models with improved reasoning capabilities.
SCOPE Benchmark
Dataset collection and construction. The benchmark is built by applying the SCOPE framework perform program-guided expansions over samples from six widely used vision-language datasets: SEED-Bench2 (26). MME (23), MM-Bench (31] GQA 221) OK-VQA [34, and Tally-QA (2). To avoid information leakage and ensure fair evaluation, only the official dev or test splits are used for expansion. Detailed statistics on dataset composition are provided in the Appendix, and static distribution of different attributes is shown in Figure 2a Based on Halstead effort scores, we define reasoning difficulty using two empirical thresholds: 4000 and 6000, which separate instances into easy, medium, and hard levels. The final dataset is divided into SCOPE-Traie and SCOPE-Tes. with 70:30 ratio. Compared with traditional static benchmarks, SCOPE enables program-centric, multi-round augmentation and difficulty-awarr diagnostics. A comparative summary with prior datasets presented Figure 2h We also provide a statistical overview of the question distribution across the three evolution stages of SCOPE, which can be found in Table [

---

Table 1: Question distribution statistics of SCOPE across training and sets.
Question Type
SCOPE-Train (Number Proportion (%) SCOPE-Test (Number Proportion
Counting
4,309 13.8%
1.579/ / 11.8%
Object Attribute (color, texture, shape)
6,471 20. 7%
2,534 18.9%
Spatial Relationships
5,377 17.2%
2,589/ / 19.3%
OCR-based Reasoning
6,127 / 19.6%
2,411 / 18.0%
Multi-hop
5,903 /
2,958 22.
Others
3,046 / 9.8%
1,318 9.9%
Total
31,233/ / 100.0%
13,389 100.0%
Table 2: Comparison of various models on SCOPE-
Model
Reasoning Difficulty
Structural Variation
Input Multiplicity
Easy
Medium
Hard
Depth
Balanced
Width
Single-input
Multi-npuu(>==)
Overall
GPT-4o
93.1
87.5
79.1
89.5
88.4
88.2
89.1
86.0
88.54
LLaVA--L5--1B
71.8
67.2
46.7
69.9
63.7
63.6
67.3
53.8
64.81
DeepSeek-V1--71
74.6
71.4
55.2
73.4
71.6
64.7
71.2
60.5
69.27
Vision-4B
74.0
66.1
47.0
68.8
66.1
64.2
67.7
57.6
65.85
Qwen2.5-VI-7B
86.4
77.1
67.9
83.5
81.2
76.9
81.2
74.4
79.97
Provision
70.8
84.7
83.4
83,4
74.5
81.78
VLB
88.9
80.6
73.4
85.6
84.9
80.7
85,4
74.5
83.41
SCOPE
92.8
84.7
76.7
89.7
87.0
86.2
89.3
77.7
87.20
IntenVL-22.5-28
83.1
76.7
60.1
79.3
77.0
73.9
77.8
69.1
76.22
Provision
83.5
77.3
63.6
79.9
79.1
74.4
79.0
70.2
77.41
-VBB
84.8
78.9
68.5
82.1
83.3
74.4
81.7
70.4
79.63
SCOPE
87.6
70.5
85.2
83,9
78.5
83.7
74.3
82.03
Human acceptance analysis. To assess the correctness and coherence of the SCOPE-Test instances, conducted human evaluations across three successive rounds of expansion. For each round, 1,000 randomly sampled instances were reviewed to verify the alignment between program outputs and target answers. The resulting human acceptance rates were 96%, 92%, and 91%, respectively, confirming that SCOPE generates logically valid and progressively challenging instances.
Evaluation metric and open-source evolution. We use accuracy as the evaluation metric, computed along the three evolution inmeension-ddiffcculi dependency structure, and input modality-to assess model performance under diverse expansion types. To promote transparency and reproducibility, we release the full SCOPE benchmark along with tools for program program-based. based, controllable dataset expansion. More importantly. SCOPE is designed as generalizable framework rather than a fixed dataset: encourage researchers to apply across domains evolve their own multimodal benchmarks. By enabling community-drivenn multi-round dataset construction, SCOPE provides foundation for open, collaborative ecosystem of benchmark development and evaluation.
5 Experiment
To evaluate the SCOPE methodology and its benchmark, we conduct series of comprehensive experiments. Section outlines the experimental setup. We then present three main evaluations: (1) SCOPE-Test enables fine-grained diagnosis of existing models across multiple reasoning dimensions (Section 5.2); (2) SCOPE-Trane improves reasoning performance across models with different parameter scales (Section 5.3); and (3) MAP's staged learning process yields notable gains in visual dialog tasks requiring sequential, context-aware reasoning (Section 5.4). Finally, we conduct ablation studies to assess the contributions of each component in our framework (Section 5.5)
outlines
5.4))
5.1 Experimental Setup
Model Setup. We train multimodal models using SCOPE- Train on two architectures of different scales: Qwen2.5-VL. LB (3] and Inter [7]. For program generation, we use GPT-4o [2]] a multimodal language model, to produce executable Python code. Unlike ViperGPT (39,, which targets end-to-end VQA, SCOPE focuses on dataset expansion and evolution. To support this, both the input image and ground- answer are provided as prompts to guide more accurate, task-aligned program synthesis.
7

---

Table 3: Comparison of various models and methods different benchmarks.
Model
MMVoturee
RealWordOA
MMBeach,ar
SeedBench2-Plus
MuirBench
MMT vol
POPE.ng
GPT-4o
69.1
75.4
83.4
72.0
68.0
65.4
86,9
LLaVA-I.5-13B
38.3
55.3
64.3
24.4
52.1
85,9
DeepSeck-VL-V VL-7B
41.5
54.2
73.2
54.0
88.1
Phi-3.5 Vision-4B 4B
43.2
53.6
76.0
62.2
53.6
Qwen2.5-VL-7B
68.5
83.5
60.9
Provision
66.4
69.1
82.7
70.1
59.8
59.3
85.6
VLB
66.8
69.0
84.1
70.7
61.3
61.4
84.8
86.8
72.3
65.2
62.6
87.2
InternVL-225-2B
60.8
60.1
74.7
60.9
40.6
54.5
90.6
Provision
60.5
61.0
73.3
61.4
39.7
52.6
89.3
60.8
60.7
75.4
40.9
55.7
90.5
SCOPP
63.4
63.0
77.0
62.1
44.3
55.2
90.7
Settings. To ensure fair and comprehensive comparison, we evaluate SCOPE against three represen- tative evolution strategies:
This method parses images into scene graphs and extracts subgraphs by selecting specific nodes and edges, thereby generating new instances while preserving relational structure.
Provision 45
VLB 42): This strategy rewrites questions and edits images through predefined templates, recombin- ing them to synthesize new instances without altering the original answers.
SCOPE (Ours): Our method constructs executable programs from original samples and systematically expands them along functional composition, input modality, and reasoning complexity, enabling scalable, unbiased, and controllable dataset evolution for multimodal benchmarks.
Baseline and Benchmark. To further assess the performance and generalizability of SCOPE, include several strong multimodal baselines spanning different model scales: GPT-4o (22, LLaVA- 1.5-13B (29). "DeepSeek-VL.. VL-7B [32). and 7ni-3.5-Vision-44 [0] For evaluation, adopt diverse suite of benchmarks covering both general-purpose and domain-specific tasks: MM Vet
[44), Real-
WorldQA 8 MMBench [31 for broad visual reasoning and real-world contextual understanding, Seedbench2-Plus (26) for text-rich visual comprehension, MMT 43]. MuirBench
40 for multi-
image inference, and POPE (27 for robustness against hallucination. Notably, the MMBench test set is not public, and SeedBench2 -eeunchhh-plssscee does not overlap with SeedBench2. Our expanded data is distinct from all test benchmarks, ensuring no risk of data leakage. We additionally use ConvBench [30].
ContextVD [4). and VisDial [9] to assess Visual Dialog capabilities.
5.2 MLLM Evaluation on SCOPE-Test
We evaluate existing multimodal models on SCOPE-TesI, focusing on performance across the three structured evolution dimensions. Results are summarized in Table 2 and key observations are as follows. Reasoning Difficulty. Model performance degrades substantially on hard instances, which typically require multi-hop reasoning or the integration of multiple intermediate results. This indicates persistent limitations in logical composition and complex inference across current MLLMs. Structural Variation. Models perform better on depth-oriented tasks, where each step builds sequentially on previous outputs, than on width-oriented tasks requiring simultaneous attention to multiple variables. This may reflect the influence of prevalent CoT tuning, which favors linear reasoning but offers limited support for managing concurrent dependencies irrelevant distractions. Source Modality. Multi-image reasoning, especially across domains, remains key challenge. GPT- 4o shows the strongest cross-domain performance, underscoring the gap between proprietary models and current open-source MILMs in broad multimodal generalization. These findings collectively underscore the need for benchmarks like SCOPE that systematically expose fine-grained reasoning challenges beyond conventional evaluations.
5.3 SCOPE-Train Enhances Model Reasoning Capabilities
To assess the effectiveness the SCOPE expansion strategy, we perform comparative study using the same initial dataset augmented through different baseline methods. Results are summarized in T leading to the following key observations. Structured expansions consistently improve performance. Compared to baseline, all three structured evolution methods Piriiidds-.oviioon.....m VLB, and

---

Table Evaluation on the visual dialog task.
Table 5: Ablation study MAP.
ConvBench Ra
ContextVD Arc
VisDial.et R&1
MMBencheas Acc
ContextVD
Acc
Qwen2.5- VL
41.5
87.4
IntemVL-22552
Provision
42.6
90.5
79.3
(SCOPE)
77.0
93.6
VLBB
43.9
91.0
82.1
SCOPE
45.2
94.3
85.6
1 w/o curriculum scheduler
76.9
93.1
2
w/o gating module
75.3
87.4
InteenVL-22.5-28
39.8
85.7
74.2
3
w/o difficulty-aware loss
75.8
91.2
Provision
41.7
89.9
78.0
w/o MAP (direct finc-tuning)
75.1
87.1
-LBB
42.1
91.1
80.5
SCOPE
44.8
93.6
81.8
5 Backbone
74.7
85.7
SCOPE-achieve achieve notable performance gains by enhancing the quality and diversity of training in- stances. This underscores the critical role of augmentation in strengthening model reasoning. SCOPE outperforms other structured methods. SCOPE consistently achieves the highest improvements across all evaluated benchmarks. Its advantage stems from broader data coverage, unconstrained by scene-graphe subgraph limitations fixed-ansee answer rewrites. Moreover, SCOPE substantially enhances performance on multi-image tasks and reduces hallucination errors, indicating stronger generalization and factual consistency. Scaling effects are more pronounced in larger models. The performance gains from SCOPE-Truin are more significant for larger models (e.g., 7B) compared to smaller ones (e.g., 2B). This trend likely reflects the increased benefit derived from SCOPE's multi-roun expansion, which introduces higher proportion of complex, high-difficulty instances that better align with the capacities of larger models. These findings show that SCOPE- enhances generalization across domains and scales with model capacity, supporting its utility for robust multimodal reasoning.
5.4 Evaluation on Visual Dialog Task
To further evaluate the advantages of SCOPE-Train and its curriculum learning strategy, we investigate their effectiveness in the context of Visual Dialog, task characterized by sequential reasoning and context-dependent understanding. Unlike conventional augmentation methods, SCOPE enables continuous and controllable expansion of each original sample while preserving its core semantics. This design enhances both sample diversity and structural alignment. As shown in Table models trained with SCOPE exhibit superior performance in multi-turn dialog reasoning tasks, particularly those requiring contextual consistency and progressive understanding. We attribute this improvement to the staged learning process: for each training instance, the model incrementally encounters three increasingly challenging expansions, allowing it to gradually build the capacity needed for handling dialog-based visual reasoning with greater fluency and robustness.
5.5
Ablation Experiment
Analysis of SCOPE extension method. To analyze the contribution of each expansion strategy to model performance, we conduct controlled ablation experiments by incrementally applying different combinations of expansion methods (R-Reasoning Path Expansion, V- V-Viulal Context Editing, C- Composition) to the same backbone, using the original dataset as a baseline. As shown in Table 6 our structured evolution strategies consistently improve model accuracy, with reasoning path expansion yielding the most notable individual gains. Furthermore, we observe clear synergistic effects when combining multiple expansion types, demonstrating that multi-dimensional augmentation provides complementary benefits beyond any single method.
Analysis on the MAP. To evaluate the effectiveness of each component within the MAP curriculum learning framework, we conduct ablation studies by individually removing each module and observing its impact on final model performance (Table 5). Results show that eliminating the curriculum scheduler leads to the smallest performance drop, suggesting that while casy-to-hard progression aids gradual learning, it does not alter the overall quantity or quality of training samples. In contrast, removing either the memory and attention modules or the difficulty-wware loss results in substantial performance degradation, indicating that these components are essential for fully exploiting the structured advantages of SCOPE's expanded data. Furthermore, we additionally include w/o MAP setting that directly fine-tunes the backbone model. The noticeable performance gap between this variant and the full MAP configuration confirms that MAP itself contributes complementary gains

---

98
Label
Accuracy
Geeenz Quic22-
R
MMBenchess
ContextVD
Accuracy on ContextVD(
Innrrvvement
Relative improverin (%)
Backbone
74.7
85.7
94
FLamvVe-25--55
1
+R
75.6
89.0
92
2
+V
75.3
88.2
3
+C
74.9
88.5
90
4
+R+V
V
76.4
91.4
5
+R+C
/
76.1
91.3
88-
6
+++C
75.8
89.7
77.0
(SCOPE)
3
Number Iterative Evalutions
Table 6: Ablation study extended data.
Figure 6: Ablation study on extended iterations.
beyond data quality. This extended analysis helps disentangle the respective effects of the dataset and the training strategy, providing clearer picture of MAP's independent impact.
Analysis the number of evolutions. We analyze model performance after each round of iterative expansion, as shown in the line plot, and focus on the relative improvement between consecutive rounds using bar plots to control for the impact of increased data volume (Figure (6). The results reveal consistent performance gains across expansion rounds, with relative improvements stabilizing after the third iteration. While further expansion could continue enhance model performance, we limit the number of rounds to three due to computational cost considerations.
Discussion: Generalizability of Visual Programming
key advantage of adopting the visual programming paradigm is its broad generalizability beyond conventional VQA. Existing program-bases bageeam-based frameworks such as ViperGPT [39] VISPROG (17), and De-fine [5] have already demonstrated that modular, Python-like programs can perform not only image QA but also visual grounding, instruction following, image editing, and even video understanding all without retraining or task-specific architectures. By representing multimodal reasoning executable code, SCOPE provides unified and interpretable interface that can be adapted to a wide range of visual tasks.
Building on this foundation, SCOPE's evolution process operates purely at the program level, meaning that the same principles of reasoning-pati expansion, visual context editing, and cross-instance composition can be applied to any task where outputs can be represented code execution results. In practice, this includes tasks such as instruction-condiiion reasoning, and visual planning. Moreover, the modular API design allows seamless extension: new domain-specifer functions can be incorporated into the library without redesigning templates or retraining the model, enabling scalable adaptation to emerging domains.
image generation, cross-frame video
In summary, the visual programming paradigm empowers SCOPE to serve as general, interpretable. and extensible framework for multimodal reasoning and generation, capable of producing structured code outputs for virtually any vision-language task.
Conclusion
In conclusion, present SCOPE, program-centric framework for controllable and verifiable dataset evolution multimodal learning. By leveraging visual programming, counterfactual reasoning, and the MAP curriculum, SCOPE enables systematic data expansion with explicit reasoning paths and adaptive difficulty control. Our constructed benchmarks, SCOPE-Train and SCOPE- Test, not only reveal critical limitations in existing models but also promote scalable and interpretable training, paving the way for more robust and generalizable MILMs.
10

---

8 Acknowledgement
This work was supported by the National Key Research and Development Program of China (2024YFB3112900) the National Natural Science Foundation of China (62441617). Zhejiang Provin- cial Natural Science Foundation of China (No. LD25F020001) and Fundamental Research Funds for the Central Universities 2226-025-00055 Ningbo Yongjiang Talent Introduction Programme (2024A-401-G) the Zhejiang NSF (LRG25F020001), Zhejiang University Education Foundation Qizhen Scholar Foundation, Wallenberg-NTU Presidential Postdoctoral Fellowship.
References
[1] Marah Ahdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.142: 2024.
[2] Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. arXiv preprint arXiv 12440, 2018
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijic Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[4] Chun-Hao Chang, George Alexandru Adam, and Anna Goldenberg, Towards robust classification model by counterfactual and invariant data generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15212-15222 152211, 2021.
[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 3558-3566 2021.
[6] Long Chen, Xin Yan. Jun Xiao, Hanwang Zhang, Shiliang Pu, and Yueting Zhuang. Counterfactual samples synthesizing for robust visual question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 19800-10800 2020.
[7] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang. Qingyun Li, Yimin Ren, Zixuan Chen, Jiapeng Luo, Jiahao Wang. Tan Jiang, Bo Wang. Conghui He, Botian Shi, Xingcheng Zhang. Han Lv, Yi Wang, Wenqi Shao, Pei Chu, Zhongying Tu, Tong He, Zhiyong Wu, Huipeng Deng, Jiaye Ge, Kai Chen, Kaipeng Zhang, Limin Wang, Min Dou, Lewei Lu, Xizhou Zhu, Tong Lu, Dahua Lin, Yu Qiao, Jifeng Dai, and Wenhai Wang. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025.
[8] X.AI Corp. Grok- 1.5 vision preview: Connecting the digital and physical worlds with our first multimodal 2024.
model.
[9] Abhishek Das. Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. Visual Dialog, In Proceedings of the IEEE Conference on Computer Vision and Patiern Recognition (CVPR), 2017.
[10] DeepSeck-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, 7. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu. Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li. Fangyun Lin. Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu. Haocheng Wang. Honghui Ding, Huajian Xin. Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Meng Li. Miaojun Wang. Mingming Li, Ning Tian, panpan Huang, Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu. Shunfeng Zhou, Shuting Pan. S. S. Li. Shuang Zhou. Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei, Tianyu Sun, T. Wang. Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang. Xiaokang Chen, Xiaotao Nie Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng
11

---

Lin, X. Q. Li. Xiangyue Jin, Xiaojin Shen. Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang, Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Wang. X. Wei, Yang Zhang. Yanhong Xu, Yao Li, Yao Zhao. Yaofeng Sun, Yaohui Wang, Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong, Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang. Yaohui Li, Yi Zheng. Yuchen Zhu, Yunxian Ma. Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. 2025.
[11] Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. leee, 2009.
[12] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang. Xiawu Zheng. Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benehmark for multimodal large language models, 2024.
[13] Mona Gandhi, Mustafa Omer Gul, Eva Prakash, Madeleine Grunde-MeLaughin McLaughlin, Ranjay Krishna, and Mancesh Agrawala. Mcasuring compositional consistency for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5046-5055, 2022.
[14] Minghe Gao, Shuang Chen, Liang Pang, Yuan Yao, Jisheng Dang, Wenqiao Zhang, Juncheng Li, Siliang Tang, Yucting Zhuang, and Tat-Seng Chua. Fact: Teaching milms with faithful, concise and transforable rationales. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 846-855, 2024.
[15] Minghe Gao, Juncheng Li, Hao Fei, Liang Pang, Wei Ji, Guoming Wang, Zheqi Lx, Wenqiao Zhang, Siliang Tang, and Yueting Zhuang. De-fine: De composing and re fin ing visual programs with auto-feedhack. In P Proceedings of the 32nd ACM International Conference on Multimedia, pages 7649-7657, 2024.
[16] Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang. King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vi: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.05233 2024.
[17] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages
June 2023.
[18] Xuehai He, Diji Yang. Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, and Xin Eric Wang. Cpl: Counterfactual prompt learning for vision and language models. arXiv preprint arXiv:2210. 10..1066, 2022.
[19] Yushi Hu, Otilia Streteu, Chun-Ta Ta Lu, Krishnamurthy Viswanathan, Kenji Hata, Enming Luo, Ranjay Krishna, and Ariel Fuxman. Visual program distillation: Distilling tools and programmatic reasoning into vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9590-9601, 2024.
[20] Jingshan Huang, Fernando Gutierrez, Harrison Strachan, Dejing Dou, Weili Huang, Barry Smith, Judith A Blake, Karen Bilbeck, Darren A Natale, Yu Lin, et al. Omniscarch: a semantic search system based on the ontology for microrna target (omit) for microrna-aageet gene interaction data, Journal of biomedical semantics, 7:1-17, 2016.
[21] Drew A Hudson and Christopher D Manning, Gqæ: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference computer vision and pattern recogmition, pages 6700-6709, 2019.
[22] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 2rXiv:2410.21276 2024.
[23] Dongzhi Jiang, Renrui Zhang. Ziyu Guo, Yanwei Li, Yu Qi. Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking fhain-of-thougeee in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint vrXiv:2502.0962. 2025.
[24] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. 2023.
12

---

[25] Tiep Le, Vasudev Lal, and Phillip Howard. Coco-counterfactual:: Automatically constructed counterfactual examples for image-text pairs. Advances Neural Information Processing Systems, 36:71195-71211, 2023.
[26] Bohao Li, Yuying Ge, Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-hench-2-plus Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404. 16790. 2024.
[27] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[28] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction wning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26296-26306 June 2024.
[30] Shuo Liu, Kaining Ying. Hao Zhang, Yue Yang, Yuqi Lin, Tianle Zhang, Chuanhao Li, Yu Qiao, Ping Luo. Wenqi Shao, et al. Convbench: A multi-turn conversation evaluation benchmark with hierarchical capability for large vision- vision-language models. arXiv preprint :r0ii:2403.2010 2024.
[31] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216-233. Springer, 2024.
[32] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang. et al. Deepseek-vl: towards real-world vision- laionnlanguage understanding. arXiv preprint arXiv:2403.05533 055225 2024.
[33] Nishtha Madaan, Inkit Padhi, Naveen Panwar, and Diptikalyan Saha. Generate your counterfactuals: Towards controlled counterfactual generation for text. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13516-13524, 2021.
[34] Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Ronzheh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cyf conference on computer vision and pattern recognition, pages 3195-3204, 2019.
[35] OpenAL. Gpt-4v(ision) system card, 2023.
[36] OpenAL. Learning to reason with 1lms, 2024.
[37] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278 25294, 2022.
[38] Xin Su, Man Luo, Kris W Pan, Tien Pei Chou, Vasudev Lal, and Phillip Howard. Sk-vqa: Synthetic knowl- edge generation scale for training context-augmented multimodal lms. arXiv preprint 240: 19593, 2024.
[39] Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 11888-18 1 11898, October 2023.
[40] Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu. Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan 7hou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding. arXiv preprint arXiv:2406.094) 2024.
[41] Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, et al. Scaling text-rich image understanding via code-guided synthetic multimodal data generation. arXiv preprint ArXiv:2502.1 2544, 2025.
[42] Yue Yang. Shuibai Zhang, Wengi Shao, Kaipeng Zhang. Yi Bin, Yu Wang, and Ping Luo. Dynamic multimodal evaluation with flexible complexity by lision-language bootstrapping, 2024.
13

---

[43] Kaining Ying. Fanqing Meng. Jin Wang, Zhiqian Li. Han Lin, Yue Yang. Hao Zhang, Wenbo Zhang. Yuqi Lin, Shuo Liu, al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint urXiv:2404. 16006, 2024.
[44] Weihao Yu, Zhengyuan Yang. Linjie Li. Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.0244 024900 2023.
[45] Jieyu Zhang, Le Xue, Linxin Song, Jun Wang. Weikai Huang, Manli Shu, Yan, Zixian Ma. Juan Carlos Niebles, Silvio Savarese, et al. Provision: Programmatically scaling vision-centric instruction data for multimodal language models. arXiv preprint arXiv:2412. 070122 2024.
[46] Yifeng Zhang, Ming Jiang, and Qi Zhao New datasets and models for contextual reasoning in visual dialog. In European Conference on Computer Vision, 2022.
14

---

Appendix
This is the Appendix for the paper "Counterfactual Evolution of Multimodal Datasets via Visual Programming". We organize the appendices and supplementary information mentioned in the paper as follows:
7.1 SCOPE Raw Data Distribution;
7.2 Detailed Distribution of SCOPE after Expansion;
7.3 A SCOPE Data Example in Annotation;
7.4 The Corresponding Picture in Figure 4;
7.5 Limitation;
7.6 API and Function Library in Program Generation
9.1 SCOPE Raw Data Distribution
We selected six publicly available datasets as the initial data sources for SCOPE evolution. The specific data distributions and selected subsets are detailed in Table 6.
Table 7: Data mixture of VQA benchmarks and datasets.
Dataset
Set
Description
Number of samples
Seedbench2
General Benchmark
5543
MMBench
General Benchmark
2521
MME
General Benchmark
2042
GQA
test-dev val
Compositional
1426
OK-VQA
External Knowledge
1186
TallyQA
test
Counting
2156
Total
14874
9.2 Detailed Distribution of SCOPE after Expansion
SCOPE underwent three rounds of evolution in total. The data distributions for the first two rounds additionally provided in Figures 7 and 8.
12197
5289
8699
4169
3807
1788
SOCPE Train
SCOPE Test
20660
162
8856
70
3866
1662
7954
3331
4759
1975
Medium Balanced
Depth Expend
Wiath Expand
Figure 7: SCOPE dataset after two evolutions
15

---

6683
2856
2167
904
4617
1989
SOCPE Train
SCOPE Test
10410
1
4462
1377
618
3627
1570
2351
989
Easy
Depth Expand
Balanced Dual-input
Wieth Expand
Single-inpul input
Figure 8: SCOPE dataset after only one evolution
9.3 A SCOPE Data Example in Annotation
We present concrete annotation example, which is followed both the SCOPE-Train and SCOPE- Test datasets.
"origin_date":
"data_san sosta_soores" "Mxx""
"source_id": "656",
"question": *will green obtained by mixing the above colors? Please yes no.",
"inage": "smggs//AW//tees1.35....
1,
9
*golden_anaeen"
"Yea"
12
"progran; "def cectte__oommn____e (Image, measiale_anawene query, ImagePatoh,
VideoSegment, lim_quezy, _ocl_to_yesne, distance, best best_image__ad Lmage_match):iv
is:def
execuue__comanee limage, my_fig, sime_weit_d between_ttteee_isee ImagaPatcch (image) \r image_ pateepptton..m Eind (Vdddd "ootor>\""' 2:3n
syntax): two colors the imagekn color_pateine shou_all (None, celor_patches,
image_patch
An
zeturn \ooo\" # Not enough colors to aixla An colorsin color_1 color_patches stmpllleeeeeleeryyy (1 is the name of this color?\") color_patched (1). simple_qusry('s "iat color_2, "aclor_2\*)(1 resuits greenin mery fi"will mixing [coler_1] Anawer or Via reault maggepptonn watch.lln_guee His_qurry (query, sorg_answer=Palse) \n show_all (None, result, Vresulti") la In return result", "progran_ansee "yes",
if Get
coler_2 show_ail (None, \a # Use qla_query to determine if mixing the twa cclors [color_21 reau in greent
show (Nose, voler_l, Pecolor_i""] name this color?>")
14
"satii analysis": "difficulty"; "easy", "extond_mathod"; null
}.
"image_count";
3,
iiviuutionddaa...
21
"question": *Will green be obtained by mixing the above two colors? Pleaze answez no."
22.
*mage"" [
"imagea/MMB Bateee//M////eeeeeeeeeeeeeee Lasmae/eesseeeeee___55 Jpg"
1.
"progran": "def cecpuutteeeemeeeeeeeeeee (image, passible_anaeee answers, query, ImagePatch,
VideoSegment, lin_quezy, boey__o_yesno, distance, ieee__imme__mateei cotpuutteeomaddddaeeee ImagaPstch (inage) \r.
isrdef
fig, sine_weit wetee/eett_tttteeeeeeeeee betwwenn_iaeee syntax): two
mmage_patch
in the image'n color show_all (None, oclor_patches Ensure there are at least two colers mixln if (en of the colorsin coles_l cotor_patches is the name of this color?)") In show (None, color \" this
pange_patoh. rind "cotor(""
patlorrptcchss"" \n 2:\n
[0]. V"nci*in in Get the
color_li") color_2 solos_pstches! simple_ (Naha
16

---

An shex_all (None, color_2, Ve use lin_query determine if mixing the two colors resulta in graenin query fywill mixing (color_1) [] and color_2) result in greer?
no. result image_patet. qum_qerry (query, saak_all (Noma, result, \"remilt"") An return rasult", "progran_answer"; "Yes", "stalic analyais"; "difficulty": "easy", "extend_method"; "Ealanced"
long mnggaonoor==alsel
},
"imace_count";"
32 )
"question" ionestion=: *If green is obsained by mixing the two colors, which color dominant
in the resulting green?", *image":
Mmagaa//ME/nneeaaarg e_bpp Jpg"
1,
"progran"; "def weeuute_command (56 mosstole_anawers query, ImagePatch, VideoSegmeat, Lla_quezy, boo bool_to_yesno, diatance, best Leag__mage__ue smage_natchh) execute_com cecouuteccomaddiiie (Smage) :/a imagaPatch (ingl \n
Answer la
colors Il
img image:
petch
color patch. "kat is the coler?>")
colors, aprand (nolor: lower (1
if Vyylllow" colors \"blua\* cclorsiin
yellox_pstch Noneln
blue_patch Noneln
for img In Imager In
patch ImagePatch (ing) lower =- \"yellow\":Mr
query
color?
yellcw_pstch patchin
ellf patch, blue_patch
s (I (V"hha is the color? (.lorer) -- Vblue)":in patchin
if yellew_ and blue_patchila
yellow_ares
yell patchhhhthh pallow_patch.e beaathhhhiggtti height
Elue_area = bluc_patch.wt:dth
stue__stch.heegnne
return ("yellow"" yellow_area blue_area else \"blue\*
retura \"no green obtained)
axzogran_enswertt
green obtained",
static analysis": { "difficulty"; "hard", "extend_method": "width"
*image_count";
45 ",
"evolution_date" _ata_____d__
"question": *If green is chrained by mixing yollow and blue, which color contributes more to the green in terms of spatial deminence scross all images, and how does their spatial relationship (left/right) affect the resulting green?", "maag""
Maaagea/KM//exeeeee http://eeeeeeaaae__eee_e_e_eee
MMD coomoneenseeressonniiie
:0
1.
52
"progran"; "def execute_ catttt__oommn____ee (1 aassible_anavers query, ImagePatch,
videnSegment, 1lm_query, bocl_ta_yesne, distance, cast_inage__atten :'n Answer is:def executo_comm cociute_oomadd_//// (imagel F\n dominant_ooloore (]An opptiall_rllaiionshiee (] for Lmn in ImagerAn.
patch (angeFatch(Img])
yellow_patch patch.
find "yelloww
blue_patch (atch (bluue" "blue)")
if
yellow_patch and sluue_atchhih
(a1loo__atcch00 blue_orea blue blua_patchi(0 width blua_patch(00..
yellov_aree
yellow_pateh [0] heiight heigh:An
\n
if yellow_area biue_arear
crraant_ooorsspppppoo (yaaloxii)) append (\"hhue)*)
Seminant_cclare neternine apstial relationshipln
else:in
In
it vellow_patehh wworisontal center e
else:in spatial wtw.rrrooooooooooooooooooo append / (b"bu is left of yellowi") In Determine the oversll domf nant color across all imagesin yel low_nont dontrant_colars-.. (\" yellowi*) blue_count dominant_coles catinant_color.ccom coo.or.coouutt (b"uue""))
gratiel_relationshpp app.pppp.pppddii/.. yeelloo
blue)")
In doninant "yy"loow* if yellow__ovee count > blue_count else \"bluei"nn In Datermine the most spatial_relationah cey=spatisl_res rew.tiiossss...ssii.s......... count) dominant_colce), and apptiial_eletionsshi
common spatial relattinnhipp'r
repataal_relattonnnll \a return E\*The deminant color is
irrogran_enswert "The dominant color is yellow, and yollow in left of blue.",
statie analysta":
"difficulty"; "hard",
*extend_methond *bal "balanced"
38
"inage_coutt:
(0)
Listing 1: Annotation example.
17

---

9.4 The Corresponding Picture in Figure 4
In Figure 4, we illustrate the process of Counterfactual Categorization from code-level perspective. Due to space limitations, the corresponding visual examples are provided in the appendix.
Figure 9: Original image
Figure 10: Associated image through Cross-Instance Composition.
9.5 Limitation
In this section, discuss several open directions and current limitations for future exploration. First, although our dataset construction relies on six publicly available datasets, SCOPE is designed as dataset-agnostic evolution algorithm and can be applied to data from various domains. We encourage future work to extend and validate its effectiveness on proprietary datasets. Second, in the MAP curriculum learning stage, the difficulty ranking of structural variation is primarily based on heuristic insights. We remain open to refining this ordering should more principled or effective strategies become available.
9.6 API and Function Library in Program Generation
The prompts employed during the code generation process are listed in detail.
iiport math
ocaass ImagePatch:
Python containing crop an image centered around parsiculas object, as well as ralevant information. Attributes
cropped_image arrey_like
array like
cropped inage Laken original image.
left, lower, right, upper int
dasaribing the position (latt/lowar/riihtt/ugas
the crop's
bounding box in the original image.
Kethods
Eind jejct_names name: stz) >iiis [[magePatch"'
18

---

keturns list of imagePatch objects containing crops of the image centered around any ohjects found in the
Image matching object_namee exiats (abject__nmm; str)->bool
Returna True the object spectfied by abjeat_name 1s round the image, Falae otherwise.
verify-propezte (propesty: s:z)->bool
keturns true if the property is met, and Felse othervise, beat _eat__ee___aa _eot___ttt__thhhhhhea match (option_ list: 11st (str), prefix: str)-
Returns the string aimple_ query ions str=tion)--aat ->atr
matches
image.
23
Returna the anawer provided, returas the
baaic queation asked about the image. no question "What is this?",
(eppthh (eethh))>>flatt
Returna the madian depth of the image crap.
27
crop(left: int, lower: int, right int, upper: Tntg-etmageFatch
Beturns new ImagePatch object containing crop of the image given coordinates.
31
def init (self, image, left: int None, Lover: int = None, right: int None, upper: irt Nona):
32
**"ialtializes an ImagePatch objtct by cropping the Image at the given coordinates and stores coordinates
attributes, It no coordinates are provided, the image la left unmodi and the coordinates are set to the dimensions the image. Parameters
37
Inage : ariay like
An arrey-like of original inage. left, lower, right, upper int
An int describing the position of the (left//bwer/rsihtt//g bordez of the crop' hounding bex the criginal image.
left self.crosped - image
None upper la
43
self.left
45
lower =
self.right image. width
solf.upper amago. height
e1s8:
celf.crooped_inee Image(i, lowerruppes, left:right] lelt = left
:0
self.upper upper
52
self.right right
self.lower lover
55
wilt.width = welf. wwt.tcoooppe_eeeeee shaaaeesshapeeeee
self.height self.cccgped__ineee... s1
57
isalf.left seih.right) / self. veltvveriiallcenta (self. lower. selr.upper) 2
self.herizontal.
def fsed(self, cbject_namm _ame:
Mist [TmagePatoh]
***Retuzna a list of ImagePatch objects natching ob ject__nmmee contained in the crop iE any otherwiss, returns an empty list. Parameters
63
objact_ name I str
the the object found
68
Returns
11st of ImagePatch jbjtc matching jece__namee contained In the crop
Examples
75
return
>>> daf execute axocure_command (image) -> sf [TmagePathh]]
223
Image_pste ImageFatch (image)
553
foo_patches imege_patch. ("
return foa_pstches >55
82
zeturn find in isels. inage, object_aame)
84
def exiats(self, object_nname name: str) sool:
***Reluzna Tzue if the object specified by jeje is found in the Inage, and Pelse otherwise.
19

---

object A string describing the name of the abject to be found in the image.
Examples
there both foos and garply hasa in the photo?
555 def avecute_commant (Image)->a0) ->strt
_mage_patch ImageFatch Limage]
96
235
io_foo = sasg__patchh.xissee ("foo")
1a_garply
(axistsa"gapplly ("garply bar")
353
return teoi_te_yessoois (is_ and le_gexply_ber)
59
100
lan lsel..find wtg.ll.ffnddcoejttt...mo
101
def
Iself, abject_name: sti, visual properly: stzl **"Returna True if the object possesses the visual property, and Ra se otherw" se, Differs from "exists" that it presupposes the existence the ob ject specified by objecs_nane, instead checking whether the object possesses the property. Parameters
103
104
105
106
107
abject_namm : atr
string describing name the object found inage. visual visual property atr
109
110
A string describing the simple visual property (eig.r color, shape, naterial) to be checked.
111
112
Examples
113
blue color?
115
>>2 def (izage -> str:
116
355
Emage_petcl ImagePatch (image)
letters_patches
aotch.Eind("letterrs
118
Question assumes only one letter patch
veritty.rooertty "blve")
120
return
cbject_neme, property)
122
123
def test__ext_matchine isels, opticn List [str], prefix: stzeNone) -> strt "*"Rcturns the string that best matches the image. Parameters
134
135
127
opticn_list. : stc
128
list
names different options
129
prefix str
130
A string with the prefixes to append to the options
132
Examples
133
134
>>> is the foo gold or whice?
135
exacute_command comennd ->str:
136
222
imsge_pat.oe ImagePateh (imaee]
137
>5>
foo coo_patches = aaage_patell find
Queation
foo patch
139
532
return foo_pstches wwwpt_eett__stteeeeeeee (['gold", "white"])
141
return bast_ bast_toxt_matcchaall...co imagee eption_list, prefix)
143
def simple_query (self, questions str None) >> strs
144
**"Rcturns the answer to a bastc quastion aaked about the image, If no queatior is provided,
145
to "Rhat is this?", The questions are about basic percepticn, and are not meant to be complex reasoning or external knowladge. Parameters
146
147
148
149
queation : sstr
string describing question asked.
151
152
examples
153
154
155
553
Which kind baz is not fredding?
156
>>> def execute coceuttecoommndeeeee (inage) str:
157
image_pateh ImageFatch baz_patches image_patch. pat.h.ffccaiaa.. find ("baz")
158
3>3
159
baz_patch baz_patehes:
160
325
if not sar_patch.vert wattch.errry..pe.
(bba"", "tredding")
161
333
retura bas_patch, aaaathhh.impli iample_uery ("What is chis bau?")
162
20

---

163
555
what color the 200?
164
555
def svacute_command (image) -> str:
165
[mage_ TmageFatoh (image]
166
252
poo_patches =
find
167
foo pat_patch che_ [0]
168
retuzn sco_petch..immle_peee ("Shat is the color?")
109
170
def secend from the left quuxy?
171
555
(Imaagd(Imago) -> str:
172
Lmage_patcl ImageFatchhinagge
173
252
bar par_potches = hatgge_pacchhaaeee finar""ar))
174
222
bar_pat boatchee.sort sort (key=lambda x: x.horisonts al
175
553
bas_patch pstccpsss
176
355
return bar_petch. bar_petch.simmle..qeee simple_query ("1s the bar quaxy?")
178
return simple_query ry (sell weserrroroeeee oroppeo__nagee question)
180
def compute__se deppt_dephhheeeeee (self):
181
"*"Retuzna the median depth of the Image crop Parameters
182
183
184
Returna
185
187
the nedian depta of the Image crop
188
189
ixamples
190
191
329 def furthest away
192
(inage) >Iaggeatchh:
193
image_patch imagePatch (image)
194
222
har_parches = smage_patch. find ("har")
195
355
bar_patches, ser: (key=lambda ber: ber.compuce_depth ()]
196
355
har_petches[[1]
197
198
depth_map compute_depi: www.......oeeeiiieeee...... return Sypth_map.mediant
199
201
def (self, left: int, loser: int, right: int, upper: int) -> ImagePotch;
200
**"geturns a new ImagePatch ezopped from the current imageEasch. Parameters
203
204
left, lowes, right, uppez
206
The beet:/ooer/ziigh//ppeee/ pixel of the cropped image.
208
209
return IzagePatch (self. crrporrped_e left, lover, right, upper)
210
211
def overlaps_ wetblapp_withheellf... left, lower, right, uppert
***Returns
czop with given coordinates everlaps with one,
213
else ralse.
Paromoters
215
216
left, lower, right, upper : int the hart://wer/right//e border grop be checked
217
218
219
220
222
True if with the given coordinates overlaps with chis else Falee
223
234
Examples
225
top qux
227
>>5 def caacute_commcnte (image) -> ImageParch:
238
232
petge_pstchhhhe TmagePatoh (mmgel]
229
535
qux_patches 1mage_patch._a_ Find ("qux"))))
230
pa qax_pat chea c0)
Ioo_palches
paagge_pacch.iamee rind
232
535
for foo in foc_patches: feo.vertical_cente return fon
www.petah.verricss_ccnee.
234
>>>
235
236
return self.left right and self.right >e left and sower <= uppes and self. upper >e lower
28def bast__imaee_matt match (istppahhht
patches: List [imagePatch], content: List (scel, return Union [imagePatch, int]:
239
***Returns the patch most likely to contain the content.
240
Paramelers
241
21

---

242
patches List [mmagePacch)
243
content List (str]
object interest
245
indern_index bool
246
True, returns the index of the pat most likely to contain the object
247
248
Returns
249
250
int
likely contain ject
252
253
return htt__iiage_aatchhiieeeee.o
content, return_indexi
254
255
25ed distancs (patch_a: ImageRatch, patch_b: bi EmagePatchl float:
257
Returns distance between the edges of ImagePatches. the palches overlap, returns negat iive distance
259
corresgonding Lo the negative intersection over union.
260
261
Porametars
362
263
patch_a ImagePatch
patch_ ImagePatch
265
266
Examples
267
208
Return the qux that is closest to the foo def conutte_oomaddddeeee (Image):
360
270
332
image_palch ImagePatch (inage)
qux_patches image_patch find gux"
272
>>>
foo_patches mmage_patch.. find foo!
273
355
foo_patch foo pro_patches
274
qux_patches sort (keyslambda distance fco_patch)
275
>3>
return qux qux_patches(()
276
277
return distance (patch_a, patch_b)
279
20ddef bool_to_yesno (bcol enswer: bool) -> str:
return "yos" if bool_answer clae "no"
280
coeroe_ coerce_Lo_nusee (etrrii
254
function takes string input returns
renoving any non-numeric
characters.
256
if the input string contains a range (e.g. "I0-15"), it returns the fizst value in the ranga.
287
return ccerce_to numericistting)
289
function using Python provide an answer the query.
ImagePatch class labove) that could executed to
291
Coconssder
Yellowing outdeliness
290
Use base Python tcomparison, sorting) for bsale logical opecations, let:/right/up/doni math
etc.
201- using the simple_query function to ask questions that require image information.
You may be given multiple images as input. image be accessed separately, 1ike: Image[U], image[1].
them when answering the quest on.
206
You-You given 1ist images Input,
in list single image, Always 208- importantas, You MUST loop each image wrap
sreat the input variable image as list.
imagePatch
209
for ing in image:
patch ImagePatch (1mg)
301
setheen ing with 11st of images, avoid veturn'ng insida the for-loop, Thatead, col eat all relevant information across images return the final 3IIf an object must be verified across multiple images, gather daza first, then retura a final sunmary:
loop.
301 You vmmust examine all images...
However, the final output should be sesingle, unified answer not 1ist per-image outpuzs.
306- Uo not return inside the locp. instead, gather all relevant information across the images result.
generate
30- Even if some intermediate reasoning is per-inage, combine the insights before returning.
309
return (anawar_img], answer_ing?, ...]
310Good:
22

---

Analyze summarize
Final "yes", integer,
sentance)
312
313Query: INSERT INSERT_QUERY QUERY HERE
314Anawer: TNSERT ANSMPR
315
316
ai7first, writa Pythen function
the original visuol question.
si8Ther, gererate follor-up question that requi rea deeper image reasoning-e.g., multi-abjeat logic, spatial relations, attribute comparison.
SiWrite second function to solve it, using higher code complexicy, lower similarity, and involving more the previoualy mentioned functions or A?Ia.
320
Mirormat your answer as follors:
smrriginal Code:
Nikoriginal_come
325Pa I ow Quest
3b<tellooup_guestiion
327
33Folloxx-p Code:
320<fol code>
Listing 2: Prompt for code generation.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, in both the abstract and introduction, we clearly state that we propose a counterfactual-basee visual programming algorithm for dataset evolution, along with corresponding evolved dataset and a supporting algorithmic framework named MAP.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in Section 7.5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
24

---

Answer: [Yes]
Justification: We provide a detailed theoretical analysis in the Method section and conduct a systematic empirical validation in the Experiment section.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, in the first part of the Experiment section, we introduce the required models, baselines, backbones, and benchmarks, ensuring that all necessary information for reproducing the main experimental results is provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
25

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, provide detailed information in the appendix, including dataset com- position, static analysis at each stage, and the prompts required for the models. Additionally, our code and data made available in the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipss cc/
public guwdes/id//eeo http:/gudddeeedeuuaaaeeeeeeeeeee Subemu ssiinnsiiin ssionPolicy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. htnips.ccppubll http:////ppbiccccuuceeeceee.ee Cae//SoddSSubmmm ss i conooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, include dedicated Setting subsection to describe the experimental setup, and the Benchmark section provides a detailed explanation of the dataset construction process and the train-test split ratio.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the statistical significance of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide the necessary APIs and prompts, and most of the experiments can be conducted by directly using the provided APIs.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://////eeeee../o/////////aae
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss the potential positive socictal impacts and have no negative societal impacts.
Guidelines:
27

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, we primarily rely on API access and do not use any harmful or sensitive data.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we use only publicly available datasets and open-source models.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
28

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use the package should be provided. For popular datasets, papprrswi sopprseithcoce com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We propose a new evolved dataset, which is included in the supplementary
materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: Yes, provided comprehensive explanation of the task and requirements, and offered appropriate compensation.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The paper describe potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
29

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: Yes, we clearly state the intended purpose of using LLMs at every point where they are involved in the workflow.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https: /ttt:::://eurrggeee httprrisss..cccooceeennee//////0
LLM) for what should or should not be described.
30


Reviews:
[{"review_id": "aE3ZFj9B2Y", "reviewer": "Reviewer_qgzN", "summary": "This paper introduces SCOPE, a novel framework for evolving multimodal datasets using symbolic visual programming and counterfactual reasoning. Key contributions include:\n1.\tVerifiable Dataset Evolution: SCOPE generates executable Python programs to model reasoning paths, intervenes via program edits along three axes—reasoning path expansion, visual context editing, and cross-instance composition (action)—and categorizes evolved instances by difficulty/structure/modality.\n2.\tBenchmarks: Constructs dynamically evolving benchmarks (SCOPE-Train/SCOPE-Test) with expert-validated samples (91–96% human acceptance).\n3.\tCurriculum Learning: Proposes MAP (Memory and Attention Path), aligning model capacity with difficulty progression through adaptive gating and loss weighting.", "questions": "Please refer to the paper weakness.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n•\tVerifiable Control: Uses executable Python programs for traceable reasoning chains, avoiding black-box LLM hallucinations.\n•\tDiagnostic Benchmark: SCOPE-Test reveals fine-grained model weaknesses, especially in multi-image/hard reasoning scenarios.\n•\tResource Accessibility: Open-sourced code/data enables community-driven evolution.\nWeaknesses\n•\tThe framework aims to improve the model through data, but its starting point depends on a very powerful model (GPT-4o). The quality and diversity of the generated program are limited by this external model, and its own biases or limitations may be passed on or even amplified in the evolution.\n•\tThe specific computational cost required for a single iteration of evolution has not been provided. Will this limit its practical application?\n•\tAs mentioned in the limitations of the article, the effectiveness of the method has not been validated in a wider range of domains.", "was_revised": true, "final_justification": "The authors have addressed my concerns regarding generalization in terms of the LLM baseline, downstream task range, and computational cost. This is an insightful paper on augmenting and enriching MLLM datasets, and it demonstrates clear effectiveness.", "rebuttal": "**[W1]** *The framework aims to improve the model through data, but its starting point depends on a very powerful model (GPT-4o). The quality and diversity of the generated program are limited by this external model, and its own biases or limitations may be passed on or even amplified in the evolution.*\n\n**[A1]** Thank you for your concern regarding the potential reliance of SCOPE on a strong backbone model (e.g., GPT-4o), which could constrain the quality and diversity of generated programs and potentially propagate its inherent biases. To address this, we provide the following explanation from four complementary perspectives:\n\n(1) Theoretical Design: SCOPE introduces three orthogonal evolution strategies—reasoning path expansion, visual context editing, and cross-instance composition—which enable controllable, multi-dimensional diversification beyond superficial modifications, reducing dependency on any single model’s outputs.\n\n(2) Code-Level Verification: Code-Level Verification: To empirically demonstrate that SCOPE progressively increases code diversity across iterations, we performed two complementary analyses. First, complexity growth: Measured by Halstead Effort, the average effort score shows a clear upward trend from approximately 3281 in the original version to 4369 after the first evolution, 5217 after the second, and 5973 after the third, indicating that the evolved code becomes increasingly complex and information-rich. Second, structural dissimilarity: We computed Abstract Syntax Tree (AST) similarity between each evolved version and the original code, observing a steady decline from 0.72 (first evolution) to 0.51 (second) and 0.34 (third), confirming that each evolutionary step introduces substantial structural variation. Together, these results provide strong evidence that SCOPE effectively enriches both the complexity and structural diversity of generated code through iterative evolution.\n\n(3) QA Distribution Analysis: We evaluated diversity in the question space by measuring distributional differences using Maximum Mean Discrepancy (MMD). Specifically, we compared the QA distribution under SCOPE with two baselines: VLB and Provision. Our results show that SCOPE achieves a significantly higher MMD value (0.38) compared to VLB (0.15) and Provision (0.18), indicating that SCOPE's QA outputs are farther from the baseline distributions and thus occupy a more uniform and dispersed semantic space across reasoning categories. A larger MMD signifies greater divergence from the baseline distributions, demonstrating that SCOPE generates a structurally and semantically more varied QA set overall.\n\n(4) Backbone Variation Experiment: To reduce reliance on a single backbone, we conducted experiments using code generated by CodeLlama, evaluating InternVL‑2.5‑2B on three benchmarks: RealWorldQA, MMBench, and ContextVD. The results show that switching to a different code pre‑training model has little impact on performance—InternVL‑2.5‑2B maintains competitive accuracy across all datasets. This experiment inspired us to plan the integration of multiple code‑generation models combined with diversity‑aware reward filtering, aiming to ensure continued structural and semantic variety while mitigating model‑specific bias.\n\n|Model|RealWorldQA|MMBench|ContextVD|\n|--|--|--|--|\n|InternVL‑2.5‑2B|60.1|74.7|85.7|\n|&nbsp;&nbsp;&nbsp;&nbsp;‑SCOPE (CodeLlama)|62.9|76.6|94.1|\n|&nbsp;&nbsp;&nbsp;&nbsp;‑SCOPE (GPT)|63.0|77.0|93.6|\n\n--- \n\n**[W2]** *The specific computational cost required for a single iteration of evolution has not been provided. Will this limit its practical application?*\n\n**[A2]** We thank the reviewer for raising the concern regarding the computational cost of a single SCOPE iteration. Our method consists of two main components:\n\n- Code Evolution: This phase generates new QA samples through programmatic function- and code-driven expansion, without relying on multimodal pipelines or iterative LLM validation loops.\n\n- Classification: This phase assigns reasoning categories to generated QAs using static code analysis (e.g., AST parsing, Halstead metrics), avoiding LLM-based filtering.\n\nCompared to prior work, MAmmoTH-VL relies heavily on multimodal LLMs in both the evolution and classification phases, introducing substantial inference overhead and high model cost. ProVision, while emphasizing that its pipeline does not depend on LLMs, explicitly states in its paper that it requires defining 24 single-image instruction data generator programs and 14 multi-image generator programs, each utilizing hundreds of pre-defined templates—resulting in significant manual engineering cost. In contrast, VLB employs iterative image manipulation and repeated LLM-based semantic verification, both contributing to considerable computational and inference overhead.\n\nIn summary, a single SCOPE iteration remains computationally lightweight and scalable—avoiding costly scene-graph generation and LLM inference—yet achieves richer structured outputs and diversity at a cost similar to prompt-based synthesis, and significantly lower than ProVision or VLB. We report the computational cost for a single evolution iteration across different baselines for comparison. The detailed results are summarized in Table below.\n\n|Method|Evolution|Classification|Human Effort (From)|\n|-|-|-|-|\n|MAmmoTH‑VL|3-5s (MLLM)| 3-5s (MLLM)|Medium (Initial dataset quality assessment)|\n|VLB|3-5s (LLM and Image iterative editing)| 1-2ms (Pre-designed difficulty level)|High|\n|Provision|1-2s (Scene-graph construction + Python generation)|1-2ms (Pre-designed question category)|Very Very High (Hundreds of pre-defined templates)|\n|**SCOPE (ours)**|2-3s (Code-Pretrain Model or Image single editing)|1-2ms (Static code analysis only)|Minimal|\n\n--- \n\n**[W3]** *As mentioned in the limitations of the article, the effectiveness of the method has not been validated in a wider range of domains.*\n\n**[A3]** Thank you for this valuable observation. To further demonstrate SCOPE's robustness and domain generality, we have planned evaluations on **four additional benchmarks**, which span different reasoning types:\n\n- **MathVista**: visual-numerical reasoning and multi-step symbolic derivation\n- **MMMU**: multi-disciplinary question answering (STEM, humanities, law, etc.)\n- **AI2D**: diagram interpretation and science-specific reasoning\n- **DocVQA**: OCR-based text-visual integration in real-world documents\n\n\n| Benchmark | Base Performance | w/ VLB | w/ SCOPE | Δ (Compare to VLB) |\n|--|--|--|--|--|\n| **MathVista** | 51.3 | 52.5 | 53.9  | +1.4  |\n| **MMMU** | 43.6   | 44.9 | 45.8 | +0.9 |\n| **AI2D** | 74.9   | 76.3 | 77.9 | +1.6 |\n| **DocVQA** | 88.7  | 89.9 | 91.8 | +1.9 |\n\n\n\n\nIn summary, these additional benchmarks will validate that SCOPE enhances reasoning consistency and multi-step control—even in domains demanding symbolic math, domain‑specific knowledge integration, or OCR-based reasoning. Importantly, we do not modify the existing API or introduce new functions for our current tasks. However, SCOPE’s design allows flexible extension into specialized domains: by defining new domain-specific functions and embedding task-relevant logic, we can seamlessly expand the function library without redesigning templates, prompt structures, or scene graphs. The three orthogonal evolution strategies—reasoning path expansion, visual context editing, and cross-instance composition—remain fully applicable across these domains, preserving lightweight implementation while supporting systematic and controlled domain adaptation."}, {"review_id": "gWpSqe4uj9", "reviewer": "Reviewer_7KCw", "summary": "This paper introduces a method to generate multimodal datasets using a combination of visual programming and counterfactual reasoning. The goal of the paper is to address limitations of existing dataset construction methods, e.g., being heuristic, template-based, or rely too heavily on model generation. The authors apply their approach on a set of existing dataset to obtain a train and test sets, which are used to train and evaluate some open VLMs. The proposed method starts by converting an image-query pair into executable code, then modifies the python code along three axes: reasoning path, visual context, and cross-instance composition to generate new samples. Experiments show that models trained on SCOPE exhibit better performance, compared to two baseline augmentation methods.", "questions": "1. How computationally expensive is your approach compared to the baselines\n2. How would you generalize your approach to cover instances where generating a program is not possible?", "limitations": "The authors address some limitations in the appendix not in the main paper", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. the idea of using interventions in the code rather than the image space is clever and novel, since code is interpretable, controllable and easy to measure its complexity.\n2. curriculum learning via MAP is well ablated and yields performance gains.\n3. overall, training on SCOPE seems to yield better model compared to two other augmentation strategies\n\n\nWeakness:\n1. The main bottleneck here is that this approach will only work with instances that *can* be described in code. This means the approach is not generalizable. \n2. The authors criticize prior work that relies heavily on model generation, yet SCOPE relies on GPT-4o for program generation. Also what if the generated code has flaws but still runs? \n3. SCOPE-test is not a challenging benchmark by any means; GPT-4o already achieves 88% accuracy.\n4. The approach is somehow complex and relies on many components, e.g., code generation and segmentation models.", "was_revised": true, "final_justification": "The author's rebuttal did not change my original assessment of the paper, which was already positive. In general, the paper makes a good contribution, and I'm leaning towards accept rather than reject.", "rebuttal": "**[W1 & Q1]** *The main bottleneck here is that this approach will only work with instances that can be described in code. This means the approach is not generalizable.* & *How would you generalize your approach to cover instances where generating a program is not possible?*\n\n**[A1]** We appreciate the reviewer’s concern regarding the perceived limitation that our approach only works with code-expressible instances. However, this concern is mitigated by substantial prior work—such as ViperGPT, Visual Programming (VISPROG), and De-fine—showing that code-driven paradigms adapt to a broad range of multimodal tasks beyond classical VQA.\n\nViperGPT composes Python programs from modular functions via LLM-generated code to perform visual grounding, image QA, video QA, multi-image comparison, symbolic math, and knowledge-aware queries. This framework requires no task-specific fine-tuning and achieves state-of-the-art zero-shot results across diverse benchmarks, evidencing that programmatic reasoning is not restricted to narrow domains.\n\nVISPROG further demonstrates how modular, Python-like programs generalize across compositional visual reasoning, image editing, and zero-shot knowledge tagging—without training data—highlighting the flexibility of code-mediated reasoning.\n\nDe-fine refines visual programs through auto-feedback loops based on static and dynamic signals, improving robustness and logical consistency in multi-step reasoning without retraining, again confirming adaptability across domains.\n\nThese foundations give SCOPE a strong basis for generalizability. Notably, we do not alter the existing API or introduce new functions—the functions inherited from ViperGPT already support visual, math, OCR, and knowledge-augmented QA tasks, as discussed in our Reviewer #qgzN response A3's evaluations on MathVista, MMMU, and science/OCR benchmarks.\n\nIf a new domain emerges beyond existing coverage, SCOPE remains extensible: new domain-specific functions can be added to the library without redesigning templates, retraining models, or changing core architecture.\n\nIn summary, by leveraging VISPROG and ViperGPT’s modular, executable program paradigm and enabling addition of domain-specific functions when needed, SCOPE achieves broad generalizability and flexible extensibility across multimodal tasks.\n\n\n**[W2]** *The authors criticize prior work that relies heavily on model generation, yet SCOPE relies on GPT-4o for program generation. Also what if the generated code has flaws but still runs?*\n\n**[A2]** We appreciate the reviewer’s concern. In fact, our critique targets pipelines relying entirely on model-generated data for generation, verification, and feedback—leading to opaque reasoning and bias—SCOPE, though using GPT‑4o for program generation, adopts a fundamentally different, interpretable paradigm.\n\nSCOPE employs executable code as an intermediate reasoning layer aligned with Chain-of-Thought. This code-mediated mechanism enhances interpretability and allows explicit control of reasoning steps, including editing or counterfactual manipulation—capabilities absent in end-to-end pipelines.\n\nMoreover, SCOPE is model-agnostic. GPT‑4o can be replaced with other pretrained code models such as CodeLlama. We also present below the results of a InternVL‑2.5‑2B trained using code generated by CodeLlama. The future work will also integrate multiple code-pretrain models to reduce backbone dependence and model-specific bias.\n\n|Model|RealWorldQA|MMBench|ContextVD|\n|--|--|--|--|\n|InternVL‑2.5‑2B|60.1|74.7|85.7|\n|&nbsp;&nbsp;&nbsp;&nbsp;‑SCOPE (CodeLlama)|62.9|76.6|94.1|\n|&nbsp;&nbsp;&nbsp;&nbsp;‑SCOPE (GPT)|63.0|77.0|93.6|\n\nFinally, human evaluation confirms the evolved code is readable, coherent, and aligned with intended reasoning, with acceptance rates of 96 %, 92 %, and 91 %, indicating that minor noise in code generation is negligible relative to the substantial gains in data quality and reasoning diversity.\n\n\n**[W3]** *SCOPE-test is not a challenging benchmark by any means; GPT-4o already achieves 88% accuracy.*\n\n**[A3]** Thank you for this insightful observation. In fact, SCOPE-Test was intentionally designed as a diagnostic benchmark, rather than a large-scale competitive suite, with the primary goal of exposing the limitations of current dataset evolution strategies. It demonstrates that models pretrained on existing datasets often fail on specific reasoning dimensions (e.g., those targeted by SCOPE-Test’s three axes), not because the models lack capacity, but because the distribution of existing data remains narrow and fails to cover critical domains. This highlights the necessity of structured data evolution rather than indiscriminately scaling pretraining corpora. Accordingly, SCOPE-Test is not meant to replace comprehensive benchmarks such as MMBench or MMMU; rather, its role is to isolate and stress-test reasoning abilities that conventional benchmarks often overlook.\n\nImportantly, SCOPE-Test retains substantial room for difficulty escalation through multi-round evolution. To illustrate, we conducted a small-scale analysis. Specifically, we selected 100 samples that underwent 10 consecutive evolution iterations and evaluated them using GPT‑4o. The results are summarized as follows:\n\n\n   \n| Evolution Rounds | Avg. Halstead Effort | GPT-4o Accuracy on SCOPE-Test (New-100)(%) |\n|--|--|--|\n|0 |3,283|97|\n|1 |4,412|93|\n|2 |5,187|89|\n|3 |6,084|88|\n|10|13,731|58|\n\n\nAs shown, additional iterations systematically increase reasoning complexity (Halstead effort ↑ by 4.2×) while reducing GPT‑4o accuracy by 39%, confirming that the benchmark’s difficulty is tunable.\n\nBeyond difficulty, SCOPE-Test reveals two critical weaknesses in current models: (1) Width limitations: difficulty in handling tasks that involve multiple interdependent variables or extended context windows; (2) Multi-image reasoning gaps: inability to integrate and reason over multiple visual inputs effectively. Both deficiencies can be directly mitigated through SCOPE’s controlled evolution strategies, which is the fundamental purpose and contribution of SCOPE-Test.\n\n\n**[W4]** *The approach is somehow complex and relies on many components, e.g., code generation and segmentation models.*\n\n**[A4]** We acknowledge that SCOPE, like prior frameworks such as ViperGPT and Visual Programming, incorporates components such as code generation and perception modules; however, these are standard elements of visual-programmatic paradigms rather than additional complexity introduced by our approach. SCOPE’s core contribution lies in aligning executable programs with chain-of-thought reasoning and introducing three orthogonal evolution strategies for controllable, systematic data diversification. Implementation remains as modular and lightweight as ViperGPT, while offering greater interpretability and verifiability through structured evolution rather than opaque model-driven augmentation. Given its strong performance and novel perspective, this program-based pipeline is a reasonable and acceptable design choice.\n\n\n**[Q1]** *How computationally expensive is your approach compared to the baselines?*\n\n**[A5]** We thank the reviewer for raising the concern regarding the computational cost of a single SCOPE iteration. Our method consists of two main components:\n\n- Code Evolution: This phase generates new QA samples through programmatic function- and code-driven expansion, without relying on multimodal pipelines or iterative LLM validation loops.\n\n- Classification: This phase assigns reasoning categories to generated QAs using static code analysis (e.g., AST parsing, Halstead metrics), avoiding LLM-based filtering.\n\nCompared to prior work, MAmmoTH-VL relies heavily on multimodal LLMs in both the evolution and classification phases, introducing substantial inference overhead and high model cost. ProVision, while emphasizing that its pipeline does not depend on LLMs, explicitly states in its paper that it requires defining 24 single-image instruction data generator programs and 14 multi-image generator programs, each utilizing hundreds of pre-defined templates—resulting in significant manual engineering cost. In contrast, VLB employs iterative image manipulation and repeated LLM-based semantic verification, both contributing to considerable computational and inference overhead.\n\nIn summary, a single SCOPE iteration remains computationally lightweight and scalable—avoiding costly scene-graph generation and LLM inference—yet achieves richer structured outputs and diversity at a cost similar to prompt-based synthesis, and significantly lower than ProVision or VLB. We report the computational cost for a single evolution iteration across different baselines for comparison. The detailed results are summarized in Table below.\n\n|Method|Evolution|Classification|Human Effort (From)|\n|-|-|-|-|\n|MAmmoTH‑VL|3-5s (MLLM)| 3-5s (MLLM)|Medium (Initial dataset quality assessment)|\n|VLB|3-5s (LLM and Image iterative editing)| 1-2ms (Pre-designed difficulty level)|High|\n|Provision|1-2s (Scene-graph construction + Python generation)|1-2ms (Pre-designed question category)|Very Very High (Hundreds of pre-defined templates)|\n|**SCOPE (ours)**|2-3s (Code-Pretrain Model or Image single editing)|1-2ms (Static code analysis only)|Minimal|"}, {"review_id": "s8J8Vv0q0K", "reviewer": "Reviewer_W2bf", "summary": "This paper presents SCOPE, a unified framework that uses visual program synthesis and counterfactual reasoning to expand and refine multimodal (image-question-answer) datasets. Starting from seed examples, SCOPE “abduces” Python programs to generate new questions, “acts” by applying targeted edits along axes like reasoning depth and visual context, and then “predicts” difficulty tiers for the resulting samples. A complementary curriculum strategy, MAP, weights examples by their programmatic complexity and routes them through specialised model components. The authors release evolving benchmarks (SCOPE-Train/Test) and show that small-to-mid-scale MLLMs, after fine-tuning on SCOPE data, improve significantly on both multimodal datasets (VQA and visual dialogue) tasks.", "questions": "- Clarify whether the multimodal dataset is intended for pre-training, post-training, or another stage of MLLM development early on in the paper.\n- In line 20, “Recent works [10, 22, 35, 36]” is misleading: three citations refer to OpenAI models (which don’t document their datasets) and one to DeepSeek-R1 (an RL-based approach). Replace these with more relevant references that support your claim about data collection and quality improvement. I’d suggest Qwen2.5-VL[1] and DeepSeek-r1[2] would be a better fit as references.\n- The scalability argument doesn’t align with pre-training, since your method relies on annotated (Image, Question, Answer) triplets. Can you pls contrast with web-scale data approaches used by foundation models (e.g. Qwen2.5VL [1], DeepSeek-r1 [2]), and specify at which model-building stage and scale you’re targeting (e.g., reasoning or VQA data for MLLMs). Notable related works include DeepSeek Math, LLaVa (and its follow-ups), and Qwen2.5-VL.\n- The related work on data augmentation isn’t thorough and needs more work to better situate itself with existing works. For image-modification methods (which I’m quite familiar with), consider engaging with VisMin [3], Demon [4],  and LANCE [5]. Additionally, please consider engaging with the program generation approach, like VisProg and ViperGPT, as they’re highly relevant but also not well discussed.\n- Specify how much baseline data (e.g., from Provision, VLM, SCOPE) is used for fine-tuning. Is the performance gain due to your method or simply increased dataset scale? Consider providing a statistics table in the main paper.\nTo isolate the impact of your created data without MAP (i.e., direct fine-tuning), indicate which table shows results “with vs. without MAP” so readers can easily compare gains from the dataset alone.\n\n[1] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[2] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\n[3] Awal, Rabiul, et al. \"VisMin: Visual Minimal-Change Understanding.\" arXiv preprint arXiv:2407.16772 (2024).\n\n[4] Li, Juncheng, et al. \"Fine-tuning multimodal llms to follow zero-shot demonstrative instructions.\" arXiv preprint arXiv:2308.04152 (2023).\n\n[5] Prabhu, Viraj, et al. \"Lance: Stress-testing visual models by generating language-guided counterfactual images.\" Advances in Neural Information Processing Systems 36 (2023): 25165-25184.", "limitations": "The limitation section, in my understanding, is not adequate. The author doesn't acknowledge the fact that they're using an existing VL dataset, which is already annotated, and hence it limits scalability by design (e.g. for web data).", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- The paper makes good use of visual programs for data generation. We've seen promising results from program generation at inference time with large models, as shown in papers like ViperGPT [1] and VisProg [2], and it's encouraging to see its usefulness for dataset creation as well. As demonstrated in the paper, program synthesis outperforms previous approaches, such as the scene graph baseline (Provision). Therefore, the paper is correct in highlighting that better vision-language datasets lead to improved model performance during fine-tuning.\n\n - The author introduces a novel curriculum approach with routing called MAP that suitably leverages the created data. \n\n- I like the experimental depth of the paper. The authors do IID (Table 1) and some OOD (Table 2/3 – I believe?). The improvement seems significant for IID. OOD result is probably good enough (as per as Table 3 is concerned). The author does a nice ablation study on both the dataset and modelling side.\n\n\n**Weaknesses**\n\n- The paper is very hard to read (has severe readability issues until section 3). The introduction is too high-level and lacks specifics. In the dataset section, it’s unclear how much data is real versus synthesised and what quantifiable gains you achieved. It’s not clear whether your contribution lies in pre-training or post-training MLLM design; please state this explicitly in the introduction section. Later on, the reader indeed picks up that it’s intended for multimodal tasks, specifically VQA-like tasks. Therefore, make clear from the outset that the work focuses mostly on VQA tasks for MLLMs.\n\n- I find the novelty of the paper from the methodology is not much. A similar dataset creation approach (or the use of VLM programs) has already been studied in VisProg [1], ViperGPT [2], LANCE [3], VisMin [4], Demon [5], and Mamoth-VL [6].\n\n- I think the paper does a bad job of establishing why one would go to such lengths for data creation for MLLM tasks; the benchmarks chosen seem to be already quite saturated in both closed-source models and the recent open-source large models (e.g. Qwen2.5-VL 72B [6]). Is this worth the trouble of program creation to make multiple inferences to create a sample? Also, I’m certain this approach is not scalable for web-scale data, i.e. refining them is more important than already curated data and extracting the last juice out of the VL datasets that the authors engaged with! By quickly checking Qwen2.5VL 72B[6], the results I found (MMBench: 88.4, RealWorldQA 75.7, MMvet: 76.2, which is significantly higher than SCOPE w/ Qwen2.5VL-7B. I believe this is very important to establish why one would use this approach instead of using a much simpler approach, such as rule-based filtering or model-based filtering done in works like Qwen2.5-VL. Given that, do we need programs in the first place? If GPT-4 can generate those programs and is good enough to answer most of the questions in the benchmarks, why not use that system to generate questions (with some demonstration) like Qwen2.5-VL is doing? Do we need counterfactuals or some modified version of the original images, where the web is full of images? Did the author try to generate examples using GPT-4 without the use of visual programs?\n\n- Related work is also not well written (see the Questions Section). It’s important to situate the contribution precisely in the context of previous work and the stage of MLLM development.\n\n- The dataset and methodology section can be written better with more concrete details. Please include clear statistics – dataset size (?), number of programs generated (?), number of models involved (?), the error rate of program execution (?) and overall cost (?) – to give readers a solid picture of how the dataset was created before the experiments. The paper should provide the details statistics of the raw VL datasets and how many samples are used in baselines too (currently, the author provides the SCOPE train test stats).  Also, clarify which models you used beyond GPT-4 and SAM (for example, which model handles image-inference subroutines) and describe the average depth of your generated programs. Finally, include a few qualitative examples comparing raw and extended data early on. I think the appendix has a fair bit of material that one could use more clearly in the main paper (or at least refer to it).\n\n- Regarding the experiments and the overall contribution, the main concern I have is which size of MLLM this work is improving? I acknowledge that GPT4 results are reported. What about large open-source models, e.g. ~72B models (those results are missing)? GPT-4 performs almost as well as SCOPE in Table 2, but that doesn't diminish SCOPE's value; to better understand the dataset’s utility across scales, it would help if the authors reported results for larger open-source models like Qwen2.5-72B [6] and InternVL2.5-78B [7]  in Table 2 (and Table 3 if relevant). Clearly, the paper shows that SCOPE is useful for 2B to 7B+ models. So, I think a large-scale open-source model’s capacity helps situate the paper's contribution more clearly within MLLM development. Maybe this dataset is only useful for small-scale models (<7B)? I don’t think the gains in Tables 2 and 3 as as significant. \n\n- I am curious what’s the accuracy of the fine-tuning without the (MAP) approach in Tables 1, 2 and 3? I understand there’s an ablation given in Table 4. If I understand correctly, that ablation doesn’t give a clear, direct fine-tuning on the dataset. It would be nice to have that result to have clarity on the method's contribution.\n\n[1] Surís, Dídac, Sachit Menon, and Carl Vondrick. \"Vipergpt: Visual inference via python execution for reasoning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Gupta, Tanmay, and Aniruddha Kembhavi. \"Visual programming: Compositional visual reasoning without training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Awal, Rabiul, et al. \"VisMin: Visual Minimal-Change Understanding.\" arXiv preprint arXiv:2407.16772 (2024).\n\n[4] Li, Juncheng, et al. \"Fine-tuning multimodal llms to follow zero-shot demonstrative instructions.\" arXiv preprint arXiv:2308.04152 (2023).\n\n[5] Guo, Jarvis, et al. \"Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.\" arXiv preprint arXiv:2412.05237 (2024).\n\n[6] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[7] Chen, Zhe, et al. \"Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.\" arXiv preprint arXiv:2412.05271 (2024).", "was_revised": true, "final_justification": "With the additional results and clarifications addressing my points and those of other reviewers, I believe the authors can make the contribution and scope clearer and more specific (what has been done in this paper only -- not what could've been possible). There's also additional concern from other reviewers on the complexity and the generalizability of the method. I strongly believe the paper needs to be clear about its contribution and scope. \n\nUpdate: I've raised my score to borderline accept following author-rebuttal discussions!", "rebuttal": "**[W1 & Q1]** *The concerns are about clarity, including the need for a more specific introduction that clearly states the work focuses on VQA tasks, clarifies whether the dataset is for pre-training or post-training, and provides details on real vs. synthetic data and the quantifiable gains achieved.*\n\n**[A1]** We thank the reviewer for the feedback and clarify the main concerns. Our introduction already states the motivation: addressing the growing mismatch between dataset evolution and model advancement—a problem that cannot be solved by simply enlarging pre-training data. We will further improve clarity by summarizing key contributions upfront. Regarding data composition, the Appendix provides full details, and we will add a concise summary in the main text to clarify that only the initial dataset is real, while all evolutionary stages are synthetic, along with quantifiable gains. SCOPE is a general paradigm applicable to both pre-training and post-training, not restricted to either. Finally, SCOPE is not limited to VQA; as a Visual Programming-based approach, it supports a broad range of multimodal tasks (e.g., video QA, image editing, multi-image reasoning).\n\n**[W2]**  *The novelty of the paper from the methodology is not much. A similar dataset creation approach (or the use of VLM programs) has already been studied.*\n\n**[A2]** We would like to clarify that our contribution does not lie in proposing Visual Programming itself or minor modifications of it. Instead, we introduce a novel data evolution algorithm that enables counterfactual editing of reasoning steps. To achieve this, we are the first to align reasoning steps with code blocks in Visual Programming, allowing intervention at the structural or variable level to alter reasoning paths—a capability not present in prior work. Furthermore, we note that several cited works, such as VisProg and ViperGPT, are not dataset creation approaches. Our method fundamentally differs by leveraging code-level interventions to generate evolved datasets for enhancing reasoning, which is the core novelty of SCOPE.\n\n**[W3 & Q3]** *The concerns focus on the necessity and scalability of program-based data generation versus simpler approaches, questioning whether the added complexity and counterfactual modifications are justified given strong existing models and large-scale web data alternatives.*\n\n**[A3]** In fact, SCOPE is designed to enable controllable, interpretable evolution of reasoning steps, not merely to enlarge datasets or match saturated benchmarks. This is important because counterfactually editing reasoning paths creates new data distributions and reasoning challenges that web-scale datasets and existing benchmarks do not capture. Visual Programming is the most effective tool for this, as it allows fine-grained structural intervention—beyond what templates or scene graphs can achieve. Regarding scalability, I’m certain this approach is quite scalable for web-scale data by generating initial questions and programs and then applying SCOPE evolution, similar to VLB or Provision. Finally, we compared SCOPE-generated challenging samples with GPT-augmented ones and found that SCOPE yields examples significantly harder even for GPT itself, demonstrating the necessity of structured program-based evolution over simple prompt-based generation.\n\n\n| Evolution Rounds | Avg. Halstead Effort | GPT-4o Accuracy on SCOPE-Test (New-100)(%) |\n|--|--|--|\n|0 |3,283|97|\n|1 |4,412|93|\n|2 |5,187|89|\n|3 |6,084|88|\n|10(GPT-Prompt)|-|79|\n|10(SCOPE)|13,731|58|\n\n\nAs for the question 3, current MLLM training pipelines are typically multi-stage, combining large-scale pre-training on web data with subsequent fine-tuning on higher-quality, task-specific datasets. For instance, Qwen2.5-VL describes a three-stage process (Sec. 2.2.2), and InternVL2.5 reports two stages (Table 3). Our method, SCOPE, is primarily designed for the fine-tuning stage, where curated data are already of higher quality than raw web-scale data. SCOPE aims to further enhance this data through counterfactual reasoning-step evolution, improving controllability and diversity beyond what web-scale pre-training can offer.\n\nContrary to the concern, SCOPE is not inherently incompatible with web-scale data. Many pre-training tasks—such as image captioning or OCR—have already been shown to be representable in programmatic form (e.g., ViperGPT and Visual Programming works). Therefore, applying SCOPE to evolve web-scale datasets is technically feasible. Finally, we stress that scaling up data size and improving data quality are orthogonal dimensions; SCOPE targets the latter, which remains critical even when training extremely large models.\n\n**[W4 & Q4]** *The related work section needs to be revised.*\n\n**[A4]** We will consider revising the Related Work section and adding some relevant citations as suggested. However, we note that image-modification is not the core aspect of our approach; in SCOPE it is used only as one of several tools, so our discussion focuses instead on Visual Programming and counterfactual reasoning, which are more central to our contribution. Additionally, our work is built on ViperGPT, we are confident that our discussion of ViperGPT is sufficiently detailed.\n\n**[W5]** *The concerns are about adding concrete details in the dataset and methodology section.*\n\n**[A5]** We would like to clarify that the requested information is already provided in the submission, although primarily organized in the Appendix for readability and space considerations. Specifically, the raw VL dataset statistics and baseline sample details are presented in Appendix Sec. 7.1, while the full data format specification is in Sec. 7.3. Each data instance in SCOPE contains one original example and three counterfactually evolved examples, all following a unified format with VQA annotations and corresponding programs. The resulting datasets include 31,233 training examples and 13,389 test examples, each paired with executable code. During program generation, we enforce execution correctness by iterative verification, ensuring a 100% success rate. Furthermore, to assess the quality of evolved data, we performed random human evaluations across three evolution iterations, achieving acceptability rates of 96%, 92%, and 91%, respectively. Regarding the methodology, in addition to GPT-4o, we used kvedit for image-editing subroutines. For better interpretability, we have already included a qualitative example in Figure 5, illustrating differences between raw and evolved samples. We will bring key statistics into the Benchmark section and cross-reference relevant figures for completeness.\n\n**[W6]** *The concern is about clarifying which model sizes benefit from SCOPE, requesting results for large open-source models to determine whether SCOPE’s utility extends beyond small-scale models and to better contextualize its contribution.*\n\n**[A6]** We appreciate the reviewer’s suggestion. Our choice of 2B and 7B models was intentional: these scales represent the most widely used and practical sizes for deployment due to their balance between performance, inference speed, and resource requirements. Most recent studies also report results primarily in this range. The key objective of SCOPE is to demonstrate that our data evolution method can bring consistent and significant improvements even at these common scales without requiring additional model capacity—an important factor for real-world applications where retraining extremely large models is infeasible.\n\nRegarding 70B-scale models such as Qwen2.5-VL-72B or InternVL2.5-78B, our current hardware resources do not allow us to retrain or fine-tune models at that scale. However, we will include the official results of these models (without fine-tuning) in Tables 2 and 3 for comparison. This will make clear that while absolute performance from very large models is higher, SCOPE provides a scalable and cost-efficient strategy to improve reasoning ability for widely used mid-sized models.\n\n**[W7 & Q5]** *The concern is about clarifying how much baseline data is used for fine-tuning, whether performance gains come from your method or increased data size, and explicitly showing results with vs. without MAP to isolate the dataset’s impact, ideally supported by a statistics table.*\n\n**[A7]** The performance improvements with SCOPE come from two factors: higher-quality data and the MAP training strategy. To ensure fairness in comparisons, we applied the same MAP approach to all baselines, including Provision and VLB. Specifically, we computed curriculum difficulty scores for their samples, adopted balanced gating, and assigned fixed difficulty-aware loss according to their data distribution. This controls for differences in training method, meaning that the performance gaps shown in Tables 1–3 primarily reflect data quality rather than training strategy. From Table 2, readers can already directly compare the gains attributable to the dataset itself under the same MAP setting.\n\nIn response to the reviewer’s suggestion, we will extend Table 4 by adding a w/o MAP setting for direct fine-tuning. This will provide a clearer view of the independent contribution of MAP versus data quality.\n\n\n| # | Setting | MMBench<sub>test</sub> | ContextVD |\n|---|---|---|---|\n| 0 | **InternVL-2.5-2B (SCOPE)**| 77.0  | 93.6  |\n| 1 | w/o curriculum scheduler| 76.9  | 93.1 |\n| 2 | w/o gating module| 75.3  | 87.4  | \n| 3 | w/o difficulty-aware loss | 75.8  | 91.2  | \n| 4 | **w/o MAP (direct fine-tuning)** |  75.1| 87.1 | \n| 5 | Backbone   | 74.7  | 85.7  | \n\n\n\n**[Q2]** *References need to be revised.*\n\n**[A8]** We acknowledge this point and will carefully review all references throughout the paper to ensure they accurately support the claims made."}, {"review_id": "De2Gr1KQ3I", "reviewer": "Reviewer_G6Ro", "summary": "This paper introduces SCOPE (Scalable Counterfactual Program Evolution), a framework for the systematic evolution of multimodal datasets. The core idea is to move beyond static benchmarks by using visual programming as a medium for counterfactual reasoning. The process involves three stages: (1) Abduction, where image-question pairs are converted into verifiable Python programs; (2) Action, where these programs are manipulated along three axes (reasoning path, visual context, cross-instance composition) to create new, more challenging instances; and (3) Prediction, where the evolved data is categorized by difficulty, structure, and input multiplicity. Based on this framework, the authors create and release the SCOPE-Train and SCOPE-Test benchmarks. They also propose MAP, a curriculum learning strategy tailored to the structured nature of the evolved data. Experiments demonstrate that models trained on SCOPE-Train achieve improved reasoning performance and that SCOPE-Test can effectively diagnose weaknesses in current state-of-the-art MLLMs.", "questions": "1. What's the distribution of question types such as counting, object attribute identification (e.g., color, texture), spatial relationship reasoning, OCR, or comparative questions?\n\n2. Does training on SCOPE improve model performance on rich-knowledge benchmarks like MMMU?", "limitations": "See Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. Strengths\n\n1.1 The core idea is well-motivated\n\nThe central contribution—using a symbolic, programmatic framework for counterfactual dataset evolution—is well-motivated. It represents an improvement from static data collection or simple template-based augmentation towards a dynamic, controllable, and verifiable data generation paradigm. \n\n1.2 Good controllability and verifiability\n\nA standout feature of this work is the inherent verifiability of the visual programming approach. Representing reasoning chains as executable code makes the entire process transparent and auditable. The three proposed intervention axes provide a structured and fine-grained mechanism for controlling the evolution of data complexity, which is a major advantage over prior \"black-box\" generation methods.\n\n1.3 The experiments results show consist improvement of SCOPE\n\nThe authors show that SCOPE-Test can uncover systematic blind spots in powerful MLLMs that are not apparent on conventional benchmarks. The performance gains from training on SCOPE-Train are consistent across multiple models and benchmarks. Furthermore, the high human acceptance rates (91-96%) for the evolved data strongly support the quality and logical coherence of the generated instances.\n\n2. Weaknesses\n\n2.1 Lack of semantic question type analysis\n\nThere is no analysis of the distribution of question types such as counting, object attribute identification (e.g., color, texture), spatial relationship reasoning, OCR, or comparative questions. It is therefore unclear if the SCOPE framework has an inherent bias towards generating certain types of questions that are easier to represent programmatically (e.g., simple counting or attribute verification) while neglecting more complex semantic reasoning tasks. A thorough analysis of this semantic distribution is crucial for understanding the true diversity and potential biases of the benchmark.\n\n2.2 Constrained by the expressiveness of the API library and the proprietary generator model\n\nThe framework's ability to generate diverse reasoning paths is fundamentally limited by the expressiveness of its predefined API library and the proprietary generator model. Complex reasoning that cannot be decomposed into the provided functions (e.g., understanding abstract concepts, social interactions, or complex causality) is implicitly out of scope. While the appendix lists the API, a discussion in the main paper about its limitations, the design principles behind it, and the potential for extension would be beneficial.", "was_revised": true, "final_justification": "My concerns have been addressed. I have raise my rating to 4", "rebuttal": "**[W1 & Q1]** *Lack of semantic question type analysis: There is no analysis of the distribution of question types such as counting, object attribute identification (e.g., color, texture), spatial relationship reasoning, OCR, or comparative questions. It is therefore unclear if the SCOPE framework has an inherent bias towards generating certain types of questions that are easier to represent programmatically (e.g., simple counting or attribute verification) while neglecting more complex semantic reasoning tasks. A thorough analysis of this semantic distribution is crucial for understanding the true diversity and potential biases of the benchmark. What's the distribution of question types such as counting, object attribute identification (e.g., color, texture), spatial relationship reasoning, OCR, or comparative questions?*\n\n**[A1]** Thank you for raising this important point. We fully agree that analyzing the semantic distribution of question types is crucial for assessing the diversity and potential biases of the benchmark. Our initial analysis focused primarily on structural diversity and reasoning complexity, as these are central to SCOPE’s programmatic evolution paradigm. However, we acknowledge the value of explicitly reporting semantic question-type coverage.\n\nTo address this, we performed an additional categorization of SCOPE-Test questions into major semantic types commonly used in multimodal reasoning benchmarks: counting, object attribute identification (color, texture, shape), spatial relationship reasoning, OCR-based reasoning, and comparative reasoning. The distribution is as follows:\n\n| **Question Type** | SCOPE-Train (Number / Proportion (%)) | SCOPE-Test (Number / Proportion (%)) |\n|--|--|--|\n| Counting| 4,309 / 13.8% | 1,579 / 11.8%  |\n| Object Attribute (color, texture, shape) | 6,471 / 20.7% | 2,534 / 18.9%  |\n| Spatial Relationships | 5,377 / 17.2%  | 2,589 / 19.3%  |\n| OCR-based Reasoning  | 6,127 / 19.6% | 2,411 / 18.0% |\n| Multi-hop | 5,903 / 18.9%   | 2,958 / 22.1%  |\n| Others  | 3,046 / 9.8%   | 1,318 / 9.9%   |\n| **Total**   | **31,233 / 100.0%**| **13,389 / 100.0%** |\n\n\nThis analysis demonstrates that SCOPE-Test is not dominated by trivial tasks such as simple counting or single-attribute queries. On the contrary, nearly half of the questions involve complex reasoning—including comparative, multi-hop, and spatial queries—while OCR-based tasks also constitute a substantial proportion, requiring symbolic reasoning and visual-text integration.\n\nFurthermore, because SCOPE’s evolution strategies operate at the programmatic level, semantic diversity naturally increases across evolution rounds as reasoning paths expand and cross-instance composition introduces multi-faceted logic. Importantly, the framework remains flexible: if future benchmarks require stronger representation of specific semantic categories, targeted function additions or evolution heuristics can easily address this without redesigning the architecture.\n\nAdditionally, we also evaluated diversity in the question space by measuring distributional differences using Maximum Mean Discrepancy (MMD). Specifically, we compared the QA distribution under SCOPE with two baselines: VLB and Provision. Our results show that SCOPE achieves a significantly higher MMD value (0.38) compared to VLB (0.15) and Provision (0.18), indicating that SCOPE's QA outputs are farther from the baseline distributions and thus occupy a more uniform and dispersed semantic space across reasoning categories. A larger MMD signifies greater divergence from the baseline distributions, demonstrating that SCOPE generates a structurally and semantically more varied QA set overall.\n\n**[W2 & Q2]** *Constrained by the expressiveness of the API library and the proprietary generator model: The framework's ability to generate diverse reasoning paths is fundamentally limited by the expressiveness of its predefined API library and the proprietary generator model. Complex reasoning that cannot be decomposed into the provided functions (e.g., understanding abstract concepts, social interactions, or complex causality) is implicitly out of scope. While the appendix lists the API, a discussion in the main paper about its limitations, the design principles behind it, and the potential for extension would be beneficial. Does training on SCOPE improve model performance on rich-knowledge benchmarks like MMMU?*\n\n**[A2]** Thank you for this insightful observation.\n\nFirst, SCOPE’s design builds upon the established visual programming paradigm introduced by frameworks such as ViperGPT, Visual Programming (VISPROG), and De-fine, which have demonstrated that programmatic reasoning with modular APIs can generalize across a wide range of multimodal tasks—well beyond classical VQA. For example, ViperGPT has successfully applied executable programs to visual grounding, image QA, video QA, multi-image comparison, symbolic math, and knowledge-augmented queries without task-specific retraining. Similarly, VISPROG and De-fine show that code-driven pipelines enable compositional reasoning, iterative refinement, and improved robustness across diverse domains. These foundations give SCOPE a strong theoretical basis for generalizability.\n\nImportantly, for all current benchmarks, we do not modify the existing API or introduce new functions—the rich function set inherited from ViperGPT is sufficient to support tasks spanning visual reasoning, mathematical problem-solving (e.g., MathVista), knowledge-intensive QA (e.g., MMMU), and OCR-based document understanding (e.g., DocVQA). Thus, the current implementation already covers a wide range of reasoning capabilities without additional engineering overhead. To further validate both generalization and extensibility, we conducted experiments on these four diverse benchmarks. The results are summarized below:\n\n\n| Benchmark | Base Performance | w/ VLB | w/ SCOPE | Δ (Compare to VLB) |\n|--|--|--|--|--|\n| **MathVista** | 51.3 | 52.5 | 53.9  | +1.4  |\n| **MMMU** | 43.6   | 44.9 | 45.8 | +0.9 |\n| **AI2D** | 74.9   | 76.3 | 77.9 | +1.6 |\n| **DocVQA** | 88.7  | 89.9 | 91.8 | +1.9 |\n\nThese results show that SCOPE consistently improves reasoning accuracy across diverse domains, outperforming VLB by up to +1.9 points. This demonstrates that SCOPE not only adapts to knowledge-intensive benchmarks like MMMU but also scales to tasks requiring OCR, symbolic reasoning, and domain-specific interpretation, confirming both its generalizability and practical effectiveness.\n\nAt the same time, SCOPE remains inherently extensible. If future benchmarks require handling concepts beyond the existing API (e.g., domain-specific logic or abstract causal reasoning), new functions can be seamlessly integrated into the library without redesigning templates, prompt structures, or scene graphs, nor retraining the underlying models. This modularity ensures that complexity is localized and controlled, rather than requiring end-to-end system changes.\n\nFinally, to address your question, our answer is: yes, training on SCOPE can improve model performance on rich-knowledge benchmarks such as MMMU. This is because SCOPE leverages external APIs during evolution, effectively distilling tool- and model-specific knowledge into the data, thereby enhancing the model’s reasoning and knowledge capabilities. SCOPE’s program-based paradigm offers interpretability, controllability, and modular extensibility, making it particularly effective for complex, multi-step reasoning tasks."}]

Author rebuttal:

Meta-review: Strengths noted across reviews include the clever program-based interventions, consistent empirical improvements, high human acceptance rates, and the release of useful resources. 

However, concerns persist about clarity and framing (pre- vs post-training use), reliance on powerful backbone models (e.g., GPT-4o), scalability to web-scale or more abstract reasoning tasks, and whether the approach is overly complex relative to the reported gains. 

Overall, despite uneven clarity and questions about generalization, the contribution is technically fine and of sufficient interest to warrant a borderline accept.
Decision: Accept (poster)