Title: Counterfactual Evolution of Multimodal Datasets via Visual Programming
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dataset Evolution

Visual Programming

Counterfactual Evolution

Multimodal Datasets
Abstract: The rapid development of Multimodal Large Language Models (MLLMs) poses increasing demands on the diversity and complexity of multimodal datasets. Yet manual annotation pipelines can no longer keep pace. Existing augmentation methods often follow fixed rules and lack verifiable control over sample diversity and reasoning complexity. To address this, we introduce Scalable COunterfactual Program Evolution (SCOPE), a framework that uses symbolic Visual Programming to guide program evolution via counterfactual reasoning. SCOPE performs the three steps of counterfactual inference: (1) Abduction, by generating verifiable programs to model reasoning associations; (2) Action, by intervening on program structure along three axes—reasoning path, visual context, and cross-instance composition; and (3) Prediction, by categorizing evolved instances by difficulty, structure, and input multiplicity. Based on this process, we build SCOPE-Train and SCOPE-Test, evolving benchmarks with expert validation. To support training, we propose MAP, a curriculum learning strategy that aligns model capacity with sample difficulty. Experiments show that SCOPE improves reasoning performance, exposes model blind spots, and enhances visual dialog capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SCOPE (Scalable Counterfactual Program Evolution), a framework for the systematic evolution of multimodal datasets. The core idea is to move beyond static benchmarks by using visual programming as a medium for counterfactual reasoning. The process involves three stages: (1) Abduction, where image-question pairs are converted into verifiable Python programs; (2) Action, where these programs are manipulated along three axes (reasoning path, visual context, cross-instance composition) to create new, more challenging instances; and (3) Prediction, where the evolved data is categorized by difficulty, structure, and input multiplicity. Based on this framework, the authors create and release the SCOPE-Train and SCOPE-Test benchmarks. They also propose MAP, a curriculum learning strategy tailored to the structured nature of the evolved data. Experiments demonstrate that models trained on SCOPE-Train achieve improved reasoning performance and that SCOPE-Test can effectively diagnose weaknesses in current state-of-the-art MLLMs.\n\nSTRENGTHS AND WEAKNESSES:\n1. Strengths\n\n1.1 The core idea is well-motivated\n\nThe central contribution—using a symbolic, programmatic framework for counterfactual dataset evolution—is well-motivated. It represents an improvement from static data collection or simple template-based augmentation towards a dynamic, controllable, and verifiable data generation paradigm. \n\n1.2 Good controllability and verifiability\n\nA standout feature of this work is the inherent verifiability of the visual programming approach. Representing reasoning chains as executable code makes the entire process transparent and auditable. The three proposed intervention axes provide a structured and fine-grained mechanism for controlling the evolution of data complexity, which is a major advantage over prior \"black-box\" generation methods.\n\n1.3 The experiments results show consist improvement of SCOPE\n\nThe authors show that SCOPE-Test can uncover systematic blind spots in powerful MLLMs that are not apparent on conventional benchmarks. The performance gains from training on SCOPE-Train are consistent across multiple models and benchmarks. Furthermore, the high human acceptance rates (91-96%) for the evolved data strongly support the quality and logical coherence of the generated instances.\n\n2. Weaknesses\n\n2.1 Lack of semantic question type analysis\n\nThere is no analysis of the distribution of question types such as counting, object attribute identification (e.g., color, texture), spatial relationship reasoning, OCR, or comparative questions. It is therefore unclear if the SCOPE framework has an inherent bias towards generating certain types of questions that are easier to represent programmatically (e.g., simple counting or attribute verification) while neglecting more complex semantic reasoning tasks. A thorough analysis of this semantic distribution is crucial for understanding the true diversity and potential biases of the benchmark.\n\n2.2 Constrained by the expressiveness of the API library and the proprietary generator model\n\nThe framework's ability to generate diverse reasoning paths is fundamentally limited by the expressiveness of its predefined API library and the proprietary generator model. Complex reasoning that cannot be decomposed into the provided functions (e.g., understanding abstract concepts, social interactions, or complex causality) is implicitly out of scope. While the appendix lists the API, a discussion in the main paper about its limitations, the design principles behind it, and the potential for extension would be beneficial.\n\nQUESTIONS:\n1. What's the distribution of question types such as counting, object attribute identification (e.g., color, texture), spatial relationship reasoning, OCR, or comparative questions?\n\n2. Does training on SCOPE improve model performance on rich-knowledge benchmarks like MMMU?\n\nLIMITATIONS:\nSee Weaknesses", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SCOPE, a unified framework that uses visual program synthesis and counterfactual reasoning to expand and refine multimodal (image-question-answer) datasets. Starting from seed examples, SCOPE “abduces” Python programs to generate new questions, “acts” by applying targeted edits along axes like reasoning depth and visual context, and then “predicts” difficulty tiers for the resulting samples. A complementary curriculum strategy, MAP, weights examples by their programmatic complexity and routes them through specialised model components. The authors release evolving benchmarks (SCOPE-Train/Test) and show that small-to-mid-scale MLLMs, after fine-tuning on SCOPE data, improve significantly on both multimodal datasets (VQA and visual dialogue) tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper makes good use of visual programs for data generation. We've seen promising results from program generation at inference time with large models, as shown in papers like ViperGPT [1] and VisProg [2], and it's encouraging to see its usefulness for dataset creation as well. As demonstrated in the paper, program synthesis outperforms previous approaches, such as the scene graph baseline (Provision). Therefore, the paper is correct in highlighting that better vision-language datasets lead to improved model performance during fine-tuning.\n\n - The author introduces a novel curriculum approach with routing called MAP that suitably leverages the created data. \n\n- I like the experimental depth of the paper. The authors do IID (Table 1) and some OOD (Table 2/3 – I believe?). The improvement seems significant for IID. OOD result is probably good enough (as per as Table 3 is concerned). The author does a nice ablation study on both the dataset and modelling side.\n\n\n**Weaknesses**\n\n- The paper is very hard to read (has severe readability issues until section 3). The introduction is too high-level and lacks specifics. In the dataset section, it’s unclear how much data is real versus synthesised and what quantifiable gains you achieved. It’s not clear whether your contribution lies in pre-training or post-training MLLM design; please state this explicitly in the introduction section. Later on, the reader indeed picks up that it’s intended for multimodal tasks, specifically VQA-like tasks. Therefore, make clear from the outset that the work focuses mostly on VQA tasks for MLLMs.\n\n- I find the novelty of the paper from the methodology is not much. A similar dataset creation approach (or the use of VLM programs) has already been studied in VisProg [1], ViperGPT [2], LANCE [3], VisMin [4], Demon [5], and Mamoth-VL [6].\n\n- I think the paper does a bad job of establishing why one would go to such lengths for data creation for MLLM tasks; the benchmarks chosen seem to be already quite saturated in both closed-source models and the recent open-source large models (e.g. Qwen2.5-VL 72B [6]). Is this worth the trouble of program creation to make multiple inferences to create a sample? Also, I’m certain this approach is not scalable for web-scale data, i.e. refining them is more important than already curated data and extracting the last juice out of the VL datasets that the authors engaged with! By quickly checking Qwen2.5VL 72B[6], the results I found (MMBench: 88.4, RealWorldQA 75.7, MMvet: 76.2, which is significantly higher than SCOPE w/ Qwen2.5VL-7B. I believe this is very important to establish why one would use this approach instead of using a much simpler approach, such as rule-based filtering or model-based filtering done in works like Qwen2.5-VL. Given that, do we need programs in the first place? If GPT-4 can generate those programs and is good enough to answer most of the questions in the benchmarks, why not use that system to generate questions (with some demonstration) like Qwen2.5-VL is doing? Do we need counterfactuals or some modified version of the original images, where the web is full of images? Did the author try to generate examples using GPT-4 without the use of visual programs?\n\n- Related work is also not well written (see the Questions Section). It’s important to situate the contribution precisely in the context of previous work and the stage of MLLM development.\n\n- The dataset and methodology section can be written better with more concrete details. Please include clear statistics – dataset size (?), number of programs generated (?), number of models involved (?), the error rate of program execution (?) and overall cost (?) – to give readers a solid picture of how the dataset was created before the experiments. The paper should provide the details statistics of the raw VL datasets and how many samples are used in baselines too (currently, the author provides the SCOPE train test stats).  Also, clarify which models you used beyond GPT-4 and SAM (for example, which model handles image-inference subroutines) and describe the average depth of your generated programs. Finally, include a few qualitative examples comparing raw and extended data early on. I think the appendix has a fair bit of material that one could use more clearly in the main paper (or at least refer to it).\n\n- Regarding the experiments and the overall contribution, the main concern I have is which size of MLLM this work is improving? I acknowledge that GPT4 results are reported. What about large open-source models, e.g. ~72B models (those results are missing)? GPT-4 performs almost as well as SCOPE in Table 2, but that doesn't diminish SCOPE's value; to better understand the dataset’s utility across scales, it would help if the authors reported results for larger open-source models like Qwen2.5-72B [6] and InternVL2.5-78B [7]  in Table 2 (and Table 3 if relevant). Clearly, the paper shows that SCOPE is useful for 2B to 7B+ models. So, I think a large-scale open-source model’s capacity helps situate the paper's contribution more clearly within MLLM development. Maybe this dataset is only useful for small-scale models (<7B)? I don’t think the gains in Tables 2 and 3 as as significant. \n\n- I am curious what’s the accuracy of the fine-tuning without the (MAP) approach in Tables 1, 2 and 3? I understand there’s an ablation given in Table 4. If I understand correctly, that ablation doesn’t give a clear, direct fine-tuning on the dataset. It would be nice to have that result to have clarity on the method's contribution.\n\n[1] Surís, Dídac, Sachit Menon, and Carl Vondrick. \"Vipergpt: Visual inference via python execution for reasoning.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Gupta, Tanmay, and Aniruddha Kembhavi. \"Visual programming: Compositional visual reasoning without training.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Awal, Rabiul, et al. \"VisMin: Visual Minimal-Change Understanding.\" arXiv preprint arXiv:2407.16772 (2024).\n\n[4] Li, Juncheng, et al. \"Fine-tuning multimodal llms to follow zero-shot demonstrative instructions.\" arXiv preprint arXiv:2308.04152 (2023).\n\n[5] Guo, Jarvis, et al. \"Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale.\" arXiv preprint arXiv:2412.05237 (2024).\n\n[6] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[7] Chen, Zhe, et al. \"Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.\" arXiv preprint arXiv:2412.05271 (2024).\n\nQUESTIONS:\n- Clarify whether the multimodal dataset is intended for pre-training, post-training, or another stage of MLLM development early on in the paper.\n- In line 20, “Recent works [10, 22, 35, 36]” is misleading: three citations refer to OpenAI models (which don’t document their datasets) and one to DeepSeek-R1 (an RL-based approach). Replace these with more relevant references that support your claim about data collection and quality improvement. I’d suggest Qwen2.5-VL[1] and DeepSeek-r1[2] would be a better fit as references.\n- The scalability argument doesn’t align with pre-training, since your method relies on annotated (Image, Question, Answer) triplets. Can you pls contrast with web-scale data approaches used by foundation models (e.g. Qwen2.5VL [1], DeepSeek-r1 [2]), and specify at which model-building stage and scale you’re targeting (e.g., reasoning or VQA data for MLLMs). Notable related works include DeepSeek Math, LLaVa (and its follow-ups), and Qwen2.5-VL.\n- The related work on data augmentation isn’t thorough and needs more work to better situate itself with existing works. For image-modification methods (which I’m quite familiar with), consider engaging with VisMin [3], Demon [4],  and LANCE [5]. Additionally, please consider engaging with the program generation approach, like VisProg and ViperGPT, as they’re highly relevant but also not well discussed.\n- Specify how much baseline data (e.g., from Provision, VLM, SCOPE) is used for fine-tuning. Is the performance gain due to your method or simply increased dataset scale? Consider providing a statistics table in the main paper.\nTo isolate the impact of your created data without MAP (i.e., direct fine-tuning), indicate which table shows results “with vs. without MAP” so readers can easily compare gains from the dataset alone.\n\n[1] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" arXiv preprint arXiv:2502.13923 (2025).\n\n[2] Guo, Daya, et al. \"Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.\" arXiv preprint arXiv:2501.12948 (2025).\n\n[3] Awal, Rabiul, et al. \"VisMin: Visual Minimal-Change Understanding.\" arXiv preprint arXiv:2407.16772 (2024).\n\n[4] Li, Juncheng, et al. \"Fine-tuning multimodal llms to follow zero-shot demonstrative instructions.\" arXiv preprint arXiv:2308.04152 (2023).\n\n[5] Prabhu, Viraj, et al. \"Lance: Stress-testing visual models by generating language-guided counterfactual images.\" Advances in Neural Information Processing Systems 36 (2023): 25165-25184.\n\nLIMITATIONS:\nThe limitation section, in my understanding, is not adequate. The author doesn't acknowledge the fact that they're using an existing VL dataset, which is already annotated, and hence it limits scalability by design (e.g. for web data).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a method to generate multimodal datasets using a combination of visual programming and counterfactual reasoning. The goal of the paper is to address limitations of existing dataset construction methods, e.g., being heuristic, template-based, or rely too heavily on model generation. The authors apply their approach on a set of existing dataset to obtain a train and test sets, which are used to train and evaluate some open VLMs. The proposed method starts by converting an image-query pair into executable code, then modifies the python code along three axes: reasoning path, visual context, and cross-instance composition to generate new samples. Experiments show that models trained on SCOPE exhibit better performance, compared to two baseline augmentation methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. the idea of using interventions in the code rather than the image space is clever and novel, since code is interpretable, controllable and easy to measure its complexity.\n2. curriculum learning via MAP is well ablated and yields performance gains.\n3. overall, training on SCOPE seems to yield better model compared to two other augmentation strategies\n\n\nWeakness:\n1. The main bottleneck here is that this approach will only work with instances that *can* be described in code. This means the approach is not generalizable. \n2. The authors criticize prior work that relies heavily on model generation, yet SCOPE relies on GPT-4o for program generation. Also what if the generated code has flaws but still runs? \n3. SCOPE-test is not a challenging benchmark by any means; GPT-4o already achieves 88% accuracy.\n4. The approach is somehow complex and relies on many components, e.g., code generation and segmentation models.\n\nQUESTIONS:\n1. How computationally expensive is your approach compared to the baselines\n2. How would you generalize your approach to cover instances where generating a program is not possible?\n\nLIMITATIONS:\nThe authors address some limitations in the appendix not in the main paper", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SCOPE, a novel framework for evolving multimodal datasets using symbolic visual programming and counterfactual reasoning. Key contributions include:\n1.\tVerifiable Dataset Evolution: SCOPE generates executable Python programs to model reasoning paths, intervenes via program edits along three axes—reasoning path expansion, visual context editing, and cross-instance composition (action)—and categorizes evolved instances by difficulty/structure/modality.\n2.\tBenchmarks: Constructs dynamically evolving benchmarks (SCOPE-Train/SCOPE-Test) with expert-validated samples (91–96% human acceptance).\n3.\tCurriculum Learning: Proposes MAP (Memory and Attention Path), aligning model capacity with difficulty progression through adaptive gating and loss weighting.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n•\tVerifiable Control: Uses executable Python programs for traceable reasoning chains, avoiding black-box LLM hallucinations.\n•\tDiagnostic Benchmark: SCOPE-Test reveals fine-grained model weaknesses, especially in multi-image/hard reasoning scenarios.\n•\tResource Accessibility: Open-sourced code/data enables community-driven evolution.\nWeaknesses\n•\tThe framework aims to improve the model through data, but its starting point depends on a very powerful model (GPT-4o). The quality and diversity of the generated program are limited by this external model, and its own biases or limitations may be passed on or even amplified in the evolution.\n•\tThe specific computational cost required for a single iteration of evolution has not been provided. Will this limit its practical application?\n•\tAs mentioned in the limitations of the article, the effectiveness of the method has not been validated in a wider range of domains.\n\nQUESTIONS:\nPlease refer to the paper weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-12"}