Visual Compositional Tuning
Authors: Xindi Wu
Hee Seung Hwang
Polina Kirichenko
Esin Tureci
Olga Russakovsky
Venue: ICLR 2026 Poster (2026)
Keywords: Compositionality
Visual instruction tuning
Complexity
Primary area: foundation or frontier models, including LLMs

Abstract: Visual instruction tuning (VIT) datasets have grown rapidly in scale, yet the informativeness of individual training samples has largely been overlooked. Recent dataset selection methods have shown that a small fraction of such datasets enriched with informative samples can lead to efficient finetuning of Multimodal Large Language Models. In this work, we explore the impact of sample complexity on informative data curation and introduce COMPACT (COMPositional Atomic-to-complex Visual Compositional Tuning), a visual compositional tuning data recipe that scales training sample complexity by combining multiple atomic visual capabilities in a single training example. Concretely, we synthesize rich and informative text questions for each image, allowing us to significantly reduce the number of training examples required for effective visual instruction tuning. COMPACT demonstrates superior data efficiency compared to existing data reduction methods. When applied to the LLAVA-665K VIT dataset, COMPACT reduces the data budget by 90% while still achieving 100.2% of the full VIT performance (compared to only 97.5% by the state-of-the-art method) across eight multimodal benchmarks. Further, training on the COMPACT data outperforms training on the full-scale VIT data on particularly complex benchmarks such as MM-Vet (+8.6%) and MMStar (+2.9%). COMPACT offers a scalable and efficient synthetic data generation recipe to improve on vision-language tasks.


Reviews:
[{"review_id": "PmMUJS2uNP", "reviewer": "Reviewer_SC5N", "summary": "This paper proposes COMPACT, where images are from LLaVA-665K, complex instructions are generated by Gemini, to improve data efficiency in multimodal instruction tuning by generating questions that require combinations of atomic visual capabilities. The task complexity is operationalized by the number of atomic capabilities involved (k-value). \nThe paper demonstrates that increasing task complexity leads to better use of visual information and yields impressive performance. Experiment results shows that with only 10% of the LLaVA-665K data, COMPACT matches or exceeds the full dataset’s performance across a variety of multimodal benchmarks. \n\nThis paper presents a practically useful and empirically strong method. The idea of compositional capability tuning is promising and clearly validated by experiments. \nHowever, conceptual and theoretical foundations remain unclear. Several core concepts, such as task complexity, informativeness, information density, and k-value, are used interchangeably without rigorous justification. The mapping from \"number of atomic capabilities\" to \"actual complexity\" is assumed rather than demonstrated. In addition, the capability definitions are hand-picked without theoretical or empirical grounding. The analysis section provides statistics but not mechanism-level explanations. As a result, important questions remain unanswered: Why does COMPACT help? Which capabilities benefit? Why does improvement transfer to tasks outside the covered perceptual abilities? \n\nIf supplemented with theoretical evidence, more in-depth analysis and argumentation, this work has the potential to become a very influential contribution, and I will raise your rating.", "questions": "NA", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. \"Atomic-to-complex visual capability tuning\" is a novel and valuable problem. This paper addresses a real gap in current data curation approaches for MLLM visual instruction tuning. The method is simple yet impactful.\n2. The proposed COMPACT dataset demonstrates significantly higher task complexity and achieves 100% full-data performance using only 10% of the original dataset. The authors validate results on 16 complex multimodal tasks, showing clear improvements over baselines.\n3. This paper have good motivation via dataset analysis. The paper conducts a detailed study on the complexity distribution of existing datasets. Especially Figure 1, which shows overrepresentation of low-k samples, provides compelling motivation for why higher-complexity samples are needed.\n4. The proposed synthetic data recipe is easy to implement and clearly \"works in practice\". The method directly offers a more data-efficient way to perform visual instruction tuning.", "weaknesses": "1. The paper repeatedly uses the terms \"task complexity\", \"informativeness\", \"effective use of information content\", and \"k-value\" as if they were equivalent. Is informativeness = complexity? Is task complexity = number of atomic capabilities? Why is k=3 more “complex” than k=1 in a meaningful sense?\nThe paper does not provide a theoretical justification nor an empirical validation for these assumptions. As a result, the k-value appears arbitrary and not a reliable measure of complexity.\n\n2. The atomic capabilities define only basic perceptual skills, making the notion of \"complex tasks\" overly simplistic. COMPACT’s complexity is defined solely as combinations of perception + attributes + spatial relations. This is a very narrow interpretation of \"complexity\", and does not align with real-world multimodal complexity, which includes OCR, counting, commonsense, math, reasoning, etc.\n\n3. The choice of the categories seems ad-hoc. Table 1 says \n\n> \"We identify 10 atomic capabilities that are necessary for general visual reasoning.\"\n\nLine 189-190:\n\n> \"... but instead provide sufficient coverage of the multimodal task space and to systematically combine tasks of increasing complexities.\"\n \nWe don't understand how you concluded that \"the 10 atomic capabilities provide sufficient coverage of the multimodal task space\". What evidence supports the conclusion? \n\nThe paper does not explain why these are necessary, why others are excluded, and whether the taxonomy is derived from theory, data statistics, or prior work. The capability selection appears subjective and weakens the foundation of the method.\n\n4. No explanation for where the improvement comes from. Benchmarks like MM-Vet and MMStar include tasks requiring OCR, math, logic, world knowledge, far beyond COMPACT’s three perceptual dimensions. Since COMPACT does not train OCR/math/logic abilities, why does it improve these tasks? The paper does not analyze this cross-capability transfer.\n\n5. No breakdown showing whether COMPACT improves perception-only tasks vs. all task categories. Without category-level gains, readers cannot tell whether COMPACT only helps “vision-centric” tasks, or whether simple perceptual compositionality generalizes/transfers to OCR/knowledge/logic. If perceptual complexity generalizes broadly, the claim becomes much stronger. But the paper does not provide the crucial analysis. \n\n6. The paper only provides descriptive correlations, not causal evidence.\nThe explanation is high-level and speculative. Causal claim \"higher complexity $\\rightarrow$ higher information density $\\rightarrow$ better learning\" remains unproven.\n\n7. Training strategy is insufficiently analyzed. The final training mixture is COMPACT + 5% simple LLaVA data. The paper states that both simple and complex samples are necessary, but does not explain why. This likely involves curriculum learning or optimization stability, but the paper provides no analysis or validation.\n\n8. Analysis section is statistics only, not mechanism. The paper largely reports distributions (k-distribution, capability distribution, correlations) but does not explain the mechanism of improvement.\nReaders want to know why this works, not just which ablation performs best.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W05YoSCMau", "reviewer": "Reviewer_Nn4a", "summary": "COMPACT (COMPositional Atomic-to-Complex Visual Capability Tuning) introduces a method for generating complex, information-dense Visual Instruction Tuning (VIT) datasets by combining multiple atomic visual capabilities (e.g., color, spatial reasoning, object recognition) into single training examples. This complexity-aware curation improves data efficiency -- achieving 100.2% of full LLaVA-665K performance using only 10% of the data, with notable gains on complex multimodal benchmarks like MM-Vet and MMStar.", "questions": "- How sensitive are results to the choice of data generator? Have you experimented with other VLMs (e.g., GPT-4V, LLaVA-NeXT)?\n\n- Is there evidence that naturally occurring questions follow a certain k-distribution? How does COMPACT's distribution compare?\n\n- The paper conflates \"compositional complexity\" with \"task complexity\" without clearly distinguishing them. \n\n\nMinor:\n- Figure 3: are you sure its in log scale?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The concept of compositional complexity (k-value) as a controllable metric for VIT data curation is intuitive and well-grounded. The exploratory experiment (Figure 1) effectively demonstrates that increasing k improves performance.\n\n- Strong improvements on compositional tasks with 90 % less data; convincing scaling curves and ablations.\n\n- The paper includes thorough ablations examining complexity ranges, instruction tuning ratios, and complexity distributions. The analysis of LLaVA-665K's complexity distribution (mean k=1.95) provides valuable insights.\n\n- Well-analyzed: Breaks down atomic capabilities, evaluates complexity distributions, and studies instruction-format mixing.", "weaknesses": "- Atomic Capability Definition:\n  - The taxonomy appears somewhat arbitrary (why these 10 capabilities specifically?)\n  - Capabilities are acknowledged as non-orthogonal (Figure 8), undermining the \"atomic\" framing\n  - Object recognition is implicitly assumed in most questions (Figure 7), suggesting the capabilities may not be properly decomposable\n\n- Evaluation scope: Only one base model (LLaVA-v1.5-7B-LoRA); unclear generality to other architectures or scales.\n\n- Error Analysis:\n  - Qualitative examples (Figure 11) cherry-pick favorable cases without systematic error analysis\n  - Insufficent analysis of failure modes or systematic biases in generated data", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uH5FjlFHQb", "reviewer": "Reviewer_Ak1x", "summary": "This paper presents COMPACT (COMPositional Atomic-to-Complex Visual Capability Tuning), a new data recipe for visual instruction tuning (VIT) in multimodal large language models (MLLMs). COMPACT introduces the idea of compositional complexity, where each training sample is constructed by combining multiple atomic visual capabilities (e.g., object recognition, spatial reasoning, color, shape). By controlling the number of combined capabilities (“k-value”), the method generates more information-dense and complex questions using Gemini-2.0-Flash, leading to significant data efficiency gains. With only 10% of LLaVA-665K data, COMPACT achieves 100.2% of the full-scale performance across benchmarks such as MM-Vet and MMStar, highlighting the benefit of complexity-aware data curation for MLLMs.", "questions": "The \"natural integration\" requirement for multi-capability questions is somewhat subjective and relies on the LLM's interpretation\n\nZero-capability samples (0.9% of LLaVA-665K) are interesting but receive minimal discussion", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Conceptually innovative: Introduces a clear and quantifiable notion of compositional complexity in VIT data, shifting focus from scale to information density.\n\n- Strong empirical evidence: Matches or exceeds full-data performance with one-tenth of the samples, demonstrating outstanding data efficiency.\n\n- Thorough evaluation: Comprehensive experiments across major multimodal benchmarks and detailed ablations on complexity levels (k-values) and instruction-tuning ratios.\n\n- The paper is well-organized, with transparent methodology, taxonomy of atomic capabilities, and plans to release the dataset.", "weaknesses": "- Dependency on proprietary models: The reliance on Gemini-2.0-Flash for both question generation and verification is a significant limitation for reproducibility and may introduce unknown biases. The cost ($86.5 for 32K samples) could also be prohibitive for scaling to larger datasets. An analysis using open-source alternatives would strengthen the work.\n\n- While the 10 atomic capabilities are well-defined, the paper acknowledges they are \"not expected to be completely orthogonal\" but provides limited justification for this specific set. The correlation analysis (Fig. 8) suggests substantial dependencies, yet the implications for the k-value metric are not fully explored. How does correlation between capabilities affect the actual complexity?\n\n- Limited scope of evaluation: The focus is exclusively on vision-centric tasks. The poor performance on knowledge-intensive benchmarks (Table 9) suggests the approach may not generalize to domains requiring external knowledge or reasoning beyond perceptual capabilities. This limits the claim of addressing \"general visual reasoning.\"\n\n\n- Verification process clarity: The quality verification step (Step 3) uses confidence thresholds and word overlap metrics, but the paper provides limited analysis of failure modes or how often verification rejects generated samples. More transparency about the quality control process would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UZYbb3OMTj", "reviewer": "Reviewer_MPVR", "summary": "The main motivation is to compress multiple capabilities into a smaller number of data samples to increase sample efficiency, doing more with less data sets that compose multiple atomic capabilities into one.", "questions": "I think this make sense, but the random baseline is very honest and seem to also suggest that using only 49k out of 665K pretty similar to the COMPACT setup, realistically, SFT is pretty light weight\n\nI would think about improving this work via framing as improving existing answer quality than just data effiency, like many atomic and subjective task here could be used to double check the quality of answers, see if they are correct, or use them in capability-specific abiliations to try to see what task are driving most gains. I would be surprised if color (which seem relatively easy) drive much of the gain", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "there are clear taxonomy of capabilities \nthere are indeed clear gains over the llava-665k datasets where it was not principally constructed.", "weaknesses": "see the question", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sxCltr3Oqs", "reviewer": "Reviewer_jUJz", "summary": "This paper focuses on the curation of informative training data to enhance MLLMs’ finetuning efficiency. It introduces COMPACT, a novel data synthesis approach that generates rich and informative text questions for each image by integrating multiple atomic visual capabilities into a single training sample. Experimental results across various benchmarks demonstrate that COMPACT significantly reduce the required number of training examples while achieving performance comparable to that of full-scale training data, highlighting its efficiency.", "questions": "1. Although the number of required training data decreases, does the incorporation of multiple atomic questions in a single COMPACT question imply that the token count for inputs and outputs hasn't reduced such significantly? \n2. In Figure 3, why do models trained on random data sometimes outperform those trained on the full dataset? Additionally, why does the notably poor performance of COMPACT on the TextVQA dataset?\t\n3. What is the task complexity of the evaluation benchmarks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1. This work defines atomic capabilities essential for general visual reasoning and introduces the k-value metric to quantify task complexity.\n2. The proposed COMPACT scales training sample complexity by incorporating multiple atomic visual capabilities within a single data, revealing that increased complexity enhances information utilization.\n3. Experiments show the effectiveness of COMPACT, which achieves comparable or even superior performance with just 10% of the training data, improving finetunig efficiency for MLLMs.", "weaknesses": "1. The entire generation, verification, and evaluation process relies on the closed-source Gemini model, which may introduce potential bias and limit reproducibility.\n2. The exploration is confined to the LLaVA-v1.5-7B-LoRA model and the LLaVA-665K VIT dataset, leaving the performance of COMPACT with other models and training datasets underexplored, especially considering that the LLaVA-665K dataset exhibits relatively low task complexity.\n3. There is a lack of comparison with other data reduction methods in experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fGxZ6BdeHW", "reviewer": "Reviewer_YsZe", "summary": "This paper presents COMPACT, a data-efficient visual instruction tuning (VIT) framework that synthesizes training samples with controlled compositional complexity. The authors introduce the k-value, representing the number of atomic visual capabilities (e.g., object recognition, spatial reasoning) required to answer a question. By generating high-k samples using Gemini-2.0-Flash and combining them with a small subset of LLaVA-665K for instruction formatting, COMPACT attains 100.2% of the full-dataset performance using only 10% of the data. It substantially outperforms baselines on complex benchmarks such as MM-Vet (+8.6%) and MMStar (+2.9%).\n\nKey contributions:\n1. A complexity-aware VIT data recipe leveraging atomic capability composition.\n2. Empirical evidence that higher-k samples enhance data efficiency.\n3. A scalable synthetic data generation framework that reduces dependence on large-scale datasets.", "questions": "1. How might COMPACT perform if atomic capabilities are expanded to include non-perceptual skills (e.g., commonsense reasoning)? Could this address the modest gains on knowledge-intensive tasks?\n2. Have you explored generating data with open-source VLMs to assess reproducibility and reduce reliance on Gemini?\n3. Does the benefits of high-ksamples diminish beyond k=3? Is there an optimal complexity ceiling for efficient learning?\n4. How does COMPACT handle images where certain atomic capabilities are inherently hard to combine?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality:\nCOMPACT introduces a novel lens for understanding and curating VIT data by framing compositional complexity through the k-value, which quantifies the number of atomic visual capabilities (e.g., object recognition, spatial reasoning, counting) required to solve a task. This provides a principled and measurable axis for dataset construction, enabling systematic control over the difficulty and diversity of visual-instruction samples—an aspect largely overlooked in prior work.\n\n2. Quality:\nThe study demonstrates strong methodological rigor, featuring comprehensive evaluations across eight multimodal benchmarks and detailed ablation analyses. These experiments confirm the method’s robustness, showing consistent improvements in both generalization and compositional reasoning while maintaining competitive performance under limited data budgets.\n\n3. Clarity:\nThe paper presents a well-structured taxonomy of atomic visual capabilities and clearly articulates the data synthesis pipeline, from capability composition to instruction formatting. The process is transparent, reproducible, and amenable to scaling, offering valuable guidance for future research in efficient multimodal data generation.\n\n4. Significance:\nBy achieving superior performance with only 32K synthetic samples, surpassing models trained on over 665K human-annotated examples, COMPACT directly challenges the prevailing assumption that larger datasets are always necessary for stronger multimodal performance. This redefines the data–efficiency frontier and highlights compositional control as a promising new direction for scaling visual instruction tuning.", "weaknesses": "1. ​​Dependency on closed-source models:​​ Data synthesis via Gemini-2.0-Flash risks reproducibility and may inherit model biases. Experiments with open-source generators would strengthen generalizability.\n2. ​​Limited scope of atomic capabilities:​​ Non-visual skills (e.g., knowledge, math) are excluded, limiting gains on benchmarks like OK-VQA. Expanding the taxonomy could broaden applicability.\n3. ​​Evaluation of compositional generalization:​​ While high-ksamples improve performance, tests for zero-shot compositionality are lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)