Title: ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints
Venue: NeurIPS 2025
Accepted: yes
Keywords: spatial reasoning

multimodal large language model

benchmark
Abstract: Spatial reasoning is a key capability in the field of artificial intelligence, especially crucial in areas such as robotics, computer vision, and natural language understanding. However, evaluating the ability of multimodal large language models (MLLMs) in complex spatial reasoning still faces challenges, particularly in scenarios requiring multi-step reasoning and precise mathematical constraints. This paper introduces ORIGAMISPACE, a new dataset and benchmark designed to evaluate the multi-step spatial reasoning ability and the capacity to handle mathematical constraints of MLLMs through origami tasks. The dataset contains 350 data instances, each comprising a strictly formatted crease pattern (CP diagram), the Compiled Flat Pattern, the complete Folding Process, and the final Folded Shape Image. We propose four evaluation tasks: Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction, and End-to-End CP Code Generation. For the CP code generation task, we design an interactive environment and explore the possibility of using reinforcement learning methods to train MLLMs. Through experiments on existing MLLMs, we initially reveal the strengths and weaknesses of these models in handling complex spatial reasoning tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ORIGAMISPACE, a novel benchmark designed to evaluate the multi-step spatial reasoning capabilities of Multimodal Large Language Models (MLLMs), with a particular focus on their ability to adhere to precise mathematical constraints. The authors argue that existing benchmarks often focus on static scenes or simple spatial transformations, lacking the complexity of sequential, constrained, and dynamic reasoning found in real-world tasks. To address this gap, they propose using the domain of origami, which inherently involves a sequence of dependent folding operations governed by strict geometric rules.\n\nExperiments on a wide range of state-of-the-art open and closed-source MLLMs (including GPT-4o and Gemini 2.5-pro) demonstrate that ORIGAMISPACE is highly challenging. All models perform significantly below human expert levels, and the results highlight the profound difficulty MLLMs have with fine-grained geometric reasoning and constraint satisfaction. The work also shows that interactive learning with compiler feedback dramatically improves performance over standard in-context learning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nSignificance and Originality: The paper identifies a clear and critical weakness in current MLLM evaluation: the lack of benchmarks for multi-step, mathematically-grounded spatial reasoning. The choice of origami as a domain is both highly original and exceptionally well-suited to this problem. It provides a task that is intuitive to understand yet programmatically verifiable, procedurally complex, and rich with implicit and explicit constraints. This is a far more rigorous test of spatial \"imagination\" than what is offered by most existing benchmarks.\n\nQuality of the Benchmark Design: The benchmark is exceptionally well-designed.\n\nThe four tasks are diverse, probing different facets of spatial reasoning from high-level pattern recognition to low-level geometric prediction and generative construction.\n\nThe use of an origami compiler is a major strength. It provides a formal, non-negotiable ground truth for evaluation, which is a significant step up from relying on human consensus or simple metric comparisons. The detailed error feedback system (CSE, GIF, PSI, AFS) is very well-thought-out and enables the novel interactive evaluation setting.\n\nThe evaluation metrics for the CP code generation task (TSS, GS, CS, FFS) are comprehensive, covering topological structure, geometric similarity, constraint satisfaction, and the final folded state. The level of detail in Appendix D is commendable and demonstrates a deep engagement with the problem.\n\nQuality of the Experimental Evaluation: The empirical study is thorough and compelling.\n\nThe authors evaluate a wide and very current set of both open-source and proprietary SOTA MLLMs, making the results highly relevant.\nThe inclusion of both layperson and expert human performance provides essential context and clearly demonstrates the difficulty of the tasks and the significant gap that remains for AI.\n\nThe comparison between in-context learning, environmental learning (with interaction), and reinforcement learning provides a key insight: that iterative refinement based on structured feedback is a powerful and necessary paradigm for solving such complex tasks.\n\nClarity and Reproducibility: The paper is extremely well-written, logically structured, and easy to follow. The extensive appendices are a model of good practice; they provide exhaustive detail on the dataset, annotation rules, compiler errors, and evaluation metrics, which will be invaluable for any researcher wishing to build upon this work.\n\nWeaknesses:\n\nDataset Scale: The primary weakness is the relatively modest scale of the dataset (350 test instances, with an additional 471 used for RL training). While the quality of each instance is clearly very high due to meticulous curation, this size might raise concerns about statistical power and the potential for models to overfit, particularly in the reinforcement learning setting.\n\nScope and Transferability: While origami is an excellent proxy for constrained spatial reasoning, the paper could benefit from a slightly broader discussion on the expected transferability of skills developed on ORIGAMISPACE to other domains like robotics (e.g., cloth folding, assembly), molecular biology (protein folding), or architectural design. This is briefly acknowledged in the limitations but could be expanded upon in the main text to strengthen the paper's impact statement.\n\nQUESTIONS:\nDataset Scale and Robustness: The dataset size of 350 test instances is modest for a benchmark of this nature. Could the authors comment on the statistical significance of the reported performance gaps between models? Have you considered or are you planning methods to scale the dataset, perhaps through procedural generation of simpler origami CPs and their corresponding folded forms, to further improve the benchmark's robustness?\n\nInput to Code Generation Task: For the End-to-End CP Code Generation task (Task 4), the input is described as a \"compiled flat layout and an image of the folded shape.\" Providing the compiled flat layout seems to give the model a significant amount of information that is not typically available in a real-world \"reverse engineering\" scenario. Could you clarify the motivation for this choice? Would it not be a more challenging and perhaps more realistic task to generate the CP code from only the folded shape image, possibly augmented with natural language instructions?\n\nGeneralization of RL Agent: The reinforcement learning results are very promising, showing that an agent can learn to interact with the compiler to improve its output. Could you provide more insight into the generalization capabilities of the trained Qwen-VL-32B agent? For instance, how well does it perform on origami designs of a complexity level or type not seen during its training? Is the learned policy mostly correcting minor syntax errors, or is it demonstrating a deeper, more generalizable understanding of geometric constraints?\nCompiler and Environment Availability: A benchmark's utility is greatly enhanced by its accessibility. Will the optimized origami compiler, the interactive environment, and the full evaluation suite be made publicly available alongside the dataset?\n\nA positive response to these questions, particularly clarifying the design choice in Question 2 and discussing the potential for scaling in Question 1, would further strengthen this already excellent paper.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations of their work in Appendix F. They correctly identify the modest dataset scale, the open question of transferability to other domains, and the potential to expand the suite of evaluation tasks as avenues for future work. This is a thoughtful and honest assessment.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper introduces a spatial reasoning benchmark for MLLMs grounded in origami folding. The benchmark defines four tasks: final shape prediction, multistep spatial reasoning, spatial relationship prediction, and CP Code prediction, which symbolically encodes crease patterns for origami compilation. The authors also introduce four evaluation directions for CP Code quality and use the benchmark to evaluate several foundational models using in-context learning, environment learning, and reinforcement learning.

STRENGTHS AND WEAKNESSES:
Strengths 

The use of origami as a grounding mechanism for spatial reasoning is novel and well-motivated. It provides a mathematically constrained yet visually interpretable setup for multistep reasoning, which avoids the overly synthetic feel of many spatial datasets. The benchmark is thoughtfully constructed with attention to dataset realism and bias reduction. The paper also explores multiple prediction setups (in-context, environment learning, RL), which makes the evaluation broad and informative for understanding MLLM capabilities across paradigms.

Weaknesses

1. The authors are urged to add OpenAI’s recent reasoning-capable multimodal models (o3)  here, given their strong performance on comparable tasks. Even a qualitative comparison would be useful.

2.  Explanations for what each crease type (mountain, valley, etc.) does or what the full syntax rules are for compiling a valid code are unclear This makes the CP Code prediction task hard to interpret. The paper would benefit from a more detailed breakdown or a concrete example of CP Code semantics.

QUESTIONS:
Questions can be found in the weakness section.

LIMITATIONS:
No
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ORIGAMISPACE, a novel benchmark dataset aimed at evaluating the multi-step spatial reasoning capabilities of multimodal large language models (MLLMs), particularly under mathematical and geometric constraints. The benchmark leverages origami-based tasks and consists of 350 instances, each including rich multimodal artifacts such as crease patterns, flat patterns, folding sequences, and final folded shapes. The authors define four tasks: Pattern Prediction, Multi-step Spatial Reasoning, Spatial Relationship Prediction and End-to-End CP Code Generation. They also propose an interactive environment for CP code generation and explore reinforcement learning (RL) techniques to enhance model training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. The use of origami as a testbed is creative and well-motivated. It naturally involves complex spatial transformations, offering a rich ground for evaluating MLLMs beyond typical vision-language tasks.\n2. The four distinct tasks reflect a thoughtful design that targets various dimensions of spatial reasoning.\n3. Mathmatical constraint is a key strength compared to existing loosely defined visual reasoning.\n\nWeaknesses:\n1. The dataset contains only 350 instances, which may limit statistical robustness and hinder the training of larger models.\n2. The mention of RL is promising, but it fails to offer details on the task structure, reward design, or preliminary performance. It is hard to evaluate the feasibility and contributions of RL here.\n3. The mathematical constraints are not well illustrated in the paper. I think it is one of you biggest contributions while the papers fails to convince reader how these constraints affect the overall performance of this problem.\n\nQUESTIONS:\n1. Could you explain further how the mathematical constrains contribution in the proposed multi-step geometric and layering reasoning? \n2. What is the impact of mathematical constrains? During each step, will be possible of precisely satisfying all mathematical constraints? if not, what is the meaning of adding these constrains? \n3. Could you elaborate how RL is set up in the experiments?\n\nLIMITATIONS:\n1. data sets limitations.\n2. Error analysis", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ORIGAMISPACE, a novel benchmark designed to evaluate the complex spatial reasoning capabilities of MLLMs. The authors argue that existing benchmarks fall short in testing multi-step reasoning under precise mathematical constraints. To address this, they use the domain of origami, which naturally involves sequential folding (multi-step reasoning) and adherence to strict geometric principles. The benchmark includes a dataset of 350 origami instances, each with a formal crease pattern (CP), folding process, and final shape. Based on this, they propose four challenging evaluation tasks, including multiple-choice questions and an interactive code generation task. Their evaluation of current MLLMs reveals significant challenges in this domain and demonstrates the potential of interactive and reinforcement learning methods to improve performance.\n\nSTRENGTHS AND WEAKNESSES:\nThe primary strength of this paper is its highly innovative and well-motivated choice of origami as a domain for evaluating spatial reasoning. This is a significant step beyond existing benchmarks. Origami inherently combines multi-step procedural understanding, 2D-to-3D spatial transformation, and strict, verifiable mathematical constraints (e.g., Kawasaki's and Maekawa's theorems), providing a rich and challenging testbed that current models struggle with. The comprehensive design of the benchmark is another major strength; it is not a single task but a suite of four distinct evaluations that probe different facets of spatial intelligence, from holistic shape prediction to fine-grained generative control.\n\nFurthermore, the technical contribution extends beyond a static dataset. The authors have improved an origami compiler and built an interactive environment for the code generation task. This is a crucial element that elevates the work, as it provides a platform not just for evaluation but for actively training and improving models via environmental feedback and reinforcement learning. The initial results from this RL exploration are very promising and open up a new research direction.\n\nWhile the work is excellent, a few minor points could be considered. The dataset size of 350 instances is modest, though this is understandable given the high complexity and manual verification required for each origami model, emphasizing quality over quantity. The direct transferability of skills learned on this benchmark to other real-world spatial reasoning domains like robotic manipulation remains an open question for future research, which is a common characteristic of specialized benchmarks.\n\nQUESTIONS:\n1.  The use of origami is highly innovative. Could you elaborate on how you see the specific skills tested here (e.g., understanding crease patterns, mathematical constraints) transferring to other complex spatial reasoning domains like robotic manipulation or architectural design?\n\n2.  The manual collection and verification process for the 350 models must have been very labor-intensive. Have you considered semi-automated methods for expanding the dataset, perhaps by procedurally generating valid CPs and using the compiler to filter them?\n\n3.  The reinforcement learning results are very promising. Could you provide more insight into the policy the agent learns? For example, does it learn to satisfy certain geometric constraints (like Kawasaki's theorem) first before attempting more complex folds?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}