Autonomous Imagination: Closed-Loop Decomposition of Visual-to-Textual Conversion in Visual Reasoning for Multimodal Large Language Models
Authors: Jingming Liu
Yumeng Li
Boyuan Xiao
Yichang Jian
Ziang Qin
Tianjia Shao
Yao-Xiang Ding
Kun Zhou
Venue: Accepted by TMLR ()

Abstract: Under pure textual modality, Large Language Models (LLMs) have demonstrated remarkable success in complex reasoning tasks by decomposing them into simpler sub-problems. However, Multimodal Large Language Models (MLLMs) still struggle with some seemingly straightforward visual tasks, such as counting and solving jigsaw puzzles. We argue that these tasks challenge the ability of {\it visual-to-textual conversion}, where MLLMs convert visual information perceived from the input scene, to textual information for further reasoning and generating the answer. If the complexity of the visual input is beyond the perceptual capability of the MLLMs, without decomposing this conversion process, simply scaling inference-time reasoning cannot solve the task because it repeatedly encounters the same perceptual bottleneck. We propose an approach, {\it autonomous imagination}, to enable MLLMs to iteratively modify visual inputs (e.g. isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closed-loop visual modification steps. We show that, without any retraining, MLLMs can now solve tasks initially beyond their perceptual capability, highlighting that closed-loop visual modification can be an effective way of decomposing the visual reasoning task into solvable substeps. Our code and data are released at https://future-item.github.io/autoimagine-site/.

Yumeng Li Published in Transactions on Machine Learning Research (09/2025)
Autonomous Imagination: Closed-Loop Decomposition of Visual-to- Texttull modal Large Language Models
Textual Conversion in Visual Reasoning for Multi-
Jingming Liu'
jmd7514577@mmiil.co com
State Key Laboratory of CADECG, Zhejiang University
yumeng.li@eju..duu. edw.cccee.u.crr
State Key Laboratory of CADECG, Zhejiang University
Boyuan Xiao State Key Loboratory of CADECG, Zhejiang University
ziaohy202@@gmaiilcoo com
Yichang Jian
1tdickens1198880maall
State Key Laboratory of CADECG, Zhejiang University
Ziang Qin
ginzzangg1993377@mmail.omm
State Key Laboratory of CAD8CG, Zhejiang University
Tianjia Shao
@isaa@@jjueduu
State Key Laboralory of CADECG, Zhejiang Universily
Yao-Xiang Ding'
Singya.gm@@gmace com
Stale Key Laboralory of CAD&CG, Zhejiang Universily
Kun Zhou
bunshou@acm.cr
State Key Loboratory of CADECG, Zhejiang University
Reviewed on OpenReview: https: // openreview. net/ permm II
Abstract
Under pure textual modality, Large Language Models (LLMs) have demonstrated remark- able success in complex reasoning tasks by decomposing them into simpler sub-problems. However, Multimodal Large Language Models (MLLMs) still struggle with some seem- ingly straightforward visual tasks, such as counting and solving jigsaw puzzles. We argue that these tasks challenge the ability of wisual-to-tertual conversion, where MLLMs con- vert visual information perceived from the input scene, to textual information for further reasoning and generating the answer. If the complexity of the visual input is beyond the perceptual capability of the MLLMs, without decomposing this conversion process, simply scaling infrencc-time reasoning cannot solve the task because it repcatedly encounters the same percept ual bottleneck. We propose an approach, aulonomous imagination, to enable MLLMs to iteratively modify visual inputs (e.g. isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closcd-loop sual modification steps. We show that, without any retraining, MLLMs can now solve tasks initially beyond their perceptual capability, highlighting that closed-loop visual modification can be an effective way of decomposing the visual reasoning task into solvable substeps. Our code and data are released at https: htttp:://fuui: /tte:://fuurreieee g1 github. 10/ autoimagine- sitgginneesste///ee
"Equal contribution. Corresponding author.

---

Published in Transactions on Machine Learning Research (09/2025)
40
GPT-4o with Autonomous Imagination
Spaiio
Resetiing
X
Cestingfe Irapiaato qUAck
Ldboat aoutt eecing Astr
Jaaen Spsrnt des ast when
O The
Grahhig veagration IEION
bodked apple sovag right
and bhobl chouedal MSD 1891
Figure 1: Our autonomous imagination approach empowers advanced MILMs to engage in iterative imagi- native reasoning, enabling them to address previously unsolvable tasks without additional training.
1 Introduction
Modern Large Language Models (LLMs) demonstrate exceptional reasoning capabilities through systematic task decomposition (Huang & Chang 2023 By breaking complex problems into sequential subtasks, models
achieve remarkable performance in textual reasoning domains (Yao et al. 2023
OpenAT 2024b). More
recently, Multimodal Large Language Models (MLLMs) (OpenAl 2024a
Liu el al.
2023
Bai et al. 2023
Wang et al. 2023a are gaining increasingly strong abilitics in visual understanding. Inspired by work on
visual prompting (Jiang et al. 2024 Lin et al.) 2024 Hong et al. 2023 Zhang et al. 2023b с Mitra et al.
2023
Zheng al. 2023 Zhang et al., 20244] Yu al. 2024), recent methods enable MLLMs to perform
visual reasoning by proactively adding visual prompts (such bounding boxes) to the image, to reduce
hallucination and handle challenging visual tasks without human intervention (Lai ct al. 2023 Wu & Xie
2023
Chen et al. 2023 Qi et al. 2024 Shao el. al. 2024 Zhou el al. 2024b).
Despite these advancements, existing methods often struggle to make MLLMs natively handle intuitive visual reasoning tasks. When challenged with tasks such as jigsaw puzzle assembly (even when provided with explicit step-by-step instructions), or simply counting objects, stale-of-the-art MLLMs consistently fail reason through the entire process. A key observation is that, instead of challenging high-level reasoning abilitics such as logical deduction, these tasks actually challenge the ability to perceive and understand complex visual scenes. Referring to the failure of existing MLLMs them, argue that ineffective task decomposition remains fundamental cause. Reasoning in complex visual input requires the strong ability of visual-to-tertual conversion, where MLLMs convert visual information perceived from the input scene to textual information for further reasoning and generating the answer. However, in current MLLM reasoning, the visual-to-textual conversion process remains an atomic operation rather than being divided into executable substeps. When the complexity of the visual scene beyond the perceptual ability of MLLMs, without decomposing this conversion step, simply scaling inference-time reasoning cannot solve the task because it repeatedly encounters the same perceptual bottleneck.
We hypothesize that, similar to textual reasoning, where intermediate textual states are necessary, MLLMs need to generate intermediate images that gradually simplify the original scene or approach the target visual state. To test this, we propose enabling MLLMs to iteratively modify visual inputs (e.g., isolating objects, rearranging puzzle pieces) into intermediate visual states, decomposing visual-to-textual conversion into closed-loop visual modification steps, without any additional training of MLLMs. For simplicity, we refer to this approach as aulonomous imagination. Technically, implement the approach to support the
2

---

Published in Transactions on Machine Learning Research (09/2025)
conversion from unstructured input visual scene into structured representation and the visual operators that efficiently modily the visual content into new states (e.g,, through transformation or erasing). This allows MLLMs to generate successive visual states that progressively simplify the original visual scene or gradually approach the target visual state in an autonomous fashion.
We conduct experiments under visual reasoning tasks that challenge the perception and understanding abilities of MLLMs under complex visual scenes, including counting, jigsaw puzzle solving, object placement, and multi-object hallucination. The results show that previously challenging visual reasoning tasks, which are beyond the perceptual capability of the MLLMs, can be effectively tackled natively by them equipped with our approach. This serves as empirical validation that a major limitation in visual reasoning stems from insufficient support for visual task decomposition, which we hope will inspire future research.
2 Related Work
2.1 Reasoning in LLMs
Numerous studies explore reasoning paradigms in natural language using their in-context learning abil- ity (Brown et al., 2020), showing that Large Language Models (LLMs) improve through reasoning-base based Kojima et al.
outputs (Wei et al. 2022
2022). Subsequent work further improves reasoning through in-
context learning by improving in-context sample selection (Rubin et al.) 2021 Lu et al. 2022, Zhang et al.
2022 Fu et al. 2022 Wang et al. 2022 Li et al. 2022b). Recently, notable advancements in OpenAl's ol [OpenAl 2024b) have demonstrated that by scaling LLMs for inference- time reasoning before answering, LLMs can be greatly enhanced in reasoning. A key insight from these advances is that proper decomposition of the reasoning process into simpler subtasks is essential for successful reasoning (Wei et al. 2022).
2.2 Visual Reasoning in MLLMs
MLLMs have evolved in visual understanding ability, initially leveraging domain-specific expert models such as HuggingGPT (Shen et al. 2023) MM-REACT (Yang et al. 2023). and VisualChatGPT (Wu et al. 2023
The focus later shifted to training LLMs with an adapter for the other modal, such as LLaVA (Liu et al. 2023). BLIP-2 (Li et al., 2022a 2023), and MoVA (Zong et al., 2024). Many recent MLLMs now exhibit native visual understanding through training in text-image pairs, with fine-tuning on
tasks [Lu et al. 2023 Bai et al.) 2023 Wang et al.) 2023a Zhu et al.] 2023 Chen et al. 2022 Luo et al.
2023
Alayrac et al.] 2022). including the state-of-the-art closed-source MLLM GPT-4o (OpenAl 2024a).
In addition to reasoning within text modalities, substantial efforts are made in reasoning in visual tasks. Existing MLLMs often face limitations in direct visual perception and are prone to generating answers with hallucinations (Zhang et al.} 2023d). Initial approaches used atlention mechanisms to improve question- answering abilities (Zhou et al. 2202), and subsequent solutions include strengthening reasoning in the visual modality by training (Lin et al. 2023 Wang al.) (023b), employing auxiliary knowledge (Mitra)
et al.
2023), and utilizing external visual perception modules Surís et al. (2023). In addition, researchers
have
proposed effective visual prompting methods to refine the focus of MLL.Ms (Jiang et al. 2024 Lin et al.
2024
Hong et al.] [0233 Zhang et al.) 2023b с Mitra et al. 2023 Zheng et al. 2023 Zhang et al. 2024 Yu
et al.
2024). See Wu et al.
(2024a) for the recent comprehensive survey. Attempts have been made to utilize
visual
prompts in the Chain of Thought (CoT) paradigm (Wei et al.
2022
Based on pioneer studies (Zhang
ct al.
2023d [Zheng et al.
2023 Zhang et al. 2023a Peng et al.,
2023
recent approaches build CoT by
enabling MLLMs
add visual prompts autonomously either through direct model training (Lai et al,
2023
Wu & Xie, 2023 Chen et al. 2023 Qi et al. 2024 Shao et al. 2024 or by utilizing external visual
processing models (Zhou et al. 2024b). By using visual prompts (e.g., bounding boxes) to highlight key information in images, the visual-to-textual conversion step becomes casier, leading to a performance boost. However, even though visual prompts simplify the perception of input images, this paradigm still restricts to do -isual-to-textual conversion only in one step, which is after all visual prompts are added in the input. As our experiments demonstrate, even when visual prompting applicd, visual-to-textual conversion remains a bottleneck in common visual reasoning tasks that are intuitive to humans, highlighting the necessity of effective task decomposition.
3

---

Published in Transactions on Machine Learning Research (09/2025)
We also discuss some recent work action planning, which addresses significantly different task from ours, while has also explored various imagination techniques, such using video generation models to simulate control processes as references (Ajay et al. 2023 Du et al. 2023), employing image generation to visualize target goals (Zhou et al., 2024a),, and improving an LLM's 's understanding of the current state, either through textual descriptions or visualizations (Liu et al. 2022 Wu et al.] 2024bb. However, these approaches are limited to closed-world scttings where the state and action spaces are predefined. Although these techniques have shown promising results in closed environments, they are designed specifically for constrained settings and cannot be directly extended to open-world contexts, where the visual scene is primitive and unstructured.
Recently, advanced image editing models that accept input of natural languages are proposed (Huang et al. 2024 Wang et a 2024). Notably, concurrent work has enables image editing models with strong reasoning abilities (Fang et al.) 2025). These models have the potential to be utilized as the imagination tools in our approach, while are more capable to deal with visual generation and editing tasks by themselves.
The closest studies to ours are from the field of robotics, where recent methods utilize the ability of MLLMs to perform object manipulation (Kapelyukh et al.) 2024 Ding et al. 2024). These methods also create a virtual space and use MLLMs evaluators to judge whether the object's ' final state matches the instruction, in order to generate a final state where the object should be moved to. However, current approaches adopt random sampling to conduct exhaustive searches in the virtual environment, which is not feasible when the possible state space large, We implement this method in the experiments to verify this argument.
3 Method
3.1 Problem Formulation
We challenge MLLMs with visual reasoning tasks that are intuitive for humans: counting, jigsaw puzzle solving, object placement, and mulli-object hallucination. For object placement, to capture a holistic top- down view, reconstruct the imput scene using 3D Gaussian Splatting (Kerbl et al. 2023) and render it from a top-down perspective. We directly feed the unstructured raw input scene into the model: no semantic pre-processing (e.g., segmentation or grounding) is applied initially, aligning with real-world settings where visual inputs are provided directly. We evaluate whether MLLMs can natively solve these tasks by proactively modifying the visual state on their own, with minimal reliance on external semantic visual understanding abilitics (c.g., we avoid using object detection models for counting, which would constitute "chcating").
3.2 Closed-Loop Reasoning Formulation
To enable existing MLLMs to proactively modily the visual scene during reasoning, we propose a plug-and- play virtual space. In this framework, the MLLM is treated as a pre-trained policy that executes a sequence of operations. We refer to this space as the imagination space for simplicity. The MLLM can move cursor to focus on an object, perform an operation on it (e.g., ignore transform it), and then go back to use the cursor again until convergence. The imagination space continuously updates the MLLM with the latest visual state by dynamically re-rendering the scene.
Note that our only external model dependency is Segment Anything 2 (SAM2) Ravi et al. 2024 which is invoked solely to isolate a visual element after the MLLMM selects via the cursor. Since our focus on testing visual perception rather than high-level logical reasoning, we directly provide the MILM with task- specific reasoning plans (e.g., "count by ignoring balls") through system prompts. This mitigates occasional failures caused by the MLLM adopting bad reasoning strategies.
Specifically, we assume that the MLLM is given an input image and input text prompt token rp, the aim of the reasoning is to obtain the prediction ion probability P(ylo, rp), where y is the final output text token. In general, to utilize CoT, the reasoning problem of predicting P(y o, ro) can be transformed into intermediate steps of predicting Poz 2,-1,, 1, ...,. T. Each 2, = {rr, 6, denotes the augmented textual prompts r, and visual information C for deepening scene understanding and pushing forward reasoning. After the final reasoning step at. T, the final output. is set to y = rr. The target transforms into predicting the following
4

---

Published in Transactions on Machine Learning Research (09/2023)
User Input
Reasoning
Output
Imagltatonn
many there in this imago?
Perceive
Lat she stregin alspsdypaleg (shoulo
Fabcuald count by groorig bable. Let's star from She and and wiite ono.
Falmad
stop
To frour on the red and whit bak need to move me Oursor sp mit perfomn Transfornlly oporation.
Groate Imagination
Condlude
Space
Awindin DiSMeT
Focux
Control
An overview of our autonomous imagination approach: The imagination space begins with unstructured input scene and undergoes an iterative reasoning process. In cach cycle, MLLMs first pereeive the current state of the imagination space, select an operation to apply, and then reassess the updated imagination space. Upon completing this reasoning sequence, MLLMs generate an answer based on the cunulative context of the process and the final state of the imagination space.
Figure 2:
joint probability:
T P(z z0a-1);
P(Zi.To,,ro)
II 1=1
(1)
where ztiita = (2r, zt ,,, and 20 = ro- What is essential in CoT design to make each step simple }
enough to match the reasoning capacity MLLMs.
In imagination space, are images rendered from the space for MLLMs perceive. We also introduce a set of operators to modify the imagined scenes into new ones. Denote by i; (ōr, and as the operator in step t. Our method transforms Eq. into
II
P ax/eaa))))) t=1
(2)
where fo, and finally rT. The one-step reasoning task P(U,, 1rl is factorized into
P(e,, a (1-))) = (44/:=/)00000000-11.00
ro=-1).
(3)
Eq. 3 shows that one-step reasoning is factorized into decision function T(a 1t-1). a scene modification function @60,)0,-1, a,), and reasoning function ((r:@0r,rrr-1)) Given the current imagined scene 6,-1 and the augmented text prompt token rz-1, the docision function chooses a specific scene modification operator de. The scene modification function then utilizes d, to update ô,-1 into å, The reasoning function finally updates Tt-1 into Tt based on ô, The decision and reasoning functions are purely based on the native reasoning ability of MLLMs. The scene modification function implemented inside the imagination space.
Comparing Eq. and Eq. 2 the imagination space transforms CoT reasoning into a closed-loop decision- making and reasoning process. During the reasoning process, the visual state is gradually simplified or approaching a target visual state: the reasoning step at step is only dependent on ô,-1, 6, but independent of all previous visual states, leads to effective task decomposition.
3.3 Imagination Space
The imagination space is designed to render images for MLLMs to perceive and supports a minimal set of operators for MLLMs to call: focus, ignore, and transform. Focus isolates relevant content for further manip- ulation, ignore enables MLLMs to disregard extrancous information, and transform allows the repositioning of desired content. This compact of operations enables MLLMs to reason and solve practical challenges themselves, demonstrated in experiments. MLLMss select operators through natural language
5

---

Published in Transactions on Machine Learning Research (09/2025)
output, which are then applied to the imagination space. The updated space is rendered and returned to MT.LMs for the subsequent reasoning step.
We discuss the detailed implementation of the operators in the following sections. The operators are imple- mented separately for the 2D space (represented as images) and the 3D space (represented via 3D Gaussians). MLLLMs are unaware of this implementation difference, as interacts exclusively with rendered images as input and executes operators identically in both scenarios.
3.4 Focus Operator
The focus operator allows MLLMs isolate and label target content from scene, creating distinct elements for further transformation. Once MLLLS identify an object of interest using a virtual cur- sor (as described in Sec. 3.7), the focus operator segments this content for focused manipulation. In
Transform
Focus
Ignore
2D space, this cursor directly conditions the Seg- ment Anything Model (SAM) (Kirillov et al., 2023 to perform segmentation. MLLMs verify segmented output to ensure alignment with their intended fo-
Transform Virtual Cursor
cus. In the 3D Gaussian space, segmentation on demand poses unique challenges: Existing meth- ods (Ye et al.) 2024 Shen et al. 2024) are designed for unconditional segmentation, where all objects are segmented without the ability specify con- ditions. However, our case requires conditional segmentation, focusing specific object selected by an input condition, making these methods un- suitable.
Transform Focused Element
Elamant Segmerted
Element Ignorad
Figure 3: Illustrations of operations within our imag- ination space: transformations can be applied fo- cused elements (including the virtual cursor), focus operations allow segmentation of cursor-sclected ele- ments, and ignore operations make cursor-selected el- ements visually invisible.
To enable the focus operator in 3D Gaussian-based representations, we introduce a method selec-
tively segment an object from an unstructured 3D
Gaussian scend] which is illustrated in Alg. 1 Given the MLLM's initial selection block, first generate virtual camera trajectory that orbits around the targeted object, creating a video sequence. This sequence is fed into the SAM2 model (Ravi et al. 2024), which applies the selection cursor condition on all rendered frames. For each frame, rays are cast from pixels within the segmentation mask, and path tracing records the radiance contributions of intersecting Gaussians. This process is expressed as volumetric integration illustrated Line 4-11. Gaussians who receive contribution higher than a threshold will receive one vote.
We select the 3D Gaussian with votes exceeding 10% of the highest vote, forming a shell around the target object's radiance field. This shell, along with the internal radiance Gaussians, is segmented collectively to define the object's 's radiance structure. Tn both 2D and 3D spaces, the segmented object is placed in separate layer, while the remaining content is retained in a base layer.
3.5 Ignore Operator
The ignore operation removes a focused object from the imagination space to prevent interference with the reasoning process. Since API access to state-of-the-art MLLMs does not support attention masks, removing the object leaves a hole the base layer, potentially causing hallucinations. To address this, we simply inpaint the hole created by the removal of the object. For both 2D and 3D imagination spaces, we project the object mask provided by OpenCV [Itseez, 2015) by simply merging the nearby pixels. The inpainted image is then used
the rendered image the region to inpaint. We directly use the implementation
We refer the detailed introduction 3D conditional segmentation algorithm our open-released code.

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm 1: 3D conditional segmentation
Input : SAM2 mask, 3D scene, thresholds 61, 62
Output: Set of marked Gaussians Maarked
Initialize vote counts: votes[g:] O for cach Gaussian 9i in the scene
foreach pizel p in masked pizels do
ray + cast past_ray(p)) ; +-1
forcach Gaussian 9s intersected by ray do
5
ag + get (a. C2 + T a,
if C; 6 then
votes|g:] + votes[gi] +1 end
T
10
if T < then break
end
12 end
max vote maximum of votes|g] over all gi
14 Gmarked +-8 0
15
foreach Gaussian do
16
if votes/g./ 2 0.1x mar mar_ then
17
Imarked f- Imarked U {9;}
18
end
19 end
for further reasoning by the MLLMs, avoiding paying attention to the inpainted region the content has been removed.
3.6 Transform Operator
Object transformation can oceur in four directions: up, down, left, and right, each represented by an alphabet character (a, b, d) to avoid interference with semantically meaningful words. In each step of the iterative imagination process, MLLMs select direction for movement. Since MLLMs struggle with determining precise distances, slandardize movement units in screen space coordinates and gradually reduce step size throughout the process, where MLLMs only control the direction of movement.
3.7 Reasoning Process
The iterative reasoning process described in Sec. 3.2 executed as follows: After initializing the imagination space, virtual cursor is positioned at the center and automatically considered focused. MLLMs perform transform operations on the cursor to reposition it within the space. Following each operation, MLLMs conduct scene modification and receive an updated image of the newly rendered imagination space.
When MLLMs identify that the cursor has selected a visual clement of interest, a focus operation is performed based on the location of the cursor. MLLMs determine that this operation has correctly segmented the intended element, the focus is shifted that object, enabling MLLMs execute either transformation ignore operation it. Once MLLMSs have either repositioned disregarded the object, the focus returns to the virtual cursor. This process iterates continuously as part of the reasoning workflow until all necessary reasoning steps are completed, culminating in final lextual response.
3.8 System Prompts
Our approach is training-free free and utilizes the native reasoning ability of the MLLMs autonomously utilize the visual imagination operators provided. We design set of general system prompts across all tasks. The

---

Published in Transactions on Machine Learning Research (09/2025)
reasoning instructions given to the MLLMs in our tasks are shown in Fig. 15 for autonomous imagination operations are illustrated in Fig. 16
The prompts used as reminders
4 Experiments
In the experiments, we challenge MLLMs with four tasks that are intuitive to humans and require strong visual understanding and reasoning abilitios: counting, simple jigsaw puzzle solving, object placement, and multi-object hallucination. We larget at verifying that compared to 1) simply scaling textual reasoning steps 2) conducting visual prompting, while still conducting visual-to-textual conversion within a single step, the closed-loop task decomposition paradigm is indeed more effective and necessary.
4.1 Benchmark and Evaluation Protocol
Counting: When faced with numerous densely packed objects, humans often rely on multi-step reasoning to reach an accurate count, as a single glance may not suffice. Existing MLLMs, however, struggle with counting, as their multistep reasoning not yet effective for this task. We include this task to assess whether the model can leverage reasoning to offset its limited direct perception, mirroring human strategies.
For evaluation, directly compare the predicted count of the model with the ground truth. In addition to reporting the success rate, we calculate the mean and variance of the counting errors to provide insight into the accuracy and consistency of each approach. We constructed 122 images with real objects with paired ground truth.
Solving simple jigsaw puzzles: Jigsaw puzzles are classic tests of visual perception and reasoning, com- monly used to assess intelligence. In this task, we evaluate the ability of MLLMs to solve simple jigsaw puzzles, aligning their problem-solving performance with that of humans to assess visual reasoning capabil- ities. In the jigsaw puzzle solving task, MLLMs are tasked with identifying and placing missing pieces in their correct locations.
For evaluation, we use a digital jigsaw puzzle game with a magnetic mechanism, where pieces automatically snap into place when positioned close to their correct locations. Success is measured by the completion rate, defined as the percentage of pieces placed their correct locations. For evaluation, we constructed 11 cases where four pieces are missing and 11 cases where six pieces are missing. The size of the puzzle ranges from 3 x 5 to x 8.
Object placement: Previous methods have shown progress in enabling MLLMs to understand and describe static scenes, such as identifying object locations (e.g., "Where is the cup?"). However, for practical use, MLLMs must also interpret dynamic instructions that convey intent, such as "Where should the cup be placed?" In the object placement lask, MLLMs are required to identify both the current and target locations of objects based on abstract. instructions (e.g., "Prepare two cups for guests in the living room"). Given the input scene of 3D Gaussians, MLLMs must determine the original locations of objects and their intended locations based on provided prompt.
For evaluation, we first measure the locating success rate, which reflects the model's ability to accurately identify the initial location of the correct object. If the model fails to do this, the placeeent task is au- tomatically marked as a failure. We then measure the placement success rate by checking whether the predicted final location falls into the marked ground-truth region. For both success rates, when multiple correct solutions exist, we provide multiple ground truth regions and selecting any valid region is considered correct. We captured a total of 17 scenes, including 271 user prompts and paired ground truth for evaluation.
Multi-object hallucination: Current MLLMs can suffer from hallucination by perceiving generating non-existing objects in the input seene. The complexity of the input seene is beyond the limitation in perceptual capability of MLLMs, which is a plausible reason for hallucination. Our approach can serve as a promising way to address this challenge, especially for the scenes where multiple objects exist. By the closed-loop iterative process of autonomous imagination, MLLMs can deal with objects one by one, as in our counting benchmark, instead of handling them all together, effectively reducing the possibility of hallucination. We conducted additional experiments the recently proposed multi-object hallueination
8

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Quantitative comparison results under counting, jigsaw puzzle solving, and object placement. See Sec. 4.1 for details of the metrics. We consider two variants of cursor-only in counting, loading to two sets of results, and GPT-4o Sampling cannot be applied to counting and locating as illustrated, please see Sec. 1.3
method in locating, so their results are identical. Furthermore, NA* indicates that under these tasks, Molmo constantly fails to output reasonable coordinates or refuses to output.
4.2 and Sec. 4.5 for details. "In object placement, cursor-only functions the same
Settings
GPT-44
Molmo
VCoT
Ours (cursor-only
GPT-40 Sampling
GPT40+Ouus
Success Rate
39.8%
50.8%
86.2%
15.5%
39.0%%/0%%
85.3%
Counting
Mean Error
0.73
0.74
0.34
3.11
1.12/not noo calculable
0.19
Variance
0.90
2.57
5.51
1.22/ not calculable
0.22
4-Piece Misssing
29.5%
31.8%
NA*
9.1%
27.3%
43.2% 30.3%
68.2%
Simple Jigsaw Puzzle
Piece Missing
9.1%
27.3%
NA*
3.3%
24.2%
51.5%
Locating
10.9%
17.3%
NA
10.4%
69.4%%
69.4%
Object Placement
Placement
3.6%
8.5%
NA*
1.5%
27.8%
17.3%
37.3%
benchmark ROPE (Chen el al. 2024a. to validate this argument. We focus on the most challenging subset in the benchmark where the state-of ofettheart model on the official leaderboard, GPT-4o, exhibits the poorest performance, which is answering about single object given multiobject image, heterogeneous object types, on unseen data split. We refer to the detailed benchmark setting from the original paper.
4.2 Baselines
We evaluate our approach against state-of-tho-art MLLM, namely GPT-4o (OpenAT 2024a) (our approach is also utilized to enhance GPT-4o in all experiments). We provide a clearly defined 2D coordinate system to GPT-4o so that output coordinates are generated without any ambiguity. We also adopt the recently proposed visual reasoning method VCoT (Shao et al. 2024) using their open-released pre-trained model. We developed bascline GPT-4o Sampling inspired by the sample-then-ovalnat imagination paradigm used in robotics (Kapelyukh et al. 2024 Ding et al.; 2024). Since their original methods are designed for robotic manipulation, we reimplement them under our method. Note that this sampling strategy can only be applied to some of the tasks in our evaluation since sampling in large solution spaces is heavily intractable. Furthermore, to demonstrate the necessity of altering the visual scene for task decomposition beyond adding visual prompts, we crafted a reasoning baseline named cursor-only that utilizes our imagination space, but restricts operations solely to transform operations of the virtual cursor. Note that this baseline serves as an ablation of our method, which has the same closed-loop control design except that scene modification is disabled and only virtual cursor moving is enabled. Furthermore, also adopt two recently proposed advanced MLLMs, namely OpenAI's ol (OpenAT. 2024b) and Molmo (Deitke et al.] 2024). ol is well known to have specifically trained reasoning abilities under pure textual modality. We utilize it to justify whether simply scaling text-time textual reasoning could solve the tasks. On the other hand, Molmo is trained to have ability to utilize visual markers as cues in visual modality for reasoning. We utilize it to justify the necessity of task dccomposition instead of conducting visual prompting with visual markers.
4.3 Counting
The results are shown in Tab. Our approach signíficantly improves the performance of original GPT- 4o, achieving a higher success rate and lower errors when mistakes occur. It worth noting that Molmo is known to be specifically trained to conduct counting with its native "pointing" ability. Our approach enables GPT-40 to perform on par with it. Note that the sampling method is not directly applicable to the counting task due to intractability, so we omit its comparisons. VCoT only supports drawing one bounding box to highlight a piece of visual evidence, hence is unsuitable for more complex reasoning tasks involving multiple objects, such as counting. We also compare with our cursor-only baseline in counting, observing that it performs similarly to GPT-4o. Since having a cursor moving around might be incffoctive for counting, we adopt a cursor-only variant, where the MLLM can draw a bounding box on the cursor location to highlight the object during counting. Although technically able draw boxes every ball to perform counting
9

---

Published in Transactions on Machine Learning Research (09/2023)
GFE4o
0.0
text
GFT46
0.7
5
10
12
34
10
10
12
13
14
15
17
58
20
70
(a) Success Rate vs. Number of (b) Solvable Rate vs. Reasoning (c) Counting by drawing bounding boxes Objects
Steps
only
Figure 4: (a)(b) show that as counting task difficulty increases linearly, scaling inference-time textual reason- ing (implemented GPT-4o-text-cot, See Sec. 4.3) fails and even performs worse than vanilla GPT-4o complexity exceeds perception limits. In contrast, our methods remain unaffected by these limits, achieving correct counting even as difficulty rises. Additionally, adding visual prompts can introduce noise, causing MLLMs to enter hallucination loops; for instance, in (c), the model incorrectly concluded there were 196 balls when only two were present. The qualitative results of counting are provided in the appendix.
GT
GPT-4o
Sampling
Cursor-anly
Visual CoT
Gouand Rash Gocelhes (61)
Coodlnee DSA cag
Cooralatte (L4Z
Coorstrate:
Coonshatee (402.477)
Gosealette (107 A03)
Qualitative comparison simple jigsaw puzzle solving, jigsaw pieces more visible to MLLMs. We illustrate the final visual state achieved by different methods after completing their reasoning processes and producing a solution. For clarity, the actual location of each coordinate in the image is highlighted with an orange line and circle.
Figure 5:
black background to make
accurately, the model quickly enters negative feedback loop: hallucinations lead it to draw more markers, which introduce additional noise and exacerbate the ballucinations. This results in success rate of 0% and makes mean error and variance not calculable, as the model cannot stop counting, further highlighting the importance of modifying the visual scene.
To further verify the essentialness of task decomposition in visual modality, we conduct additional analysis under the real object data by increasing the counting difficulty by adding more objects with partial data. The purpose was to evaluate the impact of difficulty and reasoning steps on performance. To achieve this, selected subset of cases from the whole data and supplemented them with simpler cases involving fewer objects. We then re-conducted the experiments on this partial dataset using GPT--o,, alongside a newly introduced textual CoT reasoning baseline. In the first experiment (Fig. 4aa, we show the success rate of different methods when faced with varying numbers of objects. In the second experiment (Fig. 4b) we test how different methods improve when given different limits on the number of reasoning steps. The result is shown as the solvable rate, which indicates the percentage of cases the method can solve under the current budget. When imposing such a limit, model is restricted to performing more than the specified number of steps, with each step defined performed after focusing. For the textual CoT bascline, we explicitly instructed the model on the maximum number of reasoning steps it was allowed to use. The results are shown in Fig. 4a with an additional baseline EPPT-4o-text-cot, indicating the pure text- based COT. This reveals perceptual limitations GPT-4o that cannot be overcome by simply increasing the textual CoT steps, while our method remains unaffocted. Fig. 4b shows the percentage of cases solved as reasoning steps increase. For reference, we include the one- step results from GPT-4o depicted flat line. Our method performs consistently better reasoning steps
combination of the focus operation and the subsequent operation
10

---

Published in Transactions on Machine Learning Research (09/2025)
GPT-4a
Autonomous Imagination (ours)
Sampling
Automomous Imagination (cursor only)
Visual CoT
Coodleate (450/195
Goordrate: 27g)
Conntratee
Condirate: 1114
Cowehaate
Prompt: The guest will sit on the right side the table, near the yellow cup. Where should we serve the apple?
Figure 6: Qualitative comparison of apple placement based user prompts. Coordinate locations are highlighted the image with orange lines and circles for improved visualization. Our method aligns more closely with the user's requirements, placing the apple further to the right side of the table.
grow, whereas GP-T-4otext--o fails do this and may even introduce noise, resulting poorer performance than one-step GPT-4o.
4.4 Simple Jigsaw Puzzle Solving
In the jigsaw puzzle solving task, the rotations pieces are not considered. As multiple trials are allowed, we restrict the total number of attempts to 20 and report the finish rate, defined as the proportion of missing pieces successfully placed. The sampling method involves comparing pairs of larget locations and iteratively selecting the better option until single target location remains. We guarantee that the correct answer exists within the sampling process. In our method, once jigsaw piece is moved to the imagined target location, it is considered to be placed. GPT-4o and ol select the target location by outputting coordinates, while VCoT and Molmo determine the larget location by drawing a block or marker, leveraging their specialized training for such tasks.
As shown in Tab. and Fig. 5 our method consistently outperforms all basclines. It achieves a higher locating accuracy, which contributes to higher success rate with the number of attempts. Since this visual reasoning task requires the model to choose the correct target location rather than simply locating visible elements, VCoT is ineffective at reasoning. The sampling method performs reasonably well in this task, since we set large sampling budget to ensure the correct answers can be sampled as the candidate answers. However, despite its extensive reasoning budget, it still does not match the performance of our method. This highlights the effectiveness and robustness of establishing an autonomous perception-control loop. We also demonstrate that the cursor-only baseline performs similarly primitive GPT-4o in the four-piece missing scenario. However, performs significantly better under the more challenging six-piece missing scenario, further confirming the effectiveness of our closed-loop control approach.
4.5 Object Placement
Similar to the approach used in solving simple jigsaw puzzles, the methods compared include the follow- ing: GPT-4o and ol, which output the target location by specifying coordinates; the sampling method, which iteratively compares sampled results until single option remains; VCoT and Molmo, which deter- mines placement by drawing blocks markers; and our method, which outputs the final placement after completing transformation operation. In line with how related work in robotics handles sampling-bases methods (Kapolyukh et al.) 2024 Ding et al. 2024). the sampling bascline restricts its search to subregion of the space, filtering out certain incorrect answers. For example, regions occupied by existing objects are excluded from consideration. This principle is also incorporated into the transformation operation in our method. Specifically, a focus operation is first applied to identify the platform on which the target object should reside, creating a region mask. The transformation operation then ensures that no movement is sug- gested if it would lead the object outside the defined mask. This approach improves efficiency and ensures
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Results under multi-object hallucination benchmark. Except for ol, Mohmo, and our method, the results of other basclines are cited from the official leaderboard in cithub.io/ For each base model family, we take the one with the highest performance. Mechanistically grounded MLLMs (marked with *) take visual prompts by dedicated pointer tokens. Please refer to the original paper for more details.
Models
Acc.
Models
Acc.
Models
Acc.
Models
Acc.
LLaVA-34B
30.81%
CogVLM-C
13.50%
Qwen VL-C
15.37%
Molmo
28.90%
IDEFICS
6.50%
GLaMM*
52.28%
MiniCPM-V
14.39%
ol
60.20%
Yi-VL-34B
0.41%
GromdHHG*
38.13%
GPT-4o
53.74%
GPT-4o+ Ours
65.00%
Table 3: Results under more existing benchmarks.
CLEVR (Counting) Mean Error
Where2Place (Object Placement)
Models
Success Rate
Variance
Accuracy
GPT-4o
57.3%
0.52
0.45
29.1%
CPTT-lo
74.7%
0.42
0.74
37.0%
a fair comparison between the sampling baseline and our method by reducing unnecessary exploration of invalid regions.
As shown in Tab. our method consistently outperforms the bascline methods in both object locating and placement performance. It is important to note that our cursor-only ablation baseline is identical to our full method when locating an object. Therefore, we assign them with the same locating result. Additionally, the sampling method is intractable for locating. Thus, we directly utilize the locating results of our method for placement. Methods lacking advanced visual reasoning capabilities perform poorly on the placement task, as successful placement requires inferencing about the correct spatial position for that element beyond visual recognition. The cursor-only baseline, which moves the visual marker instead of the scene modification, does not perform as effectively as our complete method. This difference underscores the necessity for substantial modification in visual task decompasition. Although the sampling baseline receives substantial visual infor- mation, it still underperforms relative to our method and even falls short of the cursor-only baseline. In contrast, in simple jigsaw puzzle solving, the sampling method achieves a relatively high success rate com- pared to the cursor-only baseline. This contrast highlights the importance of structured reasoning pathway, particularly complex, open-world scenarios where the abundant visual information could overwhelm the MLLM, preventing it from identifying the correct answer despite its presence in the sample. By following the closed-loop control reasoning process, MLLMs can progressively approach the correct answer without requiring exhaustive number of samples, resulting in greater efficiency and improved accuracy.
4.6 Multi-Object Hallucination Benchmark
We evaluated our method using the recently proposed ROPE benchmark for Multi-Object Hallueina- tion (Chen et al. 2024a), focusing on the most challenging subset where GPT-4o exhibits the poorest performance, which is answering about a single object given multi-object image, heterogeneous object types, unseen data split. Our method uses GPT-40 the base model that equips the imagination space. Please refer to the original paper for more details. Given that this benchmark emphasizes the identification of abstract objects that are not suitable for segnentation, we turn the focus operation into drawing a large reclangular region by MLLMs natively, through selecting lop-left and bottom-right corners. The benchmark indicates that MLLMs are prone to hallucinations when confronted with multiple objects that introduce additional visual distractions. We demonstrate that using our method, MLLMS can autonomously focus on the important region despite these distractions, leading to more accurate responses due to the climination of irrelevant visual information.
12

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Results for smaller MLLMs under the object placement task.
Qwen2.5- VB VL-7BB
Q0we22.GVVL-32B Base Model
InternVL2 Base Model
G-8B
InteruV12
5-2GB
Base Model
Ours
Ours
Ours
Base Model
Ouss
Locating
7.2%
19.3% 2.2%
2.9% 0.85%
16.8%
1.1%6 0%
11.6%
1.5%
16.7%
Placement
1.1%
4.2%
3.6%
0.7%
5.1%
Table 5: Comparison between RobotPoint (fine-tuned to solve Where2Place lasks) and GPT-4o enhanced with our approach.
Counting Mean Erroz
Simple Jigeaw Puzzle 6-Piece Missing
Object Placement (Ouz)
Object Placenent (Waere2Plaee)
Models
Success Rate
Variance
4-Pieoe Missing
Locating
Placing
Acmuzcy
Polat
25.0%
2.04
3.92
6.88
6.1%
1.0%
0.63%
46i8%
GPT-1o
39.8%
0.73
0.62
29.5%
9.1%
10.9%
3.6%
29.1%
GPT-1o 1o + ours
85.3%
0.19
0.22
68.2%
51.5%
69.4%
37.3%
37.0%
4.7 Discussions
Results on more existing benchmarks. To verify the robustness of our approach, we further conduct experiments under two more existing benchmarks. One is CLEVR (Johnson et al, 2017) for the counting task. In each image from CLEVR, a scene containing various number of geometric objects is presented. We randomly sample from the official test set to ensure balance over the numbers of objects in the images, which includes 178 images. Another is Where2Place (Yuan et. al., 2024h, which includes 100 real-world scenes from homes and offices in the wild for the object placement task. The results are shown in Tab. g verifying that our approach consistently improves the reasoning ability of the basc MLLM.
A Applying smaller-sized MLLMs. Our approach requires the ability of the base MLLMs to au- tonomously utilize the visual imagination operators provided. This is natively satisfied by larger state- of-the-art MLLMs, such as GPT-4o, while maybe challenging for smaller-sized MLLMs without specific training. We verify this in the counting and jigsaw puzzle solving lasks, where the smaller-sized MLLMs usually have the difficulty in realizing "when to stop": Common issues include incorrectly deciding to stop counting when there still exist remaining objects, or being hard to judge whether the jigsaw piece has been put at the right position. This shows the essentialness of studying how the autonomous imagination ability be trained, which is left as future work. On the other hand, discover that our training-free strategy
can indeed be utilized to enhance smaller-sized MLLMs (Qwen2.5 (Bai et al.. 2025) and InternVL2.5 (Chen eet al.] 2024bb) under the object placement task, which is shown in Tab. 4 verilying the potential of our approach for smaller-sized MLLMs.
Comparing with in-domain training. It is also mcaningful to compare our approach with in-domain trained models. In Tab. [ we have that Molmo achieves strong performance in the counting lask, with the specifically trained ability to utilize visual cues in counting, while performing not ideally under other tasks. We further test the RobotPoint model (Yuan et al. 2024), trained spocifically for predicting image keypoint affordances given language instructions, which is especially suitable to handle the Where2Place benchmark. We compare its performance under Where2Place, as well as our counting, jigsaw puzzle sol solving, and object placement tasks, as shown in Tab. 5 Similar to Molmo, RobotPoint shows strong performance under Where2Place that it is trained to handle, while fails to generalize its ability in other tasks. This suggests that in-domain training more suitable to address specific tasks, while improving test-time reasoning can be an casier way to improve general reasoning ability across different tasks. Morcover, as discussed in the analysis of smaller-sized MLLMs, conducting training on the general reasoning ability instead of domain- specific task solving is also promising.
Concurrent work on multimodal CoT reasoning. Concurrent to our work, there is a surge of rescarch for multimodal COT reasoning. A representative is OpenAl's o3 model (OpenAl 2025a). which is natively trained multimodal reasoning ability for "thinking with images" (OpenAl] 2025b). which utilizes visual processing tools to modify images (e.g. crop, zoom, rotate), hence realize more thorough analysis of the inputs during CoT reasoning. This paradigm is also studied by other works via visual test-time scaling [Luo
lt al. 2025
Wu et. al. 2025), supervised fine-tuning [Ti et al. 2025 Fu et al., 2025), and reinforcement
13

---

Published in Transactions on Machine Learning Research (09/2025)
Table Comparison between OpenAl's ol and o3 models under our benchmarks.
Councing Mean Error
Simple Jigsaw Puzzle
Object Placement
Models
Surcess Rate
Variance
Piece Missing
6 Piece Missing
Localing
Placing
ol
50.8%
0.74
0.90
31.8%
27.3% 66.7%
17.3%
8.5%
u3 w/o lool
50.0%
0.71
0.78
61.4%
9.5%
4.4%
63
76.2%
0.36
0.69
78.6%
57.1%
22.9%
6.2%
"There frnch
This dumboell."
ould
Figure 7: Failure cases in our method caused by hallucination, including instances where MLLMs mistakenly perceive fry that does not exist, misinterpret the wrongly focused floor as dumbbell, and incorrectly believe that puzzle piece has been placed in the correct spot.
learning (Duan et al. 2025 Xu et al. 2025). Tab. illustrates the comparison among OpenAT's ol and o3 models in our proposed benchmarks, with an additional baseline "o3 w/o tool" in which we explicitly prompt o3 to disable the tool use ability. The results are consistent with our work: Multimodal CoT usually improves over purely textual reasoning. Compared with these concurrent works, our work uniquely proposes the notion of "visual-to-textual" conversion and explicitly differentiates between visual prompting and visual imagination. This can lead to better understanding of fundamental research question: In what kinds of tasks, the ability to think with images is necessary? Furthermore, our alug-and-play autonomous imagination approach can be of independent interest to inspire more advanced multimodal CoT reasoning approaches.
Failure cases. In Fig. 7 we illustrate failure cases of our approach, which are majorly resulted from the hallucination of MLLMS.
Time cost. The time cost of our approach depends mainly on three factors: the rumning time of single call of imagination operators, single call of MILM reasoning, and the number of MILM calls. The calls of imagination operators are quite efficient. For example, each call of the SAM model takes 220ms on NVIDIA RTX 4090 on average, which is very efficient compared with MLLM reasoning. Considering the number of calls for MLLM reasoning, the average numbers of MLLM API call for our approach are: counting=117 (for all objects), jigsaw puzzle=20 (for one piece), object placement=15 (for one object),
multi-object hallucination-26 (for one object). As shown by the comparisons on GPT-4o Sampling bascline in Tab. [ our approach is more elficient than the closest previous studies to ours: sampling-baes based imagination strategies (Kapelyukh et al. 2024 Ding et al. 2024), by enabling to solve broader tasks with vast solution spaces.
5 Conclusion
In this work, we target at lackling visual reasoning problems using MLLMs where textual-to-vvsual conversion is the major bottleneck. We propose the autonomous imagination approach, which employs plug-and-play imagination space and operator design, enabling MILMs to modify visual content autonomously, leading to closed-loop decomposition of this conversion process. We conduct experiments under visual reasoning tasks that are beyond the perceptual capabilities of MILMs, while remained straightforward to humans. The results show that these tasks can be effectively tackled natively by MLLMs equipped with our approach.
Limitations and future work. Our work is a proof of concept: While closed-loop task decomposition overcomes the visual-to-textual conversion bottleneck in our demonstrated tasks, resource constraints pre-
14

---

Published in Transactions on Machine Learning Research (09/2025)
vented from training model with native closed-loop reasoning and visual modification capabilities. This also prevents us from validating our approach under large, general-purpose visual benchmarks. We hope that our work inspires future efforts to close this critical gap. Furthermore, even though we test the robustness of our approach in counting and jigsaw puzzle sovling through different lovels of difficulty (in terms of numbers of abjects/pieces) for object placement and multi-object hallucination tasks, properly measuring the lask difficulty itself a challenging problem. These tasks involve real-world scenes with complicated spatial and semantical relationships among the objects, which may not be easily captured by single measurement of hardness. We treat this an important future research problem. Finally, our current approach limits to utilize chain reasoning with fixed plans, which is not optimal in terms of time cost. Future work can improve the efficiency of reasoning with more advanced searching and planning strategies.
Acknowledgement
This work was supported by National Natural Science Foundation China (123420311,62200244)) We would like thank the reviewers for their insightful and constructive suggestions.
References
Anurag Ajay, Seungwook Han, Yilun Du, Shaung Li, Abhi Gupta, Tommi Jaakkola, Josh Tenenbaum, Leslie Kaelbling, Akash Srivastava, and Pulkit Agrawal. Compositional foundation models for hierarchical planning. arXiv preprint ar arXiv:2909.00 00882399.,8887, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 23736, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijic Wang, Jun Tang, et al. Qwen2. 5-vl technical report. ar preprint or arxi::2002. 13923, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multi- modal I1m's referential dialogue magic. arXiv preprint arXiv:2306.1 15195, 2023.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Good- man, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.067 2022.
Xuwciyi Chen, Ziqiao Ma, Xuejun Zhang, Sihan Xu, Shengyi Qian, Jianing Yang, David Fouhey, and Joyce Chai. Multi-object hallucination in vision language models. In Advances in Neural Information Processing Systems 37, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. orXiv preprint [XXv:2212.0527., 2024b.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jac Sung Park, Mohammadroza Salchi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multinodal models. arXio preprint or Xiv:22400.1 17146, 2024.
15

---

Published in Transactions on Machine Learning Research (09/2025)
Yufei Ding, Haoran Geng, Chaoyi Xu, Xiaomeng Fang, Jiazhao Zhang, Qiyu Dai, Songlin Wei, Zhizheng Zhang, and He Wang. Open6DOR: Benchmarking open-instruction 6-dof object rearrangement and a VLM-based approach. In 1CRA 2024 Workshop on 3D Visual Representations for Robot Manipulation, 2024. URL
Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. arXiv e-prints, pp. arXiv MwXXv-2302, 2023.
Chengqi Duan, Rongyao Fang, Yuging Wang, Kun Wang, Linjiang Huang, Xingyu Zeng, Hongsheng Li, and Xihui Liu. Got-rl: Unleashing reasoning capability of mllm for visual generation with reinforcement learning. aixiv preprint arXiv:2505.11022 2025.
Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, et al. Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing. arXiv preprint arxi::2003. 10639, 2025.
Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinci Flo- rencio, and Cha Zhang. Refocus: Visual editing as a chain of thought for structured image understanding. arXiv preprint ar Xix:20011.0000 055552, 2025.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning, In The Eleventh International Conference on Learning Representations, 2022.
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arxiv preprint arXiv:2312.0880 08914, 2023.
Jie Huang and Kevin Chen- Chang. Towards reasoning in large language models: A survey, 2023.
URL https://arxiv..nt org/ 2212 10403 10403
Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. In Proceedings of the IFFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8362 8371, 2024.
Itseez. Open source computer vision library.
2015.
Songtao Jiang, Yan Zhang, Chenyi Zhou, Yeying Jin, Yang Feng, Jian Wu, and Zuozhu Liu. Joint visual and text prompting for improved object-centric perception with multimodal large language models. arXiv preprint Kki::240400511, 2024.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A dingnostic datasct for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Ivan Kapelyukh, Yifei Ren, Ignacio Alzugaray, and Edward Johns. Dream2Real: Zero-shot 3D object rear- rangement with vision-language models. In IEFE International Conference on Robotics and Aulomation
(ICRA), 2024.
Bernhard Kerbl, Georgios
Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https:
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4015-4026, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot. reasoners. Advances in neural information processing systems, 55-22199-22211 2022.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprinl arxiv:208..0000 00662, 2023.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulié, and Furu Wei. Imagine while reasoning in space: Multimodal iorallization-ootthuuiii arXiv preprint 0014:22501.0154, 2025.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Muchine Learning, pp. 12888- 12900. PMLR, 2022a.
Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-inage pre-training with frozen image encoders and large language models, arXiv preprint TrXto:2301.12590 2023.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian- Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206. 02336, 2022b.
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang, Shanghang Zhang, and Hongsheng Li. Draw-and-understand: Leveraging visual prompts to enable milmss to compre- hend what you want. arXiv preprint or 2rXii:2403. 20271, 2024.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07510 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. In NeurIPS, 2023.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te- Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M. Dai. Mind's eye: Grounded language model reasoning through simulation, 2022. URL https:
//arxiv.org. www.:/..ggabbbbbe http//bb//22..... 05359
Pan Lu, Liang Qiu, Kai- Chang, Ying Nian Wu, Song-Chun Chun Zhu, Tammay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi- strucurr mathematical reasoning. arXiv preprint Xrx:22200. 14610, 2022.
Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-languaae instruction tuning for large language models. Adconces in Neural Information Processing Systems, 36, 2023.
Tiange Luo, Lajanugen Logeswaran, Justin Johnson, and Honglak Lee, Visual test-time scaling for gui agent grounding. arXiv preprint arXiv:2505.0000 00684, 2025.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig, Compositional chain-of-thought prompting for large multimodal models. arXiv preprint arXiv:2311.17076, 2023.
OpenAL. Hello 4PT-4o, 2024a. URL httpppp://pppaa...c////////////a
OpenAL.
Learning to reason with 1ms, 2024b. learning-tee tew-toor-r---ttteeeeee
URL http:///peeaal.com/nnne
OpenAl. 03 model.
https: http:////eeeaaaaeeoo///////////a
2025a.
OpenAl. Thinking with images. httt::///geaaa..commmmmaa//bbbba
2025b.
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2; Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Ji Qi, Ming Ding, Weihan Wang, Yushi Bai, Qingsong Lv, Wenyi Hong, Bin Xu, Lei Hou, Juanzi Li, Yuxiao Dong, et al. Cogcom: Train large languael models diving into details through chain of manipulations. ar Xiu preprint ar wxxi:22402.04200 2024.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXi::2408.0077 2024.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. Learning to retrieve prompts for in-context learning. arXiv preprint or wrxii:2112.08666 2021.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hongshong Li. Visual cot: Unleashing chain-of-thougge reasoning in multi-modal language models. Advances in Newral Information Processing Systems, 37, 2024.
Qiuhong Shen, Xingyi Yang, and Xiuchao Wang. Flashsplat: 2d to 3d gaussian splatting segmentation solved optimally. In European Conference on Computer Vision, pp. 456 472. Springer, 2024.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpl: Solving tasks with chatgpt and its friends in hugging face. Advances in Newral Information Processing Systems, 36, 2023.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning, In Proceedings of the IEEE/CVE international conference on computer vision, pp. 11888-11898, 2023.
Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song. et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint NViv:2311.03072 2023a.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2023b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1111 11171, 2022.
Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal 1lm as an agent for unified image generation and editing. Advances in Newral Information Processing Systems, 111128374-283999 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 -448377, 2022.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303. 0et.230000,67,,,, 2023.
Hang Wu, Hongkai Chen, Yujun Cai, Chang Liu, Qingwen Ye, Ming-Hsuan Yang, and Yiwei Wang. Dimo- gui: Advancing test-time scaling in gui grounding via modality-awarr visual reasoning. arXiv preprint arXiv:2577.0000 2025.
Junda Wu, Zhehao Zhang, Yu Xia, Xintong Li, Zhaoyang Xia, Aaron Chang, Tong Yu, Sungchul Kim, Ryan A Rossi, Ruiyi Zhang, et al. Visual prompting in multimodal large language models: A survey. arXiv preprint 2iv: 15310, 2024a.
Penghao Wu and Saining Xic. V*: Guided visual search as core mechanism in multimodal 1ms.. arXiv preprinl ar XrXi::2312. 1415 17, 2023.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Visualization-of thought elicits spatial reasoning in large language models. arXiv preprint arXin:2404.000 03622, 2024b.
Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, and Ivan Vulié. Visual planning: Let's think only with images. Xixiv preprint arXiu:250.. 11409, 2025.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint wrXiu:2303.11388 2023.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Tzhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models, 2023. URL
org/ ht///bb//3000 2305. 10601
Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaussian grouping: Segment and edit anything in 3d scenes. In Europeon Conference on Computer Vision, pp. 162-179. Springer, 2024.
Rumpeng Yu, Weihao Yu, and Xinchao Wang. Api: Attention prompting on image for large vision-language models, 2024.
Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fax. Robopoint: robotics. arXiv preprint ar arxi:24006. 10721, 2024.
language model for spatial affordance prediction for
Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, and Jiebo Luo. Cocot: Contrastive chain-of-thought prompting for large multimodal models with multiple image inputs. arXiv preprinl Xiv:24401. 02582, 2024.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-offinteres. Xiv preprint Xix:2307.036000 2023a.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context. learning? Advances in Newral Information Processing Systems, 2023b.
Yuechen Zhang, Shengju Qian, Bohao Peng, Shu Liu, and Jiaya Jia. Prompt highlighter: Interactive control for multi-modal llms. arXiv preprint ar arx:::2312.043000 2023c.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting large language models. arXiv preprint ar Xiv:22200. 0.49393,.3.... 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of- thought reasoning in language models. arXie preprint arxiv:230 00923, 2023d.
Ge Zheng, Bin Yang, Jiajin Tang, Hong- Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. Advances in Newral Information Processing Systems, 36:5168 5191, 2023.
Fashen Zhou, Yiran Qin, Zhenfei Yin, Yuzhou Huang, Ruimao Zhang, Lu Sheng, Yu Qiao, and Jing Shao. Minedreamer: Learning to follow instructions via chain-of imagiiiaiiiniiiinnnnnn for simulated- world control. arXiv preprinl arXiv:2403.1200 120377, 2024a.
Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang, Image-of-thought prompting for visual reasoning refinement in multimodal large language models, 2024b. URL
org/
abs/2405.13877
Yiyi Zhou, Tianhe Ren, Chaoyang Zhu, Xiaoshuai Sun, Jianzhuang Liu, Xinghao Ding, Mingliang Xu, and Rongrong Ji. Trar: Routing the attention spans in transformer for visual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2074-2084, 2021.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXto:2304.10592, 2023.
Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongshong Li, and Yu Liu. Mova: Adapting mixture of vision experts multimodal context. arXiv preprint arXiv:214413304 2024.
19

---

Published in Transactions on Machine Learning Research (09/2025)
SmartEdit placing pear
SmartEdit placing apple
Table 7: Quantitative results counting using Cog- CoM (Qi ct al. 2024h, by directly asking it to output the number of objects, and by prompting it to use the trained GROUNDING ability, which counts for the object. We find that though explicitly trained perform GROUNDING which counts the specific object, it somehow performs worse in our evaluation.
Settings
CogCoM
CogCOM-CROUNDING
Figure 8:
We show that existing image editing mod- els lack the capability to effectively comprehend and execute complex commands, such as placing the ap- ple/pen /pear at the left/right rightt side on the table.
Success Rate
30.3%
14.8%
Counting
Mean Error
1.26
3.77
Variance
1.39
15.47
Input
Output
Identify the empty region in the image. The landmark information does not provide sufficient details to identify specific area within the given description "sa "sa_1679 1679". So ultimately, the conclusive answer to the question discussion doesn't match my current knowledge.
Figure 9: Qualitative results on using CogCoM QQi et al. 2024) for jigsaw puzzle solving show that it generates unrelated text when asked where the highlighted puzzle piece should be placed.
A Potential Broader Impact
Our work is largely foundational and aims to improve the reasoning capabilities and reliability of multimodal language models. Reducing hallucinations in complex visual scenes could enhance safety and robustness in real-world applications. On the other hand, our approach could increase the computational cost of MLLMs, leading to energy inefficiency and environmental impact. Reproducing our work also requires access to existing MLLMs and visual tools, leading to additional complexity and costs. We hope that our efforts on open-release all our code and data could roduce this burden.
B More Experiments
Imagine with Image Editing Models. We experimented with utilizing image editing models that accept input of natural languages, such as the srateoof-th--tt model SmartEdit (Huang et al.) 2024). Idcally, such models would provide effective guidance by generating the target goal based on descriptions natural language. However, as shown Fig. we find that the image editing model is not idcal when comes following complex instructions. This underscores the significant challenge of "imagining" the target state in open-wold worndworld problems, suggesting that world model capable of accurately providing such visual guidance requires further development.
CogCoM. We have also experimented with a potentially feasible baseline CogCoM (Qi el al. 2024). How- ever, we find that it does not function well when tackling our challenges. Despite explicitly trained with a GROUNDING ability for counting objects, as shown in Tab. 7 directly prompting the model output the counting result is more accurate than performing GROUNDING. When asked complete the jigsaw puzzle finding target location, starts hallucinating by outputing unrelated text, shown in Fig. g This is
20

---

Published in Transactions on Machine Learning Research (09/2025)
potentially due to the model trained different tasks and not generalizing to challenged tasks. We decided to present the result in the appendix instead.
C Additional Qualitative Results
We present qualitative results tasks that include counting, solving simple jigsaw puzzles, and placing objects in Fig. 10 14 These images more effectively illustrate the dctailed visual reasoning processes and the changes within the imagination space. Please also see the supplementary video, which shows the reasoning process of our approach in the jigsaw puzzle solving task.
Vaua
Autromaua Imaginatin
Sumpping
Ground Trush
(AM1) Astoromoia Imaighatio
Viex
Sampiin
Cround Truhh
(sun)
GPT-46
Viea
joueer oily)
Aupromssci Imaggiation
Figure 10: Additional qualitative results of simple puzzle solving. Please zoom in for a clearer view.
21

---

Published in Transactions on Machine Learning Research (09/202)
00
Figure 11: Qualitative demonstration of the counting process autonomously performed by our approach.
22

---

Published in Transactions on Machine Learning Research (09/2025)
Figure 12: Qualitative demonstration of the counting process autonomously performed by our approach.
23

---

Published in Transactions on Machine Learning Research (09/202)
Prompt: The calculator the table is dirty, please bring some tissues.
Autonomous Imagination [ourscr only)
GPT-4o
Visual CoT
Sampling
Ground Truth
Autenomous Imagination (ours)
A student is sitting in seat with a white bag on it. Could you place the black cell phone on her table?
Autonomous Imagination (oursor only)
GPT-40
Visual CoT
Sampling
Ground Truth
Ocpaaists 8550 ANg
Condirete NA373
Goodhoas ARALSPAL
Coosfreto (900/8A)
Autonomous Imagination (ours)
Osooshogto 2207 7541
Figure 13: Additional qualitative results of object placement are provided. We also illustrate some steps in our method during the search for the target object and its placement. Please zoom in for clearer view.
24

---

Published in Transactions on Machine Learning Research (09/202)
Prompt: Could you place the pear on the plate in front dark blue bag?
Autonomous Imagination (cursor only)
GPT-4o
Visual CoT
Sampling
Ground Truth
Ossnaisate
Autenomous Imagination (ours)
Coorshraee (112280)
Prompt: The laptop's battery is dead. Please take it to the outlet to charge.
Autonamous Imagination (cursor only)
GPT-40
Visual
Sampling
Ground Truth
Osoopiatte (550 ANN
Good (00L
CoodSeete 8647
Autonomous Imagination (ours)
Cocedhote (708) 2110
Figure 14: Additional qualitative results of object placement are provided. Please zoom in for clearer view.
25

---

Published in Transactions on Machine Learning Research (09/2025)
General Reasoning Instructions for Counting
You are an assistant which counts the number of specific objects in the given image and outputs it. Please recognize distinct objects one by one, describe each unique object including its color and location, and count during this process.
General Reasoning Instructions for Jigsaw Puzzle Solving
In this task, you will receive a picture of an unfinished puzzle. The image contains a nearly complete puzzle and several pieces that have not yet been placed into the whole.
You should think step by step as follows:
1. Which of these individual puzzle pieces fits best into the whole? Describe the contents and location of the puzzle piece.
2. Establish two-dimensional rectangular coordinate system in the image, with the lower left corner as the origin, the right direction as the positive x-axis, and upward as the positive y-axis. The x-axis coordinate range of the image is O-$IMAGE_WWI _IDTH (pixels), and the y-axis coordinate range is O-SIMAGE HEIGHT (pixels).
Please output the coordinates of your selected piece in the image. You should output the answer in the last line in the format ['x y___corddiateeeyycm cooordinatt""..
General Reasoning Instructions for Object Placement
You are an assistant that decides where to place object based user requirements. First, make a plan by:
1. Listing the objects mentioned in the user requirement
2. Identifying which objects need to be moved or placed
3. Determining reasonable order to move them
Finally, summarize the objects with numbers that need to be moved in the last line, using lowercase singular form. The output format should be [['Ojject', -ireec_-umber]---
A two-dimensional rectangular coordinate system is established in the image, with the lower left corner as the origin (right positive x-axis, upward positive y-axis). The x-axis ranges O-SIMAGE_WIDTY (pixels), and y-axis ranges O-ŞIMAGE --SIMAGG_HEIIHTT (pixels).
For each object:
1. Locate it by outputting its coordinates
2. Output the target location coordinates
You should format your final answer as ['x "x_coordinate', satpeee..ycoooooeo coordinate").
Prompts for Autonomous Imagination
You can manipulate the scene using these operations:
1. Control a cursor to focus on objects
2. Transform focused objects or ignore them
3. Return to cursor control after operations
For counting: Ignore irrelevant objects while counting. For object placement: Move the cursor to focus on the target object, then transform it to the proper location.
For jigsaw puzzles: Focus on the puzzle piece and transform its ideal position.
Figure 15: General reasoning instructions provided models for visual reasoning, as well the reasoning instruction given to utilize autonomous imagination. It is not difficult for advanced MLLM to figure out this plan. As we aim to break the visual-to-textual conversion bottleneck, simplify this by directly given the reasoning plan to the MLLM to avoid occasional failure caused by the MLLM choosing bad reasoning stralegy.
26

---

Published in Transactions on Machine Learning Research (09/2025)
Cursor Control
You are now controlling the cursor. You should find the target object based on the plan you made. You will receive an image and the object to be located. There is red circle in the image, surrounded by the four letters: 'a', 'b', 'c', and 'd'. You should choose the letter closest the target object. If there are multiple target objects the image, you should aim the red circle at the closest one. Ignore artifacts like shadows and floating spots, You should output the quoted letters in the last line, that is: ['a'], ['b']. ['c'], or ['d"]. The distance will be reduced in each iteration until convergence, so you only need to choose the direction.
Checking Focus Segmentation
Your task is to check if the object shown in the image is the object you seek. The image should show a single target object against black background. Areas of the image that do not belong to the object are set to black. If not, you should judge it as ['no'] and end your check. You should imagine what the object would look like from above and compare to the image.
You should describe the object in the image, then output the check result in the last line the format: ['yes'] ['no']
After Focus
You are now focusing on the object you desire. You can choose to perform a transform operation or ignore the operation based on your own plan. Output ["transform]] or ['ignore'] the last line to perform the corresponding operation.
Transform Control
You are now moving the object the destination. The object you are controlling is highlight- ed and surrounded by at most four letters: 'a', 'b', 'c', and 'd'. Ignore artifacts like shadows and floating spots. Choose the direction leading to the destination. The distance will be reduced in each iteration until convergence, so you only need to choose the direction. You should output the quoted letters the last line, that is: ['a'], ['b'], ['c'], or ['d'].
Cursor Control (Special Focus for Multi-Object Hallucination)
You are now controlling the cursor. You should focus on the region based on the provided object ID. You should first move the cursor the top-left corner the object region. There is red circle in the image, surrounded by the four letters: 'a', 'b', 'c', and 'd'. You should output the quoted letters in the last line, that is: ['a'], ['b'], ['c'], or ['d"]. The distance will be reduced in each iteration until convergence, so you only need to choose the direction.
You have selected the top-left corner of the region. You should focus on the bottom-right corner of the region to be focused on. There is red circle in the image, surrounded by the four letters: 'a', 'b', 'c', and 'd'. You should output the quoted letters in the last line, that is: ['a'], ['b]], ['c'], or ['d']. The distance will be reduced each iteration until convergence, so you only need to choose the direction.
Figure 16: Detailed prompts instructing the MLLM about the operations it can utilize. This is given to the
MLLMM as a reminder when the MLLLM enters corresponding state during the reasoning.
27


Reviews:
[{"review_id": "6LghAUpaWW", "reviewer": "Reviewer_tdza", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**##Strength##**\n\nThe paper presents a fresh perspective on visual reasoning by focusing on visual-to-text conversion as the core bottleneck and addressing it through closed-loop scene edits. The method is intuitive, modular, and works with existing MLLMs out-of-the-box—no retraining required. Experiments are thoughtfully designed and show strong improvements across multiple tasks. The ablation studies are convincing, and the paper is generally well-written with clear figures and detailed analysis. The use of generic operators and existing tools like SAM2 also makes the system relatively easy to reproduce and extend.\n\nA key weakness lies in the paper’s heavy reliance on prompt engineering. Task-specific reasoning plans are hardcoded into prompts rather than autonomously generated, which limits the framework’s generality and scalability. This simplification, while acknowledged as a proof-of-concept, sidesteps the central challenge of enabling models to reason independently. Another concern is the limited validation of the core “perceptual bottleneck” hypothesis. While the counting experiments do convincingly show a performance drop as visual complexity increases, the other tasks—such as jigsaw puzzles, object placement, and hallucination mitigation—do not offer similarly strong or scalable evidence. These tasks lack sufficient variation in difficulty or do not directly measure how increasing complexity impacts model performance. Additionally, the evaluation is conducted on small, bespoke datasets, with only 122 images for counting and 17 scenes for object placement. This small scale makes it difficult to draw broad conclusions or assess statistical significance.\n\n\n**##Weakness##**\n\nThe system also imposes a significant computational burden. The iterative reasoning process leads to a large number of model invocations—over 100 GPT-4o calls on average per image in some tasks—which may make it impractical for real-world or real-time applications. Furthermore, the method is currently tied to a proprietary model (GPT-4o), and its applicability to open-source models remains untested. This reliance raises concerns about accessibility, reproducibility, and long-term viability. The paper also includes some unclear or underdefined technical components. For example, key terms like G_marked are introduced in algorithm boxes but not fully explained in the main text, and heuristics such as the 10% voting rule in 3D segmentation are used without justification. Finally, while the related work section addresses visual prompting and chain-of-thought methods, it misses recent and relevant efforts in closed-loop VLM agents and visual planning approaches, such as ViperGPT or “Mind’s Eye,” which weakens the contextual framing of the contribution.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Autonomous Imagination, a plug-and-play framework that enables multimodal language models (MLLMs) to reason more effectively by iteratively editing the visual scene they’re analyzing. The method operates without any additional training and uses a set of simple yet powerful operators—focus, ignore, and transform, applied in an “imagination space” that supports both 2D and 3D representations. By decomposing visual reasoning into a series of step-by-step edits, the approach helps address the \"perceptual bottleneck\" that often limits MLLMs. The framework is tested on four visual reasoning tasks—counting, jigsaw puzzles, object placement, and hallucination mitigation—and shows consistent performance gains, especially in the counting task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To strengthen the paper, the authors should broaden the validation of the perceptual bottleneck hypothesis across all tasks. Introducing more granular levels of complexity in the jigsaw and hallucination tasks, or redesigning them to better reflect scalable visual difficulty, would help support the central claim more convincingly. Another important improvement would be to reduce the framework’s reliance on hand-crafted prompts. Allowing the MLLM to autonomously generate its reasoning plan from high-level instructions would greatly enhance the method’s generality and realism, aligning it more closely with the vision of autonomous multimodal agents.\n\nIn addition, the authors are encouraged to evaluate the approach on larger, publicly available benchmarks, particularly in the counting domain, to demonstrate its robustness and external validity. Running experiments on open-source MLLMs such as LLaVA-1.6 would also help show that the method is not tightly coupled to a single proprietary model and can generalize more broadly. Given the high computational cost, profiling the API calls in more detail and exploring optimizations like batching or pruning of operators could make the approach more practical. The paper would also benefit from increased clarity and accessibility: terms like G_marked should be clearly defined in the main text, and a brief primer on the 3DGS representation would help readers unfamiliar with 3D scene reconstruction. Finally, the authors should consider expanding the related work section to better position their contributions in light of concurrent research on visual scene editing and closed-loop planning.", "broader_impact_concerns": "There are no significant negative broader impact concerns raised in the paper. The work is largely foundational and aims to improve the reasoning capabilities and reliability of multimodal language models, which is a positive direction. Notably, its ability to reduce hallucinations in complex visual scenes could enhance safety and robustness in real-world applications. However, some softer concerns remain. The method incurs high computational costs due to repeated API calls to closed-source models like GPT-4o—raising issues of energy efficiency and environmental impact. Its dependence on proprietary models also limits accessibility and reproducibility, potentially disadvantaging researchers without access to these resources. Additionally, reliance on external tools such as SAM2 and OpenCV introduces the possibility of error propagation, which is not fully addressed. While not problematic in a foundational setting, these concerns merit discussion if the approach is deployed in broader or critical contexts.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WTwWY9CIWU", "reviewer": "Reviewer_Ppp8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe proposed method is novel and intuitive. The experiments demonstrate that such autonomous imagination can significantly enhance the model's performance on some visual tasks.\n\nWeaknesses:\n1. **Generalizability of the operations.** The proposed three operations are quite customized to those evaluated tasks. Whether such types of imagination can help with other visual reasoning tasks is unclear. It may require manually designing new types of imagination for new tasks, thus limiting the scalability of the proposed methods. It would be valuable to include the evaluation on more general vision-language tasks to show that imagination at least won't hurt the performance.\n\n2. **Assumption of single path reasoning.** The proposed method assumes a single-path reasoning trajectory, which means the model cannot revisit a previous state. In some challenging cases, models may make mistakes at a particular stage. Without a mechanism to revert to previous states, the model may never find a correct solution. Therefore, the proposed approach may excel at easy tasks that can be solved in one direction, but not those that require searching multiple routes.\n\n3. **Disentangle the impact of lack of reasoning and lack of in-domain training samples.** From the results of counting in Table 1, Molmo achieves the top-1 performance without any reasoning process. The high performance of Molmo mainly comes from the high-quality counting training data. This reveals that the performance on some of those visual tasks can be enhanced by simply including in-domain training data. This may be applied to other tasks evaluated in this paper. For example, if we train the model to solve Jigsaw, do we really need inference time reasoning? Consider the higher cost of inference time scaling, maybe collecting training samples could be a more efficient way. To this end, adding experiments that demonstrate the limitations of scaling training data on certain tasks could help the community understand the necessity of inference time scaling for visual tasks.\n\n4. **Evaluate on existing benchmarks.** Most of the tasks evaluated in the paper were designed by the authors, which makes the results less convincing. The counting benchmark is only a top-down view and is less likely to have cases like overlap, which could happen frequently in real photos. Including the results on existing benchmarks and directly comparing against numbers from prior works could make the results more solid. Some tasks to consider: Pixmo-count (Deitke et al), where2place (Wang et al).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to improve the visual reasoning ability of Multimodal Large Language Models (MLLMs). They argue that scaling textual reasoning alone is insufficient to solve visual tasks like counting. This paper proposes the autonomous imagination, which allows MLLMs to iteratively modify the visual inputs with pre-defined operations, including focus, transform, and ignore. The proposed approach is evaluated on four types of tasks: counting, jigsaw puzzles, object placement, and multi-object hallucination. The experiment results show that the proposed approach can enhance the performance of MLLMs on those tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Evaluate on more tasks, including some general vision-language tasks, existing benchmarks for counting, placing, etc.\n2. Disentangle the impact of the lack of reasoning and the lack of in-domain training samples by including some fine-tuned models as baselines.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wS8sQZCaeS", "reviewer": "Reviewer_KeX5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I like the motivation of this paper, using some image operations during the inference time, for possible complicated visual tasks. The proposed framework is composed of reasonable components (while they are a bit ad-hoc IMO, please refer to the below). I appreciate the math illustration in Section 3.2. \n\nThrough experiments, they show that the proposed pipeline bring major improvements compared to GPT-4o and o1. \n\n\n---\n\nRegarding the weaknesses, I feel the design in this paper is somewhat ad-hoc. For example, as mentioned in Section 3.2, “we directly provide the MLLM with task-specific reasoning plans (e.g., ‘count by ignoring balls’) through system prompts.” Such approaches seem to benefit only very specific tasks rather than providing a generalizable design.\n\nAdditionally, the provided examples—“How many balls are there in this image?” and “The guest will sit on the right side of the table, near the yellow cup. Where should we serve the apple?”—are rather straightforward. These tasks do not appear to require the complicated autonomous imagination approach proposed.\n\nSo does the evaluation, the tested tasks \"counting, \"simple jigsaw puzzle\", and \"object placement\" are very task-specific. I wonder if the authors provided separate, hand-tailored prompts for each individual task? Also, there are many other great open-source VLMs, such as QWen and UI-TARS. Can the author also try them? Finally, why only testing on openai-o1 not openai-o3?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper addresses challenges faced by VLMs in performing intuitive visual reasoning tasks due to insufficient decomposition of the visual-to-textual conversion process. It convincingly argues that perceptual bottlenecks in VLMs cannot be overcome merely by scaling inference-time reasoning. To address this, the authors propose an approach named \"autonomous imagination,\" allowing iterative modifications of visual inputs to generate intermediate visual states. This closed-loop visual modification effectively decomposes complex visual tasks into manageable substeps, enhancing performance without retraining. Experimental results show the effectiveness of this approach across challenging tasks, such as counting and jigsaw puzzle solving.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can you address the concerns raised in the above weakness section? My major concern would be the ad-hoc image operation design and possibly applying the framework in general benchmarks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)