Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models
Authors: Yu Zeng
Wenxuan Huang
Shiting Huang
Xikun Bao
Yukun Qi
Yiming Zhao
Qiuchen Wang
Lin Chen
Zehui Chen
Huaian Chen
Wanli Ouyang
Feng Zhao
Venue: ICLR 2026 Poster (2026)
Keywords: VLMs
Visual Understanding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Although current large Vision-Language Models (VLMs) have advanced in multimodal understanding and reasoning, their fundamental perceptual and reasoning abilities remain limited. Specifically, even on simple jigsaw tasks, existing VLMs perform near randomly, revealing deficiencies in core perception and reasoning capabilities. While high-quality vision-language data can enhance these capabilities, its scarcity and limited scalability impose significant constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction Learning for Enhancing visual perception and reasoning in VLMs. AGILE formulates jigsaw solving as an interactive process, enabling the model to progressively engage with the environment. At each step, the model generates executable code to perform an action based on the current state, while the environment provides fine-grained visual feedback to guide task completion. Through this iterative cycle of observation and interaction, the model incrementally improves its perceptual and reasoning capabilities via exploration and feedback. Experimental results show that AGILE not only substantially boosts performance on jigsaw tasks of varying complexity (e.g., increasing accuracy from 9.5\% to 82.8\% under the $2 \times 2$ setting) but also demonstrates strong generalization across 9 general vision tasks, achieving an average improvement of 3.1\%. These results indicate notable enhancements in both perceptual and reasoning abilities. This work opens a new avenue for advancing reasoning and generalization in multimodal models and provides an efficient, scalable solution to the scarcity of multimodal reinforcement learning data. The code and datasets is available at https://github.com/yuzeng0-0/AGILE.

Published as a conference paper at ICLR 2026
AGENTIC JIGSAW INTERACTION LEARNING FOR EN- HANCING VISUAL PERCEPTION AND REASONING IN VISION-LANGUAGE MODELS
Yu Zeng",, Wenxuan Huang Shiting Huang', Xikun Bao', Yukun Qi', Yiming Zhao',
Qiuchen Wang', Lin Chen', Zehui Chen', Huaian Chen', Wanli Onyang? Feng Zhao If
University of Science and TechnolZengogy of China
Shanghai AI Laboratory
3 East China Normal University
"The Chinese University of Hong Kong
ABSTRACT
Although current large Vision-Language Models (VLMs) have advanced in mul- timodal understanding and reasoning, their fundamental perceptual and reasoning abilities remain limited. Specifically, even on simple jigsaw tasks, existing VLMs perform near randomly, revealing deficiencies in core perception and reasoning capabilities. While high-quality vision-language data can enhance these capabili- ties, its scareity and limited scalability impose significant constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction Learning for Enhancing visual perception and reasoning in VLMs. AGILE formulates jigsaw solving as interactive process, enabling the model to progressively engage with the envi- ronment. At each step, the model generates executable code to perform an action based on the current state, while the environment provides fine-gratee fine-grained visual feedback to guide task completion. Through this iterative cycle of observation and interaction, the model incrementally improves its perceptual and reasoning capabilities via exploration and feedhack. Experimental results show that AGILE not only substantially boosts performance on jigsaw tasks of varying complexity (e.g., increasing accuracy from 9.5% to 82.8% under the 2 x 2 setting) but also demonstrates strong generalization across general vision tasks, achieving an av- erage improvement of 3.1%. These results indicate notable enhancements in both perceptual and reasoning abilities. This work opens a new avenue for advancing reasoning and generalization in multimodal models and provides an efficient, scal- able solution to the scarcity of multimodal reinforcement learning data. The code and datasets is available at ltttps://github htt.u//...oomuuuoooo//AALLL http:////gtthtbbooo///uu////////
INTRODUCTION
Large Vision-Language Models (VLMs) have recently achieved remarkable success across wide
range of multimodal tasks, including image captioning (Chen al. 2024a c; Li et al., 2024b Zeng
et al. (2025). visual question answering (Bai et al. [0225] Zhu et al. 2025
Hurst et al., 2024 Co-
manici al. 2025 Qi al. 2025 Zhao et al. 2025 Huang et al. 2026b] Zeng et al. 2026 Wang
et al. 2026 Pan et al. (225)) document understanding (Wang et al. 2025b a), and complex real- world applications (Huang al. 2025b Han et al. 2026] Fang et al.) [025)] Huang al.. 2025a Zhang et al.] 2025a). By effectively associating visual and textual information, these models ex- hibit strong multimodal perception and reasoning capabilities. However, their performance remains severely limited on tasks that require comprehensive visual understanding and structured reasoning. In particular, we find that current VLMs often perform near random even on relatively simple 2 x Chen et al. 2023), which demand both perceptual accuracy and
jigsaw tasks CCrllucci et al. 2019
Equal Contribution
t Corresponding author

---

Published as a conference paper at ICLR 2026
(a) Swap & Observe
(b) Crop
(c) Zoom
Region of Interest book = [d. yl. x2, y2)
TIZAL
CHOOSE YOUR
HAROHARTA
WHTE
B
CLASICA
RD
Swap(B. C)
Observe(A, B,
Crep(bbox)
Zoom(bbox)
Figure 1: Description of the action space. (a) illustrates swapping two jigsaw pieces and observing the updated jigsaw state; (b) shows cropping a specific region of the jigsaw for closer inspection; and (c) depicts zooming into a selected area to examine fine-grained details.
logical inference. This suggests that existing pretraining and fine-tuning strategies are insufficient for developing robust perceptual and reasoning abilities.
Recent studies have explored reinforcement learning (RL) (Schulman al. 2017 Rafailov et al.
2023 Shao et al., (224,) Huang ct al. 2026al Ma et al., 2025 Zhang et al. [222cc] b Ren et al.) (0266
as a way to enhance these capabilities by enabling models to learn through interaction, trial-and- error, and feedback. While RL-based approaches show promise, they are fundamentally constrained
by the scarcity and limited scalability of high-quality vision-language RL data. Current methods for constructing multimodal RL datasets (Huang et al. 2025c Yang et al. 2025 Lu et al. 2023)
generally fall into two categories: human expert supervision and automated synthesis. Human- supervised datasets either prohibitively expensive or too small in scale for furge-scale training, while automated approaches that rely on closed-source models suffer from limited quality, capability constraints, and substantial API costs. These challenges collectively hinder the development of VLMs with strong reasoning and generalization abilities.
To address these limitations, we propose AGILE, Agentic jiGsaw Interaction Learning frame- work for Enhancing visual perception and reasoning in VLMs. Our approach leverages the struc- tured nature of the jigsaw puzzle a proxy task for perception and reasoning, modeling the jigsaw- solving process as a progressive interaction between the model and its environment. At each step, the model generates Python code to perform actions within well-defined action space (swapping two image tiles, observing the current jigsaw state, cropping regions for detailed observation, or zooming in for fine-grained analysis). This interaction-driven process enables the model to iter- atively refine its perceptual and reasoning capabilities while receiving explicit feedback at every step. By simulating the step-by-step dynamics of jigsaw solving, the model is encouraged to capture structural relationships among visual components and acquire more robust perception and reasoning skills.
A key feature of our approach is the use of code and rule-based data generation, which offers two major advantages. First, the difficulty of the jigsaw task can be precisely controlled by adjusting factors such the number of correctly placed tiles in the initial state and the overall jigsaw size. Second, because the ground-truth solution is inherently available, the synthetic dataset can be scaled arbitrary sizes with strict supervision, overcoming the data scarcity inherent in aanott-d or closed-source datasets. The combination of interactive training and scalable data generation yields an efficient and effective framework for advancing visual perception and reasoning in VLMs.
We validate the effectiveness of our approach through comprehensive experiments. Our method substantially improves performance across jigsaw tasks of varying complexity, for instance, raising accuracy from 9.5% to 82.8% on the 2 x 2 setting, while also demonstrating strong generalization to a wide range of vision tasks, including high-resolution image understanding, real-world scene analysis, fine-grained recognition, visual reasoning, and hallucination benchmarks. Moreover, we show that scaling the training data further boosts performance, and that under equal data budgets, ibgsaw-based training achieves results comparable to or even surpassing those obtained with gen- eral QA data. These findings underscore the potential of jigsaw tasks in addressing the scarcity of multimodal RL data. By combining interactive training with scalable data generation, significantly enhances both perceptual and higher-order reasoning abilities, pointing to a promising new direction for advancing VLMSs Our main contributions are follows:
method
2

---

Published as a conference paper at ICLR 2026
We introduce AGILE, agentic jigsaw interaction learning framework that formulates jigsaw solving as stepwise interactive process, thereby driving incremental improvements in both visual perception and reasoning capabilities of VLMs.
We present a scalable jigsaw-based data generation method that yields high-quality mul- timodal reinforcement learning datasets with controllable difficulty, providing efficient solution to the current shortage of high-quality training data.
Extensive experiments demonstrate that our approach substantially improves performance on jigsaw tasks of varying complexity while exhibiting strong generalization across diverse vision benchmarks, highlighting its effectiveness in enhancing both perception and reason- ing in VLMs.
RELATED WORK
Reinforcement Learning for Vision Language Models. Reinforcement learning (RL) (Schulman
et al. 2017 Rafailov et al. 2023, Shao et al. 2024) has been widely applied to improve reasoning in language models, as exemplified by the success of DeepSeek-RI (Guo et al. (025) in mathematical reasoning. Building on this progress, recent studies have extended RL to VLMs, with sule-baser RL in multimodal domains emerging as promising direction. For perception enhancement, RI-V
(Chen et al. 2025b) applies RL to object counting, while Perception-R1 1 Yu et al. 2025) leverages object matching and IoU as reward signals to improve grounding. DeepEyes (Zheng et al. 2025) shows how RL can encourage models to invoke visual tools, thereby expanding perceptual abilities. For reasoning. MMEureka (Meng et al., 2025) demonstrates the effectiveness of ruse-based RL in mathematical tasks. From data perspective, Vision-R1 (Huang et al.) 2025c) and R1-One (Yang et al. 2025) convert visual information into textual representations to build multimodal chain- of-thought (CoT) datasets that support stronger reasoning. Despite these advances, the lack of scal- able, high-quality RL datasets remains fundamental bottleneck, severely limiting further improve- ments in both perception and reasoning for VLMs.
Enhancing Generalization via Proxy Tasks. Rule-based reinforcement learning (RL) has shown promise but typically requires large amounts of high-quality, verifiable data. Logic-RL (Xie al., 2025a) addressed this by introducing the "Knights and Knaves" (K&K) puzzle, which offers con- trollable difficulty and algorithmically generated ground truth, enabling models acquire reasoning skills transferable to mathematical tasks. Enigmata (Chen et al. 2025a) extended this approach to broader set of text-based puzzles (e.g., cryptographic, arithmetic, logical), demonstrating that solv- ing such tasks with RL improves general reasoning without external data. RPT (Dong et al.) (025) further reframed next-token prediction as a reasoning task, using verifiable rewards enhance pre- dictive ability and strengthen the basis for reinforcement fine-tuning. Inspired by these proxy-task successes in LLMs, recent work has begun exploring similar ideas for VLMs. Code2Logic (Tong et al., 2 and ViGaL (Xie al.) 2025b) leverage code-synthesized games to improve mathe- matical reasoning, while ViCrit (Wang et al. 2025d) enhances perceptual robustness by reinforcing models to detect hallucinated entities via modified captions. Jigsaw-R1 (Wang et al. 2025c) pro- posed puzzle-based RL paradigm, but due to training limitations, its models still performed poorly on 2 jigsaw puzzles, failing fully exploit the proxy-task benefits. To address this gap, propose a agentic jigsaw interaction learning framework, which casts jigsaw solving as an interac- tive process between the model and environment. This enables iterative refinement of perception and reasoning with explicit stepwise feedback, allowing the model to better capture visual relationships and develop stronger perceptual and reasoning skills.
METHOD
In this section, we provide comprehensive description of our proposed agentic jigsaw interaction learning framework. We first introduce the jigsaw task and describe how the model interacts with the environment accomplish it (Sec. 3.1), then present the construction of the training data (Sec. 3.2). Finally, we outline the training paradigm, including both the cold-start stage and the reinforcement learning stage (Sec. 3.3
3

---

Published as a conference paper at ICLR 2026
(a) Agentic Jigsaw Interaction Reinforcement Learning Framework
Rollout Process
Trajestory
Advantage
Update
Swop
Grou
Jigaaw
Policy Model
Reword Function
A
Grop & Zoom
R Roc Ryows
Racc
Aa
(b) High-Quality Trajectories for Cold-start
(c) Interaction with Environment
Trajectory generate
Humen Refine
Policy Model
Tayk
Fecabeeti
Refine
Samini
Python
Cold-stort
GT-based filtering
Condidate
<Thslo
<aide
Fendbeelo
<Thmke
cAnower
GAA
Correct Step
Round N-l
Round N
jacts
Erer Annner
Python Intergeeter
Atnaer
Updete
Figure 2: Overview of the AGILE framework. (a) depicts the interaction process between the
model and the external environment, together with the implementation of the GRPO algorithm: (b) shows the collection of high-quality jigsaw trajectory data; and (c) illustrates the model cndel-ennironment interaction during the jigsaw rollout process.
3.1 JIGSAW TASK AND ENVIRONMENT INTERACTION
Jigsaw Task. Given an input image, we partition it into an X grid of jigsaw pieces, where task difficulty can be flexibly adjusted by varying m. If the image height or width is not divisible by m, the image is resized so that its dimensions are exact multiples of the grid. The grid is then randomly shuffled, and each piece is assigned an index in row-major order ranging from 1 (top-left) to m² (bottom-right). The shuffled configuration is denoted
Ishuf {11 Li. m
(1)
where each 1 corresponds to jigsaw block. Since the shuffling process is explicitly recorded during data generation, we can recover the ground-truth jigsaw layout as
IGT = (L(1)) (.( (m(m)))
(2)
where permutation over (1,2, current state
........m?) m??. During jigsaw solving, the model will maintain
Istate (1) (2) LI (m²)),
(3)
where n" denotes the current arrangement of pieces. At each step, the model iteratively swaps two pieces, observes the resulting configuration, and aims to reconstruct the ground-truth layout IGr-
Environment Interaction. The model interacts with the environment throughout the jigsaw- solving process in an iterative manner. As illustrated Figure I we predefine set of API for modellenvironment interactions in Python. The model expresses its actions by generating Python code, which executed by the environment to produce the corresponding resulting image. The model then uses these results to reason further and decide subsequent actions, repeating this pro- cess iteratively until the jigsaw completed. Specifically, at each step, the model can perform the following actions:
Given the current jigsaw state Istates the model calls the Swap function to exchange the positions of any two jigsaw pieces.
Swap:
Observe: Given States the model calls the Observe function to obtain the current jigsaw
progress Tobs which is then used determine the next action.
4

---

Published as a conference paper at ICLR 2026
Crop & Zoom: Given Tobe. the model crops and zooms into local region to inspect finer details and inform subsequent actions.
To better illustrate the overall procedure, the entire rollout process is presented in Algorithm| !
Algorithm Interactive Jigsaw-Solving Procedure of VLMs with the Environment
Input: Query æ, fie Policy model ne. External environment V, Maximum iteration steps T. Output: Final interaction trajectory y.
I: Initialize trajectory y 4- 0, step counter & 4- O.
2: while t < T do
3: Sample response yı :T( æ, Istuf fie: y).
Append assistant response ye to trajectory: y f- y + ye- <code> </ code> detected Y/ then
6: Parse the Python code (e.g., Swap, Ohserve, Crop. Zoom), execute in the environment V, and obtain feedback O..
7: else if <answer> </ </answer> detected in y: then return final trajectory y.
9:
end if
10:
Append environment feedback O, as user input: + + O..
11: Update step counter +- + 1.
12:
end while
13:
return final trajectory y.
3.2 DATA CONSTRUCTION
we have introduced the jigsaw task and described how the model interacts with the environment to complete it. Based on this task design, we observe that the model (e.g., Qwen- 2.5VL-7B) exhibits limited foundational capabilities, manifested in poor instruction-folloiing and incorrect Python code generation, which hinders proper interaction with the environment and intro- duces substantial training noise. Directly applying reinforcement learning (RL) thus results in low learning efficiency. To bridge this gap and more effectively train our model, we employ Gemini 2.5 Pro to collect expert trajectories for the cold-start phase, endowing Qwen-2.5VL-78 with ba- sic interactive jigsaw-solving capabilities. For this purpose, we design structured prompts to guide Gemini 2.5 Pro in interacting with the environment to complete the jigsaw tasks (The prompt tem- plates are provided in Appendix B
In Section 3.1
To further ensure data quality, apply additional filtering process: first, selecting samples where Gemini's outputs match the ground truth, and then manually verifying each interaction step in the correct samples to ensure rationality and consistency. To ensure that our model can perform diverse jigsaw reasoning actions during reinforcement learning, we carefully balance the training data. Specifically, trajectories are balanced with respect to both the number of steps (4-8) and the types of actions involved (e.g., Swap. Observe, Crop, Zoom). This design ensures that the model sufficiently exposed to full action space and learns to handle various interactions with the environment effectively. Overall, we curate dataset comprising 1.6K high-quality reasoning trajectories for the jigsaw task.
3.3 TRAINING PARADIGM: COLD-START AND REINFORCEMENT LEARNING
Cold-start. During the cold-stan start phase, we leverage the 1 1.6K high-quality jigsaw-solving trajecto- ries constructed in Section 3.2 to equip the model with basic esstuctioonfollowwe and Python code generation capabilities, ensuring effective interaction with the environment.
Reinforcement Learning. Our framework adopts Group Relative Policy Optimization (GRPO), which utilizes the average reward of multiple sampled outputs baseline, instead of relying learned value function. The policy model optimized by maximizing the following objective:
'The version is Gemi ni - www.roooeeeieeewOS-...
5

---

Published as a conference paper at ICLR 2026
Iwdl
1
To(
Mi,,s V)
[o) = BAND ( (aa(lle
min
Ar,,,
- I(si,e) (m1:((rr))))
Tota N1, V)
x, Mr V) Told (Vs, I; (1,ce: 1 1 1 d.e)
To(yi,t
clip
DEL((T (TeE).
where the rollout module samples group of trajectories {yi, M2......................... yG from the old policy Fold for each input question x through interaction with the external environment V. The advantage term As,t is computed based on the relative rewards of outputs within each group. Our reward system comprises three components:
accuracy reward, format reward, and a step reward. The total reward is computed as the sum of these components.
Accuracy Reward: We compare the model's 's generated answer Lnnwer with the ground truth IGr. If all jigsaw image blocks are correctly placed, the accuracy reward is 1; otherwise, is O.
The model receives reward of if its output follows the required structured format, with the reasoning process, code, and final answer correctly enclosed within the <think>, <code>, and <answer> tags, respectively.
Format Reward:
Step Reward: We encourage the model to complete the jigsaw in as few steps as possible. For 2 x 2 jigsaw, if each step swaps any two image blocks, most three steps are theoretically required to place all blocks correctly. Excessive actions may lead the model to perform invalid steps, reducing the proportion of effective perception and reasoning. Furthermore, model from prematurely hacking the step reward during early RL training, the step reward is applied only when the jigsaw correctly completed. If the jigsaw incorrect, the model is penalized by assigning the maximum step penalty. Formally, the step reward is defined as:
prevent the
Rstep = - (T(RRnu=1) stepuum + [(Rnc==0) slepmax
(5)
where stepaum is the number of steps the model actually used to complete the jigsaw, step is the maximum allowed steps, denotes the step penalty coefficient, which is set to -0.05, and I() is the indicator function. The final reward formulation is shown in Equation 6
R = Race + B Riormat + 7
Rsteps
(6)
where the coefficients a, 8, and y weight the relative importance of accuracy, format, and step rewards, respectively, In our experimental setup, a, B, and y are set to 0.8, 0.2, and 1.0, respectively.
4 EXPERIMENT
In this section, we present a comprehensive experimental study. We first describe the implementation details, including datasets, training procedures, and inference settings (Sec. 4.1). We then evaluate the effectiveness of our agentic jigsaw interaction learning framework on both jigsaw-solving and
general vision tasks (Sec. 4.2p. Finally, we perform ablation studies to assess the impact of jigsaw
data scale and to contrast the benefits of jigsaw data with those of general QA data (Sec. 4.3
4.1 IMPLEMENTATION DETAILS
Datasets and Models. Our training data consist of two components corresponding to the cold-start and reinforcement learning (RL) stages, In the cold-start stage, we employ 1.6K high-quality jigsaw puzzle trajectories collected in Sec. 3.2 to endow the model with basic interactive jigsaw-solving skills. In the RL stage, we construct a dataset of 15.6K images spanning diverse domains, including high-resolution visual search, OCR-based text recognition, real-world scenes, and structured dia- grams (see Appendix A for details). Each image partitioned into 2 2 patches and randomly shuffled to ensure that all patches are initially misplaced. All experiments are conducted using Qwen2.5-VI.-BB (Bai et al. (2225) as the base model.

---

Published as a conference paper at ICLR 2026
Table 1: Jigsaw Ace result. LN indicates the difficulty level, where N denotes the initial number of correct pieces. A smaller N corresponds to more scrambled jigsaw and higher difficulty, The best results are highlighted in bold, and the second-best results are underlined.
2 x 2
3 X
Model
L0
L.2
Avg.
L0
L1
L2
L3
L5
Avg.
Random
4.5
3.7
4.2
4.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
GPT-4o
38.7
37.7
47.0
41.1
1.0
1.3
2.7
3.0
3.0
7.0
7.7
13.7
4.9
Gemini-2.5-Pro
43.3
46.3
49.7
46.4
7.0
8.7
9.7
10.0
11.0
15.0
23.0
32.0
14.6
MiMo-VL-7BA.n
11.3
14.3
17.0
14.2
0.3
0.0
0.0
0.0
0.3
3.7
0.7
InterV
3.7
3.7
5.0
4.1
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
IntemVLL3-78B
3.3
4.3
3.0
3.5
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Qwen2.5VL-72h)
22.7
24.3
35.3
27.4
0.3
1.3
2.3
2.7
2.0
3.0
6.7
12.7
3.9
Qwen2.5VL-7B
6.3
6.0
16.3
9.5
0.0
0.0
0.0
0.0
0.0
0.7
1.3
1.3
0.4
+Cold-Start
12.0
32.0
22.0
22.0
0.0
0.3
0.0
0.0
0.0
0.7
0.3
0.0
0.2
+RI. 78.7
83.0
86.7
82.8
5.0
7.0
11.0
14.0
17.7
25.3
38.0
48.0
20.8
Training and Inference Setups. We conduct supervised fine-tuming (SFT) on llam--fcctrr (Zheng et al. 2024) and reinforcement learning (RL) training on verl (Sheng et al. 2024). both with full- parameter tuning. For inference and evaluation, we adopt VI.MEvaliit (Duan et al.) (002) as the framework. To ensure fair comparison, all evaluations general downstream benchmarks conducted in strict single-turn setting following VLMEvalKit, fully consistent with the evaluation protocol of all baseline models. All experiments performed 8 NVIDIA A100 GPUs with 80GB memory each. Detailed training hyperparameters are provided in the Appendix
4.2 MAIN RESULTS
Significant Improvements the Jigsaw Task. To comprehensively evaluate performance the jigsaw task, we curate a test set of 300 images spanning diverse scenarios, including high-resolution visual search, OCR-based text recognition, and real-world scenes. Each image is partitioned into 2 x 2 and 3 x 3 grids, and the performance is measured using two metrics: Acc, which equals only when all patches are correctly placed, and Score, defined as the ratio of correctly placed patches to the total number of patches. As shown in Table 1 and 2 the base model VLL 7B performs poorly without training (achieving only 9.5% accuracy even under the simplest 2 setting) while the proprietary Gemini2.5-PPP and the larger Qwen2.5-VL--22B models also struggle on this task. After supervised fine-tuning (SFT) with 1.6K cold-start trajectories and reinforcement learning (RL) with 15.6K images, Qwen2.5-VL-7B achieves substantial gains: on the 2 x setting, Ace improves from 9.5% to 82.8% and Score from 29.4% to 89.0%; on the more challenging x setting, it also generalizes well, with Acc increasing from 0.4% to 20.8% and Score from 31.1% to 62. 96, significantly surpassing Gemini2.5- that modeling jigsaw solving as interactive multi-turn dialogue enables the model progressively enhance its visual perception and reasoning abilities, thereby achieving superior performance on the jigsaw task.
and Qwen2.5- V---22. These results demonstrate
Generalization to Downstream Visual Tasks. The proposed agentic jigsaw interaction learning framework delivers substantial improvements on the jigsaw task. By leveraging explicit step-by-step feedback for iterative refinement, it enables the model to capture visual relations more effectively and to develop stronger reasoning capabilities.
To further assess whether jigsaw training enhances performance on general vision downstream tasks, we evaluate the model on benchmarks: high-resolution image understanding (HRBench4K (Wang et al. (225cc). HRBhhhh8K (Wang et al. (022cc). VStarBench (Wu & Xie (024)))), real-world scene understanding (MME-R RealWorld (Zhang et al. 2024), RealWorldOA (xAll 2024)), fine-grained visual recognition (MMVP (Tong al. 22224) BLINK (Fu et at. al. 2024)), visual reasoning (MMMU Yue et al. 2024), ) and hallucination evaluation (HallusionBench GGun et al. 2024h). As shown in Table the jigsaw trained model demonstrates strong visual generalization, achieving notable gains on HRBench4K (+4.2%), HRBench8K (45.2%) and VStarBench (+4.2%). On average, surpasses the base model Qwen2.5-VL-778 by 3.1 across all 9 benchmarks, providing compelling evidence

---

Published as a conference paper at ICLR 2026
Table Jigsaw Score result. LN indicates the difficulty level, where denotes the initial number of correct pieces. A smaller N corresponds to a more scrambled jigsaw and higher difficulty. The best results are highlighted in bold, and the second-best results are underlined.
Model
x
3
L0
L.2
Avg.
L0
L1
L2
L3
L4
L6
L7
Avg.
Random
25.2
24.7
24.8
24.9
11.4
11.3
11.0
10.9
11.1
11.4
11.3
10.8
11.2
GPT-4o
54.8
59.1
63.2
59.0
25.9
27.3
34.1
36.3
42.3
48.0
54.4
64.0
41.5
Gemini-2.5-Pro
55.3
60.1
61.6
59.0
31.9
31.8
35.7
39.4
43.1
48.7
61.8
68.6
45.1
MiMo-VLL-7B-K
21.8
27.2
29.2
26.1
5.5
6.9
8.0
10.7
15.9
19.5
20.6
28.9
14.5
LL3-mBL3-8B
24.6
24.3
25.3
24.7
7.5
10.1
15.7
16.8
20.3
25.1
28.2
31.7
19.4
IntemVLL3-78B
21.3
26.5
28.4
25.4
7.1
10.9
14,1
18.5
23.1
25.7
29.0
37.0
20.7
Qwen2.5VL-72B
40.7
46.1
56.1
47.6
21.7
25.6
30.8
33.4
34.7
40.0
46.3
$5.2
36.0
Qwen2.5VI--7B
14.4
30.1
43.8
29.4
5.8
12.0
20.3
25.6
32.7
40.9
50.2
60.9
31.1
+Cold-Sturt
40.6
45.6
45.3
43.8
6.5
6.9
7.8
10.2
10.4
13.2
15.7
17.0
11.0
+RL
86.4
88.7
91.9
89.0
41.1
46.8
52.6
58.6
65.3
71.0
77.9
83.6
62.1
Table 3: Main results. Performance comparison of different models on the 9 benchmarks, Ab- breviations: MME-RW HRB8K (HRBench&K), HalBench (HallusionBench), MMMU (MMMU. VAL).. Avg. denotes the average performance across all 9 benchmarks. A represents the relative performance gain achieved by RL compared to the base model Owen2.5-VL-7B. The best results are highlighted in bold, and second-best results are underlined.
RWQA (RealWorldQA), HRB4K (HRBench4K),
Model
MME-RW
RWQA
HRB4K
HRBSK
VStar
MMVP
BLINK
HalBench
MMMU
Avg.
1.vVA-OV-78
48.4
69.5
65.3
58.4
73.3
77.3
52.6
36.6
48.2
58.8
VL2.5BL2..58B
48.2
69.4
68.0
63.3
75.7
$4.9
$3.6
61.6
Inter VL.255 78B
49.7
78.4
74.5
72.5
75.9
83.0
63.6
57.1
65.4
68.9
Qwen2.5-VL-22
44.7
75.3
80.1
77.1
85.9
81.7
61.6
53.5
66.9
69.6
Qwen2.5-VL-7B
44.6
68.5
68.8
65.3
76.4
74.3
56.4
50.1
54.8
62.1
Cold-Start
46.2
68.4
21.0
68.4
27.5
26.2
55.7
49.8
$4.0
63.1
+ RL
48.4
70.2
73.0
70.5
80.6
78.0
58.0
51.9
55.8
65.2
Qwen2.5-VL-7B)
+3.8
+5.2
+3.7
+3.1
that jigsaw-bases training effectively enhances the model's ability to capture visual relations and strengthen reasoning skills, thereby improving its performance on general vision downstream tasks.
4.3 ANALYSIS
Impact of Jigsaw Training Data Scale. To investigate the effect data scale, we systematically vary the amount of training data and evaluate model performance both jigsaw task and gen- cral vision tasks. As shown in Figure scaling up the data yields substantial performance gains: jigsaw task accuracy increases from 22.0% to 82.8%, while HRBench4K and RealWorldOA improvements of 2.0% and 1.8%, respectively. These results highlight a clear data-to-performance trend, demonstrating that larger training sets translate directly into stronger perceptual and reason- ing abilities. Crucially, because jigasw data in our reinforcement learning stage are generated via scalable programmatic synthesis, AGILE can easily leverage larger datasets. This scalability only provides richer learning signals for continual capability growth but also offers a practical and sustainable solution to the scarcity of
multimodal RL data.
Performance Comparison: Jigsaw vs. General QA Data. We compare the effectiveness of jigsaw data against conventional General QA data for model training. As shown in Figure models trained with jigsaw data consistently outperform those trained with QA data across multiple benchmarks. Notably, combining 10K jigsaw samples with 10K QA samples yields superior performance on gen- eral vision benchmarks compared to training with 20K QA samples alone. These results highlight the unique role of jigsaw data in enhancing fundamental visual generalization, demonstrating the efficiency and controllability of jogsaw-solving key advantage of our approach lies the programmatic synthesis of jigsaw data, which enables effortless construction of large-scale, high-quality datasets, process that is otherwise costly and
proxy for visual perception and reasoning. A
8

---

Published as a conference paper at ICLR 2026
MME RealWardd
90
80
BLINK
47.5
RudWorldQ@
720
70
Jigbon (%)
42.5
Acc (5)
55
54
66
700
40
66
RedlWoridQA HRBenchAK Tigion
30
Vhoanch
HRBerch4K
68.0
20
40
80
120
160
200
240
Training Steps of RL
Qwnad 541-78
HRBercchK
S1L-2551L-18 5VL-78 (Senend Gut 201)
Figure 3: Impact of Training Data Scale. The left y-axis denotes the accuracies HRBench4K and RealWorldQA, while the right y-axis corresponds to the accuracy the jigsaw task.
Guost 5st, (Renand QA 10k Tigbew (05)
Figure 4: Comparison with General QA Data. The total number of samples is consistently maintained at 20K across both experimental se- tups.
labor-intensive for QA data. Thus, jigsaw data not only serves effective alternative to General QA data but, some cases, proves to be the superior choice. This finding underscores the poten- tial of jigsaw tasks in alleviating the scarcity of multimodal RL data and opens promising new direction for advancing multimodal model development.
Image Block Content Perception
Edge Alignment Perception
.. [skippad]
<tiinlo... [skipped)
The large shadow graphic is split between blocks
dispenser, Image Shows gray
which diagonally placed, they clearly don't align. To verify clear misalignment, I will crop the central area of the image .. [skipped] </thinlo
part of a wooden cabinet, and on the right, the top of pineapple fruit arrangement. Image Shows wooden table with trays of sandwiches and glasses of solod, The text "RV-DERBY-UK"
MADS
<thinlo The cropped image 'crop_image_ confirms the severe misalignment. The left side shows the shadow from block block C. The right side shows text from block 8 and from disrupted [skipped][/thinie
Image C: Shows doorway, wooden structure, and the fruit arrangement the left.
Image b: Shows table with polke-dot tablecloth, plates of food, and the text TBDB VEND -E" [skipped] </thinlo
continuity
Image Block Positional Reasoning
Detail Positional Reasoning
<thinlo [skipped]
<thinio [skipped)
Vertical Pairs: The images can be paired
The dark background the right side of Image and D contrasts sharply with the bright whife sheives on the left. I will focus these two parts .. [skipped] c/thinlo
vertically by animal. Image (kitten's head)
should placed directly above. Image b
(kitten's 's body). Similarly, Image (puppy's head) should placed (puppy's body).
Image B
image_ 1' clearly shows brightly colored, highly recognizable brand logos, including orange "Reese's" and the blue-red "SUPER BLOW POP" [skipped] "crop_image__
<thinko "crop, "crop_image__]
should consist of the heads of the animals side-by-side, forming title. So, Image should be next to Image C. The bottom should consist of their bodies in the basket, so Image should Image B. The woven basket pattern should continue seamlessly from
Horizontal Pairs: The
similarly styled candy logos, including "Hershey's", "Twizzlers", and "PayDay". We infer that the walls connected, together forming complete brand- wall [skipped] </thinlo
+
[skipped] </thinlo
Figure 5: Case Study. Jigsaw-solving reasoning and behaviors exhibited by our model.
Case Study. In Figure 5 we showcase several jigsaw-solving reasoning patterns and behaviors hibited by our model. These cases illustrate the emergence of high-quality perceptual and reasoning strategies in the jigsaw task, including comprehensively interpreting the visual content of individ- ual pieces to infer their spatial relations, employing cropping and zooming to examine and validate edge alignment, and reasoning about semantic consistency across pieces. Such behaviors demon-

---

Published as a conference paper at ICLR 2026
strate human-like reasoning, thereby effectively enhancing the model's perceptual and reasoning capabilities.
CONCLUSION
In this work, we propose a novel agentic jigsaw interaction learning framework that formulates jigsaw solving an interactive process enhance visual perception and reasoning in large Vision- Language Models (VLMs). By treating jigsaw solving as an iterative interaction, the model progres- sively refines its perceptual and reasoning capabilities through exploration and feedback. Our ap- proach significantly improves performance on jigsaw tasks of varying complexity and demonstrates strong generalization broader general visual tasks, including visual question answering and scene understanding, Furthermore, by analyzing the performance gains from increasing the scale of jig- saw data and comparing it with general QA data, we show that jigsaw serve an effective proxy for alleviating the scarcity of high-quality RI data, highlighting the potential of task-driven proxy training for stimulating complex multimodal perception and reasoning in VL.Ms.
ACKNOWLEDGEMENTS
This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2 2188085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Sci- ence and Technology Institution, USTC. The Al-driven experiments, simulations and model training were performed on the robotic Al-Scientist platform of Chinese Academy of Sciences.
ETHICS STATEMENT
This work adheres to ethical research standards in data collection, model training, and evaluation. All datasets used in this study are publicly available research datasets (c.g., COCO, TextVQA, HRBench, Real WorldQA,, which were collected and released under their respective licenses. No private or personally identifiable information (PII) was used.
REPRODICIBILITY STATEMENT
We are committed to ensuring the reproducibility of our results. Comprehensive implementation details, including training data and hyperparameter settings, provided in Appendices and D To further facilitate reproducibility, we have released the training code, datasets, and evaluation scripts.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 2025.
Fabio M Carlucci, Antonio D' D'Inncennte, Silvia Bucci, Barbara Caputo, and Tatiana Tommasi. Do- main generalization by solving jigsaw puzzles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2229-2238, 2019.
Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Qiying Yu, Xuefeng Li, Jiaze Chen, al. Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles. arXiv preprini ar0iv:2505. /9914, 2025a.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. https::/ggtthbb..cc http:////g.ttttobcccc///eeeeeeee
R1 V 2025b. Accessed: 2025-02-02.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. In European Confer- ence on Computer Vision, pp. 370-387. Springer, 2024a.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang. Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? Advances in Neural Information Processing Systems, 37:27056-27087. 2024b.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang. Li Yuan, et al. Sharcgpt4video: Improving video understanding and genera- tion with better captions. Advances in Neural Information Processing Systems, 37:1942 2024c.
Yingyi Chen, Xi Shen, Yahui Liu, Qinghua Tao, and Johan AK Suykens. Jigsaw-vit: Learning jigsaw puzzles in vision transformer. Pattern Recognition Letters, 166:53-60, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arxiv preprint arXiv:2507.06261 2025.
Qingxiu Dong, Li Dong, Yao Tang, Tianzhu Ye, Yutao Sun, Zhifang Sui, and Furu Wei. Reinforce- ment pre-training. arXiv preprint urXiv:2506.08007 2025.
Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang. Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluat- ing large multi-m moliiityd models. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 11198 1201, 2024.
Zhen Fang, Zhuoyang Liu, Jiaming Liu, Hao Chen, Yu Zeng. Shiting Huang, Zehui Chen, Lin Chen, Shanghang Zhang, and Feng Zhao. Dualvla: Building a generalizable embodied agent via partial decoupling of reasoning and action. arXiv preprint arXiv:2511.221344 2025.
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference Computer Vision, pp. 148-166. Springer, 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision languue models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14375- 14385, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 25....2019... 2025.

---

Published as a conference paper at ICLR 2026
Ruiyan Han, Zhen Fang, Yu Sun, Yuchen Ma, Ziheng Wang. Yu Zeng, Zehui Chen, Lin Chen, Wenxuan Huang, Wei-Jie Xu, et al. Unicorn: Towards self ieprroing unified multimodal models through self- seff-geeeratedd supervision. arXiv preprint arXiv:2601.03119, 2026.
Shiting Huang, Zhen Fang, Zehui Chen, Siyu Yuan, Junjie Ye, Yu Zeng, Lin Chen, Qi Mao, and Feng Zhao. Critictool: Evaluating self-critique capabilities of large language models in tool- calling error scenarios. arXiv preprint arXiv:2506. 13977, 2025a.
Shiting Huang, Zecheng Li, Yu Zeng, Qingnan Ren, Zhen Fang, Qisheng Su, Kou Shi, Lin Chen, Zehui Chen, and Feng Zhao. Internalizing meta-experience into memory for guided reinforcement learning in large language models. arXiv preprint arXiv:2602.10224 2026a.
Wenxuan Huang, Shuang Chen, Zheyong Xie, Shaosheng Cao, Shixiang Tang. Yufan Shen, Qingyu Yin, Wenbo Hu, Xiaoman Wang, Yuntian Tang, et al. Interleaving reasoning for better text-to- image generation. arXiv preprint arXiv:2509.0644 2025b.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint arXiv:2503.065 066749, 2025c.
Wenxuan Huang, Yu Zeng. Qiuchen Wang, Zhen Fang. Shaosheng Cao. Zheng Chu, Qingyu Yin, Shuang Chen, Zhenfei Yin, Lin Chen, et al. Vision-deepresearch:: Incentivizing deepresearch capability in multimodal large language models. arXiv preprint arXiv:2601.2200 2026b.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276. 2024.
Bohao Li, Yuying Ge. Yi Chen, Yixiao Ge, Ruimao Zhang, and Ying Shan. Seed-bench-2-plus: Benchmarking multimodal large language models with text-rich visual comprehension. arXiv preprint arXiv:2404. 16790, 2024a.
Xiaotong Li. Fan Zhang, Haiwen Diao, Yueze Wang, Xinlong Wang, and Lingyu Duan. Densefusion-Ing Merging vision experts for comprehensive multimodal perception. Advances in Neural Information Processing Systems, 37:18535-1888 18556, 2024b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Xiaoxiao Ma. Haibo Qiu, Guohui Zhang, Zhixiong Zeng. Siqi Yang, Lin Ma, and Feng Zhao. Stage: Stable and generalizable grpo for autoregressive image generation. arXiv preprint arXiv:2509.25027, 2025.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvga. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Botian Shi, Wenhai Wang. Junjun Hc, Kaipeng Zhang, et al. Mm-cureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. CoRR, 2025.
Yaning Pan, Qianqian Xie, Guohui Zhang. Zekun Wang, Yongqian Wen, Yuanxing Zhang, Haox- uan Hu, Zhiyu Pan, Yibing Huang, Zhidong Gan, et al. Mt-video-benche A holistic video un- derstanding benchmark for evaluating multimodal 1lms in multi-turn dialogues. arXiv preprint arXiv:2510. 10.1172... 2025.
Yukun Qi, Yiming Zhao, Yu Zeng, Xikun Bao, Wenxuan Huang, Lin Chen, Zehui Chen, Jie Zhao, Zhongang Qi and Feng Zhao. Ver-bench: A comprehensive evaluation framework for video chain-of-thought reasoning. arXiv preprint arXiv:2504.07956 2025.
12

---

Published as a conference paper at ICLR 2026
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 11553728-537445 2023.
Qingnan Ren, Shiting Huang, Zhen Fang, Zehui Chen, Lin Chen, Lijun Li, and Feng Zhao. Adora: Training reasoning models with dynamic advantage estimation on reinforcement learning. arXiv preprint arXiv:2602. 10019, 2026.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Jingqi Tong, Jixin Tang, Hangcheng Li, Yurong Mou, Ming Zhang, Jun Zhao, Yanbo Wen, Fan Song, Jiahao Zhan, Yuyang Lu, et al. Code2logic: Game-code-drrven data synthesis for enhancing vlms general reasoning. arXiv preprint arXiv:2505. 13886, 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9568-9578, 2024.
Qiuchen Wang, Ruixue Ding, Zehui Chen, Weiqi Wu, Shihang Wang, Pengjun Xie, and Feng Zhao. Vidorag: Visual document retrieval-augmenter generation via dynamic iterative reasoning agents. arXiv preprint arXiv:2502. 18017, 2025a.
Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, and Feng Zhao. Vrag-rl: Empower uxsionppereeption-baeee rag for visually rich in- formation understanding via iterative reasoning with reinforcement learning. arXiv preprint arXiv:2505.22019, 2025b.
Qiuchen Wang, Shihang Wang, Yu Zeng, Qiang Zhang, Fanrui Zhang, Zhuoning Guo. Bosi Zhang, Wenxuan Huang, Lin Chen, Zehui Chen, al. Vimrag: Navigating massive visual context in retrieval-augmented generation via multimodal memory graph. arXiv preprint arXiv:2602.12335, 2026.
Wenbin Wang, Liang Ding, Minyan Zeng, Xiabin Zhou, Li Shen, Yong Luo, Wei Yu, and Dacheng Tao. Divide, conquer and combine: A training- free framework for high-resolution image percep- tion in multimodal large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 7907-7915, 2025c.
Xiyao Wang. Zhengyuan Yang, Chao Feng, Yongyuan Liang, Yuhang Zhou, Xiaoyu Liu, Ziyi Zang, Ming Li, Chung- Ching Lin, Kevin Lin, et al. Vierit: A verifiable reinforcement learning proxy task for visual perception in vims. arXiv preprint arXiv:2506. 10/28, 2025d.
Zifu Wang, Junyi Zhu, Bo Tang, Zhiyu Li, Feiyu Xiong, Jiaqian Yu, and Matthew Blaschko. Jigsaw-rl: A study of rule-based visual reinforcement learning with jigsaw puzzles. arXiv preprint ar arXiv:2505.235000
2025e.
Penghao Wu and Saining Xie. V Guided visual search as core mechanism in multimodal llms. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp- 13084-13094, 2024.
XAI. Grok-1.5 Vision Preview.
2024.
13

---

Published as a conference paper at ICLR 2026
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXtv:2502.14768, 2025a.
Yunfei Xie, Yinsong Ma Shiyi Lan, Alan Yuille, Junfei Xiao, and Chen Wei. Play to generalize: Learning to reason through game play. arXiv preprint arXiv:2506.080) 2025b.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et RI-onevision: Advancing generalized multimodal rea- soning through cross-modal formalization. arXiv preprint arXiv:2503. 2025.
En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, et al. Perception-rl: Pioneering perception policy with reinforcement learning. XrXiv preprint 2xXxx:2504.07554 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3556-9567, 2024.
Yu Zeng, Yukun Qi. Yiming Zhao, Xikun Bao, Lin Chen, Zehui Chen, Shiting Huang, Jie Zhao, and Feng Zhao. Enhancing large vision-language models with ultra-detaiedd image caption gen- eration. In Proceedings of the 2025 Conference Empirical Methods in Natural Language Processing, pp. 26703-26000 26729, 2025.
Yu Zeng, Wenxuan Huang, Zhen Fang, Shuang Chen, Yufan Shen, Yishuo Cai, Xiaoman Wang, Zhenfei Yin, Lin Chen, Zehui Chen, al. Vision-dcepresearch benchmark: Rethinking visual and textual search for multimodal large language models. arXiv preprint arXiv:2602.00 22121. 2026.
Guohui Zhang, Jiangtong Tan, Linjiang Huang. Zhonghang Yuan, Mingde Yao, Jie Huang, and Feng Zhao. Infoscale: Unleashing training-free ffrenng--reeeeeeee variable-scaled image generation via effective utilization of information. arXiv preprint arXiv:2509.01421, 2025a.
Guohui Zhang, Hu Yu, Xiaoxiao Ma, Yaning Pan, Hang Xu, and Feng Zhao. Maskfocus: Fo- cusing policy optimization on critical steps for masked image generation. arXiv preprint 1187: 2025b.
Guohui Zhang, Hu Yu, Xiaoxiao Ma, JingHao Zhang. Yaning Pan, Mingde Yao, Jie Xiao, Lin- jiang Huang, and Feng Zhao. Group critical-token policy optimization for autoregressive image generation. arXiv preprint arXiv:2509.225 22485, 2025c.
Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal IIm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint a4Xiv:2408. 13257, 2024.
Yiming Zhao, Yu Zeng. Yukun Qi, Yang Liu, Lin Chen, Zehui Chen, Xikun Bao, Jie Zhao, and Feng Zhao. V2p-bench: Evaluating video-language understanding with visual prompts for better human-model interaction. arXiv preprint arXiv:250..17736, 2025.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhanghhi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint arXiv:2403.13372 2024.
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint wr.iv:2505.1436. 2025.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A DATA DISTRIBUTION
As shown in Figure our RL training corpus for jigsaw is derived from multiple sources, covering diverse range of visual scenarios:
39.7%
33.3%
High-res & Visual Search
26.9%
Text & Structured Scenes
Dense Natural Real Scenes
Figure 6: Distribution of jigsaw RL training data.
High-resolution and visual search images (39.7% %, 6.2K samples): enhance the model's fine-grained perception in high-resolution images, collect data from the VStar (Wu & Xie 2024) dataset, high-resolution natural scenes in DeepEyes (Zheng et al. 2025), and selected samples from HRBench (Wang al., 2025c). This improves the model's ability capture subtle visual cues and recognize small object attributes.
Text recognition and structured scenes (33.3%, 5.2K samples): To strengthen the model's capacity for text perception and recognition, we include text-rich images from diverse domains, such the TextVQA (Singh al. (019)) training set, InfoVQA (Mathew al. (022) book covers and posters. as well as structured visual reasoning tasks involving tables and subject-specific charts Yue al.) 2024 et 2024a Chen et al. 2024b).
Dense natural and real-world scenes (26.9%, 4.2K samples): To enhance the model's recog- nition and understanding in complex and real-world environments, we collected images from COC02017 (Lin et al. 2014) and RealWorldQA (AIl 2024) datasets.
B PROMPTS
In this section, we present the prompt used collect high-quality jigsaw trajectories, as illustrated in Figure B The same prompt is employed to prompt the Qwen2.5-VL-7B model during both the SFT and RL stages.
Prompt for Interactive Collection of
Jigsaw Trajectories
System Prompt:
You are a skilled and experienced puzzle master. will divide an image evenly into 2 x grid, get 4 image blocks, and then shuffle them. The image blocks will be labeled A, B, C, and D. Initially, the image blocks are named and arranged in the following order:
["A", "B", "C", "C", "D"]
That is, the initial layout of the image blocks is as follows:
A
CD
Your Task:
15

---

Published as a conference paper at ICLR 2026
Your goal is reconstruct the original image by observing and analyzing the visual content between the image blocks. Pay attention to the details in each image block and establish visual connections between adjacent image blocks to achieve the goal of completing the puzzle correctly. For example:
-Continuity of text information: If the image contains text or logos, observe whether the text continues naturally between adjacent blocks (e.g., characters connect smoothly and the font direction is consistent). This is one of the key clues to determine whether the image blocks should adjacent.
Structure and shape consistency: Identify possible structural elements in the image (such as buildings, roads, object outlines, etc.), and infer which image blocks should be visually spliced together to form a complete and reasonable shape.
Continuation of edge visual details: Observe the color, texture or pattern (such as sky, grass, lines or shadows) the edge of the image block to determine whether it can be visually connected to the adjacent block naturally.
Direction and angle consistency: Pay attention to the direction of image elements (such as human faces, object directions, text angles, etc.) to ensure the rationality of the overall visual direction of the image.
You swap any two image blocks each time to achieve the correct layout.
Inference Process:
1. State Representation: At each step, you need to maintain "state" list representing current arrangement of image blocks.
Example:
state = ["B", "C", "A", "D"]
This corresponds to:
Top left (index 0): B
Top right (index I): C
Bottom left (index 2): A
Bottom right (index 3): D
2. Visual Analysis: Carefully analyze the image patch content and visual details to determine the image patch location. To better complete the puzzle task, you can use additional image operations to enhance visual perception:
Crop the image area of observation image to more closely observe the visual correlation between different regions multiple image patches. Used to find clues during the puzzle verify whether the puzzle is complete (For example: it looks like the puzzle is not completely fixed, maybe because the lower left and upper right corners of observation 2magre..onn_maae22... information to further judge. First will crop the lower left area <code>..<<coder By observing the cropped lower left area, it shows that part of the text "OF H" may be related to the "appyness" text the upper right corner. To further verify and manage, will now crop the upper right image Ecode>..<<code:.. By observing the cropped image, will swap image blocks B and C...)
need to crop these two areas separately to get some visual
Zoom area to enlarge visual details (selectively observe the details of the cropped image).
These operations can help you make more informed decisions.
16

---

Published as a conference paper at ICLR 2026
Move Execution: After making a decision, generate a Python code snippet swap tiles.
Use the 'observation(statt) function to get the updated layout image for the next step.
Available Image Operations:
You can use the following Python functions to assist in inference:
1. Crop region from an image using normalized coordinates (from 0 to 1),
crop_box [xl, yl, x2, y2] prro_image__id)
crop (image, (rop_box)
Zoom an image by a specified factor (c.g., 1.5x zoom means 150% zoom).
zoom_image_ zoom (image, zoom_factor)
3. Swap tiles (e.g., top left and bottom left) and call observation function to see the jigsaw progress.
state [0], state [2] = state state observation_imaee_
observation (state)
Replace "id" with an integer to uniquely identify each operation result. The "image" pa- rameter must point to an existing image (e.g., oobservation.image... "crop_image_1"", etc.).
Format:
All Python code must be enclosed in <code>...<code>>>
All reasoning steps must be enclosed in tthink>..</thinkkk.
The final answer must be enclosed in <answer>..<<annnee <answer>..<</nsser>taag
Important:
1. For both swap and image manipulation, always explain why you are doing this before generating code.
Stop as soon as you generate a code snippet. will execute the code and return generated image for the next step.
3. You can only perform one image operation observation of the puzzle state per turn.
4. Final answer: When you sure that the puzzle is fully reconstructed (all edges aligned and the visual information is complete and continuous), return the final "state" list in the "<answer>" block.
5. Please pay close attention to the rationality of your cropping area and ensure that the cropped area is reasonable criterion for completing the puzzle.
User Prompt:
The four images are respectively labeled A, B, C, and D. Please complete the jigsaw. Image A: <image>> Image B: Seinage>Imaae C: Aamage>IImage D: <image>
17

---

Published as a conference paper at ICLR 2026
C CASE STUDY
In Figures 7 TO we present representative high-quality jigsaw trajectories collected to illustrate
how our model performs reasoning and interacts with the environment. These cases specifically demonstrate: (1) how the model generates Python code to interact with the environment, such as swapping any two jigsaw pieces, observing jigsaw progress, and cropping/zooming on regions for finer visual inspection; and (2) how the environment provides feedback that guides the model's subsequent reasoning.
The four images respactives fivlly labeled A, B. C, and Please camplete the jsan..
Imege
Image
Image
Image
the original image four shuffled blocks
*Initiia State Analysis**
Image the bottom-left
display
shalves background visible and right sign
the "Nex unn" display and lorge "Low signs. The left sign Inage "$23". looks like Shows the bottom-right of the purple "Nexcn"" display, with text like "boctor Prescribed", and ad white display, floor visible..
Image De Shows the ceiling of
and the continuations of the red
from showing '88". This looks like top-right
Gemini2.5 Pro
**Reasoning for the First Move:
partion of the main subject (the purple displey) and the start large complete os sign ("$18, 97") the left, sugecating it's the top-left correr the overall scone.
Image contoina
('$23"). It alse contoins Image shoss the bottom part of the some purple display and the shelves that ane directly below the ones Irage
Therefore. Image should be placed above Trrgge
The current loyout A the lop-left and in the top-right. I start by swapping A and B place B in the top left posit on, which
scens to be cerrect location.
My first stop to swap and B. </think
conden # Current state of the tile state a ("A". "B", 'c: 'D")
Samp biks A (top left) and (top right) stete(0] state(!] statelli striee0] call observation function to see the reoult observation_jngg observetioo(sttte) v/cade>
codk executed successfully. "observetion imwge________""
User
<think *"Curmnnt State "nolysis
The current state is ("8", "A", "D'T".
Top Left (8). The top the purple display and price sigrs.
Top Right (A): bottom the purple display.
Bottom Left (C): The botton-right of the purple dispior. oy.
Botton Right (D) The ceiling and the right side the price sigrs.
"Reesoning for
top-left cormer, shows continuation of
cornect.
6emini2.5 Pro
above shelves, Therefare, price sign (the "88" "$23 822)
store ce celling, which logically goes
be top right position, bottom part of purple display calialll botton part purple disploy white display logically goes below So, correct arrangement
soems
Currently, top-right bettom-right. swap
"A" 'c. (bottom right)
statefl], stotel3) state[3], statell]
Call observation fonction to result
obrervation imooe_ abservatinv(stote) </oode
Figure Representative high-quality jigsaw trajectory case (part l).
18

---

Published as a conference paper at ICLR 2026
successfully.
naage_
User
<thinks "CCumnntt Aralysis
The current
'0", "A'I"
Left Top-left
Top Right (by: Tap- rightight
the acans, the price sign "$23 88" connects perfectly. This ssams correct.
-Battom Left (C) Bottomirighn part of the purple disploy.
Bottom Right (A): Botton-ieft cotonnefftpee part of the purple display.
1. The top row (B. b) looks cornect. The price sign is complete, and the store background and celling clign.
*Beessoning for the next nove
Gemini2.5 Pro
2. The bottom row (c, is incorrect. The purple display spái. Tile shows the left side the botton of the purple display, and till
shows the right side.
should be ploced under 9, as they both show the left of the scone (the shelves with greer/reed boves connect, and the purple dippley conmacts vert colly).
as they both right purple display). Therefere,
placed
right
(the white display with
swop
crop region the
"Boss!"
purple diaplay
shelving </think
--codee>
verify oligrment between 8 and bax 0.4, 0.5. 0.6j
/mage wwwpeesserwation..oeo Inage. 2, bsx)
<//ceee>
The code execated suceessfally. "crop_inage_""
User
<thinko
"nCumentt State Anclysis
The curnent shote ('9", "b" *gn "A"T"
The crop showsd the battom of B and the top of c worted to check the soon between and A The crop shows the seam between (top left) and (bottoo left). This seem clearly incerrect. The "Nexiim' display doean't align, and the bochground shelving doest connect. This confirms that C is not the correct position below
Gemini2.3
Mly initial enalysi was correct: should below and should below The curment layout them swapped. naw porform
Pro
swap
state|21
Call observation function recult
observation obsenvstion(state)
The cods exeecaeed succnssfully. observetion, /angoi____n_"""
User
<thinic
**Finel State Andlysis
The current state is 1"8", 'b". "c"f"
Top Rew (B. Dy The price sign "$23.60" complete, The store's is ceiling and background theives dign correctly.
Botton Roa (A. c): The purple "Nexiim 24HR" display stard now complete and correctly assembled The text on the display flows naturally coreas the ties. The white floor is continuus.
Gemini2,5 Pro
Vertical Alignment: The top of the purple display in cornects scamlessly with the bottom part The background shelves behind the purple disploy in connect with the flor and adjocest white display All edges and vinol elements olign perfectly. The image now reconst ruched. </think.
conswrr> Pet 'D", k/ansver>
Figure Representative high-quality jigsaw trajectory case (part 2).
19

---

Published as a conference paper at ICLR 2026
imaees respectively complete jino..........
C,
User
Image A
Image
Image
Image
<think
Besed on the initial onalysis of the four inage blocks, I will begin by arranging the puzzle
""Irage get shows the top-left left corner of pastoord. has the text "THIS SIDÉ Of CARD Is F... and the left edge postmark The top and off sides have dark border, indicating it's the corner of the scarned item. This strongly suggests should be in the top-left position (incex 0).
"IImags bet shows postage stomp and the right of postmark seens to continue From one Image ".ORR ADBRESS" which completes the phrase Image This suggests
("EAROLIS,
ase
top peaition (index U.
""Image contoine baginning
address:
"bept. Pre..", "Weshington Mediccie"", 'School Medicine", 30,
suggests
Gemini2.5 5
A
suggests
shouls
My "arget 8DD" and C bottom, which corresponds to [LB", "0, 'AT."
I'll start by maving black B to the top-left corner: da this, I b ock and black
Curment state: ["A" 'gr tc". TD'
Desired oction: Seap A and Readling state: ("8", "A" "C" "D"T" </think>
<code> # Entiia state
#tote =
"or]
Swap image block (top left) and (tep right)
state(0) statefi] a state(?), state/O]
Observe layou
lnaereattonnoneee....moo oéservation(state)
The code executed successfully. Eisteretion_/mag__.. is
User
From observation imoge_"___"""""
Top-left (8) correctly placed, shows the top-left of posteard, does the stamp completes the text "THIS SIDE OF CARD IS FOR AbDRESS Bettom-left o (C) seems cerrectly pieced reletive the top
Top-right (A) insorrect. The handwritten
with enything bleck The correct block should be D, which
Gemini2 Pro
it's the start of the address.
Botton-righr right (D) incorrect. In should be the too right.
My next step to move black D to the top-r ight position, I con ochieve this by swapping block A and block D.
Current state: ["8", 'A", "C1, "Drr" Desired act or Swap (rop-right (bottom-right). Resulting state: (18).
"A""
</think>
"<oade>
'c" 'b"3
imgg (top right) and right)
state/!] stnte(3] state(t)
Shserve
observation_ Lwaggeinn__anaee22222 obsemation(stste)
c/cade>
Figure 9: Representative high-quality jigsaw trajectory case 2 (part 1).
20

---

Published as a conference paper at ICLR 2026
The coce executed successfully. "sbservation_ wrogeettooooooog_e__e
User
(think Thee current stote is ['8", 'D", 'c", 'A'T" Looking at abservation_iiee image_ 2':
The top row (8 and DJ looks correct. The text "THIS SIDE of CARD 15 FOR ADDRESS' complete, and the postmank 'MINNEAPOLIS MINN JAN 29 PM 1954" is olso complete. The top dork border continues from 0 D. The nattom row (C and A) eiso nekk correct.
Bleck (bottom-left) (eftoo--effttt () hat "Dn. R. E", "Dept, of Pre. "Washington Univ." "Eudlid Ave." Stt, L.." Black (bottom-righ right) cont iuss this with Shanki, verfive Medicine", School of Masicne", & Kingshighway': "..ouis 10,
Missour"
The handeritten text flows correctly from to
be absalutely sure, I crop crea where the hondwrriten transitions between black C block A will focus continiity of the words "Preventive Mediaie" and "Washington Univ." with "School of Medicine", and "Euelid Ave." with Kingshigheay.
6emini2.5 Pro
crop region spanning right edge
left edge focusing on
Coordinates "Koond" V_starr roughly c's h'shhtgh, y__nnd roughly height. Coordinates "x_start" 1s00, y_storr" roughly A'sheght, "yLond" roughly height.
the
chservation 2,), this translates
(middle (opprex) "yI" (middle A)= 0.5 06 0.8 (аpprox)
0. 0.55 (oppeoxc (midele (spproxd) 'y2' (middl
Let's nofine crop
specific the
and
The ve0.5 I'll from 'x004" "=00.""
The text interest ('Dept. fre.." "School of Med.." .") in vertical from about 'y=0.5" "y=0,8"
intervation__iaaee imegg__eee__... So, the crop box will [0.4, 06. 0..0 </thinks
Riodes Current stute state = ('9t, ID", 'C, '4")
Chop the region where hondwritten text from C meets
for ploser rnspectionno contiruity.
a1-0 4, ylrd (middle-iih of image. focusing on text kine)
x210 6. 92-0.8
crep, box 10-4, 0.6, 06, 0.8]
Grog_
nagge_1 Grag_
cngecctereettoor 2 2, box) s/eades
The coce executeo succesofully. "crep_/moge__"
S clos
User
Nus
<thinko The cropped image crege_inoe__"" I' shows the scom betweer. the bottom-ieft and bottom-right blecka, On the left side (Trom Block G). we can see the end of "Washingtan Unv" and 'Eclid Ave.. On the right side (from block A), we can see the beginning of "School Mediciee and 'à Kingshighway".
lines text de not seem to align perfectly. For exumple. "Univ." on the left the some line "Schoo. the right. This incorrect. The aderess should be on reporate lices, 'Dr, Shank", "Dept. of Preventive Medicin"" "Woshington Univ. School Medicinn") "Euclid Krgshhghhay Sii Lois 10, Missuuri"
Looking back et "dbservation. image______ can see: in
Gemini25 5 Pro
comnect "Shark" Line "Dept. Fre... connect "ventive Line "Euclid
"Washington should connect "School
connect
Kingshighway"
should
re-examine" ebservation_ inagg__"" carefully.
begining off. "ventive Medicine" "Wash Univ."
with "Dr. with "School "Dept. Pre" A. "Euclid Ave" cornect with '..6 Kingshighwar" *St. L.." C connect with touls 10, sssour
"Shank"
I" showed misolignment. Lat re-eveluste initial thought. "Washington Univ"" from é and "School of Medicine" from A parts the some organization, but they not be the some line. Let look at the vertical aiigment The text 'Dr. R Et from è looks horizontally aligned with "Shenk" from A. "Dapt. Fre" from C looks karizantaly aligned with "ventive ive Medicine from A. And on for the rest of the lines.
The
Thee continuity is correct. The crop may have been dightly micleoding due to the cursive handwriting lng style and line breaks, The overall image obdervation image 2" looks correct. The top now perfactly dlignad, and the botton row text is continuss,, logical address. The dark barder on the of C the right of A motch the everal scan boundary. Therefore, the puzzle solved, </thinko
vonswerrs (*B" "D". 'ct AA') "A"]]//osaee </onswer>
Figure 10: Representative high-quality jigsaw trajectory case (part 2).
D THE IMPLEMENTATION OF TRAINING
The detailed training hyperparameters are provided in Tables 4 and [ and all experiments are con- ducted 8 NVIDIA A100 GPUs, each equipped with 80 GB of memory.
21

---

Published as a conference paper at ICLR 2026
Table 4: Key hyperparameters for SFT.
Table 5: Key hyperparameters for RL.
Name
Value
Name
Value
Finetuning type
Full
Max turns
Freeze vision tower
True
Rollout num
Freeze multi-modal projector
True
Train batch size
Freeze language model
False
Mini batch size
64
Cutoff len
16384
Micro batch size per GPU
Image max pixels
401408
Learning rate
2.0e-6
Epochs
2.0
KL loss coefficient
0.0
Batch size
32
Total epochs
Gradient accumulation steps
4
Max prompt length
8192
Learning rate
1.0e-5
Single response max tokens
2048
LR scheduler type Warmup ratio
cosine
Max response length
20000
0.1
GPU memory utilization
0.7
ANALYSIS OF WANDB CURVES IN JIGSAW OPTIMIZATION
In this section, present the training curves of RL process and provide analysis of the training dynamics. We monitor the evolution of rewards (including accuracy and format rewards), response length, number of interaction turns, and validation accuracy.
2600.
24000
0.8
Resporsee Length
Reward
3000
0.-
12000
0.4
Farnat Reward
4000
01
120
Training Step
Training Stop
(a) Reward mean.
(b) Response length mean.
30
Tum Number
0.7
20
Vol
05
03
120
160
200
240
0
80
120
160
200
240
Training Step
Training Step
(c) Turn number
(d) Validation accuracy.
Figure 11: Visualization of Wandb curves in jigsaw RL optimization.
At the beginning of training, both validation accuracy and reward values are low, indicating that the model has very limited jigsaw-solving ability in the early stage. As training progresses, the number interaction turns increases briefly, suggesting that the model explores more interactions (e.g., swapping tiles, cropping for observation) in order to achieve higher accuracy. With continued training. the reward values steadily increase, reflecting a stable improvement in jigsaw-solving ca- pability through RL. After sufficient training, the model gradually reduces both the number of turns and response length, demonstrating enhanced perceptual and reasoning ability, such that it can solve jigsaw correctly with fewer interactions.
22

---

Published as a conference paper at ICLR 2026
F LIMITATIONS
Despite our best efforts, this work has several limitations. Multi-turn jigsaw interactions inevitably increase context length and introduce significant computational overhead. In the x setting, the context often exceeds the model's maximum window size, restricting our RI. training to the 2 case. Future work could investigate more efficient interaction mechanisms with external environments or integrate memory modules to mitigate context length constraints, thereby enabling RL training on larger and more complex jigsaw tasks.
G QUALITATIVE ANALYSIS ON ATTENTION PATTERNS
To further verify that AGILE brings genuine improvements in perceptual and reasoning abilities rather than merely increasing benchmark scores, we conduct additional qualitative analyses on gen- eral downstream tasks. Specifically, we visualize the attention distributions of Qwen2.5-VL-77 before and after AGILE training. As shown in attention examples (Figure [2). the original model often exhibits scattered and unstable attention allocation, focusing large irrelevant regions when processing complex high-resolution scenes or fine-grained recognition tasks. This dispersed at- tention leads to unreliable reasoning and misinterpretation of critical visual evidence. In contrast, the AGILE-trained model shows significantly more concentrated attention on key visual elements, such small objects, text regions, and structurally important areas. This improved focus enhances the model's 's ability to extract fine-grained cues and supports more reliable multimodal reasoning. These qualitative findings align with our quantitative gains across HRBench, VStarBench, and Real- WorldQA, suggesting that AGILE enhances fundamental perceptual mechanisms underlying general visual understanding.
H WHICH TASKS CAN BENEFIT MORE FROM AGILE?
Beyond overall benchmark gains, we analyze the types of perceptual and reasoning skills most enhanced by AGILE:
Fine-grained recognition. These tasks require high precision in identifying subtle visual cues, such as detecting small objects, recognizing text, or distinguishing fine-grained visual differences. AGILE enhances the model's ability to focus on most critical visual elements, improving its sen- sitivity to subtle but important features and thereby boosting fine-grained recognition performance.
Spatial relation reasoning. Benchmarks such as VStarBench contain a substantial number of questions involving spatial positional reasoning. AGILE's interaction-driven training helps the model better attend to spatial layouts and contextual relationships within a scene, substantially im- proving its ability to understand and reason about complex spatial arrangements. This capability transfers effectively to real-world spatial reasoning tasks.
Visual-textual integration. AGILE also leads to notable improvements in tasks requiring the integration of visual and textual information. Tasks involving reading product details, interpreting signs, or extracting structured information from charts benefit from AGILE's strengthened ability to jointly process textual cues grounded in visual context.
I HANDLING INVALID CODE EXECUTIONS
During reinforcement learning, the model may generate malformed or invalid code. Instead of discarding such rollouts, the environment executes the code and returns explicit error messages to the model. These erroneous interactions increase the total number of steps required to reach the correct solution and thus naturally lead to lower step rewards under our reward formulation (Eq. 5 and Eq. 6). As result, invalid code is implicitly penalized, encouraging the model to generate valid actions and interact more efficiently. This design stabilizes training without requiring complex filtering mechanisms.
23

---

Published as a conference paper at ICLR 2026
What is the color of the SUV car? The color of the SUV can red.
background, oliver SUV poriee by the weter The cornect areberr the calor of Inx Sul car is dileer
image does show SUV go not determine image""
The color of the SUV our alue. The color of the SUV car black the color of the SUV can sheer.
User
Quwan? 5V-
2P
The image whows bassball players celebruting with their bucks the comers. The visible jenseys suggest munaiguns it wearing 22 and Yonnguch is wearing 2, giving a fotel of 24.
image group playars
sum numbers TSUOURA YAMAAUCHH
and their their numbers cornespond Mesiauguma 25, Yanoguchi The puts of these numbers is 29 2 31.
celebrating,
User
Quarn SVI. 7B
AGiL2
The
Bassa on image provided. there no walcaite online source shown leanning more Tlagging Tape. image displays physical products sibli.
Baned the image provided, you are ed in Flogg Tape, there
interested Flooging relevent Information?
the company whose HANISONI VG TAPE The website chhanssn You vieit this website learn mone about their their products.
Quern2 SVL 7B
AGILE
Comparison of Attention Maps Before and After AGILE Training. indicate higher attention; cool colors indicate lower attention.
Figure 12:
Warm colors
ADDITIONAL EXPERIMENTS
J.1 SENSITIVITY TO REWARD COEFFICIENTS.
We further investigate the sensitivity of AGILE to the weighttt coefficients (o, 3, and y). As shown in Table 6 varying the values of a and yields only minor fluctuations in performance, indicating that AGILE robust to these parameters. In contrast, removing the step reward (7) produces notable performance degradation, especially VStarBench and MMVP. This finding
24

---

Published as a conference paper at ICLR 2026
ifies that the step-pes step-penalty plays critical role in encouraging efficient and meaningful interactions, preventing long, unfocused rollouts during RL.
Table Ablation on Reward Coefficients. Sensitivity analysis of the weighting coefficients a; ,, and y in the total reward.
Model
MME-RW
RWQA
HRB4K
HRBSK
VStar
MMVP
BLINK
HalBench
MMMU
Avg.
Owen2.5-VL-7B
44.6
68.5
68.8
65.3
76.4
74.3
56.4
50.1
54.8
62.1
0.8, 8 0.2,
1.0
48.4
70.2
73.0
70.5
80.6
78.0
58.0
51.9
55.8
65.2
78.5
79.0
0.8, 8
0,2, 0.0
47.4
71.1
72.4
69.8
78.5
75.3
56.7
52.8
55.6
64.4
0.9, 8 0.1. y= 1.0
48.9
70.8
73.3
69.6
79.1
78.0
57.5
51.4
55,1
64.9
J.2 RESULTS OF LARGER JIGSAW GRID SIZES AND CURRICULUM RL TRAINING
To further assess the scalability of AGILE beyond the 2 jigsaw setting, conduct an addi- tional reinforcement learning stage on 8K x jigsaw puzzles. This stage applied after the initial 15.6K 2 2 RL training, forming curriculum-style optimization principle. As shown in Table training on 3 x 3 jigsaw puzzles leads to further improvements across all benchmarks. These results demonstrate that AGILE naturally generalizes to larger and more challenging jigsaw config- urations when model capability allows, and further highlight that curriculum reinforcement learning increasingly difficult grid sizes provides effective path for continued scaling.
Table 7: Results of 3 Jigsaw RL Training.
Model
RWQA
HRB4K
VStar
MMVP
BLINK
Halliench
MMMU
Avg.
Qwen2.5-VL-7B
44.6
68.5
68.8
65.3
76.4
74.3
56.4
50.1
54.8
62.1
15.6K 2x2
48.4
70.2
73.0
70.5
80.6
78.0
58.0
51.9
55.8
65.2
15.6K 2x2 8K 3x3 RL
50.4
70.6
73.4
70.5
80.1
79.0
58.1
52.1
55.8
65.6
J.3 EXPERT TRAJECTORIES ABLATIONS
To disentangle the effect of expert cold-start trajectories and verify that AGILE's improvements are conducted additional experiments without any expert trajectories.
merely due to distillation, Specifically, we performed RI directly on the 15.6K x 2 jigsaw dataset under two conditions: (1) the full action space, and (2) an ablated action space with Crep/ Zoom removed. The results show that even in the absence of expert supervision, AGILE still delivers a +1.8% improvement over the DLL-7 baseline, demonstrating that the performance gain arises from agentic visual interaction combined with verifiable RI, rather than imitation of Gemini 2.5 Pro trajectories.
Table 8: Performance of RL without Gemini 2.5 Pro expert trajectories and with ablated action spaces.
Model
MME-RW
RWQA
HRB4K
HRB8K
VStar
MMVP
BLINK
HalBench
MMMU
Avg.
Qwen2.5.V1.77
56.4
Cold-Start
46.2
68.4
71.0
68.4
77.5
76.7
55.7
49.8
54.0
63.1
Cold-Start
70.2
73.0
80.6
78.0
58.0
51.9
65.2
Only RL. (Poll action space)
46.3
69.5
71.5
68.1
78.5
79.0
56,3
53.2
53,1
63.9
Only (No Crom/Zoom)
46.2
69.3
71.9
68.0
78.0
78.3
53.7
52.8
63.5
Furthermore, removing the Crop /cop Zoom operations leads to a clear decline in performance, under- scoring the importance AGILE's interaction design in enabling effective fine-grained perceptual reasoning.
J.4 GENERAL QA DATASET DETAILS AND EXTENDED COMPARISON WITH JIGSAW TRAINING
We provide additional clarification regarding the composition of the general QA dataset. Our 20K QA corpus is constructed entirely from high-resolution, real-world visual scenes, consisting of 15K samples from DeepEyes (Zheng et al. 2025) and 5K samples from MMEureka (Meng al. 2025
25

---

Published as a conference paper at ICLR 2026
Table Comparison between General QA and Jigsaw-based RL under equal training budgets (20K).
RWQA
Avg.
Qwen2.5 VL-1BB
44.5
68.5
68.8
65.3
76.4
74.3
56.4
90.1
54.8
62.1
20K General QA
50.5
68.9
72.4
69.5
82.7
78.3
54.8
$1.3
54.2
64.7
10K General QA 10K Jigsaw
51.6
69.8
73.0
70.6
81.7
79.0
57.3
51.2
55.7
65.5
ligsw-onnly
73.6
70.5
These datasets span diverse perceptual and reasoning categories, including fine-grained attributes, spatial relationships, counting, and commonsense scene understanding.
Accordingly, the comparison between jigsaw training and QA training is inherently fair: both train- ing regimes operate on high-resolution, real-world, and complex perceptual inputs. To further strengthen this comparison, we additionally include the 20K pure jigsaw setting. As shown in Table 9 jigsaw-based RL consistently outperforms high-resolution QA-based RL under the same training budget, demonstrating that AGILE's performance gains do not stem from differences in input reso- lution, but instead from the structured, verifiable, and spatially grounded training signal introduced by the jigsaw task.
J.5 IMPACT OF SCALING THE COLD-START (SFT) DATASET
To understand whether training longer scaling up the supervised cold-start phase affects perfor- mance, we expand the cold-start dataset to 1.5x and 2x its original size and repeated all experi- ments. As shown in Table 10 enlarging the SFT dataset leads to only marginal differences, with signs of overfitting or meaningful performance degradation.
Table 10: Effect of scaling the cold-start (SFT) dataset size. Increasing SFT data provides only marginal differences, indicating that most performance gains come from the RL stage.
Model
MME-RW
RWQA
HRB4K
HRBSK
VStar
MMVP
BLINK
HalBench
MMMU
Avg.
Qwen2.5-VI.-7B
44.6
68.5
68.8
65.3
76.4
74.3
56.4
50.1
$4.8
62.1
Cold-start (1.6K)
46.2
68.4
71.0
68.4
77.5
76.7
55.7
49.8
54.0
63.1
+RI.
48.4
70.2
73.0
70.5
80.6
78.0
58.0
51.9
55.8
65.2
Cold-start (2.4K)
46.2
68.9
71.4
68.8
77.0
76.0
57.8
50.2
54.0
63.4
+RL
48.4
73.0
69.9
78.5
79.0
57.7
50.5
56.3
64.8
Cold-start (3.2K)
46.5
67.8
71.4
68.8
76.7
58.1
49.2
$3.3
63.2
+RL
47.1
70.1
73.0
70.5
79.6
78.3
57.2
50.5
55.7
64.7
In practice, the cold-start stage serves very specific purpose: teaching the model the tool-use pattern (i.e., how to invoke the predefined Python APIs). Its goal not to solve the jigsaw task fully. Consequently, the model's performance is largely unaffected by moderate expansions of the SFT set. The primary source of generalization and performance improvement comes from the interactive RL stage, where the model learns to solve jigsaw puzzles through real environment feedback rather than through imitation.
K USE OF LLMs
Yes. use LLMs solely to assist in language polishing and improving readability. All technical content, experiments, and analyses are conducted by the authors.
26


Reviews:
[{"review_id": "f9oZiTqCGj", "reviewer": "Reviewer_Jwkt", "summary": "This paper proposes AGILE (Agentic Jigsaw Interaction Learning), a framework to improve visual perception and reasoning in large Vision-Language Models (VLMs). The authors propose a RL based paradigm where the model progressively solves visual jigsaw puzzles. AGILE formulates the puzzle as interaction between the model and an external environment, with the model generating executable Python code to perform actions such as swapping, cropping, or zooming image tiles. Through iterative observation and feedback, the model develops stronger reasoning and perception capabilities. The framework includes a cold-start phase, leveraging expert trajectories collected via Gemini 2.5 Pro, followed by a reinforcement phase using Group Relative Policy Optimization (GRPO). Extensive experiments demonstrate accuracy improvement on the trained task from 9.5% to 82.8% on 2×2 puzzles. The trained model also demonstrates generalization gains (average +3.1%) across nine diverse vision benchmarks. By constructing scalable, controllable jigsaw datasets without reliance on costly human annotation, AGILE provides a scalable solution to the scarcity of high-quality multimodal RL data.", "questions": "- Since the model generates Python code to perform jigsaw actions, how are code generation errors handled during RL training? Are invalid executions penalized, or filtered out before reward calculation?\n- The paper defines three components (accuracy, format, and step rewards). Could the authors clarify how sensitive overall performance is to the weighting coefficients (α, β, γ)? Have ablation tests been performed to determine the optimal balance among these terms?\n- The model shows improvement across nine benchmarks. Could the authors elaborate on which specific reasoning categories (e.g., fine-grained recognition vs. scene reasoning) benefit most from AGILE’s training?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper’s key strength lies in using jigsaw solving as an interactive proxy task for improving visual reasoning. Because of the rule-based data generation, this method prevents the need of annotations in a scalable manner. This reduces dataset scarcity in multimodal reinforcement learning. This design also enables control over task difficulty. \n\nBy integrating agentic behavior, code generation, and environmental feedback, the authors create a setting that closely simulates exploratory problem-solving. This becomes an essential step toward autonomous multimodal reasoning. \n\nAlthough some gains on the trained jigsaw task are expected over the base model, using AGILE achieves a jump from 9.5% to 82.8%. The model trained using AGILE also achieves gains on the general vision tasks.\n\nExperiments on the RL training data scale also indicate the importance of the RL step in improving model performance. Further experiments on using a mix of General QA and Jigsaw data over only General QA data to improve the performance indicating the importance of the jigsaw data.", "weaknesses": "- While the authors report an average 3.1% generalization improvements, it is not clear whether these improvements are mere metric increases, or because of an improvement in visual perception and reasoning. While the authors provide a small case study on the jigsaw task, qualitative analysis on the general benchmarks would be useful to quantify the gains.\n- The reliance on python code generation for performing actions can limit the model capabilities for further generalization to unstructured tasks. The authors do not mention/explore the use of more traditional tool calling methods to directly call environment APIs instead of generating python code to execute the APIs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u0kKyT6PwN", "reviewer": "Reviewer_ciSx", "summary": "This paper presents AGILE, a novel framework that enhances the visual perception and reasoning capabilities of Vision-Language Models through interactive jigsaw puzzle solving. AGILE formulates jigsaw solving as an iterative interaction between the model and its environment, allowing the model to progressively refine its perceptual and reasoning skills through exploration and feedback. The framework employs a scalable data generation pipeline to construct high-quality multimodal reinforcement learning datasets with controllable difficulty, effectively addressing the scarcity of such data. Experimental results show that AGILE significantly improves jigsaw-solving performance and generalizes well across diverse vision tasks, demonstrating its strong potential for advancing the development of VLMs.", "questions": "1. Could the authors clarify whether AGILE uses a multi-turn or single-turn setting during evaluation on general downstream tasks? If multi-turn interactions are involved, how is fairness ensured when comparing with single-turn baselines?\n2. What is the underlying motivation for introducing the Step Reward design? Could the authors elaborate on how this component influences the training dynamics and model performance? It would be valuable to include an ablation study to show the effect of removing this reward, particularly on general vision tasks.\n3. Since AGILE involves multi-turn reinforcement learning, could the authors provide a quantitative or qualitative analysis of the additional computational cost introduced by this setup? A comparison between single-turn and multi-turn RL in terms of performance–cost trade-offs would help clarify the practical implications of the proposed method.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Framework: The proposed AGILE framework creatively formulates jigsaw solving as an interactive process to enhance VLMs’ perceptual and reasoning abilities. This integration of interactive learning with jigsaw-based perception is highly novel and opens up a promising research direction for improving the core competencies of VLMs, particularly within self-supervised learning paradigms.\n2. Significant Performance Gains: Experimental results show that models trained with AGILE achieve substantial improvements on jigsaw tasks and exhibit strong generalization across various vision benchmarks, clearly demonstrating the framework’s effectiveness in boosting perception and reasoning.\n3. Data Generation Advantage: The proposed scalable data generation method is of notable value, as it effectively addresses the data scarcity challenge in multimodal reinforcement learning by producing high-quality, controllable training datasets.\n4. The paper is clearly written, logically organized, and easy to follow.", "weaknesses": "1. It remains unclear whether AGILE adopts a multi-turn or single-turn evaluation setup for general downstream tasks. The paper only mentions that both AGILE and the baselines use VLMEvalKit as the evaluation framework, but provides no further details about the evaluation protocol. If AGILE employs multi-turn interactions while the baselines use single-turn ones, this may raise concerns about the fairness of the comparison.\n2. The motivation behind the Step Reward design is not well explained. It would be helpful to understand how removing this component affects training dynamics, especially its impact on general downstream vision tasks. Including ablation experiments on this aspect would strengthen the paper’s methodological soundness and justification.\n3. Since AGILE is based on multi-turn interactive reinforcement learning, it likely introduces additional training overhead compared with single-turn RL. The authors are encouraged to discuss the computational cost and efficiency trade-offs. A comparison between single-turn and multi-turn settings in terms of both performance and training cost would make the paper more comprehensive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SmHUPRcj0T", "reviewer": "Reviewer_es5M", "summary": "This paper proposes AGILE, an agentic jigsaw interaction learning framework to enhance visual perception and reasoning in VLMs. It formulates jigsaw solving as an interactive environment where the model generates executable Python actions (Swap, Observe, Crop, Zoom), receives fine-grained visual feedback, and iteratively improves through GRPO-based RL. A cold-start stage uses 1.6K expert trajectories (filtered Gemini 2.5 Pro outputs) to teach instruction-following and code generation; RL is then performed on 15.6K programmatically generated 2×2 jigsaw tasks with controllable difficulty and verifiable rewards (accuracy, format, step). Results show large gains on jigsaw puzzles (2×2 Acc: 9.5%→82.8%; Score: 29.4%→89.0%) and nontrivial generalization across 9 vision benchmarks (+3.1% avg), with especially strong improvements on high-resolution understanding. The paper analyzes data scaling, compares jigsaw vs general QA data under a fixed budget, and includes case studies of emergent behaviors.", "questions": "See weakness.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Clear problem framing: shows that current VLMs perform near random on simple jigsaw tasks, motivating a proxy task for fundamental perception and reasoning.\n\nInteractive, verifiable RL: uses executable code and a rule-based environment, enabling precise, fine-grained feedback and scalable supervision without expensive annotation.\n\nStrong empirical gains on target task: very large improvements on 2×2 and meaningful generalization to 3×3 despite training on 2×2.\n\nCross-task generalization: consistent improvements (+3.1% avg) across diverse downstream benchmarks (HR, real-world, fine-grained, reasoning, hallucination).\n\nInsightful analyses: comparison with General QA data at equal budget; case studies illustrate learned strategies (edge continuity, text alignment, semantic consistency).", "weaknesses": "**Limited task significance and generality**\n\nFixed jigsaw formulation: The current agent–environment interaction is confined to regular grid jigsaws (uniform tiling, no occlusion, no non-rigid deformation, no noise). Such strong scene priors make it hard to cover the diverse layouts and transformations in open-world vision (e.g., free-form crops, affine/perspective distortions, occlusions, and layered structures), limiting the method’s external validity for general visual reasoning.\n\n2×2 performance saturation: Training and reward design are primarily optimized for the 2×2 setting and are near the ceiling (high Acc/Score). There is limited evidence of sustained gains on more complex structures. Overly strong 2×2 metrics may mask real capability gaps on harder tasks.\n\nWeak size generalization: While there is some transfer to 3×3, the accuracy remains low (20.8%). Moreover, larger grids (3×3, 4×4) are not directly trained in the RL stage. The absence of cross-size curricula or memory mechanisms leads to insufficient generalization from small to larger scales.\n\nFixed, non-general action space: The toolset of Swap/Observe/Crop/Zoom is highly tailored to jigsaws and does not seamlessly transfer to general multimodal interactions (e.g., instance-level operations, relation editing, cross-image search). There is a lack of validation and ablation on extensible action semantics (parameterized operations, composable tools).\n\n\n**Uncertain sources of gains on general tasks**\n\nImprovements may stem more from distillation than from the jigsaw task: The cold-start phase relies on Gemini 2.5 Pro–generated and filtered expert trajectories, which may impart strong priors to the base model in instruction-following, code formatting, and visual alignment. Current controls are insufficient to disentangle the “expert distillation effect” (SFT-driven general gains) from the independent contribution of “jigsaw agentic interaction + RL.” Suggested additions:\nRL-only (without expert trajectories) or weak-expert/random-teacher controls;\nControls with distillation from general multimodal tasks at equal scale;\nAblations removing operations tightly tied to expert demonstrations (e.g., Crop/Zoom) to see whether downstream gains persist.\n\nCoupling risk in rewards/training signals: Format rewards and step penalties may encourage “normalized outputs and low-step strategies” rather than genuine perception–reasoning improvements. It is necessary to verify whether the gains reflect core abilities rather than policy biases through analyses of failure types (code errors vs. cognitive errors) and cross-task robustness under occlusion, distortion, and noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UD8ETY5ixz", "reviewer": "Reviewer_yrdc", "summary": "This paper tackles the problem that current VLMs struggle with. For basic perceptual and reasoning tasks like 2x2 jigsaw puzzles, they perform randomly. To fix this, the authors created AGILE, a framework that turns jigsaw solving into an interactive process. The model generates code to do actions like swapping pieces, cropping, or zooming in, and gets real-time visual feedback from the environment to keep improving. The model is trained with high-quality cold-start trajectories and jigsaw images, and the results are pretty impressive: the model’s accuracy on 2x2 jigsaws jumped from 9.5% to 82.8%, and it also got 3.1% better on average across 9 general vision tasks, showing it can generalize well beyond just solving puzzles.", "questions": "As stated in weakness part.", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The authors use a clever way to solve the data scarcity issue for multimodal reinforcement learning: code-based data generation for scalable data curation.\n\n2. Interactive training design is impressive: by making the model actively engage with the jigsaw environment step by step, it builds real perceptual and reasoning skills instead of just memorizing patterns. \n\n3. General improvement on different visual tasks validates the effectiveness and generalization of the proposed method.", "weaknesses": "1. The author did not mentioned comparisons with direct apply GRPO or other RL training algorithms. As RL can improve the performance of VLMs, I would like to know the real contribution of jigsaw training. Current improvement is combined with RL training.\n\n2. The authors should provide more details about their experiments regarding the comparsion with general QAs. What does general QA dataset consist of? Also, I would like to know the result if I use high resolution general QA dataset to train the model. How it will perform? This will help others validate the contribution of using jigsaw for training more clear.\n\n3. I think the auhors should provide more on the jigsaw bench. What if giving each patch a unique ID and let the model to sort the patch to the correct order? Will it improve the overall performance. I wonder if the code excecution is the best way of doing this? How about using ID and sort them, and then use this to train the model. VLMs are not good at coding, which will make them harder to solve problems in such settings. More details and comparisons are needed. \n\n4. Cold-Start only use 1.6K data and RL using 16K data, what if training the model longer during SFT? will the improvements become fewer? Comparisons are needed regarding the scaling of training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)