Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and
  Reasoning in Vision-Language Models
Authors: name: Yu Zeng; name: Wenxuan Huang; name: Shiting Huang; name: Xikun Bao; name: Yukun Qi; name: Yiming Zhao; name: Qiuchen Wang; name: Lin Chen; name: Zehui Chen; name: Huaian Chen; name: Wanli Ouyang; name: Feng Zhao
Published: 2025-10-01 17:58:05

Abstract: Although current large Vision-Language Models (VLMs) have advanced in
multimodal understanding and reasoning, their fundamental perceptual and
reasoning abilities remain limited. Specifically, even on simple jigsaw tasks,
existing VLMs perform near randomly, revealing deficiencies in core perception
and reasoning capabilities. While high-quality vision-language data can enhance
these capabilities, its scarcity and limited scalability impose significant
constraints. To address this, we propose AGILE, an Agentic jiGsaw Interaction
Learning for Enhancing visual perception and reasoning in VLMs. AGILE
formulates jigsaw solving as an interactive process, enabling the model to
progressively engage with the environment. At each step, the model generates
executable code to perform an action based on the current state, while the
environment provides fine-grained visual feedback to guide task completion.
Through this iterative cycle of observation and interaction, the model
incrementally improves its perceptual and reasoning capabilities via
exploration and feedback. Experimental results show that AGILE not only
substantially boosts performance on jigsaw tasks of varying complexity (e.g.,
increasing accuracy from 9.5% to 82.8% under the 2 times 2 setting) but also
demonstrates strong generalization across 9 general vision tasks, achieving an
average improvement of 3.1%. These results indicate notable enhancements in
both perceptual and reasoning abilities. This work opens a new avenue for
advancing reasoning and generalization in multimodal models and provides an
efficient, scalable solution to the scarcity of multimodal reinforcement
learning data. The code and datasets is available at
https://github.com/yuzeng0-0/AGILE .