Visual Jigsaw Post-Training Improves MLLMs
Authors: name: Penghao Wu, name: Yushan Zhang, name: Haiwen Diao, name: Bo Li, name: Lewei Lu, name: Ziwei Liu

Reinforcement learning based post-training has recently emerged as a powerful
paradigm for enhancing the alignment and reasoning capabilities of multimodal
large language models (MLLMs). While vision-centric post-training is crucial
for enhancing MLLMs' intrinsic understanding of visual signals, current
post-training paradigms are predominantly text-centric, where dense visual
inputs are only leveraged to extract sparse cues for text-based reasoning.
There exist a few approaches in this direction, however, they often still rely
on text as an intermediate mediator or introduce additional visual generative
designs. In this work, we introduce Visual Jigsaw, a generic self-supervised
post-training framework designed to strengthen visual understanding in MLLMs.
Visual Jigsaw is formulated as a general ordering task: visual inputs are
partitioned, shuffled, and the model must reconstruct the visual information by
producing the correct permutation in natural language. This naturally aligns
with reinforcement learning from verifiable rewards (RLVR), requires no
additional visual generative components, and derives its supervisory signal
automatically without any annotations. We instantiate Visual Jigsaw across
three visual modalities, including images, videos, and 3D data. Extensive
experiments demonstrate substantial improvements in fine-grained perception,
temporal reasoning, and 3D spatial understanding. Our findings highlight the
potential of self-supervised vision-centric tasks in post-training MLLMs and
aim to inspire further research on vision-centric pretext designs. Project
Page: https://penghao-wu.github.io/visual_jigsaw/