PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning
Authors: Yizhen Zhang
Yang Ding
Shuoshuo Zhang
Xinchen Zhang
Haoling Li
Zhong-Zhi Li
Peijie Wang
Jie Wu
Lei Ji
Yeyun Gong
yelong shen
Yujiu Yang
Venue: NeurIPS 2025 poster ()
Keywords: vision language model
reinforcement learning
multi-image
multimodal reasoning
Primary area: applications
TL;DR: We propose a general reinforcement learning framework tailored for interleaved multimodal tasks by permutating image sequences to simulate varied positional relationships and explore more spatial and positional diversity

Abstract: Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real-world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks, and a multi-stage strategy designed to enhance the exploration-exploitation trade-off, thereby improving learning efficiency and task performance. 
Specifically, we introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional  diversity. Furthermore, we design a rollout filtering mechanism for resampling to focus on trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-used multi-image benchmarks and 3 single-image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin, achieving state-of-the-art performance on multi-image benchmarks, while preserving comparable performance on single-image tasks.

PeRL: Permutation-Enhancee Reinforcement
Learning for Interleaved Vision-Language Reasoning
Yizhen Zhang Yang Ding Shuoshuo Zhang
Xinchen Zhang Haoling Li Zhong-Zhi Li
Peijie Wang
Jie Wu
Lei Ji
Yeyun Gongti Yelong Shen
Yujiu Yang
Tsinghua University "Microsoft "CASIA
https: / thub. wtt.cooccoaaaa//aaaa chmmssstyz//PP
Abstract
Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-RI, recent emerging research has begun ex- ploring the use of reinforcement learning (RL) to enhance vision- lisiua-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real- world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL. tailored for interleaved multimodal tasks, and multi-stage strategy designed to enhance the exploration-exploitaiie trade-off, thereby improving learning efficiency and task performance. Specifi- cally, we introduce permutation of image sequences simulate varied positional relationships to explore more spatial and positional diversity. Furthermore, we design rollout filtering mechanism for resampling to focus trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-use multi- -mage benchmarks and 3 single- image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R R1-related and interleaved VLM baselines by large margin, achiev- ing state-of-the-arr performance on multi-image benchmarks, while preserving comparable performance single-image tasks.
Introduction
Real-world applications often require the ability to understand and reason over multiple images capability that humans naturally possess. response, several multi-image benchmarks (e.g., [14, 19, 35, 46)) have been introduced to support the study of such comprehensive and practically relevant tasks within the research community. Specifically, multi-image multimodal benchmarks span wide range of tasks involving diverse multimodal relationships, including: 1) Low-level visual features, such as comparing illumination, color saturation, or texture; 2) High-level semantic relationships, such emotional associations or conceptual similarities between objects; 3) Temporal reasoning, involving the understanding of event sequences or changes over time; 4) 2D/3D spatial relationships, including rotation, symmetry, and variations in camera viewpoints. Furthermore, the interleaved format of images and text, as introduced works like [3], provides a unified and flexible template that
"Equal contribution. Work done during internship of Yizhen, Shuoshuo, Haoling, Zhong Zhi Jie Microsoft Research.e3 1etiggyiizen244mmaaee tsinghua. edu. en *Corresponing authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Multi-modal benchmark performance comparison
Swwnd.5-VL..
Qwenz-VL--18
T644
WM Euroka Owen
Migician 630K
702
69.0
71.
R1 Onevision 7B 11---78--60KK
Nonyyk12
LaVA-Oneitsin
InternVL2...
70
PeRL PeRL-Ouen(Ours)
60
55
112
642
22.0
123
31
50
46.
28.8
24.3
1011
20
10-
Mantis-Eval
(ID)Multi-mmgg Natural Understanding
(00D)Multi-Image Math Reasoning
Figure 1: Compare PeRL against baseline approaches, including instruction tuned or RI-like rein- forcement learning models, across both in-domain and out-of-domain multi-image benchmarks.
supports both single-image and multi-image scenarios. This format allows for arbitrary arrangements of textual and visual inputs, enabling dynamic and context-aware interactions that more closely resemble real-world reasoning tasks.
Despite the impressive progress of vision-language models (VLMs) on single-image tasks, their abil- ity [16, 53, 54, 5] understand and reason over multiple images remains significantly underexplored. Motivated by this gap, recent works such as [19, 22, 24] have constructed multi-image instruction- tuning datasets further post-train general-purpose VLMs for multi-image tasks. In parallel, models like Flamingo [3],
[22], Migician [24] and Mantis-Idefics [19] have been trained on intericaved image-text data enhance their multi-image understanding capabilities. However, despite these efforts, interleaved VLMs still struggle with reasoning across images, as highlighted in [45]. Their limitations become especially apparent in tasks that require positional alignment, temporal sequencing, or holistic cross- cross-image understanding. Meanwhile, reinforcement learning (RL) has recently emerged as a promising direction for enhancing multimodal reasoning VLMs [34], showing encouraging results on single-image tasks. Nonetheless, these RL-augmented models also fall short when applied to more complex multi-image scenarios, particularly those demanding deep reasoning across spatial and semantic contexts.
Concretely, multi-image scenarios introduce challenges in both spatial object relationships within a single and positional across multiple images. As illustrated in Fig. 2, simply altering the order of input images can lead to incorrect predictions from both general- purpose and instruction-tun -uuned vision-language models. For instance, the general model 20wen-2.5-VL-77 each image independently, incorrectly referencing "bright yellow and blue hues" for both images. Meanwhile, the instruction-tune but misattributes it to the wrong image, indicating a failure in positional alignment. These examples highlight the limitations of current VLMs in maintaining consistent and grounded reasoning across interleaved multi-image inputs.
unniiitannnndr relationships
[7] fails to distinguish the visual attributes of
model Migician [24] correctly identifies the "red and green pattern"
To address these challenges, explore interleaved multimodal reasoning through reinforcement learning and design simple yet effective strategy to mitigate positional bias. Specifically, we develop multi-stage stage data processing pipeline that generates diverse training samples by permuting the order of input images. Since such permutations may alter the semantic alignment between text and images, we correspondingly rephrase the textual input to maintain semantic consistency. However, we observe significant imbalance in the difficulty levels of the resulting training samples leading to inefficient reinforcement learning. To mitigate this, we introduce a rollout filtering mechanism which helps stabilize training and improve learning efficiency. This approach allows our proposed model,

---

Qwen-PeRL, to focus on informative trajectories and better generalize across complex multi-image reasoning tasks.
We summarize our main contributions as follows:
We propose PeRL, simple yet effeective strategy that permutes image orders and rephrases text to address multi-image coreference, while a rollout filtering mechanism improves training efficiency by focusing informative samples.
As shown in Fig. 1, extensive experiments on both in-domain and out-of benchmarks demonstrate that our method achieves ofte-of-the-att performance on multi-image tasks and competitive results on single- image tasks.
2 Related Works
2.1 Multimodal Reasoning
Early efforts multimodal reasoning [36, 56, 25, 41, 4, 26) mainly focus on supervised fine-tuning
(SFT) especially the dataset construction with chain-of-thought (CoT) [28, 27, 61] reasoning steps. Recently. Deepseek-R1- R1-Zero [15] demonstrates remarkable reasoning capabilities, spearheading large-scale reinforcement learning (RL) research in LLM. While VLMs also require substantial reasoning capabilities for rision-anddlanguage tasks, recent investigations employ reinforcement learning to multimodal reasoning and present early progress including URSA [33]. Visual-RFT [31], LMM-R1 [37], VLM-RI [39], Reason-RFT [43], MM-EUREKA [34], R1-One Vision [55], Vision-
RI [17], RI-V [8], RI-VL [58], VL-Rethinker [47], VisualThinker-RI -i-rroo [62], Open VLThinker [12] and Perception-Be
[57]. Most of these works adopt GRPO [38] for RL scaling through data
construction [43, 17, 34, 62] or mixture [37], stati-stage training strategy [37, 12, 17, 34] or
perception-wise [31, 57, 39] or step-wise [58] reward design Distinet from these approaches, VL-Rethinker [47] strategically select high-value samples avoid vanishing advantages, while NoisyRollout [30] introduces moderately distorted images as diverse samples for broad exploration
address the challenge of imperfect visual perception.
2.2 Interleaved VLM
With the significant advancement in single-image tasks, vision-language models (VLMs) [1, 44, 2, 10, 52, 7, 52, 23, 11, 18] have emerged, prefraining on interleaved image-text datasets [48] to address more comprehensive multi-image tasks in real-world scenarios. Building on this, [23, 19, 24] further construct interleaved multi-image-text data to post-train VLMs with instruction tuning. These approaches either exploit learnable cross-attention modules with a Perceiver Resampler [3, 6, 21], or utilize linear projection layer [22, 42] to integrate pretrained language models with visual encoders. Nonetheless, multi-image tasks still face further challenges beyond the perception problem in single-image tasks, including complex spatial and positional/temporal reasoning. To address the bias, [45] proposes SoFt Attention (SoFA), a simple. training- free method to mitigate the position bias of VLMs when processing multiple images. Differently, we propose scale the interleaved VLM with reinforcement learning incentive general multimodal reasoning capability and further design comprehensive rollout strategy for coreference resolution.
3 Problem Formuation and Analysis
The multimodal understanding task is defined as predicting an output answer given a text query interleaved with images. Formally, input instance is represented the textual query and {iz}ー= is the associated set of images. The corresponding output is denoted by y. yielding dataset D {(x;, ,.)) In practice, we interleave text tokens with image representations using a special placeholder token <image> to indicate the presence and position of an image within the input sequence.
(T, where denotes
3

---

Question: image more similar reference image simage 1-7 Optioos: -image -iiigggee_ 3
Question: Which image more similar to reference image "image 1-7 Options: A. -image 35 <inage
by, Qwen-2.5.VV (.om-2..5VI--- 5-V---R Roth images frnture insects with bright, contrasting colors and are perched on leeves, which aligra with the reference image's setting. However, Image has mure similar color scheme overall composition compared the reference image.
17. Qwan-2 5-V1-TB -TB: Roth images fastore insocts viham colors and pollena, partiullatty bright yellow and blue lucs, which present buth A and However, the ovaral color specific details
insect's body and wings more closely aligned with option A compared to option D.
scheme
Migicin:: Boch images show inseets with vibeant color, but the
Migicia:: Beth imagos show insocts with vibror coler, but the reforznce image has yellow and black patten on the body. The reference image also has
refarence image has yellow and black patten on the badl, while
the other images have different color
Image A has blue
detailed viewwhile the other image has and green patten of insect's logs and assennat. Therefore, the reference image more similar the fins image.
and yellow pattern, and Image Therefore. Image A more similar to the reference image.
red and green patterns.
oo-mini-bigg high:
The reference insect (bright yellow with a black
uf-mini-hight The insect image 3 shares the same clon- gated, two-tonee body poitern and overall shape as the referan-ce 1), whereas image has different,
dursal stripe] most closely matched in overall body shape and posture image So the similar is <inage
My Curs: Thh reference image shows inscet Option shows insect with
color schome. the more simalar one sinage 35.
yellow wings, blue body, orange
insect.... Option shows insoett with rod blue colontion, standing on green lenf. The insect has different color scheme componed to
b9 <think-The The rederence image
legs. The insect is perched un a brown branch. Option D shows an insect with red wings Given these observations, the image that is similar the reference
Option becouse shares the
the reference image Option R shows inseet Considering
beight yellow waigs and orange lags, althougl it lacks the black strges and has different background -hhoondddd----------- -thunke-bboadd boxed[A]
the scheme
background, Option more similar
-think-based([])
Figure 2: A showcase highlighting the challenges faced by existing VLMs: 1) Fail to recognize or reason across multiple images; 2) VLMs exhibit positional bias, leading to inconsistent answers when the order of images is permuted even though the semantic of the input keeps the same.
3.1 Positional Bias
One of the critical challenges in multi-image multimodal reasoning is positional coreference, which requires accurately identifying the specific image that corresponds to a given textual reference. However, highlighted in [45], existing rision-language models (VLMs) still suffer from significant performance degradation when the order of input images is altered. As illustrated in Fig. 2, we summarize the major limitations of current VLMs in handling multi-image tasks. Motivated by these observations, we propose a simple yet effective strategy that diversifies image inputs through permutation, while simultaneously modifying the associated text to preserve semantic invariance. This approach encourages the model to develop more robust understanding of positional relationships across images, thereby improving generalization in complex multimodal reasoning scenarios.
3.2 Difficulty Imbalance
Mantis-Instruct [19] is the first multi-image instruction-tuning data, comprising 721K K data. Therefore, we start with the Mantis-Instruck dataset as our training data. Before training, we firstly exploit the V.L [7] model to generate several outputs and calculate the average accuracy among these generations as the difficulty score of each sample for statistical analysis. Fig. 3b represents the difficulty distribution. We find that the distribution is extremely imbalanced, leading to inefficient and unstable reinforcement learning. Inspired by this finding, we further resample the data for balanced difficulty distribution. In the following section, we discuss the pipeline to process the data according to the two observations.
3.3 Data Prepocessing
We develop multi-stage data processing pipeline that systematically filters and reformulates QA pairs from the original training corpus Mantis-Instruct [19] to construct a higher-quality training subset, as illustrated in Fig. 3a.
Initially, we apply a rule-based filter to the original dataset, screening entries based on question types, image quantity, and other relevant attributes. Then we employ GPT-4o [1] to perform format rephrasing on open-ended questions that are challenging to verify through bule-based methods. In the next phase, we implement rollout filtering mechanism using Sive2.55VL-77B [7]. For each question, we perform rollout 10 times and calculate the average accuracy of the responses. Questions then filtered based on their accuracy score, resulting in refined distribution shown in Fig. 3b.
To evaluate the semantic sensitivity of questions to image order, we introduce a semantic variation analysis step using GPT-4o. Specifically, as illustrated in Fig. 3a, given an input-output pair (a, y).

---

Score Distribution Pre- and Post-Difficltyy Filtering
Rule based filter Quastion
Format Rephress
Heter 0.38
Before filtering
2. Image
Open ended
Probabiiiyy Density
6.0
After fitteinn
4.5
Mereer 0.78
Semantic Variation
qucstion
Chick if swapping image order resgarnes charggex
3.0
Finel Dataset
1.5
variaitio lobel
Rollout Filter
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Average Score (8)
(a) Multi-Stage Stage Preprocessing Pipeline
(b) Score Distribution Pre- and Post- Difficulty Filtering
Figure 3: Overview of Data Preprocessing and Difficulty Distribution, (a) The multi-stage pipeline
applies (i) rule-based filtering. (ii) format rephrasing. (iii) rollout-based filtering and (iv) semantic variation checking to curate QA pairs. (b) KDEs of the average score 5 before (blue) and after (orange) rollout-based difficulty filtering, with the mean falling from 0.78 to 0.38.
we apply a random permutation a to reorder the image set, resulting in new input = (i,
with / = (. GPT-4o is then prompted to assess whether the permuted input preserves the semantics of the original input z, i... whether the intended answer should remain unchanged. The corresponding output ý is defined as:
(1) Here, S(x, {0, 1} is a semantic equivalence indicator obtained by prompting GPT-4o, where S = denotes the input semantics are preserved under permutation. The function A or) denotes the transformation applied to the answer when semantic equivalence does not hold. It adapts y the permuted context according to the structure of the answer space Y, such as reindexing visual choices.
S(r, (1 S(x, #)) A(y.o)
This framework facilitates controlled perturbation of the visual input via image sequence permuta- tions, aiming to simulate diverse spatial and temporal configurations and thereby encourage richer generalization and reasoning capabilities.
4 Methods
The goal of multimodal model is to train a policy model no (y([x) parameterized by 0 to optimize the correctness of model response o. In this paper, we exploit reinforcement learning (RL) framework specifically GRPO to learn to improve the multimodal reasoning as presented in Fig.4. Firstly, the policy model me (ylzr takes the input to generate group of responses {or}. Next, the reward model calculates the score r(x, o,, y) for each response o,, which adopts rule-based functions to evaluate the accuracy and format correctness. Furthermore, the group computation module estimates the advantage A, for each response o, to update the policy model.
Specifically, we propose a plug-and-pll and-play module to improve the efficient and effective training through diversified advantage estimation and dynamic resampling. 1) Before exploration, we resample rollouts with variable reward distribution to exploit learned policies effectively. This ensures the learning focuses on trajectories that contribute most to learning optimal behaviors across both spatial and positional dimensions.2) To mitigate positional bias in interleaved multi-image reasoning, introduce diversity in the rollout phase: for Single-image, we keep the original format. for multi-image positional diversity, use permutations of image sequences to simulate varied positional relationships.
4.1 GroupRelative Policy Optimization
GRPO [38] estimates the advantage score based on the reward normalization of a group of responses given specific query. Particularly, the advantage score A, and the corresponding loss are defined as:
r(x, Oi, v) 1/2 E r(x, (z. v)
A
(2)
Vol -
(r 1/ 101 r o3,
To
5

---

Policy Update
Redireace Mookl
Model
Flye
mebli aston
sinape cmain cinage i coreesee with pood hovosrte
iragre
Permutation
ragge wingg Hore i stateeen fake?
with pood
desobiing Freges
(tp.72)
Ogn
Xovarl
Fis
Computuiio
GAmeee the low mudto oho entore devtting thee misuus An image showe pards
Rewanl
Figure 4: An illustration of PeRL framework. (x1, 111 denote original input sample, while {(1,, 1 represent samples generated through permutation operation, n. is the number of permutations per sample and n is the number of rollouts per order. O. and Ai, represent the trajectory, reward and advantage of kıs rollout of sample (Iz, v1) respectively.
|o] min
To(o,(>) A,, clip To(a, (0,
LGRPO
[ota(o: - +,1+))A.)
(3)
where |o] is the group size, is the in response given the input Tota is the old policy model to generate the response o, and my the current policy model to optimize. r(x, oz v) the rule-based reward function to calculate the accuracy and format correctness of the response o;.
4.2 Permutation GRPO
Let (æ1.Vi, denote the original input sample. where I1 consists of query and one more images, and Y1 is the corresponding answer. We augment it in to a group of samples {(x2, (2), (F).)s).... ys)...((n,,111 (n,,+,++))) For multi-image input, augment it by swapping images in a randomly and modify the the answer y correspondingly. To ensure training stability, we introduce a linearly decaying factor ag (scheduled over training steps that controls the probability of applying image swapping per training batch. In the rollout progress, we input the diversified samples to the policy model to generate response o; and the reward model for further reward calculation. After that, we merge all n, + groups of responses together to calculate the baseline, final advantage and loss function
11+1 lod
R
r Oc,k, Vi)
(4)
D i=
|0:1
1=1 k=
T(I14, Oi,ks 3i) R
A
(5)
Σn-th
+1
lool (1(24, v.) R)2
M [..
k=
n,+1 los1
To(or, x2)
To Tolork (x)) Told (or,k xx)) 1 1 e, 1 1 A
[(0) = En. losl
min
As clip(
Jold (ork xx))
Am
- -DD (No Tref)
(6)

---

where o denote the bey response for imput 2;, and A4 represent the advantage of this rollout. In our experiments, we set the number of responses per order |o;| to a same value, i.e. |oi| |02|
=o.= = n. Thus the actual group size is Σ lo:|-
Note that unlike NoisyRollout [30], which performs policy update solely conditioned on the original images, we calculate policy loss of each rollout group og based on augmented inputs 2,, as shown in Equation 6. This is because the answer is also updated after permutation, continuing to use the original prompt as condition is improper and might lead to model collapse.
Algorithm PeRL: Cormmutation-Enhaaae Reinforcement Learning
Input: Current policy no, old policy =0,m. dataset D, training steps Tiuks clip parameter c, initial decay factor Coo, linear decay scheduler 17(-), permutation operation P(.), permutation number ns. rollout number per order n.
for t 1 to Tmxx do Sample batch (x1. V1) D. - (x1,))1) Set decay factor Ot, = ry(coo 6, Tmax)
for = 2 to n1 + do
Generate randomly swapped images and modify answers correspondingly, (x), y1) P((x1.y1)..0
B <- 1 J (1)
end for
for each sample (x; Vú) in B do
Generate responses {o,, Hi=z from to(o x;)
end for
Compute baseline It according to Eq.4
Compute advantages A, according to Eq.5
Update policy according Eq.6
0 +- 0 - (o)( Oold" t
end for
5 Experiment
Training Our training data comprise two parts: 22K multi-image instruction examples curated from the 721K examples in Inntis- [19], and 36K single-image examples from the K12 dataset for RL. We initialize our policy with Siven255VV--BB-ine
[7] and build on the veRL apply one random permutation per sample (n, 1) and generate six responses per order (n 6), yielding 12 rollouts per input. We set the KL coefficient 3 = 0.01, train for 2 epochs with learning rate of 1 10-6 and a batch size of 256. Further details are provided in the appendix.
framework [40]. During RL fine-tuning,
Benchmarks We conduct experiments on both multi-image benchmarks and single-image bench- marks. As the main experiment, we employ Mantis-Eval [19], BLINK [14], MMIU [35] as multi- image benchmarks. Furthermore, we evaluate the generalization on widely used single-image benchmarks including Vatt [32], Math [60] and Vishon [49]. Besides, we also evaluate our model on out-of-domain multi-image benchmarks including Remi [20] and MV-Math [50]. We directly employ the VLMEvalKit [13] to evaluate the performance.
5.1 Main Results
Baselines
The compared models adopt different training strategies. LLaVA-v1.5-7B [29], LLaVA- NeXT-Interleave [23]. LLaVA-One Vision [22], Qwen2-VL-7B [51], Qwen2.5- VL-7B [7]. Deepseek- VL-7B [52], ViterVV1.2 [11], and Intern [9] are instruction-tuned base models. Migician- 630K [24] is further fine-tuned on MGrounding-6600 660K to enhance multi-image grounding. Models [59] and wwen2-VL--BB-VIS--I
[47] are fine-tuned on synthetic VISC-150K dataset, which explicitly encodes inter-image relations.
like -is-OnneiiionniiSSCC
7

---

Table 1: Model performance across VQA and math reasoning benchmarks. Among Instruct- and RL-tuning models, bold numbers denote the best value in each column, and underlined numbers are the second-best, Task categories: multi-image natural scene understanding single-image math reasoning, and multi-image math reasoning. The last column (AVG) is the mean of all eight metrics for rows with complete scores. Models marked with are evaluated via vLL.M inference.
Model
Size
Mantis-Eval
BLINK
MMIU
MathVista
MathiVesse
Mathh
Remi
MV-MATH
AVG
Human
95.67
64.90
68.82
95.80
76.50
Random
GPT-4o
68.00
68.00
55,70
63.80
37.60
30,60
59,96
32,10
Multinodal Goneral Models
VILA-L.5
51.15
39.30
32.45
65.40
Idefics2
LLaVA-vi.5-7P
31.34
37.13
19.20
24.20
14.82
11.74
15.50
ELaVA-NeXT-Inseeeee
7B
51.30
32.40
34.00
17.99
13.81
14.70
LLaVA.OneVision
7B
64.20
48.20
40.32
63.20
26.20
18.30
13.69
18.80
36.61
LLaVA-OneVisiont VISC-----
Migician 630K
7B
69.12
51.53
52.32
58,90
30.10
19.60
8.73
14.88
38.15
InieraVL2
8B
65:40
50.90
42.00
$8.30
37.00
17:40
25.85
Iniern VL2.5
67.00
54.80
16.70
64.10
39.50
19.70
30.92
16.84
42,18
DeppscokVL-778
14.50
Qwwa2-VL-78
7B
69.60
53.20
52.76
58.20
32.50
16.30
28.62
14.53
40.71
Qwea2.5VV1-7
7H
70.80
55.23
$2,00
68.20
46
25.07
36.38
20141
47.90
Muitimodal Reasoning Models
RI-VL-78. D1-VI-78-2600"
18.77
42.62
Qwen2 7B-VCB--V
7B
69.12
55.34
52.76
58.50
R. RI-OOnvisionn-7B
7B
71.76
32.38
46.48
64.10
46.40
29.90
36.96
19.06
43.38
MM Bureka"
69.91
51.49
51.67
73.00
50.30
26.90
43.31
24.39
49.49
Noisy-K112 K12""
23.15
24.99
50.05
Qwen-PeRI, (Ours)"
76.39
58.53
73.00
49.56
28.26
43.38
25.68
51.13
A over the Baseline Model
+5.59
+3.30
+2.23
+4.80
+3.25
+3.19
+7.00
+5.27
+3.23
RI-VL-7B-2608 [58] and Vi-OonVision-- [55] before RL. MM-Eureka [34] and Noisy-K12 [30] are trained only with RL on K12 single-image math data. Our method follows the same RI-only setting, but focuses on multi-image scenario.
trained on 260K and 155K supervised samples
SOTA Results As shown in Table 1, PeRL demonstrates exceptional performance across diverse benchmarks, achieving state-of-the-art results on multimodal natural scene understanding while maintaining competitive performance on mathematical reasoning tasks.
Our model achieves superior results with 76.39 on Mantis-Eyal and 58.53 on BLINK, substantially outperforming previous best models. This achievement is particularly significant considering ap- proach utilizes substantially less training data compared to traditional supervised fine-tuning methods. On single-image mathematical reasoning tasks, PeRL demonstrates competitive performance with scores of 73.00 on Math Vista and 49.56 on Math Verse, comparable to specialized state-of-the-art models MM-Eureka and Noisy-K12.
Most notably, our model exhibits robust generalization on multi multi-image mathematical reasoning tasks despite these benchmarks being out-of-distribution relative to training paradigm. This generalization capability is particularly valuable as our model was only trained on single-image mathematical reasoning and multi-image natural scene understanding datasets.
The comprehensive evaluation across 8 metrics yields an average score of 51.13, highlighting the effectiveness of our proposed training framework tailored specifically for interleaved multimodal tasks.
5.2 Ablation Study
We investigate the effect of diversified permutation on training dynamics and generalization. As shown in Fig. 5a, Permutation GRPO gradually surpasses naive GRPO with more stable training. This is because the permutation disrupts fixed visual token positions, exposing the model to harder cases initially, but enabling it to overcome positional bias through adaptation. To further clarify the principle behind our method, we include a concise theoretical justification. Detailed derivations and formal proofs are provided in Appendix.
Permutation vs. Naive GRPO

---

Naive GRPO Permutation GRPO
Naive GRPO
Naivee GRPO
0.65
Crttic Reward AVG
Diversity
0.60
New M
0.50
0.45
25
50
75
100
125
150
175
25
50
75
100
125
150
175
Staps
Steps
(a) Reward curve comparison.
(b) Rollout diversity comparison.
Figure 5: Permutation GRPO achieves more consistent training behavior.
Table 2: Permutation Ablation: n1 is the number of permuted samples per original input. The total
number of rollouts per input 12 is held constant. n2 indicates no permutation applied, while
n, = 2 indicates two permuted samples input.
Permutation Mantis-Fval
BLINK
MMIU
MathVista
Math
MathVision
Remi
MV-MATH
75.93
56.91
52.26
71.8
49.20
26.51
39.62
24.02
76.39
58.53
54.23
73.0
49.56
28.26
41.65
25.68
77.78
59.50
54.55
72.2
50.38
27.89
39.04
24.54
74.07
57.02
52.17
70.9
46.39
27.00
38.62
23.25
Table 3: Single v.s. Multiple Image Disentanglement Ablation
Data Setting
Benchmark
Multi-image
Single-image
Mantis-Eva
BLINK MMIU
MathVista
Vathe
MathVision
Remi
MV-MATH
x
70.80
55.23
46.30
25.07
36.38
20.41
X
74.54
57.84
53.70
68.5
46.1
26.60
39.12
20.58
X
69.23
53.24
$1.67
71.9
51.4
27.20
41.67
24.26
V
76.39
58.53
54.23
73.0
49.56
28.26
To evaluate the impact on output diversity, we compute the average intra-batch cosine similarity of rollout embeddings using the BGE-Large model?. As shown in Fig. 5b, Permutation GRPO maintains higher diversity throughout training. Given the same prompt, the policy varied visual token orders, which encourages learning arder-invarian representations and improves robustness.
Permutation intensity We investigate the effect of the number of augmented permutations na. As shown in Table 2, increasing n, from 0 (i.e., the naive GRPO baseline without permutation) to 2 improves performance on most benchmarks. For instance, Mantis-Eval increases from 75.93 to 77.78, and BLINK from 56.91 to 59.50, indicating that exposure to varied input orders promotes positional invariance. The improvements spatial structures, such as BLINK and Math Verse. However, n,, = 2 yields marginal degradation on reasoning-centrin benchmarks (e.g., Math Vista and Remi) and incurs additional training cost due to reduced key-value cache reuse. These findings suggest trade-off: permutation augmentation enhances spatial generalization but may impair reasoning performance and training efficiency.
partieularly notable
multi-image tasks with complex
Single vs. Multi-Image Table 3 shows the impact of single-image and multi-image training data. Training exclusively on single-image datasets leads to limited generalization on multi-image benchmarks (e.g., significant drops on Mantis-Eval and BLINK), while training solely on multi- image data similarly reduces effectiveness on single-imag reasoning benchmarks (c.g., Math Vista). Interestingly, even without explicitly training on multi-image math reasoning data, performance improves on diitt--distribution benchmarks (Remi and MV-MATH), highlighting the model's implicit learning of both inter-image relationships and reasoning capabilities. Moreover, Math Vision benefits from multi-i image training, likely due its composite nature of muitiple concatenated math reasoning images, which aligns closely with learned multi-imann reasoning patterns.
Pattps://hugginggaaee co/ Cot.ac//.c..AAAAbbeeeeeee. large--rree----ee en v1.5

---

Conclusion
In this paper, we investigate the multimodal reasoning capabilities of VLMs on general interleaved multi-image tasks. To address the challenges posed by spatial and positional reasoning, we introduce multi-stage strategy that diversifies image order and resamples the training data distribution, enabling more effective and efficient learning. Experimental results demonstrate that resulting model, Qwen-PeRL not only achieves superior performance on multi-image benchmarks but also generalizes well to single-image tasks. As future work, aim investigate adaptive permutation strategies that can dynamically adjust image order based on task difficulty or model uncertainty.
7
Acknowledgement
This work was supported by the National Key Research and Development Program of China (No. 2024YFB280890).
References
[1] Gpt-4v(ision) system card. 2023.
[2] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, 2024.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736. 2022.
[4] Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng. Yulin Luo, Jiajun Cao, Hao Liang Ying Chen, Qi She, al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11770
2024.
[5] Ruichuan An, Sihan Yang, Renrui Zhang, Zijun Shen, Ming Lu, Gaole Dai, Hao Liang, Ziyu Guo, Shilin Yan, Yulin Luo, et al. Unictokens: Boosting personalized understanding and generation via unified concept tokens. arXiv preprint UrXiv:2505.14671 2025.
[6] Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision-language models. arXiv preprint 23Xiv:2308.01390 2023.
[7] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 139233,,... 2025.
[8] Liang Chen, Lei Li. Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generaliza- tion ability in vision-languag models with less than $3. https: //gptthb///tttt./////// com com/Deep-Agent/ R1-V, V. 2025. Accessed: 2025-02-02.
[9] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint urXiv:2412.05271, 2024.
[10] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):20010, 2024.
10

---

[11] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, al. Internl:: Scaling up vision foundation models and aligning for generie visual- risual-linguistic tasks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 24185-24199 2024.
[12] Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement. arXiv preprint arXiv:2503. 17352, 2025.
[13] Haodong Duan. Junming Yang, Yuxuan Qiao, Xinyu Fang. Lin Chen, Yuan Liu, Xiaoyi Dong. Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11198-1 11201, 2024.
[14] Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang. Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 466-166. Springer, 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning, arXiv preprint arXiv:2501.12948 2025.
[16] Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang. Jianyu Jiang, Jiawei Wang, et al. Seedl. 5-vl technical report. arXiv preprint arXiv:2505.07062. 2025.
[17] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprint wrXiv:2503.06744 2025.
[18] Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yan Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, and Yujiu Yang. Map: Multimodal uncertainty- aware vision-language pre-training model. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23262-23271, 2022.
[19] Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning, arXiv preprint arXiv:2405.01488 2024.
[20] Mehran Kazemi, Nishanth Dikkala, Ankit Anand, Petar Devic, Ishita Dasgupta, Fangyu Liu, Bahare Fatemi, Pranjal Awasthi, Sreenivas Gollapudi, Dee Guo, et al. Remi: A dataset for reasoning with multiple images. Advances in Neural Information Processing Systems, 37:60088 60109, 2024.
[21] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint wwXiv:2306.05425. 2023.
[22] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 033366, 2024.
[23] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llav--nex--mterlaavee Tackling multi culti-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895. 2024.
[24] You Li, Heyu Huang, Chi Chen, Kaiyu Huang, Chao Huang, Zonghao Guo, Zhiyuan Liu, Jinan Xu, Yuhua Li, Ruixuan Li, et al. Migician: Revealing the magic of free-form multi-image grounding in multimodal large language models. arXiv preprint arXiv:2501.05767 2025.
[25] Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang. Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jie Wang, Xiuyi Chen, et From system to system 2: survey of reasoning large language models. arXiv preprint 1711v 2025.
11

---

[26] Weifeng Lin, Xinyu Wei, Ruichuan An, Tianhe Ren, Tingwei Chen, Renrui Zhang, Ziyu Guo, Wentao Zhang, Lei Zhang, and Hongsheng Li. Perceive anything: Recognize, explain. caption. and segment anything in images and videos, 2025.
[27] Zicheng Lin, Zhibin Gou, Tian Liang, Ruilin Luo, Haowei Liu, and Yujuu Yang. Criticbench: Benchmarking IIms for critique-correct reasoning. In Annual Meeting of the Association for Computational Linguistics, 2024.
[28] Zicheng Lin, Tian Liang, Jiahao Xu, Xing Wang, Ruilin Luo, Chufan Shi, Siheng Li, Yujiu Yang, and Zhaopeng Tu. Critical tokens matter: Token-level contrastive estimation enhances Ilm's s reasoning capability, ArXiv, abs/2411. 2024.
[29] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26330 2024.
[30] Xiangyan Liu, Jinjie Ni, Zijian Wu, Chao Du, Longxu Dou, Haonan Wang, Tianyu Pang, and Michael Qizhe Shieh. Noisyrollout: Reinforcing visual reasoning with data augmentation. arXiv preprint arXiv:2504.13055, 2025.
[31] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0/785, 2025.
[32] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255 2023.
[33] Ruilin Luo, Zhuofan Zheng, Yifan Wang, Xinzhe Ni, Zicheng Lin, Songtao Jiang, Yiyao Yu, Chufan Shi, Lei Wang, Ruihang Chu, Jin Zeng, and Yujiu Yang. Unlocking multimodal mathematical reasoning via process reward model. 2025.
[34] Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint MMXiv:2503.07365 2025.
[35] Fanqing Meng, Jin Wang, Chuanhao Li, Quanfeng Lu, Hao Tian, Jiaqi Liao, Xizhou Zhu, Jifeng Dai, Yu Qiao, Ping Luo, et al. Mmiu: Multimodal multi-imag understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.0 02718, 2024.
[36] Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420-14431, 2024.
[37] Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang, Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint vrXiv:2503.07556 2025.
[38] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 0000000000.0..... 2024.
[39] Haozhan Shen, Peng Liu, Jingeheng Li, Chumxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, al. Vlm-rl: A stable and generalizable rl-style large lasion-language model. arXiv preprint arXiv:2504.07611 2025.
[40] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
[41] Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, and Yujiu Yang. Chart- mimic: Evaluating Imm's s cross-modal reasoning capability via chart-to-code generation. ArXiv, abs/2406.099611, 2024.
12

---

[42] Quan Sun, Yufeng Cui, Xiaosong Zhang. Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pages 14398-14409, 2024.
[43] Huajie Tan, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuning for visual reasoning. arXiv preprint wrxiv:2503.20752 2025.
[44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- -aan-Baptstte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:23 /8855 2023.
[45] Xinyu Tian, Shu Zou, Zhaoyuan Yang. and Jing Zhang. Identifying and mitigating position bias of multi-image vision-languee Languala models. arXiv preprint arXiv:2503.13792, 2025.
[46] Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust multi-image understanding, arXiv preprint 1xXiv:2406.09411 2024.
[47] Haozhe Wang, Chao Qu Zuming Huang. Wei Chu, Fangzhen Lin, and Wenhu Chen. VI- rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.0888. 2025.
[48] Junjie Wang, Yin Zhang, Yatai Ji, Yuxiang Zhang, Chunyang Jiang, Yubo Wang, Kang Zhu, Zekun Moore Wang, Tiezhen Wang. Wenhao Huang, Jie Fu, Bei Chen, Qunshu Lin, Minghao Liu, Ge Zhang, and Wenhu Chen. Pin: A knowledge-intensie dataset for paired and interleaved multimodal documents. ArXiv, abs/2406. 13923, 2024.
[49] Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95 2024.
[50] Peijie Wang, Zhong-Zhi Li, Fei Yin, Xin Yang, Dekang Ran, and Cheng-Lin Liu. Mv- math: Evaluating multimodal math reasoning in multi-visual contexts. arXiv preprins 2502v:2502.20808, 2025.
[51] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, al. Qwen2-vl: vl: Enhancing rision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[52] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-v12. Mixture-off -fxture-of-expprts vision- language models for advanced multimodal understanding. arXiv preprint arXiv:2412.10330 2024.
[53] Ling Yang. Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models, 2025.
[54] Ling Yang, Xinchen Zhang, Ye Tian, Chenming Shang, Minghao Xu, Wentao Zhang, and Bin Cui. Hermesflow: Seamlessly closing the gap in multimodal understanding and generation. ArXiv, abs/2502. 12148, 2025.
[55] Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615 2025.
[56] Huanjin Yao, Jiaxing Huang, Wenhao Wu, Jingyi Zhang, Yibo Wang, Shunyu Liu, Yingjie Wang, Yuxin Song. Haocheng Feng, Li Shen, et al. Mulberry: Empowering milm with ol-l reasoning and reflection via collective monte carlo tree search. arXiv preprint arXiv:2412. 18319, 2024.
13

---

[57] En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jian- jian Sun, Chunrui Han, Zheng Ge, et Perception-rl: Pioneering perception policy with reinforcement learning. arXiv preprint arXiv:2504.0753 079544 2025.
[58] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. RI-vI: Learning to reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint 22x.::5503.19933 2025.
[59] Juntian Zhang, Yuhan Liu, Wei Liu, Jian Luan, Rui Yan, al. Weaving context across images: Improving vision-language models through focus-centric visual chains. arXiv preprint arXiv:2504.20199, 2025.
[60] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wa Chang. Yu Qiao, et al. Mathverse: Does your multi-moda Ilm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[61] Xinchen Zhang, Xiaoying Zhang, Youbin Wu, Yanbin Cao, Renrui Zhang, Ruihang Chu, Ling Yang, and Yujuu Yang. Generative universal verifier as multimodal meta-reasoner, 2025.
[62] Hengguang Zhou, Xirui Li, Ruochen Wang, Minhao Cheng. Tianyi Zhou, and Cho-Jui Hsieh. R1-zero's' ss" aha moment" in visual reasoning 2b non-sft model. arXiv preprint arXiv:2503.05132, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the key contributions of PeRL, including the interleaved reasoning strategy and permutation-based training. These are consistently discussed throughout the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Table
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specfiiaatinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification:
See equation 5 4
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Algorithm
1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
Code will release after paper notification
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Seepa///guidde/odeeuueeeee www.ammmissnooooiiii.ooo.. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /Codeeuoeeeeeeeeeeeeeee /
/ cttppscccppbbpcccc/peece
onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
See Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the high computational cost of large-scale RL training and the instability across seeds, we did not report eтrоr bars statistical tests. However, all experiments were repeated under consistent conditions, and the observed trends were stable across multiple benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
See Supplemental Material
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: : htt////eurspp.ee/ee//e cc httppp/lcc//hhiccciieeeeelleeeea
Answer: [Yes]
Justification: The research adheres to the NeurIPS Code of Ethics. does not involve sensitive data, human subjects, or deployment scenarios, and all datasets used are publicly available.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: The paper presents a general reinforcement learning framework for multimodal reasoning, without deploying or evaluating on downstream applications involving human sensitive decisions. Therefore, we do not anticipate direct societal impacts.
18

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: See Section 5
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswithcee thcode com/ conddataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release any new assets. All experiments are conducted using publicly available datasets and existing open-source models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or any research with human subjects. All data used are publicly available and collected from existing datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No relation with human
Guidelines: subjects
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: core method development in this research does not involve LLMMs as any important, original, non-standard components
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / //neruurpp.... www.///oeeeence///000001 for what should or should not be described.
21

---

A
Technical Appendices and Supplementary Material
A.1 Training and Evaluation Details
Table 4: Details evaluation benchmarks.
Benchmark
Description
#samples
Mantis-eval
Multi-image General Understanding QA
217
BLINK
Multi-image General Understanding QA
1901
MMIU
Multi-image General Understanding QA
11698
Math Vista
Single-image Math Reasoning QA
1000 (testmini)
Math Verse
Single-image Math Reasoning QA
3940
Math Vision
Single-image Math Reasoning QA
3040
Remi
Multi-image General Reasoning
2600
MV-Math
Multi-image Math Reasoning
2009
We train our model on 8xH100 GPUs using the GRPO-based framework. The training batch size is 128, with micro-batch size of 4 per GPU. The maximum prompt and response lengths are 8192 and 2048 tokens, respectively. We adopt Qwen2.5- 1-------7BB--i-E
as the backbone, enabling gradient checkpointing and FSDPP without offloading. KI. loss is applied with coefficient of 0.01, while entropy regularization is disabled. The model is optimized for 2 epochs with a learning rate of 1 x 10-6
For evaluation, we use greedy decoding with temperature set to O, top-p to 1, top-k to -1, and maximum generation length of 2048. Evaluation benchmarks include MathVista, MathVerss Math Vision, and BLINK (configured via VLMEvalKit), while Mantis-Eval, MMIU, and MV-MATH (evaluated with official code via vLLM). All evaluations follow consistent decoding settings.Detaily are shown as Table and prompt A.3
A.2 Case study
As shown in Fig 6, we conduct detailed comparison of the response quality across different types of models, including the proprietary model 04-mini-high, open-source base models, and previously trained interleaved models. We observe that the base models exhibit various types of errors, which can be broadly categorized into several patterns: missing relevant images, misinterpreting images, producing incorrect answers despite correct reasoning, and reaching incorrect conclusions despite accurate observations of details. For instance, the beetle question shown above, Qwen frequently makes errors where the reasoning is correct but the final answer is wrong, or where some relevant visual information is overlooked. These are the types of failures we aim to address.
Besides, in case fig 8, we observe a consistent pattern in model behavior across the permutations of images: the model exhibits stronger alignment between the reference image and the image presented first in the sequence of options. This suggests current VLMs have a positional bias in interleaved vision tasks.
"attp:://uugginggaaco.o hatp::///uugggggooooooo///e////a
http://gggg.ccoeo///e////eeeeea V-VL-BB-- Inst-ustree
22

---

Question: How many beetles are climbing on the balls in the image?
A. Four B. Five C. None of the choices provided D. Two One
Qwen 2.5 VL: <thinkinga The image shows bectles actively engaged in ralling ball of dung. There na other beetles visible the balls or the immediate vicinity. Therefore, the number of beetles climbing on the balls is two </thinking> <answer> D. Two </answor>
LLAVA Interleave: <think> The left image shows beetle ball. The right image shows two bestles.
Therefore, the number is three. </think> <answer> D.Two</answer>
o4-mini-high:
Curs:
<thinking> in the first image there are two beetles on the
<thinking>.. sttinknng>.<thhikknng>>
ball. In the second image there is one bottle on the ball.
<answer>None of the choices providodk</answer>
That makes three beetles total, and three not ane the
listed options </thinking> <answer>> None the choices
provided </answe>>
Figure 6: A showcase for motivation
Here is a statement describing these images: The train in one of the images has just come around bend. Is it true or false?
Which star is green? A. <image>B. <image>C.
<image>
(A) False
(B) True
Before
After
image 3 is the reference image, which image of the other two is more similar to the reference image? Image 2 image 3
Could you predict the series of actions in this video and enumerate them order. <image> <image> <image>
<image> <image> <image> <image> <image>
options:
A''each bowl, hold fark, transfer soup, carry cup" B. 'hold cup. reach cup. carry cup'
C.tlip cup. hold plate, pour juice, carry spoon'
'oNe of the choices provided"
Figure 7: Training Data preprocessing pipeline explanation and several cases
23

---

<image_ 1>
<image_
<image_
Before Permuation
Which image is more similar to the reference image <image 1>? Options: A. <image_2 2> B. <image_3>
Qwen-2.5-VI--7B: To determine which image is more similar the reference image, let's analyze the key elements of both images:
Reference Image: 1) Features tall trees with dense forest setting. 2) The lighting appears to be natural, possibly early morning or late afternoon, with sunlight filtering through the trees.
Option A: 1) Also shows forest scene with tall trees. 2) There is less mist fog compared to the reference image.
Option B: 1) Displays forest path surrounded by lush greenery. 2) There is no mist or fog present, and the overall atmosphere is more open and less dense than reference image.
Comparing these elements, Option A is closer the reference image because it maintains the tall trees and dense forest setting.
After Permuation
Which image is more similar to the reference image <image_1>> Options: A. <image_3> B. <image_2>
Qwen-2.5-VL-7B: To determine which image is more similar the reference image, let's analyze the key elements of both images:
Reference Image: 1) Features tall trees with dense forest setting. 2) The lighting suggests early morning or late afternoon, with sunlight filtering through the trees. 3) The ground is covered with green foliage, and there is misty atmosphere.
Option A: Also features forest with tall trees. 2) There is a path visible, which is present in the reference image. 3) The lighting appears to be diffused, possibly due to the mist, but is less pronounced than the reference image. The ground is covered with ferns and other greenery, similar to the reference image.
The lighting is bright and direct, suggesting midday sun. 3) The ground covered with green foliage, but the overall scene lacks the misty atmosphere seen in the reference image.
Option B: 1) Features tall trees with dense forest setting. 2)
Given this analysis, option B lacks the misty atmosphere and has brighter lighting, making less similar to the reference image. The image that is most similar to the Reference Image is Option A. X
Figure 8: Case study demonstrating positional bias. The model correctly identifies the similar image when presented first (Before Permutation) but fails when the order is swapped (After Permutation).
24

---

A.3 prompt
A.3.1
Prompt for data preprocessing
During the data preprocessing stage, we employed GPT-4o in conjunction with the prompts provided below to reformat questions and perform semantic variation checks.
Format Rephrase
Your task is convert given QA conversation into multiple-choice question format, and determine whether the question follows the given question type.
Question Type Definition
1) A question is considered a Reference-Image Comparison it satisfies all the following conditions:
The question presents three
images (e.g., "<image> <image> <image>").
One image clearly identified as the reference image (e.g.. "image is the reference")
The question asks which the remaining images is most similar to or most different from the reference image.
The answer options correspond only the non-reference images.
Task Instructions
Convert the original QA pair into multiple-choice question:
Rephrase the assistant's response into an answer option (e.g.. "A", "B", "C").
Use placeholder tokens (<image>) both the question and the options.
Include only the images being compared (exclude the reference image from the options).
Format the question strictly according to the example below.
2) Determine the type of the question.
Output Format
Return JSON object with the following structure:
{
"question": "<multiple-choit question in specified format>",
"answer": "<correct option letter>", "question_type": "#ReferenceeComarisso or Other>"
Example
Input: "Question: Answer the following question: Here three images: <image 1> <image_ 2> <image_33 If image is the reference image, which image of the other two is more similar the reference image? Answer: The image that is more similar to the reference image is image 2."
Output:
{
"question"; "Answer the following question: Which image is more similar to the reference image <image_ Options: A. <image_2> B. <image_3>", "answer": "A", "ipestionnttyee:: "ReferenceCCmparison"
25

---

Naive GRPO
Permutation GRPO
30
Parcentage (8)
Percenzgen
15
2
0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
-1.00
00.5
00.50
-0.25
0.00
0.25
0.50
0.75
1.00
Advantages
Advantages
(a) Naive GRPO
(b) Permuation GRPO
Figure 9: Difference advantage between Naive GRPO and Permuation GRPO during training stage.
Semantic Variation Check
You will be given question involving one or more images, which are represented using image tokens. The token <image> represents an actual image. The tokens <image 1>, <image_22 etc., refer to specific images by their positions (e.g., <image_ refers to the first image, <image_2 to the second image).
Your task is to determine the following:
only the order of the images (e.g.. <image_1>> <image_24 2>, etc.) is changed, would the answer to the question need to change?
the question structured such that a single main image appears the question body, and other images are referenced in the choices?
Respond the following JSON format:
{
"should_ change"; true false, "is mes_uullccccicc _emmmges_ true or false
Note: This applies to both multiple-choice and fill-in-the-blank questions involving image references.
A.3.2 Prompt for training and evaluation
Reasoning Format
Instruction:
You first think about the reasoning process as an internal monologue and then provide the final answer.
The reasoning process must enclosed within <think> </think> tags.
The final answer must be put in \boxed().
A.4 Adavantage Differences
As Fig 9 shows,the distinction becomes clearer when examining the advantage distributions during training. For multi-image inputs, Naive GRPO yields distribution sharply peaked around zero, indicating that large portion of training examples contribute negligible ineffective gradient signals.
26

---

In contrast, the permutation- based GRPO introduces greater imput diversity by altering image order, which encourages the model genuinely capture positional biases. This diversification leads to more informative advantage signals, facilitating more effective gradient updates during optimization.
A.5 Theoretical Analysis
Here's theoretical analysis of why PeRL is better than GRPO in reducing positional bias, Following the notation, we start by defining GRPO's optimization problem:
To z²) To x)) A(x,,)
Eeup E (a) fe(
BKL( To Tret).
where the advantage defined as
0(x,0) r(2,0) (r(x, Varoo-manllal (r(,, +e
Recall that our reward a verifiable reward that evaluates the correctness of a reasoning or the exccution of the code, meaning that
r(x, o) (0, 13.
We note the probability of success p of the old policy:
p = Pe (x)
Po-maa m) (r(x, o) = =1).
Hence, we have for the mean and variance of a Bernoulli random variable:
E Ea (2) d') = p. This results in the following advantage:
Var Var = =p(1- - p).
1 1-P
w Vp(1 - pp) +e"
ifr(r, 0) = 1,
A(x,
w (p)
Vp(1 ((1-p))+ (f)( = O.
As established by Theorems and 2 in [1], obtain:
meer x) exp (3)
just [ut
- we (pn-1(x)) (=)e)))=00)) (
Tn(o| (x))
where
Zn-1(x) = Pro(z)) exp exp Gut (m)) + (1 - pret(z)) exp (-3we (pn-1(z))
1
Define
1
he. (p)
1 + P exp Poof
-1 kn(1-p)->
GRPO evolves
27

---

GRPO (z)
(x. PoRPO (z)).
Pa
Similar to Theorems and 2, we can show that PeRL updates as
PeRL
(x) = (le,)r)((z)) (()((2))
Pn
where p PoRT denotes the average accuracy of all permutations of the original input æ.
Proof that PeRL has less positional bias than GRPO. For n. have
GRPO Po (Imin)
Po-LL PRRL-(Fiin (Fmin)
PeRL
PeRL (Imax) S Po
GRPO (Xmax))
which holds because all models start from the same reference policy. Here, p(Tmin) and p(Tmax) represent the minimum and maximum accuracies across all image permutations, and assume these correspond to the same permutations for all methods and do not change during training.
For iteration
1, assume
GRPO Pn-1 (Tmin) S
PoRL Pa-1 ( S
Pa-1 (FFmxx) S
PoRL
Pn-1 (Tmax)..
GRPO
Since is increasing for p E [1/2, 1]. which holds for most of our training data, we obtain:
PeRL (Imax) = Reepeee(Rman (ph-R) (PPOR((()) (x)) I he (Temax)) (pr-11 (Imax)) S Derreec (p Pa
GRPO (Imax)) Pn (Imax):
GRPO
Similarly,
PeRL (xmiu) pa
GRPO
(Fmii).
Thus,
GRPO Pn
(Imin) S
Pa (Imin) S
PeRL
Pn
PeRl
(Fmax)
S Pn
GRPO
(Imax).
By induction, this inequality holds for any step under the assumption. This shows that PeRL policy's 's success probabilities are less sensitive to input permutations than GRPO's, proving effectiveness in reducing positional bias.
Enhanced Generalization via Invariance. By forcing the policy to be robust to permutations, we implicitly guide the model to learn an order-invariant representation. This means the model must base decisions on the semantic content the images themselves, rather than on superficial positional cues. Learning invariant representations is classic and powerful principle for improving generalization, as it ensures the model performs robustly on unseen data where permutations may differ from the training set.
Improved Training Stability. Permutation also unintentionally increases the effective difficulty for any given input This lowers the probability of the model answering all inputs correctly, which turn prevents the advantage estimate from collapsing toward zero and causing ineffective gradient updates issue common with overly simple samples. This process ensures more consistent advantage signal for single sample across its permutations (i.e., reduces intra-group variance), which fundamentally stabilizes the overall training process.
28


Reviews:
[{"review_id": "mVavYWCMOe", "reviewer": "Reviewer_4xuh", "summary": "The paper proposes PeRL, an RL framework to improve multi-image interleaved vision-language reasoning of VLMs by addressing positional bias and sample difficulty. It is done by (1) introducing image permutation + text rephrasing to learn order-invariant representation and (2) by filtering out easy samples to maintain a balanced training set termed rollout filtering. The training framework is based on GRPO. PeRL is SOTA on multi-image tasks and competitive on single image tasks.", "questions": "1. The paper identifies positional bias as a key failure mode of existing VLMs and proposes order-invariant training via permutations as a solution. Could the authors elaborate on why positional invariance is the right solution to this issue, as opposed to, say, explicitly modeling position-aware visual references or learning to attend over inter-image relationships in a more structured way?\n\n2. As a follow-up, while the proposed permutation-based augmentation improves order invariance, have the authors explored robustness to more complex variations such as occlusions, missing images, or distractors? Would the method still be effective in such cases, or is it narrowly targeted to positional shuffling?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- Well motivated. The paper clearly introduces an issue with existing VLMs and makes intuitive design choices in the training to solve them.\n- Good empirical improvement across multi-image benchmarks without sacrificing other aspects (single image reasoning)\n\nWeaknesses:\n- The core idea of simply permuting image order and rephrasing the text is relatively simple. I do not think it can scale well to complex real-world situations requiring hierarchal reasoning.", "was_revised": true, "final_justification": "The authors' response answered all of my concerns. I am happy with their response. I accept the paper.", "rebuttal": "Dear Reviewer 4xuh,\n\nWe sincerely thank you for your time and your constructive and insightful review. We are very encouraged that you praise our work as being **well-motivated** with intuitive design choices (S1) and for achieving SOTA empirical improvements on multi-image benchmarks **without sacrificing single-image reasoning performance (S2)**. Below, we provide detailed responses to your concerns.\n\n\n---\n\n### **W1: The core idea of simply permuting image order and rephrasing the text is relatively simple. I do not think it can scale well to complex real-world situations requiring hierarchical reasoning.**\n\n**1. Simple yet effective.**   We agree that the core concept is simple and intuitive, but we view this as a key strength of our approach. PeRL builds on the theoretically and empirically validated GRPO framework, inheriting its reliability without altering core reasoning mechanics. This design enables our model to maintain competitive performance on demanding single-image math benchmarks, even against specialized models like **MMEureka** [1] and **R1-OneVision** [2].  \n\nOur work demonstrates that it is possible to improve positional invariance while simultaneously **preserving and strengthening reasoning abilities**. This balance confirms that our “simple” yet principled approach is effective and scalable for increasingly complex tasks.  \n\n**2. Performance in more complex real-world situations.**  Addressing positional bias is, in our view, a **prerequisite for reliable higher-level reasoning**. A model cannot exhibit robust hierarchical reasoning if its predictions are sensitive to arbitrary input permutations. By first teaching the model to be **order-invariant**, we ensure it consistently integrates evidence from all images, establishing the conceptual foundation for more advanced compositional and hierarchical reasoning.  \n\nOur SOTA results across diverse multi-image and single-image benchmarks empirically support the effectiveness of this straightforward yet foundational correction.\n\n**Additional experiments on complex scenarios.** Inspired by Reviewer P1AW’s feedback, we conducted further experimentsto evaluate PeRL’s generalization to more **realistic, challenging settings**:  \n- **SlideVQA** [3]: up to 20 images per item (slide understanding)  \n- **DUDE** [4]: up to 120+ images (document understanding)  \n- **MVBench** [5] and **TempCompass** [6]: video understanding tasks  \n\n| Model          | SlideVQA ANLS | SlideVQA Exact Match | SlideVQA F1 Score | DUDE ANLS | MVBench ACC | TempCompass ACC |\n|----------------|---------------|----------------------|-------------------|-----------|-------------|-----------------|\n| Qwen-VL-2.5 7B | 56.9          | 43.9                 | 51.1              | 60.2      | 61.9        | 68.8            |\n| +PeRL          | **59.2 (+2.3)**   | **46.0 (+2.1)**          | **53.2 (+2.1)**       | **65.5 (+5.3)** | **64.5 (+2.6)** | **70.0 (+1.2)**     |\n\nWe consider video understanding to be an even more challenging and hierarchical reasoning form of multi-image reasoning, since it requires analyzing semantic and relational cues across frames.  \n\nThese findings demonstrate that PeRL not only excels on prior benchmarks but generalizes to more complex real-world settings.\n\n\n---\n\n### **Q1: Why is positional invariance the right solution to this issue, as opposed to, say, explicitly modeling position-aware visual references or learning to attend over inter-image relationships in a more structured way?**\n\nThis is an excellent question that allows us to clarify our motivation. Our method targets a specific, prevalent failure mode in VLMs: exploiting positional shortcuts when image order is semantically irrelevant to the task.  \n\nWe considered alternatives such as explicitly modeling position-aware references. For example, post-hoc attention correction methods like **SoFA** [7] can mitigate bias. However, their benefits **diminish as models become stronger**, and they function more as temporary fixes rather than addressing the **underlying positional bias introduced during pre- or post-training**.  This may stem from the fact that their architectures are specifically designed for the multi-image setting; lacking targeted training, they exhibit weaker generalization to more complex reasoning tasks.\n\nWe found that modifying attention mechanisms in this way sometimes **degrades single-image reasoning performance**, suggesting that our principled permutation approach is a more stable and generalizable solution.\n\n| Model                 | BLINK       | Mantis-Eval  | MathVista    | MathVision   |\n|-----------------------|------------|-------------|-------------|-------------|\n| LLaVA-NeXT+SoFA       | 54.9 (+2.5) | 61.23 (+2.1) | 34.2 (+0.2) | 13.8 (+0.0) |\n| Qwen-2.5-vl-7B+SoFA   | 57.5 (+2.3) | 71.4 (+0.6)  | 66.3 (-1.9) | 25.0 (-0.1) |\n| Qwen-2.5-vl-7B+PeRL   | **58.5 (+3.3)**| **76.4 (+5.6)** | **73.0 (+4.8)** | **28.3 (+3.1)** |\n\n\n\n---\n\n### **Q2: Have you explored robustness to more complex variations such as occlusions, missing images, or distractors? Would the method still be effective in such cases, or is it narrowly targeted to positional shuffling?**\nThank you for this in-depth question. We agree that robustness to such variations is a crucial long-term goal.\n\nWe argue that **mitigating the bias is a foundational step**, enabling the model to better learn inter-image relationships, not just memorize positional patterns, which is essential for handling more complex scenarios.\n\nAlthough most benchmarks do not include these variations, we performed a **preliminary experiment**:  \n- Sampled 1k random items from the BLINK dataset  \n- Simulated a “missing image” scenario by replacing images in some answer options with “None of the answers are correct”\n  \nOn this modified set, accuracy improved from **50.4% to 52.3%**.  \nWe attribute this improvement to two factors: first, for some questions, removing an incorrect option reduces difficulty by eliminating a distractor. Second, and more importantly, it indicates that our model's core understanding of inter-image relationships has genuinely improved, allowing it to better handle unexpected changes in the input. This suggests our method provides a solid foundation for robustness against more complex variations.\n\nThis indicates that our method lays a strong foundation for further robustness against even more complex variations in future research.\n\n\n---\n\n### **References**\n\n[1] Meng, Fanqing, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365 (2025).  \n\n[2] Yang, Yi, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.10615 (2025).  \n\n[3] Tanaka, Ryota, et al. SlideVQA: A dataset for document visual question answering on multiple images. Proceedings of AAAI 37.11 (2023).  \n\n[4] Van Landeghem, Jordy, et al. Document understanding dataset and evaluation (DUDE). ICCV (2023).  \n\n[5] Li, Kunchang, et al. MVBench: A comprehensive multi-modal video understanding benchmark. CVPR (2024).  \n\n[6] Liu, Yuanxin, et al. TempCompass: Do Video LLMs Really Understand Videos? ACL Findings (2024).  \n\n[7] Tian, Xinyu, et al. Identifying and mitigating position bias of multi-image vision-language models. CVPR (2025)."}, {"review_id": "qCyOZzXLb7", "reviewer": "Reviewer_P1AW", "summary": "This work proposed PeRL to solve the hallucination issue when the images are shuffled in different order. This work discussed the problem that when shuffling the images, it gives the challenges for \"spatial reasoning\" and \"position reasoning\". By data augmentation (permuting image orders, rephrases text), the model can learn the invariants with different orders. Also this method applies the filtering mechanism for better training efficiency. Based on the evaluation results on multiple datasets, showing the higher performance than the SoTA baselines.", "questions": "N/A", "limitations": "The limitation is discussed in the conclusion section.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n- The proposed method achieves the SoTA performance in a wide range of benchmarks, showing the effectiveness of the proposed method.\n- Extensive and detailed ablations studies are conducted.\n- The idea is clean, novel and effective.\n\nWeakness:\n\n- When calculating the reward, the rule-based evaluator is adopted (mentioned in Section 4). How's the rule-based evaluation works? Is the reward calculation accurately? As I know, if the reward calculation is noisy, the outcome of the training will be the issue. Is there any quantitative or qualitative analysis for this?\n- The data preprocessing relies on the LLM (GPT4o)'s output (mentioned in Section 3.3). How do you ensure the high LLM's output quality? Also, relying on the close-sourced LLM will be a potential reproductive issue, especially when the model is deprecated by the provider (e.g. OpenAI). Can any open source LLMs achieves the similar quality for data preprocessing?\n- This work focus on the multi-image reasoning. However, it only covers the images less than 5. Discussing the scenarios with much more images will be important. For example, evaluating on SlideVQA (each question is related to 20 slide images).\n\nMinor Weakness:\n- Typo: “Problem Formuation and Analysis” → “Problem Formulation and Analysis” (Section 3 title)\n\nOverall, I think this work has no major issue, so I would like to give accept rating.", "was_revised": true, "final_justification": "The authors' response has addressed all my concerns. I change my rating to accept.", "rebuttal": "Dear Reviewer P1AW,  \n\nWe sincerely thank you for your insightful and constructive feedback. We are greatly encouraged by your positive assessment, particularly that you found our core idea to be **clean, novel, and effective (S1)**. We also appreciate your recognition of our method's **state-of-the-art (SOTA) performance (S2)** across a wide range of benchmarks and your praise for our **extensive and detailed ablation studies (S3)**. Your feedback is invaluable for further improving our paper, and we address your specific questions below.\n\n---\n\n### **W1: How's the rule-based evaluation work? Is the reward calculation accurate? As I know, if the reward calculation is noisy, the outcome of the training will be the issue. Is there any quantitative or qualitative analysis for this?**\n\nIn **GRPO**, the rule-based evaluation is straightforward because our tasks are framed as **multiple-choice questions**. The model generates a chain-of-thought (CoT) reasoning process and outputs its final prediction inside a clearly marked `<answer></answer>` tag. The reward calculation is **highly reliable** because it is based on an **exact match** between the model’s `<answer>` output and the **ground truth answer option**. This process is deterministic and not subjective:  \n\n- If the predicted answer matches the correct option, the model receives a reward of **1**.  \n- Otherwise, the reward is **0**.  \n\nBecause the reward is assigned through a **discrete and unambiguous multiple-choice format**, there is **no label noise** introduced by subjective judgment or ambiguous scoring criteria.  \n\nThis is unlike open-ended tasks, where partial credit or fuzzy matching may introduce inconsistencies when judged by GPT‑4o or rule-based matching.\n\n---\n\n### **W2: The data preprocessing relies on the LLM (GPT‑4o)'s output (mentioned in Section 3.3). How do you ensure the high LLM output quality? Also, relying on the closed-source LLM will be a potential reproducibility issue. Can any open-source LLMs achieve similar quality for data preprocessing?**\n\n#### **On the Quality of GPT‑4o Output for Data Preprocessing**\nWe took several steps to ensure the **high quality and reliability** of the data preprocessing performed by GPT‑4o:\n\n- **Prompt Engineering**: We carefully designed prompts incorporating detailed rules and few-shot examples to guide the model's filtering decisions. For full transparency, the exact prompts are provided in **Appendix A.3**.  \n\n- **Manual Verification**: To quantitatively validate the output, we manually inspected a random sample of 200 filtered data points. **Our audit revealed an accuracy rate of 97.5%**, suggesting a high degree of reliability in our preprocessing pipeline.\n\n#### **On Reproducibility and Closed-Source Models**\nWe share the reviewer’s concern regarding reproducibility and explored an alternative **open-source pipeline**:\n\n- **Open-Source Alternative Experiment**: We used the **OmniCaptioner** [1] model to generate detailed captions for each image, replaced the images with placeholders (e.g., `<image_i>`), and utilized **Qwen2.5‑72B‑Instruct** language model to perform the same filtering task. This pipeline proved feasible—its accuracy was slightly lower than GPT‑4o’s but sufficient to demonstrate that **our method is reproducible with open-source models**.  \n\nWe chose GPT‑4o for our main experiments to guarantee the highest data quality, but we are confident that **future work can transition to fully open-source solutions** as their capabilities continue to improve.\n\n---\n\n### **W3: This work focuses on multi-image reasoning. However, it only covers images less than 5. Discussing scenarios with many more images will be important. For example, evaluating on SlideVQA (20 slide images per question).**\n\nThank you for this insightful suggestion. Evaluating on scenarios with a much larger number of images is indeed crucial for demonstrating the scalability and generalizability of our method.\nFirst, we would like to clarify that the MMIU [2] benchmark, already included in our paper, has a wide distribution of image counts, with 36.4% of its samples containing more than 5 images. Following your advice, we have now included a more fine-grained breakdown of performance by image count in the last table, which shows PeRL consistently improves performance as the number of images increases.\nMore importantly, inspired by your comment, we have conducted extensive new experiments on several challenging benchmarks that involve a large number of images. These include SlideVQA [3] (up to 20 images), DUDE [4] (up to 120+ images), and two video understanding tasks, MVBench [5] and TempCompass [6]. We consider video understanding an even more challenging form of multi-image reasoning, as it requires analyzing both semantic and relational cues across many frames.\n\n\n| Model          | SlideVQA ANLS | SlideVQA Exact Match | SlideVQA F1 Score | DUDE ANLS | MVBench ACC | TempCompass ACC |\n|----------------|---------------|----------------------|-------------------|-----------|-------------|-----------------|\n| Qwen-VL-2.5 7B | 56.9          | 43.9                 | 51.1              | 60.2      | 61.9        | 68.8            |\n| +PeRL          | **59.2 (+2.3)**   | **46.0 (+2.1)**          | **53.2 (+2.1)**       | **65.5 (+5.3)** | **64.5 (+2.6)** | **70.0 (+1.2)**     |\n\nAs shown in above, PeRL demonstrates **consistent and significant performance gains** across these complex benchmarks.\n\n**Why does PeRL generalize so effectively?** To understand why PeRL generalizes so effectively, we conducted a deeper analysis on SlideVQA, breaking down the results by answer and reasoning types. We report two types of answer accuracy: (i) by answer type (Non-span, Single-span, Multi-span) and (ii) by reasoning type (single-hop, multi-hop).We found that PeRL's success is largely because the model truly learns to find and synthesize cross-span evidence from multiple slides. Notably, the performance gain on \"Multi-span\" questions (+3.6) is substantially higher than that from the GRPO baseline (-1.8). Furthermore, the model's \"Multi-hop\" reasoning ability also improves, benefiting from the generalization of the strong single-image reasoning capabilities PeRL maintains.\nThis detailed analysis provides evidence that by mitigating positional bias, PeRL genuinely learns to understand the relationships between all images more holistically, thereby strengthening both its perception and reasoning capabilities.\n\n---\n**(i) Answer type by answer-type**\n\n| Model            | Non-span   | Single-span | Multi-span |\n|------------------|-----------:|-----------:|-----------:|\n| Qwen-vl-2.5-7B   | 49.3       | 60.7       | 12.2       |\n| +GRPO            | 50.5 (+1.2) | 61.4 (+0.7) | 10.4 (−1.8) |\n| +PeRL            | **50.7 (+1.4)** | **62.9 (+2.2)** | **15.8 (+3.6)** |\n\n**(ii) Answer type by reasoning type**\n\n| Model            | single-hop   | multi-hop   |\n|------------------|------------:|-----------:|\n| Qwen-vl-2.5-7B   | 58.0        | 39.3       |\n| +GRPO            | **59.5 (+1.5)** | 39.8 (+0.5) |\n| +PeRL            | 59.3 (+1.3)  | **42.0 (+2.7)** |\n \n\n\n\nFinally, we further include more detailed **MMIU** results mentioned in original paper  below, which also show robust improvement. (The ‘6–10’ column indicates performance on the MMIU subset where the number of images per sample is greater than 6 and less than 10  )\n\n| Model          | 6–10       | 11–15       | 16–20       | 21–64       |\n|----------------|-----------|------------|------------|------------|\n| Qwen-vl-2.5 7B | 36.0      | 52.8       | 54.4       | 59.1       |\n| +PeRL          | **38.3 (+2.3)** | **56.0 (+3.2)** | **57.5 (+3.1)** | **60.6 (+1.5)** |\n\n- **Typo:** “Problem Formuation and Analysis” → “Problem Formulation and Analysis” (Section 3 title)  \nThank you for pointing this out. We will correct this typo in the revised version of the paper.\n\nWe hope this response fully addresses your concerns. We are grateful for the constructive feedback, which has helped us strengthen our paper.\n\n---\n\n### **References**\n\n[1] Lu, Yiting, et al. *Omnicaptioner: One captioner to rule them all.* arXiv preprint arXiv:2504.07089 (2025).  \n\n[2] Meng, Fanqing, et al. *MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models.* ICLR (2024).  \n\n[3] Tanaka, Ryota, et al. *SlideVQA: A dataset for document visual question answering on multiple images.* AAAI (2023).  \n\n[4] Van Landeghem, Jordy, et al. *Document understanding dataset and evaluation (DUDE).* ICCV (2023).  \n\n[5] Li, Kunchang, et al. *MVBench: A comprehensive multi-modal video understanding benchmark.* CVPR (2024).  \n\n[6] Liu, Yuanxin, et al. *TempCompass: Do Video LLMs Really Understand Videos?* ACL Findings (2024)."}, {"review_id": "GCRBdfdbi6", "reviewer": "Reviewer_HKDU", "summary": "This paper introduces a data permuation method to expand the input space for multiple-image positional reasoning. Correspondingly, a modified GRPO is proposed to take the augmented trajectories to compute advantages. The method has been tested on 5 commonly used multi-image benchmarks and 3 single-image benchmarks, showing superior performance over previous approaches.", "questions": "1 what's the motivation of rule-based filter as the first preprocessing step? how much data has been left after this first step filtering?  \n\n2 what's the basis or reasons to choose which model for which preprocessing step? In section 3.3, Qwen2.5-VL-7B is adopted for rollout filtering and GPT-4o for assessing whether the answer should be modified.   \n\n3 After premutation of the images, if the answer has to be adapted, what's the process to modify the answer? Is it automatically generated or human justtification needed?  \n\n\n4 Permutation GRPO requires more rollouts to compute the advantage. What's the computational cost comparing to naive GRPO ?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:  \n1 The whole paper is well strcutred and layout is clear.   \n\n2 The method achives SOTA performance on commonly used benchmarks.  \n\n3 The proposed approach method is straightforward to understand.\n\nWeakness:  \n1 Detailed contexts can be more clearly presented. In the data processing, the authors can elaborate the filtering process and how much data is left before and after filtering.\n\n2 When the image has been permuted, if the answer has to be modified, it may have the scaling problem.\n\n3  Permutation GRPO may take more rollouts compared against native GRPO.", "was_revised": true, "final_justification": "I have read the responses by the authors and the other reviews. The authors have cleared my computational cost concenrs compared against native GRPO and will add more details about data processing in their revision. I would like to keep my original rating.", "rebuttal": "Thank you for your valuable and detailed feedback. We are motivated by your recognition of our work's **clear structure (S1)**, and **SOTA performance (S2)**. Below, we address the concerns as follows.  \n\n---\n\n### **W1&Q1: Motivation of rule-based filtering and data statistics**  \n\nOur work mainly focuses on scenarios involving multiple images. Therefore, this rule-based filter acts as a low-cost preprocessing step to filter samples that:  \n\n- Contain only a single image or no images.  \n- Cannot be verified by rule-based verifiers (e.g., Image caption or other general reasoning tasks)  \n\nStarting from the **Mantis pre-training dataset [1] (721k samples)**, this step filters out over 80% of irrelevant data, leaving **137k samples** for downstream processing. We will include these statistics in the revised paper.  \n\n---\n\n### **Q2: Rationale for model choices**  \n\nWe choose models based on two main considerations: **alignment with our RL training setup** and **the task complexity**.  \n\n1. **Qwen2.5-VL-7B for Rollout Filtering:**  We chose Qwen2.5-VL-7B because it is the base model we use for the subsequent training stage. For RL training to be effective, it is important to calibrate data difficulty to the model’s capacity. Using the base model itself for rollout filtering helps identify and remove samples that are too simple for it to learn from. After this step, the number of samples is further reduced from 137k to 22k.  \n\n2. **GPT-4o for Answer Adaptation:**  In contrast, the task of assessing and modifying answers after image permutation demands a higher level of abstract logical reasoning. This step focuses on how to accurately generate complex, high-quality training samples. We use GPT-4o for its state-of-the-art reasoning capabilities to perform this complex modification automatically, creating challenging data that is essential for our experiments.  \n\n---\n\n### **W2&Q3: Process of adapting answers after image permutation and automatically generated**  \n\nWe appreciate the reviewer’s question regarding the scalability and reliability of this step. The adaptation process is fully automated, which makes it scalable:  \n\n- We construct a detailed prompt, including explicit rules and few-shot examples, to guide GPT-4o in determining whether to modify answers after image permutation.  \n- To ensure the quality of the automatically adapted answers, we manually inspected a representative subset and observed that they were generally accurate and consistent.  \n\nWe acknowledge that fully automated processes can introduce occasional errors; however, in practice, we found the overall quality to be sufficiently high for our experimental purposes. We will clarify this process and its limitations in the revised paper to provide a more balanced discussion.  \n\n---\n\n### **Q3&W4: Permutation GRPO may take more rollouts compared against native GRPO  Permutation GRPO requires more rollouts to compute the advantage. What's the computational cost comparing to naive GRPO ?**\n\nThank you for this important question regarding computational cost. We respectfully clarify that there may be a misunderstanding about the number of rollouts.  \n\nAs stated in the caption of **Table 2 in the original paper**, the total number of rollouts per input is **held constant at 12**, which is identical to the native GRPO setup. For example, when we use one permutation ($n_s=1$), we generate two sequences (the original and the permuted). Each sequence undergoes 6 rollouts, and the advantages from all 12 are then averaged. Therefore, our method does not require more rollouts.  \n\nWhile your concern about the overall computational cost is reasonable. The overhead stems not from more rollouts, but from **reduced KV cache efficiency**. As $n_s$ increases, the frequent changes in image order alter the input context, forcing the model to switch or recompute the KV cache more often.  \n\nTo quantify this, we have performed a new analysis of the wall-clock training time per epoch for different $n_s$ values below, which we will add to the paper. This analysis confirms a modest and predictable increase in computational cost, which we believe is a reasonable trade-off for the significant multi-image performance gains shown in **Table 2**.  \n| $ n_s$  | Training time per step/min |\n|------------|----------------------------|\n| $ n_s = 0 $ | 3.7                        |\n| $ n_s = 1 $| 5.0                        |\n| $ n_s = 2 $ | 9.2                        |\n\n\n---\n### **References**\n\n[1] Jiang, Dongfu, et al. *\"Mantis: Interleaved Multi-Image Instruction Tuning.\"* Transactions on Machine Learning Research. (2024)"}, {"review_id": "0dgWxoSeKu", "reviewer": "Reviewer_rHiB", "summary": "The paper proposes a reinforcement learning-based multimodal reasoning framework, PeRL, targeting the challenge of positional reasoning in interleaved vision-language inputs. By introducing image sequence permutation augmentation and a rollout resampling mechanism, the authors effectively alleviate the positional bias and training inefficiencies that current vision-language models face in multi-image tasks. The experimental evaluation spans multiple mainstream benchmarks, and the model achieves state-of-the-art performance on both multi-image natural scene understanding tasks.", "questions": "1. Could the authors provide a clearer justification of the novelty of the proposed approach? \n\n2. Could more theoretical analysis be included? I encourage the authors to elaborate on the theoretical contribution boundaries of their method—particularly how the permutation mechanism impacts generalization, and whether it can be shown to improve training stability in a principled way.\n\n3. How can the accuracy of the permutation-based samples generated by GPT-4o be ensured?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "In my understanding, the paper has two core contributions: (1) A systematic preprocessing pipeline and semantic consistency check applied to the original multi-image dataset, which reduces and balances the QA difficulty distribution prior to training, significantly improving the stability and convergence of the reinforcement learning process; (2) An enhancement to the GRPO policy optimization algorithm, which leverages the task’s permutation property to increase the number of samples used for advantage estimation—combining responses from standard GRPO sampling and permutation-based augmentation—potentially leading to a more accurate average advantage baseline and thus better policy updates.\n\nThe paper has the following strengths: comprehensive experimental design across diverse evaluation settings, clear and well-structured presentation, and method development that reflects solid research thinking and practical value. \n\nHowever, there are also several weaknesses: \n1. The technical contributions are relatively straightforward, focusing primarily on data and training process optimization. The RL customization for this specific multimodal positional reasoning task feels insufficiently explored or underdeveloped.\n2. The paper lacks in-depth theoretical analysis or quantitative justification regarding generalization or error control, making it appear somewhat empirical.", "was_revised": true, "final_justification": "We have carefully reviewed the authors' rebuttal and the comments from other reviewers. While we appreciate the authors' responses and additional clarifications, several concerns remain unresolved. Notably, multiple reviewers, including myself, raised concerns regarding the accuracy and reliability of the permutation-based samples generated by GPT-4o. Furthermore, specific components of the method—such as the rule-based filtering in preprocessing and the design choices around model selection (e.g., using Qwen2.5-VL-7B for rollout filtering and GPT-4o for modification judgment)—lack sufficient justification. We encourage the authors to carefully revise the paper in light of all reviewer comments to better articulate the theoretical foundations, clarify the data generation process, and strengthen the empirical analysis. I'm looking forward to the final version of this paper. I will increase my score to \"boardline acceptance\" level.", "rebuttal": "We sincerely thank you for your insightful review and accurate summary of our work. We are greatly encouraged by your recognition of our two core contributions—the data pipeline and GRPO enhancement—and your positive assessment of our comprehensive experiments, clear presentation, and practical value. We address your specific points below.\n\n---\n\n### **W1&Q1: The technical contributions are relatively straightforward, focusing primarily on data and training process optimization. The RL customization for this specific multimodal positional reasoning task feels insufficiently explored or underdeveloped. Could the authors provide a clearer justification of the novelty of the proposed approach?**\n\nThank you for this critical feedback. We agree that this is a nascent area of research, and we respectfully frame our work as a foundation, rather than an exhaustive exploration of using RL for this specific challenge.\n\nTo our knowledge, this is the **first attempt to use RL to directly address positional bias** in VLMs. Our customization was deliberately designed to tackle two fundamental problems that arise when applying RL to multi-image tasks:\n\n1. **On the model's reasoning ability:** Positional bias causes critical failures in understanding inter-image relationships. For instance, a model might generate a caption with incorrect positional cues (e.g., describing the second image as the first), which misleads its own subsequent reasoning steps. By enforcing order-invariance, our RL approach improves the model's core ability to locate the correct evidence regardless of image sequence, reducing these types of pattern-matching errors.\n\n2. **On the stability of RL training:** A model sensitive to permutation poses a significant challenge for policy optimization. The same logical reasoning might receive a high reward in one image order and a low reward in another, leading to noisy and biased advantage estimates. Our method of averaging advantages across permutations directly stabilizes the training signal, ensuring the policy update is based on the actual reasoning quality, not on the arbitrary order of inputs.\n\nTherefore, while the technical implementation may seem straightforward, it is a principled solution designed to make the application of RL in this context both feasible and effective. We believe this work opens up a new and promising direction for more sophisticated RL-based solutions in the future.\n\n---\n\n### **W2&Q2: Could more theoretical analysis be included? I encourage the authors to elaborate on the theoretical contribution boundaries of their method—particularly how the permutation mechanism impacts generalization, and whether it can be shown to improve training stability in a principled way.**\n\n\nWe sincerely thank the reviewer for this thoughtful suggestion. We have conducted a deeper investigation into the theoretical underpinnings of our method.\n\n**Here's a theoretical analysis of why PeRL is better than GRPO in reducing positional bias.**\n\nFollowing the notation in [1], we start by defining GRPO’s optimization problem:\n\n$$\\max_{\\theta}\\;\n\\mathbb{E}\\_{x\\sim \\rho}\n\\\\mathbb{E}\\_{o\\sim \\pi\\_{\\mathrm{old}}(\\cdot\\mid q)}\nf\\_{\\epsilon}\\Bigl(\n\\frac{\\pi\\_{\\theta}(o\\mid x)}{\\pi\\_{\\mathrm{old}}(o\\mid x)}\n\\,A(x,o)\n\\Bigr)\n\\-\\\\beta\\\\mathrm{KL}\\bigl(\\pi\\_{\\theta}\\|\\pi\\_{\\mathrm{ref}}\\bigr),\n$$\nwhere the advantage is:\n\n$$\nA(x,o)=\\\\frac{\\\\,r(x,o)\\\\;-\\\\;\\\\mathbb{E}\\_{o'\\\\sim \\\\pi\\_{\\\\mathrm{old}}(\\\\cdot\\\\mid x)}\\\\bigl[r(x,o')\\\\bigr]\\\\;}\n{\\\\sqrt{\\\\mathrm{Var}\\_{o'\\\\sim \\\\pi\\_{\\\\mathrm{old}}(\\\\cdot\\\\mid x)}\\\\bigl[r(x,o')\\\\bigr]+\\\\epsilon}\\\\,}\n$$\n\nRecall that our reward is a verifiable reward that evaluates the correctness of a reasoning or the execution of the code, meaning that:\n\n$$ r(x,o) \\\\in \\\\{0,1\\\\} $$\n\n\nWe note the probability of success \\( p \\) of the old policy:\n\n$$ p := p\\_{\\\\theta_{\\\\text{old}}}(x) = \\\\mathbb{P}\\_{o \\\\sim \\\\pi\\_{\\\\text{old}}(\\\\cdot|x)}\\\\bigl(r(x,o) = 1\\\\bigr) $$\n\nHence, we have for the mean and variance of a Bernoulli random variable:\n\n$$ \\\\mathbb{E}\\_{o' \\\\sim \\\\pi\\_{\\\\text{old}}(\\\\cdot|x)} r(x,o') = p \\\\qquad \\\\text{and} \\\\qquad \\\\mathrm{Var}\\_{o' \\\\sim \\\\pi\\_{\\\\text{old}}(\\\\cdot|x)} r(x,o') = p(1-p). $$\n\n\nAnd this results in the following advantage:\n\n$$\nA(x,o) =\n\\\\begin{cases}\n\\\\omega_{\\\\varepsilon}^{+}(p) = \\\\dfrac{1 - p}{\\\\sqrt{p(1-p)} + \\\\varepsilon}, & \\\\text{if } r(x,o) = 1, \\\\\\\\\n\\\\omega_{\\\\varepsilon}^{-}(p) = \\\\dfrac{p}{\\\\sqrt{p(1-p)} + \\\\varepsilon}, & \\\\text{if } r(x,o) = 0.\n\\\\end{cases}\n$$\n\nAs established by Theorem 1, 2 in [1], we obtain:\n\n$$\n\\\\pi\\_n(o \\\\mid x) = \\\\frac{1}{Z\\_{n-1}(x)} \\\\, \\\\pi\\_{\\\\text{ref}}(o \\\\mid x) \n\\\\exp \\\\left(\n\\\\frac{1}{\\\\beta} \\\\left[ \n\\\\omega_\\\\varepsilon^{+}\\\\bigl(p\\_{n-1}(x)\\\\bigr) \\\\mathbf{1}\\_{\\\\{r(x,o)=1\\\\}} - \\\\omega\\_\\\\varepsilon^{-}\\\\bigl(p\\_{n-1}(x)\\\\bigr) \\\\mathbf{1}\\_{\\\\{r(x,o)=0\\\\}}\n\\\\right]\n\\\\right)\n$$\n\nwhere\n\n$$\nZ\\_{n-1}(x) = p\\_{\\\\text{ref}}(x) \\\\exp \\\\left( \\\\frac{1}{\\\\beta} \\\\omega\\_\\\\varepsilon^{+}\\\\bigl(p\\_{n-1}(x)\\\\bigr) \\\\right) + \\\\left( 1 - p\\_{\\\\text{ref}}(x) \\\\right) \\\\exp \\\\left( - \\\\frac{1}{\\\\beta} \\\\omega\\_\\\\varepsilon^{-}\\\\bigl(p\\_{n-1}(x)\\\\bigr) \\\\right)\n$$\n\nDefine:\n\n$$\nh_{\\\\varepsilon,p_{\\\\mathrm{ref}}}(p)=\\\\frac{1}{1+\\\\frac{1-p_{\\\\mathrm{ref}}}{p_{\\\\mathrm{ref}}}\\\\exp\\\\left(-\\\\frac{1}{\\\\beta}\\\\frac{1}{\\\\sqrt{p(1-p)+\\\\varepsilon}}\\\\right)},\n$$\n\n\nGRPO evolves as:\n\n$$ p_n^{GRPO}(x) = h_{\\varepsilon,p_{\\mathrm{ref}}(x)}(p^{GRPO}_{n-1}(x)). $$\n\nSimilar to Theorems 1 and 2 from [1], we can show that PeRL updates as:\n\n$$ p_n^{PeRL}(x) = h_{\\varepsilon,p_{\\mathrm{ref}}(x)}({\\bar{p}}_{n-1}^{PeRL}(x)) $$\n\nwhere  ${{\\bar{p}}_{n-1}^{PeRL}(x)}$ denotes the average accuracy of all permutations of the original input x.\n\n**Proof that PeRL has less positional bias than GRPO:**\n\nFor $n=0$, we have:\n\n$$ p_0^{GRPO}(x_{min}) \\leq p_0^{PeRL}(x_{min}) \\leq p_0^{PeRL}(x_{max}) \\leq p_0^{GRPO}(x_{max}), $$\n\nwhich holds because all models start from the same reference policy. Here, $p(x_{min})$ and $p(x_{max}) $ represent the minimum and maximum accuracies across all image permutations, and we assume these correspond to the same permutations for all methods and do not change during training process.\n\nFor iteration $n−1$ , assume:\n\n$$ p_{n-1}^{GRPO}(x_{min}) \\leq p_{n-1}^{PeRL}(x_{min}) \\leq p_{n-1}^{PeRL}(x_{max}) \\leq p_{n-1}^{GRPO}(x_{max}), $$\n\nSince $h$  is increasing for $ p \\in [1/2, 1] $, which holds for most of our training data, we obtain:\n\n$$ p_n^{PeRL}(x_{max}) = h_{\\varepsilon,p_{\\mathrm{ref}}(x_{max})}({\\bar{p}}\\_{n-1}^{PeRL}(x)) \\leq h_{\\varepsilon,p_{\\mathrm{ref}}(x_{max})}(p_{n-1}^{PeRL}(x_{max})) \\leq h_{\\varepsilon,p_{\\mathrm{ref}}(x_{max})}(p_{n-1}^{GRPO}(x_{max})) = p_n^{GRPO}(x_{max}). $$\n\nSimilarly:\n\n$$ p_n^{PeRL}(x_{min}) \\geq p_n^{GRPO}(x_{min}). $$\n\nThus:\n\n$$ p_n^{GRPO}(x_{min}) \\leq p_n^{PeRL}(x_{min}) \\leq p_n^{PeRL}(x_{max}) \\leq p_n^{GRPO}(x_{max}). $$\n\nBy induction, this inequality holds for any step n under the assumption. This shows that PeRL policy's success probabilities are less sensitive to input permutations than GRPO's, proving its effectiveness in reducing positional bias.\n\n---\n\n**Enhanced Generalization via Invariance** By forcing the policy to be robust to permutations, we are implicitly guiding the model to learn an **order-invariant representation**. This means the model must base its decisions on the semantic content of the images themselves, rather than on superficial positional cues. Learning invariant representations is a classic and powerful principle for improving generalization, as it ensures the model performs robustly on unseen data where permutations may differ from the training set.\n\n**Improve training stability** Specifically, permutation unintentionally increases the effective difficulty for any given input x. This lowers the probability of the model answering all inputs correctly, which in turn prevents the advantage estimate from collapsing toward zero and causing ineffective gradient updates—an issue common with overly simple samples.\n\nThis process ensures a more consistent advantage signal for a single sample across its permutations (i.e., **it reduces intra-group variance**), which fundamentally stabilizes the overall training process.\n\n\n\n---\n\n### **Q3: Quality Assurance for Permuted Samples Generated by GPT-4o**\n\nWe adopted a two-step strategy to ensure correctness:\n\n1. **Rule-Based Prompting:**  We designed a detailed prompt with explicit rules for how answers must change when permuting images. For instance, if all multiple-choice options are images, swapping them requires updating the correct answer label accordingly. We also included few-shot examples to guide GPT-4o. A partial prompt with key rules and examples is included in Appendix A.3.\n\n2. **Manual Validation:**  To assess the accuracy of the generated permutations, we randomly sampled 200 items for manual validation, and found that **97.5%** of them were correctly generated, which demonstrates that our approach produces highly reliable permutation results in practice.\n\nThis combination of carefully designed rule-based prompting and targeted manual validation ensured the overall accuracy of our generated data.\n\nWe sincerely hope our reply can address your concern.\n\n---\n### **References**\n\n\n[1] Mroueh, Youssef. \"Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplification.\" arXiv preprint arXiv:2503.06639 (2025)."}]

Author rebuttal:

Meta-review: The paper proposes a PeRL method for interleaved multimodal tasks, which uses image sequence permutation to increase spatial and positional diversity, and a rollout filtering mechanism for resampling to focus on important trajectories.

The original main concerns from the reviewers include:

- may not scale well to more complex tasks and more images (4xuh, P1AW)
- unclear details of the method such as the rule-based reward, data processing (P1AW, HKDU)
- lack of evaluation of data preprocessing from GPT4o (P1AW, rHiB)
- computational cost (HKDU)
- theoretical analysis (rHiB)

The rebuttal addressed most of these concerns, and thus reviewers have increased scores to 2 Accepts and 2 Borderline Accepts. The authors should update the paper according to the reviewers' comments in the final version.
Decision: Accept (poster)