Title: PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision language model

reinforcement learning

multi-image

multimodal reasoning
Abstract: Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real-world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks, and a multi-stage strategy designed to enhance the exploration-exploitation trade-off, thereby improving learning efficiency and task performance. 
Specifically, we introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional  diversity. Furthermore, we design a rollout filtering mechanism for resampling to focus on trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-used multi-image benchmarks and 3 single-image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin, achieving state-of-the-art performance on multi-image benchmarks, while preserving comparable performance on single-image tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a reinforcement learning-based multimodal reasoning framework, PeRL, targeting the challenge of positional reasoning in interleaved vision-language inputs. By introducing image sequence permutation augmentation and a rollout resampling mechanism, the authors effectively alleviate the positional bias and training inefficiencies that current vision-language models face in multi-image tasks. The experimental evaluation spans multiple mainstream benchmarks, and the model achieves state-of-the-art performance on both multi-image natural scene understanding tasks.\n\nSTRENGTHS AND WEAKNESSES:\nIn my understanding, the paper has two core contributions: (1) A systematic preprocessing pipeline and semantic consistency check applied to the original multi-image dataset, which reduces and balances the QA difficulty distribution prior to training, significantly improving the stability and convergence of the reinforcement learning process; (2) An enhancement to the GRPO policy optimization algorithm, which leverages the task’s permutation property to increase the number of samples used for advantage estimation—combining responses from standard GRPO sampling and permutation-based augmentation—potentially leading to a more accurate average advantage baseline and thus better policy updates.\n\nThe paper has the following strengths: comprehensive experimental design across diverse evaluation settings, clear and well-structured presentation, and method development that reflects solid research thinking and practical value. \n\nHowever, there are also several weaknesses: \n1. The technical contributions are relatively straightforward, focusing primarily on data and training process optimization. The RL customization for this specific multimodal positional reasoning task feels insufficiently explored or underdeveloped.\n2. The paper lacks in-depth theoretical analysis or quantitative justification regarding generalization or error control, making it appear somewhat empirical.\n\nQUESTIONS:\n1. Could the authors provide a clearer justification of the novelty of the proposed approach? \n\n2. Could more theoretical analysis be included? I encourage the authors to elaborate on the theoretical contribution boundaries of their method—particularly how the permutation mechanism impacts generalization, and whether it can be shown to improve training stability in a principled way.\n\n3. How can the accuracy of the permutation-based samples generated by GPT-4o be ensured?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

comments: SUMMARY:
This paper introduces a data permuation method to expand the input space for multiple-image positional reasoning. Correspondingly, a modified GRPO is proposed to take the augmented trajectories to compute advantages. The method has been tested on 5 commonly used multi-image benchmarks and 3 single-image benchmarks, showing superior performance over previous approaches.

STRENGTHS AND WEAKNESSES:
Strengths:  
1 The whole paper is well strcutred and layout is clear.   

2 The method achives SOTA performance on commonly used benchmarks.  

3 The proposed approach method is straightforward to understand.

Weakness:  
1 Detailed contexts can be more clearly presented. In the data processing, the authors can elaborate the filtering process and how much data is left before and after filtering.

2 When the image has been permuted, if the answer has to be modified, it may have the scaling problem.

3  Permutation GRPO may take more rollouts compared against native GRPO.

QUESTIONS:
1 what's the motivation of rule-based filter as the first preprocessing step? how much data has been left after this first step filtering?  

2 what's the basis or reasons to choose which model for which preprocessing step? In section 3.3, Qwen2.5-VL-7B is adopted for rollout filtering and GPT-4o for assessing whether the answer should be modified.   

3 After premutation of the images, if the answer has to be adapted, what's the process to modify the answer? Is it automatically generated or human justtification needed?  


4 Permutation GRPO requires more rollouts to compute the advantage. What's the computational cost comparing to naive GRPO ?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed PeRL to solve the hallucination issue when the images are shuffled in different order. This work discussed the problem that when shuffling the images, it gives the challenges for \"spatial reasoning\" and \"position reasoning\". By data augmentation (permuting image orders, rephrases text), the model can learn the invariants with different orders. Also this method applies the filtering mechanism for better training efficiency. Based on the evaluation results on multiple datasets, showing the higher performance than the SoTA baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n- The proposed method achieves the SoTA performance in a wide range of benchmarks, showing the effectiveness of the proposed method.\n- Extensive and detailed ablations studies are conducted.\n- The idea is clean, novel and effective.\n\nWeakness:\n\n- When calculating the reward, the rule-based evaluator is adopted (mentioned in Section 4). How's the rule-based evaluation works? Is the reward calculation accurately? As I know, if the reward calculation is noisy, the outcome of the training will be the issue. Is there any quantitative or qualitative analysis for this?\n- The data preprocessing relies on the LLM (GPT4o)'s output (mentioned in Section 3.3). How do you ensure the high LLM's output quality? Also, relying on the close-sourced LLM will be a potential reproductive issue, especially when the model is deprecated by the provider (e.g. OpenAI). Can any open source LLMs achieves the similar quality for data preprocessing?\n- This work focus on the multi-image reasoning. However, it only covers the images less than 5. Discussing the scenarios with much more images will be important. For example, evaluating on SlideVQA (each question is related to 20 slide images).\n\nMinor Weakness:\n- Typo: “Problem Formuation and Analysis” → “Problem Formulation and Analysis” (Section 3 title)\n\nOverall, I think this work has no major issue, so I would like to give accept rating.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe limitation is discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper proposes PeRL, an RL framework to improve multi-image interleaved vision-language reasoning of VLMs by addressing positional bias and sample difficulty. It is done by (1) introducing image permutation + text rephrasing to learn order-invariant representation and (2) by filtering out easy samples to maintain a balanced training set termed rollout filtering. The training framework is based on GRPO. PeRL is SOTA on multi-image tasks and competitive on single image tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
- Well motivated. The paper clearly introduces an issue with existing VLMs and makes intuitive design choices in the training to solve them.
- Good empirical improvement across multi-image benchmarks without sacrificing other aspects (single image reasoning)

Weaknesses:
- The core idea of simply permuting image order and rephrasing the text is relatively simple. I do not think it can scale well to complex real-world situations requiring hierarchal reasoning.

QUESTIONS:
1. The paper identifies positional bias as a key failure mode of existing VLMs and proposes order-invariant training via permutations as a solution. Could the authors elaborate on why positional invariance is the right solution to this issue, as opposed to, say, explicitly modeling position-aware visual references or learning to attend over inter-image relationships in a more structured way?

2. As a follow-up, while the proposed permutation-based augmentation improves order invariance, have the authors explored robustness to more complex variations such as occlusions, missing images, or distractors? Would the method still be effective in such cases, or is it narrowly targeted to positional shuffling?

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03