PerPO: Perceptual Preference Optimization via Discriminative Rewarding

Authors: Zining Zhu, Liang Zhao, Kangheng Lin, Jinze Yang, En Yu, Chenglong Liu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: MLLMs; RLHF; DPO

Abstract:
This paper presents Perceptual Preference Optimization (PerPO), a perception alignment method aimed at addressing the visual discrimination challenges in generative pre-trained multimodal large language models (MLLMs). PerPO employs discriminative rewarding and listwise preference optimization to align MLLMs with human visual perception processes. By utilizing the reward as a quantitative margin for ranking, our method effectively bridges generative preference optimization and discriminative empirical risk minimization. PerPO significantly enhances MLLMs’ visual discrimination capabilities while maintaining their generative strengths, mitigates image-unconditional reward hacking, and ensures consistent performance across visual tasks. This work marks a crucial step towards more perceptually aligned and versatile MLLMs. We also anticipate that PerPO will inspire the community to reconsider MLLM alignment strategies.

Decision: reject
Reviewer scores: 3, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730518285244

thread_type:
official_review

<thread_object>
creator: reviewer_oqhz

<content>
summary:
This paper introduces Perceptual Preference Optimization (PerPO), a method designed to enhance the visual discrimination abilities of multimodal large language models (MLLMs). By using discriminative rewards and listwise preference optimization, PerPO aligns MLLMs with human perception, addressing issues like image-unconditional reward hacking. It improves MLLM performance in visual tasks, such as object grounding and OCR, while maintaining generative capabilities and demonstrating generalizability across models and tasks.

strengths:
1. This paper addresses a unique and intriguing research question by exploring the discriminative abilities of multimodal large language models (MLLMs), a direction not commonly emphasized in current studies.
2. Experiments demonstrate PerPO's superiority over baseline techniques like Direct Preference Optimization (DPO) across multiple benchmarks and visual tasks, highlighting its broad applicability and effectiveness.

weaknesses:
1. The paper argues that the primary limitation in the visual discrimination capabilities of MLLMs is due to a lack of explicit perceptual alignment, but it does not provide sufficient evidence to support this claim. There is no experimental evidence that this issue arises from the alignment stage rather than other factors such as instruction tuning or an insufficient pre-training dataset. I recommend that the authors present more compelling evidence to confirm the assertion that the main challenge in the "limitations in MLLMs' visual discrimination" is mainly related to "perceptual alignment."
2. The current method may encounter limitations when applied to complex visual tasks that require significant contextual understanding. While these challenges are briefly mentioned, they are not fully investigated in the study.

questions:
1. In Table 1, the performance of the LLaVA-Next model with PerPO shows a noticeable decrease. Could the authors explain the reasons behind this performance drop?
2. Could the authors provide results on the effect of scaling up the model on the discriminative reward?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730630569284

thread_type:
official_review

<thread_object>
creator: reviewer_nbrt

<content>
summary:
The paper presents an approach termed Perceptual Preference Optimization (PerPO) aimed at addressing visual discrimination challenges in generative pre-trained multimodal large language models (MLLMs). The goal of PerPO is to enhance the models' visual discrimination capabilities while maintaining their generative strengths. The approach utilizes discriminative rewarding and listwise preference optimization to align MLLMs with human visual perception processes. The experiments demonstrate that PerPO significantly improves model performance across various visual tasks and contributes to more perceptually aligned and versatile MLLMs.

strengths:
The paper has several strengths. Firstly, it addresses an important challenge in the field of MLLMs, i.e., visual discrimination. Secondly, it presents a novel approach (PerPO) that bridges the gap between generative and discriminative functionalities of MLLMs, and does so effectively according to comprehensive empirical evaluations presented in the paper. Moreover, the PerPO framework reduces the reliance on human annotations for model training, which is a substantial contribution towards scalable and efficient training of MLLMs. Furthermore, the premise of the paper, the methodology, and the evaluations are clearly articulated and well organized, which enhances its readability and digestibility.

weaknesses:
One significant weakness is that the effectiveness of PerPO highly depends on specific datasets, potentially limiting its generalization. The paper also acknowledges a limitation where complex tasks may still require human annotations, which might not always be feasible. Moreover, there might be a lack of comprehensive experiments demonstrating the performance of PerPO across a spectrum of different domains, which would validate its claim of general applicability further.

questions:
1. Could you provide a more detailed explanation of the specific enhancements in MLLM's visual discrimination capabilities by PerPO?
2. How does the model perform when presented with complex visual tasks that require intricate discrimination capabilities?
3. How is the limitation associated with dependency on specific datasets planned to be addressed in future work?
4. Are there plans to integrate PerPO with other existing models or approaches to further improve the overall performance of MLLMs in diverse domains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730692270238

thread_type:
official_review

<thread_object>
creator: reviewer_nist

<content>
summary:
This work proposes Perceptual Preference Optimization (PerPO) for aligning multimodal large language models (MLLMs) in discriminative vision-language tasks such as object grounding and OCR. By introducing the discriminative reward (measured by the discrepancy between the prediction and the ground truth) into the pairwise weighting in Listwise Preference Optimization (LiPO), PerPO enables sample-efficient alignment of generative MLLMs in discriminative tasks. Empirical results validates that PerPO achieves better performance than DPO and SFT in visual discriminative tasks such as object grounding and dense OCR.

strengths:
1. The writing is clear and smooth, providing great explanation of the background and the proposed method.

2. Compared with DPO and SFT, PerPO enables more sample-efficient alignment for visual discriminative tasks.

3. PerPO also improves general image understanding and mitigates image-unconditional reward hacking.

weaknesses:
1. The evaluation of general vision-language comprehension is based on LLaVA-Bench-in-the-Wild (LLaVA$^W$), a very tiny benchmark with less than 100 samples. The scores may not sufficiently reflect the MLLM image understanding ability. Larger, widely adopted benchmarks such as VQAv2 and MM-Bench are preferred.

2. In Section 5.1, it is claimed that "discriminative reward also aligns well with human," but the results are evaluated by GPT-4o, not human users.

3. From Figure 2(b), the performance on RefCOCO+ seems to saturate after data size > 5k, and more such data can hurt general vision-language capabilities (shown by lowered LLaVA$^W$ scores). Therefore, including more finetuning data does not seem to support PerPO in this scenario. Do SFT and DPO show similar trends? Is PerPO only better than SFT and DPO when the finetuning data scale is small?

4. The tested MLLM and visual perception tasks are somewhat limited (only LLaVA-1.5 and LLaVA-NeXT on object grounding and dense OCR). Thus, the generalizability of PerPO may be unclear. It would be better to include results on more MLLMs (e.g., Cambrian-1, LLaVA-OneVision, Qwen2-VL) and visual tasks (e.g., LISA or GLaMM on segmentation).

questions:
Please check the weakness section above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716755239

thread_type:
official_review

<thread_object>
creator: reviewer_ppgq

<content>
summary:
This paper proposes PerPO, a listwise reward optimization algorithm to address the poor visual discrimination performance of MLLMs and the reward hacking issues in DPO. Specifically, the authors propose to (1) directly use the task-specific score in the output space as the discriminative reward; (2) using a list of negative discrimination rewards can reduce reward hacking. While the authors show a variety of empirical results, this paper lacks a more informative explanation on why such a method can address the critical issue it mentioned.

strengths:
+ This paper discusses an important topic by investigating the reward optimization issue in MLLMs, and they believe this fundamental issue derives from a misspecified reward definition and reward hacking in DPO. They propose corresponding measures to address issue.
+ This paper contains many empirical results.

weaknesses:
+ While this paper provides many empirical results (e.g. the authors constantly referring to fig.1), the authors only explain their method with intuition, while a more in-depth or theoretical analysis is expected for a paper like this. Only empirical results are not convincing enough.
+ Since this paper simply uses the scalar task-specific discriminative score as the reward, why don't the authors compare their method to PPO while using the reward they defined? It seems equation (8) also seeks to optimize a list of accumulated scalar rewards like PPO did. Could the authors provide more thorough theoretical analysis on the difference of PerPO and PPO, especially in terms of optimization objective and smoothness?
+ It is not very clear why the authors use a list of "negative" rewards? Why woul negative only rewards be sufficient and good enough to optimize using their objective?

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732723899370

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
We are grateful to the reviewers for their comprehensive review and valuable insights. Your thoughtful comments have significantly contributed to improving the quality of our manuscript. Here, we have compiled a summary of the revisions to clearly outline the updates made to the original manuscript. Furthermore, given the similarity of some comments and questions, we have summarized common issues and provided a general response.

1. **Manuscript updates.** We have listed the modifications made to the original manuscript. For detailed content, please refer to the revised version, where the changes are highlighted in blue font.
    1. Add the introduction of "negative samples" in lines 73-76.
    2. Revise the expression related to "negative reward" in lines 51-53 and 157-158.
    3. Replace the GPT-4o judgment results in Fig.3 with human users' assessment and revise the content in lines 416-421. 
    4. Add the experimental results for PerPO’s generalization assessment in Appendix A.1.
    5. Add the performance of PerPO on general vision tasks in Appendix A.2.
    6. Add the assessment details of PerPO by GPT-4o and human users in Appendix A.3.
    
2. **Common issues.** We have summarized and addressed the common concerns raised by the reviewers as follows.
    1. **PerPO's Performance in Complex Visual Tasks.** Despite being trained on specific discriminative tasks, PerPO’s enhanced perceptual capabilities have resulted in notable improvements across complex visual tasks. To substantiate this claim, we conducted additional evaluations using more general visual benchmarks, including **MM-Vet [1], MM-Bench [2], MMMU [3], and VQAv2 [4]**.
    2. **PerPO's Generalization Capability.** To validate our method’s generalizability, we conducted additional comparative experiments utilizing the **LLaVA-OneVision [5]** framework. Moreover, we extended our investigation to encompass **LLaVA-Next-13B [6]**, aiming to assess the efficacy of discriminative rewards in scaling up the model. Notably, PerPO consistently exhibited superior performance across all experimental settings when compared to alternative methods.
    3. **Future Work Plan for PerPO.** To align with a broader range of visual scenarios in the future, we intend to **integrate multiple expert models for data construction and incorporate diverse metrics as our discriminative rewards**. Leveraging the composite discriminative reward, we can sample diverse preference data to achieve perceptual alignment in complex visual tasks.

We genuinely hope our summary offers meaningful insights. We have thoughtfully addressed each comment from every reviewer and look forward to your support and approval.

**References**
[1] Yu et al., Mm-vet: Evaluating large multimodal models for integrated capabilities.
[2] Liu et al., Mmbench: Is your multi-modal model an all-around player?
[3] Yue et al., Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.
[4] Goyal et al., Making the v in vqa matter: Elevating the role of image understanding in visual question answering.
[5] Li et al., Llava-onevision: Easy visual task transfer.
[6] Liu et al., LLaVA-NeXT: Improved reasoning, OCR, and world knowledge.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper addresses visual discrimination in multimodal large language models (MLLMs) by introducing Perceptual Preference Optimization (PerPO), a method that extends listwise preference optimization with discriminative rewards based on the discrepancy between predictions and ground truth. A key innovation lies in the use of a list of negative discrimination rewards to counter reward hacking issues commonly encountered in Direct Preference Optimization (DPO). This design enables sample-efficient alignment of generative MLLMs with human perception, improving performance in visual tasks such as object grounding and OCR. Claimed advantages over prior approaches include enhanced task-specific visual discrimination and robust generalization across models and tasks. Empirical results validate PerPO’s superiority over DPO and SFT.

Four reviewers evaluated this paper, highlighting its major strengths as its ability to bridge the gap between generative and discriminative functionalities of MLLMs, its significant contributions to visual discrimination tasks, its superiority over baseline methods such as DPO and SFT, and the framework's reduced reliance on human annotations.

However, reviewers raised several concerns about the paper, particularly its reliance on empirical results and intuition without sufficient theoretical analysis. The distinction between PerPO and PPO, specifically regarding optimization objectives and smoothness, was highlighted as an area needing deeper exploration. Questions were also raised about the use of negative-only rewards, with reviewers questioning their sufficiency and optimality for the proposed objective. Another significant concern was the limited scope of evaluation, which predominantly focused on small benchmarks (e.g., LLaVA-Bench-in-the-Wild) and a narrow range of tasks (e.g., object grounding and OCR), leaving broader generalizability unverified. Furthermore, claims about perceptual alignment being the primary limitation in MLLMs' visual discrimination were not substantiated with sufficient evidence, as alternative factors such as instruction tuning or pre-training dataset limitations were not thoroughly examined.

During the discussion phase, the authors demonstrated PerPO’s performance on complex visual tasks with additional evaluations conducted on general benchmarks such as MM-Vet, MM-Bench, MMMU, and VQAv2, and also validated the method’s generalization capability with experiments performed using LLaVA-OneVision and LLaVA-Next-13B.

The reviewers partially acknowledged the authors' clarifications and additional experimental results. However, Reviewers pPGQ and NbRT expressed ongoing concerns about the novelty of the proposed approach, noting that it primarily represents a direct application of LiPO with a discriminative reward. Furthermore, while additional results on more benchmarks and tasks and with advanced models were provided during the rebuttal phase, a comprehensive and systematic evaluation was not fully presented due to time constraints.

The ACs concur that the lack of comprehensive and systematic evaluation is a legitimate concern and a notable weakness in the current manuscript. Additionally, the novelty of the proposed approach should be more clearly explained, and the claims regarding perceptual alignment as the primary limitation in MLLMs' visual discrimination should be more thoroughly examined. The manuscript also requires substantial rewriting to better organize the new results and, in particular, integrate them into the main paper. Recognizing the validity of the weaknesses raised by the reviewers, the ACs assess that these shortcomings outweigh the current merits of the paper. Consequently, the ACs cannot recommend the acceptance of the current manuscript.