Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

Authors: Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, Tat-Seng Chua
Primary area: foundation or frontier models, including LLMs
Keywords: Large Models; Alignment; Hallucination

Abstract:
The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model’s own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.  Our code is avaliable at \url{https://anonymous.4open.science/r/FISAO-57F0/}.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730278386386

thread_type:
official_review

<thread_object>
creator: reviewer_vdxf

<content>
summary:
This paper first demonstrates the advantages of token-level over sentence-level approaches in handling hallucinations. It then utilizes the similarity between each text token and visual features as a reward to achieve a token-level preference optimization. This optimization method employs expert models to make judgments on entity targets, enabling automatic construction of preference data. Ultimately, it achieves a certain level of performance improvement.

strengths:
This paper:
1. analyzes the significant advantages of token-level processing over sentence-level processing in handling hallucinations, laying a solid foundation for subsequent research on token-level rewards.
2. claims a token-level visual reward, which relies on expert models to discriminate entity targets. Although the approach is not novel, the idea is commendable.
3. conducts some simple experiments and analyses to verify the effectiveness of the method.

weaknesses:
1. The paper overuses formula derivations while glossing over specific methodological details, making the overall writing quite confusing. Especially in section 3.2 and equation 5, there’s a lack of necessary logic, making it difficult to understand.
2. I understand that the data construction method mentioned in the article still relies on fuzzy matching with ground truth based on entities. This seems to be a rather manual and primitive way of data construction, potentially introducing significant systematic errors. Have you tried more precise token selection methods?
3. Overall, I find the experiments still insufficient. The experimental baselines are quite outdated, and the tested benchmarks are not comprehensive enough. Particularly, why is there a claim about handling hallucinations, but the final experiments lack an evaluation of hallucinations? Also, why is there almost no improvement on POPE? What’s the reason for this?

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730654668338

thread_type:
official_review

<thread_object>
creator: reviewer_zuh4

<content>
summary:
The paper proposes Fine-Grained Self-Alignment Optimization (FiSAO) to improve vision-language alignment in VLMs. The core idea is to leverage token-level, fine-grained feedback from the model's own visual encoder to guide the training process. The method avoids the need for external annotations or tools by utilizing the vision encoder as a fine-grained verifier, providing token-level rewards during training.

strengths:
- The paper addresses a critical challenge in morden VLMs. The approach of token-level, fine-grained rewards to improve alignment make much sense to me. 

- The paper presents a theoretical framework, including mathematical proofs.

- The experimental evaluation is thorough and well-designed. The experiments support the benefits of token-level rewards in reducing hallucinations and improving benchmark performance.

weaknesses:
1. The theoretical analysis assumes that the CLIP provides perfect vision-langauge alignment, which is a bit unrealistic as CLIP models are known to have alignment errors in practice. Under this assumption, it’s unsurprising that it can be proved that incorporating the CLIPScore improves the performance. Also, the presented theorem cannot support the benifit of _fine-grained_ reward, which is the core point of this paper. Additionally, the framework also relies on linear relationships between inputs and outputs and the infinite data setting.

2.  The introduction of the MDP perspective seems unnecessary and is not well-justified within the context of the paper. Framing language generation as an MDP may not capture the long-range dependencies and context inherent in language, potentially violating the Markov property. At the end of Section 3.3.1, the author state that “This perspective highlights how fine-grained rewards can be applied ...”--I am unsure about how introducing the MDP framework demonstrates this point. 

3. Regarding Section 3.1

   Overall, this section looks a bit unclear to me. Firstly, the calculation process of token-level reward is not introduced, leading to confusion. Whether it is token-level CLIPScore? (additional question: whether per-sentence average of token-level reward is the same as sentence-level reward?) 

   The authors state that sentence-level rewards show a weak correlation with conventional metrics, implying that sentence-level rewards are not effective. However, metrics like BLEU and ROUGE are known to have limitations in measuring semantic similarity, consider using CiDEr or SBERT semantic simialrity score can make the results more convincing. Also, the paper does not clarify whether token-level rewards have a stronger correlation.

5. Issues with Clarity and Readibility

   - Symbols like $S$ in Equation (7) and the normalization function \( \mathcal{N} \) are not properly defined when first introduced. The S also overlapps with the S in MDP.

   - Figure 1 shows distributions of two types of rewards before the concept and calculation of ‘reward’ is introduced.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718127850

thread_type:
official_review

<thread_object>
creator: reviewer_ik73

<content>
summary:
The paper presents Fine-Grained Self-Alignment Optimization (FiSAO) to improve the alignment of visual and linguistic modalities in vision-language large models (VLLMs). It addresses the challenges of misalignment caused by independent pre-training, which can lead to biased outputs and hallucinations. By leveraging token-level feedback from the vision encoder, FiSAO enhances alignment more effectively than traditional methods that rely on coarse feedback. The findings suggest that this approach reduces the need for external data and improves performance in tasks requiring precise integration of visual and language information.

strengths:
- The paper is well-structured and clearly articulated, making it accessible for readers.
- The introduction of FiSAO is grounded in sound analysis and presents a logical approach to addressing alignment issues.
- Extensive experimental validation supports the effectiveness of FiSAO, showcasing its practical applicability.
- The authors offer valuable insights into enhancing alignment in vision-language large models, contributing meaningfully to the field.

weaknesses:
- In the comparison with other state-of-the-art methods, FiSAO did not achieve results on MM-Vet that are comparable to leading approaches. It would be beneficial to include an analysis of the factors contributing to this outcome and to outline potential strategies for improvement in future iterations. Understanding the specific challenges faced in this context could guide refinements to FiSAO and the design of models.
- Efficiency of FiSAO should be analysed as well.  And how the dependency of pretrained vision model affects the performance?

questions:
Please see above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731029045059

thread_type:
official_review

<thread_object>
creator: reviewer_sjcy

<content>
summary:
This paper introduces Fine-Grained Self-Alignment Optimization (FiSAO), a method designed to improve vision-language alignment in Vision-Language Large Models (VLLMs). Unlike traditional alignment techniques that use coarse, sentence-level feedback, FiSAO leverages token-level feedback from the model’s visual encoder, requiring no additional external data or annotations. This token-level approach addresses hallucination issues with more precise differentiation between hallucinated and correctly grounded outputs.

strengths:
- Token-level rewards is an emerging area of research that is worth exploring. So far, only contemporary work has begun to consider token-level reward methods for VLMs, with this paper being one of the first works in this space.
- The methodological framework is well-defined, with both empirical and theoretical foundations.
- Evaluation is comprehensive, with many baselines and benchmarks.

weaknesses:
- Table 1 does not seem to be very representative of relevant literature. Neither the motivation nor the related work covers existing methods for NLP token-level reward [1,2]. Would be nice to also cover concurrent works for VLMs [3,4] so that differences with recent work are better highlighted (albeit not taken into consideration as a weakness in my review, to the contrary, this shows this area of research is very important).

- It is unclear why not use OpenCHAIR which addresses the limitations of CHAIR for benchmarking object hallucinations.

- Figure 2 alone does not robustly support the observation that sentence-level rewards are unreliable indicators of model performance. To be exact, Figure 2 suggests there exists no *linear* relationship between the CLIP-based sentence rewards and the conversational metrics. Sentence-level CLIP scores might still have value, but in ways not captured by simple linear correlation with BLEU or ROUGE. Perhaps the figure could also show the linear correlation for token-level rewards to make a comparative argument, which would be stronger. 

- Theoretical analysis assumes linear transformations and relationships between the latent representations of the image and text modalities (e.g., in the data generative model for v and t). It also relies on orthogonal matrices U_v and U_t​ for projecting latent variables to high-dimensional representations. While this simplifies the theoretical analysis, it assumes a degree of independence or decorrelation between features that may not exist in complex multimodal settings. There are other simplifications such as sub-Gaussian noise, Gaussian generative likelihood, an infinite data setting, and modeling token-level feedback as a regression problem, all of which understandably simplify (and make possible) the analysis but perhaps should be stated clearly as assumptions.

- The current method relies on predefined sets of objects and labels (from datasets like Detic and COCO) to define common objects for fine-grained reward calculation and the fine-grained reward calculation uses predefined thresholds to distinguish correct and hallucinated tokens. Seems both would require careful tuning.

- Calculating token-level rewards for every token in a generated sequence seems computationally intensive? The paper contains no analysis of the performance - computational overhead tradeoffs against sentence-level rewards.

- The reward in Eq. (8) is too difficult to parse, perhaps consider improving clarity in the draft notation presentation format.

- Please explain what is POVID, etc., in the main paper? The appendix does not seem to be a good place to add this information since these methods are crucial in understanding the results. Also briefly mention what are the eval metrics used for each Table 2 column? I could not locate this information easily.  

- Also, it is unclear if Table 2 shows results where FiSAO is added on top of other methods such as POVID, Human-Prefer, etc. Would it make more sense to evaluate the method with just RLHF sentence-level or token-level feedback to show clear differences? 

[1] Yoon, Eunseop, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. "TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback." In Findings of the Association for Computational Linguistics ACL 2024, pp. 14969-14981. 2024.

[2] Yang, Kailai, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. "Selective Preference Optimization via Token-Level Reward Function Estimation." arXiv preprint arXiv:2408.13518 (2024).

[3] Fu, Deqing, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. "TLDR: Token-Level Detective Reward Model for Large Vision Language Models." arXiv preprint arXiv:2410.04734 (2024). **[Concurrent work]**

[4] Xu, Yuancheng, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. "GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment." arXiv preprint arXiv:2410.08193 (2024). **[Concurrent work]**

questions:
- What are the major differences between this work and recent token-level reward papers in NLP and VLMs?

- What is the correlation between CLIP-based **token-level** rewards and conversational metrics?

-  What are the eval. metrics used for each Table 2 column?

-  To my understanding, Table 2 shows results where FiSAO is added **on top of** other methods such as POVID, Human-Prefer, etc.? Would it make more sense to evaluate the method with just RLHF sentence-level or token-level feedback to show clear differences? 

I find the work interesting and timely, and look forward to the rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731389972665

thread_type:
official_review

<thread_object>
creator: reviewer_3rvt

<content>
summary:
This paper addresses the issue of modality misalignment in Vision-Language Language Models (VLLM). The authors demonstrate that a token-level reward is more effective than a sentence-level reward for aligning modalities. They apply a specially designed token-level reward during preference tuning to mitigate modality misalignment.

strengths:
1. The proposed self-training approach does not need any additional data and external modules to mitigate the misalignment issue.
2. They find that coarse feedback, such as sentence-level rewards, shows a weak correlation with hallucination detection.
3. They are the first method to introduce token-level rewards for VLLMs preference tuning.

weaknesses:
See questions.

questions:
1. In Figure 2, you show that the correlation between CLIP-based sentence rewards and conventional evaluation metrics is weak. Could you also present the correlation between token-level rewards and conventional evaluation metrics?

2. In the Table 3 experiment, it would be valuable to see the performance improvements across all baselines when using your approach. While line 429 mentions that the LLaVA backbone with your method surpasses existing approaches, it is worth noting that the original LLaVA-1.5 already performed well compared to other models. Demonstrating consistent improvements across various VLLMs could serve as strong evidence of your approach’s generalizability.

3. It is clear from Table 3 that your approach does not allow LLaVA-1.5 to surpass the higher-performing baselines on the MM-Vet benchmark. Additionally, Table 4 shows that fine-grained rewards do not yield an advantage on some benchmarks, including MME^c, POPE, and CHAIR. I am particularly interested in understanding which types of benchmarks or backbone architectures benefit most from your approach and where it shows superior performance.

4. In Figure 4, you illustrate the comparison of reward distributions for generated objects on LLaVA-1.5 before and after training. It seems reasonable that training the model with an RL algorithm on the same reward function results in higher average rewards. However, I am unclear about the purpose of showing the shift in reward distributions. Could you elaborate on the rationale behind presenting this distribution shift? Additionally, could you provide a detailed explanation for the design of Equation (8) and the reasoning behind its specific form?

5. Currently, new MLLMs leverage not only CLIP-based models but also other visual encoders, such as DINO-v2 (self-supervised) and SigLip. How adaptable is your approach when applied to MLLMs that incorporate multiple visual encoders? Could you explain any modifications or considerations necessary for your method to handle such configurations effectively?

6. Your approach primarily focuses on object hallucination. I am curious to know whether it also helps mitigate other types of hallucinations, such as those related to actions or spatial relationships. Additionally, could you elaborate on how the size of the expanded set C contributes to the observed performance improvements?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method for vision-language models that uses token-level rewards to improve alignment without requiring additional data. 

### Strengths:
1. Well-grounded theoretical framework
> "The methodological framework is well-defined, with both empirical and theoretical foundations" (Reviewer SJcY)
2. Novel conceptual contribution in token-level processing approach
> "analyzes the significant advantages of token-level processing over sentence-level processing in handling hallucinations, laying a solid foundation for subsequent research on token-level rewards" (Reviewer vDxF)

3. No additional data requirement and practical efficiency
> "The proposed self-training approach does not need any additional data and external modules to mitigate the misalignment issue" (Reviewer 3Rvt)

4. Comprehensive experimental validation 
> "The experimental evaluation is thorough and well-designed. The experiments support the benefits of token-level rewards in reducing hallucinations and improving benchmark performance" (Reviewer ZUH4)

### Weaknesses:
1. Limited improvement on certain benchmarks 
> "It is clear from Table 3 that your approach does not allow LLaVA-1.5 to surpass the higher-performing baselines on the MM-Vet benchmark" (Reviewer 3Rvt)

2. Initial theoretical presentation needed better justification
> "The theoretical analysis assumes that the CLIP provides perfect vision-langauge alignment, which is a bit unrealistic" (Reviewer ZUH4)

3. Computational overhead concerns
> "The paper contains no analysis of the performance - computational overhead tradeoffs against sentence-level rewards" (Reviewer SJcY)


### Justification:
While the paper has limitations on some benchmarks, it makes a valuable contribution to vision-language alignment research. The authors demonstrated strong responsiveness during rebuttal, addressing major concerns with new experiments and analyses. The method is practical (requiring no additional data) and shows competitive performance across most metrics. The computational efficiency analysis and expanded evaluations added during rebuttal strengthen the paper's practical value.