Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment
Authors: Chenhang Cui
An Zhang
Yiyang Zhou
Zhaorun Chen
Gelei Deng
Huaxiu Yao
Tat-Seng Chua
Venue: ICLR 2025 Poster (2025)
Keywords: Large Models; Alignment; Hallucination
Primary area: foundation or frontier models, including LLMs
TL;DR: A self-alignment method that utilizes a fine-grained verifier to improve vision-language alignment without the need for additional data.

Abstract: The recent advancements in large language models (LLMs) and pre-trained vision models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. Despite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and external datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optimization), a novel self-alignment method that utilizes the model’s own visual encoder as a fine-grained verifier to improve vision-language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FiSAO significantly improves vision-language alignment, even surpassing traditional preference tuning methods that require additional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of token-level rewards being applied to such models.  Our code is avaliable at \url{https://anonymous.4open.science/r/FISAO-57F0/}.

Published as a conference paper at ICLR 2025
FINE-GRAINED VERIFIERS: PREFERENCE MODELING AS NEXT-TOKEN PREDICTION IN VISION-LANGUAGE ALIGNMENT
Chenhang Cui
An Zhang Yiyang Zhou²
Zhaorun Chen
Gelei Deng*
Huaxiu Yao² Tat-Seng Chua?
NNational University of Singapore, PUNC-Chapel Hill, Universiity Chicago, *Nanyang Technological University
ABSTRACT
The recent advancements in large language models (LLMs) and pre-trained vi- sion models have accelerated the development of vision-language large models (VLLMs), enhancing the interaction between visual and linguistic modalities. De- spite their notable success across various domains, VLLMs face challenges in modality alignment, which can lead to issues like hallucinations and unsafe content generation. Current alignment techniques often rely on coarse feedback and exter- nal datasets, limiting scalability and performance. In this paper, we propose FiSAO (Fine-Grained Self-Alignment Optinization), novel self-alignment method that utilizes the model's 's own visual encoder as fine-grained verifier to improve vision- language alignment without the need for additional data. By leveraging token-level feedback from the vision encoder, FISAO significantly improves vision-lane langugee alignment, even surpassing traditional preference tuning methods that require ad- ditional data. Through both theoretical analysis and experimental validation, we demonstrate that FiSAO effectively addresses the misalignment problem in VLLMs, marking the first instance of leken-level rewards being applied such models. Our code is avaliable at
com wwwch/ccccccccoo FILAFFISAAOOOOO ICLR
INTRODUCTION
The advent of large language models (LLMss) (Bronn al. 2020 Touvron al. 2023 Yang et al. 2024) and pre-trained vision models (Radford et al.) 2021a Liu et al. (023c) has propelled vision- language large models (VLLMs) by advancing connections between visual and linguistic modalities through linear projection (Li et al.) 2023b) or q-former (Dai et al. 2023b). These VLLMS have demonstrated notable capabilities across diverse domains such as medical applications (Liu al. 2023b), autonomous driving (Zhou et al. 2023a), and embodied intelligence (Peng et al., 2023) ) However, challenges remain precisely aligning vision and language modalities for integrated inference due to their independent pre-training (Jang et al. 22233 Liu et al. 2024a). This pre-training process often results incompatible modaaiit--ppecific representations, hindering the formation of a coherent aligned representation space during joint training (Jang et al. [003). Misalignment between modalities can lead to safety risks such as biased or inappropriate content generation (Gong et al.
2023
Tu et al. 2023) and hallucinations, where outputs not grounded in visual input (Wang et al. These risks are particularly concerning in tasks like visual question answering (Cui et al.
2023
2023 Fan et al. 2024), OCR (Shi et al., 2023), and image captioning (Gunjal et al. 2024). where
precise alignment is critical.
To address these misalignment issues, recent works have explored strategies such as instruction tun- ing (Liu et al. [2023a Chen et al. (2024b) preference tuning (Yu et al.) 2023a), and post-processing methods (Zhou al. 2023b Yin al. 2023). However, most prevalent alignment methods rely heavily on external datasets (Zhou et al. 2024a), models (Yin et al. 2023), or costly human annota- tions et al. 2023a). Preference tuning, for example, requires extensive manual labeling, either from human experts (Sun et al., 2023 Yu et al.) (023a) or commercial models (Lee et al. 2023 Li et al. 2023b), which imposes significant costs on building reward datasets and limits scalability.
'An Zhang is the corresponding author.

---

Published as a conference paper at ICLR 2025
Table 1: Feature comparison of different preference tuning approaches.
Model Name
Reward Model Additional Data GPT-Assisted
Vlfeedback (Li al. 2023d
Human- Preference (Sun el al. 2023
POVID (hou at. 2024a
FISAO
Worse still, these alignment methods often rely on coarse feedback, such as sentence-level (Zhou et 20245] Deng et al. 2024) or output-level rewards (Li et al. (023d), framing the reward modeling task as a simple classifccttion problem that scores outputs as desirable or undesirable. Focusing solely on assigning a numerical score for an entire output fails to leverage VLLMs' token-level generation capabilities, limiting their ability to perform detailed reasoning and precise objective identification.
To mitigate the abovementioned limitations, we propose Fine- Tine-Grinned Self- belf-Allinmeet Optimization (FiSAO), a method for precisely self-aligning modalities in VLLMs using token-level fine- dine-grained feedback from the vision encoder. Our findings indicate that coarse feedback shows a weak correlation with hallucination detection, while fine-graing ome-grained reward more effectively differentiates between hallucinated and correct outputs (see Section(.1]] In other words, when using hallucination detection as a proxy for alignment measurement, token-level feedback from the vision encoder offers more informative signals for preference tuning compared to coarse scores. Our theoretical analysis further confirms that this fine-grained feedback improves modality alignment (see Section 3.2). Additionally, FiSAO eliminates the need for external annotations or tools by leveraging its vision encoder as fine-grained verifier, rewarding each generated token based on its alignment with the visual input. As result, FiSAO effectively harnesses the model's 's text generation capabilities and demonstrates superior performance compared to preference tuning methods that rely on additional data. We compare FiSAO with other preference tuning approaches in Table [
Our primary contributions can be summarized as follows: We first empirically analyze the dif- ferences between coarse and fine-grained rewards in addressing misalignment issues, finding that coarse feedback from pre-trained vision encoders, such as sentence-level rewards, shows weak correlation with hallucination detection, whereas token-level rewards offer more precise signals for modality alignment. Building these findings, we propose novel self-training approach, FiSAO, which leverages token-level feedback from the model's own visual encoder, eliminating the need for additional data external tools. To the best of our knowledge, FiSAO is the first method intro- duce token-level rewards for VLLMs. We further demonstrate FiSAO's effectiveness in mitigating misalignment through both empirical results and theoretical analysis.
PRELIMINARIES
This section reviews the standard pipeline of preference tuning for VLLMs. as outlined in prior works (Ziegler et al., 2019, Ouyang et al. 2022. Yu et al. 2023a). The process typically consists of three phases: 1) Supervised Fine-Tuning (SFT), 2) Reward Modeling, and 3) Policy Optimization.
Supervised Fine-Tuning (SFT) Phase. Preference tuning for VLLMs usually begins by jointly training a pre-trained language model and a pre-trained vision encoder on a high-quality instruction dataset (Li et al. 2023b: Dai et al. 2023b), resulting in SFT model denoted as TSFT.
Reward Modeling Phase. Given text and visual input e as the prompt, the SFT model TSFT is used to generate pair of responses (y1, 32) TTFT (y/z, r). This pair is then evaluated by humans or AI, with one response marked as preferred yo and the other as less preferred yt, denoted as Ya V/ (r. This preference is assumed to follow latent reward model r" r" xr v), which is not directly observable. To model this underlying preference, the Bradley-Terry (BT) model is commonly employed to define the preference distribution p":
exp(r* (I., U. U.W)) p" (yw 311=) exp(r"(x, v, + exp(r (x, m))
(I)
Given static dataset of comparisons D (x(1), v00) ,l )H sampled from p', we can parametrize a reward model (2 v.y) and estimate its parameters using maximum likelihood estimation. By formulating the estimation of reward model ro(a,, v, y) as binary classification
2

---

Published as a conference paper at ICLR 2025
Halluciated Objects
Fallucinated Objects Objects
Objects
Dansity
Density
36
Score
(a) Distributions of Token-Levd Reward
(b) Distributions of Sentence-Leve Reward
Figure I: Comparison of token-level (Ia) and sentence-level [bb] reward distributions for hallucinated
and correct objects in the LLaVA 1.5 model. Further comparisons can be found in Appendix A.2.2
problem, we define the negative log likelihood loss LR follows:
LR(r D) (s.,,,.....) blog (ra v, yw) (a v, [n))];
where a denotes the logistic function, and reward model T (2 v, is typically initialized from SFT model TSFT, with linear layer added top of the final transformer block to produce scalar output representing the reward prediction (Yu et al.) 2023a). Due to the high costs associated with constructing reward model Tæ. such as annotation and training, some preference tuning methods employ external models or tools to directly provide rewards (Hessel et al. 2021).
Policy Optimization Phase. The goal of the policy optimization phase is to refine the policy model Te using feedback from the reward model re, formulated as:
max Eg, (.w.o..l..ooooo......... (r (Ir, v, v,,)] - Dxi. (Tollor, (TTer e)],
(3)
where controls the deviation from the reference policy nee which is initialized as TSFT- This constraint is essential, as it prevents the model from deviating significantly from the original model Tet. maintains generation diversity, and prevents mode collapse high-reward answers (). Due the discrete nature of language generation, Eqn. iss not differentiable. To solve this issue, the standard
approach (Ziegler et al.) | Ouyang et al. 2022) has been proposed to construct modified reward function r(z, y) (T, v. Blogg np 2r, v) log [ v)) and then maximize it using Proximal Policy Optimization (PPO) (Schulman al. 2017).
Although the above preference tuning pipeline enhances models with impressive capabilities (Rafailov ctt al. 2023), it is considerably more complex than supervised learning, incurring significant computa- tional costs. In light of this, recent alignment methods, such as DPO (Ratailov et al. 2023), have been proposed to streamline the process by conducting preference tuning directly on human-preferred responses without the need for reward model.
FISAO
This section first presents empirical findings (Section 3.1). demonstrating that token-level rewards tend to yield improved alignment in Vision-Laanuaage Learning Models (VLLLMs) compared to sentence-level rewards. theoretical justification for the effectiveness of FiSAO is then provided in Section 3.2 Following this, Sections 3.3 and 3.4] the two-step preference tuning process employed by AO, consisting of reward modeling and policy optimization. The overall framework of FiSAO is illustrated in Figure 3] while Table I compares FiSAO with other preference tuning approaches. Unlike other methods, FISAO eliminates the need for reward model training, additional data, high-cost human annotators.
3.1 EMPIRICAL FINDINGS
Hallucinations in VLLMs occur when these models generate content that is not grounded in the input image (Liu et al. 2024a). such as referencing non non-existent objects, often indicating weak alignment between the visual and linguistic modalities (Liu al. 2024a). To investigate vasion-language
3

---

Published as a conference paper at ICLR 2025
0.25
Data Points
Data Points
Regression Line
0.20
Regrossion
0.06
ROUGE
0.15
0.04
0.10
0.00
Score
Score
(a) Coreelatonn with BLEU
(b) Conrelation with ROUGE
Figure 2: Correlation between the CLIP-based sentence rewards and conventional evaluation metrics:
BLEU (2a) and ROUGE 2bb. A small Pearson correlation coefficient (r) indicates weak correlation. More comparison is detailed in Appendix A.2.2
alignment in VLLMM, examine its relationship hallucinations. VLLMS commonly extract features using pretrained vision encoders, such as CLIP (Radford et al. 2021 and Grounding DINO (Liu al. 2023c). These pretrained vision encoders are trained jointly on vision and language modalities, resulting ting in a more reliable object recognition (Kuo et al. 2022). Consequently, we propose utilizing the vision encoder of the VLLM verifier investigate two distinct types of reward signals: the sentence-leve signal, which is commonly employed in prior research (Hessel ct al. 2021 Zhou et al. 2024b), and the token-level signal, which has remained largely unexplored.
To facilitate this investigation, we conducted two experiments: (1) we plot the distribution of scores across the sentence-level and token-level signals for both hallucinated and correctly identified objects, and (2) we examine the relationship between sentence-level rewards and conventional evaluation metrics for VLLMs, such as BLEU and ROUGE. The scores are obtained by calculating the dot product of the text and image embeddings derived from the pretrained vision encoder within the VLLM. We generate captions for 5,000 images randomly sampled from the COCO training dataset and utilize the widely recognized CHAIR hallucination benchmark (Rohrbach et al. 2018 identify correctly identified and hallucinated objects. We present our observations as follows:
Token-level rewards differentiate objects better than sentence-level rewards. Figure I presents comparison of score distributions for hallucinated and correct objects generated by LLaVA-1.5 using Iwo types of rewards: token-level and sentence-level. In the token-level reward distribution (Figure Tap. we observe that hallucinated objects are generally associated with lower scores compared to correct objects. In contrast, in the sentence-level reward distribution (Figure lb), the two distributions largely overlap. with both hallucinated and correct objects peaking around the same score range (60-70). This indicates that, at the sentence level, the reward signal struggles to distinguish between hallucinated and correct objects.
Sentence-level rewards show weak correlation with conventional metrics. Figure 2 iillustrate the relationship between CLIP scores and conventional evaluation metrics BLEU and ROUGE for the generated captions. The scatter plots for BLEU (left) and ROUGE (right) depict the distribution of data points and their corresponding regression lines. From these figures, it is evident that there is very weak correlation between the scores and both BLEU and ROUGE, with correlation coefficients of -0.01 for each. Specifically, high sentence-level score does not necessarily indicate high-quality sentence. This observation suggests that sentences-level rewards may not be reliable indicators of model performance.
3.2 THEORETICAL FRAMEWORK FOR INCORPORATING PRE-TRAINED VISION MODELS' FEEDBACK INTO MODEL TRAINING
In this section, we present theoretical framework demonstrating how integrating feedback from pre-trained vision models enhance the performance of VLLMs. Under certain assumptions, show that utilizing the vision feedback leads to improved quality of model outputs compared to relying solely supervised fine-tuning.
We consider a VLLM and decompose the input prompt into x = (v,t) E Rd x Rd representing the image and text prompts, respectively. A Ithough text data generally consists of discrete tokens. following previous work (Nakada et al. 2023| Chen et al., 2023) Liu et al.) 2024d] Zhou et al. 2024b), model these tokens continuous random vectors in this section. Specially, assume
4

---

Published as a conference paper at ICLR 2025
the following data generative model for and t:
V == U, + Ee, and t U,2 + Et:
where U. E Od x and U2 E Qderr are orthonormal matrices representing decoders that transform the latent fow-dimeensiona) signals 2u, 2g E I' to images and text, respectively. Here, &, and & are noise vectors, and we assume they follow sub-gaussin distributions with well-conditioned covariance matrices and sub- gaussian norms upper bounded by a universal constant. We consider the infinite data setting. common simplification to avoid the influence of sample randomness (Kim et al. 2019)] Ghorbani et al. 2021; Ye et al. 2023a). According to (Nakada et al. 2023), with an abundance of image-text pairs, the learned visual CLIP embedding (z) and textual CLIP embedding Fr()) converge to U_A v and U, t, respectively. To simplify our analysis without loss of generality, we consider single score for each response y and define the feedback from pre-trained vision encoders as
Rí (y) = (U, v U, y)- We assume the ground truth Yinah = V/ + v+VV l+ where VT E IRd xdv and Vi" E Rd xd. are weight matrices, and by is a noise term. In our method, we assume that To, (y (x)
with 8 = (V1. 122) follows a Gaussian distribution: Te, (v/ x) x exp (-22= ly (V1v v + ?2t) where 11 E IR xdv and V/2 E Id xd, are the weight matrices for the image and text inputs, respectively, and > 0 is the standard deviation.
To better illustrate the contribution of using vision feedback compared to pure supervised fine-tuming (SFT), we consider the supervised fine tne-ttnnng score as (y) -11 (Vi + V22 The merged score then becomes
(5) where À E 1]. As R(y) depends on X, we denote the solution by (()). In the special case where A = 0, this corresponds to the setting where we do not use feedback from pre-trained vision encoders at all. To assess the quality of the text output y, we approach as a regression problem where there is an associated outcome linked to the ground-truth text output Y/auth 2 = 8 Y/uuh, with 3 E Rd. The quality of y is evaluated using the loss function
R(y) = (1 A)- (v) + A Ry(v),
L(y)=
min E[[ 3 {1
BERd
Note that in this context, a lower value of Lyy) indicates better quality of the text output y. Conse- quently, we derive the following theorem.
Theorem 3.1. Suppose that 7े. (y 2) lies in the LLM space { x) 0 E }}. Then, there exists
some X > 0, such that I Ex (x]=)) [yy] < (x)))
The proof can be seen in Appendix A.3. from pre-trained vision encoders (where X > 0) can enhance VLLMs' performance.
Our theoretical analysis implies that integrating feedback
3.3
REWARD MODELING FOR FISAO
3.3.
GENERATION FROM THE PERSPECTIVE SEQUENTIAL DECISION-MAKINS
In this section, we introduce a novel perspective on preference tuning for VLLMMs conceptualizing it decision-making process that takes next-token prediction. As discussed in Section 3.1 it is more appropriate to utilize token-level feedback from the fine-grained verifier. Therefore, we consider preference tuning as decision- making process undertaken by agent. In this context, after observing the input text and image, a VLLM policy Te takes actions by predicting the next token. Here, consider a standard finite state Markov decision process (MDP) for VLLMs (Puterman 2014), represented as tuple M = (S, A, P. 7, R). In this context, S is the set of states 8, representing the current context history of generated tokens in the VILM. The set A denotes the actions a, which correspond to the possible next tokens that the VLLM can generate. The transition probabilities P E A(S)sxA indicate the probability of transitioning from one state to another given an action. The discount factor y E (0, 1] is typically set to in our case, focusing on the undiscounted scenario. Lastly, is a bounded reward function R S
+ R. providing feedback or reward for the new state.
VLLM मe taking action in state and transitioning
Given an appropriate reward function in M, the optimal policy Th. E II the solution to the optimization problem of maximizing the expected discounted total future reward:
T
max Ea #EI1
7 R(st, azs S1+1)
(6)
5

---

Published as a conference paper at ICLR 2025
VLLM
Vision Encoder
Average
0.61
Deszrbbe this
This image features a col and dee.
Lubel: Hullacisation
Step Reward Modeling
VLLM
Vision Encoder
Token-level reward
visual
Gueation: Generate
bokens
y<t
Calculation
tectual tokens
'cat'
Forward Backward
Step 2 line-Grained Preference Palicy Optimization
Figure 3: The overall framework of FiSAO. We employ two steps to achieve self-alignment from tineggrained feedback: (1) caleulate the fine- oee-grained reward based on the baseline score obtained from correct and hallucinated tokens. (2) optimize the preference policy using this reward to align the model's responses during training.
This perspective highlights how fine tne-grained rewards can be applied enhance and guide VLLMs, enhancing the lision-langaue alignment VLLMs.
3.3.2 ESTIMATION OF BASELINE SCORES FOR GROUND TRUTH AND HALLUCINATED DISTRIBUTIONS
To fairly evaluate the model's performance using feedback from the fine-grained verifier, it is crucial establish baseline score. In this section, estimate the baseline reward for the reward calculation process. Assume that the model generates set of responses Y = {3' y² ...,. y²} visual inputs and queries (x' ( J.).... (x*", v²) from the training dataset. Object tokens of these responses can be divided into two subsets: Yg: and Ynal- Here, YgI represents the object tokens that are correctly aligned with the corresponding visual input, determined by the ground truth labels, while Yhal consists of the tokens that are identified as hallucinated or misaligned with the corresponding visual input. For each correct object set O' and hallucinated object set Ó' in i-th response, we calculate score using the dot product between the features of object token and the visual input 2, derived from the fine ime-grained verifier. Finally, the average scores for correct objects Het and hallucinated objects Phat are calculated follows:
in response to
Mgt Σi |0
Σ S(o,,u'), Phal i=1 0yE0S
Σ S(of, u*),
(7)
Σ.=1
|11$|
where ||. denotes cardinality of a set. Eqn. 7 can help define the boundary used to calculate the final reward for fine-grained preference policy optimization.
3.3.3 FINE-GRAINED REWARD CALCULATION
In this section, we calculate fine-grained rewards for preference tuning. Formally, let the model's response a query z with the visual input from the original dataset be denoted as {y/ 3/, .2...... yT To better select tokens suitable for providing feedback, we choose common objects from the existing dataset. First, we construct an entity set using the labels from Detic (Zhou et al. (2022) and COCO LLn et al. 2015). Then, expand the original set to C' by including similar words and plural forms. Detailed information can be found the Appendix A.I.I To better incorporate the feedback from the fine-grained verifier, calculate the negative and positive reward boundaries based the baseline

---

Published as a conference paper at ICLR 2025
scores of correct and hallucinated responses, described in Section 3.3.2 We apply the following {R(se, formula to calculate the fine- ameeggrined reward R { de, $141))==1
(S v), (Hmml - 2)) - &xu (er( Mee. (x, yee. w)], y E C&S(M, v) < |haal
R(st, S1,11) =
N(S(ye, u), (Hg + X)) [Dkl[TneL yee, (e., if y C&S((V, v) Ps
(8)
0.
otherwise
where (yt, 11) is the dot product score of the yt and of the pre-trained vision encoder, À is the margin, { is a scaling factor for the KI. divergence penalty, N(...) is normalization function, Mg and Phal are the average scores of the correct and hallucinated tokens, respectively. More details can be seen in Appendix A.1.5
3.4 FINE-GRAINED PREFERENCE POLICY OPTIMIZATION FOR FISAO
Following (Ouyang et al. 2022 Yu et al. 2023a our approach employs a clipped-PPO method to train the model. This method involves cutting the probability ratios mitigate large updates, ensuring stable and reliable training. Unlike standard PPO, our approach learns from fine-grained feedback at the token level for each state. By incorporating fine-grained preference signals, FiSAO ensures better vision- laiinnlanguaae alignment in VLLMs. The objective function is defined as:
L(0) Eas- Σ min clip())) 1,1+ +)) ((
where rf) is the probability ratio, Rg the advantage estimate and is a hyperparameter that determines the clipping range, and clip(.) is clipping function that constrains the value of rx)).. The probability ratio (0) is calculated as:
To (x, yeee v) (Mt x Meer v)
Ty (0)
(10)
where Tert and ne are the policies before and after the update, respectively. We show the detailed process of FiSAO in Algorithm]]
Algorithm 1 FiSAO
Require: Dataset: D { oz N Reference model: Tref; Policy model: πT; PPO training
epochs
Ensure: Updated policy model mp
1: for each (Ir, v) E 'D do
2: Generate the response from query and image {y1. U2...., yT = x2( v)
3: for each state y/ in {yo, .I...., } do
4: Compute the score R(st, Qt, St+1) from Eqn.
5:
for each epoch in do
6:
Calculate probability ratio (f) from Eqn. Update To using Eqn.
7:
8:
return m
EXPERIMENT
In this section, we evaluate FiSAO on the modality alignment of Vision-Language Large Models (VLLMs), showcasing its effectiveness in enhancing models' performance. Our investigation aims to answer the following questions: (1) Does FiSAO enhance the visual understanding capabilities of VLLMs compared to previous approaches? (2) How does the primary component of FiSAO contribute to performance across different benchmarks? (3) Does our method modify the reward distribution of objects in the model's output before and after training?
7

---

Published as a conference paper at ICLR 2025
Table 2: The performance of FiSAO across all benchmarks. Bold indicates the best result and
underline indicates the second-best result within each model group (LLaVA vs. InstructBlip). For CHAIRş and CHAIRj, smaller better.
Comprehensive Benchmark
VQA
COCO Benchmark
Method
MME"
MME
SEED
MM Vel
SQA'
POPE
GQA
Cap_val val
CHAIRSs
CHAIR;
LLAVA-15
1510.7
148.2
58.6
30.5
66.8
85.9
Vifeedback
1432.7
321.8
59.3
31.2
66.2
83,7
63.2
54.8
40.3
13.2
Human-Prrffe
1490.6
335.0
58.1
31.1
65.8
81.5
61.3
50.4
38.7
11.3
POVID
1452.8
325.3
60.2
31,8
68.8
86.9
61.7
57.3
35.2
8,3
+ FISAO
1522.6
349.0
60.6
30.5
69.3
85.7
62.0
61.2
39.9
9.9
InstructBlip
1237.5
292.1
38.5
26.0
43.5
84.8
48.0
65.5
60.3
11.9
Vifeedback
1241.3
298.9
40.4
26.6
44.6
78.5
47.7
64.0
56.5
9.7
Human-Prefer
1250.9
304.2
39.3
26.6
44.1
79.0
47.5
64.8
51.2
10.8
POVID
1255.
301.8 8
38.3
26.3
43.4
84.6
48.3
66.5
51.5
10.5
FISAO
1305.3
308.9
40.0
26.9
83.7
48.2
8.8
Table 3: Comparison of FiSAO and other open-sourced state-of-the-art VLLMs.
Method
MME²
MME
SEED
MMB
MetVet SQA' GQA
BLIP-2
1293.8
290.0
46.4
38.1
22.4
61.0
41.0
InstructBlip
1237.5
292.1
38.5
36.0
26.0
43,5
48.0
Qwen- VL-Chat
1487.6
360.7
58.2
60.6
47.3
68.2
57.5
mPL.UG-Ow12
1450.2
313.2
57.8
64.5
36.2
68.7
56.1
LLaVA-1.5
1510.7
348.2
58.6
64.3
30.5
66.8
62.0
FiSAO (LLaVA-1.5)
1522.6
349.0
60.6
64.8
30.5
69.3
62.0
4.1 EXPERIMENTAL. SETUP
Implementation Details. We employ LLaVA-1.5 7B (Liu et al. 2024b) and InstructBLIP Dai et al, 2023b) as the backbone models. During the preference tuning process, we adapt Low-Rank Adaptation (LoRA) (Hu et al. 202I) fine-tuning. We select the first 8k data from the ALaVA-Instruue 150k dataset (Li al. (2023b). As both InstructBLIP and LLaVA are trained using the 1 50k dataset, no additional data is introduced into our model training. Training is conducted over one epoch, with Proximal Policy Optimization (PPO) being applied for four epochs per sample, utilizing four A100 80GB GPUs. Fine-tuning LLaVA-1.5 7B takes approximately six hours, while fine-tuning InstructBLIP 13B requires around ten hours. For more detailed information on training hyperparameters and training data, please refer to Appendix A.1.5.
Evaluation Benchmarks. We conduct evaluations on three types of benchmarks: comprehensive benchmarks, general VQA benchmarks and COCO benchmarks. Specifically, these include: (1)
Comprehensive benchmarks (MME (Fu al., 2024), SEEDbench (Li al., (223a), MMbench (Liu
et al. 2024c), MM-Vet Yu et al. 2023b)): (2) VQA (ScienceQA (SQA) (Lu et al. 2022), POPE (Li
jt al. 2023e). GQA (Hudson & Manning 2019)); (3) Caption benchmark (L.i et al. (022)) (Average
score of BLEU, ROUGE-L and CIDER), CHAIR (Rohrbach et al. 2019) ) The detailed information
is in Appendix A.I.3
We compare FISAO with previous preference tuning approaches, including Silkie (VIfeed- (Sun et al. 2023), and POVID (Zhou compare FiSAO with other state-of the art open-source VLLMs. including BLII-2 (Li et al. 2023c), InstructBIIIP (Dai et al. 2023a), Qwen-VL-Chat (Bai et al. mPL.UG-Ow22 Yee et al. (023c). More details can be seen in Appendix A.1.4
Baselines.
back) Li et al. 2023d). LLaVA- RLHF
et al. 2024a). Furthermore,
2023
4.2 EXPERIMENTAL RESULTS ON BENCKMARKS (RQ1)
Comparison with Other Preference Tuning Approaches. As shown in Table ??, our method demonstrates clear advantages over other preference tuning approaches, which often require training reward models or incur high data costs. The superiority of FiSAO lies in its use of fine-grained verifier, which more effectively captures the intrinsic preferences of VLLMs and achieves stronger modality alignment between the pre-trained vison models and LLMs. Additionally, on the LLaVA backbone, FiSAO surpasses existing approaches, delivering average performance improvement of
8

---

Published as a conference paper at ICLR 2025
8.7%. This underscores FiSAO's effectiveness in leveraging fine-graines token-level rewards to align visual and textual modalities seamlessly.
Comparison with Other Open-Sourced VLLMs. Table 3 compares FiSAO with other state-of-the- art VLLMs. Our method, implemented on the LLaVA-1.5 architecture, achieves competitive results across multiple benchmarks, demonstrating its effectiveness in various tasks such as vision question answering and image captioning. This highlights FiSAO's capability in integrating fine-griined token-level rewards to enhance modality alignment in VLLMs.
4.3 ANALYSIS (RQ2&RQ3)
Ablation Study. Table summarizes the results of the ablation study conducted on FiSAO. Each row represents different configuration: the presence (v) absence (x) of fine-grained rewards and PPO training. When fine-grained rewards are not used regardless of PPO training, performance metrics are notably lower across all benchmarks compared configurations where fine-grained rewards are employed. Introducing PPO training alone shows an improvement, but the most significant gains are observed when both fine-grained rewards and PPO training are utilized. This combination achieves the highest scores, demonstrating the effectiveness of integrating both strategies in enhancing model performance and alignment across various evaluation tasks. These findings underscore the importance of fine-grained token-level rewards in optimizing VLLMs such FiSAO for multimodal tasks.
How does Reward Margin Effect Model's Performance? We present how different reward margins impact the model's performance across various benchmarks in Table 5 The table highlights how varying the reward margin À affects the performance of LLaVA-1.5 + FiSAO across multiple benchmarks. The results indicate notable variations in performance metries based on the choice of reward margin. Specifically, when the margin is either too small or too large, a decline is observed in metrics such as CHAIR,, suggesting diminishing returns with extreme reward margins. Although overall performance remains relatively stable, these findings underscore the importance of optimizing the reward margin to balance precision and generalization in FiSAO for enhancing the performance of VLLMs.
How does FISAO Alter the Reward Distribu- tion of Objects in the Model's Output before and after Training? To better demonstrate how our method enhances vision-langure laagua- alignment and ensures the generation of high-scoring ob- jects, we visualize the reward distribution of generated objects on the CHAIR benchmark,
0.30
Rewards Traniing
0.25
0.20
Density
0.15
depicted in Figure 4 The figure illustrates that VLLMs tend to generate objects with lower scores before training. This result indicates that the reward distribution before training more dispersed and misaligned with the preferences of the visual encoder. After applying our method, the reward distribution shifts the right, reflect- ing improved lision-language aligmment. This shift signifies that fine-gre mne-grai feedback leads to enhanced overall performance in VL.LMs.
0.10
8.5
0.00
-8
-6
Reward from the Fine-Grained Verifier
Figure 4: Comparison of reward distributions for generated objects on LLaVA-1.5 before and after Training.
Case Study on Sentence-Level Reward and Token-Level Reward. In this section, we conduct case study where two sentences from an image are selected for evaluation using both token-level and sentence-level scoring. From Figure 5 we can observe that the sentence-level score is not sensitive to hallucinatory sentences, it assigns similar scores to both sentences. In contrast, token-level scoring more effectively identifies hallucinatory objects.
RELATED WORK
Recent advancements in large language models (Brown et al. 2020 Liu et al., 2022 Touvron et al., 2023) and pre-trained vision models (Radford et al. 202 la) have enabled the creation of Vision-Large

---

Published as a conference paper at ICLR 2025
image wearing umbrellas, young we lorling
holding and
gir holding g umbrella appeors guidding girt
walk.
Hullucinaion:
Santerra-laowel Score: 0.547
lokan-Lavel Scoret womant 5/b umbrellie 0./28
sreee, stme dtanding walking reatbs. In total there three other indiviouals in the imagee some them coser the womon and the gir. Additorally there &s handllag viS the surse, iuil Lefunging to ome of the people present.
There are svenci other people
Halluciaation vf
Sentence Level Score: 0,559
Token Level Score: handllgg 0:574 people: 636
Figure 5: Case study on sentence-level reward and token-level reward.
Table 4: Ablation study results. Each row illustrates different configuration, indicating the presence (V) or absence (x) of fine-grained rewards and PPO training.
Comprehensive Benchmark
VQA
COCO-cap Benchmark
Fine-grainee
PPO
MME"
MME
SEED
MMB
MM-Vet
SQA'
POPE
GQA
Cap_val
CHAIRs
CHAIR:
1431.9
340.0
59.6
64.0
30.6
67.7
85.7
61.4
54.5
54.0
11.0
1509.3
350.4
59.5
64.1
30.5
67.5
85,9
60.9
56.6
55.3
11.4
V
1522.6
349.0
60.6
64.8
30.7
69.3
85.7
62.0
61.2
39.9
9.9
Table 5: Performance of FISAO with varying margins
Comprehensive Benchmark
VQA
COCO-cup Benchmark
A
MME
MME
SEED
MMB
MM-Vet
SQA!
POPE
GQA
Cap_val
CHAIRs CHAIR;
5
1509.3
350.4
60.4
64.1
30.6
67.5
84.4
61.7
57.1
53.3
1522.6
349.0
60.6
64.8
30.7
69.3
85.7
62.0
61.2
20
1501.4
348.6
59.2
64.5
31.0
67.9
85.1
61.6
59.7
56.5
13.6
Language Models (VLLMs), which effectively integrate language and vision capabilities. These models have significantly improved automation in various fields, including medical applications (Liu et al. 2023b), recommendation (Sheng et al., 2024), autonomous driving (Zhou et al.) (023a)) agent- based evaluation (Zheng et al. 2224)) and embodied agents (Peng et al.) 2023). The typical architecture of VLLMs involves aligning the embedding spaces of both modalities using techniques such as Qformer or fully connected layers (Zhu et al.) [023 Ye al. 2023b Li al. 2023b).
However, VLLMs face challenges in precise alignment due to independent pre-training of language and vision models, leading to safety concerns (Gong al. 2023 Tu et al. 2023), hallucinations (Wang
et al. 2023), and reasoning deficiencies (Ghosh et al. (024). Traditional vision-language models (VLMs) have focused on image-tex: text alignment through methods like no-attention frameworks (Lu
al. 2019), anchor points (Li al., 2020). and contrastive learning (Radford et al. (202 Recently,
alignment strategies can be classified into alignment from training data, which leverages high-quality
datasets for supervised fine-tuning (SFT), and alignment from feedback, which involves fine-tuning
based on human or Al feedback (Sun et al. 2023 Yu et al. 2023a Zhou et al.] [024a [i et al. 2023d
Zhao et al. 2002).. Feedback-based methods often use Proximal Policy Optimization (PPO) (Sun
et al. 2023) and Direct Preference Optimization (DPO) (Zhao et al., 2023 Li et al. 2023d Chen
al. 2024a). Despite their potential, these methods face challenges such as high costs in dataset construction and need for external tools. Additionally, some approaches use sentence- rewards, which do not fully leverage the text-generation capabilities that large language models (LLMs) are fundamentally designed for. By concentrating on assigning numerical score to the entire instance, these methods overlook the VL.LMs' inherent capability to generate responses, including detailed reasoning steps. The detailed version is shown in Appendix A.4
6 CONCLUSION
In this study, we addressed the alignment issues prevalent in Vision-Language Large Models (VLLMs) by investigating the integration of pre-trained vision encoders with large language models. Through comprehensive analysis, we introduced a novel self-training method using ane-ggrined Proximal Policy Optimization (PPO) that does not rely additional data. This method leverages the model's visual encoder as a reward model to enhance alignment at the token level, demonstrating superior performance compared to existing preference tuning approaches.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is supported by the NExT Research Centre and Special Funding for Students' Overseas Research Internships of University of Electronic Science and Technology of China (UESTC). We sincerely appreciate the reviewers and the AC for their valuable suggestions throughout the review process.
ETHICS STATEMENT
This paper aims to enhance vision-language alignment for Vision-Language Large Models (VLLMs) and do not obey the ICLR code of ethics.
REPRODUCIBILITY STATEMENT
All the results in this work are reproducible. We provide detailed settings for our experiments in Table
REFERENCES
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization. text reading, and beyond, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL htw.tttpss/pprrceedngg..cc neur wtcesscccppppeeee cppppprr...p......eppeeeeeee...e fi files/paper
Paper..dr.
Beitao Chen, Xinyu Lyu, Lianli Gao, Jingkuan Song, and Heng Tao Shen. Alleviating hallucinations in large vasion-language models through inaciiattioninducee optimization. arXiv preprint vrXiv:2405.15356 2024a.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models align and interact with humans via natural language feedback. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 14239-14250, 2024b.
Zixiang Chen, Yihe Deng, Yuanzhi Li, and Quanquan Gu. Understanding transferable representation learning and zero-shot transfer in clip. arXiv preprint arXiv:2310.0092. 2023.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of ballucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint arXiv:231 0et::2311.0222 1.03287, 2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purppon vision-language models with instruction tuning, 2023a.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purppsse vision-language models with instruction tuning, 2023b.
Ailin Deng. Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding, arXiv preprint arXiv:2402. 15300, 2024.
11

---

Published as a conference paper at ICLR 2025
Yue Fan, Jing Gu, Kaiwen Zhou, Qianqi Yan, Shan Jiang, Ching Chen Kuo, Xinze Guan, and Xin Eric Wang. Muffin or chihuahua? challenging large vision-language models with multipanel vqa. arXiv preprint ar 4rXiv:2401.15847, 2024.
Ching-Chen
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Linearized two-layers neural networks in high dimension. 2021.
Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Utkarsh Tyagi. Oriol Nieto, Zeyu Jin, and Dinesh Manocha. Vdgd: Mitigating Ivlm ballucinations in cognitive prompts by bridging the visual perception gap. arXiv preprint arXiv:2405. 15683, 2024.
Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. Figstep: Jailbreaking large vision-language models via typographic visual prompts. arXiv preprint arXiv:231 1.05608, 2023.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models, 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. CoRR, abs/2104.08718, 2021. URL https://
abs 221/22104.00 08718.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 1xXi:2206609660 2021.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Jiho Jang, Chaerin Kong. Donghyeon Jeon, Seonhoon Kim, and Nojun Kwak. Unifying vision- language representation space with single-tower transformer. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 37, pp. 980-988, 2023.
Michael P Kim, Amirata Ghorbani, and James Zou. Multiaccuracy: Black-box post-processing for fairness in classification. In Proceedings of the 2019 AAAI/ACM Conference on Al, Ethics, and Society, pp. 247-254, 2019.
Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-ylm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209.15633 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimoal models, March 2024. httptttst///ttttitt wttp://....oo.e..e.........ccm
Lab/
Imms eval
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Ilms with generative comprehension, 2023a.
Chunyuan Li. Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-vision assistant for biomedicine in one day. arXiv preprint 00tv:2306000899 2023b.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023c.
12

---

Published as a conference paper at ICLR 2025
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang. Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665 2023d.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pp. 137. Springer, 2020.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucination large vision-language models. arXiv preprint arXiv:2305.10355 2023e.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306. 14565, 2023a.
Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. survey hallucination in large vision-language models, arXiv preprint ar arXiv:2402.00253 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2024b.
Junling Liu, Ziming Wang, Qichen Ye, Dading Chong, Peilin Zhou, and Yining Hua. Qilin- med-vl: Towards chinese large vision-language model for general healthcare. arXiv preprint arXiv:2310.17956 2023b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 2023c.
Xiao Liu, Da Yin, Jingnan Zheng, Xingjian Zhang, Peng Zhang. Hongxia Yang, Yuxiao Dong, and Jie Tang. OAG-BERT: towards unified backbone language model for academic knowledge services. In KDD '22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14- 18, 2022. ACM, 2022.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024c.
Yuhang Liu, Zhen Zhang, Dong Gong, Biwei Huang, Mingming Gong, Anton van den Hengel, Kun Zhang, and Javen Qinfeng Shi. Revealing multimodal contrastive representation learning through latent partial causal models. arXiv preprint arXivv2402.06223 2024d.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task task-agnostic visiolinguistic representations for vision-and- -annauuggee tasks. Advances in neural information processing systems, 32, 2019.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chur Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. The 36th Conference on Neural Information Processing Systems (NeurIPS), 2022.
Ryumei Nakada, Halil Ibrahim Gulluk, Zhun Deng, Wenlong Ji. James Zou, and Linjun Zhang. Understanding multimodal contrastive learning and incorporating unpaired data. In International Conference Artificial Intelligence and Statistics, pp. 4348-4 4480. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feed- back. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 27730-27744, Curran Asso- ciates, Inc., 2022. URL http:////////peeeeeaaaaao....o.. files//pprreeeppeeeerppppep pdf
Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021a
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https?
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv:1809.02/56, 2018.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017
Leheng Sheng, An Zhang, Yi Zhang, Yuxin Chen, Xiang Wang, and Tat-Seng Chua. Language representations can be what recommenders need: Findings and potentials. arXiv preprint arXiv:2407.05441, 2024.
Yongxin Shi, Dezhi Peng, Wenhui Liao, Zening Lin, Xinhong Chen, Chongyu Liu, Yuyi Zhang, and Lianwen Jin. Exploring oct capabilities of gpt-4v (ision): A quantitative and in-depth evaluation. arXiv preprint arXiv:2310. 16809, 2023.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXivv2309.144: 14525, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Haoqin Tu, Chenhang Cui, Zijun Wang, Yiyang Zhou, Bingchen Zhao, Junlin Han, Wangchunshu Zhou, Huaxiu Yao, and Cihang Xie. How many unicorns are in this image? safety evaluation benchmark for vision llms. arXiv preprint arXiv:2311.16100 16101, 2023.
Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye. Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. arXiv preprint arXiv:2308. /5/26, 2023.
14

---

Published as a conference paper at ICLR 2025
An Yang. Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Haotian Ye, James Zou, and Linjun Zhang. Freeze then train: Towards provable representation learning under spurious correlations and feature noise. In International Conference on Artificial Intelligence and Statistics, pp. 8968-8990, PMLR, 2023a.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:204..4417, 2023b.
Qinghao Ye, Haiyang Xu, Jiabo Ye. Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration, 2023c.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint urXiv:2310.16045 2023.
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy milmss via behavior alignment from grained correctional human feedback. arXiv preprint arXiv:2312.00849 2023a.
Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023b.
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivims through hallucination-avv aware direct preference optimization, 2023.
Jingnan Zheng, Han Wang, An Zhang, Tai D. Nguyen, Jun Sun, and Tat-Seng Chua. Ali-agent: Assessing umss alignment with human values via agent-based evaluation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Xingcheng Zhou, Mingyu Liu, Bare Luka Zagar, Ekim Yurtsever, and Alois C Knoll. Vision language models in autonomous driving and intelligent transportation systems. arXiv preprint arXiv:2310.14414, 2023a.
Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV, 2022.
Yiyang Zhou, Chenhang Cui, Jaehong Yoon, Linjun Zhang, Zhun Deng, Chelsea Finn, Mohit Bansal, and Huaxiu Yao. Analyzing and mitigating object hallucination in large vision-language models. arXiv preprint arXiv:2310.007: 2023b.
Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.1 11411,
1411, 2024a.
Yiyang Zhou, Zhiyuan Fan, Dongjic Cheng. Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang. Yun Li, Linjun Zhang, and Huaxiu Yao. Calibrated self selfrrewan vision language models. arXiv preprint 240.:2445..1622, 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny, Minigpt-4: Enhancing Vision-language understanding with advanced large language models, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: (ttp:::990000999 2019.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 EXPERIMENTAL SETTINGS
A.1.1 DETAILS OF ENTITY SET
First, we construct an entity set using the labels from Detic (Zhou et al. 2022) and COCO (in et al. 2013). We present the case of these datasets' labels in Table 6 Then, we expand the original set to C by including similar words and plural forms using the inflect library and the wordnet module from the nltk library. The expanded set C' contains 5678 words compared to the original set, which contains 1204 words. The inflect library used to generate plural and singular forms of the original labels, while the wordnet module from nltk is employed to find synonyms. This method allows to create a comprehensive entity set by considering various linguistic forms, thus enhancing the robustness of our dataset.
Original Word
Expanded Words
apple
apples
handbag
bag, handbags, pocketbook, purse grip, suitcases
suitcase
bagel
bagels
boat
boats, sauceboat, boat
bob
dock, cork, bobs
bread
breads, lucre, lolly, staff of life
cat
purge, chuck, cats
chair
moderate, chairs, chairperson, lead, chairman ducks, duck, dip, douse jars, clash, shock
duck
jar
person
someone, person, individual, somebody, people, soul
shirt
shirts
taco
tacos, greaser, wetback, taco cycle, wheels, roll
wheel
Table 6: Cases of original Words and their expanded forms.
A.1.2 OVERVIEW OF THE BACKBONE MODELS
LLaVA-1.5 a multimodal model designed for general-purpon visual and language understanding. It integrates vision encoder with the Vicuna language model, making it capable of processing images and generating text-based responses. The model is an open-source chatbot that has been fine-tuned on multimodal mnstruction-follwviin data generated by GPT. It is built upon the transformer architecture, specifically leveraging the LLaMA/Vicuna foundation models.
InstructBLIP is sophisticated vision- language model designed to follow detailed instructions. It is built upon the BL.P-- architecture, incorporating a vision encoder, a language model, and a Query Transformer (Q-Former) that bridges the two components. The Q-Former module specifically enhanced to handle instruction text tokens, allowing it to extract task-relevant features from images effectively.
A.1.3 DETAILS OF EVALUATION BENCHMARK
MME Fu et al. multimodal tasks. It measures models' capabilities across two key areas: perception and cognition, using 14 specially designed subtasks that test interpretative and analytical skills.
2024 comprehensive benchmark for evaluating the performance of LVLMS in
SEED-Bench (Li et al. 2023a) focuses on evaluating the generative comprehension abilities of LVLMS.. It includes dataset of 19K multiple-choice questions with detailed human annotations,
16

---

Published as a conference paper at ICLR 2025
spanning 12 evaluation dimensions that cover both spatial and temporal understanding in image and video modalities.
MMBench (Liu et al. 2024c employs a dual approach: it provides an extensive dataset that broadens the range and variety of evaluation questions, and introduces the innovative CireularEval strategy, which uses ChatGPT to convert free-form predictions into structured choices.
MM-Vet Yu et al. 2023b) is benchmark created to evaluate the diverse competencies of LVLMs. It organizes complex multimodal tasks into 16 unique integrations based on six core vision-language capabilities, offering detailed analysis of model performance across various question types and answer styles.
2022 is multimodal benchmark aimed at assessing and diagnosing AI ScienceQA (Lu et
systems' multi-hop reasoning and interpretability in the science domain. It includes dataset of around 21K multiple-choice questions across various scientific topics, complete with detailed answer annotations, related lectures, and explanations.
GQA (Hudson & Manning 2019) is dataset designed for advanced visual reasoning real- world scenarios, using scene graph-based structures generate 22 million diverse, semantically- programmed questions. It features novel set of evaluation metries focused on consistency, grounding, and plausibility, setting high standard for vision-language task assessment.
POPE (ii et al., 2023e) an evaluation method for examining object hallucination LVLMs. It transforms the evaluation into a binary classification task, asking LVLMs simple Yes-or-No questions to identify hallucinated objects. POPE employs various object sampling strategies to reveal model tendencies towards hallucination.
The COCO-caption benchmark assesses image captioning models using BLEU, ROUGE, and CIDEr scores, providing a comprehensive measure of caption quality. We calculate the average these scores and multiply by 100 to obtain the final score. This benchmark utilizes the COCO dataset, emphasizing the accuracy and relevance of generated captions. Detailed evaluation methodology and task specifies can found in the 1 1mms_eva repository, specifically under the
val directory
CHAIR (Rohrbach et al. 2019) is well-known tool for evaluating object hallucination in image captioning tasks. It includes two variants: CHAIRI and CHAIRS, which assess object hallucination at the instance and sentence levels, respectively. Specifically, we randomly sampled 500 images from the COCO (Lin et al. 2015) validation set and evaluated object hallucination using the CHAIR metric.
A.1.4 DETAILS OF BASELINES
Silkie (VIfeed- back) (Li et al. 2023d) focuses on improving large vision language models (LVLMs) by using preference distillation. The authors created a vision-language feedback (VL.eedback) dataset, consisting of multi-modal instructions and responses generated by 12 different LVLMs. The model pool includes prominent models like GPT-4V and LLaVA- series. By applying direct preference optimization (DPO) on this dataset, they developed the Silkie model, which shows significant improvements in perception and cognition capabilities.
LLAVA-RLHF (Human--peeference) (Sun et al. 2023) explores the integration of reinforce- ment learning with human feedback (RLHF) to enhance vision-language models. The LLaVA series, built on Vicuna models and fine-tuned with GPT-4 generated multi-modal data, further improved by aligning visual faithfulness and human preferences. This approach aims to ensure that the generated responses are more aligned with human expec- tations and the visual content they describe, providing more reliable and contextually accurate output
pettss:/giithiie com Fvol igg MMS- l wtwpraaaateeeeeeeee mms_mma__aaseee mam__vall tasks
17

---

Published as a conference paper at ICLR 2025
Backhone Model
LLaVA-1.5
InstructBLIn
Parameter
7B
13B CLIP- VIT-L
Reward Model
CLIP- ViT-LV
Dataset
LLLVV Instruct
LLaVA --In-ruucttt
Fine- Tuming Method
LoRA
LoRA
Number of Epochs
PPO Training Epochs
4
GPUs Used
4 A100 80GB GPUs
4 A100 80GB GPUs
Training Time
-6 hours
-10 hours
LoRA
128
128
LoRA Alpha
256
256
Learning Rate
5e-7
4c-6
LoRA Parameter
all linear
linear
0.2
0.2
10
10
Table Training parameters for LLaVA-1.5 7B and InstructBLIIP 13B models.
POVID (Zhou et al. 2024a) a framework for generating non-preferred responses in Vision-Language Large Models (VLLMs) aimed preference optimization. The framework employs strategies: hallucination text responses and noisy image responses at token and instance levels. This approach helps in understanding and optimizing VLLMs by inten- tionally producing outputs that are less preferred, thus identifying areas for improvement in model performance and user interaction.
A.1.5 HYPERPARAMETER DETAILS
In this section, we show the detailed information on training hyperparameters and training data in Table 7 Specifically, for the normalized function (V.--), ), we calculate the score for correct objects
((9c.)) (+))++)) and for hallucinated objects ((pr,))
as Sma (Mg++A)
(Agt-A)-Sun Smae Smin and Smax represent the
minimum and maximum possible scores, respectively. In this way, constrain the reward within the range of - to 1.
A.2 ADDITIONAL ANALYSIS
A.2.1 DETAILED ANALYSIS ON COCO-CAPTION BENCHMARK
Table provides comprehensive comparison of various methods evaluated on COCO-caption benchmark. Our method, denoted as FiSAO, demonstrates significant improvements across multiple metrics, highlighting its efficacy in enhancing caption generation quality. On the LLaVA backbone, FiSAOconsistently outperforms the baseline and other preference-tuning methods across all BLEU metrics, as well as METEOR, ROUGE L, and CIDEr scores. These results underscore the robustness of Ain capturing nuanced textual and visual features, achieving superior alignment and coherence in the generated captions. Similarly, for the InstructBLIP backbone, FiSAOmaintains a competitive edge, achieving high scores across the evaluation metrics and outperforming other peefreenc--tuning approaches. The improvements observed with FiSAOhighlight its effectiveness in leveraging fine- grained token-level rewards to enhance the alignment between visual and textual modalities.
A.2.2 ADDITIONAL ANALYSIS ON SENTENCE-LEVEL REWARD
We present the sentence-level rewards of the generated captions on InstructBLIP Figure 6 We can observe the low distinction between correct and hallucinated captions. We also show comparison of Fine- Fine-Grained and sentence-leved reward distribution Figure Figure where the sentence-level reward shows explicit correlation with traditional evaluation scores. This comparison highlights that the Grained reward distribution tends to be more useful, offering detailed view of the
18

---

Published as a conference paper at ICLR 2025
Table 8: Evaluation results on COCO-caption benchmark.
Method
Bleu
Bleu 2
Bleu
Bleu
METEOR
ROUGE L
CIDEr
LLaVA
0.7312
0,5641
0.4150
0.2976
0.2929
0.5559
1.1038
Vlfeedback
0.7149
0.5487
0.3734
0.2788
0.2835
0.5398
1,0969
+ Human-Prefch
0.6741
0,5047
0.3613
0.2519
0.2864
0.5329
0,9142
POVID
0.7360
0.5680
0.4197
0.3030
0.2954
0.5601
1.1305
+ FiSAO 0.7925
0.6259
0.4681
0.3407
0.2811
0.5774
1.1970
InstructBLIP
0.8220
0.6682
0.5199
0.3973
0.2982
0.5984
1.3498
+ Vlfeedback
0.7919
0.6346
0.4886
0.3689
0.3000
0.5874
1.3055
Human-PPeeer
0.8034
0.6431
0.5068
0.3759
0.3104
0.6012
1.2902
POVID
0.8204
0.6671
0.5198
0.3977
0.3009
0.6002
1.3619
FISAO
0.8239
0.6707
0,5231
0.4008
0.2985
0.5994
1.3526
Inluucictted Objects
Hallucinated Objects
Correct Objects
Correct Objects
Density
Density
20
40
60
100
40
50
60
70
80
90
Score
Score
(a) Fine-Grrie Grained Reward
(b) Sentence-Lever Reward
Figure 6: Comparison of fine-grained and sentence-level reward distributions in InstructBLIIP.
model's performance. These analyses further demonstrate that using Fine-Grained rewards is more effective than sentence-level rewards.
A.2.3 ADDITIONAL. ANALYSIS ON REWARD DISTRIBUTION OF OBJECTS
To further illustrate how our method enhances the alignment between visual encoders and VLLMs, we present the reward distribution of hallucinated objects in Figure Figure The figure shows that, before training, the reward distribution for hallucinated objects in both LLaVA and InstructBLIP is more scattered and less aligned with the visual encoder's preferences. After applying our method, the reward distribution shifts to the right, indicating improved alignment and consistency with the visual encoder. This shift demonstrates that the model's rewards now more accurately reflect the visual encoder's 's evaluations, thereby enhancing the overall performance of vision-language alignment.
A.2.4 CASE STUDIES
In this section, we present detailed case studies comparing the outputs of our model with LLaVA 1.5. The case studies highlight the strengths of FiSAOin generating detailed image descriptions. As shown in Figure TO] FiSAOfocuses on providing comprehensive overview, including contextual details such environment and the placement of objects (e.g., handbag, table settings). This approach ensures that the description covers relevant aspects of the scene. LLaVA 1.5 includes specific interactions and objects that enhance the vividness of the scene. However, sometimes generates objects that are not actually present in the images.
19

---

Published as a conference paper at ICLR 2025
0.07
0.0
Data Points Regression
0.20
0.03'
Data Points Regression
0.06
ROUGE
10.15
0.04
0.03
0.10
0.01
0.00
50
Score
Score
(a) BLEU
(b) ROUGE
-0.02
Data Points Regression
0.02
Data Points
Regression Line
METEOR
0.15
CIDEr
0.10
0.05
0-
50
70
40
50
60
Score
Score
(c) METEOR
(d) CIDEr
Figure 7: Correlation between sentence reward and conventional evaluation metrics on InstructBLIP..
0.25
-0.01
Data Paints Regression
Data Points
Regression Line
0.06
BLEU
ROUGE
0.15
0.04
0.10
0.02
50
70
60
Score
Score
(a) BLEU
(b) ROUGE
0.25
Datá Paints Regression
1.2
0.03
Data
METEOR
0.20
1.0
Regression Line
0.15
CIDEr
0.8
0.10
0.2
0.0
50
70
60
Score
Score
(c) METEOR
(d) CIDEr
Figure 8: Correlation between sentence reward and conventional evaluation metrics on LLaVA
20

---

Published as a conference paper at ICLR 2025
Rewarde- Bekon The sing Rewente- Adw Truniri
0.25
Rewarce- Selcis Traking
Rasarca- Aftr Tran
0.2
0.30
Density
Denaity
0.15
0.10
0.0
0.6
0.00
0.00
Reward from the Fine-Grained Verifier
Reward from the Fine-Grained Verifier
(a) LLaVA
(b) InstructBLIP
Figure 9: Reward distribution comparison before and after training.
Prompt: Desctibe this image in detail.
Ours
LLaVA 1.5
The image festures a group of people starding seerre larps diephant There am least five people in the scrne, with some standiing closse further The euephant positiozed bowards right si the Image, taking up sygufflcmt portoo
The image festures group of people standing neer large elephant a drmtamn The elephant is posiboned the centrr the with trunk octanied low the T There least
elephent and others
the scene, with others further the eleplazi, people interssing
starding
adkitiin the peogle and eleghant,
as
person feeding a carvol, curvol is visible the
handbag located near the leftt side the image, possibly
elephane's mouth. and the person is holding in their band.
belonging to one of the people in the group. I'ne scane
The other people are abserving the inbermtion, likely
appeors la sot in dirt srre, which is commun enviramentt for elepharrs.
enjusing the usique engerienaa of leing so cluse to such
magesti creatum.
Promit: Describe this image detail.
intage dedicioe pieza silling dining regredents. including chaese, olivas, and syincch There sovrrd altees acstiond across the puzza, with some located noar the cetker and ollers lowarks the siges The spinchh is also er evendy distributted acroas the puzza, adding a vibrant groen color to the dish. in aaliition the
picze with The image feelune cheese, spinsch, placed tropungs steluding
table,
tooped with
dining tible. pizza into mody coton. The table is ser with a fork and kualie indi sling Uhat lbe planor is mestt le enjuyyed with utatalls.
and
In additionn to the
there twn cups the table,
there two other present the pestioned
the tabile
near the left cocner and the other neer the bawl
left comer
the table. located towards the top right comeer. thinng table oocopies most the image, with the ylzza tiking up significent poction the tuble's suffcce.
night
comes, is
right side, table orcrytes moot image, with the piece leing the main Locas.
Figure 10: Case studies on LLaVA 1.5.
A.3 WIYY DOES FEEDBACK FROM PRETRAINED VISION ENCODERS CONTRIBUTE TO THE MODEL'S PERFORMANCE THEORETICAL ANALYSIS
A.3.1 PROOF OF THFOREM 3.T
We begin by considering the distribution of the generated response Yp given by nे (y x). Since Up arg max,, R(y), this distribution is a point mass. The global minimizer will converge to a. (y) x).
By our method, we have
yp arg max [(1 A) -V UUUTUUU,TUU v/
(LI)
Simplifying, we rewrite the optimization problem as
Up arg min [|ly (Vi' + Vitt - v,
21

---

Published as a conference paper at ICLR 2025
A where 7 1-
Taking the derivative with respect to y and setting it to zero yields
2 ( (Vi V2t)) -UUU v= 0.
Solving for Up, we obtain
Dp +V V2zt + U.U...
This shows that integrating vision feedback effectively increases the weight on the visual input.
Next, we consider the loss function
M(y)
min BER":
E (= - 3T, v)
(12)
where 2 = - Struth and Youth = V + V22 + Ey.
Substituting the expressions for y, and Blruth we have
I(yp) min E
(8 Ytruth 18 v²)))²
(13)
3
Expanding, we get
I(yp) min E (1TT -8 (1,u..,))) 3 ),)
(14)
We introduce assumption that cy contains component that can be estimated via vision feedback. Suppose
c = KU.UU v +2,
(15)
where ? is noise independent of u, and is scalar.
Therefore,
Vluthh = V7 v + Vg + KU.U u ė.
(16)
Now, since
1 (A) = V + Vi' + ? U,UJo.
(17)
2
the vision feedback term helps to estimate part of ey-
We define the mean squared error:
MSEA = EE ||1000 (2)
Stnuth
(18)
Substituting,
MSEA [1(2 1:) U.U -c]²?
(19)
For 0,
MSE = E. || -KU.U v - -=11²]
(20)
The difference is
AMSE MSE MSE = (2 1x) n:2 E [[IUtU,T w/]
(21)
Setting 2K (which implies X
2x 2x+I
0), we have
AMSE 2 [| [/]]0]]] <..
(22)
22

---

Published as a conference paper at ICLR 2025
Thus, there exists À > 0 such that
Ba (v m) [L(Y]]
E
< B Ba (alm)) [L()]
(23)
This proves the theorem.
By selecting a suitable À > O, we have demonstrated that integrating vision feedback can reduce the expected loss. Therefore, incorporating vision feedback helps the model to predict the output more accurately, which proves Theorem 3.1
A.4 RELATED WORK
A.4.1 VISION-LARGE LANGUAGE MODEL.
Recently, the development of large language models (Brown al.] 2020) Touvron al. 2023 and pre-trained vision models (Radford et al. aa. has paved the way for Vision-Large Language ModeI(VLLLMs). These advanced models, which can comprehend both text and images, have greatly enhanced our capacity to automate complex tasks accross various areas such as medical application (Liu et al., 2023b), autonomous driving (Zhou et al. 2023a) and embodied agent (Peng itt al. 2023). The fundamental architecture of VLLMs typically integrates both language and vision models. This integration involves aligning the embedding spaces both modalities using Qformer or a simple fully connected layer (Zhu et al.) 2023; Ye et al.) 2023b] [i al., (023b). However, Vision-Language Large Models (VLLMs) still face the problem of misalignment, as both models typically pre-trained independently before being aligned through vision- vision-language joint training. This misalignment can lead several issues, such safety concerns, where the model may produce inappropriate or biased content (Gong et al. 2023 Tu et al. (023)) hallucinations in VLLMs, where the model generates information not grounded in the images, thus deviating from observable reality (Wang et al., 2023), and deficiencies in logical reasoning (Ghosh et al., 2024), where the model fails to coherently integrate visual and textual information, resulting in inaccurate outputs.
A.4.2 VISION-LANGUAGE ALIGNMENT
Traditional vision- language models (VLMs) have primarily aimed enhance image-text alignment using methods such as the co-attention framework (Lu et al.) 2019), anchor points (Li al. 20200), and contrastive learning (Radford et al.) 20216). With the significant advancements in large language models (LLMs), recent approaches have explored novel directions to integrate visual encoders with LLMs, enabling better comprehension of vision-language multi-modal tasks. Aligning visual and linguistic modalities can primarily be categorized into two approaches: alignment from training data and alignment from feedback. Alignment from training data involves using high-quality datasets for SET (Supervised Fine-Tuning) training, including diverse instructions and dataset compression. This method relies on the diversity and quality of the training data to improve the model's performance. Alignment from feedback focuses on fine-tuning the model using feedback of human (Sun et al. 2023 Yu al. 2023a) or other models like CLIP (Zhou al. 2024a) and large models (Li et al. 2023d Zhao al. 2023 Two primary methods for learning from feedback in VLLMs are Proximal Policy Optimization (PPO) (Sun al. 2023) and Direct Preference Optimization (DPO) (Zhao et al., 2023 Li et al.) 2023d Chen et al. 2024a). However, these methods encounter challenges. They may generate out-of distrributiiinoo data that fails to significantly enhance the model's performance and entail significant expenses in dataset construction.
A.5 LIMITATIONS
One limitation of FiSAO is its dependency on the quality and robustness of the pre-trained vision models. If the visual encoder contains inherent biases inaccuracies, these issues be propagated through the reward model, potentially affecting the overall alignment process.
23

---

Published as a conference paper at ICLR 2025
A.6 BROADER IMPACTS
The proposed enhancement in Vision-Language Large Models (VLLMs) through fine-gre tne-grained policy optimization presents several significant broader impacts across various fields and societal dimensions. FiSAOcontributes to the field of AI by providing a novel approach to self-training without the need for additional data. This can inspire further research into data-efficient training methods, fostering innovation and reducing the environmental impact associated with large-scale data collection and processing. Besides, enhanced vision-language alignment can significantly improve the performance of assistive technologies, such as screen readers and automated transcription services, making digital content more accessible to people with disabilities. This aligns with global efforts to promote inclusivity and equal access information and technology.
24


Reviews:
[{"review_id": "qE12Djewp3", "reviewer": "Reviewer_3Rvt", "summary": "This paper addresses the issue of modality misalignment in Vision-Language Language Models (VLLM). The authors demonstrate that a token-level reward is more effective than a sentence-level reward for aligning modalities. They apply a specially designed token-level reward during preference tuning to mitigate modality misalignment.", "questions": "1. In Figure 2, you show that the correlation between CLIP-based sentence rewards and conventional evaluation metrics is weak. Could you also present the correlation between token-level rewards and conventional evaluation metrics?\n\n2. In the Table 3 experiment, it would be valuable to see the performance improvements across all baselines when using your approach. While line 429 mentions that the LLaVA backbone with your method surpasses existing approaches, it is worth noting that the original LLaVA-1.5 already performed well compared to other models. Demonstrating consistent improvements across various VLLMs could serve as strong evidence of your approach’s generalizability.\n\n3. It is clear from Table 3 that your approach does not allow LLaVA-1.5 to surpass the higher-performing baselines on the MM-Vet benchmark. Additionally, Table 4 shows that fine-grained rewards do not yield an advantage on some benchmarks, including MME^c, POPE, and CHAIR. I am particularly interested in understanding which types of benchmarks or backbone architectures benefit most from your approach and where it shows superior performance.\n\n4. In Figure 4, you illustrate the comparison of reward distributions for generated objects on LLaVA-1.5 before and after training. It seems reasonable that training the model with an RL algorithm on the same reward function results in higher average rewards. However, I am unclear about the purpose of showing the shift in reward distributions. Could you elaborate on the rationale behind presenting this distribution shift? Additionally, could you provide a detailed explanation for the design of Equation (8) and the reasoning behind its specific form?\n\n5. Currently, new MLLMs leverage not only CLIP-based models but also other visual encoders, such as DINO-v2 (self-supervised) and SigLip. How adaptable is your approach when applied to MLLMs that incorporate multiple visual encoders? Could you explain any modifications or considerations necessary for your method to handle such configurations effectively?\n\n6. Your approach primarily focuses on object hallucination. I am curious to know whether it also helps mitigate other types of hallucinations, such as those related to actions or spatial relationships. Additionally, could you elaborate on how the size of the expanded set C contributes to the observed performance improvements?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed self-training approach does not need any additional data and external modules to mitigate the misalignment issue.\n2. They find that coarse feedback, such as sentence-level rewards, shows a weak correlation with hallucination detection.\n3. They are the first method to introduce token-level rewards for VLLMs preference tuning.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTc06alAiK", "reviewer": "Reviewer_SJcY", "summary": "This paper introduces Fine-Grained Self-Alignment Optimization (FiSAO), a method designed to improve vision-language alignment in Vision-Language Large Models (VLLMs). Unlike traditional alignment techniques that use coarse, sentence-level feedback, FiSAO leverages token-level feedback from the model’s visual encoder, requiring no additional external data or annotations. This token-level approach addresses hallucination issues with more precise differentiation between hallucinated and correctly grounded outputs.", "questions": "- What are the major differences between this work and recent token-level reward papers in NLP and VLMs?\n\n- What is the correlation between CLIP-based **token-level** rewards and conversational metrics?\n\n-  What are the eval. metrics used for each Table 2 column?\n\n-  To my understanding, Table 2 shows results where FiSAO is added **on top of** other methods such as POVID, Human-Prefer, etc.? Would it make more sense to evaluate the method with just RLHF sentence-level or token-level feedback to show clear differences? \n\nI find the work interesting and timely, and look forward to the rebuttal.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Token-level rewards is an emerging area of research that is worth exploring. So far, only contemporary work has begun to consider token-level reward methods for VLMs, with this paper being one of the first works in this space.\n- The methodological framework is well-defined, with both empirical and theoretical foundations.\n- Evaluation is comprehensive, with many baselines and benchmarks.", "weaknesses": "- Table 1 does not seem to be very representative of relevant literature. Neither the motivation nor the related work covers existing methods for NLP token-level reward [1,2]. Would be nice to also cover concurrent works for VLMs [3,4] so that differences with recent work are better highlighted (albeit not taken into consideration as a weakness in my review, to the contrary, this shows this area of research is very important).\n\n- It is unclear why not use OpenCHAIR which addresses the limitations of CHAIR for benchmarking object hallucinations.\n\n- Figure 2 alone does not robustly support the observation that sentence-level rewards are unreliable indicators of model performance. To be exact, Figure 2 suggests there exists no *linear* relationship between the CLIP-based sentence rewards and the conversational metrics. Sentence-level CLIP scores might still have value, but in ways not captured by simple linear correlation with BLEU or ROUGE. Perhaps the figure could also show the linear correlation for token-level rewards to make a comparative argument, which would be stronger. \n\n- Theoretical analysis assumes linear transformations and relationships between the latent representations of the image and text modalities (e.g., in the data generative model for v and t). It also relies on orthogonal matrices U_v and U_t​ for projecting latent variables to high-dimensional representations. While this simplifies the theoretical analysis, it assumes a degree of independence or decorrelation between features that may not exist in complex multimodal settings. There are other simplifications such as sub-Gaussian noise, Gaussian generative likelihood, an infinite data setting, and modeling token-level feedback as a regression problem, all of which understandably simplify (and make possible) the analysis but perhaps should be stated clearly as assumptions.\n\n- The current method relies on predefined sets of objects and labels (from datasets like Detic and COCO) to define common objects for fine-grained reward calculation and the fine-grained reward calculation uses predefined thresholds to distinguish correct and hallucinated tokens. Seems both would require careful tuning.\n\n- Calculating token-level rewards for every token in a generated sequence seems computationally intensive? The paper contains no analysis of the performance - computational overhead tradeoffs against sentence-level rewards.\n\n- The reward in Eq. (8) is too difficult to parse, perhaps consider improving clarity in the draft notation presentation format.\n\n- Please explain what is POVID, etc., in the main paper? The appendix does not seem to be a good place to add this information since these methods are crucial in understanding the results. Also briefly mention what are the eval metrics used for each Table 2 column? I could not locate this information easily.  \n\n- Also, it is unclear if Table 2 shows results where FiSAO is added on top of other methods such as POVID, Human-Prefer, etc. Would it make more sense to evaluate the method with just RLHF sentence-level or token-level feedback to show clear differences? \n\n[1] Yoon, Eunseop, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. \"TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback.\" In Findings of the Association for Computational Linguistics ACL 2024, pp. 14969-14981. 2024.\n\n[2] Yang, Kailai, Zhiwei Liu, Qianqian Xie, Jimin Huang, Erxue Min, and Sophia Ananiadou. \"Selective Preference Optimization via Token-Level Reward Function Estimation.\" arXiv preprint arXiv:2408.13518 (2024).\n\n[3] Fu, Deqing, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. \"TLDR: Token-Level Detective Reward Model for Large Vision Language Models.\" arXiv preprint arXiv:2410.04734 (2024). **[Concurrent work]**\n\n[4] Xu, Yuancheng, Udari Madhushani Sehwag, Alec Koppel, Sicheng Zhu, Bang An, Furong Huang, and Sumitra Ganesh. \"GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment.\" arXiv preprint arXiv:2410.08193 (2024). **[Concurrent work]**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xkztBDTEUF", "reviewer": "Reviewer_iK73", "summary": "The paper presents Fine-Grained Self-Alignment Optimization (FiSAO) to improve the alignment of visual and linguistic modalities in vision-language large models (VLLMs). It addresses the challenges of misalignment caused by independent pre-training, which can lead to biased outputs and hallucinations. By leveraging token-level feedback from the vision encoder, FiSAO enhances alignment more effectively than traditional methods that rely on coarse feedback. The findings suggest that this approach reduces the need for external data and improves performance in tasks requiring precise integration of visual and language information.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-structured and clearly articulated, making it accessible for readers.\n- The introduction of FiSAO is grounded in sound analysis and presents a logical approach to addressing alignment issues.\n- Extensive experimental validation supports the effectiveness of FiSAO, showcasing its practical applicability.\n- The authors offer valuable insights into enhancing alignment in vision-language large models, contributing meaningfully to the field.", "weaknesses": "- In the comparison with other state-of-the-art methods, FiSAO did not achieve results on MM-Vet that are comparable to leading approaches. It would be beneficial to include an analysis of the factors contributing to this outcome and to outline potential strategies for improvement in future iterations. Understanding the specific challenges faced in this context could guide refinements to FiSAO and the design of models.\n- Efficiency of FiSAO should be analysed as well.  And how the dependency of pretrained vision model affects the performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oYep8ClVKe", "reviewer": "Reviewer_ZUH4", "summary": "The paper proposes Fine-Grained Self-Alignment Optimization (FiSAO) to improve vision-language alignment in VLMs. The core idea is to leverage token-level, fine-grained feedback from the model's own visual encoder to guide the training process. The method avoids the need for external annotations or tools by utilizing the vision encoder as a fine-grained verifier, providing token-level rewards during training.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper addresses a critical challenge in morden VLMs. The approach of token-level, fine-grained rewards to improve alignment make much sense to me. \n\n- The paper presents a theoretical framework, including mathematical proofs.\n\n- The experimental evaluation is thorough and well-designed. The experiments support the benefits of token-level rewards in reducing hallucinations and improving benchmark performance.", "weaknesses": "1. The theoretical analysis assumes that the CLIP provides perfect vision-langauge alignment, which is a bit unrealistic as CLIP models are known to have alignment errors in practice. Under this assumption, it’s unsurprising that it can be proved that incorporating the CLIPScore improves the performance. Also, the presented theorem cannot support the benifit of _fine-grained_ reward, which is the core point of this paper. Additionally, the framework also relies on linear relationships between inputs and outputs and the infinite data setting.\n\n2.  The introduction of the MDP perspective seems unnecessary and is not well-justified within the context of the paper. Framing language generation as an MDP may not capture the long-range dependencies and context inherent in language, potentially violating the Markov property. At the end of Section 3.3.1, the author state that “This perspective highlights how fine-grained rewards can be applied ...”--I am unsure about how introducing the MDP framework demonstrates this point. \n\n3. Regarding Section 3.1\n\n   Overall, this section looks a bit unclear to me. Firstly, the calculation process of token-level reward is not introduced, leading to confusion. Whether it is token-level CLIPScore? (additional question: whether per-sentence average of token-level reward is the same as sentence-level reward?) \n\n   The authors state that sentence-level rewards show a weak correlation with conventional metrics, implying that sentence-level rewards are not effective. However, metrics like BLEU and ROUGE are known to have limitations in measuring semantic similarity, consider using CiDEr or SBERT semantic simialrity score can make the results more convincing. Also, the paper does not clarify whether token-level rewards have a stronger correlation.\n\n5. Issues with Clarity and Readibility\n\n   - Symbols like $S$ in Equation (7) and the normalization function \\( \\mathcal{N} \\) are not properly defined when first introduced. The S also overlapps with the S in MDP.\n\n   - Figure 1 shows distributions of two types of rewards before the concept and calculation of ‘reward’ is introduced.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jOToBDJ3zG", "reviewer": "Reviewer_vDxF", "summary": "This paper first demonstrates the advantages of token-level over sentence-level approaches in handling hallucinations. It then utilizes the similarity between each text token and visual features as a reward to achieve a token-level preference optimization. This optimization method employs expert models to make judgments on entity targets, enabling automatic construction of preference data. Ultimately, it achieves a certain level of performance improvement.", "questions": "See Weaknesses", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper:\n1. analyzes the significant advantages of token-level processing over sentence-level processing in handling hallucinations, laying a solid foundation for subsequent research on token-level rewards.\n2. claims a token-level visual reward, which relies on expert models to discriminate entity targets. Although the approach is not novel, the idea is commendable.\n3. conducts some simple experiments and analyses to verify the effectiveness of the method.", "weaknesses": "1. The paper overuses formula derivations while glossing over specific methodological details, making the overall writing quite confusing. Especially in section 3.2 and equation 5, there’s a lack of necessary logic, making it difficult to understand.\n2. I understand that the data construction method mentioned in the article still relies on fuzzy matching with ground truth based on entities. This seems to be a rather manual and primitive way of data construction, potentially introducing significant systematic errors. Have you tried more precise token selection methods?\n3. Overall, I find the experiments still insufficient. The experimental baselines are quite outdated, and the tested benchmarks are not comprehensive enough. Particularly, why is there a claim about handling hallucinations, but the final experiments lack an evaluation of hallucinations? Also, why is there almost no improvement on POPE? What’s the reason for this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)