Aligning Visual Contrastive learning models via Preference Optimization
Authors: Amirabbas Afzali
Borna khodabandeh
Ali Rasekh
Mahyar JafariNodeh
Sepehr Kazemi Ranjbar
Simon Gottschalk
Venue: ICLR 2025 Poster (2025)
Keywords: contrastive learning
preference optimization
alignment
reinforcement learning from human feedback
robustness
computer vision
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: In this work we propose preference-based optimization, to finetune and align contrastive learning-based models such as CLIP.

Abstract: Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Preference Optimization (PO) methods such as Reinforcement  Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have been applied to align generative models with human preferences, their use in contrastive learning has yet to be explored. This paper introduces a novel method for training contrastive learning models using different PO methods to break down complex concepts. Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task. In particular, we focus on enhancing model robustness against typographic attacks and inductive biases, commonly seen in contrastive vision-language models like CLIP. Our experiments demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring fairness, robustness, and alignment with specific preferences. We evaluate our method for tackling typographic attacks on images and explore its ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.

Published as a conference paper at ICLR 2025
ALIGNING VISUAL CONTRASTIVE LEARNING MODELS VIA PREFERENCE OPTIMIZATION
Amirabbas Afzali & Borna Khodabandeh [amir8a 18afal1,,, borna 0kh gma com
Ali Rasekh
L33 Research Center,
Leibniz Universität Hannover, Germany ali, www.kheesssssssc..
Mahyar JafariNodeh
Sepehr Kazemi
Massachusetts Institute of Technology, USA mahyarjn@mt t. edu
sepehrkazemi 90 www.l..c.... com
Simon Gottschalk
L3S Research Center, Leibniz Universität Hannover, Germany sottkkhaaee
ABSTRACT
Contrastive learning models have demonstrated impressive abilities to capture se- mantic similarities by aligning representations the embedding space. However, their performance can be limited by the quality of the training data and its in- herent biases. While Preference Optimization (PO) methods such as Reinforce- ment Learning from Human Feedback (RLHF) and Direct Preference Optimiza- tion (DPO) have been applied to align generative models with human preferences, their use in contrastive learning has yet to be explored. This paper introduces novel method for training contrastive learning models using different PO methods break down complex concepts. Our method systematically aligns model behav- ior with desired preferences, enhancing performance on the targeted task. In par- ticular, we focus on enhancing model robustness against typographic attacks and inductive biases, commonly seen in contrastive vision-language models like CLIP. Our seperimentt"ddmmoostrarc
demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring faimesss robustness, and alignment with specific preferences. We evaluate our method for tackling typographic attacks on images and explore its ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.
INTRODUCTION
In recent years, Vision-Language Models (VLMs) such as CLIP (Radford et al.) 202 have revo- lutionized downstream vision-language tasks like classification (Conde & Turgutlu 202I), object detection (Zhong al. 202T), segmentation (Xu al. 2022), and image generation (Shaaria al.) 2022]. These models are trained on large-scale web data, such as the 400 million text-image pairs used for training CLIP. However, despite their success, VLMs exhibit several vulnerabilities. For instance, adversarial attacks (Goodfellow et al. 2015 Moosavi-Dezfoolf et al. (016)) and backdoor vulnerabilities (Bai al.) [0044] Liang et al.) 2024a) can lead erroneous classifications with high confidence. CLIP has also been shown to be vulnerable to typographic attacks, where text within im-
ages causes misclassification (Goh al. 202
Additionally, biases such as gender and racial bias
"Equal contribution.
'The code is available GitHub.

---

Published as a conference paper at ICLR 2025
(Ruggeri& & Nozza 2023 Zhang et al. 2022 Bolukbasi et al. 2016; Jialu Wang et al. 2021 can be amplified by these models due to biases in their training datasets. Another dispreferred behavior oc- curs when models focus on tasks unrelated to the intended objective, as highlighted by (Menon et al. 2022]. These challenges underscore the need for alignment methods in large pre-trained models.
Generative models, including Large Language Models (LLMs), have also been widely adopted for solving complex problems across various domains. Examples in the literature include GPT-4 (Achiam et al., 2023), Mistral (Albert Q. Jiang al., (023), and LLaMA (Touvron et al.] 2023). These models are trained on massive datasets of unlabeled text, learning general language repre- sentations. Some of these LLMs require additional Supervised Fine- TuninT (SFT) to specialize for specific tasks using labeled data. However, even with fine-tuning, LLMs can produce outputs that misalign with human values or safety expectations, such as the unreliability of L.MSs in au- tonomous driving (Chen et al. 2024). To mitigate this, alignment techniques are applied, where models are further trained using a reward model that captures human preferences (Stiennon et al.) 2022 Ouyang et al., 2022 Bai et al. 2022). Popular alignment approaches include Reinforcement Learning from Human Feedback (RLHF) (Christiano et al.] 2023) and Direct Preference Optimiza- tion (DPO) (Rafailov al. 2024), both of which extend the capabilities of these models beyond what SFT alone can achieve. These alignment techniques have improved the safety and relevance of LLM outputs by making them more aligned with human preferences.
Clark et al, 2023), but to the best of our knowledge, they have not been extensively applied to contrastive learning models like CLIP. In this work, we extend the preference optimization paradigm to non- generative VLMs. Our approach builds on the contrastive learning framework while simultaneously aiming to preserve the pretrained knowledge- concept known as continual learning (Garg et al. 2024
Alignment methods have been explored in VLMs (Sun et al. 2023 Gallego 2023
Wang et al. 2023).
In this work, we extend the preference optimization paradigm to contrastive learning models, exem- plified on enhancing robustness against typographic attacks and mitigating gender biases in image classification and retrieval. By systematically aligning the model's 's behavior with human prefer- ences, we aim to preserve its pretrained knowledge while improving its performance in sensitive areas such as fairness and robustness. The main contributions of this paper are follows:
We extend and compare three PO methods, originally developed for aligning generative models, to non-generative contrastive learning models. This way, we improve model align- ment, preserving the pretrained model knowledge while optimizing for desired behaviors.
We propose controlling model behavior by adjusting the singular values of learnable linear transformation. This allows fine-tuning regarding specific concepts.
Our experiments demonstrate the benefits of applying preference optimization in con- trastive learning tasks, such as training for robustness against typographic attacks and dis- entangling gender information from the embedding space, mitigating gender bias.
2 FOUNDATIONS & RELATED WORK
Preference Optimization and RLHE Reinforcement Learning from Human Feedback (RLHF) aligns model behavior with human preferences by training the model based on feedback from human annotators. In the standard RLHF paradigm, annotators rank model responses. For example, given an input prompt a preference inequality might be expressed as yı < Ywe meaning response yo is preferred over response y1 for prompt x. These preferences are structured into datasets D = [Zz, Voe 31 ). The preference modeling often follows mnergy-based methods like the Bradley-Terry model (Bradley & Terry 1952), where the probability of preferring yoe over yi is modeled as below, where f² is the "true" reward function underlying the preferences:
(x.Ww)
(' > yr x)) ex (x.W)) + er" (maac) = a (r" (I, yw) - r" (z. y1))
(1)
Since obtaining the true reward r" from a human is impossible, reward model ra is learned by approximating the true reward function. This model is optimized by minimizing the negative log- likelihood of the human preference data:
Ln(rr) Bar. log(a (r vw) ro(x,, v())]]
(2)
2

---

Published as a conference paper at ICLR 2025
Once the reward function is learned, the RL pipeline can be used to improve the generation policy. To prevent significant deviation from the pretrained reference model, a KL-divergence-shaee reward is incorporated, with the RL objective defined as:
max Re (r y)] - (To()) We
where >> controls the degree of allowed divergence from Tef- Unlike RLHF, which often slow and unstable due to its reliance sampling and reinforcement learning, DPO (Rafailov et al.) 2024) offers more efficient and stable alternative by directly defining loss function and bypassing reinforcement learning, allowing for standard optimization. However, DPO can sometimes overfit to preference datasets. To address this, Identity Preference Optimization (IPO) (Cheshlaghi Azar al. 2024) bypasses the Bradley-Terry modelization and introduces new loss function to decrease overfitting by controlling the gap between the likelihood ratios of the model and a reference model. Defining the policy ratio as below:
To (Vo yer (x)
Ni (Wer Mi5 x)) log
(4)
Tee (yn Tre xx)
We can now express both the DPO and IPO objectives in terms of this ratio:
(To Fref) E ((c..n..w)w) |- log (Bloro (Yues yt, 2))l
(5)
(Non (Yw, y1: M1
LIPO (TTO: Tret) ((2.
(6)
Additionally, newer methods like Kahneman- Tversky Optimization (KTO) (Ethayarajh et al. 2024) only require binary signal of whether output is desired or undesired. making it more practical for many real-world applications:
ExTO(To, Tee)) B (wwy)))((()((( - (ao y: ())]
where the weights w(y) depend whether the outcome is desired or undesired, and ukto adjusts the model based on the reference distribution:
uKrO y. 3) = [o<TTo v) - 2ref)
~ Vdesired x
Au ~
Mundesired |x
w(y)
(aa TKTO (z, v)) y Vundosired
AD
y Fdesired (x
Nolyla) where zxt = E (To (v/(a(( and TkTо(I; y) = 3og "=((")l))) Further details
of mentioned methods are provided in Appendix A These methods have been widely adopted in
training generative models, including diffusion models like Denoising Diffusion Policy Optimiza-
tion (DDPO) (Black et al.] (024). This work investigates how these methodologies can be applied to contrastive learning- models. This paper focuses on two critical applications: mitigating typographic attacks and addressing biases, such as gender bias in contrastive visual models like CLIP.
Typographic attacks. Typographic attacks exploit model's tendency to prioritize text over visual content (Goh et al. 2021), causing CLIP to misclassify images based on misleading text. For ex- ample, writing "cal" on image of a dog could lead CLIP to incorrectly label it as "an image of a cat". " This represents a clear example of undesired behavior in CLIP and is an ideal test case for preference optimization. Recent works, such (Azuma & Matsui 2023 Joanna Materzynska al. 2022 IIharco et al. 2022). address this shortcoming through various fine-tuning schemes.
Inductive biases. Biases such gender and racial bias (Ruggeri & Nozza 2023 Zhang et al. 2022 Bolukbasi et al., 2016 Jialu Wang et al. 2021] are prevalent in
models that often unintentionally encode and maintain societal biases, leading skewed predictions dispropor- tionately affecting marginalized groups. Another common issue is task bias, where models focus on unintended aspects of the task, further amplifying distortions in their outputs (Menon et al. 2022).
METHODOLOGY
Figure I gives overview of approach with joint training objective: preference-baie based contrastive optimization using a dataset of preference labels 2 Ppet and regularization using a regu- larization dataset Dreg. Details of our methodology are given in the following.
3

---

Published as a conference paper at ICLR 2025
plana dog
plano dogg
Appickea (ob
Text
Encoder
preference labels
regularication labels
g
Play
Image Ensoder
That Shar Jar")
Image Encoder
rmn
Encodor
Image
(a (ae')
(x', N.oyl U1) E D poct
x. E Diog
(T,, Treti Dreet) + Xug freg (To, Trati Dreg)
Figure 1: Overview our proposed approach. On the left side, we calculate the prefer- ence optimization loss Lrrer(To, Teet' Prret using the preference dataset, and the output log- its T(yw) ((x) from both models. On the right side, the regulatory loss freg (To, Trel; Dreg) is calculated using the regularization dataset. The snowflake icons denote frozen encoders.
3.1 PROBLEM FORMULATION: CONTRASTIVE LEARNING AN MDP
To effectively apply preference optimization to contrastive learning models, we first need to frame the problem within the reinforcement learning paradigm, modeling as a Markov decision pro- cess (MDP). Specifically, we define the contrastive learning task one-step MDP, denoted as M = (S, Po. P. R, 7), where the state space, A the action space, Po is the initial state distribution, P is the transition function, R is the reward function, and is the discount factor. Since is one-step process, the transition function and the discount factor are unnecessary, as states do not continue beyond the first step. In this setup, the model operates input (either text or image) represents the state s S, and the most similar text or image is selected the action A. Additionally, Po defines the distribution of initial states, and represents the reward function.
retrieval system: the
The policy Ta(a)s) is defined analogously the contrastive learning objective, typically using softmax function over the similarity score fø(r, y) = I=(r)) where Ig XMimage - Rdd and To Niext + Rd are the encoders for the image and text, respectively. Here, Ximage and Mext represent the input distributions of images and texts, and is the dimensionality of the latent space where both image and text representations are embedded. 0 represents the model parameters, and T is the temperature. The action space is chosen from fixed set of possible outputs A = y 2 {ys Ne(L...... where K represents the total number of classes. The components M(S, A, Po. R) are defined as follows:
(o((s) - p(x)) ((ssaa) r(x, v)
efolu.r)
(a(a))) A
2
E efaly).))
This MDP framework can be applied to different tasks and provides flexibility in designing policies. We specifically utilize methods like DPO and KTO due to their implicit reward modeling, which allows more effective model alignment with desired behaviors.
3.2 PREFERENCE AND REGULARIZATION DATASETS
In our framework, the desired response of the model given an input x (e.g., an adversarial image) is considered the preferred output Yu- This represents the correct human-aligned response that aim to optimize for. On the other hand, the dispreferred output YI refers to the model's potentially biased response, the target of adversarial attack, or undesirable outcome (such as bias related
4

---

Published as a conference paper at ICLR 2025
to gender or other attributes). The model learns to be robust against adversarial attacks specific biases by comparing these two outputs.
We use two datasets to facilitate this learning:
Preference Dataset pret: This dataset contains pairs (z, yur vt). where yu is the preferred output and yi is the dispreferred output. The model learns to differentiate between desired and undesired outcomes through this dataset, improving robustness against adversarial attacks and biases.
Regularization Dataset Digg: This dataset consists of clean samples (images or text) with- out adversarial manipulation. It is used during training to ensure the model retains its accuracy on clean data.
By combining these datasets, the model is trained to align with specific behavior or to be robust against adversarial inputs while maintaining performance on clean data and other downstream tasks.
3.3 PREFERENCE-BASE CONTRASTIVE OPTIMIZATION
Building on the proposed formulation, preference optimization can be effectively applied to con- trastive learning models by guiding the optimization process toward desired behaviors. Moreover, we can shape the policy to learn specific behaviors or patterns by defining them preferred. Fur- thermore, in Section 4.3 we highlight a significant advantage of this optimization paradigm as a fine-tuning method, helping retain the pre-trained knowledge of the model while adapting it to new preferences and support it with experiment results.
Recalling Eq.
4). we first focused on policy ratio, h.o (yl v1, x)) which
write
(8) Given that the objective of the model me is to simultaneously increase the probability of the pre- ferred label To (yw ar) and decrease the probability of the dispreferred label To the term heo (Yw: Mt, ) is positive. Note that jet in this framework is kept frozen.
hap (Ners x1,
log Ta (Vm.la)))) log To 5o(31l2)
log [rer(Uunlæ)) log Ter((ll
Lemma 3.1 Under the assumption that the text encoder is frozen, i.e., Ted To T, the policy ratio for models using the contrastive learning policy, in the methods such as DPO or IPO can be expressed as 2
hio (yw M. x)) } - In) (T -(w))))
Based on Lemma 3.1 the term hro depends only on the term (Io(x) ( (2))). This indicates that the loss objective is designed to adjust [x()) to align more closely with the preferred text embedding difference, i.e., ((ww) - T())), while maintaining appropriate proximity to the reference embedding (x)( Having this in mind, we now raise the following question:
How does the Gradient update work in case of DPOПPO?
First, we need to formulate the gradient of loss correctly in order to analyze it. The following Corollary provides more insightful way of writing the gradient.
Corollary 3.2 Using Lemma 3.1 we can obtain the gradient of the loss as the following
OIa
VoLipng Ro, Tref) = Wpree (Yw: y1; x)
(T(ww)
-T(x),,
(10)
00
(7)
(IT)
where the gradient weight (I) in DPO is Wpeer (Nwr v) al-Bhzo (Yr yr x)), and in IPO, (peer(yw: z1 x)
hx, (Yw, V1. 2z) Intuitively, the term (II) shows that adjusting the im- age embedding [x in the direction of T(ww) - T(u), also encourages the model to better align
"See Appendix B.1 for the proof.
Further details are provided in Appendix
B.3
5

---

Published as a conference paper at ICLR 2025
with the preferred outputs. Importantly, if the distribution of πTş deviates significantly from the refer- ence distribution Tet. the absolute value of the policy ratio increases. As noted previously, this effect reduces the update rate in the direction of (II), acting mechanism to control the model and keep it closer to the reference policy, preventing excessive divergence from the reference distribution.
In the KTO method, since there is direct comparison between Yo and yt, do get the same equation as in [ However, the main idea still applies. The model adjusts the image embedding (x) to match human preferences, but does so through a different optimization approach.
3.3.1 REGULARIZATION
In all our training methods, we explicitly constrain the model to remain proximal to the reference model on the preference dataset D pret. However, the preference dataset pret is typically small and often fails to capture the entire data manifold. In many cases, it primarily consists of adversarial and out-of-distribution samples. To address this limitation, we introduce additional regularization terms into our training process, ensuring the trained model maintains proximity to the reference model. This motivates the introduction of a regularization term feg. designed to maintain proximity between the trained model wa and the reference model ref as follow:
log (1)xx) Trog (3/x)]
(Teg(T,, Tyefi Dreg)= = (xa.(T (Tr((()))))) Egn E Bywa(ula))
(1I)
where Preg is the set of clean images; for example, the typographic attack task, we construct Preg using clean images from the training dataset. By adding this term to the loss function, the model can better maintain its accuracy on true labels, even when facing adversarial challenges, by using the regularization dataset Drog to ensure robustness.
A shortcoming of these preference losses also suggests introducing additional regularization terms. From this point on, for the sake of simplicity, we denote (e(Nulz)
Terlya ar) = 21 and xe [z)
= I2. To
)u((u] [n)
better understand the effect of regularization, we will build up on top of the following theorem.
Theorem 3.3 (Feng et al.
2024 The partial derivatives of Eq. with respect to iı and x2 are
given by:
@Lpro(Z1: d @Lrro(21;22)
T2
1.
(12)
dx
We find a similar result in the case of IPO
Proposition 3.4 The partial derivatives of Eq.
66 with respect to xı and x2 are given by:
@Lrro(11:72)
1 @Lro((11322)
T 1.
(13)
dx1
In our problem formulation in Section 3.1 we define the true label as you and the adversarial label (e.g., typographic label) as yt. According to Theorem 3.3 and Proposition 3.4 the rate of increase in the probability of the preferred response Yo is lower than the rate of decrease in the probability of the dispreferred response yt. Therefore, as suggested by (Feng al. 2024). model primarily focuses on reducing the likelihood of the dispreferred response. This also motivates us to introduce more regulatory terms to ensure proximity.
Finally, for training, we propose using a total loss function from the dataset D = (Dpret: Drey) as follows, where Creg the regularization loss and Spref is the preference optimization objective, computed using one of Eqs. 5 or 7
L( Tref; D) ( Feef; Drret) Creg Tref; Dreg).
(14)
For example, consider giving images as input to CLIP, where the model chooses the corresponding caption from fixed caption set. The selection process follows the policy we defined, using the softmax function over the similarity scores between the image and each caption in the set, as shown The training process using the proposed loss (Eq. (14) is outlined in A Algorithm
in Figure I
*See Appendix B.2 for the proof.

---

Published as a conference paper at ICLR 2025
Algorithm 1 Preference-based contrastive optimization
Require: dataset D (Drret, Dwg), Model Te, Reference model Ted, Regularization coef. Are
1: for each batch b D do
2: Dycet, Dieg +
Get batch preference regularization data
3: Compute (v)x)) - Softmax(fu (v)) for v E {0, ref)
Compute model and ref. distributions
4:
lprett f ( iTret; Opect)
b Compute preference loss using one Eqs. 5
(6) or
5:
freg +- (Tee(TT,, Teel beee)
>. Compute regularization loss as in Eq. (11 zero if Dreg I
6:
Aut 6- 4-4poo + Amy heg
D. Total loss
7:
Update model Ty by minimizing lut
8:
end for
3.4 LINEAR TRANSFORMATIONS AND ADAPTATIONS
In many cases, fine-tuning the entire model may be unnecessary and require more interpretation and control. Therefore, we employ a linear head on top of the encoders. This linear projection layer offers more interpretable approach, providing a plug-and-play framework where the projection can be learned during training and then applied as needed.
This linear layer also introduces linear-algebraic perspective to the adaptation process. Suppose the learned transformation is represented by the learnable matrix W. This matrix transforms both vectors I(x) and T②y) to WI(x) and WT( respectively, thereby transforming the similarity function f)y,x) = T(x)"T(w)/+ into a new function f(y, x) the Singular Value Decomposition (SVD), we express W = UEV'. Therefore, we have:
Utilizing
f(y, I W WTW
(V
(15)
In this formulation, the unitary matrix selects the important directions, and the diagonal matrix Σ = diag( (1. (o....., ox) determines the degree of attenuation amplification along each direction ey- Specifically, each o, determines whether the corresponding direction U; is strengthened (o, > 1) or weakened (a, 1). As detailed in Appendix E our training process keeps the model close to the reference, as expected. Consequently, each a; remains relatively close to 1, and the overall matrix WT' stays close to the identity matrix.
This observation enables post-training adjustments, allowing us to strengthen or weaken each as needed. One approach achieve this by directly tuning each singular value d;, either through linear interpolation or by applying matrix powers. In our work, we utilize matrix powers to modify the singular values via a transformation scaling parameter & I. implementing the transformation Σ E as follows:
W. = UV'VT,
(16)
where E¹ diag(cí.....o')). This approach provides flexibility in adapting the model, enabling fine-grained control over the transformation applied to the encoded representations.
For large positive or negative values of f, we recommend normalizing the output embeddings, as done in the original CLIP model. This normalization prevents certain directions in the embedding space from becoming excessively large. While normalization still allows for the strengthening or weakening specific directions, it ensures that the length of the vector remains bounded, instead of becoming arbitrarily large. This approach maintains the intended alignment or misalignment with certain directions without causing instability in the embedding space.
4 EXPERIMENTS AND RESULTS
In this section, experimentally justify method capabilities, using the CLIP model contrastive learning model. An explanation of the datasets we used can be found in Appendix
Setup: For all experiments, we used 8 A100 GPUs, each with 40GB of memory. For the typographic attack experiments, hyperparameters such as 3 and were selected based on our empirical studies described in the appendix. Specifically, = 1.5, = 0.01 for KTO. In disentangling gender bias, we also set B = = 1. All models were trained for three epochs with batch size of 512. The learning rate was to 2 x 10-5, and
used B À 1 for DPO,
> 0.01 for IPO, and

---

Published as a conference paper at ICLR 2025
Table 1: Classification accuracy on various datasets: O (Original dataset) and T (Typographic dataset). The last row highlights the differences between KTO, our best-performing proposed method, and the bert-perforning baseline for each dataset. Positive differences indicate improve- ments achieved by the proposed method.
Cattech001
OxfordPees
StanfordCars
Flowerst02
PGVCAireaatt
SUNSY7
EureSAT
Arg:
T
T
0
O
T
CLIP
53.54
63.97
$7.35
58.95
58.72
21.02
66.32
53.52
18,99
10.85
42.57
25.53
61.74
54,02
42.98
4,56
58.05
31.31
Maserrymias+
50.53
74.75
75.01
63.61
40.33
15:79
51.86
54.95
13.23
8.28
36.28
33.03
$1.06
99.52
37.30
16.32
45.25
35.77
PAINT
83.48
83:57
85.25
76.35
55.30
33.64
64.75
54.90
10.73
14.46
42.61
36.00
61.09
53.62
38,20
17.31
56.74
46.31
Prlls
89.26
79.54
67.22
72.86
57.47
25.64
63.82
44.12
19.36
14.49
40.64
31.60
61.41
43.90
43.85
9.85
57.87
40,58
Oun: (DPP))
87,50
55.43
55.25
79.72
55.03
1 34.33
50.60
55,70
16.21
13.87
39.36
38.45
61.02
55.54
09.83
28.32
56.41
49.00
Oun (IPO)
65.73
93.78
55.32
80.44
53.67
35.02
54.50
52,90
17.07
15.36
40.53
39.94
61.91
58.05
46.12
43.23
55.72
51.14
Ours (KTO)
$7.67
86.02
55.41
81.02
57.76
37.04
59.10
58,00
17.27
15.59
40.74
40.33
02.53
99.01
46.26
36.94
57.09
51.74
DiTerenex
11.61
12:45
1181
14.47
10.9
13.60
15.63
13.08
11.09
11.30
11.87
13.73
10.83
15.39
12.41
119.63
10.78
15.43
employed the Adamax optimizer with a linear warmup and a cosine scheduler, setting the warmup ratio to 0.1. We set the random seed to 0 for all experiments ensure reproducibility.
4.1 TYPOGRAPHIC ROBUSTNESS
We further evaluate our method on the typographic attack mitigation task. We set up the prob- lem described in Section g using the template "an image of a <class i> i>" (or more dataset-specific template for non-generic images), designating the typographic class as the undesired outcome y/ and the original caption as the desired outcome ya--
Baselines: We evaluate our method against the following baselines: pre-trained CLIP (Radford al. 2021), D) (Materzynska et al. 2022). PAINT (Ihaaroo et al. 2022), and Defense- preffx (Azuma &
Matsui 2023), with results collected from (Azuma & Matsui 2023) for comparability. (Materzynska
et al. (2022) propose using orthogonal projections to create subspaces in CL.IP's image encoder that separate the processing of written words from visual concepts. PAINT (Patching with Interpolation) is a technique for adapting ppen-vocabulary models to downstream tasks, while Lefense-Prefic is specifically designed to counter typographic attacks on CLIP by inserting learned token before class names in text prompts. We use ImageNet-100, a 100-class subset of ImageNet (Russakovsky et al. 2015), to train the model. Typographic attack images are generated by adding misleading text labels to the original images using open-source implementations from (Azuma & Matsui 2023).
Further details of our baselines are provided in Appendix F
Our results shown in Table I indicate that our methods effectively prevent typographic attacks with little loss of prior knowledge, with improvements of up to 19.63 in accuracy over the best performing
baseline 5
4.1.1 CONTROL BETWEEN OPTICAL CHARACTER RECOGNITION AND OBJECT DETECTION
In this experiment, we apply linear adaptations and retrain the network, then evaluate the train- ing results. We then interpolate between models using previously discussed linear transformation methods, varying the transformation scaling I, defined in Eq. (16) from -4.0 to 4.0 to assess per- formance. By doing this, measure the effectiveness of our controlling scheme between the tasks of Optical Character Recognition (OCR) and Object Detection (OD).
Our results in Figure 2 demonstrate that we can confidently control the trade-off between OCR and OD without any additional training and with minimal impact on overall accuracy. This method allows us choose which task prioritize flexibly, and we can visualize the performance frontier, showing that improvements in OCR accuracy come at the cost of OD accuracy and vice versa. The ability of this linear layer to manage the trade-off suggests that the concept is linearly separable within the CLIP embedding space.
"oore analyses are shown in Appendices and
8

---

Published as a conference paper at ICLR 2025
Modele Prefrained model
0.7
Objject deseection accuresry
0.6
0.5
Clann data
0.4
0.
accurscy
Objeci delection accurwy
03
0.3
02
0.2
0.1
0.
Transformation scaling
OCR task accuracy
(a) Accuracy on typographic samples and percentage of typo- (b) Frontier of a DPO fine-tuned model, graphic label predictions versus transformation scaling factor t. showing OCR vs. OD accuracy across As t increases, the model favors object labels over typographic with varying t. labels while maintaining accuracy.
Figure 2: Comparisons of optical character recognition (OCR) and object detection (OD).
4.2 DISENTANGLING GENDER UNDERSTANDING
In this setup, we demonstrate how our approach can reverse contrastive learning model's under- standing of concept (e.g.. gender) without significantly altering other aspects of its performance.
For demonstration, we focus on flipping CLIP's understanding of gender. We utilize dataset of labeled images featuring men and women engaged in various activities and occupations from (Zhng et al. 2022). The problem setup is as follows: given choices like Vmmn "The man is <activity>" and Jweman = "The woman is <activity>* the CLIP model typically picks one of these answers. To reverse the gender understanding, we designate the label with the correct activity but flipped gender as the preferred answer yo and the original gender- specific label yı. As the regulatory term, we use the same dataset of images. However, we employ androgynous target labels such "The person in <activity> to ensure that our model does not forget any information only the linear layer and then visualize the effect of varying the transformation scaling / between O and 1. Our results confirm that we have high degree of control over CLIP's understanding of gender, as demonstrated in Figure
detecting the correct activity. We repeat our learning process using
Testuned Origial
0.8
0.5
Accurayy
0.6
Pobabiliy
0.6
0.5
Grate Ixteste
0.3
0.2
6.1
0.0
Transformation Scaling
(a) Example image showing a man work- ing.
(b) Model predictions before and after ap-
(c) As f increases from 0 to 1, gender- specific predictions are reversed. t" marks the point where gender information is neu- tralized, leading balanced male and fe- male predictions. Task detection accuracy remains stable across all models.
plying our gender- flipping method, show- ing changes in the predicted captions:
(I) "The man in the photo working." "
(II) "The woman in the photo is working."
Figure Analyses of the models' understanding of gender.
To further evaluate our results, we apply the modified CLIP models to an image retrieval task, dis- tinct from the original text retrieval task used for training. We assess how this impacts the retrieved images for specific text prompts by computing the similarity scores between the prompts and im- ages as simmage text)= Io (image))"Ta
Using the dataset from (Zhang et al. 2022). issue
9

---

Published as a conference paper at ICLR 2025
Figure 4: Images retrieved for the caption "an image of police" with three different policies from top to bottom: reversed understanding of gender (6W, 4M), pretrained CLIP model (2W, 8M), neutralized understanding of gender (5W, 5M), i.e., I. tt.
prompts describing individuals in various occupations to analyze the model's behavior and showcase an instance in Figure
4.3 PRESERVING PRETRAINED KNOWLEDGE VIA PREFERENCE OPTIMIZATION
In all of the aforementioned preference optimization methods, implicit explicit constraint exists to keep the model as close as possible to the reference model. Specifically, including KL-divergence regularization facilitates adjustments without deviating significantly from the ref- erence model wref- To support this claim, Figure [5a] shows the KL-divergence between the out- put distributions of the model and the reference model on the original dataset, measured as after fine-tuning for the typographic robustness task using dif-
ExwPan, [DKL (Te( ferent optimization methods.
Compared to Cross-Entropy optimization, our proposed methods result in lower divergence, demon- strating better retention of the model's pre-trained knowledge. Figure 5bllusstrees that we can con- trol this deviation (i.e., retention of pre-trained knowledge) during fine- fine-tuning by adjusting the B hyperparameter, consistent with Eq. (3). All models (a) and (b) were trained on the Imagener100 and FOODIO1 datasets, respectively, and the models in (b) were optimized using the IPO method.
60
8.1
Nodel
(a) Our methods vs. Cross-Entropy optimization (CE).
(b) Effect of changing the hyperparameter B.
Figure KI--divergence studies.
CONCLUSION
This paper introduces a novel approach for improving contrastive learning models through Prefer- ence Optimization (PO). Our approach systematically aligns model behavior with desired prefer- ences, enhancing robustness against adversarial attacks and effectively mitigating biases in vision- language models like CLIP. Our experiments demonstrate improved performance standard con- trastive learning methods in adversarial robustness and fairness. This work could open new pathways for applying preference-basee optimization to non-generative models.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially funded by the Federal Ministry for Digital and Transport (BMDV), Germany ("MoToRes"" 19F2271A) and by the Federal Ministry for Economic Affairs and Climate Action (BMWK), Germany ("ATTENTION!", 01MJ22012D).
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini arXiv:2303. 08774, 2023.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7B, October 2023. URL http::/ /tppiv//axiv../// wwg. abs 2200 06825 arXiv:2310.0632 [cs].
Hiroki Azuma and Yusuke Matsui. Defense-Prefix for Preventing Typographic Attacks CLIP, 04512 arXiv:2304.04512 [cs].
September 2023, URLhttp://axxxv..rrg////
Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. BadCLIP: Trigger- Aware Prompt Learning for Backdoor Attacks CLIP, March 2024. URL http:///rxiv.
org/abs/ 2311 61 94 arXiv:2311.16199 [cs].
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, April 2022. URL /http::/arxx..///////////// org/ http:/bb//2220/ 2204 05862 arXiv:2204.085 0,5862 [cs].
Fábio M. Bayer, Renato J. Cintra, and Francisco Cribari-Neto. Beta seasonal autoregressive moving average models. Journal of Statistical Computation and Simulation, 8811)):961-2281, July 2018. ISSN 1563-5163 663. doi: 10. 00010800004495552188.411111
10.1080 01000009499953 20186655..2211.8.. 101.81199999 1491974
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training Diffusion Mod- els with Reinforcement Learning, January 2024. URL http::/aarxii.orgg/bb///2
13301 McXiv:2305.1330 [cs].
Tolga Bolukbasi, Kai- Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings, July 2016.
/tRp:ht://axxxivvooo abs www. 06520 1XXi:16077006500 [cs, stat].
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- discriminative compo- nents with random forests. In Computer vision-ECCV 20/4: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
Ralph Allan Bradley and Milton E. Terry. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika, (9(3/44:324-3455 1952. ISSN 0006-3444. doi: 10. 2307/2334029. URL //tp:////www.// jstor, www.or/gg...aae.eeeeeeeeeeeeeeee htttttaable2233300 Publisher: [Oxford University Press, Biometrika Trust].
Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with lmms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Roboties and Au- tomation (ICRA), pp. 10093-14000 IEEE, 2024.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, February 2023. URL /ttt::/ttttpiiiaia /wtp/iai.ooooo/../..........eoeo arXiv: 706.03741 [es, stat].
abs/1706. 03741

---

Published as a conference paper at ICLR 2025
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. De- scribing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet. Directly Fine-Tuning Diffusion Mod- els on Differentiable Rewards, September 2023. URL http::/arxiv..or//aatat
17400 arXiv:2309.17400 [cs].
Marcos V. Conde and Kerem Turgutlu. CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Work- doi: 10.1 ww..1999/VWW53009.. 202100444, URL
shops (CVPRW), pp. 3951-3955 June 2021.
http:://rxii org/ haae:/bb//222000 2204. 14244 arXiv:2204. 142444 [cs].
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Cas- tricato, and Edward Raff. Vggan-clip: Open domain image generation and editing with natural language guidance. In European Conference on Computer Vision, pp. 88-105. Springer, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 255. leee, 2009.
Kawin Ethayarajh. Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model Alignment as Prospect Theoretic Optimization, February 2024. URL httt:::/axxii..rrrr abs/2402. 01306 arXiv:2402.01 [cs].
Li Fei-Fei, Rob Fergus, and Pietro Perona.
Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference on computer vision and pattern recognition workshop. pp. 178-178 IEEE, 2004.
Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, and Wenqiang Lei. Towards analyzing and understanding the limitations of dpo: A theoretical perspective, 2024. URL https: //arxiv. org/abs/ 204. 04626
Victor Gallego. Fast Adaptation with Bradley-Tenry Preference Models in Texa-To-Image Classi- fication and Generation, September 2023. URL arXiv:2308.0799 07999999..999... [es].
07929
Saurabh Garg, Mehrdad Farajtabar, Hadi Pouransari, Raviteja Vemulapalli, Sachin Mehta, Oncel Tuzel, Vaishaal Shankar, and Fartash Faghri. TiC-CLIP: Continual Training of CLIP Models, March 2024. URL (tLp:ttt://// http:::/aaxiv..rrog abs///ab///// 0300.112 16226 arXiv:2310.16226 [cs].
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In Sanjoy Dasgupta, Stephan Mandt, and Yingzhen Li (eds.), Proceed- ings of The 27th International Conference on Artificial Intelligence and Statistics, volume 238 of Proceedings of Machine Learning Research, pp. 4447-4455, PMLR, 02 04 May 2024. URL https: htttps://proceeiingg mlr wwe.sp..es///eemme.. v2 38/ hwtp///////hhha/eee wwwshhaghhihiiee azar 24a html.
Gabriel Goh, Nick Cammarata t, Chelsea Voss t. Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal Neurons in Artificial Neural Networks. Distill, 6(3): e30, March 2021. ISSN 2476-0757. doi: 0.23915/dissiill000033 URL Dub/2021/multimcdall
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2015. URL
wwt//bs//aa/aa////oo 6572
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEFE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7)22217-2226, 2019.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song. Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching vocabuocy models by interpolating weights, October 2022. URL httt:://ara arXiv:2208.054: 055592 [es].
httg//bs/22000 05592
12

---

Published as a conference paper at ICLR 2025
Jialu Wang, Yang Liu, and Xin Eric Wang. Are Gender-Neutral Queries Really Gender-Neutral. Mitigating Gender Bias in Image Search, September 2021. URL /tLlbtt///aaxi.. org/ abs/ 2109. 05433 arXiv:2109.05433 [cs].
Joanna Materzynska, Antonio Torralba, and David Bau. Disentangling visual and written concepts in CLIP, June 2022. URL/http://arxet
org/ abs/ 07835 arXiv:2206.07835 [cs].
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-graing ane-grained categorization. In Proceedings of the IEEE international conference computer vision work- shops, pp. 561, 2013.
Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Ee-Chien Chang, and Xiaochun Cao. Revisiting backdoor attacks against large vision-language models, 2024a. URL https: 18844
Xize Liang, Chao Chen, Shuang Qiu, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Ropo: Robust preference optimization for large language models, 2024b. URL
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- -rae-graindd visual classification of aircraft. arXiv preprint 1306:v13306.1555 2013.
Joanna Materzynska, Antonio Torralba, and David Bau. Disentangling visual and written concepts 07835
in clip, 2022.
Sachit Menon, Ishaan Preetam Chandratreya, and Carl Vondrick. Task Bias in Vision- Language Models, December 2022. URL
arXiv:2212.04412 [cs].
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. DeepFool: a simple and accurate method to fool deep neural networks, July 2016. URL http::ttt////x..rrggg/ abs/
1511. 04599
MEXiv:1511 11115111199999 [cs].
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixh Indian conference on computer vision, graphics image processing, pp. 722-729 IEEE, 2008.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL
wrt. 2203.. 2tt.//0.0//2200.015555 02155
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021.
org/abs
arXiv:2103.00003 [cs].
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. TtLlttts////axxv.rrrt abs http//bbb///5../ 220.//280000000
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? CoRR, abs/1192..188.1 2019. URL httt:://axiv.coo abs/ 1902. 10811
Gabriele Ruggeri and Debora Nozza. A Multi-dimensional study on Bias in Vision-Language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the
Association for Computational Linguistics: ACL 2023. pp. 2023. Association for Computational Linguistics. doi: 10. 10.186633vv12022 http:////1222....nnn------- acc.43. URL https: //acs antho ogy. ogy.org/2023, ndi ndings-aci 403
Toronto, Canada, July
13

---

Published as a conference paper at ICLR 2025
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeey Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge, January 2015. URL http://arxiv: htg/abs/14055 0575 arXiv:1409.00 [cs].
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image dif- fusion models with deep language understanding, 2022. URL wtt.httsss//exxi...rrg///////
2205 0165.11487 487
Yang Shu, Xingzhuo Guo, Jialong Wu, Ximei Wang, Jianmin Wang, and Mingsheng Long. Clipood: Generalizing clip to distr--iistiituioonns, 2023. URL
00864
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, February 2022. htthttp:: /ttp:///aa.vvv.v www.xiiii...gaaaaeeeeeeeeeeeeeee 2009 C1325 2XXv:2009.01322 [cs].
Zhiqing Sun, Sheng Shen, Shengeao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang. Yiming Yang, Kurt Keutzer, and Trevor Darrell. Aligning Large Multimodal Models with Factually Augmented RLHF, September 2023. URL http:
serxiv.org/aas..
14525 arXiv:2309.1453 14525 [cs].
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin. Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Founda- tion Language Models, February 2023. URL arXiv:2302.13971 [cs].
Amos Tversky. Advances in Prospect Theory: Cumulative Representation of Uncertainty, volume 5, pp. 493-519. 01 2016. ISBN 978-3-319-20450-33 doi: 10.1007/978-33 01019/978---1119245--22222
Liyuan Wang, Xingxing Zhang. Hang Su, and Jun Zhu. A Comprehensive Survey of Continual Learning: Theory, Method and Application, January 2023. URLhttps://axxxv 2302. 00487v3
abs/
Jaanxing Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119: 3-22, 2016
Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. sim- ple baseline for open-vocabulary semantic segmentation with pre-trained vision-languagg model, 2022.
Yi Zhang, Junyang Wang, and Jitao Sang. Counterfactually Measuring and Eliminating Social Bias in Vision-Lamguage Pre-training Models, July 2022. URLp:t::///axxt org. org/abs/2207.
01 01056 arXiv:2207.0n1 0000600066611 [cs].
Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao. RegionCLIP: Region-based Language-Iman Pretraining, December 2021. URL http://aaxx.v.orr/bbb//// 09106 arXiv:21 12.0912 [cs].
14

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Theoretical Foundations of RLHF
16
A.1 Deriving Preference Optimization a Supervised Learning Framework
16
B Proofs
18
Proof of Lemma 5.1
18
B.2 Proof Proposition 3.4
B.3 The Gradient the DPO and IPO Objectives
19
C Features learned learned by our method
19
D
Latent visualization using VQGAN
19
E Near Orthogonality of
F Datasets & Baselines
22
F Datasets
22
22 Baselines
22
G Hy dyperparameter Sensitivity Analysis
24
Effect of 3
24
Ablation Study on the Regularization Loss Lag and the Effect of
24
Beta Moving Average Exponential Moving Average
25
Sample Efficiency Analysis
26
Impact of Number of Typographic Instances per Image
27
Performance Comparison of DPO, IPO, and KTO
27
J.1
Sample Efficiency
27
J.2
Sensitivity
................................
27
L3
Performance Across Difterent Datasets
28
J.4
Weight Analysis of Different Variants
28
Experiments on Gradient-based Áttacks
28
15

---

Published as a conference paper at ICLR 2025
A THEORETICAL FOUNDATIONS OF RLHF
RLHF extends the traditional reinforcement learning paradigm by incorporating human preferences into the learning process (Ouyang et al. (022). This feedback can take the form of direct preference comparisons between different actions or sequences of actions, or it can involve explicit corrections when the agent's is behavior is undesirable. Human judgments are converted into a reward signal that helps the agent refine its policy. RLHF has gained attention in areas like natural language processing and fine-tuning large language models, where specifying a clear reward function is difficult and aligning the agent's s behavior with human values or intentions is critical. This method offers a more robust approach to training Al systems in tasks that require subjective judgment or nuanced decision-making.
In the context of aligning Large Language Models (LLMs), RLHF plays a crucial role. The process usually begins with supervised fine-tuning (SFT) of a pre-trained model on task-specific datasets, which helps the model learn to generate appropriate responses for various queries, resulting a reference model denoted as Tee- This model is refined using RLHF, which consists of three core steps.
1. Supervised Fine-tuning (SFT): Pre-trained language models often require initial supervised fine-tuning phase to follow human instructions better. This phase aligns the model's responses with human expectations. After this phase, obtain model Te which serves the initial policy for further training.
2. Reward Model Training: Once the SFT model is trained, the next step in RLHF is train reward model Tw (zt, y). where x is an input prompt and is the corresponding response. The reward model is trained reflect human preferences between different responses. Given two responses yo (preferred, or "win") and yt (less preferred, or "lose"), the Bradley- -erry model is commonly used model human preferences:
p" (yo 11(z) ('v(z.8o) + etv(a.)))
(17)
which can be rewritten
p" (yw > y1 = (z, Bw) - To (x)
(18)
where is the sigmoid function. To train the reward model, the following log-loss function is minimized over a dataset of human preferences D = ((x(0, (u6) v/ D)
- -B [log (r (w) rw (z, 3)))]-
(19)
3. Fine-tuning with Reinforcement Learning: In the final stage of RLHF, the trained reward model is kept fixed, and the policy Rl. (initialized from T is fine-tuned using the PPO algorithm. The goal is to optimize the policy maximize the expected reward predicted by the reward model while maintaining closeness to the supervised fine-tuned model through Kullback-Leibler (KL) divergence regularization. The optimization problem formalized as:
max E Rn D.ywohl (v)=)) [rw v) 3 (RI ( (vla)) SFT ()))
(20)
where DKL (v]=) m SFT (z]=)) represents the KL divergence between the RL policy and the SFT policy. The coefficient 3 controls the trade-off between encouraging exploration and maintaining consistency with the supervised model. KL divergence serves two purposes: it acts as an entropy bonus, encourages exploration, and ensures that the RL. policy stays within the supervised fine-tuned model.
This structured process ensures that LLMss are capable of solving complex tasks and remain aligned with human preferences and ethical guidelines, which is essential in applications where human values play crucial role.
A.1 DERIVING PREFERENCE OPTIMIZATION AS SUPERVISED LEARNING FRAMEWORK
In traditional RLHF, RL algorithms like PPO are required to optimize the policy since the reward signal is not differentiable. However, RLHF is often slow, primarily due to the need for sampling
16

---

Published as a conference paper at ICLR 2025
generations, and could be more stable in practice, especially in distributed settings. As result, cent research has focused on designing glosed-form loss functions that maximize the margin between preferred and dispreferred generations. One prominent approach is Direct Preference Optimization, which has gained popularity due to its mathematical equivalence with RLHF.
Direct Preference Optimization (DPO). DPO (Rafailov et al., 2024) leverages offline preference data to directly optimize the policy without relying on reinforcement learning-baae methods like PPO. DPO demonstrates that the optimal solution Eq. (20), T.. satisfies the following equation:
No To(y xr)
(z. y) = Blog
+ 3 log (
(21)
[T)()/
Here, re represents the reward model, Ty is the policy model, and Tief is the reference model. Both Ty and uet are initialized from the same SFT (Supervised Fine-Tuning) model, but only me is further optimized during DPO, while Tef remains unchanged. ((x) is the partition function, and 8 is nyper-parameter that controls the intensity of the reward signal.
The preference probability Po is derived from pairwise comparisons using the Bradley-Tern model. Substituting Eq. (2I) into Eq. (19) yields the following loss function for DPO:
To(Mull)) 3 log (ve (x)
Tee )
To (rulll (z)
dopo To; Tet) = (o log a (3log
(22)
In Eq. denotes the sigmoid function, and D represents the dataset consisting of pairwise preferences. Each preference triplet (I,, Yw: U1) includes a prompt æ; preferred response Yur and a less preferred response y1- This formulation allows DPO to optimize directly on pairwise prefer- ence data, avoiding the instability of RL methods while maintaining mathematical equivalence with traditional RLHF.
Identity Preference Optimization (IPO). As the original DPO loss function has shown limitations
in practice, such overfitting to the preference dataset (Gheshlaghi Azar et al. 2024), IPO was pro- posed as an extension. IPO introduces a regularization term to the DPO loss to mitigate overfitting by controlling the gap between the log- liglelllioood ratios of the preferred and dispreferred outputs for both the model and the reference model. The IPO loss function defined as:
LIoe(Te; Tret) = - -B (e DDD (10g
TT (To (x) [r) me (e)er (z)) To ( [Te (Vwl (x))
3-1 27
(23)
2
By adding this regularization, IPO ensures better generalization of the model, avoiding overfitting specific preference patterns and maintaining stability in performance across different datasets.
Kahneman- Tversyy Optimization (KTO). In language modeling, collecting large-scale prefer- ence datasets can be difficult, but relatively easier determine whether an output is desired or undesired. This challenge has inspired KTO (Ethayarajh et al.) 2024). method that bypasses the need for detailed preference data by using binary signal to guide optimization. KTO is based on prospect theory (Terskky 2016), directly maximizing the utility of generations using model of human decision- -aaiin..
KTO has demonstrated higher data efficiency and better handling of imbalanced datasets compared to DPO. The KTO loss function is defined as:
LxTO (9, Trer) = Ez.y- (w(y) (x yı 9))],
(24)
where (2ro(1, yi B) adjusts based on whether the output is desired or not:
[o(Troo (z, y) 2ref) y ~ Vdosired x ( TxTo v)) Vundesised
Au
y ~ Vundesired |x
'K1o(z,y;)) =
w(y)
AD
y Bdesized jr
Here, Zre represents the reference reward, and rkTO v) 3 log (w [x)(y(=)
The weighting function
w(y) differentiates between desired and undesired outputs, where An and dr are parrmteprs controlling the weighting of each.
17

---

Published as a conference paper at ICLR 2025
KTO's key advantage is its ability focus maximizing the reward for desired outputs without inflating the KL divergence term, effectively balancing utility and regularization. This makes KTO a practical solution for real-world upplications. where collecting detailed preference data is infeasible.
In all three methods- IPO, and KTO-the the policy Ty is directly optimized without the need for reward model. We incorporate these methods into experiments to compare their effectiveness in various aspects, such as reducing bias and improving adversarial robustness and overall model performance.
B PROOFS
For simplicity, in all the sections that follow, we assume T
B.1 PROOF OF LEMMA 3.1
As seen in DPO and IPO.for preference optimization, using a preference dataset he key objective is to optimize based preference dataset by calculating h (Iw,, V1, define in 4 using our defined policy No with the softmax output, we find the following:
Con on)
( (1) (T) (yr x)) Mree TTee (Bwla)))
Liy efolvsi))
2ny exfell (ffi (MLim))
NT (Jhrr Vl, x)) log
log
cToly)
chatt
Ev. ef00101213)
log ())
fa(Uw, - fo(u,, + Fror(nn x)) - f (ro((yw;
-Io(x) (Vhw) - Io()=) To (x) Trer(Ww) + [n Tra(lw))) /r
Io(()) (No)) - Tx())) - [ne((x)" (Trer - (ymr))
(25)
Assuming the text encoder To is frozen, such that To Tedt T, the expression simplifies to: h (yw: Wt x) (Io Ins( (z)) (T T(x)))
(26)
Our loss functions aim to increase lur (yw, M1 x). starting from the initialized value of 0 (since initially set to πT)) With some proximity constraints, as in DPO and IPO:
CDPO (TO, Tyes) E(s.gr-0wwwD I- log (Bhrr (Nhrs yl, x))]
(27)
Luvo (To: Tret) (1 (
( (Bhr 31 xx)
B
(28)
B.2 PROOF OF PROPOSITION 3.4
By rewriting LIPO(TO, Teet) with 21 and Ø2, we obtain:
LIPO(TO, neet) = E Ble ( ( ( 22,
3-1
(W1
2
8-1
27
log
I1
-(
(29)
x2
For (Cpo(111z)) dzg
8 @L1o0(113
B
1
x2
=2
log
82
2
(30)
Ø
8
log
Ø2
2
18

---

Published as a conference paper at ICLR 2025
For
dxg
Ö (111
31
8-1 1
21
12
2
log
dr²
2
x2
(31)
11
-1
2
log
22
2
22
and for the second part we have:
(L1Po(11:22))
@CD
log
x2
21
g-1
2
dx1
2
log
8
(32)
x2
22
J2
37
[in are two probability ratios, where To (y(r)) E [0, and (y)()) [0, 11. Assuming (y)))) is the probability of the fixed reference model, can assume Tyerl the (x) = and (z)(()(a)) = j. where (a, 2 1). In this case, we have I1 E [0, a] and æ2 [0, bj. As the DPO optimization progresses, #1 tends increase and 22 tends decrease. Consequently, To (Mh (x) will be greater than and (3) (x) will be smaller than 5. In other words, this implies that 21
and finally given that 01 xo zo
and 22 (w la))
no [M/]]]))] is greater than 1, 22
x [) less than and therefore 22 21.
Kaal a))
Red (1n at)
B.3 THE GRADIENT OF THE DPO AND IPO OBJECTIVES
The gradients these loss functions highlight how the image encoder In adjusted to align more closely with the desired output:
OLg
].
V Co Tres) - o B (Wor M1
(T(ww) - T(v)))
(33)
de
higher when prosimal reference
(x()) direction of (Vym) T(xe)
[ 1 2 1 (Bar y1: x))
[Ig 00
VoLioo Tres)
(T(ww) T(y)))
(34)
positive when proximal reference
increase
(2())) in T((x)) T(m))
These gradients show that both DPO and IPO adjust the image embedding [g(x)) the direction of T(yw) T(t)), encouraging the model to better align with the preferred outputs.
C FEATURES LEARNED BY OUR METHOD
In Figure 6 we show saliency maps of vanilla CLIP and our fine-tuned model on images with typographic attacks. CLIP focuses on irrelevant parts of images, especially typographic texts, which results in false predictions. On the other hand, our model capable of ignoring the text and focuses on the visual concept of the image, which results in correct prediction.
Further, in Figure 7 we provide the predictions of several models on another example image with typographic attack. While the fine-tuned model remains robust, correctly classifying the attacked image, the original model misclassifies it. The inverse model confidently selects the typographic label but preserves classification accuracy.
These analyses showcase how applying our method forces the model to focus on the suitable features of the data rather than different sorts of attacks such typographic.
D LATENT VISUALIZATION USING VQGAN
In this section, we further investigate our fine-tuned CLIP on typographical attack datasets. We use VOGAN-CLIP (Crowson et al. 2022) as the generative model which uses CLIP in its backbone.
19

---

Published as a conference paper at ICLR 2025
Image
SRBAS
Saliency map (CLIP)
Saliency map (fine-tuned)
Figure 6: Saliency maps of vanilla CLIP and our fine-tuned model given four different images with typographic attacks.
photo dumpling
photo chm chowler
(typo.)
clam
Ieverse (clean)
chowder
Pretriined (typo)
Pretrained (cleum)
Fine aune (typa.)
Fine uundd (clean)
Profubility
(a) Example image showing dumplings af- ter typographic attack with mislcading text ("clam chowder").
(b) Predictions of six models.
Figure 7: Model predictions given an example image with typographic attack.
The results are shown in Figure g
As can seen, our fine-tuned CLIP completely ignores text in its image encoder representation and only focuses on visual concepts which increase its reliability and trustworthiness.
E NEAR ORTHOGONALITY OF W
Previous works such as Materzynska et al. 2022 also utilized a linear projection for their opti- mization objective. However, when these projections stray away from orthogonal projections, our pretrained knowledge might be lost; this is a simple consequence of the following fact:
f(y,x) - I(x) W WT(y) f(y.z)= = I(x) I(x) T())
} f(,,)) - f(, -00 - I))(w))
< <<< - I | W - -]], (35)
where the second equality is result of CLIPs' normalized embeddings, as it is evident from Eq. 35). deviation from the base model is controlled by [WT W - /]}, and ideally, do not want the deviation to be too large. Previous works such as (Materzynska et al. (022) added explicit loss
objectives such R(W)= | W f maintain proximity, and they have demonstrated the
20

---

Published as a conference paper at ICLR 2025
Vive
TME E
CLIP
MALE
FOCUS
LOVV
Time
Tiit
1:90
Ours
Figure 8: Retrieved images using VQGAN-CLIP (Crowson et al. 2022) using the captions "focus", "Love", "Male police" and "Time" for image generation. As be seen, vanilla CLIP puts the texts on the images, but our model fine-tuned on the typographical attack dataset SUN ignores such image texts and generates only visual concepts.
importance of this orthogonality loss. However, our method does not let и and Nee diverge easily. Therefore, expect unitary by default. After training the model using our method, considering that pretrained knowledge has been maintained, we already expect the matrix W to be almost orthogonal. Nonetheless, we quantify this proximity in Figure g
W
Deviation from Ideatity
Singular
1.0
0.000
175
120
0.8
-0.100
125
0.6
00.00
230
200
75
-2.005
03
-0.000
25
0.0
0.015
120
255
304
120
256
304
0.05
0.90
0.95
1.00
3.05
3.10
1.15
1.20
1.25
Singular value
(a) Model trained on achieving typographic attack robustness.
W W
Deviation from Ideatity manix
Singular value distribution
400
120
0.01
0.6
F 2:0
200
0.4
-0.01
02
-0.02
00
0.8 Singolar value
(b) Model trained on reversing gender understanding.
Figure 9: On the orthogonality of W: The left plots show the overall heatmaps of W W, where it is evident that the matrix close to the identity matrix. The middle plots display the residuals, which range between -0.03 and 0.03, much smaller than the diagonal values. On the right, we show the distributions of the singular values, which are concentrated around 1.0, indicating that W close to being orthogonal and near the identity matrix.
21

---

Published as a conference paper at ICLR 2025
The original CLIP model outputs normalized embeddings. Using arbitrary linear projection can change the norm of the embeddings and hurt model performance and robustness. However, in typical training scenario, this is not much of issue since transformation is close to an orthogonal transformation.
|| = x W W> & | T W|| = I W| = | W - I+
We [[z|
W - π|| + |||| = + | - 1| x1
=
Wr
(36)
However, when we amplify the singular values using matrix exponents, as discussed in Section this difference grows exponentially; therefore, we should be careful and apply normalization.
3.4
F DATASETS & BASELINES
While human-curated preference sets are ideal, our methodology can still be effectively applied by designing the pretraining task in a semi-supervised manner. In both of our experiments, we did not rely on publicly available preference sets human-curatee preferences. Instead, we designed them ourselves, tailored to the specific task we wanted to fine-tune on. For instance, in the first experiment, we generated synthetic typographic attacks and used the original and targeted labels as preference labels. For debiasing the model, we used a dataset of images showing individuals of each gender performing various tasks and fine-tuned the model on an auxiliary task of reversing the model's gender understanding, again without curated preferences. This demonstrates that preference optimization can be effectively applied using task sspcciic,,,,, sami-supreiseed strategies.
F.1 DATASETS
Table presents the statistics of the datasets used in this paper. To evaluate the classification
curacy of our method on both original and typographic images (results in Table I]. we consider 9
datasets: ImageNet-1000 (Deng al. 2009). Caltech101 (Fei-Feii et al.] [004], OxfordPets PPakkh et al. 2012). StanfordCars (Krause et al. 2013). Flowers 102 (Nilsback & Zisserman 2008), FGV-
CAirCrafts MMjj et al.) 2013). DTD (Cimpoi et al. 2014). SUN397 (Xiao et al., 2016) and Eu- roSAT (Helber et a (211)). These datasets are not typographic attack datasets, so we follow the procedure (Azuma Matsui 2023) to generate typographic attack images from these datasets.
We also consider three real-world datasets designed for typographic attacks in Table Materzynska (Materzynska et al. 2022]. PAINT (Ilharco et al. 2022), and RTA-100 (Azuma & Matsui] (003).
Figure shows some samples from the RTA- 100 dataset. Additionally, for the Gender Bias results and analysis, we considered the VL-Bias (Zhang et al. 2022) dataset. Also, we consider Food101
[Bossard et al. 2014) for additional experiments, including saliency maps in Section C
Table 2: Classification accuracy real-world typographic datasets.
Method
Materzynska
PAINT
RTA 100
Avg.
CLIP
43.27
50.00
47.20
46.82
Materzynska+
77.78
55.45
$7.60
63.61
PAINT
$3.22
58.18
53.60
55,00
Defense: Prefix
71.93
63.64
58.00
64.52
(DPO)
$7.29
60.35
65.40
Ours (IPO)
77.98
59.38
65.27
67.54
Ours (KTO)
80.36
56.25
62.19
66.27
F.2 BASELINES
In the typographic attack experiments, we used the following works as our baseline.
Materzynska et al. 2022). focus disentangling the representation of written words and visual concepts in CLIP's image encoder. They achieve this through orthogonal projec- tions, creating subspaces that isolate eliminate the model's ability to process text. This
22

---

Published as a conference paper at ICLR 2025
Fuler
iPad
Caste
Vue is Hooo
Tone.
Birrer
indde
hetrs drjer
Figure 10: Samples of the RTA-100 dataset.
Table 3: Datasets statistics.
Number of Classes
Number of Images
Dataset
ImageNet-100 100
100
130,000
Caltech 101
101
9,146
OxfordPets
37
7,393
StanfordCars
196
16,185
Flowers
102
8,189
FGVCAircraft
100
10,000
DTD
47
5,640
SUN397
397
108,754
EuroSAT
10
27, 27,000
Food101
101
101,000
Materzynska
19
171
PAINT
89
110
RTA-100
100
1,000
VI.-Bias
65
24,000
disentanglement helps reduce text artifacts in image generation and offers defense against typographic attacks.
(lharro et al. 2022 introduce PAINT (Patching with Interpolation), a method that lin- early interpolates between the weights of model before and after fine-tuning on a specific patching task (6patch = a8 (1- a))n).... This approach improves accuracy on tasks where the zero-shot CLIP model performs poorly, while preserving performance on tasks like Im- ageNet. PAINT is computationally efficient, as it doesn't 't require additional parameters or retraining from scratch. It also exhibits "broad transfer." " where patching on one task can improve accuracy related tasks, even with different classes.
(Azuma & Matsui 2023) present Defense-Prehh (DP). technique that inserts DP token before a class name in text prompts to make resilient to typographic attacks. This method focuses on modifying the text input to CLIP rather than changing the model itself. DP sig- nificantly enhances accuracy on typographic attack datasets while maintaining the model's zero-shot capabilities. The authors also demonstrate its applicability to downstream tasks like object detection, where it effectively reduces the impact of typographic attacks without requiring additional training.
23

---

Published as a conference paper at ICLR 2025
Table I does not include comparison of our method with other baselines on the ImageNet dataset. This is because the experimental results of other methods were conducted on the ImageNetV2 vali- dation subset, which was not publicly available at the time. Instead, present results subset the ImageNerV2 dataset, referred to as the "matched-frequene" subset (Recht et al. 2019). Table comparison of our method with the pretrained CLIP model under the typographic attack scenario.
different
Table 4: Classification accuracy results the ImageNetV2 dataset.
Method
CLIP
54,21
33.23
Ours (DPO)
$3,80
49,18
Ours (IPO)
47.61
42.75
Ours (KTO)
53.17
50.35
G HYPERPARAMETER SENSITIVITY ANALYSIS
In this section, we analyze the impact of several syper-parameten such f and X. We utilize the FOODIO1 dataset for training and SUN as the zero-shot dataset to assess model generalization. All experiments are conducted for 3 epochs, with learning rate of 2 x 10-5, and coefficient y 0.7 in the Beta Moving Average.
G.1 EFFECT OF 8
The hyper-parameter controls the deviation from the policy in the loss function in Eq. (14). To assess its impact, we experiment with values between 0.01 and 1.5 on both the in-domain and zero- shot datasets.
As shown in Figure I our results indicate that IPO is highly sensitive to B. Additionally, the performance of both DPO and KTO decreases for excessively large values on the zero-shot dataset.
To donain Ancuracy (%)
Zero-Baer Anturacy
110
Spepaape 100
Tyoograste
C00
(a) In-domain accuracy.
(b) Zero-shot accuracy.
Ablation study on the effect of the hyper-parameter 8 classification accuracies for clean and typographic datasets. The results highlight the performance DPO, IPO, and KTO models and their impact on in-domain and zero-shot accuracy.
Figure 11:
G.2 ABLATION STUDY THE REGULARIZATION LOSS LusO AND EFFECT OF
The hyperparameter X serves the weight of the regularization term in our loss function. This regularizer is crucial for maintaining model performance on clean datasets during debiasing and ad- versarial training procedures. As shown in Figure 12] vary À between 0.01 and 2.0 to analyze its impact on both in-domain and zero-shot datasets, highlighting a trade-off. In the zero-shot dataset, excessively large À values result in improved performance the clean dataset but decrease in per- formance on the typographic dataset. Conversely, very small values of (e.g., 0.01) lead to lower performance the clean dataset and higher performance the typographic dataset.
24

---

Published as a conference paper at ICLR 2025
170
(50)
Pedemminn Accurey
Zero sbut
(a) In-domain accuracy.
(b) Zero-shot accuracy.
Ablation study the effect of the hyper- perer-prrmmtter À classification accuracies for clean and typographic datasets. The results highlight the performance DPO, IPO, and KTO models and their impact in-domain and zero-shot accuracy.
Figure 12:
H BETA MOVING AVERAGE VS. EXPONENTIAL MOVING AVERAGE
Fine-tuning enhances -peciicc performance but can degrade model's generalization, particu- larly in zero-shot tasks. Exponential Moving Average (EMA) tends to accelerate this degradation by diminishing the pre-trained model's 's influence over time, whereas Beta Moving Average (BMA) (Bayer et al., 2018) retains more knowledge from the pre-trained model, achieving a better bal- ance between task-specific adaptation and generalization. This balance is essential for tasks like debiasing, adversarial robustness, and zero-shot classification, as evidenced in VLMs for out-of- distribution detection (Shu al. 2023). BMA applies a temporal ensemble approach, weighting each training checkpoint along the fine-tuning trajectory by Beta distribution to compute the final model state.
T
@TB
Σ 1=
a2
8,,
(37)
k= 0*k
where Oe is determined by Beta distribution and controls how much influence each model has in the final ensemble. Unlike EMA, which rapidly diminishes the contribution of earlier models (including the pre-trained model), BMA ensures that 0g remains a significant part of the model's knowledge, promoting better zero-shot generalization.
The weights a are drawn from the Beta distribution, normalized over the training steps:
+ 0.5
0t = Betaaaa 7)
(38)
T +
where y is hyper-parameter that controls the balance between the pre-trained and fine-tuned mod- els. By setting y < 1, we ensure that both the pre-trained and fine-tuned models contribute more heavily to the final averaged model, thus reducing the forgetting problem of EMA.
To make BMA efficient in practice, it can be implemented as a moving average:
(11 0%
@BMA a4 0g
BBMA
(39)
E
a
which allows us to compute the model's temporal ensemble without needing to store all the inter-
mediate checkpoints.
In all of our reported results, we employed the BMA update strategy to enhance zero-shot perfor- mance. The specifics of our training procedure are detailed in Algorithm 2 where: denotes the original image, 2 the perturbed image (e.g., typographic image). the label, and ý the adversarial label (e.g., typographic label). As demonstrated in Table [] compare the performance of BMA and EMA. The hyperparameters and À for each method are the same as those used during their training ImageNet.
25

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of the effect of BMA and EMA optimization strategies on the in-domain dataset (FOOD101) and the zero-shot dataset (SUN).
(a) In-domain accuracy
(b) Zero-shot accuracy
Original
Typographic
Original
Typographic
Method
BMA
EMA
BMA
EMA
Method
BMA
EMA
BMA
EMA
CLIP
78.88
51.43
CLIP
61.00
35.33
DPO
79.20
78.99
76.14
75.49
DPO
61.87
62.26
55.56
54.08
IPO
81.94
80.67
78.12
76.90
IPO
62.67
61.85
59.22
58.22
KTO
81.58
79.99
80.49
80.63
KTO
62.37
61.76
60.18
59.20
Algorithm Preference optimization for contrastive learning with BMA details
Require: Pre-trained CLIP model Tup: Dataset D pPprer Dreg), Regularization coef. Areg learn-
ing rate 1)
1: Initialize BMA model: 08MA detach(Oo)
2: Initialize policy: 0g f- 80
3: for = 1
do
4: Speet, Dreg f {(I, y, U. 0)}
D Get batch of preference / regularization data Compute preference loss using Ugret {(., 7)
5:
(prer <- Lpp (T0,-1 Tret; Dpret) free f (T Trefi breg)
6:
Compute regularization loss using bxeg { x)}
7: hor <- lreg + Areg freg
Total loss
8: Update parameters of policy: Of <- 07- V 07 1ot
9: Calculate a of the current model as in Eq. (38)
Update the BMA model BEMA in Eq. (39
11: end for
Ensure: The final BMA model OPMA
where: x Original image, Typographic image, y Label, ÿ Typographic label
1 SAMPLE EFFICIENCY ANALYSIS
In this section, we examine how efficiently our model learns with different dataset sizes. This analysis helps us understand how well the models perform with less data and how their accuracy changes as the dataset grows (see Figure As expected, based on the foundation in Section 4.2 of Gheshhaggh Azar et al. 2024), when there insufficient training data, DPO tends to overfit, leading to poor performance on both In-domain and Zero-shot datasets, while KTO performs better in this situation.
Specifically, we focus on the impact of the number of typographic instances per image and how this affects both the in-domain and zero-shot generalization performance.
Table 6: Impact of number of typographic instances per image on in-domain (FOOD101) and zero- shot (SUN) performance. Results show accuracy (%) for different settings.
Number
FOOD101 (In-domain)
SUN (Zero-shot)
of Instances
BMA
EMA
BMA
EMA
74.85
75.24
58.27
57.47
3
75.22
75.25
58.83
58.46
5
75.59
75.49
58.18
58.07
10
76.25
75.78
57.88
57.29
26

---

Published as a conference paper at ICLR 2025
PSL Crtssooy ww
Clie
(a) In-domain accuracy.
(b) Zero-shot accuracy.
Figure 13: Ablation study on the effect of dataset size on classification accuracies for clean and typographic datasets. The results highlight the performance of DPO, IPO, and KTO models and their impact on in-domain and zero-shot accuracy.
1.1 IMPACT OF NUMBER TYPOGRAPHIC INSTANCES PER IMAGE
To investigate the effect of varying the number of typographic instances per image, we conducted experiments using 1, 3, 5, and 10 typographic variations for each image on the FOODIO1 dataset (in-domain) and the SUN dataset (zero-shot). We evaluate the effect of varying the number of typographic instances per image on model performance. The results, summarized in Table suggest that increasing the number of instances does not significantly improve accuracy, indicating that this augmentation technique may not be effective for enhancing the model's 's generalization further.
PERFORMANCE COMPARISON OF DPO, IPO, AND KTO
In this section, we provide detailed analysis of the performance of DPO, IPO, and KTO in our framework. The comparison focuses on key aspects such as sample efficiency, sensitivity to the B hyperparameter, and performance across different datasets.
J.1 SAMPLE EFFICIENCY
As discussed in Section I and illustrated in Figure 13 KTO demonstrates superior performance in scenarios where training data is scarce. This result aligns with our expectations, as KTO transforms each preference triplet (I, Yur 31) Pper into two samples: (æ, Bássired) and (æ, Bundesirea). By effec- tively doubling the number of training samples, KTT achieves greater sample efficiency, making particularly beneficial in data-constrained settings.
In contrast, DPO performs the poorest among the three methods when training data is insufficient. This is consistent with the findings of (Gheshlaghi Azar al. 2024), where DPO's tendency to overfit in low-data regimes is highlighted. Overfitting leads to poor generalization, resulting in sub- optimal performance on both in-domain and zero-shol datasets. IPO, while not sample-efficient as KTO, outperforms DPO in such scenarios due to its relatively balanced approach to learning from preference data.
J.2 SENSITIVITY TO 3
Figure 11 highlights the sensitivity of each method to the 3 hyperparameter, which governs the strength of the policy regularization. IPO particularly sensitive to variations in 3.. Large values lead to uniform policy distribution Te. causing the model to assign significant probabilities to all actions. This behavior reduces specificity and harms performance, particularly on zero-shot datasets.
For both DPO and KTO, excessively high 3 values similarly degrade performance. However, КTO exhibits slightly greater robustness compared to DPO in this regard, likely due to its ability to lever- age additional training samples. To achieve reasonable performance with IPO, stronger constraints are required to retain the policy close to the reference model, as noted by (Gheshlaghi Azar et al, 2024). Fine-tuning 3 is therefore crucial for optimizing IPO performance.
27

---

Published as a conference paper at ICLR 2025
J.3 PERFORMANCE ACROSS DIFFERENT DATASETS
Our ablation results in Figure 13 and the reported results in Table 1 indicate that with sufficient training data, all three methods DPO, IPO, and XTO-exhibil comparable performance on both original and adversarial datasets. This convergence in performance is expected. as the final loss function in all three methods is derived from the same main objective (Eq. (3), albeit with different design principles and optimization approaches. However, under specific conditions, their distinct characteristics lead to notable differences, highlighting the unique strengths and limitations of each method.
Nonetheless, specific strengths emerge in certain contexts:
Synthetic Datasets: As shown in Table I KTO outperforms DPO and IPO on synthetic datasets, achieving better average performance both the original and typographic vari- ants.
Real-World Datasets: As shown in Table 2 among the three methods real-world datasets.
IPO achieves the best average performance
J.4 WEIGHT ANALYSIS OF DIFFERENT VARIANTS
Regarding the specific analysis of the weight of different variants according to Eq. (T0) we provide the following explanation:
DPO: The gradient weight is given by:
Wpeer(yus yr; x) = -0 (yow y1, x)),
where (-) is the sigmoid function. As -Bh (Nw. yt, x) increases, the weight decreases, which results in lower emphasis on inputs with higher values of ha as they are already good enough. This deemphasizing prevents the model from straying too far from the base model.
IPO: The gradient weight is defined as:
Wpper (Bhs M0;
heg (2hrs M1,
2,3
Here, Nrg (the yt, x) increases, the weight linearly decreases, becoming
hno (Nws M1, = 23 In effect, this would stop the model from straying further than 23*
ideally optimizing to around hTo (yw: 31, = 29
Other Optimization Schemes: Any other preference optimization scheme that uses the differential reward can be studied. One such case could be ROPO, as introduced by (Liang
et al. 2024b).
We compare the different weighting schemes Figure
Figure 15 illustrates how the function correlates with dataset difficulty, highlighting the relation- ship between model performance and the inherent challenges posed by different datasets.
K EXPERIMENTS ON GRADIENT-BASED ATTACKS
In this experiment, we evaluate the effectiveness our proposed method in enhancing model ro- bustness against gradient- attacks, specifically the Projected Gradient Descent (PGD) attack. PGD a widely used method for generating adversarial examples by iteratively perturbing the input to maximize the model's loss.
The setup is as follows: we use targeted PGD attacks to create an image x', which the model predicts the caption y for instead of yu. where yn is the target caption of the adversary. Our methodology from the typographic attack section remains largely the same, with a few adjustments. Specifically, attack the model in online fashion during training. The training accuracies of the model
28

---

Published as a conference paper at ICLR 2025
2.0
WepolX), =
Wxpo(X), 8 0.5
1.5
Wropo(X), a= 8, /
Wpref
0.5
1/28
0.0
-0.5
Non-Optimal Region
-1.0
-1.0
-0.5
0.0
0.5 Dngly.. x)
1.0
1.5
2.0
Figure 14: Visualization different weighting schemes.
Values of hya yo: x)
10
EuroSat
Airewi
flncers
Pets
Caltech
ImageNitt.co
Dataset
Figure 15: The violin plot represents the distribution of the function in DPO method across differ- ent datasets. From left to right, the datasets progressively become less challenging. By "challenging. we refer to the performance As seen, the dataset the far right corresponds training set, which shows higher and more accurate f values.
the pretrained model on the typographic variation of each dataset.
on clean and perturbed data are illustrated in Figures 16 and 17 Additionally, our results on the validation set of the dataset provided in Table 7
In these experiments, for e = 4/255, we used 700 iterations with batch size of 256, and for 2/255, used 1000 iterations with batch size of 256.
29

---

Published as a conference paper at ICLR 2025
Overall, it is evident that the clean data accuracy remains the same while adversarial robustness is increasing.
60
95
85
80
20
10
70
(a) Adversarially perturbed data accuracy
(b) Clean data accuracy
Figure 16: Results of training on adversarial robustness, CIFAR1O dataset, = 2/255.
Mamning MAplow
25
98
15
5
Stap
Stop
(a) Adversarially perturbed data accuracy
(b) Clean data accuracy
Figure 17: Results of training on adversarial robustness, CIFAR10 dataset, = 4/255..
Dataset
Epsilon (c)
Accuracy on Adversarial Images
Accuracy on Clean Images
CIFAR-10 10
2/255
62.89%
98.44% 97.27%
CIFAR-10 10
4/255
21.88%
Table 7: Evaluation Model Accuracy on CIFAR-10 Dataset for Clean and Adversarially Perturbed Images.
30


Reviews:
[{"review_id": "JPdl09gxSf", "reviewer": "Reviewer_PfoD", "summary": "This paper introduces an alignment method designed for contrastive models, such as CLIP, using aligned and unaligned image-text pairs. In this setup, each image input has a preferred (or aligned) response and an dispreferred (or unaligned) output. The model is trained to differentiate between these two responses using preference optimisation designed as a one-step Markov decision process. Therefore, they use a preference dataset that pairs images with aligned and unaligned responses, and a regularisation dataset containing clean examples to maintain the model's ability to generalise to other downstream tasks. Importantly, they don't fine-tune the full model but train a single linear projection layer on top of the frozen text and image encoders. \n\nTo further control the model's behaviour, the authors modify the singular values of the learned linear transformation. Specifically, they apply a singular value decomposition (SVD) to the weight matrix of this layer and scale all singular values using a scaling parameter $t$. This intervention technique builds on the intuition that the linear transformation transforms the original similarity function between image and text spaces.\n\nThey evaluate the effectiveness of their method in two settings. First, they evaluate its effect on typographic robustness by comparing it against baseline models (incl. standard CLIP, PAINT, Defense-Prefix) across nine datasets (incl. ImageNet, Flowers102, and EuroSAT). The preference dataset is created by adding misleading text to the original images of each dataset. They find that their method performs on par or better than prior methods, with a few exceptions. Despite improving the robustness, some performance gaps between the original and typographic dataset remain; for example, a gap of around 20 % on StanfordCars. Using the intervention technique leveraging the SVD of the linear projection layer, they show that they can modify the trade-off between OCR and object detection performance. In the second setting, they explore the possibility to disentangle gender representations. They train the linear transformation using a dataset of images depicting men and women during activities, and show that by scaling the singular values they can reverse gender-specific representations, including a specific scaling factor where the gender information is effectively neutralised, without significant degradation on the downstream task.", "questions": "- The role of the transformation scaling parameter $t$ in the results in Table 1 remains unclear to me. Is the parameter varied between all settings or kept constant?\n- The scaling range from -2 to 1.2 in Figure 2 has likely been chosen because the performance improves up until that point. But what happens if you scale between e.g. -4 and 4? It would be interesting to see even if performance starts to deteriorate after some threshold.\n- Have you tried training separate linear projection layers for the text and image decoders?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "- This paper appears to be the first paper to apply preference optimisation to contrastive models, and presents an interesting use of SVD to control model behaviour. \n- Optimising robustness and mitigating (gender) biases are of significant interest, especially in high-risk domains. \n- The evaluation results suggest comparable and often better performance than alternative approaches in improving robustness while enabling a (to some degree) interpretable intervention technique. \n- The paper is well written and easy-to-follow.", "weaknesses": "- Despite improving robustness over baseline methods in some datasets, none of the methods consistently outperforms other methods (see Table 1). \n- The baseline methods, PAINT and Defense-prefix, and their differences to the proposed method are not explained in the paper.  \n\nMinor Comments: \n- Line 23: Incomplete sentence „Our experiments We demonstrate“. \n- Line 256: Comma instead of dot used. \n- Line 258: Comma should be a dot, and dot should be a comma. \n- Line 289: „this“ -> „This“ \n- The differences in Table 1 appear to computed inconsistently. While most of the time the differences are computed based on the best alternative method incl. the base model (e.g. OxfordPets), the difference for DTD O is computed with respect to PAINT, whereas CLIP seems to performs better. Overall, I think it would be easier to follow if all differences would be reported relative to the base CLIP model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FiqPArEc7N", "reviewer": "Reviewer_PYYa", "summary": "This paper introduces a method for aligning contrastive learning models (CLIP), with human preferences using preference optimization techniques such as DPO and IPO. By formulating contrastive learning as a one-step MDP and fine-tuning CLIP with these techniques, the authors enhance the model robustness of CLIP against typographic attacks and mitigate biases, particularly around gender. Experimental results show improvement on multiple datasets.", "questions": "Broadly speaking, for a general image-text task, e.g., VQA or retrieval, is there any guidance to design the preferences? The easiest way is following standard RLHF and curating a set with actual human preferences, but could the authors kindly suggest any other auxiliary information we can leverage?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "Originality: This is the first work to improve contrastive learning models through Preference Optimization. The idea of leveraging true labels and typographic labels for preferences, instead of curating a separate preference set from human annotation, is novel and interesting.\n\nClarity: This paper is well-written and has very clear motivations, backgrounds, methods, and experiments. \n\nSignificance: The topic of aligning human preferences in contrastive learning is impactful, as models like CLIP are now used widely, yet many undesirable behaviors such as gender biases still exist.", "weaknesses": "Significance: this paper relies on a preference dataset, which requires heavy annotations and the preference set will be very small compared to the training set of CLIP. Also, the preference would be very task-specific (e.g., typographic or gender), limiting the generalizability of the approach to new, unseen attacks or biases.\n\nQuality: the inclusion of SVD makes it much slower to fine-tune on a larger scale. Also, the experiments focus on controlled, relatively smaller-scale datasets (the largest being ImageNet100), so the effectiveness of the approach is yet to be seen on diverse, complex large-scale datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M9ihN68pDr", "reviewer": "Reviewer_ck2S", "summary": "The paper revisits well-known alignment techniques, such as Direct Preference Optimization (DPO) and Identity Preference Optimization (IPO), in the representation space learned by CLIP. The idea is simple yet effective: reformulate the policy $\\pi$ in DPO and IPO by using the similarity scores between preference texts, $y_w$ (preferred) and $y_l$ (unpreferred), and the given adversarial image $x'$. The authors evaluate the proposed method on typographic attacks and show that it improves the CLIP model’s robustness to these attacks while preserving performance on the original datasets (without typographic attacks). To mitigate the overfitting issue of training large models on small datasets, the authors propose training a linear layer (parameterized by $W$) appended to the visual encoder, with both the pre-trained visual and text encoders frozen. Additionally, the authors propose applying SVD decomposition over $W$ as $W=U\\Sigma^tV$, allowing the alignment magnitude to be controlled by $t\\in\\mathcal{R}$. The authors demonstrate that a learned alignment for gender bias can be effectively controlled by adjusting $t$.", "questions": "The reviewer appreciates the well-organized background of DPO, IPO, and KTO, as well as the new perspective on these methods with CLIP. The paper focuses on providing new insights into preference optimization with CLIP. However, the most significant difference between DPO, IPO, and KTO—i.e., per-example weighting—is not discussed sufficiently in the paper (KTO might require further discussion). Additionally, the lack of a thorough comparison and the absence of clear distinctions in Table 1 together weaken the contribution. Therefore, the reviewer’s main questions are as follows:\n\n1. What are the differences among the proposed method variants? Which types of inputs or datasets are weighted more or preferred for each variant? \n2. Could these variants be unified into a single method that outperforms the other baselines in Table 1?\n\nThe reviewer is open to reconsidering the rating if the authors could address these questions (including those in Weakness section).\n\nSome typos:\n* In Ln. 023, “Our experiments” appears incorrectly inserted.\n* Table 2 lacks a reference. The possible related reference is in Ln. 1073 in the appendix.\n* Several papers in the references are duplicated: Ln. 551 and Ln. 554; Ln. 559 and Ln. 562; Ln. 681 and Ln. 687; Ln. 750 and Ln. 754.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed method is simple yet effective.\n2. The authors provide a new perspective on IPO and DPO concerning the representation space learned by CLIP.\n3. The alignment controllability through $t$ is effective.\n4. The background and motivation are well-organized.", "weaknesses": "1. Clarity needs improvement.\n    * $\\mathcal{L}_{pref}$ in (10) appears without a definition. In Corollary 3.2, it is assumed to be either the DPO loss or IPO loss, while the experiments further include the case of KTO loss.\n    * In (9), $\\mathcal{I}_{ref}$ is frozen and has no trainable parameters, contributing solely to per-example weighting when substituted in (5), (6), and (7). It is recommended to clarify this in advance.\n    * In Fig.1, $\\mathcal{L}_{pref}$ is computed with the given triplet $(y_w, y_l, x’)$, where $x’$ is an adversarial image. The presence of multiple negative text representations, such as $\\tau_1$, $\\tau_2$, and $\\tau_3$, is confusing without specifying either $y_w$ or $y_l$ as text inputs.\n    * The overall loss in (13) is iterated over two different datasets, $D_{pref}$ and $D_{reg}$, simultaneously. Further explanation is needed on how the inputs $(y_w, y_l, x’)\\in D_{pref}$ and $x\\in D_{reg}$ are paired or sampled.\n    * The bottom row (differences) in Table 1 is confusing, and it cannot correctly demonstrate the trade-off between O (Original dataset) and T (Target dataset) for each variant of the proposed method. The reviewer recommends indicating the improvement or degradation alongside each accuracy as $\\color{green}{(+1.0)}$ or $\\color{red}{(-1.0)}$ relative to the base model, i.e., CLIP, for clarity.\n2. Lack of a concrete conclusion over comparisons with baselines. The results in Table 1 deserve more discussion. Examples are listed below.\n    * No method in Table 1 consistently outperforms the others. Is there a large domain gap between different datasets that prevents any method from generalizing well across all of them?\n    * PAINT significantly outperforms the proposed method (including all variants: DPO, IPO, and KTO) on both O and T in the ImageNet* column. Is the constraint of a single trainable linear layer in the proposed method too restrictive?\n    * A comparison between different variants of the proposed method would be valuable. For example, what types of inputs are weighted more in different variants according to (10)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I823MaPQjq", "reviewer": "Reviewer_y7hh", "summary": "This paper introduces Preference Optimization for training the contrastive learning model CLIP, aiming to enhance the model's robustness against typographic attacks and mitigate gender biases. This approach aligns the model with human preferences. Experimental results on datasets such as ImageNet, Caltech101, and OxfordPets demonstrate the effectiveness of this method.", "questions": "1. Could this method be applied to other tasks apart from enhancing robustness against typographic attacks and mitigating gender biases?\n2. Could you provide the ablation study results for components $\\mathcal{L}_{pref}$ and $\\mathcal{L}_{reg}$ in the loss function?\n3. I do not understand the image in the left part of Figure 1. What does the obscured dog in the left part of Figure 1 signify?\n4. What is the role of section 3.4 in your method? Why is fine-tuning the model mentioned in Section 3.4?\n5. Could you provide evidence for your claim “the overall matrix $W^TW$ remains close to the identity matrix” in Section3.4？ \n6. Section 3 does not clearly introduce the method. In the last part of Section 3, you should package and summarize your method to give readers an overall understanding.\n7. In Table 1, although the accuracy on the typographic dataset has increased compared to other methods, the accuracy on the original dataset has generally decreased. Therefore, the method has harmed the pretrained knowledge.\n8. Could you provide an analysis of the different performances of DPO, IPO, and KTO in your method in Section 4?\n9. What is the relationship between Section 4.4.1 and Section 4.1? I am not sure about the role of Section 4.4.1.\n10.  What is the meaning of transformation scaling t in Section 4?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "It is significant to explore aligning non-generative model with human preferences using Preference Optimization.\n\nThis paper is well-motivated.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)