Aligning Visual Contrastive learning models via Preference Optimization

Authors: Amirabbas Afzali, Borna khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr Kazemi Ranjbar, Simon Gottschalk
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: contrastive learning, preference optimization, alignment, reinforcement learning from human feedback, robustness, computer vision

Abstract:
Contrastive learning models have demonstrated impressive abilities to capture semantic similarities by aligning representations in the embedding space. However, their performance can be limited by the quality of the training data and its inherent biases. While Preference Optimization (PO) methods such as Reinforcement  Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) have been applied to align generative models with human preferences, their use in contrastive learning has yet to be explored. This paper introduces a novel method for training contrastive learning models using different PO methods to break down complex concepts. Our method systematically aligns model behavior with desired preferences, enhancing performance on the targeted task. In particular, we focus on enhancing model robustness against typographic attacks and inductive biases, commonly seen in contrastive vision-language models like CLIP. Our experiments demonstrate that models trained using PO outperform standard contrastive learning techniques while retaining their ability to handle adversarial challenges and maintain accuracy on other downstream tasks. This makes our method well-suited for tasks requiring fairness, robustness, and alignment with specific preferences. We evaluate our method for tackling typographic attacks on images and explore its ability to disentangle gender concepts and mitigate gender bias, showcasing the versatility of our approach.

Decision: accept (poster)
Reviewer scores: 8, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1729858195387

thread_type:
official_review

<thread_object>
creator: reviewer_y7hh

<content>
summary:
This paper introduces Preference Optimization for training the contrastive learning model CLIP, aiming to enhance the model's robustness against typographic attacks and mitigate gender biases. This approach aligns the model with human preferences. Experimental results on datasets such as ImageNet, Caltech101, and OxfordPets demonstrate the effectiveness of this method.

strengths:
It is significant to explore aligning non-generative model with human preferences using Preference Optimization.

This paper is well-motivated.

weaknesses:
See questions.

questions:
1. Could this method be applied to other tasks apart from enhancing robustness against typographic attacks and mitigating gender biases?
2. Could you provide the ablation study results for components $\mathcal{L}_{pref}$ and $\mathcal{L}_{reg}$ in the loss function?
3. I do not understand the image in the left part of Figure 1. What does the obscured dog in the left part of Figure 1 signify?
4. What is the role of section 3.4 in your method? Why is fine-tuning the model mentioned in Section 3.4?
5. Could you provide evidence for your claim “the overall matrix $W^TW$ remains close to the identity matrix” in Section3.4？ 
6. Section 3 does not clearly introduce the method. In the last part of Section 3, you should package and summarize your method to give readers an overall understanding.
7. In Table 1, although the accuracy on the typographic dataset has increased compared to other methods, the accuracy on the original dataset has generally decreased. Therefore, the method has harmed the pretrained knowledge.
8. Could you provide an analysis of the different performances of DPO, IPO, and KTO in your method in Section 4?
9. What is the relationship between Section 4.4.1 and Section 4.1? I am not sure about the role of Section 4.4.1.
10.  What is the meaning of transformation scaling t in Section 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730039793377

thread_type:
official_review

<thread_object>
creator: reviewer_ck2s

<content>
summary:
The paper revisits well-known alignment techniques, such as Direct Preference Optimization (DPO) and Identity Preference Optimization (IPO), in the representation space learned by CLIP. The idea is simple yet effective: reformulate the policy $\pi$ in DPO and IPO by using the similarity scores between preference texts, $y_w$ (preferred) and $y_l$ (unpreferred), and the given adversarial image $x'$. The authors evaluate the proposed method on typographic attacks and show that it improves the CLIP model’s robustness to these attacks while preserving performance on the original datasets (without typographic attacks). To mitigate the overfitting issue of training large models on small datasets, the authors propose training a linear layer (parameterized by $W$) appended to the visual encoder, with both the pre-trained visual and text encoders frozen. Additionally, the authors propose applying SVD decomposition over $W$ as $W=U\Sigma^tV$, allowing the alignment magnitude to be controlled by $t\in\mathcal{R}$. The authors demonstrate that a learned alignment for gender bias can be effectively controlled by adjusting $t$.

strengths:
1. The proposed method is simple yet effective.
2. The authors provide a new perspective on IPO and DPO concerning the representation space learned by CLIP.
3. The alignment controllability through $t$ is effective.
4. The background and motivation are well-organized.

weaknesses:
1. Clarity needs improvement.
    * $\mathcal{L}_{pref}$ in (10) appears without a definition. In Corollary 3.2, it is assumed to be either the DPO loss or IPO loss, while the experiments further include the case of KTO loss.
    * In (9), $\mathcal{I}_{ref}$ is frozen and has no trainable parameters, contributing solely to per-example weighting when substituted in (5), (6), and (7). It is recommended to clarify this in advance.
    * In Fig.1, $\mathcal{L}_{pref}$ is computed with the given triplet $(y_w, y_l, x’)$, where $x’$ is an adversarial image. The presence of multiple negative text representations, such as $\tau_1$, $\tau_2$, and $\tau_3$, is confusing without specifying either $y_w$ or $y_l$ as text inputs.
    * The overall loss in (13) is iterated over two different datasets, $D_{pref}$ and $D_{reg}$, simultaneously. Further explanation is needed on how the inputs $(y_w, y_l, x’)\in D_{pref}$ and $x\in D_{reg}$ are paired or sampled.
    * The bottom row (differences) in Table 1 is confusing, and it cannot correctly demonstrate the trade-off between O (Original dataset) and T (Target dataset) for each variant of the proposed method. The reviewer recommends indicating the improvement or degradation alongside each accuracy as $\color{green}{(+1.0)}$ or $\color{red}{(-1.0)}$ relative to the base model, i.e., CLIP, for clarity.
2. Lack of a concrete conclusion over comparisons with baselines. The results in Table 1 deserve more discussion. Examples are listed below.
    * No method in Table 1 consistently outperforms the others. Is there a large domain gap between different datasets that prevents any method from generalizing well across all of them?
    * PAINT significantly outperforms the proposed method (including all variants: DPO, IPO, and KTO) on both O and T in the ImageNet* column. Is the constraint of a single trainable linear layer in the proposed method too restrictive?
    * A comparison between different variants of the proposed method would be valuable. For example, what types of inputs are weighted more in different variants according to (10)?

questions:
The reviewer appreciates the well-organized background of DPO, IPO, and KTO, as well as the new perspective on these methods with CLIP. The paper focuses on providing new insights into preference optimization with CLIP. However, the most significant difference between DPO, IPO, and KTO—i.e., per-example weighting—is not discussed sufficiently in the paper (KTO might require further discussion). Additionally, the lack of a thorough comparison and the absence of clear distinctions in Table 1 together weaken the contribution. Therefore, the reviewer’s main questions are as follows:

1. What are the differences among the proposed method variants? Which types of inputs or datasets are weighted more or preferred for each variant? 
2. Could these variants be unified into a single method that outperforms the other baselines in Table 1?

The reviewer is open to reconsidering the rating if the authors could address these questions (including those in Weakness section).

Some typos:
* In Ln. 023, “Our experiments” appears incorrectly inserted.
* Table 2 lacks a reference. The possible related reference is in Ln. 1073 in the appendix.
* Several papers in the references are duplicated: Ln. 551 and Ln. 554; Ln. 559 and Ln. 562; Ln. 681 and Ln. 687; Ln. 750 and Ln. 754.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690160271

thread_type:
official_review

<thread_object>
creator: reviewer_pyya

<content>
summary:
This paper introduces a method for aligning contrastive learning models (CLIP), with human preferences using preference optimization techniques such as DPO and IPO. By formulating contrastive learning as a one-step MDP and fine-tuning CLIP with these techniques, the authors enhance the model robustness of CLIP against typographic attacks and mitigate biases, particularly around gender. Experimental results show improvement on multiple datasets.

strengths:
Originality: This is the first work to improve contrastive learning models through Preference Optimization. The idea of leveraging true labels and typographic labels for preferences, instead of curating a separate preference set from human annotation, is novel and interesting.

Clarity: This paper is well-written and has very clear motivations, backgrounds, methods, and experiments. 

Significance: The topic of aligning human preferences in contrastive learning is impactful, as models like CLIP are now used widely, yet many undesirable behaviors such as gender biases still exist.

weaknesses:
Significance: this paper relies on a preference dataset, which requires heavy annotations and the preference set will be very small compared to the training set of CLIP. Also, the preference would be very task-specific (e.g., typographic or gender), limiting the generalizability of the approach to new, unseen attacks or biases.

Quality: the inclusion of SVD makes it much slower to fine-tune on a larger scale. Also, the experiments focus on controlled, relatively smaller-scale datasets (the largest being ImageNet100), so the effectiveness of the approach is yet to be seen on diverse, complex large-scale datasets.

questions:
Broadly speaking, for a general image-text task, e.g., VQA or retrieval, is there any guidance to design the preferences? The easiest way is following standard RLHF and curating a set with actual human preferences, but could the authors kindly suggest any other auxiliary information we can leverage?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718305065

thread_type:
official_review

<thread_object>
creator: reviewer_pfod

<content>
summary:
This paper introduces an alignment method designed for contrastive models, such as CLIP, using aligned and unaligned image-text pairs. In this setup, each image input has a preferred (or aligned) response and an dispreferred (or unaligned) output. The model is trained to differentiate between these two responses using preference optimisation designed as a one-step Markov decision process. Therefore, they use a preference dataset that pairs images with aligned and unaligned responses, and a regularisation dataset containing clean examples to maintain the model's ability to generalise to other downstream tasks. Importantly, they don't fine-tune the full model but train a single linear projection layer on top of the frozen text and image encoders. 

To further control the model's behaviour, the authors modify the singular values of the learned linear transformation. Specifically, they apply a singular value decomposition (SVD) to the weight matrix of this layer and scale all singular values using a scaling parameter $t$. This intervention technique builds on the intuition that the linear transformation transforms the original similarity function between image and text spaces.

They evaluate the effectiveness of their method in two settings. First, they evaluate its effect on typographic robustness by comparing it against baseline models (incl. standard CLIP, PAINT, Defense-Prefix) across nine datasets (incl. ImageNet, Flowers102, and EuroSAT). The preference dataset is created by adding misleading text to the original images of each dataset. They find that their method performs on par or better than prior methods, with a few exceptions. Despite improving the robustness, some performance gaps between the original and typographic dataset remain; for example, a gap of around 20 % on StanfordCars. Using the intervention technique leveraging the SVD of the linear projection layer, they show that they can modify the trade-off between OCR and object detection performance. In the second setting, they explore the possibility to disentangle gender representations. They train the linear transformation using a dataset of images depicting men and women during activities, and show that by scaling the singular values they can reverse gender-specific representations, including a specific scaling factor where the gender information is effectively neutralised, without significant degradation on the downstream task.

strengths:
- This paper appears to be the first paper to apply preference optimisation to contrastive models, and presents an interesting use of SVD to control model behaviour. 
- Optimising robustness and mitigating (gender) biases are of significant interest, especially in high-risk domains. 
- The evaluation results suggest comparable and often better performance than alternative approaches in improving robustness while enabling a (to some degree) interpretable intervention technique. 
- The paper is well written and easy-to-follow.

weaknesses:
- Despite improving robustness over baseline methods in some datasets, none of the methods consistently outperforms other methods (see Table 1). 
- The baseline methods, PAINT and Defense-prefix, and their differences to the proposed method are not explained in the paper.  

Minor Comments: 
- Line 23: Incomplete sentence „Our experiments We demonstrate“. 
- Line 256: Comma instead of dot used. 
- Line 258: Comma should be a dot, and dot should be a comma. 
- Line 289: „this“ -> „This“ 
- The differences in Table 1 appear to computed inconsistently. While most of the time the differences are computed based on the best alternative method incl. the base model (e.g. OxfordPets), the difference for DTD O is computed with respect to PAINT, whereas CLIP seems to performs better. Overall, I think it would be easier to follow if all differences would be reported relative to the base CLIP model.

questions:
- The role of the transformation scaling parameter $t$ in the results in Table 1 remains unclear to me. Is the parameter varied between all settings or kept constant?
- The scaling range from -2 to 1.2 in Figure 2 has likely been chosen because the performance improves up until that point. But what happens if you scale between e.g. -4 and 4? It would be interesting to see even if performance starts to deteriorate after some threshold.
- Have you tried training separate linear projection layers for the text and image decoders?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732529021345

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all reviewers

comment:
We sincerely thank all reviewers for their careful reading and valuable comments. Here we address some common questions that are of interest to multiple reviewers.

The experimental results of our baseline methods were conducted by [1] on the ImageNetV2 validation subset, which was no longer publicly available at the time of our experiments. Instead, we evaluated our method on a different subset of the ImageNetV2 dataset, referred to as the "matched-frequency" subset. Table 4 provides a detailed comparison of our method with the pretrained CLIP model under the typographic attack on the ImageNet dataset.

Additionally, we have identified and addressed an issue in our presentation of Table 1. The KTO method significantly outperforms all baselines on the typographic dataset while performing nearly identically to the state-of-the-art (SOTA) method on the original dataset. We have revised Table 1 to make these distinctions clearer and provide a more accurate portrayal of the performance of our proposed methods across different datasets.

[1] Hiroki Azuma and Yusuke Matsui. Defense-prefix for preventing typographic attacks on clip. In Proceedings of the IEEE/CVF International Conference on Computer Vision
</content>
</thread_object>

</thread>
<thread>
creation date:
1733307825635

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Rebuttal Period

comment:
We sincerely thank all reviewers for their careful reading and valuable comments. During the rebuttal period, we addressed all typos, grammatical errors, and writing mistakes in the manuscript. We also carefully responded to reviewers' concerns and incorporated their feedback to improve the clarity and quality of the paper. Specifically, we revised several sections to resolve misunderstandings, ensuring the text is more accessible to a broader audience.

We improved the presentation of our main results in Table 1, highlighting the significant performance gains of our method compared to baseline approaches. The revised table now provides a more intuitive and clear report of our findings. Additionally, as suggested by reviewers, we expanded the explanations of baseline methods in Appendix F, ensuring readers have sufficient context for comparison.
Figure 2 was updated by increasing the scaling range for improved visualization. We also added a detailed explanation of this adjustment in the main text, complemented by further analysis in Appendix E.

We refined the definition of $\mathcal{L}_{\text{pref}}$ based on reviewer suggestions, ensuring it is precise and easy to understand. To strengthen the evaluation of our method, we included a detailed comparison of different variants of our proposed approach, which is now presented in Appendix J.

Additionally, we emphasized the comparative study design of our work, particularly highlighting the superior performance of Kahneman-Tversky Optimization (KTO). Updates were made to the abstract, contributions, and conclusion to better reflect this focus. To streamline the main text, some references were moved to the appendix as footnotes.

One reviewer (y7hh) raised an insightful question about the potential applicability of our method to tasks beyond enhancing robustness against typographic attacks and mitigating gender biases.

Inspired by this question, we conducted an additional experiment to test the versatility of our framework. Specifically, we evaluated our method's effectiveness in enhancing the robustness of the CLIP model against adversarial perturbations, such as PGD attacks. The results of this new experiment, along with implementation details, are provided in Appendix K, demonstrating the broader applicability of our framework. Unfortunately, we have not received further feedback from the reviewer regarding this addition.

Best, Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers largely appreciated the paper's novel approach of applying Preference Optimization (PO) to enhance the CLIP contrastive learning model's robustness and reduce biases. Both Direct Preference Optimization (DPO) and Identity Preference Optimization (IPO) show promise in mitigating typographic attacks and bias. Evaluations on various datasets demonstrate comparable or superior results compared to existing methods. The method's clarity and novelty were widely praised, with some reviewers suggesting further elaboration on specific experiments and results. On the other hand, the manuscript would benefit from a clearer explanation of the trade-offs between methods like DPO, IPO, and KTO, as well as a more thorough discussion of their respective effectiveness in different settings. Furthermore, while the approach was seen as promising, several reviewers noted that the experiments primarily involved smaller datasets, which might not fully demonstrate the method's scalability to more complex, real-world tasks. Despite these critiques, the general consensus remained positive, with the paper seen as a valuable contribution to improving contrastive learning models. We ask the authors to incorporate the feedback and additional results from the discussion phase.