Finetuning CLIP to Reason about Pairwise Differences

Authors: Dylan Sam, Devin Willmott, João D. Semedo, J Zico Kolter
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: CLIP, finetuning, vision language models

Abstract:
Vision-language models (VLMs) such as CLIP are trained via contrastive learning between text and image pairs, resulting in aligned image and text embeddings that are useful for many downstream tasks. A notable drawback of CLIP, however, is that the resulting embedding space seems to lack some of the structure of their purely text-based alternatives. For instance, while text embeddings have been long noted to satisfy analogies in embedding space using vector arithmetic, CLIP has no such property. In this paper, we propose an approach to natively train CLIP in a contrastive manner to reason about differences in embedding space. We finetune CLIP so that the differences in image embedding space correspond to text descriptions of the image differences, which we synthetically generate with large language models on image-caption paired datasets. We first demonstrate that our approach yields significantly improved capabilities in ranking images by a certain attribute (e.g., elephants are larger than cats), which is useful in retrieval or constructing attribute-based classifiers, and improved zeroshot classification performance on many downstream image classification tasks. In addition, our approach enables a new mechanism for inference that we refer to as comparative prompting, where we leverage prior knowledge of text descriptions of differences between classes of interest, achieving even larger performance gains in classification. Finally, we illustrate that the resulting embeddings obey a larger degree of geometric properties in embedding space, such as in text-to-image generation.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730057064841

thread_type:
official_review

<thread_object>
creator: reviewer_i5go

<content>
summary:
This paper proposes PC-CLIP, a method to finetune vision-language models to better reason about pairwise differences between images by aligning embedding differences with LLM-generated descriptions of visual differences. The approach improves CLIP's performance on difference-based classification from almost random while maintaining zero-shot classification performance. The quality of the embeddings is also evaluated through text-to-image generation.

strengths:
- The paper is well-structured and clearly written, explaining the methodology and results effectively. The experiments are comprehensive, and I also find the methodology for evaluating the quality of the embeddings through text-to-image generation very interesting and insightful.
- The method is novel, and the results of the difference-based classification look promising. The method gains significant performance from the almost random performance of vanilla CLIP. This capability can potentially benefit downstream models.

weaknesses:
See Questions.

questions:
- The method requires the LLM to generate the differences between two captions for both training and performing difference-based classification. However, the experiment only covers LLaMA2-13B-chat-hf as the LLM. I wonder if the authors have any insight into whether the training is based on LLaMA2 but the inference is using any smaller/larger models. Would the change in the distribution of the output impact the performance of the model?
- I'm curious if this approach could be combined with other CLIP improvement techniques. For example, NegCLIP [1] improves compositional reasoning by focusing more on the text encoder. This method feels like a nice complement to it.

[1] When and Why Vision-Language Models Behave like Bags-Of-Words, and What to Do About It?, Yuksekgonul et al., 2023
</content>
</thread_object>

</thread>
<thread>
creation date:
1730172235633

thread_type:
official_review

<thread_object>
creator: reviewer_juc5

<content>
summary:
This paper presents PC-CLIP, an approach to improve CLIP models' ability to reason about differences between images through fine-tuning CLIP with LLM-generated comparisons. The key idea is to align differences in CLIP's image embedding space with natural language descriptions of visual differences between image pairs generated by LLM. The authors demonstrate this enables new capabilities like difference-based classification while maintaining or improving standard zero-shot performance. They also introduce comparative prompting to leverage class-level differences during inference.

strengths:
- New capabilities: The method enables valuable new capabilities like difference-based classification and comparative prompting while maintaining or improving CLIP's core zero-shot performance.

- Comprehensive empirical validation: The work provides extensive experimental validation across multiple tasks, datasets, and evaluation metrics, including classification, embedding analysis, and generation. The baselines are also very reasonable (fine-tune CLIP on the COCO original and rewrite captions).

weaknesses:
- Scaling up: The experiment focuses only on a single CLIP model size (ViT-L/14), it is unclear whether the approach still remains useful when the model is larger, and whether it is more or less useful.

- Limited improvements: While the method shows consistent improvements on standard classification tasks, many of the gains are relatively small (1-2% absolute improvement). It is unclear whether the cost is worth such small improvements.

- LLM dependency: The method's core dependency on LLM-generated comparisons introduces risks from hallucination and noise in the training signal, with limited analysis of how comparison quality impacts downstream performance or how to ensure robust comparison generation.

questions:
- What is the difference between Table 2 and Table 3? The numbers in Table 3 are much lower than in Table 2. Also, the PC-CLIP baseline numbers are much lower than CLIP (e.g., CUB 59.43 vs 52.57). Also, Table 3 is never referenced in the main text.

Also see weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687508727

thread_type:
official_review

<thread_object>
creator: reviewer_sgyp

<content>
summary:
The paper argues that the CLIP text embeddings do not exhibit the structure of purely text-based embeddings. Specifically, the difference between the CLIP text embeddings in the representation space does not encode the semantic difference between two captions in the raw space. To solve this problem, the authors propose to finetune the CLIP model on LLM-generated difference descriptions for the pair of image-text data sampled from a dataset. Post-training, the model can perform difference-based classification and zero-shot classification very well. In addition, the paper’s solution gives rise to the possibility of comparative prompting which can potentially improve the model classification on confused classes (e.g., crab and lobster), and embedding arithmetic with text-to-image generative models.

strengths:
- The paper proposes an effective method to solve the highlighted problem. Specifically, the synthetic data generation using LLMs for any pair of images is reasonable. 
- The paper showcases the utility of their method using difference-based classification, zero-shot evaluation, and evaluating the quality of the learned method.

weaknesses:
- The paper lacks a good motivation on why CLIP models should exhibit the structure of purely-language based text embeddings. The CLIP pretraining never encourages such structure to emerge so it is not unusual to observe such behavior. In particular, the motivation never justifies why difference understanding is a desirable property.
- The paper’s problem does not seem relevant in the context of large multimodal models. Specifically, we have several models such as LLaVA, Qwen-VL that should be able to perform difference-based classification by design. In addition, we would not need to create difference descriptions during inference with these models. 
- The paper is quite detached from the prior work on difference based image captioning [1,2]. It is a well-studied field where the focus is on understanding the fine-grained differences between the two images. In reality, we would need PC-CLIP for such tasks more than difference-based classification. There is no evaluation on such tasks. 
- There is prior work that aims to address a similar problem [3,4]. The paper neither discusses nor compares with any of these methods. One of them is quite old, and warrants some level of comparison. 
- Comparative prompting is not well-grounded. Specifically, how will we know if the f_A is misaligned? Figure 3 expects that f_B is well-aligned but f_A is not, what happens if both f_A and f_B are misaligned? 
- In Table 2, we observe that comparative prompting leads to worse results than non-comparative prompting for CIFAR100, CUB, and almost similar results for SUN397. This suggests that this method may not be effective many times.
- Additionally, comparative prompting requires a user to create difference captions and tune the alpha hyperparameter. It is not clear whether such efforts are worth the gains on completely unseen tasks and scenarios in comparison to a CLIP model which does not require any of these.
- Zero-shot CLIP results are not provided for a more standard dataset such as ImageNet. A part of this is probably due to the cost involved in finding the confusion classes and creating difference prompts for them.
- Apart from zero-shot classification, CLIP model achieved high-level robustness to natural distribution shifts (ImageNetv2/R/A/S). It is unclear how PC-CLIP would perform in such cases. Additionally, there are no linear probing results to understand the quality of the learned vision embeddings. 


[1] Robust Change Captioning: https://arxiv.org/abs/1901.02527 \
[2] Spot the diff: https://arxiv.org/pdf/1808.10584 \
[3] CLIP4IDC: CLIP for Image Difference Captioning (https://arxiv.org/pdf/2206.00629) \
[4] OneDiff: A Generalist Model for Image Difference Captioning (https://arxiv.org/pdf/2407.05645)

questions:
Mentioned in the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730873638146

thread_type:
official_review

<thread_object>
creator: reviewer_i1cp

<content>
summary:
The paper proposes a method to train CLIP image/text encoders such that the difference between CLIP's image embeddings are aligned with the text descriptions of the image differences. The authors further show through empirical results that this results in better performance on difference-based classification while maintaining neutral performance on zero-shot prompting performance.

strengths:
The strengths of the paper are as follows:

* The proposed method is simple and can be easily extended as a main objective during CLIP pretraining (not just as a finetuning objective).

* The proposed method shows significant improvement on tasks involving difference based classification.

weaknesses:
The weaknesses of the paper are:

* The main contribution of the paper is limited. Finetuning CLIP on the proposed objective (aligning g(I_1 ) - g(I_2) with f(T_{1,2})) is a novel contribution, however in itself is not sufficient. Similarly, the empirical studies in this paper are not very impressive. While Table 1 results on difference-based classification are impressive, the rest of the results show very marginal improvements over the baselines.

* There seems to be lot of repetition in the form of summarizing the same set of results/observations over and again. The authors must reduce such repetition and include more details from the appendix, such as the training details.

questions:
* In Sec 4.1, it is mentioned that the 1000 randomly sampled images were paired and a filtering was applied to remove poor quality generations -- are there any ablations without such a filtering? This is important in the context of scaling up this approach to millions of images where we might end up with lower quality generations, so learning how much lower quality generations impact the final performance could be a good indicator of the scalability of this approach.
* In L419, I think the authors meant to say "highly confused in Table 3" instead of "highly confused in Section 4.3".
</content>
</thread_object>

</thread>
<thread>
creation date:
1732230822654

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Response

comment:
We thank the reviewers for their thoughtful and descriptive feedback. We are glad that reviewers found that our finetuning objective is “novel” [i1CP, i5go], allowing for “valuable new capabilities like difference-based classification and comparative prompting” [Juc5], with a “comprehensive empirical evaluation” [Juc5, i5go], which have “interesting and insightful” evaluations of the embedding space.

In response to reviewer comments, we have provided many new experiments in this rebuttal that we believe strengthen the contributions of the paper. 
 

## Image-difference Captioning and Retrieval

As suggested by reviewer SGyP, we have added some new experiments in terms of difference and image-pair retrieval and captioning, following the experimental guidelines in [1]. We observe that when using PC-CLIP instead of CLIP in the pipeline of [1], we observe better performance in terms of both retrieval (i.e., picking the pair of images described by a text difference) and captioning (i.e., describing the difference between images), as well as compared to the other captioning baselines from their paper. 

| Retrieval Model | Recall @ 5 | Retrieval Recall @ 10 | 
| --- | --- | --- | 
| CLIP + IDC | 3.0 | 3.7 | 
| PC-CLIP + IDC | **3.6** | **5.2** | 


| Caption Models | BLEU-4 | METEOR | CIDEr-D | ROUGE-L | 
| --- | --- | --- | --- | --- | 
| IFDC | 8.7 | 11.7 | 37.00 | 29.90 | 
| VACC | 8.1 | 12.5 | 34.5 | 32.10 | 
| CLIP + IDC | 10.61 | **12.82** | 41.17 | 32.96 | 
| PC-CLIP + IDC | **10.96** | **12.82** | **43.09** | **33.24** |

## Natural Distribution Shifts

Furthermore, we have added more results on natural distribution shift benchmarks of ImageNet-a and ImageNet-R, where we again achieve slight performance improvements over the CLIP baseline.

|  | Imagenet-A | Imagenet R |
| --- | --- | --- |
| CLIP | 69.07 | 90.33 |
| PC-CLIP | **69.2** | **90.47** |

## Ablations to Noise in Generations

In response to the request of Reviewer we performed an additional ablation where we finetune PC-CLIP on the full unfiltered set of data. We observe the following results (where the best-performing method is bolded and the second-best-performing method is italicized):

| Zeroshot Dataset  | CIFAR100 | CUB | EuroSAT | Flowers | SUN |
| --- | --- | --- | --- | --- | --- |
| CLIP | 85.59 | **81.72** | 54.96 | 81.51 | 72.46 |
| PC-CLIP Unfiltered | *85.81* | *80.46* | **58.81** | *81.57* | *73.11* |
| PC-CLIP | **86.12**   | 80.08  | *57.15* | **81.95** | **73.58** |

| Difference-based Dataset | AwA2 | CIFAR100 Size | CUB | Flowers Color |  |
| --- | --- | --- | --- | --- | --- |
| CLIP | 51.74 ± 1.34   | 54.92 ± 1.11  | 53.32 ± 0.22 | 52.97 ± 2.12 | 
| PC-CLIP Unfiltered | *57.70 $\pm$ 0.37* | *65.40 $\pm$ 1.24* | **64.92 $\pm$ 0.16** | *67.08 $\pm$ 2.03* | 
| PC-CLIP | **58.52 ± 0.46** |  **67.44 ± 1.29**  | *64.91 ± 0.21* |  **67.55 ± 2.11** | 

We observe that while PC-CLIP with filtering achieves the strongest performance on a majority of tasks, PC-CLIP Unfiltered outperforms vanilla CLIP weights on 4 of the 5 tasks, and even outperforms PC-CLIP with filtering on one task. This supports that our finetuning method is robust to noise in the LLM generations.

## PC-CLIP at Larger CLIP Model Scale

As requested by Reviewer Juc5, we provide an experiment where we train a ViT-H/14 model and show the results, compared to standard CLIP features with this model. We observe the following results:
|  | CIFAR100 | CUB | EuroSAT | Flowers | SUN |
| --- | --- | --- | --- | --- | --- |
| CLIP (ViT-H) | 87.60 | 86.42 | 53.56 | **89.06** | 75.64 |
| PC-CLIP (ViT-H) | **87.85** | **86.85** | **53.93** | 88.91 | **75.68** |

This demonstrates that PC-CLIP is indeed effective at larger CLIP model scales and can scale up.


We thank the reviewers again for their time and effort in providing detailed reviews. We have uploaded our revision, with changes highlighted in red. We now address other reviewer comments in their individual threads below. 


[1] CLIP4IDC: CLIP for Image Difference Captioning
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies how can improve the CLIP image and text embeddings on the pairwise difference reasoning task. Concretely, the author observed that there is King - man + woman = Queen capability in the text embedding model, while such capability is missing in the CLIP embedding. To encourage such capability, the author proposed to use LLM to explicitly output the difference b/w two captions in text. Then CLIP is used to encode the LLM output and try to match it with the difference of image embeddings of the corresponding two images.

Strength:
1. The paper is easy to follow
2. The idea of pairwise difference encoding is novel and interesting.

Weakness:
1. The author didn't clearly justify the reason of why such capability is needed for CLIP embedding. Although the author argues the potential benefits for image generation and saw improvement on the CLIPScore, the CLIPScore is not very reliable in capturing detail description difference. The difference in Figure 4 is quite limited. There also seems no side by side human preference comparison to justify the effectiveness of PC-CLIP text embedding model in the image generation task.
2. Scaling up the size of CLIP achieves limited improvement.
3. Comparative prompting seems not be well grounded and not always improving performance. The significance of comparative prompting is limited.

Given all the weakness and the reviewers rating, I would recommend reject.