Semi-Supervised CLIP Adaptation by Enforcing Semantic and Trapezoidal Consistency

Authors: Kai Gan, Bo Ye, Min-Ling Zhang, Tong Wei
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Semi-supervised learning, Vision-language pre-training

Abstract:
Vision-language pre-training models, such as CLIP, have demonstrated strong capability in rapidly adapting to downstream tasks through fine-tuning, and have been widely applied across various tasks. However, when the downstream tasks are constrained by limited image-text paired data, CLIP struggles to effectively address the domain gap between the pre-training and the target tasks. To address this limitation, we propose a novel semi-supervised CLIP training method coined SemiCLIP that leverages a small amount of image-text pairs alongside a large volume of images without text descriptions to enhance CLIP’s cross-modal alignment. To effectively utilize unlabeled images, we introduce semantic concept mining to improve task-specific visual representations by matching images with relevant concepts mined from labeled data. Leveraging matched semantic concepts, we construct learnable surrogate captions for unlabeled images and optimize a trapezoidal consistency to regulate the geometric structure of image-text pairs in the representation space. Experimental results demonstrate that our approach significantly improves the adaptability of CLIP in target tasks with limited labeled data, achieving gains ranging from 1.72\% -- 6.58\% for zero-shot classification accuracy and 2.32\% -- 3.23\% for image-text retrieval performance on standard benchmarks. The source code is available at https://github.com/Gank0078/SemiCLIP.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729064844870

thread_type:
official_review

<thread_object>
creator: reviewer_jqns

<content>
summary:
The paper proposes a semi-supervised CLIP training method, termed SemiCLIP, which leverages a large amount of unlabeled images when only limited image-text pairs are available. SemiCLIP consists of two training stages: supervised pre-training and semi-supervised fine-tuning. In the supervised pre-training stage, the paper introduces a concept classifier along with the standard contrastive loss to enhance image-text alignment using labeled data. The semi-supervised fine-tuning stage includes two key components: concept-level semantic consistency, which ensures that the model maintains consistency in understanding visual concepts, and caption-level trapezoidal consistency, designed to improve cross-modal alignment by refining the geometric structure between image-text pairs. Experimental results show promising improvements over existing baselines across diverse domains.

strengths:
- The paper is well written and easy to follow.
- Experiments are comprehensive.
- Results show consistent improvements over baselines across diverse domains.

weaknesses:
The motivation behind caption-level trapezoidal consistency (Section 3.3.2) is not sufficiently explained. 

- The surrogate caption is generated by concatenation of templated keywords. Even though the templates are learnable, the distribution of generated captions is likely to differ significantly from the labeled captions. It would be helpful if the underlying motivation for this choice of surrogate caption generation process is explained in detail. (One simple alternative might be generating surrogate captions based on keywords using a large language model.)
- The main objective of trapezoidal consistency loss is to effectively utilize less-than-ideal surrogate captions due to their inexact nature. However, it is unclear why diagonal and legs regularization is beneficial, whereas the direct usage of contrastive loss is not, given the same inexact caption. It would be useful to explain why trapezoidal consistency is better suited to handle noisy captions. (A simpler alternative could involve using a soft-label in the contrastive loss, specifically for surrogate captions.)

I look forward to authors’ clarification and am willing to increase the score based on their clarification.

questions:
- Effect of linear classifier: Semantic concept classifier could be performed by CLIP text encoder, akin to zero-shot classification. Is there a specific reason for using linear classifier head instead?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729244280046

thread_type:
official_review

<thread_object>
creator: reviewer_psom

<content>
summary:
In this manuscript, the authors focus on efficiently mitigating the domain gap in semi-supervised learning of CLIP on downstream tasks. Specifically, the authors first leverage labeled images to extract the concept list (used as pseudo labels) of the target dataset, and employ CLIP loss and classification loss on pseudo labels for supervised learning. After initialization on labeled set, the authors propose concept-level semantic consistency and caption-level trapezoidal consistency to optimize CLIP on the whole dataset. Experimental results show that the proposed method can effectively

strengths:
1. The correlation between images and concepts is intuitive, which can be effectively used to optimize CLIP in semi-supervised learning scenarios. 

2. The experimental results are promising. 

3. The writing is polished and easy to understand.

weaknesses:
1. My main concern is the analysis of prompting strategy of \hat{T}_{j}. The position of prompts, the initialization of prompts, or even using fixed prompts rather than learnable prompts can be analyzed. The authors can conduct empirical analysis to investigate which prompting strategy is better. For example, comparing performance with prompts at different positions (e.g. beginning, middle (similar to current configuration), and end of all concepts), different initialization (zero init or normal init for prompt), and fixed vs learnable prompts. All these factors may affect the quality of pseudo captions for unlabeled images. 

2. The quality of pseudo labels (i.e., mined semantic concepts) may be critical for semi-supervised learning. The authors could conduct some analysis (e.g., mining concepts with different granularities) to investigate the effect from quality of pseudo labels. For example, "your current configuration" vs "mined concepts but unified into some common objects or attributes" vs "oracle fine-grained concepts (e.g., mining concepts from the whole original dataset)".

3. The authors can also conduct ablation study regarding different percentages of labeled data (e.g. 1%, 5%, 10%, 25%, 50% labeled data) to evaluate the robustness of proposed training framework. Intuitively, a good semi-supervised learning framework may still robust against low percentage labeled data. Nevertheless, in low percentage labeled data settings, the lack of ground-truth caption may restrict the quality of mined concepts.

questions:
For Table 1~4, we recommend  the authors to show the upper bound (supervised learning on all data) to reveal the gap between semi-supervised learning and fully-supervised learning.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730640686193

thread_type:
official_review

<thread_object>
creator: reviewer_52fh

<content>
summary:
This paper presents a new semi-supervised training method for vision-language pre-training models like CLIP, called SemiCLIP. The method is designed to improve CLIP's adaptability to downstream tasks when there's limited image-text paired data. SemiCLIP uses a small amount of image-text pairs and a large volume of images without text descriptions to enhance cross-modal alignment. It introduces semantic concept mining to improve visual representations by matching images with relevant concepts from labeled data. The method also creates learnable surrogate captions for unlabeled images and optimizes a trapezoidal consistency to regulate the geometric structure of image-text pairs. The experiments show that SemiCLIP significantly improves CLIP's adaptability, increasing zero-shot classification accuracy by 1.72% - 6.58% and image-text retrieval performance by 2.32% - 3.23% on standard benchmarks.

strengths:
+ The paper proposes a new method to leverage unlabeled images alongside limited labeled data, enhancing CLIP's cross-modal alignment.

+ Authors design a caption-level trapezoidal consistency loss to appropriately aggregate mined concepts, which is new to me.

weaknesses:
- The method used to mine semantice concepts had been widely used in semi-supervised works thus is not quite novel to me.
- Why does the distance between I_i and Tˆ_j should be consistent with the distance between I_j and T_i? The reason for the consistency for diagonals constrains is not clear to me.

questions:
1. Since the paper mainly focus on the CLIP downstream adaptation, I suggest authors change the title from "SEMI-SUPERVISED CLIP TRAINING" to "SEMI-SUPERVISED CLIP ADAPTING".

2. More explanation towards trapezoidal distance hypothesis. Why it is necessary to restrict the trapezoid to have equal-length legs and diagonals?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730702619758

thread_type:
official_review

<thread_object>
creator: reviewer_wcrk

<content>
summary:
The paper presents SEMICLIP, a semi-supervised training method that enhances CLIP’s performance with limited number of image-text paired data. It utilizes a small number of labeled pairs along with a large set of unlabeled images by employing semantic concept mining to create pseudo-labels for the unlabeled data. The method introduces trapezoidal consistency regularization to maintain geometric relationships between image-text pairs, optimizing the model's alignment. Experimental results show that SEMICLIP bring improvements on zero-shot classification and image-text retrieval performance on various domains datasets.

strengths:
- The paper is written in a clear manner, making the concepts and methodologies easy to understand. This clarity enhances the overall accessibility of the work, allowing readers to grasp the key ideas without difficulty.
- Extensive experiments to evaluate the effectiveness of the proposed method. The experiments are conducted on 8 classification benchmarks and 6 retrieval benchmarks.

weaknesses:
- The proposed Semantic Concepts Mining method is not novel. Previous work [1] has already used CLIP as a concept labeler to construct pseudo labels for contrastive learning and image captioning. While [1] conducted experiments under the pretraining setting, this paper focuses on small domain datasets.
- Caption-level trapezoidal consistency builds incrementally on CyCLIP [2] in a semi-supervised setting. CyCLIP introduced cross-modal and in-modal consistency. While the paper does provide some comparisons between CyCLIP and SEMICLIP, it primarily emphasizes SEMICLIP focuses on unlabeled images and surrogate captions. However, this change may not be particularly novel.
- The experiments are primarily conducted on specific domain datasets. What about the performance on general benchmarks, such as the retrieval benchmarks Flicker30k and COCO, as well as classification on ImageNet?

[1] Huang, Runhui, et al. "Nlip: Noise-robust language-image pre-training." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.

[2] Goel, Shashank, et al. "Cyclip: Cyclic contrastive language-image pretraining." Advances in Neural Information Processing Systems 35 (2022): 6704-6719.

questions:
Please refer to the weaknesses section to see the exact questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709200431

thread_type:
official_review

<thread_object>
creator: reviewer_zosv

<content>
summary:
-- This paper proposes a new semi-supervised CLIP training method SEMICLIP that adapts CLIP to target tasks using only a small amount of image-text pairs.

-- This paper designs concept-level consistency and caption-level trapezoidal consistency for learning from unlabeled images to enhance visual representations and improve cross-modal alignment, respectively.

-- Extensive experiments demonstrate that the proposed method achieves state-of-the-art results in both zero-shot classification and image-text retrieval tasks.

strengths:
-- SEMICLIP mines candidate semantic concepts from labeled data and learns to associate images with concepts, which may be a new way of using unlabeled images in vision-language pre-training.

-- This paper proposes the trapezoidal consistency to enhance the multi-modal alignment by exploiting the geometric structure of trapezoids in the representation space.

-- This paper uses prompts-driven templates and predicts concepts to construct surrogate captions for unlabeled images.

weaknesses:
-- The proposed method consists of two stages. The finetune stage heavily relies on the concept ming from the pretraining stage. The contribution of each stage is hard to evaluate.

-- The experiments are only conducted on task-specific datasets. What is their performance on common zero-shot classification and multimodal retrieval?

questions:
-- In SEMANTIC CONCEPTS MINING,  the linear classifier is initialized from the concept features of clip text encoder. Do the parameters of this module update during pre-training?

-- It seems this linear classifier is a close-set classifier, why not use the open-set method?

-- Does each concept have its own prompts-driven template [V ]1 [V ]2 [V ]3? Is the prompts-driven template [V ]1 [V ]2 [V ]3 shared for all concepts?

-- It is unclear whether the model is trained from scratch or initialized from pre-trained CLIP.

-- The sizes of datasets in the experiments are unclear.

-- Why do the proposed models suppress the fine-tuned models (CLIP (fine-tuned))? And what is the upper bound for performance?

-- Why is the lower base not used in Figure 1(b)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731913746319

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Manuscript Changes and Looking Forward to Discussing with Reviewers

comment:
We sincerely appreciate all reviewers for their time and effort. The valuable comments have been instrumental in improving our paper. In addition to our responses, we have also made updates to the manuscript to address concerns. The details are as follows: 

(1) For avoid misunderstanding, we provide a clearer explanation of the training of the linear classifier $\psi$ in Section 3.2. 

(2) To illustrate the upper bound of performance, we add the results of fully supervised training to Tables 1-5 and provide the corresponding analysis in Appendix E. 

(3) In order to provide a clear view of the data, we include a description of the dataset size in Appendix F. 

(4) To compare with the differences from Nlip, we cite and compare it in Appendix G. 

(5) In order to make the title more aligned with the paper’s content, we replaced 'training' with 'adaptation'. 

(6) To highlight the role of the prompting strategy, we compare the effects of different prompt positions, initializations, and learnability on performance in Appendix H. 

(7) To evaluate the robustness of SemiCLIP, we conduct ablation study regarding different percentages of labeled data in Appendix I.

(8) To evaluate the performance of SemiCLIP on general benchmarks, we provoide results for COCO datasets in Appendix J.

(9) To emphasize the novelty of trapezoidal consistency, we further elaborate in Appendix C on its differences from CyCLIP and the reasons for its effectiveness.

We are open to any additional questions or feedback reviewers may have. 

Best Regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
(a) Scientific Claims and Findings

The paper introduces SEMICLIP, a semi-supervised training method for CLIP models that enhances performance using limited image-text pairs. SEMICLIP employs semantic concept mining to create pseudo-labels for unlabeled data and introduces trapezoidal consistency regularization to maintain geometric relationships between image-text pairs. The method is evaluated on zero-shot classification and image-text retrieval tasks, showing improvements over existing baselines. Reviewers highlight the method's potential to improve cross-modal alignment and visual representation using unlabeled images.

(b) Strengths

Reviewer zoSv appreciates the novel approach of using semantic concepts and trapezoidal consistency to enhance visual representations and cross-modal alignment. WcrK commends the clear writing and extensive experiments across multiple benchmarks. 52fH notes the innovative use of trapezoidal consistency loss, while Psom highlights the intuitive correlation between images and concepts. JQns finds the paper well-written, with comprehensive experiments showing consistent improvements over baselines.

(c) Weaknesses

The reviewers identify several weaknesses. zoSv points out the difficulty in evaluating the contribution of each training stage and the limited scope of datasets used. WcrK questions the novelty of the semantic concepts mining method and the incremental nature of trapezoidal consistency. 52fH suggests that the method for mining semantic concepts is not novel and questions the rationale behind trapezoidal consistency. Psom raises concerns about the prompting strategy and the quality of pseudo-labels. JQns finds the motivation for trapezoidal consistency insufficiently explained and questions the surrogate caption generation process.

(d) Decision Reasons

On balance, AC agrees with positive points raised by the reviewers which outweigh the negative ones. The decision to accept the paper is based on its approach to semi-supervised training for CLIP models and the promising experimental results. The method's ability to leverage unlabeled images and improve cross-modal alignment is a significant contribution, as highlighted by reviewers zoSv and JQns. While there are concerns about the novelty of certain components and the explanation of trapezoidal consistency, the overall strengths in innovation, experimental validation, and potential impact outweigh these weaknesses. The paper's contributions to enhancing CLIP's adaptability and performance make it a valuable addition to the conference.