Semi-Supervised CLIP Adaptation by Enforcing Semantic and Trapezoidal Consistency
Authors: Kai Gan
Bo Ye
Min-Ling Zhang
Tong Wei
Venue: ICLR 2025 Poster (2025)
Keywords: Semi-supervised learning
Vision-language pre-training
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We propose a method to leverage limit image-text pairs and abundant unlabeled images to improve the vision-language pre-training

Abstract: Vision-language pre-training models, such as CLIP, have demonstrated strong capability in rapidly adapting to downstream tasks through fine-tuning, and have been widely applied across various tasks. However, when the downstream tasks are constrained by limited image-text paired data, CLIP struggles to effectively address the domain gap between the pre-training and the target tasks. To address this limitation, we propose a novel semi-supervised CLIP training method coined SemiCLIP that leverages a small amount of image-text pairs alongside a large volume of images without text descriptions to enhance CLIP’s cross-modal alignment. To effectively utilize unlabeled images, we introduce semantic concept mining to improve task-specific visual representations by matching images with relevant concepts mined from labeled data. Leveraging matched semantic concepts, we construct learnable surrogate captions for unlabeled images and optimize a trapezoidal consistency to regulate the geometric structure of image-text pairs in the representation space. Experimental results demonstrate that our approach significantly improves the adaptability of CLIP in target tasks with limited labeled data, achieving gains ranging from 1.72\% -- 6.58\% for zero-shot classification accuracy and 2.32\% -- 3.23\% for image-text retrieval performance on standard benchmarks. The source code is available at https://github.com/Gank0078/SemiCLIP.

Published as a conference paper at ICLR 2025
SEMI-SUPERVISED CLIP ADAPTATION BY ENFORCING SEMANTIC AND TRAPEZOIDAL CONSISTENCY
Kai Gan Bo Ye Min-Ling Zhang Tong Wei
School of Computer Science and Engineering, Southeast University, Nanjing 210096, China 2 Key Laboratory of Computer Network and Information Integration (Southeast University). Ministry of Education, China {gank, yeb, zhangml, weit weit] edu..c..edd.co cn.
ABSTRACT
Vision-language pre-training models, such CLIP, have demonstrated strong capability in rapidly adapting to downstream tasks through fine-tuning, and have been widely applied across various tasks. However, when the downstream tasks are constrained by limited image-text paired data, CLIP struggles to effectively address the domain gap between the pre-training and the target tasks. To address this limitation, we propose a novel semi-superviin supervisd CLIP adaptation method coined SEMICLIP that leverages a small amount of image-text pairs alongside a large vol- ume of images without text descriptions to enhance CLIP's cross-modal alignment. To effectively utilize unlabeled images, we introduce semantic concept mining improve task-specific visual representations by matching images with relevant concepts mined from labeled data. Leveraging matched semantic concepts, we con- struct learnable surrogate captions for unlabeled images and optimize trapezoidal consistency to regulate the geometric structure of image-text pairs in the repre- sentation space. Experimental results demonstrate that our approach significantly improves the adaptability of CLIP in target tasks with limited labeled data, achiev- ing gains ranging from 1.72% 6.58% for zero-shot classification accuracy and 2.32% 3. 3.23% for image-texs text retrieval performance on standard benchmarks. The source code is available https: /gi thub. com/ Gamm//nnn000 8 Semi CI.IP.
INTRODUCTION
Vision-language pre- pre-training has emerged prominent learning paradigm, especially CLIP (Rad- ford et al., 2021), which has benefited various downstream tasks (Li et al., 2021e; Yao et al., 2021; Gao et al., 2022; Li et al., 2021a; Gan & Wei, 2024; Xia et al., 2024) due its advanced capabiliities in image-text semantic representation. However, when adapting CLIP to specific domains such as re- mote sensing (Arampacha et al., 2021) and medical imaging (Kim al., 2023) with limited image-text pairs, the model fails to effectively mitigate the domain gap, resulting in unsatisfactory performance. Previous research has primarily concentrated on the development of large-scale pre-training datasets for each domain (Johnson et al., 2019; Yang et al., 2020; Schuhmann et al., 2022), necessitating significant human and time resources. Fortunately, in most scenarios, capturing devices can readily images, prompting us to consider whether leveraging
obtain a substantial number of domain- a large volume of unlabeled images can enhance the generalization capabilities of vision-languag pre-training for specific downstream tasks.
Semi-supervised learning (SSL) is common way leverage unlabeled data to improve generalization performance, and remarkable progress has been made in this research field (Sohn et al., 2020; Zhang al., 2021; Ye al., 2024; Chen al., 2023). However, most existing SSL methods are designed for classification tasks and cannot be directly used in lision-language pre-training. Some research has explored the use of additional tools, such as object detectors (Li al., 2021b: Wang et al., 2023a) and large language models (Yang et al., 2023; Mirza et al.. 2024), to improve the multi-modal alignment of the CL.IP model cases where the quantity of image-text pairs is constrained. Recently, S-CL.IP (Mo et al., 2023) proposes generating keyword-level and caption-level pseudo-labels for unlabeled
"Corresponding author

---

Published as a conference paper at ICLR 2025
Image caption:
I D
upper hose
hridge is dark green
T;
"Eight storage tanks are surrounded by green meadows."
Predicted concepts:
legs
legs
diogonals
tanks, storage, lawn, meadows
Image caption:
PW P
"Blue seriously stretchy dark
iower
wash mid-rise jegging."
E
Concept Mining
Predicted concepts:
denim, jeggings, skinny, knee
Learaable Pronges
planes, airport. rursaay,, terninal
Concataatioon
(a)
(b)
Figure 1: (la) Relevant concepts predicted by the proposed semantic concepts mining module for unlabeled images. Notably, the true captions are missing in practice, and are placed in the figure merely as a reference; (lb) Illustration of our proposed trapezoidal consistency regularization. A learnable surrogate caption is constructed for each unlabeled image based on mined concepts.
images to enable semi-supervised CLIP adaptation. The core idea is that the true caption of unlabeled images can be linearly reconstructed from the captions of labeled data within the same mini-batch Although S-CLIP achieves significant performance improvements compared to the original CLIP, it can only leverage the relational information within the mini-batch and still unable to make the most effective of all the labeled data, resulting in unsatisfactory performance in real-world scenarios.
This paper proposes a novel semi-supervised CLIP adaptation method, SEMICLIP. For semi- supervised CLIP training, labeled and unlabeled images should share the same task domain, which suggests that a large number of common concepts can be leveraged between them. Therefore, we propose mining candidate concept database from labeled data, and training classifier to match images and concepts. We also generate pseudo-labels for unlabeled images using the classifier and construct learnable surrogate captions facilitate semi-supervised training. Overall, SEMICLIP follows the "supervised pre-training, semi- cemi-supervised fine-tuning" paradigm for SSL.
During the supervised pre- pre-training, SEMICLIP employs semantic concepts mining module to extract candidate concepts from labeled data and learn to associate images with the concepts via classification loss top of the visual representations. SEMICLIP also optimizes the image-text contrastive loss for labeled data. Once the concepts mining module is trained, we can predict relevant concepts for unlabeled images and use them as pseudo-labels fine-tune the task-specific visual representations. As illustrated in Figure la, the predicted concepts demonstrate a high correlation with images, even though some concepts may not appear in the true image captions.
During the sumi-supereised fine-tuning, we propose concept-level and caption-level consistency regularization to fully exploit unlabeled images. At the concept level, for unlabeled images, optimize the model's predictions for strongly augmented images to remain consistent with those for weakly augmented images, thereby improving the model's generalization. At the caption level, propose trapezoidal consistency to constrain the geometric relationships between image-text pairs in the representation space.
As illustrated in Figure 1b, image-text pairs labeled data form the upper base the trapezoid, while unlabeled images and their corresponding surrogate captions constitute the lower base. The proposed trapezoidal consistency first includes a contrastive loss which seeks to minimize the length (distance) of the upper base (representing accurate image-text pairs), and another two squared losses to restrict the trapezoid to have equal-length legs and diagonals. However, the length of the lower base (representing less accurate pairs) is adaptively adjusted by the model, which allows the model to maintain robust alignment for reliable pairs, while giving flexibility to handle noise in the less reliable pairs, particularly when dealing with unlabeled images. Our main contributions can be summarized as follows:
We propose novel semi--upervised CLIP adaptation method, SEMICLIP, which mines candidate semantic concepts from labeled data and learns to associate images with concepts, providing new way of using unlabeled images in vision-language pre-training.
2

---

Published as a conference paper at ICLR 2025
We propose the trapezoidal consistency, which uses prompts- driven templates and predicted concepts to construct surrogate captions for unlabeled images, and enhances the multi- alignment by exploiting the geometric structure of trapezoids in the representation space.
Extensive experimental analyses show that SEMICLIP achieves state-of-the tff-the-art zero-shot and image-text retrieval performance on multiple benchmarks across various domains, e.g.. SEMICLIP improves S-CLIP by over 3.75% on average in zero-shot classification tasks.
RELATED WORKS
Vision-language pre-training (Radford et al., 2021; Jia et al., 2021; Pham et al., 2023; Huang et al., 2023) have demonstrated remarkable success in various downstream tasks through mee-tuning. Various multiple cross- cross-modality loss functions have been introduced as training objectives, including image-text matching (Chen et al., 2020; Li et al., 2019; Lu et al., 2019; Tan Bansal, 2019), masked language modeling (Dong et al., 2023; Bao et al., 2022; Wang et al., 2023b), masked image modeling (Zhou et al., 2020; Xie et al., 2022), and contrasive loss (I.i et al., 2021a; 2020a;b). CLIP, as one of the most representative works, aims to achieve alignment between image and text representations by learning a joint embedding. CLIP demonstrates remarkable effectiveness in tasks such as zero-shot classification (Radford al., 2021) image-text retrieval (Luo et al., 2022), few-shot learning (Liu et al., 2022), and long-tailed learning (Shi et al., 2024).
Semi-supervised learning aims to improve the generalization performance by leveraging unlabeled data through entropy minimization (Grandvalet & Bengio, 2004), consistency regularization (Sohn et al., 2020), distribution alignment (Kim et al., 2020), and contrastive learning (Lee et al., 2022; Yang et al., 2022). Most of these methods focus on selecting reliable pseudo-labels throughout the training process. In particular, FixMatch (Sohn et al., 2020) selects lseudo-labels with confidence scores greater than a predetermined threshold (c.g., 0.95). However, for SSL problems vision-lant asioonlanguaaee pre-training, the above methods cannot be directly applied due to the unique caption for each image. SEMICLIP mines semantic concepts from image captions in labeled data to construct "pseudo-labele for unlabeled images, and the pseudo-label selection and consistency regularization can be applied.
Training CLIP with limited image-text pairs has gained substantial attention for adapting CLIP to downstream tasks. For example, MedCLIP (Wang et al., 2022) replaces the InfoNCE loss (Oord et al., 2018) with a semantic matching loss grounded in medical knowledge mitigate false negatives in contrastive learning. WFH (Wang et al., 2023a) generates visual hallucinations from textual inputs, which are subsequently paired with originally unpaired texts, facilitating more diverse interactions across modalities. SimVLM (Wang et al., 2021b) is trained with a prefix language modeling objective through exploiting large-scale weak supervision. However, these methods rely on additional information or models, such as class labels, pre-trained object detectors, and large language models. The method most related to our work is S-CLIP (Mo et al., 2023), which assigns pseudo-labels for be recovered by combining image captions in
unlabeled images by assuming that the true captions labeled data. Nevertheless, the performance of S-CLIP is far from satisfactory on large-scale datasets because S-CLIP finds the nearest neighbors of each unlabeled image within the current mini-batch to construct the caption, while the labeled data outside the batch is not effectively utilized.
SEMICLIP: TOWARDS SEMI-SUPERVISED CLIP ADAPTATION
In this section, we present the SEMICLIP approach which consists of three key modules, i.e., semantic concepts mining, concept-level semantic consistency, and caption-level trapezoidal consistency.
3.1 PRELIMINARIES
In general langun-l tasks (Radford al., 2021; Jia et al., 2021), common to assume that there exists abundant paired images and texts D' {[T, T:)) for contrastive learning. However, in realistic scenarios, images can be readily captured. whereas obtaining the corresponding captions entails significant costs. Usually, we can access substantial number of unlabeled images D" = M without captions, where M > N. The most popular vision-language model CLIP (Radford et al., 2021) learns joint embedding space that establishes connection between images I, and texts T, The training objective of CLIP is to maximize the cosine similarity between the embeddings of matching tmate-text pairs while minimizing the similarity between non-matching
3

---

Published as a conference paper at ICLR 2025
pairs. Specifically, CLIP consists of a visual encoder f-(+) and text encoder fr(.), so it can generate embeddings 1 = (I( for images and T = (T,, for texts. Then the embeddings are normalized to the unit hypersphere. Specifically, the CLIP loss takes the following form:
N
exp (1T)/) + log Σ (
exp TT)TT)
LAL 2N
log
(I)
Σ
where (:-) represents the cosine similarity for normalized embeddings and is a temperature scaler which is simply set to I in our experiments. After pre-training, CLIP can perform zero-shot classification by encoding class names as text with appropriate templates (e.g., "a photo fclass name)") and computing the similarities between the test image embedding and each class embedding. CLIP predicts the class associated with the highest image-text similarity.
3.2 SEMANTIC CONCEPTS MINING
During the supervised pre-training, SEMICLIP performs Semantic Concepts Mining (SCM), which consists of mining semantic concepts from the available captions and leveraging labeled data for CLIP adaptation. Despite the limited information contained within the texts, is highly likely that the concepts it encompasses are also present in unlabeled images. Thus, we first employ NLTK (Bird al., 2009) to extract the corresponding nouns from the captions in labeled data. Considering the potential noise associated with certain nouns, we filter nouns to ensure their frequency exceeds threshold of ju, which is set in our experiments. However, some nouns appear excessively often such as the word "Figure" in the SciCap dataset (Hsu et al., 2021) which contains numerous scientific figures. These overly frequent nouns (occurrence rate exceeds 30%) are not beneficial for improving performance in downstream tasks. After extraction, we recognize a total of V noun entities {Co) ) which are regarded concepts exist within the domain corresponding to the task, and for each captioned image 4, can identify its multi-concepts encoding yı, E {0, 1} where positions set to 1 if the image contains the concept. Notably, an image can associate with multiple concepts, hence the label encoding yr, can consist of more than entry of 1s.
In semi-supervised classification tasks, models trained on labeled data can generate pseudo-labels for unlabeled data, where labels are similar to the concepts defined above. We thus inquire: Can the model exploit concepts in captioned images to annotate corresponding concepts in unlabeled images? For this purpose, we train linear classifier v with extracted concepts. Specifically, due to the rich semantic representation capabilities of the CLIP text encoder fr(.), we employ the template (e.g.. udes [Concept] }"") integrating the extracted concepts to generate hand-crafted "a photo includes
descriptions and compute their features 18, I which are then utilized to initialize the weights in th. Subsequently, optimize & through soft cross-entropy loss to match the labeled images with their corresponding concepts as follows:
LSCM
yr, y1.
log (TF)
where 1yn.1 = =VV yr, (v) is utilized to normalize the yr, so that it can conform to the properties
of probabilities. Through minimizing Equation (2), &b can predict concepts for any Nomain-related unlabeled images. this stage, SEMICLIP simultaneously minimizes Equation (1) and Equation (2), and in Figures 2a and 2b, we find that the alignment to concepts can facilitate the model's adaptation to the domain of training data, resulting in improved performance on downstream tasks compared to
S-CLIP in some scenarios. The objective for supervised pre-training is L LCUP + LSCM-
At the end of the training, we assign pseudo labels J1; which assign 1 to the positions with top-k predicted confidence (Sohn et al., 2020) to unlabeled images leveraging & to help understand the semantic information in unlabeled images and achieve further alignment SSL fine-tuning.
3.3 LEARNING FROM UNLABELED IMAGES
To further improve the CLIP model for the target task, leverage unlabeled images and semantic concepts associated with them to conduct semi-super fine-tuning, which includes concept-level semantic consistency and caption-level trapezoidal consistency.
4

---

Published as a conference paper at ICLR 2025
3.3.1 CONCEPT-LEVEL SEMANTIC CONSISTENCY
When we obtain pseudo labels for unlabeled images, an intuitive strategy in SSL is to train & for further alignment between pseudo-concepts and unlabeled images. Popular SSL methods FixMatch (Sohn et al., 2020) and ReMixMatch (Berthelot et al., 2019) align the predictions of strongly augmented images (e.g., applying RandomAugment (Cubuk et al., 2020) to images) with those of weakly augmented images, which partially mitigates confirmation bias (Arazo et al., 2020; Wang et al., 2021a) in SSL tasks and contributes to performance improvement. It is noteworthy that strong data augmentation may hurt the performance (Mu et al., 2022) for vision-language pre-training due to the alterations the color or position of objects within the images, resulting in a mismatch between the images and their corresponding captions. However, since the concepts we consider primarily consist of nouns that denote entities, which are not significantly affected by data augmentation, making them highly suitable for concept-level semantic consistency:
|D"I
UI, 10.
SSMM =
log 2e (fr(((())
(3)
where ((4y) represents the specific strong augmentation scheme for 15. Incorporating concept-level semantic consistency improves the model's robustness to spurious feature patterns (Sohn al., 2020; Wei & Gan, 2023) and augments the model's perceptual ability regarding concepts. We will elucidate the performance benefits of employing strong data augmentation as compared weak data augmentation in Sec. 4.6. It is noteworthy that popular strategies for semi-supervised classification, such as strong data augmentation and confidence filtering (for U1, generation), are utilized within SEMICLIP, bridging the gap between SSL classification and cemi-supervieed CLIP adaptation.
3.3.2 CAPTION-LEVEL TRAPEZOIDAL. CONSISTENCY
The concepts predicted by &b are highly relevant to the unlabeled images, revealing the information of entities contained in the images. However, relying solely on the concepts present in the image is insufficient to effectively summarize all the content contained within the image due to the absence of descriptive attributes for the concepts and their interrelations. Therefore, how to aggregate concepts appropriately constitutes a crucial challenge.
Inspired by the remarkable performance of context optimization (Zhou et al., 2022b;a) for automated prompt engineering, we introduce learnable prompts which can integrate and link corresponding concepts in the images, striving to create meaningful connections between the various concepts. Specifically, we leverage the prompts-driven template concept in the image, where [V] represent learnable vectors that establish connections among various concepts through optimization during training. To expedite the model's 's training adaptation, we employ the token embeddings of photo includes" to initialize [V] Considering the presence of multiple concepts in each image, we concatenate their prompts to construct surrogate caption:
to describe each
i [V ( [v][*) [v]] (V]"")(C..)
We select the top-k concepts (C...) ) extracted from I, with the highest classification confidence (Sohn et al., 2020) in ((T5), where 81.4 represent the indexes for top-k concepts.
Although learnable prompts allow us to generate more accurate surrogate captions for unlabeled images, these captions can be coarse-grained Hence, we adopt caption-level trapezoidal consistency, which includes constraints for the upper base, legs, and diagonals of the trapezoid constructed through the labeled image-text pairs, unlabeled images, and surrogate captions. Specifically, we select a subset of unlabeled images 1 with top-/% cosine similarity between images and their corresponding surrogate captions, denoted as D = (1," M where Mp A P% represents the number N+ and the of selected unlabeled images. Subsequently, define D = DU {T,, N
trapezoidal consistency as follows:
D
Trepp
LCUI
( (TJ 1
(5)
apper base
diagonole
lege
The diagonal constraint ensures consistent similarity across multiple image-text pairs. Simultancously. the equal length of legs constraint ensures that the similarity between images and the similarity between texts are treated equally in the feature space.
5

---

Published as a conference paper at ICLR 2025
To understand why trapezoidal consistency works for surrogate captions. draw inspiration from two aspects:
As illustrated in Figure Ib, the consistency for diagonals constrains the distance between I; and G3 to be consistent with the distance between I, and T; Similarly, The consistency for legs regulates the distance between and 1j to align with the distance between T; and Fÿ-- Equation (1) encourages I and T; to be closer, leading to a corresponding decrease in the length of the upper base. The quadrilateral characterized by vertices 4, 1,. T, and TT will be optimized toward an isosceles trapezoid, which does not need any distance constraints regarding the length of its lower base. This is equivalent to imposing no direct alignment constraint between I; and Tj. thereby mitigating the negative impact of inaccurate T3.
Despite the absence of direct alignment between I and F the trapezoid consistency allows both to interact with samples from within and outside the modality (Goel et al., 2022), ensuring coherence among samples within each modality and substantially enhancing the model's overall alignment ability.
Overall, during semi-supervised fine-tuning, concept-level semantic consistency and caption-level trapezoidal consistency are jointly optimized by minimizing: & LSM Lrap-
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets and Evaluation Metrics. We conduct extensive experiments on four publicly available datasets to evaluate the performance of SEMICLIP. Following previous method S-CLIP (Mo et al., 2023), the datasets include Remote sensing datasets (Yang & Newsam, 2010; Zhang et al., 2014; Lu et al., 2017), Fashion datasets (Han et al., 2017; Rostamzadeh et al., 2018: Vasileva et al., 2018), SciCap dataset (Hsu et al., 2021), and Simpsons dataset (Attia, 2018; Adler, 2023). Under the default setting, we subsample 10% image-text pairs of the training dataset randomly as labeled data, leaving the rest as unlabeled data. The models are evaluated on zero-shot classification and image-text retrieval tasks, with performance measured by Top-1 classification accuracy (%) and recall at (R @ @k). Results are reported as the mean and standard deviation across three random seeds.
Implementation details. We use the publicly available CLIP model (Ilharco et al., 2021) backbone. All experiments are conducted on four NVIDIA A6000 GPUs with a batch size of 64 per GPU. To maintain equitable memory usage in SSL, each mini-batch consists of 32 image-text pairs and 32 unpaired images. We utilize ViT-B-16 as the default vision encoder in our experiments, and experiments related to other vision encoders are shown in Appendix A.I. We train the model 25 epochs in the supervised pre- are-training, and 15 epochs for semi-supervised fine-tuning, We employ AdamW (Loshchilov, 2017) alongside a weight decay set at 5 10-4 and apply the default cosine learning rate scheduling with warmup for the first 10 steps. The learning rate is set to 5 x 10-5 for remote sensing and fashion datasets and 1 x 10 10-6 for SciCap and Simpsons datasets. We establish default values of 30 for P and 4 for k.
Competing Methods. We compare our method with the pre-trained CLIP, denoted CLIP (original), and the CLIP model fine-tuned using only image-text pairs in labeled data, denoted as CLIP (fine- tuned). Oracle (fully supervised fine-tuned) is the supervised learning on all training data. We also compare SEMICLIP with another three semi-supern supesvupe CLIP adaptation approaches, i.c., Hard-PL, Soft-PL, and S-CLIP, More details about these methods can be found in Appendix
4.2 REMOTE SENSING DATASETS
Following the settings in Arampacha et al. (2021), we train vision-language models on a merged dataset, termed RS-ALL, which consists of RSICD (Lu et al., 2017), UCM (Yang & Newsam, 2010), and Sydney (Zhang et al., 2014). Considering the potentially broader domain distribution of unlabeled data, we not only utilize the default 90% of RS-ALL designated as unlabeled data (L=U), but also incorporate the RESISC45 dataset (Cheng et al., 2017) as unlabeled data (L=UU).
Zero-shot classification. We utilize the validation sets from the classification variants of the RSICD and UCM datasets, referred RSICD-CLS and UCM-CLS, respectively. Furthermore,

---

Published as a conference paper at ICLR 2025
Method
Data
RSICD CLS
UCM
WHI
RSSCN7
AID
CLIP (original)
59.2
60.2
81.2
69.0
59.6
CLIP (fine-tune)
75.7
80.1
92.2.11
70.7
79.8
Oracle (fully supervised fine-tuned)
L+ U
87.5+is
67.9e2a
94,2m15
75.7=29
89.3+12
Hard-PL. (Lee et al., 2013) 2013)
78.0=22
72.4 (-7.7)
92.3 (+0.1)
71.7:44 (+1.0)
81.4
Soft-PL (Assran al., 2021)
L=U
81.7-0s (+6.0)
78.9+52
95.3+6a (+3.1)
72.0-15 (+1.3)
85,8-16
S-CLIP (Mo et ul., 2023)
81.4.1a (+5.7)
81.3 (+1.2)
95.9 (+3.7)
75.1 (+4.4)
86.4 (+6.6)
SEMICIIP (ours)
83.8_ (+8.1)
85.6.114 (+5.5)
96.611 (+4.4)
75.3 (+4.6)
87.4gm
Hard-PL (Lee et al., 2013)
79,9-25 (+4.2)
76.2+10 (-3.9)
91 91.0-13 (-1:2)
4144 (40.7)
82.4=20
Soft-PL. (Assman et al., 2021)
LyUU
78. 38 (+2.6)
78.3 (-1.8)
95.3 (+3.1)
73.8 (43.1)
82.7as7 (+2.9)
S-CLIP (Mo 2023)
78.3=31 (+2.6)
79.5+40 (-0.6)
93.8+15 (+1.6)
73.9=50 (43.2)
84.9 (+5.1)
SEMICLIP (ours)
84.1 (+8.4)
85,4+28 (+5.3)
96.5+117 (+4.3)
72.9 (42.2)
86.2-10
Table 1: Zero-shot classification results on remote sensing datasets. We compare the original CLIP, supervised CLIP fine-tuned on labeled data (L), and semi-supervised methods that utilize unlabeled data sampled from the same (L=U) or different (LşU) distribution as the labeled data. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one. The best results are in bold.
Image text R@S
Text >iage R@S
Method
Data
RSICD
UCM
Sydney
RSICD
UCM
Sydney
CLIP (original)
12.6
46.7
44.8
13.9
39.5
44.8
CLIP (fine-tuned)
25.7+
55.2+0s
46.6+3.
24.9
56.3+16
49,4
CLIP (fine-tuned)
L+ U
34.3=113
63.5+11
63.6122
32.0
66.61111
65.4122
Hard-PL. et al., 2013)
27.3-11
54.8+16
52.3436
24.2
54.6210
55.2469
Soft-PL (Assran et al., 2021)
L-U
27.3=09
54.8+22
52.3+22
16.1
55.6+44
56.3+53
S-CLIP (Mo et al., 2023)
27.5=111
$7.0+15
51.1+43
25.6=06
57.3+4.1
50.6+26
SEMICLIP (ours)
1..6
59.2+22
53.4
26.4
58.4+-24
58.0+20
Hard-PL (Lee al., 2013)
25,9=16
$2.7+22
50.6270
22,4
54.3114
51.7452
Soft-PL. (Assran ct al., 2021)
L
27.4=16
54.1+22
51.1#10
24.4207
54.4+24
52.3+53
S-CI.IP (Mo et al., 2023)
27.2=06
55.7230
50.0246
26.1
52.9455 55.7429 26.3+04
SEMICLIP (ours)
28.5
59.5+36
54.6436
59.5+13
58.6+44
Table Image-text retrieval results on remote sensing datasets, following the same setup of Table
assess the generalization capability of SEMICLIP on WHU-RS19 (Xia et al., 2009), RSSCN7 (Zou et al., 2015), and AID (Xia et al., 2017), which are not included in the training datasets. Results in Tab. demonstrate the performance advantages of SEMICLIP. Compared with S-CLIP, SEMICLIP demonstrates average performance gain of 1.72% and 2.94% for L=U and L settings, respectively. LșU represents the scenarios that demand higher robustness of the model against unseen domains and domain shifts, where SEMICLIP achieves more significant performance gains compared to previous methods. Interestingly, our method even outperforms the oracle in some zero-shot settings, and we provide more detailed explanation of this phenomenon in Appendix E.
Image-text retrieval. We evaluate the image-Xtex and text text-ximage retrieval performance on the validation sets of the RSICD, UCM, and Sydney datasets. Tab. 2 presents the R @@5 results, where we can observe that the proposed SEMICLIP surpasses all competing methods across all settings, achieving average 4.48% gain over fine-tuned CLIP. More results can be found in Appendix A.3.
4.3 FASHION DATASETS
We evaluate the models on the union of Fashion200k (Han et al., 2017). FashionGen (Rostamzadeh et al., 2018), and Polyvore Outfits (Vasileva et al., 2018) datasets. Given that each text caption can associate with multiple image views, we randomly select one image during each iteration.
Zero-shot classification. The zero-shot classification performance is evaluated on the classification validation sets of Fashion200k, FashionGen, and Polyvore Outfits. It worth noting that we con- sidered two different class hierarchies, namely super-class and sub-class, in the Fashion200k and FashionGen datasets. The results in Tab. 3 indicate that SEMICLIP significantly outperforms S-CLIP, with an average gain of 12.15% on sub-class in particular. We attribute this to the richness of the concepts extracted by SEMICLIP through semantic concepts mining, and the concepts may encom-
7

---

Published as a conference paper at ICLR 2025
Fashion200k
FashionGen
Polyvore
Method
Super-clar
Super
class
Class
CLIP (original)
73.2
27.7
34,8
26.4
70,2
CLIP (line-tumed)
79.0
35.1
35,4
24.5=24
60.4+223
Oracie (Jully supervised line-tuned)
88.6+113
45,3+44
33.3+25
45.5=73
76.3+13
Hord-Pr. ee al., 2013)
44.9 24.1)
23.9
24.2.19 (111))
18.3 18.3124
(2611)
Soft-PL. (Assran ot al., 2021)
82.5=22 (+3.5)
36.6=14 (+1.5)
44.8=35 (+9.4)
33.6+14 (+9.1)
73.6+17 (+13.2)
S-CLIP (Mo et 2025)
85.1 +00 (+6.1)
38.4-07 (+3.3)
44.0=46 (+8.6)
29.6+48 (45.1)
73.9+14 (+13.5)
SEMICLIP (ours)
85.8+04 (+6.8)
44.7-10
5144 (+16.0)
47.6=17 (+23.1)
74,4 (+14.0)
Table 3: Zero-shot classification results fashion datasets. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one.
Image-tton ttxxt R@5
Text Text-iinage R@5
Method
Fashion200k
FashionGen
Polyvore
Fashion200k
FashionGen
Polyvore
CLIP (original)
12.5
23.5
24.3
12.3
27.7
27.4
CLIP (fine-tuned)
13.7404
32.1
16.3205
13,5292
31.9202
16.2203
Oracle (fully supervised fine-tuned)
34.9:0.7
63.420.
48.5200
34,9=00
35.3200
49.3205
Hard-PL (Lec al., 2013)
12.6+0.7
26.8+0.7
14.9+0a
14,5+as
30.2=0.4
17.3=00
Soft-PL (Assran et al., 2021)
15.2+00.
35.0+003
18.8400
15,9
37.41:07
21.9=02
S-CLIP (Mo et al., 2023)
16.9+04
37.9+04
22.6+04
17,4+00
40.9+004
24.2=005
SEMICLIP (ours)
22.1+:0.4
48.4
27.7:103
22.5
49.7
27.6=002
Table 4: Image-text retrieval results on fashion datasets, following the same setup of Table 3.
pass some fine-grained information. Coupled with subsequent concepts-level semantic consistency, SEMICLIP can significantly improve fine-grat tinengranedd classification performance.
Image-text retrieval. We evaluate the image-text retrieval performance on the validation sets of three fashion datasets. We can observe that SEMICLIP outperforms S-CLIP by an average gain of 6.35% in terms of @@5 in Tab. 4. Compared to results for remote sensing datasets, it is evident that SEMICLIP exhibits a more pronounced advantage, as the Fashion datasets present greater challenges with large-scale and diverse samples. Such settings Methods like S-CLIP, which utilize nearest neighbor strategies for pseudo-labeling, often struggle to find suitable neighbors within single batch.
representative of real-world scenarios.
4.4
MORE DATASETS
We conduct experiments on SciCap (Hsu et al., 2021) and Simpsons (Adler, 2023) datasets to further verify the performance superiority image-text retrieval.
SciCap. The images in SciCap belong to the scientific figures domain, which presents a significant challenge for the model. The results in Tab. 5 indicate that SEMICLIP improve the performance averaged 4.03% compared to fine-tuned CLIP. This further confirms that SEMICLIP can enhance the performance pre-trained models by leveraging substantial amount of unlabeled data, even in challenging domain settings.
Simpsons. Due to the Simpsons dataset only contains limited image-text pairs, we integrate unlabeled images from an additional dataset called Simpsons Characters (Attia, 2018). From Tab. 5, we can conclude that SEMICLIP achieves averaged 6.55% and 6.95% enhancement for R@ I and R@5 respectively compared to S-CLIP. The results demonstrate that even when image-text pairs are extremely scarce, SEMICLIP still maintain a significant performance advantage.
4.5 IN-DEPTH ANALYSIS FOR SEMICLIP
In this section, we conduct extensive studies to demonstrate the effectiveness of SEMICLIP and analysis for the sensitivity of hyperparameters can be found in Appendix A.2.
Extracted concepts. In Figure la, we provide examples illustrating the concepts extracted by the semantic concepts extractor for the unlabeled images. We note that some extracted concepts, like "tanks" and "jeggings", ; are indeed present in the corresponding real captions of the images. However, there are also concepts like "lawn" and "skinny" that do not appear in the captions, but these non-
8

---

Published as a conference paper at ICLR 2025
SciCap
Simpsons
Image Mext
-ext
Text- >image
Image
-ext
Text viaage
Method
@@1
R@5
R@1
R@5
R@1
R005
R@1
R@5
(original)
13.8
14.2
13.2
35.5
32.9
CLIP (fine-tuned)
11.020e
19.0.oa
11.
18.1an0
19.3
48.2
14.5
45.6210
Oracle (fully supervised fine tuned)
14,8209
25.5-14
14.8205
25.72n3
28.3231
52.3+56
22.3=311
55.824.1
Hurd-PL (Lee et ul., 2013)
11.2-02
189-
12.2.01
20.1
16.7427
42.1
15,4
Soft-PL (Assran et al., 2021)
11.5=02
19.4-03
12.2+02
20.5+02
18.4+13
40,4+440
15.8=13
43,4+11
S-CLIP (Mo 2023)
11.6-05
20.6-05
12.6=044
21.1
15,4220
43.9a55
17:5=15
47,4
SEMICLIP (ours)
12.3=04
21.0-c4
13.0=00
21.5+02
24.1+08
50.4=27
21.9-15
54.+
Table 5: Image-text retrieval results on SciCap (scientific figures) and Simpsons (comics) datasets. We train the model on each dataset and evaluate their validation sets.
14
10
Lackelpiis Undineee Zin Unbioca or Zine
Gan of accursey (8)
Gain of resieva (RQ2
SPT
8
SPT
10
SSFT
SSFT
Prequency.
25
15-
R$
Fashion
0
SdCap
Simpeons
0.5
0.6
0.7
0.8
Cosine Distance
(a)
(b)
(c)
Figure 2: (2a) Gain of averaged zero-shot classification for S-CLIP and different stages in SEMICLIP. "SPT" and "SSFT" respectively denote supervised pre-training and semi-supervised fine-tuning: (2b) Gain of averaged image-text retrieval for S-CLIP and different stages in SEMICLIP; (2c) The cosine distance for image-text pairs. "Labeled pairs" and "Unlabeled w/ LTmp" denote the distance between images and captions in labeled data and the distance between unlabeled images and surrogate captions when training with Irap- "Unlabeled w/ LCLIP" " represent the distance when directly aligning unlabeled images and surrogate captions through CLIP loss.
existent concepts closely align with the content of the images. This indicates that the classifier ei can effectively perceive the concepts images, which is beneficial for generating surrogate captions.
Gain of performance. Figures 2a and 2b illustrate the performance gains of S-CLIP and SEMICLIP different stages compared to fine-tuned CLIP. It evident that all three methods surpass fine-tuned CLIP to some extent, and SEMICLIP in semi cani-supervised fine-tuning shows a significant performance advantage over S-CLIP. Surprisingly, in SPT, SEMICLIP can completely outperform fine-tuned CLIP and even surpass S-CLIP in some settings, such as zero-shot and image-text retrieval performance fashion datasets. This indicates that leveraging concepts extracted from captions and trained by Equation (2) can enhance the model's perceptual capability regarding concepts which lays solid foundation for the subsequent semi-supervised fine-tuning.
Structure of trapezoid. As described Sec. 3.3.2, the Equation (5) will optimize the structure illustrated in Figure lb resemble an isosceles trapezoid. In Figure 2c, we present the frequency distribution of the cosine distances between labeled image-text pairs and the cosine distances between unlabeled images and surrogate captions. We observe that the distance between unlabeled images and surrogate captions is generally larger when we apply Liap. indicating that the lower base is longer than the upper base. However, if CLIP loss directly imposed on unlabeled images and surrogate captions, it causes their distances to become excessively close, resulting in significant decline in zero-shot classification by 4.07% on average. This indicates that the direct alignment unlabeled images and surrogate captions suffers from performance degradation due to the noise present in surrogate captions, whereas trapezoidal consistency effectively mitigates this issue and significantly improves performance through interactions between samples from both image and text modalities.
4.6 ABLATION ANALYSIS
We apart the factors that contribute significantly the success of SEMICLIP in Tab. 6.

---

Published as a conference paper at ICLR 2025
Remote sensing
Fashion
Ablations
ZS
I2T
T21
ZS
12T
T21
SEMICIIP
85.7
32.4
31.2
60.8+13
24.1
24.5=11
w/o Strong augmentation
84.4+19
31.9
30.3+1.
57.7122
22.3+19
22.5
w/o CSC
82.7+18
32.0=21
30.6
59.1
23.5+21
23.9
w/o Consistency for diagonals
82.0+26
31.9=09
30.8
56,4115
22.6113
23.9
w/o Consistency for legs
83.8+22
31.4+3.7
30.5+211
59.7+112
23.8+1.1
24.0+22
w/o CTC
81.9
31.7=28
30.7
54.1114
22.1423
22.4
w/o Learnable prompts
84.6424
31.5212
30.5
59.3119
23.2411
23.8423
Table 6: Ablation studies. We investigate the impact of the core components of the SEMICLIP on remote sensing and fashion datasets. "ZS", "I2T", "T2I" denote the averaged zero-shot classification, image-*text retrieval and text-iimage retrieval performance. "CSC" and "CTC" represent the concept-level semantic consistency and caption-level trapezoidal consistency, respectively.
Impact of strong data augmentation. We apply strong data augmentation to unlabeled images during the concept-level semantic consistency Equation (3). Upon replacing strong augmentation with weak augmentation, we notice significant performance decline across all settings, i.e., 3.1% zero-shot accuracy drop in fashion datasets. It suggests that strong data augmentation significantly contributes to the improvement of zero-shot performance.
Impact of concept-level semantic consistency (CSC), CSC is employed further leverage extracted concepts to enhance the model's 's awareness of concepts in task-specific domains. The exclusion of CSC results in averaged 1.15% performance drop. It noteworthy that in some cases, the removal of strong augmentation leads to more severe performance decline than the removal of CSC (e.g., the "ZS" fashion). We believe this due to the confirmation bias when applying weak augmentation for consistency, which can cause the model's biases to accumulate, severely impairing performance.
Impact of consistency for the diagonals of trapezoid. The consistency for diagonals can symmetrize the similarity across two mismatched image-text pairs. When removing the consistency for diagonals, observe reduction in performance averaging 1.85%. This suggests that cross-modal interaction and consistency constraints are beneficial for enhancing the model's 's alignment capabilities.
Impact of consistency for the legs of trapezoid. SEMICL.IP utilizes consistency for legs to enhance the intra-modality similarity. While the benefits derived from the consistency for legs are relatively small compared to the diagonals, with an average performance decrease of 0.92% upon its removal, nevertheless underscores the potential of constraints for legs to enhance overall performance by ensuring coherence among samples within each modality.
Impact of caption-level trapezoidal consistency (CTC). The complete removal of CTC undoubtedly leads to significant performance decline, with reductions of 5.25% and 1.33% observed on the zero-shot and image-text retrieval tasks, respectively. This highlights that CTC plays an important role in strengthening the interaction and alignment between samples. More importantly, it facilitates the integration of surrogate captions into the training, further leveraging unlabeled images to improve the performance and effectively mitigating noise potentially introduced by surrogate captions.
Impact of learnable prompts in surrogate caption. In order to investigate the role of learnable prompts in constructing surrogate captions for unlabeled images, we detached the gradients and achieve
froze the prompts after initialization. Results reveal that applying learnable prompts an averaged 0.97% improvement in model performance, which we believe stems from the dynamic adaptation of prompts that better aligns surrogate captions with task-specific domains.
5 CONCLUSION
This paper proposes a new semi-supervised CLIP adaptation method that effectively adapts CLIP to target tasks using only small number of image-text pairs. The proposed trapezoidal consistency can effectively tackle inaccurate surrogate captions, achieving robust representation learning. Extensive experiments demonstrate that our method achieves state-of-the-art results in both zero-shot classi- fication and image-text retrieval tasks. We believe that trapezoidal consistency has the potential to flexibly adapt to different multi-modal superieper learning tasks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Science Foundation of China (62206049, 62176055), Postgraduate Research& Innovation Program of Jiangsu Province (KYCX25 (544),,.0544) and the Big Data Computing Center of Southeast University. We would like thank anonymous reviewers for their constructive suggestions.
Ethics Statement. This research fully complies with the specified ethical standards. No human participants or sensitive data were used, and the datasets utilized are open and ethically sourced. The research methodology and results have been carefully evaluated prevent any potential misuse, discrimination, or bias. All procedures align with legal, privacy, and security regulations. There are no conflicts of interest or external funding that could influence the study's findings. We have maintained transparency and integrity throughout the research process to ensure that the results are accurately and ethically represented.
Reproducibility Statement. We have made significant efforts to ensure the reproducibility of our results. The source code used in our experiments is included in the supplementary materials, along with detailed README file that provides step-by-step instructions and the necessary commands to reproduce the experiments. All the hyperparameters and experimental settings are specified in Sec. 4.1 to facilitate replication.
REFERENCES
Doron Adler. Simpsons blip captions.
/Noooodd8
2023.
Arto Arampacha, Vidhani Dev, Goutham Venkatesh, Bhaskar Mayank, Ghosh Ritobrata, and Pal Sujit. Fine tuning clip with remote sensing (satellite) images and captions. https://hugginggaac. co/ sto/blo/// fine-tiiii tune clip-rsicd, 2021.
Eric Arazo, Diego Ortego, Paul Albert, Noel O' O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In 2020 International joint conference neural networks (IJCNN, pp. 1-8, 2020.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Armand Joulin, Nicolas Ballas, and Michael Rabbat. Semi-supervised learning of visual features by non-parametrically predicting view assignments with support samples. In ICCV, 2021.
Alexandre Attia. The simpsons characters data. https: //www. wwtpg/ll...//daaaeeeeeeeeeeeeeee alexattia/the, wwsiiissssssaaaass
datasst, 2018.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture- -f-mmodaity-experts. Advances in Neural Information Processing Systems, 35:32897-32912, 2022.
David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring. arXiv preprint arXiv:1911.09780 2019.
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. 2009.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj. and Marios Savvides. Softmatch: Addressing the quantity- trade-off in semi-supervised learning. arXiv preprint, 2023.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120, 2020.
11

---

Published as a conference paper at ICLR 2025
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 2017.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Xiaoyi Dong, Jianmin Bao, Yinglin Zheng, Ting Zhang, Dongdong Chen, Hao Yang, Ming Zeng, Weiming Zhang, Lu Yuan, Dong Chen, et al. Maskelip: Masked self-distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10995-11005, 2023.
Kai Gan and Tong Wei. Erasing the bias: Fine-tuning foundation models for semi-supervised learning. In Forty-first International Conference on Machine Learning. 2024.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramid- clip: Hierarchical feature alignment for vision- langugg model pretraining. Advances in Neural Information Processing Systems, 1-:35959-35977 2022.
Shashank Goel, Hritik Bansal, Sumit Bhatia, Ryan Rossi, Vishwa Vinay, and Aditya Grover. Cyclip: Cyclic contrastive language-image pretraining. Advances in Neural Information Processing Systems, 35:6704-6719, 2022.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. Advances in Neural Information Processing Systems, 17, 2004.
Xintong Han, Zuxuan Wu, Phoenix X Huang, Xiao Zhang. Menglong Zhu, Yuan Li, Yang Zhao, and Larry S Davis. Automatic spatially-award fashion concept discovery. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1463-1171, 2017.
Ting- Yao Hsu, C Lee Giles, and Ting-Hao's Kenneth' Huang. Scicap: Generating captions for scientific figures. arXiv preprint arXiv:2 10. 1624, 2021.
Runhui Huang, Yanxin Long. Jianhua Han, Hang Xu, Xiwen Liang, Chunjing Xu, and Xiaodan Liang. Nlip: Noise-robust language-image pre-training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 926-934, 2023.
Gabriel Ilarco,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and Dision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916, 2021.
Alistair EW Johnson, Tom Pollard, Seth Berkowitz. Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, de-identified publicly available database of chest radiographs with free-text reports. Nature scientific data, 2019.
Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distri- bution aligning refinery of pseudo-label for imbalanced semi-supervised learning. Advances in neural information processing systems, 33:14567-11479 2020.
Younghyun Kim, Sangwoo Mo, Minkyu Kim, Kyungmin Lee, Jaeho Lee, and Jínwoo Shin. Bias- to-text: Debiasing unknown visual biases through language interpretation. arXiv preprint arXiv:2301.11104, 2023.
Dong-Hyun Lee al. Pseudo-label: The simple and efficient supirvupee learning method for deep neural networks. In Workshop on challenges in representation learning. ICML, 2013.
Doyup Lee, Sungwoong Kim, Ildoo Kim, Yeongjae Cheon, Minsu Cho, and Wook-Shin Han. Con- trastive regularization for semi-supervised learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 391 1-9900, 2022.
12

---

Published as a conference paper at ICLR 2025
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34:9694 9705, 2021a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Kei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprini urXiv:/906 0tte:/9088..355,, 2019.
Liunian Harold Li, Haoxuan You, Zhecan Wang, Alireza Zareian, Shih-Fu Chang, and Kai-Wei Chang. Unsupervised vision-and-language pre-training without parallel images and captions. In NAACL, 2021b.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. arXiv preprint arXiv:2012. 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, pp. 121 2020b.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive language- imngge---aageeee pre-training paradigm. arXiv preprini arXiv:21 2021c.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context, In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang. Mohit Bansal, and Colin A Raffel. Few-shot murameter-efficien fine-tuning is better and cheaper than in-contex, learning. Advances Neural Information Processing Systems, 35:1950- 19600-9996,0000 2022.
I Loshchilov. Decoupled weight decay regularization arXiv preprint 1711v 2017.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-lange lnn-lag tasks. Advances Neural Information Processing Systems, 32, 2019.
Xiaoqiang Lu, Binqiang Wang, Xiangtao Zheng, and Xuelong Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing, 2017.
Huaishao Luo, Lei Ji, Ming Zhong, Yang Chen, Wen Lei, Nan Duan, and Tianrui Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing, 508: 293-304, 2022.
Muhammad Jehanzeb Mirza, Leonid Karlinsky, Wei Lin, Horst Possegger, Mateusz Kozinski, Rogerio Feris, and Horst Bischof. Lafter: Label-free tuning of zero-shot classifier using language and unlabeled image collections. Advances in Neural Information Processing Systems, 36, 2024.
Sangwoo Mo, Minkyu Kim, Kyungmin Lee, and Jinwoo Shin. S-clip: Semi-supervised vision- language learning using few specialist captions. Advances in Neural Information Processing Systems, 36:61187-61212. 2023.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xic. Slip: Self-supervision meets language-image pre-training. In European Conference Computer Vision, pp. 529-544, 2022.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03744 2018.
Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for zero-shot transfer learning. Neurocomputing, 126658,,,,, 2023.
13

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 9748-8763, 2021.
Negar Rostamzadeh, Seyedarian Hosseini, Thomas Boquet, Wojciech Stokowiee, Ying Zhang, Christian Jauvin, and Chris Pal. Fashion-gen: The generative fashion dataset and challenge. arXiv preprint arXiv: 08,/7,000 2018.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open sorge-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.084000 2022.
Jiang-Xin Shi, Tong Wei, Zhi Zhou, Jie-Jing Shao, Xin- Yan Han, and Yu-Feng Li. Long-tail tail learning with foundation model: Heavy fine-tuning hurts. In Forty-first International Conference on Machine Learning, 2024.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Do- gus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances neural information processing systems, 23:596-608, 2020.
Hao Tan and Mohit Bansal, Lxmert: Learning cross-modality encoder representations from trans- formers. arXiv preprint arXiv: /et.//9990.9)) 2019.
Mariya Vasileva, Bryan Plummer, Krishna Dusad, Shreya Rajpal, Ranjitha Kumar, and David Forsyth. Learning type-aware embeddings for fashion compatibility. In Proceedings of the European Conference on Computer Vision, pp. 390-405, 2018.
Tzu-Jui Julius Wang, Jorma Laaksonen, Tomas Langer, Heikki Arponen, and Tom E Bishop. Learning by hallucinating: laniuanla pre- pre-training with weak supervision. In WACV, 2023a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision- languuee tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19175-19186, 2023b.
Ximei Wang, Jinghan Gao, Mingsheng Long, and Jianmin Wang. Self-tuning for data-efficient deep learning. International Conference on Machine Learning, pp. 10738-10748, 2021a.
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. In EMNLP, 2022.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint 1tii::2008119000 2021b.
Tong Wei and Kai Gan. Towards realistic long-tailed semi-supervised learning: Consistency is all you need. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 3469-3478, 2023.
Gui-Song Xia, Wen Yang, Julie Delon, Yann Gousseau, Hong Sun, and Henri Maître. Structural high-resolution satellite image indexing. 2009.
Gui-Song Xia, Jingwen Hu, Fan Hu, Baoguang Shi, Xiang Bai, Yanfei Zhong, Liangpei Zhang, and Xiaoqiang Lu. Aid: benchmark data set performance evaluation of aerial scene classification. IEEE Transactions Geoscience and Remole Sensing, 2017.
Shiyu Xia, Jiaqi Lv, Ning Xu, and Xin Geng. Ambiguity-inducee contrastive learning for instance- dependent partial label learning. In IJCAI, pp. 3615-3621, 2022.
14

---

Published as a conference paper at ICLR 2025
Shiyu Xia, Jiaqi Lv, Ning Xu, Gang Niu, and Xin Geng. Towards effective visual representations for partial-label learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19589-15598, 2023.
Shiyu Xia, Yuankun Zu, Xu Yang, and Xin Geng. Initializing variable-sized vision transformers from learngene with learnable transformation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.
Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9653-9663, 2022.
Fan Yang, Kai Wu, Shuyi Zhang, Guannan Jiang, Yong Liu, Feng Zheng, Wei Zhang, Chengjie Wang, and Long Zeng. Class-aware contrastive semi-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14421-14430, 2022.
Kaicheng Yang, Jiankang Deng, Xiang An, Jiawei Li, Ziyong Feng, Jia Guo, Jing Yang, and Tongliang Liu. Alip: Adaptive language-image pre-training with synthetic caption. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Xuewen Yang, Heming Zhang, Di Jin, Yingru Liu, Chi-Hao Wu, Jianchao Tan, Dongliang Xie. Jue Wang, and Xin Wang. Fashion captioning: Towards generating accurate descriptions with semantic rewards. In Proceedings of the European Conference Computer Vision, 2020.
Yi Yang and Shawn Newsam. Bag-of-visual-words and spatial extensions for land-use classification. In SIGSPATIAL international conference on advances in geographic information systems, 2010.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2 1.07783, 2021.
Bo Ye, Kai Gan, Tong Wei, and Min-Ling Zhang. Bridging the gap: Learning pace synchronization for open- open-world semi-supervised learning. In Proceedings of the Thirty- -hird International Joint Conference Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pp. 5362- 5370. ijcai.org, 2024.
Bowen Zhang, Yidong Wang. Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. NeurlPS, 34: 18408-18419, 2021.
Fan Zhang, Bo Du, and Liangpei Zhang, Saliency-guided unsupervised feature learning for scene classification. IEEE Transactions on Geoscience and Remote Sensing, 2014.
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 168 1-16225, 2022a
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-233 -2348, 2022b.
Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 13041-13049, 2020.
Qin Zou, Lihao Ni, Tong Zhang, and Qian Wang. Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 2015.
15

---

Published as a conference paper at ICLR 2025
A ADDITIONAL EXPERIMENTS
A.1 DIFFERENT NEURAL NETWORK ARCHITBCTURES
We conduct experiments to evaluate the performance of SEMICLIP in zero-shot classification and image-text retrieval with different neural network architectures. As shown in Tabs. 7 and 8, SEMICLIP achieves average performance improvement of 3.52% on zero-shot classification and 0.28% on image-text retrieval compared to S-CLIP when adopting ResNet-50 as the backbone. In addition, SEMICLIP exhibits averaged performance advantage of 2.62%, 4.20%, and 1.10% over S- CLIP in zero-shot classification, image->tex, retrieval, and text- retrieval when implemented with the ViT-B-32 architecture. This indicates that SEMICLIP can achieve consistent and significant performance advantages for different neural network architectures and model sizes.
Model
Method
RSICD-CLS
UCM-CLS
WHU-RS19
RSSCN7
AID
CLIP (original)
45.3
50.5
65.5
58.9
47.8
ResNet-50
CLIP (fin-ttnnd)
58.3+00
63.5+24
76.5
61.9
63.1
S-CLIP
66.9=17 (48.6)
66.7=1u (43.2)
86.9420 (+10.4)
66.2+11 (44.3)
73.0aus (49.9)
SEMICLIP
72.0=21 (+13.7)
73.6+14 (+10.))
87.2+24 (+10.7)
08.9:04447.0
75.6+10 (+12.5)
CLIP (original)
56.2
$8.5
76.3
62.3
55.6
VII-B-32
CLIP (fin-tuned)
73.6+3u
79.0+44
91.1+13
70.4+31
79.6+25
S-CLIP SEMICLIP
78.5 5805 (+4.9)
77.444 (-166)
94, 94,92us (43.8)
71. (+0.7)
84.2 (+4.6)
81.3.0a (+7.7)
84.3 (+5.3)
95.2208 (+4.1)
73.0 (+2.6)
85,4 (+5.8)
CLIP (original) CLIP (fine-tuned)
59.2
60.2
81.2
59.6
ViT-B-16
75.7=23
80.1
92.2+15
70.7
79.8
S-CLIP
81.4 (+5.7)
81.3
95.9218 (+3.7)
75.1 13.1 (+4.4)
86.4 (+6.6)
SEMICLIP
83.8 (+8.1)
85.6 (45.5)
96.6212 (+4.4)
75.3 (+4.6)
27 (+7.6)
Table 7: Zero-shot classification results on remote sensing datasets using different neural architectures. Parentheses indicate the performance gap from the supervised CLIP, and bolds denote the best results within each architecture.
A.2 HYPERPARAMETER SENSITIVITY
Despite the impressive performance attained by our method, the inclusion of some parameters leads us question whether these outstanding outcomes are result of parameter tuning. Figure presents the results of the parameter sensitivity experiments for P (top-P% unlabeled images and surrogate captions are selected for training), & (number of selected concepts for each unlabeled image), and j (frequency threshold for concepts initialization). We observe that variations in P and i do not result significant fluctuations in performance. However, when μı exceeds 5, the performance for zero-shot exhibits noticeable decline. We believe this is due to a larger μ filtering out too many concepts, resulting in an insufficient number of concepts available for further consistency, which ultimately impacts model performance. Nevertheless, considering the strong performance of SEMICLIP across the majority of sensitivity experiments, it can be concluded that the performance advantage stems from its thoughtful design rather than extensive reliance parameter tuning.
Accurac (%)
Accuracy (%)
RBK (%6)
RBK (86)
Accurary
Z5
88-
12T
86-
-28
(((%)
Figure 3: The sensitivity P (%), k and on remote sensing datasets. "ZS", "I2T", "T2I" denote the averaged zero-shot classification, image retrieval and text->image retrieval performance across various hyperparameter settings.
16

---

Published as a conference paper at ICLR 2025
Image >txx R@5
Text >image R@s
Model
Method
RSICD
UCM
Sydney
RSICD
UCM
Sydney
CLIP (original)
9.4
34.3
36,2
10.1
24.8
51.7
ResNct-50
CLIP (fine-tuned)
15.4=17
41.3211
47.1
15.1
40.9=14
56.1424
S-CLIP
18.4
45.7+11
50.0130
16.8212
43.5=11
55.1
SEMICLIP
18.6=19
46.2+:10
49.6
17.1
44.3
55,4+1x
CLIP (original)
12.1
37.6
41.4
12.9
35.2
37.9
VIT-B-32
CLIP (fine-tuned)
24.5=00
56.3115
47.1
24.621.
57.0=11
50.0146
S-CLIP
25.0=0.8
55.7+225
43.1
23,9
59.2
51.7
SEMICLII
26.7=111
60.82188
48.9
25.3210
60.023.1
52.8226
CLIP (original)
12.6
46.7
44,8
13.9
39.5
44.8
ViT-B-16
CLIP (fine-tuned)
25.7=223
55,2.108
46,6130
24.9:0.6
56,3
49 49.2
S-CLIP
27.5+11
57.0+15
51.1443
25.6440.
57.3+4.1
50.64426
SEMICLIP
27.6:1.6
59.2122
53.4146
26.4
58.4
58.0120
Table Image-text retrieval results on remote sensing datasets using different neural architecture. Bolds denote the best results within the same setups.
Image -text R@1
Text-+image R@1
Method
Data
RSICI
UCM
Sydney
RSICD
UCM
Sydney
CLIP (original)
4.5
17.1
10.3
3.6
9.0
12.1
CLIP (fin--tnndd)
L
10.6
18.9427
20.7+.0
7.3
14.9
12.6#1a
Hard-PI. (1.cc al., 2013)
11.4117
17.0=29
17.9
6.820.
14.1
20.7+10
Soft-PL (Assran et al., 2021)
L=U
11.6:11
19,2=111
20.7+330
7.1104
13.72006
20.3
S-CLIP (Mo et al., 2023)
11.4003
19.5=13
17.8+26
7.1408
14.0=1.4
16.7=10
SEMICLIP (ours)
12.0106
20.3
21.8240
7.62:99
14,9
21.3236
Hard-PL (Lee al., 2013)
11.1115
17.1-24
19.0133
6.11002
16.0
16.715.3
Soft-PL (Assran et al., 2021)
LyU
11.2:0a
16.0
16.7
6.5
15.1
18,4
S-CLIP (Mo et al., 2023)
10.9406
18.9-15
20.4122
7.2106
14,9126 16.5212
SEMICLIP (ours)
11.5
22.1:05
20.7
7.31:0.5
17.7
17.8
Table 9: Image-text retrieval results of R@1 on remote sensing datasets, following the same setup of
Table 1. Bolds denote the best results within the same setups.
A.3 RESULTS FOR @@1
Tabs. and 10 illustrate the R@1 performance of image-text retrieval for remote sensing and fashion datasets. For remote sensing datasets, SEMICLIP achieves performance advantages of 1.42% and 1.70% over S-CLIP in image-texx -tet and text->image retrieval, respectively. Similarly, on the fashion datasets, SEMICLIP achieves performance gains of 4.33% and 3.70% for image->tex and text-imaae retrieval. It is evident that SEMICLIP achieves more significant advantage on fashion datasets than remote sensing datasets, which we attribute the larger sample size and greater diversity of the fashion datasets. This diversity makes it difficult for S-CLIP to identify suitable neighboring samples to generate effective pseudo-labels
A.4 RESULTS FOR SMALLER BATCH SIZE
All experiments in our main paper are conducted on four NVIDIA A6000 GPUs with batch size of 64 per GPU. Given the potential limitations of computational resources, we conducted experiments to evaluate SEMICLIP on single GPU, where the batch size is only one-quarter of that used in previous experiments. Tab. 11 reveals that SEMICLIP demonstrates significant performance advantage over S-CLIP, achieving an average improvement of 7.54% in zero-shot classification. Notably, the performance of S-CLIP is even 1.72% lower than that of fine-tuning CLIP with only labeled image-text pairs, indicating its inability to achieve satisfactory performance with a smaller batch size due to the reliance neighboring information within the batch and motivation under the paradigm of partial label learning (Xia al., 2022; 2023). In contrast, SEMICLIP is less affected by batch size, consistently demonstrating robust improvements in the model's alignment capabilities.
17

---

Published as a conference paper at ICLR 2025
Image- R@
Text- Text- R@1
Method
Fashion200k
FashionGen
Polyvore
Fashion200k
FashionGen
Polyvore
CLIP (original)
4.7
10.9
11.4
4.5
12.5
13.0
CLIP (fine-tuned)
4.8g03
12.9=04
6.4202
4,2400
13.0
6.22o.1
Hard-PL. (Lec ct al., 2013)
4.8=02
11.1=02
6,5+03
5.1
13.7+04
7.8=03
Soft-PL (Assran et al., 2021)
5.64003
14.7=u4
8,2402
5,340.
15,4+05
9.8+4.0
S-CLIP (Mo et al., 2023)
6,6#003
16.7=14
10,4+002
6.1402
18.9
10,9+04
SEMICLIP (ours)
9,3
24.0
13.4
8,7
25.2
13.1
Table 10: Image-text retrieval results of R on fashion datasets, following the same setup of Table 3. Bolds denote the best results within the same setups.
Method
RSICD-CLS
UCM-CLS
WHU-RS19
RSSCN7
AID
CLIP (original)
59.2
60.2
81.2
69.0
59.6
CLIP (fie-ttnedd)
75.3
77.2476
92.3#29
72.4a32
79.7
Hard-PL. (Lee al., 2013)
78.5an5 (43.2)
70.1=47 (-9.))
91.6284 (-0.7)
62.4a48 (-10.0)
82.2a29 (+2.5)
SoftPPL (Assran al., 1)
77.8+34 (+2.5)
71.2=33 (-6.0)
94.3
64.9=53 (-7.5)
82.6 (+2.9)
S-CLIP (Mo al., 2023)
74.2=16 (-L.)
69.7=30 (-7.5)
93.3213 (+1.0)
71.3210 (-1.1)
79.8216 (40.1)
SEMICLIP (ours)
83.5 (+8.2)
37.5
25.1 (42.8)
72.5 (40.1)
87.4210 (+7.7)
Table 11: Zero-shot classification results for batch size 64 of single GPU on remote sensing datasets. We compare the original CLIP, supervised CLIP fine-tuned on labeled data, and semi- supervised methods that utilize unlabeled data sampled from the same distribution as the labeled data. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one. Bolds denote the best results within the same setups.
B COMPETING METHODS
We denote the pre-trained CLIP as CLIP (original) and the model fine-tuned only on image-text pairs CLIP fine-tuned). Following S-CLIP, we compare SEMICLIP with extensions of SSL methods, including Hard-PL and Soft-PL.
Hard-PL. For an unlabeled image 1j. Hard-PL. searches for the labeled image within the same batch that is most similar to I5 in terms of image embeddings and aligns its corresponding text with 1 through Equation (1).
Soft-PL. Soft-PL employs soft nearest neighbor (Assran et al., 2021) to achieve alignment for unlabeled images. Unlike Hard-PL, Soft-PL takes into account the cosine similarities between unlabeled image I and multiple nearest neighbors in labeled data, and applies the softmax function to obtain the soft pseudo-label which tends to have a value close to one for visually similar images. More details can be found in Assran et al. (2021); Mo et al. (2023).
S-CLIP. S-CLIP (Mo et al., 2023) features two innovative pseudo-labeling strategies: the caption- level pseudo-label combines captions from paired images by solving an optimal transport problem, while the keyword-level pseudo-label is derived from the nearest paired image's 's caption, employing partial label learning. It assumes set of candidate labels for supervision instead of exact labels, enhancing contrastive learning and the integration of the language modality. S-CLIP and SEMICLIP are designed to improve performance by effectively utilizing additional unpaired data during training.
C MORE EXPLANATION FOR CAPTION-LEVEL TRAPEZOIDAL CONSISTENCY
Distinctions between CyCLIP and Caption-level Trapezoidal Consistency. Considering that there exists some resemblance between CyCLIP and caption-level trapezoidal consistency (CTC), delineate their differences and contributions of CTC below:
CTC introduces CyCLIP into the semi-supervised learning scenario by utilizing unlabeled images and surrogate captions, thereby extending the applicability of CyCLIP beyond complete image-text pairs
We found that CTC can partially alleviate the impact of noise in the captions on training and provided a geometric explanation through trapezoidal geometrie representation aspect.
18

---

Published as a conference paper at ICLR 2025
CTC offers new insights for future work addressing than-ideal alignment in image-text pairs, which a widespread issue in practical settings.
Why is CTC effective? We explain the reasons why CTC is effective below:
Traditional contrastive learning methods, such as CLIP, aim reduce the distance between matching image-text pairs when learning image-text representations. However, they do not impose constraints on the overall geometric structure of the data, which leads to inconsistent predictions between the image and text spaces, especially in semi-supervised scenarios with only a small number of image-text pairs. Trapezoidal consistency addresses this issue by introducing equal-length legs and diagonals consistency regularization terms. These regularizers constrain the similarity gaps between mismatched image-text pairs as well as Imageeimage and text-text pairs, resulting in more consistent and structured representation space, thereby improving prediction consistency.
Trapezoidal consistency can ensure geometric alignment between image and text represen- tations, allowing the model to make more consistent predictions when reasoning in both the image and text spaces. This means that the image and text representations learned by trapezoidal consistency can be more easily interchanged, leading to improved performance on downstream tasks.
The rigid separation between the positive pairs and negative pairs enforced by the contrastive loss CLIP may degrade performance when some pairs in the negative batch belong similar entity (Mu et al., 2022). Trapezoidal consistency poses constraints on the overall geometry of all the data pairs rather than forcing rigid separation, which enables the interaction of information between intra-modal and moosl samples, even in data-scarce scenarios.
D MORE EXAMPLES FOR PREDICTED CONCEPTS
Figure 4 presents additional examples of predicted concepts for unlabeled images on remote sensing and fashion datasets. We observe that although some concepts do not explicitly appear in the ground- truth captions, they are still highly semantically relevant to the unlabeled images, as illustrated by "stadium" Figure 4a and "midi" in Figure 4d. The strong semantic alignment of the extracted concepts with the images provides solid foundation for the subsequent concept-level semantic consistency and caption-level trapezoidal consistency in proposed SEMICLIP.
E WHY SEMICLIP CAN OUTPERFORM ORACLE IN SOME SETTINGS?
Interestingly, we find that our method can outperforms the oracle in some zero-shot settings. We believe this is due to the severe imbalance in the proportions of the sub-datasets within the training sets, RS-ALL and Fashion. When training on the full dataset, sub-datasets with a smaller proportion, such UCM in Tab. 1, perform poorly on their corresponding pero-shot test set(UCM-CLSS which resulted in slightly lower overall performance. We find this to be an interesting phenomenon, and it warrants further research into how mitigate the issue of imbalance in the proportions of the sub-datasets.
The oracle outperforms SEMICLIP in retrieval performance, indicating that increasing the amount of labeled data will significantly improve retrieval performance. This also points to a potential direction for further improvement of SEMICLIP in the future.
F THE SIZE OF DATASETS
For remote sensing datasets, the RSICD, UCM, and Sydney datasets contain 8734, 1680, and 497 image-text pairs, respectively. The three datasets constitute RS-ALL, and we randomly select 10% of the image-text pairs from the training set labeled data, with the remaining pairs treated as unlabeled. During the inference, the sizes for classification datasets are shown Tab. 12.
For fashion datasets, the Fashion200k, FashionGen, and Polyvore datasets contain 61753, 60147, and 71967 image-text pairs, respectively. The sizes for classification datasets shown in Tab. 13.
19

---

Published as a conference paper at ICLR 2025
Image caption:
Image caption:
"Some sparse buildings and several ponds are resort area with many farmlands.
"Several boats are in port near many red buildings,"
Predicted concepss concepts:
Predicted concepts:
resort, pond, lake, bareland
port, boats, wherf. buildiigs
Image caption:
Image caption:
"Many buildings and green plants around baseball field."
green pond surrounded by cropped meadow."
Predicted concepts:
Predicted concepts:
stadium. playground, baseball, field
pond, farmlands, pool, meadows
(a)
(b)
Image caption:
Image caption:
"Multicolor faux fur panel biker jacket."
"Black women's 's dara skirt."
Predicted concepts: women, skirt, midi, knit
Predicted concepts: jacket, biker, fur, leather
Image caption:
Image caption
"Black scoop neck silk blend Lean tee."
"Gray erin wrapover sleeveless neck tie blouse."
Predicted concepts:
Predicted concepts:
tee, neck, sleeve, blouse
tie, blouse, neck, silk
(c)
(d)
Figure 4: (4a and 4b) More examples for predicted concepts for unlabeled images in remote sensing datasets; (4c and 4d) More examples for predicted concepts for unlabeled images in fashion datasets.
Table 12: The size of zero-shot classification datasets in the remote sensing domain.
RSICD-CLS
UCM-CLS
WHU-RSI9
RSSCN7
AID
Number of images Number of classes
1094
2100
1005
2800
10000
31
21
19
7
30
Table 13: The size of zero-shot classification datasets in the fashion domain.
Fashion200k
FashionGen
Polyvore Class
Super-class
Sub-class
Super-class
Sub-class
Number images Number classes
29,785
32,528
14,657
5
31
48
121
11
For other datasets, SciCap contains 106934 image-text pairs and Simpsons only contains 720 pairs.
G THE COMPARISON WITH NLIP
Due to concept labeler in Nlip (Huang et al., 2023) looks similar to semantic concepts mining, the differences follows:
We train a linear classifier achieve better image-concept alignment in task speciffe domain, while concept labeler in Huang et al. (2023) obtains concepts through amage-concepl retrieval, which may not enable effective retrieval certain specialized domains due niche nature of some concepts.
Concepts generated in Nlip are considered as conditional input to the cross-mode modal decoder, which requires a large amount of data for training, significantly increasing the overhead. However, SEMICLIP can be trained with only a small amount of labeled data, achieving significant improvements in model performance in semi-supervised settings.
20

---

Published as a conference paper at ICLR 2025
H THE ANALYSIS OF PROMPTING STRATEGY
We analyze the prompting strategy experimentally for more in-depth investigation.
Comparing performance with prompts at different positions. We conducted experiments on the effect of prompts at different positions, with all prompts initialized using normal distribution for fair comparison. Overall, as shown in Tab. 14, the performance slightly better when the prompts positioned in the middle, which is why
this approach in experiments.
Table 14: Effects of incorporating prompts various positions.
Position
Beginning
Middle
End
7S
12T
T2T
75
12T
T21
75
12T
T21
Accuracy or Recall 84.2
32.2
30.9
84.9
32.3
30.6
84.4
31.7
30.3
Different Prompts initialization. From the results in Tab. 15, we found that zero initialization performed poorly in zero-shot tasks, with 1.1% lower performance compared to SEMICLIP, but showed no significant decline in retrieval performance. Normal initialization performs slightly worse than our proposed method in both zero-shot and retrieval tasks. Overall, SEMICLIP's initialization method achieves the most robust and stable performance.
Table 15: Effects of incorporating prompts of different initialization.
Initialization
Zero
Normal
SemiCLIP
7S
12T
T2T
75
12T
T2T
7S
12T
T2I
Accuracy or Recall
84.6
32.5
31.0
84.9
32.3
30.6
85.7
32,4
31.1
Fixed vs learnable prompts. The results in Tab. 6 show that Iearnable prompts achieve an average performance advantage of 0.9% over fixed prompts, which indicates that learnable prompts facilitate improved model adaptation to task-specific domains.
I EFFECTS OF DIFFERENT LABELED DATA PERCENTAGES
The experimental results in Figure show that SEMICLIP consistently outperforms S-CLIP across different label proportion settings. In settings with a low proportion of labeled data, the quality of extracted concepts impacted. However, methods like S-CLIP, which rely on labeled data to construct neighbor labels, will experience more significant performance drop in such scenarios. The above experiments demonstrate the robustness of the proposed training framework under different proportions of labeled data.
35.0
95
32.6
aesuren (8)
80
90.0
(2T ROOK
30
27.5
35
(Iredknnd)
CUP (freedanwd)
(fne-luned)
80
S-CUPP
S-CUPP
22.6
S-CLIP
SemiCLI
SemiCUP
SemiCLIP
1%
6%
10%
25%
60%
1%
6%
10%
25%
60%
1%
6%
10%
26%
50%
Labeled Percentages
Labeled Percentages
Labeled Percontaages
Figure 5: Ablation study regarding different percentages of labeled data.
J EXPERIMENTS ON GENERAL BENCHMARKS
To evaluate the performance of SEMICLIP on general benchmarks, we conducted comparative experiments the COCO (Lin et al., 2014) dataset. The results in Tab. 16 indicate that SEMICLIP
21

---

Published as a conference paper at ICLR 2025
1% labeled
10% labeled
Image Hext
Text->image
Image-Atex
Text-oimage
Method
R@
R@5
R@
R@S
R@
R@5
R@
R@5
CLIP (fine-tuned)
33.7=13
60.2=00
33.6=11
60.5=20
35.6
65. 65.0228
36.9+1.5
65.0+1.
S-CLIP (Mo et al., 2023)
29.5=18
54.8209
27.8214
53.821.
33.2112
60.613.1
31.7416
59.1
SEMICLIP (ours)
37.8 122
64,4133
37.6112
65.7.33
41.4.13
70.3
42.6.19
69.8.2.
Table 16: The retrieval performance on COCO.
can achieve significant performance improvements on general benchmark over CLIP (fine-tuned) and S-CLIP. It is worth noting that S-CLIP's performance shows an average decrease of 4.5% compared to CLIP (fine-tuned), aligning with the paper's s claim (Mo et al., 2023) that S-CLIP experiences performance drops when trained on small number of image-tex: pairs in common datasets like COCO. However, the superior performance our proposed SEMICLIP is unaffected by the different types of datasets, achieving significant improvements both commonly used datasets and task- specific datasets.
22


Reviews:
[{"review_id": "iHukJyE589", "reviewer": "Reviewer_zoSv", "summary": "-- This paper proposes a new semi-supervised CLIP training method SEMICLIP that adapts CLIP to target tasks using only a small amount of image-text pairs.\n\n-- This paper designs concept-level consistency and caption-level trapezoidal consistency for learning from unlabeled images to enhance visual representations and improve cross-modal alignment, respectively.\n\n-- Extensive experiments demonstrate that the proposed method achieves state-of-the-art results in both zero-shot classification and image-text retrieval tasks.", "questions": "-- In SEMANTIC CONCEPTS MINING,  the linear classifier is initialized from the concept features of clip text encoder. Do the parameters of this module update during pre-training?\n\n-- It seems this linear classifier is a close-set classifier, why not use the open-set method?\n\n-- Does each concept have its own prompts-driven template [V ]1 [V ]2 [V ]3? Is the prompts-driven template [V ]1 [V ]2 [V ]3 shared for all concepts?\n\n-- It is unclear whether the model is trained from scratch or initialized from pre-trained CLIP.\n\n-- The sizes of datasets in the experiments are unclear.\n\n-- Why do the proposed models suppress the fine-tuned models (CLIP (fine-tuned))? And what is the upper bound for performance?\n\n-- Why is the lower base not used in Figure 1(b)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- SEMICLIP mines candidate semantic concepts from labeled data and learns to associate images with concepts, which may be a new way of using unlabeled images in vision-language pre-training.\n\n-- This paper proposes the trapezoidal consistency to enhance the multi-modal alignment by exploiting the geometric structure of trapezoids in the representation space.\n\n-- This paper uses prompts-driven templates and predicts concepts to construct surrogate captions for unlabeled images.", "weaknesses": "-- The proposed method consists of two stages. The finetune stage heavily relies on the concept ming from the pretraining stage. The contribution of each stage is hard to evaluate.\n\n-- The experiments are only conducted on task-specific datasets. What is their performance on common zero-shot classification and multimodal retrieval?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oIepbHm7lN", "reviewer": "Reviewer_WcrK", "summary": "The paper presents SEMICLIP, a semi-supervised training method that enhances CLIP’s performance with limited number of image-text paired data. It utilizes a small number of labeled pairs along with a large set of unlabeled images by employing semantic concept mining to create pseudo-labels for the unlabeled data. The method introduces trapezoidal consistency regularization to maintain geometric relationships between image-text pairs, optimizing the model's alignment. Experimental results show that SEMICLIP bring improvements on zero-shot classification and image-text retrieval performance on various domains datasets.", "questions": "Please refer to the weaknesses section to see the exact questions.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is written in a clear manner, making the concepts and methodologies easy to understand. This clarity enhances the overall accessibility of the work, allowing readers to grasp the key ideas without difficulty.\n- Extensive experiments to evaluate the effectiveness of the proposed method. The experiments are conducted on 8 classification benchmarks and 6 retrieval benchmarks.", "weaknesses": "- The proposed Semantic Concepts Mining method is not novel. Previous work [1] has already used CLIP as a concept labeler to construct pseudo labels for contrastive learning and image captioning. While [1] conducted experiments under the pretraining setting, this paper focuses on small domain datasets.\n- Caption-level trapezoidal consistency builds incrementally on CyCLIP [2] in a semi-supervised setting. CyCLIP introduced cross-modal and in-modal consistency. While the paper does provide some comparisons between CyCLIP and SEMICLIP, it primarily emphasizes SEMICLIP focuses on unlabeled images and surrogate captions. However, this change may not be particularly novel.\n- The experiments are primarily conducted on specific domain datasets. What about the performance on general benchmarks, such as the retrieval benchmarks Flicker30k and COCO, as well as classification on ImageNet?\n\n[1] Huang, Runhui, et al. \"Nlip: Noise-robust language-image pre-training.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 1. 2023.\n\n[2] Goel, Shashank, et al. \"Cyclip: Cyclic contrastive language-image pretraining.\" Advances in Neural Information Processing Systems 35 (2022): 6704-6719.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cYZcmFlHGs", "reviewer": "Reviewer_52fH", "summary": "This paper presents a new semi-supervised training method for vision-language pre-training models like CLIP, called SemiCLIP. The method is designed to improve CLIP's adaptability to downstream tasks when there's limited image-text paired data. SemiCLIP uses a small amount of image-text pairs and a large volume of images without text descriptions to enhance cross-modal alignment. It introduces semantic concept mining to improve visual representations by matching images with relevant concepts from labeled data. The method also creates learnable surrogate captions for unlabeled images and optimizes a trapezoidal consistency to regulate the geometric structure of image-text pairs. The experiments show that SemiCLIP significantly improves CLIP's adaptability, increasing zero-shot classification accuracy by 1.72% - 6.58% and image-text retrieval performance by 2.32% - 3.23% on standard benchmarks.", "questions": "1. Since the paper mainly focus on the CLIP downstream adaptation, I suggest authors change the title from \"SEMI-SUPERVISED CLIP TRAINING\" to \"SEMI-SUPERVISED CLIP ADAPTING\".\n\n2. More explanation towards trapezoidal distance hypothesis. Why it is necessary to restrict the trapezoid to have equal-length legs and diagonals?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper proposes a new method to leverage unlabeled images alongside limited labeled data, enhancing CLIP's cross-modal alignment.\n\n+ Authors design a caption-level trapezoidal consistency loss to appropriately aggregate mined concepts, which is new to me.", "weaknesses": "- The method used to mine semantice concepts had been widely used in semi-supervised works thus is not quite novel to me.\n- Why does the distance between I_i and Tˆ_j should be consistent with the distance between I_j and T_i? The reason for the consistency for diagonals constrains is not clear to me.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GpagzwFPuY", "reviewer": "Reviewer_Psom", "summary": "In this manuscript, the authors focus on efficiently mitigating the domain gap in semi-supervised learning of CLIP on downstream tasks. Specifically, the authors first leverage labeled images to extract the concept list (used as pseudo labels) of the target dataset, and employ CLIP loss and classification loss on pseudo labels for supervised learning. After initialization on labeled set, the authors propose concept-level semantic consistency and caption-level trapezoidal consistency to optimize CLIP on the whole dataset. Experimental results show that the proposed method can effectively", "questions": "For Table 1~4, we recommend  the authors to show the upper bound (supervised learning on all data) to reveal the gap between semi-supervised learning and fully-supervised learning.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The correlation between images and concepts is intuitive, which can be effectively used to optimize CLIP in semi-supervised learning scenarios. \n\n2. The experimental results are promising. \n\n3. The writing is polished and easy to understand.", "weaknesses": "1. My main concern is the analysis of prompting strategy of \\hat{T}_{j}. The position of prompts, the initialization of prompts, or even using fixed prompts rather than learnable prompts can be analyzed. The authors can conduct empirical analysis to investigate which prompting strategy is better. For example, comparing performance with prompts at different positions (e.g. beginning, middle (similar to current configuration), and end of all concepts), different initialization (zero init or normal init for prompt), and fixed vs learnable prompts. All these factors may affect the quality of pseudo captions for unlabeled images. \n\n2. The quality of pseudo labels (i.e., mined semantic concepts) may be critical for semi-supervised learning. The authors could conduct some analysis (e.g., mining concepts with different granularities) to investigate the effect from quality of pseudo labels. For example, \"your current configuration\" vs \"mined concepts but unified into some common objects or attributes\" vs \"oracle fine-grained concepts (e.g., mining concepts from the whole original dataset)\".\n\n3. The authors can also conduct ablation study regarding different percentages of labeled data (e.g. 1%, 5%, 10%, 25%, 50% labeled data) to evaluate the robustness of proposed training framework. Intuitively, a good semi-supervised learning framework may still robust against low percentage labeled data. Nevertheless, in low percentage labeled data settings, the lack of ground-truth caption may restrict the quality of mined concepts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OgJV5IckQx", "reviewer": "Reviewer_JQns", "summary": "The paper proposes a semi-supervised CLIP training method, termed SemiCLIP, which leverages a large amount of unlabeled images when only limited image-text pairs are available. SemiCLIP consists of two training stages: supervised pre-training and semi-supervised fine-tuning. In the supervised pre-training stage, the paper introduces a concept classifier along with the standard contrastive loss to enhance image-text alignment using labeled data. The semi-supervised fine-tuning stage includes two key components: concept-level semantic consistency, which ensures that the model maintains consistency in understanding visual concepts, and caption-level trapezoidal consistency, designed to improve cross-modal alignment by refining the geometric structure between image-text pairs. Experimental results show promising improvements over existing baselines across diverse domains.", "questions": "- Effect of linear classifier: Semantic concept classifier could be performed by CLIP text encoder, akin to zero-shot classification. Is there a specific reason for using linear classifier head instead?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and easy to follow.\n- Experiments are comprehensive.\n- Results show consistent improvements over baselines across diverse domains.", "weaknesses": "The motivation behind caption-level trapezoidal consistency (Section 3.3.2) is not sufficiently explained. \n\n- The surrogate caption is generated by concatenation of templated keywords. Even though the templates are learnable, the distribution of generated captions is likely to differ significantly from the labeled captions. It would be helpful if the underlying motivation for this choice of surrogate caption generation process is explained in detail. (One simple alternative might be generating surrogate captions based on keywords using a large language model.)\n- The main objective of trapezoidal consistency loss is to effectively utilize less-than-ideal surrogate captions due to their inexact nature. However, it is unclear why diagonal and legs regularization is beneficial, whereas the direct usage of contrastive loss is not, given the same inexact caption. It would be useful to explain why trapezoidal consistency is better suited to handle noisy captions. (A simpler alternative could involve using a soft-label in the contrastive loss, specifically for surrogate captions.)\n\nI look forward to authors’ clarification and am willing to increase the score based on their clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)