S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions
Authors: Sangwoo Mo
Minkyu Kim
Kyungmin Lee
Jinwoo Shin
Venue: NeurIPS 2023 poster (2023)
Keywords: vision-language model
semi-supervised learning
specialist domain
TL;DR: We propose a semi-supervised learning method for CLIP, enabling its training on specialist domains with few image-text pairs.

Abstract: Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs.

S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions
Sangwoo Mo 1.2
Minkyu Kim 1.3
Kyungmin Lee'
Jinwoo Shin
KAISS University of Michigan *KRAFTON
Abstract
Vision- Vision-language models, such contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, propose S-CLIP, semi-supervised learning method for training CLIP that utilizes additional unpaired images. S- CLIP employs two labelo- strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs."
Introduction
Pre-trained vision-language models have achieved remarkable success, providing a foundation for numerous downstream tasks [1]. However, these models often struggle when applied to specialized domains such remote sensing or medical imaging
[2]. This because the models are trained on web-crawled data that may not fully capture the diversity and complexity of these domains [3, 4]. To address this issue, previous research has focused on constructing large-scale pre-training datasets for each domain [5, 6]. However, annotating caption for each specialized domain can be expensive and cone-consu,ing, which limits their applicability across various domains.
Several studies have attempted to reduce the number of image-text pairs used for vision-language pre-training. However, these approaches often rely on additional information, such pre-trained object detectors [7-9], or class-annotated images [10], which may not be applicable to specialized domains. Other approaches leverage ceffsuuee learning utilize unpaired data [11-14], but they do not fully exploit the information provided by image-text pairs. Research on leveraging few image-text pairs improve vision-language pre-training remains underexplored.
In various areas of machine learning, particularly in image classification, semi-supervised learning [15] is popular approach for training model using limited annotations. This technique typically relies on pseudo-labeling [16], which predicts labels for unlabeled data by propagating information
'Wrr done at KAIST. Corresponds to: swmo@umich. edu Code: htttp:://githbb cww/almmm/naocccccccc coom/alinaab/scllft
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Nearest image in the batch
Zero-shat accuracy (%)
66.9
on RSICD-OLS
(+8.6)
55.6
58.3
(-2.7)
Their captions are olfferent! (naive paeudi-labeliig falls)
45.3
huge rectangle contains baseball field, a swimming pool, two basketball courts and eight tennis courts"
"two tonnis courts and some buildings are in two sides of a road"
CLIP
CLIP-FT Hard-PL
S-CLIP (ours)
(a) Problem of naive pseudo- labelin (use the nearest caption)
(b) Comparison of models
Figure I: Motivation. (a) Images and captions are from the remote sensing dataset, RSICD. Unlike class labels, captions vary widely across images, making the naive pseudo-labeling of assigning the nearest caption ineffective. (b) Zero-shot accuracy on the RSICD-CLE dataset. Bracket indicates the gap from CLIP fine-tuned on remote sensing datasets, called CLIP-FT. The original CLIP struggles in this specialized domain, and CLIP-FT gives large gain. As discussed in (a), naive pseudo-labeling (Hard-Pl.) harms CLIP-FT, while our S-CLIP provides substantial improvement.
from labeled data and refining the model using them. Combining this approach with self self-supervised learning [17] has significantly improved classifiers using only a few class labels [18-22].
A natural question is whether semi-supervised learning, particularly pseudo-labeling, can improve vision- vision-language pre-training. However, the techniques from image classification cannot be directly applied to vision-language models since captions are diverse and often uniquely associated with each image, Thus, the naive pseudo-labeling approach of assigning the caption of the nearest labeled image² (Hard-PL. [16] can mislead the models. Figure 1a demonstrates that even visually similar images often have different captions, although they may share keywords such as "tennis court." result, Figure 1b shows that Hard-PL harms the performance of CLIP fine-tuning. This observation motivates the proper design of pseudo-labeling method that effectively utilizes captions.
Contribution. We propose S-CLIP, a novel semi-supervised learning method for vision-language pre- training, particularly CLIP [23]. S-CLIP introduces two novel pseudo-labeling methods specifically designed for contrastive learning and the language modality, as illustrated in Figure 2:
Caption-level pseudo-label. We assume that the semantics of an unlabeled image can be expressed as a combination of those of labeled images. To achieve this, we define the caption-level pseudo- label as a probability distribution over the labeled images, which guides the relationship of an unlabeled image with its corresponding captions. Specifically, the pseudo-labels solving an optimal transport [24, 25) problem between the unlabeled and labeled images. This approach prevents pseudo-label from collapsing few captions and ensures robust training, particularly when dealing with distribution shifts between unlabeled and labeled images.
obtained by
Keyword-level pseudo-label. We assume that an unlabeled image shares keywords with visually similar images, even if their full captions are not identical. Therefore, define the keyword-level pseudo-label as one of the keywords in the nearest labeled image to an unlabeled image. This approach creates a candidate set of target keywords instead of single exact one, and the training can be formulated as a partial label learning [26] problem. This loss helps the model understand the local components of unlabeled images, leveraging the structure of language.
We note that both pseudo-labels are complementary. Caption-level pseudo-label helps the model understand the global structure of language, which is more effective for image-text retrieval. Keyword- level pseudo-label helps the model understand the local phrases, which is more effective for zero-shot classification. Combining them achieves the best both worlds (Section 5.5).
We demonstrate the effectiveness of our method in various specialist domains with limited available image-text pairs, including remote sensing [27], fashion [28], scientific figures [29], and comics [30] domains. In the remote sensing domain, S-CLIP outperforms CLIP fine-tuning and semi-supervised learning competitors in five zero-shot classification and six image-text retrieval tasks. For instance, S-CLIP improves zero accuracy WHU-RS19
[31] 19 by 10.4% and image-to-text retrieval R@5
We use the terms "paired" and "labeled" interchangeably, as well as "unpaired" and "unlabeled," depending on the context of vision- lisoualaangugg pre-training semi- ceni-superiised learning.

---

Pseudo-label =
Combination of the captions (from yisually similar images the batch)
Pseudo-label
Candidate keywords (from the caption the nearest image)
0.4 x
0.3 x
huge rectangle
baseball field
E
swimming pool tennis courts
To ...ennii courts..."
T.. buildings..."
(a) Caption-level pseudo-label
(b) Keyword-lewe pseudo-label
Figure 2: Method overview. Conceptual illustration of our proposed S-CLIP, utilizing (a) caption- level and (b) keyword-level pseudo-labels. In (a), the pseudo-label is given by probability distri- bution over the captions, obtained by solving an optimal transport problem between unlabeled and labeled images. In (b), the pseudo-label is given by the candidate set of keywords from the caption of the nearest labeled image, which is learned by the partial label learning algorithm.
on UCM [32] by 4.4% compared to CLIP fine-tuning. S-CLIP remains robust even when unlabeled images are drawn from different dataset. In the comics domain, the Simpsons [30] dataset only includes 800 image-text pairs. Therefore, we incorporate the Simpsons Characters [33] dataset, which consists of 30,000 images without captions. Here, S-CLIP improves the text->image retrieval R of CLIP from 11.8% to 15.8%, providing a relative gain of 33%.
2 Related work
Vision-language pre-training (VLP). Pre-training vision-language models has achieved remarkable success, providing a foundation for many downstream tasks [1]. Numerous approaches have been proposed, including reconstructing masked inputs [34-38], learning joint embedding of vision and language modalities [23, 39-42], generating language descriptions from image inputs [43-45], and connecting pre-trained unimodal image and language models [46-48]. CLIP [23] is a representative model that learns a joint embedding between vision and language modalities. CLIP has demonstrated effectiveness in various applications, including zero-shot classification [23], image-text retrieval [49], open-vocabulary object segmentation [50, 51], and dittr---isttiittion detection [52].
VLP for specialist domains. Despite their success in natural image domains, VLP models often struggle when applied to specialist domains like remote sensing [2]. This is because they are trained on web-crawled data that may not capture the diversity and complexity of these domains [3, 4]. Data in these fields often scarce, has limited accessibility, and requires expert annotations, making difficult to obtain the image-text pairs needed for VLP training. While attempts have been made to train VLP models for specialized domains, including medical [10, 53-59J, fashion [60-63], and remote sensing [64], previous research has mainly focused on constructing large-scale pre-training datasets for each domain [5, 6], which can be expensive and time-consuming.
VLP with limited pairs. Several studies trained VLP models with a limited number of image-text pairs. Hsu al. [65] used adversarial domain adaptation regularize unpaired data embeddings similar to paired data. However, this approach can cause problems when the unpaired data is drawn from different distribution. MedCLIP [10] utilized images with class labels reduce the need for captions. Nevertheless, it still requires class labels, limiting the use of large unlabeled data. Other studies explored unsupervised or semi-supervised VLP training in image domains [7-9]. However, they rely on a pre-trained object detector to align detected objects with keywords in captions. making them unsuitable for specialist domains. Other works leverage sulf-super learning on each modality [-]4, but do not fully exploit the information provided by image-text pairs. In contrast, our method leverages image-text pairs and is generally applicable without extra information.
Semi-supervised learning, Semi-supervised learning [15] aims to train a model using small labeled dataset and large unlabeled dataset. Techniques for semi- cemi-supervised learning can be categorized into two types: pseudo-labeling (or label propagation) [16, 66], which utilizes information from the small labeled dataset, and self-supervised learning [17], which learns representations from the unlabeled dataset. Combining these techniques has achieved remarkable success in semi-supervised learning, particularly for training classifiers with few class labels [18-22]. Prior works on robust sunervuper learning focus scenarios where the distributions of unlabeled and labeled data
3

---

shift [67-69]. This issue naturally arises in vision- languaee pre-training due to the uniqueness of captions for each image. Our method addresses this issue using soft labels and performs robustly even when the unlabeled images are from different dataset than the labeled images.
3 Background
Before we demonstrate our remi-supervised vision- language pre-training method, we provide a brief overview contrastive language-iman -image pre-training and semi- semi-supervised learning.
3.1 Contrastive language-image pre-training
Contrastive language-iman image pre-training (CLIP) [23] is a popular vision-language model that learns joint embedding space connecting images x and texts y.² CLIP is trained using batch of paired images and texts { y: ] through contrastive learning, where those from the same pair are treated as positive and different pairs as negative. The training objective of CLIP involves two classification tasks: predicting a text given an image p(y(r) and predicting an image given a text p(xly) Labels are assigned to the N samples in the batch, corresponding to their pairs the target. The embeddings are normalized to have a unit norm, and CLIP predicts the label using a softmax function with a temperature parameter 7 > O, applied to the cosine similarity of the embeddings.
Specifically, we denote the softmax classifier o- as a function of input and target embeddings. This allows us to represent p(y|rr) = o- (Iz, {ur ) E RN and p(rly) = or {r,);-)) E R which denote the class probabilities of CLIP. These probabilities are calculated as follows: ply = ma(x) =
exp(# /1/ /r/T)/(2)" j=1 exp(æ y//T))- The CLIP model minimizes the following objective:
N
CCLI
Σ 2N
1
(H(p(ulz.)).)) + H (p(z))))...))
(1)
where H denotes the cross-entropy loss and E R is a one-hot vector with the i-th element being one. The CLIP model can be used for zero-shot classification or image-text retrieval by computing cosine similarity in the embedding space. For example, in zero-shot classification, the text embedding of the class prompt that closest to an image embedding can be identified.
3.2 Semi-supervised learning
Semi-supervised learning (Semi-SL) [15] aims to train a model using a small number of labeled data {a,, y/ 1 and large number of unlabeled data { i... when M > N.. Various Semi- SL methods have been proposed to enhance neural network performance, particularly for image classification [19-22].* These methods typically rely on pseudo-labels [16]. obtained by predicting classes for unlabeled data using model. The model is then refined by minimizing the cross-entropy M loss N7 En= H (p(vlu,) 4s) with respect to the prediction p(y|u, E RC and pseudo-label q, E RC for unlabeled data u4 Pseudo-labels can take the form of a hard label, represented by a one-hot vector identifying the class with the highest probability [16], or a soft label that calibrates this hard label [21]. Pseudo-labed minimize prediction entropy by sharpening specific classes [70].
Semi-SI. for CLIP. Assigning pseudo-labels labels in CLIP poses challenges due to the unique captions associated with each image. The assumption that unlabeled data belongs to the same class as labeled data does not hold here. Therefore, we propose novel Semi-SL method for CLIP.
4 S-CLIP: Semi-supervisedd Vision-Language Pre-training
We extend the training of CLIP to incorporate unpaired images {u, MM alongside the image-text pairs (2i, yi V.Y To achieve this, introduce S-CLIP, which integrates two novel pseudo-labeling methods considering contrastive learning and the language modality. The conceptual illustrations of the proposed pseudo-labels and training objectives are shown in Figure and Figure 3, respectively.
We use the notations æ and y to represent image and text embeddings computed by the CLIP encoders. However, for simplicity, we often omit the term "emheddings" when referring to them. In image classification, the targets y are represented by one-hot vectors of size C for C classes.

---

Captions
Captions
Keywords
Y/
>/
Y1
Y/2
Yy
k₁
K2
Paired Images
X1
paars fffu frop
Unpaired images
from OT plan
Unpaired images
14
from keyword set
X2
14-2
14
:
XW
14
One-hot labels
Soft labels
Sparse soft labels
(a) CLIP loss CCLP
(h) Caption loss Leaption
Keyword loss Lieyword
Figure 3: Training objective. Conceptual illustrations of the training objectives of S-CLIP: (a) is the original CLIP loss Eq. (1) using paired images and texts; (b) and (c) are our proposed caption- and keyword-level pseudo-label losses in Eq. (3) and Eq. (4), respectively, applied unpaired images. Caption-level pseudo-labels are soft labels obtained from the optimal transport (OT) plan r, while key keyword-level pseudo-labels are sparse soft labels obtained from the candidate keyword set K..
4.1 Caption-level pseudo-label
Although the semantics of an unlabeled image may not precisely match any captions yr, visually similar images often share similar semantics. Therefore, the caption for unlabeled image u can be expressed as a combination of the captions }ur To achieve this, we use caption-level pseudo-labels, which represent probability distribution over the captions. These pseudo-labels serve as the target for the N-way classification problem in contrastive learning, with batch size of N. Figure 2a demonstrates how caption-level pseudo-label for an unlabeled image obtained.
The pscudo-labels are derived from the relationship between unlabeled and labeled images, formulated an optimal transport (OT) [24] problem. We define the cost function represent the number of unlabeled and labeled images, respectively. The cost is given by the negative cosine similarity between the normalized embeddings of the unlabeled image u, and the labeled image 2j, i.e., Cij = - Zy. Then, we solve following entropic regularized OT [25] problem:
RMx N where M and N
min (T,C) AH(T), T(p, q) = (T E M [ = p. 1 = q)- TEll(p.q)
Here, II(p, q) represents set of transportation plans that satisfy the flow constraint. The sums of flows from sources and to sinks match the vectors p E RM and q E R. respectively, with both p and q adding up to one. We set them as uniform probabilities, and they worked in our experiments, even when unlabeled and labeled data have distribution shifts. IN denotes the all-one vector of dimension Σ.,, Fú log i, with scale À smooths the transportation plan,
N. The entropic regularizer H (T) enabling an efficient solution using the Sinkhorn Knopp [25] algorithm.
Once we obtain the transportation plan r, we define the pscudo-label for an unlabeled image u, by normalizing its plan sum up one. Specifically, the j-th element of the pseudo-label & R is computed as / Ej Tij- This pseudo-label is then utilized to guide the probability distribution over N captions, where the prediction p(y) (4) E R is determined by softmax classifier o, (u,, {y, ) The loss function for the caption-leve pseudo-label aims to minimize the following:
Ceaption
H(plylus) 9:).
M
The prediction p(yl (4) relates the unlabeled image u; to the text embeddings while the pseudo- label 9 relates the image embeddings x. Consequently, the proposed loss function aligns the relationships between image and text embeddings, suggested in supervised CLIP [7 [71-73]. In our experiments, we compare the usage of text and image embeddings for pseudo-label computation, and using image embeddings outperformed using text embeddings due to their robustness in novel specialist domains. We also explore computing pseudo-embeddings [9], but yields inferior results synthesizing pseudo-embeddings is more challenging than inferring pseudo-labels.
5

---

Effect of OT. Optimal transport balances the flow between unlabeled and labeled images, ensuring robust pseudo-label assignments without concentration on a few labels. In vision-langued tasks, diverse and imbalanced captions often cause image embeddings to focus on a few nearest captions, resulting in the collapse of embeddings. This challenge is magnified when employing semi-supervised learning with distribution shifts [67]. We empirically validate the detrimental effect of naive pseudo- labeling approaches, especially in the presence of distribution shifts.
Compared to soft-nearest neighbor pseudo-labels (Soft-PL) [21], our method achieves balance through Sinkhorn iterations. Specifically, reverts to Soft-PL when the iteration count is zero. In our experiments, we use 10 iterations, and increasing the iteration count has only a marginal impact on performance. The computational cost is negligible as the process occurs in a -iw-demnsiioal embedding space. We set the scale of the entropic regularizer to match the softmax temperature, т. This adjustment scales the cost function by 1/T, aligning with the cosine similarity scale learned by CLIP. More information on the OT and Soft-PI. relationship is in Appendix A.
OT for vision-language models. Several works have utilized OT for vision- -angunge models, but for problems other than semi-supervised learning. We discuss these works in Appendix B.
4.2 Keyword-level pseudo-label
We propose a keyword-level pseudo-labeling approach to overcome the limitations of caption-level pseudo-labels in capturing the meanings of words in captions. By leveraging the compositionality of language, we assume that visually similar images may share keywords, even if the entire captions differ. This enables us to guide unlabeled images using keywords from their nearest neighbors. It is important to note that an image may have multiple keywords, and visually similar images share only a subset. This aligns with the partial label learning (PLL) [26] problem, where the target label is unknown, and we have access to candidate shares the keyword "tennis court" with its nearest neighbor but not "baseball field." Consequently, the nearest neighbor provides a candidate set of keywords, as illustrated in Figure 2b.
of labels. For instance, in Figure 1a, the image
To identify candidate keywords for an unlabeled image u, we assume a pre-defined set of keywords These keywords can be obtained from class names, if
with corresponding embeddings K } 1. available, or extracted by using algorithms like YAKE [74] applied to captions. Given an unlabeled image u, we find the nearest labeled image using the OT assignments from the previous section. Let (ci...., q be the indices of keywords present in labeled image x, denoted as K,, = .ke....... .....,..) }. This forms the candidate set of target keywords for u. While we simply use the keywords explicitly from the caption, some additional steps can be taken to infer synonyms or related concepts.
In the context of partial label learning, we assume that the ground-truth target label for the unlabeled image u belongs to the candidate set Ky. PLL aims to minimize the loss by selecting the minimum value among candidate labels, minkek, H (p(k) lu), k), rather than specifying a single target label. However, relying on a single minimum can be risky. Previous works have addressed this by using soft labels for candidates [75, 76). This involves minimizing H (p(k (u), 9), where q E RK is a soft label with positive values for keywords in the candidate set and zero for others.
Similarly, we define the pseudo-label y for the unlabeled image u, based on the similarity between the embeddings of u, and the keywords kj in the candidate set K,, Specifically, the j-th element of the pseudo-label qi is calculated using a softmax function a o-(u;, K.,) for k E K,, and is set to zero otherwise. This pseudo-label is then utilized to guide the probability distribution over K keywords, where the prediction p(k])u) E R is determined by softmax classifier a, (u,, K). The loss function for the keyword-Level pseudo-label aims to minimize the following:
M
1 M
Lkeywordd
H(p(klu:)
Both the prediction p(k]u,,) and the pseudo-label qi are related to the unlabeled image and keywords. However, the restricted candidate set effectively guides the learning process through PLL.
Training objective. We apply the CLIP loss to paired images and texts and the proposed pseudo-label losses to unpaired images. The pseudo-label losses are halved to match the scale of the image part in the CLIP loss. In summary, our final training objective becomes LCLP + 1/2 (Eeaption + Lkeyward).

---

Table 1: Zero-shot classification results remote sensing datasets. We compare the original CLIP, supervised CLIP fine-tuned on labeled data (L), and semi-supervised methods that utilize unlabeled data sampled from the same (L=U) or different (L=U) distribution as the labeled data. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one. Bolds denote the best results among the semi-supervised methods within the
same setups. S-CLIP consistently improves zero-shot accuracy, even when the unlabeled data has a distribution shift. In contrast, naive pseudo-labeling often harms the accuracy.
Method
Data
RSICD-CLS
UCM-CLS
WHU-RS19
RSSCN7
AID
CLIP (original)
45,3
50.5
65.5
58.9
47.8
CLIP (fine-tune)
L.
58.3203
63.5gM
76.5232
61.9212
63.1
Hard-PL
56.6=3. (-1.7)
61.6#22 (-1.9)
78.1 (+1.6)
63.9+21 (+2.0)
63.2+26 (40.1)
Soft-PL [21]
L=U
62.510x (+4.2)
65.7 (+2.2)
83.7 (+7.2)
65.7.06 (+3.8)
68.0.o0
(+4.9)
S-CLIP (ours)
66.9217 (+8.6)
66.7216 (+3.2)
86.9220 (+10.4)
66.2211 (+4.3)
73.0 (+9.9)
Hard-PL. [16]
55.821a (+2.5)
61.0262 (-2.5)
76.3211 (-0:2)
62.5 (+0.6)
62.3 (-0.8)
Soft-PL [21]
L
56.1 (-2.2)
62.4 (-1.1)
79.5 (+3.0)
63,3+26 (+1.4)
62.4a21 (-0.7)
S-CLIP (ours)
65.3121 (+7.0)
66.0 (+2.5)
86.3 (+9.8)
68.3 (+6.4)
70.8 (+7.7)
5
Experiments
We demonstrate the effectiveness of our method in various specialist domains with limited available image-text pairs, including remote sensing, fashion, scientific figures, and comics domains.
Setup. We use a pre-trained CLIP model on natural images [23], called CLIP (original). However, this model often struggles in specialist domains. Therefore, we fine-tune the model with domain-spectioo image-text pairs, called CLIP (fine-tune). We compare this supervised model with semi- cupi-super methods using unpaired images. We use a batch size of 64 per GPU, with total of 4 GPUs. To ensure fair GPU memory usage in semi-supervised learning, we employ 32 image-caption pairs and 32 unpaired images for each mini-batch. Models are evaluated on zero-shot classification and image-text retrieval tasks, measuring Top-1 classification accuracy (%) and recall at K (R@K, K). We report the average and standard deviation across three random seeds. We follow the training recipe of OpenClip [77] if not specified. Additional experimental details are in Appendix C.
5.1 Baselines
Our proposed S-CLIP includes two novel pseudo-labeling methods. Thus, we mainly compare S- CLIP with other pseudo-labeling approaches, which are extensions of semi-supervised classification techniques. Our baselines, Hard-PL and Soft-PL, extend the hard (top-1) and soft nearest neighbor approaches for pseudo-label assignment. We then utilize these methods as caption-level pestiooo pseudo-labels, similar Eq. (3). The specific forms of each method are as follows.
Hard-PL. Given an unlabeled image u4 Hard-PL identifies the top-1 nearest labeled image" de- based their cosine similarities, calculated i* arg max, (w, 22y) Subsequently, the pseudo- label of u; is determined by the corresponding caption, denoted as qi = yy-. This pseudo-label serves as the target for predicting (.), and the loss is computed as Σ, (p(s)uu:)
PL. Soft-PL calibrates the pseudo-label of Hard-PL by considering all the relations between labeled and unlabeled images, known soft nearest neighbor [21], instead of selecting only the top-1 nearest image. For an unlabeled image u, and a set of labeled images )z,). these relations determined by applying softmax to their cosine similarities. The pseudo-labe is defined q, or (uz, {z, AN E RV where the pscudo-label has a higher value close to one for visually similar images. We use the same softmax temperature 7 as CLIP, which is used to compute the similarities between the labeled image and text embeddings. The loss is computed as Σ, H (p(ylu.),
5.2 Remote sensing datasets
We train vision-language models using the union of RSICD [27], UCM [32], and Sydney [78], named RS-ALL, following the setup of [64]. For supe-supe learning, we subsample 10% of image-text
'We also tested the version directly finding the nearest text for both Hard- PL and Soft-PL. However, the similarities in visual embeddings give more robust results, as discussed in the ablation study.
7

---

Table 2: Image-text retrieval results remote sensing datasets, following the same setup of Table 1. Bolds denote the best results among the semi-supervised methods within the same setups. S-CLIP performs best in most cases, except the supervised baseline works best for text text->image retrieval on the Sydney dataset. Naive pseudo-labeling also helps. but the gain is often unstable.
Image Mext R@5
Text vimgge R@5
Method
Data
RSICD
UCM
Sydney
RSICD
UCM
Sydney
CLIP (original)
9.4
34.3
36.2
10.1
24.8
51.7
CLIP (fine-tune)
15.4
41.3
47.1
15.1
10.9
56.1
Hard-PL [16]
16.1+
40.8++29
43.1++30
15.7403
40,5+30
47.7+5.3
Soft-PL [21]
L=U
17.0++09
43.2+339
42,0+4.3
16,5+a
42.9+33
50.2+4.9
S-CLIP (ours)
18.4
45.7
50.0
16,8
43.5
55.1
Hard-PL. [16]
15.8
42.1
47.1
15.3404
42.1
50.0=17
Soft-PL [21]
L
16.5+111
40.2:13.2
46.6144
15,4102
40.8:0.5
54.0
S-CLIP (ours)
17.1:08
43.5=335
48.9
15.8+07
42.5=11
$2.3=1.0
pairs as labeled data (L), while the remaining 90% of images (L-UU) or unlabeled images from the RESISC45 [79] dataset (LzJU) served as unlabeled data. In Appendix D, we demonstrate that our method consistently gives improvements over CLIP across various setups. These include different neural architectures, varied ratios of image-text pairs, and even scenarios where we use all image-text pairs from RS-ALL in conjunction with unlabeled images from RESISC45.
For zero-shot classification, we use the validation sets of the classification versions of the RSICD and UCM datasets, denoted as RSCID-CLS and UCM-CLS, respectively. To evaluate the generalization abilities, we test models on unseen datasets such as WHU-RS19 [31], RSSCN7 [80], and AID [81]. For image-text retrieval, we use the validation sets of the RSICD, UCM, and Sydney datasets.
Zero-shot classification. Table 1 presents the zero-shot classification results. S-CLIP consistently and significantly outperforms all supervised CLIP fine-tuning and semi-supervised methods. One can make several observations. Firstly, fine-tuning CLIP improves performance over the original CLIP, highlighting the importance of adapting models to the target specialist domain. Secondly, our proposed pseudo-labeling techniques are crucial, as naive pseudo-labeling often harms accuracy. Thirdly, semi-supervised learning methods improve performance on unseen datasets, such as RSSCN7, demonstrating the usefulness of observing more unlabeled images in generalization. Lastly, our method is robust to the distribution shifts (L=UU) setup, which leverages an external source of unlabeled images, a common scenario practice. In contrast, Soft-PL improves performance in the same distribution (L=U) setup but fails on the distribution shifts scenario, confirming the effectiveness of optimal transport in balancing the assignments of pseudo-labels.
Image-text retrieval. Table 2 presents the image-text retrieval results. S-CLIP consistently improves both image->txt and text-vimage retrieval, except in one case where the supervised baseline performs the best. This confirms that S-CLIP also benefits fine language understanding for retrieval. The trend is consistent across evaluation metrics, seen the R@ I results in the appendix.
5.3 Fashion datasets
We train vasion-language models using the union of Fashion200k [28), FashionGen [82], and Polyvore Outfits [83] datasets. We subsample 10% of the image-text pairs as labeled data and the remaining 90% of images as unlabeled data. We evaluate the zero-shot accuracy on the validation sets of all three datasets, considering both super-class and sub-class accuracies for Fashion200k and FashionGen. The class names in Polyvore match the same level of super-class as those in the other datasets.
Table 3 presents the zero-shot classification results. S-CLIP consistently outperforms all supervised and sumi-super methods. The improvement is more significant for super-class classification but less pronounced for sub-class classification and image-text retrieval. This is because the current leeudo-labeling approach assumes that the semanties of unlabeled image can be represented by the captions in a batch. However, this assumption may not hold true for fine-grained semantics, as certain fine- grained captions may be missing from the batch. Therefore, our method could further enhanced by increasing the batch size or incorporating queue of caption embeddings [84, 85].
8

---

Table 3: Zero-shot classification results fashion datasets. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one. Bolds denote the best results. S-CLIP consistently outperforms all supervised and semi-supervised methods.
Fashion200k
FashionGen
Polyvore
Method
Super-elass
Sub-class
Super-class
Sub-class
Class
CLIP (original)
73.4
29.3
35.9
22.1
58.4
CLIP (fine-tune)
79.4
38.4
41.5
33.1
64.2=3.
Hard-PL [16]
74.6#27 (-4.8)
35.0#26 (-3.4)
36.6439 (-4.9)
28.8+0a (-4.3)
65.5-u7 (+1.3)
Soft-PL. [21]
80.0+11 (40.6)
37.1 (-1.3)
45.5+32 (+4.0)
33,2=00 (40.1)
73.6=19 (+9.4)
S-CLIP (ours)
82.0+28 (+2.6)
39,5 (+1.1)
55.3 (+13.8)
38.7 13.7=19 (45.6)
74.6 (+10.4)
Table Image-text retrieval results on SciCap (scientific figures) and Simpsons (comies) datasets. We train the model on each dataset and evaluate on their validation sets. Bolds denote the best results. S-CLIP consistently outperforms all supervised and semi-supervised methods.
SciCap
Simpsons
Image- ttxt
Text timaee
Image -text
Text=
Method
@
R@5
R@1
R
R@1
R@5
R 1
R@5
CLIP (original)
9.2
14.7
9.0
14.5
15.8
40.8
10.5
27.6
CLIP (fine-tune)
10.5
17.9
10.2:0.1
17.2
16.7+20
41.41111
11.8+23
41.7=15
Hard-PL [16]
9.7
16.8++0.6
9.2+06
16.2.a07
15.4+1.5
41.7+33
12.7+115
40.8=23
SoftPPL [21]
9.1
15.5+110
9.7+003
16.8405
15.4+22
39.9+27
12.9+19
36.4=27
S-CLIP (ours)
10.9
18.8+0.
11.1:03
18.8
18.4+2.
43.9415
15,8+11
42.1=01
5.4 More captioning datasets
We conduct experiments two specialist domains, namely science figures and comics. For this purpose, we use the SciCap [29] and Simpsons [30] datasets. We train the models on each dataset and evaluate their image-text retrieval performance the validation sets. The results presented in Table 4, and detailed discussions can be found the following paragraphs.
SciCap. We subsample 10% of the image-text pairs as labeled data and the remaining 90% of images as unlabeled data. S-CLIP improves upon CLIP, even for the scientific figures domain, which exhibits substantial gap from the natural image domain that CLIP is pre-trained on. This confirms the practical impact of S-CLIP in extending the applicability of CLIP to various specialist domains.
Simpsons. The Simpsons dataset contains only 800 image-text pairs, so we incorporate unlabeled images from another dataset called Simpsons Characters [33]. S-CLIP significantly improves CLIP, even in the presence of distribution shifts between unlabeled and labeled images. For example, it boosts the text=>image retrieval @@1 from 11.8% to 15.8% (+33%). This confirms the practical impact of S-CLIP in specialist domains with limited available image-text pairs.
5.5 Ablation studies
We conduct an ablation study on the design choices for S-CLIP. The results are reported in Table 5, which compares the average values of zero-shot accuracy (across five tasks in Table 1) and image-text retrieval R @ (across six tasks in Table 2) for each design. The observations are as follows:
(a) Training objective. Caption-level pseudo-labels are more beneficial for image-text retrieval, while keyword-level pseudo-labels are better for zero-shot classification. Intuitively, keyword-level pseudo-labels help the model understand specific words, while caption-level pseudo-labels aid in fine-grained language comprehension. Combining both produces the best results.
(b) Sinkhorn iteration. Sinkhorn iterations significantly improve Soft-PL- (= Iter. 0) by balancing pseudo-labels. However, saturates after 10 iterations, so we have used it as the default in our experiments. Increasing the number of iterations ensures more even distribution of pseudo- labels. For example, the pseudo-labels from Soft-PI. often collapse to some index in the batch. Sinkhorn iterations effectively regularize this behavior, reducing the maximum confidence of

---

Table 5: Ablation study on design choices. We report the average values of zero-shot accuracy (left) and image+>text retrieval (right). Bolds denote the best results. See Section 5.5 for discussions.
(a) Training objective
Sinkhorn iteration
(c) Choice of keywords
CLIP (fine-tune)
64.7
Iter. 70.6
9.2
YAKE-100
70.0
9.8
Caption-level
69.6
10.5
Iter.
71.4
9.4
YAKE-200
72.2
9.3
Keyword-level
70.0
9.9
Iter.
71.6
10.2
YAKE-300
70,6
10.2
Both
71.9
10.6
Iter. 10
71.9
10.6
Class name
71.9
10.6
(d) Pscudo-target
(c) Embedding for OT
(f) Partial label loss
Pscudo-embed 68.8
9.2
Text
71.6
9.7
Hard-max 66.1
10.2
Pseudo-label
71.9
10.6
Image
71.9
10.6
Soft-max 71.9
10.6
prediction from 100% to 40%. This regularization is more important in the LAU setup, where Soft-PL often harms the performance, while ours consistently provides improvement.
(c) Choice of keywords. We compare class names and YAKE-K, which implies that K keywords are extracted using the YAKE [74] algorithm. Both types of keywords work well, although class names provide more consistent improvement. YAKE provides both nouns, such "building," and adjectíves like "green," "many," or "large," in contrast to the class names, which consist only of nouns. This enables YAKE to understand diverse semantics beyond object recognition, making keyword-level pseudo-labeling applicable to any domain. This applicability is further confirmed on the SciCap and Simpsons datasets, where class names are unavailable.
(d) Pseudo-target. Given pscudo-label qi E IR-V for unlabeled image u4, one synthesize a pseudo-embedding 2; Σ, 9i Y/ E Rd, where di denotes the j-th element of qı and is the dimension of the embeddings V/- Then, one can consider {u:, = as paired data and apply contrastive learning to it in addition the true pairs {a,, yi However, we found that pseudo-embedding! is less effective than pseudo-labeling. This is because accurately estimating the pseudo-embedding is harder than assigning soft probabilities the true embeddings. Additionally, the pseudo-embeddings lie on an interpolation of true embeddings, which can introduce confusing negatives and potentially harm the effectiveness of contrastive learning.
(e) Embedding for OT. We compute pseudo-labels by examining the relationship between unlabeled images and either labeled images or labeled text directly. The relationship with labeled images proves be more effective in assigning pseudo-labels. Specifically, using text embeddings directly also works reasonably well in the L-U setup. However, often fails entirely in the L-AU setup. This is because the corresponding captions for unlabeled images unknown. In contrast, visual similarities can be robustly computed even when the images are unseen.
(f) Partial label loss. We the soft label given by a (ug, K.) as the target for PLL. We also the hard version, using the top-1 similar keyword from the candidate set, i.e., i* = arg max, (1: kg). However, performs worse, aligning with the observations from prior PLL literature.
We provide additional ablation studies in Appendix E, including results in the general image domain, additional pseudo-labelin baselines, qualitative examples of pseudo-labels, training curves, regu- larized fine-tur tuning approaches, selection of mini-batcher during training, additional baselines using keyword information, and statistics of keywords in captions.
6 Conclusion
We propose S-CLIP, a semi- semi-supervised extension of CLIP that leverages unpaired images to train in specialist domains with limited image-text pairs. We demonstrate its superiority across various domains such as remote sensing, fashion, scientific figures, and comics. We hope that S-CLIP expands the applicability of CLIP. Limitations and broader impacts are discussed in Appendix F.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial
10

---

Intelligence Graduate School Program (KAIST); No.2021-0-02000 Artificial Intelligence Innovation Hub; No.2022-0-00959 Few-shot Learning of Casual Inference in Vision and Language for Decision Making).
References
[1] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108 07258, 2021.
[2] Y. Kim, S. Mo, M. Kim, K. Lee, J. Lee, and J. Shin. Bias-to-text: Debiasing unknown visual biases through language interpretation. arXiv preprint arXiv:2301. 104, 2023.
[3] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-tex dataset for automatic image captioning. In Annual Conference of the Association for Computational Linguistics, 2018.
[4] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:22 10. 08402, 2022.
[5] A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, G. Mark, and S. Horn. Mimic-exr, de-identified publicly available database of chest radiographs with free-text reports. Nature scientific data, 2019.
[6] X. Yang, H. Zhang, D. Jin, Y. Liu, C.-H. Wu, J. Tan, D. Xie, J. Wang, and X. Wang. Fashion captioning: Towards generating accurate descriptions with semantic rewards. In European Conference on Computer Vision, 2020.
[7] L. H. Li, H. You, Z. Wang, A. Zareian, S.-F. Chang, and K.-W. Chang. Unsupervised vision- and-language pre-training without parallel images and captions. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
[8] M. Zhou, L. Yu, A. Singh, M. Wang, Z. Yu, and N. Zhang. Unsupervised vision-and ann-lang pre-training via retrieval-based multi-granular aligmment. In Conference on Computer Vision and Puttern Recognition, 2022.
[9] T.-. J. Wang, Laaksonen, T. Langer, H. Arponen, and T. E. Bishop. Learning by hal- lucinating: Vision-language pre-training with weak supervision. In Winter Conference on Applications of Computer Vision, 2023.
[10] Z. Wang, Z. Wu, D. Agarwal, and J. Sun. Medelip: Contrastive learning from umpaired medical images and text. In Conference on Empirical Methods in Natural Language Processing, 2022.
[1I] Li, F. Liang, L. Zhao, Y. Cui,
Ouyang, J. Shao, F. Yu, and J. Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations, 2022.
[12] N. Mu, A. Kirillov, D. Wagner, and S. Xie. Slip: Self-supervision meets language- pre-training. In European Conference on Computer Vision. 2022.
[13] E. Segal, B. Bogin, and J. Berant. Training vision-language models with less bimodal supervision. In Automated Knowledge Base Construction, 2022.
[14] R. Windsor, A. Jamaludin, T. Kadir, and A. Zisserman. Vision- languae modelling for radiological imaging and reports in the low data regime. arXiv preprint arXiv:2303.1764 2023.
[15] O. Chapelle, B. Scholkopf, and A. Zien. Semi-supervised learning. 2006. Cambridge, Mussachusettes: The MIT Press View Article, 2, 2006.
[16] D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, 2013.
[17] X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang. J. Zhang, and Tang, Self-supervised learning: Generative contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
[18] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch:
A holistic approach to semi-supervised learning. In Neural Information Processing Systems, 2019.
11

---

[19] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi- semi-supervised learning with consistency and confidence. In Neural Information Processing Systems, 2020.
[20] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Conference on Computer Vision and Pattern Recognition, 2020.
[21] M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat. Semi- supervised learning of visual features by pon-parmmetrically predicting view assignments with support samples. In International Conference on Computer Vision, 2021.
[22] M. Bošnjak, P. H. Richemond, N. Tomasev, F. Strub, J. C. Walker, F. Hill, L. H. Buesing, R. Pas- canu, C. Blundell, and J. Mitrovic. Semppl: Predicting pscudo-labels for better contrastive representations. In International Conference on Learning Representations, 2023.
[23] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervi- sion. In International Conference on Machine Learming, 2021.
[24] C. Villani et al. Optimal transport: old and new. Springer, 2009.
[25] M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Neural Information Processing Systems, 2013.
[26] T. Cour, B. Sapp, and B. Taskar. Learning from partial labels. Journal of Machine Learning Research, 2011.
[27] X. Lu, B. Wang, X. Zheng, and X. Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing, 2017.
[28] X. Han, Z. Wu, P. X. Huang, X. Zhang, M. Zhu, Y. Li, Y. Zhao, and L. S. Davis, Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision, pages 1463-1471, 2017.
[29] T.-Y. Hsu, C. L. Giles, and T.-H. Huang. Scicap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP, 2021.
[30] D. Adler. Simpsons blip captions. https://hugginggace.u htt.://aac...ttttttttttttrrrrrrr
-impsons- captions, 2023.
[31] G.-S. Xia, W. Yang. J. Delon, Y. Gousseau, H. Sun, and H. Maître, Structural high-resolution satellite image indexing. 2009.
[32] Y. Yang and S. Newsam. Bog-of-visual-words and spatial extensions for land-use classification. In SIGSPATIAL international conference on advances in geographic information systems, 2010.
[33] A. Attia. The simpsons characters data. https: https://wwwr http://wwwwwgl...... wtwggge.coo/daaaeeeeeee com/datasets/ seeettiatthesiipsooee wharssaaaaaaateedattaeeeeee 2018.
[34] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. In Annual Conference of the Association for Computational Linguistics, 2020.
[35] Y.-C. Chen, I.. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision, 2020.
[36] W. Kim, B. Son, and I. Kim. Vilt: Vision-andd-anguuage transformer without convolution or region supervision. In International Conference on Machine Learning. 2021.
[37] Z. Wang, Yu, A. W. Yu, Dai, Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint wrXiv:2108.1004, 2021.
[38] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022.
[39] Jia, Y. Yang, Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Q Le, Y.H.. Sung, Z. Li, and T. Duerig. Scaling up visual and lasion-language representation learning with noisy text supervision. In International Conference on Machine Learning, 2021.
12

---

[40] H. Pham, Z. Dai, G. Ghiasi, H. Liu, A. W. Yu, M.-T. Luong, M. Tan, and Q. V. Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050 2021.
[41] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Conference Computer Vision and Puttern Recognition, 2022.
[42] Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. arXiv preprint arXiv:2212.00779) 2022.
[43] K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In Conference on Computer Vision and Pattern Recognition, 2021.
[44] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping iaggagee-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, 2022.
[45] J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. Transactions Machine Learning Research, 2022.
[46] M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and Hill. Multimodal few-shot learning with frozen language models. In Neural Information Processing Systems, 2021.
[47] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Milli- can, M. Reynolds, et al. Flamingo: visual language model for few-shot learning. In Neural Information Processing Systems, 2022.
[48] J. Li, D. Li, S. Savarese, and Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[49] H. Luo, L.. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. ClipAclip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing. 508:293-304 2022.
[50] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022.
[51] M. Xu, Z. Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai. A simple baseline for open- vocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision, 2022.
[52] J. Jeong, Y. Zou, T. Kim, D. Zhang, A. Ravichandran, and O. Dabeer. Winclip: Zero-/few-shot anomaly classification and segmentation. In Conference on Computer Vision and Pattern Recognition, 2023.
[53] X. Wang, Y. Peng, Lu, Z. Lu, and R. M. Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Conference on Computer Vision and Pattern Recognition, 2018.
[54] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference, 2022.
[55] S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung. Gloria: A multimodal global-local repre- sentation learning framework for label-efficient medical image recognition. In International Conference on Computer Vision, 2021.
[56] R. Liao, D. Moyer, M. Cha, K. Quigley, S. Berkowitz, S. Horng, P. Golland, and W. M. Wells. Multimodal representation learning via maximization of local mutual information. In Medical Image Computing and Computer Assisted Intervention, 2021.
[57] B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, Naumann, A. Nori, Alvarez- Valle, et al. Making the most of text semantics to improve biomedical vasion-language processing. In European Conference Computer Vision, 2022.
[58] P. Müller, G. Kaissis, C. Zou, and Rueckert. Joint learning of localized representations from medical images and reports. In European Conference Computer Vision 2022.
13

---

[59] S. Zhang, Y. Xu, N. Usuyama. J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong, et al. Large-scale domain specifii pretraining for biomedical vision-language processing, arXiv preprint arXiv:2303 00915, 2023.
[60] D. Gao, Jin, B. Chen, M. Qiu, P. Li, Y. Wei, Y. Hu, and H. Wang. Fashionbert: Text and image matching with adaptive loss for eross-modal retrieval. In International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020.
[61] M. Zhuge, D. Gao, D.-P. Fan, L. Jin, B. Chen, H. Zhou, M. Qiu, and L. Shao. Kaleido-bert: Vision-language pre-training on fashion domain. In Conference on Computer Vision and Pattern Recognition, 2021.
[62] S. Goenka, Z. Zheng, A. Jaiswal, R. Chada, Y. Wu, V. Hedau, and P. Natarajan. Fashionvip: Vision language transformer for fashion retrieval with feedback. In Conference on Computer Vision and Pattern Recognition, 2022.
[63] X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang. Fashionvil: Fashion-focused vision-and-language representation learning. In European Conference on Compuler Vision, 2022.
[64] A. Arampacha, V. Dev, G. Venkatesh, B. Mayank, G. Ritobrata, and P. Sujit. Fine tuning clip with remote sensing (satellite) images and captions. https://huugingffe co/blog/
clip-rsicd, 2021.
[65] T.-M. H. Hsu. W.-H. Weng, W. Boag, M. McDermott, and P. Szolovits. Unsupervised multimodal representation learning across medical images and reports. In NeurlPS Machine Learning for Health Workshop, 2018.
[66] Y. Bengio, O. Delalleau, and N. Le Roux. label propagation and quadratic criterion. 2006.
[67] A. Oliver, Odena, C. Raffel, E. D. Cubuk, and Goodfellow. Realistic evaluation of deep sepi-supereised learning algorithms. In Neural Information Processing Systems, 2018.
[68] J. Park, S. Yun, J. Jeong, and J. Shin. Opencos: Contrastive semi supervised learning for handling open-set unlabeled data. arXiv preprint arXiv:2 2021.
[69] S. Mo, J.-C. Su, C.-Y. Ma, M. Assran, Misra, L. Yu, and S. Bell. Ropaws: Robust semi- supervised representation learning from uncurated data. In International Conference on Learning Representations, 2023.
[70] Y. Grandvalet and Y. Bengio. superiuper learning by entropy minimization. In Neural Information Processing Systems, 2004.
[71] J. Zhou, Dong, Z. Gan, L. Wang, and F. Wei. Non-contrastive learning meets language- immge pre-training arXiv preprint wxXv:2210000934. 2022.
[72] S. Goel, H. Bansal, S. Bhatia, R. Rossi, V. Vinay, and A. Grover. Cyclip: Cyclic contrastive
language-image pretraining. In Neural Information Processing Systems, 2022.
[73] Y. Gao, J. Liu, 7. Xu, T. Wu, W. Liu, J. Yang, K. Li, and X. Sun. Softelip: Softer cross- alignment makes clip stronger. arXiv preprint arXiv:2303. 17561, 2023.
[74] R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, C. Nunes, and A. Jatowt. Yake! keyword extraction from single documents using multiple local features. Information Sciences, 509: 257-289, 2020.
[75] Lv, M. Xu, L. Feng, G. Niu, X. Geng, and M. Sugiyama. Progressive identification of true labels for partial-label learning. In International Conference on Machine Learning, 2020.
[76] H. Wang, R. Xiao, Y. Li, L. Feng, G. Niu, G. Chen, and Zhao. Pico: Contrastive label disam- biguation for partial label learning, In International Conference on Learning Representations, 2022.
[77] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, 2021.
[78] F. Zhang, B. Du, and L. Zhang. Saliency-guide unsupervised feature learning for scene classification. IEEE transactions on Geoscience and Remote Sensing, 2014.
[79] G. Cheng, Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art Proceedings of the IEEE, 2017.
14

---

[80] Q. Zou, L. Ni, T. Zhang, and Q. Wang. Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters, 2015.
[81] G.-S. Xia, Hu, F. Hu, Shi, X. Bai, Y. Zhong, L. Zhang, and X. Lu. Aid: benchmark data set for performance evaluation of acrial scene classification. IEEE Transactions on Geoscience and Remote Sensing, 2017.
[82] N. Rostamzadch, S. Hosseini, T. Boquet, W. Stokowiee. Y. Zhang, C. Jauvin, and C. Pal. Fashion-gen: The generative fashion dataset and challenge. arXiv preprint 0rXiv:1806.083/ 2018.
[83] M. I. Vasileva, B. A. Plummer, K. Dusad, S. Rajpal, R. Kumar, and D. Forsyth. Learning type-aware embeddings for fashion compatibility, In Proceedings of the European conference on computer vision (ECCV), pages 390-405, 2018.
[84] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, 2020.
[85] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Puttern Recognition, 2020.
[86] S. Yuan, K. Bai, L. Chen, Y. Zhang, C. Tao, C. Li, G. Wang, R. Henao, and L. Carin. Weakly supervised cross-domain alignment with optimal transport. In British Machine Vision Conference, 2020.
[87] G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang. Prompt learning with optimal transport for lision-anguage models. In International Conference on Learning Representations, 2023.
[88] K. Kim, Y. Oh, and C. Ye. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprint arXiv:2301 12171, 2023.
[89] T. Gupta, A. Vahdat, G. Chechik, X. Yang. J. Kautz, and D. Hoiem. Contrastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision, 2020.
[90] B. Wu, Cheng, P. Zhang. P. Vajda, and J. E. Gonzalez. Data efficient language-supervised zero-shot recognition with optimal transport distillation. In International Conference on Learning Representations, 2022.
[91] F. Taherkhani, Dabouei, S. Soleymani, Dawson, and N. M. Nasrabadi. Transporting labels via hicrarchical optimal transport for semi-supervised learning, In European Conference on Computer Vision, 2020.
[92] F. Taherkhani, A. Dabouei, S. Soleymani, J. Dawson, and N. M. Nasrabadi. Self-supervised wasserstein pscudo-labeling for semi-supervised image classification. In Conference on Computer Vision and Pattern Recognition, 2021.
[93] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With little help from my friends: Nearest-neighbo contrastive learning of visual representations. In International Conference on Computer Vision, 2021.
[94] M. Zheng, F. Wang, S. You, C. Qian, C. Zhang, X. Wang, and C. Xu. Weakly supervised contrastive learning. In International Conference on Computer Vision, 2021.
[95] M. Azabou, M. G. Azar, R. Liu, C.-H. Lin, E. C. Johnson, K. Bhaskaran- -air, M. Dabagia, B. Avila- L. Kitchell, K. B. Hengen, al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2102. 10106, 2021.
[96] P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and
D. Krishnan. Supervised contrastive learning. In Neural Information Processing Systems, 2020.
[97] M. Assran, N. Ballas, L. Castrejon, and M. Rabbat. Supervision accelerates pre-training in con- trastive semi-supervised learning of visual representations. arXiv preprint 1000003006.10803, 2020.
[98] B. Qu, X. Li, D. Tao, and X. Lu. Deep semantic understanding of high resolution remote semsing image. In International conference computer, information and telecommunication systems, 2016.
[99] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition, 2016.
15

---

[100] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- highami, M. Minderer, G. Heigold. S. Gelly, el. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[101] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L.. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
16

---

A Relationship between OT and Soft-PL
The entropic-regularized optimal transport problem described in Eq. (2) can be efficiently solved using the Sinkhorn-Knon Knopp [25] algorithm. For our case, the cost function C E MR is given by the negative cosine similarity: Ci5 = u, 2y. The Sinkhorn- Knopp algorithm proceeds as follows: first, it computes the exponentiated cost matrix K = exp(_x), where denotes the scale of the entropic regularizer. Then, the algorithm iteratively updates two vectors: E RMxx and IR X which are initialized with uniform probability. The transportation plan I can be estimated using the exponentiated cost matrix K and the current vectors u and v. Specifically, it is given by T= u @ K @ v, where @ denotes element-wise multiplication.
At the initialization zero, the transportation plan I' is obtained by normalizing the exponentiated cost matrix K since the vectors and given by qi / Σ; Kij, which is equivalent to the Soft-PL a (u,, xy) except for the softmax temperature, which is denoted as A. This observation motivates us to set À = T, matching the same temperature CLIP, for computing the relation between embeddings. From this perspective, our proposed OT approach enhances the robustness of Soft-PL in pseudo-label assignment, especially in cases with distribution shifts between unlabeled and labeled images. Our ablation study demonstrates that increasing the number of Sinkhorn iterations improves the training process.
uniform. Consequently, our caption-level pseudo-label qi is
After initialization, the vectors and v are updated to align with the exponentiated cost matrix K and satisfy the flow constraints and q. Specifically, given the current vectors u(() and ((), the next iteration of vectors is obtained as u Kv and v(++1) K u This iterative process converges to the optimal solution. Our ablation study demonstrates that small number of iterations is sufficiently effective, and we employ 10 iterations in our experiments.
B Additional related work
Optimal transport for vision-language models. Several prior works have applied optimal transport to vision-language models. However, most of them focus transportation between image and caption, connecting objects and phrases [35, 36, 86-88J. This is known as weakly supervised phrase grounding [89], and it assumes supervised setting with image-text pairs. OTTER [90] calculates transportation between images and text, which aligns with our approach. However, they also assume supervised setting and focus calibrating the one-hot target of CLIP. To our knowledge, our work is the first to apply optimal transport for semi-supervised CLIP training.
Optimal transport for semi-supervised learning. Some prior works have utilized optimal transport for semi- supervised learning, specifically in image classification [91, 92]. These studies demonstrate the benefits of optimal transport in pseudo-labeling class labels. However, argue that OT plays more critical role in vision-language models due to inherent distribution shifts between labeled and unlabeled image captions. Our experiments reveal that naive pscudo-labeling often has detrimental effects, while OT significantly improves sunevsupe CLIP training.
Partial label learning. Partial label learning (PLL) [26] relaxes the classification problem by using a candidate set as the target instead of the exact one. This involves optimizing for the best candidate within the set. Instead of determining the hard assignment of the candidate, PRODEN [75] computes soft labels for the elements the set to smooth the optimization process. PiCO [76] further enhances this approach by incorporating contrastive learning. Our work extends the techniques of PL. to novel problem: keyword-leww pscudo-labeling for Dision-language models.
Label propagation in contrastive learning. The concept of pseudo-labeling (or label propagation) also applied in image-only contrastive learning. In addition to using the same image with different augmentations [84, 85], some studies explore the utilization of neighboring images as positive pairs [93-95], Similar concept can be extended to supervised [96] and semi supervisd [22, 97] contrastive learning, where images sharing the same class label are considered positive pairs.
17

---

Experimental details
C.1 Dataset details
Remote sensing, We use the union of RSICD [27], UCM [32], and Sydney [78] datasets for training. The captions for the UCM and Sydney datasets were annotated by Qu et al. [98]. Figure 4 displays the example images and captions, while Table 6 presents the statistics of each dataset. Since an image may have multiple associated captions, we randomly select one for each iteration.
'a playground with eight basketball fields near is surrounded by many green trees"
"many houses arranged neally and some roads crossed them with some plants roadside"
"a residential area with houses arranged neatly and some roads go through this
(a) RSICD
(b) UCM
(c) Sydney
Figure 4: Examples of image-text pairs in the remote sensing domain.
Table 6: Statistics image-text paired datasets in the remote sensing domain.
RSICD
UCM
Sydney
pairs
8,734
1,680
497
We use the validation sets of RSCID-CLS [27], UCM-CLS [32], WHU-RS19 [31], RSSCN7 [80], and AID [81] datasets for zero-shot classification. RSCID-CLS and UCM-CLS the classification versions of RSCID and UCM, respectively. Table presents the statistics of each dataset.
Table 7: Statistics of classification datasets in the remote sensing domain.
RSICD-CIS
UCM-CLS
WHU-RS19
RSSCN7
AID
# images
1,094
2,100
1,005
2,800
10,000
# of classes
31
21
19
30
Fashion. We use the union of Fashion200k [28], FashionGen [82]. and Polyvore Outfits [83] datasets for training. Figure displays the example images and captions, while Table 8 presents the statistics of each dataset. Since item may have multiple associated views of images corresponding the caption, we randomly select one view for each iteration. For captions, we concatenate the titles and descriptions from FashionGen and Polyvore, and the sentences from Fashion200k.
"grey classic u-neck t-shirt. short sleeve t-shirt heather grey. tonal logo flag u-neck collar."
"off-white striped flared pants. multicoloured cotton silk striped flared pants from off-white.
"white printed silk-jersey wrap dress midnight blue"
(a) Fashion200k
(b) FashionGen
(c) Polyvorc
Figure 5: Examples of image-text pairs in the fashion domain.
18

---

Table 8: Statistics of image-text paired datasets in the fashion domain.
Fashion200k
FashionGen
Polyvore
# of pairs
61,753
60,147
71,967
We the validation sets of Fashion200k, FashionGen, and Polyvore datasets for zero-shot classifi- cation. Table 9 presents the statistics of each dataset. For Fashion200k and FashionGen, we report both the super-class and the sub-class. The class names in Polyvore match the same level of the super-class as those in the other datasets, For example, the super-class contains class names such as "dress," while the sub-class contains class names such as "casual and day dress."
Table 9: Statistics of classification datasets in the fashion domain.
Fashion200k
FashionGen
Polyvore
Super-class
Sub-class
Super-class
Sub-class
Class
images
29,785
32,528
14,657
# of classes
5
31
48
121
11
Scientific figures. We the SciCap [29] dataset for training. Figure displays the example images and captions, while Table 10 presents the statistics of each dataset. We use figures that do not have subfigures, denoted the "SeiCap-No-Subfig-ngg subset, for simplicity.
Comics. We use the Simpsons [30] dataset as labeled data and the Simpsons Character [33] dataset as unlabeled data. Figure 6 displays the example images and captions, while Table 10 presents the statistics each dataset. We use 90% of pairs from the Simpsons dataset as training split and remaining 10% as the validation split. Since the training split contains only 720 image-text pairs, it is necessary to incorporate images from different datasets like Simpsons Characters.
Aum (100)
430
'Fi.. 2. variable 2-D initial probe distribution used for CFO runs reported here"
'simpsons wearing headphones and looking like a guy"
(a) SciCap
(b) Simpsons
Figure 6: Examples image-text pairs in the scientific figure and comies domains.
Table 10: Statistics image-text paired datasets in the scientific figure and comics domains.
SciCap
Simpsons
pairs 106,834
720
19

---

C.2 Implementation details
Training. We utilize the OpenCLIP [77] library implement our algorithms. By default, we follow the original training configuration. We use a batch size of 64 per GPU, with a total of 4 GPUs. The learning rate set to 5e-5, and we apply the default cosine learning rate scheduling with warmup period of the first 10 steps. We train all models until the performance saturate, which can vary over the number of image-text pairs. Specifically, for remote sensing, we train models for 25 epochs, for fashion 10 epochs, for scientific figures 5 epochs, and for comics datasets 10 epochs. We run three trials and report the average and standard deviation for all experiments.
We add the supervised CLIP loss and pseudo-label losses for semi-supervised methods. We halve the scale of pseudo-label losses match the scale of the image part in the CLIP loss. We the scale of the entropic regularizer to be the same as the softmax temperature of CLIP, denoted as = and set the number of Sinkhorn iterations to 10. Note that our method does not introduce any additional hyperparameters, except for the selection of keyword candidate sets discussed below.
Keywords. We the available class names for remote sensing and fashion datasets while extracting the keywords using the YAKE| [74] algorithm from the captions for the SciCap and Simpsons datasets. For remote sensing, keywords, of which around 30 words appeared in the captions. For fashion, we combine the super- class names from Fashion200k and FashionGen obtain 56 keywords, which are more effective than sub-class names since the captions often do not contain the exact sub-class names.
combine the class names from RSICD-CLS and UCM-CLS to obtain 45
Our ablation study presents that the keywords extracted by YAKE are also effective, even outperform- ing the class names in some cases. In the case of remote sensing, the YAKE keywords contain not only nouns like "building" but also adjectives like "green," "many," "large," which help the model understand the fine-grained information the scene. Moreover, we utilize YAKE- 100 keywords for SciCap and Simpsons when class names are unavailable, and this approach has proven be effective. Thus, we believe that a reasonable choice of keywords would firmly benefit our method.
Computation. The computation bottleneck mostly lies in computing embeddings rather than the loss function. Thus, all methods require a similar amount of time for a single training iteration, except for S-CLIP, which takes longer due to the additional forwarding of keywords. Training duration varies depending on the number of image-text pairs, taking a few minutes to a few hours. Semi supervipe methods take twice as long supervised CLIP since draw batches of labeled and unlabeled data with an even size, necessitating twice as many iterations for the same epoch.
Evaluation. We use the evaluation code provided by OpenCLIP. For zero-shot classification, we use the default template: "a photo of [class]." We use this template for fashion datasets but use the template "an aerial photograph of [class]" for remote sensing, following the setup of [64].
20

---

D Additional remote sensing results
D.1 Different neural architectures
Table 11 presents the zero-shot classification results using the ResNet [99] and VIT [100] models. S-CLIP significantly improves the supervised baseline in all considered scenarios. Note that the larger model shows better overall accuracy, for both the original and fine-tuned CLIP. Here, S-CLIP gives reliable improvements when scaling up larger models.
Table 11: Zero-shot classification results on remote sensing datasets using different neural architec- tures. Parentheses indicate the performance gap from the supervised CLIP, and bolds denote the best results within each architecture. S-CLIP give consistent improvements, scaling to larger networks.
Model
Method
RSICD-CLS
UCM-CLS
WHU-RS19
RSSCN7
AID
CLIP (original)
45.3
50.5
65.5
58.9
47.8
ResNet-50
CLIP (fin--tnee)
58.3203
63.5234
76.5132
61.9112
63.1213
S-CLIP (ours)
66.9
(13.2)
86.9
66.2 (+4.3)
73.0 (49.9)
CLJP (original)
55.8
58.6
76.4
62.1
55.7
VIT-B/32
CLIP (fine-tnne)
72.3
76.8210
90.4
68.9122
78.9205
S-CLIP (ours)
77.5= (+5.2)
78.3=24 (+1.5)
93.2 (+2.8)
69.6 (+0.7)
83.7 (+4.8)
CLIP (original)
58.9
60.1
80.9
69.4
59.6
VIT-B/16
CLIP (finc-tnne)
72.4=06
78.7=27
90.0
71.9388
77.9209
S-CLII (ours)
79.5 (+7.1)
82.3 (+3.6)
93.9 (+2.1)
76.3 (+4.4)
85.2209 (+7.3)
D.2 Different image-text pair ratios
Table 12 presents the zero-shot classification results for different image-text pair ratios. Using only 10% of pairs (2x fewer than CLIP), S-CLIP achieves performance comparable to CLIP using 20% of pairs. Moreover, with 30% pairs (3x fewer than CLIP), S-CLIP matches the performance CLIP using 100% of pairs and even outperforms it, possibly due the keyword-level pseudo-Labeling.
Table 12: Zero classification results remote sensing datasets using different image-text pair ratios. Parentheses indicate the performance gap from the supervised CLIP, and bolds denote the best results within each pair ratio. S-CLIP gives consistent improvements, reducing required pairs.
Method
Ratio
RSICD-CLS
UCM-CLS
WHU-RS19
RSSCN7
AID
CLIP (original)
0%
45.3
50.5
65.5
58.9
47.8
CLIP (fine-tune)
58.3-03
63.5+34
76.5+32
619+12
63.1=13
10%
S-CLIP (ours)
66.9 (48.6)
16.7=16643.2
86.9 (+10.4)
66.2 (+4.3)
73.0=03 (+9.9)
CL.IP (fine-tune)
66.2-1.1
69.7
82.8215
65.6234
71.821
20%
S-CLIP (ours)
77.6111 (+11.4)
72.0 (+2.3)
90.7.18 (+7.9)
67.9276 (+2.3)
83.7207 (+11.9)
CLIP (Gime-tune)
70.8=14
70.7
85,4
63.6213
75.7
30%
S-CLIP (ours)
80.1=10 (49.3)
74.0=1a (43.3)
94.8=12 (49.4)
69.6:23 (+6.0)
87.8 (+12.1)
CLIP (fine-tune)
100%
77.5
76.5
93.9
71.2a00
83.621
21

---

D.3 Using all image-text pairs
We used 10% of the image-text pairs as labeled data to compare two scenarios, L=U and LžU, in the main paper. Having demonstrated the effectiveness of S-CLIP in the LøU scenario, the next question whether it would also be helpful when using all paired data and extra unlabeled data. Table 13 presents the results of using the entire RS-ALL dataset as labeled data and RESISC45 as the unlabeled data. S-CLIP shown to effective even this challenging scenario.
Table 13: Zero-shot classification results on remote sensing datasets using all captioned images and extra unlabeled images, where the unlabeled data is from different distribution than the labeled data. S-CLIP also provides improvements in this most challenging scenario.
Model
Method
RSICD-CLS
UCM-CLS
WHU-RS19
RSSCN7
AID
CLIP (fine-tune)
77.5+15
76.5+a5
93.9=09
71.2+a3
83.6=16
ResNet-50
S-CLIP (ours)
81.1 (+3.6)
(+1.2)
95.8=09 (+1.)
72.3+28 (+1.1)
87.7=04 (+4.1)
CLIP (fin--tune)
85 3210
87.3207
95.8210
75.7
92.0gns
VIT-B/32
S-CLIP (ours)
(7.0006
88.6+14 (+1.3)
97.2-04 (+1.4)
76.2 (+0.5)
92.5=0a (+0.5)
CLIP (finc-tunc)
87.0g03
88.5
96.7
78.4
89.2
VIT-B/16
S-CLIP (ours)
87.4212
88.9 (+0.4)
97.3204 (+0.6)
1001
93.1 (+3.9)
D.4 Image-text retrieval R@1
Table 14 presents the R@1 results of image-text retrieval, complementing Table in the main paper which shows the R@5 results. S-CLIP outperforms the supervieee methods in most cases, except the supervised baseline works best for image- stegt retrieval on the UCM dataset.
Table 14: Image-text retrieval results on remote sensing datasets, following the same setup of Table 1. Bolds denote the best results among the semi-supervised methods within the same setups. S-CLIP performs the best in most cases, similar to the observation in Table 2.
Image-ttet R@1
Text Text- R@1
Method
Data
RSICD
UCM
Sydney
RSICD
UCM
Sydney
CLIP (original)
2.1
10.3
2.2
20.7
CLIP (fine-tune)
L
3.6202
10.6-2.1
13.2
3.7203
10.0
14.4=20
Hard-PL. [16]
3.8=0.1
9.7=27
14.4210
3.9207
9.2+18
10.9=10
Soft-PL [21]
L=U
4.0=a0.
11.0=a8
11.5
4.2
9,8+115
16.1=4.
S-CLIP (ours)
4.2
11.6-11
14.9
4.2=006
11.1
17.8=1.
Hard-PL [16]
3,8
8.1=10
13.4
3.4
8.9414
12.6=1.
Soft-PL [21]
L
3.8
8.4=1.2
10.9
3.5.03
9,4
14.9=1.
S-CLIP (ours)
4.2
9.8=0.6
13.8
3.9
10.8
17.8=5.
22

---

E
Additional ablation studies
E.1
General image domain (COCO) results
Table 15 presents results on the COCO [101] dataset. We observed that fine-tuning models using limited image-caption pairs degrades performance since the original CLIP already performs well. Therefore, we selected a subset of the "sports" " category where the original CLIP performs relatively weakly. Following other setups, we used 10% of the data as labeled data and the rest as unlabeled data. We kept the training configuration consistent with our paper but ran 10 epochs to ensure model convergence. S-CLIP model outperforms others, even in this general image domain.
Table 15: Image-text retrieval results for the COCO sports category, where the original CLIP performs relatively weakly. S-CLIP also performs well in the general image domain.
Method
Image-to-text retrieval
Text-to-image retrieval
CLIP (original)
40.30
38.
CLIP (fine-tune)
46.59
47.76
Hard-PL
48.72
47.12
Soft-PL.
48.72
47.23
S-CLIP (ours)
49.79
48.40
E.2
Additional pseudo-labeling baselines
Table 16 presents a comparison with pseudo-labeling baselines adapted from the sate-of-the- art semi- supervised image classification methods. Specifically, employ the pseudo-labeling techniques of SemPPL [22] and RoPAWS [69]. SemPPL uses the mode k-NN predictions. In our case, since captions are unique for each image, assign uniform probability over k-NN samples, which refer to as "k-NN" pseudo-labeling. On the other hand, RoPAWS utilizes an analytic fixed teint solution for label propagation, which we refer as "LabProp" pseudo-labeling. We used these baselines for caption-level pseudo-labeling and compare them with OT. We do not used keyword-level pseudo-labeling for a fair comparison. OT outperforms other pseudo-labeling approaches.
Table 16: Comparison with pseudo-labeling approaches adopted from the SOTA semi-supervised image classification methods, following the setup of Table 5. OT performs the best.
Method
Zero-shot accuracy
Image-text retrieval
k-NN (k=5)
61.1
8.8
LabProp
68.2
9.4
OT (ours)
69.6
10.5
E.3
Qualitative examples of pseudo-labels
Figure 7 and Figure 8 provide qualitative examples of caption-level and keyword-level pscudo-labels, respectively. Optimal transport (OT) matches visually and semantically similar labeled images for unlabeled queries, providing meaningful caption-level pseudo-labels. In addition, the nearest labeled images share overlapping keywords, providing meaningful keyword-level pseudo-labels.
23

---

Query
Nearest Labeled Images (Top-5)
Timmy green trees are around buildig with swiming poof
aurrounded
"there are houses near the road behind which foreet
"the river very long and has a very wide wrea
building asiimming poot mony sparse grain trees a
"many g green pond
Boveet
boats port many buildings
"many port many
lown
'some boats wherf
angps, land?
port plece
"maay bulldings and green troes and boo sldes of road in dense renidieniia arms""
large number of factories been built in the faclory"
"the residendal buildings on both dides the road are neatly arranged
"houses of dffferent calors are led ondenty along the winding roads'
"on both sices of the road were densely brown houses"
"many grean treed around many buildings with many cans indust industriil aeas"
Figure 7: Qualitative examples of caption-level pseudo-labels. Optimal transport (OT) can identify visually and semantically similar images, including those representing forests, ports, and dense buildings. As result, caption-level pseudo-labels offer accurate semantics for unlabeled images.
mony groen surrounded Vreaa and
"mrny builddnnge green commencial ares'
rives"
stadium focBbll Sold"
many meedows"
advanced land"
buildings read"
Query
Gup
Nearest Labeled Images
bridge come green
"many green treps "mary groen istoilum with parking
"many buildings green
trapezodd playground"
sumruunded by
buildings"
Figure 8: Qualitative examples of keyword-level pseudo-labels. The images closest to the query, found by OT, often share the keywords in their captions (column 1-2). Even in cases where the images do not share all the keywords, they still have some overlapping keywords (column 3-6).
24

---

E.4
Training curves
Figure 9 presents the training curves showing the loss and evaluation metrics. The validation accuracy for zero-shot classification on RSICD-CLS and image-text retrieval on RS-ALL is reported. The trends are similar in other datasets. Supervised CLIP is prone to overfitting, as indicated by the CLIP validation loss, but this is regularized by Hard-PI. and Soft-PL. Our proposed S-CLIP yields the greatest benefits, with superior classification and retrieval performance.
4.75
CLIP (base)
Hard-PL.
4.50
S-CLIP (ours]
CLIP (base)
CLIP (base)
Hard-PL
Hard-PL
Soft-PL
Soft-PL
3.75
S-CLIP (ours)
S. CLP (ours)
1.0
15
20
25
10
15
20
5
10
15
20
25
(a) CLIP validation loss
(b) Zero-shot classification
(c) Image-text retrieval
Figure 9: Trends in loss and evaluation metrics during training. S-CLIP regularizes overfitting, as
shown by the CLIP validation loss. As a result, it performs the best classification and retrieval.
E.5 Locked image text tuning
Table 17 presents the results of fine-tuning after locking (freezing weights) either the image or text encoders, suggested by LiT [41]. In method, locking either encoder leads to degradation in performance. This occurs because both the image and text data from the specialist domain are unseen, requiring the model to learn this novel information. In contrast, in the base CLIP, locking the text encoder yields similar results to having no lock. This is due to the base CLIP suffering from overfitting and inadequately learning the text information. These results provide additional evidence that pseudo-labeling with S-CLIP improves the language understanding of models.
Table 17: Ablation study on locked image or text encoders. We report the average values of zero-shot accuracy (left) and imagex>ttet retrieval (right). Locking either the image text encoder degrades performance, as it prevents the model learning new information from specialist domains.
Locked image
Locked text
No lock
CLIP (fine-tune)
48.7
64.8
64.7
9.3
S-CLIP (ours)
63.1
8.7
67.1
10.2
71.9
10.6
E.6 Selection of mini-batches during training
Table 18 presents the effect of batch selection by applying different strategies S-CLIP. Using smaller batch size (16 x 4 = 64) instead of the original (64 x 4 = 256) significantly decreased the performance. However, collecting similar images helped mitigate this drop. We simplified the process by arranging batches based on their indices, sorted over visual similarities, and used them without shuffling. This method, "sorted batch," resulted in reasonable gain. Exploring a more effective active batch sampling strategy would be an interesting future direction.
Table 18: Ablation study on the mini-batch sampling strategies, following the setup of Table 5. Using a larger and sorted batch improves performance by providing more informative pseudo-labeke
Batch size
Zhot accuracy
Image- text retrieval
64 (random batch)
65.8
8.3
64 (sorted batch)
67.3
256 (random batch)
71.9
10.6
'The reported average values may appear similar, but each dataset has distinct values.
25

---

E.7 Baselines using keyword information
Table 19 presents the results compared to the baselines using keyword information. Specifically, we incorporated two losses: (a) keyword loss on labeled data using multi-label classification, and (b) keyword loss on unlabeled data using our pseudo-labeling approach (nearest image is given by OT). To assess the impact of both losses, we compared CLIP-FT, CL)P-FT+(a), and CLIP-FT+(a)+hh)) For the ablation study, we did not apply caption-level pseudo-labeling. Both (a) and (b) contribute the overall performance, confirming that our proposed pseudo-labeling loss relies not only on keyword information but also on exploiting information from unlabeled data.
Table 19:
Comparison to the baselines using keyword information, following the setup of Table 5. Both keyword information and semi-supervised learning contribute to the final performance.
Method
Zero-shot accuracy
Image-text retrieval
CLIP-FT
64.7
9.3
keyword info.
67.3
9.7
semi supersuid
70.6
10.1
E.8 Keyword statisties in captions
We trained our models using captioned datasets, and the class names were derived from the classifi- cation counterparts of the dataset. Consequently, captions often contain multiple class names. For instance, 45.8% of captions in the RSICD dataset contain more than two class names. The remaining 50% of captions have only single class name, which results in our keyword-level loss being reduced to classification loss. It is worth noting that we developed a method to handle scenarios where class names are unavailable. By utilizing YAKE-200 keywords, we found that 94.8% of captions contain more than two keywords, confirming the necessity of partial label learning.
F Limitations and broader impacts
Linitations. Our pseudo-labeling approach is effective for zero-shot classification but less effective for image-text retrieval, which requires fine- fine-grained language understanding. This is because the current pseudo-labeling approach assumes that captions in a batch can capture the semantics of unlabeled images, which may not hold true for fine-grained contexts. To address this issue, one can increase the batch size [84] or incorporate a queue of caption embeddings [85].
In addition, the current OT formulation of caption-level pseudo-labels assumes uniform sink and source constraints. However, this assumption can be refined in the case of distribution shifts. For example, techniques from robust semi-supervised learning [68, 69] can be utilized to estimate the in-domain-ness and subsequently regularize pseudo-labeling based on this estimation.
Lastly, keyword-level pseudo-labeling could be enhanced. The current method relies on the exact inclusion of keywords in the caption form the candidate set. One can relax this criterion to include synonyms or related concepts by incorporating additional inference steps. In addition, the extracted keywords from captions may vary in their levels. Integrating a word hierarchy to account for different levels of keywords would be interesting future research direction.
Broader impacts. Our paper aims broaden the applicability of CLIP to specialist domains. However, this broader scope may present challenges when using the model with domains that include harmful content. These challenges arise from the data itself, rather than the model itself. Therefore, is crucial to have adequate data regularization to effectively address these concerns.
26


Reviews:
[{"review_id": "qkBXOFR5Lb", "reviewer": "Reviewer_EbWV", "summary": "This paper proposes a method S-CLIP, to enhance CLIP in a semi-supervised way by leveraging captions (via optimal transport) and keywords (via image-keyword similarity) from other unpaired images in the same batch. This is beneficial for specialized domains like remote sensing, where image-text data is usually limited. Results show better performance than the traditional pseudo-labelling methods, especially in OOD domains.", "questions": "Given this method is sensitive to the batch data (e.g. statement in 5.2), it would be more solid if we could consider some data preprocessing strategies, e.g., to batch similar images in the same batch as much as possible? Or at least we should study how batch size affects the results in ablations?\n\nIn line 237, “we employ 32 image-caption pairs and 32 unpaired images”: this seems contrary with the problem to solve, i.e., N << M?\n\nCould we give some qualitative examples for caption-level and keyword-level selection? E.g., given a few images, which are the selected text (weighted captions/keywords) look like in the same batch?\n\nminor comment1: in Figure 1b, Hard-PL is “-2.7” instead of “-1.7”?\n\nminor comment2: the entire section 3 background could be removed to save more space for ablations/analysis, which is more valuable to readers (given the background intro is too basic)?\n\nminor comment3: explain what “YAKE-100/200/300” mean in Table 5?\n\nminor comment4: “(f) Partial” to be consistent with other items.\n", "limitations": "No specific concern.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Addressing low-data regimes for contrastive learning. Such a problem is common in specialized domains like remote sensing.\n\nNovel and simple method, but showing promising results compared to the standard pseudo-labelling methods, especially in OOD domains.\n\nExperiments and ablations are comprehensive.\n\nWell written and easy to read.\n", "weaknesses": "In section 5 experiments, missing experiment details for reproducing, such as model size, epoch, training schedule (lr, weight decay etc.)...\n\nBesides the traditional pseudo-labelling methods, it would be nicer if we could compare with other SOTA pseudo-labelling methods for vision-language training – if there were no such SOTA methods, we could claim that in the paper.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer EbWV,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the problem significance, novel and simple method, comprehensive experiments and ablations, and clear presentation. We will address your concerns and questions in the response below.\n\n---\n**[W1] Experimental details**\n\nWe have stated all the experimental details, such as epochs and training schedules, in Appendix D.2, as well as in our code provided in the supplementary material. We will clarify that line 240 of Section 5, stating “Additional details are in Appendix D.,” includes such information in the final manuscript.\n\n---\n**[W2] Comparison to other pseudo-labeling methods**\n\nDesigning pseudo-labeling methods for vision-language training is an underexplored research question. Specifically, our focus is on adapting CLIP for specialist domains, making prior approaches not directly applicable. For instance, prior works often require a pre-trained object detector to align detected objects with keywords in captions [7-9] or a pre-trained decoder to synthesize data in missing modalities [R1], which may not be suitable for specialist domains. Therefore, we believe that our work is SOTA for the problem we consider, and we will add this claim in the final manuscript as per your suggestion.\n\nNevertheless, for readers' further interest, we provide additional baselines that adapt the SOTA semi-supervised image classification methods. Specifically, we employ the pseudo-labeling technique of SemPPL [22] and RoPAWS [69], both published in ICLR 2023. SemPPL uses the mode of the k-NN prediction. In our case, since captions are unique for each image, we assign a uniform probability over k-NN samples, which we refer to as \"k-NN\" pseudo-labeling. On the other hand, RoPAWS utilizes an analytic fixed-point solution of label propagation, which we refer to as \"LabProp\" pseudo-labeling. We use these baselines for caption-label pseudo-labeling and compare them with OT. We do not apply keyword-level pseudo-labeling for ablation. The table below presents the results, following the setup of Table 5. OT outperforms other pseudo-labeling approaches.\n\n\n\\begin{array}{lcc}\n\\text{Method} & \\text{Zero-shot accuracy} & \\text{Image-text retrieval} \\newline\n\\hline\n\\text{k-NN (k=5)} & 61.1 & 8.8 \\newline\n\\text{LabProp} & 68.2 & 9.4 \\newline\n\\text{OT (ours)} & 69.6 & 10.5 \\newline\n\\end{array}\n\n[R1] Devillers et al. Semi-supervised Multimodal Representation Learning through a Global Workspace. arXiv, 2023/06/27. \n\n---\n**[Q1] Selection of mini-batches during training**\n\nThank you for the insightful suggestion! We conducted additional experiments to verify the effect of batch selection by applying different strategies to S-CLIP. The table below presents the results, following the setup of Table 5. Using a smaller batch size (16x4=64) instead of the original (64x4=256) significantly decreased the performance. However, collecting similar images helped mitigate this drop. We simplified the process by arranging batches based on their indices, sorted over visual similarities, and used them without shuffling. This method, \"sorted batch,\" resulted in a reasonable gain. Exploring a more effective active batch sampling strategy would be an interesting future direction.\n\n\n\\begin{array}{lcc}\n\\text{Batch size} & \\text{Zero-shot accuracy} & \\text{Image-text retrieval} \\newline\n\\hline\n\\text{64 (random batch)} & 65.8 & 8.3 \\newline\n\\text{64 (sorted batch)} & 67.3 & 8.9 \\newline\n\\text{256 (random batch)} & 71.9 & 10.6 \\newline\n\\end{array}\n\n---\n**[Q2] Size of the labeled vs. unlabeled data**\n\nWe used 32 image-caption pairs and 32 unpaired images for each batch per GPU, where N and M indicate the total numbers of paired and unpaired images, respectively. In our experiments, N << M; for instance, M is 9 times larger than N when we subsample 10% of images as paired ones. We will further clarify in the final manuscript that 32 implies the batch size, not the dataset size, as also mentioned on line 236.\n\n---\n**[Q3] Qualitative examples of pseudo-labels**\n\nWe provide the qualitative examples of pseudo-labels in the attached PDF in the general response. The results demonstrate that optimal transport (OT) matches visually and semantically similar labeled images for unlabeled queries, providing meaningful caption-level pseudo-labels. In addition, the nearest labeled images share overlapping keywords, providing meaningful keyword-level pseudo-labels.\n\n---\n**Minor comments**\n\n1, 4: Thank you for the clarification. We will correct the typos in the final manuscript.\\\n2: We included the background section for self-containment. However, we will consider relocating it to the Appendix in the final manuscript.\\\n3: YAKE-N implies that we utilize the top-N keywords from the YAKE algorithm. We will provide further clarification on this in the final manuscript.\n\n---\nPlease let us know if you have any further concerns.\n\nSincerely,\\\nAuthors"}, {"review_id": "edwwUyl4pA", "reviewer": "Reviewer_CjR3", "summary": "Fine-tuning CLIP in specialized domains suffer the problem of limited available image-text pairs. This paper propose to address this challenge via semi-supervised learning methods with the additional unpaired images. Two pseudo-labeling strategies including the caption-level pseudo labeling and keyword-level pseudo labeling are proposed to construct valuable information about the unpaired images.\nThe problem addressed in the paper is meaningful and the method proposed in the article is demonstrated effective through comprehensive experiments.\n", "questions": "(1) \"Baseline\" in table 5 is a little confusing. Is that refers to the \"CLIP (fine-tune)\" method or the \"hard-PL\" or \"soft-PL\" baselines ?\n\n(2) It seems the role of the proposed OT-based \"Caption-level pseudo-label\" is similar as the soft-PL baseline. I suggest the authors provide a comparison of the effect of using the OT-based method and the soft-PL method on top of CLIP fine-tuned; or replace the OT-based method with the soft-PL, ablation on S-CLIP to show the effectiveness.\n\n(3) I suggest the accuracy of training on 100% labels (full data) should be reported, as the upper bound performance to be reffered.", "limitations": "No", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(+) This article is well written, and the description of the method pipeline is clear and easy to understand. \n\n(+) Experiments are sufficient.\n\n(+) The problem it solves is meaningful.", "weaknesses": "(-) \"Baseline\" in table 5 is a little confusing. See question (1).\n\n(-) The effectiveness of OT-based \"Caption-level pseudo-label\" compared with soft-PL should be further verified.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer CjR3,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the problem significance, sufficient experiments, and clear presentation. We will address your concerns and questions in the response below.\n\n---\n**[W1] Definition of “Baseline” in Table 5**\n\n\"Baseline\" implies CLIP (fine-tune), since our goal was to ablate the effect of caption-level and keyword-level pseudo-labeling on top of this. We will clarify this in the final manuscript.\n\n---\n**[W2] Advantage of OT over Soft-PL**\n\nOT becomes identical to Soft-PL when we set the number of Sinkhorn iterations to 0, as mentioned in line 304. The rationale behind this can be found in Appendix A.2, where the Sinkhorn algorithm is initialized as Soft-PL and progressively becomes balanced as the iterations proceed. In short, Soft-PL often assigns the pseudo-labels concentrated on certain captions, resulting in the collapse during training. OT regularizes this collapse, leading to the assignment of better pseudo-labels. The ablation study in Table 5b demonstrates the advantage of OT over Soft-PL with performance improvements over Sinkhorn iterations. We will further clarify this in the final manuscript.\n\nFor your convenience, we have copied the results in the table below.\n\\begin{array}{lcc}\n\\text{Sinkhorn iteration} & \\text{Zero-shot accuracy} & \\text{Image-text retrieval} \\newline\n\\hline\n\\text{Iter. 0 (= Soft-PL)} & 70.6 & 9.2 \\newline\n\\text{Iter. 1} & 71.4 & 9.4 \\newline\n\\text{Iter. 5} & 71.6 & 10.2 \\newline\n\\text{Iter. 10} & 71.9 & 10.6 \\newline\n\\end{array}\n\n---\n**Q1** and **Q2** are answered in W1 and W2, respectively.\n\n---\n**[Q3] Comparison with training using 100% labels**\n\nWe provide a comparison of CLIP-FT and S-CLIP using different ratios of image-text pairs in Table 12 in Appendix E.2, including training on 100% of labels. We found that S-CLIP, using only 20% of labels, matches the performance of CLIP-FT trained on 100% of labels.\n\nFor your convenience, we have copied the results in the table below.\n\n\\begin{array}{lccc}\n\\text{Method} & \\text{Pair ratio (\\\\%)} & \\text{RSICD-CLS} & \\text{UCM-CLS} \\newline\n\\hline\n\\text{CLIP (original)} & 0 & 45.3 & 50.5 \\newline\n\\hline\n\\text{CLIP (fine-tune)} & 10 & 58.3 & 63.5 \\newline\n\\text{S-CLIP (ours)} & 10 & 66.9 & 66.7 \\newline\n\\hline\n\\text{CLIP (fine-tune)} & 20 & 66.2 & 69.7 \\newline\n\\text{S-CLIP (ours)} & 20 & 77.6 & 72.0 \\newline\n\\hline\n\\text{CLIP (fine-tune)} & 100 & 77.5 & 76.5 \\newline\n\\end{array}\n\n---\nPlease let us know if you have any further concerns.\n\nSincerely,\\\nAuthors"}, {"review_id": "tSF4n2MrL1", "reviewer": "Reviewer_7J1f", "summary": "The paper introduces an additional training method for vision-language models like contrastive language-image pre-training (CLIP). The proposed method consists of semi-supervised learning with two different pseudo labeling including caption- and keyword-level pserudo-label. These two pseudo labelings are complementarily understand image through different types of text representation. The experimental section describes the effectiveness of the proposed method in terms of zero-shot image recognition and image-text retrieval in several representative datasets. A relatively small image-text paired dataset can improve the zero-shot recognition and image-text retrieval with the proposed pseudo labelings.", "questions": "Please see above-mentioned comment in 'weaknesses'.", "limitations": "There are no negative limitations and societal impacts.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper serves a method for semi-supervised learning on image-text paired datasets. Usually, it is quite difficult to construct the kind of dataset, however, the paper proposed that a simple yet effective idea to solve a problem in vision language pre-training like CLIP. \n\n+ This paper also points out that limited focused areas such as remote sensing and fashion cannot be effectively used a vision-language models like CLIP as is. Therefore, the paper insists that an easily adaptive labeling approach and semi-supervised fine-tuning from pseudo labels are important for CLIP to adapt a specific image dataset.\n\n+ In the experimental section, the proposed method S-CLIP is much better than that of used the original CLIP and fine-tuned CLIP (CLIP-FT). It is reasonable to assign soft labels rather than hard labels (Hard-PL) in this problem. \n\n+ The paper is well-written and effectively communicates the main ideas in the paper, making it accessible to a broad audience. Moreover, there contains various visualizations, e.g., Figures 1 and 2, to confirm that the learnt representations of vision-language pre-training are meaningful. ", "weaknesses": "- CLIP is pre-trained on image-text paired dataset widely extracted from the major space on the Internet. Therefore, it is understandable that the dataset collection does not work well in situations where only a small amount of paired data is collected, or where the paired data is not steadily available on the web. On the other hand, does the proposed dataset collection method only work on remote sensing, fashion, scientific figures, and comics? If you could show the vision-language application based on the proposed dataset collection in more situations, the paper must be more effective to broader readers. Probably, the approach could be shown in more generic situations, however, there are only 4 types of situations. Though the authors do not necessarilly add the experimental results, how about discussing the effectiveness for general purpose dataset collection?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 7J1f,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the problem significance, simple yet effective method, reasonable experiments, and clear presentation. We will address your concerns and questions in the response below.\n\n---\n**[W1] Dataset collection beyond the considered four domains**\n\nWe would like to clarify that our paper focuses on effective learning from a given limited set of image-text pairs, rather than on collecting such datasets. Our proposed semi-supervised learning method can be applied to any vision-language domain, and for our experiments, we have chosen four domains where we could access publicly available datasets.\n\nNevertheless, we agree with your point that our work can achieve broader applicability across domains beyond the four we have chosen. Examples include medical imaging [R1], manufacturing visual anomalies [R2], or scientific images such as cells [R3]. For such domains, users may require caption annotations from human experts, such as doctors, factory workers, or scientists. We believe our semi-supervised learning framework would significantly reduce the annotation costs for collecting datasets from these specialists. We will include this discussion on dataset collection strategy in the final manuscript.\n\n[R1] https://physionet.org/content/mimic-cxr/2.0.0/ \\\n[R2] https://www.mvtec.com/company/research/datasets/mvtec-ad \\\n[R3] https://www.rxrx.ai/\n\n---\nPlease let us know if you have any further concerns.\n\nSincerely,\\\nAuthors"}, {"review_id": "1GljHUP1lf", "reviewer": "Reviewer_XrtM", "summary": "The paper studies semi-supervised learning for contrastive vision-language pre-training, during which unlabelled/unpaired data is also accessible. The authors formulate the pseudo-labeling as an optimal transport (OT) problem and propose to use the Sinkhore algorithm to produce pseudo labels, which is validated to be superior to vanilla had/soft PL solution. Comparisons with other works in literature utilizing OT are properly discussed. Furthermore, the paper proposes to use the key-level pseudo label to account for partial label learning. Experiments on several data domains are conducted to validate its effectiveness.", "questions": "- The title is somewhat confusing. The paper studies semi-supervised fine-tuning on pre-trained CLIP weights, which is not suitable to phrase as \"semi-supervised pre-training\", where you are dealing with an unlabelled general domain. \n- When keywords are obtained from class names, does one labeled image normally only contain just one keyword? If yes, does this reduce to a classification loss (since the target is one-hot)?\n", "limitations": "All limitations are properly addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is technically sound and easy to follow.\n- The evaluation is self-sufficient with proper baselines, which helps to corroborate the effectiveness of the proposed loss.\n- The ablative study is sufficient and showcases the impact of different composing parts of the method.\n- Consistent performance gain over a variety of downstream datasets.", "weaknesses": "- For keyword-level pseudo-label, the compared baseline is somewhat not fair, since one could also leverage the keyword information for labeled/paired data in the vanilla fine-tuning setting, i.e., fine-tuning pre-trained CLIP with CLIP loss and multi-class classification loss. Additional experiments should be conducted to verify the gain derives from the semi-supervised part of it instead of simply leveraging the label information.\n- It would also be nice to showcase results in general data (e.g., the ones evaluated in the CLIP paper, Flicker, etc.). Manual data split may be necessary but it helps to corroborate the generalizability of the proposed method.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer XrtM,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the technical soundness, sufficient evaluation and ablation, and clear presentation. We will address your concerns and questions in the response below.\n\n---\n**[W1] Baseline using the keyword information**\n\nFollowing your suggestion, we have included baseline results that leverage keyword information. Specifically, we incorporated two losses: (a) keyword loss on labeled data using multi-label classification, and (b) keyword loss on unlabeled data using our pseudo-labeling approach (nearest image is given by OT). To assess the impact of both losses, we compared CLIP-FT, CLIP-FT+(a), and CLIP-FT+(a)+(b). For the ablation study, we did not apply caption-level pseudo-labeling. The table below presents the results, following the setup of Table 5. Both (a) and (b) contribute to the overall performance, confirming that our proposed pseudo-labeling loss relies not only on keyword information but also on exploiting information from unlabeled data.\n\n\\begin{array}{lcc}\n\\text{Method} & \\text{Zero-shot accuracy} & \\text{Image-text retrieval} \\newline\n\\hline\n\\text{CLIP-FT} & 64.7 & 9.3 \\newline\n\\text{+ keyword loss on L} & 67.3 & 9.7 \\newline\n\\text{+ keyword loss on L and U} & 70.6 & 10.1 \\newline\n\\end{array}\n\n---\n**[W2] Experiments on the general image domain**\n\nWe conducted additional experiments on the COCO dataset. Here, we observed that fine-tuning models using limited image-caption pairs degrades the performance, as the original CLIP already performs well. Thus, we used a subset of the \"sports\" category where the original CLIP performs relatively weakly. Following the setup described in our main paper, we used 10% of the data as labeled data and the rest as unlabeled data. We kept the training configuration consistent with our paper but ran 10 epochs to ensure model convergence. We report the image-text retrieval performance based on the R@5 metric. The table below shows that our proposed S-CLIP model outperforms others, even in this general image domain.\n\n\\begin{array}{lcc}\n\\text{Method} & \\text{Image-to-text retrieval} & \\text{Text-to-image retrieval} \\newline\n\\hline\n\\text{CLIP (original)} & 40.30 & 38.17 \\newline\n\\text{CLIP (fine-tune)} & 46.59 & 47.76 \\newline\n\\text{Hard-PL} & 48.72 & 47.12 \\newline\n\\text{Soft-PL} & 48.72 & 47.23 \\newline\n\\text{S-CLIP (ours)} & 49.79 & 48.40 \\newline\n\\end{array}\n\n---\n**[Q1] Usage of “pre-training” in the title**\n\nWe used the term \"pre-training\" to highlight our goal of training a vision-language model capable of zero-shot classification and image-text retrieval. Although our approach can pre-train models from scratch, we fine-tuned CLIP to transfer its learned representations. Nevertheless, based on your suggestion, we will revise our paper title to use the term \"learning\" instead of \"pre-training\" to be less confusing.\n\n---\n**[Q2] Number of keywords per caption**\n\nWe trained our models using captioned datasets, while the class names were derived from the classification counterparts of the dataset. Consequently, captions often contain multiple class names. For instance, 45.8% of captions in the RSICD dataset contain more than two class names. However, we acknowledge that your point is fair; the remaining 50% of captions only have a single class name, resulting in our keyword-level loss reducing to a classification loss in such cases. It is worth noting that we developed a method to handle scenarios where class names are unavailable. By utilizing YAKE-200 keywords, we found that 94.8% of captions contain more than two keywords, thus confirming the necessity of partial label learning. We will incorporate this discussion in the final manuscript.\n\n---\nPlease let us know if you have any further concerns.\n\nSincerely,\\\nAuthors"}, {"review_id": "swZTB9VFOl", "reviewer": "Reviewer_Q2NM", "summary": "The paper presents S-CLIP, a novel semi-supervised learning method for training contrastive language-image pre-training (CLIP) models in specialized domains where limited image-text pairs are available. Vision-language models like CLIP have achieved impressive results in natural image domains, but they struggle when applied to specialized domains like remote sensing due to the scarcity of paired data. To address this, S-CLIP leverages additional unpaired images and introduces two pseudo-labeling strategies: caption-level and keyword-level pseudo-labeling.\n\nCaption-level pseudo-labeling assigns pseudo-labels to unlabeled images based on the captions of paired images. This is achieved by solving an optimal transport problem between the unlabeled and labeled images, ensuring that the pseudo-labels represent a probability distribution over the labeled images. On the other hand, keyword-level pseudo-labeling assigns pseudo-labels based on the keywords in the captions of visually similar paired images. It formulates the training as a partial label learning problem, considering a candidate set of target keywords instead of a single exact one.\n\nThe experiments conducted in various specialist domains, including remote sensing, fashion, scientific figures, and comics, demonstrate the effectiveness of S-CLIP. It outperforms CLIP fine-tuning and other semi-supervised learning competitors in zero-shot classification and image-text retrieval tasks. For example, in the remote sensing domain, S-CLIP improves zero-shot accuracy by 10.4% and image-to-text retrieval R@5 by 4.4% compared to CLIP fine-tuning. Moreover, S-CLIP remains robust even when the unlabeled images are from a different dataset. Overall, S-CLIP shows promise in enhancing CLIP training using only a few image-text pairs.", "questions": "- Could you clarify why the soft-PL, hard-PL were the only semi supervised baselines chosen for comparison despite the presence of a vast body of literature for semi supervised learning?", "limitations": "No ethical limitations as such", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Effective Pseudo-Labeling Strategies: The paper proposes two pseudo-labeling strategies, caption-level and keyword-level pseudo-labeling, which effectively guide the training process. The caption-level pseudo-labeling leverages optimal transport to assign pseudo-labels based on the captions of paired images, ensuring robust training even with distribution shifts. The keyword-level pseudo-labeling assigns pseudo-labels based on keywords in visually similar paired images, capturing local components of unlabeled images. The combination of these strategies results in improved zero-shot classification and image-text retrieval tasks.\n\n- Experimental Demonstrations: The paper provides extensive experimental evaluations in various specialist domains, including remote sensing, fashion, scientific figures, and comics. The results consistently show that S-CLIP outperforms CLIP fine-tuning and other semi-supervised learning competitors, achieving significant improvements in zero-shot classification and image-text retrieval tasks. The robustness of S-CLIP, even when unlabeled images are from a different dataset, further strengthens its effectiveness.\n\n- Relevance and Applicability: The paper addresses an important and practical problem in the field of vision-language models. The proposed S-CLIP method has the potential to enhance the training of CLIP models in specialized domains with limited paired data. The findings are relevant not only for researchers working on vision-language models but also for practitioners in various domains where image-text understanding is crucial.\n\n- Clarity and Presentation: The paper is well-written, organized, and presents the proposed method and experimental results clearly. The figures and illustrations aid in understanding the concepts and methodologies. The strengths and weaknesses of the proposed method are discussed in a concise and informative manner.", "weaknesses": "- Baselines with other semi supervised approaches are severely limited with the added baselines being severely outdated. Several much newer few shot learning approaches are present and should be compared against. A summary could be found in the paper: A Survey on Deep Semi-supervised Learning by Yang et al. for example: SEMI-MAE: MASKED AUTOENCODERS FOR SEMI-SUPERVISED\nVISION TRANSFORMERS by Yu et al or Semi-supervised Multimodal Representation Learning through a Global Workspace by Devillers et al\n- A lot of the core content like limitations, experimental setup etc has been moved to the appendix which I feel is key to the understanding of the paper and unfairly grants the author extra writing space\n- The approach combines keyword and caption level pseudo labelling, both of which have individually been well studied in literature. The combination of both is certainly a novel effort, but the contribution of the same doesnt seem like a strong enough novelty for a venue liek Neurips\n- The choice of using OT as the distance measure between 2 images is interesting, it would be nice to see how other image distance measures compare to it", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer Q2NM,\n\nThank you for your valuable feedback and comments. We appreciate your remarks on the strengths of our paper, including the problem significance, effective method, extensive experiments, and clear presentation. We will address your concerns and questions in the response below.\n\n---\n**[W1] Comparison to other semi-supervised learning approaches**\n\nIn our paper, we focus on developing a pseudo-labeling method for vision-language models, an underexplored research question in previous literature. As a result, we selected the two most representative pseudo-labeling approaches, Hard-PL and Soft-PL, as our baseline. These techniques can be integrated into existing semi-supervised learning frameworks, such as in combination with consistency regularization, as discussed in lines 100-102.\n\nFor instance, your suggested Semi-MAE integrates Hard-PL alongside the MAE loss, and we can easily incorporate this into our method by applying the MAE loss to unlabeled images. On the other hand, your suggested global workspace (GW) necessitates a decoder for each modality, like image VAE and text BERT. However, obtaining these pre-trained models may not be feasible for the specialist domains we are focusing on. Considering that existing methods either complement our work or are not directly applicable to our specialized domain setup, we believe our baselines are not outdated.\n\nNevertheless, for readers' further interest, we provide additional baselines that adapt the SOTA semi-supervised image classification methods. Specifically, we employ the pseudo-labeling technique of SemPPL [22] and RoPAWS [69], both published in ICLR 2023. SemPPL uses the mode of the k-NN prediction. In our case, since captions are unique for each image, we assign a uniform probability over k-NN samples, which we refer to as \"k-NN\" pseudo-labeling. On the other hand, RoPAWS utilizes an analytic fixed-point solution of label propagation, which we refer to as \"LabProp\" pseudo-labeling. We use these baselines for caption-label pseudo-labeling and compare them with OT. We do not apply keyword-level pseudo-labeling for ablation. The table below presents the results, following the setup of Table 5. OT outperforms other pseudo-labeling approaches.\n\n\n\\begin{array}{lcc}\n\\text{Method} & \\text{Zero-shot accuracy} & \\text{Image-text retrieval} \\newline\n\\hline\n\\text{k-NN (k=5)} & 61.1 & 8.8 \\newline\n\\text{LabProp} & 68.2 & 9.4 \\newline\n\\text{OT (ours)} & 69.6 & 10.5 \\newline\n\\end{array}\n\n---\n**[W2] Many contents are in the Appendix**\n\nWe made our best efforts to convey our main ideas within the limited nine pages. As a result, we moved limitations and experimental setups to the Appendix, considering them less essential but valuable information for in-depth readers. We will include the mentioned contents in the final manuscript, where an additional content page is allowed.\n\n---\n**[W3] Combination of known pseudo-labeling techniques**\n\nWe note that the use of pseudo-labels for image-caption pairs was an underexplored research question. As a result, the concepts of caption-level and keyword-level pseudo-labeling were not well-studied in the literature. Our contribution lies in the design of these approaches rather than simply combining existing techniques. In the subsequent paragraphs, we recap the discussions on how our proposed approaches differ from the existing pseudo-labeling techniques for image classification.\n\n*Caption-level pseudo-labeling.* As we discussed in lines 631-636 in the Appendix, the OT-based approach, similar to our caption-level pseudo-labeling, was previously studied for semi-supervised image classification. However, we argue that the balancing effect of OT is more critical for vision-language models, given the inherent distribution shifts between labeled and unlabeled image captions. Our novel problem setup enhances the practical applicability of OT, which has been less recognized in the literature.\n\n*Keyword-level pseudo-labeling.* The need for partial label learning arises due to the possibility of a caption containing multiple keywords, which does not happen for image classification tasks. Our proposed technique is designed to address this unique problem in semi-supervised vision-language models, which is novel to our best knowledge.\n\n---\n**[W4] Other distance measures than OT**\n\nWe compared several distance measures to define pseudo-labels, including Soft-PL and OT, based on the embedding similarities between image-image and image-text. For instance, Table 5e illustrates that image-image similarity outperforms image-text similarity. To further demonstrate the impact of distance measures, we present additional results in the table below. We solely apply the caption-level pseudo-labeling for ablation. We follow the setup of Table 5 but use the L$\\neq$U setup instead of the L=U setup to emphasize the robustness of image-image distance. The results confirm the significance of the proper choice of the distance measure. Exploring other distances would be an interesting future direction.\n\n\n\\begin{array}{lcc}\n\\text{Distance} & \\text{Zero-shot accuracy} & \\text{Image-text retrieval} \\newline\n\\hline\n\\text{Soft-PL (image-text)} & 50.3 & 8.2 \\newline\n\\text{Soft-PL (image-image)} & 60.6 & 8.7 \\newline\n\\text{OT (image-text)} & 64.9 & 8.9 \\newline\n\\text{OT (image-image)} & 69.1 & 9.7 \\newline\n\\end{array}\n\n---\n**[Q1] Why using Hard-PL and Soft-PL as baselines**\n\nAs discussed in W1, our focus is on designing pseudo-labels for image-caption pairs, which are a core component of semi-supervised learning and are orthogonal to other parts such as self-supervision (or consistency regularization). Therefore, we have chosen two of the most representative pseudo-labeling techniques, Hard-PL and Soft-PL, as our baseline.\n\n---\nPlease let us know if you have any further concerns.\n\nSincerely,\\\nAuthors"}]

Author rebuttal:
To answer the Q3 of reviewer EbWV, we have attached a PDF containing qualitative examples of caption-level and keyword-level pseudo-labels.

Meta-review: This paper proposes to enhance CLIP in a semi-supervised way with the help of captions. The proposed approach could benefit domains where image-text data are difficult to collect.

Concerns were raised mainly regarding the implementation details and baseline comparisons. The authors provided a rebuttal and after the discussion period, all reviewers vote for acceptance. The AC recommends acceptance, and encourages the authors to revise paper accordingly.
Decision: Accept (poster)