Title: S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions
Venue: NeurIPS 2023
Accepted: yes
Keywords: vision-language model

semi-supervised learning

specialist domain
Abstract: Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs.

S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions

Abstract

Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. SCLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by 10% for zero-shot classification and 4% for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs. 1

1 Introduction

Pre-trained vision-language models have achieved remarkable success, providing a foundation for numerous downstream tasks [1]. However, these models often struggle when applied to specialized domains such as remote sensing or medical imaging [2]. This is because the models are trained on web-crawled data that may not fully capture the diversity and complexity of these domains [3, 4]. To address this issue, previous research has focused on constructing large-scale pre-training datasets for each domain [5, 6]. However, annotating caption for each specialized domain can be expensive and time-consuming, which limits their applicability across various domains.

Several studies have attempted to reduce the number of image-text pairs used for vision-language pre-training. However, these approaches often rely on additional information, such as pre-trained object detectors [7-9], or class-annotated images [10], which may not be applicable to specialized domains. Other approaches leverage self-supervised learning to utilize unpaired data [11-14], but they do not fully exploit the information provided by image-text pairs. Research on leveraging a few image-text pairs to improve vision-language pre-training remains underexplored.

In various areas of machine learning, particularly in image classification, semi-supervised learning [15] is a popular approach for training a model using limited annotations. This technique typically relies on pseudo-labeling [16], which predicts labels for unlabeled data by propagating information

https://github.com/alinlab/s-clip

'the

huge rectangle

contains a

baseball field

,

a

swimming pool

,

two basketball courts

and

eight tennis courts

'

'

two tennis courts

and some

buildings

are in two sides of a

road

'

Nearest image

in the batch

Their captions are different! (naive pseudo-labeling  fails)

S-CLIP

(ours)

Hard-PL

CLIP-FT

CLIP

66.9

(+8.6)

55.6

(-2.7)

58.3

45.3

Zero-shot accuracy (%)

on RSICD-CLS

(a) Problem of naive pseudo-labeling (use the nearest caption)

(b) Comparison of models

from labeled data and refining the model using them. Combining this approach with self-supervised learning [17] has significantly improved classifiers using only a few class labels [18-22].

A natural question is whether semi-supervised learning, particularly pseudo-labeling, can improve vision-language pre-training. However, the techniques from image classification cannot be directly applied to vision-language models since captions are diverse and often uniquely associated with each image. Thus, the naive pseudo-labeling approach of assigning the caption of the nearest labeled image 2 (Hard-PL) [16] can mislead the models. Figure 1a demonstrates that even visually similar images often have different captions, although they may share keywords such as 'tennis court.' As a result, Figure 1b shows that Hard-PL harms the performance of CLIP fine-tuning. This observation motivates the proper design of pseudo-labeling method that effectively utilizes captions.

Contribution. We propose S-CLIP, a novel semi-supervised learning method for vision-language pretraining, particularly CLIP [23]. S-CLIP introduces two novel pseudo-labeling methods specifically designed for contrastive learning and the language modality, as illustrated in Figure 2:

Caption-level pseudo-label. We assume that the semantics of an unlabeled image can be expressed as a combination of those of labeled images. To achieve this, we define the caption-level pseudolabel as a probability distribution over the labeled images, which guides the relationship of an unlabeled image with its corresponding captions. Specifically, the pseudo-labels are obtained by solving an optimal transport [24, 25] problem between the unlabeled and labeled images. This approach prevents pseudo-labels from collapsing to a few captions and ensures robust training, particularly when dealing with distribution shifts between unlabeled and labeled images.

Keyword-level pseudo-label. We assume that an unlabeled image shares keywords with visually similar images, even if their full captions are not identical. Therefore, we define the keyword-level pseudo-label as one of the keywords in the nearest labeled image to an unlabeled image. This approach creates a candidate set of target keywords instead of a single exact one, and the training can be formulated as a partial label learning [26] problem. This loss helps the model understand the local components of unlabeled images, leveraging the structure of language.

We note that both pseudo-labels are complementary. Caption-level pseudo-label helps the model understand the global structure of language, which is more effective for image-text retrieval. Keywordlevel pseudo-label helps the model understand the local phrases, which is more effective for zero-shot classification. Combining them achieves the best of both worlds (Section 5.5).

We demonstrate the effectiveness of our method in various specialist domains with limited available image-text pairs, including remote sensing [27], fashion [28], scientific figures [29], and comics [30] domains. In the remote sensing domain, S-CLIP outperforms CLIP fine-tuning and semi-supervised learning competitors in five zero-shot classification and six image-text retrieval tasks. For instance, S-CLIP improves zero-shot accuracy on WHU-RS19 [31] by 10.4% and image-to-text retrieval R@5

=

+

...

0.4

×

Pseudo-label =

Combination of the captions

(from visually similar images in the batch)

0.3

×

'…

tennis courts

…'

'…

buildings

…'

Candidate set of keywords

(from the caption of the nearest image)

Pseudo-label =

huge rectangle

baseball field

swimming pool

tennis courts

∈

(a) Caption-level pseudo-label

(b) Keyword-level pseudo-label

on UCM [32] by 4.4% compared to CLIP fine-tuning. S-CLIP remains robust even when unlabeled images are drawn from a different dataset. In the comics domain, the Simpsons [30] dataset only includes 800 image-text pairs. Therefore, we incorporate the Simpsons Characters [33] dataset, which consists of 30,000 images without captions. Here, S-CLIP improves the text → image retrieval R@1 of CLIP from 11.8% to 15.8%, providing a relative gain of 33%.

2 Related work

Vision-language pre-training (VLP). Pre-training vision-language models has achieved remarkable success, providing a foundation for many downstream tasks [1]. Numerous approaches have been proposed, including reconstructing masked inputs [34-38], learning a joint embedding of vision and language modalities [23, 39-42], generating language descriptions from image inputs [43-45], and connecting pre-trained unimodal image and language models [46-48]. CLIP [23] is a representative model that learns a joint embedding between vision and language modalities. CLIP has demonstrated effectiveness in various applications, including zero-shot classification [23], image-text retrieval [49], open-vocabulary object segmentation [50, 51], and out-of-distribution detection [52].

VLP for specialist domains. Despite their success in natural image domains, VLP models often struggle when applied to specialist domains like remote sensing [2]. This is because they are trained on web-crawled data that may not capture the diversity and complexity of these domains [3, 4]. Data in these fields is often scarce, has limited accessibility, and requires expert annotations, making it difficult to obtain the image-text pairs needed for VLP training. While attempts have been made to train VLP models for specialized domains, including medical [10, 53-59], fashion [60-63], and remote sensing [64], previous research has mainly focused on constructing large-scale pre-training datasets for each domain [5, 6], which can be expensive and time-consuming.

VLP with limited pairs. Several studies trained VLP models with a limited number of image-text pairs. Hsu et al. [65] used adversarial domain adaptation to regularize unpaired data embeddings similar to paired data. However, this approach can cause problems when the unpaired data is drawn from a different distribution. MedCLIP [10] utilized images with class labels to reduce the need for captions. Nevertheless, it still requires class labels, limiting the use of large unlabeled data. Other studies explored unsupervised or semi-supervised VLP training in image domains [7-9]. However, they rely on a pre-trained object detector to align detected objects with keywords in captions, making them unsuitable for specialist domains. Other works leverage self-supervised learning on each modality [11-14], but do not fully exploit the information provided by image-text pairs. In contrast, our method leverages image-text pairs and is generally applicable without extra information.

Semi-supervised learning. Semi-supervised learning [15] aims to train a model using a small labeled dataset and a large unlabeled dataset. Techniques for semi-supervised learning can be categorized into two types: pseudo-labeling (or label propagation) [16, 66], which utilizes information from the small labeled dataset, and self-supervised learning [17], which learns representations from the unlabeled dataset. Combining these techniques has achieved remarkable success in semi-supervised learning, particularly for training classifiers with few class labels [18-22]. Prior works on robust semi-supervised learning focus on scenarios where the distributions of unlabeled and labeled data

shift [67-69]. This issue naturally arises in vision-language pre-training due to the uniqueness of captions for each image. Our method addresses this issue using soft labels and performs robustly even when the unlabeled images are from a different dataset than the labeled images.

3 Background

Before we demonstrate our semi-supervised vision-language pre-training method, we provide a brief overview on contrastive language-image pre-training and semi-supervised learning.

3.1 Contrastive language-image pre-training

Contrastive language-image pre-training (CLIP) [23] is a popular vision-language model that learns a joint embedding space connecting images x and texts y . 3 CLIP is trained using a batch of paired images and texts { x i , y i } N i =1 through contrastive learning, where those from the same pair are treated as positive and different pairs as negative. The training objective of CLIP involves two classification tasks: predicting a text given an image p ( y | x ) and predicting an image given a text p ( x | y ) . Labels are assigned to the N samples in the batch, corresponding to their pairs as the target. The embeddings are normalized to have a unit norm, and CLIP predicts the label using a softmax function with a temperature parameter τ > 0 , applied to the cosine similarity of the embeddings.

Specifically, we denote the softmax classifier σ τ as a function of input and target embeddings. This allows us to represent p ( y | x ) = σ τ ( x, { y i } N i =1 ) ∈ R N and p ( x | y ) = σ τ ( y, { x i } N i =1 ) ∈ R N , which denote the class probabilities of CLIP. These probabilities are calculated as follows: p ( y = y i | x ) = exp( x · y i /τ ) / ( ∑ N j =1 exp( x · y j /τ )) . The CLIP model minimizes the following objective:

where H denotes the cross-entropy loss and e i ∈ R N is a one-hot vector with the i -th element being one. The CLIP model can be used for zero-shot classification or image-text retrieval by computing cosine similarity in the embedding space. For example, in zero-shot classification, the text embedding of the class prompt that is closest to an image embedding can be identified.

3.2 Semi-supervised learning

Semi-supervised learning (Semi-SL) [15] aims to train a model using a small number of labeled data { x i , y i } N i =1 and a large number of unlabeled data { u i } i M =1 , i.e., when M ≫ N . Various SemiSL methods have been proposed to enhance neural network performance, particularly for image classification [19-22]. 4 These methods typically rely on pseudo-labels [16], obtained by predicting classes for unlabeled data using a model. The model is then refined by minimizing the cross-entropy loss 1 M ∑ M i =1 H ( p ( y | u i ) , q i ) with respect to the prediction p ( y | u i ) ∈ R C and pseudo-label q i ∈ R C for unlabeled data u i . Pseudo-labels can take the form of a hard label, represented by a one-hot vector identifying the class with the highest probability [16], or a soft label that calibrates this hard label [21]. Pseudo-labels minimize prediction entropy by sharpening specific classes [70].

Semi-SL for CLIP. Assigning pseudo-labels in CLIP poses challenges due to the unique captions associated with each image. The assumption that unlabeled data belongs to the same class as labeled data does not hold here. Therefore, we propose a novel Semi-SL method for CLIP.

4 S-CLIP: Semi-supervised Vision-Language Pre-training

We extend the training of CLIP to incorporate unpaired images { u i } i M =1 alongside the image-text pairs { x i , y i } N i =1 . To achieve this, we introduce S-CLIP, which integrates two novel pseudo-labeling methods considering contrastive learning and the language modality. The conceptual illustrations of the proposed pseudo-labels and training objectives are shown in Figure 2 and Figure 3, respectively.

Captions

Paired images

𝑦

!

𝑦

"

𝑦

#

⋯

𝑥

!

𝑥

"

𝑥

#

⋯

One-hot labels

(

■

from true pairs)

(a) CLIP loss

L

CLIP

Captions

Unpaired images

𝑦

!

𝑦

"

𝑦

#

⋯

Soft labels

(

■

from OT plan

Γ

)

𝑢

!

𝑢

"

𝑢

$

⋯

(b) Caption loss

L

caption

Keywords

Unpaired images

𝑘

!

𝑘

"

𝑘

#

⋯

Sparse soft labels

(

■

from keyword set

K

$

)

𝑢

!

𝑢

"

𝑢

%

⋯

L

keyword

4.1 Caption-level pseudo-label

Although the semantics of an unlabeled image u may not precisely match any captions y i , visually similar images often share similar semantics. Therefore, the caption for unlabeled image u can be expressed as a combination of the captions { y i } . To achieve this, we use caption-level pseudo-labels, which represent a probability distribution over the captions. These pseudo-labels serve as the target for the N -way classification problem in contrastive learning, with a batch size of N . Figure 2a demonstrates how caption-level pseudo-label for an unlabeled image is obtained.

The pseudo-labels are derived from the relationship between unlabeled and labeled images, formulated as an optimal transport (OT) [24] problem. We define the cost function C ∈ R M × N , where M and N represent the number of unlabeled and labeled images, respectively. The cost is given by the negative cosine similarity between the normalized embeddings of the unlabeled image u i and the labeled image x j , i.e., C ij = 1 -u i · x j . Then, we solve following entropic regularized OT [25] problem:

Here, Π( p , q ) represents a set of transportation plans that satisfy the flow constraint. The sums of flows from sources and to sinks match the vectors p ∈ R M + and q ∈ R N + , respectively, with both p and q adding up to one. We set them as uniform probabilities, and they worked in our experiments, even when unlabeled and labeled data have distribution shifts. 1 N denotes the all-one vector of dimension N . The entropic regularizer H (Γ) = -∑ i,j Γ ij log Γ ij with scale λ smooths the transportation plan, enabling an efficient solution using the Sinkhorn-Knopp [25] algorithm.

Once we obtain the transportation plan Γ , we define the pseudo-label for an unlabeled image u i by normalizing its plan to sum up to one. Specifically, the j -th element of the pseudo-label q i ∈ R N is computed as Γ ij / ∑ j Γ ij . This pseudo-label is then utilized to guide the probability distribution over N captions, where the prediction p ( y | u i ) ∈ R N is determined by a softmax classifier σ τ ( u i , { y i } N i =1 ) . The loss function for the caption-level pseudo-label aims to minimize the following:

The prediction p ( y | u i ) relates the unlabeled image u i to the text embeddings y , while the pseudolabel q i relates it to the image embeddings x . Consequently, the proposed loss function aligns the relationships between image and text embeddings, as suggested in supervised CLIP [71-73]. In our experiments, we compare the usage of text and image embeddings for pseudo-label computation, and using image embeddings outperformed using text embeddings due to their robustness in novel specialist domains. We also explore computing pseudo-embeddings [9], but it yields inferior results as synthesizing pseudo-embeddings is more challenging than inferring pseudo-labels.

Effect of OT. Optimal transport balances the flow between unlabeled and labeled images, ensuring robust pseudo-label assignments without concentration on a few labels. In vision-language tasks, diverse and imbalanced captions often cause image embeddings to focus on a few nearest captions, resulting in the collapse of embeddings. This challenge is magnified when employing semi-supervised learning with distribution shifts [67]. We empirically validate the detrimental effect of naive pseudolabeling approaches, especially in the presence of distribution shifts.

Compared to soft-nearest neighbor pseudo-labels (Soft-PL) [21], our method achieves balance through Sinkhorn iterations. Specifically, it reverts to Soft-PL when the iteration count is zero. In our experiments, we use 10 iterations, and increasing the iteration count has only a marginal impact on performance. The computational cost is negligible as the process occurs in a low-dimensional embedding space. We set the scale of the entropic regularizer to match the softmax temperature, λ = τ . This adjustment scales the cost function by 1 /τ , aligning it with the cosine similarity scale learned by CLIP. More information on the OT and Soft-PL relationship is in Appendix A.

OT for vision-language models. Several works have utilized OT for vision-language models, but for problems other than semi-supervised learning. We discuss these works in Appendix B.

4.2 Keyword-level pseudo-label

We propose a keyword-level pseudo-labeling approach to overcome the limitations of caption-level pseudo-labels in capturing the meanings of words in captions. By leveraging the compositionality of language, we assume that visually similar images may share keywords, even if the entire captions differ. This enables us to guide unlabeled images using keywords from their nearest neighbors. It is important to note that an image may have multiple keywords, and visually similar images share only a subset. This aligns with the partial label learning (PLL) [26] problem, where the target label is unknown, and we have access to a candidate set of labels. For instance, in Figure 1a, the image shares the keyword 'tennis court' with its nearest neighbor but not 'baseball field.' Consequently, the nearest neighbor provides a candidate set of keywords, as illustrated in Figure 2b.

To identify candidate keywords for an unlabeled image u , we assume a pre-defined set of keywords with corresponding embeddings K = { k i } K i =1 . These keywords can be obtained from class names, if available, or extracted by using algorithms like YAKE [74] applied to captions. Given an unlabeled image u , we find the nearest labeled image x using the OT assignments from the previous section. Let { c 1 , . . . , c l } be the indices of keywords present in labeled image x , denoted as K u = { k c 1 , . . . , k c l } . This forms the candidate set of target keywords for u . While we simply use the keywords explicitly from the caption, some additional steps can be taken to infer synonyms or related concepts.

In the context of partial label learning, we assume that the ground-truth target label for the unlabeled image u belongs to the candidate set K u . PLL aims to minimize the loss by selecting the minimum value among candidate labels, min k ∈ K u H ( p ( k | u ) , k ) , rather than specifying a single target label. However, relying on a single minimum can be risky. Previous works have addressed this by using soft labels for candidates [75, 76]. This involves minimizing H ( p ( k | u ) , q ) , where q ∈ R K is a soft label with positive values for keywords in the candidate set and zero for others.

Similarly, we define the pseudo-label q i for the unlabeled image u i based on the similarity between the embeddings of u i and the keywords k j in the candidate set K u . Specifically, the j -th element of the pseudo-label q i is calculated using a softmax function σ τ ( u i , K u ) for k ∈ K u and is set to zero otherwise. This pseudo-label is then utilized to guide the probability distribution over K keywords, where the prediction p ( k | u ) ∈ R K is determined by a softmax classifier σ τ ( u i , K ) . The loss function for the keyword-level pseudo-label aims to minimize the following:

Both the prediction p ( k | u i ) and the pseudo-label q i are related to the unlabeled image and keywords. However, the restricted candidate set effectively guides the learning process through PLL.

Training objective. We apply the CLIP loss to paired images and texts and the proposed pseudo-label losses to unpaired images. The pseudo-label losses are halved to match the scale of the image part in the CLIP loss. In summary, our final training objective becomes L CLIP + 1 / 2 ( L caption + L keyword ) .

̸

̸

5 Experiments

We demonstrate the effectiveness of our method in various specialist domains with limited available image-text pairs, including remote sensing, fashion, scientific figures, and comics domains.

Setup. Weuse a pre-trained CLIP model on natural images [23], called CLIP (original). However, this model often struggles in specialist domains. Therefore, we fine-tune the model with domain-specific image-text pairs, called CLIP (fine-tune). We compare this supervised model with semi-supervised methods using unpaired images. We use a batch size of 64 per GPU, with a total of 4 GPUs. To ensure fair GPU memory usage in semi-supervised learning, we employ 32 image-caption pairs and 32 unpaired images for each mini-batch. Models are evaluated on zero-shot classification and image-text retrieval tasks, measuring Top-1 classification accuracy (%) and recall at K (R@K). We report the average and standard deviation across three random seeds. We follow the training recipe of OpenClip [77] if not specified. Additional experimental details are in Appendix C.

5.1 Baselines

Our proposed S-CLIP includes two novel pseudo-labeling methods. Thus, we mainly compare SCLIP with other pseudo-labeling approaches, which are extensions of semi-supervised classification techniques. Our baselines, Hard-PL and Soft-PL, extend the hard (top-1) and soft nearest neighbor approaches for pseudo-label assignment. We then utilize these methods as caption-level pseudo-labels, similar to Eq. (3). The specific forms of each method are as follows.

Hard-PL. Given an unlabeled image u i , Hard-PL identifies the top-1 nearest labeled image 5 x i ∗ based on their cosine similarities, calculated as i ∗ = arg max j ( u i · x j ) . Subsequently, the pseudolabel of u i is determined by the corresponding caption, denoted as q i = y i ∗ . This pseudo-label serves as the target for predicting p ( y | u i ) , and the loss is computed as ∑ i H ( p ( y | u i ) , q i ) .

Soft-PL. Soft-PL calibrates the pseudo-label of Hard-PL by considering all the relations between labeled and unlabeled images, known as soft nearest neighbor [21], instead of selecting only the top-1 nearest image. For an unlabeled image u i and a set of labeled images { x i } , these relations are determined by applying softmax to their cosine similarities. The pseudo-label is defined as q i = σ τ ( u i , { x i } N i =1 ) ∈ R N , where the pseudo-label has a higher value close to one for visually similar images. We use the same softmax temperature τ as CLIP, which is used to compute the similarities between the labeled image and text embeddings. The loss is computed as ∑ i H ( p ( y | u i ) , q i ) .

5.2 Remote sensing datasets

We train vision-language models using the union of RSICD [27], UCM [32], and Sydney [78], named RS-ALL, following the setup of [64]. For semi-supervised learning, we subsample 10% of image-text

̸

̸

pairs as labeled data (L), while the remaining 90% of images (L=U) or unlabeled images from the RESISC45 [79] dataset (L = U) served as unlabeled data. In Appendix D, we demonstrate that our method consistently gives improvements over CLIP across various setups. These include different neural architectures, varied ratios of image-text pairs, and even scenarios where we use all image-text pairs from RS-ALL in conjunction with unlabeled images from RESISC45.

For zero-shot classification, we use the validation sets of the classification versions of the RSICD and UCM datasets, denoted as RSCID-CLS and UCM-CLS, respectively. To evaluate the generalization abilities, we test models on unseen datasets such as WHU-RS19 [31], RSSCN7 [80], and AID [81]. For image-text retrieval, we use the validation sets of the RSICD, UCM, and Sydney datasets.

̸

Zero-shot classification. Table 1 presents the zero-shot classification results. S-CLIP consistently and significantly outperforms all supervised CLIP fine-tuning and semi-supervised methods. One can make several observations. Firstly, fine-tuning CLIP improves performance over the original CLIP, highlighting the importance of adapting models to the target specialist domain. Secondly, our proposed pseudo-labeling techniques are crucial, as naive pseudo-labeling often harms accuracy. Thirdly, semi-supervised learning methods improve performance on unseen datasets, such as RSSCN7, demonstrating the usefulness of observing more unlabeled images in generalization. Lastly, our method is robust to the distribution shifts (L = U) setup, which leverages an external source of unlabeled images, a common scenario in practice. In contrast, Soft-PL improves performance in the same distribution (L=U) setup but fails on the distribution shifts scenario, confirming the effectiveness of optimal transport in balancing the assignments of pseudo-labels.

Image-text retrieval. Table 2 presents the image-text retrieval results. S-CLIP consistently improves both image → text and text → image retrieval, except in one case where the supervised baseline performs the best. This confirms that S-CLIP also benefits fine language understanding for retrieval. The trend is consistent across evaluation metrics, as seen in the R@1 results in the appendix.

5.3 Fashion datasets

We train vision-language models using the union of Fashion200k [28], FashionGen [82], and Polyvore Outfits [83] datasets. We subsample 10% of the image-text pairs as labeled data and the remaining 90% of images as unlabeled data. We evaluate the zero-shot accuracy on the validation sets of all three datasets, considering both super-class and sub-class accuracies for Fashion200k and FashionGen. The class names in Polyvore match the same level of super-class as those in the other datasets.

Table 3 presents the zero-shot classification results. S-CLIP consistently outperforms all supervised and semi-supervised methods. The improvement is more significant for super-class classification but less pronounced for sub-class classification and image-text retrieval. This is because the current pseudo-labeling approach assumes that the semantics of unlabeled image can be represented by the captions in a batch. However, this assumption may not hold true for fine-grained semantics, as certain fine-grained captions may be missing from the batch. Therefore, our method could be further enhanced by increasing the batch size or incorporating a queue of caption embeddings [84, 85].

5.4 More captioning datasets

We conduct experiments on two more specialist domains, namely science figures and comics. For this purpose, we use the SciCap [29] and Simpsons [30] datasets. We train the models on each dataset and evaluate their image-text retrieval performance on the validation sets. The results are presented in Table 4, and detailed discussions can be found in the following paragraphs.

SciCap. We subsample 10% of the image-text pairs as labeled data and the remaining 90% of images as unlabeled data. S-CLIP improves upon CLIP, even for the scientific figures domain, which exhibits a substantial gap from the natural image domain that CLIP is pre-trained on. This confirms the practical impact of S-CLIP in extending the applicability of CLIP to various specialist domains.

Simpsons. The Simpsons dataset contains only 800 image-text pairs, so we incorporate unlabeled images from another dataset called Simpsons Characters [33]. S-CLIP significantly improves CLIP, even in the presence of distribution shifts between unlabeled and labeled images. For example, it boosts the text → image retrieval R@1 from 11.8% to 15.8% (+33%). This confirms the practical impact of S-CLIP in specialist domains with limited available image-text pairs.

5.5 Ablation studies

We conduct an ablation study on the design choices for S-CLIP. The results are reported in Table 5, which compares the average values of zero-shot accuracy (across five tasks in Table 1) and image-text retrieval R@1 (across six tasks in Table 2) for each design. The observations are as follows:

(a) Training objective. Caption-level pseudo-labels are more beneficial for image-text retrieval, while keyword-level pseudo-labels are better for zero-shot classification. Intuitively, keyword-level pseudo-labels help the model understand specific words, while caption-level pseudo-labels aid in fine-grained language comprehension. Combining both produces the best results.

(b) Sinkhorn iteration. Sinkhorn iterations significantly improve Soft-PL (= Iter. 0) by balancing pseudo-labels. However, it saturates after 10 iterations, so we have used it as the default in our experiments. Increasing the number of iterations ensures a more even distribution of pseudolabels. For example, the pseudo-labels from Soft-PL often collapse to some index in the batch. Sinkhorn iterations effectively regularize this behavior, reducing the maximum confidence of

̸

prediction from 100% to 40%. This regularization is more important in the L = U setup, where Soft-PL often harms the performance, while ours consistently provides improvement.

(c) Choice of keywords. We compare class names and YAKEK , which implies that K keywords are extracted using the YAKE [74] algorithm. Both types of keywords work well, although class names provide more consistent improvement. YAKE provides both nouns, such as 'building,' and adjectives like 'green,' 'many,' or 'large,' in contrast to the class names, which consist only of nouns. This enables YAKE to understand diverse semantics beyond object recognition, making keyword-level pseudo-labeling applicable to any domain. This applicability is further confirmed on the SciCap and Simpsons datasets, where class names are unavailable.

(d) Pseudo-target. Given a pseudo-label q i ∈ R N for an unlabeled image u i , one can synthesize a pseudo-embedding z i = ∑ j q ij · y j ∈ R d , where q ij denotes the j -th element of q i , and d is the dimension of the embeddings y j . Then, one can consider { u i , z i } as paired data and apply contrastive learning to it in addition to the true pairs { x i , y i } . However, we found that pseudo-embedding is less effective than pseudo-labeling. This is because accurately estimating the pseudo-embedding is harder than assigning soft probabilities to the true embeddings. Additionally, the pseudo-embeddings lie on an interpolation of true embeddings, which can introduce confusing negatives and potentially harm the effectiveness of contrastive learning.

̸

(e) Embedding for OT. We compute pseudo-labels by examining the relationship between unlabeled images and either labeled images or labeled text directly. The relationship with labeled images proves to be more effective in assigning pseudo-labels. Specifically, using text embeddings directly also works reasonably well in the L = U setup. However, it often fails entirely in the L = U setup. This is because the corresponding captions for unlabeled images are unknown. In contrast, visual similarities can be robustly computed even when the images are unseen.

(f) Partial label loss. We use the soft label given by σ τ ( u i , K u ) as the target for PLL. We also test the hard version, using the top-1 similar keyword from the candidate set, i.e., i ∗ = arg max j ( u i · k j ) . However, it performs worse, aligning with the observations from prior PLL literature.

We provide additional ablation studies in Appendix E, including results in the general image domain, additional pseudo-labeling baselines, qualitative examples of pseudo-labels, training curves, regularized fine-tuning approaches, selection of mini-batches during training, additional baselines using keyword information, and statistics of keywords in captions.

6 Conclusion

We propose S-CLIP, a semi-supervised extension of CLIP that leverages unpaired images to train in specialist domains with limited image-text pairs. We demonstrate its superiority across various domains such as remote sensing, fashion, scientific figures, and comics. We hope that S-CLIP expands the applicability of CLIP. Limitations and broader impacts are discussed in Appendix F.

References

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258 , 2021.

Y. Kim, S. Mo, M. Kim, K. Lee, J. Lee, and J. Shin. Bias-to-text: Debiasing unknown visual biases through language interpretation. arXiv preprint arXiv:2301.11104 , 2023.

P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Conference of the Association for Computational Linguistics , 2018.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

A. E. Johnson, T. J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C.-y. Deng, R. G. Mark, and S. Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Nature scientific data , 2019.

X. Yang, H. Zhang, D. Jin, Y. Liu, C.-H. Wu, J. Tan, D. Xie, J. Wang, and X. Wang. Fashion captioning: Towards generating accurate descriptions with semantic rewards. In European Conference on Computer Vision , 2020.

L. H. Li, H. You, Z. Wang, A. Zareian, S.-F. Chang, and K.-W. Chang. Unsupervised visionand-language pre-training without parallel images and captions. In Annual Conference of the North American Chapter of the Association for Computational Linguistics , 2021.

M. Zhou, L. Yu, A. Singh, M. Wang, Z. Yu, and N. Zhang. Unsupervised vision-and-language pre-training via retrieval-based multi-granular alignment. In Conference on Computer Vision and Pattern Recognition , 2022.

T.-J. J. Wang, J. Laaksonen, T. Langer, H. Arponen, and T. E. Bishop. Learning by hallucinating: Vision-language pre-training with weak supervision. In Winter Conference on Applications of Computer Vision , 2023.

Z. Wang, Z. Wu, D. Agarwal, and J. Sun. Medclip: Contrastive learning from unpaired medical images and text. In Conference on Empirical Methods in Natural Language Processing , 2022.

Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations , 2022.

N. Mu, A. Kirillov, D. Wagner, and S. Xie. Slip: Self-supervision meets language-image pre-training. In European Conference on Computer Vision , 2022.

E. Segal, B. Bogin, and J. Berant. Training vision-language models with less bimodal supervision. In Automated Knowledge Base Construction , 2022.

R. Windsor, A. Jamaludin, T. Kadir, and A. Zisserman. Vision-language modelling for radiological imaging and reports in the low data regime. arXiv preprint arXiv:2303.17644 , 2023.

O. Chapelle, B. Scholkopf, and A. Zien. Semi-supervised learning. 2006. Cambridge, Massachusettes: The MIT Press View Article , 2, 2006.

D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , 2013.

X. Liu, F. Zhang, Z. Hou, L. Mian, Z. Wang, J. Zhang, and J. Tang. Self-supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering , 2021.

D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Neural Information Processing Systems , 2019.

K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Neural Information Processing Systems , 2020.

Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Conference on Computer Vision and Pattern Recognition , 2020.

M. Assran, M. Caron, I. Misra, P. Bojanowski, A. Joulin, N. Ballas, and M. Rabbat. Semisupervised learning of visual features by non-parametrically predicting view assignments with support samples. In International Conference on Computer Vision , 2021.

M. Bošnjak, P. H. Richemond, N. Tomasev, F. Strub, J. C. Walker, F. Hill, L. H. Buesing, R. Pascanu, C. Blundell, and J. Mitrovic. Semppl: Predicting pseudo-labels for better contrastive representations. In International Conference on Learning Representations , 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , 2021.

C. Villani et al. Optimal transport: old and new . Springer, 2009.

M. Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. In Neural Information Processing Systems , 2013.

T. Cour, B. Sapp, and B. Taskar. Learning from partial labels. Journal of Machine Learning Research , 2011.

X. Lu, B. Wang, X. Zheng, and X. Li. Exploring models and data for remote sensing image caption generation. IEEE Transactions on Geoscience and Remote Sensing , 2017.

X. Han, Z. Wu, P. X. Huang, X. Zhang, M. Zhu, Y. Li, Y. Zhao, and L. S. Davis. Automatic spatially-aware fashion concept discovery. In Proceedings of the IEEE international conference on computer vision , pages 1463-1471, 2017.

T.-Y. Hsu, C. L. Giles, and T.-H. Huang. Scicap: Generating captions for scientific figures. In Findings of the Association for Computational Linguistics: EMNLP , 2021.

D. Adler. Simpsons blip captions. https://huggingface.co/datasets/Norod78/ simpsons-blip-captions , 2023.

G.-S. Xia, W. Yang, J. Delon, Y. Gousseau, H. Sun, and H. Maître. Structural high-resolution satellite image indexing. 2009.

Y. Yang and S. Newsam. Bag-of-visual-words and spatial extensions for land-use classification. In SIGSPATIAL international conference on advances in geographic information systems , 2010.

A. Attia. The simpsons characters data. https://www.kaggle.com/datasets/ alexattia/the-simpsons-characters-dataset , 2018.

L. H. Li, M. Yatskar, D. Yin, C.-J. Hsieh, and K.-W. Chang. Visualbert: A simple and performant baseline for vision and language. In Annual Conference of the Association for Computational Linguistics , 2020.

Y.-C. Chen, L. Li, L. Yu, A. El Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision , 2020.

W. Kim, B. Son, and I. Kim. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning , 2021.

Z. Wang, J. Yu, A. W. Yu, Z. Dai, Y. Tsvetkov, and Y. Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904 , 2021.

W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning , 2021.

H. Pham, Z. Dai, G. Ghiasi, H. Liu, A. W. Yu, M.-T. Luong, M. Tan, and Q. V. Le. Combined scaling for zero-shot transfer learning. arXiv preprint arXiv:2111.10050 , 2021.

X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Conference on Computer Vision and Pattern Recognition , 2022.

Y. Li, H. Fan, R. Hu, C. Feichtenhofer, and K. He. Scaling language-image pre-training via masking. arXiv preprint arXiv:2212.00794 , 2022.

K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In Conference on Computer Vision and Pattern Recognition , 2021.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning , 2022.

J. Yu, Z. Wang, V. Vasudevan, L. Yeung, M. Seyedhosseini, and Y. Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research , 2022.

M. Tsimpoukelli, J. L. Menick, S. Cabi, S. Eslami, O. Vinyals, and F. Hill. Multimodal few-shot learning with frozen language models. In Neural Information Processing Systems , 2021.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. In Neural Information Processing Systems , 2022.

J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomputing , 508:293-304, 2022.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations , 2022.

M. Xu, Z. Zhang, F. Wei, Y. Lin, Y. Cao, H. Hu, and X. Bai. A simple baseline for openvocabulary semantic segmentation with pre-trained vision-language model. In European Conference on Computer Vision , 2022.

J. Jeong, Y. Zou, T. Kim, D. Zhang, A. Ravichandran, and O. Dabeer. Winclip: Zero-/few-shot anomaly classification and segmentation. In Conference on Computer Vision and Pattern Recognition , 2023.

X. Wang, Y. Peng, L. Lu, Z. Lu, and R. M. Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In Conference on Computer Vision and Pattern Recognition , 2018.

Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine Learning for Healthcare Conference , 2022.

S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung. Gloria: A multimodal global-local representation learning framework for label-efficient medical image recognition. In International Conference on Computer Vision , 2021.

R. Liao, D. Moyer, M. Cha, K. Quigley, S. Berkowitz, S. Horng, P. Golland, and W. M. Wells. Multimodal representation learning via maximization of local mutual information. In Medical Image Computing and Computer Assisted Intervention , 2021.

B. Boecking, N. Usuyama, S. Bannur, D. C. Castro, A. Schwaighofer, S. Hyland, M. Wetscherek, T. Naumann, A. Nori, J. Alvarez-Valle, et al. Making the most of text semantics to improve biomedical vision-language processing. In European Conference on Computer Vision , 2022.

P. Müller, G. Kaissis, C. Zou, and D. Rueckert. Joint learning of localized representations from medical images and reports. In European Conference on Computer Vision , 2022.

S. Zhang, Y. Xu, N. Usuyama, J. Bagga, R. Tinn, S. Preston, R. Rao, M. Wei, N. Valluri, C. Wong, et al. Large-scale domain-specific pretraining for biomedical vision-language processing. arXiv preprint arXiv:2303.00915 , 2023.

D. Gao, L. Jin, B. Chen, M. Qiu, P. Li, Y. Wei, Y. Hu, and H. Wang. Fashionbert: Text and image matching with adaptive loss for cross-modal retrieval. In International ACM SIGIR Conference on Research and Development in Information Retrieval , 2020.

M. Zhuge, D. Gao, D.-P. Fan, L. Jin, B. Chen, H. Zhou, M. Qiu, and L. Shao. Kaleido-bert: Vision-language pre-training on fashion domain. In Conference on Computer Vision and Pattern Recognition , 2021.

S. Goenka, Z. Zheng, A. Jaiswal, R. Chada, Y. Wu, V. Hedau, and P. Natarajan. Fashionvlp: Vision language transformer for fashion retrieval with feedback. In Conference on Computer Vision and Pattern Recognition , 2022.

X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang. Fashionvil: Fashion-focused vision-and-language representation learning. In European Conference on Computer Vision , 2022.

A. Arampacha, V. Dev, G. Venkatesh, B. Mayank, G. Ritobrata, and P. Sujit. Fine tuning clip with remote sensing (satellite) images and captions. https://huggingface.co/blog/ fine-tune-clip-rsicd , 2021.

T.-M. H. Hsu, W.-H. Weng, W. Boag, M. McDermott, and P. Szolovits. Unsupervised multimodal representation learning across medical images and reports. In NeurIPS Machine Learning for Health Workshop , 2018.

Y. Bengio, O. Delalleau, and N. Le Roux. label propagation and quadratic criterion. 2006.

A. Oliver, A. Odena, C. A. Raffel, E. D. Cubuk, and I. Goodfellow. Realistic evaluation of deep semi-supervised learning algorithms. In Neural Information Processing Systems , 2018.

J. Park, S. Yun, J. Jeong, and J. Shin. Opencos: Contrastive semi-supervised learning for handling open-set unlabeled data. arXiv preprint arXiv:2107.08943 , 2021.

S. Mo, J.-C. Su, C.-Y. Ma, M. Assran, I. Misra, L. Yu, and S. Bell. Ropaws: Robust semisupervised representation learning from uncurated data. In International Conference on Learning Representations , 2023.

Y. Grandvalet and Y. Bengio. Semi-supervised learning by entropy minimization. In Neural Information Processing Systems , 2004.

J. Zhou, L. Dong, Z. Gan, L. Wang, and F. Wei. Non-contrastive learning meets language-image pre-training. arXiv preprint arXiv:2210.09304 , 2022.

S. Goel, H. Bansal, S. Bhatia, R. Rossi, V. Vinay, and A. Grover. Cyclip: Cyclic contrastive language-image pretraining. In Neural Information Processing Systems , 2022.

Y. Gao, J. Liu, Z. Xu, T. Wu, W. Liu, J. Yang, K. Li, and X. Sun. Softclip: Softer cross-modal alignment makes clip stronger. arXiv preprint arXiv:2303.17561 , 2023.

R. Campos, V. Mangaravite, A. Pasquali, A. Jorge, C. Nunes, and A. Jatowt. Yake! keyword extraction from single documents using multiple local features. Information Sciences , 509: 257-289, 2020.

J. Lv, M. Xu, L. Feng, G. Niu, X. Geng, and M. Sugiyama. Progressive identification of true labels for partial-label learning. In International Conference on Machine Learning , 2020.

H. Wang, R. Xiao, Y. Li, L. Feng, G. Niu, G. Chen, and J. Zhao. Pico: Contrastive label disambiguation for partial label learning. In International Conference on Learning Representations , 2022.

G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, 2021.

F. Zhang, B. Du, and L. Zhang. Saliency-guided unsupervised feature learning for scene classification. IEEE transactions on Geoscience and Remote Sensing , 2014.

G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE , 2017.

Q. Zou, L. Ni, T. Zhang, and Q. Wang. Deep learning based feature selection for remote sensing scene classification. IEEE Geoscience and Remote Sensing Letters , 2015.

G.-S. Xia, J. Hu, F. Hu, B. Shi, X. Bai, Y . Zhong, L. Zhang, and X. Lu. Aid: A benchmark data set for performance evaluation of aerial scene classification. IEEE Transactions on Geoscience and Remote Sensing , 2017.

N. Rostamzadeh, S. Hosseini, T. Boquet, W. Stokowiec, Y. Zhang, C. Jauvin, and C. Pal. Fashion-gen: The generative fashion dataset and challenge. arXiv preprint arXiv:1806.08317 , 2018.

M. I. Vasileva, B. A. Plummer, K. Dusad, S. Rajpal, R. Kumar, and D. Forsyth. Learning type-aware embeddings for fashion compatibility. In Proceedings of the European conference on computer vision (ECCV) , pages 390-405, 2018.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning , 2020.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Conference on Computer Vision and Pattern Recognition , 2020.

S. Yuan, K. Bai, L. Chen, Y. Zhang, C. Tao, C. Li, G. Wang, R. Henao, and L. Carin. Weakly supervised cross-domain alignment with optimal transport. In British Machine Vision Conference , 2020.

G. Chen, W. Yao, X. Song, X. Li, Y. Rao, and K. Zhang. Prompt learning with optimal transport for vision-language models. In International Conference on Learning Representations , 2023.

K. Kim, Y. Oh, and J. C. Ye. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprint arXiv:2301.12171 , 2023.

T. Gupta, A. Vahdat, G. Chechik, X. Yang, J. Kautz, and D. Hoiem. Contrastive learning for weakly supervised phrase grounding. In European Conference on Computer Vision , 2020.

B. Wu, R. Cheng, P. Zhang, P. Vajda, and J. E. Gonzalez. Data efficient language-supervised zero-shot recognition with optimal transport distillation. In International Conference on Learning Representations , 2022.

F. Taherkhani, A. Dabouei, S. Soleymani, J. Dawson, and N. M. Nasrabadi. Transporting labels via hierarchical optimal transport for semi-supervised learning. In European Conference on Computer Vision , 2020.

F. Taherkhani, A. Dabouei, S. Soleymani, J. Dawson, and N. M. Nasrabadi. Self-supervised wasserstein pseudo-labeling for semi-supervised image classification. In Conference on Computer Vision and Pattern Recognition , 2021.

D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman. With a little help from my friends: Nearest-neighbor contrastive learning of visual representations. In International Conference on Computer Vision , 2021.

M. Zheng, F. Wang, S. You, C. Qian, C. Zhang, X. Wang, and C. Xu. Weakly supervised contrastive learning. In International Conference on Computer Vision , 2021.

M. Azabou, M. G. Azar, R. Liu, C.-H. Lin, E. C. Johnson, K. Bhaskaran-Nair, M. Dabagia, B. Avila-Pires, L. Kitchell, K. B. Hengen, et al. Mine your own view: Self-supervised learning through across-sample prediction. arXiv preprint arXiv:2102.10106 , 2021.

P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. In Neural Information Processing Systems , 2020.

M. Assran, N. Ballas, L. Castrejon, and M. Rabbat. Supervision accelerates pre-training in contrastive semi-supervised learning of visual representations. arXiv preprint arXiv:2006.10803 , 2020.

B. Qu, X. Li, D. Tao, and X. Lu. Deep semantic understanding of high resolution remote sensing image. In International conference on computer, information and telecommunication systems , 2016.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition , 2016.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations , 2020.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision , 2014.

A Relationship between OT and Soft-PL

The entropic-regularized optimal transport problem described in Eq. (2) can be efficiently solved using the Sinkhorn-Knopp [25] algorithm. For our case, the cost function C ∈ R M × N is given by the negative cosine similarity: C ij = 1 -u i · x j . The Sinkhorn-Knopp algorithm proceeds as follows: first, it computes the exponentiated cost matrix K = exp( C -λ ) , where λ denotes the scale of the entropic regularizer. Then, the algorithm iteratively updates two vectors: u ∈ R M × 1 and v ∈ R 1 × N , which are initialized with uniform probability. The transportation plan Γ can be estimated using the exponentiated cost matrix K and the current vectors u and v . Specifically, it is given by Γ = u ⊙ K ⊙ v , where ⊙ denotes element-wise multiplication.

At the initialization zero, the transportation plan Γ is obtained by normalizing the exponentiated cost matrix K since the vectors u and v are uniform. Consequently, our caption-level pseudo-label q i is given by q i = K ij / ∑ j K ij , which is equivalent to the Soft-PL σ λ ( u i , x i ) except for the softmax temperature, which is denoted as λ . This observation motivates us to set λ = τ , matching the same temperature as CLIP, for computing the relation between embeddings. From this perspective, our proposed OT approach enhances the robustness of Soft-PL in pseudo-label assignment, especially in cases with distribution shifts between unlabeled and labeled images. Our ablation study demonstrates that increasing the number of Sinkhorn iterations improves the training process.

After initialization, the vectors u and v are updated to align with the exponentiated cost matrix K and satisfy the flow constraints p and q . Specifically, given the current vectors u ( t ) and v ( t ) , the next iteration of vectors is obtained as u ( t +1) = p K v ( t ) and v ( t +1) = q K ⊤ u ( t +1) . This iterative process converges to the optimal solution. Our ablation study demonstrates that a small number of iterations is sufficiently effective, and we employ 10 iterations in our experiments.

B Additional related work

Optimal transport for vision-language models. Several prior works have applied optimal transport to vision-language models. However, most of them focus on transportation between an image and a caption, connecting objects and phrases [35, 36, 86-88]. This is known as weakly supervised phrase grounding [89], and it assumes a supervised setting with image-text pairs. OTTER [90] calculates transportation between images and text, which aligns with our approach. However, they also assume a supervised setting and focus on calibrating the one-hot target of CLIP. To our knowledge, our work is the first to apply optimal transport for semi-supervised CLIP training.

Optimal transport for semi-supervised learning. Some prior works have utilized optimal transport for semi-supervised learning, specifically in image classification [91, 92]. These studies demonstrate the benefits of optimal transport in pseudo-labeling class labels. However, we argue that OT plays a more critical role in vision-language models due to inherent distribution shifts between labeled and unlabeled image captions. Our experiments reveal that naive pseudo-labeling often has detrimental effects, while OT significantly improves semi-supervised CLIP training.

Partial label learning. Partial label learning (PLL) [26] relaxes the classification problem by using a candidate set as the target instead of the exact one. This involves optimizing for the best candidate within the set. Instead of determining the hard assignment of the candidate, PRODEN [75] computes soft labels for the elements in the set to smooth the optimization process. PiCO [76] further enhances this approach by incorporating contrastive learning. Our work extends the techniques of PLL to a novel problem: keyword-level pseudo-labeling for vision-language models.

Label propagation in contrastive learning. The concept of pseudo-labeling (or label propagation) is also applied in image-only contrastive learning. In addition to using the same image with different augmentations [84, 85], some studies explore the utilization of neighboring images as positive pairs [93-95]. Similar concept can be extended to supervised [96] and semi-supervised [22, 97] contrastive learning, where images sharing the same class label are considered positive pairs.

C Experimental details

C.1 Dataset details

Remote sensing. We use the union of RSICD [27], UCM [32], and Sydney [78] datasets for training. The captions for the UCM and Sydney datasets were annotated by Qu et al. [98]. Figure 4 displays the example images and captions, while Table 6 presents the statistics of each dataset. Since an image may have multiple associated captions, we randomly select one for each iteration.

(a) RSICD

(b) UCM

(c) Sydney

We use the validation sets of RSCID-CLS [27], UCM-CLS [32], WHU-RS19 [31], RSSCN7 [80], and AID [81] datasets for zero-shot classification. RSCID-CLS and UCM-CLS are the classification versions of RSCID and UCM, respectively. Table 7 presents the statistics of each dataset.

Fashion. We use the union of Fashion200k [28], FashionGen [82], and Polyvore Outfits [83] datasets for training. Figure 5 displays the example images and captions, while Table 8 presents the statistics of each dataset. Since an item may have multiple associated views of images corresponding to the caption, we randomly select one view for each iteration. For captions, we concatenate the titles and descriptions from FashionGen and Polyvore, and use the sentences from Fashion200k.

(a) Fashion200k

(b) FashionGen

(c) Polyvore

We use the validation sets of Fashion200k, FashionGen, and Polyvore datasets for zero-shot classification. Table 9 presents the statistics of each dataset. For Fashion200k and FashionGen, we report both the super-class and the sub-class. The class names in Polyvore match the same level of the super-class as those in the other datasets. For example, the super-class contains class names such as 'dress,' while the sub-class contains class names such as 'casual and day dress.'

Scientific figures. We use the SciCap [29] dataset for training. Figure 6 displays the example images and captions, while Table 10 presents the statistics of each dataset. We use figures that do not have subfigures, denoted as the 'SciCap-No-Subfig-Img' subset, for simplicity.

Comics. We use the Simpsons [30] dataset as labeled data and the Simpsons Character [33] dataset as unlabeled data. Figure 6 displays the example images and captions, while Table 10 presents the statistics of each dataset. We use 90% of pairs from the Simpsons dataset as the training split and the remaining 10% as the validation split. Since the training split contains only 720 image-text pairs, it is necessary to incorporate images from different datasets like Simpsons Characters.

'Fig. 2. variable 2-D initial probe

distribution used for CFO runs

reported here'

(a) SciCap

'simpsons wearing headphones  and

looking like a guy'

(b) Simpsons

C.2 Implementation details

Training. We utilize the OpenCLIP [77] library to implement our algorithms. By default, we follow the original training configuration. We use a batch size of 64 per GPU, with a total of 4 GPUs. The learning rate is set to 5e-5, and we apply the default cosine learning rate scheduling with a warmup period of the first 10 steps. We train all models until the performance saturate, which can vary over the number of image-text pairs. Specifically, for remote sensing, we train models for 25 epochs, for fashion 10 epochs, for scientific figures 5 epochs, and for comics datasets 10 epochs. We run three trials and report the average and standard deviation for all experiments.

We add the supervised CLIP loss and pseudo-label losses for semi-supervised methods. We halve the scale of pseudo-label losses to match the scale of the image part in the CLIP loss. We set the scale of the entropic regularizer to be the same as the softmax temperature of CLIP, denoted as = τ , and set the number of Sinkhorn iterations to 10. Note that our method does not introduce any additional hyperparameters, except for the selection of keyword candidate sets discussed below.

Keywords. We use the available class names for remote sensing and fashion datasets while extracting the keywords using the YAKE [74] algorithm from the captions for the SciCap and Simpsons datasets. For remote sensing, we combine the class names from RSICD-CLS and UCM-CLS to obtain 45 keywords, of which around 30 words appeared in the captions. For fashion, we combine the superclass names from Fashion200k and FashionGen to obtain 56 keywords, which are more effective than sub-class names since the captions often do not contain the exact sub-class names.

Our ablation study presents that the keywords extracted by YAKE are also effective, even outperforming the class names in some cases. In the case of remote sensing, the YAKE keywords contain not only nouns like 'building' but also adjectives like 'green,' 'many,' or 'large,' which help the model understand the fine-grained information of the scene. Moreover, we utilize YAKE-100 keywords for SciCap and Simpsons when class names are unavailable, and this approach has proven to be effective. Thus, we believe that a reasonable choice of keywords would firmly benefit our method.

Computation. The computation bottleneck mostly lies in computing embeddings rather than the loss function. Thus, all methods require a similar amount of time for a single training iteration, except for S-CLIP, which takes longer due to the additional forwarding of keywords. Training duration varies depending on the number of image-text pairs, taking a few minutes to a few hours. Semi-supervised methods take twice as long as supervised CLIP since we draw batches of labeled and unlabeled data with an even size, necessitating twice as many iterations for the same epoch.

Evaluation. We use the evaluation code provided by OpenCLIP. For zero-shot classification, we use the default template: 'a photo of a [class].' We use this template for fashion datasets but use the template 'an aerial photograph of a [class]' for remote sensing, following the setup of [64].

D Additional remote sensing results

D.1 Different neural architectures

Table 11 presents the zero-shot classification results using the ResNet [99] and ViT [100] models. S-CLIP significantly improves the supervised baseline in all considered scenarios. Note that the larger model shows better overall accuracy, for both the original and fine-tuned CLIP. Here, S-CLIP gives reliable improvements when scaling up to larger models.

D.2 Different image-text pair ratios

Table 12 presents the zero-shot classification results for different image-text pair ratios. Using only 10% of pairs (2 × fewer than CLIP), S-CLIP achieves performance comparable to CLIP using 20% of pairs. Moreover, with 30% of pairs (3 × fewer than CLIP), S-CLIP matches the performance of CLIP using 100% of pairs and even outperforms it, possibly due to the keyword-level pseudo-labeling.

D.3 Using all image-text pairs

We used 10% of the image-text pairs as labeled data to compare two scenarios, L=U and L = U, in the main paper. Having demonstrated the effectiveness of S-CLIP in the L = U scenario, the next question is whether it would also be helpful when using all paired data and extra unlabeled data. Table 13 presents the results of using the entire RS-ALL dataset as labeled data and RESISC45 as the unlabeled data. S-CLIP is shown to be effective even in this challenging scenario.

̸

D.4 Image-text retrieval R@1

Table 14 presents the R@1 results of image-text retrieval, complementing Table 2 in the main paper which shows the R@5 results. S-CLIP outperforms the semi-supervised methods in most cases, except the supervised baseline works best for image → text retrieval on the UCM dataset.

̸

̸

E Additional ablation studies

E.1 General image domain (COCO) results

Table 15 presents results on the COCO [101] dataset. We observed that fine-tuning models using limited image-caption pairs degrades performance since the original CLIP already performs well. Therefore, we selected a subset of the 'sports' category where the original CLIP performs relatively weakly. Following other setups, we used 10% of the data as labeled data and the rest as unlabeled data. We kept the training configuration consistent with our paper but ran 10 epochs to ensure model convergence. S-CLIP model outperforms others, even in this general image domain.

E.2 Additional pseudo-labeling baselines

Table 16 presents a comparison with pseudo-labeling baselines adapted from the sate-of-the-art semisupervised image classification methods. Specifically, we employ the pseudo-labeling techniques of SemPPL [22] and RoPAWS [69]. SemPPL uses the mode of k-NN predictions. In our case, since captions are unique for each image, we assign a uniform probability over k-NN samples, which we refer to as 'k-NN' pseudo-labeling. On the other hand, RoPAWS utilizes an analytic fixed-point solution for label propagation, which we refer to as 'LabProp' pseudo-labeling. We used these baselines for caption-level pseudo-labeling and compare them with OT. We do not used keyword-level pseudo-labeling for a fair comparison. OT outperforms other pseudo-labeling approaches.

E.3 Qualitative examples of pseudo-labels

Figure 7 and Figure 8 provide qualitative examples of caption-level and keyword-level pseudo-labels, respectively. Optimal transport (OT) matches visually and semantically similar labeled images for unlabeled queries, providing meaningful caption-level pseudo-labels. In addition, the nearest labeled images share overlapping keywords, providing meaningful keyword-level pseudo-labels.

Query

Nearest Labeled Images (Top-5)

'a park

with many green buildings

Query

Nearest

Labeled Images

'the

baseball field

field

is surrounded by lush

trees and

buildings

'

'many

buildings

and

green trees are in an

industrial

area'

'four

baseball fields

are

surrounded by many

green trees and

meadows

'

'this is a

bridge

, several

cars, grassland and a

river

'

'a

bridge

is on a

river

with some green trees in

two sides of it'

'many

buildings

are

near a

stadium

with a

football  field'

'many green trees are

around a

stadium

with a

parking lot

near it'

'the trapezoid

commercial

is next to

the

playground

'

trees and a

pond

is

surrounded by many

green trees and

buildings

and a road'

'many green trees and a

large

pond

are in a

park

surrounded by many

'

E.4 Training curves

Figure 9 presents the training curves showing the loss and evaluation metrics. The validation accuracy for zero-shot classification on RSICD-CLS and image → text retrieval on RS-ALL is reported. The trends are similar in other datasets. Supervised CLIP is prone to overfitting, as indicated by the CLIP validation loss, but this is regularized by Hard-PL and Soft-PL. Our proposed S-CLIP yields the greatest benefits, with superior classification and retrieval performance.

0

5

10

15

20

25

3.75

4.00

4.25

4.50

4.75

CLIP (base)

Hard-PL

Soft-PL

S-CLIP (ours)

(a) CLIP validation loss

5

10

15

20

25

40

50

60

70

CLIP (base)

Hard-PL

Soft-PL

S-CLIP (ours)

(b) Zero-shot classification

5

10

15

20

25

0

2

4

6

CLIP (base)

Hard-PL

Soft-PL

S-CLIP (ours)

(c) Image-text retrieval

E.5 Locked image or text tuning

Table 17 presents the results of fine-tuning after locking (freezing weights) either the image or text encoders, as suggested by LiT [41]. In our method, locking either encoder leads to a degradation in performance. This occurs because both the image and text data from the specialist domain are unseen, requiring the model to learn this novel information. In contrast, in the base CLIP, locking the text encoder yields similar results to having no lock. 6 This is due to the base CLIP suffering from overfitting and inadequately learning the text information. These results provide additional evidence that pseudo-labeling with S-CLIP improves the language understanding of models.

E.6 Selection of mini-batches during training

Table 18 presents the effect of batch selection by applying different strategies to S-CLIP. Using a smaller batch size ( 16 × 4 = 64 ) instead of the original ( 64 × 4 = 256 ) significantly decreased the performance. However, collecting similar images helped mitigate this drop. We simplified the process by arranging batches based on their indices, sorted over visual similarities, and used them without shuffling. This method, 'sorted batch,' resulted in a reasonable gain. Exploring a more effective active batch sampling strategy would be an interesting future direction.

E.7 Baselines using keyword information

Table 19 presents the results compared to the baselines using keyword information. Specifically, we incorporated two losses: (a) keyword loss on labeled data using multi-label classification, and (b) keyword loss on unlabeled data using our pseudo-labeling approach (nearest image is given by OT). To assess the impact of both losses, we compared CLIP-FT, CLIP-FT+(a), and CLIP-FT+(a)+(b). For the ablation study, we did not apply caption-level pseudo-labeling. Both (a) and (b) contribute to the overall performance, confirming that our proposed pseudo-labeling loss relies not only on keyword information but also on exploiting information from unlabeled data.

E.8 Keyword statistics in captions

We trained our models using captioned datasets, and the class names were derived from the classification counterparts of the dataset. Consequently, captions often contain multiple class names. For instance, 45.8% of captions in the RSICD dataset contain more than two class names. The remaining 50% of captions have only a single class name, which results in our keyword-level loss being reduced to a classification loss. It is worth noting that we developed a method to handle scenarios where class names are unavailable. By utilizing YAKE-200 keywords, we found that 94.8% of captions contain more than two keywords, confirming the necessity of partial label learning.

F Limitations and broader impacts

Limitations. Our pseudo-labeling approach is effective for zero-shot classification but less effective for image-text retrieval, which requires fine-grained language understanding. This is because the current pseudo-labeling approach assumes that captions in a batch can capture the semantics of unlabeled images, which may not hold true for fine-grained contexts. To address this issue, one can increase the batch size [84] or incorporate a queue of caption embeddings [85].

In addition, the current OT formulation of caption-level pseudo-labels assumes uniform sink and source constraints. However, this assumption can be refined in the case of distribution shifts. For example, techniques from robust semi-supervised learning [68, 69] can be utilized to estimate the in-domain-ness and subsequently regularize pseudo-labeling based on this estimation.

Lastly, keyword-level pseudo-labeling could be enhanced. The current method relies on the exact inclusion of keywords in the caption to form the candidate set. One can relax this criterion to include synonyms or related concepts by incorporating additional inference steps. In addition, the extracted keywords from captions may vary in their levels. Integrating a word hierarchy to account for different levels of keywords would be an interesting future research direction.

Broader impacts. Our paper aims to broaden the applicability of CLIP to specialist domains. However, this broader scope may present challenges when using the model with domains that include harmful content. These challenges arise from the data itself, rather than the model itself. Therefore, it is crucial to have adequate data regularization to effectively address these concerns.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents S-CLIP, a novel semi-supervised learning method for training contrastive language-image pre-training (CLIP) models in specialized domains where limited image-text pairs are available. Vision-language models like CLIP have achieved impressive results in natural image domains, but they struggle when applied to specialized domains like remote sensing due to the scarcity of paired data. To address this, S-CLIP leverages additional unpaired images and introduces two pseudo-labeling strategies: caption-level and keyword-level pseudo-labeling.\n\nCaption-level pseudo-labeling assigns pseudo-labels to unlabeled images based on the captions of paired images. This is achieved by solving an optimal transport problem between the unlabeled and labeled images, ensuring that the pseudo-labels represent a probability distribution over the labeled images. On the other hand, keyword-level pseudo-labeling assigns pseudo-labels based on the keywords in the captions of visually similar paired images. It formulates the training as a partial label learning problem, considering a candidate set of target keywords instead of a single exact one.\n\nThe experiments conducted in various specialist domains, including remote sensing, fashion, scientific figures, and comics, demonstrate the effectiveness of S-CLIP. It outperforms CLIP fine-tuning and other semi-supervised learning competitors in zero-shot classification and image-text retrieval tasks. For example, in the remote sensing domain, S-CLIP improves zero-shot accuracy by 10.4% and image-to-text retrieval R@5 by 4.4% compared to CLIP fine-tuning. Moreover, S-CLIP remains robust even when the unlabeled images are from a different dataset. Overall, S-CLIP shows promise in enhancing CLIP training using only a few image-text pairs.\n\nSTRENGTHS:\n- Effective Pseudo-Labeling Strategies: The paper proposes two pseudo-labeling strategies, caption-level and keyword-level pseudo-labeling, which effectively guide the training process. The caption-level pseudo-labeling leverages optimal transport to assign pseudo-labels based on the captions of paired images, ensuring robust training even with distribution shifts. The keyword-level pseudo-labeling assigns pseudo-labels based on keywords in visually similar paired images, capturing local components of unlabeled images. The combination of these strategies results in improved zero-shot classification and image-text retrieval tasks.\n\n- Experimental Demonstrations: The paper provides extensive experimental evaluations in various specialist domains, including remote sensing, fashion, scientific figures, and comics. The results consistently show that S-CLIP outperforms CLIP fine-tuning and other semi-supervised learning competitors, achieving significant improvements in zero-shot classification and image-text retrieval tasks. The robustness of S-CLIP, even when unlabeled images are from a different dataset, further strengthens its effectiveness.\n\n- Relevance and Applicability: The paper addresses an important and practical problem in the field of vision-language models. The proposed S-CLIP method has the potential to enhance the training of CLIP models in specialized domains with limited paired data. The findings are relevant not only for researchers working on vision-language models but also for practitioners in various domains where image-text understanding is crucial.\n\n- Clarity and Presentation: The paper is well-written, organized, and presents the proposed method and experimental results clearly. The figures and illustrations aid in understanding the concepts and methodologies. The strengths and weaknesses of the proposed method are discussed in a concise and informative manner.\n\nWEAKNESSES:\n- Baselines with other semi supervised approaches are severely limited with the added baselines being severely outdated. Several much newer few shot learning approaches are present and should be compared against. A summary could be found in the paper: A Survey on Deep Semi-supervised Learning by Yang et al. for example: SEMI-MAE: MASKED AUTOENCODERS FOR SEMI-SUPERVISED\nVISION TRANSFORMERS by Yu et al or Semi-supervised Multimodal Representation Learning through a Global Workspace by Devillers et al\n- A lot of the core content like limitations, experimental setup etc has been moved to the appendix which I feel is key to the understanding of the paper and unfairly grants the author extra writing space\n- The approach combines keyword and caption level pseudo labelling, both of which have individually been well studied in literature. The combination of both is certainly a novel effort, but the contribution of the same doesnt seem like a strong enough novelty for a venue liek Neurips\n- The choice of using OT as the distance measure between 2 images is interesting, it would be nice to see how other image distance measures compare to it\n\nQUESTIONS:\n- Could you clarify why the soft-PL, hard-PL were the only semi supervised baselines chosen for comparison despite the presence of a vast body of literature for semi supervised learning?\n\nLIMITATIONS:\nNo ethical limitations as such", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies semi-supervised learning for contrastive vision-language pre-training, during which unlabelled/unpaired data is also accessible. The authors formulate the pseudo-labeling as an optimal transport (OT) problem and propose to use the Sinkhore algorithm to produce pseudo labels, which is validated to be superior to vanilla had/soft PL solution. Comparisons with other works in literature utilizing OT are properly discussed. Furthermore, the paper proposes to use the key-level pseudo label to account for partial label learning. Experiments on several data domains are conducted to validate its effectiveness.\n\nSTRENGTHS:\n- The paper is technically sound and easy to follow.\n- The evaluation is self-sufficient with proper baselines, which helps to corroborate the effectiveness of the proposed loss.\n- The ablative study is sufficient and showcases the impact of different composing parts of the method.\n- Consistent performance gain over a variety of downstream datasets.\n\nWEAKNESSES:\n- For keyword-level pseudo-label, the compared baseline is somewhat not fair, since one could also leverage the keyword information for labeled/paired data in the vanilla fine-tuning setting, i.e., fine-tuning pre-trained CLIP with CLIP loss and multi-class classification loss. Additional experiments should be conducted to verify the gain derives from the semi-supervised part of it instead of simply leveraging the label information.\n- It would also be nice to showcase results in general data (e.g., the ones evaluated in the CLIP paper, Flicker, etc.). Manual data split may be necessary but it helps to corroborate the generalizability of the proposed method.\n\nQUESTIONS:\n- The title is somewhat confusing. The paper studies semi-supervised fine-tuning on pre-trained CLIP weights, which is not suitable to phrase as \"semi-supervised pre-training\", where you are dealing with an unlabelled general domain. \n- When keywords are obtained from class names, does one labeled image normally only contain just one keyword? If yes, does this reduce to a classification loss (since the target is one-hot)?\n\nLIMITATIONS:\nAll limitations are properly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an additional training method for vision-language models like contrastive language-image pre-training (CLIP). The proposed method consists of semi-supervised learning with two different pseudo labeling including caption- and keyword-level pserudo-label. These two pseudo labelings are complementarily understand image through different types of text representation. The experimental section describes the effectiveness of the proposed method in terms of zero-shot image recognition and image-text retrieval in several representative datasets. A relatively small image-text paired dataset can improve the zero-shot recognition and image-text retrieval with the proposed pseudo labelings.\n\nSTRENGTHS:\n+ The paper serves a method for semi-supervised learning on image-text paired datasets. Usually, it is quite difficult to construct the kind of dataset, however, the paper proposed that a simple yet effective idea to solve a problem in vision language pre-training like CLIP. \n\n+ This paper also points out that limited focused areas such as remote sensing and fashion cannot be effectively used a vision-language models like CLIP as is. Therefore, the paper insists that an easily adaptive labeling approach and semi-supervised fine-tuning from pseudo labels are important for CLIP to adapt a specific image dataset.\n\n+ In the experimental section, the proposed method S-CLIP is much better than that of used the original CLIP and fine-tuned CLIP (CLIP-FT). It is reasonable to assign soft labels rather than hard labels (Hard-PL) in this problem. \n\n+ The paper is well-written and effectively communicates the main ideas in the paper, making it accessible to a broad audience. Moreover, there contains various visualizations, e.g., Figures 1 and 2, to confirm that the learnt representations of vision-language pre-training are meaningful.\n\nWEAKNESSES:\n- CLIP is pre-trained on image-text paired dataset widely extracted from the major space on the Internet. Therefore, it is understandable that the dataset collection does not work well in situations where only a small amount of paired data is collected, or where the paired data is not steadily available on the web. On the other hand, does the proposed dataset collection method only work on remote sensing, fashion, scientific figures, and comics? If you could show the vision-language application based on the proposed dataset collection in more situations, the paper must be more effective to broader readers. Probably, the approach could be shown in more generic situations, however, there are only 4 types of situations. Though the authors do not necessarilly add the experimental results, how about discussing the effectiveness for general purpose dataset collection?\n\nQUESTIONS:\nPlease see above-mentioned comment in 'weaknesses'.\n\nLIMITATIONS:\nThere are no negative limitations and societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
Fine-tuning CLIP in specialized domains suffer the problem of limited available image-text pairs. This paper propose to address this challenge via semi-supervised learning methods with the additional unpaired images. Two pseudo-labeling strategies including the caption-level pseudo labeling and keyword-level pseudo labeling are proposed to construct valuable information about the unpaired images.
The problem addressed in the paper is meaningful and the method proposed in the article is demonstrated effective through comprehensive experiments.

STRENGTHS:
(+) This article is well written, and the description of the method pipeline is clear and easy to understand. 

(+) Experiments are sufficient.

(+) The problem it solves is meaningful.

WEAKNESSES:
(-) "Baseline" in table 5 is a little confusing. See question (1).

(-) The effectiveness of OT-based "Caption-level pseudo-label" compared with soft-PL should be further verified.

QUESTIONS:
(1) "Baseline" in table 5 is a little confusing. Is that refers to the "CLIP (fine-tune)" method or the "hard-PL" or "soft-PL" baselines ?

(2) It seems the role of the proposed OT-based "Caption-level pseudo-label" is similar as the soft-PL baseline. I suggest the authors provide a comparison of the effect of using the OT-based method and the soft-PL method on top of CLIP fine-tuned; or replace the OT-based method with the soft-PL, ablation on S-CLIP to show the effectiveness.

(3) I suggest the accuracy of training on 100% labels (full data) should be reported, as the upper bound performance to be reffered.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method S-CLIP, to enhance CLIP in a semi-supervised way by leveraging captions (via optimal transport) and keywords (via image-keyword similarity) from other unpaired images in the same batch. This is beneficial for specialized domains like remote sensing, where image-text data is usually limited. Results show better performance than the traditional pseudo-labelling methods, especially in OOD domains.\n\nSTRENGTHS:\nAddressing low-data regimes for contrastive learning. Such a problem is common in specialized domains like remote sensing.\n\nNovel and simple method, but showing promising results compared to the standard pseudo-labelling methods, especially in OOD domains.\n\nExperiments and ablations are comprehensive.\n\nWell written and easy to read.\n\nWEAKNESSES:\nIn section 5 experiments, missing experiment details for reproducing, such as model size, epoch, training schedule (lr, weight decay etc.)...\n\nBesides the traditional pseudo-labelling methods, it would be nicer if we could compare with other SOTA pseudo-labelling methods for vision-language training – if there were no such SOTA methods, we could claim that in the paper.\n\nQUESTIONS:\nGiven this method is sensitive to the batch data (e.g. statement in 5.2), it would be more solid if we could consider some data preprocessing strategies, e.g., to batch similar images in the same batch as much as possible? Or at least we should study how batch size affects the results in ablations?\n\nIn line 237, “we employ 32 image-caption pairs and 32 unpaired images”: this seems contrary with the problem to solve, i.e., N << M?\n\nCould we give some qualitative examples for caption-level and keyword-level selection? E.g., given a few images, which are the selected text (weighted captions/keywords) look like in the same batch?\n\nminor comment1: in Figure 1b, Hard-PL is “-2.7” instead of “-1.7”?\n\nminor comment2: the entire section 3 background could be removed to save more space for ablations/analysis, which is more valuable to readers (given the background intro is too basic)?\n\nminor comment3: explain what “YAKE-100/200/300” mean in Table 5?\n\nminor comment4: “(f) Partial” to be consistent with other items.\n\nLIMITATIONS:\nNo specific concern.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}