SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Label propagation in contrastive learning. The concept of pseudo-labeling (or label propagation) is also applied in image-only contrastive learning. In addition to using the same image with different augmentations [84, 85], some studies explore the utilization of neighboring images as positive pairs [93-95]. Similar concept can be extended to supervised [96] and semi-supervised [22, 97] contrastive learning, where images sharing the same class label are considered positive pairs.
Remote sensing. We use the union of RSICD [27], UCM [32], and Sydney [78] datasets for training. The captions for the UCM and Sydney datasets were annotated by Qu et al. [98]. Figure 4 displays the example images and captions, while Table 6 presents the statistics of each dataset. Since an image may have multiple associated captions, we randomly select one for each iteration.
"aplayground with eight basketball fields near it is surrounded by many green trees"
(a) RSICD
"many houses arranged neatly and some roads crossed them with some plants in the roadside"
(b) UCM
" a residential area with houses arranged neatly and some roads go through this area
(c) Sydney
Figure 4: Examples of image-text pairs in the remote sensing domain.
Table 6: Statistics of image-text paired datasets in the remote sensing domain.
RSICD	UCM	Sydney
# of pairs	8,734	1,680	497
We use the validation sets of RSCID-CLS [27], UCM-CLS [32], WHU-RS19 [31], RSSCN7 [80], and AID [81] datasets for zero-shot classification. RSCID-CLS and UCM-CLS are the classification versions of RSCID and UCM, respectively. Table 7 presents the statistics of each dataset.
Table 7: Statistics of classification datasets in the remote sensing domain.
RSICD-CLS	UCM-CLS	WHU-RS19	RSSCN7	AID
# of images	1,094	2,100	1,005	2,800	10,000
# of classes	31	21	19	7	30
Fashion. We use the union of Fashion200k [28], FashionGen [82], and Polyvore Outfits [83] datasets for training. Figure 5 displays the example images and captions, while Table 8 presents the statistics of each dataset. Since an item may have multiple associated views of images corresponding to the caption, we randomly select one view for each iteration. For captions, we concatenate the titles and descriptions from FashionGen and Polyvore, and use the sentences from Fashion200k.
"white printed silk-jersey wrap dress midnight blue"
(a) Fashion200k
Figure 5: Examples of image-text pairs in the fashion domain.
"grey classic u-neck t-shirt. short sleeve t-shirt in heather grey. tonal logo flag at u-neck collar."
(b) FashionGen
"off-white striped flared pants. multicoloured cotton and silk striped flared pants from off-white."
(c) Polyvore
Table 8: Statistics of image-text paired datasets in the fashion domain.
Fashion200k	FashionGen	Polyvore
# of pairs	61,753	60,147	71,967
We use the validation sets of Fashion200k, FashionGen, and Polyvore datasets for zero-shot classification. Table 9 presents the statistics of each dataset. For Fashion200k and FashionGen, we report both the super-class and the sub-class. The class names in Polyvore match the same level of the super-class as those in the other datasets. For example, the super-class contains class names such as "dress," while the sub-class contains class names such as "casual and day dress."
Table 9: Statistics of classification datasets in the fashion domain.
Fashion200k	FashionGen	Polyvore
Super-class	Sub-class	Super-class	Sub-class	Class
# of images	29,785	32,528	14,657
# of classes	5	31	48	121	11
Scientific figures. We use the SciCap [29] dataset for training. Figure 6 displays the example images and captions, while Table 10 presents the statistics of each dataset. We use figures that do not have subfigures, denoted as the "SciCap-No-Subfig-Img" subset, for simplicity.
**Comics.** We use the Simpsons [30] dataset as labeled data and the Simpsons Character [33] dataset as unlabeled data. Figure 6 displays the example images and captions, while Table 10 presents the statistics of each dataset. We use  $90\%$  of pairs from the Simpsons dataset as the training split and the remaining  $10\%$  as the validation split. Since the training split contains only 720 image-text pairs, it is necessary to incorporate images from different datasets like Simpsons Characters.
"Fig. 2. variable 2-D initial probe distribution used for CFO runs reported here"
(a) SciCap
"simpsons wearing headphones and looking like a guy"
(b) Simpson's
Figure 6: Examples of image-text pairs in the scientific figure and comics domains.
Table 10: Statistics of image-text paired datasets in the scientific figure and comics domains.
SciCap	Simpsons
# of pairs	106,834	720
Training. We utilize the OpenCLIP [77] library to implement our algorithms. By default, we follow the original training configuration. We use a batch size of 64 per GPU, with a total of 4 GPUs. The learning rate is set to 5e-5, and we apply the default cosine learning rate scheduling with a warmup period of the first 10 steps. We train all models until the performance saturate, which can vary over the number of image-text pairs. Specifically, for remote sensing, we train models for 25 epochs, for fashion 10 epochs, for scientific figures 5 epochs, and for comics datasets 10 epochs. We run three trials and report the average and standard deviation for all experiments.
We add the supervised CLIP loss and pseudo-label losses for semi-supervised methods. We halve the scale of pseudo-label losses to match the scale of the image part in the CLIP loss. We set the scale of the entropic regularizer to be the same as the softmax temperature of CLIP, denoted as  $= \tau$ , and set the number of Sinkhorn iterations to 10. Note that our method does not introduce any additional hyperparameters, except for the selection of keyword candidate sets discussed below.
Keywords. We use the available class names for remote sensing and fashion datasets while extracting the keywords using the YAKE [74] algorithm from the captions for the SciCap and Simpsons datasets. For remote sensing, we combine the class names from RSICD-CLS and UCM-CLS to obtain 45 keywords, of which around 30 words appeared in the captions. For fashion, we combine the super-class names from Fashion200k and FashionGen to obtain 56 keywords, which are more effective than sub-class names since the captions often do not contain the exact sub-class names.
Our ablation study presents that the keywords extracted by YAKE are also effective, even outperforming the class names in some cases. In the case of remote sensing, the YAKE keywords contain not only nouns like "building" but also adjectives like "green," "many," or "large," which help the model understand the fine-grained information of the scene. Moreover, we utilize YAKE-100 keywords for SciCap and Simpsons when class names are unavailable, and this approach has proven to be effective. Thus, we believe that a reasonable choice of keywords would firmly benefit our method.
Computation. The computation bottleneck mostly lies in computing embeddings rather than the loss function. Thus, all methods require a similar amount of time for a single training iteration, except for S-CLIP, which takes longer due to the additional forwarding of keywords. Training duration varies depending on the number of image-text pairs, taking a few minutes to a few hours. Semi-supervised methods take twice as long as supervised CLIP since we draw batches of labeled and unlabeled data with an even size, necessitating twice as many iterations for the same epoch.
Evaluation. We use the evaluation code provided by OpenCLIP. For zero-shot classification, we use the default template: "a photo of a [class]." We use this template for fashion datasets but use the template "an aerial photograph of a [class]" for remote sensing, following the setup of [64].
D Additional remote sensing results
D.1 Different neural architectures
Table 11 presents the zero-shot classification results using the ResNet [99] and ViT [100] models. S-CLIP significantly improves the supervised baseline in all considered scenarios. Note that the larger model shows better overall accuracy, for both the original and fine-tuned CLIP. Here, S-CLIP gives reliable improvements when scaling up to larger models.
Table 11: Zero-shot classification results on remote sensing datasets using different neural architectures. Parentheses indicate the performance gap from the supervised CLIP, and bolds denote the best results within each architecture. S-CLIP give consistent improvements, scaling to larger networks.
Model	Method	RSICD-CLS	UCM-CLS	WHU-RS19	RSSCN7	AID
ResNet-50	CLIP (original)	45.3	50.5	65.5	58.9	47.8
CLIP (fine-tune)	58.3±0.3	63.5±3.4	76.5±3.2	61.9±1.2	63.1±1.3
S-CLIP (ours)	66.9±1.7 (+8.6)	66.7±1.6 (+3.2)	86.9±2.0 (+10.4)	66.2±1.1 (+4.3)	73.0±0.3 (+9.9)
ViT-B/32	CLIP (original)	55.8	58.6	76.4	62.1	55.7
CLIP (fine-tune)	72.3±1.2	76.8±3.0	90.4±0.6	68.9±2.2	78.9±0.5
S-CLIP (ours)	77.5±1.0 (+5.2)	78.3±3.6 (+1.5)	93.2±1.9 (+2.8)	69.6±0.8 (+0.7)	83.7±1.8 (+4.8)
ViT-B/16	CLIP (original)	58.9	60.1	80.9	69.4	59.6
CLIP (fine-tune)	72.4±0.6	78.7±2.7	90.0±1.4	71.9±3.8	77.9±0.9
S-CLIP (ours)	79.5±1.5 (+7.1)	82.3±0.4 (+3.6)	93.9±2.1 (+2.1)	76.3±0.3 (+4.4)	85.2±0.9 (+7.3)
D.2 Different image-text pair ratios
Table 12 presents the zero-shot classification results for different image-text pair ratios. Using only  $10\%$  of pairs  $(2\times$  fewer than CLIP), S-CLIP achieves performance comparable to CLIP using  $20\%$  of pairs. Moreover, with  $30\%$  of pairs  $(3\times$  fewer than CLIP), S-CLIP matches the performance of CLIP using  $100\%$  of pairs and even outperforms it, possibly due to the keyword-level pseudo-labeling.
Table 12: Zero-shot classification results on remote sensing datasets using different image-text pair ratios. Parentheses indicate the performance gap from the supervised CLIP, and bolds denote the best results within each pair ratio. S-CLIP gives consistent improvements, reducing required pairs.
Method	Ratio	RSICD-CLS	UCM-CLS	WHU-RS19	RSSCN7	AID
CLIP (original)	0%	45.3	50.5	65.5	58.9	47.8
CLIP (fine-tune)	10%	58.3±0.3	63.5±3.4	76.5±3.2	61.9±1.2	63.1±1.3
S-CLIP (ours)	66.9±1.7 (+8.6)	66.7±1.6 (+3.2)	86.9±2.0 (+10.4)	66.2±1.1 (+4.3)	73.0±0.3 (+9.9)
CLIP (fine-tune)	20%	66.2±1.1	69.7±0.2	82.8±1.5	65.6±3.4	71.8±1.1
S-CLIP (ours)	77.6±1.1 (+11.4)	72.0±0.7 (+2.3)	90.7±1.9 (+7.9)	67.9±2.6 (+2.3)	83.7±0.7 (+11.9)
CLIP (fine-tune)	30%	70.8±1.6	70.7±2.3	85.4±1.7	63.6±1.5	75.7±0.8
S-CLIP (ours)	80.1±1.7 (+9.3)	74.0±1.1 (+3.3)	94.8±1.2 (+9.4)	69.6±2.8 (+6.0)	87.8±0.5 (+12.1)
CLIP (fine-tune)	100%	77.5±1.5	76.5±0.5	93.9±0.9	71.2±0.3	83.6±1.6
D.3 Using all image-text pairs
We used  $10\%$  of the image-text pairs as labeled data to compare two scenarios,  $\mathrm{L} = \mathrm{U}$  and  $\mathrm{L}\neq \mathrm{U}$ , in the main paper. Having demonstrated the effectiveness of S-CLIP in the  $\mathrm{L}\neq \mathrm{U}$  scenario, the next question is whether it would also be helpful when using all paired data and extra unlabeled data. Table 13 presents the results of using the entire RS-ALL dataset as labeled data and RESISC45 as the unlabeled data. S-CLIP is shown to be effective even in this challenging scenario.
Table 13: Zero-shot classification results on remote sensing datasets using all captioned images and extra unlabeled images, where the unlabeled data is from a different distribution than the labeled data. S-CLIP also provides improvements in this most challenging scenario.
Model	Method	RSICD-CLS	UCM-CLS	WHU-RS19	RSSCN7	AID
ResNet-50	CLIP (fine-tune)	77.5±1.5	76.5±0.5	93.9±0.9	71.2±0.3	83.6±1.6
S-CLIP (ours)	81.1±0.2 (+3.6)	77.7±1.4 (+1.2)	95.8±0.9 (+1.9)	72.3±2.8 (+1.1)	87.7±0.4 (+4.1)
ViT-B/32	CLIP (fine-tune)	85.3±1.0	87.3±0.7	95.8±1.0	75.7±1.4	92.0±0.5
S-CLIP (ours)	87.0±0.6 (+1.7)	88.6±1.4 (+1.3)	97.2±0.4 (+1.4)	76.2±0.2 (+0.5)	92.5±0.8 (+0.5)
ViT-B/16	CLIP (fine-tune)	87.0±0.3	88.5±0.6	96.7±0.1	78.4±0.8	89.2±1.4
S-CLIP (ours)	87.4±1.2 (+0.4)	88.9±1.6 (+0.4)	97.3±0.4 (+0.6)	79.1±1.0 (+0.7)	93.1±1.4 (+3.9)
D.4 Image-text retrieval R@1
Table 14 presents the R@1 results of image-text retrieval, complementing Table 2 in the main paper which shows the R@5 results. S-CLIP outperforms the semi-supervised methods in most cases, except the supervised baseline works best for image→text retrieval on the UCM dataset.
Table 14: Image-text retrieval results on remote sensing datasets, following the same setup of Table 1. Bolds denote the best results among the semi-supervised methods within the same setups. S-CLIP performs the best in most cases, similar to the observation in Table 2.
Method	Data	Image→text R@1	Text→image R@1
RSICD	UCM	Sydney	RSICD	UCM	Sydney
CLIP (original)	-	2.1	7.1	10.3	2.2	7.1	20.7
CLIP (fine-tune)	L	3.6±0.2	10.6±2.1	13.2±2.0	3.7±0.3	10.0±0.5	14.4±2.0
Hard-PL [16]	3.8±0.1	9.7±2.7	14.4±1.0	3.9±0.7	9.2±1.8	10.9±1.0
Soft-PL [21]	L=U	4.0±0.3	11.0±0.8	11.5±2.0	4.2±0.4	9.8±1.5	16.1±4.3
S-CLIP (ours)	4.2±0.1	11.6±1.1	14.9±1.0	4.2±0.6	11.1±0.5	17.8±1.0
Hard-PL [16]	3.8±0.3	8.1±1.0	13.4±1.2	3.4±0.2	8.9±1.4	12.6±1.0
Soft-PL [21]	L≠U	3.8±0.4	8.4±1.2	10.9±1.0	3.5±0.3	9.4±1.0	14.9±1.0
S-CLIP (ours)	4.2±0.6	9.8±0.6	13.8±0.2	3.9±1.1	10.8±1.5	17.8±5.0
E Additional ablation studies
E.1 General image domain (COCO) results
Table 15 presents results on the COCO [101] dataset. We observed that fine-tuning models using limited image-caption pairs degrades performance since the original CLIP already performs well. Therefore, we selected a subset of the "sports" category where the original CLIP performs relatively weakly. Following other setups, we used  $10\%$  of the data as labeled data and the rest as unlabeled data. We kept the training configuration consistent with our paper but ran 10 epochs to ensure model convergence. S-CLIP model outperforms others, even in this general image domain.
Table 15: Image-text retrieval results for the COCO sports category, where the original CLIP performs relatively weakly. S-CLIP also performs well in the general image domain.
Method	Image-to-text retrieval	Text-to-image retrieval
CLIP (original)	40.30	38.17
CLIP (fine-tune)	46.59	47.76
Hard-PL	48.72	47.12
Soft-PL	48.72	47.23
S-CLIP (ours)	49.79	48.40
E.2 Additional pseudo-labeling baselines
Table 16 presents a comparison with pseudo-labeling baselines adapted from the sate-of-the-art semi-supervised image classification methods. Specifically, we employ the pseudo-labeling techniques of SemPPL [22] and RoPAWS [69]. SemPPL uses the mode of k-NN predictions. In our case, since captions are unique for each image, we assign a uniform probability over k-NN samples, which we refer to as "k-NN" pseudo-labeling. On the other hand, RoPAWS utilizes an analytic fixed-point solution for label propagation, which we refer to as "LabProp" pseudo-labeling. We used these baselines for caption-level pseudo-labeling and compare them with OT. We do not used keyword-level pseudo-labeling for a fair comparison. OT outperforms other pseudo-labeling approaches.
Table 16: Comparison with pseudo-labeling approaches adopted from the SOTA semi-supervised image classification methods, following the setup of Table 5. OT performs the best.
Method	Zero-shot accuracy	Image-text retrieval
k-NN (k=5)	61.1	8.8
LabProp	68.2	9.4
OT (ours)	69.6	10.5
E.3 Qualitative examples of pseudo-labels
Figure 7 and Figure 8 provide qualitative examples of caption-level and keyword-level pseudo-labels, respectively. Optimal transport (OT) matches visually and semantically similar labeled images for unlabeled queries, providing meaningful caption-level pseudo-labels. In addition, the nearest labeled images share overlapping keywords, providing meaningful keyword-level pseudo-labels.
Figure 7: Qualitative examples of caption-level pseudo-labels. Optimal transport (OT) can identify visually and semantically similar images, including those representing forests, ports, and dense buildings. As a result, caption-level pseudo-labels offer accurate semantics for unlabeled images.
Figure 8: Qualitative examples of keyword-level pseudo-labels. The images closest to the query, as found by OT, often share the keywords in their captions (column 1-2). Even in cases where the images do not share all the keywords, they still have some overlapping keywords (column 3-6).
E.4 Training curves
Figure 9 presents the training curves showing the loss and evaluation metrics. The validation accuracy for zero-shot classification on RSICD-CLS and image  $\rightarrow$  text retrieval on RS-ALL is reported. The trends are similar in other datasets. Supervised CLIP is prone to overfitting, as indicated by the CLIP validation loss, but this is regularized by Hard-PL and Soft-PL. Our proposed S-CLIP yields the greatest benefits, with superior classification and retrieval performance.
(a) CLIP validation loss
(b) Zero-shot classification
Figure 9: Trends in loss and evaluation metrics during training. S-CLIP regularizes overfitting, as shown by the CLIP validation loss. As a result, it performs the best classification and retrieval.
(c) Image-text retrieval
E.5 Locked image or text tuning
Table 17 presents the results of fine-tuning after locking (freezing weights) either the image or text encoders, as suggested by LiT [41]. In our method, locking either encoder leads to a degradation in performance. This occurs because both the image and text data from the specialist domain are unseen, requiring the model to learn this novel information. In contrast, in the base CLIP, locking the text encoder yields similar results to having no lock. This is due to the base CLIP suffering from overfitting and inadequately learning the text information. These results provide additional evidence that pseudo-labeling with S-CLIP improves the language understanding of models.
Table 17: Ablation study on locked image or text encoders. We report the average values of zero-shot accuracy (left) and image  $\leftrightarrow$  text retrieval (right). Locking either the image or text encoder degrades performance, as it prevents the model learning new information from specialist domains.
Locked image	Locked text	No lock
CLIP (fine-tune)	48.7	8.6	64.8	9.4	64.7	9.3
S-CLIP (ours)	63.1	8.7	67.1	10.2	71.9	10.6
E.6 Selection of mini-batches during training
Table 18 presents the effect of batch selection by applying different strategies to S-CLIP. Using a smaller batch size  $(16 \times 4 = 64)$  instead of the original  $(64 \times 4 = 256)$  significantly decreased the performance. However, collecting similar images helped mitigate this drop. We simplified the process by arranging batches based on their indices, sorted over visual similarities, and used them without shuffling. This method, "sorted batch," resulted in a reasonable gain. Exploring a more effective active batch sampling strategy would be an interesting future direction.
Table 18: Ablation study on the mini-batch sampling strategies, following the setup of Table 5. Using a larger and sorted batch improves performance by providing more informative pseudo-labels.
Batch size	Zero-shot accuracy	Image-text retrieval
64 (random batch)	65.8	8.3
64 (sorted batch)	67.3	8.9
256 (random batch)	71.9	10.6
The reported average values may appear similar, but each dataset has distinct values.
E.7 Baselines using keyword information
Table 19 presents the results compared to the baselines using keyword information. Specifically, we incorporated two losses: (a) keyword loss on labeled data using multi-label classification, and (b) keyword loss on unlabeled data using our pseudo-labeling approach (nearest image is given by OT). To assess the impact of both losses, we compared CLIP-FT, CLIP-FT+(a), and CLIP-FT+(a)+(b). For the ablation study, we did not apply caption-level pseudo-labeling. Both (a) and (b) contribute to the overall performance, confirming that our proposed pseudo-labeling loss relies not only on keyword information but also on exploiting information from unlabeled data.
Table 19: Comparison to the baselines using keyword information, following the setup of Table 5. Both keyword information and semi-supervised learning contribute to the final performance.
Method	Zero-shot accuracy	Image-text retrieval
CLIP-FT	64.7	9.3
+ keyword info.	67.3	9.7
+ semi-supervised	70.6	10.1
E.8 Keyword statistics in captions
We trained our models using captioned datasets, and the class names were derived from the classification counterparts of the dataset. Consequently, captions often contain multiple class names. For instance,  $45.8\%$  of captions in the RSICD dataset contain more than two class names. The remaining  $50\%$  of captions have only a single class name, which results in our keyword-level loss being reduced to a classification loss. It is worth noting that we developed a method to handle scenarios where class names are unavailable. By utilizing YAKE-200 keywords, we found that  $94.8\%$  of captions contain more than two keywords, confirming the necessity of partial label learning.
F Limitations and broader impacts
Limitations. Our pseudo-labeling approach is effective for zero-shot classification but less effective for image-text retrieval, which requires fine-grained language understanding. This is because the current pseudo-labeling approach assumes that captions in a batch can capture the semantics of unlabeled images, which may not hold true for fine-grained contexts. To address this issue, one can increase the batch size [84] or incorporate a queue of caption embeddings [85].
In addition, the current OT formulation of caption-level pseudo-labels assumes uniform sink and source constraints. However, this assumption can be refined in the case of distribution shifts. For example, techniques from robust semi-supervised learning [68, 69] can be utilized to estimate the in-domain-ness and subsequently regularize pseudo-labeling based on this estimation.
Lastly, keyword-level pseudo-labeling could be enhanced. The current method relies on the exact inclusion of keywords in the caption to form the candidate set. One can relax this criterion to include synonyms or related concepts by incorporating additional inference steps. In addition, the extracted keywords from captions may vary in their levels. Integrating a word hierarchy to account for different levels of keywords would be an interesting future research direction.
Broader impacts. Our paper aims to broaden the applicability of CLIP to specialist domains. However, this broader scope may present challenges when using the model with domains that include harmful content. These challenges arise from the data itself, rather than the model itself. Therefore, it is crucial to have adequate data regularization to effectively address these concerns.