SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Similarly, we define the pseudo-label  $q_{i}$  for the unlabeled image  $u_{i}$  based on the similarity between the embeddings of  $u_{i}$  and the keywords  $k_{j}$  in the candidate set  $\mathsf{K}_{u}$ . Specifically, the  $j$ -th element of the pseudo-label  $q_{i}$  is calculated using a softmax function  $\sigma_{\tau}(u_i,\mathsf{K}_u)$  for  $k\in \mathsf{K}_u$  and is set to zero otherwise. This pseudo-label is then utilized to guide the probability distribution over  $K$  keywords, where the prediction  $p(k|u)\in \mathbb{R}^{K}$  is determined by a softmax classifier  $\sigma_{\tau}(u_i,\mathsf{K})$ . The loss function for the keyword-level pseudo-label aims to minimize the following:
$$
\mathcal {L} _ {\text {k e y w o r d}} = \frac {1}{M} \sum_ {i = 1} ^ {M} H \left(p \left(k \mid u _ {i}\right), q _ {i}\right). \tag {4}
$$
Both the prediction  $p(k|u_i)$  and the pseudo-label  $q_{i}$  are related to the unlabeled image and keywords. However, the restricted candidate set effectively guides the learning process through PLL.
Training objective. We apply the CLIP loss to paired images and texts and the proposed pseudo-label losses to unpaired images. The pseudo-label losses are halved to match the scale of the image part in the CLIP loss. In summary, our final training objective becomes  $\mathcal{L}_{\mathrm{CLIP}} + 1 / 2\left(\mathcal{L}_{\mathrm{caption}} + \mathcal{L}_{\mathrm{keyword}}\right)$ .
Table 1: Zero-shot classification results on remote sensing datasets. We compare the original CLIP, supervised CLIP fine-tuned on labeled data (L), and semi-supervised methods that utilize unlabeled data sampled from the same  $(\mathrm{L} = \mathrm{U})$  or different  $(\mathrm{L}\neq \mathrm{U})$  distribution as the labeled data. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one. Bolds denote the best results among the semi-supervised methods within the same setups. S-CLIP consistently improves zero-shot accuracy, even when the unlabeled data has a distribution shift. In contrast, naive pseudo-labeling often harms the accuracy.
Method	Data	RSICD-CLS	UCM-CLS	WHU-RS19	RSSCN7	AID
CLIP (original)	-	45.3	50.5	65.5	58.9	47.8
CLIP (fine-tune)	L	58.3±0.3	63.5±3.4	76.5±3.2	61.9±1.2	63.1±1.3
Hard-PL [16]	56.6±3.5 (-1.7)	61.6±2.2 (-1.9)	78.1±2.5 (+1.6)	63.9±2.1 (+2.0)	63.2±2.6 (+0.1)
Soft-PL [21]	L=U	62.5±0.8 (+4.2)	65.7±2.7 (+2.2)	83.7±2.7 (+7.2)	65.7±0.6 (+3.8)	68.0±0.7 (+4.9)
S-CLIP (ours)	66.9±1.7 (+8.6)	66.7±1.6 (+3.2)	86.9±2.0 (+10.4)	66.2±1.1 (+4.3)	73.0±0.3 (+9.9)
Hard-PL [16]	55.8±1.2 (-2.5)	61.0±6.2 (-2.5)	76.3±1.1 (-0.2)	62.5±2.7 (+0.6)	62.3±1.4 (-0.8)
Soft-PL [21]	L≠U	56.1±3.5 (-2.2)	62.4±1.8 (-1.1)	79.5±3.6 (+3.0)	63.3±2.6 (+1.4)	62.4±2.1 (-0.7)
S-CLIP (ours)	65.3±2.1 (+7.0)	66.0±0.4 (+2.5)	86.3±1.1 (+9.8)	68.3±2.2 (+6.4)	70.8±2.5 (+7.7)
We demonstrate the effectiveness of our method in various specialist domains with limited available image-text pairs, including remote sensing, fashion, scientific figures, and comics domains.
Setup. We use a pre-trained CLIP model on natural images [23], called CLIP (original). However, this model often struggles in specialist domains. Therefore, we fine-tune the model with domain-specific image-text pairs, called CLIP (fine-tune). We compare this supervised model with semi-supervised methods using unpaired images. We use a batch size of 64 per GPU, with a total of 4 GPUs. To ensure fair GPU memory usage in semi-supervised learning, we employ 32 image-caption pairs and 32 unpaired images for each mini-batch. Models are evaluated on zero-shot classification and image-text retrieval tasks, measuring Top-1 classification accuracy (\%) and recall at K (R@K). We report the average and standard deviation across three random seeds. We follow the training recipe of OpenClip [77] if not specified. Additional experimental details are in Appendix C.
Our proposed S-CLIP includes two novel pseudo-labeling methods. Thus, we mainly compare S-CLIP with other pseudo-labeling approaches, which are extensions of semi-supervised classification techniques. Our baselines, Hard-PL and Soft-PL, extend the hard (top-1) and soft nearest neighbor approaches for pseudo-label assignment. We then utilize these methods as caption-level pseudo-labels, similar to Eq. (3). The specific forms of each method are as follows.
Hard-PL. Given an unlabeled image  $u_{i}$ , Hard-PL identifies the top-1 nearest labeled image $^{5}$ $x_{i*}$  based on their cosine similarities, calculated as  $i^{*} = \arg \max_{j}(u_{i} \cdot x_{j})$ . Subsequently, the pseudolabel of  $u_{i}$  is determined by the corresponding caption, denoted as  $q_{i} = y_{i*}$ . This pseudo-label serves as the target for predicting  $p(y|u_{i})$ , and the loss is computed as  $\sum_{i} H(p(y|u_{i}), q_{i})$ .
Soft-PL. Soft-PL calibrates the pseudo-label of Hard-PL by considering all the relations between labeled and unlabeled images, known as soft nearest neighbor [21], instead of selecting only the top-1 nearest image. For an unlabeled image  $u_{i}$  and a set of labeled images  $\{x_i\}$ , these relations are determined by applying softmax to their cosine similarities. The pseudo-label is defined as  $q_{i} = \sigma_{\tau}(u_{i},\{x_{i}\}_{i = 1}^{N})\in \mathbb{R}^{N}$ , where the pseudo-label has a higher value close to one for visually similar images. We use the same softmax temperature  $\tau$  as CLIP, which is used to compute the similarities between the labeled image and text embeddings. The loss is computed as  $\sum_{i}H(p(y|u_{i}),q_{i})$ .
5.2 Remote sensing datasets
We train vision-language models using the union of RSICD [27], UCM [32], and Sydney [78], named RS-ALL, following the setup of [64]. For semi-supervised learning, we subsample  $10\%$  of image-text
We also tested the version directly finding the nearest text for both Hard-PL and Soft-PL. However, the similarities in visual embeddings give more robust results, as discussed in the ablation study.
Table 2: Image-text retrieval results on remote sensing datasets, following the same setup of Table 1. Bolds denote the best results among the semi-supervised methods within the same setups. S-CLIP performs best in most cases, except the supervised baseline works best for text→image retrieval on the Sydney dataset. Naive pseudo-labeling also helps, but the gain is often unstable.
Method	Data	Image→text R@5	Text→image R@5
RSICD	UCM	Sydney	RSICD	UCM	Sydney
CLIP (original)	-	9.4	34.3	36.2	10.1	24.8	51.7
CLIP (fine-tune)	L	15.4±1.7	41.3±1.8	47.1±6.5	15.1±1.0	40.9±1.6	56.1±2.4
Hard-PL [16]	16.1±0.2	40.8±2.9	43.1±3.0	15.7±0.7	40.5±3.0	47.7±5.3
Soft-PL [21]	L=U	17.0±0.9	43.2±3.9	42.0±4.3	16.5±0.1	42.9±3.3	50.2±4.9
S-CLIP (ours)	18.4±0.6	45.7±1.4	50.0±3.0	16.8±1.2	43.5±1.5	55.1±2.0
Hard-PL [16]	15.8±1.0	42.1±5.7	47.1±4.0	15.3±0.4	42.1±3.1	50.0±1.7
Soft-PL [21]	L≠U	16.5±1.1	40.2±3.2	46.6±4.6	15.4±0.2	40.8±0.5	54.0±4.0
S-CLIP (ours)	17.1±0.8	43.5±3.5	48.9±2.6	15.8±0.7	42.5±1.5	52.3±1.0
pairs as labeled data (L), while the remaining  $90\%$  of images  $(\mathrm{L} = \mathrm{U})$  or unlabeled images from the RESISC45 [79] dataset  $(\mathrm{L} \neq \mathrm{U})$  served as unlabeled data. In Appendix D, we demonstrate that our method consistently gives improvements over CLIP across various setups. These include different neural architectures, varied ratios of image-text pairs, and even scenarios where we use all image-text pairs from RS-ALL in conjunction with unlabeled images from RESISC45.
For zero-shot classification, we use the validation sets of the classification versions of the RSICD and UCM datasets, denoted as RSCID-CLS and UCM-CLS, respectively. To evaluate the generalization abilities, we test models on unseen datasets such as WHU-RS19 [31], RSSCN7 [80], and AID [81]. For image-text retrieval, we use the validation sets of the RSICD, UCM, and Sydney datasets.
Zero-shot classification. Table 1 presents the zero-shot classification results. S-CLIP consistently and significantly outperforms all supervised CLIP fine-tuning and semi-supervised methods. One can make several observations. Firstly, fine-tuning CLIP improves performance over the original CLIP, highlighting the importance of adapting models to the target specialist domain. Secondly, our proposed pseudo-labeling techniques are crucial, as naive pseudo-labeling often harms accuracy. Thirdly, semi-supervised learning methods improve performance on unseen datasets, such as RSSCN7, demonstrating the usefulness of observing more unlabeled images in generalization. Lastly, our method is robust to the distribution shifts  $(\mathrm{L} \neq \mathrm{U})$  setup, which leverages an external source of unlabeled images, a common scenario in practice. In contrast, Soft-PL improves performance in the same distribution  $(\mathrm{L} = \mathrm{U})$  setup but fails on the distribution shifts scenario, confirming the effectiveness of optimal transport in balancing the assignments of pseudo-labels.
Image-text retrieval. Table 2 presents the image-text retrieval results. S-CLIP consistently improves both image  $\rightarrow$  text and text  $\rightarrow$  image retrieval, except in one case where the supervised baseline performs the best. This confirms that S-CLIP also benefits fine language understanding for retrieval. The trend is consistent across evaluation metrics, as seen in the R@1 results in the appendix.
5.3 Fashion datasets
We train vision-language models using the union of Fashion200k [28], FashionGen [82], and Polyvore Outfits [83] datasets. We subsample  $10\%$  of the image-text pairs as labeled data and the remaining  $90\%$  of images as unlabeled data. We evaluate the zero-shot accuracy on the validation sets of all three datasets, considering both super-class and sub-class accuracies for Fashion200k and FashionGen. The class names in Polyvore match the same level of super-class as those in the other datasets.
Table 3 presents the zero-shot classification results. S-CLIP consistently outperforms all supervised and semi-supervised methods. The improvement is more significant for super-class classification but less pronounced for sub-class classification and image-text retrieval. This is because the current pseudo-labeling approach assumes that the semantics of unlabeled image can be represented by the captions in a batch. However, this assumption may not hold true for fine-grained semantics, as certain fine-grained captions may be missing from the batch. Therefore, our method could be further enhanced by increasing the batch size or incorporating a queue of caption embeddings [84, 85].
Table 3: Zero-shot classification results on fashion datasets. Parentheses indicate the performance gap from the supervised CLIP, where values highlighted in green indicate gaps larger than one. Bolds denote the best results. S-CLIP consistently outperforms all supervised and semi-supervised methods.
Method	Fashion200k	FashionGen	Polyvore
Super-class	Sub-class	Super-class	Sub-class	Class
CLIP (original)	73.4	29.3	35.9	22.1	58.4
CLIP (fine-tune)	79.4±1.5	38.4±1.5	41.5±1.4	33.1±1.9	64.2±3.1
Hard-PL [16]	74.6±2.7 (-4.8)	35.0±2.6 (-3.4)	36.6±3.9 (-4.9)	28.8±0.8 (-4.3)	65.5±0.7 (+1.3)
Soft-PL [21]	80.0±1.1 (+0.6)	37.1±0.7 (-1.3)	45.5±3.2 (+4.0)	33.2±0.3 (+0.1)	73.6±7.9 (+9.4)
S-CLIP (ours)	82.0±2.8 (+2.6)	39.5±1.1 (+1.1)	55.3±2.6 (+13.8)	38.7±1.9 (+5.6)	74.6±1.8 (+10.4)
Table 4: Image-text retrieval results on SciCap (scientific figures) and Simpsons (comics) datasets. We train the model on each dataset and evaluate on their validation sets. Bolds denote the best results. S-CLIP consistently outperforms all supervised and semi-supervised methods.
Method	SciCap	Simpsons
Image→text	Text→image	Image→text	Text→image
R@1	R@5	R@1	R@5	R@1	R@5	R@1	R@5
CLIP (original)	9.2	14.7	9.0	14.5	15.8	40.8	10.5	27.6
CLIP (fine-tune)	10.5±0.2	17.9±0.2	10.2±0.1	17.2±0.2	16.7±2.0	41.4±1.1	11.8±2.3	41.7±1.5
Hard-PL [16]	9.7±0.3	16.8±0.6	9.2±0.6	16.2±0.7	15.4±1.5	41.7±3.3	12.7±1.5	40.8±2.3
Soft-PL [21]	9.1±0.5	15.5±1.0	9.7±0.3	16.8±0.5	15.4±2.0	39.9±2.7	12.9±1.9	36.4±2.7
S-CLIP (ours)	10.9±0.3	18.8±0.3	11.1±0.3	18.8±0.4	18.4±2.3	43.9±1.5	15.8±1.3	42.1±0.1
5.4 More captioning datasets
We conduct experiments on two more specialist domains, namely science figures and comics. For this purpose, we use the SciCap [29] and Simpsons [30] datasets. We train the models on each dataset and evaluate their image-text retrieval performance on the validation sets. The results are presented in Table 4, and detailed discussions can be found in the following paragraphs.
SciCap. We subsample  $10\%$  of the image-text pairs as labeled data and the remaining  $90\%$  of images as unlabeled data. S-CLIP improves upon CLIP, even for the scientific figures domain, which exhibits a substantial gap from the natural image domain that CLIP is pre-trained on. This confirms the practical impact of S-CLIP in extending the applicability of CLIP to various specialist domains.
Simpsons. The Simpsons dataset contains only 800 image-text pairs, so we incorporate unlabeled images from another dataset called Simpsons Characters [33]. S-CLIP significantly improves CLIP, even in the presence of distribution shifts between unlabeled and labeled images. For example, it boosts the text→image retrieval R@1 from  $11.8\%$  to  $15.8\%$ $(+33\%)$ . This confirms the practical impact of S-CLIP in specialist domains with limited available image-text pairs.
We conduct an ablation study on the design choices for S-CLIP. The results are reported in Table 5, which compares the average values of zero-shot accuracy (across five tasks in Table 1) and image-text retrieval R@1 (across six tasks in Table 2) for each design. The observations are as follows:
(a) Training objective. Caption-level pseudo-labels are more beneficial for image-text retrieval, while keyword-level pseudo-labels are better for zero-shot classification. Intuitively, keyword-level pseudo-labels help the model understand specific words, while caption-level pseudo-labels aid in fine-grained language comprehension. Combining both produces the best results.
(b) Sinkhorn iteration. Sinkhorn iterations significantly improve Soft-PL ( $= \text{Iter. 0}$ ) by balancing pseudo-labels. However, it saturates after 10 iterations, so we have used it as the default in our experiments. Increasing the number of iterations ensures a more even distribution of pseudolabels. For example, the pseudo-labels from Soft-PL often collapse to some index in the batch. Sinkhorn iterations effectively regularize this behavior, reducing the maximum confidence of
Table 5: Ablation study on design choices. We report the average values of zero-shot accuracy (left) and image  $\leftrightarrow$  text retrieval (right). Bolds denote the best results. See Section 5.5 for discussions.
(a) Training objective	(b) Sinkhorn iteration	(c) Choice of keywords
CLIP (fine-tune)	64.7	9.3	Iter. 0	70.6	9.2	YAKE-100	70.0	9.8
+ Caption-level	69.6	10.5	Iter. 1	71.4	9.4	YAKE-200	72.2	9.3
+ Keyword-level	70.0	9.9	Iter. 5	71.6	10.2	YAKE-300	70.6	10.2
+ Both	71.9	10.6	Iter. 10	71.9	10.6	Class name	71.9	10.6
(d) Pseudo-target	(e) Embedding for OT	(f) Partial label loss
Pseudo-embed	68.8	9.2	Text	71.6	9.7	Hard-max	66.1	10.2
Pseudo-label	71.9	10.6	Image	71.9	10.6	Soft-max	71.9	10.6
prediction from  $100\%$  to  $40\%$ . This regularization is more important in the  $\mathrm{L} \neq \mathrm{U}$  setup, where Soft-PL often harms the performance, while ours consistently provides improvement.
(c) Choice of keywords. We compare class names and YAKE- $K$ , which implies that  $K$  keywords are extracted using the YAKE [74] algorithm. Both types of keywords work well, although class names provide more consistent improvement. YAKE provides both nouns, such as "building," and adjectives like "green," "many," or "large," in contrast to the class names, which consist only of nouns. This enables YAKE to understand diverse semantics beyond object recognition, making keyword-level pseudo-labeling applicable to any domain. This applicability is further confirmed on the SciCap and Simpsons datasets, where class names are unavailable.
(d) Pseudo-target. Given a pseudo-label  $q_i \in \mathbb{R}^N$  for an unlabeled image  $u_i$ , one can synthesize a pseudo-embedding  $z_i = \sum_j q_{ij} \cdot y_j \in \mathbb{R}^d$ , where  $q_{ij}$  denotes the  $j$ -th element of  $q_i$ , and  $d$  is the dimension of the embeddings  $y_j$ . Then, one can consider  $\{u_i, z_i\}$  as paired data and apply contrastive learning to it in addition to the true pairs  $\{x_i, y_i\}$ . However, we found that pseudo-embedding is less effective than pseudo-labeling. This is because accurately estimating the pseudo-embedding is harder than assigning soft probabilities to the true embeddings. Additionally, the pseudo-embeddings lie on an interpolation of true embeddings, which can introduce confusing negatives and potentially harm the effectiveness of contrastive learning.
(e) Embedding for  $OT$ . We compute pseudo-labels by examining the relationship between unlabeled images and either labeled images or labeled text directly. The relationship with labeled images proves to be more effective in assigning pseudo-labels. Specifically, using text embeddings directly also works reasonably well in the  $L = U$  setup. However, it often fails entirely in the  $L \neq U$  setup. This is because the corresponding captions for unlabeled images are unknown. In contrast, visual similarities can be robustly computed even when the images are unseen.
(f) Partial label loss. We use the soft label given by  $\sigma_{\tau}(u_i, \mathsf{K}_u)$  as the target for PLL. We also test the hard version, using the top-1 similar keyword from the candidate set, i.e.,  $i^* = \arg \max_j (u_i \cdot k_j)$ . However, it performs worse, aligning with the observations from prior PLL literature.
We provide additional ablation studies in Appendix E, including results in the general image domain, additional pseudo-labeling baselines, qualitative examples of pseudo-labels, training curves, regularized fine-tuning approaches, selection of mini-batches during training, additional baselines using keyword information, and statistics of keywords in captions.
We propose S-CLIP, a semi-supervised extension of CLIP that leverages unpaired images to train in specialist domains with limited image-text pairs. We demonstrate its superiority across various domains such as remote sensing, fashion, scientific figures, and comics. We hope that S-CLIP expands the applicability of CLIP. Limitations and broader impacts are discussed in Appendix F.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2019-0-00075, Artificial
Intelligence Graduate School Program (KAIST); No.2021-0-02068, Artificial Intelligence Innovation Hub; No.2022-0-00959, Few-shot Learning of Casual Inference in Vision and Language for Decision Making).
[100] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[101] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, 2014.
A Relationship between OT and Soft-PL
The entropic-regularized optimal transport problem described in Eq. (2) can be efficiently solved using the Sinkhorn-Knopp [25] algorithm. For our case, the cost function  $\mathbf{C} \in \mathbb{R}^{M \times N}$  is given by the negative cosine similarity:  $C_{ij} = 1 - u_i \cdot x_j$ . The Sinkhorn-Knopp algorithm proceeds as follows: first, it computes the exponentiated cost matrix  $K = \exp\left(\frac{C}{-\lambda}\right)$ , where  $\lambda$  denotes the scale of the entropic regularizer. Then, the algorithm iteratively updates two vectors:  $\mathbf{u} \in \mathbb{R}^{M \times 1}$  and  $\mathbf{v} \in \mathbb{R}^{1 \times N}$ , which are initialized with uniform probability. The transportation plan  $\Gamma$  can be estimated using the exponentiated cost matrix  $K$  and the current vectors  $\mathbf{u}$  and  $\mathbf{v}$ . Specifically, it is given by  $\Gamma = \mathbf{u} \odot K \odot \mathbf{v}$ , where  $\odot$  denotes element-wise multiplication.
At the initialization zero, the transportation plan  $\Gamma$  is obtained by normalizing the exponentiated cost matrix  $\mathbf{K}$  since the vectors  $\mathbf{u}$  and  $\mathbf{v}$  are uniform. Consequently, our caption-level pseudo-label  $q_{i}$  is given by  $q_{i} = \mathrm{K}_{ij} / \sum_{j}\mathrm{K}_{ij}$ , which is equivalent to the Soft-PL  $\sigma_{\lambda}(u_i,x_i)$  except for the softmax temperature, which is denoted as  $\lambda$ . This observation motivates us to set  $\lambda = \tau$ , matching the same temperature as CLIP, for computing the relation between embeddings. From this perspective, our proposed OT approach enhances the robustness of Soft-PL in pseudo-label assignment, especially in cases with distribution shifts between unlabeled and labeled images. Our ablation study demonstrates that increasing the number of Sinkhorn iterations improves the training process.
After initialization, the vectors  $\mathbf{u}$  and  $\mathbf{v}$  are updated to align with the exponentiated cost matrix  $K$  and satisfy the flow constraints  $\mathbf{p}$  and  $\mathbf{q}$ . Specifically, given the current vectors  $\mathbf{u}^{(t)}$  and  $\mathbf{v}^{(t)}$ , the next iteration of vectors is obtained as  $\mathbf{u}^{(t + 1)} = \frac{\mathbf{p}}{\mathrm{K}\mathbf{v}^{(t)}}$  and  $\mathbf{v}^{(t + 1)} = \frac{\mathbf{q}}{\mathrm{K}^{\top}\mathbf{u}^{(t + 1)}}$ . This iterative process converges to the optimal solution. Our ablation study demonstrates that a small number of iterations is sufficiently effective, and we employ 10 iterations in our experiments.
B Additional related work
Optimal transport for vision-language models. Several prior works have applied optimal transport to vision-language models. However, most of them focus on transportation between an image and a caption, connecting objects and phrases [35, 36, 86-88]. This is known as weakly supervised phrase grounding [89], and it assumes a supervised setting with image-text pairs. OTTER [90] calculates transportation between images and text, which aligns with our approach. However, they also assume a supervised setting and focus on calibrating the one-hot target of CLIP. To our knowledge, our work is the first to apply optimal transport for semi-supervised CLIP training.
Optimal transport for semi-supervised learning. Some prior works have utilized optimal transport for semi-supervised learning, specifically in image classification [91, 92]. These studies demonstrate the benefits of optimal transport in pseudo-labeling class labels. However, we argue that OT plays a more critical role in vision-language models due to inherent distribution shifts between labeled and unlabeled image captions. Our experiments reveal that naive pseudo-labeling often has detrimental effects, while OT significantly improves semi-supervised CLIP training.
Partial label learning. Partial label learning (PLL) [26] relaxes the classification problem by using a candidate set as the target instead of the exact one. This involves optimizing for the best candidate within the set. Instead of determining the hard assignment of the candidate, PRODEN [75] computes soft labels for the elements in the set to smooth the optimization process. PiCO [76] further enhances this approach by incorporating contrastive learning. Our work extends the techniques of PLL to a novel problem: keyword-level pseudo-labeling for vision-language models.