SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sangwoo Mo $^{1,2}$  Minkyu Kim $^{1,3}$  Kyungmin Lee $^{1}$  Jinwoo Shin $^{1}$ $^{1}$ KAIST  $^{2}$ University of Michigan  $^{3}$ KRAFTON
Vision-language models, such as contrastive language-image pre-training (CLIP), have demonstrated impressive results in natural image domains. However, these models often struggle when applied to specialized domains like remote sensing, and adapting to such domains is challenging due to the limited number of image-text pairs available for training. To address this, we propose S-CLIP, a semi-supervised learning method for training CLIP that utilizes additional unpaired images. S-CLIP employs two pseudo-labeling strategies specifically designed for contrastive learning and the language modality. The caption-level pseudo-label is given by a combination of captions of paired images, obtained by solving an optimal transport problem between unpaired and paired images. The keyword-level pseudo-label is given by a keyword in the caption of the nearest paired image, trained through partial label learning that assumes a candidate set of labels for supervision instead of the exact one. By combining these objectives, S-CLIP significantly enhances the training of CLIP using only a few image-text pairs, as demonstrated in various specialist domains, including remote sensing, fashion, scientific figures, and comics. For instance, S-CLIP improves CLIP by  $10\%$  for zero-shot classification and  $4\%$  for image-text retrieval on the remote sensing benchmark, matching the performance of supervised CLIP while using three times fewer image-text pairs. $^{1}$
Pre-trained vision-language models have achieved remarkable success, providing a foundation for numerous downstream tasks [1]. However, these models often struggle when applied to specialized domains such as remote sensing or medical imaging [2]. This is because the models are trained on web-crawled data that may not fully capture the diversity and complexity of these domains [3, 4]. To address this issue, previous research has focused on constructing large-scale pre-training datasets for each domain [5, 6]. However, annotating caption for each specialized domain can be expensive and time-consuming, which limits their applicability across various domains.
Several studies have attempted to reduce the number of image-text pairs used for vision-language pre-training. However, these approaches often rely on additional information, such as pre-trained object detectors [7-9], or class-annotated images [10], which may not be applicable to specialized domains. Other approaches leverage self-supervised learning to utilize unpaired data [11-14], but they do not fully exploit the information provided by image-text pairs. Research on leveraging a few image-text pairs to improve vision-language pre-training remains underexplored.
In various areas of machine learning, particularly in image classification, semi-supervised learning [15] is a popular approach for training a model using limited annotations. This technique typically relies on pseudo-labeling [16], which predicts labels for unlabeled data by propagating information
*Work done at KAIST. Corresponds to: swmo@umich.edu
$^{1}$ Code: https://github.com/alinlab/s-clip
(a) Problem of naive pseudo-labeling (use the nearest caption)
(b) Comparison of models
Figure 1: Motivation. (a) Images and captions are from the remote sensing dataset, RSICD. Unlike class labels, captions vary widely across images, making the naive pseudo-labeling of assigning the nearest caption ineffective. (b) Zero-shot accuracy on the RSICD-CLS dataset. Bracket indicates the gap from CLIP fine-tuned on remote sensing datasets, called CLIP-FT. The original CLIP struggles in this specialized domain, and CLIP-FT gives a large gain. As discussed in (a), naive pseudo-labeling (Hard-PL) harms CLIP-FT, while our S-CLIP provides substantial improvement.
from labeled data and refining the model using them. Combining this approach with self-supervised learning [17] has significantly improved classifiers using only a few class labels [18-22].
A natural question is whether semi-supervised learning, particularly pseudo-labeling, can improve vision-language pre-training. However, the techniques from image classification cannot be directly applied to vision-language models since captions are diverse and often uniquely associated with each image. Thus, the naive pseudo-labeling approach of assigning the caption of the nearest labeled image $^{2}$  (Hard-PL) [16] can mislead the models. Figure 1a demonstrates that even visually similar images often have different captions, although they may share keywords such as "tennis court." As a result, Figure 1b shows that Hard-PL harms the performance of CLIP fine-tuning. This observation motivates the proper design of pseudo-labeling method that effectively utilizes captions.
Contribution. We propose S-CLIP, a novel semi-supervised learning method for vision-language pretraining, particularly CLIP [23]. S-CLIP introduces two novel pseudo-labeling methods specifically designed for contrastive learning and the language modality, as illustrated in Figure 2:
- Caption-level pseudo-label. We assume that the semantics of an unlabeled image can be expressed as a combination of those of labeled images. To achieve this, we define the caption-level pseudo-label as a probability distribution over the labeled images, which guides the relationship of an unlabeled image with its corresponding captions. Specifically, the pseudo-labels are obtained by solving an optimal transport [24, 25] problem between the unlabeled and labeled images. This approach prevents pseudo-labels from collapsing to a few captions and ensures robust training, particularly when dealing with distribution shifts between unlabeled and labeled images.
- **Keyword-level pseudo-label.** We assume that an unlabeled image shares keywords with visually similar images, even if their full captions are not identical. Therefore, we define the keyword-level pseudo-label as one of the keywords in the nearest labeled image to an unlabeled image. This approach creates a candidate set of target keywords instead of a single exact one, and the training can be formulated as a partial label learning [26] problem. This loss helps the model understand the local components of unlabeled images, leveraging the structure of language.
We note that both pseudo-labels are complementary. Caption-level pseudo-label helps the model understand the global structure of language, which is more effective for image-text retrieval. Keyword-level pseudo-label helps the model understand the local phrases, which is more effective for zero-shot classification. Combining them achieves the best of both worlds (Section 5.5).
We demonstrate the effectiveness of our method in various specialist domains with limited available image-text pairs, including remote sensing [27], fashion [28], scientific figures [29], and comics [30] domains. In the remote sensing domain, S-CLIP outperforms CLIP fine-tuning and semi-supervised learning competitors in five zero-shot classification and six image-text retrieval tasks. For instance, S-CLIP improves zero-shot accuracy on WHU-RS19 [31] by  $10.4\%$  and image-to-text retrieval R@5
2We use the terms "paired" and "labeled" interchangeably, as well as "unpaired" and "unlabeled," depending on the context of vision-language pre-training or semi-supervised learning.
Figure 2: Method overview. Conceptual illustration of our proposed S-CLIP, utilizing (a) caption-level and (b) keyword-level pseudo-labels. In (a), the pseudo-label is given by a probability distribution over the captions, obtained by solving an optimal transport problem between unlabeled and labeled images. In (b), the pseudo-label is given by the candidate set of keywords from the caption of the nearest labeled image, which is learned by the partial label learning algorithm.
on UCM [32] by  $4.4\%$  compared to CLIP fine-tuning. S-CLIP remains robust even when unlabeled images are drawn from a different dataset. In the comics domain, the Simpsons [30] dataset only includes 800 image-text pairs. Therefore, we incorporate the Simpsons Characters [33] dataset, which consists of 30,000 images without captions. Here, S-CLIP improves the text  $\rightarrow$  image retrieval R@1 of CLIP from  $11.8\%$  to  $15.8\%$ , providing a relative gain of  $33\%$ .
Vision-language pre-training (VLP). Pre-training vision-language models has achieved remarkable success, providing a foundation for many downstream tasks [1]. Numerous approaches have been proposed, including reconstructing masked inputs [34-38], learning a joint embedding of vision and language modalities [23, 39-42], generating language descriptions from image inputs [43-45], and connecting pre-trained unimodal image and language models [46-48]. CLIP [23] is a representative model that learns a joint embedding between vision and language modalities. CLIP has demonstrated effectiveness in various applications, including zero-shot classification [23], image-text retrieval [49], open-vocabulary object segmentation [50, 51], and out-of-distribution detection [52].
VLP for specialist domains. Despite their success in natural image domains, VLP models often struggle when applied to specialist domains like remote sensing [2]. This is because they are trained on web-crawled data that may not capture the diversity and complexity of these domains [3, 4]. Data in these fields is often scarce, has limited accessibility, and requires expert annotations, making it difficult to obtain the image-text pairs needed for VLP training. While attempts have been made to train VLP models for specialized domains, including medical [10, 53-59], fashion [60-63], and remote sensing [64], previous research has mainly focused on constructing large-scale pre-training datasets for each domain [5, 6], which can be expensive and time-consuming.
VLP with limited pairs. Several studies trained VLP models with a limited number of image-text pairs. Hsu et al. [65] used adversarial domain adaptation to regularize unpaired data embeddings similar to paired data. However, this approach can cause problems when the unpaired data is drawn from a different distribution. MedCLIP [10] utilized images with class labels to reduce the need for captions. Nevertheless, it still requires class labels, limiting the use of large unlabeled data. Other studies explored unsupervised or semi-supervised VLP training in image domains [7-9]. However, they rely on a pre-trained object detector to align detected objects with keywords in captions, making them unsuitable for specialist domains. Other works leverage self-supervised learning on each modality [11-14], but do not fully exploit the information provided by image-text pairs. In contrast, our method leverages image-text pairs and is generally applicable without extra information.
Semi-supervised learning. Semi-supervised learning [15] aims to train a model using a small labeled dataset and a large unlabeled dataset. Techniques for semi-supervised learning can be categorized into two types: pseudo-labeling (or label propagation) [16, 66], which utilizes information from the small labeled dataset, and self-supervised learning [17], which learns representations from the unlabeled dataset. Combining these techniques has achieved remarkable success in semi-supervised learning, particularly for training classifiers with few class labels [18-22]. Prior works on robust semi-supervised learning focus on scenarios where the distributions of unlabeled and labeled data
shift [67-69]. This issue naturally arises in vision-language pre-training due to the uniqueness of captions for each image. Our method addresses this issue using soft labels and performs robustly even when the unlabeled images are from a different dataset than the labeled images.
Before we demonstrate our semi-supervised vision-language pre-training method, we provide a brief overview on contrastive language-image pre-training and semi-supervised learning.
3.1 Contrastive language-image pre-training
Contrastive language-image pre-training (CLIP) [23] is a popular vision-language model that learns a joint embedding space connecting images  $x$  and texts  $y$ .<sup>3</sup> CLIP is trained using a batch of paired images and texts  $\{x_i, y_i\}_{i=1}^N$  through contrastive learning, where those from the same pair are treated as positive and different pairs as negative. The training objective of CLIP involves two classification tasks: predicting a text given an image  $p(y|x)$  and predicting an image given a text  $p(x|y)$ . Labels are assigned to the  $N$  samples in the batch, corresponding to their pairs as the target. The embeddings are normalized to have a unit norm, and CLIP predicts the label using a softmax function with a temperature parameter  $\tau > 0$ , applied to the cosine similarity of the embeddings.
Specifically, we denote the softmax classifier  $\sigma_{\tau}$  as a function of input and target embeddings. This allows us to represent  $p(y|x) = \sigma_{\tau}(x,\{y_i\}_{i=1}^N) \in \mathbb{R}^N$  and  $p(x|y) = \sigma_{\tau}(y,\{x_i\}_{i=1}^N) \in \mathbb{R}^N$ , which denote the class probabilities of CLIP. These probabilities are calculated as follows:  $p(y = y_i|x) = \exp(x \cdot y_i / \tau) / (\sum_{j=1}^{N} \exp(x \cdot y_j / \tau))$ . The CLIP model minimizes the following objective:
$$
\mathcal {L} _ {\mathrm {C L I P}} = \frac {1}{2 N} \sum_ {i = 1} ^ {N} \left(H \left(p \left(y \mid x _ {i}\right), \mathbf {e} _ {i}\right) + H \left(p \left(x \mid y _ {i}\right), \mathbf {e} _ {i}\right)\right), \tag {1}
$$
where  $H$  denotes the cross-entropy loss and  $\mathbf{e}_i\in \mathbb{R}^N$  is a one-hot vector with the  $i$ -th element being one. The CLIP model can be used for zero-shot classification or image-text retrieval by computing cosine similarity in the embedding space. For example, in zero-shot classification, the text embedding of the class prompt that is closest to an image embedding can be identified.
3.2 Semi-supervised learning
Semi-supervised learning (Semi-SL) [15] aims to train a model using a small number of labeled data  $\{x_i, y_i\}_{i=1}^N$  and a large number of unlabeled data  $\{u_i\}_{i=1}^M$ , i.e., when  $M \gg N$ . Various Semi-SL methods have been proposed to enhance neural network performance, particularly for image classification [19-22]. These methods typically rely on pseudo-labels [16], obtained by predicting classes for unlabeled data using a model. The model is then refined by minimizing the cross-entropy loss  $\frac{1}{M} \sum_{i=1}^{M} H(p(y|u_i), q_i)$  with respect to the prediction  $p(y|u_i) \in \mathbb{R}^C$  and pseudo-label  $q_i \in \mathbb{R}^C$  for unlabeled data  $u_i$ . Pseudo-labels can take the form of a hard label, represented by a one-hot vector identifying the class with the highest probability [16], or a soft label that calibrates this hard label [21]. Pseudo-labels minimize prediction entropy by sharpening specific classes [70].
Semi-SL for CLIP. Assigning pseudo-labels in CLIP poses challenges due to the unique captions associated with each image. The assumption that unlabeled data belongs to the same class as labeled data does not hold here. Therefore, we propose a novel Semi-SL method for CLIP.
4 S-CLIP: Semi-supervised Vision-Language Pre-training
We extend the training of CLIP to incorporate unpaired images  $\{u_i\}_{i=1}^M$  alongside the image-text pairs  $\{x_i, y_i\}_{i=1}^N$ . To achieve this, we introduce S-CLIP, which integrates two novel pseudo-labeling methods considering contrastive learning and the language modality. The conceptual illustrations of the proposed pseudo-labels and training objectives are shown in Figure 2 and Figure 3, respectively.
3We use the notations  $x$  and  $y$  to represent image and text embeddings computed by the CLIP encoders. However, for simplicity, we often omit the term "embeddings" when referring to them.
4In image classification, the targets  $y$  are represented by one-hot vectors of size  $C$  for  $C$  classes.
(a) CLIP loss  $\mathcal{L}_{\mathrm{CLIP}}$
(b) Caption loss  $\mathcal{L}_{\mathrm{caption}}$
Figure 3: Training objective. Conceptual illustrations of the training objectives of S-CLIP: (a) is the original CLIP loss in Eq. (1) using paired images and texts; (b) and (c) are our proposed caption-level and keyword-level pseudo-label losses in Eq. (3) and Eq. (4), respectively, applied to unpaired images. Caption-level pseudo-labels are soft labels obtained from the optimal transport (OT) plan  $\Gamma$ , while keyword-level pseudo-labels are sparse soft labels obtained from the candidate keyword set  $K_u$ .
(c) Keyword loss  $\mathcal{L}_{\mathrm{keyword}}$
4.1 Caption-level pseudo-label
Although the semantics of an unlabeled image  $u$  may not precisely match any captions  $y_{i}$ , visually similar images often share similar semantics. Therefore, the caption for unlabeled image  $u$  can be expressed as a combination of the captions  $\{y_{i}\}$ . To achieve this, we use caption-level pseudo-labels, which represent a probability distribution over the captions. These pseudo-labels serve as the target for the  $N$ -way classification problem in contrastive learning, with a batch size of  $N$ . Figure 2a demonstrates how caption-level pseudo-label for an unlabeled image is obtained.
The pseudo-labels are derived from the relationship between unlabeled and labeled images, formulated as an optimal transport (OT) [24] problem. We define the cost function  $\mathbf{C} \in \mathbb{R}^{M \times N}$ , where  $M$  and  $N$  represent the number of unlabeled and labeled images, respectively. The cost is given by the negative cosine similarity between the normalized embeddings of the unlabeled image  $u_{i}$  and the labeled image  $x_{j}$ , i.e.,  $C_{ij} = 1 - u_{i} \cdot x_{j}$ . Then, we solve following entropic regularized OT [25] problem:
$$
\min  _ {\Gamma \in \Pi (\mathbf {p}, \mathbf {q})} \langle \Gamma , C \rangle - \lambda H (\Gamma), \quad \Pi (\mathbf {p}, \mathbf {q}) = \left\{\Gamma \in \mathbb {R} _ {+} ^ {M \times N} \mid \Gamma \mathbf {1} _ {N} = \mathbf {p}, \Gamma^ {\top} \mathbf {1} _ {M} = \mathbf {q} \right\}. \tag {2}
$$
Here,  $\Pi (\mathbf{p},\mathbf{q})$  represents a set of transportation plans that satisfy the flow constraint. The sums of flows from sources and to sinks match the vectors  $\mathbf{p}\in \mathbb{R}_+^M$  and  $\mathbf{q}\in \mathbb{R}_+^N$ , respectively, with both  $\mathbf{p}$  and  $\mathbf{q}$  adding up to one. We set them as uniform probabilities, and they worked in our experiments, even when unlabeled and labeled data have distribution shifts.  $\mathbf{1}_N$  denotes the all-one vector of dimension  $N$ . The entropic regularizer  $H(\Gamma) = -\sum_{i,j}\Gamma_{ij}\log \Gamma_{ij}$  with scale  $\lambda$  smooths the transportation plan, enabling an efficient solution using the Sinkhorn-Knopp [25] algorithm.
Once we obtain the transportation plan  $\Gamma$ , we define the pseudo-label for an unlabeled image  $u_{i}$  by normalizing its plan to sum up to one. Specifically, the  $j$ -th element of the pseudo-label  $q_{i} \in \mathbb{R}^{N}$  is computed as  $\Gamma_{ij} / \sum_{j} \Gamma_{ij}$ . This pseudo-label is then utilized to guide the probability distribution over  $N$  captions, where the prediction  $p(y|u_{i}) \in \mathbb{R}^{N}$  is determined by a softmax classifier  $\sigma_{\tau}(u_{i}, \{y_{i}\}_{i=1}^{N})$ . The loss function for the caption-level pseudo-label aims to minimize the following:
$$
\mathcal {L} _ {\text {c a p t i o n}} = \frac {1}{M} \sum_ {i = 1} ^ {M} H \left(p \left(y \mid u _ {i}\right), q _ {i}\right). \tag {3}
$$
The prediction  $p(y|u_i)$  relates the unlabeled image  $u_i$  to the text embeddings  $y$ , while the pseudo-label  $q_i$  relates it to the image embeddings  $x$ . Consequently, the proposed loss function aligns the relationships between image and text embeddings, as suggested in supervised CLIP [71-73]. In our experiments, we compare the usage of text and image embeddings for pseudo-label computation, and using image embeddings outperformed using text embeddings due to their robustness in novel specialist domains. We also explore computing pseudo-embeddings [9], but it yields inferior results as synthesizing pseudo-embeddings is more challenging than inferring pseudo-labels.
Effect of OT. Optimal transport balances the flow between unlabeled and labeled images, ensuring robust pseudo-label assignments without concentration on a few labels. In vision-language tasks, diverse and imbalanced captions often cause image embeddings to focus on a few nearest captions, resulting in the collapse of embeddings. This challenge is magnified when employing semi-supervised learning with distribution shifts [67]. We empirically validate the detrimental effect of naive pseudolabeling approaches, especially in the presence of distribution shifts.
Compared to soft-nearest neighbor pseudo-labels (Soft-PL) [21], our method achieves balance through Sinkhorn iterations. Specifically, it reverts to Soft-PL when the iteration count is zero. In our experiments, we use 10 iterations, and increasing the iteration count has only a marginal impact on performance. The computational cost is negligible as the process occurs in a low-dimensional embedding space. We set the scale of the entropic regularizer to match the softmax temperature,  $\lambda = \tau$ . This adjustment scales the cost function by  $1 / \tau$ , aligning it with the cosine similarity scale learned by CLIP. More information on the OT and Soft-PL relationship is in Appendix A.
OT for vision-language models. Several works have utilized OT for vision-language models, but for problems other than semi-supervised learning. We discuss these works in Appendix B.
4.2 Keyword-level pseudo-label
We propose a keyword-level pseudo-labeling approach to overcome the limitations of caption-level pseudo-labels in capturing the meanings of words in captions. By leveraging the compositionality of language, we assume that visually similar images may share keywords, even if the entire captions differ. This enables us to guide unlabeled images using keywords from their nearest neighbors. It is important to note that an image may have multiple keywords, and visually similar images share only a subset. This aligns with the partial label learning (PLL) [26] problem, where the target label is unknown, and we have access to a candidate set of labels. For instance, in Figure 1a, the image shares the keyword "tennis court" with its nearest neighbor but not "baseball field." Consequently, the nearest neighbor provides a candidate set of keywords, as illustrated in Figure 2b.
To identify candidate keywords for an unlabeled image  $u$ , we assume a pre-defined set of keywords with corresponding embeddings  $K = \{k_i\}_{i=1}^K$ . These keywords can be obtained from class names, if available, or extracted by using algorithms like YAKE [74] applied to captions. Given an unlabeled image  $u$ , we find the nearest labeled image  $x$  using the OT assignments from the previous section. Let  $\{c_1, \ldots, c_l\}$  be the indices of keywords present in labeled image  $x$ , denoted as  $K_u = \{k_{c_1}, \ldots, k_{c_l}\}$ . This forms the candidate set of target keywords for  $u$ . While we simply use the keywords explicitly from the caption, some additional steps can be taken to infer synonyms or related concepts.
In the context of partial label learning, we assume that the ground-truth target label for the unlabeled image  $u$  belongs to the candidate set  $\mathsf{K}_u$ . PLL aims to minimize the loss by selecting the minimum value among candidate labels,  $\min_{k\in \mathsf{K}_u}H(p(k|u),k)$ , rather than specifying a single target label. However, relying on a single minimum can be risky. Previous works have addressed this by using soft labels for candidates [75, 76]. This involves minimizing  $H(p(k|u),q)$ , where  $q\in \mathbb{R}^K$  is a soft label with positive values for keywords in the candidate set and zero for others.