SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qi Qian $^{1*}$  Yuanhong Xu $^{2}$  Juhua Hu $^{3}$
<sup>1</sup> Alibaba Group, Bellevue, WA 98004, USA
2 Alibaba Group, Hangzhou, China
$^{3}$  School of Engineering and Technology, University of Washington, Tacoma, WA 98402, USA
{qi.qian, yuanhong.xuyh}@alibaba-inc.com, juhuah@uw.edu
Vision-language pre-training methods, e.g., CLIP, demonstrate an impressive zero-shot performance on visual categorizations with the class proxy from the text embedding of the class name. However, the modality gap between the text and vision space can result in a sub-optimal performance. We theoretically show that the gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP and the optimal proxy for vision tasks may reside only in the vision space. Therefore, given unlabeled target vision data, we propose to learn the vision proxy directly with the help from the text proxy for zero-shot transfer. Moreover, according to our theoretical analysis, strategies are developed to further refine the pseudo label obtained by the text proxy to facilitate the intra-modal proxy learning (InMaP) for vision. Experiments on extensive downstream tasks confirm the effectiveness and efficiency of our proposal. Concretely, InMaP can obtain the vision proxy within one minute on a single GPU while improving the zero-shot accuracy from  $77.02\%$  to  $80.21\%$  on ImageNet with ViT-L/14@336 pre-trained by CLIP.
Vision-language pre-training in CLIP aims to align image-text pairs to facilitate multi-modal understanding for downstream tasks [26]. By optimizing the contrastive loss consisting of images and their corresponding text descriptions, CLIP demonstrates an impressive zero-shot transfer of the pre-trained model to generic tasks. Concretely, given a downstream data set, each class has a proxy obtained from the text embedding about the corresponding class name, and then images can be categorized by finding the nearest class proxy. With class names only, zero-shot accuracy on ImageNet [28] can achieve  $77.02\%$  with ViT-L/14@336 [9] pre-trained by CLIP.
After the success of CLIP, many research efforts have been devoted to improving the transfer performance on downstream tasks. Most of developed methods can be cast into two categories, that is, few-shot refinement and zero-shot enhancement. Few-shot methods require a small amount of labeled data from the target task to refine the prediction [12, 41, 42]. While these methods show a better performance than zero-shot learning, labeled examples from the target domain may not be available in real-world applications. Therefore, many methods focus on improving the zero-shot learning itself by exploring side information. On one hand, many pre-trained large language models (LLMs), e.g., GPT-3 [4], can be leveraged to obtain better text proxy for classification [20]. On the other hand, unlabeled data from the target domain can be adopted to fine-tune an adapter network or input prompts [23, 30] for alignment. In this work, we focus on understanding the behavior of CLIP by exploiting the unlabeled data without external LLMs or additional architectures for zero-shot categorization.
When considering pre-trained CLIP models and excluding any auxiliary networks for zero-shot transfer, many existing methods try to optimize input text prompts to align images and the text of class names better [30, 33, 42]. However, recent observations demonstrate that the modality gap between text (i.e., language) and vision is still significant after the optimization of CLIP [18]. Consequently, the efforts on improving the text proxy can be ineffective for the vision task.
To mitigate the problem from the gap between the text and vision space, we propose to obtain proxies of classes in the vision space directly. First, our theoretical analysis in Proposition 1 shows that the modality gap will be preserved due to a small temperature parameter in the contrastive loss of CLIP, which confirms the empirical observation in [18]. Furthermore, Proposition 2 indicates that the optimal proxy for the vision task can only be obtained from the vision space. Therefore, with the gap between the text and vision space, directly using the text proxy makes the zero-shot transfer sub-optimal. To recover the optimal vision proxy, we develop an intra-modal proxy learning (InMaP) method to obtain visual proxies with pseudo labels from text proxies. The proposed method is illustrated in Fig. 1. With fixed features extracted by CLIP encoders, the optimization problem in InMaP is convex without encoders in the loop, which can be solved efficiently with gradient descent [3]. Our main contributions can be summarized as follows.
Figure 1: Illustration of our proposed intramodal proxy learning (InMaP). Triangles denote the class proxy while circles denote the vision data. Different classes are denoted with different colors. By recovering the vision proxy, it can align with vision data better (i.e., from solid line as in the text space to dashed line in the vision space).
- We theoretically show that the modality gap cannot be reduced sufficiently by minimizing the contrastive loss in CLIP. While the optimal proxy for image classification is in the vision space, the gap degenerates the zero-shot performance of the text proxy as shown in Theorem 1.
- We propose a novel algorithm to obtain the intra-modal proxy directly from the vision space. By recovering the vision proxy with help from the text proxy and an appropriate temperature suggested by Proposition 3 for calibration, effective proxies can be observed from unlabeled data.
- Our Theorem 2 indicates that the accuracy of pseudo labels from the text proxy is essential to recover the proxy in the vision space. Therefore, we further develop strategies to refine pseudo labels according to the data distribution by optimal transport (OT) [34].
- Experiments on 14 downstream vision tasks demonstrate both the effectiveness and efficiency of our proposal. Concretely, InMaP can achieve  $80.21\%$  accuracy on ImageNet with ViT-L/14@336 using a single GPU within one minute.
Before CLIP, zero-shot learning that aims to identify examples of novel classes without any labeled training data has attracted much attention since [22]. However, most of existing works are only able to discover new classes closely related to the training classes [1, 5, 6, 11, 22, 38], where they share the similar attributes, and have to train an individual model for each task. On the contrary, pre-training on large-scale data with the contrastive loss aligns visual and language features and makes a single CLIP model applicable for diverse downstream tasks in a straightforward way. Compared with conventional zero-shot methods, the pre-training data in CLIP may be overlapped with downstream tasks, which can result in data leak for evaluation. While the issue can be addressed by eliminating the overlapping data, the influence on the performance is negligible as discussed in [26].
After pre-training with image-text pairs, the obtained CLIP models can be transferred to various downstream tasks with few-shot or zero-shot strategies. We briefly review these two related directions as follows.
Few-shot transfer CLIP demonstrates that an ensemble of input text prompts for the text encoder can improve the zero-shot performance by a significant margin. Therefore, many methods try to
further refine the prompts. Given a small set of labeled data from the target vision tasks, CoOp [42] optimizes input prompts for the text encoder with labels. By formulating the prompts as learnable variables, it can obtain task-dependent prompts without any hand-crafted designs. On the contrary, CLIP-Adapter [12] optimizes obtained output features from both encoders. However, it adds additional bottleneck layers for training, which increases the complexity of optimization. To reduce the efforts of training with encoders, Tip-Adapter [41] demonstrates a training-free method that incorporates the prediction from a cache model consisting of the labeled image data and that from the text proxy. In this work, we also consider to minimize the training efforts by excluding encoders in optimization and the intra-modal proxy can be obtained from extracted features.
Zero-shot transfer Since labeled data are not always available, many efforts are devoted to improving zero-shot transfer. Some methods explore other large pre-trained models as the supplementary for CLIP. VisDesc [20] leverages GPT-3 to generate rich context descriptions for given class names, and thus shows superior performance over the simple prompt in CLIP. SuS-X [33] includes additional images to mitigate the modality gap. Concretely, auxiliary images can be obtained either from a large data set (e.g., LAION [29]) or a text-to-image generation model (e.g., stable diffusion [27]). The additional large models will inevitably increase the inference time, hence, recent methods consider to leverage the unlabeled target data for fine-tuning. UPL [15] and TPT [30] utilize the unlabeled data to optimize learnable input text prompts. SVL-Adapter [23] trains an additional encoder from the unlabeled data with self-supervised learning for ensemble. Unlike existing methods, our proposal does not need any additional model and we learn the proxy for each class from the vision space directly.
In this section, we first analyze the modality gap in CLIP, and then elaborate the proposed method that recovers the proxy in the vision space to reduce the influence from the modality gap.
3.1 Cross-Modal Learning in CLIP
By pre-training paired images and text, a vision-language model, e.g., CLIP [26], is capable of classifying unlabeled images in the zero-shot manner. Let  $\tilde{f}(\cdot)$  and  $g(\cdot)$  denote the encoder for images and text in the pre-trained CLIP model, respectively. Given a set of unlabeled image data  $\{x_i\}_{i=1}^n$  and the unique class names for the set  $\{z_j\}_{j=1}^C$ , their corresponding visual and text representations can be extracted as
$$
\mathbf {x} _ {i} = f (x _ {i}); \quad \mathbf {z} _ {j} = g (z _ {j})
$$
where  $\mathbf{x}_i\in \mathcal{R}^d$  and  $\mathbf{z}_j\in \mathcal{R}^d$  have the same dimension and a unit norm.  $d$  is the dimension of features.  $\mathbf{z}_j$  can be considered as the proxy for the  $j$ -th class.
With the features of the  $i$ -th image and all text proxies, the image can be classified as
$$
y _ {i} = \arg \max _ {j} \mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {j}
$$
where  $y_{i}$  is the prediction from the zero-shot classification. The capacity of zero-shot prediction is from the alignment between vision and language that is optimized by the pre-training objective in CLIP. Given image-text pairs as  $(x_{i}, t_{i})$ , two encoders are learned by minimizing the contrastive loss
$$
\sum_ {i} - \log \frac {\exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {i} / \tau)}{\sum_ {j} ^ {m} \exp (\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {j} / \tau)} - \log \frac {\exp (\mathbf {t} _ {i} ^ {\top} \mathbf {x} _ {i} / \tau)}{\sum_ {j} ^ {m} \exp (\mathbf {t} _ {i} ^ {\top} \mathbf {x} _ {j} / \tau)}
$$
where  $m$  is the mini-batch size,  $\mathbf{t}_i = g(t_i)$  is the extracted text representation of  $t_i$  with a unit norm, and  $\tau$  is the temperature for the normalized cross entropy loss.
While CLIP demonstrates an impressive zero-shot performance, the cross entropy loss focuses on cross-modal ranking and can be sub-optimal for intra-modal vision categorization. It is because that cross entropy loss is closely related to a triplet loss [25] as
$$
\forall j, \quad \mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {i} - \mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {j} \geq 0; \quad \mathbf {t} _ {i} ^ {\top} \mathbf {x} _ {i} - \mathbf {t} _ {i} ^ {\top} \mathbf {x} _ {j} \geq 0
$$
which mainly aims to rank the multi-modal positive pair over negative pairs rather than to mix the text and vision space for modality gap reduction. Moreover, a small temperature, e.g., 0.01 in CLIP, will
amplify the difference between pairs and preserve the distance between modalities. The phenomenon can be stated as follows. The detailed proof can be found in the appendix.
Proposition 1. Assuming  $\frac{\exp(\mathbf{x}_i^\top\mathbf{t}_i / \tau)}{\sum_j^m\exp(\mathbf{x}_i^\top\mathbf{t}_j / \tau)} = \delta$ , we have
$$
\mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {k} + c _ {1} \tau \leq \mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {i} \leq \mathbf {x} _ {i} ^ {\top} \mathbf {t} _ {k} + c _ {2} \tau
$$
where  $k$  denotes the nearest negative pair as  $k = \arg \max_{j:j\neq i}\mathbf{x}_i^\top \mathbf{t}_j$ .  $c_1$  and  $c_2$  are constants.  $c_1 > 0$  when  $\delta > 1/2$  and  $c_2 > 0$  when  $\delta > 1/m$ .
Remark Proposition 1 implies that when  $\delta$  is sufficiently large, the relevant text has a higher rank than the irrelevant text, which is the key for zero-shot classification. Moreover, it shows that with the same prediction  $\delta$ , the absolute distance between modalities, i.e.,  $\| \mathbf{x}_i - \mathbf{t}_i\|_2^2 = 2 - 2\mathbf{x}_i^\top \mathbf{t}_i$ , partially depends on the value of temperature, i.e.,  $\| \mathbf{x}_i - \mathbf{t}_i\|_2^2 \geq 2 - 2\mathbf{x}_i^\top \mathbf{t}_k - 2c_2\tau$ . The analysis indicates that optimizing cross entropy loss with a small  $\tau$  will not pull the text and vision space together.
Some recent work empirically observed that inter-modal distribution is significantly different from the intra-modal distribution in CLIP [18, 33], which confirms our analysis. Consequently, the proxies of class names from the text space may not be sufficient to capture the distribution of vision space, which results in the degenerated zero-shot performance. In lieu of using the text proxy directly for zero-shot visual categorization, we propose to obtain the class proxy in vision space as follows.
3.2 Intra-Modal Proxy Learning
With ground-truth labels  $\{y_i\}$ , the standard supervised learning in vision space can be written as
$$
\min  _ {W} \sum_ {i} - \log \left(\frac {\exp \left(\mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {y _ {i}} / \tau_ {I}\right)}{\sum_ {j} ^ {C} \exp \left(\mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {j} / \tau_ {I}\right)}\right) \tag {1}
$$
where  $\mathbf{w}_j$  is the learnable proxy for the  $j$ -th class and  $W = [\mathbf{w}_1, \ldots, \mathbf{w}_C] \in \mathcal{R}^{d \times C}$ .  $\tau_I$  is the temperature for the optimization with vision data. With an equivalent form, it is easy to show that  $\mathbf{w}_j$  is in the vision space spanned by  $\{\mathbf{x}_i\}$ .
Proposition 2. Let  $W^{*}$  be the optimal solution to
$$
\min  _ {W} \sum_ {i} - \log \left(\frac {\exp \left(- \| \mathbf {x} _ {i} - \mathbf {w} _ {y _ {i}} \| _ {2} ^ {2} / (2 \tau_ {I})\right)}{\sum_ {j} ^ {C} \exp \left(- \| \mathbf {x} _ {i} - \mathbf {w} _ {j} \| _ {2} ^ {2} / (2 \tau_ {I})\right)}\right)
$$
which is equivalent to Eqn. 1 if  $\mathbf{w}_j$  as well as  $\mathbf{x}_i$  has a unit norm. Then, we have
$$
\forall j, \quad \mathbf {w} _ {j} ^ {*} = \Pi_ {\| \mathbf {w} \| _ {2} = 1} \frac {\sum_ {i : y _ {i} = j} (1 - p _ {i , j}) \mathbf {x} _ {i} - \sum_ {k : y _ {k} \neq j} p _ {k , j} \mathbf {x} _ {k}}{\sum_ {i : y _ {i} = j} (1 - p _ {i , j}) - \sum_ {k : y _ {k} \neq j} p _ {k , j}}
$$
where  $p_{i,j} = \frac{\exp(-\|\mathbf{x}_i - \mathbf{w}_j^*\|_2^2 / (2\tau_I))}{\sum_k^C\exp(-\|\mathbf{x}_i - \mathbf{w}_k^*\|_2^2 / (2\tau_I))}$  and  $\Pi_{\| \mathbf{w}\| _2 = 1}$  projects the vector to be with a unit norm.
Proof. It is directly from K.K.T condition [3].
Remark Proposition 2 demonstrates that the optimal proxy for vision categorization should be in the vision space. While proxies from the text space can work as the substitute for vision proxies in CLIP, the distance between the text and vision space (i.e., the modality gap) as illustrated in Proposition 1 will degenerate the performance for vision tasks.
To further demonstrate the challenge from the gap between modalities and simplify the analysis, we decompose the proxy of class names as
$$
\mathbf {z} _ {j} = \sqrt {a} \mathbf {z} _ {j} ^ {x} + \sqrt {1 - a} \mathbf {z} _ {j} ^ {\perp}
$$
where  $\mathbf{z}_j^x$  is from the vision space spanned by  $\{\mathbf{x}_i\}$  and  $\mathbf{z}_j^\perp$  shows the component from the orthogonal subspace such that  $\mathbf{z}_j^{x\top}\mathbf{z}_j^\perp = 0$ . Both  $\mathbf{z}_j^x$  and  $\mathbf{z}_j^\perp$  have the unit norm.  $\mathbf{z}_j^\perp$  can be considered as encoding the specific information for the text modality. With the decomposition, we can observe that it is possible to recover the optimal prediction if the vision space is covered by the text space as follows.
Proposition 3. Let  $p_{i,j}$  and  $p_{i,j}^{\prime}$  denote the prediction probability obtained with vision proxies  $\{\mathbf{w}_j^*\}$  and text proxies  $\{\mathbf{z}_j\}$ , respectively.  $\tau_T$  and  $\tau_I$  denote the temperature in CLIP and that in vision proxy learning, respectively. If  $\mathbf{z}_j^x = \mathbf{w}_j^*$  and  $0 < a < 1$ , we have  $p_{i,j}^{\prime} = p_{i,j}$  when  $\tau_I = \tau_T / \sqrt{a}$ .
Remark Proposition 3 shows that if the text space contains the whole vision space, the intramodal prediction can be recovered from the cross-modal prediction using a larger temperature as  $\tau_{I} = \tau_{T} / \sqrt{a}$ , where  $a$  measures the overlap between modalities.
However, recent observations indicate that the text and vision space learned by CLIP are distinct with a clear margin [18] and it is hard for the text space to cover the vision space. The difference between the text and vision proxy can be lower-bounded in the following theorem.
Theorem 1. Let  $Z$  and  $W^{*}$  denote the text proxies and optimal vision proxies, where  $Z = \sqrt{a} Z^{x} + \sqrt{1 - a} Z^{\perp}$  and  $Z^{x}$  consists of a rank  $r$  approximation of  $W^{*}$  as  $Z^{x} = U_{r}A^{\top}$ .  $W^{*} = U\Sigma V^{\top} = \sum_{i}^{d'} s_{i}u_{i}v_{i}^{\top}$ , where  $d' = \min \{d,C\}$  and  $s_1 \geq \dots \geq s_{d'} \geq 0$ . Then, we have
$$
\| Z - W ^ {*} \| _ {F} ^ {2} \geq 2 C (1 - \sqrt {a}) + \sqrt {a} \sum_ {i = r + 1} ^ {d ^ {\prime}} s _ {i} ^ {2}
$$
Remark Theorem 1 shows that the gap between text and vision proxy comes from two parts. The former term indicates the distance to the irrelevant text space as  $1 - \sqrt{a}$ . The latter one depicts the approximation loss from the low-rank overlapping between text space and vision space with  $r < d'$ . Due to the inherent difference between these two modalities, the distance between  $Z$  and  $W$  is hard to be minimized.
Therefore, to improve the zero-shot performance in the vision space, we consider to obtain the vision proxy in lieu of directly using the text proxy. The main challenge comes from the fact that the label  $\{y_{i}\}$  is unavailable in zero-shot classification. Fortunately, the intra-modal proxy can be learned by mimicking the proxy from the other modal. Concretely, we propose to minimize the KL divergence between distributions from the text and vision proxy as
$$
\min  _ {W} L \left(P ^ {\prime}, W\right) = \sum_ {i} D _ {\mathrm {K L}} \left(P _ {i} ^ {\prime} \mid \mid P _ {i}\right) \tag {2}
$$
where  $P_{i}^{\prime}$  and  $P_{i}$  indicate the distribution estimated by the text proxy  $Z$  and the learnable vision proxy  $W$ , respectively.
$$
p _ {i, j} ^ {\prime} = \frac {\exp \left(\mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {j} / \tau_ {T}\right)}{\sum_ {k} ^ {C} \exp \left(\mathbf {x} _ {i} ^ {\top} \mathbf {z} _ {k} / \tau_ {T}\right)}; \quad p _ {i, j} = \frac {\exp \left(\mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {j} / \tau_ {I}\right)}{\sum_ {k} ^ {C} \exp \left(\mathbf {x} _ {i} ^ {\top} \mathbf {w} _ {k} / \tau_ {I}\right)}
$$
Proposition 3 suggests that  $\tau_{I}$  for  $P_{i}$  should be larger than  $\tau_{T}$  in  $P_{i}^{\prime}$  to calibrate the magnitude of the distribution and our ablation study confirms the analysis.
Unlike prompt learning methods [30], the problem in Eqn. 2 is defined with extracted features, which does not require the backward operation with large encoder networks for optimization. Moreover, it is convex and the optimal vision proxy can be obtained efficiently by the standard gradient descent [3].
By learning with the pseudo label predicted from the text proxy, we can recover the optimal proxy in the vision space as shown in the following theorem.
Theorem 2. If simplifying the loss function as  $L(P', W) = -\sum_{i,j} P_{i,j}' \log (P_{i,j})$  and assuming that it is  $\mu$ -strongly convex in  $W$  and letting
$$
W ^ {\prime *} = \arg \min  _ {W} L (P ^ {\prime}, W); \quad W ^ {*} = \arg \min  _ {W} L (Y, W)
$$
where  $Y_{i}$  is the ground-truth label distribution for  $\mathbf{x}_i$ , we have
$$
\| W ^ {\prime *} - W ^ {*} \| _ {F} ^ {2} \leq \frac {2}{\mu} \| P ^ {\prime} - Y \| _ {F} \| \log \left(P _ {W ^ {\prime *}}\right) - \log \left(P _ {W ^ {*}}\right) \| _ {F}
$$
Remark Theorem 2 shows that the distance between the recovered vision proxy and the optimal vision proxy is bounded by the gap between pseudo labels predicted by text proxy and the ground-truth label distribution. After pre-training CLIP models on a large scale data set, it can approximate the distribution of real data, which enables learning of the intra-modal proxy.
Algorithm 1 Intra-Modal Proxy Learning (InMaP)
1: Input: Unlabeled image set  $\{x_{i}\}$ , class names  $\{z_{j}\}$ , pre-trained CLIP encoders  $(f, g)$ , iterations  $T_{w}, T_{p}$ , temperature  $\tau_{T}, \tau_{I}$ , threshold  $\alpha$
2: Extract features  $\{\mathbf{x}_i\}$  and  $\{\mathbf{z}_j\}$  using pre-trained encoders  $f$  and  $g$ , respectively.
3: Obtain pseudo labels by solving Eqn. 3 with  $T_{p}$  iterations.
4: Refine pseudo labels with the threshold as in Eqn. 5.
5: Obtain vision proxies  $W$  by solving Eqn. 2 with  $T_{w}$  iterations.
6: return  $y_{i} = \arg \max_{j}\mathbf{x}_{i}^{\top}\mathbf{w}_{j}$
3.3 Pseudo Label Refinement
According to Theorem 2, the accuracy of pseudo label is essential for recovering the appropriate vision proxy. Therefore, we develop strategies to improve the quality of pseudo labels.
First, since we have the whole test set rather than a single example for zero-shot learning, the labels obtained from the text proxy can be refined according to the geometry of the target vision data. Given the logits matrix  $M \in \mathcal{R}^{n \times C}$  with  $M_{i,j} = \mathbf{x}_i^\top \mathbf{z}_j$ , the original pseudo label can be obtained by solving the problem
$$
P ^ {\prime} = \arg \max  _ {P} \langle P, M \rangle + \tau_ {T} H (P) \quad s. t. \quad \forall i, \sum_ {j} P _ {i, j} = 1 / n; \quad \forall i, j, P _ {i, j} \geq 0
$$
where  $H(P)$  computes the entropy of  $P$  and the obtained results should be multiplied by  $n$  as the predicted labels. By incorporating a reference distribution  $q \in \mathcal{R}^C$  over classes, the problem can be rewritten as
$$
\max  _ {P} \langle P, M \rangle + \tau_ {T} H (P) \quad s. t. \quad \forall i, \sum_ {j} P _ {i, j} = 1 / n; \quad \forall j, \sum_ {i} P _ {i, j} = q _ {j}; \quad \forall i, j, P _ {i, j} \geq 0 \tag {3}
$$
which is known as Sinkhorn distance [8] and is an approximation of optimal transport distance. The refined pseudo label can be obtained by an efficient iterative method [8].
The last challenge is from observing the appropriate reference distribution. Without any prior knowledge, a balanced distribution  $q = \mathbf{1} / C$  is a popular selection in practice. By investigating the prediction from the text proxy, it implies a distribution as  $\hat{q}_j = \sum_i P_{i,j}' / n$ . Then, a reference distribution can be obtained by further smoothing  $\hat{q}$  as
$$
q _ {j} = \frac {\hat {q} _ {j} ^ {\gamma}}{\sum_ {k} \hat {q} _ {k} ^ {\gamma}} \tag {4}
$$
where  $\gamma \leq 1$  is the smoothing parameter and  $\gamma = 1$  implies the original distribution from the text proxy.
Moreover, inspired by the semi-supervised learning [31], soft label with high confidence can be converted into one-hot label to eliminate the influence from irrelevant classes. Let  $\alpha$  be the threshold. After obtaining the pseudo label from Eqn. 3, we have