SlowGuess/NAACL_2022_OCR huggingface dataset:
Yuanhao Xiong*
yhxiong@cs.ucla.edu
Wei-Cheng Chang
Amazon Inc.
weicheng.cmu@gmail.com
Hsiang-Fu Yu
rofu.yu@gmail.com
The eXtreme Multi-label text Classification (XMC) problem concerns finding most relevant labels for an input text instance from a large label set. However, the XMC setup faces two challenges: (1) it is not generalizable to predict unseen labels in dynamic environments, and (2) it requires a large amount of supervised (instance, label) pairs, which can be difficult to obtain for emerging domains. In this paper, we consider a more practical scenario called Extreme Zero-Shot XMC (EZ-XMC), in which no supervision is needed and merely raw text of instances and labels are accessible. Few-Shot XMC (FS-XMC), an extension to EZ-XMC with limited supervision is also investigated. To learn the semantic embeddings of instances and labels with raw text, we propose to pre-train Transformer-based encoders with self-supervised contrastive losses. Specifically, we develop a pre-training method MACLR, which thoroughly leverages the raw text with techniques including Multi-scale Adaptive Clustering, Label Regularization, and self-training with pseudo positive pairs. Experimental results on four public EZ-XMC datasets demonstrate that MACLR achieves superior performance compared to all other leading baseline methods, in particular with approximately  $5 - 10\%$  improvement in precision and recall on average. Moreover, we show that our pre-trained encoder can be further improved on FS-XMC when there are a limited number of ground-truth positive pairs in training. Our code is available at https://github.com/amzn/pecos/tree/mainline/examples/MACLR.
The eXtreme Multi-label text Classification (XMC) problem aims at tagging a text input with most relevant subset of labels from an extremely large output space. Many web-related applications can be formulated as an XMC task with encouraging results,
such as finding the best matching products from a large catalog in e-commerce systems (Medini et al., 2019; Chang et al., 2021), auto-completing queries given its prefix on search engines (Yadav et al., 2021), predicting search keywords for dynamic advertising (Prabhu et al., 2018; Chang et al., 2020b), tagging categories of Wikipedia articles from a large label taxonomy (Dekel and Shamir, 2010; Chalkidis et al., 2019), to name just a few.
The current XMC setup is built on full label coverage and full supervision, where full label coverage means labels to be predicted have appeared in the training set and full supervision indicates it requires a significant number of annotated (instance, label) pairs. In detail, it is assumed that an XMC algorithm has access to raw text of instances and labels, together with their corresponding relations during training, as shown in Figure 1.
However, there are several limitations of this XMC setting. First of all, due to the assumption of full label coverage, it is typical in XMC approaches to simply treat labels as IDs for classification and thus they are restricted to making predictions within observed labels. This assumption is unrealistic since the label set usually keeps growing over time, e.g., newly added websites or products which are absent during training yet crucial for applications such as recommendation and advertising. Besides, collecting labeled pairs is time-consuming, expensive and sometimes infeasible, for example, launching an e-commerce system in the emerging locale, where no user behavioral signals are available. In spite of these constraints, most existing methods (Dahiya et al., 2021b; You et al., 2019; Mittal et al., 2021; Dahiya et al., 2021a) followed this XMC setup. It can be seen in Figure 2 that Astec (Dahiya et al., 2021b), one of the state-of-the-art extreme classifiers, is incapable of handling the scenario without supervision, which leads to zero performance in both Precision@5 and Recall@100. Moreover, the increasing trend in Astec's perfor
*Work done during an internship at Amazon Inc.
Human Language Technologies, pages 5455 - 5468
Figure 1: Four different settings in XMC. Four essential components are considered: instances (raw text), labels (raw text), supervision (positive pairs), and label coverage. In detail, we divide label coverage into 3 groups: full, partial, and none. * in FS-XMC emphasizes that only a limited amount of supervision is available. We can see that EZ-XMC is the most general and practical setting, where no supervision and label coverage is required.
mance along with the label ratio suggests that it depends highly on the supervision level and is hard to generalize to unseen labels. This motivates us to investigate how to design an effective XMC model with zero supervision.
(a) Precision@5.
(b) Recall@100.
Figure 2: Performance of three representative XMC methods on LF-Amazon-131K at different ratios of label coverage. A subset covering [0, 1, 5, 10, 25, 50, 100]  $(\%)$  of the whole label set is sampled for fine-tuning.
In this paper, we consider an essential yet under-explored XMC setting, called Extreme Zero-shot XMC (EZ-XMC). As depicted in Figure 1, we can access raw text of both instances and labels but do not know their corresponding relations in EZ-XMC. Moreover, we do not make any assumption on the label coverage, so the labels in the testing set may or may not appear in the training stage. An extension to EZ-XMC with a limited number of training pairs, Few-shot XMC (FS-XMC), is also taken into account in our paper. Either EZ-XMC or FS-XMC occurs frequently in the real world since informative and abundant (instance, label) pairs are never easy to obtain. Also, it is more practical and worthwhile to reduce labor for manual annotation by solving problems under EZ-XMC. Note that generalized zero-shot XMC (GZ-XMC) proposed in a recent work (Gupta et al., 2021) can be regarded as a special case of EZ-
XMC. GZ-XMC allows that the set of test labels is not completely overlapped with training labels but still requires supervision from positive pairs, as shown in Figure 1. From Figure 2, we can observe that ZestXML (Gupta et al., 2021) designed for GZ-XMC also suffers the issue of no supervision.
A natural question then arises: how should we deal with EZ-XMC problems? Despite the name, EZ-XMC is barely easy to tackle. Fortunately, although dedicated supervision signals are lacking, raw text of instances and labels, e.g., product descriptions and categories, are still accessible in EZ-XMC. Thus it is of vital importance to effectively leverage self-information of these data to train a model for classification. To overcome challenges encountered in EZ-XMC, we turn to solving the problem from a different perspective without learning classifiers explicitly. In particular, XMC can be cast into a problem which learns a sentence encoder  $\mathcal{E}$  to map instances and labels into dense embeddings, and predictions are made through approximate nearest neighbor search algorithms in the latent space (Shrivastava and Li, 2014). Motivated by recent progresses in self-supervised learning (Gao et al., 2021; Chen et al., 2020; He et al., 2020b; Devlin et al., 2019), we propose MACLR (Multiscale Adaptive Clustering & Label Regularization), a two-stage pre-training procedure with those unpaired raw data to obtain a sentence encoder  $\mathcal{E}$  under EZ-XMC. As to FS-XMC, fine-tuning the encoder on a few paired data is sufficient for the performance boost. Figure 2 demonstrates that MACLR achieves superior performance when no supervision is available and achieves much higher recall than Astec and ZestXML by a large margin,
even under the higher label coverage ratio.
- We propose an essential Extreme Zero-Shot XMC (EZ-XMC) setting without any assumptions on supervision and label coverage, which has not been explored in previous work and is more practical in real applications.
- We leverage unlabeled data to pretrain the sentence encoder  $\mathcal{E}$  with improved Inverse Cloze Task in Stage I of MACLR. In particular, multi-scale adaptive clustering and label regularization are proposed to utilize raw text thoroughly. In Stage II, we further self-train the encoder with pseudo positive pairs constructed from  $\mathcal{E}$  in Stage I as well as TF-IDF model with complementary information.
- Comprehensive experiments are conducted on four public benchmark EZ-XMC datasets. Results demonstrate that our pre-trained encoder can outperform existing unsupervised baseline methods notably. As an example, MACLR achieves Recall@100 of  $54.99\%$ , nearly the same level as Astec (one of the SOTA XMC methods) (Dahiya et al., 2021b) trained with a supervised subset covering around  $70\%$  labels on LF-Amazon-131K.
- MACLR can also achieve comparable or even better performance under the few-shot setting than those models heavily dependent on supervised information. For example, MACLR is better than the SOTA ZestXML (Gupta et al., 2021) in Recall@100 over  $20\%$  (57.55% v.s. 32.69%) when fine-tuned on the subset covering  $1\%$  labels of LF-Amazon-131K.
Extreme multi-label classification Various extreme classifiers have been proposed to address the large output space challenge of XMC problems. We can broadly categorize them into two groups: partitioned-based models with linear classifiers (Prabhu et al., 2018; Prabhu and Varma, 2014; Yu et al., 2020) that partition labels with hierarchical trees, leading to sub-linear inference time complexity, and embedding-based methods (Bhatia et al., 2015; Jain et al., 2019; Guo et al., 2019) that learn a classifier for each label and leverage approximated nearest neighbor (Malkov and Yashunin, 2018; Guo et al., 2016) to index labels in the large output space. There are also deep learning models such as Atten
tionXML (You et al., 2019), Astec (Dahiya et al., 2021b), SiameseXML (Dahiya et al., 2021a), and XR-Transformer (Zhang et al., 2021) that further improve the accuracy of those linear counterparts with various advanced encoder architectures. Nevertheless, none of those XMC methods can handle the EZ-XMC setup: they not only suffer from the lack of supervised signals, but also fail to generalize to unseen cold-start labels in the test set. The only exception is ZestXML (Gupta et al., 2021), a recently proposed XMC method that was designed to address the generalized zero-shot XMC (GZ-XMC) problem where a number of labels for prediction are absent during training. While ZestXML partially resolves the generalization challenge of cold-start labels, just like those conventional XMC models, it still depends heavily on a large number of training data with positive (instance, label) pairs.
Self-supervised learning techniques The past few years have witnessed great promise in selfsupervised learning (Lan et al., 2020; Chen et al., 2020; He et al., 2020b; Devlin et al., 2019; Khosla et al., 2020; Gao et al., 2021), where a pre-training task is defined using only data's self-information. Learned representations from the pre-training task can be then leveraged in a wide range of downstream tasks in various domains, such as image classification (Chen et al., 2020; He et al., 2020b) and object detection (Li et al., 2020) in computer vision, and open-domain question answering (Lee et al., 2019; Guu et al., 2020) in natural language processing. Specifically, we focus on contrastive approaches for Sentence-BERT (Reimers and Gurevych, 2019) models in this paper, where the intuition is to pull semantically close neighbors together and push apart non-neighbors via noise contrastive estimation or N-pair losses. Various effective pre-training tasks such as Inverse Cloze Task (ICT) (Lee et al., 2019) and SimCSE (Gao et al., 2021) have been shown to improve the performance of Sentence-BERT models.
In this section, we present the problem formulation of EZ-XMC. With  $\mathcal{X}$  and  $\mathcal{V}$  denoting the set of instances and labels respectively, the general XMC problem can be viewed as learning a scoring function  $f:\mathcal{X}\times \mathcal{V}\to \mathbb{R}$ .  $f(\cdot ,\cdot)$  maps an (instance, label) pair  $(x,y)$  to a similarity score, which is used to make a prediction through approximate nearest neighbor search algorithms. In previous set
tings such as XMC and GZ-XMC, a considerable amount of relevant (instance, label) pairs  $\{(x_i,y_i)\}$  are available. On the contrary, in EZ-XMC, we have no knowledge about corresponding relations between instances and labels, but only their raw text, as shown in Figure 1. In this case, existing approaches that depend on the relevant pairs fail to learn an effective scoring function, even with a few paired data under FS-XMC.
Recent progresses in self-supervised learning have shown that a generalized sentence encoder can be learned through elaborately designed pretraining tasks even without any supervision (Lee et al., 2019; Chang et al., 2020a), and then adapted to different downstream tasks directly or via slight finetuning. On the other hand, the scoring function  $f$  can be modeled as  $f(x,y) = \langle \mathcal{E}(x),\mathcal{E}(y)\rangle$ , where  $\mathcal{E}$  is a sentence encoder producing semantical dense embeddings, and  $\langle \cdot ,\cdot \rangle$  is the similarity measurement such as inner product and cosine similarity. Without loss of generality, inner product is adopted in the paper as the similarity metric between embeddings of instances and labels. Thus, we formulate the problem as training an encoder  $\mathcal{E}$  with raw text of  $\mathcal{X}$  and  $\mathcal{Y}$  through a pre-training task for EZ-XMC. As to the few-shot scenario FS-XMC, we can fine-tune  $\mathcal{E}$  for improvement.
In this section, we introduce a two-stage pretraining procedure, MACLR, to thoroughly leverage unpaired data with raw text for EZ-XMC. Specifically, we present the general framework in Section 4.1, and then dive into details of two stages, pre-training with the improved Inverse Cloze Task and self-training with pseudo positive pairs, in Sections 4.2 and 4.3 respectively. A complete algorithm is presented in Algorithm 1 in Appendix 4.4.
The framework of our pre-training procedure is shown in Figure 3. MACLR consists of two stages:
- Stage I: title-context pairs are constructed for the Inverse Cloze Task, and the encoder  $\mathcal{E}$  is then trained on these pairs together with two proposed techniques, multi-scale adaptive clustering and label regularization.
- Stage II: More pseudo positive pairs are crafted using different score functions modeled by the encoder from Stage I and TF-IDF respectively.  $\mathcal{E}$  is further trained on additional
pairs to improve the encoding performance. Details of each component in our pre-training framework are discussed in the following sections.
Figure 3: Framework of our pre-training procedure.
4.2 Stage I: Pre-training with improved ICT
Inverse Cloze Task (Lee et al., 2019) is a frequently used pre-training task for the sentence encoder. Specifically, for an instance  $x = \{s_1, \ldots, s_n\}$  consisting of  $n$  sentences, ICT randomly samples a sentence to serve as the pseudo positive label  $\hat{y} = s_i$  where  $i \sim [1, n]$ . Then the rest of  $x$  is the pseudo instance  $\hat{x} = \{s_1, \ldots, s_{i-1}, s_{i+1}, \ldots, s_n\}$ . In XMC, due to the property that the label usually summarizes the instance with one short sentence, which works similarly as the title  $s_1$ , we directly utilize (context, title) pairs in the form of  $(\hat{x} = \{s_2, \ldots, s_n\}, \hat{y} = s_1)$ . This construction works as the analog of the ground truth (instance, label) pairs and capture the semantics of a sentence. With these pseudo pairs, the contrastive training objective for a mini-batch of  $N$  pairs is as follows:
$$
\mathcal {L} _ {\text {c o n t r a s t i v e}} = - \sum_ {i = 1} ^ {N} \log \frac {\exp \left(\mathcal {E} \left(\hat {x} _ {i}\right) \cdot \mathcal {E} \left(\hat {y} _ {i}\right)\right)}{\sum_ {j = 1} ^ {N} \exp \left(\mathcal {E} \left(\hat {x} _ {i}\right) \cdot \mathcal {E} \left(\hat {y} _ {j}\right)\right)} \tag {1}
$$
Based on ICT, we also develop two techniques, multi-scale adaptive clustering and label regularization, to fully leverage the information of unpaired instances and labels.
4.2.1 Multi-scale Adaptive Clustering
In the original ICT scheme, we can construct only one positive pair for a particular instance. It is relatively hard in contrastive learning without enough positive examples, especially for extreme multi-label classification where one instance might be associated with more than one label, and a label is also likely to point to several different instances at the same time. Thus a question arises naturally:
is it possible to construct more positive pairs from purely unpaired raw data to intergrate richer information into the pre-training process? We solve it by the unsupervised K-means clustering. In detail, we divide pseudo (context, title) pairs from ICT into  $K$  clusters through K-means based on the embeddings of all instances. Then if  $C(\hat{x}_i) = C(\hat{x}_j)$ , i.e.,  $\hat{x}_i$  and  $\hat{x}_j$  belong to the same cluster,  $(\hat{x}_i, \hat{y}_j)$  and  $(\hat{x}_j, \hat{y}_i)$  are regarded as positive pairs besides original ICT pairs. Furthermore, supervised contrastive loss is adopted for training the encoder with a minibatch of  $N$  pairs based on the cluster assignment:
$$
\mathcal {L} _ {\text {c l u s t e r}} = \sum_ {i = 1} ^ {N} \frac {- 1}{| P _ {\mathcal {Y}} (i) |} \sum_ {p \in P _ {\mathcal {Y}} (i)} \log \frac {\exp \left(\mathcal {E} \left(\hat {x} _ {i}\right) \cdot \mathcal {E} \left(\hat {y} _ {p}\right)\right)}{\sum_ {j = 1} ^ {N} \exp \left(\mathcal {E} \left(\hat {x} _ {i}\right) \cdot \mathcal {E} \left(\hat {y} _ {j}\right)\right)} \tag {2}
$$
Here,  $P_{\mathcal{Y}}(i) = \{p \in \{1, \dots, N\} : C(\hat{x}_i) = C(\hat{x}_p)\}$  is the set of indices of all positives for  $\hat{x}_i$  in the batch, and  $|P_{\mathcal{Y}}(i)|$  is its cardinality. Minimizing Equation (2) pulls close the representations of instances and their positive labels within the same cluster and pushes away the representations of those from different clusters.
Besides, since the ultimate goal is the minimization of Equation (1), we propose a multi-scale approach with adaptive training, which guides the encoder to learn the easier tasks with sufficient positive examples, and then master harder tasks gradually. This approach allows the encoder to learn from the coarse scale to the fine scale of clustering assignment, and is similar to the idea of curriculum learning (Bengio et al., 2009) to first focus on learning from a subset of simple examples, and expanding to include the remaining harder samples. Our adaptive training process can be conducted by modifying the cluster size to adjust the task difficulty accordingly. To be specific, we initialize the cluster assignment with the number of clusters  $K_{0}$ , and double the cluster size every  $T$  steps. The cluster assignment is also updated every  $T_{\mathrm{update}}$  steps along with the training of  $\mathcal{E}$ . Such a process lasts for half of the total training steps  $T_{\mathrm{total}}$  to take advantage of positive examples from constructed clusters. The obtained intermediate encoder from this adaptive procedure is expected to satisfactorily capture the semantics of a sentence and is ready to deal with the optimization of Equation (1). Then for the rest half of training steps, we turn to the hardest setting treating each instance as one independent cluster, which exactly falls into the contrastive training objective in Equation (1). Our multi-scale adaptive clustering is illustrated in Figure 4.
Figure 4: An example of multi-scale adaptive clustering. Here different colors represent different clusters. (a) In the beginning, there is only one cluster and  $\{\hat{y}_j\}_{j=1}^4$  are all positive labels for  $\hat{x}_1$ . (b)  $K$  is doubled to 2 and now  $\hat{y}_1$  and  $\hat{y}_3$  are positive to  $\hat{x}_1$ . (c) Finally,  $K$  is equal to 4 where each instance itself is a cluster, and hence  $\hat{x}_1$  only has one positive label  $\hat{y}_1$ . The process is similar for the rest of the instances.
4.2.2 Label Regularization
In addition to leveraging information from the instance side, we also have access to the raw texts of the whole label set and can utilize them to boost the encoder's performance from the label side (Mueller et al., 2022; Müller et al., 2022). Intuitively, for a randomly sampled label, with a high probability it is an negative example to the instance of interest. We can take advantage of this intuition to make the embedding of the instance far from its irrelevant labels. Instead of increasing the distance directly, it is more stable and effective to adopt contrastive losses. To avoid overfitting, we choose a new positive example for each instance instead of its corresponding pseudo label from ICT which has been used in  $\mathcal{L}_{\mathrm{cluster}}$ . More concretely,  $\hat{x}_i^+$  is selected exactly the same as  $\hat{x}_i$ , since the dropout layer is placed in the standard training of Transformer-based models and can be viewed as a minimal form of data augmentation (Gao et al., 2021). By feeding the same sentence to the encoder  $\mathcal{E}$ , two embeddings with different dropout masks are obtained, i.e.,  $\hat{h}_i = \mathcal{E}(\hat{x}_i, z_i)$  and  $\hat{h}_i^+ = \mathcal{E}(\hat{x}_i^+, z_i^+)$  where  $z$  represents a random mask for dropout.  $\hat{h}_i \neq \hat{h}_i^+$  due to the dropout noise, but they hold similar semantics from the same sentence and thus can be used as a positive pair for contrastive learning. The procedure of label regularization is depicted in Figure 5. At each step, we sample  $M$  real labels from the label set  $\mathcal{V}$ , and the regularization term is computed as follows:
$$
\mathcal {L} _ {\text {l a b e l}} = \sum_ {i = 1} ^ {N} - \log \frac {\exp \left(\hat {h} _ {i} \cdot \hat {h} _ {i} ^ {+}\right)}{\sum_ {j = 1} ^ {M} \exp \left(\hat {h} _ {i} \cdot \mathcal {E} \left(y _ {j} ^ {-}\right)\right) + \exp \left(\hat {h} _ {i} \cdot \hat {h} _ {i} ^ {+}\right)} \tag {3}
$$
Through minimizing  $\mathcal{L}_{\mathrm{label}}$ , the encoder learns to pull the instance away from its irrelevant labels
and incorporate the dropout augmentation at the same time. Together with  $\mathcal{L}_{\mathrm{cluster}}$ , we have the final objective function for pre-training in the Stage I as
$$
\mathcal {L} = \mathcal {L} _ {\text {c l u s t e r}} + \mathcal {L} _ {\text {l a b e l}}. \tag {4}
$$
(b)
Figure 5: An illustration of label regularization. (a) shows that  $\hat{x}_i$  is expected to be far from sampled irrelevant labels  $\{y_j^-\}_{j=1}^4$ , while (b) indicates the identical  $\hat{x}_i$  is added as a positive example for label regularization.
4.3 Stage II: Self-training with multi-viewed pseudo pairs
After the pre-training procedure in Section 4.2, we can obtain an intermediate encoder  $\mathcal{E}_I$ . But are there any ways to further improve the encoder? Inspired by self-training in semi-supervised learning (Yalniz et al., 2019; Xie et al., 2020; He et al., 2020a; Zoph et al., 2020),  $\mathcal{E}_I$  can be leveraged to make predictions on those unpaired training instances themselves, to generate pseudo positive pairs. These pseudo pairs are much better than random guessing and can serve as a distinct view from ICT pairs. On the other hand, similar pseudo pairs can be constructed by other unsupervised methods such as TF-IDF, which provide different and complementary information about the instance.
With multi-viewed pseudo positive pairs, we can conduct further training on the encoder in State II from a new perspective and self-improve  $\mathcal{E}_I$ . The detailed process works as follows:
1) Compute the similarity score using  $\mathcal{E}_I$  for each training instance  $x_{i}$ , and select labels with top-k maximum scores as its pseudo labels;
2) Generate labels similarly with TF-IDF, except that  $\mathcal{E}(x)$  and  $\mathcal{E}(y)$  are replaced with their TF-IDF vectors;
3) Mix pseudo positive pairs from 1) and 2) together, and train  $\mathcal{E}_I$  on them with Equation (2).
4.4 MACLR Algorithm