SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
Yandong Li, Di Huang, Danfeng Qin, Liqiang Wang, and Boqing Gong. 2020. Improving object detection with selective self-supervised self-training. In European Conference on Computer Vision, pages 589-607. Springer.
Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. IEEE transactions on pattern analysis and machine intelligence, 42(4):824-836.
Tharun Medini, Qixuan Huang, Yiqiu Wang, Vijai Mohan, and Anshumali Shrivastava. 2019. Extreme classification in log memory using count-min sketch: A case study of amazon search with  $50\mathrm{m}$  products. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13244-13254.
Anshul Mittal, Noveen Sachdeva, Sheshansh Agrawal, Sumeet Agarwal, Purushottam Kar, and Manik Varma. 2021. ECLARE: Extreme classification with label graph correlations. In Proceedings of the Web Conference 2021, pages 3721-3732.
Aaron Mueller, Jason Krone, Salvatore Romeo, Saab Mansour, Elman Mansimov, Yi Zhang, and Dan Roth. 2022. Label semantic aware pre-training for few-shot text classification. arXiv preprint arXiv:2204.07128.
Thomas Müller, Guillermo Pérez-Torró, and Marc Franco-Salvador. 2022. Few-shot learning with siamese networks and label tuning. arXiv preprint arXiv:2203.14655.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In Proceedings of the 2018 World Wide Web Conference on
World Wide Web, WWW 2018, Lyon, France, April 23-27, 2018, pages 993-1002. ACM.
Yashoteja Prabhu and Manik Varma. 2014. Fastxml: a fast, accurate and stable tree-classifier for extreme multi-label learning. In The 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, New York, NY, USA - August 24 - 27, 2014, pages 263-272. ACM.
Anand Rajaraman and Jeffrey David Ullman. 2011. Mining of massive datasets. Cambridge University Press.
Sashank J. Reddi, Satyen Kale, Felix X. Yu, Daniel Niels Holtmann-Rice, Jiecao Chen, and Sanjiv Kumar. 2019. Stochastic negative mining for learning with large output spaces. In The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 1940-1949. PMLR.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Anshumali Shrivastava and Ping Li. 2014. Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2321-2329.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2020. MPNet: Masked and permuted pretraining for language understanding. In Advances in Neural Information Processing Systems.
Nandan Thakur, Nils Reimers, Andreas Rückle, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Qizhe Xie, Minh-Thang Luong, Eduard H. Hovy, and Quoc V. Le. 2020. Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10684-10695. IEEE.
Nishant Yadav, Rajat Sen, Daniel N. Hill, Arya Mazumdar, and Inderjit S. Dhillon. 2021. Session-aware query auto-completion using extreme multi-label ranking. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD '21, page 3835-3844, New York, NY, USA. Association for Computing Machinery.
I Zeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. 2019. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905.00546.
Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5812-5822.
Hsiang-Fu Yu, Kai Zhong, Jiong Zhang, Wei-Cheng Chang, and Inderjit S Dhillon. 2020. PECOS: Prediction for enormous and correlated output spaces. Journal of Machine Learning Research.
Jiong Zhang, Wei-cheng Chang, Hsiang-fu Yu, and Inderjit S Dhillon. 2021. Fast multi-resolution transformer fine-tuning for extreme multi-label text classification. In Advances in Neural Information Processing Systems.
Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. 2020. Rethinking pre-training and self-training. Advances in neural information processing systems, 33:3833-3845.
Datasets and Source Codes Statistics of four datasets are presented in Table 5. All XMC datasets used in the paper are available in the Extreme Classification Repository (Bhatia et al., 2016)<sup>1</sup>, except for LF-Amazon-1M, which is available from the ZestXML paper (Gupta et al., 2021)<sup>2</sup>. Besides, sources code can be accessed in this link.
Table 5: Dataset statistics.  $\mathbf{N}_{\text {train }}$ ,  $\mathbf{N}_{\text {test }}$  and  $\mathbf{N}_{\text {label }}$  are the number of training points, test points, and labels respectively.  $\mathbf{D}_{\text {BoW}}$  is the dimensionality of Bag-of-Words (BoW) features.
Dataset	\( {\mathrm{N}}_{\text{train }} \)	\( {\mathrm{N}}_{\text{test }} \)	\( {\mathrm{N}}_{\text{label }} \)	\( {\mathrm{D}}_{\text{BoW }} \)
LF-Amazon-131K	294,805	134,835	131,073	80,000
LF-WikiSeeAlso-320K	693,082	177,515	312,330	80,000
LF-Wikipedia-500K	1,813,391	783,743	501,070	500,000
LF-Amazon-1M	914,179	1,465,767	960,106	1,000,000
Compared Baselines of EZ-XMC Here we provide detailed description of each baseline method under the EZ-XMC setting.
- TF-IDF (Rajaraman and Ullman, 2011), which represents instances and labels by sparse TF-IDF features and retrieves top labels for each instance based on the similarity of TF-IDF features;
- XR-Linear (Yu et al., 2020), a hierarchical linear model trained with pseudo positive pairs constructed from TF-IDF;
- GloVe (Pennington et al., 2014), which adopts dense average word embeddings with the dimension of 300 trained on co-occurrence statistics to measure similarity between instances and labels;
- Sentence-BERT (SentBERT) (Devlin et al., 2019; Reimers and Gurevych, 2019), a sentence encoder modeled as a Siamese-Transformer to derive semantically meaningful embeddings for instances and labels;
- Paraphrase MPNet (MPNet) (Song et al., 2020), another Sentence-BERT model originally designed for searching sentence paraphrases;
- SimCSE (Gao et al., 2021), a Siamese-Transformer pre-trained with the contrastive objective using dropout noise as augmentation;
- ICT (Lee et al., 2019), another Siamese-Transformer pre-trained with the contrastive objective using (context, title) pairs.
Evaluation Metrics As mentioned before, we adopt precision and recall as our evaluation metrics. In detail,  $P@k$  and  $R@k$  are defined as follows:
$$
P @ k = \frac {1}{k} \sum_ {i \in \operatorname {r a n k} _ {k} \left(y ^ {\prime}\right)} y _ {i}, \quad R @ k = \frac {1}{\sum_ {l} y _ {l}} \sum_ {i \in \operatorname {r a n k} _ {k} \left(y ^ {\prime}\right)} y _ {i}. \tag {5}
$$
$y \in \{0,1\}^L$  and  $y' \in R^L$  are the ground truth vector and the prediction vector respectively.  $\mathrm{rank}_k$  returns the indices of the top- $k$  highest elements.
Hyper-parameters We use a Siamese Transformer model to embed both instances and labels. The encoder consists of a 12 layers BERT-base model, topped with a linear head projecting hidden state of the [CLS] token into a 512-dimensional embedding. The sequence length of the instance and the label is set to be 288 and 64 respectively. We pre-train the model on eight V100 GPUs for 100,000 steps with an Adam optimizer and batch size of 32 per GPU in both Stage I and Stage II. This pre-training process takes about 1 day. We adopt an initial learning rate  $1 \times 10^{-5}$  with the warm-up ratio 0.1, followed by a linear learning rate decay. For fine-tuning, the learning rate of Adam is set to  $5 \times 10^{-6}$  with 2000 training steps for the  $1\%$  label ratio and 10K training steps for
http://manikvarma.org/downloads/XC/XMLRepository.html
2https://github.com/nilesh2797/zeustxml
Table 6: Mean and variance of MACLR performance of four independent runs under EZ-XMC on LF-Amazon-131K.
Method	Precision	Recall
@1	@5	@10	@1	@3	@5	@10	@100
MACLR	17.64 ± 0.11	15.24 ± 0.01	11.81 ± 0.01	10.11 ± 0.04	24.13 ± 0.04	30.14 ± 0.04	37.13 ± 0.03	54.88 ± 0.01
the  $5\%$  label ratio. In the Stage I, we use the initial cluster size  $K_{0} = 2048$  and set  $T_{K} = 10000$  and  $T_{\mathrm{update}} = 5000$ . In Stage II, top 3 ranked labels from predictions of the encoder and TF-IDF are selected to constitute the pseudo set for self-training.
For hyper-parameters of all baselines, we follow their default setups. All experiments are conducted on the AWS p3dn.24xlarge instance, consisting of 96 Intenl Xeon CPUs with 768 GB of RAM and 8 Nvidia V100 GPUs with 32 GB of memory each. It takes about half a day to complete the pre-training procedure of MACLR.
We present error bars of four independent runs to validate our MACLR results are statistically significant under EZ-XMC in Table 6. It can be observed that the variance is small, showing that our method can produce similar results with different random seeds, and MACLR is statically better than other baselines compared with results in Table 1. Therefore, we run each method for four times and report the best performance in the main paper.
B Hyperparameter Study
We provide details of hyperparameter tuning process. For hyperparameters in the Stage I including  $T_{K}$ ,  $T_{\mathrm{update}}$ , and  $K_{0}$ , we choose them from the following ranges:  $T_{K} \in [5000, 10000, 25000]$ ,  $T_{\mathrm{update}} \in [1000, 5000, 10000]$ , and  $K_{0} \in [1024, 2048, 4096]$ . Instead of grid search for all possible combinations (in total  $3^{3} = 27$ ), we tried to investigate the impact of different hyperparameters first and then determined the value. Specifically, we choose the median setting of  $(T_{K} = 10000, T_{\mathrm{update}} = 5000, K_{0} = 2048)$  as the baseline, and for each hyperparameter to be evaluated, we fix the remaining two and observe the performance. Experimental results can be found in Table 7, 8 and 9. We summarize our findings below:
- With  $T_{\mathrm{update}} = 5000, K_0 = 2048, T_K = 10000$  performs the best. We assume that when  $T_K$  is smaller as 5000, the training steps are not sufficient to obtain satisfactory representations under the current cluster size; and when  $T_K$  is larger as 25000, the algorithm spends too much time on one cluster size and
fails to explore different scenarios based on curriculum learning.
- With  $T_{K} = 10000$ ,  $K_{0} = 2048$ ,  $T_{\mathrm{update}} = 1000$  outperforms the other two values. It is expected since the cluster assignment would be more accurate and the learned embeddings would be better accordingly.
- With  $T_{K} = 10000$ ,  $T_{\mathrm{update}} = 5000$ ,  $K_{0} = 2048$  is the best among three values. We hypothesize the trajectory of  $K_{0} = 2048$  gives the appropriate curriculum learning process.
Based on these observations, the best setting should be  $(T_{K} = 10000, T_{\mathrm{update}} = 1000, K_{0} = 2048)$ . However, we can see that  $T_{\mathrm{update}} = 1000$  is only slightly better than  $T_{\mathrm{update}} = 5000$  and considering the cost of updating cluster assignment, we select  $T_{\mathrm{update}} = 5000$  instead.
Table 7: Impact of  $T_K$  with fixed  $T_{\mathrm{update}} = 5000$  and  $K_0 = 2048$  on LF-Amazon-131K.
Hyperparameter	Precision	Recall
@1	@3	@5	@1	@3	@5	@10	@100
TK=5000	15.35	12.78	9.67	8.22	20.56	26.20	33.03	50.24
TK=10000	16.37	13.71	10.65	9.29	21.63	27.03	33.93	51.45
TK=25000	14.46	12.05	9.24	7.68	19.96	25.78	32.64	49.16
Table 8: Impact of  $T_{\mathrm{update}}$  with fixed  $T_K = 10000$  and  $K_0 = 2048$  on LF-Amazon-131K.
Hyperparameter	Precision	Recall
@1	@3	@5	@1	@3	@5	@10	@100
\(T_{update}=1000\)	16.61	13.82	10.90	9.63	21.72	27.35	34.32	51.82
\(T_{update}=5000\)	16.37	13.71	10.65	9.29	21.63	27.03	33.93	51.45
\(T_{update}=10000\)	15.26	12.66	9.72	8.42	20.59	26.14	32.98	50.04
Table 9: Impact of  ${K}_{0}$  with fixed  ${T}_{\text{update }} = {5000}$  and  ${T}_{K} = {10000}$  on LF-Amazon-131K.
Hyperparameter	Precision	Recall
@1	@3	@5	@1	@3	@5	@10	@100
K0=1024	15.82	13.03	9.94	8.74	20.85	26.40	33.06	50.28
K0=2048	16.37	13.71	10.65	9.29	21.63	27.03	33.93	51.45
K0=4096	15.98	13.24	10.05	8.87	21.14	26.69	33.36	50.62
For the hyperparameter  $\mathbf{k}$  in the Stage II, we use the previous setting  $(T_{K} = 10000, T_{\mathrm{update}} = 5000, K_{0} = 2048)$  for pretraining in the stage I, and tune  $k$  from [1,3,5]. Results are shown in
Table 10 in the paper. It can be observed that  $k = 3$  leads to the best performance. This is due to the fact that  $k = 1$  did not introduce a sufficient number of meaningful pseudo (instance, label) pairs while  $k = 5$  was likely to add to the noise of the training set with irrelevant (instance, label) pairs.
Table 10: Impact of the number of pseudo labels  $k$  on LF-Amazon-131K.
Hyperparameter	Precision	Recall
@1	@3	@5	@1	@3	@5	@10	@100
k = 1	17.15	14.92	11.38	9.72	23.76	29.55	36.68	54.46
k = 3	18.13	15.42	11.93	10.35	24.45	30.43	37.28	54.99
k = 5	16.74	14.06	10.93	9.48	23.51	29.12	36.19	53.98
Therefore, the default hyperparameter choice of MACLR is determined as  $(T_{K} = 10000, T_{\mathrm{update}} = 5000, K_{0} = 2048, k = 3)$ , which is tuned on LF-Amazon-131K. Moreover, as we can see that MACLR is not very sensitive to the choice of hyperparameters and performs better than all the baselines, we assume that this configuration would perform similarly on other datasets and use it without further tuning.
C Additional Experiments on FS-XMC
Table 11: Results of FS-XMC where the training subset covers  $1\%$  positive pairs from the whole set.
Method	Precision	Recall
@1	@3	@5	@1	@3	@5	@10	@100
LF-Amazon-131K
XR-Linear	5.37	2.66	1.68	2.81	3.92	4.09	4.26	4.99
Astec	3.29	2.04	1.41	1.93	3.33	3.77	4.06	5.06
SiameseXML	7.14	3.74	2.41	4.22	6.17	6.55	6.95	8.09
ZestXML	12.91	11.31	8.91	7.20	17.69	22.51	28.27	42.40
SentBERT	15.08	11.81	9.06	8.38	18.42	22.89	28.62	46.38
MPNet	15.26	12.30	9.42	8.56	19.35	23.98	29.91	48.06
MACLR (ours)	18.92	16.17	12.62	10.98	25.64	32.16	39.46	58.24
LF-WikiSeeAlso-320K
XR-Linear	6.97	3.43	2.31	3.74	5.02	5.44	5.84	6.87
Astec	5.58	3.35	2.48	3.22	5.43	6.51	7.95	11.76
SiameseXML	9.87	5.22	3.59	5.84	8.57	9.53	10.60	13.04
ZestXML	10.40	8.18	6.49	5.57	11.65	14.52	18.81	33.20
SentBERT	18.85	14.23	11.22	11.16	21.77	26.94	33.78	55.88
MPNet	18.04	13.27	10.44	10.51	19.99	24.62	30.86	52.52
MACLR (ours)	20.49	15.50	12.24	12.34	23.88	29.43	36.76	59.82
In this section, we present additional experimental results for the setting of FS-XMC on LF-Amazon-131K and LF-WikiSeeAlso-320K. Instead of sampling a few-shot subset by the label coverage ratio, we turn to sampling based on the pair ratio. Specifically, suppose a training set  $\mathcal{D}_{\mathrm{train}} = \{(x_i,y_i)\}$  has  $|\mathcal{D}_{\mathrm{train}}|$  positive pairs. Each time we randomly sample a small ratio of  $\delta$  (1% or 5% in our paper) pairs from the total set to constitute the few-shot subset. Each subset has  $\delta |\mathcal{D}_{\mathrm{train}}|$  pairs for fine-tuning. Detailed results of  $\delta = 1\%$
Table 12: Results of FS-XMC where the training subset covers  $5\%$  positive pairs from the whole set.
Method	Precision	Recall
@1	@3	@5	@1	@3	@5	@10	@100
LF-Amazon-131K
XR-Linear	11.20	5.82	3.80	5.98	8.56	9.18	9.80	12.79
Astec	10.71	6.50	4.52	6.12	10.23	11.67	13.35	18.15
SiameseXML	11.88	8.72	5.93	8.50	13.68	15.23	16.80	20.28
ZestXML	12.86	11.28	8.91	7.10	17.62	22.43	28.42	49.41
SentBERT	16.94	13.59	10.52	9.55	21.23	26.55	33.14	51.81
MPNet	17.48	13.58	10.61	9.95	21.38	26.83	33.60	52.31
MACLR (ours)	19.75	16.45	12.87	11.18	25.99	32.70	40.38	59.82
LF-WikiSecAlso-320K
XR-Linear	13.13	6.88	4.70	7.00	9.64	10.54	11.49	14.20
Astec	15.61	8.73	6.23	8.77	13.17	15.02	17.36	24.30
SiameseXML	16.51	9.68	6.96	9.40	14.78	16.97	19.48	25.26
ZestXML	17.68	8.51	6.85	10.63	12.01	15.20	20.08	43.10
SentBERT	20.12	15.01	11.87	12.05	23.01	28.40	35.52	58.41
MPNet	19.88	14.90	11.76	11.85	22.75	27.96	35.03	57.26
MACLR (ours)	21.80	16.61	13.12	13.27	25.74	31.59	39.25	62.13