SlowGuess/NAACL_2022_OCR huggingface dataset:
After showing the gap between the implementation and the formulation in Liu et al. (2017), the true performance of XML-CNN should be re-examined. In this section, we conduct a comprehensive ablation study for XML-CNN to investigate the claims in the original paper. We then investigate more deeply on dynamic max-pooling to determine its usefulness for XMTC tasks. The results bring us a similarly competitive but simpler baseline for XMTC tasks.
We consider a random 80/20 split of the training data to generate a training subset and a validation subset for hyperparameter selection. We follow Liu et al. (2017) to truncate the documents to 500 words, represent each word as a 300-dimensional GloVe word embedding (Pennington et al., 2014) and pad the sequences in each batch when needed. The word embeddings are considered as trainable parameters during training. We have carefully conducted hyperparameter selection. The procedure and other details are in Appendix C.
We follow Liu et al. (2017) to train the models with 50 epochs on the whole training set after hyperparameter tuning and then evaluate the test set. We evaluate each method on three datasets: EUR-Lex, Wiki10-31K, and AmazonCat-13K. In Section 4.2, we conduct the ablation study by including one larger dataset: Amazon-670K. All of them are in English. The datasets are obtained from the repository of You et al. (2019) and we follow Liu et al. (2017) to reduce the vocabulary set.
4.2 Ablation Studies of XML-CNN
To understand how each component introduced in Liu et al. (2017) really works, we conduct an ablation study as what the authors have done. Specifically, the effects of using Eq. (4) as the loss, adding a linear hidden layer, and introducing dynamic max-pooling are checked. By the results shown in Ta
ble 7, we can re-examine what the authors have claimed in Liu et al. (2017). To begin, from the first and the second rows, the loss function Eq. (4) indeed improves the scores  $2\% -6\%$  on each dataset. The results validate the claim that Eq. (4) is more suitable for multi-label tasks than Eq. (3).
Next, while adding a hidden layer is claimed to be beneficial in Liu et al. (2017), our results show that the hidden layer is slightly harmful on EUR-Lex and Wiki10-31K when the standard max-pooling is applied; see rows 2 and 4 in Table 7. It works when dynamic max-pooling is employed, but the improvements are not significant. The authors also claimed that the hidden layer could reduce the number of parameters; see Eq. (5). This claim is true when  $h$  is relatively small compared with the number of convolutional features. However, we noticed that larger  $h$ 's such as 512 and 1,024 are always preferable after hyperparameter tuning. In these cases, the number of parameters may not be reduced.
Lastly, we discuss the effect of dynamic max-pooling. In the situation of not adding a hidden layer, dynamic max-pooling slightly improves upon the standard max-pooling on most but not all datasets. If a hidden layer is included in the architecture, dynamic max-pooling also gives moderate improvements. However, dynamic max-pooling may require more network parameters due to multiple pools. To check its usefulness, we investigate more in Section 4.3.
4.3 Further Investigation on Dynamic Max-Pooling
We conduct two experiments to understand whether dynamic max-pooling always benefits XML-CNN. The first experiment is a comparison between different numbers of pools. The second experiment compares dynamic max-pooling with standard max-pooling by using a similar total number of parameters. The experiment results and more discussions are in Appendix D.1 and Appendix D.2. The investigations tell us:
- Using too many pools may deteriorate the performance.
- Under similar total numbers of parameters, standard max pooling is more preferable than dynamic max-pooling.
For dynamic max-pooling, the output size  $t \times d$  is proportional to the pool size  $d$ . For standard max-pooling ( $d = 1$ ), we can enlarge the number of filters  $t$  to have a similar number of parameters; see details in Appendix D.2.
See details in Appendix A.4.
method	EUR-Lex	Wiki10-31K
P@1	P@3	P@5	P@1	P@3	P@5
XML-CNN-impl	77.39	62.32	51.28	83.98	70.03	60.18
XML-CNN-paper	78.94	65.77	54.15	84.70	71.80	61.03
AmazonCat-13K	Amazon-670K
P@1	P@3	P@5	P@1	P@3	P@5
XML-CNN-impl	94.73	80.29	64.97	38.02	34.13	31.20
XML-CNN-paper	94.78	80.03	64.52	35.69	31.89	29.08
Table 6: Comparison between the two XML-CNN implementations. XML-CNN-impl is the actual implementation of Liu et al. (2017). XML-CNN-paper is our implementation that follows Liu et al. (2017).
loss function	hidden layer	max-pooling	EUR-Lex	Wiki10-31K	Note
P@1	P@3	P@5	P@1	P@3	P@5
Eq. (3)	N	standard	72.78	59.84	49.94	80.70	64.83	55.43	Kim-CNN (Kim, 2014)
Eq. (4)	N	standard	80.93	66.38	55.34	82.78	68.07	57.63
Eq. (4)	N	dynamic	77.88	64.58	53.38	83.37	70.64	60.16
Eq. (4)	Y	standard	76.56	62.92	51.84	81.73	68.82	58.65
Eq. (4)	Y	dynamic	78.94	65.77	54.15	84.70	71.80	61.03	XML-CNN (Liu et al., 2017)
AmazonCat-13K	Amazon-670K
P@1	P@3	P@5	P@1	P@3	P@5
Eq. (3)	N	standard	92.85	76.90	61.76	27.23	24.65	22.70	Kim-CNN (Kim, 2014)
Eq. (4)	N	standard	93.41	78.11	62.95	33.38	29.99	27.47
Eq. (4)	N	dynamic	93.65	78.56	63.41	34.61	30.91	28.25
Eq. (4)	Y	standard	94.73	79.64	63.94	33.86	30.27	27.69
Eq. (4)	Y	dynamic	94.78	80.03	64.52	35.69	31.89	29.08	XML-CNN (Liu et al., 2017)
Table 7: An ablation study of XML-CNN. For max-pooling, "standard" means the standard way of using the single maximal value, while "dynamic" means to use Eq. (6).
4.4 What We May Claim about XML-CNN
We conclude our findings in this section as follows:
Eq. (4) is indeed more suitable for multi-label tasks than Eq. (3).
- For the hidden layer, there is a minor tradeoff between the number of parameters and the performance. A negative way to interpret this is that introducing the hidden layer does not always improve the performance. However, a positive interpretation is that with a slight performance loss, a hidden layer can effective reduce the number of parameters when the output size of the pooling operation is large.
- Dynamic max-pooling is not as beneficial as increasing the number of convolutional filters.
After our careful re-investigation, our suggestion to future studies of XMTC is that instead of using XML-CNN as a baseline, the following simpler settings can be considered.
- If there is no memory concern, Kim-CNN is suitable for its similar performance to XML-CNN.
- Otherwise, a simplified version of XML-CNN without dynamic max-pooling, namely Kim
CNN with an additional hidden layer, is sufficiently strong and space-efficient as the baseline.
This work aims to highlight the importance of validating existing works. From the investigation of XML-CNN, we learned that there are many pitfalls when developing new architectures. We correct the issues in the authors' implementation, carefully re-examine the claims about XML-CNN and recommend suitable baselines for future studies. Though not proposing a new method, we hope this work encourages the community to reproduce and re-examine influential works. This may help the community build future works on top of correct materials.
This work is partially supported by ASUS Intelligence Cloud Services and the Ministry of Science and Technology of Taiwan via the grants MOST 110-2628-E-002-013. We also thank the National Center for High-performance Computing (NCHC) of National Applied Research Laborato
ries (NARLabs) in Taiwan for providing computational resources.
Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. Rethinking complex neural network architectures for document classification. In NAACL-HLT.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In KDD.
Wei-Cheng Chang, Hsiang-Fu Yu, Kai Zhong, Yiming Yang, and Inderjit S. Dhillon. 2020. Taming pretrained transformers for extreme multi-label text classification. In KDD.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multipooling convolutional neural networks. In ACL.
Sander Dieleman, Jan Schlüter, Colin Raffel, Eben Olson, Søren Kaae Sønderby, Daniel Nouri, et al. 2015. Lasagne: First release.
Sayan Ghosh, Eugene Laksana, Stefan Scherer, and Louis-Philippe Morency. 2015. A multi-label convolutional neural network approach to cross-domain action unit detection. In ACII.
Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. 2014. Deep convolutional ranking for multilabel image annotation. In ICLR.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP.
Jie-Jyun Liu, Tsung-Han Yang, Si-An Chen, and Chih-Jen Lin. 2021. Parameter selection: Why we should pay more attention to it. In ACL.
Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. 2017. Deep learning for extreme multi-label text classification. In SIGIR.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable prediction of medical codes from clinical text. In *NAACL*.
Hao Peng, Jianxin Li, Yu He, Yaopeng Liu, Mengjiao Bao, Lihong Wang, Yangqiu Song, and Qiang Yang. 2018. Large-scale hierarchical text classification with recursively regularized deep graph-CNN. In WWW.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. 2018. Parabel: Partitioned label trees for extreme classification with application to dynamic search advertising. In WWW.
Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller. 2015. Striving for simplicity: The all convolutional net. In ICLR Workshop.
Ronghui You, Zihan Zhang, Ziye Wang, Suyang Dai, Hiroshi Mamitsuka, and Shanfeng Zhu. 2019. AttentionXML: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In NeurIPS.
A The Challenges of Reproducing Liu et al. (2017)
The authors evaluated XML-CNN on six datasets from the Extreme Multi-Label Repository.9 Unfortunately, some of the datasets have been changed on the repository to different number of data/labels (with a similar name). Furthermore, for some of them, the repository does not provide the raw-text documents, making it hard to preprocess the documents to the embedding needed for XMLCNN. Fortunately, we find the repository of AttentionXML.10 (You et al., 2019) where two datasets (EUR-Lex and Amazon-670K) of raw text match the statistics of the datasets in the XML-CNN work. We then choose the smaller EUR-Lex as the first attempt to reproduce XML-CNN faithfully.
The released code includes only the training but not the validation/evaluation procedure. In the original paper (Liu et al., 2017), it is mentioned that  $25\%$  of training data is reserved as the validation set for hyperparameter selection. However, the details such as how to generate the validation set, which metric was considered in validation, and whether they re-trained the model with the whole training set are not specified in the paper. Therefore, we cannot exactly replicate the results in Table 1. We ran the released code and observed that with only  $75\%$  of training data, the results are always worse than ones reported in Liu et al. (2017). Thus, we presume that in Liu et al. (2017), the authors reported the results of models trained on the whole training set by using the selected hyperparameters.
A.3 Lasagne vs PyTorch
As mentioned in Sec 3.1, we implement a PyTorch-based program that is as close to the released Lasagne code as possible. We fix the common hyperparameters such as the number of filters and the dropout rate as ones provided in the authors' implementation. Then we train the whole training set and follow their setting to report the test scores at the 50-th epoch. The results of their and our implementations are respectively shown in the
second and the third rows in Table 8. The minor differences between the scores are possible because ensuring everything to be the same from the beginning to the end is tremendously difficult. For example, optimizers implemented in Lasagne and PyTorch are not entirely the same. What we have confirmed is that for the network part, under the same input, the two implementations generate exactly the same output and loss values. Therefore, we conclude that our implementation can be used together with theirs in checking the reproducibility. However, both are still worse than the results of Liu et al. (2017) in the first row of Table 8. This fact encouraged us to investigate more on the data processing step done in Liu et al. (2017).
A.4 Vocabulary Set
In Liu et al. (2017), the authors compared XML-CNN with some linear-based algorithms, which use bag-of-word (BOW) features to deal with document inputs. The BOW features usually only consider vocabularies with higher frequency to reduce the dimensionality. To fairly compare XML-CNN with linear models, the authors removed vocabularies which are not used in the BOW features. While the Extreme Multi-Label Repository provides BOW features of EUR-Lex and we assume that they were used in Liu et al. (2017), the vocabulary set of the BOW features is not accessible now. Fortunately, we obtained from the repository owner the vocabulary set used in generating their BOW features. The results of XML-CNN with the reduced vocabulary set are shown in the fourth and the fifth rows in Table 8. By using the reduced vocabulary set, the results of both Lasagne and PyTorch implementations are improved to be closer to the ones in Table 1. As a result, we conclude that the authors' result on EUR-Lex can be reproduced, though many issues must be addressed in the entire process.
B Implementation Details of Section 3.2
To implement Eq. (6), we follow Adhikari et al. (2019) to use AdaptiveMaxPool1d<sup>11</sup> and consider 4 pools, i.e.,  $d = 4$ . Notice that Eq. (6) does not handle the situation where the sequence length is not divisible by  $d$ . Adaptive max-pooling solve the problem by allowing some overlapping between
<sup>9</sup>http://manikvarma.org/downloads/XC/XMLRepository.html
10The repository is free for non-commercial use. https: //github.com/yourh/AttentionXML
11https://pytorch.org/docs/stable/ generated/torch.nn.AdaptiveMaxPoolld. html
Implementation	Framework	Vocabulary set	P@1	P@3	P@5
Results reported in Liu et al. (2017)	76.38	62.81	51.41
Public code by Liu et al. (2017)'s authors	Lasagne	All	72.08	56.32	46.20
Our code mimicking the above	PyTorch	All	74.05	59.61	48.24
Public code by Liu et al. (2017)'s authors	Lasagne	Reduced	74.28	58.98	48.16
Our code mimicking the above	PyTorch	Reduced	75.50	60.47	49.38
Table 8: Reproducing results reported in Liu et al. (2017) on EUR-Lex by using the authors' and our implementations.
parameter	range
learning rate	{0.0001, 0.0003, 0.001, 0.003, 0.01}
# of filters	{96, 192, 384, 768}
embedding dropout	{0.2, 0.4, 0.6, 0.8}
# pools	{2, 8}
hidden layer	{256, 512, 1024}
hidden layer dropout	{0.2, 0.4, 0.6, 0.8}
Table 9: The range of hyperparameters used for selection.
pools. Therefore, it generates exactly  $d$  outputs from documents with varying lengths.
C Details of Experimental Setup
From the hyperparameter ranges listed in Table 9, we apply Optuna (Akiba et al., 2019) to select the best hyperparameters from 48 random trials. In the validation procedure, we optimize P@1 for EUR-Lex, AmazonCat-13K, and Amazon-670K, and P@3 for Wiki10-31K. We do not optimize P@1 for Wiki10-31K because there is a dominant class that is associated with about  $80\%$  of data. Each trial is stopped if the validation metric does not improve for 10 epochs or when it reaches 50 epochs.
In the original papers of Kim-CNN and XML-CNN, both described the use of filters with different window sizes in the convolutional layer. In Kim (2014) and Liu et al. (2017), filter sizes 3, 4, 5 and 2, 4, 8 are respectively used. However, as shown in Table 10, using multiple filter sizes does not have a significant benefit compared with using a fixed filter size 8. Furthermore, among single filter-size settings, the filter size 8 is generally competitive, so we use it in our ensuing investigation.
The experiments are conducted on Azure with an Nvidia Tesla V100 GPU, taking  $< 1$ ,  $< 1$ , 6, 20 GPU hours for one trial on EUR-Lex, Wiki10-31K, AmazonCat-13K, and Amazon-670K respectively.
D Further Investigation on Dynamic MaxPooling
D.1 Effect of the Number of Pools
In dynamic max-pooling, a crucial hyperparameter is the number of pools  $d$ . Nevertheless, in the public code of the XML-CNN work (Liu et al., 2017), due to the unusual setting in Eq. (7),  $d$  is not a fixed number but depends on the document length. Consequently, discussion about the number of pools is lacking in the original work.
In Table 11, we conduct a comparison by using  $d \in \{1,2,8,32,64\}$ . On all datasets,  $d = 2$  and  $d = 8$  have the best performance. Increasing the number of pools to more than 8 not only leads to worse results on some problems, but also costs more memory and training time. Our results indicate that while the goal of dynamic max-pooling is to extract multiple features from each CNN filter, using too many pools may deteriorate the performance instead.
D.2 Investigation on Dynamic Max-Pooling by Fixing the Number of Parameters
As discussed in Sec 4.2, we noticed that the number of parameters in XML-CNN increases along with the number of pools in dynamic max-pooling. Assume the number of filters is  $t$  and the number of pools is  $d$ . In XML-CNN, the total number of filters after the pooling layer is  $t \times d$ , while Kim-CNN still only has  $t$  filters. It is unclear whether the improvement of dynamic max-pooling is caused by the richer information from multiple pools or simply the larger number of parameters. We investigate this issue by comparing XML-CNN with different numbers of pools but ensuring the similar number of parameters. From Table 12, we observe that XML-CNN with 1 pool (i.e., without dynamic max-pooling) outperforms XML-CNN with 2 or 8 pools. The result reveals that the architectural modification of dynamic max-pooling may not be that useful.
Though increasing the number of filter also introduced more parameters in CNN, the number is negligible compared to the number of parameters in
method	filter sizes	EUR-Lex	Wiki10-31K	AmazonCat-13K
P@1	P@3	P@5	P@1	P@3	P@5	P@1	P@3	P@5
Kim-CNN	[2]	78.45	63.41	52.46	83.30	69.34	59.17	93.48	78.21	63.03
[4]	79.04	64.23	52.66	83.19	68.78	58.26	93.66	78.56	63.35
[8]	80.23	66.12	54.48	83.07	69.98	59.96	93.75	78.97	63.93
[2, 4, 8]	79.90	66.43	54.86	82.65	68.05	56.93	93.51	78.14	62.93
XML-CNN	[2]	75.83	62.35	51.67	82.91	69.20	58.97	94.59	79.80	64.32
[4]	77.70	62.97	52.27	82.84	70.20	59.73	94.87	80.27	64.79
[8]	77.98	65.11	53.90	82.68	69.40	59.43	94.55	79.72	64.16
[2, 4, 8]	78.37	64.78	53.61	80.47	68.23	58.24	94.89	80.02	64.25
Table 10: P@K results of comparison between fixed filter size and multiple filter sizes.
# pools	EUR-Lex	Wiki10-31K	AmazonCat-13K
P@1	P@3	P@5	P@1	P@3	P@5	P@1	P@3	P@5
d = 1	76.40	62.78	51.88	80.89	67.89	58.17	94.73	79.64	63.95
d = 2	77.98	65.11	53.90	82.68	69.40	59.43	94.55	79.72	64.16
d = 8	76.79	63.28	52.10	84.19	71.55	61.14	94.79	80.04	64.49
d = 32	66.57	52.51	42.56	82.94	69.20	59.20	94.46	79.45	63.78
d = 64	68.28	54.14	44.19	83.01	69.91	59.13	94.29	79.00	63.27
Table 11: Effect of number of pools in dynamic max-pooling
the output layer, where the output size  $L$  is usually extremely large.
E NDCG results
This section shows NDCG@K results reported by Liu et al. (2017) and in our experiments. Table 13 shows NDCG@K results reported by Liu et al. (2017). Table 14 shows NDCG@K results corresponding to Table 4. Table 15 shows NDCG@K results of our ablation study (Table 7). The observations from NDCG@K results are similar to those from P@K results.
# of filters	# of pools	EUR-Lex	Wiki10-31K	AmazonCat-13K
P@1	P@3	P@5	P@1	P@3	P@5	P@1	P@3	P@5
256	1	76.66	63.70	52.64	83.07	69.35	58.89	94.52	79.66	64.23
128	2	77.36	62.93	51.79	83.62	68.92	58.35	94.34	79.51	64.14
1024	1	78.37	65.65	54.42	83.42	70.66	60.50	94.90	80.29	64.79
128	8	75.91	62.12	51.33	84.11	69.94	59.22	94.30	79.50	64.13
Table 12: A comparison between different settings of XML-CNN with the same number of parameters.
RCV1	Amazon-670K
N@1	N@3	N@5	N@1	N@3	N@5
Kim-CNN-Eq.(3)	93.54	88.2	87.26	15.19	14.6	14.12
XML-CNN	96.88	92.63	92.22	35.39	33.74	32.64
EUR-Lex	Wiki-30K
N@1	N@3	N@5	N@1	N@3	N@5
Kim-CNN-Eq.(3)	42.84	36.95	33.83	78.93	60.52	51.96
XML-CNN	76.38	66.28	60.32	84.06	76.35	68.94
Amazon-12K	Wiki-500K
N@1	N@3	N@5	N@1	N@3	N@5
Kim-CNN-Eq.(3)	90.31	83.87	81.21	23.38	15.45	13.64
XML-CNN	95.06	89.48	87.06	59.85	48.67	46.12
Table 13: NDCG@K results reported in Liu et al. (2017), where Kim-CNN-Eq.(3) indicates the setting to optimize Eq. (3) rather than Eq. (4).
Method	CNN sweeping direction	Dynamic Max-pooling	N@1	N@3	N@5	Note
Kim-CNN	embeddings	N/A	45.38	36.72	33.04	Actual implementation of Liu et al. (2017)
Kim-CNN	words	N/A	75.83	64.75	58.93	Procedure described in Kim (2014)
XML-CNN	embeddings	Eq. (7)	75.96	64.31	58.20	XML-CNN-impl: actual implementation of Liu et al. (2017)
XML-CNN	words	Eq. (7)	58.09	48.30	43.81
XML-CNN	embeddings	Eq. (6)	63.03	51.92	46.88
XML-CNN	words	Eq. (6)	75.73	65.31	59.54	XML-CNN-paper: procedure described in Liu et al. (2017)
Table 14: NDCG results of different implementations of the convolutional layer and dynamic max-pooling for Kim-CNN and XML-CNN.
loss function	hidden layer	max-pooling	EUR-Lex	Wiki10-31K	Note
N@1	N@3	N@5	N@1	N@3	N@5
Eq. (3)	N	standard	72.78	63.12	58.03	80.70	68.40	60.95	Kim-CNN (Kim, 2014)
Eq. (4)	N	standard	80.93	70.07	64.42	82.78	71.48	63.39
Eq. (4)	N	dynamic	77.88	67.93	62.13	83.37	73.61	65.61
Eq. (4)	Y	standard	76.56	66.40	60.60	81.73	71.81	64.01
Eq. (4)	Y	dynamic	78.94	69.11	63.09	84.70	74.84	66.61	XML-CNN (Liu et al., 2017)
AmazonCat-13K	Amazon-670K
N@1	N@3	N@5	N@1	N@3	N@5
Eq. (3)	N	standard	92.85	85.90	83.50	27.23	26.02	25.20	Kim-CNN (Kim, 2014)
Eq. (4)	N	standard	93.41	87.03	84.77	33.38	31.70	30.60
Eq. (4)	N	dynamic	93.65	87.41	85.22	34.61	32.69	31.49
Eq. (4)	Y	standard	94.73	88.68	86.20	33.86	32.03	30.88
Eq. (4)	Y	dynamic	94.78	88.99	86.72	35.69	33.72	32.45	XML-CNN (Liu et al., 2017)
Table 15: An ablation study of XML-CNN evaluated by NDCG@K. For max-pooling, "standard" means the standard way of using the single maximal value, while "dynamic" means to use Eq. (6).