SlowGuess/NAACL_2022_OCR huggingface dataset:
Si-An Chen $^{1,2}$ , Jie-Jyun Liu $^{2}$ , Tsung-Han Yang $^{2}$ , Hsuan-Tien Lin $^{1}$ , and Chih-Jen Lin $^{1}$
$^{1}$ National Taiwan University  
 $\{\mathrm{d}09922007, \mathrm{htlin}, \mathrm{cjlin}\} @ \mathrm{csie.ntu.edu.tw}$ $^{2}$ ASUS Intelligent Cloud Services  
 $\{\mathrm{eleven1\_liu}, \mathrm{henry1\_yang}\} @ \mathrm{asus.com}$
The power and the potential of deep learning models attract many researchers to design advanced and sophisticated architectures. Nevertheless, the progress is sometimes unreal due to various possible reasons. In this work, through an astonishing example we argue that more efforts should be paid to ensure the progress in developing a new deep learning method. For a highly influential multi-label text classification method XML-CNN, we show that the superior performance claimed in the original paper was mainly due to some unbelievable coincidences. We re-examine XML-CNN and make a re-implementation which reveals some contradictory findings to the claims in the original paper. Our study suggests suitable baselines for multi-label text classification tasks and confirms that the progress on a new architecture cannot be confidently justified without a cautious investigation.
Deep learning has been a popular research topic in NLP due to its superior performance. The intrinsic structure of deep learning allows researchers to enhance the model performance by introducing more complex network architectures. Nevertheless, the increasing complexity brings difficulties to ensure the true architectural progress. For example, Adhikari et al. (2019) have shown that LSTM architectures with appropriate regularization are either competitive or superior to more recent models. As another example, Liu et al. (2021) report that the lack of hyperparameter tuning in an influential work (Mullenbach et al., 2018) makes the progress of subsequent network developments questionable. Complex architectures are more difficult to train, involve more hyperparameters, and are riskier to unintentional implementation. Because new architectures are usually modified from previous ones, a questionable work may make the research progress unclear. Therefore, re-examining or reproducing
influential architectures are now considered important in the community.
In this work, we re-examine XML-CNN (Liu et al., 2017), an influential work in extreme multi-label text classification (XMTC), as a case study to demonstrate the demands of inspecting existing architectures. XML-CNN has been viewed as an essential baseline in subsequent works (Peng et al., 2018; Prabhu et al., 2018; You et al., 2019; Chang et al., 2020; Adhikari et al., 2019) with more than hundreds of citations. XML-CNN roots from Kim-CNN (Kim, 2014), a classical architecture for multi-class text classification. The authors of XML-CNN proposed several modifications from Kim-CNN to accommodate the XMTC task and empirically claim that all modifications bring significant improvements.
Despite XML-CNN's popularity, we identified two serious implementation issues that make the original claims uncertain. First, the authors introduced dynamic max-pooling into XML-CNN, but the implementation is actually far from the intended formulation. Second, a bug in the experiment code caused the dimensions of convolution operations accidentally swapped. The two issues coincidentally make XML-CNN competitive, leading the authors to illusively claim superiority over KimCNN and usefulness of dynamic max-pooling in the original paper (Liu et al., 2017).
Our contribution can be summarized as follows.
- We point out, analyze, and correct the issues in the authors' XML-CNN implementation. Our implementation is made public to help the community build future works on top of the correct implementation
- We re-examine the claims about XML-CNN. Our results demonstrate that the progress from KimCNN to XML-CNN may not be as significant as claimed in Liu et al. (2017), and again confirm that careful attention is needed on ensuring true architectural progress.
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1987 - 2000 July 10-15, 2022 ©2022 Association for Computational Linguistics
- Our investigation suggests that instead of XML-CNN, Kim-CNN or a simpler variant of XML-CNN should be considered as a baseline in XMTC tasks.
The paper is organized as follows: in Section 2, we introduce Kim-CNN, XML-CNN, and their differences. We conduct an investigation on XML-CNN in Section 3. The investigation includes inspection of the authors' code and our analysis on why it coincidentally works. We then conduct a fair and thorough comparison between Kim-CNN and XML-CNN in Section 4. Finally, we conclude this work in Section 5. Supplementary materials and programs used for experiments are available at https://www.csie.ntu.edu.tw/~cjlin/papers/xmlcnn/.
2 XML-CNN: CNN for Multi-Label Text Classification
For multi-label text classification, each instance is an  $n$ -word document that is associated with a subset of  $L$  possible categories. The relationship between the document and the categories can be modeled by a convolutional neural network (CNN), as pioneered for multi-class text classification by Kim-CNN (Kim, 2014). The architecture was later extended to XML-CNN (Liu et al., 2017) for multi-label text classification. Here we introduce the two architectures along with a focus on the key modifications.
2.1 CNN for Text Classification
Kim-CNN (Kim, 2014) is the first work that applies convolutional neural networks in text classification. The architecture is illustrated in Fig. 1a. Kim-CNN preprocesses a document by first encoding the  $i$ -th word to a  $k$ -dimensional embedding vector  $\boldsymbol{x}_i \in \mathbb{R}^k$  (Pennington et al., 2014). We denote an  $n$ -word document by  $\boldsymbol{x}_{1:n}$ , where  $\boldsymbol{x}_{i:j} = [\boldsymbol{x}_i, \dots, \boldsymbol{x}_j]^\top \in \mathbb{R}^{(j - i + 1)\times k}$  represents a sub-sequence from the  $i$ -th to the  $j$ -th word in the document.
A convolutional operation applies a filter  $\boldsymbol{w}_i \in \mathbb{R}^{m \times k}$  to a sub-sequence of  $m$  words to produce a new feature:
$$
c _ {i} = f \left(\boldsymbol {w} _ {i} \cdot \boldsymbol {x} _ {i: i + m - 1} + b _ {i}\right), \tag {1}
$$
where  $f$  is an activation function such as ReLU,  $b_{i}\in \mathbb{R}$  is a bias term and the "·" operator means the sum after component-wise products between two matrices. The filter is applied to all  $m$ -word sub-sequences in the document to form a feature
map  $\pmb{c} = [c_1, \dots, c_{n - m + 1}] \in \mathbb{R}^{n - m + 1}$ . Suppose Kim-CNN uses  $t$  filters and let  $\pmb{c}^{(1)}, \dots, \pmb{c}^{(t)}$  be the corresponding feature maps. A max-pooling layer is then applied to summarize the features as  $\pmb{z} = [\max(\pmb{c}^{(1)}), \dots, \max(\pmb{c}^{(t)})] \in \mathbb{R}^t$ . Lastly, a dropout layer and a fully-connected layer is used to predict a score vector
$$
\boldsymbol {s} = \tilde {\boldsymbol {W}} (\boldsymbol {z} \odot \boldsymbol {r}) + \tilde {\boldsymbol {b}} \in \mathbb {R} ^ {L}, \tag {2}
$$
where  $\odot$  is the element-wise multiplication operator,  $\tilde{W} \in \mathbb{R}^{L \times t}$ ,  $\tilde{\boldsymbol{b}} \in \mathbb{R}^L$  are learnable parameters and each  $r_i$  of  $\boldsymbol{r} \in \mathbb{R}^t$  is a dropout random variable that follows a Bernoulli distribution.
Kim-CNN was originally proposed for multiclass classification based on the cross-entropy loss
$$
- \sum_ {i = 1} ^ {L} y _ {i} \log p _ {i}, \quad \text {w h e r e} p _ {i} = \frac {e ^ {s _ {i}}}{\sum_ {j = 1} ^ {L} e ^ {s _ {j}}} \tag {3}
$$
is the estimated probability of the  $i$ -th class,  $s_i$  is the  $i$ -th element of  $\mathbf{s}$  that denotes the score of the  $i$ -th class, and  $\mathbf{y} \in \{0,1\}^L$  denotes the ground truth of the instance. If the  $i$ -th label is associated with the document, then  $y_i = 1$ ; otherwise,  $y_i = 0$ . By the construction of  $p_i$  in Eq. (3),  $\sum_{i} p_i$  is forced to be 1, which is natural for multi-class classification. For multi-label classification, however, it is not clear whether requiring all  $p_i$ 's to sum to one would be too restrictive, given that there can be multiple  $y_i$ 's with  $y_i = 1$ . Nevertheless, the loss has been considered for some multi-label works (Gong et al., 2014; Ghosh et al., 2015).
2.2 From CNN to XML-CNN
XML-CNN is a pioneering work that extends KimCNN from multi-class text classification to XMTC. The architecture of XML-CNN is illustrated in Fig. 1b. It extends Kim-CNN with three modifications:
- using a label-wise binary cross-entropy loss instead of the cross-entropy loss in Eq. (3),
- adding an additional linear hidden layer with dropout,
- introducing dynamic max-pooling (Chen et al., 2015) to extract multiple features from each CNN filter.
For the first modification, the authors noticed the issue of the cross-entropy loss discussed in Section 2.1. They then allow the model to flexibly
(a) Kim-CNN
Figure 1: Architectures of Kim-CNN and XML-CNN.
(b)XML-CNN
predict multiple positive labels by taking the independent binary cross-entropy loss instead:
$$
- \sum_ {i = 1} ^ {L} \left[ y _ {i} \log \left(\sigma \left(s _ {i}\right)\right) + \left(1 - y _ {i}\right) \log \left(1 - \sigma \left(s _ {i}\right)\right) \right], \tag {4}
$$
where  $\sigma(s) = \frac{1}{1 + e^{-s}}$  is the sigmoid function.
For the second modification, the additional linear layer may help to reduce the number of parameters, allowing the model to be stored in common GPU devices when  $L$  is extremely large. Let  $h$  be the number of elements in the added hidden layer. XML-CNN reduces the number of parameters after the CNN layer from  $t \times L$  in the original Kim-CNN to
$$
t \times h + h \times L \tag {5}
$$
when  $h$  is sufficiently small.
For the third modification, the authors applied dynamic max-pooling (Chen et al., 2015) in XML-CNN to capture multiple features from different parts of the document. In contrast to the traditional max-pooling, which calculates the maximum along the whole sequence, dynamic max-pooling divides the sequence into multiple pools and then collects the maximum values within each pool to get some fine-grained features. Given a filter map  $c \in \mathbb{R}^n$ , the formulation with  $d$  pools is:
$$
D (\boldsymbol {c}) = \left[ \max  \left\{\boldsymbol {c} _ {1: \frac {n}{d}} \right\}, \dots , \max  \left\{\boldsymbol {c} _ {n - \frac {n}{d} + 1: n} \right\} \right] \in \mathbb {R} ^ {d}. \tag {6}
$$
The output becomes
$$
\boldsymbol {z} = \left[ D (\boldsymbol {c} ^ {(1)}), \dots , D (\boldsymbol {c} ^ {(t)}) \right] \in \mathbb {R} ^ {d t}
$$
instead of  $\left[\max (\pmb{c}^{(1)}),\dots ,\max (\pmb{c}^{(t)})\right]\in \mathbb{R}^t$  in Kim-CNN.
2.3 Claims about XML-CNN
Liu et al. (2017) compared their proposed XML-CNN with Kim-CNN by reporting  $\mathrm{P@K}$  on six datasets, as shown in Table 1.  $\mathrm{P@K}$  calculates for each document the percentage of correct predictions (i.e., precision) among the top  $K$  predicted labels and reports the average over all test documents. Table 1 clearly indicates significant improvements from Kim-CNN to XML-CNN on all datasets. To examine the impact of each new component in XML-CNN, the authors further conducted ablation studies to make the following claims.
Eq. (4) is more suitable than Eq. (3) for multi-label classification problems.
- The additional linear layer improves both the performance and the scalability.
- Dynamic max-pooling further improves the performance significantly.
The impressive progress of XML-CNN makes it a standard benchmark for XMTC (e.g., Peng et al., 2018; Prabhu et al., 2018; You et al., 2019; Chang et al., 2020; Adhikari et al., 2019). However, we will show in this study that the progress may not be as significant as the authors claimed. While the first modification is included in our evaluation in Section 4.2, our focus is on the other two modifications, which correspond to the differences between XML-CNN and Kim-CNN-Eq.(4), the multi-label version equipped with the binary cross-entropy loss in Eq. (4). Subsequently, Kim-CNN-Eq.(4) will be shorthanded Kim-CNN for simplicity.
Though we use  $\pmb{c} \in \mathbb{R}^{n - m + 1}$  earlier, we let  $\pmb{c} \in \mathbb{R}^n$  here for easier explanation.
<sup>2</sup>Due to the space limit, we leave their and our NDCG@K results in the supplementary materials.
RCV1	Amazon-670K
P@1	P@3	P@5	P@1	P@3	P@5
Kim-CNN-Eq.(3)	93.54	76.15	52.94	15.19	13.78	12.64
XML-CNN	96.86	81.11	56.07	35.39	31.93	29.32
EUR-Lex	Wiki-30K
P@1	P@3	P@5	P@1	P@3	P@5
Kim-CNN-Eq.(3)	42.84	34.92	29.01	78.93	55.48	45.05
XML-CNN	76.38	62.81	51.41	84.06	73.96	64.11
Amazon-12K	Wiki-500K
P@1	P@3	P@5	P@1	P@3	P@5
Kim-CNN-Eq.(3)	90.31	74.34	58.78	23.38	11.95	8.59
XML-CNN	95.06	79.86	63.91	59.85	39.28	29.81
Table 1: Results reported in Liu et al. (2017), where Kim-CNN-Eq.(3) indicates the setting to optimize Eq. (3) rather than Eq. (4).
3 Investigation into XML-CNN
In this section, we point out a significant gap between the formulations in the XML-CNN paper and the authors' implementations. We first reproduce the results in Liu et al. (2017) to ensure what the authors have done. We then confirm the reported superiority of XML-CNN over Kim-CNN is actually due to some coincidences.
3.1 The Challenges of Reproducing Liu et al. (2017)
The authors have released their implementation of XML-CNN on GitHub.<sup>3</sup> They wrote the code in Lasagne (Dieleman et al., 2015), an outdated deep learning framework. To facilitate a thorough comparison, we implement a PyTorch-based program<sup>4</sup> that is as close to the released Lasagne code as possible. Though their implementation is available, to our surprise, reproducing XML-CNN results on the same datasets is more challenging than expected. We leave details of solving various challenges in Appendix A. In particular, we find that some data sets used in Liu et al. (2017) are no longer available, so similar ones are considered; see data statistics in Table 2.
We choose EUR-Lex for checking the reproducibility due to the following reasons.
- The dataset is publicly available and from Tabel 2 it has the same statistics as in Liu et al. (2017).
- The improvement of XML-CNN is significant as shown in Table 1.
- The size is relatively small but adequate.
The results of the authors and our implementations
(a) Normal CNN that follows Eq. (1) to go along the words.
Figure 2: Two implementations of CNN.
(b) CNN in the public code of Liu et al. (2017) that goes along the embedding dimension.
are respectively shown in the second and the third rows in Table 3. The difference between the two implementation is even smaller than the difference between XML-CNN's paper numbers to its public implementation. This justifies that our results are close enough for reproducing the numbers. We conclude that the author's result on EUR-Lex can be reproduced, though many issues must be addressed in the entire process.
3.2 Problematic Gap Between Implementations and Formulations
Though we can reproduce the results reported in Liu et al. (2017), in checking their programs, we surprisingly found some significant gaps between the implementation and the formulations in their paper. The first one is that their implementation of the convolutional operation is completely different from Eq. (1). We illustrate the two CNNs in Fig. 2. In the authors' implementation, the convolution operation sweeps along the embeddings rather than the words, as shown in Fig. 2b. That is, it seems the authors did not implement what they intended to do.
Another problem is about the dynamic max-
<sup>3</sup>https://github.com/jimmy646/XML-CNN
4The implementation is based on LibMultiLabel (released under the MIT license): https://github.com/ASUS-AICS/LibMultiLabel
Dataset	# training data	# test data	# labels
EUR-Lex	15,449	3,865	3,956
Wiki-30K → Wiki10-31K	12,959 → 14,146	5,992 → 6,616	29,947 → 30,938
Amazon-12K → AmazonCat-13K	490,310 → 1,186,239	152,981 → 306,782	12,277 → 13,330
Amazon-670K	490,449	153,025	670,091
Wiki-500K → Wiki-500K	1,646,302 → 1,779,881	711,542 → 769,421	501,069 → 501,008
Table 2: The datasets used in Liu et al. (2017). Some are no longer available, so similar ones are considered in this work and “ $\rightarrow$ ” indicates the difference.
Implementation	Framework	P@1	P@3	P@5
Results reported in Liu et al. (2017)	76.38	62.81	51.41
Public code by Liu et al. (2017)'s authors	Lasagne	74.28	58.98	48.16
Our code mimicking the above	PyTorch	75.50	60.47	49.38
Table 3: Reproducing results reported in Liu et al. (2017) on EUR-Lex by using the authors' and our implementations.
pooling. The authors set the default pool size to 2 and stride to 1 in their public implementation:
$$
\left[ \max  \left\{\boldsymbol {c} _ {1: 2} \right\}, \max  \left\{\boldsymbol {c} _ {2: 3} \right\}, \dots \max  \left\{\boldsymbol {c} _ {n - 1: n} \right\} \right], \tag {7}
$$
which differs from Eq. (6) in that overlapped pools are used. Further, given that the aim of max-pooling is to extract information from each pool, a size-two pool is unusually small. We do not know why the authors implemented dynamic max-pooling in this form, but we will show that this odd implementation, together with the wrong convolution mentioned earlier, surprisingly works well.
To compare with these unusual settings, we generate another implementation of XML-CNN by following Eq. (1) and Eq. (6). The details of our implementation can be found in Appendix B. According to Liu et al. (2017), this version should be what its authors intend to have. Table 4 shows the results on EUR-Lex by various ways to implement Kim-CNN and XML-CNN. Other settings (e.g., hyperparameters) are kept to be the same as those in the authors' implementation; see also Section 3.1 and supplementary materials. From Table 4, we have the following observations.
- For each category (Kim-CNN or XML-CNN), the last row indicates the setting described in the original papers. If the CNN input is changed to the wrong one, the results of both Kim-CNN and XML-CNN become dramatically worse (rows 1 and 5). On the other hand, if the implementation of dynamic max-pooling follows Eq. (7) rather than Eq. (6), the result of XML-CNN also significantly deteriorates (row 4).
However, if both inappropriate settings for CNN and dynamic max-pooling are applied, the resulting procedure corresponds to the actual implementation by Liu et al. and works surprisingly well (row 3). In contrast, without the help of Eq. (7), Kim-CNN by the inappropriate CNN implementation performs poorly. In such a situation, Kim-CNN's scores are quite similar to the results of Kim-CNN reported in Liu et al. (2017), as shown in Table 1. So we presume that in Liu et al. (2017), Kim-CNN was implemented with the inappropriate CNN setting.
In sum, the implementation seems not what Liu et al. intended to do in their paper. Thus, their conclusions based on the unintentional implementations may be questionable. In particular, in Table 4 Kim-CNN is competitive if an implementation following its original paper (Kim, 2014) is considered.
For better distinction in subsequent discussions, we name the two XML-CNN implementations respectively corresponding to Liu et al.'s paper and public code as follows.
- XML-CNN-paper: XML-CNN following Eq. (1) and Eq. (6).
- XML-CNN-impl: XML-CNN using CNNs sweeping along embeddings and Eq. (7).
3.3 The Two Implementations of XML-CNN: Analysis
We try to explain why XML-CNN-impl can achieve competitive results. For the analysis, we first argue that conceptually, the unusual dynamic max-pooling Eq. (7) is similar to not doing pooling. The reason is because the small pool size  $= 2$  implies that at least half of  $c$  elements are retained. Then we design an experiment to compare the combina
As mentioned in Section 2.3, we note again that KimCNN in all our experiments, unless specified, optimizes the same loss in Eq. (4) as XML-CNN.
Method	CNN sweeping direction	Dynamic Max-pooling	P@1	P@3	P@5	Note
Kim-CNN	embeddings	N/A	45.38	34.02	27.72	Actual implementation of Liu et al. (2017)
Kim-CNN	words	N/A	75.83	61.08	50.19	Procedure described in Kim (2014)
XML-CNN	embeddings	Eq. (7)	75.96	60.56	49.23	XML-CNN-impl: actual implementation of Liu et al. (2017)
XML-CNN	words	Eq. (7)	58.09	45.19	37.06
XML-CNN	embeddings	Eq. (6)	63.03	48.31	39.32
XML-CNN	words	Eq. (6)	75.73	61.82	50.82	XML-CNN-paper: procedure described in Liu et al. (2017)
tions of the following settings.
- CNN sweeping direction: embeddings or words
- Pooling implementation: standard max-pooing or no pooling
Table 5 shows the  $\mathrm{P@1}$  scores on predicting the test set of EUR-Lex. We have the following observations.
- Results of the (embeddings, no pooling) setting are similar to those of the (embeddings, Eq. (7)) setting in Table 4. This confirms our earlier argument that Eq. (7) is close to no pooling.
- If CNN sweeps along the words, the standard max-pooling is significantly better than no pooling. A possible explanation is that when CNN sweeps along the words, some sub-sequence of words are shown to be more important than others. Then the standard max-pooling is helpful to identify them. This situation is similar to that in image classification, where max-pooling is effective to extract "sharp" features (Springenberg et al., 2015).
- If CNN sweeps along the embeddings, an opposite situation occurs. No pooling is much better than standard max-pooling. Because all the embeddings can be considered equally useful, the resulting features after convolutional operation have similar importance. For such "smooth" features, it is known in image classification that average pooling or no pooling are recommend (Springenberg et al., 2015). In other words, standard max-pooling can extract little information in such a case and may lead to worse performance.
In Section 3.4, we will present results to further
Table 4: Results of different implementations of the convolutional layer and dynamic max-pooling for Kim-CNN and XML-CNN. The standard CNN input should be  $k \times n$ . Other settings are the same as those for the last two rows of Table 3.
max{c}	No pooling
words	74.67	53.61
embeddings	58.14	76.48
Table 5: P@1 of combinations of CNN sweeping directions and pooling methods for implementing XML-CNN. Note that the first column differs from the first two rows in Table 4 because we now have a hidden layer.
support the above analysis.
3.4 The Two Implementations of XML-CNN: Performance Comparison
Table 6 shows a comprehensive comparison between XML-CNN-impl and XML-CNN-paper on more datasets. In contrast to Table 4 where we follow the hyperparameters used in Liu et al. (2017), we tune the hyperparameters for both methods in Table 6. We observe that XML-CNN-paper outperforms XML-CNN-impl on EUR-Lex and Wiki10-31K. Following the discussion in Section 3.3, the reason may be that XML-CNN-impl lacks the ability to learn position-agnostic features when documents are long. Note that for EUR-Lex and Wiki10-31K, the documents are truncated to 500 words because of the long document length. On the other hand, XML-CNN-impl works competitively on AmazonCat-13K and Amazon-670K. Though the documents are also truncated to 500 words when needed, the average document lengths of these two sets are less than 250.
In sum, XML-CNN-paper should be preferable
<sup>6</sup>Details of experimental settings and hyperparameter search are in Section 4.1 and Appendix C.
because of the more reasonable architecture and better performance on long documents. Moreover, XML-CNN can deal with variable sentence lengths, while XML-CNN-impl cannot because the network architecture depends on the sentence length. We consider XML-CNN-paper as the setting for XMLCNN in subsequent experiments.
4 The True Performance of XML-CNN