SlowGuess/ACL_2020_OCR huggingface dataset:
Boli Chen, Xin Huang, Lin Xiao, Liping Jing  
Beijing Key Lab of Traffic Data Analysis and Mining  
Beijing Jiaotong University, Beijing, China  
{18120345, 18120367, 17112079, lpjing}@bjtu.edu.cn
Although deep neural networks are effective at extracting high-level features, classification methods usually encode an input into a vector representation via simple feature aggregation operations (e.g. pooling). Such operations limit the performance. For instance, a multi-label document may contain several concepts. In this case, one vector can not sufficiently capture its salient and discriminative content. Thus, we propose Hyperbolic Capsule Networks (HYPERCAPS) for Multi-Label Classification (MLC), which have two merits. First, hyperbolic capsules are designed to capture fine-grained document information for each label, which has the ability to characterize complicated structures among labels and documents. Second, Hyperbolic Dynamic Routing (HDR) is introduced to aggregate hyperbolic capsules in a label-aware manner, so that the label-level discriminative information can be preserved along the depth of neural networks. To efficiently handle large-scale MLC datasets, we additionally present a new routing method to adaptively adjust the capsule number during routing. Extensive experiments are conducted on four benchmark datasets. Compared with the state-of-the-art methods, HYPERCAPS significantly improves the performance of MLC especially on tail labels.
The main difference between Multi-Class Classification (MCC) and Multi-Label Classification (MLC) is that datasets in MCC have only served mutually exclusive classes, while datasets in MLC contain much more correlated labels. MLC allows label co-occurrence in one document, which indicates that the labels are not disjointed. In addition, a large fraction of the labels are the infrequently occurring tail labels (Bhatia et al., 2015), which is also referred as the power-law label distribution.
Figure 1: The power-law label distribution of EURLEX57K with Y-axis on log-scale. Division is based on average number of training instances.
Figure 1 illustrates the label distribution of EURLEX57K (Chalkidis et al., 2019). A multi-label document usually has serval head and tail labels, and hence contain several concepts about both its head and tail labels simultaneously.
Recent works for text classification, such as CNN-KIM (Kim, 2014) and FASTTEXT (Joulin et al., 2017), focus on encoding a document into a fixed-length vector as the distributed document representation (Le and Mikolov, 2014). These encoding based deep learning methods use simple operations (e.g. pooling) to aggregate features extracted by neural networks and construct the document vector representation. A Fully-Connected (FC) layer is usually applied upon the document vector to predict the probability of each label. And each row in its weight matrix can be interpreted as a label vector representation (Du et al., 2019b). In this way, the label probability can be predicted by computing the dot product between label and document vectors, which is proportional to the scalar projection of the label vector onto the document vector as shown in Figure 2. For example, label "movie" should have the largest scalar projection onto a document about "movie". However, even
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3115-3124 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 2: Illustration of the FC layer in the encoding based methods.
the learned label representation of "music" can be distinguished from "movie", it may also have a large scalar projection onto the document.
Moreover, multi-label documents always contain several concepts about multiple labels, such as a document about "sport movie". Whereas the document vector representation is identical to all the labels, and training instances for tail labels are inadequate compared to head labels. The imbalance between head and tail labels makes it hard for the FC layer to make prediction, especially on tail labels. In this case, one vector can not sufficiently capture its salient and discriminative content. Therefore, the performance of constructing the document vector representation via simple aggregation operations is limited for MLC.
Capsule networks (Sabour et al., 2017; Yang et al., 2018a) has recently proposed to use dynamic routing in place of pooling and achieved better performance for classification tasks. In fact, capsules are fine-grained features compared to the distributed document representation, and dynamic routing is a label-aware feature aggregation procedure. (Zhao et al., 2019) improves the scalability of capsule networks for MLC. However, they only use CNN to construct capsules, which capture local contextual information (Wang et al., 2016). Effectively learning the document information about multiple labels is crucial for MLC. Thus we propose to connect CNN and RNN in parallel to capture both local and global contextual information, which would be complementary to each other. Nevertheless, Euclidean capsules necessitate designing a non-linear squashing function.
Inspired by the hyperbolic representation learning methods which demonstrate that the hyper
bolic space has more representation capacity than the Euclidean space (Nickel and Kiela, 2017; Ganea et al., 2018a), Hyperbolic Capsule Networks (HYPERCAPS) is proposed. Capsules are constrained in the hyperbolic space which does not require the squashing function. Hyperbolic Dynamic Routing (HDR) is introduced to aggregate hyperbolic capsules in a label-aware manner. Moreover, in order to fit the large label set of MLC and improve the scalability of HYPERCAPS, adaptive routing is presented to adjust the number of capsules participated in the routing procedure.
The main contributions of our work are therefore summarized as follows:
- We propose to connect CNN and RNN in parallel to simultaneously extract local and global contextual information, which would be complementary to each other.
- HYPERCAPS with HDR are formulated to aggregate features in a label-aware manner, and hyperbolic capsules benefits from the representation capacity of the hyperbolic space.
- Adaptive routing is furthermore presented to improve the scalability of HYPERCAPS and fit the large label set of MLC.
- Extensive experiments on four benchmark MLC datasets demonstrate the effectiveness of HYPERCAPS, especially on tail labels.
In order to make neural networks work in the hyperbolic space, formalism of the Möbius gyrovector space is adopted (Ganea et al., 2018b).
An  $n$ -dimensional Poincaré ball  $\mathcal{B}^n$  is a Riemannian manifold defined as  $\mathcal{B}^n = \{\pmb{x} \in \mathbb{R}^n \mid \| \pmb{x} \| < 1\}$ , with its tangent space around  $\pmb{p} \in \mathcal{B}^n$  denoted as  $T_p \mathcal{B}^n$  and the conformal factor as  $\lambda_p := \frac{2}{1 - \|p\|^2}$ . The exponential map  $\exp_p : T_p \mathcal{B}^n \to \mathcal{B}^n$  for  $\pmb{w} \in T_p \mathcal{B}^n \setminus \{\pmb{0}\}$  is consequently defined as
$$
\exp_ {\boldsymbol {p}} (\boldsymbol {w}) = \boldsymbol {p} \oplus \left(\tanh  \left(\frac {\lambda_ {\boldsymbol {p}}}{2} \| \boldsymbol {w} \|\right) \frac {\boldsymbol {w}}{\| \boldsymbol {w} \|}\right). \tag {1}
$$
To work with hyperbolic capsules, Möbius operations in the Poincaré ball also need to be formulated.
Möbius addition for  $\pmb{u},\pmb{v}\in \mathcal{B}^n$  is defined as
$$
\boldsymbol {u} \oplus \boldsymbol {v} = \frac {(1 + 2 \langle \boldsymbol {u} , \boldsymbol {v} \rangle + \| \boldsymbol {v} \| ^ {2}) \boldsymbol {u} + (1 - \| \boldsymbol {u} \| ^ {2}) \boldsymbol {v}}{1 + 2 \langle \boldsymbol {u} , \boldsymbol {v} \rangle + \| \boldsymbol {u} \| ^ {2} \| \boldsymbol {v} \| ^ {2}}, \tag {2}
$$
where  $\langle \cdot ,\cdot \rangle$  denotes the Euclidean inner product.
Thus Möbius summation can be formulated as
$$
\mathop {\mathbf {m}} _ {i = m} ^ {n} \boldsymbol {p} _ {i} = \boldsymbol {p} _ {m} \oplus \dots \oplus \boldsymbol {p} _ {n}, p _ {i} \in \mathcal {B} ^ {n}. \tag {3}
$$
Mobius scalar multiplication for  $k\in \mathbb{R}$  and  $\pmb {p}\in \mathcal{B}^n\setminus \{\mathbf{0}\}$  is defined as
$$
k \otimes \boldsymbol {p} = \tanh  (k \tanh  ^ {- 1} (\| \boldsymbol {p} \|)) \frac {\boldsymbol {p}}{\| \boldsymbol {p} \|}. \tag {4}
$$
And  $k\otimes \pmb {p} = \mathbf{0}$  when  $\pmb {p} = \mathbf{0}\in \mathcal{B}^n$
The definition of Möbius matrix-vector multiplication for  $M \in \mathbb{R}^{m \times n}$  and  $\pmb{p} \in \mathcal{B}^n$  when  $Mp \neq 0$  is as follows
$$
M \otimes \boldsymbol {p} = \tanh  (\frac {\| M \boldsymbol {p} \|}{\| \boldsymbol {p} \|} \tanh  ^ {- 1} (\| \boldsymbol {p} \|)) \frac {M \boldsymbol {p}}{\| M \boldsymbol {p} \|}. \tag {5}
$$
And  $M\otimes p = 0$  when  $Mp = 0$
HDR is developed based on these operations.
3 Local and Global Hyperbolic Capsules
Neural networks are generally used as effective feature extractors for text classification. Kernels of CNN can be used to capture local n-gram contextual information at different positions of a text sequence, while hidden states of RNN can represent global long-term dependencies of the text (Wang et al., 2016). Hence, we propose to obtain the combination of local and global hyperbolic capsules by connecting CNN and RNN in parallel, which would be complementary to each other.
Given a text sequence of a document with  $T$  word tokens  $\pmb{x} = [x_{1},\dots,x_{T}]$ , pre-trained  $w$ -dimensional word embeddings (e.g. GLOVE (Pennington et al., 2014)) are used to compose word vector representations  $\pmb{E} = [e_1,\dots,e_T] \in \mathbb{R}^{T \times w}$  upon which CNN and RNN connected in parallel are used to construct local and global hyperbolic capsules in the Poincaré ball. Figure 3 illustrates the framework for HYPERCAPS.
3.1 Local Hyperbolic Capsule Layer
N-gram kernels  $\pmb{K} \in \mathbb{R}^{k \times w}$  with different window size  $k$  are applied on the local region of the word representations  $\pmb{E}_{t:t+k-1} \in \mathbb{R}^{k \times w}$  to construct the local features as
$$
l _ {t} = \varphi (\boldsymbol {K} \circ \boldsymbol {E} _ {t: t + k - 1}), \tag {6}
$$
where  $\circ$  denotes the element-wise multiplication and  $\varphi$  is a non-linearity (e.g. ReLU). For simplicity, the bias term is omitted.
With totally  $d$  channels, the local hyperbolic capsules at position  $t$  can be constructed as
$$
\boldsymbol {l} _ {t} = \exp_ {\mathbf {0}} \left(\left[ l _ {t} ^ {(1)}, \dots , l _ {t} ^ {(d)} \right]\right) \in \mathcal {B} ^ {d}. \tag {7}
$$
Therefore, a  $k$ -gram kernel with 1 stride can construct  $T - k + 1$  local hyperbolic capsules. The local hyperbolic capsule set is denoted as  $\{\pmb{u}_1,\dots ,\pmb{u}_L\}$ .
3.2 Global Hyperbolic Capsule Layer
Bidirectional GRU (Chung et al., 2014) is adopted to incorporate forward and backward global contextual information and construct the global hyperbolic capsules. Forward and backward hidden states at time-step  $t$  are obtained by
$$
\begin{array}{l} \begin{array}{c} \overrightarrow {h _ {t}} = \operatorname {G R U} (\overrightarrow {h _ {t - 1}}, e _ {t}), \\ \longleftarrow \quad \longleftarrow \end{array} \tag {8} \\ \overleftarrow {\boldsymbol {h} _ {t}} = \operatorname {G R U} (\overleftarrow {\boldsymbol {h} _ {t + 1}}, \boldsymbol {e} _ {t}). \\ \end{array}
$$
Each of the total  $2T$  hidden states can be taken as a global hyperbolic capsule using the exponential map, i.e.  $\vec{g_t} = \exp_0(\vec{h_t})$ , and equally for the backward capsules. The global hyperbolic capsule set is denoted as  $\{\pmb {u}_1,\dots ,\pmb {u}_G\}$
3.3 Hyperbolic Compression Layer
As discussed in (Zhao et al., 2019), the routing procedure is computational expensive for a large number of capsules. Compressing capsules into a smaller amount can not only relieve the computational complexity, but also merge similar capsules and remove outliers. Therefore, hyperbolic compression layer is introduced. Each compressed local hyperbolic capsule is calculated as a weighted Möbius summation over all the local hyperbolic capsules. For instance,
$$
\boldsymbol {u} _ {l} = \mathfrak {M} _ {\boldsymbol {u} _ {k} \in \left\{\boldsymbol {u} _ {1}, \dots , \boldsymbol {u} _ {L} \right\}} r _ {k} \otimes \boldsymbol {u} _ {k} \in \mathcal {B} ^ {d}, \tag {9}
$$
where  $r_k$  is a learnable weight parameter. And likewise for compressing global hyperbolic capsules.
Let set  $\{\pmb{u}_1, \dots, \pmb{u}_P\}$  denote the compressed local and global hyperbolic capsules together, which are then aggregated in a label-aware manner via HDR.
4 Hyperbolic Dynamic Routing
The purpose of Hyperbolic Dynamic Routing (HDR) is to iteratively aggregate local and global hyperbolic capsules into label-aware hyperbolic capsules, whose activations stand for probabilities of the labels.
Figure 3: Illustration of HYPERCAPS framework.
4.1 Label-Aware Hyperbolic Capsules
With the acquirement of the compressed local and global hyperbolic capsule set  $\{\pmb{u}_1,\dots ,\pmb{u}_P\}$  in layer  $\ell$  , let  $\{\pmb {v}_1,\dots ,\pmb {v}_Q\}$  denote the label-aware hyperbolic capsule set in the next layer  $\ell +1$  , where  $Q$  equals to the number of labels.
Following (Sabour et al., 2017), the compressed hyperbolic capsules are firstly transformed into a set of prediction capsules  $\{\hat{u}_{j|1},\dots,\hat{u}_{j|P}\}$  for the  $j$ -th label-aware capsule, each of them is calculated by
$$
\hat {\boldsymbol {u}} _ {j \mid i} = \boldsymbol {W} _ {i j} \otimes \boldsymbol {u} _ {i} \in \mathcal {B} ^ {d}, \tag {10}
$$
where  $W_{ij}$  is a learnable parameter.
Then  $\pmb{v}_j$  is calculated as a weighted Möbius summation over all the prediction capsules by
$$
\boldsymbol {v} _ {j} = \mathfrak {M} _ {\hat {\boldsymbol {u}} _ {j | i} \in \{\hat {\boldsymbol {u}} _ {j | 1}, \dots , \hat {\boldsymbol {u}} _ {j | P} \}} c _ {i j} \otimes \hat {\boldsymbol {u}} _ {j | i}, \tag {11}
$$
where  $c_{ij}$  denotes the coupling coefficient that indicates the connection strength between  $\hat{\pmb{u}}_{j|i}$  and  $\pmb{v}_j$ .
The coupling coefficient  $c_{ij}$  is iteratively updated during the HDR procedure and computed by the routing softmax
$$
c _ {i j} = \frac {\exp (b _ {i j})}{\sum_ {k} \exp (b _ {i k})}, \tag {12}
$$
where the logits  $b_{ij}$  are the log prior probabilities between capsule  $i$  and  $j$ , which are initialized as 0.
Once the label-aware hyperbolic capsules are produced, each  $b_{ij}$  is then updated by
$$
b _ {i j} = b _ {i j} + \Re \left(d _ {\mathcal {B}} \left(\boldsymbol {v} _ {j}, \hat {\boldsymbol {u}} _ {j \mid i}\right)\right), \tag {13}
$$
where  $d_{\mathcal{B}}(\cdot ,\cdot)$  denotes the Poincaré distance, which can be written as
$$
d _ {\mathcal {B}} (\boldsymbol {u}, \boldsymbol {v}) = \cosh^ {- 1} \left(1 + \frac {1}{2} \lambda_ {\boldsymbol {u}} \lambda_ {\boldsymbol {v}} \| \boldsymbol {u} - \boldsymbol {v} \| ^ {2}\right). \tag {14}
$$
And  $\mathfrak{K}$  is a Epanechnikov kernel function (Wand and Jones, 1994) with
$$
\mathfrak {K} = \left\{ \begin{array}{l l} \gamma - x, & x \in [ 0, \gamma) \\ 0, & x \geq \gamma \end{array} \right. \tag {15}
$$
where  $\gamma$  is the maximum Poincaré distance between two points in the Poincaré ball, which is  $d_{\mathcal{B}}(\pmb {p},\mathbf{0})$  with  $\| \pmb {p}\| = 1 - \epsilon (\epsilon = 10^{-5})$  to avoid numerical errors.
HDR is summarized in Algorithm 1. Different from the routing procedure described in (Sabour et al., 2017), HDR does not require the squashing function since all the hyperbolic capsules are constrained in the Poincaré ball.
4.2 Adaptive Routing
The large amount of labels in MLC is one major source of the computational complexity for the routing procedure. Since most of the labels are unrelated to a document, calculating the label-aware hyperbolic capsules for all the unrelated labels is redundant. Therefore, encoding based adaptive routing layer is used to efficiently decide the candidate labels for the document.
The adaptive routing layer produces the candidate probability of each label by
$$
\boldsymbol {c} = \sigma \left(\boldsymbol {W} _ {c} \frac {1}{T} \sum_ {\boldsymbol {e} _ {i} \in \boldsymbol {E}} \boldsymbol {e} _ {i} + \boldsymbol {b} _ {c}\right), \tag {16}
$$
Table 1: Statistics of the datasets:  $N_{train}$  and  $N_{test}$  are the numbers of training and test instances,  $W_{train}$  and  $W_{test}$  are their average word numbers,  $L$  is the average label number per instance,  $I$  is the average number of training instances per label,  $\# H$  and  $\# T$  are the numbers of head and tail labels,  $H$  and  $T$  are their average number of training instances respectively.
Dataset	Ntrain	Ntest	Wtrain	Wtest	L	I	#H	H	#T	T
AAPD	49,356	6,484	163.34	164.14	2.41	2,199.03	17	5,002.23	37	911.08
RCV1	23,149	781,265	259.47	269.23	3.21	715.50	27	2,209.44	76	184.76
ZHIHU	2,699,969	299,997	38.14	35.56	2.32	3,165.92	442	7,144.31	1,557	2,036.54
EUR-LEX57K	51,000	6,000	726.46	725.37	5.06	53.45	711	273.72	3,560	9.46
Algorithm 1 Hyperbolic Dynamic Routing
1: procedure  $\mathrm{HDR}(\hat{\pmb{u}}_{j|i},r,\ell)$
2: Initialize  $\forall i,j:b_{ij}\gets 0$
3: for  $r$  iterations do
4: for all capsule  $i$  in layer  $\ell$  and capsule  $j$  in layer  $\ell + 1$ :  $c_{ij} \gets \text{softmax}(b_{ij}) \quad \triangleright$  Eq. 12
5: for all capsule  $j$  in layer  $(\ell +1)$
$$
\boldsymbol {v} _ {j} \leftarrow \mathfrak {M} _ {i} c _ {i j} \otimes \hat {\boldsymbol {u}} _ {j | i}
$$
6: for all capsule  $i$  in layer  $\ell$  and capsule  $j$  in layer  $\ell + 1$ :
$$
b _ {i j} \leftarrow b _ {i j} + \mathfrak {K} \left(d _ {\mathcal {B}} \left(\boldsymbol {v} _ {j}, \hat {\boldsymbol {u}} _ {j \mid i}\right)\right)
$$
7: return  $v_{j}$
where  $\sigma$  denotes the Sigmoid function.  $W_{c}$  and the bias  $b_{c}$  are learnable parameters updated by minimizing the binary cross-entropy loss (Liu et al., 2017)
$$
\mathcal {L} ^ {c} = - \sum_ {j = 1} ^ {Q} \left(y _ {j} \log \left(c _ {j}\right) + \left(1 - y _ {j}\right) \log \left(1 - c _ {j}\right)\right), \tag {17}
$$
where  $c_{j}\in [0,1]$  is the  $j$ -th element in  $\pmb{c}$  and  $y_{j}\in$ $\{0,1\}$  denotes the ground truth about label  $j$ . The adaptive routing layer selects the candidate labels during test. Label-aware hyperbolic capsules are then constructed via HDR to predict probabilities of these candidate labels.
During the training process, negative sampling is used to improve the scalability of HYPERCAPS. Let  $\mathcal{N}^{+}$  denote the true label set and  $\mathcal{N}^{-}$  denote the set of randomly selected negative labels, the loss function is derived as
$$
\mathcal {L} ^ {f} = - \left(\sum_ {j \in \mathcal {N} ^ {+}} \log \left(a _ {j}\right) + \sum_ {j \in \mathcal {N} ^ {-}} \log \left(1 - a _ {j}\right)\right), \tag {18}
$$
where  $a_{j} = \sigma (d_{\mathcal{B}}(\boldsymbol {v}_{j},\mathbf{0}))$  is activations of the  $j$  -th label-aware capsules, which is proportional to the distance from the origin of the Poincaré ball.
The proposed HYPERCAPS is evaluated on four benchmark datasets with various label number from 54 to 4271. We compare with the state-of-the-art methods in terms of widely used metrics. Performance on tail labels is also compared to demonstrate the superiority of HYPERCAPS for MLC. An ablation test is also carried out to analyse the contribution of each component of HYPERCAPS.
Datasets Experiments are carried out on four publicly available MLC datasets, including the small-scale AAPD (Yang et al., 2018b) and RCV1 (Lewis et al., 2004), the large-scale ZHIHU $^{1}$  and EUR-LEX57K (Chalkidis et al., 2019). Labels are divided into head and tail sets according to their number of training instances, i.e. labels have less than average number of training instances are divided into the tail label set. Their statistics can be found in Table 1.
Evaluation metrics We use the rank-based evaluation metrics which have been widely adopted for MLC tasks (Bhatia et al., 2015; Liu et al., 2017), i.e. Precision@k ( $P@k$  for short) and nDCG@k, which are respectively defined as
$$
P @ k = \frac {1}{k} \sum_ {j \in r a n k _ {k} (a)} y _ {j}, \tag {19}
$$
$$
n D C G @ k = \frac {\sum_ {j \in r a n k _ {k} (a)} y _ {j} / \log (j + 1)}{\sum_ {j = 1} ^ {\min  (k , \| y \| _ {0})} 1 / \log (j + 1)}, \tag {20}
$$
where  $y_{j} \in \{0,1\}$  denotes the ground truth about label  $j$ ,  $\text{rank}_k(a)$  denotes the indices of the candidate label-aware hyperbolic capsules with  $k$  largest activations in descending order, and  $\| y \|_0$  is the true label number for the document instance.
1https://www.biendata.com/competition/ zhihu/data/.
Table 2: Results on all the labels in  $P@k$  and  $nDCG@k$ , bold face indicates the best of each line.
Dataset	Metric	FASTTEXT	SLEEC	XML-CNN	SGM	REGGNN	NLP-CAP	HYPERCAPS
AAPD	P@1	75.33	75.85	76.31	77.90	79.92	81.75	85.37
P@3	53.83	54.36	54.41	55.76	57.31	59.63	61.89
P@5	37.57	37.89	37.83	38.58	39.50	41.97	42.51
nDCG@3	71.22	71.54	72.12	73.73	75.77	78.40	81.64
nDCG@5	75.78	75.98	76.39	78.05	80.03	83.70	85.87
RCV1	P@1	95.40	95.35	96.86	95.37	96.53	97.05	97.10
P@3	79.96	79.51	81.11	81.36	81.69	81.27	82.04
P@5	55.64	55.06	56.07	53.06	56.23	56.33	57.06
nDCG@3	90.95	90.45	92.22	91.76	92.28	92.47	93.03
nDCG@5	91.68	90.97	92.63	90.69	92.67	93.11	93.66
ZHIHU	P@1	49.40	50.22	49.68	50.32	50.67	53.73	56.50
P@3	31.50	32.21	32.27	31.83	32.43	33.83	35.77
P@5	23.23	23.81	24.17	23.95	24.23	25.10	26.27
nDCG@3	46.52	47.57	46.65	46.90	47.97	48.89	50.61
nDCG@5	49.16	50.34	49.60	50.47	50.70	51.19	52.89
EUR-LEX57K	P@1	86.18	89.43	85.33	89.11	90.46	90.83	91.42
P@3	73.18	76.73	74.40	78.03	79.29	80.72	82.18
P@5	60.15	63.59	61.21	65.02	65.83	69.14	70.53
nDCG@3	77.42	80.98	78.59	82.30	83.45	84.13	86.05
nDCG@5	73.21	76.96	74.36	78.50	79.40	81.91	83.28
The final results are averaged over all the test instances.
Baselines To demonstrate the effectiveness of HYPERCAPS on the benchmark datasets, six comparative text classification methods are chosen as the baselines. FASTTEXT (Joulin et al., 2017) is a representative encoding-based method which use average pooling to construct document representations and MLP to make the predictions. SLEEC (Bhatia et al., 2015) is a typical label-embedding method for MLC, which uses  $k$ -nearest neighbors search to predict the labels. XML-CNN (Liu et al., 2017) employs CNN as local n-gram feature extractors and a dynamic pooling technique as aggregation method. SGM (Yang et al., 2018b) applies the seq2seq model with attention mechanism, which takes the global contextual information. REGGNN (Xu et al., 2019) uses a combination of CNN and LSTM with a dynamic gate that controls the information from these two parts. NLP-CAP (Zhao et al., 2019) is a capsule-based approach for MLC, which reformulates the routing algorithm. NLP-CAP use only CNN to construct capsules, and it applies the squashing function onto capsules.
Implementation Details All the words are converted to lower case and padding is used to handle the various lengths of the text sequences. Maximum length of AAPD, RCV1 and EUR-LEX57K is set to 500, while maximum length of ZHIHU is
50. To compose the word vector representations, pre-trained 300-dimensional GLOVE (Pennington et al., 2014) word embeddings are used for AAPD, RCV1 and EUR-LEX57K, while ZHIHU uses its specified 256-dimensional word embeddings. The dimension of the Poincaré ball is set to 32 with a radius  $1 - \epsilon$  ( $\epsilon = 10^{-5}$ ) to avoid numerical errors. Multiple one-dimensional convolutional kernels (with window sizes of 2, 4, 8) are applied in the local hyperbolic capsule layer. The number of compressed local and global hyperbolic capsules is 128. Adaptive routing layer is not applied on the small-scale datasets AAPD and RCV1. The maximum candidate label number is set to 200 for the large-scale datasets ZHIHU and EUR-LEX57K. For the baselines, hyperparameters recommended by their authors are adopted.