Venue: ACL 2020

# Hyperbolic Capsule Networks for Multi-Label Classification

Boli Chen, Xin Huang, Lin Xiao, Liping Jing

Beijing Key Lab of Traffic Data Analysis and Mining

 Beijing Jiaotong University, Beijing, China

{18120345, 18120367, 17112079, lpjing}@bjtu.edu.cn

###### Abstract

Although deep neural networks are effective at extracting high-level features, classification methods usually encode an input into a vector representation via simple feature aggregation operations (_e.g._ pooling). Such operations limit the performance. For instance, a multi-label document may contain several concepts. In this case, one vector can not sufficiently capture its salient and discriminative content. Thus, we propose Hyperbolic Capsule Networks (HyperCaps) for Multi-Label Classification (MLC), which have two merits. First, hyperbolic capsules are designed to capture fine-grained document information for each label, which has the ability to characterize complicated structures among labels and documents. Second, Hyperbolic Dynamic Routing (HDR) is introduced to aggregate hyperbolic capsules in a label-aware manner, so that the label-level discriminative information can be preserved along the depth of neural networks. To efficiently handle large-scale MLC datasets, we additionally present a new routing method to adaptively adjust the capsule number during routing. Extensive experiments are conducted on four benchmark datasets. Compared with the state-of-the-art methods, HyperCaps significantly improves the performance of MLC especially on tail labels.

## 1 Introduction

The main difference between Multi-Class Classification (MCC) and Multi-Label Classification (MLC) is that datasets in MCC have only serval mutually exclusive classes, while datasets in MLC contain much more correlated labels. MLC allows label co-occurrence in one document, which indicates that the labels are not disjointed. In addition, a large fraction of the labels are the infrequently occurring _tail labels_[2], which is also referred as the power-law label distribution. Figure 1 illustrates the label distribution of EURLex57k[1]. A multi-label document usually has serval head and tail labels, and hence contain several concepts about both its head and tail labels simultaneously.

Recent works for text classification, such as CNN-Kim[15] and FastText[16], focus on encoding a document into a fixed-length vector as the _distributed document representation_[11]. These encoding based deep learning methods use simple operations (_e.g._ pooling) to aggregate features extracted by neural networks and construct the document vector representation. A Fully-Connected (FC) layer is usually applied upon the document vector to predict the probability of each label. And each row in its weight matrix can be interpreted as a label vector representation [17]. In this way, the label probability can be predicted by computing the dot product between label and document vectors, which is proportional to the scalar projection of the label vector onto the document vector as shown in Figure 2. For example, label "_movie_" should have the largest scalar projection onto a document about "movie". However, even

Figure 1: The power-law label distribution of EURLex57k with y-axis on log-scale. Division is based on average number of training instances.



the learned label representation of "_music_" can be distinguished from "_movie_", it may also have a large scalar projection onto the document.

Moreover, multi-label documents always contain several concepts about multiple labels, such as a document about "sport movie". Whereas the document vector representation is identical to all the labels, and training instances for tail labels are inadequate compared to head labels. The imbalance between head and tail labels makes it hard for the FC layer to make prediction, especially on tail labels. In this case, one vector can not sufficiently capture its salient and discriminative content. Therefore, the performance of constructing the document vector representation via simple aggregation operations is limited for MLC.

Capsule networks Sabour et al. (2017); Yang et al. (2018a) has recently proposed to use dynamic routing in place of pooling and achieved better performance for classification tasks. In fact, capsules are fine-grained features compared to the distributed document representation, and dynamic routing is a label-aware feature aggregation procedure. Zhao et al. (2019) improves the scalability of capsule networks for MLC. However, they only use CNN to construct capsules, which capture local contextual information Wang et al. (2016). Effectively learning the document information about multiple labels is crucial for MLC. Thus we propose to connect CNN and RNN in parallel to capture both local and global contextual information, which would be complementary to each other. Nevertheless, Euclidean capsules necessitate designing a non-linear squashing function.

Inspired by the hyperbolic representation learning methods which demonstrate that the hyperbolic space has more representation capacity than the Euclidean space Nickel and Kiela (2017); Ganea et al. (2018a), Hyperbolic Capsule Networks HyperCaps is proposed. Capsules are constrained in the hyperbolic space which does not require the squashing function. Hyperbolic Dynamic Routing (HDR) is introduced to aggregate hyperbolic capsules in a label-aware manner. Moreover, in order to fit the large label set of MLC and improve the scalability of HyperCaps, adaptive routing is presented to adjust the number of capsules participated in the routing procedure.

The main contributions of our work are therefore summarized as follows:

* We propose to connect CNN and RNN in parallel to simultaneously extract local and global contextual information, which would be complementary to each other.
* HyperCaps with HDR are formulated to aggregate features in a label-aware manner, and hyperbolic capsules benefits from the representation capacity of the hyperbolic space.
* Adaptive routing is furthermore presented to improve the scalability of HyperCaps and fit the large label set of MLC.
* Extensive experiments on four benchmark MLC datasets demonstrate the effectiveness of HyperCaps, especially on tail labels.

## 2 Preliminaries

In order to make neural networks work in the hyperbolic space, formalism of the _Mobius gyrovector space_ is adopted Ganea et al. (2018b).

An \(n\)-dimensional _Poincare ball_\({\cal B}^{n}\) is a _Riemannian manifold_ defined as \({\cal B}^{n}=\{\mathbf{x}\in\mathbb{R}^{n}~{}|~{}\lVert\mathbf{x}\rVert<1\}\), with its _tangent space_ around \(\mathbf{p}\in{\cal B}^{n}\) denoted as \(T_{\mathbf{p}}{\cal B}^{n}\) and the _conformal factor_ as \(\lambda_{\mathbf{p}}:=\frac{2}{1-\lVert\mathbf{p}\rVert^{2}}\). The _exponential map_\(\exp_{\mathbf{p}}:T_{\mathbf{p}}{\cal B}^{n}\to{\cal B}^{n}\) for \(\mathbf{w}\in T_{\mathbf{p}}{\cal B}^{n}\setminus\{\mathbf{0}\}\) is consequently defined as

\[\exp_{\mathbf{p}}(\mathbf{w})=\mathbf{p}\oplus(\tanh(\frac{\lambda_{\mathbf{p}}}{2}\lVert\mathbf{ w}\rVert)\frac{\mathbf{w}}{||\mathbf{w}\rVert}). \tag{1}\]

To work with hyperbolic capsules, Mobius operations in the Poincare ball also need to be formulated.

_Mobius addition_ for \(\mathbf{u},\mathbf{v}\in{\cal B}^{n}\) is defined as

\[\mathbf{u}\oplus\mathbf{v}=\tfrac{(1+2\langle\mathbf{u},\mathbf{v}\rangle+\lVert\mathbf{v}\rVert ^{2})\mathbf{u}+(1-\lVert\mathbf{u}\rVert^{2})\mathbf{v}}{1+2\langle\mathbf{u},\mathbf{v}\rangle+ \lVert\mathbf{u}\rVert^{2}\lVert\mathbf{v}\rVert^{2}}, \tag{2}\]

where \(\langle\cdot,\cdot\rangle\) denotes the Euclidean inner product.

Figure 2: Illustration of the FC layer in the encoding based methods.

 

Thus _Mobius summation_ can be formulated as

\[\mathop{\varprojlim}_{i=m}^{n}\mathbf{p}_{i}=\mathbf{p}_{m}\oplus\cdots\oplus\mathbf{p}_{n}, p_{i}\in\mathcal{B}^{n}. \tag{3}\]

_Mobius scalar multiplication_ for \(k\in\mathbb{R}\) and \(\mathbf{p}\in\mathcal{B}^{n}\setminus\{\mathbf{0}\}\) is defined as

\[k\otimes\mathbf{p}=\tanh(k\tanh^{-1}(\|\mathbf{p}\|))\frac{\mathbf{p}}{\|\mathbf{p}\|}. \tag{4}\]

And \(k\otimes\mathbf{p}=\mathbf{0}\) when \(\mathbf{p}=\mathbf{0}\in\mathcal{B}^{n}\).

The definition of _Mobius matrix-vector multiplication_ for \(\mathbf{M}\in\mathbb{R}^{m\times n}\) and \(\mathbf{p}\in\mathcal{B}^{n}\) when \(\mathbf{M}\mathbf{p}\neq\mathbf{0}\) is as follows

\[\mathbf{M}\otimes\mathbf{p}=\tanh(\tfrac{\|\mathbf{M}\mathbf{p}\|}{\|\mathbf{p}\|}\tanh^{-1}(\| \mathbf{p}\|))\tfrac{\mathbf{M}\mathbf{p}}{\|\mathbf{M}\mathbf{p}\|}. \tag{5}\]

And \(\mathbf{M}\otimes\mathbf{p}=\mathbf{0}\) when \(\mathbf{M}\mathbf{p}=\mathbf{0}\).

HDR is developed based on these operations.

## 3 Local and Global Hyperbolic Capsules

Neural networks are generally used as effective feature extractors for text classification. Kernels of CNN can be used to capture local n-gram contextual information at different positions of a text sequence, while hidden states of RNN can represent global long-term dependencies of the text Wang et al. (2016). Hence, we propose to obtain the combination of local and global hyperbolic capsules by connecting CNN and RNN in parallel, which would be complementary to each other.

Given a text sequence of a document with \(T\) word tokens \(\mathbf{x}=[x_{1},\dots,x_{T}]\), pre-trained \(w\)-dimensional word embeddings (_e.g._Glove Pennington et al. (2014)) are used to compose word vector representations \(\mathbf{E}=[\mathbf{e}_{1},\ldots,\mathbf{e}_{T}]\in\mathbb{R}^{T\times w}\), upon which CNN and RNN connected in parallel are used to construct local and global hyperbolic capsules in the Poincare ball. Figure 3 illustrates the framework for HyperCaps.

### Local Hyperbolic Capsule Layer

N-gram kernels \(\mathbf{K}\in\mathbb{R}^{k\times w}\) with different window size \(k\) are applied on the local region of the word representations \(\mathbf{E}_{t:t+k-1}\in\mathbb{R}^{k\times w}\) to construct the local features as

\[l_{t}=\varphi(\mathbf{K}\circ\mathbf{E}_{t:t+k-1}), \tag{6}\]

where \(\circ\) denotes the element-wise multiplication and \(\varphi\) is a non-linearity (_e.g. ReLU_). For simplicity, the bias term is omitted.

With totally \(d\) channels, the local hyperbolic capsules at position \(t\) can be constructed as

\[\mathbf{l}_{t}=\exp_{\mathbf{0}}([l_{t}^{(1)},\dots,l_{t}^{(d)}])\in\mathcal{B}^ {d}. \tag{7}\]

Therefore, a \(k\)-gram kernel with \(1\) stride can construct \(T-k+1\) local hyperbolic capsules. The local hyperbolic capsule set is denoted as \(\{\mathbf{u}_{1},\dots,\mathbf{u}_{L}\}\).

### Global Hyperbolic Capsule Layer

Bidirectional GRU Chung et al. (2014) is adopted to incorporate forward and backward global contextual information and construct the global hyperbolic capsules. Forward and backward hidden states at time-step \(t\) are obtained by

\[\begin{split}&\overrightarrow{\mathbf{h}_{t}}=\text{GRU}( \overrightarrow{\mathbf{h}_{t-1}},\mathbf{e}_{t}),\\ &\overleftarrow{\mathbf{h}_{t}}=\text{GRU}(\overleftarrow{\mathbf{h}_{ t+1}},\mathbf{e}_{t}).\end{split} \tag{8}\]

Each of the total \(2T\) hidden states can be taken as a global hyperbolic capsule using the exponential map, _i.e._\(\overrightarrow{\mathbf{g}_{t}}=\exp_{\mathbf{0}}(\overrightarrow{\mathbf{h}_{t}})\), and equally for the backward capsules. The global hyperbolic capsule set is denoted as \(\{\mathbf{u}_{1},\dots,\mathbf{u}_{G}\}\).

### Hyperbolic Compression Layer

As discussed in Zhao et al. (2019), the routing procedure is computational expensive for a large number of capsules. Compressing capsules into a smaller amount can not only relieve the computational complexity, but also merge similar capsules and remove outliers. Therefore, hyperbolic compression layer is introduced. Each compressed local hyperbolic capsule is calculated as a weighted Mobius summation over all the local hyperbolic capsules. For instance,

\[\mathbf{u}_{l}=\mathop{\varprojlim}_{\mathbf{u}_{k}\in\{\mathbf{u}_{1},\dots,\mathbf{u}_{L}\}}r _{k}\otimes\mathbf{u}_{k}\in\mathcal{B}^{d}, \tag{9}\]

where \(r_{k}\) is a learnable weight parameter. And likewise for compressing global hyperbolic capsules.

Let set \(\{\mathbf{u}_{1},\dots,\mathbf{u}_{P}\}\) denote the compressed local and global hyperbolic capsules together, which are then aggregated in a label-aware manner via HDR.

## 4 Hyperbolic Dynamic Routing

The purpose of Hyperbolic Dynamic Routing (HDR) is to iteratively aggregate local and global hyperbolic capsules into label-aware hyperbolic capsules, whose activations stand for probabilities of the labels 



### Label-Aware Hyperbolic Capsules

With the acquirement of the compressed local and global hyperbolic capsule set \(\{\mathbf{u}_{1},\dots,\mathbf{u}_{P}\}\) in layer \(\ell\), let \(\{\mathbf{v}_{1},\dots,\mathbf{v}_{Q}\}\) denote the label-aware hyperbolic capsule set in the next layer \(\ell+1\), where \(Q\) equals to the number of labels.

Following Sabour et al. (2017), the compressed hyperbolic capsules are firstly transformed into a set of prediction capsules \(\{\hat{\mathbf{u}}_{j|1},\dots,\hat{\mathbf{u}}_{j|P}\}\) for the \(j\)-th label-aware capsule, each of them is calculated by

\[\hat{\mathbf{u}}_{j|i}=\mathbf{W}_{ij}\otimes\mathbf{u}_{i}\in\mathcal{B}^{d}, \tag{10}\]

where \(\mathbf{W}_{ij}\) is a learnable parameter.

Then \(\mathbf{v}_{j}\) is calculated as a weighted Mobius summation over all the prediction capsules by

\[\mathbf{v}_{j}=\mathop{\mathfrak{M}}_{\hat{\mathbf{u}}_{j|i}\in\{\hat{\mathbf{u}}_{j|1}, \dots,\hat{\mathbf{u}}_{j|P}\}}c_{ij}\otimes\hat{\mathbf{u}}_{j|i}, \tag{11}\]

where \(c_{ij}\) denotes the coupling coefficient that indicates the connection strength between \(\hat{\mathbf{u}}_{j|i}\) and \(\mathbf{v}_{j}\).

The coupling coefficient \(c_{ij}\) is iteratively updated during the HDR procedure and computed by the _routing softmax_

\[c_{ij}=\frac{\exp(b_{ij})}{\sum_{k}\exp(b_{ik})}, \tag{12}\]

where the logits \(b_{ij}\) are the log prior probabilities between capsule \(i\) and \(j\), which are initialized as \(0\).

Once the label-aware hyperbolic capsules are produced, each \(b_{ij}\) is then updated by

\[b_{ij}=b_{ij}+\mathfrak{K}(d_{\mathcal{B}}(\mathbf{v}_{j},\hat{\mathbf{u}}_{j|i})), \tag{13}\]

where \(d_{\mathcal{B}}(\cdot,\cdot)\) denotes the _Poincare distance_, which can be written as

\[d_{\mathcal{B}}(\mathbf{u},\mathbf{v})=\cosh^{-1}(1+\frac{1}{2}\lambda_{\mathbf{u}} \lambda_{\mathbf{v}}\lVert\mathbf{u}-\mathbf{v}\rVert^{2}). \tag{14}\]

And \(\mathfrak{K}\) is a _Epanechnikov kernel function_Wand Jones (1994) with

\[\mathfrak{K}=\begin{cases}\gamma-x,&x\in[0,\gamma)\\ 0,&x\geq\gamma\end{cases} \tag{15}\]

where \(\gamma\) is the maximum Poincare distance between two points in the Poincare ball, which is \(d_{\mathcal{B}}(\mathbf{p},\mathbf{0})\) with \(\lVert\mathbf{p}\rVert=1-\epsilon\) (\(\epsilon=10^{-5}\)) to avoid numerical errors.

HDR is summarized in Algorithm 1. Different from the routing procedure described in Sabour et al. (2017), HDR does not require the squashing function since all the hyperbolic capsules are constrained in the Poincare ball.

``` Input: A sequence of labeled capsules \(\{\mathbf{u}_{1},\dots,\mathbf{u}_{P}\}\), \(\{\mathbf{v}_{1},\dots,\mathbf{v}_{Q}\}\), \(\{\mathbf{w}_{1},\dots,\mathbf{w}_{Q}\}\), \(\{\mathbf{w}_{2},\dots,\mathbf{w}_{Q}\}\), \(\{\mathbf{w}_{q},\mathbf{w}_{1},\dots,\mathbf{w}_{Q}\}\), \(\{\mathbf{w}_{r},\mathbf{w}_{l}\}\), \(\{\mathbf{w}_{s},\mathbf{w}_{t}\}\), \(\{\mathbf{w}_{t},\mathbf{w}_{s}\}\), \(\{\mathbf{w}_{t},\mathbf{w}_{t}\}\), \(\{\mathbf{w}_{s},\mathbf{w}



where \(\sigma\) denotes the _Sigmoid_ function. \({\mathbf{W}}_{c}\) and the bias \({\mathbf{b}}_{c}\) are learnable parameters updated by minimizing the _binary cross-entropy_ loss Liu et al. (2017)

\[\mathcal{L}^{c}=-\sum_{j=1}^{Q}\big{(}y_{j}log(c_{j})+(1-y_{j})log(1- c_{j})\big{)}, \tag{17}\]

where \(c_{j}\in[0,1]\) is the \(j\)-th element in \({\mathbf{c}}\) and \(y_{j}\in\{0,1\}\) denotes the ground truth about label \(j\). The adaptive routing layer selects the candidate labels during test. Label-aware hyperbolic capsules are then constructed via HDR to predict probabilities of these candidate labels.

During the training process, negative sampling is used to improve the the scalability of HyperCaps. Let \(\mathcal{N}^{+}\) denote the true label set and \(\mathcal{N}^{-}\) denote the set of randomly selected negative labels, the loss function is derived as

\[\mathcal{L}^{f}=-\big{(}\sum_{j\in\mathcal{N}^{+}}log(a_{j})+ \sum_{j\in\mathcal{N}^{-}}log(1-a_{j})\big{)}, \tag{18}\]

where \(a_{j}=\sigma(\textit{dg}({\mathbf{v}}_{j},{\bf 0}))\) is activations of the \(j\)-th label-aware capsules, which is proportional to the distance from the origin of the Poincare ball.

## 5 Experiments

The proposed HyperCaps is evaluated on four benchmark datasets with various label number from 54 to 4271. We compare with the state-of-the-art methods in terms of widely used metrics. Performance on tail labels is also compared to demonstrate the superiority of HyperCaps for MLC. An ablation test is also carried out to analyse the contribution of each component of HyperCaps.

### Experimental Setup

DatasetsExperiments are carried out on four publicly available MLC datasets, including the small-scale AAPD Yang et al. (2018) and RCV1 Lewis et al. (2004), the large-scale Zhihu1 and EUR-Lex57k Chalkidis et al. (2019). Labels are divided into head and tail sets according to their number of training instances, _i.e._ labels have less than average number of training instances are divided into the tail label set. Their statistics can be found in Table 1.

Footnote 1: [https://www.biendata.com/competition/zhihu/data/](https://www.biendata.com/competition/zhihu/data/).

Evaluation metricsWe use the rank-based evaluation metrics which have been widely adopted for MLC tasks Bhatia et al. (2015); Liu et al. (2017), _i.e. Precision@k_ (_P@k_ for short) and _nDCG@k_, which are respectively defined as

\[\textit{P@k}=\frac{1}{k}\sum_{j\inrank_{k}(a)}y_{j}, \tag{19}\] \[\textit{nDCG@k}=\frac{\sum_{j\inrank_{k}(a)}y_{j}/log(j+1)}{\sum _{j=1}^{\min(k,\|y\|_{0})}1/log(j+1)}, \tag{20}\]

where \(y_{j}\in\{0,1\}\) denotes the the ground truth about label \(j\), \(rank_{k}(a)\) denotes the indices of the candidate label-aware hyperbolic capsules with \(k\) largest activations in descending order, and \(\|y\|_{0}\) is the true label number for the document instance.

\begin{table}
\begin{tabular}{l r r l l l l l l l} \hline \hline Dataset & \(N_{train}\) & \(N_{test}\) & \(W_{train}\) & \(W_{test}\) & \(L\) & \(I\) & \(\#H\) & \(H\) & \(\#T\) & \(T\) \\ \hline AAPD & 49,356 & 6,484 & 163.34 & 164.14 & 2.41 & 2,199.03 & 17 & 5,002.23 & 37 & 911.08 \\ RCV1 & 23,149 & 781,265 & 259.47 & 269.23 & 3.21 & 715.50 & 27 & 2,209.44 & 76 & 184.76 \\ Zhihu & 2,699,969 & 299,997 & 38.14 & 35.56 & 2.32 & 3,165.92 & 442 & 7,144.31 & 1,557 & 2,036.54 \\ EUR-Lex57k & 51,000 & 6,000 & 726.46 & 725.37 & 5.06 & 53.45 & 711 & 273.72 & 3,560 & 9.46 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Statistics of the datasets: \(N_{train}\) and \(N_{test}\) are the numbers of training and test instances, \(W_{train}\) and \(W_{test}\) are their average word numbers, \(L\) is the average label number per instance, \(I\) is the average number of training instances per label, \(\#H\) and \(\#T\) are the numbers of head and tail labels, \(H\) and \(T\) are their average number of training instances respectively.

 

The final results are averaged over all the test instances.

BaselinesTo demonstrate the effectiveness of HyperCaps on the benchmark datasets, six comparative text classification methods are chosen as the baselines. FastText(Joulin et al., 2017) is a representative encoding-based method which use average pooling to construct document representations and MLP to make the predictions. SLEEC Bhatia et al. (2015) is a typical label-embedding method for MLC, which uses _k-nearest neighbors_ search to predict the labels. XML-CNN Liu et al. (2017) employs CNN as local n-gram feature extractors and a dynamic pooling technique as aggregation method. SGM Yang et al. (2018b) applies the _seq2seq_ model with attention mechanism, which takes the global contextual information. RegGNN Xu et al. (2019) uses a combination of CNN and LSTM with a dynamic gate that controls the information from these two parts. NLP-Cap Zhao et al. (2019) is a capsule-based approach for MLC, which reformulates the routing algorithm. NLP-Cap use only CNN to construct capsules, and it applies the squashing function onto capsules.

Implementation DetailsAll the words are converted to lower case and padding is used to handle the various lengths of the text sequences. Maximum length of AAPD, RCV1 and EUR-Lex57k is set to 500, while maximum length of Zihhu is 50. To compose the word vector representations, pre-trained 300-dimensional Glove Pennington et al. (2014) word embeddings are used for AAPD, RCV1 and EUR-Lex57k, while Zhihu uses its specified 256-dimensional word embeddings. The dimension of the Poincare ball is set to 32 with a radius \(1-\epsilon\) (\(\epsilon=10^{-5}\)) to avoid numerical errors. Multiple one-dimensional convolutional kernels (with window sizes of 2, 4, 8) are applied in the local hyperbolic capsule layer. The number of compressed local and global hyperbolic capsules is 128. Adaptive routing layer is not applied on the small-scale datasets AAPD and RCV1. The maximum candidate label number is set to 200 for the large-scale datasets Zhihu and EUR-Lex57k. For the baselines, hyperparameters recommended by their authors are adopted.

### Experimental Results

The proposed HyperCaps is evaluated on the four benchmark datasets by comparing with the six baselines in terms of _P@k_ and _nDCG@k_ with \(k=1,3,5\). Results on all the labels averaged over the test instances are shown in Table 2. _nDCG@1_ is omitted since it gives the same value as _P@1_. It is notable that HyperCaps obtains competitive results on the four datasets.

The encoding-based FastText is generally inferior to the other baselines as it applies the average pooling on word vector representations, which ig 

nores word order for the construction of document representations. The typical MLC method SLEEC takes advantage of label correlations by embedding the label co-occurrence graph. However, SLEEC uses TF-IDF vectors to represent documents, thus word order is also ignored. XML-CNN uses a dynamic pooling technique to aggregate the local contextual features extracted by CNN, while SGM uses attention mechanism to aggregate the global contextual features extracted by LSTM. RegGNN is generally superior to both of them as it combines the local and global contextual information dynamically and takes label correlations into consideration using a regularized loss. However, the two capsule-based methods NLP-Cap and HyperCaps consistently outperform all the other methods owing to dynamic routing, which aggregates the fine-grained capsule features in a label-aware manner.

Moreover, NLP-Cap only uses CNN to extract the local contextual information, while HyperCaps benefits from the parallel combination of local and global contextual information. In addition, NLP-Cap applies the non-linear squashing function for capsules in the Euclidean space, while HDR is designed for hyperbolic capsules, which take advantage of the representation capacity of the hyperbolic space. Therefore, HyperCaps outperforms NLP-Cap as expected. This result further confirms that the proposed HyperCaps with HDR is effective to learn the label-aware hyperbolic capsules for MLC.

### Performance on Tail Labels

In MLC, tail labels have low occurring frequency and hence are hard to predict compared to head labels. The performance on tail labels of the four benchmark datasets is evaluated in terms of _nDCG@k_ with \(k=1,3,5\). Figure 4 shows the results of the five deep learning based MLC methods, _i.e._ XML-CNN, SGM, RegGNN, NLP-Cap and HyperCaps._nDCG@l_ is smaller than _nDCG@3_ on AAPD, RCV1 and Zhihu since most of their test instances contain less than three tail labels. It is remarkable that HyperCaps outperforms all the other methods on tail labels.

RegGNN takes advantage of the local and global contextual information and label correlations, thus it outperforms XML-CNN and SGM. The two capsule-based methods NLP-Cap and HyperCaps are both superior to the other methods, which indicates that the label-aware dynamic routing is effective for the prediction on tail labels. In addition, the fact that HyperCaps significantly improves the prediction performance compared to NLP-Cap implies that the representation capacity of the hyperbolic space and the combination of local and global contextual information are helpful for learning on tail labels. The results demonstrate the superiority of the proposed HyperCaps on tail labels for MLC.

Figure 4: Results on tail labels in _nDCG@k_.

Figure 5: Results of ablation test on EUR-Lex57k in _P@k_. L denotes local capsules, G denotes global capsules, H denotes HDR.

 



### Ablation Test

An ablation test would be informative to analyze the effect of varying different components of the proposed HyperCaps, which can be taken apart as local Euclidean capsules only (denoted as L), global Euclidean capsules only (denoted as G), a combination of the local and global Euclidean capsules (denoted as L + G), and a combination of the local and global hyperbolic capsules (denoted as L + G + H). Euclidean capsules (in L, G and L + G) are aggregated via the origin dynamic routing [17], while hyperbolic capsules (in L + G + H) are aggregated via our HDR.

Figure 5 shows the results on EUR-Lex57k in terms of _P@k_ with \(k=1,3,5\). In order to make the comparison fair, the number of total compressed capsules is equally set to 256 for all the four models. Adaptive routing is also applied with the maximum candidate label number set equally to 200. Generally, the proposed combination of local and global contextual information contributes to the effectiveness of the model (L + G). Therefore, it is practical to combine the local and global contextual information via dynamic routing. HDR furthermore improves the performance by making use of the representation capacity of the hyperbolic space. Overall, each of the components benefits the performance of HyperCaps for MLC.

In summary, extensive experiments are carried out on four MLC benchmark datasets with various scales. The results demonstrate that the proposed HyperCaps can achieve competitive performance compared with the baselines. In particular, effectiveness of HyperCaps is shown on tail labels. The ablation test furthermore confirms that the combination of local and global contextual information is practical and HyperCaps benefits from the representation capacity of the hyperbolic space.

## 6 Related Work

### Multi-Label Classification

Multi-label classification (MLC) aims at assigning multiple relevant labels to one document. The MLC label set is large compared to Multi-class classification (MCC). Besides, the correlations of labels (_e.g._ hierarchical label structures [49]) and the existence of tail labels make MLC a hard task [8].

As data sparsity and scalability issues arise with the large number of labels, XML-CNN [10] employs CNN as efficient feature extractor, whereas it ignores label correlations, which are often used to deal with tail labels. The traditional MLC method SLEEC [8] makes use of label correlations by embedding the label co-occurrence graph. The seq2seq model SGM [30] uses the attention mechanism to consider the label correlations, while RegGNN [29] applies a regularized loss specified for label co-occurrence. RegGNN additionally chooses to dynamically combine the local and global contextual information to construct document representations.

### Capsule Networks

Capsule networks are recently proposed to address the representation limitations of CNN and RNN. The concept of capsule is first introduced by [11]. [17] replaces the scalar output features of CNN with vector capsules and pooling with dynamic routing. [11] proposes the EM algorithm based routing procedure between capsule layers. [1] proposes to regard dynamic routing as an information aggregation procedure, which is more effective than pooling. [30] and [12] investigate capsule networks for text classification. [20] then presents a capsule compression method and reformulates the routing procedure to fit for MLC.

Our work is different from the predecessors as we design the Hyperbolic Dynamic Routing (HDR) to aggregate the parallel combination of local and global contextual information in form of hyperbolic capsules, which are constrained in the hyperbolic space without the requirement of non-linear squashing function. In addition, adaptive routing is proposed to improve the scalability for large number of labels.

### Hyperbolic Deep Learning

Recent research on representation learning [13] indicates that hyperbolic space is superior to Euclidean space in terms of representation capacity, especially in low dimension. [14] generalizes operations for neural networks in the Poincare ball using formalism of Mobius gyrovector space. Some works lately demonstrate the superiority of the hyperbolic space for serval natural language processing tasks, such as textual entailment [14], machine translation [15] and word embedding [16]. Our work presents

the Hyperbolic Capsule Networks (HyperCaps) for MLC.

## 7 Conclusion

We present the Hyperbolic Capsule Networks (HyperCaps) with Hyperbolic Dynamic Routing (HDR) and adaptive routing for Multi-Label Classification (MLC). The proposed HyperCaps takes advantage of the parallel combination of fine-grained local and global contextual information and label-aware feature aggregation method HDR to dynamically construct label-aware hyperbolic capsules for tail and head labels. Adaptive routing is additionally applied to improve the scalability of HyperCaps by controlling the number of capsules during the routing procedure. Extensive experiments are carried out on four benchmark datasets. Results compared with the state-of-the-art methods demonstrate the superiority of HyperCaps, especially on tail labels. As recent works explore the superiority of hyperbolic space to Euclidean space for serval natural language processing tasks, we intend to couple with the hyperbolic neural networks [14] and the hyperbolic word embedding method such as PoincareGlove[23] in the future.

This work was supported in part by the National Natural Science Foundation of China under Grant 61822601, 61773050, and 61632004; the Beijing Natural Science Foundation under Grant Z180006; National Key Research and Development Program (2017YFC1703506); the Fundamental Research Funds for the Central Universities (2019JBZ110). We thank the anonymous reviewers for their valuable feedback.

## References

* Banerjee et al. (2019) Siddhartha Banerjee, Cem Akkaya, Francisco Perez-Sorrosal, and Kostas Tsioutsiouliklis. 2019. Hierarchical transfer learning for multi-label text classification. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6295-6300.
* Bhatia et al. (2015) Kush Bhatia, Himanshu Jain, Purushottam Kar, Manik Varma, and Prateek Jain. 2015. Sparse local embeddings for extreme multi-label classification. In _Advances in Neural Information Processing Systems 28_, pages 730-738.
* Chalkidis et al. (2019) Ilias Chalkidis, Emmanouil Fergadiotis, Prodromos Malakasiots, and Ion Androutsopoulos. 2019. Large-scale multi-label text classification on EU legislation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6314-6322.
* Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In _NIPS 2014 Workshop on Deep Learning_.
* Du et al. (2019a) Chunning Du, Haifeng Sun, Jingyu Wang, Qi Qi, Jianxin Liao, Chun Wang, and Bing Ma. 2019a. Investigating capsule network and semantic feature on hyperplanes for text classification. pages 456-465.
* Du et al. (2019b) Cunxiao Du, Zhaozheng Chin, Fuli Feng, Lei Zhu, Tian Gan, and Liqiang Nie. 2019b. Explicit interaction model towards text classification. In _Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence_, pages 6359-6366.
* Ganea et al. (2018a) Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018a. Hyperbolic entailment cones for learning hierarchical embeddings. In _Proceedings of the 35th International Conference on Machine Learning_, pages 1646-1655.
* Ganea et al. (2018b) Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018b. Hyperbolic neural networks. In _Advances in neural information processing systems 31_, pages 5345-5355.
* Gong et al. (2018) Jingjing Gong, Xipeng Qiu, Shaojing Wang, and Xuaming Huang. 2018. Information aggregation via dynamic routing for sequence encoding. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 2742-2752.
* Gulcehre et al. (2019) Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. 2019. Hyperbolic attention networks. In _International Conference on Learning Representations_.
* Hinton et al. (2011) Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. 2011. Transforming auto-encoders. In _International Conference on Artificial Neural Networks_, pages 44-51. Springer.
* Hinton et al. (2018) Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. 2018. Matrix capsules with EM routing. In _International Conference on Learning Representations_.
* Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_, pages 427-431.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1746-1751 

* Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In _Proceedings of the 31st International Conference on Machine Learning_, pages 1188-1196.
* Lewis et al. (2004) David D Lewis, Yiming Yang, Tony G Rose, and Fan Li. 2004. Rcv1: A new benchmark collection for text categorization research. _Journal of machine learning research_, 5(Apr):361-397.
* Liu et al. (2017) Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. 2017. Deep learning for extreme multi-label text classification. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_, pages 115-124.
* Nickel and Kiela (2017) Maximillian Nickel and Douwe Kiela. 2017. Poincare embeddings for learning hierarchical representations. In _Advances in Neural Information Processing Systems 30_, pages 6338-6347.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 1532-1543.
* Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017. Dynamic routing between capsules. In _Advances in Neural Information Processing Systems 30_, pages 3856-3866.
* Tifrea et al. (2019) Alexandru Tifrea, Gary Becigneul, and Octavian-Eugen Ganea. 2019. Poincare glove: Hyperbolic word embeddings. In _International Conference on Learning Representations_.
* Wand and Jones (1994) Matt P Wand and M Chris Jones. 1994. _Kernel smoothing_. Chapman and Hall/CRC.
* Wang et al. (2016) Xingyou Wang, Weijie Jiang, and Zhiyong Luo. 2016. Combination of convolutional and recurrent neural network for sentiment analysis of short texts. In _Proceedings of the 26th International Conference on Computational Linguistics_, pages 2428-2437.
* Xu et al. (2019) Yunlai Xu, Xiangying Ran, Wei Sun, Xiangyang Luo, and Chongjun Wang. 2019. gated neural network with regularized loss for multi-label text classification. In _2019 International Joint Conference on Neural Networks_, pages 1-8.
* Yang et al. (2018a) Min Yang, Wei Zhao, Jianbo Ye, Zeyang Lei, Zhou Zhao, and Soufei Zhang. 2018a. Investigating capsule networks with dynamic routing for text classification. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3110-3119.
* Yang et al. (2018b) Pengcheng Yang, Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018b. SGM: Sequence generation model for multi-label classification. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 3915-3926.
* Zhao et al. (2019) Wei Zhao, Haiyun Peng, Steffen Eger, Erik Cambria, and Min Yang. 2019. Towards scalable and reliable capsule networks for challenging NLP applications. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 1549-1559.

## Appendix A Label Distributions

Figure 1 and Figure 6 show the label distributions of the four benchmark datasets. Head and tail labels are divided based on the average number of training instances (listed in Table 1), _i.e._ labels have less than average number of training instances are tail labels. We observe that this division generally follows the _Pareto Principle_, as nearly 80% of labels are divid

Figure 6: Label distributions of the other three benchmark datasets. y-axes of Zhihu is on log-scale