Text2Tree: Aligning Text Representation to the Label Tree Hierarchy for Imbalanced Medical Classification
Authors: Jiahuan Yan
Haojun Gao
Zhang Kai
Weize Liu
Danny Chen
Jian Wu
Jintai Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: text mining
medical text representation
imbalanced text classification
TL;DR: Explore medical text classification by learning text representations from data imbalance and scarcity perspective and align text representations with the disease label hierarchy.

Abstract: Deep learning approaches exhibit promising performances on various text tasks. However, they are still struggling on medical text classification since samples are often extremely imbalanced and scarce. Different from existing mainstream approaches that focus on supplementary semantics with external medical information, this paper aims to rethink the data challenges in medical texts and present a novel framework-agnostic algorithm called Text2Tree that only utilizes internal label hierarchy in training deep learning models. We embed the ICD code tree structure of labels into cascade attention modules for learning hierarchy-aware label representations. Two new learning schemes, Similarity Surrogate Learning (SSL) and Dissimilarity Mixup Learning (DML), are devised to boost text classification by reusing and distinguishing samples of other labels following the label representation hierarchy, respectively. Experiments on authoritative public datasets and real-world medical records show that our approach stably achieves superior performances over classical and advanced imbalanced classification methods. Our code is available at https://github.com/jyansir/Text2Tree.

Text2Tree:
Aligning Text Representation to the Label Tree Hierarchy for Imbalanced Medical Classification
Jiahuan Yan', Haojun Gao², Kai Zhang", Weize Liu', Danny Chen", .Jian W Jintai Chen 1.5,+
'College of Computer Science and Technology. Zhejjang University, Hangzhou, China School of Medicine, Zhejiang University, Hangzhou, China
Pollytechnic Institute, Zhejiang University, Hangzhou, China
DDeparmeent of Computer Science and Engineering, University of Notre Dame, IN, USA
Computer Science Department, University of Illinois Urbana-Chamaa Chammaain,,,,,,, II, USA
(iyansir, joagh, zhangkai1999, weizeliu, wujian2000) @zju.edu.cn, dchen nd.edu, jtchen721 @ gggmail.com
Abstract
There have been various classical strategies for general imbalanced classification and long-tailed multi-label classification (Huang et al., 2021) machine learning studies. Such studies probably overly focused on data in rare categories (e.g.. by re-sampling (Chawla et al., 2002; Menardi et al., 2014), re-weighting (Kumar et al., 2010; Lin et al., 2017; Chang et al., 2017: Li et al., 2020; Wang al., 2022a), ensemble learning (Khosh- goftaar al., 2007; Liu al., 2020), data aug- mentation (Mariani et al., 2018; Chu et al.. 2020; Zada et al., 2022)) to alleviate distribution bias, or employed sophisticated learning paradigms such as contrastive learning (Wang et al., 2021), transfer learning (Ke et al., 2022), and prompt- tuning (Zhang et al., 2022)) learn fine-grained representations of hard cases. But, in medical text classification, many rare diseases lack sufficient data representation learning and the incidence rates vary greatly for different discases, which can- not be completely solved by the general imbalanced classification approaches since they overlook the underlying dependency of medical terminologies.
Deep learning approaches exhibit promising performances on various text tasks. How- ever, they are still struggling on medical text classification since samples often ex- tremely imbalanced and scarce. Different from existing mainstream approaches that fo- supplementary semantics with exter- nal medical information, this paper aims to rethink the data challenges in medical texts algo-
and present novel framework-agnen rithm called Text2Tree that only utilizes in- ternal label hierarchy in training deep learning models. We embed the ICD code tree struc- ture labels into cascade attention modules label representa- tions. Two new lcarning schemes, Similarity Surrogate Learning (SSL) and Dissimilarity Mixup Learning (DML), are devised to boost text classification by reusing and distinguish- ing samples of other labels following the la- bel representation hierarchy, respectively. Ex- periments on authoritative public datasets and real- wollw medical records show that our ap- proach stably achieves superior performances over classical and advanced imbalanced clas- sification methods. Our code is available at thub. wtw..th..........oooo....tt /eamm/yyyysrrrrextrree... https://gi thwb.
for learning hierarchy-awee
Although the current studies of medical text classification either leveraged the label descrip- tions (Chen and Ren, 2019; Zhou et al., 2021; Yang et al.. 2022) label dependency (Xie et al., 2019; Cao et al., 2020) for precise pairwise sample-label matching (Mullenbach et al., 2018), or incorpo- rated external knowledge sources (c.g., Wikipedia) to enrich semantic information (Prakash et al., 2017; Bai and Vucetic, 2019; Wang et al., 2022b), they did not explicitly cope with the data imbalance and scarcity issues. Therefore. imbalanced medical text classification is still an open challenge.
1 Introduction
Medical text classification widely recognized as an urgent yet challenging problem due to its ex- tremely imbalanced data distribution, large variety of rare labels (Johnson et al., 2016; Ziletti et al., 2022), and complicated label relationship (Tsai al., 2021; Vu al., 2021). Various downstream clinical tasks have been derived from this problem, including ICD coding (Mullenbach et al., 2018; Yuan et al., 2022; Yang et al., 2022) and automated diagnosis (Chen et al., 2020b), showeasing its po- tential values in modern clinical practice with ma- chine learning approaches (Berner, 2007).
Targeting this problem, this paper makes the first effort on learning medical text representations to resolve the data imbalance and scarcity issues
with the support of disease label dependency. As shown in Fig. 1, the right part demonstrates an example of CCD-10-CM codes (a clinical modi-
"Corresponding authors.

---

fication of ICD-10 codes) about pneumonia and Iypertensioon-rllleed discase dependeney in a tree structure. Among all the types of pneumonia, fun- gal pneumonia stands out as an exceptionally rare and tifeethreatening variant (Meersseman et al., 2007) that poses significant challenges to early de- tection (Morrell et al., 2005). Consequently, di- agnosed cases of fungal pneumonia are notably scarce, and we aim supplement our analysis with existing samples from the other diseases. An in- tuitive assumption is that two diseases with some common clinical presentations (c.g., symptom, dis- case course, treatment) can be used to compensate for each other. Naturally, we can associate fungal pneumonia (J1 6.8) with COVID-19 (J12.82), since both of them are respiratory system diseases (J00- 199) and pneumonia (J09-J18) while the diagnosed cases of COVID-I9 are plentiful. Similarly, chlamy- dial pneumonia (J16.0) shares a closer parent node (J16) and can be a better supplementary source, while some more dissimilar ones (e.g., circulatory system diseases, 100-199) are not so beneficial. Uti- lizing the tree hierarchy, we are able to access prior knowledge of the similarity between disease types, which is helpful in dealing with the data imbalance and scarcity issues.
formation flow along tree edges. Next, pairwise label similarity is calculated based on the label representations. Based on the similarity, samples from other labels will be treated as surrogate posi- tive anchors to provide extra contrastive signal by SSL, or apply mixup to give new samples by DML. The SSL branch learns to gather more information from more similar labels to explicitly reuse texts in representation learning following the label hi- erarchy, while DML generates and classifies hard cases, adversarially preventing excessive manifold distortion resulted from SSL.
Our main contributions are as follows:
We first explore the medical text representa- tion learning problem from the data imbalance and scarcity perspective, and propose the new Text2Tree algorithm that aligns text repre- sentations to the label tree hierarchy.
In contrast to previous methods that reused samples with the same labels, we propose SSL and DML to leverage samples from diverse la- bels facilitate representation learning based on the underlying label hierarchy.
Comprehensive experiments show that our Text2Tree algorithm stably outperforms ad- vanced framework-agnoss imbalanced clas- sification algorithms, without any external medical resource.
Motivated by this, we present new framework- agnostic algorithm that aligns text representation label tree hierarchy (called Text2Tree), which is tailored for better classification by incorporating an ICD code embedding tree to guide medical text representations. Text2Tree has three major com- ponents: (i) a hierarchy-aware label representation learning module (HLR), (ii) similarity surrogate learning (SSL), and (iii) dissimilarity mixup learn- ing (DML) approaches. Unlike general imbalanced classification methods that compensate a sample by reusing samples with the same disease labels or itself, (ii) and (iii) are designed to reuse samples from other labels under the specification of the la- bel hierarchy learned by (i). In this paper, the used "codes" refer to discase labels with underlying hi- erarchy, and we do not distinguish "code", "label", and "node".
2
Related Work
2.1 General Imbalanced Classification
Imbalanced classification refers to ubiquitous ma- chine learning problem in practical applications with long-tailed data distribution (Sun et al., 2009; Zhang et al., 2023). A major category of classical algorithms attempts to alleviate this problem from the data perspective, such as re-sampling (Menardi et al., 2014), weiieinht (Lin et al., 2017), ensem- ble learning (Liu et al., 2020), and data augmenta- tion (Chu et al., 2020; Zada et al., 2022). The core essence of these methods is to fix the skewed distri- bution or thoroughly harness the available data, and thus they can be easily adopted in both statistical machine learning and deep learning paradigms.
The Text2Tree training procedure proceeds as follows. First, cascade attention modules are built based on the prior structure of the medical ICD code tree. Only embeddings of labels on adjacent tree layers are interacted in each module, and label representations are computed layer by layer. Hence, a strict interaction fashion is kept to constrain in-
Another line of work seeks to address class im- balance the data representation level. A common assumption is that better feature representations make better classifiers (Wang et al., 2021), which is intuitive for deep learning models. Among these methods, prototype learning (Liu et al., 2019; Zhu

---

212.8 Adenovtral pra.moria
ICD-10 (ROOT)
212 22.5 Other vinal sne-moria
212 3 Vinil prsummnta (unspecited)
312,42 coronavitus disease 2019
J00-J99
100-199
115 Racheral preumonts
115.2 Preummore (Kiesstela preumontie)
315.5 Becterbol preumonie (unspecfted)
other infectious organisms
J15
J16
110
115
215.8 Chibmydia prcumenia
215.5 Pncumonia (other spocifed infectious
organisms, 6.g. Fungel pnoamonio)
Peceroscia edated IOCM
J12.0
J12.8
J15.2
J16.0
J16.8
110.0
115.0
115.1
118 Essental (priaryy) Ixpartension
Hypertensive heart disosse
Fungal
Essential
112 hypertensive chroric kidnay
112.0 With heart falure
111.9
J12.81
J12.82
pneumonia hypertension
renel disesse
SARS
COVID-19
112.5 Stage through stage chrons
kidney disease, or unspecited
113
hyportonsie
2.0 Label Similarity
115.2 Secondary hyportension (unspecfiad) Hyperionizorkkeed ICD- CM
Rure disease
Comman diseuse
0.0
Hierarchical ICD-10 codes on pneumonia related and hypertension related diseases (left) can be organized into tree structure (right). The code tree contains prior knowledge on disease label similarity that can guide data reuse from other labels. Here "similarity" a soft metric that can be computed according to the code tree.
Figure I:
and Yang, 2020) and contrastive learning (Wang et al., 2021; Kang et al., 2021) select reasonable chor samples for feature alignment; transfer learn- ing (Cui al., 2018; Kang et al., 2021; Ke al., 2022) acquires a better initial representation based on pre-training or samples from other domains. More recently, prompt-tuning (Zhang et al., 2022) is also trending technique to take advantage of representation ability of large pre-trained models.
et al., 2020) first introduced prior hierarchy knowl- edge with structure encoders for modeling label dependency in HTC, (Chen al., 2021) further performed label-text semantic matching in a joint embedding space to distinguish target labels from incorrect labels, (Wang et al., 2022d) incorporated a contrastive learning framework by masking unim- portant tokens to generate positive samples from the original ones, and (Wang et al., 2022c) used soft prompts to fuse label hierarchy into pre-trained models for better adaption to HTC. Recent stud- ies (Jiang et al., 2022) also used combination of local and global views take advantage of both types of approaches.
2.2 Hierarchical Text Classification
Hierarchical text classification (HTC) is chal- lenging sub-field of multi-label text classifica- tion (Wehrmann et al., 2018; Chen al., 2021; Wang et al., 2022c,d; Jiang et al., 2022). The labels HTC fall under different levels label tree, while medical text classification a flat classifica- tion problem (each label in ICD coding is a specific disease at the lowest level) though in this paper, an underlying label hierarchy exists based on the medical code tree.
Methodology
In this section, we separately describe the three key components of our proposed Text2Tree algorithm: hierarchy-aware label representation (HLR) learn- ing module, similarity surrogate learning (SSL), and dissimilarity mixup learning (DML).
HTC methods can be categorized into local and global approaches. The local ones (Wehrmann et al., 2018; Shimura et al., 2018; Banerjee et al., 2019) leveraged label hierarchical information to separately build a classifier for each label level in the label tree. Currently, the global series be- come prevalent for their better performance (Chen al., 2021). Methods of this type treated HTC multi-label text classification problem on all the nodes in the label tree, and the main concern is to propose effective frameworks for better hi- erarchy encoders and label representation. (Zhou
3.1 Hierarchy-aware Label Representation
Given prior label dependency (e.g., an ICD code tree as shown in Fig. 1), it is intuitive to encode hierarchy information with a graph encoder. Dif- ferent from the recent work on HTC (Wang et al., 2022d) that used complicated neural architectures (e.g., GraphFormer (Ying et al., 2021)), here we propose a cascade tree attention module intro- duce dependency bias among labels. Given a code tree of maximum level (we define the ROOT to be at level 0), we model a representation

---

No interaction
Interaction
Selective interaction
Used codes
Code 3128
Paren 312
Salecteo aaling Skmp 1
N Baisin
116
Salle
Y
Corde
ROOT
232
332.8
Wg
W₂
W,
K
Attention
RO.
Att.
J12
Art.
J12.0
representation
J12
J12.8
K
J16
Code 112.8
amsbeddding
Level 0-1
Level 1-2
Level 2-3
attention masks
attention masks
attention masks
(a) The computing process of cascade attention layers
(b) Computing 112.8 label representation from related labels
Figure 2: The HLR module consists of cascade attention layers that derive attention masks from the code tree hierarchy (left). The representation label contains information from itself, its parent and selected siblings (right). "Used codes Y defines the disease label set (at the lowest level of ICD tree) for classification.
code from its parent code p and sibling codes $ = {j pj) = pirent(i) = p.j # i}. We formulate the code representation h4 as follows:
ROOT are compulsory, and the ones between an L1 label and its siblings are seleetive. Similarly, the second attention layer only includes level-2 (L2) and L1 codes, and when we calculate the repre- sentation of an 1.2 label, using its parent (an L1 label) representation hp rather than embedding Ep (3) helps pass information from the higher levels (the K; = [k; kpp KJ],V, V; [ri, Ups Vyl,
9: = Wiger K; = WAe,, = W
ky = Whej. 0j = Ey = sigmoid(s,).
(2)
kp = Wkhp; Ppp = Wohp
(4) ROOT and the parent's siblings). There are L cas- cade attention layers in total. In Eq. (4), the key and value matrices of label are composed of the corresponding vectors of itself, its parent (com- pulsorily included) and siblings in J (selectively included by threshold clipping on sibling selectors cj). In Eq. (5), the final label representation h; is composed of attentively fetched information and its embedding Ca-
h: Attention (g: Kú, Vi) + Ci.
In Eq. (4), J Sn l, 0.5} is the selected sibling subset, K, and V, are key and value ma- trices composed of the corresponding vectors in Eq. (2). Inspired by a recent Transformer architec- ture with selective feature interactions (Yan et al., 2023), we design a sibling selector Ej in Eq. (2) by performing activation on learnable parameter sj. indicating whether the sibling j should inter- act with node in the attention module to achieve data-driven sibling aggregation. The core of the whole process to attentively fetch information from the parent label and selected sibling labels to express the representation of label i (see Fig. 2). Here e denotes the learnable embedding of label i, and W. q, W.. and W. represent transformations to the query. key, and value vectors in attention mechanism. Note that in Eq. (3), we use the parent node representation hp generate its key and value vectors. The reason for this that process la- bel representations different levels layer by layer (see the left part of Fig. 2). The first attention layer only contains level-1 1 (L1) labels and the ROOT, where the interactions between LI labels and the
We define the representation and embedding the ROOT as equal (hROOT = éRooT) since the ROOT has no parent and siblings. The straight- through trick (Bengio et al., 2013) is used to solve the undifferentiable issue of selecting siblings in Eq. (4). Before training, all label embeddings be randomly initialized or assigned with average of BERT token embeddings if label texts are given.
3.2 Similarity Surrogate Learning
Contrastive learning has been extensively validated as effective representation learning method in the image (Chen al., 2020e; He et al., 2020) and text (Gunel et al., 2021) domains. Prevailing con- trastive methods can be roughly categorized into self sulf-super and sully-suer ones (Khosla et al., 2020). The basic idea of both types is to

---

slightly adjust the distance between samples and anchors for better data representation space. A classical form of the supervised contrastive learn- ing (SCL) loss is:
further extend Eqq (6) based on the similarity score:
exp(z: 25/T)
f(i,j) = Sim(i, log
Σ exp(zi za/T)
aEA(w)
exp(zi 2p/T) log Σ exp(zi 24 za/T) PCP())
Lgst Σ EET
Σ f(i,a), (8)
Leal= = iEI
Σ
Σ
Σ Sim(i, a) aEA(1)
P(i)
aεA(()
(cA(s)
(6)
where f(i.j) is a soft contrastive term between sample and its surrogate anchor j. which encour- ages the algorithm to contrastively reuse samples with high label similarity Sim(i, Label repre- sentations h are learnable in Sec. 3.1, and thus the surrogate anchor selection is data-driven.
where i E I = [I,....N} denotes the sample index in a batch (or dataset), A(i) = / {i).
P(i) = {pl p E A( = Wi 2; is the en- coded features of sample and c R is scalar temperature. In this paper, we use the hidden state of the first token (CLS)) after the BERT encoder (i.e., 2j (z) Eq. (6) intuitively ex- tends positive anchors from self- gelf-generated samples an unsupervised manner to samples with the same label in a supervised manner, thus effectively leveraging label information (Khosla et al., 2020). But, rare labels are common in medical text clas- sification. For example, in the top-100 frequent labels of the PubMed dataset, the top-10 labels ac- count for 40.2% of the total label amount, while the rarest 50 labels occupy only 23.8% (see Fig. Appendix A). Among 8,692 unique ICD-9 codes of the MIMIC-III dataset (Johnson et al., 2016), 4,115 codes occur less than 6 times (Yang et al., 2022). In our real-world datasets, there is also a large amount of rare diseases (see the figures in Appendix A).
Actually, our SSL loss (Eq. (8)) can be viewed as a combination of the SCL loss and an extra term:
Last Σ
Σ
S(i
Σ
Si
EEI
PEP(()
PCP(s)
Σ Σ Sim(i,b)
Σ f(i,b)
ICl
bE
BEB(1)
=Teeft Teright:
(9)
:. :: E P()), Up = Vi
..iimii, = Σ
Sim(i, p) (P()),
pEP(()
: = LLse = Lest = Loo! + Lrights
where B(i) A(1) \ P(i) denotes samples with diffcrent labels from sample Obviously, our SSL loss leverages samples with other labels to contribute contrastive signal (since Lright > 0 (Eq. (9))), and such extra signal is likely to relief the data scarcity issue on rare discase labels by alle- viating the sparsity of optimization signal. Overall, Lg. (9) indicates a progressive relationship that SSL a generalized form of SCL, with extra flexibility of user-defined sample similarity function (Eq. (7)) based on the specific real-world application. SSL is approximately equivalent to SCL for a sample in common labels, while for a sample in the low re- source scenario (e.g.. rare disease, online learning, (7)
Samples with rare labels are unlikely to match any positive sample in batch (the value P()) tends to
be zero in Eq. (6)), making it intractable to acquire contrastive signal for such samples.
To tackle this problem, we propose Similarity Surrogate Learning (SSL) based on the label simi- larity, which treats any other sample in a batch as potentially positive surrogate anchor. Specifically, we define the similarity score between samples i and j based on their label representations:
h byg
Sim(i,j) byy 12
small batch size for limited hardwares), SSL makes contrastive signal tractable for such hard cases.
where hy, is the label representation of sample i. For multi-label classification, we calculate the av- erage representation hy, of all the codes, as:
3.3 Dissimilarity Mixup Learning
As typical interpolation- augmentation tech- nique, Mixup (Zhang al., 2018) has been widely adapted to NLP settings (Chen al., 2020a; Sun et al., 2020) and proved to be an effective data- adaptive regularization to reduce overfitting (Zhang
1
Σ hy-
hy,
Y
WCY
Here, we simply choose the cosine similarity. We

---

PubMed, we use a recent Kaggle version since it well sorted and contains 50K research articles from the PubMed repository, and we retain the top- 100 3-level MeSH labels (i.e., each level of MeSH ID "C01.784" is "C", "C01" and "C01.784"). Der- matology and Gastroenterology are two datasets cleaned from outpatient records of the two largest departments in a top hospital during the last three years, and Inpatient is cleaned from the inpatient records from famous healthcare institution. All the three real-world datasets are annotated with ICD-10 codes, and two clinical graduates cleaned them according to the discipline in Appendix B. For data splitting, we use 20% samples as test set, 16% as evaluation set, and the rest 64% training set. For each multi-class dataset, we further keep the ratios of each label in the three sets the same. al., 2021). We exploit this strategy to reuse in- formation of samples with less similar labels by introducing Dissimilarity Mixup Learning (DML), Different from the ordinary Mixup strategy that samples weights from the Beta distribution (A ~v Beta(a, a)), we directly assign the weights accord- ing to the similarity score (Eq. (7)), as:
À 0.5(1 + Sim(i.))) j)), 2 = Az + (1 -))z
(10)
ÿ - Ay, + (1 - A)ys,
where 2 is the same as that in Sec. 3.2, and yi is in the one-hot representation. DML driven by label representations learned in Sec. 3.1, and tends to mix a bigger portion of more dissimilar samples, so as to generate hard samples. Notably, our DML needs no hyperparameter compared to the ordinary Mixup, and is more user-friendly.
The dataset statistics are given in Table More detailed dataset information is provided in Ap- pendix A.
3.4 The Overall Training
The metrics Macro-FI and Micro-FI are used for measuring the classification results.
Prediction is made based on BERT pooler and a simple fully connected layer after mixing the encoded features. The final loss function is the combination of classification loss and the SSL loss:
Dataset
N
Y
Avg((,)
Avg(
MIMIC-III
11.4K
33
450.61
4.01
PuhMed
50.0K
100
122.88
8.52
Dermatology
20.5K
59
44.97
Gastroenterology
35.0K
58.31
ý FC((en)Pooler())),
Inpatient
2.6K
69.22
(1 Lop + ALsst:
(11)
Table 1: Dataset statistics. N is the number of samples. |Y| the number of classes, and Avg(l.) and Avg(lva the average token length and average label amount per sample, respectively.
where Lee is cross entropy loss for multi-class clas- sification and is binary cross entropy loss for multi- label classification, and À hyperparameter con- trolling the loss weight. In backward propagation, we detach the gradient in Eq. (10) and only op- timize HLR through Eq. (8). We illustrate and discuss backward gradient flow in Appendix D.
Baselines. For systematic and fair comparison, we utilize various aggossc imbalanced classification algorithms and advanced hicrarchi- cal text classification (HTC) methods. 1) Fine- tune: The ordinary finetune paradigm. 2) Re- sampling: A classical method to alleviate distri- bution bias; we use RandomOverSamppl (ROS) implemented the Imbalanced learn Python pack- age for multi-class tasks and distribution bal- ance loss (DBLoss) designed by (Huang et al., 2021) for multi-label tasks, because the ordinary re-sampling methods are not effective. 3) Re- weighting: We choose FocalLoss (Lin et al., 2017) since it is typical dynamically weighted loss for hard cases. 4) Prevailing contrastive learning: In- cluding self contrastive learning (SelfCon) and su- pervised contrastive learning (SupCon) (Khosla et al., 2020). 5) MixUp (Sun et al., 2020): A typi- cal interpolation- based data augmentation method.
4 Experiments
4.1 Experimental Setup
Datasets and Evaluation Metrics. We experi- ment on two authoritative medical text datasets: MIMIC-III (Johnson et al., 2016) and PubMed and three in-house real-world datasets: Dermatol- ogy. Gastroenterology, and Inpatient. Here, the first two public datasets are for multi-label classifica- tion, and the three in-house ones are for multi-class classification. In experiments, for MIMIC-I1I, we use only 33 discase labels in the top-50 version, and convert ICD-9 codes into ICD-10 codes. For maliilaol--ext-clasffffcttto

---

HTC: We choose HGCLR (Wang al., 2022d) and HPT (Wang et al., 2022c) because they are recent state-of-the-art methods in global HTC. Note that some baselines are supplementary rather than competitive counterparts because they can be jointly deployed with Text2Tree in practice (c.g., re-sampling).
performs badly on the small-sized Inpatient. Fur- ther, we see the best performance of SupCon on Inpatient, which may be due to the small data size with a large label space and detailed text seman- tics. The Inpatient dataset has the smallest scale but the most label amount and the longest average token length among the three multi-class datasets, and therefore is hard to directly learn ideal data representations. SupCon is able to precisely use label information to exploit fine-grained semantics in long texts, while Text2Tree is highly dependent on the quality of the learned label representations. which is inferior on the datasets in which all the labels are scarce (see Fig. 9, Appendix A).
Implement Details. For all the algorithms, we use bert-base-uncased for the English datasets and for the Chinese ones. We imple- ment the experiment code with PyTorch on Python 3.8. All the experiments are run on NVIDIA RTX 3090. The optimizer CLR according to (Wang et al., 2022d) and is AdamW (Loshchilov and Hutter, 2018) for the oth- ers with the default configuration except for the learning rate. For hyperparameter tuning, we use gird search for each method with detailed hyperpa- rameter spaces in Appendix C. Here we define the search space of HGCLR according to the recom- mended settings in (Wang al., 2022d), and use default settings for HPT in (Wang et al., 2022c). We select hyperparameters with respect to Macro- Fl and Micro-F1 separately on the evaluation set. Due to the differences of the HTC label system (i.e., if predict "C01.784" MeSH label with HTC meth- ods, the training framework should predict "C", "C01" and "C01.784" simultaneously), we consider extra higher level labels in HGCLR training loss and ignore them during metric calculation. The maximum token length is 512 for MIMIC-III and is 128 for the rest datasets. We uniformly adopt early stop with 10 epochs for fine-tuning based on the evaluation metric.
Adam for HG-
As expected, we find that HGCLR and HPT usu- ally perform inferiorly. As advanced hierarchical text classification (HTC) methods, they need to make prediction from all the levels in framework training, and thus would incur more redundancy to classify extra labels in flat text classification. Both of them perform relatively better the multi-class datasets (i.e., the three real-world datasets) than on the multi-label ones (i.e., MIMIC-III and PubMed), since samples in multi-classification contain less higher level labels and thus the impact is weakened. Compared to HTC methods, our Text2Tree has more diverse contrastive signal because we treat any other sample as a potentially positive anchor. Besides, they takes more computational resource and training time, because HGCLR calculates clas- sification signal on both the original samples and masked ones, doubling the training batch size, and HPT adopts the graph neural network for hierachy injection, which is computationally incfficient com- pared to attention-based HLR.
4.2 Main Results and Analyses
Although in this paper we experiment with Text2Tree on flat text classification in the medical domain, it can be easily extended to HTC settings or other general domains once the hierarchical label dependency is given (c.g., code tree or graph).
The main results are presented in Table 2. Our Text2Tree algorithm achieves the best Macro-F1 scores on four datasets, and stably ranks in the top 3 in both Micro-F1 and Macro-Fl across the five datasets, while the performances of the other meth- ods are highly unstable. This demonstrates that incorporating label hierarchy to guide data reuse from other labels helps medical text representations essentially, especially for texts from samples of rare diseases (it helps Macro-F1).
4.3 Ablation Study
To further validate the effectiveness of each key component in Text2Tree, we perform ablation study by separately removing one of 1) Similar- ity Surrogate Learning (SSL), 2) Dissimilarity Mixup Learning (DML), and 3) the Hierarchy- aware Label Representation Learning (HILR) mod- ule. Table presents the ablation study results on two datasets (see Appendix E for the full re- sults). Note that we detach the gradient in Eq. (10)
We find that MixUp performs well on PubMed, which is probably attributed the data scale, since random interpolation more likely to produce more diverse data representations when the data source is sufficient. Hence, we can also see that it

---

MIMIC-III
PubMed
Dermato.
Gastro,
Inpatient
Method
Maero
Miero
Macro
Micro
Macro
Miero
Macro
Micro
Macro
Micro
Finetune
42.22
51.65
57.17
65.30
53,93
56.31
47.64
50.52
71.22
71.12
ROS (Menardi et ul., 2014)
53.40
53.76
44.52
45.08
72.44
72.28
DBLos (Huang et al., 2021)
44.27
53.69
56.97
64.65
LocalLoss (Lin ct ul., 2017)
43.39
51.95
57.04
65.08
53.49
55.46
47.02
49.35
72.77
72.28
SelCCon (Khosla et al., 2020)
41.28
50.47
57.71
65.58
54.04
57.07
47.68
49.94
70.69
68.65
SupCon (Khosla et al., 2020)
42.21
52.02
57.53
65.43
54.69
56,83
47.66
50.90
73.44
73.43
MixUp (Sun et al., 2020)
42.11
51.72
58.29
65.82
54.58
56.16
48.40
49.64
70.98
70.79
HIGCLR (Wang et al., 2022d)
41.59
51.53
55.77
64,50
53.34
56.50
48.53
51.70
71.07
71.29
HPT (Wang et al., 2022c)
41.32
51.11
57.64
65.73
53.82
56.02
47.58
49.51
73.08
72.96
Text2Tree (ours)
44.75
53.60
58.70
66.10
55.23
$7.27
48.88
51.26
73.01
72.77
Table 2: Medical text classification results. The best results marked in bold while the second best ones are underlined. "Marco" and "Micro" are for Macro-FI and Micro-F1, respectively.
Ablation Text2Tree
MIMIC-I11
PubMed
First Level Code of MeSH AxXXXX
DXLAXX
44.75/53.60
58.70/66.10
EXLAXX
NooXXX
w/o SSL
43.63/52.90
57.15/65.27
w/o DML
43.64/51.99
57.25/65.53
w/o HLR
43.07/52.47
57.90/65.57
Table 3: Ablation study results on the MIMIC-III and
PubMed datasets. We report macro-Fl/miccro-f re- sults when separately removing an individual compo- nent of Text2Tree. Ablation study results on all the five datasets given in Appendix E.
Figure 3D t-SNE visualization of the label represen- tations on the PubMed dataset, before (left) and after (right) the Text2Tree training. Labels with the same first-level codes are in the same color.
(see Sec. 3.4), and thus HLR optimization is solely based Eq. (8) in SSL. In the "w/o SSL" group, we retain the gradient in Eq. (10) to allow the HLR module to be optimizable. In the group "w/o HLR", there no label representation, and thus we choose the combination of supervised contrastive learn- ing (SupCon) and the ordinary mixup (MixUp) as substitution.
4.4 Visualization of Label Representations
To further illustrate the effectiveness of our HLR module, visualize the label representations the PubMed dataset (see Fig. 3). Here we choose four largest first level groups in 100 labels (McSH codes starting with "A", "D", "E", "N") and sim- ply assign the same color to labels of the same group. Starting with code (label) representations calculated from randomly initialized code (label) embeddings (the left figure), the HLR module in- deed captures hierarchical bias with simple cascade attention modules in Sec. 3.1, and learns to cluster similar medical labels (the ones under the same first level). The SSL and DML processes are guided with reasonable label representations (the right fig- ure) to reuse information from the other labels.
can be seen that when keeping HLR, dropping either SSL or DML will harm the performances. is hard to judge which SSL and DML is more sig- nificant since they exhibit different levels of Macro- Fl or Micro-F1 decline across the five datasets.
When deploy SupCon and MixUp jointly (the w/o HLR group), it is interesting observe that this combination usually does not exceed the bet- ter dingle-method performance (the combination worse than the better method across four datasets, see Appendix E and Table 2), which may be at- tributed to the incompatibility between MixUp and SupCon. The ordinary mixup randomly interpo- lates samples and may distort the fine-grained data representation space learned by SupCon, while our SSL and DML are respectively designed to pre- fer samples with more and less similar labels to alleviate this incompatibility, thus making HLR a necessary part of TexeeTree.
4.5 Effeet on Rare Labels
To analyze the effect of Text2Tree on rare medi- cal labels, we present Macro-F1 differences com- pared the ordinary finetune on PubMed label groups when deploying different baselines (see Fig. 4). Overall, all the validated algorithms can boost performance on the rarest label group (labels 1~10), while DBLoss and FocalLoss are prone to

---

Baselines
designed SSL and DML representation learn- ing methods require precise label representations. To provide such precision, we tailor the HLR mod- ule with lightweight cascade attention to introduce label hierarchy bias according to the code tree.
CoLoss
rocallesa
Supton
Toaz
A Macro-FF
References
Tian Bai and Slobodan Vucetic. 2019. Improving med- ical code prediction from clinical text via incorpo- rating online knowledge sources. In WWW, pages 72-82.
1-12
11:22
21-33
22-44
41-30
52-90
Label ID (in increasing frequency order)
Figure 4: Macro-Fl changes on label groups (grouped by increasing- frequency ordered label IDs) on PubMed when adopting different algorithms to the ordinary fine- tune paradigm. Full results are given Appendix
Siddhartha Banerjee, Cem Akkaya, Francisco Perez- Sorrosal, and Kostas Tsioutsiouliklis, 2019. Hierar- chical transfer learning for multi-label text classifica- tion. In ACI., pages 6295-6300.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1rXiv:1308..55
sacrificing generalization ability on more frequent labels (see their F1 changes on labels 11-100).
SupCon is likely to attain conservative improve- ment on rare labels (labels 1-50) but fail to excel in common cases (labels 51-100). MixUp and Text2Tree show strong generalization on all the cases, while Text2Tree provides more stable and better Fl gains in most cases and is the best choice for the rarest labels.
Eta S Berner. 2007, Climical Decision Support Systems, volume 233. Springer.
Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Sheng- ping Liu, and Weifeng Chong. 2020. HyperCore: Hy- perbolic and co-graph representation for automatic ICD coding. In ACL, pages 3105-3114.
Haw-Shiuan Chang. Erik Learned Miller, and Andrew McCallum. 2017. Active bias: Training more accu- rate neural networks by emphasizing high variance samples. In NeurIPS.
5 Conclusions
In this paper, we proposed Text2Tree, new -aanosi learning algorithm that aligns text representations to discase label hierarchy with novel learning methods of contrastively reusing data, so as to resolve the data imbalance and scarcity issues in medical text classification. Com- pared with various state-of-the-art general imbal- anced text classification methods, the superiority of Text2Tree was verified on 5 real-world datasets. We believe that Text2Tree will serve as strong baseline in imbalanced medical text classification and could be extended to other domains when cor- responding prior label dependency is provided.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall. and W Philip Kegelmeyer. 2002. SMOTE: Synthetic minority over-sampling technique. Journal of Artifi- cial Inielligence Research, 16:321-357.
Haibin Chen, Qianli Ma, et al. 2021. Hierarchy-ave aware label semantics matching network for hierarchical text classification. In ACL, pages 4370-4379.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020a. Mix- Text: Linguistically-nnfone interpolation of hidden space for semi-supervised text classification. In ACL, pages 2147-2157.
Jun Chen, Xiaoya Dai. Quan Yuan, Chao Lu, and Haifeng Huang. 2020b. Towards interpretable clin- ical diagnosis with Bayesian network ensembles stacked on entity-aware CNNs. In ACL, pages 3143- 3153.
Limitations
While Text2Tree is able to exploit the label hi- erarchy to align medical text representations and achieve stable improvement among various classi- cal and advanced imbalanced classification meth- ods on the validated medical text classification datasets, one limitation that Text2Tree explic- itly needs prior label dependency to work, com- pared to the other general algorithms. Also, the
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020c. simple framework for contrastive learning of visual representations. In ICML. pages 1597-1607. PMLR
Yuwen Chen and Jiangtao Ren. 2019. Automatic ICD code assignment utilizing textual descriptions and hierarchical structure of ICD code. In BIBM, pages 348-353. IEEE

---

Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. 2020. Feature space augmentation for long-taid oongtailedd data. In ECCV, pages 694-710. Springer.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang. Boqing Gong, and Stella X Yu. 2019. Large-scale long-tailed recognition in open world. In CVPR, pages 2537-2546.
Yin Cui, Yang Song, Chen Sun, Andrew Howard, and Serge Belongie. 2018. Large scale fine-grainet cate- gorization and domain-specific transfer learning. In CVPR, pages 4109 4118.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In ICLR.
Giovanni Mariani, Florian Scheidegger, Roxana Istrate, Costas Bekas, and Cristiano Malossi. 2018. BAGAN: Data augmentation with balancing GAN. arXiv preprint ar3.v:1803.09655.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning, In ICLR.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast unsu- pervised visual representation learning. In CVPR, pages
Wouter Meersseman, Katrien Lagrou, Johan Maertens, and Eric Van Wijngaerden. 2007. Invasive aspergillo- the intensive care unit. Clinical Infectious Diseases, 45(2):205-216.
Yi Huang, Buse Giledereli, Abdullatif Köksal, Arzucan Özgür, and Elif Ozkirimli. 2021. Balancing meth- ods for multi-label text classification with long-tailed class distribution. In EMNLP, pages 8153-8161.
Giovanna Menardi et al. 2014. Training and assess- ing classification rules with imbalanced data. Data Mining and Knowledge Discovery, 28:92-122.
Matthew Morrell, Victoria J Fraser, and Marin H Kollef. 2005. Delaying the empiric treatment of Candida bloodstream infection until positive blood culture re- sults are obtained: potential risk factor for hospital mortality. Antimicrobial Agents and Chemotherupy. 49(9):36403364
Ting Jiang. Deqing Wang, et al. 2022. Exploiting global and local hierarchies for hierarchical text classifica- tion. In EMNLP, pages 4030-4039.
Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. MIMIC-III. freely acces- sible critical care database. Scientific Data, 3(1):1-9.
James Mullenbach, Sarah Wiegreffe, Jon Duke, Jimeng Sun, and Jacob Eisenstein. 2018. Explainable predic- tion of medical codes from clinical text. In NAACL, pages 1101-1111. Association for Computational Linguistics (ACL).
Bingyi Kang. Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. 2021. Exploring balanced feature spaces for representation learning. In ICLR.
Aaditya Prakash, Siyuan Zhao, Sadid Hasan, Vivek Datla, Kathy Lee, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2017. Condensed memory networks for clinical diagnostic inferencing. In AAAI.
Zixuan Ke, Mohammad Kachuee, and Sungjin Lee. 2022. Domain- Domain-aware aware contrastive knowledge transfer for multi-domain imbalanced data. In WASSA, pages 25-36.
Kazuya Shimura, Jiyi Li, and Fumiyo Fukumoto. 2018. HFT-CNN: Learning hicrarchical category structure for multi-label short text categorization. In EMNLP. pages 81 1-816.
Taghi M Khoshgoftaar, Moiz Golawala, and Jason Van Hulse. 2007. An empirical study of learning from imbalanced data using random forest. In ICTAI, volume 2. pages 310-317. IEEE
Lichao Sun, Congying Xia, et al. 2020. Mixup- Transformer: Dynamic data augmentation for NLP tasks. In COLING, pages 3436-3440.
Prannay Khosla, Piotr Teterwak, et al. 2020. Supervised contrastive learning. In NeurIPS.
M Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In NeurIPS.
Yanmin Sun, Andrew KC Wong, and Mohamed S Kamel. 2009. Classification of imbalanced data: A review. IJPRAI, 23(04)6687-119
Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2020. Dice loss for data- imbalanced NLP tasks. In ACI. pages 465-476.
Shang- Tsai, Chao- Wei Huang, and Yun-Nung Chen. 2021. Modeling diagnostic label correlation for auto- matic ICD coding. In NAACL, pages 4043-4052.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In ICCV, pages 2980-2988.
Thanh Vu, Dat Quoe Nguyen, and Anthony Nguyen. 2021. A label attention model for ICD coding from clinical text. In IJCAL, pages 3335-3341.
Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tie-Yan Liu. 2020. Self-paces ensemble for highly imbalanced massive data classification. In ICDE, pages 841-852. IEEE.
Cheng Wang, Jorge Balazs, György Szarvas, Patrick Ernst, Lahari Poddar, and Pavel Danchenko. 2022a. Calibrating imbalanced classifiers with focal loss: An empirical study. In EMNLP, pages 145-153.

---

Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang. 2021. Contrastive learning based hybrid net- works for long-tailed image classification. In CVPR, pages 943-952.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. 2023. Deep long sail-tailed learning: A survey. TPAMI.
Jie Zhou, Chumping Ma. Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gong- shen Liu, 2020. Hierarchy-awan global model for hierarchical text classification. In ACI,, pages 1106- 1117.
Tao Wang. Linhai Zhang, Chenchen Ye, Junxi Liu, and
Deyu Zhou. 2022b. A novel framework based on medical concept driven attention for explainable med- ical code prediction via external knowledge. In ACL, pages 1407-1416.
Tong Zhou, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Kun Niu, Weifeng Chong, and Shengping Liu, 2021. Automatic ICD coding via interactive shared representation networks with self-distillation mecha- nism. In ACI, pages 5948-5957.
Zihan Wang, Peiyi Wang, et al. 2022c. HPT: hierarchy- aware prompt tuning for hierarchical text classifica- tion. In EMNLP, pages 3740-3751.
Zihan Wang, Peiyi Wang. et al. 2022d. Incorporating hierarchy into text encoder: A contrastive learning approach for hierarchical text classification. In ACL. pages 7109-7119.
Linchao Zhu and Yi Yang. 2020. Inflated episodic mem- ory with region self-attention for cong-tailed visual recognition. CVPR, pages 4344 4353.
Angelo Ziletti, Alan Akbik, Christoph Berns, Thomas Herold, Marion Legler, and Martina Viell. 2022. Medical coding with biomedical Transformer ensem- bles and zero/few-shot learning. NAACL, pages 176-187.
Jonatas Wehrmann, Ricardo Cerii, and Rodrigo Bar- 2018. Hierarchical multi-label classification net- works. ICML, pages 5075-5084.
Xiancheng Xie, Yun Xiong, Philip S Yu, and Yangyong Zhu. 2019. EHR coding with multi-scal feature at- tention and structured knowledge graph propagation. In CIKM, pages 649-658.
Jiahuan Yan, Jintai Chen. Yixuan Wu, Danny Z Chen, Jian Wu. 2023. T2G-Former: Organizing tabular features into relation graphs promotes heterogeneous feature interaction. In AAAL
Zhichao Yang. Shufan Wang, Bhanu Pratap Singh Rawat, Avijit Mitra, and Hong Yu. 2022. Knowledge injected prompt based fine-tur mee-tunin for multi-label few- shot ICD coding. arxiv preprint arXiv:2210.03304
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. 2021. Do Transformers really perform badly for graph representation? In NeurIPS.
Zheng Yuan, Chuanqi Tan, and Songfang Huang. 2022. Code synonyms do matter: Multiple synonyms matching network for automatic ICD coding. In ACL, pages 808-814.
Shiran Zada, Itay Benou, and Michal Irani. 2022. Pure noise the rescue of insufficient data: Improving im- balanced classification by training random noise images. In JCML, pages 25817-25833, PMLR.
Chen Zhang, Lei Ren, Jingang Wang, Wei Wu, and Dawei Song. 2022. Making pretrained language models good long-tailed learners. In EMNLP, pages 3298-3312.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paz. 2018. Mixup: Beyond empirical risk minimization. In ICLR.
Linjun Zhang. Zhun Deng, et al. 2021. How does Mixup help with robustness and generalization? In ICLR.

---

A Detailed Dataset Information
C Baseline Settings
Details of used datasets shown in Table 4. We present label distributions of PubMed (Fig. 5), MIMIC-III (Fig. 6). Dermatology (Fig. 7), Gas- troenterology (Fig. 8) and Inpatient (Fig. 9). In- patient simulates extreme scarcity of medical data, where the most discases have less than 30 records.
C.1 Implementation
For ROS we use the version implemented in Imbalanced-learn python package. For DBLoss, MixUp and HGCLR reuse the implementation in the original paper (Huang al., 2021; Sun al., 2020; Wang et al., 2022d). For FocalLoss we ex- tend the multi-class version in (Lin et al., 2017) to the multi-label version for MIMIC- and PubMed. For SelfCon and SupCon, we reuse the implemen- tation in (Khosla et al., 2020) and adapt the code from image classification task to text classification scenario.
B Data Clean Instructions
We introduce three real-world electronic medical record (EMR) datasets, including two outpatient record datasets and inpatient record dataset. The outpatient records consist of 786,923 records from the Gastroenterology department and 635,302 records from the Dermatology department of top hospital. The inpatient records are sourced from a healthcare institution, with total of 9,032 records. To ensure the quality of the textual data, the follow- ing records are excluded:
C.2 Hyperparameter Tuning
For DBLoss we use recommended setting in (Huang et al., 2021), we select hyperparame- ters of baselines with grid search on the hyper- parameter spaces provided in the following tables (Table 5- Table 9). We set learning rate space for all baselines to { 5e - 6, le 5, 3e - 5, 5e 5}. We use batch size of 16 for MIMIC-III and 64 for the rest datasets.
Duplicated records: Records with repetitive content are removed to avoid redundancy in the dataset.
D Data and Gradient Flow in Text2Tree
Trivial records: Records that contain simple follow-up visits or medication prescriptions, which provide valuable information, such as "no change in medical history", "stable condition" "continuing treatment" elim- inated.
Fig. 10 illustrates data flow and gradient flow our used Text2Tree. Detaching gradient from DML is an empirical choice, for both DML and SSL can impact HLR, we concerned gradients from both will destabilize the learning process and select the best gradient strategy by macro-F1. We provide results on different graident policies in Table 10.
Label leakage: Records with descriptions that inadvertently reveal the diagnostic labels, such as "gastritis discovered during routine check-up". are excluded to prevent potential label leakage.
E Additional Results
Table 11 reports results of ablation study on the five datasets. Fig. 11 presents Macro-F1 changes in all the label groups of PubMed for different baselines.
Records with a text length less than 15 characters are discarded as they lack enough information.
The main diagnosis in each medical record ex- tracted as the label and mapped to the ICD-10 coding system. Descriptions related to COVID- 19, which were requested during the pandemic but lack relevant diagnostic information, are removed. Due to the evident long-tail distribution observed in the Gastroenterology and Dermatology records. only diseases with frequency greater than 50 in outpatient records and 20 inpatient records are included datasets.

---

Dataset
M
Avgll.)
Avg(gg))
dev
C.S.
Lang-
MIMIC-II
11368
450.61
4.01
8066
1573
1729
multi-label
ICDI0
English
PubMed
50000
100
122.88
8.52
32000
8000
10000
multi-label
McSI1
English
Dermatology
20522
44.97
13103
3256
4163
multi-class
ICD10
Chinese
Gastroenterology
34952
35
58.31
22351
5574
7027
multi-class
ICD10
Chinese
Inpatient
2603
98
69.22
1627
370
606
multi-class
ICD10
Chinese
Table 4: Detailed dataset statistics. "C.S." means code system.
tap-11- 11-40 labels
36,0%
the neral
top-10 babek
Melit Label
Figure 5: Data distribution of PubMed. The occurrence frequency of top-10 10 labels counts for 40.2 % in total top- 100 labels, while the rarest labels only account for 23.9 %.
Parameter
Search space
a
(0.25, 0.5, 0.75] (0.5, 1.0, 3.0)
iterations 4 x 3 x = 48
Table 5: Hyperparameter space for FocalLoss.
Parameter
Search space
lemperature
{0.5, 1.0, 2.0, 3.0} (0.001, 0.005, 0.01, 0.05)
À
# iterations
x x 4= 64
Table 6: Hyperparameter space for SelfCon and SupCon. À is hyperparameter in controlling the loss weight (similar to Eq. (I1)).
Parameter
Search space
temperature
{0.5, 1.0, 2.0,3.0) (0.001, 0.005, 0.01, 0.05]
iterations
4 x 4
Parameter
Search space
(0.1, 0.5, 1.0, 2.0, 8.0,
Table 9: Hyperparameter space for Text2Tree.
# iterations
x 6= 24
Table 7: Hyperparameter space for MixUp. a the hyperparameter in Beta distribution.
Parameter
Search space
[0.005, 0.01, 0.02, 0.05] {0.1, 0.3, 0.5, 1.0) 1.0
X
temperature
# iterations
4 x 4 = 64
Table 8: Hyperparameter space for HGCLR. and are threshold and contrastive loss weight.

---

ICD-10 CM Label
Figure 6: Data distribution of MIMIC-IIL We only use top-50 labels and retain 33 disease labels in them.
KCD-10-CM
Figure 7: Data distribution Dermatology. We cleaned 59 common diseases dermatology department.
ICD-10-CM Labol
Figure 8: Data distribution of Gastroenterology. We cleaned 35 common discases in gastroenterology department.
Ablation
MIMIC-III
PubMed
Dermato.
Gastro.
Inpatient
det DML (baseline)
44.75/53.60
58.70/66.10 10
55.23/57.27
48.88/51.26
73.01/72.77
det SSL
44.36/53.72
58.32/66.23
54.31/56.55
48.62/51.25 47.92/51.00
72.18/72.77
det
44.13/53.38
57.87/66.13
54.12/57.84
72.83/72.28
Table 10: Results of different gradient policies on all the five datasets. det indicates gradient detach.

---

ICD:10-CM Label
Figure 9: Data distribution of Inpatient. There are 98 diseases in the dataset of inpatient records, and the most diseases only have than 30 records, leading to scarcity issue each label.
Backward gradient flow
DNN
X
SSL
Loss
Forward data flow
HLR
Y
DML
ICD 10
detached
Figure Forward data and backward gradient illustration Text2Tree. We empirically detach the gradient from DML for stable learning.
Ablation
MIMIC-III
PubMed
Dermato.
Gastro.
Inpatient
Text2Tree (baseline)
44.75/53.60
58.70/66.10
55.23/57.27
48,88/51.26
73.01/72.77
w/o SSL
43.63/52.90
57.15/65.27
54.53/56.55
48.34/50.80
71.24/71.45
w/o DML
43.64/51.99
57.25/65.53
54.31/57.89
47.51/51.00
72.35/72.61
w/o HLR
43.07/52.47
57.90/65.57
53.95/55.22
47.19/50.29
71.50/71.78
Table 11: Ablation results all the five datasets.

---

Baselines
DBLoss
FocalLoss
MixUp
2
SupCon
Text2Tree
Macro-F1 (%)
1-10
11-20
21-30
31-40
41-50
51-60
61-70
71-80
81-90
91-100
Label ID (in increasing frequency order)
Figure 11: Full version Macro-Fl changes on grouped labels (labels are ordered by their frequency and evaluation is performed each label group, each group contains 10 labels) PubMed when adopting different algorithms the ordinary fine-tuning paradigm.


Reviews:
[{"review_id": "7M3OTPPewG", "reviewer": "Reviewer_LHTo", "summary": "", "questions": "Questions related to HLR\n+ Is the HLR applied to each level just once? Could you explain the motivation behind this particular design choice?\"\n+ GRAM [3] also used graph neural networks to represent medical ontologies.How does your proposed approach offer key advantages compared to GRAM?\n+ HLR incorporates information from both parent and sibling nodes, but what would occur if it were to aggregate information from a single direction? That is, only from either the parent or sibling nodes.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to sincerely thank you for finding our task very practical, our approach novel and well designed. We hope the point-by-point responses can clarify all your concerns.\n\n> Q1: Contents in L355-365 are incomplete. The description of Equation (9) is insufficient and the contribution of SSL is not articulated.\n\nA1: Thanks for your thoughtful and careful comments. We would like to state that our SSL loss leverages samples from other labels to contribute contrastive signal (since $L_{right}$ > 0 (Eq. (9)), and such extra signal is likely to relief data scarcity problem in rare disease labels by alleviating sparsity of positive samples. Following your suggestion, we would like to explain Equation (9) and summarize the contribution of SSL by comparing to normal supervised contrastive loss (SupCon) as follows.\n\n- **Equation (9) Explanation**: Derivation of Eq. (9) indicates a progressive relationship that SSL is a generalized form of commonly used SupCon, i.e., when we treat any other samples in a batch size (or in a dataset) as potentially positive anchors (soft anchor), SupCon is generalized into SSL. In this paper, the weight between a sample and its soft anchor is their pair-wise label similarity (see Eq. (7)), which is normalized to 0~1. Hence, when a soft anchor has the same label with one sample, the weight is 1, and all soft anchors of the same label constitute the left loss ($L_{left}$, in Eq. (9)), which is equal to SupCon. The right loss ($L_{right}$, in Eq. (9)) represents contrast signal from samples with different labels. SSL can be very flexible with user-defined sample similarity function.\n\n- **Contributions of SSL**: In this work we define sample similarity as pair-wise label similarity. (1) If a sample is from common classes, a large amount of contrast signal comes from anchors with the same label, where SSL is approximately equivalent to SupCon; (2) When a sample in low resource like rare diseases, our SSL is helpful to alleviate sparsity of contrast signal by treating samples with similar labels as a soft (positive) anchor, while normal SupCon cannot provide supervision signal for optimization in this case. (3) Moreover, when hareware is limited (e.g., online learning in mobile phone), the training batch size is small, where SupCon may be ineffective but SSL promise a better upper bound of optimization space. By using our designed SSL rather than SupCon, contrastive signal becomes tractable for hard samples from rare labels.\n\nWe will refine the contribution of SSL and explain Eq. (9) more clear in the final version. Thank you!\n\n> Q2: (1) The proposed HLR differs significantly from conventional graph neural networks such as GCN [1] and GAT [2], but the advantages of HLR are not well-explained in the paper. (Please refer to the Questions section for further discussion.) (2) Key advantages compared to GRAM [3]?\n\nA2: Thank you for your insightful concerns and identifying the distinctions of our HLR (implemented with cascade attention module, CAM) from GNNs. \n\n**As for question (1)**, there're two key designs of HLR:\n\n- Each ICD code compulsorily links its parent, this promises codes with common ancestors tend to form more similar representations because of shared ancestor information.\n\n- Each ICD code selectively links its siblings. It was controlled by parametric indicators in Eq. (2), with the motivation that codes in the same level are probably similar, e.g. fungal pneumonia is a scare disease but can share similar clinical representation with common virus pneumonia (e.g., COVID-19).\n\nThe motivation and design require to use a dynamical edge selection strategy for the label graph during training, while ordinary GNNs assume a pre-determined static graph structure.\n\nThe differences between GNNs and HLR are:\n\n1. Our HLR can dynamically select sibling codes with parametric indicators during training (as stated in Line 255-262, Page 4), while GNNs applied in label tree typically used pre-determined and static edges without connecting sibling codes [5]. Notably, though GAT can also dynamically computes the weights of each code, the edges are pre-determined and it has no chance to capture inter-sibling relationships.\n\n2. Our HLR selects similar diseases from the same level to compensate for a scarce one (as stated in Line 89-102, Page 2), while GNNs applied in other medical concept tree (e.g. GRAM [3]) cannot help nodes fetch important information from their sibling nodes.\n\n3. Our HLR empirically outperforms commonly used graph encoders in computation efficiency. To assess the training speed of HLR (CAM implemented), we compared the training times of CAM-based, GAT-based, and Graphformer-based [4] HLR on the five datasets. Following the label tree structure of [5], on a GTX 3090Ti, the average training backward time per sample is shown in the table below. It can be clearly seen that our proposed CAM is empirically faster during training, and the superiority becomes even more pronounced when the code amount in ICD tree is larger (e.g., the Inpatient dataset has more than 200 codes in the tree and achieve the most substantial acceleration).\n\n| avg. train backward time per sample (s) | MIMIC-III | PubMed | Dermato. | Gastro. | Inpatient |\n| :------------------------------------------- | :-------: | :----: | :------: | :-----: | :-------: |\n| HARL(3-layer CAM, ours)                      |  0.0221  | 0.0056 |  0.0115  | 0.0094 |  0.0288  |\n| HARL(1-layer Graphformer)                    |  0.0382  | 0.0186 |  0.0322  | 0.0191 |  0.1153  |\n| HARL(1-layer GAT)                            |  0.0433  | 0.0192 |  0.0301  | 0.0182 |  0.1297  |\n\nWe will highlight the differences between HLR and GNNs and include the empirical comparison in the final version. Thank you again for your constructive suggestions!\n\n\n**As for question (2)**, the graph-based attention in GRAM was applied in sequential diagnosis prediction where input features are structured patient visit (standard medical concepts in features are available), while HARL was used in ICD coding task with unstructured medical texts, which is vastly different in terms of application. If the graph-based attention could be used in ICD coding task, their key differences and our advantages would be:\n\n- **Key Difference**: “HARL incorperates label embedding tree to guide medical text representations”, as stated in Line 106-111, Page 2. HARL organizes knowledge in ICD label level to guide text feature learning, while GRAM operates at the feature level to augment and interact features with medical ontologies.\n\n- **Advantage**: As discussed in question (1), our HLR component uses a dynamical edge selection strategy, which may help to pass and learn more information. In contrast, GRAM was implemented in a GNN manner with static graph structure and did not contain edges among siblings. Such static graph structure likely misses additional information, especially when the amount of sibling nodes is large.\n\n> Q3: Additional experiments with label description (sentence) similarity?\n\nA3: Thank you for your suggestion. We initialized all label embeddings (in HLR) randomly to demonstrate the effectiveness obtained by only utilizing label tree structure. To further demonstrate the superiority of our HLR, we conduct additional experiment results on HLR using: (a) label hierarchy only (randomly init., in the paper) (b) label description only (fixed embeddings).\n\n|                        |  MIMIC-III  |   PubMed    |  Dermato.   |   Gastro.   |  Inpatient  |\n| :--------------------- | :---------: | :---------: | :---------: | :---------: | :---------: |\n| (a)rand. init.(out HLR)| 44.75/53.60 | 58.70/66.10 | 55.23/57.27 | 48.88/51.26 | 73.01/72.77 |\n| (b)text semantics only | 42.89/51.76 | 58.45/66.24 | 55.08/56.61 | 47.56/50.79 | 71.37/71.62 |\n\nIt can be seen group (b) often underperforms our HLR. This outcome could potentially be attributed to the precision of label descriptions that some meaningful texts are sensible for humans but tough for a language model, whose text representation ability relies on the pre-training corpus. Furthermore, the embeddigs of group (b) remain fixed. Compared to the uncontrollable label texts, the label hierarchy is explicit enough to lead label embedding learning where HARL benefits significantly. We will include and discuss on these results in the final version.\n\n> Q4: The performance gain from the proposed approach is small？\n\nA4: Thank you for your careful concerns.\n\n1. **Main Focus & Performance**: We would like to gently point out that this work focuses on coping with scarce diseases and imbalanced classification in the ICD coding tasks with model-agnostic manner. We have actually and stably improved the performance on scarce diseases. See figure 10, page 14, all macro-F1 gains of HARL on PubMed scarce label groups (label 1~90) stably exceed 1\\% while compared methods are unstable. In Table 2, page 8, HARL stably outperforms across five real-world datasets while other baselines are data-biased solutions. Notably, since HARL focus on scarce diseases for health care fairness, the stable performances on metric macro could be more important.\n\n2. **Comparison on Previous Works**: In fact, this task is of great significance and is extensively researched in previous works, because every bit of progress will further advance medical equity, genuinely enabling patients to break free from suffering [6,7,8]. The previous works were solely evaluated on MIMIC dataset, while we conducted extensive evaluations on multiple real-world datasets from hospitals to demonstrate a more stable and conving improvement, which is critical and beneficial to clinical practice.\n\n3. **Significant Potential**: Since HARL is a model-agnostic workflow that can be unconditionally applied and improve any neural architectures for medical text. For fair comparison, we employ a simple base model (normally finetuned on a BERT) to emphasize the contributions stemming from the approaches we have devised. Thus, HARL holds considerable significance, as it can be employed in conjunction with other approaches or models to acquire further performance improvement and universal benefits.\n\n> Q5: How HLR is applied to each label level and the design motivation?\n\nA5: HLR learns label embeddings with cascade attention module (CAM), we addtionally prepared a figure for illustration of processing a 2-level label tree, which will be added in the final version. Here we elaborate the HLR process literally.\n\nWe denote label embeddings as symbol $e$ and representations as $h$. Each label's $h$ is acquired from attention calculation with its $e$ as query, its parent’s $h$, selected siblings’ $e$ and its $e$ as keys and values. All $e$ are randomly initialized. The attention is performed layer by layer and the final output of CAM are leaf node labels' $h$, which are further used to calculate pair-wise label similarity in SSL and DML. Each attention layer only processes labels of adjacent levels. There're two key designs:\n\n- Each label compulsorily links its parent, this promises labels with common ancestors tend to form more similar $h$ because of shared ancestor information.\n- Each label selectively links its siblings, which is controlled by parametric indicators in Eq. (2), with the motivation that labels of the same level are probably similar, e.g. fungal pneumonia is a scare disease but can share similar clinical representation with common virus pneumonia (e.g., COVID-19).\n\nWe would like to include detailed illustration and elaboration for friendly comprehension in the final version.\n\n> Q6: What if aggregate information from either parent or sibling node rather than both?\n\nA6: We are glad to hear your keen consideration. We discussed design motivation of dynamical sibling selection in the responses for “Q2 (question (1))” and “Q5”. Although solely considering parent or sibling may miss extra information, we would like to add experiments on solely aggregating from parent or siblings.\n\n|               |  MIMIC-III  |   PubMed   |  Dermato.  |   Gastro.   |  Inpatient  |\n| :------------ | :---------: | :---------: | :---------: | :---------: | :---------: |\n| both          | 44.75/53.60 | 58.70/66.10 | 55.23/57.27 | 48.88/51.26 | 73.01/72.77 |\n| parent only   | 44.38/53.12 | 57.89/65.84 | 55.36/57.12 | 48.23/50.87 | 72.98/73.06 |\n| siblings only | 43.25/52.09 | 57.41/65.13 | 54.60/56.87 | 48.46/50.73 | 71.95/71.38 |\n\nIt can be seen only including siblings is often less effective than other case. This observation aligns with our motivation, as not all siblings contribute beneficial information, and that is the reason we need to select them in a data-driven manner. Conversely, only including parent ensures a better performance than only relying on siblings, as it promises an errorless information source.\n\n#### References\n\n\\[1\\] T. Kipf and M. Welling, \"Semi-Supervised Classification with Graph Convolutional Networks\", ICLR. 2017\n\n\\[2\\] P. Velickovic et al., \"Graph Attention Networks\", ICLR. 2018\n\n\\[3\\] Edward Choi et al. \"GRAM: Graph-based Attention Model for Healthcare Representation Learning\", KDD. 2017\n\n\\[4\\] Ying C, et al. \"Do transformers really perform badly for graph representation?.\" NeurIPS. 2021\n\n\\[5\\] Wang Zi, et al. \"Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification.\" ACL. 2022\n\n\\[6\\] Xie X, Xiong Y, et al. \"EHR coding with multi-scale feature attention and structured knowledge graph propagation.\" CIKM. 2019\n\n\\[7\\] Cao Pengfei, et al. \"HyperCore: Hyperbolic and co-graph representation for automatic ICD coding.\" ACL. 2020\n\n\\[8\\] Yang Z, et al. \"Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding.\" EMNLP. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ This work addresses a very practical problem and the proposed approaches are well designed.\n+ Section 4.1 and Appendices B and C provide a thorough description of the experimental setting and implementation details, allowing the reimplementation of results.\n+ The ablation study demonstrates that each design component proposed by the authors contributes to performance improvement. They also highlight that hierarchy-aware label representation learning (HLR) and other contrastive learning methods are incompatible. Moreover, the proposed approach works better on the rare labels compared to baselines.\n+ The proposed approach demonstrates better performance on the classification of rare labels compared to baselines.\n", "reasons_to_reject": "+ The contents in L355-365, which are crucial for understanding the proposed SSL, are incomplete. Specifically, (1) the description of Equation (9) is insufficient and (2) the contribution of SSL is not articulated.\n+ The proposed HLR differs significantly from conventional graph neural networks such as GCN [1] and GAT [2], but the advantages of HLR are not well-explained in the paper. (Please refer to the Questions section for further discussion.)\n+ The authors compute the similarity between two labels based on the graphical structure. However, since each label also has its own name and description, it would be possible to compute similarity based on natural language representations. To convincingly demonstrate the benefits of hierarchical label representation, the authors should conduct similar experiments using sentence similarity.\n+ The performance gain from the proposed approach is small, and this is also true for rare labels.\n\n[1] T. Kipf and M. Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR 2017\n\n[2] P. Velickovic et al., Graph Attention Networks, ICLR 2018", "questions_to_authors": "Questions related to HLR\n+ Is the HLR applied to each level just once? Could you explain the motivation behind this particular design choice?\"\n+ GRAM [3] also used graph neural networks to represent medical ontologies.How does your proposed approach offer key advantages compared to GRAM?\n+ HLR incorporates information from both parent and sibling nodes, but what would occur if it were to aggregate information from a single direction? That is, only from either the parent or sibling nodes.", "paper_topic_and_main_contributions": "The authors address the issues of data imbalance and scarcity in biomedical classification. To this end, they propose a novel, framework-agnostic algorithm called Hierarchy Aligned Representation Learning (HARL). HARL captures the hierarchical structure of labels with a cascaded tree attention module, further applying it to contrastive learning. In particular, the authors propose a novel training objective called \"similarity surrogate learning,\" which allows the use of all samples in a batch as negative examples. Additionally, they introduce a new mixup strategy to reuse sample information from less similar labels. On five biomedical text classification datasets (MIMIC-III, PubMed, Dermatology, Gastroenterology, and Inpatient), HARL exhibits similar or slightly better performance compared to the baseline. The authors evaluate HARL and baselines specifically for rare labels, demonstrating that HARL outperforms baselines in these settings.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[3] GRAM:  Graph-based Attention Model for Healthcare Representation Learning, Edward Choi et al., KDD 2017", "Typos_Grammar_Style_and_Presentation_Improvements": "L365: incomplete sentence\nCaption of Table 2: “Marco” -> “macro”"}, "extra_text": {}}, {"review_id": "s31qm0303D", "reviewer": "Reviewer_8duU", "summary": "", "questions": "Question A: What is the performance of the HARL algorithm proposed in this paper? How does it compare with the baseline approaches? \nQuestion B: Does the baseline method use any other metrics as evaluation criteria? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your postive comments and finding our topic important, proposed approach reasonable and experiment adequate. We hope our responses can clarify all your concerns.\n\n> Q1: The description of the method is not detailed.\n\nA1: Thank you. HARL consists of HLR, SSL and DML, in which SSL and DML utilize label representations from HLR. We describe HLR process and relationship of three components separately. \n\n1. **HLR Process**: In this work, HLR is implemented with cascade attention module (CAM). We have prepared an additional figure for illustration of processing a 2-level label tree in the final version. Here we elaborate the HLR process literally.\n\n   We denote label embeddings as the symbol $e$ and representations as $h$. Each label's $h$ is acquired from attention calculation with its $e$ as query, its parent’s $h$, selected siblings’ $e$ and its $e$ as keys and values. All $e$ are randomly initialized. The attention process is performed layer by layer, and the final outputs of the CAM are leaf node labels' $h$, which are further used to calculate pair-wise label similarity in SSL and DML. Each attention layer processes only the labels of adjacent levels. There're two key designs:\n\n    - Each label compulsorily links its parent, this promises labels with common ancestors tend to form more similar $h$ because of shared ancestor information.\n    - Each label selectively links its siblings, which is controlled by parametric indicators in Eq. (2), with the motivation that labels of the same level are probably similar, e.g., fungal pneumonia is a scare disease but can share similar clinical representation with common virus pneumonia (e.g., COVID-19).\n\n    **HLR Contributions**: HLR takes the advantage of this design to help the rare disease code fetch important information from its sibling codes and parent code, while traditionally used graph encoders for the label tree only considered information from parent label. We firstly designed and applied HLR that can dynamically select sibling codes in the label tree.\n\n2. **Component Relationship**: We have prepared a data / gradient flow figure during HARL training process to illustrate the relationship of three components in the final version. Here, we provide a detailed explanation of this relationship.\n\n   In forward process, HLR will embed input ICD codes with ICD label tree and output code embeddings to SSL and DML. SSL and DML perform code similarity based learning and dissimilarity based learning on medical text hidden states (encoded by any DNN text encoder), and jointly contribute the final loss. In backward process, we empirically detach the gradient from DML to HLR, considering gradient from both SSL and DML may unstabilize the learning process. For model inference, DNN directly performs ICD coding prediction task as usual.\n\nWe highly agree that the HARL framework presentation should be more detailed. We will diligently polish and included prepared figures and elaboration for friendly comprehension.\n\n> Q2: Analysis of DBLoss based performances on the MIMIC-III dataset.\n\nA2: Thank you for your thoughtful suggestion. DBLoss is firstly proposed to solve long-tailed multi-label classification by reducing redundant information of label co-occurrence [1]. The probable reason it performs well is that MIMIC-III includes detailed inpatient medical records annotated with many coexisting diseases, where scenario label co-occurrence is frequent. However, when label co-occurrence is irregular, or each sample has only one label, DBLoss cannot take advantage, while HARL is fueled by label representations that can be applied to both multi-label or multi-class data if label information is accessible. For instance, on PubMed dataset (in fine-grained label version), label co-occurrence is spare and complicated, where DBLoss underperforms other baselines. Besides, DBLoss is unable to be deployed in multi-class datasets.\n\nWe would like to further discuss and clearify performance of DBLoss in the final version, thank you!\n\n> Q3: The writing should be improved.\n\nThank you. We are currently collaborating with a native-speaking professor on certain projects, and he has kindly agreed to assist us in refining this paper. We believe its final version will be more clear and easily understood.\n\n> Q4: HARL performance and comparison methods with baselines?\n\nA4: We will present HARL performance and baseline comparison methods separately here.\n\n1. **Elaboration of HARL Performance**: We use macro-F1 & micro-F1 on five real-world medical datasets on ICD coding task to to evaluate HARL and baselines, HARL achieves best macro-F1 on 4 datasets and best micro-F1 on 3 datasets, and stably ranks top 2 in across all datasets (in Table 2, page 7). Besides, all macro-F1 gains of HARL on PubMed scarce label groups (label 1~90) stably exceed 1\\% while other ordinary methods are unstable (in figure 10, page 14).\n\n2. **Baseline Comparison**: We elaborate our comparison methods into baseline selection and settings.\n\n   - Baseline Selection: HARL is a model-agnostic workflow, thus we select 6 model-agnostic algorithms from four different roadmaps. For completeness, we firstly include an extra baseline from recent research of hierarchical text classification (HTC) [2], which is overlooked in previous ICD coding works.\n   - Baseline Settings: All baselines and HARL use the same BERT base model and ordinary finetune head for fair comparison, with each baseline using a recommended hyper-parameter space.\n\n> Q5: Other evaluation criteria for baseline methods?\n\nA5: In order to visualize the performance promotion on rare labels, we reported and analyzed baseline macro-F1 gains on PubMed label groups (grouped in frequency order) as macro score is more important for health care fairness and rare disease detection. We will add other clinical metrics (e.g., brier score) in the final version.\n\n#### References\n\n\\[1\\] Huang, Yi, et al. \"Balancing methods for multi-label text classification with long-tailed class distribution.\" EMNLP. 2021\n\n\\[2\\] Wang, Zihan, et al. \"Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification.\" ACL. 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Sample imbalance and scarcity are common problems in medical text classification. The task is worth studying.\n2. The idea of incorporating code-embedded trees to guide the expression of medical texts is reasonable and could be applied to some relative tasks.\n3. The experiments demonstrate the effectiveness of the proposed approach on 5 real-world datasets.\n", "reasons_to_reject": "1. The description of the method is not comprehensive or detailed. For example, use an overall framework diagram to clarify the interactions between the three modules（HLR, SSL&DML）.\n2. In the experimental part of this paper, the DBLoss method performs close to the method proposed in this paper on the MIMIC-III dataset and lacks analyses.\n3. The writing of this paper should be improved.", "questions_to_authors": "Question A: What is the performance of the HARL algorithm proposed in this paper? How does it compare with the baseline approaches? \nQuestion B: Does the baseline method use any other metrics as evaluation criteria? ", "paper_topic_and_main_contributions": "This paper addresses the problem of medical text classifications from the perspective of sample imbalance and scarcity. The main contribution of this paper is that the proposed HARL algorithm realizes the hierarchical alignment of textual and labeled representations and establishes a benchmark for the classification of unbalanced medical texts. In this paper, the effectiveness of the method is demonstrated through adequate medical text classification experiments and ablation experiments.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ppRjFvsdPW", "reviewer": "Reviewer_xmzY", "summary": "", "questions": "Question A: Why detach the gradient in Equation 10? It seems like the label representation module can be guided either by SSL or DML so why choose SSL instead of DML or use both?\n\nQuestion B: What is the advantage of the cascade tree attention module? I see neither explanation nor experiments about why this is a better choice than a GAT or a Graphformer. In previous HTC works, graph encoders can have just one layer so they are more light-weighted.\n\nQuestion C: What does j refer to in Equation 10? Does one sample mixup with every other sample in a batch or just a random one?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for finding our approach novel, useful and potentially generalized. We hope our replies address all of your concerns.\n\n> Q1: Section 3.2?\n\nA1: Thank you. We would like to state that derivation of Eq. (9) indicates our SSL loss leverages samples from other labels to contribute contrastive signal (since $L_{right}$ > 0 (Eq. (9)), and such extra signal is likely to relief data scarcity problem in rare disease labels by alleviating sparsity of positive samples.\nWe will make it more clear in the final version. Thank you!\n\n> Q2: Weak baselines, strong HTC baselines missing?\n\nA2: Thank you for the question. Actually, we have referred the metioned models in the related work, see Line 221-222 (about HPT [1]) and Line 224-226 (about HBGL [2]) in Page 2.\n\nBesidse, we included HTC baseline HGCLR [3] for its using an ordinary finetune head and adopting conservative learning idea as HARL, while HPT and HBGL used prompt-tuning and a label tree MLM head separately. The clarification of baseline selection is stated at:\n\n- “We introduce various **framework-agnostic** algorithms for fair comparison”, in Line 431-433, Page 6.\n- “Group **ordinary finetune** is included”, in Line 434, Page 6. Baselines were all use ordinary finetune head.\n\nWe highly agree with your reminding on insufficient advanced HTC baselines and are very willing to conduct additional experiments. Here are the results, “N” in “arch-specific” denotes HARL directly works on hidden states, though in the paper we used ordinary finetune on BERT, practically it can be adopted to any architectures e.g., prompt-tuning, semantic matching. We report average results on the same 5 random seeds in the paper.\n\n|       |  MIMIC-III  |   PubMed    |  Dermato.   |   Gastro.   |  Inpatient  |  arch-specific   |\n| :---- | :---------: | :---------: | :---------: | :---------: | :---------: | :--------------: |\n| HGCLR | 41.59/51.53 | 55.77/64.50 | 53.34/56.50 | 48.53/51.70 | 71.07/71.29 | ordinary finetune|\n| HBGL  | 42.65/52.37 | 57.38/65.03 | 53.61/55.90 | 47.36/50.02| 72.18/72.63  | label tree MLM |\n| HPT   | 41.32/51.11 | 57.64/65.73 | 53.82/56.02 | 47.58/49.51 | 73.08/72.96 | prompt-tuning |\n| HARL  | 44.75/53.60 | 58.70/66.10 | 55.23/57.27 | 48.88/51.26 | 73.01/72.77 |        N        |\n\nWe will include these results in the final version. However, we would like to gently point out that we are working on medical text representation for the ICD coding task, specifically from the perspective of rare diseases, rather than HTC, and this topic have several key differences compared to the HTC field.\n\n- **Task Differnece**: ICD coding is typically a flat classification problem with all disease labels at the same level, whereas HTC techniques target multi-level label systems.\n\n- **Roadmap Difference**: HARL is an integrated data processing workflow (like well-known DIVIDEMIX [4]) that engages in flat classification instead of HTC.\n\n- **High Label Difficulty**: A typical ICD tree contains numerous parent nodes with sparse children, while in the commonly used HTC benchmark label trees, parents often possess many children.\n\n- **Key Contribution**: HARL firstly design a model-agnostic workflow with underlying ICD tree structure that focuses on imbalanced and scarce disease text and convey a processing spirit. Previous ICD coding works have never evaluated with HTC models \\[6,7,8,9\\], we are the first to include HTC baseline for completeness.\n\n- **Implementation Difference**: Actually, we use the cascade attention module for dynamical edge selection during training (we assumed edges among sibings are uncertain and selective, see “Q5&A5” for the design motivation and detailed technical differences), while HTC models typically use GNNs or Graphformer [5] that need pre-determined and static edges beforhand. Besides, label texts are used in HTC models but not used in our HARL.\n\nHere are baseline settings and analysis:\n\n- **Baseline Settings**: We follow graph strcture setting of \\[1,2,3\\] that assume only edges to parents are retained and used recommended model hyper-parameter spaces, other training settings are the same as HGCLR in our paper. We use `transformers==4.18.0` for HARL, HGCLR and HPT, and an old version `2.10.0` for HBGL as [HBGL GitHub](https://github.com/kongds/HBGL/blob/master). To faithfully follow the HTC label system, we reorganize ICD codes (i.e., turn “Virus Diseases(C01.925)” into “Diseases(C) -> Infections(C01) -> Virus Diseases(C01.925)”), HARL directly classifies leaf nodes without label texts and HTC models classify all (label texts accessible), early stop and final metrics are based on leaf nodes.\n\n- **Results**: In most cases, neither HPT nor HBGL can consistantly outputform HGCLR and HARL in ICD coding tasks, which might be caused by directly adopting HTC models to flat ICD coding task, and information from important sibling nodes is missing. Notably, HBGL and HPT achieve better performances on PubMed and Inpatient (than HGCLR), this may due to the great text abundance of medical abstracts and inpatient records, where label tree MLM and prompt-tuning are more suitable to match label words from input texts compared to ordinary finetuned methods.\n\nThank you for your careful comments, we will clarify the detailed difference between medical text representation under ICD coding and HTC in the related works.\n\n> Q3: Weak ablation, fail to mention comparison with current solutions (normal contrastive loss & mixup) for SSL and DML separately?\n\nA3: Thank you for your careful concerns. We conducted ablations on:\n\n- “w/o SSL”, as stated in Sec. 4.3, Page 7-8, by removing SSL part from HARL.\n- “w/o DML”, as stated in Sec. 4.3, Page 7-8, by removing DML part from HARL.\n- “w/o HRL”, as stated in Line 545-548, Page 7, by constructing combination of normal supervised contrastive loss (SupCon) and mixup.\n\nAll the ablation results demonstrate each component is well designed and improves performance. Following your suggestions, we add the further ablation groups:\n\n- “Compare SSL with normal contrastive loss” means we need a group like “- SSL + SupCon”.\n- “Compare DML with normal mixup” means we need a group like “- DML + mixup”.\n\nWe highly agree with a further ablation and deeper analysis on single component of SLL and DML, and would like to conduct additional experiments to compare SSL with normal contrastive loss (see \"- SSL + SupCon\" below) and compare DML with normal mixup (- DML + mixup).\n\n|             |  MIMIC-III  |    PubMed   |  Dermato.  |   Gastro.   |  Inpatient  |\n| :---------- | :---------: | :---------: | :---------: | :---------: | :---------: |\n| HARL        | 44.75/53.60 | 58.70/66.10 | 55.23/57.27 | 48.88/51.26 | 73.01/72.77 |\n| -SSL+SupCon | 43.25/52.68 | 56.49/65.35 | 54.99/56.83 | 47.89/50.83 | 72.80/73.76 |\n| -DML+mixup  | 42.83/52.32 | 55.92/64.47 | 53.96/56.47 | 47.43/50.46 | 72.18/71.83 |\n\nIt can be seen replacing DML with normal mixup will consistently hurt the performance, and in most cases it can be worse by replacing SSL with SupCon as well (except micro-F1 of Inpatient). Notably, substitution on DML will lead to more performance drop than the one on SSL, which is probably attributed to the direct impact of DML on text hidden states for final classification. Overall, replacing either SSL or DML with their normal solutions cannot benefit from hierarchy-aware label representation and hurts HARL performance.\n\nWe will make addtional ablation analysis more clear and detailed in the final version.\n\n> Q4: Why detach in Eq. (10) (DML)? What about only detach SSL, or use gradients from both?\n\nA4: Thank you for the detailed question. HARL is trainable in all 3 cases (detach only DML or SSL, or use both gradients). Actually, this is an empirical choice, for both DML and SSL can impact HLR, we concerned gradients from both will destabilize the learning process and select the best gradient strategy by macro-F1. Here are results on 3 gradient policies.\n\n|                           |  MIMIC-III  |   PubMed   |  Dermato.  |   Gastro.   |  Inpatient  |\n| :------------------------ | :---------: | :---------: | :---------: | :---------: | :---------: |\n| det. DML(current) | 44.75/53.60 | 58.70/66.10 | 55.23/57.27 | 48.88/51.26 | 73.01/72.77 |\n| det. SSL                  | 44.36/53.72 | 58.32/66.23 | 54.31/56.55 | 48.62/51.25 | 72.18/72.77 |\n| no det.                   | 44.13/53.38 | 57.87/66.13 | 54.12/57.84 | 47.92/51.00 | 72.83/72.28 |\n\nWe very appreciate for your mention on our undetailed gradient design, and would like to include and clear the empirical reason behind “detach SSL” in the final version. We have also prepared a data / gradient flow figure during HARL training process for friendly comprehension.\n\n> Q5: Advantages and necessity of using cascade attention module (CAM) in HLR compared to commonly used one-layer GAT or Graphformer in HTC works? Really light-weighted?\n\nA5: Thank you for your thoughtful comments. We have described the goal of CAM as:\n\n- “Selection of similar diseases from the same level to compensate for a scarce one”, as stated in Line 89-102, Page 2.\n- “Dynamical selection of sibling codes with parametric indicators during training”, as stated in Line 255-262, Page 4.\n- “Use information from the parent and selected siblings to represent a code”, as stated in Line 259-262, Page 4.\n\nWe elaborate the necessity into three parts: design motivation, technical difference and advantages.\n\n- **Design Motivation**: There're two key designs of HLR.\n\n  1. Each ICD code compulsorily links its parent, this promises codes with common ancestors tend to form more similar representations because of shared ancestor information.\n\n  2. Each ICD code selectively links its siblings. It was controlled by parametric indicators in Eq. (2), with the motivation that codes in the same level are probably similar, e.g. fungal pneumonia is a scare disease but can share similar clinical representation with common virus pneumonia (e.g., COVID-19).\n\n  The motivation and design require to use a dynamical edge selection strategy for the label graph during training, while commonly used GNNs or Graphformer in HTC works \\[1,2,3\\] assume a pre-determined and static edges beforhand.\n\n- **Technical difference** between common graph encoder (e.g., GAT and Graphformer) and HLR are\n\n  1. Our HLR can dynamically select sibling codes with parametric indicators during training (as stated in Line 255-262, Page 4), while GNNs or Graphformer typically used pre-determined and static edges without connecting sibling codes [1,2,3]. Notably, though GAT can also dynamically computes the weights of each code, the edges are pre-determined and it has no chance to capture inter-sibling relationships.\n\n  2. Our HLR is able to select similar diseases from the same level to compensate for a rare one (as stated in Line 89-102, Page 2), while GNNs or Graphformer cannot help nodes fetch information from their sibling nodes.\n   \n- **Advantages of CAM** compared to common graph encoder: Besides the ability of dynamical sibling selection (discussed above), CAM has better computation complexity (than Graphformer) and empirical training time.\n\n  1. Computation Complexity Analysis on Graphformer & CAM: Given an ICD tree with $N$ nodes, the complexity of a one-layer Graphformer is $O(N^2 + N)$, and a 3-layer CAM (assume $a,b,c$ nodes each layer) is $O(a^2+b^2+c^2)$, where $a+b+c \\approx N$, the complexity difference can be more pronounced when $N$ is larger, for CAM maintains 3 small attention maps while Graphformer is a large one. Besides, Graphformer includes extra processing on spatial and edge encodings.\n   \n  2. Empirical Training Backward Time Evaluation: To further empirically assess the computation efficiency of the CAM, we would like to include an additional experimental comparison of training backward time on five training sets, we use CAM, GAT, Graphformer to implement HLR separately (GAT and Graphformer follow graph structure as \\[1,2,3\\], substitution codes of GAT and Graphformer are from HPT GitHub repository), and average training backward time per sample is reported as follows.\n   \n  3. Inference Time: After HARL training process, DNN models perform inference directly without HARL components and has no computation difference.\n\n| avg. train backward time per sample (s) | MIMIC-III (N=74,L=512) | PubMed (N=167,L=128) | Dermato. (N=107,L=128) | Gastro. (N=60,L=128) | Inpatient (N=206,L=128) |\n| :--------------------------------------------- | :---------------------------: | :-------------------------: | :---------------------------: | :-------------------------: | :----------------------------: |\n| HARL(3-layer CAM)                              |            0.0221            |           0.0056           |            0.0115            |           0.0094           |             0.0288             |\n| HARL(1-layer Graphformer)                      |            0.0382            |           0.0186           |            0.0322            |           0.0191           |             0.1153             |\n| HARL(1-layer GAT)                              |            0.0433            |           0.0192           |            0.0301            |           0.0182           |             0.1297             |\n\nIn the table, the “N” denotes the code amount in the ICD tree, the “L” denotes maximum token length used for the dataset. From the dataset dimension, it is reasonable that MIMIC-III and Inpatient has the longer average training time than others for the long document and large label amount separately. From the method dimension, GAT and Graphformer need longer time to optimize, and this difference becomes even more pronounced when “N” is larger (e.g., on Inpatient, HARL with CAM is 4~5 times faster than the one with GAT or Graphformer).  Empirically, CAM is faster than GAT, this may benefit from the hardware acceleration for attention parallel calculation.\n\nWe highly agree that the advantages and the necessity of using CAM implemented HLR should be emphatically discussed and demonstrated. We will include the detailed motivation, empirical evaluation to make the advantages more clear in the final version.\n\n> Q6: What $j$ means in Eq. (10)?\n\nA6: In Eq. (10), $j$ represents all other samples in a batch size. We follow the same formulaic symbols in the early works of mixup applications in NLP [10,11], and will add the denotation in the final version.\n\n#### References\n\n\\[1\\] Wang Z, et al. \"HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification.\" EMNLP. 2022\n\n\\[2\\] Jiang T, et al. \"Exploiting Global and Local Hierarchies for Hierarchical Text Classification.\" EMNLP. 2022\n\n\\[3\\] Wang Z, et al. \"Incorporating Hierarchy into Text Encoder: a Contrastive Learning Approach for Hierarchical Text Classification.\" ACL. 2022\n\n\\[4\\] Li J, et al. \"DivideMix: Learning with Noisy Labels as Semi-supervised Learning.\" ICLR. 2020\n\n\\[5\\] Ying C, et al. \"Do transformers really perform badly for graph representation?.\" NeurIPS. 2021\n\n\\[6\\] Xie X, Xiong Y, et al. \"EHR coding with multi-scale feature attention and structured knowledge graph propagation.\" CIKM. 2019\n\n\\[7\\] Chen Y, et al. \"Automatic ICD code assignment utilizing textual descriptions and hierarchical structure of ICD code.\" BIBM. 2019\n\n\\[8\\] Cao P, et al. \"HyperCore: Hyperbolic and co-graph representation for automatic ICD coding.\" ACL. 2020\n\n\\[9\\] Yang Z, et al. \"Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding.\" EMNLP. 2022\n\n\\[10\\] Chen J. \"MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification.\" ACL. 2020\n\n\\[11\\] Sun L, et al. \"Mixup-Transformer: Dynamic Data Augmentation for NLP Tasks.\" COLING. 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper first explores the medical text classification problem from a data imbalance perspective and comes up with a solution.\n2) This paper proposes a general solution for contrastive learning with the imbalance problem, which can be adapted to any scenario with imbalanced training data.\n3) Experiments show that every component of the proposed model is useful.", "reasons_to_reject": "1) Section 3.2 is unfinished (line 365).\n2) Strong baselines are missing. Although this paper is about medical text classification, this problem can be easily transformed into hierarchical text classification (HTC). Even though the authors select one of the HTC models, i.e. HGCLR, as a baseline, stronger HTC models fail to compare. Works like\n\n   * Wang, Zihan, et al. \"HPT: Hierarchy-aware Prompt Tuning for Hierarchical Text Classification.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n   * Jiang, Ting, et al. \"Exploiting Global and Local Hierarchies for Hierarchical Text Classification.\" Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022.\n\n    achieve far better results than HGCLR so they should be compared. Among them, HPT (the first one) is adept at handling the imbalance and low resource situations, which may have solved the problem already.\n3) The ablation results are weak. Although the authors demonstrate each component is useful, they fail to mention how each component is better than current solutions. E.g., how about using the normal contrastive learning loss instead of the proposed SSL or using an ordinary mixup strategy instead of DML? There is no experiment about how the proposed design is necessary instead of using existing works.\n\n**Edit after rebuttal**\n\nThe authors explain the above three points in detail with lots of extra experiments. I will not change my score because it's unfair for papers that have done the experiments part well before the rebuttal.", "questions_to_authors": "Question A: Why detach the gradient in Equation 10? It seems like the label representation module can be guided either by SSL or DML so why choose SSL instead of DML or use both?\n\nQuestion B: What is the advantage of the cascade tree attention module? I see neither explanation nor experiments about why this is a better choice than a GAT or a Graphformer. In previous HTC works, graph encoders can have just one layer so they are more light-weighted.\n\nQuestion C: What does j refer to in Equation 10? Does one sample mixup with every other sample in a batch or just a random one?", "paper_topic_and_main_contributions": "This paper proposes a Hierarchy Aligned Representation Learning (HARL) for medical text classification and attempts to address the imbalanced problem where some classes have insufficient training data. Besides an ordinary BERT and a linear classifier, HARL contains two more components: 1) Similarity Surrogate Learning adopts supervised contrastive learning by viewing all samples in a batch as weighted positive samples; 2) Dissimilarity Mixup Learning trains the model with weighted mixup samples. The authors use similarity scores of label representations as the weights for the two components, and the label representations are learned by a cascade attention module.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 143 imblance -> imbalance\n2) There should be a main figure showing how the three components interact with each other. There are so many symbols and equations and it is hard to understand without a figure. Also, there are $\\lambda$ in both Equations 10 and 11 and it seems like they are not the same one.\n3) The discussion in the methodology section (lines 297-333) can be moved to the introduction, showing the necessity and challenge of the imbalance problem first before coming up with a solution."}}]

Author rebuttal:

Meta-review: This paper proposes HARL, a method for performing the automatic coding task (i.e. predicting diagnosis codes based on clinical text). HARL uses attention on diagnosis label taxonomy to learn useful features for rare/imbalance diagnosis codes, and empirically showed favorable performance. All reviewers agreed on the value of the task itself, and the performance is promising, there were several strong concerns such as: 1) HARL's performance on predicting rare diagnosis codes (which is the core motivation of this paper) are comparable to baselines; 2) HARL makes an exaggerated claim of novelty regarding the use of diagnosis code hierarchy, which has been explored multiple times before in the healthcare domain 3) HARL fails to accurately position its core claim between medical text classification and general imbalanced classification algorithm.
Decision: Accept-Findings