Venue: NAACL 2024

Create! Don't Repeat: A Paradigm Shift in Multi-Label Augmentation through Label Creative Generation

Letian Wang, Xianggen Liu, Jiancheng Lv\({}^{*}\)

College of Computer Science, Sichuan University, China

Engineering Research Center of Machine Learning and Industry Intelligence, China

letianwang@stu.scu.edu.cn, {liuxianggen, lvjiancheng}@scu.edu.cn

Corresponding Author.

###### Abstract

We propose Label Creative Generation (LCG), a new paradigm in multi-label data augmentation. Beyond repeating data points with fixed labels, LCG creates new data by exploring innovative label combinations. Within LCG, we introduce Tail-Driven Conditional Augmentation (TDCA), combining tail-driven label sampling and label-conditioned text generation for balanced, consistent data augmentation. Our approach has demonstrated a **100.21%** increase in PSP@1 across three datasets, successfully mitigating the long-tail effect in MLTC and markedly enhancing model performance.

## 1 Introduction

Multi-label Text Classification (MLTC), prevalent in fields such as sentiment analysis and recommendation systems, grapples with the dual challenges of an extensive label space and a pronounced long-tail distribution. This imbalance is exemplified in Wiki10-31K dataset, where a mere **1.5%** of labels have more than **100** training instances, leaving the vast majority with a scarcity of training data. Table 1 reveals that while numerous studies claim to have addressed or alleviated the long-tail issue, advancements in recent years, particularly on PSP@k, have been gradual. Relying solely on advancements in neural architectures appears to be ineffective over time. There is a need for data augmentation.

### Inadequacies of Current DA Methods

Current data augmentation (DA) methods in text classification have shown promise, yet their effectiveness in MLTC remains limited. Prevailing DA strategies typically focus on employing various approaches to replicate data while maintaining labels unchanged. These methods fall into two distinct categories: paraphrase-based augmentation and conditional augmentation (CA). The former includes model-free methods like easy data augmentation Wei and Zou (2019), as well as model-required techniques such as backtranslation Sennrich et al. (2016); Edunov et al. (2018). CA employs conditional generation (often label-conditioned) for data augmentation to synthesize texts that are more controlled yet diverse Li et al. (2020); Liu et al. (2020). Such strategies offered benefits in binary or multi-class text classification, often regarded as a means to bolster model robustness Bayer et al. (2023). However, as illustrated in Figure 1, conventional approaches like EDA and BT are seldom effective and might even

Figure 1: PSP@5 for various expansion ratios on MeSH-12K dataset, comparing four data augmentation methods: Easy Data Augmentation (EDA), Back-Translation (BT), Conditional Augmentation (CA), and TDCA. The Expansion Ratio (E/R) quantifies augmentation: 1 indicates no augmentation and \(1+x\%\) shows \(x\%\) augmentation over the original dataset.

 

impede model performance, particularly at greater data expansion scales. CA shows improved performance but remains modest.

Intrinsically marked by the existence of multiple labels per instance, MLTC poses distinct challenges to conventional data augmentation methods. Paraphrasing-based approaches, which solely alter the text while maintaining its original intent and keeping the labels unchanged, fail to enhance performance by a large margin. Moreover, the noise introduced in the paraphrasing process can adversely affect model performance, a negative impact that becomes more pronounced with an increasing proportion of augmented data.

### LLMs in MLTC

The recent validation of scaling laws in large language models (LLMs) like GPT Brown et al. (2020); Bubeck et al. (2023) and LLMa Touvron et al. (2023a, b) has revolutionized previously infeasible tasks and advanced tasks that had plateaued. Research exploring LLMs in multi-label text classification has been emerging. For example, Kocon et al. (2023) evaluated ChatGPT in text classification, observing its lag behind traditional SOTA models, with the gap widening in complex classification scenarios. However, its performance in emotion recognition was noteworthy. Loukas et al. (2023) corroborated this, highlighting ChatGPT's potential in few-shot and zero-shot classification. This is in line with the training goal of LLMs, which involves predicting the next token from preceding ones.

Post-training on large-scale general corpora, their formidable understanding and common sense skills lend them efficacy in text generation tasks and common classification tasks with fewer categories Ray (2023). Nonetheless, they grapple with issues like hallucinations and prompt sensitivity, underperforming in knowledge-intensive tasks like large-scale information retrieval and fine-grained text classification Li et al. (2023).

Given the extremely large number of labels, employing LLMs for MLTC directly is impractical. An alternative is leveraging LLMs for data augmentation, as explored in some recent works. AugGPT Dai et al. (2023), for instance, continued with the conventional approach, employing ChatGPT for paraphrasing texts while preserving the original labels, yielding marginal improvements. Van Nooten and Daelemans (2023) took a more straightforward route. They provided ChatGPT with nine distinct labels along with the corresponding examples to generate additional text-labels pairs for tweet classification.

### Create! Don't Repeat

The untapped potential of LLMs opens new avenues for conditional data generation to liberate us from the limitations of simply repeating existing data points, enabling the synthesis of entirely new ones. Hence, we introduce a new multi-label data augmentation paradigm, Label Creative Generation (LCG), and within LCG, we propose Tail-Driven Conditional Augmentation (TDCA). TDCA comprises (1) a Metropolis-Hastings algorithm based tail-driven label sampling for crafting more balanced label combinations with consideration of label correlations; and (2) a contrastive label-conditioned generation approach, which fine-tunes LLMs to generate texts that not only accurately reflect each label in the sampled label combinations but also emulate the style of the original dataset.

TDCA enables the transference of extensive correlations from head labels to tail labels through LLM. Our experimental results demonstrate a substantial alleviation of the long-tail effect, evidenced by a **100.21%** average enhancement in PSP@1 across Eurlex-4K, MeSH-12K, and Wiki10-30K datasets. Remarkably, with increasing expansion ratios, we observe an ascending trend in PSP@k, devoid of the detrimental noise impact commonly associated with traditional data augmentation methods, a finding corroborated by t-SNE visualizations in Figure 5. Ablation studies affirm the efficacy and validity of TCDA.

Our contributions are threefold:

Figure 2: Tag frequency distribution post data augmentation. Compared to traditional data augmentation methods (EDA, BT, and CA), TDCA demonstrates more effective mitigation of the long-tail effect.

 

1. We introduce a new paradigm in multi-label data augmentation, Label Creative Generation. This approach, to the best of our knowledge, is pioneering in the field as it does not rely on pre-existing label combinations but instead creates new ones.
2. Within the framework of LCG, we propose a novel method named Tail-Driven Conditional Augmentation. TDCA enables the generation of balanced label combinations through the construction of a dual-weighted label graph and employs tail-driven label sampling based on the Metropolis-Hastings algorithm. Moreover, it utilizes contrastive label-conditioned generation to produce augmented texts that are both representative of the assigned labels and coherent with the original dataset.
3. Our experiments conducted across three datasets show that TDCA significantly reduced the long-tail effect in MLTC. The results show notable improvements in PSP@1, P@10, and N@10, with increases of **100.21%**, **16.58%**, and **11.65%**, respectively.

## 2 Tail-Driven Conditional Augmentation

Multi-label Text Classification (MLTC) aims to assign a subset of labels \(Y\subseteq L\) to each instance \(x\), where \(L=\{l_{1},l_{2},...,l_{N}\}\) represents the entire possible label space. This task can be formalized as learning a mapping function \(f(x)\to 2^{L}\), predicting the power set of \(L\). The approach of TDCA is straightforward: It commences with tail-driven label sampling to create label combinations and proceeds to contrastive label-conditioned generation, synthesizing text that aligns with these labels.

### Tail-Driven Label Sampling

In MLTC, two primary characteristics emerge: (1) Each instance is associated with multiple labels, which exhibit significant correlations; (2) As depicted in Table 2, multi-label datasets encounter a pronounced long-tail effect, with tail labels dominating the label space yet being represented in only a small fraction of training instances. To address this, a balanced and correlation-aware label sampling approach is necessary. Consequently, we present Tail-Driven Label Sampling, which incorporates the Dual-Weighted Label Graph and Metropolis-Hastings sampling.

#### 2.1.1 Dual-Weighted Label Graph

The Dual-Weighted Label Graph (DWLG) is formally defined as \(G=(V,E,W_{v},W_{e})\). It comprises vertices \(V\), each signifying a distinct label, and edges \(E\), linking pairs of vertices based on their co-occurrence in the dataset. Vertex weights are consolidated in \(W_{v}=\{w_{v}(i)~{}|~{}i\in V\}\), with \(w_{v}(i)\) representing the occurrence frequency of label \(i\). The edge weights, denoted as \(w_{e}(i,j)\) in \(W_{e}\), capture the co-occurrence strength between labels \(i\) and \(j\). This dual-weighted architecture yields an integrated perspective of both individual label properties and their interconnections, fundamental for advanced label sampling and data exploration1.

Footnote 1: Our DWLG diverges from the traditional Label Correlation Graph concept introduced by Mittal et al. (2021), departing from random walk-based graph construction and incorporating dual weights for a clearer depiction of label dynamics.

#### 2.1.2 Metropolis-Hastings Label Sampling

Employing the principles of Markov chain theory, Metropolis-Hastings (M-H) sampling (Metropolis et al., 1953; Hastings, 1970) can adeptly adjust to the target distribution, which is perfectly in tune with our aim to achieve a more balanced sampling strategy in long-tail distributions. We initiate M-H sampling of the DWLG by starting from a tail label, then iteratively transitions to other labels, directed by both the transition kernel and acceptance rate. This procedure is maintained until an adequate count of labels is sampled or a pre-determined step limit is reached.

Firstly, the transition kernel \(q(i\to j)\) calculates the likelihood of moving from the current label

Figure 3: Tail-driven label sampling. Vertex size shows label frequency in the dataset, while edge thickness and color depth indicate label co-occurrence frequency.

 

to an alternative label \(j\) within the DWLG during sampling. Considering the inter-label correlations, we define \(q(i\to j)\)in the following manner:

\[q(i\to j) = \frac{e^{w_{e}(i,j)}}{\sum_{k\in\text{neighbors}(i)}e^{w_{e}(i,k)}}, \tag{1}\]

where \(w_{e}(i,j)\) denotes the edge weight between labels \(i\) and \(j\) in the DWLG.

Then, drawing upon the principles of information entropy Shannon (1948), the target distribution \(p(i)\) is defined to encapsulate the significance and scarcity of label \(i\):

\[p(i)=\frac{e^{s(i)/T}}{\sum_{k}e^{s(k)/T}}. \tag{2}\]

Here, \(s(i)=-\log(w_{v}(i))\) is the importance score of label \(i\), and \(T\) is a temperature parameter that moderates the distribution's smoothness.

Finally, the acceptance rate \(\alpha(i\to j)\) evaluates whether to accept the transition from label \(i\) to label \(j\), thereby steering the sampling outcomes towards the target distribution:

\[\alpha(i\to j)=\min\left(1,\frac{p(j)\cdot q(j\to i)}{p(i)\cdot q(i\to j)} \right). \tag{3}\]

### Contrastive Label-conditioned Generation

Conditional generation, recognized for its capability to produce texts that are both diverse and controlled, has improved data augmentation in binary and multi-class scenarios. However, previous studies have largely concentrated on the labelled duplication of existing data points. The true potential of conditional generation extends far beyond this practice.

Integrating LLMs into conditional generation introduces several challenges. When dealing with a sampled labels set: (1) An excess of labels leads to extended input sequences, complicating LLMs' ability to reflect each label in the generated text, often causing omissions; (2) LLMs' sensitivity to prompt selection results in erratic text generation quality; (3) Existing LLMs, enhanced with RLHF, tend to generate superfluous explanatory content, which can be counterproductive for data augmentation; (4) Texts generated by LLMs exhibit stylistic discrepancies with the original dataset.

We fine-tune LLM on the original dataset to mitigate these issues. Inspired by Song et al. song2023 and Rafailov et al. fafailov2023, we devised two targeted loss functions for multi-label classification dataset augmentation: Label Match Loss (\(\mathcal{L_{CM}}\)), ensuring generated texts align with each input label, and Style Consistency Loss (\(\mathcal{L_{SC}}\)), which aids in producing texts that are coherent, controllable, and in stylistic harmony with the original dataset.

#### 2.2.1 Style Consistency Loss

Initially, a subset \(\{X^{1},Y^{1};\dots;X^{n},Y^{n}\}\) is randomly extracted from the training set, wherein \(X=\{x_{1},\ldots,x_{|X|}\}\) denotes a text, comprising a series of tokens \(x\), and \(Y=\{y_{1},\ldots,y_{|Y|}\}\) represents a set of multiple labels associated with \(X\). Regarding \(Y\), we concat it with a prompt to create a composite text \(c(Y)\), such as "Generate text for the labels \([y_{1},\ldots,y_{|Y^{i}|}]\)" to serve as an input for the LLM. To ensure that the text \(X_{aug}\) produced by the LLM aligns with \(X\), the Style Consistency Loss is as follows:

\[\mathcal{L_{SC}}=-\sum_{t}\log P_{\phi}(x_{t}|c(Y),x_{1,\cdots,t-1}). \tag{4}\]

Here, \(\phi\) signifies the parameters of the LLM. Given the input \(c(Y)\) and the prior tokens \(x_{1,\cdots,t-1}\), our goal is to fine-tune the model with greater probability to predict the next token \(x_{t}\) 



#### 2.2.2 Label Match Loss

To effectively align the augmented input \(X_{aug}\) with the corresponding label combinations \(Y\), we not only employ \(\mathcal{L}_{\mathcal{SC}}\) to illustrate ideal generation but also aim to guide the LLM in distinguishing between good and bad ones. In the randomly selected subset of the training set \(\{\mathbf{X^{1}},\mathbf{Y^{1}};X^{2},Y^{2};\ldots;X^{n},Y^{n}\}\), where each \(Y^{i}\) is unique, the Jaccard similarity Jaccard (1912) is utilized to evaluate and rank the degrees of similarity, ranging from \(Y^{1}\) and \(Y^{2}\), to \(Y^{1}\) and \(Y^{n}\). For labels \(Y^{1}\), its associated text \(X^{1}\) is considered a **positive** example, while texts ranging from \(X^{2}\) to \(X^{n}\) are deemed **negative** examples, exhibiting progressively higher degrees of dissimilarity.

We generalize the InfoNCE loss van den Oord et al. (2018) using the Plackett-Luce model Plackett (1975); Luce (1959), deriving the Label Match Loss as follows:

\[\mathcal{L}_{\mathcal{LM}}=-\sum_{i=1}^{n-1}\log\frac{\exp\left(\frac{r_{ \phi}(X^{i})}{\mathcal{T}_{i}^{j}}\right)}{\sum_{j=i}^{n}\exp\left(\frac{r_{ \phi}(X^{ij})}{\mathcal{T}_{i}^{j}}\right)}, \tag{5}\]

where \(\phi\) denotes the parameters of the LLM, \(\mathcal{T}\) represents the temperature coefficient, and \(r_{\phi}(X^{i})\) indicates the likelihood of the LLM generating \(X^{i}\). For the set \(\{\mathbf{X^{1}},\mathbf{Y^{1}};X^{2},Y^{2};\ldots;X^{n},Y^{n}\}\), we initially compare \(X^{1}\) with each of \(X^{2}\), ..., \(X^{n}\), followed by a comparison of \(X^{2}\) with \(X^{3}\), ..., \(X^{n}\), aiming to align the LLM-generated \(X_{aug}\) with the label \(Y\). Specifically, \(r_{\phi}(X)\) is defined as:

\[r_{\phi}(X)=\frac{1}{|X|}\sum_{t=1}^{|X|}\log P_{\phi}(x_{t}|c(Y^{1}),x_{1, \cdots,t-1}), \tag{6}\]

where \(X\) encompassing tokens \(x_{1},\cdots,x_{|X|}\), with \(c(Y^{1})\) representing the label of the positive sample integrated with the prompt.

Each comparison involves modulation of suppression for negative samples at varying degrees through temperature coefficients \(\mathcal{T}\):

\[\mathcal{T}_{i}^{j>i}=\frac{1}{s(Y^{1},Y^{i})-s(Y^{1},Y^{j})}, \tag{7}\]

\[\mathcal{T}_{i}^{i}=\min_{j>i}\mathcal{T}_{i}^{j}. \tag{8}\]

Here, \(s(Y^{i},Y^{j})\) denotes the Jaccard similarity between the label sets \(Y^{i}\) and \(Y^{j}\).

#### 2.2.3 Optimization Objective

Combining the label match loss and style consistency loss, the final loss function is:

\[\mathcal{L}=\mathcal{L}_{\mathcal{LM}}+\lambda\mathcal{L}_{\mathcal{SC}}. \tag{9}\]

The hyperparameter \(\lambda\) balances the importance of label matching and style consistency. Optimizing \(\mathcal{L}\) ensures that the generated text is not only

Figure 4: Fine-tuning LLM in Contrastive Label-conditioned Generation. \(\mathcal{L}_{\mathcal{LM}}\) ensures alignment of generated texts with input labels. \(\mathcal{L}_{\mathcal{SC}}\) supports the production of similar texts with the original dataset.



relevant to the sampled labels \(Y\) but also stylistically coherent with the actual text \(X\), striking a balance between accuracy and authenticity.

## 3 Experiments

### Datasets

We employ three benchmark datasets: Eurlex-4K, MeSH-12K, and Wiki10-30K ("K" denotes the label count within each dataset). **Eurlex-4K**[9] comprises a corpus of European Union legal documents. We utilize its raw text, as provided in Ye et al. yu2020, without applying stemming or stop-word removal. **MeSH-12K**, a subset culled from the BioASQ datasets2Tsatsaronis et al. (2015), is composed of article titles and abstracts from PubMed, annotated with Medical Subject Headings (MeSH) as their labels. **Wiki10-30K**, originating from Wikipedia, is a refined iteration of Wiki10-31K3 Zubiaga (2012). While prior studies merely utilized the label IDs in Wiki10-31K for classification, we undertook a meticulous review and cleansing of this dataset. This entailed the exclusion of content-lacking instances and labels constituted solely of punctuations (_e.g._, "", "", "") or NLTK-listed stopwords (_e.g._, "and", "or"). More statistical information can be found in Table 2.

Footnote 2: [http://participants-area.bioasq.org](http://participants-area.bioasq.org)

Footnote 3: [https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multilabel.html](https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multilabel.html)

### Evaluation Metrics

To assess the performance of each method, we use the following evaluation metrics: \(P@k\), \(PSP@k\), and \(N@k\).

**Precision at k (\(P@k\))** measures the precision of the top \(k\) predicted labels in matching the true labels. For an instance with its top-k predicted label set \(\hat{Y}@k\) and actual label set \(Y\), \(P@k\) is defined as:

\[P@k=\frac{|\hat{Y}@k\cap Y|}{k}, \tag{10}\]

where \(|\hat{Y}_{k}\cap Y|\) is the number of correct predictions in the top \(k\) labels. On the Wiki10-30K, if a model accurately predicts the head labels \((L_{>100}=461)\) and ignores the rest, it would achieve a \(P@1\) of 90.8%. This high precision, however, overlooks the informative tail labels.

**Propensity Scored Precision at k (\(PSP@k\))** mitigates this limitation by modifying precision to give higher weights to less frequent labels. Given the top-\(k\) predicted label set \(\hat{Y}_{k}\) and the actual label set \(Y\), \(PSP@k\) is defined as:

\[PSP@k=\frac{\sum_{y\in\hat{Y}@k\cap Y}\frac{1}{\pi_{y}}}{k}. \tag{11}\]

Here, \(\pi_{y}\) represents the propensity score of label \(y\), quantifying the likelihood of encountering label \(y\) in the training dataset.

**Normalized Discounted Cumulative Gain at k (\(N@k\))** evaluates the ranking quality of predicted labels, considering both the relevance of the labels and their ranking positions. The Discounted Cumulative Gain (DCG) for a predicted label \(\hat{y}_{i}\) at rank \(i\) is calculated as:

\[DCG@k=\sum_{i=1}^{k}\frac{\mathbb{I}(\hat{y}_{i}\in Y)}{\log_{2}(i+1)}, \tag{12}\]

where \(\mathbb{I}(\cdot)\) is an indicator function. \(N@k\) is the ratio of \(DCG@k\) to \(iDCG@k\):

\[iDCG@k=\sum_{i=1}^{\min(k,|Y|)}\frac{1}{\log_{2}(i+1)}. \tag{13}\]

\[N@k=\frac{DCG@k}{iDCG@k}. \tag{14}\]

### Experiment Settings

We compare TDCA with three data augmentation baselines: EDA, BT, and CA. EDA Wei and Zou (2019) involves synonym replacement and random insertion/swap/deletion. In BT, we 

select French, Chinese, Russian, Italian, and Spanish as intermediate languages for English paraphrasing via mllb-208-distilled model Costajussa et al. (2022). CA and TDCA utilize LLaMa-based LLM, Owen-7B-Chat Bai et al. (2023), for label-conditioned text generation. For MLTC, we utilized LightXML Jiang et al. (2021) with bert-base-uncased Devlin et al. (2019). The EurLex-4K setup included a 1e-4 learning rate, 15 epochs, batch size of 16, and max token length of 512, with SWA Izmailov et al. (2018) applied post 10 epochs (step size 200). For Wiki10-30K, we extended training to 30 epochs (SWA step size 300), maintaining other parameters. MeSH-12K followed the Wiki10-30K settings, with a batch size of 8 and a max token length of 256. TDCA utilized Metropolis-Hastings sampling from labels with <100 instances, with a 1000-step limit and temperature of 10. Fine-tuning involved a \(\lambda\) value of 0.2 for loss balance, 2 epochs, a 512 sequence length, and a 5e-6 learning rate. All experiments were conducted on 8 NVIDIA A100 GPUs.

## 4 Results

### Performance Comparison

Performance assessments were carried out on the EurLex-4K, MeSH-12K, and Wiki10-30K on various expansion ratios to evaluate the effectiveness of our proposed TDCA compared to EDA, BT, and CA. As Table 3 indicates, all methods mitigated the long-tail effect in MLTC datasets to varying degrees: EDA and BT marginally enhanced \(PSP@k(k=1,3,5)\) by 1.89% and 1.95%, respectively. CA's diverse text generation led to a 9.45% increase, limited by unchanged labels. TDCA, integrating tail-driven sampling, impressively reduced the long-tail effect with an 85.61% improvement.

Moreover, data augmentation's effectiveness correlates with the number of dataset labels. TDCA enhanced \(PSP@1\) by 16.46% on EurLex-4K, 98.78% on MehSH-12K, and 185.51% on Wiki10-30K. This result aligns with the expectation that more labels intensify the long-tail effect, enhancing the utility of data augmentation.

Figure 5: A t-SNE van der Maaten and Hinton (2008) visualization of the MeSH-12K dataset comparing various text data augmentation methods: (a) BT, (b) EDA, and (c) our proposed TDCA, where represents original data, and represents augmented data.

\begin{table}
\begin{tabular}{l l c r r r r r r r r r r} \hline
**Dataset** & **Methods** & **E/R** & **PSP@1** & **PSP@3** & **PSP@5** & **P@1** & **P@3** & **P@5** & **P@10** & **N@3** & **N@5** & **N@10** \\ \hline  & _Raw_ & 1 & _42.65_ & _50.46_ & _53.89_ & _85.56_ & _73.11_ & _60.91_ & _38.35_ & _76.48_ & _70.46_ & _73.82_ \\  & EDA & 1+100\% & 42.33 & 50.50 & 54.53 & 85.54 & 72.77 & 60.84 & 38.33 & 76.18 & 70.33 & 73.78 \\
**CurLex** & BT & 1+25\% & 42.77 & 50.73 & 54.42 & 86.03 & 73.42 & 61.15 & 38.61 & 76.84 & 70.79 & 74.34 \\  & CA & 1+150\% & 43.53 & 52.30 & 55.89 & 86.70 & 74.04 & 61.66 & 38.93 & 77.48 & 71.33 & 74.86 \\  & **TDCA** & **1+400\%** & **49.67** & **57.87** & **61.19** & **88.15** & **76.49** & **64.04** & **40.68** & **79.75** & **73.79** & **77.70** \\ \hline  & _Raw_ & 1 & _18.04_ & _21.56_ & _23.32_ & _90.46_ & _74.64_ & _63.61_ & _46.04_ & _78.32_ & _69.84_ & _58.44_ \\  & EDA & 1+200\% & 18.13 & 21.93 & 24.04 & 89.31 & 74.65 & 63.66 & 45.90 & 78.06 & 69.70 & 58.20 \\
**MeSH** & BT & 1+200\% & 18.16 & 21.66 & 23.64 & 89.20 & 73.89 & 62.70 & 45.18 & 77.45 & 68.91 & 57.49 \\  & CA & 1+500\% & 19.33 & 24.14 & 26.65 & 90.03 & 77.85 & 66.99 & 49.07 & 80.74 & 72.63 & 61.34 \\  & **TDCCA** & **1+1000\%** & **35.84** & **42.08** & **46.45** & 90.20 & **82.67** & **75.61** & **59.92** & **84.42** & **79.23** & **70.57** \\ \hline  & _Raw_ & 1 & _16.22_ & _16.66_ & _17.33_ & _87.89_ & _77.42_ & 68.06 & _51.79_ & _79.84_ & _72.77_ & _60.47_ \\  & EDA & 1+50\% & 16.67 & 17.30 & 18.13 & 87.24 & 76.43 & 67.00 & 50.82 & 78.92 & 71.78 & 59.52 \\
**Wiki** & BT & 1+50\% & 16.66 & 17.50 & 18.28 & 87.32 & 76.98 & 67.59 & 51.42 & 79.37 & 72.29 & 60.07 \\  & CA & 1+200\% & 17.83 & 19.15 & 20.34 & 87.09 & 76.48 & 67.52 & 51.20 & 78.97 & 72.16 & 59.89 \\  & **TDCA** & **1+3000\%** & **46.31** & **39.02** & **36.92** & 87.12 & **78.27** & **72.03** & **58.79** & **80.31** & **75.48** & **65.88** \\ \hline \end{tabular}
\end{table}
Table 3: Performance comparison of DA techniques (EDA, BT, CA, TDCA) against raw dataset on Wiki10-30K, EurLex-4K, and MeSH-12K. Metrics for each method are reported at their **optimal** E/R (Expansion Ratio).

 

Regarding \(P@K\) and \(N@K(k=1,3,5,10)\), label-conditioned augmentation methods, CA and TDCA, consistently improved performance, with CA increasing \(P@K\) by 1.42% and \(N@K\) by 1.09% on average, and TDCA by 8.16% and 5.95%. EDA and BT, however, had a marginal negative impact: EDA decreased \(P@k\) and \(N@K\) by 0.63% and 0.64%, while BT showed similar declines by 0.52% and 0.49%. This is further supported by t-SNE visualizations (Figure 5), where TDCA's augmented data demonstrate better integration with original data, unlike EDA and BT.

TDCA not only effectively counters the long-tail effect in MLTC datasets but also bolsters prediction precision and ranking quality. The improvements are progressive with larger values of \(k\) in all metrics, which indicates that label-conditioned augmentation is more adaptable to MLTC's extremely large label space. Furthermore, the quantitative experiments (Table 4) reveal that TDCA outperforms traditional methods at equivalent ratios. While EDA and BT exhibit increasingly negative effects with more augmented data, TDCA's benefits progressively amplify. For comprehensive details, refer to training logs in Appendix B.

The case study in Appendix A uncovers an intriguing aspect: the original dataset often exhibits an implicit link between labels and text, significantly challenging the learning process of models. The LLM-based CA and TDCA render these labels more explicit in the generated texts, facilitating an easier learning of correspondences.

### Ablation Analysis

Table 4 presents the results of TDCA and its three variant models under different expansion ratios. These variants are: CA (TDCA without M-H based tail-driven sampling, _i.e._, TDCA w/o M-H), TDCA w/o F-T (TDCA without contrastive labelled-conditioned fine-tuning), and CA w/o F-T (TDCA w/o M-H & F-T ).

The varying shades of red (high) and green (low) in the table illustrate that TDCA and TDCA w/o F-T significantly outperform both CA (TDCA w/o M-H) and CA w/o F-T (TDCA w/o F-T & M-H) across all metrics. This highlights the pivotal role of M-H based tail-driven sampling in TDCA, validating our proposed Label Creative Generation. Furthermore, the comparison between TDCA and TDCA w/o F-T, as well as between CA and CA w/o F-T, confirms the effectiveness of contrastive fine-tuning. Overall, these results demonstrate that each component of the TDCA structure is effective and contributes to its overall performance.

## 5 Conclusion

In this paper, we introduce a new multi-label data augmentation paradigm named Label Creative Generation (LCG). Under LCG, we propose Tail-Driven Conditional Augmentation (TDCA). TDCA facilitates the creation of balanced label combinations by dual-weighted label graph and tail-driven label sampling. Furthermore, TDCA employs a contrastive label-conditioned generation to produce augmented texts that match each label and maintain consistency with the original dataset. Empirical evaluations on three datasets with varying label counts demonstrate the effectiveness of TDCA. Our approach significantly surpasses existing data augmentation methods, effectively mitigates the long-tail effect, and enhances model prediction performance in MLTC.

\begin{table}
\begin{tabular}{c c|c c c c c c c c c c c c c c c c c} \hline  & \multicolumn{3}{c}{**EDA**} & \multicolumn{3}{c}{**BT**} & \multicolumn{3}{c}{**CA w/o F-T**} & \multicolumn{3}{c}{**CTA**} & \multicolumn{3}{c}{**w/o M-HD**} & \multicolumn{3}{c}{**PCA w/o F-T**} & \multicolumn{3}{c}{**TDCAL**} \\ \cline{2-17}  & PSP & P & N & PSP & P & N & PSP & P & N & PSP & P & N & PSP & P & N & PSP & P & N \\ \hline \multirow{11}{*}{Enplex} & 1+259\% & **93.91** & 60.58 & **70.04** & 54.22 & 61.15 & 70.79 & 54.88 & 61.53 & 71.08 & **84.42** & 61.17 & 70.44 & **84.00** & 61.05 & 70.77 & 55.61 & 61.73 & 71.31 \\  & 1+50\% & 542\% & 65.04 & 70.44 & **51.57** & 62.06 & **69.95** & 55.51 & 61.11 & 71.18 & 54.51 & 61.07 & 70.71 & 54.28 & 61.16 & 71.03 & 56.57 & 62.03 & 71.77 \\  & 1+100\% & 543\% & 60.74 & 73.53 & 67.03 & 71.52 & 51.51 & 61.74 & 70.81 & 53.59 & 65.50 & 71.21 & 55.73 & 61.92 & 71.28 & **53.97** & **73.57** & **73.17** \\  & 1+200\% & 543\% & 60.91 & **59.14** & 53.85 & 60.42 & 69.82 & 56.04 & 61.76 & 71.39 & 56.36 & 62.01 & 71.55 & 57.21 & 62.76 & **72.59** & **60.31** & 63.72 & 73.26 \\  & 1+500\% & 544\% & 60.02 & **60.48** & 51.96 & 60.31 & 69.01 & 56.42 & 61.92 & 71.52 & 56.43 & 62.00 & 71.50 & **60.47** & **60.77** & **60.75** & **72.54** & **61.81** & **61.08** & **72.81** \\ \hline \multirow{11}{*}{MeSH} & 1+259\% & **233.81** & 63.27 & 63.97 & 23.55 & 63.66 & 60.72 & **53.59** & 63.70 & 70.02 & **42.27** & 64.25 & 70.36 & 70.24 & 43.86 & 64.20 & 71.05 & 24.40 & 64.65 & 70.55 \\  & 1+50\% & **23.61** & 63.31 & 69.44 & **23.67** & 63.41 & 60.61 & 24.64 & 64.65 & 70.63 & 24.68 & 65.07 & 70.95 & 25.49 & 64.78 & 70.69 & 25.80 & 65.65 & 71.44 \\  & 2+100\% & 23.90 & 63.27 & 60.73 & 69.28 & 25.03 & 65.09 & 71.04 & 25.31 & 65.61 & 71.34 & 26.65 & 61.15 & 71.78 & 27.61 & 66.93 & 72.28 \\  & 1+200\% & 240\% & 64.36 & 69.70 & 62.44 & 62.02 & 68.91 & 59.66 & 65.69 & 71.26 & 26.65 & 62.22 & 29.24 & **61.99** & **31.90** & **60.96** & **78.74** \\ \hline \multirow{11}{*}{Wekit} & 1+500\% & 24.50 & 64.31 & 69.88 & **23.00** & 61.81 & 67.91 & 26.36 & 66.09 & 71.70 & 26.65 & 66.99 & 72.63 & **36.22** & **22.21** & **76.55** & **38.99** & **73.06** & **78.01** \\ \cline{1-1} \cline{1-1 



## Limitations

* This method is primarily effective for textual labels with semantic content. It is less suited for labels represented by numerical IDs or non-descriptive identifiers.
* Compared to EDA and other non-model-based data augmentation techniques, our approach is more resource-intensive. However, its resource consumption is comparable to seq2seq model-based methods like back-translation.
* Due to computational resource constraints, we were unable to fully explore the upper limits of TDCA's performance improvements. On the EurLex dataset, we observed optimal expansion ratios between 400% to 500%. Beyond this range, performance tended to plateau or slightly decrease. In contrast, on the MeSH-12K and Wiki10-30K datasets, with more extensive labels, we experimented with expansion ratios up to 1000% and 3000%, respectively, without reaching an apparent performance ceiling.
* Our exploration did not extend to closed-source LLMs such as ChatGPT, Bard, or Claude, limited by API access. Nonetheless, considering the promising results achieved with the un-tuned Qwen-7B model (_i.e._, TDCA w/o F-T in ablation study), we believe that employing closed-source LLMs via API calls can yield comparable or superior results.
* We observed limitations in fine-tuning (FT), potentially due to the simplicity of our data synthesis task (generating text from labels), where using prompts alone could teach a LLM this task. However, we believe that the LCG extends beyond this. For instance, in recommendation systems, LCG could be used on the MovieLens dataset to generate diverse user information for movie recommendations, such as address, age, postal code, etc. Nevertheless, relying solely on prompts for an LLM might fall short, whereas FT ensures this capability.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China (No. 62206192); the Natural Science Foundation of Sichuan Province (No. 2023NS-36 FSC1408); the Science and Technology Major Project of Sichuan Province; and the Fundamental Research Funds for the Central Universities (No. 1082204112364). We sincerely thank the ARR 2023 Dec reviewers for their insightful feedback and extend our great gratitude to Yang Li and Guanyu Qiao for the stimulating discussions at the inception of this research, which significantly enriched this work.

## References

* Bai et al. [2021] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jiawei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023. Quen technical report. _CoRR_, abs/2309.16609.
* Bayer et al. [2023] Markus Bayer, Marc-Andre Kaufhold, and Christian Reuter. 2023. A survey on data augmentation for text classification. _ACM Comput. Surv._, 55(7):146:1-146:39.
* Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Bubeck et al. [2023] Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. _CoRR_, abs/2303.12712.
* Costa-jussa et al. [2023] Marta R. Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Meja Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti

Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzman, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyval Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. _CoRR_, abs/2207.04672.
* Dai et al. (2023) Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Wei Liu, Ninghao Liu, Sheng Li, Dajiang Zhu, Hongmin Cai, Lichao Sun, Quanzheng Li, Dinggang Shen, Tianming Liu, and Xiang Li. 2023. Auggpt: Leveraging chatpft for text data augmentation. _arXiv preprint arXiv:2302.13007_, v3.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* November 4, 2018_, pages 489-500. Association for Computational Linguistics.
* Ge et al. (2022) Zhiqi Ge, Yuanyuan Guan, Ximing Li, and Bo Fu. 2022. Consistent, balanced, and overlapping label trees for extreme multi-label learning. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, October 17-21, 2022_, pages 551-560. ACM.
* Hastings (1970) W. K. Hastings. 1970. Monte Carlo sampling methods using Markov chains and their applications. _Biometrika_, 57(1):97-109.
* Izmailov et al. (2018) Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. In _Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018_, pages 876-885. AUAI Press.
* Jaccard (1912) Paul Jaccard. 1912. The distribution of the flora in the alpine zone.1. _New Phytologist_, 11(2):37-50.
* Jain et al. (2016) Himanshu Jain, Yashoteja Prabhu, and Manik Varma. 2016. Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_, pages 935-944. ACM.
* Jiang et al. (2021) Ting Jiang, Deqing Wang, Leilei Sun, Huayi Yang, Zhengyang Zhao, and Fuzhen Zhuang. 2021. Lightxml: Transformer with dynamic negative sampling for high-performance extreme multi-label text classification. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_, pages 7987-7994. AAAI Press.
* Kocon et al. (2023) Jan Kocon, Igor Cichecki, Oliwiher Kaszyca, Mateusz Kochanek, Dominika Szydlo, Joanna Baran, Julta Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclera, Anna Kocon, Bartlomiej Koptyra, Viktoria Mielesczencko-Kowsrezvic, Piotr Milkowski, Marcin Oleksky, Maciej Piasecki, Lukasz Radlinski, Konrad Wojtanski, Stanislaw Rozniak, and Przyms-law Kazienko. 2023. Chatgtpct: Jack of all trades, master of none. _Inf. Fusion_, 99:101861.
* Li et al. (2023) Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023_, pages 6449-6464. Association for Computational Linguistics.
* Li et al. (2020) Kun Li, Chengo Chen, Xiaojun Quan, Qing Ling, and Yan Song. 2020. Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7056-7066, Online. Association for Computational Linguistics.
* Liu et al. (2017) Jingzhou Liu, Wei-Cheng Chang, Yuexin Wu, and Yiming Yang. 2017. Deep learning for extreme multi-label text classification. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017_, pages 115-124. ACM.
* Liu et al. (2020) Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Sorousd Vosoughi. 2020. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 9031-9041. Association for Computational Linguistics.
* Loukas et al. (2023) Lefteris Loukas, Ilias Stogiannidis, Prodromos Malaka-siatis, Stavros Vassos. 2023. Breaking the bank with ChatGPT: Few-shot text classification for finance. In _Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecasting_, pages 74-80, Macao. Association for Computational Linguistics.
* Mencia and Furnkranz (2008) Eneldo Loza Mencia and Johannes Furnkranz. 2008. Efficient pairwise multilabel classification for large-scale problems in the legal domain. In _Machine 

Learning and Knowledge Discovery in Databases_, pages 50-65, Berlin, Heidelberg. Springer Berlin Heidelberg.
* Luce (1959) R. D. Luce. 1959. _Individual Choice Behavior: A Theoretical Analysis_. Wiley, New York.
* Metropolis et al. (1953) Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. 1953. Equation of state calculations by fast computing machines. _The Journal of Chemical Physics_, 21(6):1087-1092.
* Mittal et al. (2021) Anshul Mittal, Noveen Sachdeva, Sheshansh Agrawal, Sumeet Agarwal, Purushottam Kar, and Manik Varma. 2021. ECLARE: extreme classification with label graph correlations. In _WWW '21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_, pages 3721-3732. ACM / IW3C2.
* Plackett (1975) R. L. Plackett. 1975. The analysis of permutations. _Journal of the Royal Statistical Society. Series C (Applied Statistics)_, 24(2):193-202.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. _CoRR_, abs/2305.18290.
* Ray (2023) Partha Pratim Ray. 2023. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. _Internet of Things and Cyber-Physical Systems_, 3:121-154.
* Semrich et al. (2016) Rico Semrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
* Shannon (1948) C. E. Shannon. 1948. A mathematical theory of communication. _The Bell System Technical Journal_, 27(3):379-423.
* Song et al. (2023) Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2023. Preference ranking optimization for human alignment. _CoRR_, abs/2306.17492.
* Touvron et al. (2023a) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Bapistte Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Auvelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Touvron et al. (2023b) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esioub, Jude Fernandes, Jeremy Fu, Wenyu Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Koumanu, Artem Korenecy, Punit Singh Kouura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diena Liskovich, Yinghai Lu, Tuning Mao, Xavier Martinet, Todor Mihaloylov, Pushkar Mishra, Igor Molbybo, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Runta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Rambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023b. Llama 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotiis, Ioannis Partalas, Matthias Zschunke, Michael R Rivers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, Yannis Almirantis, John Pavlopoulos, Nicolas Baskiots, Patrick Gallinari, Thierry Artieres, Axel Ngonga, Norman Heino, Eric Gaussier, Liliana Barrio-Alvers, Michael Schroeder, Ion Androutsopoulos, and Georgios Paliquras. 2015. An overview of the bioasa large-scale biomedical semantic indexing and question answering competition. _BMC Bioinformatics_, 16:138.
* van den Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. _CoRR_, abs/1807.03748.
* Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of Machine Learning Research_, 9(86):2579-2605.
* Van Nooten and Daelemans (2023) Jens Van Nooten and Walter Daelemans. 2023. Improving Dutch vaccine hesitancy monitoring via multi-label data augmentation with GPT-3.5. In _Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis_, pages 251-270, Toronto, Canada. Association for Computational Linguistics.
* Wei and Zou (2019) Jason W. Wei and Kai Zou. 2019. EDA: easy data augmentation techniques for boosting performance on text classification tasks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_, pages 6381-6387. Association for Computational Linguistics.
* Xiong et al. (2023) Jie Xiong, Li Yu, Xi Niu, and Youfang Leng. 2023. Xrr: Extreme multi-label text classification with candidate retrieving and deep ranking. _Information Sciences_, 622:115-132.
* Ye et al. (2020) Hui Ye, Zhiyu Chen, Da-Han Wang, and Brian D. Davison. 2020. Pretrained generalized autoregressive model with adaptive probabilistic label clusters for 

extreme multi-label text classification. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_, volume 119 of _Proceedings of Machine Learning Research_, pages 10809-10819. PMLR.
* You et al. (2019) Ronghui You, Zihan Zhang, Zive Wang, Suyang Dai, Hiroshi Mamitsuka, and Shantefang Zhu. 2019. Attentionxml: Label tree-based attention-aware deep model for high-performance extreme multi-label text classification. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5812-5822.
* Zubiaga (2012) Arkaitz Zubiaga. 2012. Enhancing navigation on wikipedia with social tags. CoRRabs/1202.5469.

 

Case Study

**MeSH terms (labels)** Community Health Planning; Consultants; **Data Collection**; Delivery of Health Care, Integrated; **Evaluation Studies as Topic;** Health Care Surveys; Health Plan Implementation; Health Services Needs and Demand; Iowa; Public Health Administration; State Health Plans; **United States****

**ORIGNAL Determinants of successful community health system planning. Needs assessment and other community health planning projects are being conducted in many parts of the country. Whether these projects are successful, in that they lead to implementation of recommendations, has not been systematically assessed. This study describes how local public health officials in Iowa view the planning projects that were conducted in their communities. Planning projects occurred in over half of the counties in the state. Most were led by local public health agencies. Hospitals were participants in three-fourths of the projects. Successful implementation was associated with the number of planning techniques used and the breadth of representation on the steering committee. The most common recommendations related to the expansion of prevention/promotion services and community health nursing. Public health nursing had been integrated with local hospitals in one-third of the counties, with no reductions in performance.**

**EDA determinants of the community health system other needs assessment and planning community health planning projects are being conducted in many parts of the country whether health the arc successful assessed that they lead to implementation of recommendations has not been on in related study describes how local public health officials in Iowa view agencies planning projects that were conducted in their systematically planning projects services in over half of the counties in the state most successful led by local public techniques this hospitals were participants in three fourths of the projects successful implementation was associated with third number of planning these used and the breadth of representation communities the steering committee projects most common recommendations the to the expansion of prevention promotion occurred and community health nursing public health nursing had been integrated with local hospitals in one the of were counties with no reductions in performance**

**BTDeterminants of successful community health system planning. Needs assessment and other community health planning projects are being carried out in many parts of the country. The success of these projects, because they lead to the implementation of recommendations, has not been systematically evaluated. This study describes how local public health officials in Iowa view planning projects in their communities. More than half of the counties in the state have planned projects. Most of them are led by local public health institutions. Hospitals are involved in three-quarters of the projects. The successful implementation is related to the number of planning technologies used and the representativeness of the steering committee. The most common suggestions involve expanding prevention/publicity services and community health care. In one third of counties, public health care has been integrated with local hospitals, but the performance has not declined.**

**CA Evaluation of Community Health Planning Consultants in Iowa: A Survey of Data Collection and Delivery of Integrated Health Care Services. Community health planning consultants play a critical role in the implementation of state health plans and addressing the health services needs of communities. This study aimed to evaluate the effectiveness of community health planning consultants in Iowa in facilitating the delivery of integrated health care services. A survey was conducted among community health planning consultants to collect data on their roles, responsibilities, and experiences in implementing state health plans. The survey also assessed the challenges faced by consultants in collecting and analyzing data, delivering integrated health care services, and evaluating the impact of their interventions. The results showed that community health planning consultants in Iowa are effective in facilitating the delivery of integrated health care services, but face challenges in data collection and evaluation. The study recommends the development of standardized data collection tools and training programs for consultants to enhance their skills in data collection and analysis. Additionally, the study highlights the need for ongoing evaluation of the effectiveness of community health planning consultants in addressing the health services needs of communities and implementing state health plans.** 



**MeSH terms (labels)**

Fanconi Anemia; Stress, Mechanical; Surface Properties; Syndrome; Chromosomes, Human, Pair 13; Founder Effect; Mites; Thromboxane-A Synthase; Benzoquinones; Female; Thromboxane B2; Leukotrienes

**TDCAC Exploring the Role of Mechanical Stress and Surface Properties in Fanconi Anemia Syndrome: Insights from the Founder Effect, Chromosomes, and Mites. Fanconi Anemia (FA) is a rare genetic disorder characterized by chromosomal instability, bone marrow failure, and an increased risk of cancer. Previous studies have highlighted the involvement of oxidative stress in FA pathogenesis, but the role of mechanical stress and surface properties remains unclear. This review aims to examine the impact of mechanical stress and surface properties on FA, particularly in the context of the founder effect, chromosomes, and mites. We discuss the potential mechanisms through which these factors may contribute to FA, such as the activation of thromboxane-A synthase and the production of benzoquinolnes and leukotrienes. Additionally, we explore the role of gender in FA and the potential differential effects of these factors in female patients. By synthesizing current knowledge with insights from various research fields, this review sheds light on the complex interplay between mechanical stress, surface properties, and FA pathogenesis, providing a foundation for future investigations in this area.

**MeSH terms (labels)**

Electronics; Blood Urea Nitrogen; Income; Oregon; Personal Autonomy; Ethnic Groups; Hypercholesterolaemia; Hospital Administration; Computers; Health Knowledge, Attitudes, Practice; Research Subjects; Cardiac Care Facilities

**TDCA The Association between Income and Health Outcomes among Various Ethnic Groups in Oregon: A Study of Personal Autonomy, Hypercholesterolememia, and Hospital Administration Using Electronics and Computers: This study aimed to investigate the relationship between income and various health outcomes among different ethnic groups residing in Oregon. The study utilized electronic data and computer technology to analyze data from multiple sources, including hospital administration records, research subjects' health knowledge, attitudes and practices, and cardiac care facilities. Specifically, the study examined the impact of income on personal autonomy, blood urea nitrogen levels, and the prevalence of hypercholesterolemia within different ethnic groups. Results showed a significant association between income and personal autonomy, with lower income groups experiencing higher levels of restriction in decision-making about their health. Additionally, income was found to be positively correlated with blood urea nitrogen levels and the prevalence of hypercholesterolemia, suggesting that higher income may play a protective role against these health concerns. The implications of these findings for hospital administrations and future research on health disparities among various ethnic groups are discussed.

**MeSH terms (labels)**

Polyethylene Glycols; Bacterial Proteins; Point Mutation; Regulon; antigens, CD4; Transplantation Immunology; Calorimetry; Deoxycholic Acid; T-Lymphocytes; Receptors, Lymphocyte Honning; Surface Properties; Mice, inbred CBA

**TDCAC Investigating the Impact of Polyethylene Glycols on Bacterial Proteins and Point Mutation in the Regulon of CD4Antigens in Transplantation Immunology using Calorimetry and Deoxycholic Acid in CBA Mice. This study aims to explore the effects of polyethylene glycols (PEGs) on bacterial proteins and point mutation in the regulon of CD4 antigens in transplantation immunology. Through the use of calorimetry and deoxycholic acid, we will investigate how PEGs alter the surface properties of T-lymphocytes and receptors involved in lymphocyte homing. The study will be conducted in CBA mice, a commonly used model for transplantation research. Results from this study will provide valuable insights into the role of PEGs in modifying the immune response, potentially leading to the development of targeted therapies for transplantation immunology.

Figure 7: Illustration of TDCA performance on the MeSH-12K dataset. Similar to Figure 6, red indicates head labels (with over 100 instances in the training set), and blue denotes tail labels (with fewer than 10 instances). But TDCA exhibits fewer red head labels and a greater number of blue tail labels. Despite the low frequency of occurrence, the labels are still related due to the construction of the Dual-Weighted Label Graph and Tail-Driven Label Sampling, as seen in the first example: **Fanconi Anemia** is linked to **Chromosomes, Human, Pair 13** due to mutations causing this inherited disease. Similarly, **Syndrome** is a term that includes conditions like **Fanconi Anemia**. The enzyme **Thromboxane-A Syntase** is crucial in producing **Thromboxane B2**. Furthermore, **Leukotrienes**, known for their role in inflammation, can interact with **Thromboxane B2** during certain physiological or pathological conditions. These interrelations facilitate the LLMs’ task of generating coherent texts, and the augmented data is meaningful in real-world contexts.





## Appendix B Training Logs

Figure 8: Training logs of **TDCA** at different expansion ratios for PSP, P, and N metrics on MeSH-12K.

Figure 9: Training logs of **CA** at different expansion ratios for PSP, P, and N metrics on MeSH-12K.

Figure 10: Training logs of **EDA** at different expansion ratios for PSP, P, and N metrics on MeSH-12K.