Contrastive Bootstrapping for Label Refinement
Year: 2023

Abstract:
Traditional text classification typically categorizes texts into pre-defined coarse-grained classes, from which the produced models cannot handle the real-world scenario where finer categories emerge periodically for accurate services. In this work, we investigate the setting where fine-grained classification is done only using the annotation of coarse-grained categories and the coarse-to-fine mapping. We propose a lightweight contrastive clustering-based bootstrapping method to iteratively refine the labels of passages. During clustering, it pulls away negative passage-prototype pairs under the guidance of the mapping from both global and local perspectives. Experiments on NYT and 20News show that our method outperforms the state-of-the-art methods by a large margin.1

heading: 1 Introduction
text: Traditional text classification often categorize into a set of coarse-grained classes, which falls short in real-world scenarios where finer categories emerge. To this end, coarse-to-fine text classification is introduced (Mekala et al., 2021), which performs fine-grained classification given only annotation of coarse-grained categories and the coarse-to-fine mapping. Then, it finetunes a pre-trained language model for each coarse prototype.2 However, this two-step method could be sub-optimal. For example, it is vulnerable to the noise which is propagated and accumulated through the pipeline. Besides, it requires finetuning and saving a pre-trained language model for each coarse prototype which is heavyweight.
To this end, we propose a lightweight bootstrapping method based on contrastive clustering to iter-
1Code is available at https://github.com/recorderh ou/contrastive_bootstrapping_label_refinement
2We use prototype and category interchangeably.
atively refine the labels of passages.3 To be more specific, the method starts with an epoch of warmup on the weakly-labeled dataset. During warm-up, it pulls away negative passage-prototype pairs under the guidance of the mapping from both global and local perspectives, i.e., coarse inter-cluster and fine inter-cluster perspectives. After the warm-up, the distances between clusters are not significant which causes misclassification. Instead of continuing training on the weakly-labeled dataset which might greatly increase the noise (Figure 1(b)), we perform a bootstrapping process which finetunes the model on the selected dataset and updates the selected dataset by the finetuned model alternately. To mitigate the noise, we propose a selection strategy to identify high-quality pairs in terms of similarity and distinction. To further boost our method, we adopt a modified similarity metric from (Lample et al., 2018) and use the gloss knowledge to augment the prototype representation. As shown in (Figure 1(c)), the resulting clusters are well separated with less noise.
Our contributions are summarized as follows: • We propose a lightweight bootstrapping
method based on contrastive clustering to ad-
3We focus on passage-level classification as it is consistent with prior studies (Mekala et al., 2021). Though, without loss of generality, the studied problem as well as the proposed method can be extended to classifying natural language text in other granularities.
976
dress the problem of coarse-to-fine text classification. • Our method outperforms the state-of-the-art methods on two widely-used datasets. Further analysis verifies the effectiveness of our proposed techniques.

heading: 2 Proposed Method
text: This section describes the technical details of the proposed method, starting with the task description.

heading: 2.1 Task Description
text: We follow the task definition of coarse-to-fine text classification in previous work (Mekala et al., 2021). Given n passages {p1, ..., pn} with their corresponding coarse-grained labels {c1, ..., cn}, along with the coarse-to-fine mapping T , our goal is to assign a fine-grained label to each passage. The key notations used in our paper are defined as follows: (1) C = {C1, C2, ..., Cm} denotes the coarse prototypes. (2) F = {F1,F2, ...,Fk} denotes the fine prototypes. (3) T : C → F denotes the coarse-to-fine mapping, a surjective mapping which separates F into |C| non-overlapping partitions. (4) Spf = T (ci) denotes the fine-grained candidate prototype of pi, which is also dubbed as p for simplicity. (5) Snf = F/Spf denotes fine prototypes not belonging to T (ci). (6) Snc = C/ci denotes coarse prototypes in C other than ci.

heading: 2.2 Our Method
text: Training Process As illustrated in Figure 2, we start with an epoch of warm-up, during which we optimize two contrastive losses Lglobal, Llocal on the weakly-labeled dataset and only the Lglobal on the unlabeled dataset. The two contrastive losses are detailed in the following paragraphs. Then, we conduct several epochs of bootstrapping with the above model. At each bootstrapping step, we first select a small set of passages on which labels are predicted with high confidence by the model. Then, we finetune the model on the selected dataset with the same losses as warm-up. We repeat the finetuning and the selection alternately.
Initial Weak Supervision Following previous work, we consider samples that exclusively contain the label surface name as their respective weak supervision. More details can be referred to the prior study.
Passage and Prototype Representation We encode passages {p1, ..., pn} and all prototypes C∪F into the same embedding space with a pretrained language model. The resulting passage representation and prototype representation are denoted as p and l respectively. During the training process, the prototype representations are dynamically updated to fit the current passage representations. Specifically, we use the last hidden representation of [CLS] as their representations.
Similarity Metric Cosine similarity is often used to measure semantic similarity of embedding representations. However, in high-dimensional spaces, some “hub” vectors may be close to many other vectors while some other vectors are instead being isolated. For example, a passage’s representation p may get high cosine with a large number of labels in Spf due to such hubness issues. In this case, a high similarity score does not necessarily lead to a high discrepancy among labels. Selecting a highlyscored label from the hub as the seed is potentially detrimental to our pairing-based method. Inspired by cross-domain similarity local scaling (Lample et al., 2018), we adopt a modified similarity metric c(p, l) to prevent passage vectors from becoming hubs:
c(p, l) = cos(p, l)−KNN(p) (1)
KNN(p) = 1
K ∑ max l∈F K{cos(p, l)} (2)
where KNN(.) denotes K nearest neighbors.
Warm-up Viewing a passage as an anchor, we expect that its semantic similarity to the correct fine-grained prototype should be closer than any other fine-grained candidate prototypes. We regard the distance in the representation space as the similarity. Specifically, we optimize the following
margin ranking loss:
Lglobal = 1|Spf | ∑
l∈Spf l′∈Snf
max{c(p, l)− c(p, l′) + γ, 0}
(3)
where γ is a hyper-parameter denoting the margin. We use all fine candidate prototypes in Spf as positive examples and randomly sample the same number of prototypes from Snf as negative examples. We view this loss as a global loss to cluster samples according to their coarse labels (Figure 3).
For instances labeled in the initial weak supervision stage, we adopt another margin ranking loss:
Llocal = max{sec_max− c(p, l) + σ, 0} (4) sec_max = max
l′∈Spf ,l′!=l c(p, l′) (5)
We regard this loss as a local loss to cluster samples according to their fine-grained labels (Figure 1 (a)).
Bootstrapping After the warm-up, representations show an inclination to form clusters. Yet, the distances between them are not significant enough to separate the classes. To further get compact clusters, we perform bootstrapping which finetunes the model on the selected dataset and updates the selected dataset by the finetuned model alternately. Instead of using the initial weak supervision which might greatly increase the noise as observed, we propose a selection strategy to select high-quality passage-prototype pairs. Specifically, we assign a pseudo label to each passage by their similarity (Eq.(6)). Apart from similarity, we assume high-quality pairs should also be discriminative (Eq.(7)):
l = arg max l∈Spf c(p, l) (6)
c(p, l)− max l′∈Spf ,l′!=l c(p, l′) > β (7)
where β is a threshold updated at each epoch. We construct a confident set CS with top r% pairs satisfying these two conditions. We update β with the lowest similarity in CS. Then, we optimize Eq.(4) and Eq.(3) on CS and the rest passages accordingly.
Gloss Knowledge Since the surface names alone can not well represent the semantics of labels, we enrich them with external semantic knowledge. To be more specific, we select the first two sentences in each surface name’s first Wikipedia webpage to augment the original surface name with a predefined template (Table 3). We adopt the format of “template, surface name, gloss” and use the last hidden representation of [CLS] as their representation.
Prediction It is worth noticing that applying our similarity metric c(p, l) do not change the relative ranking among labels in Spf compared with the cosine similarity. For simplicity, we use cosine similarity for prediction.
l = arg max l∈Spf cos(p, l) (8)

heading: 3 Experiments
text: In this section, we describe the experimental evaluation for the proposed method.

heading: 3.1 Datasets and Metrics
text: For a fair comparison with prior work, we use the same hierarchical datasets used by We report both Macro-F1 and Micro-F1 for evaluation on the following two datasets.
The 20 Newsgroups (20News) The passages in 20News was organized into 5 coarse-grained newsgroups and 20 fine-grained newsgroups corresponding to different topics (Table 2). Passages in 20News were partitioned evenly across the 20 different fine-grained newsgroups.4 Following (Mekala et al., 2021), we omitted the 3 miscellaneous newsgroups (“misc.forsale,” “talk.politics.misc” and “talk.religion.misc”) and expanded the abbreviation to full words.
The New York Times (NYT) This dataset contains 5 coarse-grained topics and 25 subtopics (Table 2). The NYT dataset is highly skewed with the coarse-grained topic “sports” containing more than 80% passages.

heading: 3.2 Main Results
text: We compare our model with the previous work (Mekala et al., 2021), as well as several zeroshot weakly supervised text classification methods
4http://qwone.com/~jason/20Newsgroups/
(Wang et al., 2021b; Meng et al., 2020a) following previous works. We reproduce them using their implementation.567
As shown in Table 1, our method outperforms the baselines by 5.67% in Micro-F1 and 5.54% in Macro-F1 on the NYT dataset, as well as 3.97% in Micro-F1 and 3.04% in Macro-F1 on 20News dataset.

heading: 3.3 Analysis
text: To verify the effectiveness of different model components , we conduct ablation studies to test each of those.
Effect of Bootstrapping The “w/o bootstrap” results in Table 1 report the performance with warm-up only. These results are consistently lower than those with bootstrapping. Specifically, bootstrapping improves the warm-up by 3.15% MicroF1, 7.40% Macro-F1 and 1.63% Micro-F1, 3.30% Macro-F1 on NYT and 20News respectively. Figure 1(a)(c) shows passage representations are more separated from each other.
Effect of Selection Strategy We replace the selection strategy in bootstrapping with the initial weakly-labeled samples. From the “w/o bootstrap” results in Table 1, we can see that, our selection strategy brings an improvement of 4.26% MicroF1, 7.46% Macro-F1 on NYT. It is better to use the seed dataset on 20News. We hypothesize that this observation is because the seed dataset has a more balanced label distribution than our selected
5https://github.com/yumeng5/LOTClass 6https://github.com/ZihanWangKi/XClass 7https://github.com/dheeraj7596/C2F
high-quality samples on 20News. We also incorporate our selection strategy to the C2F baseline in the bootstrapping stage. As shown in Table 1 row “C2F w/ our select,” this strategy improves the performance of C2F by 1.43% Micro-F1, 1.17% MacroF1 on 20News and 0.41% Micro-F1 on NYT, exhibiting the effectiveness of our strategy.
Effect of Similarity Metric We replace our similarity metric with the cosine similarity. From Table 1 “w/o similarity” we can see that, our similarity metric brings along an improvement of 3.39% in Micro-F1, 7.46% in Macro-F1 on NYT, and 16.43% in Micro-F1 and 22.46% in Macro-F1 on 20News. From Figure 4, we can see that 63% of samples belonging to the “Law Enforcement” prototype are misclassified using the cosine similarity. However, 18% are misclassified using our similarity metric, verifying its effectiveness. Besides, results for “w/ Manhattan similarity” and “w/ Euclidean similarity” show that alternating cosine similarity in c(p, l) causes performance drops of 35.81% (5.53%) in Micro-F1, 40.72% (6.57%) in Macro-F1 and 50.19% (0.18%) in Micro-F1, 50.43% (0.73%) in Macro-F1 on 20News and NYT data, further proving the effectiveness of our similarity metric.
Effect of Gloss Knowledge We remove the gloss knowledge and use the label surface name only. Comparing the “w/o gloss” results in Table 1 with the full-setting ones, we observe that the gloss knowledge brings an improvement of 2.73% in Micro-F1, 9.42% in Macro-F1 on NYT and 4.86% in Micro-F1, 6.91% in Macro-F1 on 20News. Figure 5 further shows the effect of gloss knowledge on different prototypes.
Extending to the setting without coarse-to-fine mapping We extend our method to the setting without the coarse-to-fine mapping. In other words, the only supervision is the gold coarse labels. We modify Lglobal as follows:
Lc_global = max{c(p, lc)− c(p, l′c) + γ, 0} (9)
where we use the golden coarse label lc as the positive example and randomly sample one coarse label l′c from Snc as the negative example. The “w/o fine” results in Table 1 show that the performance does not degrade much when the association between coarse and fine-grained labels does not exist, showing the feasibility of our method in a more general setting.

heading: 4 Related Work
text: Previous works in weakly supervised text classification have explored different kinds of weak supervision. (1) a set of related keywords. (Mekala and Shang, 2020) augment and disambiguate the initial seed words with contextualized and highly labelindicative keywords. (Meng et al., 2020b) identify keywords for classes by querying replacements for class names using BERT and pseudo-labels the documents by heuristics with the selected keywords. (2) a few labeled documents. (Tang et al., 2015) represent the labeled documents and different levels of word co-occurrence information as a largescale text network. (Meng et al., 2018) propose a pseudo-document generator that leverages the seed labeld documents to generate pseudo-labeled documents for model pre-training. (3) label surface names. (Wang et al., 2021b) propose an adaptive representation learning method to obtain label and document embedding, and cluster them to pseudolabel the corpus. Our setting is different from theirs
in that we use coarse-grained annotation to improve the fine-grained text classification.
Contrastive learning (He et al., 2020; Chen et al., 2020; Khosla et al., 2020) aims at learning representations by contrasting the positive pairs and negative pairs. In NLP, existing works can be primarily categorized into two distinct streams. Unsupervised contrastive learning seeks to contrast grouped or perturbed instances to generate more robust representation of unlabeled textual data (Gao et al., 2021; Wei et al., 2021; Kim et al., 2021; Wang et al., 2021a). On the contrary, supervised contrastive learning (Suresh and Ong, 2021; Zhou et al., 2021; Yu et al., 2021; Huang et al., 2022) is label-aware and seeks to create representations for differently labeled data with more discrepancy. Our work has shown that supervised contrastive learning incorporating label names, with minimal external knowledge, improves the model’s performance in label refinement.

heading: 5 Conclusion
text: In this paper, we study the task of coarse-to-fine text classification. We propose a novel contrastive clustering-based bootstrapping method to refine the label in an iterative manner. Experiments on two real-world datasets for coarse-to-fine text classification verify the effectiveness of our method. Future work could consider extending this method to other fine-grained decision-making tasks that could potentially benefit from coarse-grained labels, such as various kinds of lexical semantic typing tasks (Huang et al., 2022). Another meaningful direction is to consider incorporating other partial-label learning techniques (Zhang et al., 2016) that are relevant to coarse-to-fine prediction tasks.
Limitations
Our paper has the following limitations: (1) In realworld applications, the label hierarchy may be more than two levels. It is worth extending our method to such a setting and empirically verifying it. (2) Our selection strategy simply takes top r% confident samples, which might result in class imbalance problem. Alleviating the imbalance problem may further improve our performance. We leave them as future work.

heading: Acknowledgement
text: We appreciate the reviewers for their insightful comments and suggestions. We would like to express our gratitude to the authors of the C2F paper (Mekala et al., 2021) for their collective effort in open-sourcing the dataset and code. Their released materials played a vital role in our research.
Shudi Hou, Yu Xia and Sujian Li were supported by National Key R&D Program of China (No. 2020AAA0109703). Muhao Chen was supported by the National Science Foundation of United States Grant IIS 2105329, a subaward of the INFER Program through UMD ARLIS, an Amazon Research Award and a Cisco Research Award.

heading: A Dataset Statistics
text: We list the statistics of the datasets in Table 2.

heading: B Templates
text: We list the templates used in Table 3.

heading: C Effect of gloss knowledge on different prototypes
text: We show the confusion matrix over all fine prototypes in Figure 5.
D Implementation Details
We use RoBERETa-base (Liu et al., 2019) as the encoder. The models are trained on one GeForce RTX 3090 GPU. We set the batch size as 8. We do one epoch of warmup and four epochs of bootstrapping. We use the predictions from the last epoch as the final predictions. We use AdamW (Loshchilov and Hutter, 2017) as the optimizer. r is set as 15 for NYT and 1 for 20News. γ and σ are set as 0.05 for both NYT and 20News. We run our model 3 times using different random seeds. We used t-SNE (Pedregosa et al., 2011; Buitinck et al., 2013) for the visualization in this paper.

heading: E Selection of r
text: We select the value of r from set {1, 5, 10, 15, 20}. For each coarse prototype Ci, we calculate the ratio of initial weak supervision WCi in category Ci to the total number of instance ICi in Ci, we denote the ratio as RCi = WCi/ICi . After that, we select the r closest to min
Ci∈C {RCi}. As shown in Table 4a
and Table 4b, the minimal RCi in NYT dataset is 13.43%, closest to 15, while the minimal RCi in 20News dataset is 2.05%, closest to 1.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: After Conclusion
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: 3.1
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We obtain the license and will not distribute it .
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? We use the dataset following their intended use.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments? 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix D
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix D and E

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix D

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix D
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: Future work could consider extending this method to other fine-grained decision-making tasks that could potentially benefit from coarse-grained labels, such as various kinds of lexical semantic typing tasks (Huang et al. , 2022). Another meaningful direction is to consider incorporating other partial-label learning techniques (Zhang et al. , 2016) that are relevant to coarse-to-fine prediction tasks. Limitations
Our paper has the following limitations: (1) In realworld applications, the label hierarchy may be more than two levels. It is worth extending our method to such a setting and empirically verifying it. (2) Our selection strategy simply takes top r% confident samples, which might result in class imbalance problem. Alleviating the imbalance problem may further improve our performance. We leave them as future work