Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification
Authors: Chengyu Dong
Zihan Wang
Jingbo Shang
Venue: EMNLP 2023 Main (2023)
Keywords: text classification
weak supervision
label noise
label bias
TL;DR: We propose simple methods to make weakly-supervised text classification based on seed-word matching achieve state-of-the-art.

Abstract: Recent advances in weakly supervised text classification mostly focus on designing sophisticated methods to turn high-level human heuristics into quality pseudo-labels. In this paper, we revisit the seed matching-based method, which is arguably the simplest way to generate pseudo-labels, and show that its power was greatly underestimated. We show that the limited performance of seed matching is largely due to the label bias injected by the simple seed-match rule, which prevents the classifier from learning reliable confidence for selecting high-quality pseudo-labels. Interestingly, simply deleting the seed words present in the matched input texts can mitigate the label bias and help learn better confidence. Subsequently, the performance achieved by seed matching can be improved significantly, making it on par with or even better than the state-of-the-art. Furthermore, to handle the case when the seed words are not made known, we propose to simply delete the word tokens in the input text randomly with a high deletion ratio. Remarkably, seed matching equipped with this random deletion method can often achieve even better performance than that with seed deletion.

Debiasing Made State-of-the-art: Revisiting the Simple Seed-based Weak Supervision for Text Classification
Chengyu Dong
Zihan Wang
Jingbo Shang"
University of California San Diego
(cdong, ziw224, jshang] @uesd. edu
Abstract
thus often dubbed pseudo-labels, are then em- ployed to train the text classifier following stan- dard fully supervised or semi-supervised training framework. State-of-the-ae methods mostly fo- cus on designing sophisticated human guidance to obtain high-quality labels, through contextual- ized weak supervision (Mekala and Shang. 2020), prompting language models (Meng al., 2020; Zhao et al., 2022), clustering for soft matching (Wang al., 2021), and complicated interactions between seeds (Zhang et al., 2021).
Recent advances in weakly supervised text classification mostly focus on designing so- phisticated methods to turn high-level human heuristics into quality pseudo-labels. In this pa- we revisit the seed matching- matehing-based method, which is arguably the simplest way to gener- ate pseudo-labele and show that its power was greatly underestimated. We show that the lim- ited performance of seed matching is largely due to the label bias injected by the simple seed-match rule, which prevents the classifier from learning reliable confidence for select- ing high-quality pseudo-labels. Interestingly, simply deleting the seed words present in the matched input texts mitigate the label bias and help learn better confidence. Subsequently, the performance achieved by seed matching be improved significantly, making it par with or even better than the state-of-the- art. Furthermore, handle the case when the seed words are not made known, propose to simply delete the word tokens in the input text randomly with a high deletion ratio. Remark- ably, seed matching equipped with this random deletion method can often achieve even better performance than that with seed deletion. We refer to our method as SimSeed, which is pub- licly available
In this paper, revisit the seed matching-based weak supervision (denoted as Vanilla) (Mekala and Shang, 2020; Meng et al., 2018; Tao al., 2015), which is arguably the simplest way to generate assudo-labels, and show that its power was greatly underestimated. Specifically, this simple method matches input text with label if the user-prow ssor-provided seed words of this label are contained in the input text. For example, in sentiment analysis, doc- ument will be labeled "positive" if it contains the word "happy". text classifier is then trained based on all these pseudo-labels.
One can expect non-trivial number of emrors in the seed matching-based pseudo-labels. In an ideal case, we can select only those correct pseudo- labels for training, the accuracy of the text classifier can be significantly boosted. For example, on the 20 Newsgroups dataset, ideally with only those correct pscudo-labels one can get an accuracy of 90.6% 90.6%, compared to 80.1% obtained on all pseudo- labels (see more in Section 4). In practice, to select those correct labels, common way is to use the confidence score of classifier trained on pseudo- labels (Rizve al., 2021). However, those high- confidence pseudo-labels may not be correct the weakly-supervisedd setting, likely because the classifier may fail to learn reliable confidence on these noisy pseudo-labed (Mekala et al., 2022).
1 Introduction
Recently, weakly supervised text classification, be- cause of its light requirement of human effort, has been extensively studied (Mekala and Shang, 2020; Wang et al., 2021; Zhao et al., 2022; Meng et al., 2020; Zhang et al., 2021; Meng et al., 2018; Tao et al., 2015; Park and Lee, 2022). Specifically, it requires only high-level human guidance label the text, such as few rules provided by human experts that match the text with the labels. These labels, which are not necessarily correct and are
Corresponding author.
In this paper, we take a deep dive into this prob- lem and find that, surprisingly. the high noise rate
'nttps://gi thub. www.bccoooo.....oommmmmmooo..oot www.coo////ceeeeeeeee shw nahakenk SimSeed

---

20News-Coarse
20News-Fine
less noisy pseudo-labels can indeed improve the classification accuracy significantly, making seed matching- weak supervision on par with or sometimes even better than the state-of-the-ar
0.125
Vanilla
Noise Rate
0.100
Seed Deletion
Random Flipping Noise
0.20
0.075
0.15
0.050
0.10
We further investigate the scenario where the seed words are not made known. We propose to delete every word token in the input text randomly and independently. This simple random deletion method can improve confidence learning even more as shown in Figure 1. Our theorctical analysis also shows that this random deletion method can mitigate the label bias with a high probability and therefore recover the seed deletion in effect. It is worth noting that both of these methods introduce no additional hyperparameters.
0.025
0.05
0.00
0.4
0.6
0.8
1.0
0.4 0.6 0.6. 1.0 Selection Ratio
Selection Ratio
Noise rate in the subset of pseudo-labels selected based on the confidence score of a classifier trained on the pseudo-labeled data. We show the noise rates at multiple selection ratios. ("Vamilla")): the classi- fier is trained on the original pseudo-labeled data given by seed matching. ("Random Flipping Noise"): the classifier is trained on noisy pseudo-labeled data syn- thesized using the ground-truth labels, where the data examples, the overall noise rate, noise rate in each class, and the noise transition rate between any two classes are identical to the label noise induced by seed matching. ("Seed Deletion"): "): The classifier is trained on pseudo- labeled data where the seeds are deleted from the text. (Random Deletion): The classifier is trained on pscudo- labeled data where the words in the text are randomly deleted.
Figure 1:
In summary, our contributions are as follows. We revisit the seed matching- weak super- vision and find that its effectiveness is mainly limited by the label bias injected by the seed- matching rule.
We show that simply deleting seed words from the pseudo-labeled texts can significantly alle- viate the label bias and improve the confidence estimation for pseudo-label selection, as well as end-to-end classification accuracy achieved by seed matching, on par with or even better than the tfthe-.f-
among the pseudo-labels is often not an obstacle to learning reliable confidence at all. In fact, on set of synthesized pseudo-labels labels where the noise rate is exactly the same as those given by seed matching,
We further propose the random deletion method handle the case when the seed words are un- known and demonstrate its effectiveness both empirically and theoretically.
but the noisy labels are generated by randomly flipping true labels into other classes, the confi-
dence learned by a classifier can genuinely reflect the correct labels, as shown in Figure 1.
Thereeore, we argue that the poor confidence learned on realistic pseudo-labels is largely at- tributed to the strong but likely erroneous corre- lation between the text and pscudo-label injected by the seed-matching rule, which we refer to as label bias. Such a bias can be easily learned by the text classifier upon training, thus yielding spu- riously high confidence any text matching the seed word and ruining the pseudo-label selection.
Preliminaries and Related Work
Seed matching as simple weak supervision. Seed matching (Meng et al., 2018) is probably one of the simplest weak supervision. Specifically, for each label class, user provides a set of seed words that are indicative of it. A given document is annotated as the label class whose seed words appear in the document, annotated as the label class whose seed words appear most frequently if multiple such label classes exist. Sophisticated weak supervisions have also been proposed to generate pseudo-labels with better quality, such as meta data (Mekala et al., 2020), context (Mekala and Shang, 2020), sentence representation (Wang et al., 2021), predictions of masked language models (Meng et al., 2020) and keyword-gras predictions (Zhang et al., 2021). Confidence learning matters for pseudo-label selection. Since label errors prevail in the pscudo- labels generated by weak supervision, it is often
To defend against such a label bias, we pro- pose to simply delete the seed words present in the text upon training a classifier on the pseudo- labeled data, which effectively prevents the classi- fier from learning the biased correlation between sceds and the corresponding pscudo-labels. As shown in Figure 1, such simple seed deletion method can significantly improve the confidence score of the trained classifier and thus help select labuds-labels with fewer label errors at every se- lection ratio. Empirical results verify that these

---

necessary to select the pseudo-labels before incor- porating them into training (Wang et al., 2021; Mekala et al., 2022). common method to se- lect those pseudo-labell is to use the model confi- dence, namely the probability score associated with a deep classifier's prediction, to determine whether a given pseudo-label is correct or not (Guo et al., 2017). However, such confidence often cannot gen- uinely reflect the correctness of the pseudo-label generated by weak supervision, in that pseudo- labels with high confidence are not necessarily cor- rect (Mekala et al., 2022)
document would produce a minor gain.
3.2 Random deletion
In real-world applications, the seed words provided by the user may not always be accessible due to pri- vacy concerns or simply because they are lost when processing and integrating the data. We show that it is still feasible to perform seed deletion in a proba- bilistic manner without knowing seed words, while remaining effective for confidence-based pseudo- label selection.
To achieve this, in fact, we only have to delete the words randomly and independently in a given document. Specifically, give a deletion ratio p. for every document x = {t1, 12, -",, tn), we randomly sampled few positions M (d1, i2, '"' (pp]), where [-] denotes the ceiling of number. We then generate a corrupted document by deleting words at those positions, namely
Backdoor attack and defense. A problem related to seed matching is the backdoor attack for text clas- sification based on trigger words (Dai et al., 2019; Kurita et al., 2020; Chen et al., 2021). Such attacks corrupt a dataset by inserting a particular word (i.e.. trigger) into several documents and change their corresponding labels as ones specified by the at- tacker. A model fine-tuned on such a dataset will predict the label specified by the attacker when- ever document contains the trigger word. This is largely because the model would overfit the ma- licious correlation between the trigger word and the specified label, which is similar to the prob- lem when learning pseudo-labels generated by seed matching. Therefore, trigger-based backdoor at- tacks may be defended by the methods proposed in this work as well, especially random deletion since the attacker will not reveal the trigger word.
{tyli e {1, 2, .., n), $ .M.. Now on the corrupted dataset D = {(²,)) we can train a classifier 9 and utilize its confidence score P,(iil>) for pseudo- label selection, similar to seed deletion. Random deletion as a probabilistic seed deletion. Despite its simplicity, we show with high proba- bility, random deletion can mitigate the label bias induced by seed matching. The intuition here is that since the document only contains one or few seed words, by random deletion it is very likely we can delete all seed words while retaining at least some other words that can still help learning.
3 Method
Specifically, we consider a particular type of cor- rupted document å that contains no seed word, iee, â n S3 Ø, but contains at least one word that is indicative of the true class, i.e., à. n C. # Ø, where C, denotes the set of words that are indica- tive of the class y. Since such document no longer contains the seed word, its pseudo-label is not spu- riously correlated with the text. At the same time, contains daassinddcative words that can help the classifier learn meaningful features.
3.1 Seed deletion
We describe the details of seed deletion. Specifi- cally, we denote an input document composed of set of words as x = (t1, t.. " } and its pseudo- label given by seed matching as y. We denote the set of seed words from class y as Syy Now for each document in the pseudo-labeled dataset, we gener- ate a corrupted document & by deleting any seed word in x that is associated with its pseudo-label, namely & {t|t x, & Sy). We then train classifier 0 on the corrupted dataset Đ = {(t, 9)} and use its confidence score at the pseudo-label ()) as an uncertainty measure to select the correct pseudo-labels.
We then investigate the probability that docu- ment becomes such particular type after random deletion. We term such probability as the seed- deletion rate rSD. which is defined as
Note that when generating the uncertainty mea- sure on a (document, pseudo-label) pair, one can either evaluate the classifier on the original docu- ment or the corrupted document. Empirically we found that evaluating the classifier on the original
T'D := P(1(fn 0, CC # #)), (1)
where 1(-) is the indicator function. an ideal case where FSD = 1, we can completely recover the effect of seed deletion on eliminating label bias.

---

Now since each word in the document is inde- pendently deleted, we have
Datasets. We report the text classification perfor- mance on the following datasets, including New York Times (NYT), 20 Newsgroups (20News), AG- News (Zhang et al., 2015), Rotten tomatoes (Pang and Lee, 2005), as well as NYT and 20News datasets with their fine-grained labels respectively. We select these datasets as they cover diverse data properties in text classification, such topic clas- sification and sentiment analysis, long and short input documents, seare--graiee and fine-grained labels, and balanced and imbalanced label distribu- tions.
SDD = =p" 10100pp00110000 (1-pp"),
(2)
where ns |Syl denotes the number of seed words in the document and n, = |Cyl denotes the number of words in the document that are indica- tive of the class. One may find that when ne > Yls TSD can be quite close to 1 as long as p is large. Estimate the best deletion ratio. We estimate the best deletion ratio for random deletion based on some reasonable assumptions. First, it is easy to see that based on Eq. (2), the optimal deletion ratio is
For seed word matching, we consider the seed words used in (Mekala and Shang, 2020; Wang et al., 2021). Table shows the statistics of these datasets and the corresponding pseudo-labels given by seed matching.
T F2y + n2p
which depends on both the number of seed words n, and the number of class-indicative words De in the document. For ng. we can simply set it as 1 since the pseudo-labed labell of document is usually determined by one or two seed words. For ne. we assume that all words document are indicative of the true class, except stop words and punctuation. These estimations are acceptable as p is almost al- ways close to 1 and is quite robust to the change of nt and n. as long as ne is large (See Figure 2). This condition is likely to be true for realistic datasets (See Table 1). Note that for simplicity, we set one single deletion ratio for specific dataset. Thus we n. the median number of class-indicative words over all documents in a dataset.
Training setting. We adhere to the follow- ing experiment settings for all methods un- less otherwise specified. We utilize BERT (Devlin et al., 2019) as the text classifier since we found that BERT can produce reliable confidence estimates with our methods in our experiments.
For pscudo-label selection, we select 50% of the high-quality pseudo-labels for all methods and datasets. For random deletion specifically, the deletion ratio following the estimation in Sec- tion 3.2. The estimated best deletion ratio of each dataset can be found in Figure 3.
set
Finally, we employ the standard self- self-training pro- tocol to utilize both the documents labeled by weak supervision and additional documents that are not labeled. Note that we selected subset of pseudo- labels before self-nraining, those ascudo-labeled documents that were not selected will be merged into unlabeled documents. For the self-training process specifically, we train text classifier on the labeled documents and generate predictions unlabeled documents. The top 7 fraction of predic- tions with the highest confidence are then treated as new pseudo-labets and will be merged with the existing pseudo-labels for another training. In our experiments, we conduct this self-training process for 5 iterations.
0.9
0.8
0.7
0g = 1
0.6
2
0g 5
05-
0
20
40
60
100
nc
Figure 2: The optimal deletion ratio for random deletion with respect to the number seed words n, and the number of class-indicative words ne based on Eq. (3).
4 Experiments
Note that one can use BERT to select high- quality pscudo-labels alone following our methods while employing advanced text classifiers for sub- sequent self training, which may further improve the performance.
4.1 Experiment setup
We evaluate the performance of seed word match- ing equipped with seed deletion or random deletion on text classification.
Comparative methods. We compare our proposed

---

Table 1: Statistics of the dataset and the corresponding pseudo-labets given by seed matching, For sequence length, report the median across all habele- documents reduce the impact of outliers, along with the median absolute deviation in the bracket.
# Pseudo-labeled Docs
Pscudo-label Noise Rate (%)
Sequence Length (Pseudo-labeled Docs)
n. (Estimated)
Dataset # Does
Labels
(Pseudo-labeled Docs)
AGNews
120,000
4
32,359
16.26
39(6)
27(()
20News-Coarse
17.871
5
7,671
12.50
286 136)
14066)
NYT-Coarse
13,081
5
9,460
11.47
922(222)
462( 105)
20News-Fine
17,871
17
10,455
25.67
257(
129(58)
NYT-Fine 13,081
26
8,229
31.80
940(214)
467(100)
Motten-Tomatten
10,662
2
990
28.38
26(8)
13(4)
Table 2: Classification performance achieved by vamilla seed matching and seed matching equipped with various pseudo-label selection methods. indicates methods that are not fair comparison and are listed only for reference. We conduct each experiment for 5 times, report the average, and denote the standard deviation in the bracket. We report both Macro-FI and Micro-FI for classification accuracy.
AGNews
29News-Coarse
NYT-Coarse
20News-Fine
NYT-Fine
Rotten- Tomatoos
Method
Mi-FI
Ma-FI
Mi-FI
Ma-F1
Mi-FI
Ma-FI
Mi-FI
Ma-FI
Mi-F1
Ma-F1
Mi-F1
Ma-FI
Oracle' 86.3(0.3) 86,2(0.3) 90.6(0.4) 90.600.4) 97.000.)
93.9(0.4)
81.1(0.) 81.1(0.3)
96.5(0.1)
92.0(0.4)
79.6(1.0) 79.6(1.0)
Vauilla 83.9(0.6) 83.900.6) 80.5(0.6) 80.1(0.6) 87.8/022) 78.40..4 68.2(0.6)
69.000.7)
73.0(0.7)
68.70088
71.4(1.) 71.3(1.)
Standard Confidence 82.2/2.1) 82.012.1) 78.9(1.)
80.0(1.)
88.3(4.1)
79.1(2.8)
64.4(2.2)
66.6(188
45.8(0.5)
58.610.3)
72.012.) 72.2(2.4)
O2U-NNet 79.8(0.5) 79.810 80.910.3)
78.5(0.2) 92.9(0.4) 85.9(0.7) 71.1(0.4)
71.2(0.8) 14.7(10:2)
74.1(1.4) 8,70(733)
74.0(1.4)
LOPS 79.5(0.9) 79.5(0.6)
81.(010)
80.7(0.4) 94.6(0.4)
88.4(0.5)
73.8(0.))
72.7(1.0)
84.3(0.5)
81.60.3
70.4(0.4)
70.400.4)
Seed-Deletion 84.3(0.7) 84.2(0.7)
86.4(0.9)
(6.1 92.4(1.) 85.0(2.0)
73.7(0.7) 75.000.5)
81.7(1.5)
79.4(1.1) 70.4(1.) 70.3(13)
Random-Deletion 86.2(0.5) (6.1(0.5)
84.4(0.9)
84.8(088)
91.((13) 83.3(1.8)
76.3(0.8)
76.8(0.7)
84.611.4)
79.6(1.1)
73.6(4.3)
73.4(4.5)
Paraphrase 85 4(0.3) 400.))
86.9(12)
86.((12)
94.000.8
88.5(0.7)
75.600.6)
76.800.5)
76.1(4.0)
74.8(2.4)
75.400.1)
75.3(0.1)
MI M-pplRee 85 1)0.1) 85.8 87.4(0.1) 87.500.2) 94.5(0.1)
88.9(0.2) 73.6(1.0 .)) 74.8(088) 1(0.))
80.010.4)
76.7(15) 76.70-5)
method with the following baselines.
4.2 Main results
Seed-based weak supervision. We present the classification performance achieved by different methods in Table 2. One may find that seed dele- tion and random deletion significantly boost the performance of seed matching- based weakly- supervised classification. On some datasets (e.g.. AGNews), random deletion can approach the oracle selection with almost no gap. This demonstrates the performance of the simple seed matching- weak supervision is greatly underestimated.
Vanilla: Self-training on all the pseudo-labeled provided by seed matching, without pseudo-label selection.
Standard confidence: Train a classifier on all the pseudo-laheled documents and use its confidence score to select a subset of high-quality pseudo- labels.
O2U-Ner (Huang et al., 2019): Train a classifier all the pseudo-labeled documents and use the normalized loss of each document throughout the training as a metric to select pseudo-labels.
Seed deletion and random deletion are also on par with or significantly better than other pseudo- label selection methods including those using so- phisticated confidence measures such as the nor- malized loss in O2U-Net and the learning order in LOPS. In fact, seed deletion and random deletion are still using the standard confidence score of a classifier to select pseudo-labels, albeit they first corrupt the pseudo-labeled documents for training the classifier. Nevertheless, the performance im- provement compared to the standard confidence score is huge. For example, the improvement on NYT-Fine is as large as 20% in terms of Macro- F1 and 40% in terms of Micro-F1. This demon- strates that confidence- pseudo-label selec- tion is greatly underestimated.
LOPS (Mekala et al., 2022): Train a classifier on all the pseudo-labeles documents and use the learning order cached during training to se- lect pseudo-labels. We follow the setting recom- mended in their paper and set 7 50%.
Oracle*: * Based on the true labels, select only those correct pseudo-label, for self-training. Note that this is not realistic method and is used only for comparison.
Whenever it is necessary to train classifier to obtain the confidence, we train for 4 epochs, in line with the setting in LOPS.

---

Table 3: Classification performance of text classification using a variety of weak supervisions. Results on sophis- ticated weak supervisions are cited from the corresponding paper. indicates that result is reported as accuracy instead of Fi-score in the original paper. while we still include it here since the dataset is SottenTomaatees here since is not reported in most of the listed papers.
We neglect
AGNews
20News-Courd Course
NYT-Coarse
20News-Fine
NYT-Fine
Method
Mi-FI
Ma-FI
Mi-FI
Ma-FI
Mi-FI
Ma-FI
Mi-FI
Ma-FI
Mi-FI
Ma-FI
ConWea
73.4
73.4
74.3
74.6
93.1
87.2
68.7
68.7
87.4
77.4
X-Class
82.4
82.3
58.2
61.1
96.3
93.3
70.4
70.4
86.6
74.7
LOTClass
84.9
84.7
47.0
35.0
70.1
30.3
12.3
10.6
5.3
4.1
ClassKG
88.8"
80
75
96
83
78
77
92
80
LIME
87.2
87.2
79.7
79.6
Seed Deletion
84.3
84.2
86.4
86.1
92.4
85.0
73.7
75.0
81.7
79.4
Random Deletion
86.2
86.1
84.4
84.8
91.7
83.3
76.3
76.8
84.6
79.6
Compare with sophisticated weak supervision. In Table 3, we compare the performance of seed word matching equipped with seed word deletion or random deletion with those methods using sophis- ticated weak supervision sources, listed as follows.
Alternative seed-agnostic debiasing methods. We explore alternative methods to delete the seed words and mitigate the label bias in seed matching- based weak supervision, without knowing the seed words. We consider the following alternatives.
ConWea (Mekala and Shang, 2020) uses pre- trained language models to contextualize the weak supervision in iterative manner.
MLM-replace: We randomly mask subset of words in the document and use BERT to pre- dict the masked words. The document is then corrupted by replacing the masked words with the predictions. This follows the idea of random deletion to delete the seed words probabilistically. Such a method is widely used in other applica- tions (Gao et al., 2021; Clark et al., 2020).
X-Class (Wang al., 2021) learns class- orisntore document representations based the label sur- face names. These document representations are aligned to the classes to obtain pseudo labels.
LOTClass (Meng et al., 2020) obtains synonyms for the class names using pretrained language models and constructs category vocabulary for each class, which is then used to pseudo-label the documents via string matching.
Paraphrase: We generate the paraphrase of a doc- ument using the T5 model (Raffel et al., 2019) fine-tuned on paraphrase dataset APWS (Zhang al., 2019). We the publicly available im- plementation (Duerr, 2021). This is a straight- forward method delete the seed words.
ClassKG (Zhang et al., 2021) constructs key- word graph discover the correlation between keywords. Pseudo-labeling a document would be translated into annotating the subgraph that represents the document.
Since these alternative methods only serve as a reference for main methods, we search their best hyperparameter, namcly the mask ratio for MLM-replace and the token-generation tempera- ture for paraphrase respectively. As shown in Ta- ble 2, these alternative methods can work as well or better than random deletion. However, in prac- tice, we would prefer using random deletion since it requires no extra model or knowledge source.
LIME (Park and Lee, 2022) combines seed matching with an entailment model to better pseudo-label documents and refine the final model via self-training.
For each method, we report the results obtained di- rectly from the corresponding paper. If the results on some datasets are not reported in the original pa- per, we cite the results in follow-up papers if there are any. We found that with seed deletion or ran- dom deletion, seed word matching can be almost good as even better than those sophisticated weak supervisions. We thus believe seed word matching can be a competitive baseline and should be considered when developing more complicated weak supervisions.
4.3 Study on random deletion
Deletion ratio in random deletion. We verify whether our estimation of the best deletion ratio is reasonable. In Figure 3, we modulate the deletion ratio and check the classification performance of random deletion for different datasets. One can find that as the deletion ratio increases, the performance first increases and then decreases, which is aligned

---

AGNews
20News
NYT-Coarse
20News- Fine
NYT-Fine
notten-Tomatoes
3.8
8.6
3.75
8
3.50
3.50
0.4
02
3.00
90
Macro-F3
66-
75
Sood Da sbor
Zandert Coridance
73
62
0.3
05 1.
0.0
0.3
10
0.0
055 1.2
0.0
2.5 10
00
1.2
5.0
055
1.5
Deletion Ratio
Deletion Ratio
Deletion Ratio
Deletion Ratio
Deletion Ratio
Deletion Ratio
Figure 3: (Top) The seed-deletion rate SDD given different deletion ratios (Eq. (2)), where n, and n. are estimated for each dataset mentioned in Section 3.2. (Bottom) The classification performance of random deletion given different deletion ratios. "(p")" denotes the one using the best deletion ratio estimated for each dataset. We also denote the performance of seed deletion and standard confidence for comparison.
with the trend of the seed-deletion rate TSD ana- lyzed in Section 3.2. The performance peaks when the deletion ratio is large (z 0.9), which matches estimation of the best deletion ratio. Further- more, one may find that the best deletion ratio rel- atively smaller for datasets with a shorter sequence length (c.g., AGNews and Rotten-Tonee Tomatoes), com- pared to that for datasets with a long sequence length (e.g., 20News and NYT), which is also pre- dicted by our estimation.
standard random deletion. We experiment on two representative datasets including 20News-Coarse and 20News-Fine due to computational constraints. Figure 4 shows that when seed words are always retained, random deletion achieves significantly worse performance than the standard random dele- tion, although the former is merely deleting one or two words fewer. On the other hand, when seed words are already deleted, further random deletion only slightly improves the performance compared to the standard random deletion. These pieces of evidence demonstrate that the benefit of random deletion may be partly attributed to a regulariza- tion effeet, but the deletion of sceds and thus the mitigation of label bias still one important factor. Compare with additional regularization meth- ods. Since random deletion may introduce an ad-
How does random deletion work?. One may no- tice that in Table 2, random deletion can outperform seed deletion on a few datasets. This indicates that random deletion has an additional regularization effect on top of deleting seeds, potentially due to more intense data augmentation.
2ONews-Coarse
20News-
Random Deletion
20News
Random Deletion
Random Deletion (Retain seeds)
Drop (all)
Random Deletion [Delete seeds)
Drop (wd-embed)
Drop2D (wd-embed)
Macro-F1
Macro-F1
Drap2D (embed)
Figure 4: Classification performance achieved by vari- ations of random deletion with seed words always re- tained in the document ("Retain seeds") and with seed words always deleted completely ("Delete seeds").
72
Macro-F1
Macro-F1
Figure 5: Classification performance using dropout as regularization method for pseudo-label selection. We try two types of dropout including Dropout ("Drop"") and Dropout-2D ("Drop2D"). We try dropout at various positions in the transformer architecture, including all dropouts layers in the original transformer ("all"), the embedding layer only ("embed"), and the word embed- ding layer only ("wd embed").
However, we note that random deletion works not entirely because of this additional regulariza- tion effect. To show this, we conduct ablation ex- periments with two additional methods. The first is random deletion but with the seed words always re- tained in the document. The second is random dele- tion but with seed words first deleted completely. For a fair comparison, we search the best delc- tion ratio for these different methods including the
ditional regularization effect, we compare it with other regularization methods that can potentially reduce the label bias. We will mainly compare dif- ferent types of dropout (see below) that are widely

---

Table 4: Original document versus corrupted document after random deletion, along with the corresponding pseudo-label and true label. Here the example documents are randomly picked from the pseudo-labeled data in the 20News-Coarse dataset.
Corrupted Document (Random Deletion)
Original Document
Pseudo-label
Computer
incrensing number serial ports distribution world neep posting bost muce, jpl. nasa. guv in article ( steven langlois wrote does unyone mac which will increase
distributtion poating4 available upplication working must independenily. such any the serial? ink bug only system the, them then of the, to using
Seed Word
mac
know there are any devices available the number of scrial ports available for use simultancussly? i would like connect up serial devices my for application am working on.
True-label
Computer
Pseudo-label
Computer
the here pointless options. you know, stuff you get on a car that has no earthly use? power windows like my power windowx, think they's worth it however, craise control is pretty dumb option. what's is the point? you 'e on long trip. you floor the gas and keep your eyes on the rear view for cops. right? power seats are preety unlucky enough to have to share your car...
dumb options in artiele charkss parr
options you ).? keep are enough have like do puper. breath.
Seed Word
windows
too, uniless you're
True-label
Sports
Pseudo-label
Science
pro abortion femninist leader endorses trashing of lree speech rights in article { gordon fitch) ) writes ( doug holtsinger writes arrested defying judge's order at aboction protest rally the miami herald, april II, megregor's s order prohibits anti abortion pickets wilhin 36 fict of the property line of aware woman center for choice. even the they may not display pictures dead fetuses sing chant load enough to be heard by pasients inside the climic. ...
leader trashting judge 11 judge abortion the display pictures lond similar appeaa group from have rock and then see the homes did speech of hear there from to expression on putticular information, others if be considered the 'ists arson to else the
Seed Word
circuit
1993 circuit judge
True-label
Polities
used in classification tasks, since it is mostly simi- lar to random deletion. We defer other commonly seen regularization methods the appendix. We consider applying dropout to different positions in the transformer model. For a fair comparison, we search the best dropout ratio for these dropout methods, as well as the best deletion ratio for our random deletion. We experiment on two represen- tative datasets due to computational constraints.
tion effect. Case study. We manually inspect the pseudo- labeled documents after random deletion see if the label bias can be mitigated. In Table 4, we ran- domly pick some example documents after random deletion and find that the seed words are indeed deleted and some class-indicative words are still present to allow effective classification.
5 Conclusion and Future Work
Dropout (Hinton et al., 2012) is a classic regu- larization method to prevent overfitting, We sim- ply utilize the dropout layers built in the original transformer architecture, including those after the embedding layers, self-attention layers, and feed- forward layers, following previous work (Gao et al., 2021).
In this paper, we revisit the simple seed matching- based weakly supervised text classification method and show that if its pseudo-labels are properly de- biased, it can achieve state-of- -o-the--rttttttt- accuracy on many popular datasets, outperforming more sophis- ticated types of weak supervision. Specifically, our controlled experiments show that confidence-bll basee selection of seed matching-baser ineffective largely because of the label bias injected by the simple, yet erroneous seed-match rule. We propose two effective debiasing methods, seed dele- tion, and random deletion, which can mitigate the label bias and significantly improve seed matching- based weakly supervised text classification.
Dropout-2D (Tompson et al., 2014) is different from vanilla Dropout in that it drops the entire channel as a whole. We only apply this to the embedding layer in the transformer to drop the entire embedding of a word or position.
useudo-labelt is
As shown in Figure 5, random deletion consis- tently outperforms other regularization methods. The only regularization method that can compete with random deletion is Dropout-2D applied on the word embedding layer specifically. However, one may note that this dropout variation is in fact almost equivalent to random deletion since the entire embedding of a word will be randomly dropped. These again demonstrate that random deletion works not simply because of a regulariza-
In future work, we plan to extend this debias- ing methodology to broader problems and methods. For example, for weakly supervised text classifi- cation, we wish to explore generalization of the debiasing strategy to more sophisticated types of weak supervision. It will also be interesting to de- velop a backdoor defense framework around the

---

proposed methods, especially random deletion.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models. In ICMI. 2027 Workshop on Adversarial Machine Learning.
Limitations
We have shown that randomly deleting the words work well for seed-matching- babeedgg weak supervi- sion without knowing the seed words. However, this idea might not generalize straightforwardly to more sophisticated types of weak supervision. We have tried to apply random deletion to X-Class, but have not observed a significant improvement in the text classification performance. We hypothesize that this is because the pseudo-labels in X-Class are not generated based on seed word matching, but rather based on the similarity between the label em- beddings provided by pretrained language models. We believe label debiasing method universally applicable to all types of weak supervision is still under-explored.
Kevin Clark, Minh-Thaae Thang Luong, Quoc Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders discriminators rather than generators. ArXiv, 00os2203.105555
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019, A backdoor attack against Istm- sstm-based text classification systems. IEEE Access, 1113872-1388777
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Pupers), pages 4171-4186 Minneapolis, Minnesota. Association for Computational Linguistics.
Ethical Consideration
Sehastian Duemr 2021. Sedueen/tS-aawrapraas hug- ging face.
This paper analyzes the difficulty of identifying la- bel errors in the pseudo-labels generated by simple seed-matching weak supervision. This paper pro- posed methods better identify such label errors and improve weakly supervised text classifiers. We do not anticipate any major ethical concerns.
Yarin Gal and Zoubin Ghahramani. 2015. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. ArXiv, abs/1506.02111
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings, ArXiv, abs/2104.08821.
A Acknowledgements
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q Wein- berger. 2017. On calibration of modern neural net- works. In International Conference on Machine Learning, pages 1321-1330. PMLR.
We thank the anonymous reviewers for their help- ful feedback. Our work is sponsored in part by NSF CAREER Award 2239440. NSF Proto-OKN Award 2333790, NIH Bridge2A1 Center Program Cisco-UCSD
Geoffrey E. Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov, 2012. Im- proving neural networks by preventing co-adaptation of feature detectors. ArXiv, abs/ 22w7/05000000
under award 1 JU54HG012510-00 Sponsored Research Project, as well as generous gifts from Google, Adobe, and Teradata. Any opin- ions, findings, and conclusions or recommenda- tions expressed herein are those of the authors and should not be interpreted necessarily represent- ing the views, either expressed or implied, of the U.S. Government. The U.S. Government autho- rized to reproduce and distribute reprints for gov- emnment purposes not withstanding any copyright annotation hereon.
Jinchi Huang, Lie Qu, Rongfei Jia, and Binqiang Zhao. 2019. O2u-net: A simple noisy label detection ap- proach for deep neural networks. In Proceedings of the IEEE/CVF International Conference on Com- puler Vision, pages 3326-3334 3334.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pre-trained models. arXiv preprint arXi::2004.0666
Dheeraj Mckala, Chengyu Dong, and Jingbo Shang. 2022. Lops: Learning order inspired pseudo-label selection for weakly supervised text classification. ArXiv, 122222
References
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas,
David Krueger, Emmanuel Bengio, Maxinder Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. 2017. closer look memorization in deep net- works. ArXiv, www//106.00 706.05394.
Dheeraj Mekala and Jingbo Shang. 2020. Contextu- alized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 323- 333.

---

Dheeraj Mekala, Xinyang Zhang, and Jingbo Shang. 2020. Meta: Metadata-empowere weak supervision for text classification. In Conference Empirical Methods in Natural Language Processing.
Yuan Zhang, Jason Baldridge, Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. ArXiv, abs//904.00 130.
Xuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu, and Lei Li. 2022. Pre-trained language models can be fully zero-shot learners. arXiv preprint arXiv:2212.06950.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervisee supervised neural text classification. In Proceedings of the 27th ACM International Con- ference Information and Knowledge Management, pages 983-992. ACM.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji. Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the ACL.
Seongmin Park and Jihwa Lec. 2022. Lime: Weakly- supervised text classification without seeds. ArXiv, abs/2210.06722)
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lec, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/19 10683.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh Singh Rawat, and Mubarak Shah. 2021. defense of pseudo-labeling An uncertainty-avee aware pseudo-label selection framework for semi-supervised learning. ArXiv, abs/2 10.069...
Fangho Tao, Chao Zhang, Xiusi Chen, Meng Jiang, Tim Hanratty, Lance Kaplan, and Jiawei Han. 2015. Doc2cube: Automated document allocation to text cube via dimension-awan joint embedding. Dimen- sion, 2016:2017.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. 2014. Efficient ob- ject localization using convolutional networks. 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 648 656.
Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2021. X-class: Text classification with extremely weak su- pervision. In Proceedings of the 202] Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3043-3053 Online. Association for Computational Linguistics.
Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021. Weakly supervss text clas- sification based on keyword graph. In Conference on Empirical Methods in Natural Language Processing.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28:649-657.

---

A Appendix
training epochs for early stopping respectively. As shown in Figures 7 and random deletion consis- tently outperforms the baseline for different con- fidence regularization methods. Random deletion also achieves more robust performance across dif- ferent hyperparameler settings, which is important for
Effect of selection ratio. As mentioned before, our methods do not introduce additional hyperpa- rameters. Nevertheless, for pseudo- selection in general, the selection fraction could be an impor- tant hyperparameter as the noise rate in the selected subset of pseudo-labels can vary significantly if select different fractions, as also shown in Figure Therefore, we check the performance of the stan- dard confidence-basse pseudo-labee selection and the confidence-bart -based selection equipped with seed deletion and random deletion, as the selection frac- tion varies. l'igure 6 shows that our proposed meth- ods are consistently better than standard confidence and achieve relatively robust performance the selection fraction varies. The performance peaks when the selection fraction is moderate (~ 50%) for different datasets.
classification since we often lack large clean dataset to select the best hyper- parameter.
2ONews-Coarse
20News-Fine
86
74
Macro-F1
72
82-
Vanilla Delletion
68
109
102
100
10:
Number of Passes
Number of Passes
Figure 7: Classification performance using MC-dropout to obtain better confidence for pscudo-label selection. We test different numbers of passes for MC-dropout.
20News-Coad
20News-Fine
85
Macro-F1
75-
2ONews-Coarse
20News-Fine
75
70
Vanili
77.5
Rand Delction
Seed Deletion
65
Macro-F1
75.0
0.8
7..5
Selection Ratio
Selection Ratio
Figure The classification performance when selecting the pscudo-labels at different fractions.
75
Vanilb
Dal
10a
100
10a
Number of Epochs
Number of Epochs
Experiments on additional confidence regular- ization methods. We experiment on additional methods for regularizing the confidence learning of the text classifier, including the following.
Figure 8: Classification performance using early stop- ping as a regularization method to obtain better confi- dence for pseudo-label selection. We check the perfor- mance when early stopping at different epochs.
MC-Dropout (Gal and Ghahramani, 2015) ran- domizes the network inference process by dropping intermediate activations. The aver- age output over multiple inferences utilized as a more reliable confidence score.
Early stopping is often utilized as regulariza- tion to help mitigate overfitting. Empirically, it is observed that an early-stopped model is less prone to learning noisy data (Arpit et al., 2017).
Here, we treat each method baseline and com- pare it with the corresponding method combined with random deletion. For each method, we mod- ulate its most important hyperparameter, namely number passes for MC-Dropout and number of


Reviews:
[{"review_id": "mUjmmV1PPR", "reviewer": "Reviewer_PSwo", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. Please see our reponse below.\n\n## ____\n__Q. Clarification of the results in Table 3__\n\nWe assume the reviewer is referring to the results reported in Table 3. However, note that here the comparison between the competitive methods and ours is not entirely fair, since they often employ different base classifiers and/or additional datasets. For example, ClassKG [1] uses Longformer [2] for the classification of long-text datasets, which include the 20News-groups and New York-Times datasets (4 out of 5 datasets we used in our experiments). And LIME [3] employs BART-large [4] as the model to select pseudo-labels. They also fine-tune the model on MultiNLI [5] before being used for pseudo-label selection.\n\nNevertheless, our method can still achieve results that are quite close to these methods. On 4 out of 5 datasets, the gap between our result and the best existing result is less than or equal to 1 (Macro-F1). We believe this demonstrates that our method can be on par with the state-of-the-art while being significantly simpler.\n\n## ____\n__Reference__\n\n[1] Weakly-supervised Text Classification based on Keyword Graph. Zhang et al., 2021.\n\n[2] Longformer: The Long-document Transformer. Beltagy et al., 2020.\n\n[3] Lime: Weakly-supervised text classification without seeds. Park et al., 2022.\n\n[4] BART: Denoising Sequence-to-sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Lewis et al., 2020.\n\n[5] A Broad-coverage Challenge Corpus for Sentence Understanding through Inference. Williams et al., 2018.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method although simple, is very interesting and shows competitive results or even outperforms other more complicated methods.\n- The authors provide an extensive study about how the deletion of seed words should be done and perform experiments that give insights on how the random deletion works and the decision of the deletion ratio.\n", "reasons_to_reject": "The proposed method has lower scores than its competitors in 3 out of the five datasets used for evaluation.", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper the authors propose an improvement for the seed matching-based method for weak supervision. They define the label bias injected by the seed-matching rule and argue that this is the cause of the poor performance of the method. They implement two methods to delete seed words from pseudo-labeled texts and show that they improve the results.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C7pwGBX66c", "reviewer": "Reviewer_wbLQ", "summary": "", "questions": "1) Why did you select BERT? Do you expect significant changes in the results if a different classifier is used?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. Please see our reponse below.\n\n## ____\n__Q. BERT as the text classifier__\n\nWe believe BERT is arguably the most widely used base model for text classification. Existing works on weakly-supervised text classification also often employ BERT (or its variation) as the single base classifier to demonstrate the results, e.g., LOTClass [1] and X-Class [2].\n\nFurthermore, we expect our methods can still help for other base classifiers.\nThis is because our methods are specifically designed to better select the correct weak pseudo-labels, by leveraging the confidence estimates of a model. In our experiments, we found that BERT can produce reliable confidence estimates with our methods. \nOnce the subset of pseudo-labels is determined, one can in fact train with any text classifier, which should achieve consistently better classification performance with higher-quality pseudo-labels.\n\n## ____\n**Reference**\n\n[1] Text Classification Using Label Names Only: A Language Model Self-training Approach. Meng et al., 2020. \n\n[2] X-class: Text Classification With Extremely Weak Supervision. Wang et al., 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The novelty and the contributions of the paper are very well justified. The analyses are extensive and show a clear advantage of the method over exsiting approaches, including state-of-the-art. The methods are well described and easy to implement/adjust to wider range of the datasets and classification tasks. ", "reasons_to_reject": "The use of a sinlge classifier - BERT. The results might differ if different classifiers are used. ", "questions_to_authors": "1) Why did you select BERT? Do you expect significant changes in the results if a different classifier is used?", "paper_topic_and_main_contributions": "The aim of the paper is to improve the performance of weakly supervised text classifiers by revisiting the seed matching-based method for generating pseudo-labels. Prelimiary study presented in the paper shows that the limited performance of seed matching-based methods is due to a bias caused by the seed words which prevents the classifier from learning reliable confidence for high quality pseudo labels. The authors propose two simple techniques for overcoming this problem, based on seed word deletion and random deletion. The seed word deletion method is based on deleting the seed words from the the pseudo-labeled texts. The random deletion method aims to handle cases where the seed words are unknown and it is based on random token deletion in a probablistic manner. Both methods are shown to mitigate the label bias problem caused by the seed words and help improve the performance of the classifier. For their experiments, the authors used 4 datasets (New York Times, 20 Newsgroups, AG-News, and Rotten tomatoes) and utilise BERT as text classifier. The authors compared the proposed methods with a wide range of seed-based weak supervision approaches as well as more sophisticated weak suprivision techniques. Results showed that the proposed methods outperform other seed-based approaches and also perform on par  with or even better than more sophisticated techniques.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "htZVOVgNuS", "reviewer": "Reviewer_Lv81", "summary": "", "questions": "- Question A: What are the actual deletion ratios used for the 6 datasets in Table 3 (maybe I overlooked this info?)?\n- Question B: Since the deletion ratio p seems to be quite high at p>= 0.9 according to Sec 4.3, this means that per document <= 10% of the class-indicative tokens are retained, plus punctuation. If a given class contains >10 documents, I’d assume due to the random/uniform sampling, each class-indicative word may still occur in one of the docs. What happens in the case of very long-tail classes with e.g. <10 documents - in this case, at least some of the class-indicative words are lost since they get deleted in all documents, correct? How is the performance in this case? And also - for very long documents, doesn’t deleting so many class-indicative words mean that the ratio of punctuation (non-informative) to words (informative) is very skewed towards the former? Does that affect classification performance? (Or maybe I've misunderstood the deletion procedure?)", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. Please see our reponse below.\n\n## ____\n__Q. Clarification of the deletion ratio__\n\nThroughout the paper, we determine the deletion ratio of random deletion based on our estimation in Section 3.2 (Equation (3)). One can find the estimated deletion ratio for each dataset in Figure 3. We are sorry for the confusion and will make it clear in the revision.\n\n## ____\n__Q. Performance on long-tail classes and long documents__\n\nFirst, we note that our random deletion is not static but performed along with training. This means that for classes with very few documents (long-tail classes), the classifier can still see all the class-indicative words as long as the training is conducted for sufficiently many epochs. \nOne can also employ standard strategies for dealing with highly imbalanced datasets (e.g., upsampling), where random deletion should properly fit in, since the seed-deletion rate (i.e., Equation (1), the probability that any training document contains no seed word but at least one class-indicative word) remains the same.\n\nSecond, for long documents, the ratio between class-indicative words and non-class-indicative words should remain approximately the same when we perform random deletion, even for a very high deletion ratio. This is because the deleted words are uniformly sampled, which means the class-indicative words and non-class-indicative words have an equal chance of being deleted. \n\nFinally, we would like to emphasize that the performance of the text classifier when training with random deletion is not a major concern. This is because random deletion (as well as seed deletion) is designed to improve the confidence estimates of the text classifier, such that they can be used to select the correct pseudo-labels. After the high-quality pseudo-labels are selected, one can use any classifier to achieve the best performance.\n\nIn fact, existing works have shown that the performance of classification models is usually orthogonal to its confidence estimates [1]. In our experiments, we also observe that the classifier used to select the pseudo-labels often has low accuracy but well-calibrated confidence estimates.  \n\n## ____\n__Reference__\n\n[1] On Calibration of Modern Neural Networks. Guo et al., 2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- paper is well organized and easy to follow\n- very interesting analysis on \"debiasing\" seed-based pseudo-labeling    \n- experimental results are convincing", "reasons_to_reject": "- I can’t find any", "questions_to_authors": "- Question A: What are the actual deletion ratios used for the 6 datasets in Table 3 (maybe I overlooked this info?)?\n- Question B: Since the deletion ratio p seems to be quite high at p>= 0.9 according to Sec 4.3, this means that per document <= 10% of the class-indicative tokens are retained, plus punctuation. If a given class contains >10 documents, I’d assume due to the random/uniform sampling, each class-indicative word may still occur in one of the docs. What happens in the case of very long-tail classes with e.g. <10 documents - in this case, at least some of the class-indicative words are lost since they get deleted in all documents, correct? How is the performance in this case? And also - for very long documents, doesn’t deleting so many class-indicative words mean that the ratio of punctuation (non-informative) to words (informative) is very skewed towards the former? Does that affect classification performance? (Or maybe I've misunderstood the deletion procedure?)", "paper_topic_and_main_contributions": "The paper addresses the task of weakly supervised text classification, specifically when employing seed word lists per class to generate pseudo-labels. Naively training a SSL model on these pseudo-label may result in the classifier learning a “biased”, simple correlation between those seed words and the class label, which does not generalize well. The authors show that this seed bias can be overcome by deleting seed words from the document before training. In addition, they show that a random deletion strategy works even better when a rather larger fraction of words per document is deleted. Experiments on 6 text classification datasets show that both deletion approaches can achieve SotA accuracy, compared to more sophisticated pseudo-labeling strategies.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work focuses on the problem of weakly labeled data for text classification and proposes a simple yet effective method. Reviewers unanimously applauded the motivation, presentation, and novelty of the work. Only minor issues were regarding breadth of experiments and missing references, none of which detract from the quality of the work.
Decision: Accept-Main