MEGClass: Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities
Authors: Priyanka Kargupta
Tanay Komarlu
Susik Yoon
Xuan Wang
Jiawei Han
Venue: EMNLP 2023 Findings (2023)
Keywords: text classification
extremely weak supervision
weakly-supervised learning
document representations
representation learning
pseudo-document generation
TL;DR: This paper proposes an extremely weakly-supervised text classification method using mutually-enhancing text granularities (word, sentence, and document-level context).

Abstract: Text classification is essential for organizing unstructured text. Traditional methods rely on human annotations or, more recently, a set of class seed words for supervision, which can be costly, particularly for specialized or emerging domains. To address this, using class surface names alone as extremely weak supervision has been proposed. However, existing approaches treat different levels of text granularity (documents, sentences, or words) independently, disregarding inter-granularity class disagreements and the context identifiable exclusively through joint extraction. In order to tackle these issues, we introduce MEGClass, an extremely weakly-supervised text classification method that leverages Mutually-Enhancing Text Granularities. MEGClass utilizes coarse- and fine-grained context signals obtained by jointly considering a document's most class-indicative words and sentences. This approach enables the learning of a contextualized document representation that captures the most discriminative class indicators. By preserving the heterogeneity of potential classes, MEGClass can select the most informative class-indicative documents as iterative feedback to enhance the initial word-based class representations and ultimately fine-tune a pre-trained text classifier. Extensive experiments on seven benchmark datasets demonstrate that MEGClass outperforms other weakly and extremely weakly supervised methods.

MEGClass:
Extremely Weakly Supervised Text Classification via Mutually-Enhancing Text Granularities
Priyanka Kargupta', Tanay Komarlu', Susik Yoon', Xuan Wang', Jiawei Han' DDpartment of Computer Science, University of Illinois at Urbana-Champaign
DDeparmentt of Computer Science at Virginia Tech [pk36, tkomarlu2, susik, hanj)@illinois. xuanwêvt.. edu
Abstract
(1) education at all levels The taacher tought the stupionts at school today...
(2) education at document level, politics at the word level The teacher explained the checks and balanoes between the evecutive and judiciai bronches the students during government closs roday
Text classification is essential for organizing unstructured text. Traditional methods rely on human annotations or, more recently, a set of class seed words for supervision, which can be costly, particularly for specialized or emerging domains. To address this, using class surface names alone as extremely weak supervision has been proposed. However, existing approaches treat different levels of text granularity (docu- ments, sentences, or words) independently, dis- regarding inter-granularity class disagreements and the context identifiable exclusively through joint extraction. In order to tackle these issues, we introduce MEGClass, extremely weakly supervised text classification method that lever- ages Mutuaaly-Enhanccin Text Granularities. MEGClass utilizes coarse- and fine- mee-graieed context signals obtained by jointly considering a document's most -aas-indieettve words and sentences. This approach enables the learning of a contextualized document representation that captures the most discriminative class indi- cators. By preserving the heterogeneity of po- tential classes, MEGClass can select the most informative class-indicative documents itera- tive feedback to enhance the initial word-basee based class representations and ultimately fine-tune a pre- rre-trained text classifier. Extensive exper- iments on seven benchmark datasets demon- strate that MEGClass outperforms other weakly and extremely weakly supervised methods.
(3) politics and education at all levels
The Calfarnia legisiature possed a new bill increasing the vollory of elementary and high school teachers....
These are the three document types featured within a corpus. While existing methods can only distin- guish between (1) and (3), MEGClass addresses all three types as well as minimizing (3) from the constructed pseudo-training dataset.
Figure 1:
pervision of domain experts. Recent works have ex- plored weakly supervised text classification, where a few labeled documents or class seeds are pro- vided to serve as weak supervision (Meng et al., 2018; Mckala and Shang, 2020; Agichtein and Gra- vano, 2000; Tao et al., 2018; Zeng et al., 2022). These works typically compile a pseudo-training dataset from given corpus by assigning pseudo- label each document based on its alignment specific class. The pseudo-training dataset aims to be a confident substitution for an expert-curated collection blaas-indicaatve documents, ultimately being used to fine-tune text classifier.
The motivation of this work begins with the as- sumption that there are three types of documents with ground truth class topic, as illustrated in Fig- ure 1: documents diseussing (1) the class topic at the word, sentence, and document level, (2) the class topic at the document-level but with mul- tiple other topies mentioned at the word or sen- tence level, or (3) multiple topics as well as the class topic at all levels. Existing weakly supervised methods focus on word-level (Meng et al., 2018, 18, 2020b; Mekala and Shang, 2020) or document- level (Wang et al., 2021) information independently and only ever associate one pseudo-label to docu- ment, which may allow them to differentiate doc- uments of type (1) from those of type (3). How- ever, this notion is fundamentally unrealistic for real-world data, which often takes on the inter-
1 Introduction
Text classification a fundamental task in Natural Language Processing (NLP) that enables automatic labeling of massive text corpora, which is necessary in many downstream applications (Rajpurkar et al., 2016; Zhang et al., 2022b; Tang et al., 2015). Prior works train text classifiers in fully-supervised manner (Yang al., 2016, 2019; Zhang et al., 2015) that requires substantial amount of train- ing data, which is expensive and consumin,,,,g, especially in emerging domains that require the su-

---

granularity class disagreements exhibited in type (2), and hence encounters the risk of the word- and sentence-level topics overriding the true document- level class. For example, the education- news ar- ticle of type (2) uses strong political terms to dis- cuss the government being the subject matter of a teacher's lesson to students, which may mistak- enly lead it being classified as "politics" instead of "education". In general, documents may also feature several indeterminate sentences that do not confidently indicate any sort of class alignment.
fies class-indicative keywords and uses them represent classes and sentences. MEGClass then automatically weighs the importance of each sen- tence in classification for more accurately estimat- ing a document's class distribution. MEGClass further leverages a multi-head attention network to learn document representation that reflects the critical information at multiple text granularities. Finally, MEGClass takes the most confident docu- ment representations to enhance our initial word- based class representations through our iterative feedback approach, helping our model better under- stand what an e.g., "education" " document should look like at all levels of granularity. Because our approach can handle all three doc- ument types when constructing the final pseudo- training dataset, allow fine-tuned final clas- sifier be more robust to challenging real-world documents. Comparing with existing weakly and extremely weakly supervised methods, MEGClass achieves a stronger performance on most datasets, especially with longer documents and fine-grained classes. Our contributions are summarized as fol- lows:
The existing classification methods unfortu- nately do not address such inconsistency concerns among different levels of granularity. Several approaches (Meng et al., 2018, 2020b) generate pseudo-labels based the individual words sen- lences instead of the entire document and evaluate their confidence in aligning to a single class. This may lead to two glaring issues: (1) many words and sentences are only considered high quality and class-indicative based on the context of their parent document, and (2) they could be representative of classes different from the document's 's fundamental topic (c.g., "checks and balances" and "judicial branches" in conftict with "education" in Figure 1). On the other hand, another approach (Wang et al., 2021) heavily relies on both word-level and document-level, but independently- resulting in both overconfident and vague document represen- tations used for generating pscudo-labels.
1. To the best of our knowledge, this is the first work to exploit mutually enhancing text granularities for extremely weakly supervised text classification (only given the class label names).
2. We propose a novel method, MEGClass, which produces a quality pseudo-training dataset through class distribution estimation and contextualized document embeddings, fined by iterative feedback. The pseudo- training dataset is used to fine-tune a text clas- sifier.
In this study, we attack this problem by utilizing all three levels of text granularity (words, sentences, and documents) in a manner that allows them to mutually enhance each other's significance in un- derstanding which class a given document aligns with. Furthermore, we consider a document's class distribution instead of single class such that we rank documents with narrower distribution higher and can correct any initial misclassifications (c.g., education > polities in Figure 1) through itera- tive feedback. Doing so will allow us to exploit three levels of insight into the document's ss local (word/ssenencce) and global (document) class dis- tribution and jointly factor them into our under- standing of the document and final pscudo-training dataset construction.
3. Experiments on seven datasets demonstrate that MEGClass outperforms existing weakly and extremely weakly supervised meth- ods, significantly in long-document datasets and competitively in shorter, coarse-grained datasets.
Reproducibility:
We release our data and source code to facilitate further studies
Related Work
In this study, we propose a novel framework MEGClass perform extremely weakly super- vised text classification using mutually enhancing text granularities. Specifically, when only the class surface names are given, MEGClass first identi-
2.1 Weakly Supervised Text Classificationn
In real world applications, the acquisition of hu- man annotations from domain experts is time-

---

consuming and expensive. Thus, prior methods leverage supervision from existing large knowl- edge bases such as Wikipedia to serve as distant supervision (Chang et al., 2008; Song and Roth, 2014; Gabrilovich and Markovitch, 2007), while other works utilize heuristic rules (Badene et al., 2019; Ratner et al., 2016; Shu et al., 2020) or seed keywords (Meng et al., 2018; Mekala and Shang, 2020; Agichtein and Gravano, 2000; Tao et al., 2018). While they mitigate significant human la- beling efforts, it is often impractical to assume that a large knowledge base is available.
sidering multi-text granularities.
2.3 Exploiting Multi-Text Granularities In addition to both word- and document-level granularities being utilized in weakly supervised text classification, other tasks such as topic min- ing, story discovery, and document summarization have exploited various text granularity levels. Re- cent studies on topic mining utilize PLMs by em- ploying context-aware word representations and sentence-level context (Zhang et al., 2022a). Fur- thermore, recent studies for online story discovery utilize pretrained sentence representations of ticles along with thematic keywords (Yoon et al., 2023b) and continual self-supervision (Yoon et al., 2023c). For interpretable topic discovery, Top- Clus (Meng et al., 2022) leverage low-dimensional latent topics for attention-based embedding and clustering. PDSum (Yoon et al., 2023a) proposes prototype-based continuous summarization of evolving multi-document set streams, by incorpo- rating word- and sentence-level themes extract representative sentences.
2.2 Extremely Weakly Supervised Text Classification
Recently, methods exploring the extremely weakly supervised setting are achieving motivating results. These methods exploit only the provided class la- bel names to discover class-indicative keywords and generate pseudo-labels for classifier training. LOTClass (Meng et al., 2020b) utilizes pre-trained language models (PLMs) as a linguistic knowledge base to derive class- -ansiiniie keywords. Simi- larly, Con Wea (Mekala and Shang, 2020) lever- ages contextualized representations of word oc- currences and seed words to disambiguate class keywords. X- X-Class (Wang et al., 2021) discovers keywords to construct static class & document rep- resentations which are clustered to generate pseudo- labels. ClassKG (Zhang et al., 2021) leverages Graph Neural Network (GNN) iteratively ex- plore feyword-keyword correlation for each class. WDDC (Zeng et al., 2022) utilizes cloze style prompts to obtain words that summarize the docu- ment content. They propose a latent variable model to learn word distribution that maps to the pre- defined categories as well as document classifier simultaneously. NPPrompt (Zhao et al., 2022) pro- poses fine-tune PLMs on related words obtained through embedding similarities. This fine-tuned PLM is leveraged to prompt candidate labels which are aggregated into semantically similar labels for the final document prediction. Finally, PIEClass (Zhang et al., 2023) is concurrent work that uti- lizes zero-shot prompting of PLMS in order to get pscudo labels based contextualized text under- standing. However, all of these methods fail to account for intergranular class disagreements and place full confidence in the initially chosen target class. Furthermore, these methods ignore the rich information that can be derived from jointly con-
3 Framework
3.1 Preliminaries
Problem Formulation. We address an extremely weakly supervised text classification task, which expects an input of class names and set of doc- uments and outputs single class label per doc- ument. Specifically, let document d; D be list of sentences, where we assume that d, primarily falls under a specific class Ck- Each class C E C has a surface name, which we utilize as initial insight into what the class is about. The set of documents with ground truth class of is denoted as Ck. We also consider sen- list of words.
[81, . s/,,]
tence sj [wi, W2, ..... [|
Class Representations. Weakly supervised text classification problems can be initially approached by getting class representations using a pre-trained language model. Previous works (Aharoni and Goldberg, 2020; Wang et al., 2021) find that using simple and weighted average of contextualized word representations is effective for preserving the domain information of documents and hence effec- tive for clustering relevant documents in each class. For instance, X-Class (Wang et al., 2021) estimates class representations by a harmonic mean of the T static word representations (Ethayarajh, 2019) closest to the class surface name.

---

Initialization (Section 3.1) Representations
Sheudo-Training Dataset Refinement (Sectiioont 3.2-3,4)
Fine-Tune Text Classifier (Section 3.5) single- classification
science polltics rellgion
Class-Ore Sries-OreentedFoe Sentence Representations
jossiden
Inee
With
assear ocastiions
Care MEGClass Framework
Class Distribution Estimation (Section 3.2)
Contextualized Embeddings (Section 3.3)
Iterative Faedback (Section 3.4)
The powecdent met unth
(0.4,05, 6o)
science:
they ran expadmenns on rustation offecis.
(10,0....0)
politics: religion: 0.0
Selt-amention
Pooling
Select most confidint documents via norrowmess of their closs distribution
and punsohwrr fonight
The euprrmmenttt senults
(0.9, 0.0
Conturtustive sentainees
Contortualeed docermants
Update word-based class representations with top documents
Figure 2: We propose MEGClass, which given a raw text input corpus and eseer-ppe class names, learns contextualized document representations that reflect the most discriminative indicators of its potential class using regularized contrastive loss. It then utilizes the most confident documents for each class to update its understanding the respective class, ultimately constructing a pseudo dataset to fine-tune a text classifier.
Class-Oriented Sentence Representations. Each word in a sentence is assigned a weight based on the similarity between its closest class and the average of all class representations, finally to de- rive class-orienter sentence representation. Any class (Meng al., 2020a; Zhang al., 2022a) or sentence representations (Reimers and Gurevych, 2019) can be used for this purpose (explored further in Appendix A.3). For MEGClass, we use the initialization introduced in X-Class's (Wang et al., 2021), but we do so based on only the sentence-level context and not the document-level context as X-Class does.
correlates with how informative it is in indicating a single distinct class. Hence, for classifying sports, sentences that can represent two or more classes (e.g. "the player got the ball") are vague and should be weighed less than sentence related single class (e.g. "he hit home run", which is only rele- vant to baseball).
We quantify this by noting sentence sj 's cosine- similarity to its top two classes individually (af and of respectively). The larger the gap (af a²) is between the two top cosine-similarities, the more discriminative sj is to its top class. For example, "he hit homerun" would be significantly closer baseball than tennis and hence would have a larger class gap than "the player got the ball".
Figure 2 shows the overall framework of MEG- Class, which exploits these class and sentence rep- resentations to learn contextualized document rep- resentations along with varying class distributions, to derive pseudo dataset to fine-tune classifier.
In addition to measuring class discriminative- ness, we must also consider that a sentence sy significance is relative to the other sentences in the document d4, and that the sum of all a document's sentence weights must equal to 1. Thus, we choose to divide each sentence's class gap by the sum of all sentence class gaps within document d;:
3.2 Class Distribution Estimation
In order to learn the most critical indicators of document's potential true class, we must first iden- tify which classes viable candidates. To provide flexibility and correct any initial misclassifications, we choose represent a document's set of class candidates as a distribution as opposed to binary label for each class. A narrower class distribution indicates higher confidence in assigning a class label to the given document. A naive method to de- termine the target class distribution is to average document's sentence representations and compute the cosine-similarity between the average and each class representation. This unrealistically assumes that all sentences have equal significance doc- ument's underlying context. Consequently, given our task, we claim that a sentence's 's significance
2
jjweight
(1)
This allows us to reduce the impact that vague sen- tences (close to multiple classes) have on the target class distribution. In Figure 4, we demonstrate the superior performance of this metric (6% greater) compared to other techniques.
Once we have the confidence of cach sentence 8j within the document d, perform weighted label ensemble where each sentence will vote for its most similar class Ck with its vote weight wj.eight and the voting results are aggregated into a class distribution P(d E Cc):

---

sentences csj. indicative of a sentence's relative significance its parent document dj:
P(d Ck) =
Σ Sjweeigtt
(2)
EjEd;
dd
CD(d) = cd, = E
Cycsj E
Red
3.3 Contextualized Representations
(4)
[d:l
(lllee)))
Contextualized Sentence Representations. As we discuss in Section 3.1. the sentence representa- tions computed independently from the other sentences within document. However, sentences that seem vague isolation (e.g.. "he walked to the podium") may become significant when key information from other sentences is introduced (e.g., "the president held a press conference today"). Hence, we aim to incorporate the pair-wise rela- tionships between the sentences sj in document d. We accomplish this by learning contextualized sentence representations csj using multi-head self-attention mechanism (Vaswani al., 2017):
C3j
clla((sz))
k=1
where hed is the number of hidden dimensions, aj is an attention weight indicating the relative importance of csj for representing cdj, derived by attention layer La (cs,) tanh esj + b)V with learnable weights W, b, and V.
Then, we minimize a weighted regularized con- trastive loss (Chen et al., 2020) to ensure that the contextualized document representations ed2 and sentence attention weights are updated to reflect all viable class candidates Ct2 relative to the target class distribution:
CS(d:) = [cs1, .8,, (.l E IR (.) xhea
10 P(ds Ck)
(3)
Lw C,d())) P(dss
-4n(kmhs [E E d:l) + sy d:l))
k=1
(5)
where hes is the number hidden dimensions, Egjsy E d,] denotes the initial class-oriented sen- tence representations for document d, Imha is multi-head self attention layer, and 4n feed for- ward layer with layer normalization.
x log Ex elcoss(cd,
where the temperature scaling value is T, and the target class distribution is P(d; E Ck). Note that this regularized contrastive loss designed specifi- cally for MEGClass as unlike previous methods (Mekala and Shang, 2020; Meng et al., 2020b; Wang et al., 2021; Zhang et al., 2021; Yoon et al., 2023a,c), we do not artificially fit a contextualized document representation to single class without the confidence to support it. By retaining the class candidates within the contextualized representation itself, we can also determine which documents are the most confidently single-topic.
By computing contextualized sentence represen- tations, we incorporate both the word-level (via class-oriented weights as discussed Section 3.1) and the document-level information (via multi- head self-attention). However, order to construct a contextualized document representation, we must identify which sentences contribute the most to- wards the document's fundamental context.
Contextualized Document Representations. Unlike Section 3.2, where we had to determine each sentence's significance based its class- discriminatiiveness, now have target class distribution and document-level context integrated into our sentence representations. Recent docu- ment summarization methods show that learning sentence-level attention weights in representing documents can help highlight documents' theme (Yoon et al., 2023a). Similarly, in our case, we attentively combine the contextualized sentences to represent their parent document, while promoting the document's underlying theme be akin the target class distribution that we have computed. Specifically, we utilize attentive pooling to learn attention weights for each of the contextualized
3.4 Iterative Feedback
Thus far, we have been using the initial word-based class representations (Section 3.1) as our reference for a target class. However, now that we can iden- tify the documents with the narrowest class distri- bution and hence highest class-confidence, we can utilize them to enhance their corresponding word- based class representations. Hence, the iterative feedback will provide more insight into discover- ing new potential class indicators within the other less confident documents.
Updating the Class Set. Following the appli- cation of principal component analysis (PCA) to

---

PLM-based text clustering seen in (Aharoni and Goldberg, 2020), we first reduce any potential re- dundant noise in both our contextualized document and class representations using PCA. Each trans- formed representation will then have its closest class as its pseudo-label and confidence score equal to the corresponding cosine-similarity. For each class Ck. we rank the documents (that have pseudo-label oạ) based on their confidence and add the top 4% documents to each class's set, where each set already contains the initial word-based class representation. For each following iteration, the documents are re-ranked, and the class sets are consequently recomputed.
to train our text classifier (Devlin al., 2019).
Experiments
4.1 Experimental Setup
Datasets. We seven publicly available benchmark datasets to provide wide range of label granularity. domain, and document length. We covered sentiment-based reviews (Yclp). lengthy news articles (20News, NYT-TopiC, NYT- Location), concise summaries with abstract classes (Books), and fine-grained classes (NYT-Fine, 20News-Fine). Table 4 in the Appendix specifies the label names used for each dataset:
(1) Yelp (Zhang al., 2015) is for sentiment polarity classification of Yelp business reviews, adapted from the Yelp Dataset Challenge in 2015. (2) Books (Wan and McAuley, 2018) is a dataset of book titles and descriptions used for book genre categorization, collected from a popular book review website Goodreads. (3) 20News (Lang, 1995) is long-document dataset collected from 20 different newsgroups for topic categorization. (4) NYT-Topic (Meng et al., 2020a) is long-document dataset collected from the New York Times for topic categorization. (5) NYT-Loc (Meng et al., 2020a) uses the same corpus as NYT-Topic, collected from the New York Times, but is used for location categorization. (6) NYT-Fine (Wang et al., 2021) is long-documen dataset collected from the New York Times with 26 fine-grained classes (c.g., surveillance, the affordable care act). (7) 20News-Fine (Lang, 1995) is dataset collected in the same manner as 20News, but is partitioned into 20 fine-grained classes (c.g., graphics, windows, baseball).
In order utilize this feedback, we replace the initial class representations with the average of each class set and use the learned contextual- ized sentence representations cs², and correspond- ing sentence weights aj for recomputing the target class distribution (Liquation 2). This allows us to in- corporate both document-leve and sentence-level feedback throughout our iterations. We avoid word- level feedback in order to mitigate the efficiency issues seen previous works such as (Zhang et al., 2021) and (Zeng et al., 2022). Overall, we find that using an updated class distribution for re-learning new contextualized representations significantly boosts performance (as shown in our experiments section) and avoids significant error propagation (more details in Appendix A.6).
3.5 Text Classifier Fine-Tuning
Unlike prior methods, MEGClass does not assume that document will exclusively map single class. Specifically, we find that retaining docu- ment's class distribution information is crucial to identifying documents that discriminatively align with a single class (e.g. a 90% politics, 10% health article is a stronger candidate than 70% politics, 30% health article) for our meeuo--rainingg dataset. NYT
Dataset
Domain
Grannlarity Classes
Does
Yelp 20News
Sentiment
Charse
38,000
News
Course
17.871
NYT-Topic
News
Coarse
31,997
News
Course
10
31,997
Using the confidence score as described in Sec- tion 3.4, we select the top 8% of the documents for each class. These documents and their pseudo- labels are used to fine-tune pre- pre-trained text clas- sifier as ground-truth in order for our method to be generalized and applied to unseen documents (Appendix A.2). This is an established and stud- ied noisy training scenario (Goldberger and Ben- Reuven, 2017; Angluin and Laird, 1988). Thus, since we know how confident we are on each docu- ment, we can select the most confident documents
Books
Goodreads
Course
33,594
NYT.Fine
News
26
13,081
20News.
News
Table 2: Dataset Statistics.
Compared Methods. We compare MEGClass to the following six methods that have shown the most promising results thus far. We use Micro-/Macro- Fl as our evaluation metrics (Appendix A.9). More details of the parameter and supervision settings of each model can be found in Appendix A.7. (1) NPPrompt (Zhao et al., 2022) collects related

---

Model
Yelp
20News
NYT-Topic
NYT--Loc
NYT-FFime
20News-Fine
Supervised
95.7/95.7
96.60996660
95,98/5.01
96.0955.0
81.081.0
98.0 98.0966.
96. 39/99966.36
ConWea
71.4/122
75.73/73.26
81.67/71.41
85.31/83.81
52.3/52.6
76.23/69.82
48.7/48.7
LOTClas
87.4/87.2
73.78/72.53
67.11/43.38
19.9/16.1
15.0/20.11
9.49.6
X-Closs
8/68
73.17/73.07
79.01668.6
19.51/89
53.6/54.2
66/66677.6 7,36
58. 7/577/885 5
ClassKG
91,2/91.2
81.0/82.07
72.06/65.76
86 06.84/83.35
55.0/54,71
88.86/70.5
52,29/52.1
WDDC-MLM
81.2/8111
81,21/68.88 82
81,5/69.2
$S.84/56.99 91
53.86/53.75
87,41168.34
1/,0/50.
NPPrompt
81.2/8111
68.9/68.8 8
64.6/64.2
53.9/53.8
49.6/49.7
55.2/54,9
48.6/48.3
MEGClas
87.41/87.411
81.72/80.63
85,42/68.03
93.06/91.93
56.35/55,71
89.24/71006
66.37/64.24
MEG-Iait
78,17/77.44
75.93/74.57
79.85164,77
62,84/65.57
50.28/50.51
82.48/66.77 74
52,30/54.02
MEG-CX
84.24/84.2
76.90/75.26
78.71865.11
82.31/80.65
52,39/51.21
86.54769.46
64. 21
Table I: Evaluations of Compared Methods and MEGClass. Our results are averaged over five trials with the variance provided well. Both micro-/macro Ft scores are reported due to imbalanced datasets. Supervised performance provides an upper bound. denotes the second-best method.
words through embedding similarities obtained from a pre-trained language model. These related words are leveraged as labels to prompt genera- tive language model and aggregated as the match- ing result. (2) WDDC-MLM (Zeng et al., 2022) utilizes cloze style prompts obtain words sum- marizing the document. They learn latent vari- able model to map a word distribution to the pre- defined categories as well as document classifier simultancously. (3) ClassKG (Zhang et al., 2021) constructs a keyword graph with co-occurrence re- lations and self-trains sub-graph annotator to gen- erate pseudo labels for text classifier training. The class predictions iteratively update the class key- words. (4) X- (Wang et al., 2021) leverages BERT to generate class-oriented document repre- sentations. The document-class pairs are formed by clustering and used to fine-tune a text classifier. (5) LOTClass (Meng et al., 2020b) constructs a category vocabulary for each class, using a PI.M fine-tuned using a self-training and soft-labeling strategy. (6) CenWea (Mekala and Shang, 2020) uses a PLM to obtain contextualized representa- tions of keywords. It then trains text classifier and expands seed words iteratively. (7) Supervised (Devlin et al., 2019) fine-tunes BERT text classi- fier on a task- speciifc training dataset with an 80/20 train-test split.
the class with the highest cosine similarity between the transformed contextualized document represen- tations and the transformed class representations, as discussed in beginning of Section 3.4.
4,2 Overall Performance Results
Table 1 shows our results on the Yelp, 20News, NYT, and Books datasets. Overall, MEGClass performs the best on the long text coarse- granind datasets. However, MEGClass trails ClassKG on the Yelp dataset while being competitive with other weakly supervised approaches. It is important to note that Yelp contains shorter text documents when compared against News datasets due the nature of Yelp reviews, which does not benefit from the varying levels of text granularity as much as longer documents. In addition, ClassKG has dras- tically longer run time as demonstrated in Figure 3. Finally, we show that MEGClaas significantly beats the other datasets on NONews-Fine and NYT- Topic micro-fl and approaches fully supervised performance on the NYT-Loc dataset (first row of Table 1). We expand upon these results through a case study in Appendix A.4 and A.5.
Baseline Runtime Comparison
ClasskG
30
WDDC
We denote our method as MEG our ex- perimental settings and hyperparameter sensitivity analyses are included in Appendix A.7 and 4.4.
NPPrompt
15T
MEBClOss
182
LOTC/Oss
135
We run set of ablation studies in order to better understand the impact that each one of our mod- ules has on the overall methodology. We have two ablation versions. MEG-Init refers to the labels obtained when we select the class with the maxi- mum weight in the initial target class distribution specified in Equation 2. MEG-CX refers to the labels obtained in the first iteration when we select
X Cioss
07
20
25
30
35
60
NYT Tine 20News
Figure 3: Running time of compared methods.
TineeGrained Classification. In Table 1, we also examine MEGClass's 's performance on fine-graina text classification on the NYT-Fine and 20News-

---

Fine datasets, which have annotations for 26 and 20 fine-gres sine-grained classes respectively. We compare our method against other state-of-theeart weakly super- vised methods and show that MEGClass performs the best. We largely attribute this performance to longer text documents and the fine-grained label space containing highly distinct classes. With the expanded keywords, our contrastive learning pro- cess leverages the distinct label space learn the attributes that are common between data classes and attributes that set apart a data class from an- other.
Weighted tabel Ensemble Ablation Study
Velp
20Newx
NYT-Toplc
NYT-loc
MYT-Fite
20Newe-Fine
Dotooet
Equal Sentence Weight
Sentence Centrality
Glore Dksomminntivvneee
Figure Effects of sentence weight computation met- ric during iteration #1 for weighted label ensemble pro- posed Equation 1.
Iterative Feedback
0.82
0.89
Ablation Studies. In our thorough evaluation, we analyzed the relative contributions of each com- ponent and found that the impact of our learned contextualized representations iterative feedback class distribution estimation. Specifically, the quality of the joint representations that we learn de- termines which documents we deem as sufficiently confident to enhance the class representations as erative feedback, which then heavily influences the accuracy of the estimated class distribution used for the next iteration. We can see this through the immediate boost in performance between MEG- Init and MEG-CX in Table in Figure 6, as well as the overall high quality documents chosen across all different datasets, shown in Table 10.
0.6
0.8
088
0.77
Iteration
--00hews --- Topic
-NYY-LLO
Class Set hypprpporrmeeeereensttttte (k)
0.88
0.83
0.78
0.68
0.8
Int0l do
7.5
10
20
(top % documents choson)
20Novs
-NT-TTopic
Figure 5: Top. Micro-FI scores after transforming con- textualized document representations over 4 iterations. Bottom. Sensitivity of top k% documents chosen to update class set as iterative feedback. "Initial cls repr" denotes the initial class word-based representations.
4.3 Weighted Label Ensemble Analysis
In Figure 4, compare different mechanisms for computing the sentence weights used in the weighted label ensemble (Section 3.2). We com- pare using equal sentence weights, sentence cen- trality (Liang et al., 2021), and class discrimina- tiveness (what MEGClass proposes in Equation 1). Sentence centrality is popular sentence weight metric in document summarization (Liang et al.. 2021), measuring the similarity between given sentence at position and all sentences with posi- tions 2 ++ 1. Its values tend to reflect leading- sentence bias. Figure demonstrates that the class discriminativeness metric we propose performs the best overall, especially on long-document datasets like NYT where the leading sentences do not reflect all of the main topics. This demonstrates the need for class discrmmicrim weighted label ensemble.
7.5% of documents chosen have average micro- /macro FI score of 90.75/86.99 across all datasets, including Books (see Table 10 in Appendix A.6 for the full results).
We expand upon these results by showing the qualitative (Table 3) quality of the set of class- indicative documents. We can see that relative to the overall difficulty of the dataset, the accuracy of documents chosen is high across all classes (as seen through the macro-Fl), and when we observe the highest-ranked document for each 20News class, the document is very representative of the class- even after the first iteration.
4.4 Hyperparameter Sensitivity
Iterative Feedback Analysis. The top graphs in Figure 5 demonstrate the significant benefit of utilizing the top-k% class-indicative documents as feedback for updating the target class distribu- tion. Furthermore, after the first iteration, the top
We introduce four core hyperparameters in MEG- Class, and we conduct hyperparameter sensitivity analyses for each one. At the high-level, our frame- work consists of three main parts: (1) initializa- tion (Section 3. (2) pseudo-training dataset re-

---

Class Sehence
Document Escerpt
tion and classifier fine-tuning steps (e.g. for the final classifier fine-tuning, choosing the top % of documents as the pseudo-training dataset).
L..J isotope changes doe't I normally a chemistty, a consumpption of only heavy water would he fatal, and that seads watered only
heavy waler do not sprout. Does myyne know about this?
also heard this. The reason that deuterium does not have exactly the same reaction
hydrogen causes lower velocity, Baltzmann constant, mumble). [..]]
extra (which
Polities
[..] had hit part in the mach larer political agenda President Clinton. [..] NJ ussembly voies to overiurn ussuul weapon ban.
0.8
Regularization TeriiiionTeeeeree Temperature Sensitivity (1)
Feb 28th Compound in Waco attacked. On Feb, 25th the New Jorsey assembly voted to overturn the assoult weapon ban in that
0.7
Macro-11
state. It looked like might be a tight vote. hut the Senate in N.J.
0.5
was going vote overtumn the [..]
Religion
[..] the Oriental Orthodos and Eastern Orthodox did sign common statsment Christology,, which heresy Monophy sitism was condemmed. So the Captic Orthodox Church does not believe Monophysiricm, Sorry! What does the Coptic Church believe about the will und enengy of Chris?? [...
0.5
0.4
02
tepertttt -NNT-Loc - -rT--Coars
20New
NYT-FFo
Table 3: Class Set Examples. The highedt-ranked docu- ment after one iteration in three of the 20News top-k% class sets (other two classes are comparable in quality).
Figure 6: Sensitivity of the temperature scaling value for the weighted regularized contrastive loss. Macro-F1 scores are computed after the first iteration.
finement (Sections 3.2-3.4), and (3) classifier fine- tuning (Section 3.5). The second part is main component, which involves the following three sub- steps:
Epoch # Hyperparameter Sensitivity
087
0.82
Mico-F
0.7
1. Class distribution estimation (Section 3.2) +
0.2
there are no hyperparameters for estimating the initial class distribution of each document.
0.62
# of Epochs NYT--Fin -NYT-Coare
-0NNww
MYT-Lo
2. Contextualized embeddings (Section 3.3)
for learning document representations that jointly integrate sentence and document level information, the hyperparameters we intro- duce are the training epochs and regularization temperature used in the weighted regularized contrastive loss. Specifically, in Figure 6 we find that MEGClass's performance remains stable for temperature T values below 0.3. We also demonstrate in Figure 7 that increasing the number of epochs does not benefit MEG- Class significantly. In fact, MEGClass starts to stagnate after Epoch 3 and is more depen- dent on the top k% of documents selected.
Figure 7: Sensitivity of the number of training epochs during the first iteration for learning the contextualized sentence and document representations.
5 Conclusion
In this work, we propose novel method MEG- Class for extremely weakly supervised text clas- sification by exploiting mutually-enhancing text granularities. MEGClass learns new document rep- resentations that reflect the strongest class indica- tors with respect to all three levels of text granu- larities. MEGClass identifies the top documents with narrower class distribution (more confidently single-topic) to use as feedback for the next itera- tion, and it ultimately constructs pseudo-datase of documents for fine-tuning a pre-trained text clas- sifier. We demonstrate MEGClass's 's strong perfor- mance and stability through extensive experiments on varying domains (news and reviews) and label spaces (topics, locations, and sentiments). Due to MEGClass's novel use of mutually enhancing text granularities, further exploration can be done to uti- lize MEGClass for information extraction at vary- ing text granularities. Furthermore, more sophisti- cated nonlinear dimensionality reduction methods can be a potential aspect to explore. Finally, han- dling class hicrarchies may improve MEGClass's ss -raigrai classification abilities.
3. Iterative feedback for refining the first two sub- steps (Section 3.4) - we introduce the follow- ing hyperparameters: (1) k for choosing the top k% contextualized document representa- tions to enhance each class representation. (2) the number of iterations of feedback. Figure presents a sensitivity analysis on both of these, and shows that we reach a relatively stable set of confident documents at k = 7.5% that significantly enhances the quality of the ini- tial keyword-based class sets across all seven datasets.
For a fair comparison with the baselines, we use the same hyperparameter settings for the initializa-

---

Ethics Statement
In Proceedings of the Fifth ACM Conference on Dig- ital Libraries, DL '00, page 85 94, New York, NY. USA. Association for Computing Machinery.
Based on our current methodology and results, we do not expect any significant ethical concerns, given that text classification is a standard problem across NLP upplications and basing it on extremely weak supervision helps as barrier to any user- inputted biases. However, one minor factor to take into account any hidden biases that exist within the pre-trained language models used as a result of any potentially biased data that they were trained on. We used these pre- pre-trained language models for identifying semantical similarities between class names and documents//sentences and did not ob- serve any concerning results, as it is low-risk consideration for the domains that we studied.
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters pretrained language models. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 7747- 7763, Online. Association for Computational Lin- guistics.
Dana Angluin and Philip D. Laird. 1988. Learning from noisy examples. Machine Learning, 2:343-370.
Sonia Badene, Kate Thompson, Jean- lean-Pierre Lorré, and Nicholas Asher. 2019. Data programming for learn- ing discourse structure. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 640-645, Florence, Italy, Associa- tion for Computational Linguisties.
Limitations
Ming-Wee Wei Chang, Lev-Arie Ratinov, Dan Roth, and Vivek Srikumar, 2008. Importance of semantic rep- resentation: Dataless classification. In AAAI Confer- ence on Artificial Intelligence.
In our experiments, we find that the expanded key- words can be vague and unrelated for topics de- scribed by singular homonym. For example, the class label name "Cosmos" in the NYT--Fne dataset generates keywords related to the New York Cos- mos rather than Astronomy. When we modify the label names by replacing homonyms with labels that provide surface-level context for each topic, we find that this increases MEGClass's macro-f1 performance by approximately 6.64 points (77.70),
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton, 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference on machine learning, pages 1597-1607. PMLR.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-41866 Minneapolis, Minnesota. Association for Computational Linguistics.
demonstrating that the selection of label names and seed words is critical to provide contextual infor- mation for topics. This information is especially crucial for fine-grate fine-grained text classification fine- grained classes like "international business" and "economy" can be indistinguishable to pre-trained language models without local or global context from sentences and documents.
Kawin Ethayarajh. 2019. How contextual contextu- alized word representations? comparing the geom- etry of bert, elmo, and gpt-2 embeddings. In Con- ference on Empirical Methods in Natural Language Processing.
Acknowledgements
Evgeniy Gabrilovich and Shaul Markovitch. 2007. Computing semantic relatedness using wikipedia- based explicit semantic analysis. In Proceedings of the 20th International Joint Conference on Artif- ical Intelligence, IJCAI'07, page 1606-1611, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
This research was supported in part by the US DARPA KAIROS Program No. FA8750- 19-2-1004 and the National Research Founda- tion of Korea (Basic Science Research Program: 0923IIAAAAA14043766665 Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessar- ily represent the views, either expressed or implied, of DARPA or the U.S. Government.
Jacob Goldberger and Ehud Ben-Reuven. 2017. Train- ing deep neural-networks using a noise adaptation layer. In International Conference Learning Rep- resentations.
Ken Lang. 1995. Newsweeder: Learning to filter net- news. In Proceedings of the Twelfth International Conference on International Conference on Machine Learning, ICML 19,, page 331-339, San Francisco. CA, USA. Morgan Kaufmann Publishers Inc
References
Eugene Agichtein and Luis Gravano. 2000. Snowball: Extracting relations from large plain-text collections.

---

Xinnian Liang, Shuangzhi Wu. Mu Li, and Zhoujun Li. 2021. Improving unsupervised extractive summa- rization with facet-aware modeling. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 1685-1697.
International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR "20, page 1051-1060, New York, NY, USA. Associa- tion for Computing Machinery.
Yangqiu Song and Dan Roth. 2014. On dataless hi- erarchical text classification. In Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelli- gence, AAAI 14, page 1579 1585. AAAI Press.
Dheeraj Mekala and Jingbo Shang, 2020. Contextu- alized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguisties.
Duyu Tang. Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. Conference on Empirical Methods in Natural Lan- guage Processing, pages 1422-1433 1432, Lisbon, Portu- gal. Association for Computational Linguistics.
Yu Meng, Jiaxin Huang, Guangyuan Wang, Zihan Wang. Chao Zhang. Yu Zhang, and Jiawei Han. 2020a. Dis- criminative topic mining via category-name guided text embedding. In Proceedings of The Web Confer- ence 2020. pages 2121-2132.
Proceedings of the 2015
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-superviser neural text classification. In proceedings of the 27th ACM International Con- ference on information and knowledge management, pages 983-992.
Fangbo Tao, Chao Zhang, Xiusi Chen, Meng Jiang, Tim Hanratty, Lance Kaplan, and Jiawei Han. 2018. Doc2cube: Allocating documents to text cube with- out labeled data, In 2018 IEFF International Confer- ence on Data Mining (ICDM), pages 1260-1263
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020b. Text classification using label names only: A language model selfftraining approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages Online. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances neural information processing systems, 30.
Mengting Wan and Julian McAuley, 2018. Item rec- ommendation on monotonic behavior chains. In Proceedings of the 12th ACM conference on recom- mender systems, pages 86-94.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Yu Zhang, and Jiawei Han. 2022. Topic discovery via latent space clustering of pretrained language model representa- tions. In Proceedings of the ACM Web Conference 2022, pages 3 142--1552.
Zihan Wang, Dheeraj Mekala, and Jingbo Shang. 2021. X-class: Text classification with extremely weak su- pervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 3043-3053, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoe V. Le. 2019, Xlnet: Generalized autoregressive pretraining for lan- guage understanding. CORR, abs/ 1906.08237.
Alexander Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. 2016. Data program- ming: Creating large training sets, quickly. In Pro- ceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16. page 3574-3582, Red Hook, NY, USA. Curran Asso- ciates Inc.
Zichao Yang, Diyi Yang, Chris Dyer. Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Associa- tion for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLA pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Susik Yoon, Hou Pong Chan, and Jiawei Han. 2023a. PDSum: Prototype-driven continuous summarization of evolving ings of the ACM Web Conference 2023, pages 1650- 1661.
sets stream. In Proceed-
Kai Shu, Subhabrata Mukherjee, Guoqing Zheng, Ahmed Hassan Awadallah, Milad Shokouhi, and Su- san Dumais. 2020. Learning with weak supervision for email intent detection. In Proceedings of the 43rd
Susik Yoon, Dongha Lee, Yunyi Zhang, and Jiawei Han. 2023b. Unsupervised story discovery from continu- ous news streams via scalable thematic embedding.

---

In Proceedings of the ACM SIGIR conference on Re- search and Development in Information Retrieval 2023.
Dataset
Class Words
Yelp
bod, good
20News
compuler, sports, science, polities, religion
NYT-
basiness, polinis, sports, health, education, estate, ants soi- technology
Topic
Susik Yoon, Yu Meng, Dongha Lee, and Jiawei Han. 2023c. SCStory: Self-supervised and continual on- line story discovery. In Proceedings of the ACM Web Conference 2023, pages 1853-1864.
NYT-Loc
united, states, ima. japan, china, britain, russi, germany. canada, france, Italy
Books
children, praphic, Fantasy_peramontee peranormal, his- biography, mysteryy_ thrillerrrrreeeee grime, poctry, romance, young. acult
NYT-Fine
music, haschall, international business, abortion, military. foutball, golf, television. economy, dance, succer, cosmos, surveillanee, law __nffrremmet, besketball, federal budget. movies, stacks and honds, gun, control, energy companies. envirnment, bockey, the affordable care act immigration, tenmis, gay Lights
Ziqian Zeng, Weimin Ni. Tianqing Fang. Xiang Li, Xinran 7hao, and Yangqiu Song. 2022. Weakly su- pervised text classification using supervision signals from a language model. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 2295-2305, Seattle, United States. Association for Computational Linguistics.
20News- Hine
atheism, graphics. microsoll ibm, mac, motif, autos, motor- eycles, bascball, bockey, encryption. electroniss, medicine, space, christian, guns,
Table 4: Label names used for each dataset
Lu Zhang. Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021. Yeakly-supervisee text classi- fication based on keyword graph. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 2803-2813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Datasef
Train
Test
Yelp
91.27/91.17
85.85/85.30 30
20News
91.24/9.200
71.00/70,1 12
NYT-Topic
94.75/72.28
85.06/67.00 .59
NYT
97.53/97.37
92.65/90.99
Books
66.24 66.24666.47 47
43 61.64/16666
93.49/78.00
85.96/63.02
Fine
76.39/72.79
57,28/86.77
Table 5: Micro-/Macro-FI performance of fine-tuned text classifier on train and test documents.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification.
Yu Zhang, Yunyi Zhang. Martin Michalski, Yucheng Jiang, Yu Meng, and Jiawei Han. 2022a. Effective seed-guided topic discovery by integrating multiple types of contexts.
use train-test split of 50-50, where the 50% train- ing dataset consists of the top 8 50% document pseudo-dataset (Section 3.5) and the remaining 50% in the testing dataset are the lower-ranked documents unseen by the final text classifier. We can see that the pseudo-dataset which MEGClass constructs is quite robust to unseen documents, es- pecially given that the test documents are explicitly not as single-topic confident (hence ranked in the bottom 50% of documents).
Yunyi Zhang, Fang Guo, Jiaming Shen, and Jiawei Han. 20226. Unsupervised key event detection from mas- sive text corpora. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. ACM.
Yunyi Zhang, Minhao Jiang, Meng. Yu Zhang, and Jiawei Han. 2023, PIEClass: Seakly-supervised text classification with prompting and noise-robus. iterative eusemble training. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing.
A.3 Varying Class and Sentence Representations
In this section, we demonstrate the MEGClass can be applied to any initialization of sentence and class representations. For clearly comparing the performance and overall quality of the different representations, we present the final classifier's per- formance in Table when using the original class and class-oriented sentence representations (CO) versus Traieenccerannooomerrs---oo class and sen- tence representations (ST) respectively (Reimers and Gurevych, 2019).
Xuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming li Wu, and Lei Li. 2022. Pre-trained language models can be fully zero-shot learners. In Annual Meeting of the Association for Computational Linguistics.
A Appendix
A.1 Dataset Classes
Table 4 contains the label names that we use for
each dataset.
For computing the class and sentence repre- sentations using Sentencee we use the model all-mpnet-base- phrase (e.g. "This is [class name]" for Yelp and Books; "This article about [class name]" for 20News, Topic, NYT-Loc, and NYT-Fine)
A.2 Robustness of Final Text Classifier
We conduct a performance comparison to verify the robustness of our fine-tuned pre-trained text classifier proposed in Section 3.5 against unseen documents, as shown in Table 5. Specifically, we
encode a class

---

Dataset
C0
ST
A.4 Case Study
Yelp
87.41/87.41
90.6990.58
20News
81.72/80.63
69.12/6644
Topic
85.42/68.03
$2.70161.78
In this section, we walk through the critical mod- ules of the MEGClass framework by showing their respective intermediate results on the NYT-Loc dataset and their respective impact in determining final classification for document.
LYT
93,06/91.93
56,48/65.45 45
Books
56.35/55.71
50.68/51.2
NYT-Fine
89.24/71.06
90.32/77.28
Fine
66.37/64.24
38.00/44.09
Table 6: Miero-/Macro-FI final classifier performance comparison between using class-oriented representa- (ST) sen-
In Table 8, we list the top 12 class-indicative words (in order of descending rank) identified through the algorithm provided by XClass (Wang et al., 2021) for each class in NYT-Loc. In some cases, based on the term-extraction algorithm, cer- tain classes may have smaller number of terms identified. These terms are utilized for computing the initial word-based class representations.
tions (CO) and Sentence Transformes---booe tence and class representations on MEGClass's 's original framework and parameter settings.
lteration
United States
Iraq
Britain
0.4344
0.23
0,223
0.3295
0.1954
0.4751
0.2153
0.1773
0.0895
0.1424
0.7680
Furthermore, in Table 9, we include three dif- ferent documents which MEGClass classifies cor- rectly and baseline methods, X-Class and ClassKG, misclassify. and MEGClass on the hand classifies correctly. From the three examples, we can see that the incorrect classes that X-Class and ClassKG choose are typically mentioned within the docu- ment either explicitly implicitly through strong class-indicative terms. However, just as shown in Figure 1, in document type (2), we cannot solely trust the word-level context to be able to indepen- dently identify the document-level context. Hence, these word-level red herrings (e.g. "Iraq' 's weapons" in Document 3) distract both X-Class and ClassKG from the true document-leww level context (e.g. the cent legislative vote in the British Parliament and scandal with the prime minister).
Table 7: Class Distribution Updates For Document 3. We only show the classes with non-zero probabilities.
to represent each class, as well as encoding each sentence individually.
From Table 6, we can see that each type of representation performs well on different datasets. Specifically, Sentence' Transformers SentenceTransformen seems to sur- pass the original class-oriented, representations Yelp and NYT-Fine, notably datasets which have abstract and indistinct classes respectively (e.g. Yelp: good vs. bad; NYT-Fine: phrase class- names, overlap between classes). On the other hand, the class-oriented representations perform well on datasets with distinct and well-represented classes.
In Table 7, we additionally show how the class distribution that we estimate using the weighted label ensemble (Section 3.2) updates over each it- eration as we refine the class sets with confident documents. We can see how critical incorporat- ing confidence documents iterative feedback is for Document 3 (excerpt provided in Table 9), which initially has a slightly higher probability of incorrectly belonging to "Iraq" but is corrected to "Britain" the second iteration.
Class
Class Words
United
united states, united, it states, american, america, USB, ameri- americanism, nation, country. world
States
Irag
iraq. iragi. baghdad, iragis, haghdadis, iragiya, haghdadi, kuwaiti, kuwalt, irags, kuwailis, saldam
China
china, chinese, beijing, beijingers, guamii guangsi, nanzhung. guangning
Brilain
britain. brilish, england, uk. kingdom, london, londomers, britons, briton, anglo, brits, britian
Russia
nasia, russian, russians, moscow, soviei, siberia, soviets, nov- gorod, mssiya, leningrad, siherian, muscovite
Germany
germany. german, germans, deuischland, berliner. berlin. herliners, genmanys, prossia, prussian, saxony, bavaria
Canada
canada, canadian, canadions, quehee, quchocers, toront, mon- treal, montrealers, quehecois, ontario, ottawa, manitoha frunce, french. paris, parisia,, frenchman, frenchmen, french. wommn, fruncais, frenchness, bordcoux, parisians, francaise Itll, nalian, Italians italano, Italia, luscany, veneto, macionale, florentine
A.5
Comparison with ChatGPT
We conduct performance comparison between MEGCCass and ChatGPT on the NYT-Topic and
Inll
milanese, milan,
NYT-Fiine datasets. For prompting ChatGPT, we use the following template in an effort to prevent Gha from generating new class names:
Table 8: Class-Indicative Words. The top 12 (or all if number of critical words identified was smaller as mentioned in Section 3.1) class-indicative terms listed in-order of similarity for the NYT-Loc dataset.
Document: [document text]

---

Gold Britain
MEGClass
ClasKG Iraq
Document Excerpl
United
an immense shaped table brimming with fooaal uranngemenis, candelabra, fresh fruit and diexying arry of gold plated tableware and glasses. queen
guests at queen elizabeth's banquet were
Statess
head of the table, with president bush her right. laurn huch, the first lady, sat herween prince philip and prince charles, the queen's 'een's bushand and son. kari rove, the president's top political adviser. among the handful americans
took
part the royal procession, which included the queen's budyguurds, wearing red juckets and carrying ccremionial halberds [..]
Irau
Iray
Britain
France
the foreign ministers the world's leading powers drupped diplomatic niveties, argued, shook their fingers and expressed exasperation in public struggle to overcome their bitter impasse iray, efforts proved futile, but they made another day high drama the usually somnolent sccurity council. "dominique, that's a false choice!" britain is foreign socretary, jack strew, declared deminique villepin. french foraign minister [..] the
everyone laced force. [...] powell responding
whether to disarm saddamm hussein pescefully earlier presentation by hans the chief inspecter biological and chemical weapons, who hud just finished saying that iraq was making significunt strides in disarming.
Britain
Britain
Iraq
Iraq
prime minister tony blair narrowly defented revolt in his own lahor party on tuesday night over legislation in parfiument revamp the country' higher education syslem, thus avoiding political humillation that threatened to bring down his goverament. the close vote. 316 to 311 in favor of substantially raising tuition fees, guve important lift
bluir the of potentially greater challenge to his government on wednesday. when lord button issucs the lindings of his inverti into the events surrounding the death of dr. david kelly. he was the specialist on inag's weapons whose concerns, privately expressed
bbc. formed busis news reports that bluir and his aides had overstated the intelligence iray's illicil weapoes programs make stronger case for war. the university linuncing bill will require brilish univeraity students, who like mosi europeams make only momninal tuition contribations degree, to begin paying as much as 5,500 a year starting in 2006 L...
the cost college
Table 9: Examples excerpts of documents classified correctly by MEGClass and incorrectly by X- X-Class and ClassKG Selected from the NYT-Loc dataset; we include an excerpt of the document, its ground truth (gold) label, and the corresponding predictions from each model. We bold the key indicators of the ground truth class and underline the potential red- red-herrings for the other models' classifications.
Dataset
Micro-Fl
Macro-F1
its. This event naturally requires incorporating eco- nomic terms to describe the contextual information without compromising the document-level context clearly indicating "international business" as the true class.
Yelp
95,86
95.36
20News
95.07
94.03
NYT-Topic
93.61
79.34
NYT-Loe
96.91
96.99
NYT-Fine
95.77
87.91
20News. Fine
79.94
77.30
Table 10: Shows accuracy of the most confident k
Similarly, the last document in Table 11 details the government shutdown regarding the repealing of the medical device tax proposed within the af- fordable care act. Despite the presence of federal government and budget terms at the word-level, it is clear to readers that when jointly considering word-level and document-level context, the primary subject of conversation is the affordable care act despite the similar settings in which federal bud- get documents take place. These examples clearly demonstrate the need for considering multiple text granularities jointly as MEGClass proposes.
7.5% of documents chosen as iterative feedback after the first iteration.
For the above document, select the closest class label from ONLY the following options: [list of class label names]? Again, please only pick from the provided list.
As demonstrated in Table 11, we find that while ChatGPT performs well on datasets with clearly distinct and coarse-grained classes (e.g. "politics" and "science" in NYT-Topic), often misclassi- fies documents of type (2) in Figure 1; this con- sists of classes are fine-graing fine-grained and their respective documents will contain keywords of similar fine- grained classes (e.g. document under "interna- tional business" will likely contain many economic terms). This seen through the first document in Table 11, which discusses a European carmaker shutting down a plant due to a restructuring plan as a result of a tightening budget and falling prof-
A.6 Iterative Feedback Specifications
In Table 10, we show the performance of the top = 7.5% ranked documents chosen as iterative feedback after the first iteration. We can see that relative to the overall difficulty of each dataset, the accuracy of documents chosen is high across all classes, indicating that based on both quantitative (Table 10) and qualitative (Table 3 measures, the chosen iterative feedback is high in quality.
It is important to note that for each itcration, the top documents within the class set are up-

---

ChatGPT
MEGClus/Gold
Ducument Excerpt
Economy
Interaational Business
[..] faced shrinking domestic market hurning through cash, peugent shutting the plant at cosi of (000 jobs, put at restructuring plan meant to save .s billion curos, or J billion. by next year. the company says closing the plant esscutial for reducing overcapacity, problem for many european carmakers, plants that operate helow maximam efficiency may lose money on every car they produce. compuny hus five other big automotive plants france, and aulnay's importance had declined amad obbing profitability for the subcompact cars produced. [-.]
Economy
International Business
the european partiamentt on tuesday scrapped propusals by health officials that electronic cigarettes be tightly regulated as modical deviess, setting stage for dehate
united states the
extent of regulation. europeun lawmakers endorsed a permissive approach to the sule and use of e-cigaretles, alihough the products could not be sold legally anyone younger than. the food and drug administration the united states has said wants to issue regulations on the misoiin-deelivery devices soon. [-.]
Federal Budget
The Affordable Care Act
with the govermment entering its first day of the shurdown, repablicans and democrats were heginning to flout that they buped would lead a compremniee"
can week on sumething.. believe, on the medicul device tux." senater richurd durbin of illiniis, the no. 2 democrat, said on cnn. the lax. which helps pay for president house republicans recently included repeal as part of larger package of demands that the senate rejected munday. though demorrats have repeatedlly insisted thit they would not accept budget deal that contains extrancous policy provisions like the delays and defunding of the affordable care act that republicans have sought mr. durbin's i have caveat: that the rovenue from the so-called medical device be replaced repealed, the same time, senator rand puul, republican of kentucky, sugpested that congress pass wack budget that would allow government operations continse while domoerats and repablicans talkedd. [-..]
health care overhaul. hos boen
contantion
parties
seemed to open the door slightly. he did
Table 11: Examples of documents misclassified by ChatGPT. Selected from the NYT-Fine dataset; we include excerpt of the document well the corresponding predictions from ChatGPT versus MEGClass, where MEGClass matches the ground truth (gold) class label. We bold the key indicators of the ground truth class and underline the potential red herrings.
dated/replaced, not added to (in order to avoid duplicates as well as avoid overfitting to certain documents throughout the iterations in case of any misclassifications).. Furthermore, in order to avoid error propagation in case of certain document mis- classifications within our class sets, each iteration we re-learn new contextualized sentence/documentt representations from our initial class-oriented rep- resentations.
base of NPPrompt The hyperparameters are set to the default values.
WDDC-MLM (Zeng et al., 2022): We used the codebase of WDDC-MLM** The hyperparameters are set to the default values. We specifically choose to compare with WDDC-MLM as opposed to their other proposed model, WDDC-Doc, where the for- mer uses the supervision signals from a masked lan- guage model (MLM) as opposed to the document itself for the latter. We do this because WDDC- MLM has better performance than WDDC-Doc across the majority of the tested datasets.
A.7 Experimental Settings
For implementing MEGClass, we use the follow-
ClassKG (Zhang et al., 2021): We used the code- base of ClassKG', The hyperparameters are set the default values. We trained the graph neural net- work for 10 iterations.
ing hyperparameters across all datasets: T= 100,
= =le- -1e-3, 3, maximum sentence length 150, number of self-attention heads = 2, regularization 7 = 0.1, epochs = 4, k- = 7.5%, and å = 50%. For the datasets (Yelp, 20News, NYTiTopic, and NYT-Loc, 20News-Fine) with all well-defined classes (e.g. single word label names), we conduct 4 iterations of iterative feedback. For the datasets that contain phrases as label names (Books and NYT-Finne, we conduct 2 itcrations. For our word representations, we use bert-base-uncased, which leads to the hidden dimensions hes = -768 and had = 768.
X-Class (Wang et al., 2021) We used the code- base of X- As before, the hyperparameters are set to the default values.
LOTClass (Meng et al., 2020b) We use the code- base of LOTClass?. As before, the hyperparame- ters are set to the default values.
Wea (Mekala and Shang, 2020) We used the codebase of Con Wea We utilize the class label names as seed words provided in the source code.
We compare MEGClass with six other baseline text classification methods. Each bascline method is evaluated using the extremely weak supervision setting and hyperparameters as specified in their original implementation. We conduct all experi- ments on a single NVIDIA RTX A6000 GPU.
Supervised (Devlin et al., 2019) We use the Hug-
https:/ggittub.omm//uunnnngggg conn/uuannnnnnn/hh/PPPPo "https//githhu..oom//
" "htps://gituuu.coom// wom/ZZhanWanggK//CCsss
-NPrrompt (Zhao et al., 2022): We used the code-
" "htts:///gttubb hhaps://gitub..om//mcc///// http:p/hww...oocccccccoooooooooa

---

gingl'ace Transformer Python Interface" train the BERT model on the specified dataset using an 80-20 train-test split with the bert-base model. The text classifier is trained over 10 epochs and a batch size 32.
A.9 Evaluation Metries
Following the prior text classification studies (Zeng et al., 2022; Zhang et al., 2021; Mekala and Shang, 2020; Wang et al., 2021; Meng et al., 2018), we evaluate our methods and the baselines using two metrics: Micro-Fl and Macro-F1.
A.8 Implementation & Complexity Analysis
Macro-F1. Macro-F1 calculated using Macro- Precision (Pma) and Macro-Recal (Rma) where
We detail the architecture of the multi-head self- attention (MHS) network (Section 3.3) below, which Icarns the contextualized sentence and doc- ument representations. The input and hidden di- mensions are 768 (to preserve the original BERT embedding dimensions), and we only use two self- attention heads to minimize complexity much as possible. We train it from scratch (no pre-training), and has exactly 2,955,265 total parameters.
Il O Gml tm
Pma
E mmc [M
I'm n mm tm Micro-F1. Micro-FF is calculated using Micro- Precision (Pmi) and Micro-Recall (Rm) where
Ryna
E m
Input: x <- Initial sentence representations
Emeas n tml
Pmù
EmEMtm
Layer 1: 31 value=æ)
MHSA(query=zr, key=x,
Roi Eme ntml
Layer 2: 22 LayerNorm(E1 + æ) (contex- tualized sentences)
Ementt
Macro-F1 and Micro-F1 are calculated using the Fl score formula with their respective granular precision and recall scores.
Layer 3: æz <- Tanh(Linear(z22)
Layer 5: w < Softmax(Linear(2s, out- dim=l)) (sentence weights)
Fi x Precision Recall
Precision Recall
Output: od <- w X 33 (contextualized docu- ments)
Given Na (# of documents), N₂ (# of classes), Nº (# of epochs), N2 (batch size), and Nep (pa- rameters multi-head self-attention network), we specify a time complexity analysis of MEGClass's core framework:
Class distribution estimation: O(NaNk)
Contextualized embeddings: O(N.N,NCDD
Document Selection for Iterative Feedback: O(NaNg))
The primary time bottleneck comes from the ini- tialization step before the core framework, where sentence representation must be computed for each sentence, so this ultimately depends the type of encoder used for this step (c.g. X-Class's class- oriented representations, Sentence as shown in Section A.3).


Reviews:
[{"review_id": "dsN64tlkSd", "reviewer": "Reviewer_pjys", "summary": "", "questions": "A: l. 298: How do you justify to only look at the top 2 classes? Out of simplicity? Did you conduct ablation studies wrt. to this choice? What if a sentence pretty well discriminates e.g. two classes from the rest but is unable to distinguish among them?\nB: The overall model size/complexity does not become very clear. I understand that you use BERT as a classifier, but what are the specifications of the whole architecture before BERT?\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and positive comments and helpful questions. We are thankful that you found MEGClass to be very relevant for practitioners, innovative, and intuitively understandable. We have taken your questions very seriously, and hopefully our response provides more clarity on our method’s systematic approach and time complexity.\n\n_So it feels kind of handcrafted and potentially feels somehow brittle since the only hyperparameters that are more closely inspected are the temperature and #epochs._\n\n- We introduce four core hyperparameters in MEGClass, and to clarify, we conducted hyperparameter sensitivity analyses for each one in the original paper. At the high-level, our framework consists of three main parts: (1) initialization (Section 3.1), (2) pseudo-training dataset refinement (Sections 3.2-3.4), and (3) classifier fine-tuning (Section 3.5). The second part is our main component, which involves the following three substeps:\n\n     i. Class distribution estimation (Section 3.2) → there are **no hyperparameters** for estimating the initial class distribution of each document.\n\n     ii. Contextualized embeddings (Section 3.3) → for learning document representations that jointly integrate sentence and document level information), the hyperparameters we introduce are the **training epochs and regularization temperature** used in the weighted regularized contrastive loss. **Figures 7 and 8 demonstrate their general stability and that our selected values for both have stable performance across six diverse datasets.**\n\n     iii. Iterative feedback for refining the first two substeps (Section 3.4) → we introduce the following hyperparameters: (1) k for choosing the **top k%** contextualized document representations to enhance each class representation, and (2) the **number of iterations** of feedback. **Figure 6 presents a sensitivity analysis on both of these, and shows that we reach a relatively stable set of confident documents at the same k% across all six datasets.**\n\n- **For a fair comparison with the baselines, we use the same hyperparameter settings for the initialization and classifier fine-tuning steps** (e.g. for the final classifier finetuning, choosing the top $\\delta=50$% of documents as the pseudo-training dataset). \n\n_The pipeline consists of many components for each of which it does not become entirely clear how high the contribution to the overall performance actually is._\n\n- In our thorough evaluation, we analyzed the relative contributions of each component and found that the impact of our learned contextualized representations (ii) > iterative feedback (iii) > class distribution estimation (i). Specifically, the quality of the joint representations that we learn determines which documents we deem as sufficiently confident to enhance the class representations as iterative feedback, which then heavily influences the accuracy of the estimated class distribution used for the next iteration. We can see this immediate boost in performance from (ii) and (iii) in Figure 6, as well as the overall high quality documents chosen across all different datasets, shown in Table 10. We can make this explicit comparison between components more apparent in our existing “ablation studies” subsection under Section 4.2.\n\n_The overall model size/complexity does not become very clear. I understand that you use BERT as a classifier, but what are the specifications of the whole architecture before BERT?_\n\n- Following our previous response, the architecture of substep (ii), the multi-head self-attention network that learns the contextualized sentence and document representations, is:\n\n     - Input: $x$ ← Initial sentence representations\n     - Layer 1: $x_1$ ← MultiHeadSelfAttention(query=$x$, key=$x$, value=$x$)\n     - Layer 2: $x_2$ ← LayerNorm($x_1$ + $x$) (contextualized sentences)\n     - Layer 3: $x_3$ ← Tanh(Linear($x_2$))\n     - Layer 5: $w$ ← Softmax(Linear($x_3$, out-dim=1)) (sentence weights)\n     - Output: $cd$ ← $w \\bigtimes x_3$ (contextualized documents)\n\n- The input and hidden dimensions are 768 (to preserve the original BERT embedding dimensions), and we only use two self-attention heads to minimize complexity as much as possible. We train it from scratch (no pre-training), and it has exactly 2,955,265 total parameters.\n\n- Given $N_d$ (# of documents), $N_k$ (# of classes), $N_e$ (# of epochs), $N_b$ (batch size), and $N_{CD}$ (parameters in multi-head self-attention network), here is a time complexity analysis of MEGClass’s core framework:\n\n     i. Class distribution estimation: $O(N_dN_k)$\n\n     ii. Contextualized embeddings: $O(N_eN_bN_{CD})$\n\n     iii. Document Selection for Iterative Feedback: $O(N_dN_k)$\n\n- The primary time bottleneck comes from the initialization step before the core framework, where a sentence representation must be computed for each sentence, so this ultimately depends on the type of encoder used for this step (e.g. X-Class’s class-oriented representations, SentenceTransformers as shown in Section A.2). We can add this analysis to the Appendix!\n\n_How do you justify to only look at the top 2 classes? Out of simplicity? Did you conduct ablation studies wrt. to this choice? What if a sentence pretty well discriminates e.g. two classes from the rest but is unable to distinguish among them?_\n\n- We examine the _distinction_ between the top two classes; if the top two classes are equally strong, their distinction is 0, and it has no value for single-label classification. For example, a document which maps 50% to economy, 50% to international business provides no value for single-label text classification. On the other hand, a _<70%, 30%> split provides good value for single-label classification_. As shown in Figure 5, our ablation studies demonstrate the superiority of our sentence weight computation based on these top two classes over other common weighing mechanisms. We do acknowledge that your point of taking into account more than two classes would be very beneficial if we wanted to extend our setting to the multi-label classification. This could be a topic for future work!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a practically very relevant scenario for practitioners and there is a lack of existing benchmark data sets for evaluating a model’s capabilities with respect to solving this task.\n- The method is innovative and intuitively understandable \n- The presentation of the paper is overall very clear and easy to follow\n- The analysis is very thorough; the choice of data sets, performance metrics, and ablations are meaningful and well justified \n\n", "reasons_to_reject": "- The pipeline consists of many components for each of which it does not become entirely clear how high the contribution to the overall performance actually is. So it feels kind of handcrafted and potentially feels somehow brittle since the only hyperparameters that are more closely inspected are the temperature and #epochs.\n- The overall model size / complexity does not become very clear.", "questions_to_authors": "A: l. 298: How do you justify to only look at the top 2 classes? Out of simplicity? Did you conduct ablation studies wrt. to this choice? What if a sentence pretty well discriminates e.g. two classes from the rest but is unable to distinguish among them?\nB: The overall model size/complexity does not become very clear. I understand that you use BERT as a classifier, but what are the specifications of the whole architecture before BERT?\n\n", "paper_topic_and_main_contributions": "The authors of this paper propose a pipeline for learning from weak supervision signals of different granularities. They distinguish between document- and sentence-/word-level, both of which are taken into account in their approach, whereas only the document-level was taken into account in previous research. In their work, they exploit the information on different granularities to refine both the sentence and the document representations, as well as the word-based class representations via iterative feedback loops. While surpassing the compared weakly-supervised baselines on 5 out of 6 benchmarks, there still remains a notable gap to the fully supervised baseline.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 14/15, l. 44, l. 64 (and other places): Write weakly-supervised either w/ or w/o hyphen\nl. 472 afaik the data set is called “20newsgroups”\nFig. 5 is not appropriate; the cut-off y-axis conveys a misleading message as differences between the different bars are grossly overemphasized."}}, {"review_id": "rZoCroO1hS", "reviewer": "Reviewer_PUtu", "summary": "", "questions": "- Could you please provide an explanation for the substantial disparity between the experimental results presented in your main study and those reported in the original paper? Addressing this discrepancy would help dispel any unwarranted doubts and provide a clearer understanding of the findings.\n- It would be beneficial to elucidate the rationale behind the omission of certain State-of-the-Art (SOTA) methodologies, such as (LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification), from the comparative analysis. Similarly, clarifying the reasons for not comparing with fine-grained datasets like DBpedia and 20News would enhance the transparency of your research design.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your constructive feedback. We are encouraged that you find MEGClass’s use of multi-granularity information intriguing and that our method’s performance is “remarkable”. We have taken your constructive feedback seriously and have added a plethora of experiments for substantiating the statistical significance of our method’s SOTA performance, as well as on the datasets you have suggested. We are happily addressing your comments on our framework diagrams and are working on refining them.\n\n_While the paper commences with a toy example, the motivation behind the research is not adequately articulated. This leaves a gap in understanding the precise challenges the authors aim to address in weakly supervised text classification, leading to some confusion._\n\n- Document classification using class name only (with no training examples nor human annotations) is a challenging but useful task. This becomes even more challenging with complex/long text since a document about class A could contain many words in class B (e.g., a document on education may contain words or sentences related to politics).  Existing methods do not consider different levels of granularity and treat all sentences as equally significant for document-level classification. MEGClass automatically weighs the importance of each sentence in classification (lowers the weight of the sentences that indicate no class or multiple classes). We further developed a multi-head self-attention network to learn a document representation that reflects the critical information at multiple text granularities. Finally, we take the most confident document representations to enhance our initial word-based class representations through our iterative feedback approach, helping our model better understand what a class A document should look like at all levels of granularity. Because our multi-granularity approach can deal with these more complex documents and consider them for our final pseudo-training dataset, we allow our final classifier to be fine-tuned on more complex examples and consequently be more robust to challenging real-world documents.\n\n_The absence of paired t-tests undermines the persuasiveness of the primary experimental results, weakening the credibility of the findings._\n\n- Thank you for your feedback! In order to complete paired t-tests, we would have to run evaluation for each dataset on all of the baselines, this would require a substantial amount of time (e.g. ClassKG takes 30-40 hours). Hence, we include below the mean and variance of five trials conducted for each dataset, and we compare it with the most competitive baseline respectively. We can see that across the five trials, despite some variance (due to the randomized weight initialization when training the multi-head self-attention network and fine-tuning the final classifier), **MEGClass still reliably surpasses the SOTA for five out of six baselines, as originally stated**. We can update Table 1 with these averaged results and respective variances.\n\n|  | Test #1 | Test #2 | Test #3 | Test #4 | Test #5 | Mean | Var | Top Baseline |\n|---|---|---|---|---|---|---|---|---|\n| NYT-Topic | 86.04/68.84 | 86.04/68.48 | 85.13/67.87 | 85.21/67.77 | 84.70/67.17 | **85.42/68.03** | 0.283/0.338 | 81.67/71.54 (ConWea) |\n| NYT-Loc | 93.2/91.8 | 93.53/92.54 | 92.53/91.52 | 93.02/91.87 | 93.02/91.93 | **93.06/91.93** | 0.105/0.112 | 89.51/89.68(X-Class) |\n| Books | 56.3/55.7 | 56.40/55.46 | 56.54/55.76 | 56.33/55.70 | 56.16/55.91 | **56.35/55.71** | 0.015/0.015 | 55.0/54.7(ClassKG) |\n| NYT-Fine | 89.72/71.61 | 89.19/71.34 | 89.42/70.75 | 88.45/70.91 | 89.46/70.7 | **89.24/71.06** | 0.187/0.126 | 88.86/70.5(ClassKG) |\n| 20News | 82.36/81.46 | 81.45/80.30 | 81.60/80.61 | 81.72/80.49 | 81.46/80.3 | **81.72/80.63** | 0.113/0.185 | 81.0/82.0 (ClassKG) |\n| Yelp | 87.53/87.53 | 87.26/87.26 | 87.64/87.64 | 87.36/87.35 | 87.28/87.28 | 87.41/87.41 | 0.022/0.022 | **91.2/91.2** (ClassKG) |\n\n_Could you please provide an explanation for the substantial disparity between the experimental results presented in your main study and those reported in the original paper? Addressing this discrepancy would help dispel any unwarranted doubts and provide a clearer understanding of the findings._\n\n- Some of the compared baselines were originally designed for weakly supervised text classification (class label names + class-indicative keywords). To make the comparison fair, we modified their weak supervision to match our extremely weak supervision setting (only class label names). This led to some results for certain baselines being lower than what they reported. However, some are also higher as shown in the following table:\n\n| Baseline Model | Dataset | Their Reported F1 | Our Reported F1 |\n|---|---|---|---|\n| ConWea | 20News | 62/57 | **75.73/73.26** |\n| X-Class | NYT-Topic | **79.02**/68.55 | 79.01/**68.62** |\n| ClassKG | 20News | 80/75 | **81.0/82.0** |\n\n_It would be beneficial to elucidate the rationale behind the omission of certain State-of-the-Art (SOTA) methodologies, such as (LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification), from the comparative analysis._\n\n- LOPS is not a standalone weakly supervised text classification method, rather it enhances the performance of existing weakly supervised methods as a “performance-boost plug-in”. LOPS requires the pseudo-labels from _existing methods_ as input and additional class seed words (easier supervision level than ours) to optimize classifier fine-tuning. Integrating a performance boost plug-in into MEGClass is not within the scope of this paper, which compares the core weakly supervised text classification method with other core baselines. However, this is a promising research direction that we will explore in the future. Otherwise, to the best of our knowledge, we have included all recent and strong SOTA weakly/extremely weakly supervised text classification methods.\n\n_Similarly, clarifying the reasons for not comparing with fine-grained datasets like DBpedia and 20News would enhance the transparency of your research design._\n\n- We did not include DBPedia and 20News-Fine because not all of the baselines included them. Nonetheless, we covered sentiment-based reviews (Yelp), lengthy news articles (20News, NYT-Topic, NYT-Location), concise summaries with abstract classes (Books), and fine-grained classes (NYT-Fine). However, DBPedia explicitly defines the document’s topic in the first sentence, restricting its evaluation to word-level information and thus lacking challenging real-world scenarios with word or sentence-level content conflicting with the true document-level topic. For 20News-Fine, we favored NYT-Fine for evaluating fine-grained performance over 20News-Fine due to its training noise (e.g. gibberish documents). Still, we present MEGClass's performance on both datasets below with the original hyperparameters detailed in our paper. With the limited time, we compared MEGClass with the Macro-F1 scores of extremely weakly supervised baselines reported recently in [1]. **Notably, our method outperforms the SOTA by ~5.5 points on 20News-Fine and trails only behind ClassKG on DBPedia (Figure 4), an inefficient word-level approach.**\n\n- [1] Wang, Z., Wang, T., Mekala, D., & Shang, J. (2023). A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches. arXiv preprint arXiv:2305.12749.\n\n|  | LOTClass | X-Class | ClassKG | MEGClass (us) |\n|---|---|---|---|---|\n| 20News-Fine | 9.40 | 58.78$^\\dagger$ | 52.29 | **64.27** |\n| DBPedia | 57.98 | 89.50 | **94.75** | 92.14$^\\dagger$ |\n_$\\dagger$ denotes the second-best method._\n\n_The schematic representation of the model lacks clarity and adherence to academic standards. Some components appear to be directly sourced from other papers or online without proper references. Additionally, inconsistencies in fonts and stretching of diagrams in the experimental section compromise the overall presentation._\n\n- Thank you for your feedback, we appreciate the valuable comments regarding our presentation. We will fix the inconsistent font formatting of diagrams and minor typographical errors in the final version of the paper. We will work on making the main schematic representation much clearer in the updated version and have also replaced the diagram used for the contextualized sentences component but nonetheless added the Transformer paper's citation to our references list, thank you for pointing this out!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- I find the concept of incorporating multi-granularity information (word-level, sentence-level, and document-level) for weakly supervised text classification quite intriguing. The authors effectively substantiate their approach through a thorough set of experiments, highlighting the remarkable performance of their methods.\n- The authors notably create a pseudo-training dataset by estimating label distributions and iteratively refining it through feedback for fine-tuning the classifier.\n- The introduction of a regularized contrastive loss to update contextualized document representations and sentence attention weights in alignment with the target class distribution is a noteworthy design choice.", "reasons_to_reject": "- The schematic representation of the model lacks clarity and adherence to academic standards. Some components appear to be directly sourced from other papers or online without proper references. Additionally, inconsistencies in fonts and stretching of diagrams in the experimental section compromise the overall presentation.\n- While the paper commences with a toy example, the motivation behind the research is not adequately articulated. This leaves a gap in understanding the precise challenges the authors aim to address in weakly supervised text classification, leading to some confusion.\n- The absence of paired t-tests undermines the persuasiveness of the primary experimental results, weakening the credibility of the findings.\n- The experimental evaluation should be extended to include fine-grained datasets like DBpedia and 20News, both of which possess fine-grained labels. This broader testing would enhance the paper's applicability and insights.\n- Another round of proofreading is recommended to rectify minor typographical errors ('homerun') and instances of inappropriate diction, ensuring the paper maintains a high standard of clarity and professionalism.", "questions_to_authors": "- Could you please provide an explanation for the substantial disparity between the experimental results presented in your main study and those reported in the original paper? Addressing this discrepancy would help dispel any unwarranted doubts and provide a clearer understanding of the findings.\n- It would be beneficial to elucidate the rationale behind the omission of certain State-of-the-Art (SOTA) methodologies, such as (LOPS: Learning Order Inspired Pseudo-Label Selection for Weakly Supervised Text Classification), from the comparative analysis. Similarly, clarifying the reasons for not comparing with fine-grained datasets like DBpedia and 20News would enhance the transparency of your research design.", "paper_topic_and_main_contributions": "The paper presents MEGClass, a novel framework for text classification under extremely weak supervision using mutually enhancing text granularities. Unlike traditional methods relying on human annotations or class seed words, MEGClass utilizes class surface names as supervision. By jointly considering documents, sentences, and words, MEGClass captures context and inter-granularity class disagreements. It employs a Multi-Head Attention Network to create contextualized document representations, refining them through iterative feedback. Experimental results across benchmark datasets demonstrate MEGClass's superiority over other weakly and extremely weakly supervised methods, especially for longer documents and fine-grained classes. MEGClass's contributions include its innovative use of class surface names, context-aware representation learning, and the generation of a refined pseudo-training dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- The authors should consider citing the published version of the referenced work instead of relying solely on the arXiv citation. This practice would enhance the credibility and accuracy of the references in the paper. For instance: [1] Wang, Zihan, Dheeraj Mekala, and Jingbo Shang. \"X-Class: Text Classification with Extremely Weak Supervision.\" In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3043-3053. 2021."}}, {"review_id": "YKHu924sIK", "reviewer": "Reviewer_YqtT", "summary": "", "questions": "1. Maybe I missed the details, but I didn't get what is the source of the unlabeled dataset used. Are you using the original training data from each dataset (unlabeled)? Or is there some common general text dataset like Wikipedia to select the best sentences in the pseudo training data?\n2. For Figure 4, it would be better to mention the environment used for performing the experiment.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate your constructive and positive feedback, and are glad that you find MEGClass to be well-motivated and thoroughly explained. We hope that our response has helped clarify any confusion, and we will definitely integrate your comments into the updated version of the paper.\n\n_The iterative feedback mechanism is well-studied to improve the quality of labeled training data for weak supervision in text classification [1]._\n\n- While WeSHClass [1] does have an iterative feedback mechanism component, they use each class’s raw confident documents to _finetune_ its respective classifier (k classifiers for k classes). **What has not been previously well studied and simultaneously what MEGClass proposes** is that instead of directly using the raw documents themselves to construct a pseudo-dataset each iteration, we can use the learned _contextualized document representation_ of each confident document to directly update its respective word-based _class representation_. Not only does this prove to be successful (Figure 6, Table 10) and incorporates the idea of multiple text granularities even at the iterative feedback level, but it also avoids fine-tuning large models every single iteration and instead only once (at the very end).\n\n- [1] Meng, Yu, et al. \"Weakly-supervised hierarchical text classification.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.\n\n_The proposed novelty of using multiple hierarchical granularity is limited._\n\n- From our knowledge, while previous methods do consider word- and document-level text granularities, they (1) do not consider sentence-level granularities (MEGClass does), (2) do not consider all three levels of text granularities _jointly_ (MEGClass recognizes the dependencies between all three granularities and integrates them into a learned representation through its multi-head self-attention network– jointly updating both the learned sentence weights and document representations), and (3) consider all parts of the document to be equally significant to the document-level classification task (MEGClass accounts for vague/irrelevant sentences through its novel sentence-based weighted label ensemble as described in Section 3.2 and contextualized sentence weight learning in Equation 4). **Hence, our proposed novelty is with respect to our joint granularity consideration rather than the mere inclusion of multiple text granularities.**\n\n_Maybe I missed the details, but I didn't get what is the source of the unlabeled dataset used. Are you using the original training data from each dataset (unlabeled)? Or is there some common general text dataset like Wikipedia to select the best sentences in the pseudo training data?_\n\n- Yes, we just use the original training dataset without the labels. Since we do not use any of the labels provided anyways, we choose to evaluate on all of the training documents to show that our method can perform well on a large number of documents, where 50% of the dataset is _not_ seen by the final classifier we fine-tune (Section 3.5). Also, just to clarify, we select the best _documents_ for the pseudo training dataset– this is beneficial compared to simply using sentences since this is more adaptable to the expected input being documents with multiple topics likely being discussed at the sentence and word level. MEGClass’s angle is to be able to pick out the best pseudo-training documents that confidently discuss their respective class topic, even if they contain words or sentences that overlap with other classes.\n\n_For Figure 4, it would be better to mention the environment used for performing the experiment._\n\n- **We have included the experimental settings for each baseline experiment in Section A.6.** Specifically, each baseline method is evaluated using the extremely weak supervision setting (class label names only, no additional seed words provided) and hyperparameters as specified in their original implementation. We conduct all experiments on a single NVIDIA RTX A6000 GPU. We will add the reference to Section A.6 in the figure’s caption.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents both qualitative and quantitative results to demonstrate that the proposed MEGClass method outperforms existing weak supervision approaches. The case study in the ablation section highlights the steps involved in classifying a document into one of the predefined classes.\n2. The proposed MEG-Class method is well-motivated and thoroughly explained. Each level of text granularity builds upon the previous one, and they are ultimately combined iteratively to enhance the quality of the generated pseudo training data.\n3. The authors conducted experiments on six different benchmark datasets, achieving significant improvements over the baselines.\n4. The writing in the paper is very clear and easy to comprehend.\n", "reasons_to_reject": "1. The iterative feedback mechanism is well-studied to improve the quality of labeled training data for weak supervision in text classification [1].\n2. The proposed novelty of using multiple hierarchical granularity is limited.\n\n[1] Meng, Yu, et al. \"Weakly-supervised hierarchical text classification.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.", "questions_to_authors": "1. Maybe I missed the details, but I didn't get what is the source of the unlabeled dataset used. Are you using the original training data from each dataset (unlabeled)? Or is there some common general text dataset like Wikipedia to select the best sentences in the pseudo training data?\n2. For Figure 4, it would be better to mention the environment used for performing the experiment.\n", "paper_topic_and_main_contributions": "The authors propose a novel weakly supervised text classification method that utilizes various granularities, such as words, sentences, and documents. They also introduce an iterative feedback mechanism to construct pseudo-labeled training data, which is subsequently utilized for training a task-specific classifier. The paper comprehensively elucidates the problem's premise and underscores the significance of employing distinct text granularities in text classification tasks. The authors conduct experiments on several benchmark datasets, achieving state-of-the-art performance on the majority of them. Furthermore, they conduct various ablation studies to provide substantial evidence supporting the claims presented in the paper.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "7KuUXTnJdM", "reviewer": "Reviewer_7Hfk", "summary": "", "questions": "1) Can you comment on the runtime performance of various methods across datasets?\n\n2) In Section 3.3, multi-head self-attention was mentioned. Is this something custom-made or similar to the transformer architecture?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your feedback and suggestions. We are happy that you find MEGClass’s idea for integrating multiple levels of granularity, especially for more challenging documents, to be helpful as well as substantiated through our “thorough” evaluation. We have taken your suggestions seriously and have conducted additional experiments to supplement our existing ablation studies/runtime analysis and better substantiate the need for MEGClass.\n\n_No quantitative results were presented in the main paper on how the methods will perform compared to chatGPT/LLMs in a zero-shot setting in finding class representative samples. A section in the paper's appendix mostly focuses on qualitative results._\n\n- We focused on qualitative results in Section A.4 because the GPT 4 API is not open-source. ChatGPT also requires extensive prompt engineering while not guaranteeing a consistent output format nor the predicted label being within the set of provided classes. Additionally, **through our qualitative comparison in Section A.4, we demonstrate ChatGPT’s existing challenges to deal with documents that contain keywords from similar fine-grained classes** (e.g. terms from the “economy” class used in an “international business” document), where it is necessary to jointly consider the word-level and document-level context for properly understanding the primary subject of conversation– which MEGClass does. \n\n- Nonetheless, given that LLaMA-13b qualifies as a significantly large language model (as stated, 13 billion parameters) and is open source, we have presented a quantitative comparison between MEGClass and zero-shot LLaMA-13b for the full NYT-Fine corpus (only one dataset given the rebuttal time constraint). After parsing out the answer from the generated response and running evaluation, we can see that **MEGClass significantly surpasses LLaMA-13b on this dataset**. We find that the reason behind this is that LLaMA-13b tends to predict the class “football” (originally indicating American Football in the NYT-Fine corpus) for the majority of all sports classes, which represent the majority of the dataset. Additionally, it rarely selects fine-grained classes like “international business” or “law enforcement”, indicating a lack of class understanding.\n\n| Dataset/Model | MEGClass | LLaMA-13b |\n|---|---|---|\n| NYT-Fine | **89.72/71.61** | 35.25/35.59 |\n\n_The paper reports results on only bert-base-uncased. Requires a study using various LLM as base embeddings to understand the impact on different base embeddings in MEGClass._\n\n- In Section A.2, we chose to study different initializations of sentence and class representations as they are the primary input into our core framework. We specifically compare the sentence representations we use (detailed in Section 3.1; applies X-Class’s class-oriented word weighting mechanism to the bert-base-uncased embeddings) with all-mpnet-base-v2 (SentenceTransformers) and analyze this impact. The initialized sentence representation uses bert-base-uncased so that we can provide a fair comparison with the rest of the baselines, which all primarily use bert-base-uncased as their base model. Nonetheless, we include a comparison between bert-base-uncased and bert-large-uncased on the 20News, Books, and NYT-Topic datasets (just three chosen for the sake of time; the three span different document lengths, writing styles, and domains). _It is important to note that a recent paper [1] studying extremely weakly supervised methods has observed the drop in performance from bert-base and bert-large and a failure with roberta-base on X-Class’s weighing mechanism_, and they hypothesized that this may be due to the “distribution of similarities of representations generated by a language model [being] different” across models. Since we use X-Class’s class-oriented representations for sentences and their class representations, this may explain the drop in performance. Users can switch out these initial sentence representations used instead, as we tested in Section A.2, which will have a more reasonable impact on MEGClass’s performance:\n\n|  | 20News | Books | NYT-Topic |\n|---|---|---|---|\n| bert-base-uncased | **82.36/81.46** | **56.3/55.7** | **86.04/68.84** |\n| bert-large-uncased | 73.64/69.47 | 33.0/33.1 | 68.69/59.53 |\n| roberta-base | 16.28/10.89 | 7.74/5.08 | N/A |\n\n- [1] Wang, Z., Wang, T., Mekala, D., & Shang, J. (2023). A Benchmark on Extremely Weakly Supervised Text Classification: Reconcile Seed Matching and Prompting Approaches. arXiv preprint arXiv:2305.12749.\n\n_Can you comment on the runtime performance of various methods across datasets?_\n\n- Yes, across the datasets, the runtime performance is proportional to the 20News runtime analysis results (same order) and simply varies based on the number of documents and classes. We show the comparison for NYT-Fine as it has the highest number of classes (26) and hence can provide more insight into how the different methods scale with respect to the number of classes. MEGClass, LOTClass, and X-Class have their performance minimally affected, whereas ClassKG and WDDC scale in units of hours. This similar type of behavior is observed across the other datasets:\n\n| (Class #, Doc #) | MEGClass | LOTClass | X-Class | WDDC | ClassKG |\n|---|---|---|---|---|---|\n| 20News (5, 17.9K) | 1h 20m | 1h | 42m | 5h 54m | 30h |\n| NYT-Fine (25, 13.1K) | 1h 55m | 1h 21m | 1h  | 8h 19m | 42h |\n\n_In Section 3.3, multi-head self-attention was mentioned. Is this something custom-made or similar to the transformer architecture?_\n\n- We use the standard multi-head self attention block that is also used in the Transformer architecture, but it is only one block in our “custom-made”, light-weight network implementation (we define it in our source code under MEGClass/model.py). Our specific custom-made network learns the contextualized sentence representations, their corresponding sentence weights, and the contextualized document representations based on the former two. Using this custom network, we can directly update the sentence weights themselves based on our novel weighted regularized contrastive loss.\n\n_Can the Figure 3 caption describe the figure more elaborately?_\n\n- We included longer descriptions of each step within the figure itself, which covers the majority of the high-level details we discuss in Section 3.4, but we will consider how to make the figure and caption more clear!\n\n_Some parts of A.6 need to bring in Section 4.1. It was hard to figure out that bert-base-uncased was used._\n\n- We moved Experiment Setup to the Appendix for the sake of brevity, but we can move these details into Section 4.1 in the updated version since an additional page in the final version is allowed!\n\n_In the Subsection \"contextualized document representations,\" can we add some reference for regularized contrastive loss? Equation 5 looks similar to Equation 6 in [1]. If not, can you comment?_\n\n- Yes this is a fair citation that we originally missed, we can add this reference for regularized contrastive loss. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) Idea of utilizing words, sentences, and documents level context in an integrated fashion to identify the true label of a document where only one class topic is discussed at the document level but several topics are discussed at the word or sentence level. The method seems to benefit more for longer documents.\n\n2) Evaluation is thorough. The paper compares performance improvements in F1 scores and provides a run-time comparison of various methods.", "reasons_to_reject": "1) No quantitative results were presented in the main paper on how the methods will perform compared to chatGPT/LLMs in a zero-shot setting in finding class representative samples. A section in the paper's appendix mostly focuses on qualitative results. \n\n2) The paper reports results on only bert-base-uncased. Requires a study using various LLM as base embeddings to understand the impact on different base embeddings in MEGClass.", "questions_to_authors": "1) Can you comment on the runtime performance of various methods across datasets?\n\n2) In Section 3.3, multi-head self-attention was mentioned. Is this something custom-made or similar to the transformer architecture?", "paper_topic_and_main_contributions": "The paper proposes a sample selection strategy for training text classification models using only class surface names. To select the most representative samples, the paper proposes to utilize word, sentence, and document-level context in an integrated fashion through a learned function. The paper also studies iterative feedback mechanisms to augment the informativeness of the class surface names.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) Can the Figure 3 caption describe the figure more elaborately?\n\n2) Some parts of A.6 need to bring in Section 4.1. It was hard to figure out that bert-base-uncased was used."}, "extra_text": {"Missing_References": "1) In the Subsection \"contextualized document representations,\" can we add some reference for regularized contrastive loss? Equation 5 looks similar to Equation 6 in [1]. If not, can you comment? \n\nReferences\n1) Contrastive Regularization for Semi-Supervised Learning"}}]

Author rebuttal:

Meta-review: The reviewers all recognized the importance of this task for practitioners and are excited about how intuitive the presented methods are, and I agree that this is an important and interesting task. This could be even better supported, however, as one reviewer raised a concern about the motivation of the paper and specifically the toy example given in Section 1. I also found this hard to follow; perhaps the authors can select a more intuitive and/or compelling example, which would address this concern. Generally though, the paper is very well motivated and this is recognized by all the reviewers.

Multiple reviewers expressed confusion over the system explanation and visualization. This part of the paper should be clarified and additional context should be given about the sources of the various components. I found this part confusing as well. 

Reviewers gave ambiguous soundness scores, but I believe these concerns were mostly addressed in the rebuttal (e.g., the variation tests in response to Reviewer PUtu).  I agree these results should have been included in the original paper, but they do address this particular concern. The authors also provided an analysis of additional datasets, on top of the many datasets already included in the submitted paper.
Decision: Accept-Findings