Cross-task Knowledge Transfer for Extremely Weakly Supervised Text Classification
Year: 2023

Abstract:
Text classification with extremely weak supervision (EWS) imposes stricter supervision constraints compared to regular weakly supervised classification. Absolutely no labeled training samples or hand-crafted rules specific to the evaluation data are allowed. Such restrictions limit state-of-the-art EWS classification methods to indirect weak labeling techniques that assign unnatural label uncertainty estimates. We present PLAT, a framework that creates weak labels by leveraging recent developments in zero-shot text classification. PLAT employs models trained for sub-tasks other than classification to label documents. Most importantly, PLAT refrains from assigning overly confident weak labels and improves soft-label training performance for downstream classifiers. Classifiers trained with PLAT significantly outperform those trained on weak labels generated by the previous state-of-the-art in extremely weakly supervised text classification.

heading: 1 Introduction
text: We undertake the low-resource task of categorizing an unlabeled set of documents using just candidate category labels. The task is a stricter subtask of weakly supervised text classification – weak labels cannot be obtained even through utilizing a small training set or hand-crafted rules based on domain knowledge. Such task formulation mimics a realistic and practical scenario where one has to classify a set of documents into a label from a pre-defined label set, using only class names. Following Wang et al. (2021) we call this task classification with extremely weak supervision (EWS).
Due to such additional constraints on sources of supervision, models under EWS cannot trivially adapt recent state-of-the-art approaches under regular weak supervision. Best-performing methods for classification under EWS usually involve mining a set of category-indicative keywords from pre-trained language models (Meng et al., 2018;
Mekala and Shang, 2020; Türker et al., 2020; Meng et al., 2020; Zeng et al., 2022). At evaluation time, each document is compared to the keyword set of each label. The weak label for a document is the label with the keyword set most similar to words that constitute the document. This divorcement of feature extraction and label assignment introduces additional noise during weak labeling, causing unnatural assignment of label confidence and oversensitivity to training size (Wang et al., 2021).
We overcome such limitations in EWS by leveraging pre-trained language models to create weak labels in an end-to-end fashion. Most importantly, we eliminate the keyword-collection step in currently popular EWS approaches. We employ language models trained on non-classification tasks (textual entailment, next sentence prediction, and multiple-choice question-answering) as weak labelers for classification. Our research bridges weaklysupervised noisy-label training with recent developments in prompt-based low-shot text classification (Yin et al., 2019; Keskar et al., 2019). Our framework realizes both the robustness of noisylabel training and the label efficiency of prompting. We use publicly available, off-the-shelf models for each source task in our experiments.
Our contributions are as follows:
• We analyze the limitations of popular existing methods in EWS, especially in their unnatural assignment of pseudo-label confidence.
• We present PLAT1, a framework that utilizes models trained in subtasks other than classification to create weak labels for classification. Downstream classifiers trained with our weak labels significantly outperform the previous state-of-the-art in difficult EWS datasets.
• We analyze how cross-task weak labels act as better pseudo-labels, with roots in existing
1Pseudo-Labeling Across Tasks
5329
research in knowledge distillation.

heading: 2 Background

heading: 2.1 Weakly supervised text classification
text: Broadly, two lines of research exist in weaklysupervised text classification: obtaining better weak labels (Hancock et al., 2018; Chatterjee et al., 2020a; Rao et al., 2021; Zhang et al., 2021a, 2022a), and streamlining the training of downstream classifiers with the obtained noisy labels (Onoe and Durrett, 2019; Ren et al., 2020; Mekala et al., 2022; Yu et al., 2022; Kuang et al., 2022). PLAT focuses on improving the former by creating weak labels via knowledge transfer from models trained on tasks other than classification.
Weak labels were traditionally assigned using manually written rules (Cachay et al., 2021; Zhang et al., 2022a). Since rule-based labeling necessitates domain knowledge and hand-crafted rules specific to each dataset, much research efforts focused on automatic rule generation. However, even automatically generated rules cannot be used in situations that require EWS because the process either necessitates a small labeled dataset of the same classification task (Varma and Ré, 2018; Banerjee et al., 2019; Sukumaran et al., 2022), or human feedback is required in the iterative learning process (Zhang et al., 2022b). Under EWS, we require a method that fully automates the weaklabeling process, without any classification datasets or dataset-specific domain knowledge.
PLAT employs cross-task knowledge transfer to achieve completely automated weak-labeling without any labeled classification data.

heading: 2.2 Cross-task knowledge transfer
text: In cross-task knowledge transfer, a model trained for a specific source task solves a different target task. Cross-task knowledge transfer is useful when labeled training data is scarce for the target task but is abundant or unnecessary for the source task (Egonmwan et al., 2019; Lin et al., 2021). Such preconditions make cross-task knowledge transfer naturally suitable for pseudo-labeling in weakly supervised training. In Egonmwan et al. (2019), for instance, question-answering models are used to create weak summary labels.
Because data efficacy of weak labelers is a prerequisite under weak supervision, recently popular zero-shot classification methods based on prompting (Brown et al., 2020; Liu et al., 2021; Sanh
et al., 2022) are appealing approaches to automatic label creation. In the EWS setup, only cross-task zero-shot labelers can be used, because the task prohibits any labeled data for classification. Although cross-task knowledge transfer with a classification target task is extensively researched (Hancock et al., 2018; Wang et al., 2019; Khodorchenko, 2019; Rao et al., 2021; Chatterjee et al., 2020a), we are the first to explore prompt-based, cross-task distillation for weak-labeling in text classification.
Zhang et al. (2021a) also uses language model prompting for weak label generation, but its weaklabeler is a text classification model and thus is not a cross-task setup. In concurrent work, Smith et al. (2022) also prompts language models for zero-shot weak label generation. The research leverages multi-task models trained with multiple source tasks, either with extremely large scale (GPT-3) or already on text classification source tasks (T0++). In contrast, our work focuses on cross-task knowledge distillation capabilities of data-efficient, single-task models, each trained for a different non-classification source task. Smith et al. (2022) focuses on zero-shot capabilities that emerge from extreme-scale text generation models, while our work explores methods to handle various model output types (open-ended, binary, and multiple-choice) for weak labeling. We further provide qualitative analysis on the confidence assigned to each weak label.

heading: 2.3 Common approaches to text classification under EWS
text: Popular methods under the EWS constraint employ a keyword-set matching scheme for weak labeling. Keywords for each label are auto-generated by mining pre-trained language models. Throughout this paper, we call these methods keyword-based EWS.
WeSTClass (Meng et al., 2018) augments training data by creating pseudo-documents from seed words for each class. ConWea (Mekala and Shang, 2020) uses masked language models to discern overlapping keywords with context. Contextinfused keyword set for each class is matched with documents for weak labeling. WESSTEC (Türker et al., 2020) queries a knowledge base for information about each label and calculates the similarity between a document’s vector representation to each label knowledge embedding. LOTClass (Meng et al., 2020) enriches each label’s keyword set by collecting possible replacement words for
every label from masked language models. X-Class (Wang et al., 2021) force-aligns document representations to label embeddings for weak labeling and achieves state-of-the-art results in EWS. All aforementioned methods train a downstream classifier such as BERT (Devlin et al., 2019) or RoBERTa (Liu et al., 2019) as the second step in their pipelines. In succeeding work, ClassKG (Zhang et al., 2021b) posits EWS as a keywordsubgraph annotation task and takes keyword correlation into account.
PLAT departs from existing conventions by eliminating the keyword collection process from the EWS pipeline. We directly mine weak predictions instead of keywords from trained language models.
In a similar and concurrent work as ours, WDDC (Zeng et al., 2022) also uses zero-shot prompting in pre-trained language models to create weak labels. WDDC uses cloze prompts to extract keyword sets (to be compared against document words, as in most aforementioned works), which is significantly different from PLAT that assigns classification labels directly with the prompts.
We choose LOTClass and X-Class as baselines in our experiments, for their state-of-the-art results and reproducibility.

heading: 3 Problems with keyword-matching EWS
text: Compared to simple supervised classification, EWS methods mentioned in Section 2.3 introduce two additional steps to the training pipeline: building category-indicative keyword sets and assigning weak labels to unlabeled documents using the built keyword sets. Our investigations show that the disjoint nature of such approaches leads to unwanted
side effects that degrade weak-label quality.

heading: 3.1 Uninformative label confidence
text: Accurate estimates of label uncertainty are important in noisy training scenarios commonly used in weakly-supervised classification (Meng et al., 2020; Yuan et al., 2020). The keyword-matching process used in state-of-the-art EWS forces weak labelers to gauge weak label confidence indirectly. We find that weak labels obtained this way are often coupled with unreliable label confidence that is sensitive to hyperparameters such as the size of the keyword set or evaluation set. In X-Class, measuring the distance of a document’s embedding from its label cluster center is the only way to measure label uncertainty. In LOTClass, prediction confidence is the number of keywords a document contains in its pseudo-label keyword set.
Even though PLAT uses non-classification models for weak labeling, much more natural confidence is assigned to its weak labels. Label confidence of correct predictions are higher on average compared to those of wrong predictions (Figure 1). In contrast, LOTClass and X-Class assigns similar confidence to both. Weak labels created by LOTClass and X-Class show drastic drops in pseudolabel count as the label confidence threshold increases (Figure 2). Such overconfidence in label quality estimates can hinder downstream classifier performance (Wei et al., 2022; Jiang et al., 2021).

heading: 3.2 Inability to handle complex class names
text: Keyword-based EWS relies on mining words within documents in the evaluation set and extracting category-indicative words for each class. Therefore, even state-of-the-art methods require class names to be either lexically or contextually descriptive. In clickbait classification, for example, the word “clickbait” does not exist among news headlines the model has to classify. In such cases, keyword-based EWS methods have no anchor within the documents to extract categoryindicative keywords for the word "clickbait”. Wang et al. (2021) shows existing EWS methods falter when the label names do not appear in documents to be weakly labeled.
We observe the same phenomenon even with keyword-based EWS methods that consider language context. Robustness further deteriorates when label names are more complex, such as consisting of multiple tokens. Most keyword-based EWS use masked language models for contextaware keyword search, and it is not straightforward to consolidate sequence vectors as a single, contextual vector to be used in clustering algorithms in keyword-based EWS. Our method sidesteps such limitations by flexibly handling any label name through language model prompting.

heading: 3.3 Sensitivity to dataset size
text: Weak-label quality of keyword-based EWS also relies heavily on the size of the test set. While EWS methods require no labeled documents for training, existing methods still require a sizable count of unlabeled data to perform well. At their core, most keyword-based EWS methods aim to generate cluster-centers for each class by leveraging textual information in unlabeled documents. A smaller evaluation set will naturally result in lower-quality cluster boundaries. To overcome such reliance on
dataset size, we propose a way to weakly label each document in a zero-shot manner. We confirm this intuition by comparing F1 scores of weak labels created by keyword-based EWS (X-Class) and with PLAT (Figure 3) at varying confidence thresholds.

heading: 4 PLAT
text: PLAT draws inspiration from recent findings in zero-shot language model prompting (Yin et al., 2019; Keskar et al., 2019; Ma et al., 2021) to obtain weak labels for unlabeled documents. In the EWS setting, PLAT leverages source models trained on a single non-classification task to solve classification tasks.
PLAT follows the typical two-step weaklysupervised classification pipeline. In the first phase, PLAT creates weak labels for classification. In the second phase, a final classifier is trained using the obtained weak labels. The novelty of PLAT lies in improving the weak labeling phase with cross-task knowledge distillation. We aim to keep the PLAT framework source-model-agnostic and make the weak labelers hot-swappable, taking advantage of parallel advances in zero-shot NLP.

heading: 4.1 Cross-task weak labeling
text: We test three different models for weak labeling, each trained on a single task: entailment (PLATENT), next sentence prediction (PLATNSP), and multiple-choice question answering (PLATQA). Although each task and corresponding model have different input and output formats, adding appropriate prompts can reduce all tasks into indirect classification tasks.
Let X = {x0, . . . , xm} be a set of unlabeled documents to be classified. A weak labeler must categorize xi as a single class from the set of all possible classes C = {c0, ..., cn}.
For every xi ∈ X , PLAT’s pseudo-labelers generate two kinds of labels: a hard label Hi ∈ C, which is the single most likely class that xi belongs to, and a soft label Si, which is a categorical distribution over C expressing the class probability of xi. We train separate downstream classifiers using hard and soft labels, to compare how taking label uncertainty into account can help in weakly-supervised classification.

heading: 4.1.1 Entailment (PLATENT)
text: Yin et al. (2019) explores zero-shot text classification through entailment. Similarly, we pose classification as an entailment task by ranking entailment probabilities of xi and each verbalized class. A verbalized class (Schick and Schütze, 2021) is the every class name in the form of a sentence, adapted to appear as an input to the entailment model.
Verbalizers can be adapted for each classification task. For topic classification, the verbalizer could be “This text is about <class name>.” For spam detection, the verbalizer to represent the spam class could be “This is an ad". The full set of verbalizers used is detailed in the Experiments section. VENT is a set of all verbalized class names:
VENT = {verbalizer(c) | c ∈ C}. (1)
For every xi ∈ X , we construct a set of all pairs of xi and each verbalized label v ∈ VENT , between all of which we calculate textual entailment:
PairsiENT = {(xi, v) | v ∈ VENT }. (2)
Entailment model MENT is a model that takes a sentence pair (s1, s2) and calculates the probabilities that sentence s1 entails, contradicts, or has no relation to sentence s2. In this work, we only use entailment probabilities.
We use MENT to calculate the entailment probability of every (xi, v) ∈ PairsiENT .
ProbsiENT =
{MENT (xi, v) | (xi, v) ∈ PairsiENT }. (3)
The hard label Hi for xi is argmax(ProbsiENT ), and the soft label Si is softmax(ProbsiENT ).

heading: 4.1.2 Next sentence prediction (PLATNSP)
text: Ma et al. (2021) finds that next sentence prediction (NSP) and reverse NSP models perform on par with entailment in zero-shot text classification. In our experiments, NSP and reverse NSP weak labelers had a negligible difference in final classifier performance. We choose reverse NSP for higher reported classification scores in Ma et al. (2021).
We use the same verbalizers (VNSP = VENT ) as in entailment-based weak labeling in the preceding section. For every xi ∈ X , we construct a set of all pairs of xi and each verbalized label v ∈ VNSP , similar to PairsiENT in 4.1.1:
PairsiNSP = {(xi, v) | v ∈ VNSP }. (4)
For all v ∈ V , we calculate probabilities that xi appears after each v. NSP Model MNSP takes a sentence pair (s1, s2) and calculates the probability that s2 appears after s1.
ProbsiNSP =
{MNSP (v, xi) | (xi, v) ∈ PairsiNSP }. (5)
The hard label Hi for xi is argmax(ProbsiNSP ), and the soft label Si is softmax(ProbsiNSP ).

heading: 4.1.3 Multiple-choice question-answering (PLATQA)
text: A multiple-choice question-answering (QA) model MQA takes a context, a question, and answer choices, and returns the distribution of answer possibility over the answer choices. To pose QA as a classification task, we set the context as each xi,
the question as a dataset-specific prompt p, and the answer choices as verbalized versions of all classes. The question forces the model to select one verbalized element of C as an answer.
The full set of prompts and verbalizers used in PLATQA is detailed in the experiments section. Even though prompt p is dataset-specific, its construction does not require domain knowledge, and instead depends on the type of classification (I.e. topic classification, location classification, etc.).
Formally defined,
ProbsiQA = MQA(xi, p, VQA), (6)
where
VQA = {verbalizer(c) | c ∈ C}. (7) The hard label Hi for xi is argmax(ProbsiQA) and the soft label Si is softmax(ProbsiQA).

heading: 4.2 Final classifier training
text: A separate text classifier is trained with obtained weak labels. We use BERT in all our experiments. The final classifier is the output model of PLAT.

heading: 4.2.1 Training with hard labels
text: Given a set of hard labels {H0, ...,Hi} created by a weak-label generator, we train a downstream classifier B by maximizing the likelihood of predicting
the weak label given the document. The loss function is a standard cross-entropy objective:
Lhard = − m∑
i=0
∑ j∈C y(Hi) log(B(xi)j), (8)
where y(Hi) is 1 only if j = Hi and 0 otherwise. B(xi)j is the prediction confidence of classifier for class cj ∈ C on document xi.

heading: 4.2.2 Training with soft labels
text: We adopt a similar objective function when training with confidence-aware soft labels. The final classifier is trained to minimize the divergence between the one-hot model prediction and the soft confidence distribution from the weak labeler over the set of all possible class names. The classifier’s objective function becomes:
Lsoft = − m∑
i=0
∑ j∈C Sji log(B(xi)j), (9)
where Sji is the weak label confidence of specific class j ∈ C from overall confidence distribution assigned to xi.

heading: 5 Experiments and Results
text: We qualitatively analyze PLAT in two aspects: accuracy of generated weak labels, and prediction
classification performance of the final classifier trained with the weak labels. For the latter, we measure classifier performance in both hard- and softlabel (confidence-aware) training. The same configuration for training the final classifier is applied to all variants of PLAT and baseline weak labelers. Classifier performance is measured in macro- and micro-F1 scores.

heading: 5.1 Datasets
text: We test PLAT on topic and clickbait classification datasets. For topic classification, we use AGNews (Zhang et al., 2015), Yahoo Topics (Zhang et al., 2015), and DBpedia (Zhang et al., 2015). We use Clickbait Detection (Chakraborty et al., 2016) for clickbait classification. Table 1 provides a detailed description of each benchmark dataset.
For topic classification datasets, we use the verbalizer "This text is about <class name>" for all models and the prompt "What is this text about?" for PLATQA. For clickbait classification, we use the verbalizers "This is <news/spam>" and the prompt "Is this news or spam?" for PLATQA.

heading: 5.2 Source models for weak-labeling
text: We use publicly available cross-task labelers in all variants of PLAT. For PLATENT, we use BART2 (Lewis et al., 2020) trained on MNLI (Williams et al., 2018). For PLATNSP, we use BERT3 trained
2https://huggingface.co/facebook/bart-large-mnli 3https://huggingface.co/bert-large-cased
with standard token unmasking and NSP objectives. For PLATQA, we use RoBERTa4 model trained on RACE (Lai et al., 2017).
To train the final classifier with weak labels generated by aforementioned models, we fine-tune a pre-trained BERT model5 with a constant learning rate of 5e−5. We use an AdamW optimizer (Loshchilov and Hutter, 2018) with β1 = 0.9, β2 = 0.999, eps = 1e−6, and no weight decay.

heading: 5.3 Hard label training results
text: Classification performance of the final classifier for baseline weak-labelers and variants of PLAT is detailed in Table 2. Weak labels generated with PLAT yield notably higher F1 scores compared to those generated by baselines, except on DBpedia. The high performance of X-Class on DBpedia can be attributed to longer average document length and greater test set size. Compared to other datasets, DBpedia provides a greater amount of raw text from which keyword-based baselines can mine category-indicative keywords. In such settings, PLAT’s zero-shot capability does not provide an advantage as great in scenarios with fewer resources.
4https://huggingface.co/LIAMF-USP/roberta-largefinetuned-race
5https://huggingface.co/bert-base-cased

heading: 5.4 Confidence-aware training results
text: Classification performance after confidence-aware training is detailed in Table 3. PLAT also notably outperforms baselines with soft labels, taking better advantage of confidence-aware training. Our results confirm past research in knowledge distillation that accurate estimates of label uncertainty lead to better model calibration (Chatterjee et al., 2020b; Rizve et al., 2021).
Earlier works in EWS try retaining only labels with confidence over a certain threshold δ. In a noisy-training scenario, a trade-off exists between retaining a large number of training examples and average label confidence. Our work confirms findings in Wang et al. (2021) that excessively high label δ degrades final classifier performance (Figure 4). While tuning the threshold parameter results in a higher increase in F1 scores for PLAT, we report scores at δ = 0 for a fair comparison with previous work and to eliminate δ as a hyperparameter.

heading: 5.5 Classification difficulty analysis
text: We analyze how the performance of each weak labeler changes according to the classification difficulty of each sample. Classification difficulty of a sample is defined as the number of weak labelers that made wrong predictions. Since we compare 5 models, the maximum difficulty is 5. PLAT
assigns a much more “natural” confidence distribution, where the model is confident about lowdifficulty questions while comparatively uncertain about high-difficulty questions (Figure 5). Models that fail to show such graduality tend to make inaccurate predictions (LOTClass in AGNews and Yahoo, X-Class in Clickbait, and PLATNSP in DBpedia), especially on more difficult samples.

heading: 6 Conclusion
text: We present three variants of PLAT, a framework for text classification under extremely weak supervision. By eliminating keyword-based weak labeling, PLAT sidesteps the brittle dependence on evaluation set size and hyperparameters found in previous state-of-the art methods. PLAT is a flexible framework that leverages prompting to generate weak labels with more natural confidence estimates.
PLAT makes no assumptions about the training dynamics of its source models. Therefore, evolutions of source models are completely orthogonal to developments in PLAT. The black-box treatment of its weak labeler models enables the usage of completely unsupervised weak labelers – a potential already demonstrated by PLATNSP. We expect future developments in unsupervised solutions to enable even more resource-efficient classification under PLAT.
Limitations
We identify the following limitations of PLAT and strategies to overcome such drawbacks:
• Performance of the final classifier is dependent on the black-box source weak labeler. We believe this limitation can be worked around in a real-word setting by ensembling source models to vote on a likely weak label for practical accuracy gains.
• Best-performing source models might differ for different tasks. The dataless nature of EWS prevents precursory accuracy evaluations while choosing the source weak labeler model. However, quality of candidate weak labelers can be gauged indirectly. Users can examine confidence distributions of weak labels (as in Figure 1 and Figure 2) as an indicator of pseudo-label "naturalness". They can also perform difficulty analysis (as shown in Figure 5(a)) that does not require any labeled data. In a real-world scenario, ensemble weak labelers will be used, eliminating the need to choose a single best source model.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Last section
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 5
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Low resource study; computing infrastructure was almost all-encompassing.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Detailed in footnotes
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: We expect future developments in unsupervised solutions to enable even more resource-efficient classification under PLAT. Limitations
We identify the following limitations of PLAT and strategies to overcome such drawbacks:
• Performance of the final classifier is dependent on the black-box source weak labeler. We believe this limitation can be worked around in a real-word setting by ensembling source models to vote on a likely weak label for practical accuracy gains. • Best-performing source models might differ for different tasks. The dataless nature of EWS prevents precursory accuracy evaluations while choosing the source weak labeler model. However, quality of candidate weak labelers can be gauged indirectly. Users can examine confidence distributions of weak labels (as in Figure 1 and Figure 2) as an indicator of pseudo-label "naturalness". They can also perform difficulty analysis (as shown in Figure 5(a)) that does not require any labeled data. In a real-world scenario, ensemble weak labelers will be used, eliminating the need to choose a single best source model