SlowGuess/NAACL_2022_OCR huggingface dataset:
Guoqing Zheng
zheng@microsoft.com
Kai Shu
kshu@iit.edu
Building machine learning models for natural language understanding (NLU) tasks relies heavily on labeled data. Weak supervision has been proven valuable when large amount of labeled data is unavailable or expensive to obtain. Existing works studying weak supervision for NLU either mostly focus on a specific task or simulate weak supervision signals from ground-truth labels. It is thus hard to compare different approaches and evaluate the benefit of weak supervision without access to a unified and systematic benchmark with diverse tasks and real-world weak labeling rules. In this paper, we propose such a benchmark, named WALNUT<sup>1</sup>, to advocate and facilitate research on weak supervision for NLU. WALNUT consists of NLU tasks with different types, including document-level and token-level prediction tasks. WALNUT is the first semi-weakly supervised learning benchmark for NLU, where each task contains weak labels generated by multiple real-world weak sources, together with a small set of clean labels. We conduct baseline evaluations on WALNUT to systematically evaluate the effectiveness of various weak supervision methods and model architectures. Our results demonstrate the benefit of weak supervision for low-resource NLU tasks and highlight interesting patterns across tasks. We expect WALNUT to stimulate further research on methodologies to leverage weak supervision more effectively. The benchmark and code for baselines are available at aka.ms/walnut_benchmark.
To tackle natural language understanding (NLU) tasks via supervised learning, high-quality labeled examples are crucial. Recent advances on large pre-trained language models (Peters et al., 2018; Devlin et al., 2018; Radford et al., 2019) lead to impressive gains on NLU benchmarks, including
Giannis Karamanolakis
- 8 NLU tasks with weak rules
Doc-level classification
$\mathrm{O}$  Token-level classification
- 5 clean/weak splits per task
Few Labeled Data
psaandnS Aepaa-ianss
Weakly Labeled Data
Figure 1: WALNUT, a benchmark with 8 NLU tasks with real-world weak labeling rules. Each task in WALNUT comes with few labeled data and weakly labeled data for semi-weakly supervised learning.
S
GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019), at the assumption that large amount of labeled examples are available. For many real-world applications, however, it is expensive and time-consuming to manually obtain large-scale high-quality labels, while it is relatively easier to obtain auxiliary supervision signals, or weak supervision, as a viable source to boost model performance without expensive data annotation process.
Learning from weak supervision for NLU tasks is attracting increasing attention. Various types of weak supervision have been considered, such as knowledge bases (Mintz et al., 2009; Xu et al., 2013), keywords (Karamanolakis et al., 2019; Ren et al., 2020), regular expression patterns (Augenstein et al., 2016), and other metadata such as user interactions in social media (Shu et al., 2017). Also, inspired by recent advances from semi-supervised learning, semi-weakly supervised learning methods which leverage both a small set of clean labels and a larger set of weak supervision (Papandreou et al., 2015; Hendrycks et al., 2018; Shu et al., 2019; Mazzetto et al., 2021; Karamanolakis
<sup>1</sup>WALNUT: Semi-WeAkly supervised Learning for Natural language Understanding Testbed
Human Language Technologies, pages 873 - 899
et al., 2021; Maheshwari et al., 2021; Zheng et al., 2021) are emerging to further boost task performance. However, a unified and systematic evaluation benchmark supporting both weakly and semi-weakly supervised learning for NLU tasks is rather limited. On the one hand, many existing works only study specific NLU tasks with weak supervision, thus evaluations of proposed techniques leveraging weak supervision on a small set of tasks do not necessarily generalized onto other NLU tasks. On the other hand, some works rely on simulated weak supervision, such as weak labels corrupted from ground-truth labels (Hendrycks et al., 2018), while real-world weak supervision signals can be far more complex than simulated ones. Furthermore, existing weakly and semi-weakly supervised approaches are evaluated on different data with different metrics and weak supervision sources, making it difficult to understand and compare.
To better advocate and facilitate research on leveraging weak supervision for NLU, in this paper we propose WALNUT (Figure 1), a semi-weakly supervised learning benchmark of NLU tasks with real-world weak supervision signals. Following the tradition of existing benchmarks (e.g., GLUE), we propose to cover different types of NLU tasks and domains, including document-level classification tasks (e.g., sentiment analysis on online reviews, fake news detection on news articles), and token-level classification tasks (e.g., named entity recognition in news and biomedical documents). WALNUT provides few labeled and many weakly labeled examples (Figure 1) and encourages a consistent and robust evaluation of different techniques, as we will describe in Section 3.
In addition to the proposed benchmark, in Section 4.2 we shed light on the benefit of weak supervision for NLU tasks in a collective manner, by evaluating several representative weak and semiweak supervision methods for and several base models of various sizes (e.g., BiLSTM, BERT, RoBERTa), leading to more than 2,000 groups of experiments. Our large-scale analysis demonstrates that weak supervision is valuable for low-resource NLU tasks and that there is large room for performance improvement, thus motivating future research. Also, by computing the average performance across tasks and model architectures, we show surprising new findings. First, simple techniques for aggregating multiple weak labels (such as unweighted majority voting) achieve better per
formance than more complex weak supervision paradigms. Second, weak supervision has smaller benefit in larger base models such as RoBERTa, because larger pre-trained models can already achieve impressively high performance using just a few clean labeled data and no weakly labeled data at all. We identify several more challenges on leveraging weak supervision for NLU tasks and shed light on possible future work based on WALNUT.
The main contributions of this paper are: (1) We propose a new benchmark on semi-weakly supervised learning for NLU, which covers eight established annotated datasets and various text genres, dataset sizes, and degrees of task difficulty; (2) We conduct an exploratory analysis from different perspectives to demonstrate and analyze the results for several major existing weak supervision approaches across tasks; and (3) We discuss the benefits and provide insights for potential weak supervision studies for representative NLU tasks.
2.1 Weak Supervision for NLU
Document-level classification Existing works on weakly supervised learning for document-level classification attempt to correct the weak labels by incorporating a loss correction mechanism for text classification (Sukhbaatar et al., 2014; Patrini et al., 2017). Other works further assume access to a small set of clean labeled examples (Hendrycks et al., 2018; Ren et al., 2018; Varma and Ré, 2018; Shu et al., 2020b). Recent works also consider the scenario where weak signals are available from multiple sources (Ratner et al., 2017; Meng et al., 2018; Ren et al., 2020) to exploit the redundancy as well as the consistency in the labeling information. Despite the recent progress on weak supervision for text classification, there is no agreed upon benchmark that can guide future directions and development of NLU tasks in semi-weakly supervised setting.
Token-level classification Weak supervision has also been studied for token-level classification (sequence tagging) tasks, focusing on Named Entity Recognition (NER). One of the most common approaches is distant supervision (Mintz et al., 2009), which uses knowledge bases to heuristically annotate training data. Besides distant supervision, several weak supervision approaches have recently addressed NER by introducing various types of
labeling rules, for example based on keywords, lexicons, and regular expressions (Fries et al., 2017; Ratner et al., 2017; Shang et al., 2018; Safranchik et al., 2020; Lison et al., 2020; Li et al., 2021). WALNUT integrates existing weak rules into a unified representation and evaluation format.
2.2 NLU Benchmarks
Accompanying the emerging of large pre-trained language models, NLU benchmarks has been a focus for NLP research, including GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019). On such benchmarks, the major focus is put on obtaining best possible performance (He et al., 2020) under the full training setting, which assumes that a large quantity of manually labeled examples are available for all tasks. Few-shot NLU benchmarks exist (Schick and Schütze, 2021; Xu et al., 2021; Ye et al., 2021; Mukherjee et al., 2021), however these do not contain weak supervision. Though research in weak supervision in NLU has gained significant interest (Hendrycks et al., 2018; Shu et al., 2019; Zheng et al., 2021), most of these work either focus on a small set of tasks or simulate weak supervision signals from ground-truth labels, hindering its generalization ability to real-world NLU tasks. The lack of a unified test bed covering different NLU task types and data domains motivates us to construct such a benchmark to better understand and leverage semi-weakly supervised learning for NLU in this paper.
Different from existing work based on crowdsourcing (Hovy et al., 2013; Gokhale et al., 2014) to obtain noisy labels, we focus specifically on the semi-weakly supervised learning setting, where we collect tasks with weak labels obtained from human-written labeling rules. (Zhang et al., 2021) is concurrent work that also features weak supervision for various (not necessarily text-based) tasks and assumes a purely weakly supervised setting, i.e., no clean labeled data is available. In contrast, WALNUT focuses on NLU tasks under a more realistic semi-weakly supervised setting and, as we show in Section 3, a small amount of clean labeled data plays an important role in determining the benefit of weak supervision for a target task.
3 WALNUT
3.1 Benchmark Construction Principles
We first describe the design principles guiding the benchmark construction.
Task Selection Criterion We aim to create a testbed which covers a broad range of NLU tasks where real-world weak supervision signals are available. To this end, WALNUT includes eight English text understanding tasks from diverse domains, ranging from news articles, movie reviews, merchandise reviews, biomedical corpus, wikipedia documents, to tweets. The eight tasks are categorized evenly into two types, namely document classification and token classification (sequence labeling). It's worth noting that we didn't create any labeling rules ourselves to avoid bias, but rather opted with labeling rules which already exist and are extensively studied by previous research. Therefore, WALNUT does not include other NLU tasks, such as natural language inference and question answering, as we are not aware of previous research with human labeling rules for these tasks.
While many previous works studied weak supervision in a purely weakly supervised setting, recent advances in few-shot and semi-supervised learning suggest that a small set of cleanly labeled examples together with unlabeled examples greatly helps boosting the task performance. Though large scale labeled examples for a task is difficult to collect, we acknowledge that it's rather practical to collect a small set of labeled examples. In addition, recent methods leveraging weak supervision also demonstrate greater gains of combining a small set of labeled examples with large weakly labeled examples (Hendrycks et al., 2018; Shu et al., 2019; Zheng et al., 2021). Therefore, WALNUT is designed to emphasize the semi-weakly supervised learning setting. Specifically, each dataset contains both a small number of clean labeled instances and a large number of weakly-labeled instances. Each weakly-labeled instance comes with multiple weak labels (assigned by multiple rules) and a single aggregated weak label derived from weak rules. Note that this way WALNUT can be naturally used to support the conventional weakly supervised setting by ignoring the provided clean labels.
Consistent and Robust Evaluation To address discrepancies in evaluation protocols from existing research on weak supervision and to better account for the small set of clean examples per task, WALNUT is constructed to promote systematic and robust evaluations across all eight tasks. Specif
Table 1: Statistics of the eight document- and token-level tasks in WALNUT. See Section 3.2 for details.
Dataset	AGNews	IMDB	Yelp	GossipCop	CoNLL	NCBI	WikiGold	LaptopReview
Label granularity	doc.	doc.	doc.	doc.	token	token	token	token
Task	topic	sentiment	sentiment	fake	NER	NER	NER	NER
Domain	news	movies	restaurants	news	news	biomed	web	tech
# Classes	4	2	2	2	9	3	9	3
# Train-clean (|Dc|)	80	40	40	40	180	60	360	150
# Train-weak (|Dw|)	4,439	16,626	10,954	6,462	13,861	532	995	2,286
# Dev	12,000	2,500	3,800	1,430	3,250	99	169	609
# Test	12,000	2,500	3,800	957	3,453	99	170	800
# Weak rules	9	8	8	3	50	12	16	12
ically, for each task, we first determine the number of clean examples to sample with pilot experiments (with the rest treated as weakly labeled examples by applying the corresponding weak labeling rules), such that the weakly supervised examples can be still helpful with the small clean examples present (typically 20-50 per class; see Sec. 3.3 for details); second, to consider sampling uncertainty, we repeat the sampling process for the desired number of clean examples 5 times and provide all 5 splits in WALNUT. Methods on WALNUT are expected to be using all 5 pre-computed splits and reporting the mean and variance of its performance.
To summarize, WALNUT can facilitate research on weakly- and semi-weakly supervised learning by offering the following:
Eight NLU tasks from diverse domains;
- For each task, five pairs of clean and weakly labeled samples for robust evaluation;
- For each individual weakly labeled example, all weak labels from multiple rules and a single aggregated weak label.
3.2 Task Categories
Here, we describe the eight tasks in WALNUT (Table 1), grouped into four document-level classification tasks (Section 3.2.1) and four token-level classification tasks (Section 3.2.2).
3.2.1 Document-level Classification
The goal of document-level classification tasks is to classify a sequence of tokens  $x_{1}, \ldots, x_{N}$  to a class  $c \in C$ , where  $C$  is a pre-defined set of classes. We consider binary and multi-class classification problems from different application domains such as sentiment classification (Zhang et al., 2015), fake news detection (Shu et al., 2020c), and topic classification (Zhang et al., 2015). Concretely, we include the following widely-used document-level
text classification datasets: AGNews (Zhang et al., 2015), Yelp (Zhang et al., 2015), IMDB (Maas et al., 2011) and GossipCop (Shu et al., 2020a).
For Yelp, IMDB, and AGNews, the weak rules are derived from the text using keyword-based heuristics, third-party tools as detailed in (Ren et al., 2020). For GossipCop, the weak labeling rules are derived from social context information accompanying the news articles, including related users' social engagements on the news items (e.g., user comments in Twitter). For example, a weak labeling rule for fake news can be "If a news piece has a standard deviation of user sentiment scores greater than a threshold, then the news is weakly labeled as fake news." (Shu et al., 2020c).
3.2.2 Token-level Classification
The goal of token-level classification tasks is to classify a sequence of tokens  $x_{1},\ldots ,x_{N}$  to a sequence of tags  $y_{1},\ldots ,y_{N}\in C^{\prime}$ , where  $C^\prime$  is a pre-defined set of tag classes (e.g., person or organization). As one of the most common tokenlevel classification tasks, Named Entity Recognition (NER) deals with recognizing categories of named entities (e.g., person, organization, location) and is important in several NLP pipelines, including information extraction and question answering.
We include in WALNUT the following four NER datasets from different domains, for which weak rules are available: CoNLL (Sang and De Meulder, 2003), the NCBI Disease corpus (Doogan et al., 2014), WikiGold (Balasuriya et al., 2009) and the LaptopReview corpus (Pontiki et al., 2016) from the SemEval 2014 Challenge. For the CoNLL and WikiGold dataset, we use weak rules provided by (Lison et al., 2020). For the NCBI and LaptopReview dataset, we use weak rules provided by (Safranchik et al., 2020).
(a) AG News
(b)IMDB
(d) Gossipcop
(e) CoNLL
(f) NCBI
(g) WikiGold
Figure 2: F1 score by varying (in the x-axis) the number of clean instances per class considered in the clean training set  $(D_C)$ . The importance of weak supervision is more evident for settings with smaller numbers of instances, where the gap in performance between the "Clean" approach and "Clean+Weak" approach is larger. For a robust evaluation across tasks, WALNUT provides five clean/weak splits per task. See Section 3.3 for details.
(h) LaptopReview
3.3 Dataset Pre-Processing
To construct a semi-weakly supervised learning setting, we split the training dataset for each task into a small subset with clean labels  $(D_C)$  and a large subset with weak labels  $(D_W)$ . For robust evaluation, we create five different clean/weak train splits as we noticed that the model performances may vary with different clean train instances. The validation/test sets are always the same across splits.
Because of different dataset characteristics (e.g., differences in number of classes, difficulty), we choose the size for  $D_{C}$  per dataset via pilot studies. (After having selected the instances for the  $D_{C}$ , we consider the remaining instances as part of the  $D_{W}$  split.) We defined the size of  $D_{C}$  such that we demonstrate the benefits of weak supervision and at the same time leave substantial room for improvement in future research. To this end, we compare the performances of the same base classification model (e.g., BiLSTM), trained using only  $D_{C}$  ("Clean" approach) v.s. using both  $D_{C}$  and  $D_{W}$  ("Clean+Weak" approach). As shown in Figure 2, for each dataset, we choose a small size of  $D_{C}$ , such that the "Clean+Weak" approach has a substantially higher F1 score than the "Clean" approach and at the same time the "Clean" approach has no trivial F1 score.
The statistics of the pre-processed datasets included in WALNUT are shown in Table 1.
4 Baseline Evaluation in WALNUT
In this section, we describe the baselines and evaluation procedure (Section 4.1), and discuss evalua
tion results in WALNUT (Section 4.2). Our results highlight the value of weak supervision, important differences across different baselines, and the potential utility of WALNUT for future research on weak supervision.
4.1 Baselines and Evaluation Procedure
We evaluate several baseline approaches in WALNUT by considering different base models (text encoders) and different (semi-)weakly supervised learning methods to train the base model.
Encoder Models To encode input text, we experiment with various text encoders, ranging from shallow LSTMs to large pre-trained transformer-based encoders (Vaswani et al., 2017). In particular, we consider a series of models with increasing model size: Bi-directional LSTM with Glove embeddings (Pennington et al., 2014), DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), BERT-large, and RoBERTa-large. For each text encoder, a classification head is placed on top of the encoder to perform the task. For more details on the base model configurations see Appendix A.1.
Learning Methods Given the semi-weakly supervised setting in WALNUT, we evaluate eight supervision approaches in the following categories:
- Learning from clean labeled examples only. The model is trained on only the small amount of available clean examples  $D_{C}$ , a naive baseline method leveraging no weak supervision, which we denote as  $\mathbf{C}$ .
- Learning from weakly labeled examples only. The model is trained on all weakly labeled examples  $D_W$ . To produce a single weak label from the multiple labeling rules for training, we aggregate the rules via two methods: majority voting (denoted by  $\mathbf{W}$ ) and Snorkel (Ratner et al., 2017) (denoted by Snorkel).
- Learning from both clean and weakly labeled examples. The model is trained with both  $D_C$  and  $D_W$  in a weakly-supervised setting. The first two baselines in this category is simply concatenating  $D_C$  and the aggregated weak labels (from either  $\mathbf{W}$  and Snorkel), and the model is trained on the combination. We denote these two as  $\mathbf{C} + \mathbf{W}$  and  $\mathbf{C} + \mathbf{Snorkel}$ , respectively. We also test three recent semiweakly supervised learning methods which proposed better ways to leverage both  $D_C$  and  $D_W$ : GLC which is a loss correction approach (Hendrycks et al., 2018), MetaWN which is a meta-learning approach to learn the importance of weakly labeled examples (Shu et al., 2019; Ren et al., 2018) and MLC, a meta-learning approach to learn to correct the weak labels (Zheng et al., 2021).
To establish an estimate of the ceiling performance on WALNUT, for each task we also train with all clean training examples in the original dataset (denoted by Full Clean).
Experimental Procedure For a robust evaluation, we repeat each experiment five times on the five splits of  $D_C$  and  $D_W$  (clean and weak examples for each task; see Section 3.3), and report the average scores and the standard deviation across the five runs. In WALNUT, we report the average micro-average F1 score on the test set. Datasets and code for WALNUT are publicly available at aka.ms/walnut_benchmark.
4.2 Experimental Results and Analysis
Table 2 shows the main evaluation results on WALNUT. Rows correspond to supervision methods for the base model, columns correspond to tasks, and each block corresponds to a different base model. Unless explicitly mentioned, in the rest of this section we will compare approaches based on their average performance across tasks (rightmost column in Table 2).
As expected, training with Full Clean achieves the highest F1 score, corresponding to the high-resource setting where all clean labeled data are available. Such method is not directly comparable to the rest of the methods but serves as an estimate of the ceiling performance for WALNUT. Training with only limited clean examples achieves the lowest overall F1 score: in the low-resource setting, which is the main focus in WALNUT, using just the available clean subset  $(D_C)$  is not effective.
Weak supervision is valuable for low-resource NLU. "W" and "Snorkel" achieve better F1 scores than "C" for many base models: even using only weakly-labeled data in  $D_W$  is more effective than using just  $D_C$ , thus demonstrating that simple weak supervision approaches can be useful in the low-resource setting. Approaches such as "C+W" and "C+Snorkel" lead to further improvements compared to "C" and "Snorkel", thus highlighting that even simple approaches for integrating clean and weak labeled data (here by concatenating  $D_C$  and  $D_W$ ) are more effective than considering each separately.
There is no clear winner in WALNUT. Our results in Table 4.2 indicate that the performance of weak supervision techniques varies substantially across tasks. Therefore, it is important to evaluate such techniques in a diverse set of tasks to achieve a fair comparison and more complete picture of their performance. The performance of various techniques also varies across different splits (See Table 14 in Appendix for variances of all experiments). Interestingly, "C+W" and "C+Snorkel" sometimes perform better than more complicated approaches, such as GLC, MetaWN and MLC.