SlowGuess/NAACL_2021_OCR huggingface dataset:
Giannis Karamanolakis\* Subhabrata Mukherjee Guoqing Zheng' Ahmed Hassan Awadallah Columbia University, New York Microsoft Research gkaraman@cs.columbia.edu {submukhe, zheng, hassanam} @microsoft.com
State-of-the-art deep neural networks require large-scale labeled training data that is often expensive to obtain or not available for many tasks. Weak supervision in the form of domain-specific rules has been shown to be useful in such settings to automatically generate weakly labeled training data. However, learning with weak rules is challenging due to their inherent heuristic and noisy nature. An additional challenge is rule coverage and overlap, where prior work on weak supervision only considers instances that are covered by weak rules, thus leaving valuable unlabeled data behind.
In this work, we develop a weak supervision framework (ASTRA $^1$ ) that leverages all the available data for a given task. To this end, we leverage task-specific unlabeled data through self-training with a model (student) that considers contextualized representations and predicts pseudo-labels for instances that may not be covered by weak rules. We further develop a rule attention network (teacher) that learns how to aggregate student pseudo-labels with weak rule labels, conditioned on their fidelity and the underlying context of an instance. Finally, we construct a semi-supervised learning objective for end-to-end training with unlabeled data, domain-specific rules, and a small amount of labeled data. Extensive experiments on six benchmark datasets for text classification demonstrate the effectiveness of our approach with significant improvements over state-of-the-art baselines.
The success of state-of-the-art neural networks crucially hinges on the availability of large amounts of annotated training data. While recent advances on language model pre-training (Peters et al., 2018;
*Most of the work was done while the first author was an intern at Microsoft Research.
1ASTRA: weAkly-supervised Self-TRAining. Our code is publicly available at https://github.com/microsoft/ASTRA.
Figure 1: Our weak supervision framework, ASTRA, leverages domain-specific rules, a large amount of (task-specific) unlabeled data, and a small amount of labeled data via iterative self-training.
Devlin et al., 2019; Radford et al., 2019) reduce the annotation bottleneck, they still require large amounts of labeled data for obtaining state-of-the-art performances on downstream tasks. However, it is prohibitively expensive to obtain large-scale labeled data for every new task, therefore posing a significant challenge for supervised learning.
In order to mitigate labeled data scarcity, recent works have tapped into weak or noisy sources of supervision, such as regular expression patterns (Augenstein et al., 2016), class-indicative keywords (Ren et al., 2018b; Karamanolakis et al., 2019), alignment rules over existing knowledge bases (Mintz et al., 2009; Xu et al., 2013) or heuristic labeling functions (Ratner et al., 2017; Bach et al., 2019; Badene et al., 2019; Awasthi et al., 2020). These different types of sources can be used as weak rules for heuristically annotating large amounts of unlabeled data. For instance, consider the question type classification task from the TREC dataset with regular expression patterns such as: label all questions containing the token "when" as numeric (e.g., "When was Shakespeare born?"). Approaches relying on such weak rules typically suffer from the following challenges. (i) Noise. Rules by their heuristic nature rely on shallow patterns and may predict wrong labels for many instances. For example, the question "When would
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 845-863 June 6-11, 2021. ©2021 Association for Computational Linguistics
such a rule be justified?" refers to circumstances rather than numeric expressions. (ii) Coverage. Rules generally have a low coverage as they assign labels to only specific subsets of instances. (iii) Conflicts. Different rules may generate conflicting predictions for the same instance, making it challenging to train a robust classifier.
To address the challenges with conflicting and noisy rules, existing approaches learn weights indicating how much to trust individual rules. In the absence of large-scale manual annotations, the rule weights are usually learned via mutual agreement and disagreement of rules over unlabeled data (Ratner et al., 2017; Platanios et al., 2017; Sachan et al., 2018; Bach et al., 2019; Ratner et al., 2019; Awasthi et al., 2020). For instance, such techniques would up-weight rules that agree with each other (as they are more likely to be correct), and down-weight such rules otherwise. An important drawback of these approaches is low coverage since rules assign weak labels to only a subset of the data, thus leading to low rule overlap to compute rule agreement. For instance, in our experiments on six real-world datasets, we observe that  $66\%$  of the instances are covered by fewer than 2 rules and  $40\%$  of the instances are not covered by any rule at all. Rule sparsity limits the effectiveness of previous approaches, thus leading to strong assumptions, such as, that each rule has the same weight across all instances (Ratner et al., 2017; Bach et al., 2019; Ratner et al., 2019), or that additional supervision is available in the form of labeled "exemplars" used to create such rules in the first place (Awasthi et al., 2020). Most importantly, all these works ignore (as a data pre-processing step) unlabeled instances that are not covered by any of the rules, thus leaving potentially valuable data behind.
Overview of our method. In this work, we present a weak supervision framework, namely ASTRA, that considers all task-specific unlabeled instances and domain-specific rules without strong assumptions about the nature or source of the rules. ASTRA makes effective use of a small amount of labeled data, lots of task-specific unlabeled data, and domain-specific rules through iterative teacher-student co-training (see Figure 1). A student model based on contextualized representations provides pseudo-labels for all instances, thereby, allowing us to leverage all unlabeled data including instances that are not covered by any heuristic rules. To deal with the noisy nature of heuristic rules and pseudo
labels from the student, we develop a rule attention (teacher) network that learns to predict the fidelity of these rules and pseudo-labels conditioned on the context of the instances to which they apply. We develop a semi-supervised learning objective based on minimum entropy regularization to learn all of the above tasks jointly without the requirement of additional rule-exemplar supervision.
- We propose an iterative self-training mechanism for training deep neural networks with weak supervision by making effective use of task-specific unlabeled data and domain-specific heuristic rules. The self-trained student model predictions augment the weak supervision framework with instances that are not covered by rules.
- We propose a rule attention teacher network (RAN) for combining multiple rules and student model predictions with instance-specific weights conditioned on the corresponding contexts. Furthermore, we construct a semi-supervised learning objective for training RAN without strong assumptions about the structure or nature of the weak rules.
- We demonstrate the effectiveness of our approach on several benchmark datasets for text classification where our method significantly outperforms state-of-the-art weak supervision methods.
2 Self-Training with Weak Supervision
We now present our approach, ASTRA, that leverages a small amount of labeled data, a large amount of unlabeled data, and domain-specific heuristic rules. Our architecture has two main components: the base student model (Section 2.1) and the rule attention teacher network (Section 2.2), which are iteratively co-trained in a self-training framework.
Formally, let  $\mathcal{X}$  denote the instance space and  $\mathcal{Y} = \{1,\dots ,K\}$  denote the label space for a  $K$  class classification task. We consider a small set of manually-labeled examples  $D_{L} = \{(x_{l},y_{l})\}$ , where  $x_{l}\in \mathcal{X}$  and  $y_{l}\in \mathcal{V}$  and a large set of unlabeled examples  $D_U = \{x_i\}$ . We also consider a set of pre-defined heuristic rules  $R = \{r^j\}$ , where each rule  $r^j$  has the general form of a labeling function that considers as input an instance  $x_{i}\in \mathcal{X}$  (and potentially additional side information), and
Figure 2: Our ASTRA framework for self-training with weak supervision.
either assigns a weak label  $q_i^j \in \{0,1\}^K$  (one-hot encoding) or does not apply, i.e., does not assign a label for  $x_i$ . Our goal is to leverage  $D_L$ ,  $D_U$ , and  $R$  to train a classifier that, given an unseen test instance  $x' \in \mathcal{X}$ , predicts a label  $y' \in \mathcal{Y}$ . In the rest of this section, we present our ASTRA framework for addressing this problem.
2.1 Base Student Model
Our self-training framework starts with a base model trained on the available small labeled set  $D_{L}$ . The model is then applied to unlabeled data  $D_{U}$  to obtain pseudo-labeled instances. In classic self-training (Riloff, 1996; Nigam and Ghani, 2000), the student model's pseudo-labeled instances are directly used to augment the training dataset and iteratively re-train the student. In our setting, we augment the self-training process with weak labels drawn from our teacher model that also considers rules in  $R$  (described in the next section). The overall self-training process can be formulated as:
$$
\begin{array}{l} \min  _ {\theta} \mathbb {E} _ {x _ {l}, y _ {l} \in D _ {L}} [ - \log p _ {\theta} (y _ {l} \mid x _ {l}) ] + \\ \lambda \mathbb {E} _ {x \in D _ {U}} \mathbb {E} _ {y \sim q _ {\phi^ {*}} (y | x)} [ - \log p _ {\theta} (y \mid x) ] \tag {1} \\ \end{array}
$$
where,  $p_{\theta}(y|x)$  is the conditional distribution under student's parameters  $\theta$ ;  $\lambda \in \mathbb{R}$  is a hyper-parameter controlling the relative importance of the two terms; and  $q_{\phi^*}(y|x)$  is the conditional distribution under the teacher's parameters  $\phi^*$  from the last iteration that is fixed in the current iteration.
2.2 Rule Attention Teacher Network (RAN)
Our Rule Attention Teacher Network (RAN) aggregates multiple weak sources of supervision with trainable weights and computes a soft weak label
$q_{i}$  for an unlabeled instance  $x_{i}$ . One of the potential drawbacks of relying only on heuristic rules is that a lot of data get left behind. Heuristic rules by nature (e.g., regular expression patterns, keywords) apply to only a subset of the data. Therefore, a substantial number of instances are not covered by any rules and thus are not considered in prior weakly supervised learning approaches (Ratner et al., 2017; Awasthi et al., 2020). To address this challenge and leverage contextual information from all available task-specific unlabeled data, we leverage the corresponding pseudo-labels predicted by the base student model (from Section 2.1). To this end, we apply the student to the unlabeled data  $x \in D_U$  and obtain pseudo-label predictions as  $p_{\theta}(y|x)$ . These predictions are used to augment the set of already available weak rule labels to increase rule coverage.
Let  $R_{i} \subset R$  be the set of all heuristic rules that apply to instance  $x_{i}$ . The objective of RAN is to aggregate the weak labels predicted by all rules  $r^{j} \in R_{i}$  and the student pseudo-label  $p_{\theta}(y|x_{i})$  to compute a soft label  $q_{i}$  for every instance  $x_{i}$  from the unlabeled set  $D_{U}$ . In other words, RAN considers the student as an additional source of weak rule. Aggregating all rule labels into a single label  $q_{i}$  via simple majority voting (i.e., predicting the label assigned by the majority of rules) may not be effective as it treats all rules equally, while in practice, certain rules are more accurate than others.
RAN predicts pseudo-labels  $q_{i}$  by aggregating rules with trainable weights  $a_{i}^{(\cdot)} \in [0,1]$  that capture their fidelity towards an instance  $x_{i}$  as:
$$
q _ {i} = \frac {1}{Z _ {i}} \left(\sum_ {j: r ^ {j} \in R _ {i}} a _ {i} ^ {j} q _ {i} ^ {j} + a _ {i} ^ {S} p _ {\theta} (y | x _ {i}) + a _ {i} ^ {u} u\right), \tag {2}
$$
where  $a_{i}^{j}$  and  $a_{i}^{S}$  are the fidelity weights for the
heuristic rule labels  $q_{i}^{j}$  and the student assigned pseudo-label  $p_{\theta}(y|x_{i})$  for an instance  $x_{i}$ , respectively;  $u$  is a uniform rule distribution that assigns equal probabilities for all the  $K$  classes as  $u = [\frac{1}{K},\dots ,\frac{1}{K}]$ ;  $a_{i}^{u}$  is the weight assigned to the "uniform rule" for  $x_{i}$ , which is computed as a function of the rest of the rule weights:  $a_{i}^{u} = (|R_{i}| + 1 - \sum_{j:r^{j}\in R_{i}}a_{i}^{j} - a_{i}^{S})$ ; and  $Z_{i}$  is a normalization coefficient to ensure that  $q_{i}$  is a valid probability distribution.  $u$  acts as a uniform smoothing factor that prevents overfitting for sparse settings, for instance, when a single weak rule applies to an instance.
According to Eq. (2), a rule  $r^j$  with higher fidelity weight  $a_i^j$  contributes more to the computation of  $q_i$ . If  $a_i^j = 1 \forall r^j \in \{R_i \cup p_\theta\}$ , then RAN reduces to majority voting. If  $a_i^j = 0 \forall r^j \in \{R_i \cup p_\theta\}$ , then RAN ignores all rules and predicts  $q_i = u$ . Note the distinction of our setting to recent works like Snorkel (Ratner et al., 2017), that learns global rule-weights  $a_i^j = a^j \forall x_i$  by ignoring the instance-specific rule fidelity. Our proposed approach is flexible but at the same time challenging as we do not assume prior knowledge of the internal structure of the labeling functions  $r^j \in R$ .
In order to effectively compute rule fidelities, RAN considers instance embeddings that capture the context of instances beyond the shallow patterns considered by rules. In particular, we model the weight  $a_{i}^{j}$  of rule  $r^j$  as a function of the context of the instance  $x_{i}$  and  $r^j$  through an attention-based mechanism. Consider  $h_i \in \mathbb{R}^{d'}$  to be the hidden state representation of  $x_{i}$  from the base student model. Also, consider the (trainable) embedding of each rule  $r^j$  as  $e_j = g(r^j) \in \mathbb{R}^d$ . We use  $e_j$  as a query vector with sigmoid attention to compute instance-specific rule attention weights as:
$$
a _ {i} ^ {j} = \sigma \left(f \left(h _ {i}\right) ^ {T} \cdot e _ {j}\right) \in [ 0, 1 ], \tag {3}
$$
where  $f$  is a multi-layer perceptron that projects  $h_i$  to  $\mathbb{R}^d$  and  $\sigma(\cdot)$  is the sigmoid function. Rule embedding allows us to exploit the similarity between different rules in terms of instances to which they apply, and further leverage their semantics for modeling agreement. RAN computes the student's weight  $a_i^S$  using the same procedure as for computing the rule weights  $a_i^j$ .
Note that the rule predictions  $q_{i}^{j}$  are considered fixed, while we estimate their attention weights. The above coupling between rules and instances via their corresponding embeddings  $e_{j}$  and  $h_{i}$  al
(a) Rule predictions disagree.
Figure 3: Variation in unsupervised entropy loss with instance-specific rule predictions and attention weights encouraging rule agreement. Consider this illustration with two rules for a given instance. When rule predictions disagree ( $q^1 \neq q^2$ ), minimum loss is achieved for attention weights  $a^1 = 0$ ,  $a^2 = 1$  or  $a^1 = 1$ ,  $a^2 = 0$ . When rule predictions agree ( $q^1 = q^2$ ), minimum loss is achieved for attention weights  $a^1 = a^2 = 1$ . For instances covered by three rules, if  $q^1 = q^2 \neq q^3$ , the minimum loss is achieved for  $a^1 = a^2 = 1$  and  $a^3 = 0$ .
(b) Rule predictions agree.
ows us to obtain representations where similar rules apply to similar contexts, and model their agreements via the attention weights  $a_{i}^{j}$ . To this end, the trainable parameters of RAN ( $f$  and  $g$ ) are shared across all rules and instances. Next, we describe how to train RAN.
2.3 Semi-Supervised Learning of ASTRA
Learning to predict instance-specific weights  $a_{i}^{(\cdot)}$  for the weak sources (including rules and student pseudo-labels) is challenging due to the absence of any explicit knowledge about the source quality and limited amount of labeled training data. We thus treat the weights  $a_{i}^{(\cdot)}$  as latent variables and propose a semi-supervised objective for training RAN with supervision on the coarser level of  $q_{i}$ :
$$
\mathcal {L} ^ {R A N} = - \sum_ {\left(x _ {i}, y _ {i}\right) \in D _ {L}} y _ {i} \log q _ {i} - \sum_ {x _ {i} \in D _ {U}} q _ {i} \log q _ {i}. \tag {4}
$$
Given task-specific labeled data  $D_{L}$ , the first term in Eq. (4) minimizes the cross-entropy loss between the teacher's label  $q_{i}$  and the corresponding clean label  $y_{i}$  for the instance  $x_{i}$ . This term penalizes weak sources that assign labels  $q_{i}^{(\cdot)}$  that contradict with the ground-truth label  $y_{i}$  by assigning a low instance-specific fidelity weight  $a_{i}^{(\cdot)}$ .
The second term in Eq. (4) minimizes the entropy of the aggregated pseudo-label  $q_{i}$  on unlabeled data  $D_{U}$ . Minimum entropy regularization is effective in settings with small amounts of labeled
TREC	SMS	YouTube	CENSUS	MIT-R	Spouse
Labeled Training Data (|DL|)	68	69	100	83	1842	100
Unlabeled Training Data (|DU|)	5K	5K	2K	10K	65K	22K
Test Data	500	500	250	16K	14K	3K
#Classes	6	2	2	2	9	2
#Rules	68	73	10	83	15	9
Rule Accuracy (Majority Voting)	60.9%	48.4%	82.2%	80.1%	40.9%	44.2%
Rule Coverage (instances in DU covered by ≥ 1 rule)	95%	40%	87%	100%	14%	25%
Rule Overlap (instances in DU covered by ≥ 2 rules)	46%	9%	48%	94%	1%	8%
Algorithm 1 Self-training with Weak Supervision
Input: Small amount of labeled data  $D_{L}$ ; task-specific unlabeled data  $D_{U}$ ; weak rules  $R$
Outputs: Student  $p_{\theta}^{*}(\cdot)$ , RAN Teacher  $q_{\phi}^{*}(\cdot)$
1: Train student  $p_{\theta}(\cdot)$  using  $D_{L}$
2: Repeat until convergence:
2.1: Train teacher  $q_{\phi}(\cdot)$  using  $D_L$ ,  $D_U$  through Eq. (2) and (4)
2.2: Apply  $q_{\phi}(y \mid x, R, p_{\theta})$  to  $x \in D_U$  to obtain pseudo-labeled data:  $D_{RAN} = \{(x_i, q_i)\}_{x_i \in D_U}$  through Eq. (2)
2.3: Train  $p_{\theta}(\cdot)$  using  $D_L$ ,  $D_{RAN}$  through Eq. (1)
data by leveraging unlabeled data (Grandvalet and Bengio, 2005), and is highly beneficial in our setting because it encourages RAN to predict weights that maximize rule agreement. Since the teacher label  $q_{i}$  is obtained by aggregating weak labels  $q_{i}^{(\cdot)}$ , entropy minimization encourages RAN to predict higher instance-specific weights  $a_{i}^{(\cdot)}$  to sources that agree in their labels for  $x_{i}$ , and lower weights when there are disagreements between weak sources – aggregated across all the unlabeled instances.
Figure 3 plots the minimum entropy loss over unlabeled data over two scenarios where two rules agree or disagree with each other for a given instance. The optimal instance-specific fidelity weights  $a_{i}^{(\cdot)}$  are 1 when rules agree with each other, thereby, assigning credits to both rules, and only one of them when they disagree. We use this unsupervised entropy loss in conjunction with cross-entropy loss over labeled data to ensure grounding.
End-to-end Learning: Algorithm 1 presents an overview of our learning mechanism. We first use the small amount of labeled data to train a base student model that generates pseudo-labels and
augments heuristic rules over unlabeled data. Our RAN network computes fidelity weights to combine these different weak labels via minimum entropy regularization to obtain an aggregated pseudolabel for every unlabeled instance. This is used to re-train the student model with the above student-teacher training repeated till convergence.
Datasets. We evaluate our framework on the following six benchmark datasets for weak supervision from Ratner et al. (2017) and Awasthi et al. (2020). (1) Question classification from TREC-6 into 6 categories (Abbreviation, Entity, Description, Human, Location, Numeric-value); (2) Spam classification of SMS messages; (3) Spam classification of YouTube comments; (4) Income classification on the CENSUS dataset on whether a person earns more than \(50K or not; (5) Slot-filling in sentences on restaurant search queries in the MIT-R dataset: each token is classified into 9 classes (Location, Hours, Amenity, Price, Cuisine, Dish, Restaurant Name, Rating, Other); (6) Relation classification in the Spouse dataset, whether pairs of people mentioned in a sentence are/were married or not.
Table 1 shows the dataset statistics along with the amount of labeled, unlabeled data and domain-specific rules for each dataset. For a fair comparison, we use exactly the same set of rules as in the previous work for the benchmark datasets. These rules include regular expression patterns, lexicons, and knowledge bases for weak supervision. Most of these rules were constructed manually, except for the CENSUS dataset, where rules have been automatically extracted with a coverage of  $100\%$ .
On average across all the datasets,  $66\%$  of the instances are covered by fewer than 2 rules, whereas  $40\%$  are not covered by any rule at all - demonstrating the sparsity in our setting. We also report the accuracy of the rules in terms of majority voting
Method	Learning to Weight	Unlabeled (no rules)
Rules	Instances
Majority	-	-	-
Snorkel (Ratner et al., 2017)	✓	-	-
PosteriorReg (Hu et al., 2016)	✓	-	-
L2R (Ren et al., 2018a)	-	✓	-
ImplyLoss (Awasthi et al., 2020)	✓	✓	-
Self-train	-	-	✓
ASTRA	✓	✓	✓
Table 2: ASTRA learns rule-specific and instance-specific attention weights and leverages task-specific unlabeled data where no rules apply.
on the task-specific unlabeled datasets. Additional details on the dataset and examples of rules are presented in the Appendix.
Evaluation. We train ASTRA five times for five different random splits of the labeled training data and evaluate on held-out test data. We report the average performance as well as the standard deviation across multiple runs. We report the same evaluation metrics as used in prior works (Ratner et al., 2017; Awasthi et al., 2020) for a fair comparison.
Model configuration. Our student model consists of embeddings from pre-trained language models like ELMO (Peters et al., 2018) or BERT (Devlin et al., 2019) for generating contextualized representations for an instance, followed by a softmax classification layer. The RAN teacher model considers a rule embedding layer and a multilayer perceptron for mapping the contextualized representation for an instance to the rule embedding space. Refer to the Appendix for more details.
Baselines. We compare our method with the following methods: (a) Majority predicts the majority vote of the rules with ties resolved by predicting a random class. (b) LabeledOnly trains classifiers using only labeled data (fully supervised baseline). (c) Self-train (Nigam and Ghani, 2000; Lee, 2013) leverages both labeled and unlabeled data for iterative self-training on pseudolabeled predictions over task-specific unlabeled data. This baseline ignores domain-specific rules. (e) Snorkel+Labeled (Ratner et al., 2017) trains classifiers using weakly-labeled data with a generative model. The model is trained on unlabeled data for computing rule weights in an unsupervised fashion, and learns a single weight per rule across all instances. It is further fine-tuned on labeled data. (f)  $L2R$  (Ren et al., 2018b) learns to re-weight noisy or weak labels from domain-specific rules via meta-learning. It learns instance-specific but