SlowGuess/NAACL_2021_OCR huggingface dataset:
We now describe implementation details for each component in ASTRA: our base student model and our rule attention teacher network. Table 8 shows our hyperparameter search configuration. We choose optimal hyperparameters by manual tuning based on the development performances. Table 9 shows the hyperparameters and model architecture details for each dataset. For a fair comparison, we use the same architectures as previous approaches but we expect further improvements by exploring different architectures.
Base Student Model Our student model consists of an instance embedding layer (e.g., ELMO (Peters et al., 2018), BERT (Devlin et al., 2019), logistic regression), a multilayer perceptron with two hidden layers, and a softmax classification layer for predicting labels.
Rule Attention Teacher Network Our RAN teacher model consists of a 128-dimensional rule embedding layer, a multilayer perceptron for mapping the contextualized representation for an instance to the rule embedding space, and a sigmoid attention layer.
Iterative Teacher-Student Training At each iteration, we train the RAN teacher on unlabeled data and fine-tune on clean labeled data. Also at each iteration, we train the student on pseudolabeled teacher data and fine-tune on clean labeled data. We consider a maximum number of 25 self-training iterations (with early stopping of patience 3 epochs) and keep the models' performances for the iteration corresponding to the highest validation performance.
We evaluate our framework on the following six benchmark datasets for weak supervision from Ratner et al. (2017) and Awasthi et al.  $(2020)^{2}$ . All datasets are in English. Table 11 shows detailed
dataset statistics. We consider the same test sets with previous work. For a robust evaluation of our model's performance, we split each dataset into five random train/validation/unlabeled splits and report the average performance and standard deviation across runs. For a fair comparison, we use the same splits and evaluation procedure across all methods and baselines.
TREC: Question classification from TREC-6 into 6 categories: Abbreviation, Entity, Description, Human, Location, Numeric-value. Table 12 reports a sample of regular expression rules out of the 68 rules used in the TREC dataset. TREC has 13 keyword-based (coverage=62%) and 55 regular expression-based (coverage=57%) rules.
SMS: Binary Spam vs. Not Spam classification of SMS messages. SMS has 16 keyword-based (coverage  $= 4\%$ ) and 57 regular expression-based (coverage  $= 38\%$ ) rules.
YouTube: Binary Spam vs. Not Spam classification of YouTube comments.3 YouTube has 5 keyword-based (coverage  $= 48\%$ ), 1 regular expression-based (coverage  $= 23\%$ ), 1 length-based (coverage  $= 23\%$ ), and 3 classifier-based (coverage  $= 46\%$ ) rules.
CENSUS: Binary income classification on the UCI CENSUS dataset on whether a person earns more than \(50K or not. This is a non-textual dataset and is considered to evaluate the performance of our approach under the low sparsity setting, since the 83 rules are automatically extracted and have a coverage of \(100\%\).
MIT-R: Slot-filling in sentences on restaurant search queries in the MIT-R dataset: each token is classified into 9 classes (Location, Hours, Amenity, Price, Cuisine, Dish, Restaurant Name, Rating, Other). MIT-R has 5 keyword-based (coverage=6%) and 10 regular expression-based (coverage=10%) rules.
Spouse: Relation classification in the Spouses dataset<sup>4</sup>, whether pairs of people mentioned in a sentence are/were married or not. Spouse has 6 keyword-based (coverage=23%), 1 heuristic-based
3https://archive.ics.uci.edu/ml/ machine-learning-databases/00380/ YouTube-Spam-Collection-v1.zip 4https://www.dropbox.com/s/ jmrvyaqew4zp9cy/spouse_data.zip
$^{2}$ https://github.com/awasthiabhijeet/Learning-From-Rules
(coverage  $= 4\%$ ), and 2 distant supervision-based (coverage  $= 0.2\%$ ) rules.
A.3 Experimental Result Details
We now discuss detailed results on each dataset. To be consistent with previous work, we report accuracy scores for the TREC, Youtube, and CENSUS dataset and macro-average F1 scores for the SMS, Spouse, and MIT-R datasets.
A.3.1 Ablation Studies
Table 10 reports detailed ablation results per dataset. The right column computes the average accuracy across datasets.
A.3.2 Case Study: TREC-6 Dataset
Table 12 shows a sample of rules from the TREC-6 dataset. Those rules capture regular expression patterns to predict one of the 6 question categories for a question. Tables 13-26 show examples of individual instances in TREC-6, the corresponding rule predictions, the student pseudo-labels, as well as our RAN that aggregates rule and student predictions with attention weights  $a$  to compute a single (Teacher) label.
Hyperparameter	Values
Learning rate	1e-1, 1e-2, 1e-3, 1e-4, 1e-5
Fine-tuning rate	1e-3, 1e-4, 1e-5, 1e-6, 1e-7
Type of pseudo-labels	soft, hard
Table 8: Hyperparameter search.
TREC	SMS	Youtube	CENSUS	MIT-R	Spouse
Instance vector type	ELMO	ELMO	LogReg	Categorical	ELMO	BERT
Instance vector dimensionality	1024	1024	16,634	105	1024	768
Learning rate	1e-4	1e-4	1e-4	1e-3	1e-4	1e-4
Fine-tuning rate	1e-5	1e-5	1e-5	1e-5	1e-4	1e-5
Type of pseudo-labels	soft	soft	hard	soft	soft	soft
Pseudo-training epochs (patience: 5)	25	25	25	25	25	25
Fine-tuning epochs (patience: 5)	70	70	70	70	70	70
Self-training iterations (patience: 3)	25	25	25	25	25	25
Training batch size	16	16	16	128	256	16
Unsupervised batch size	256	256	256	256	256	256
Rule embedding dimensionality	128	128	128	128	128	128
Table 9: Selected hyperparameters.
TREC (Acc)	SMS (F1)	Youtube (Acc)	CENSUS (Acc)	MIT-R (F1)	AVG (Acc)
ASTRA (Teacher)	80.3 (2.4)	95.3 (0.5)	95.3 (0.8)	83.1 (0.4)	76.9 (0.6)	88.1
ASTRA (Student)	79.2 (2.1)	95.7 (0.5)	95.5 (0.5)	82.8 (0.1)	76.6 (0.9)	87.7
Hard Student Pseudo-labels in RAN in Eq. (2)	77.6 (1.2)	94.5 (0.7)	95.3 (0.8)	83.0 (0.7)	75.9 (0.9)	87.0
No minimum entropy regularization in Eq. (4)	75.8 (2.5)	95.7 (0.7)	95.1 (0.8)	83.0 (0.3)	72.6 (0.4)	86.9
No cross-entropy loss in Eq. (4)	74.1 (2.3)	93.9 (0.4)	95.5 (0.6)	83.1 (0.6)	71.9 (0.4)	86.7
No student pseudo-labels in RAN in Eq. (1)	75.3 (4.3)	95.8 (0.2)	91.4 (2.2)	83.1 (0.4)	71.9 (1.0)	85.3
Table 10: Ablation studies.
TREC	SMS	Youtube	CENSUS	MIT-R	Spouse
|DL|	68	69	100	83	1842	100
|DU|	4884	1586	4502	64,888	10,000	22,254
Validation	500	500	150	5561	16281	2711
Test Size	500	500	250	16281	14256	2701
#Classes	6	2	2	2	9	2
#Rules	68	73	10	83	15	9
Rule Precision (Majority Voting)	63.7%	97.3%	78.6%	80.7%	84.1%	66.6%
Rule Accuracy (Majority Voting)	60.9%	48.4%	82.2%	80.1%	40.9%	44.2%
Rule Coverage (instances in DU covered by ≥ 1 rule)	95%	40%	87%	100%	14%	25%
Rule Overlap (instances in DU covered by ≥ 2 rules)	46%	9%	48%	94%	1%	8%
Table 11: Dataset statistics.
Rule	Label	Pattern
Rule 5	HUM	( |^) (which|who|what|what) [^{\w}] * ([^{\s}]+ ) * (person|man|woman|human|poet|poet) [^{\w}] * ( |$)
Rule 8	HUM	( |^) (who|what|what) [^{\w}] * (\w+ ) {0,1} (person|man|woman|human|president|president) [^{\w}] * ( |$)
Rule 24	ENTY	( |^) (what|what) [^{\w}] * (\w+ ) {0,1} (is|is) [^{\w}] * (*([^{\s}]+ ) * (surname|address|name|name) [^{\w}] * ( |$)
Rule 29	NUM	( |^) (which|what|what) [^{\w}] * (*([^{\s}]+ ) * (time|day|month|hours|minute *|seconds|year|date|date) [^{\w}] * ( |$)
Rule 32	NUM	( |^) (year|year) [^{\w}] * ( |$)
Rule 41	NUM	( |^) (what|what) [^{\w}] * ([^{\s}]+ ) * (percentage |share|number|population|population) [^{\w}] * ( |$)
Rule 42	DESC	( |^) (explain|describe|what|what) [^{\w}] * ( |$)
Rule 54	DESC	( |^) (how|what|what) [^{\w}] * (*\w+ ) {0,1} (do|does|does) [^{\w}] * ( |$)
Rule 61	HUM	( |^) (called|alias|nicknamed|nicknamed) [^{\w}] * ( |$)
Rule 68	ABBR	( |^) (what|what) [^{\w}] * (\w+ ) {0,1} (does|does) [^{\w}] * (* ([^{\s}]+ ) * (stand for) [^{\w}] * ( |$)
Table 12: Sample of REGEX rules from the TREC-6 dataset capturing the various question categories (HUM: Human, ENTY: Entity, NUM: Numeric Value, DESC: Description, ABBR: Abbreviation)
Text	Why is a ladybug helpful?
Clean label	DESC
RAN Teacher	DESC
Weak Source	Label	Weight	Feature
Student	DESC	a=1.0	hi (contextualized instance embedding)
Table 13: TREC example. No rules apply. The student generalizes beyond rules by considering contextualized instance embeddings and assigns the instance to the DESCRIPTION class. Our RAN teacher assigns an attention weight of 1 to the student and predicts the right class.
Text	Which president was unmarried ?
Clean label	HUM
RAN Teacher	NUM
Weak Source	Label	Weight	Feature
Student	HUM	a=1.0	hi (contextualized instance embedding)
Table 14: TREC example: While no rules apply to this instance, the student leverages the semantics of the sentence (i.e., that "president" corresponds to a person) to predict the HUMAN (HUM) class. Thus, our RAN predicts the right label instead of discarding this instance.
Text	What is a baby turkey called?
Clean label	ENTY
RAN Teacher	ENTY
Weak Source	Label	Weight	Feature
Student	DESC	a=1.0	h_i (contextualized instance embedding)
Rule 24	ENTY	a=1.0	( |^) (what|what) [^{\w}] * (\w+ ) {0,1} (is|is) [^{\w}] *
* ([^{\s}] + ) * (surname|address|name|name) [^{\w}] * ( |$)
Rule 42	DESC	a=0.0	( |^) (explain|describe|what|what) [^{\w}] * ( |$)
Rule 61	HUM	a=0.0	( |^) (called|alias|nicknamed|nicknamed) [^{\w}] * ( |$)
Table 15: TREC example: Student is wrong but unsure. Teacher predicts the right label and fixes student's mistake. Teacher correctly down-weights rule 42 and rule 61 that provide the wrong prediction but erroneously up-weights the Student. As in this case the Student is uncertain, about the label, the final aggregated prediction of the Teacher is mostly influenced by Rule 24.
Text	What currency do they use in Brazil?
Clean label	ENTY
RAN Teacher	ENTY
Weak Source	Label	Weight	Feature
Student	ENTY	a=1.0	hi (contextualized instance embedding)
Rule 42	DESC	a=0.0	(|^)(explain|describe|what|what)[^{\w}]*( |$)
Rule 54	DESC	a=0.0	(|^)(how|what|what)[^{\w}]*
★ (\w+ ){0,1} (do|does|does)[^{\w}]* ( |$)
Table 16: TREC example. The student is crucial for RAN to predict the right label (ENTITY) as both rules predict the wrong label. RAN correctly down-weights the two rules and up-weights the Student.
Text	What is an atom?
Clean label	DESC
RAN Teacher	DESC
Weak Source	Label	Weight	Feature
Student	DESC	a=1.0	hi (contextualized instance embedding)
Rule 42	DESC	a=1.0	(|^)(explain|describe|what|what)[^{\w}]*( |$)
Table 17: TREC example: Rule 42 was down-weighted in the previous two examples but is up-weighted here, demonstrating that RAN effectively leverages the contextualized instance representation to predict instance-specific rule weights.
Text	What was President Lyndon Johnson's reform program called?
Clean label	ENTY
RAN Teacher	ENTY
Weak Source	Label	Weight	Feature
Student	ENTY	a=1.0	\(h_i\)(contextualized instance embedding)
Rule 8	HUM	a=1.0	( |^) (who|what|what) [^{\backslash}w] * (\w+ ) {0,1} (person|man|woman|human|president|president) [^{\backslash}w] * ( |$)
Rule 24	ENTY	a=1.0	( |^) (what|what) [^{\backslash}w] * (\w+ ) {0,1} (is|is) [^{\backslash}w] * *( [^{\backslash}s]+ ) * ( surname|address|name|name) [^{\backslash}w] * ( |$)
Rule 42	DESC	a=0.0	( |^) (explain|describe|what|what) [^{\backslash}w] * ( |$)
Rule 61	HUM	a=0.0	( |^) (called|alias|nicknamed|nicknamed) [^{\backslash}w] * ( |$)
Table 18: TREC example.
Text	What is the percentage of water content in the human body?
Clean label	NUM
RAN Teacher	NUM
Weak Source	Label	Weight	Feature
Student	DESC	a=0.0	\(h_i\) (contextualized instance embedding)
Rule 5	HUM	a=0.0	( |^)(which|who|what|what)[^{\backslash}w] * ([^{\backslash}s]+ )*(person|man|woman|human|poet|poet)[^{\backslash}w]* ( |$)
Rule 41	NUM	a=0.2	( |^)(what|what)[^{\backslash}w] * ([^{\backslash}s]+ )*(percentage|share|number|population|population)[^{\backslash}w]* ( |$)
Rule 42	DESC	a=0.0	( |^)(explain|describe|what|what)[^{\backslash}w]* ( |$)
Table 19: TREC example.
Text	How fast is alcohol absorbed?
Clean label	NUM
RAN Teacher	NUM
Weak Source	Label	Weight	Feature
Student	NUM	a=1.0	hi(contextualized instance embedding)
Table 20: TREC example: While no rules apply to this instance, the student associates "How fast" with the NUMBER (NUM) class.
Text	Which mountain range in North America stretches from Maine to Georgia?
Clean label	LOC
RAN Teacher	LOC
Weak Source	Label	Weight	Feature
Student	LOC	a=1.0	h_i (contextualized instance embedding)
Table 21: TREC example: While no rules apply to this instance, the student associates the context with the LOCATION (LOC) class.
Text	When is the official first day of summer?
Clean label	NUM
RAN Teacher	NUM
Weak Source	Label	Weight	Feature
Student	NUM	a=1.0	hi(contextualized instance embedding)
Table 22: TREC example: While no rules apply to this instance, the student associates “when,” “day,” and “summer” to NUMBER (NUM) class.
Text	What is Australia's national flower?
Clean label	ENTY
RAN Teacher	DESC
Weak Source	Label	Weight	Feature
Student	DESC	a=1.0	h_i (contextualized instance embedding)
Rule 42	DESC	a=0.0	( |^) (explain|describe|what|what) [^{\backslash}w]*( |$)
Table 23: TREC example: Both the Student and Rule 42 provide a wrong prediction
Text	What is the name of the chocolate company in San Francisco?
Clean label	HUM
RAN Teacher	ENTY
Weak Source	Label	Weight	Feature
Student	ENTY	a=1.0	hi (contextualized instance embedding)
Rule 42	DESC	a=0.0	(|^)(explain|describe|what|what)[^{\w}]*( |$)
Rule 53	ENTY	a=1.0	(( |^)(name|name)[^{\w}]*( |$))
Rule 59	ENTY	a=1.0	(|^)(which|what|what)[^{\w}]*
*( [^{\s}] + ) * (organization|trust|company|company)[^{\w}]*( |$)
Rule 67	ENTY	a=0.6	(|^)(what|what)[^{\w}]* ( \w+ ) {0,1} (is|is)[^{\w}]*
*( [^{\s}] + ) * (surname|address|name|name)[^{\w}]* ( |$)
Table 24: The clean label in this case is Human, while from the text we understand that the label is Entity.
Text	What does I.V. stand for?
Clean label	ABBR
RAN Teacher	ABBR
Weak Source	Label	Weight	Feature
Student	ABBR	a=1.0	hi (contextualized instance embedding)
Rule 42	DESC	a=0.0	(|^)(explain|describe|what|what)[^{\w}]*( |$)
Rule 54	DESC	a=0.1	(|^)(how|what|what)[^{\w}]*
*( \w+ ) {0,1} (do|does|does)[^{\w}]* ( |$)
Rule 68	ABBR	a=1.0	(|^)(what|what)[^{\w}]* ( \w+ ) {0,1} (does|does)[^{\w}]*
*( [^{\s}]+ ) *(stand for)[^{\w}]* ( |$)
Table 25: TREC example.
Text	What year did the Titanic sink ?
Clean label	NUM
RAN Teacher	NUM
Weak Source	Label	Weight	Feature
Student	NUM	a=1.0	hi (contextualized instance embedding)
Rule 29	NUM	a=1.0	( |^)(which|what|what) [^{\w}] *
*([^{\s}] + ) * (time|day|month|hours|minute
*|seconds|year|date|date) [^{\w}] * ( |$)
Rule 32	NUM	a=1.0	( |^)(year|year) [^{\w}] * ( |$)
Rule 42	DESC	a=0.0	( |^)(explain|describe|what|what) [^{\w}] * ( |$)
Table 26: TREC example.