SlowGuess/NAACL_2022_OCR huggingface dataset:
MetaWN: Code is adapted from https://github.com/xjtushujun/meta-weight-net. Batch size is 8 for the 4 document-level classification datasets and 4 for the 4 token-level classification datasets. The meta-network is a three-layer feed-forward network with hidden dimension of 128. Each experiment trains for 10 epochs with the checkpoint with the best validation performance saved for evaluation on test set.
- MLC: Code is adapted from https://github.com/microsoft/MLC. Batch size is 8 for the 4 document-level classification datasets and 4 for the 4 token-level classification datasets. The meta-network is a three-layer feed-forward network with hidden dimension of 128; the label embedding dimension used in the meta-network is 64. Each experiment trains for 10 epochs with the checkpoint with the best validation performance saved for evaluation on test set.
To encode input text, we experiment with various text encoders, ranging from shallow LSTMs to large pre-trained transformerbased encoders (Vaswani et al., 2017):
- BiLSTM-based encoder: the BiLSTM implementations are all based on 50-dimensional pre-trained glove word embeddings (Pennington et al., 2014) and bi-directional LSTMs with hidden size 128. Note that our implementation is different than other BiLSTM implementations used by previous work, which are based on 100-dimensional word embeddings and LSTM hidden size 300. This renders our BiLSTM models roughly 3 times smaller than those used by previous work, thus the numbers are not directly comparable. We chose a smaller model capacity for BiLSTMs to contrast the performance with larger models including DistilBERT and others to show the importance of model capacity on WALNUT. During training, we use a learning rate of 0.005 for all BiLSTM-based models.
- Transformer-based encoders: we consider pre-trained DistilBERT (Sanh et al., 2019), BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), BERT-large, and RoBERTa-large.
We fine-tune these models (via the huggingface library) using task-specific classification heads on top of the encoder and a learning rate of 0.00001.
B.1 Document-level classification
- AGNews: and multi-class topic classification (world vs. sports vs. business vs. sci/tech) on news articles from the AGNews dataset (Zhang et al., 2015).
- Yelp: binary sentiment classification (negative vs. positive) of Yelp restaurant reviews (Zhang et al., 2015).
- IMDB: binary sentiment classification (negative vs. positive) of IMDB movie reviews (Maas et al., 2011).
- GossipCop: binary fake news detection (fake vs. not fake) on news articles from the GossipCop $^3$  fact-checking websites. The GossipCop dataset is part of the fake news detection benchmark FakeNewsNet (Shu et al., 2020a). (We only include the results of Gossipcop to represent fake news classification task as the results for Politifact are similar.)
B.2 Token-level classification
According to the BIO tagging scheme, "B," "I," and "O," represent the beginning, inside, and outside, of a named entity span, respectively. (Not extracting any values corresponds to a sequence of "O"-only tags.) Consider, for example, named entity recognition in the CoNLL dataset:
Tokens: Barack Obama lives in Washington Tags: B-PER I-PER O O B-LOC
- CoNLL: the CoNLL 2003 dataset (Sang and De Meulder, 2003) contains news articles from Reuters (split into sentences). In total, there are 35,089 entities from 4 types: organization (ORG), person (PER), location (LOC), and miscellaneous (MISC). Tag classes  $C'$ : ['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']
- NCBI: the NCBI Disease corpus (Doğan et al., 2014) contains PubMed abstracts with
6,866 disease mentions. Tag types:  $\left[\mathrm{O}^{\prime},\right.$  B', I]
- WikiGold: the WikiGold dataset (Balasuriya et al., 2009) contains English Wikipedia articles that were randomly selected and manually annotated with the same entity types as CoNLL. Tag classes  $C'$ : ['O', 'B-PER', 'I-PER', 'B-ORG', 'I-ORG', 'B-LOC', 'I-LOC', 'B-MISC', 'I-MISC']
- LaptopReview: the Laptop Review corpus from the SemEval 2014 Challenge (Pontiki et al., 2016) contains 3,012 mentions to laptop features. Tag types  $C'$ : ['O', 'B', 'I']
Table 5 shows detailed statistics for token-level classification datasets. More dataset statistics are provided in Table 1. Tables 6-13 show detailed information for all rules. Figures 4-10 show examples of weak rules for various datasets.
<sup>3</sup>https://www.gossipcop.com/
Table 5: Extra token-level statistics for the token-level classification datasets.
CoNLL	NCBI	WikiGold	LaptopReview
# train tokens	203,621	135,572	31,560	41,525
# dev tokens	51,362	23,789	3,683	9,970
# test tokens	46,435	24,219	3,762	11,884
Table 6: List of rules for the AGNews dataset. The rules are the same as the tagging rules in (Zhang et al., 2015). The Python implementations can be found in: https://github.com/weakrules/Denoise-multi-weak-sources/blob/master/rules-noisy-labels/Agnews/angews_rule.py
Rule name	Description
1. world1	Keyword-based detection of the WORLD topic
2. world2	Keyword-based detection of the WORLD topic
3. sports1	Keyword-based detection of the SPORTS topic
4. sports2	Keyword-based detection of the SPORTS topic
5. sports3	Keyword-based detection of the SPORTS topic
6. tech1	Keyword-based detection of the TECH topic
7. tech2	Keyword-based detection of the TECH topic
8. business1	Keyword-based detection of the BUSINESS topic
9. business2	Keyword-based detection of the BUSINESS topic
Table 7: List of rules for the IMDB dataset. The rules are the same as in (Zhang et al., 2015). The Python implementations can be found in: https://github.com/weakrules/Denoise-multi-weak-sources/blob/master/rules-noisy-labels/IMDB/imdb_rule.py
Rule name	Description
1. expression_nexttime	Regex-based detection of POSITIVE sentiment (re-watching expressions)
2. expression Recommend	Regex-based detection of POSITIVE sentiment (recommendation expressions)
3. expression_value	Regex-based detection of POSITIVE sentiment (value expressions)
4. keyword comparé	Keyword-based detection of NEGATIVE sentiment based on movie comparisons
5. keyword_general	Keyword-based detection of POSITIVE and NEGATIVE sentiment
6. keyword_ACTOR	Keyword-based detection of POSITIVE sentiment regarding the actors
7. keyword_finish	Keyword-based detection of NEGATIVE sentiment
8. keyword_plot	Keyword-based detection of POSITIVE and NEGATIVE sentiment regarding the plot
Table 8: List of rules for the Yelp dataset. The rules are the same as in (Zhang et al., 2015). The Python implementations can be found in: https://github.com/weakrules/Denoise-multi-weak-sources/blob/master/rules-noisy-labels/Yelp/yelp_rule.py
Rule name	Description
1. textblob_lf	Model-based detection of POSITIVE and NEGATIVE sentiment (TextBlob model)
2. keyword_recommend	Regex-based detection of POSITIVE sentiment (recommendation expressions)
3. keyword_general	Regex-based detection of POSITIVE and POSITIVE sentiment (general expressions)
4. keyword_mood	Keyword-based detection of POSITIVE and NEGATIVE sentiment based on the user's mood
5. keyword_service	Keyword-based detection of POSITIVE and NEGATIVE sentiment relevant to the service
6. keyword_price	Keyword-based detection of POSITIVE and NEGATIVE sentiment regarding the prices
7. keyword_environment	Keyword-based detection of POSITIVE and NEGATIVE sentiment relevant to the ambience
8. keyword_food	Keyword-based detection of POSITIVE and NEGATIVE sentiment relevant to the food
Table 9: List of rules for the GossipCop dataset. The rules are the same as in (Shu et al., 2020a) (page 8 in http://www.cs.iit.edu/~kshu/files/ecml_PKdd_mwss.pdf).
Rule name	Description
1. mean Scores	User interaction-based detection of FAKE news: If a news piece has a standard deviation of user sentiment scores greater than a threshold τ1, then the news is weakly labeled as FAKE news.
2. std Scores	User interaction-based detection of FAKE news: If the mean value of users' absolute bias scores - sharing a piece of news - is greater than a threshold τ2, then the news piece is weakly-labeled as FAKE news.
3. credibility_score	User interaction-based detection of FAKE news: If a news piece has an average credibility score less than a threshold τ3, then the news is weakly-labeled as FAKE news.
Rule name
1. date_detector
2. time_detector
3. money_detector
4. proper_detector
5. infrequent_proper_detector
6. proper2_detector
7. infrequent_proper2_detector
8. nnp_detector
9. infrequent_nnp_detector
10. compound_detector
11. infrequent_compound_detector
12. miscdetector
13. legal_detector
14. company_type_detector
15. full_name_detector
16. number_detector
17. snips
18. core_web_md
19. core_web_md+c
20. BTC
21.  $\mathrm{BTC + c}$
22. SEC
23.  $\mathrm{SEC} + \mathrm{c}$
24.edited_core_web_md
25.edited_core_web_md+c
26. wiki(case)
27. wiki_uncased
28. multitoken_wikicased
29. multitoken_wiki_uncased
30. wiki_smallcased
31. wiki_small_uncased
32. multitoken_wiki_smallcased
33. multitoken_wiki_small_uncased
34. geo_cased
35. geo_uncased
36. multitoken.geo_cased
37. multitoken.geo_uncased
38. crunchbase_cased
39. crunchbase_uncased
40. multitoken_crunchbasecased
41. multitoken_crunchbase_uncased
42. productcased
43. product_uncased
44. multitoken_productcased
45. multitoken_product_uncased
46. doclevel_voter
47. doc_historycased
48. doc_history_uncased
49. docmajoritycased
50. doc_majority_uncased
Heuristic detection of entities of type DATE
Heuristic detection of entities of type TIME
Heuristic detection of entities of type MONEY
Heuristic detection of proper names based on casing
+ including at least one infrequent token
Heuristic detection of sequences of tokens with NNP as POS-tag
+ including at least one infrequent token (rank  $>15000$  in vocabulary)
Heuristic detection of proper noun phrases with compound dependency relations
Heuristic detection of entities of type NORP, LANGUAGE, FAC or EVENT
Heuristic detection of entities of type LAW
Gazetteer using a large list of company names
Heuristic function to detect full person names
Heuristic detection of entities CARDINAL,ORDINAL, PERCENT and QUANTITY
Probabilistic parser specialised in the recognition of dates,
+ times, money amounts, percents, and cardinal/ordinal values
NER model trained on Ontonotes 5.0
NER model trained on Ontonotes  $5.0 +$  postprocessing
NER model trained on the Broad Twitter Corpus
NER model trained on the Broad Twitter Corpus + postprocessing
NER model trained on SEC-filings
NER model trained on SEC-filings + postprocessing
NER model trained on Ontonotes  $5.0+$  alternative postprocessing
NER model trained on Ontonotes  $5.0 +$  alternative postprocessing
Gazetteer (case-sensitive) using Wikipedia entries
Gazetteer (case-insensitive) using Wikipedia entries
Same as above, but restricted to multitoken entities
Gazetteer (case-sensitive) using Wikipedia entries with non-empty description
Gazetteer (case-insensitive) using Wikipedia entries with non-empty description
Gazetteer (case-sensitive) using the Geonames database
Gazetteer (case-insensitive) using the Geonames database
Gazetteer (case-sensitive) using the Crunchbase Open Data Map
Gazetteer (case-insensitive) using the Crunchbase Open Data Map
Gazetteer (case-sensitive) using products extracted from DBPedia
Gazetteer (case-insensitive) using products extracted from DBPedia
Consider all appearances of the same entity string in the document
Considers already introduced entities in the document (case-sensitive)
982 Considers already introduced entities in the document (case-insensitive)
Considers all entities in the document (case-sensitive)
Considers all majority labels in the document (case-insensitive)
Table 10: List of rules for the CoNLL dataset. The Python implementation of CoNLL rules is provided in the "skweak" repo: https://github.com/NorskRegnesentral/skweak/blob/670fcdec680930ce3e497886d06d61e6a1f2c195/examples/ner/conl12003_ner.py
Table 11: List of rules for the NCBI dataset. The rules are the same as the tagging rules in (Safranchik et al., 2020). Python implementations: https://github.com/BatsResearch/safranchik-aaaai20-code/blob/master/NCBI-Disease/train_generate_models.py
Rule name	Description
1. CoreDictionaryUncased	AutoNER dictionary (biomedical entities)
2. CoreDictionaryExact	AutoNER dictionary (biomedical entities, exact match)
3. CancerLike	Heuristic detection of entities that are relevant to cancer
4. CommonPrefixes	Heuristic detection of entities that are relevant to common diseases
5. Deficiency	Heuristic detection of entities that are relevant to deficiencies
6. Disorder	Heuristic detection of entities that are relevant to disorders
7. Lesion	Heuristic detection of entities that are relevant to lesions
8. Syndrome	Heuristic detection of entities that are relevant to syndromes
9. BodyTerms	UMLS dictionary entries for terms that are relevant to body parts
10. OtherPOS	Heuristic detection of parts of speech that are not relevant to any disease
11. StopWords	Heuristic detection of stop words that are not relevant to any disease
12. Punctuation	Heuristic detection of punctiations that are not relevant to any disease
Table 12: List of rules for the WikiGold dataset. The Python implementation of WikiGold rules is provided in the "skweak" repo: https://github.com/NorskRegnesentral/skweak/blob/670fcdec680930ce3e497886d06d61e6a1f2c195/examples/ner/conl12003_ner.py
Rule name	Description
1. BTC	NER model trained on the Broad Twitter Corpus
2. core_web_md	NER model trained on Ontonotes 5.0
3. crunchbasecased	Gazetteer (case-sensitive) using the Crunchbase Open Data Map
4. crunchbase_uncased	Gazetteer (case-insensitive) using the Crunchbase Open Data Map
5. full_name_detector	Heuristic function to detect full person names
6. geocased	Gazetteer (case-sensitive) using the Geonames database
7. geo_uncased	Gazetteer (case-insensitive) using the Geonames database
8. misc_detector	Heuristic detection of entities of type NORP, LANGUAGE, FAC or EVENT
9. wikicased	Gazetteer (case-sensitive) using Wikipedia entries
10. wiki_uncased	Gazetteer (case-insensitive) using Wikipedia entries
11. multitoken crunchbasecased	Same as above, but restricted to multitoken entities
12. multitoken crunchbase_uncased	Same as above, but restricted to multitoken entities
13. multitoken.geocased	Same as above, but restricted to multitoken entities
14. multitoken.geo_uncased	Same as above, but restricted to multitoken entities
15. multitoken_wikicased	Same as above, but restricted to multitoken entities
16. multitoken_wiki_uncased	Same as above, but restricted to multitoken entities
Table 13: List of rules for the LaptopReview dataset. The rules are the same as the tagging rules in (Safranchik et al., 2020). Python implementations: https://github.com/BatsResearch/safranchik-AAai20-code/blob/master/LaptopReview/train_generate_models.py
Rule name	Description
1. CoreDictionary	AutoNER dict with entries of terms that are relevant to electronics
2. OtherTerms	Heuristic detection of laptop entities based on a pre-defined keyword list
3. ReplaceThe	Heuristic detection of laptop entities based on the “replace the” phrase
4. iStuff	Heuristic detection of laptop entities based on uppercase letters
5. Feelings	Heuristic detection of laptop entities based on common expressions
6. ProblemWithThe	Heuristic detection of laptop entities based on common expressions
7. External	Heuristic detection of laptop entities based on common hardware expression
8. StopWords	Heuristic detection of stop words that are not relevant to electronics
9. Punctuation	Heuristic detection of punctuation that are not relevant to electronics
10. Pronouns	Heuristic detection of pronouns that are not relevant to electronics
11. NotFeatures	Heuristic detection of terms that are not relevant to laptop features
12. Adv	Heuristic detection of adverbs that are not relevant to electronics
Figure 4: Example of weak rule from the Yelp dataset (rule 6: keyword_price from Table 8).
Figure 5: Example of weak rule from the Yelp dataset (rule 1: textblob_lf from Table 8).
Figure 6: Example of weak rule from the CoNLL dataset (rule 3: money_detector from Table 10). This rule heuristically detects entities that are relevant to money.
Figure 7: Example of weak rule from the CoNLL dataset (rule 16: number_detector from Table 10). This rule heuristically detects entities that are relevant to numbers.
Figure 8: Example of weak rule from the NCBI dataset (rule 3: CancerLike from Table 11). This rule heuristically detects entities that are relevant to cancer.
Figure 9: Example of weak rule from the NCBI dataset (rule 11: StopWords from Table 11). This rule heuristically detects stop words and assigns the 'O' tag to the corresponding tokens by assuming that they are not relevant to any disease.
Figure 10: Example of weak rule from the LaptopReview dataset (rule 5: Feelings from Table 13). This rule heuristically detects entities that are relevant to laptop features based on keywords that express the user's feelings.
Table 14 shows standard deviation results for all datasets, methods, and base models. The rightmost column responds the average standard deviation (AVG std) across tasks, which we also reported in Table 3.
Analysis of individual weak rules. Tables 15-21 show performance results for each weak rule for the datasets in WALNUT. We evaluate two different strategies for majority voting in case of an instance that is not covered by any rules: (1) "Strict" counts the instance as misclassified and (2) "Loose" assigns a random label to the instance. Most rules have very low F1 score while there are a few rules with a relatively high F1 score.
Figure 3 shows the precision-recall scatter plots for each weak rule individually. (We skip the scatter plot for GossipCop as it has just 3 rules.) Several rules have relatively high precision but most rules have very low recall.
Table 14: Standard deviation results on WALNUT.
Method	AGNews	IMDB	Yelp	GossipCop	CoNLL	NCBI	WikiGold	LaptopReview	AVG
BiLSTM
Full Clean	0.2	0.5	0.3	1.0	7.2	1.6	0.5	2.5	1.7
C	1.1	2.9	4.4	2.0	1.0	1.5	0.9	5.0	2.4
W	6.1	0.5	2.4	0.9	3.2	1.5	0.6	1.2	2.1
C+W	0.3	0.9	1.2	1.1	6.7	1.1	0.6	2.9	1.9
Snorkel	3.9	0.8	0.6	0.4	2.3	2.1	0.7	1.4	1.5
C+Snorkel	0.4	1.0	1.2	1.5	2.8	1.7	0.6	2.1	1.4
GLC	1.3	0.3	0.9	1.7	7.2	1.2	0.5	0.9	1.7
MetaWN	1.8	0.3	2.0	1.3	0.0	1.3	0.4	0.5	0.9
MLC	2.1	0.3	3.8	1.6	0.0	1.1	0.3	0.7	1.2
DistilBERT
Full Clean	0.2	0.3	0.3	1.5	0.4	0.4	0.7	3.5	0.9
C	6.2	6.8	7.4	6.4	3.6	2.0	1.3	1.7	4.4
W	3.9	1.1	1.5	1.1	0.9	1.2	0.3	2.0	1.5
C+W	0.6	0.2	0.9	1.2	0.8	1.4	0.4	3.6	1.1
Snorkel	3.0	1.6	0.6	0.5	1.1	1.5	0.4	2.2	1.4
C+Snorkel	0.7	0.5	2.0	0.8	0.9	1.9	0.4	3.6	1.4
GLC	2.8	0.5	1.5	2.0	2.1	1.8	0.2	1.5	1.5
MetaWN	1.6	1.3	0.8	2.2	1.8	1.4	0.3	0.6	1.3
MLC	2.5	0.6	0.7	1.6	1.2	1.8	0.4	2.3	1.4
BERT
Full Clean	0.1	0.5	0.2	1.0	0.6	0.5	1.0	1.8	0.7
C	0.9	8.1	5.2	1.8	1.3	0.8	1.3	2.5	2.7
W	2.7	0.5	1.1	2.2	1.2	2.8	0.9	1.8	1.7
C+W	0.4	0.6	1.4	1.5	0.9	1.4	0.8	1.5	1.1
Snorkel	2.3	3.7	1.3	0.9	1.3	3.5	1.0	1.3	1.9
C+Snorkel	1.0	0.5	0.6	0.6	1.6	1.7	0.8	2.6	1.2
GLC	1.6	0.8	2.2	2.8	2.4	1.2	0.4	1.2	1.6
MetaWN	1.1	1.0	1.0	2.4	1.6	0.5	0.3	1.4	1.2
MLC	2.0	0.8	1.3	1.4	2.1	2.8	0.2	0.4	1.4
RoBERTa
Full Clean	0.1	0.4	0.2	1.0	0.3	0.7	1.0	2.0	0.7
C	2.0	5.4	5.9	5.2	2.3	2.1	1.7	4.1	3.6
W	1.2	0.7	1.2	2.4	1.4	1.5	0.9	2.7	1.5
C+W	0.9	1.7	1.4	1.0	1.6	1.5	0.6	5.3	1.8
Snorkel	3.2	2.3	2.9	0.6	2.0	1.1	0.9	2.9	2.0
C+Snorkel	0.7	2.2	1.6	1.8	1.8	3.1	0.8	5.7	2.2
GLC	1.3	0.7	1.8	2.3	3.2	0.4	0.4	0.8	1.4
MetaWN	2.7	0.9	1.4	2.1	0.7	0.9	0.3	1.1	1.3
MLC	1.6	1.2	1.0	1.3	1.2	2.7	0.2	3.1	1.6
BERT-large
Full Clean	0.1	0.4	0.3	0.6	1.0	1.4	1.2	3.1	1.0
C	22.6	3.7	5.8	3.2	4.0	2.4	2.3	3.4	5.9
W	1.1	2.4	1.2	1.4	1.2	2.0	1.1	1.0	1.4
C+W	2.1	1.6	0.9	1.8	1.6	1.9	0.9	3.8	1.8
Snorkel	2.2	1.5	0.5	1.5	0.7	4.0	1.1	1.6	1.6
C+Snorkel	0.9	1.4	0.8	1.4	1.0	4.5	1.1	2.8	1.7
GLC	2.0	0.9	1.1	1.2	1.7	1.2	0.9	2.0	1.4
MetaWN	1.9	1.0	3.9	2.0	1.5	1.0	0.9	23.0	4.4
RoBERTa-large
Full Clean	0.07	0.33	0.16	0.59	0.7	0.7	0.8	2.0	0.7
C	1.8	9.6	7.8	1.0	1.5	1.2	0.7	4.7	3.5
W	0.8	0.7	0.5	2.7	2.0	4.1	1.8	2.9	1.9
C+W	1.2	1.6	1.6	2.6	1.9	1.3	0.7	5.0	2.0
Snorkel	0.8	2.5	2.5	1.9	1.2	2.9	1.9	4.4	2.3
C+Snorkel	2.1	2.5	1.5	2.3	2.5	3.1	0.7	2.8	2.2
GLC	1.7	1.0	2.1	1.2	28.4	1.2	0.8	3.8	5.0
MetaWN	2.0	16.6	3.0	15.6	1.4	1.5	0.6	2.9	5.5
Table 15: Performance of each rule on AGNews.
Rule	AG News
Unlabeled	train	validation	test
Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1
rule 1	0.179	0.078	0.109	0.179	0.114	0.140	0.182	0.077	0.108	0.180	0.079	0.110
rule 2	0.157	0.082	0.108	0.156	0.121	0.136	0.159	0.082	0.108	0.154	0.081	0.106
rule 3	0.162	0.093	0.118	0.162	0.134	0.147	0.160	0.094	0.118	0.166	0.094	0.120
rule 4	0.192	0.011	0.021	0.192	0.018	0.033	0.192	0.012	0.022	0.193	0.011	0.021
rule 5	0.187	0.064	0.095	0.189	0.090	0.122	0.190	0.067	0.099	0.188	0.068	0.099
rule 6	0.140	0.053	0.077	0.141	0.100	0.117	0.137	0.054	0.077	0.141	0.051	0.075
rule 7	0.161	0.052	0.079	0.163	0.096	0.121	0.163	0.050	0.077	0.163	0.051	0.078
rule 8	0.136	0.114	0.124	0.138	0.168	0.152	0.134	0.113	0.123	0.137	0.118	0.127
rule 9	0.152	0.007	0.014	0.153	0.011	0.020	0.149	0.007	0.013	0.154	0.008	0.014
Majority (strict)	0.649	0.426	0.512	0.814	0.812	0.812	0.645	0.424	0.509	0.650	0.429	0.514
Majority (loose)	0.618	0.620	0.617	0.814	0.812	0.812	0.611	0.613	0.610	0.618	0.620	0.618
Table 16: Performance of each rule on IMDB.
IMDB
Rule	unlabeled	train	validation	test
Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1
rule 1	0.182	0.001	0.001	0.000	0.000	0.000	0.333	0.000	0.001	0.000	0.000	0.000
rule 2	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
rule 3	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
rule 4	0.497	0.405	0.446	0.502	0.478	0.489	0.505	0.404	0.448	0.513	0.423	0.463
rule 5	0.538	0.044	0.073	0.549	0.045	0.075	0.408	0.039	0.067	0.481	0.046	0.077
rule 6	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000	0.000
rule 7	0.457	0.109	0.176	0.463	0.120	0.190	0.448	0.095	0.156	0.459	0.115	0.183
rule 8	0.655	0.006	0.012	0.644	0.004	0.009	0.630	0.008	0.015	0.667	0.008	0.015
Majority (strict)	0.495	0.426	0.457	0.749	0.745	0.745	0.501	0.423	0.458	0.511	0.448	0.476
Majority (loose)	0.708	0.707	0.706	0.749	0.745	0.745	0.710	0.708	0.708	0.740	0.739	0.739
Table 17: Performance of each rule on Yelp.
Rule	Yelp
Unlabeled	train	validation	test
Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1
rule 1	0.642	0.047	0.085	0.638	0.053	0.094	0.643	0.052	0.093	0.614	0.042	0.076
rule 2	0.214	0.029	0.051	0.221	0.036	0.063	0.213	0.028	0.050	0.239	0.031	0.054
rule 3	0.501	0.328	0.371	0.504	0.393	0.419	0.514	0.338	0.381	0.492	0.324	0.367
rule 4	0.498	0.064	0.114	0.485	0.081	0.139	0.501	0.069	0.121	0.491	0.066	0.117
rule 5	0.502	0.101	0.163	0.503	0.122	0.191	0.489	0.090	0.147	0.519	0.105	0.168
rule 6	0.426	0.035	0.065	0.433	0.046	0.083	0.417	0.036	0.066	0.398	0.036	0.066
rule 7	0.486	0.044	0.081	0.484	0.053	0.095	0.509	0.044	0.081	0.479	0.039	0.071
rule 8	0.553	0.049	0.085	0.556	0.060	0.103	0.515	0.049	0.086	0.553	0.053	0.092
Majority (strict)	0.508	0.389	0.411	0.762	0.700	0.692	0.515	0.392	0.415	0.498	0.381	0.404
Majority (loose)	0.710	0.677	0.663	0.762	0.700	0.692	0.719	0.683	0.671	0.706	0.672	0.659
Table 18: Performance of each rule on GossipCop.
GossipCop
Rule	train	validation	test
Prec	Rec	F1	Prec	Rec	F1	Prec	Rec	F1
rule 1	0.632	0.629	0.627	0.614	0.610	0.607	0.629	0.627	0.625
rule 2	0.648	0.622	0.604	0.643	0.620	0.604	0.658	0.630	0.613
rule 3	0.740	0.731	0.728	0.754	0.746	0.744	0.732	0.726	0.724
majority	0.758	0.732	0.725	0.757	0.728	0.721	0.760	0.740	0.735