SlowGuess/NAACL_2022_OCR huggingface dataset:
Larger base models achieve better overall performance. We further aggregate statistics across tasks, methods, and base models in Table 3. The bottom row reports the average performance across methods for each base model and leads to a consistent ranking in F1 score among base models: BiLSTM  $\leq$  DistilBERT  $\leq$  BERT-base  $\leq$  RoBERTa-base. Observing higher scores for larger transformer models such as RoBERTa agrees with previous observations (Brown et al., 2020). Interestingly, switching from BERT-base to BERT-large (and from RoBERTa-base to RoBERTa-large) in base model architecture leads to marginal improvement, suggesting the need to explore more effective learning methods leveraging weak supervision.
2For token-level F1, we use the conlleval implementation: https://huggingface.co/metrics/seqeval
Table 2: Main results on WALNUT with F1 score (in %) on all tasks. The rightmost column reports the average F1 score across all tasks. (MLC is not shown for BERT-large and RoBERTa-large due to OOM.)
Method	AGNews	IMDB	Yelp	GossipCop	CoNLL	NCBI	WikiGold	LaptopReview	AVG
BiLSTM (20M parameters)
Full Clean	89.4	83.1	86.4	64.5	31.9	69.9	21.8	62.6	63.7
C	79.5	56.2	59.5	50.8	00.8	58.2	15.8	42.3	45.4
W	78.0	75.2	70.8	62.0	11.1	52.3	02.7	49.4	50.2
Snorkel	79.9	75.4	76.0	61.4	06.7	52.5	02.7	49.4	52.1
C+W	82.0	75.6	70.2	64.1	17.2	56.8	15.7	51.2	52.5
C+Snorkel	82.9	75.4	66.5	62.6	07.7	59.2	10.7	53.8	52.4
GLC	56.5	72.2	63.7	60.5	05.1	58.9	08.7	55.2	47.6
MetaWN	55.2	72.7	65.5	58.2	00.0	53.9	03.4	51.6	45.1
MLC	55.3	72.3	65.7	52.5	00.0	52.5	05.9	51.5	43.7
DistilBERT-base (66M parameters)
Full Clean	92.1	88.8	93.7	75.1	88.6	75.7	79.7	75.8	83.7
C	80.8	71.2	73.1	55.3	51.4	57.7	69.5	53.0	64.0
W	72.2	75.0	70.2	70.8	66.9	62.0	57.4	53.8	66.0
Snorkel	70.2	70.7	65.9	68.4	64.3	62.9	56.3	54.0	65.1
C+W	83.3	74.8	71.5	71.4	66.9	66.2	64.0	57.3	68.5
C+Snorkel	84.3	81.7	81.8	69.1	64.6	67.8	64.4	57.5	71.4
GLC	67.8	74.1	68.1	67.3	72.4	72.8	71.7	66.8	70.1
MetaWN	70.0	74.4	69.3	70.0	65.7	64.2	58.5	58.2	66.3
MLC	70.4	74.3	69.4	69.6	69.2	66.2	58.3	58.0	66.9
BERT-base (110M parameters)
Full Clean	92.5	90.0	74.7	74.7	89.4	78.4	81.1	76.2	82.1
C	82.9	63.8	60.3	57.1	67.3	66.6	71.9	54.6	65.6
W	72.3	75.5	69.6	69.0	67.5	59.5	56.7	55.9	65.8
Snorkel	73.7	72.9	65.6	68.2	65.1	60.9	53.8	56.2	66.0
C+W	80.1	81.8	71.3	68.4	68.4	67.9	65.0	59.2	68.9
C+Snorkel	76.2	82.6	75.3	67.1	65.9	69.9	64.3	59.6	70.1
GLC	68.8	75.7	68.8	68.1	74.7	74.7	70.7	65.8	70.9
MetaWN	72.8	75.2	68.1	69.8	66.9	66.7	58.9	59.2	67.2
MLC	73.0	74.7	70.0	71.3	70.4	68.4	58.5	59.7	68.2
RoBERTa-base (125M parameters)
Full Clean	92.8	92.4	95.9	77.2	91.2	83.1	87.2	80.2	87.5
C	84.1	74.5	70.2	57.4	72.9	72.9	78.2	61.3	71.4
W	66.4	76.1	70.4	71.4	64.9	69.9	64.1	58.9	67.8
Snorkel	71.9	70.1	66.3	69.2	61.2	70.0	61.8	59.7	67.5
C+W	70.6	76.5	70.4	72.2	64.1	74.0	71.6	61.2	68.9
C+Snorkel	74.6	68.2	66.4	71.4	62.2	73.4	72.2	61.6	68.8
GLC	67.6	74.9	69.0	68.0	74.6	79.1	79.6	71.5	73.0
MetaWN	69.6	75.4	69.0	71.8	63.8	69.9	63.5	62.5	68.2
MLC	70.4	74.5	69.9	72.9	68.3	74.3	63.1	63.6	69.6
BERT-large (336M parameters)
Full Clean	92.5	91.4	94.9	73.5	90.2	80.5	82.8	78.9	85.6
C	72.5	65.4	68.4	57.8	67.2	69.7	73.9	51.1	65.8
W	68.5	75.9	70.7	69.3	65.7	62.0	57.1	54.2	65.4
Snorkel	73.3	70.9	65.8	70.0	63.6	67.3	57.2	54.4	66.5
C+W	73.4	74.8	71.8	70.2	66.7	70.7	66.9	55.7	67.6
C+Snorkel	73.6	71.3	65.9	71.3	63.6	69.7	63.4	57.2	67.0
GLC	67.1	74.6	67.3	69.8	71.8	76.1	68.1	65.4	70.0
MetaWN	71.6	74.2	67.0	70.8	64.4	70.1	53.9	45.9	64.7
RoBERTa-large (355M parameters)
Full Clean	93.1	94.4	96.9	78.5	91.3	83.5	87.7	80.4	88.2
C	86.1	69.1	84.8	69.1	76.4	77.7	77.1	60.6	75.1
W	74.3	77.7	70.5	73.2	63.6	67.4	61.2	57.3	68.2
Snorkel	75.5	72.6	67.1	69.3	61.1	68.1	61.0	59.2	67.9
C+W	71.9	77.4	70.6	71.6	63.8	71.2	70.4	60.2	68.5
C+Snorkel	74.0	69.0	66.5	73.7	61.1	71.8	69.1	62.4	68.5
GLC	67.8	75.8	68.7	64.1	56.7	80.0	78.3	68.4	70.0
MetaWN	68.6	66.2	71.1	64.6	63.2	69.5	59.3	61.5	65.5
Rules (no base model)
Rules	61.8	73.9	65.9	73.5	61.3	64.7	52.2	60.0	64.1
Table 3: Average F1 score across the eight tasks in WALNUT. The bottom row computes the average F1 score across tasks and supervision methods. The three right-most columns report the average F1 score across model architectures and all tasks ("All"), document-level tasks ("Doc"), and token-level tasks ("Token").
Method	BiLSTM	DistilBERT	Base Model Architecture	Average Results
BERT	RoBERTa	BERT-large	RoBERTa-large	All	Doc	Token
Full Clean	63.7	83.7	82.1	87.5	85.6	88.2	81.8	86.6	77.0
C	45.4	64.0	65.6	71.4	65.8	75.1	64.5	68.7	60.3
W	50.2	66.0	65.8	67.8	65.4	68.2	63.9	71.9	55.9
Snorkel	52.1	65.1	66.0	67.5	66.5	67.9	64.2	70.4	57.9
C+W	52.5	68.5	68.9	68.9	67.6	68.5	65.8	73.6	58.0
C+Snorkel	52.4	71.4	70.1	68.8	67.0	68.5	66.3	73.0	59.7
GLC	47.6	70.1	70.9	73.0	70.0	70.0	66.9	68.6	65.3
MetaWN	45.1	66.3	67.2	68.2	64.7	65.5	62.8	69.2	56.4
AVG	51.1	69.4	69.6	71.6	69.1	71.5
Table 4: Overall performance gain and gap of all weak supervision methods (Weak Sup, by averaging performance of W, Snorkel, C+W, C+Snorkel, GLC, MetaWN and MLC) against no weak supervision (C) and full clean training. Note that RoBERTa-large in included here, as the standard deviation of its performance with different splits on tasks varies significantly (See Table 14 in Appendix) hence using its performance mean as an indicator is less conclusive.
BiLSTM	DistilBERT	BERT	RoBERTa	BERT-large	AVG
Perf. gain: Weak Sup – C	3.90	4.21	2.98	-2.07	1.32	2.07
Perf. gap: Full Clean – Weak Sup	14.42	15.47	13.58	18.13	18.51	16.00
Weak supervision has smaller benefit in larger base models. Another question that we attempt to address in WALNUT is on whether weak supervision equally benefits each base model architecture. To quantify such benefit we compare the performance differences between models trained using semi-weak supervision and models trained using clean data only. The "Weak Sup" approach in Table 4 is computed as the average F1 score across all semi-weak supervision methods (C+W, C+Snorkel, GLC, and MetaWN). The performance gap between "Weak Sup" and "C" (training with few clean data only) is smaller for larger models. Additionally, the performance gap between "Full Clean" (full clean data training) and "Weak Sup" approach is larger for larger models. The two above observations highlight that weak supervision has smaller benefit in larger models. An important future research direction is to develop better learning algorithms and improving the effectiveness of weak supervision in larger models.
Analysis of weak rules. For now, we have focused on the evaluation of base models trained using weak labels generated by multiple weak labeling rules. It is interesting also to decouple the base model performance from the rule aggregation
technique (e.g., majority voting, Snorkel) that was used to generate the training labels, which is an essential modeling component for weak supervision. The bottom row in Table 2 ("Rules") reports the test performance of rules computed by taking the majority voting of weak labels on the test instances. (For test instances that are not covered by any rules, a random class is predicted.) Such majority label is available in our pre-processed datasets. Interestingly, "Rules" sometimes outperforms base models trained using weak labels ("W", "Snorkel"). Note however that "Rules" assumes access to all weak labels on the test set, which might not always be available. On the other hand, the base model learns text features beyond the heuristic-based rules and does not require access to rules during test time and thus can be applied for any test instance.
For a more in-depth analysis of the rule quality, WALNUT also supports the analysis of individual rules and multi-source aggregation techniques, such as majority voting or Snorkel. Figure 3 shows a precision-recall scatter plot for each rule on each of the dataset in WALNUT. For instance, in the CoNLL dataset rules vary in characteristics, where most rules have a relatively low recall while there are a few rules that have substantially higher recall than the rest. Across datasets, we observe that rules
(a) AG News.
(b)Yelp.
(c)IMDB.
(d) CoNLL.
(e) NCBI.
(f) WikiGold.
(g) LaptopReview.
Figure 3: Scatterplots of precision-recall for weak supervision rules. Each point corresponds to a rule. (GossipCop is omitted as it contains only three rules.)
have higher precision than recall, as most rules are sparse, i.e., apply to a small number of instances in the dataset (e.g., instances containing a specific keyword). Similar trends are observed on other datasets as well. For detailed descriptions of all weak rules in all datasets, refer to Table 6 - 13 in the appendix.
Motivated by the lack of a unified evaluation platform for semi-weakly supervised learning for low-resource NLU, in this paper we propose a new benchmark WALNUT covering a broad range of data domains to advocate research on leveraging
both weak supervision and few-shot clean supervision. We evaluate a series of different semi-weakly supervised learning methods with different model architecture on both document-level and token-level classification tasks, and demonstrate the utility of weak supervision in real-world NLU tasks. We find that no single semi-weakly supervised learning method wins on WALNUT and there is still gap between semi-weakly supervised learning and fully supervised training. We expect WALNUT to enable systematic evaluations of semi-weakly supervised learning methods and stimulate further research in directions such as more effective learning paradigms leveraging weak supervision.
We would like to thank Yichuan Li for helping adapt several weak supervision baselines for document-level classification tasks, and the reviewers for constructive feedback.
Isabelle Augenstein, Tim Roktaschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885.
Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R Curran. 2009. Named entity recognition in wikipedia. In Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources (People's Web), pages 10-18.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
Jason Fries, Sen Wu, Alex Ratner, and Christopher Ré. 2017. Swellshark: A generative model for biomedical named entity recognition without labeled data. arXiv preprint arXiv:1704.06360.
Chaitanya Gokhale, Sanjib Das, AnHai Doan, Jeffrey F Naughton, Narasimhan Rampalli, Jude Shavlik, and Xiaojin Zhu. 2014. Corleone: Hands-off crowdsourcing for entity matching. In Proceedings of the 2014 ACM SIGMOD international conference on Management of data, pages 601-612.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. 2018. Using trusted data to train deep networks on labels corrupted by severe noise. In NeurIPS.
Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with mace. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120-1130.
Giannis Karamanolakis, Daniel Hsu, and Luis Gravano. 2019. Leveraging just a few keywords for fine-grained aspect detection through weakly supervised co-training. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan. 2021. Self-training with weak supervision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 845-863.
Yinghao Li, Pranav Shetty, Lucas Liu, Chao Zhang, and Le Song. 2021. BERTifying the hidden Markov model for multi-source weakly supervised named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing.
Pierre Lison, Jeremy Barnes, Aliaksandr Hubin, and Samia Touileb. 2020. Named entity recognition without labelled data: A weak supervision approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1518-1533.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150.
Ayush Maheshwari, Oishik Chatterjee, Krishnateja Killamsetty, Ganesh Ramakrishnan, and Rishabh Iyer. 2021. Semi-supervised data programming with subset selection. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021.
Alessio Mazzetto, Cyrus Cousins, Dylan Sam, Stephen H Bach, and Eli Upfal. 2021. Adversarial multi class learning under weak supervision with performance guarantees. In Proceedings of the 38th International Conference on Machine Learning.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 983-992.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011.
Subhabrata Mukherjee, Xiaodong Liu, Guoqing Zheng, Saghar Hosseini, Hao Cheng, Greg Yang, Christopher Meek, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021. Few-shot learning evaluation in natural language understanding. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
George Papandreou, Liang-Chieh Chen, Kevin P Murphy, and Alan L Yuille. 2015. Weakly-and semi-supervised learning of a deep convolutional network for semantic image segmentation. In Proceedings of the IEEE international conference on computer vision, pages 1742-1750.
Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. 2017. Making deep neural networks robust to label noise: A loss correction approach. In CVPR.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237.
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In International workshop on semantic evaluation, pages 19-30.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. 2017. Snorkel: Rapid training data creation with weak supervision. Proceedings of the VLDB Endowment, 11(3):269-282.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. arXiv preprint arXiv:1803.09050.
Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. 2020. Denoising multi-source weak supervision for neural text classification. arXiv preprint arXiv:2010.04582.
Esteban Safranchik, Shiying Luo, and Stephen Bach. 2020. Weakly supervised sequence tagging from noisy rules. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5570-5578.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Jingbo Shang, Jialu Liu, Meng Jiang, Xiang Ren, Clare R Voss, and Jiawei Han. 2018. Automated phrase mining from massive text corpora. IEEE Transactions on Knowledge and Data Engineering, 30(10):1825-1837.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-weight-net: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems, pages 1917-1928.
Kai Shu, Deepak Mahudeswaran, Suhang Wang, Dongwon Lee, and Huan Liu. 2020a. Fakenewsnet: A data repository with news content, social context, and spatiotemporal information for studying fake news on social media. *Big Data*, 8(3):171-188.
Kai Shu, Subhabrata Mukherjee, Guoqing Zheng, Ahmed Hassan Awadallah, Milad Shokouhi, and Susan Dumais. 2020b. Learning with weak supervision for email intent detection. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1051-1060.
Kai Shu, Amy Sliva, Suhang Wang, Jiliang Tang, and Huan Liu. 2017. Fake news detection on social media: A data mining perspective. ACM SIGKDD explorations newsletter, 19(1):22-36.
Kai Shu, Guoqing Zheng, Yichuan Li, Subhabrata Mukherjee, Ahmed Hassan Awadallah, Scott Ruston, and Huan Liu. 2020c. Leveraging multi-source weak social supervision for early detection of fake news. arXiv preprint arXiv:2004.01732.
Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. 2014. Training convolutional networks with noisy labels. arXiv preprint arXiv:1406.2080.
Paroma Varma and Christopher Ré. 2018. Snuba: automating weak supervision to label training data. VLDB.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: a stickier benchmark for general-purpose language understanding systems. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 3266-3280.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Hu Yuan, Huilin Xu, Guoao Wei, Xiang Pan, and Hai Hu. 2021. Few clue: A chinese few-shot learning evaluation benchmark. CoRR, abs/2107.07498.
Wei Xu, Raphael Hoffmann, Le Zhao, and Ralph Grishman. 2013. Filling knowledge base gaps for distant supervision of relation extraction. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross-task generalization in nlp.
Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. 2021. Wrench: A comprehensive benchmark for weak supervision. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649-657.
Guoqing Zheng, Ahmed Hassan Awadallah, and Susan Dumais. 2021. Meta label correction for noisy label learning. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 35.
Limitations and Broader Impact
The proposed benchmark is likely to stimulate research on weakly supervised learning for NLU, and offer the research community on weakly supervised learning a unified testbed for evaluating new methodologies developed for low-resource NLU. For many practical NLU applications, large amount of manually labeled data is unavailable or expensive to obtain due to either cost or privacy concerns, resorting to proxy signals such as weak supervision is a viable solution to mitigate this annotation scarce problem. We hope WALNUT would provide such an evaluation environment to advocate progress in this direction.
Limitations. Due to the lack of existing real-world weak supervision for many NLU tasks, WALNUT does not include NLU tasks such as Natural Language Inference for which it is hard to construct weak supervision rules. Also, currently WALNUT only considers English data; a possible extension is to also include multi-lingual corpus with weak supervision available to boost the performance of multi-lingual language models with weakly supervised learning.
We implement all baseline experiments with PyTorch and each experiment runs on a single NVIDIA GPU. Below are hyper-parameter specifications for all baseline methods (hyperparameters not mentioned below are given default values):
- Full Clean, C, C+W, Snorkel, C+Snorkel: Batch size is 32 for document-level classification datasets and 16 for the token-level classification datasets. The code for Snorkel is adapted from: https://github.com/snorkel-team/snorkel. Each training experiment is conducted for the 10 epochs with the checkpoint with the best validation performance saved for evaluation on the test set.
- GLC: Code is adapted from https://github.com/mmazeika/glc. Batch size is 16 for the 4 document-level classification datasets and 8 for the 4 token-level classification datasets. Each experiment trains for 10 epochs with the checkpoint with the best
validation performance saved for evaluation on test set.