SlowGuess/NAACL_2022_OCR huggingface dataset:
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
M. Pawan Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. In Advances in Neural Information Processing Systems 23: 24th Annual Conference on Neural Information Processing Systems 2010. Proceedings of a meeting held 6-9 December 2010, Vancouver, British Columbia, Canada, pages 1189-1197. Curran Associates, Inc.
Samuli Laine and Timo Aila. 2017. Temporal ensembling for semi-supervised learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, volume 3, page 2.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database, 2016.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. BOND: bert-assisted open-domain named entity recognition with distant supervision. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pages 1054-1064. ACM.
Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and Jim Glass. 2013. Query understanding enhanced by hierarchical parsing structures. In IEEE ASRU Workshop. IEEE.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Tomasz Malisiewicz, Abhinav Gupta, and Alexei A. Efros. 2011. Ensemble of exemplar-svms for object detection and beyond. In IEEE International Conference on Computer Vision, ICCV 2011, Barcelona, Spain, November 6-13, 2011, pages 89-96. IEEE Computer Society.
Julian J. McAuley and Jure Leskovec. 2013. Hidden factors and hidden topics: understanding rating dimensions with review text. In Seventh ACM Conference on Recommender Systems, RecSys '13, Hong Kong, China, October 12-16, 2013, pages 165-172. ACM.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018, pages 983-992. ACM.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9006-9017, Online. Association for Computational Linguistics.
Hossein Mobahi, Mehrdad Farajtabar, and Peter L. Bartlett. 2020. Self-distillation amplifies regularization in hilbert space. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020. Uncertainty-aware self-training for few-shot text classification. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yilin Niu, Fangkai Jiao, Mantong Zhou, Ting Yao, Jingfang Xu, and Minlie Huang. 2020. A self-training method for machine reading comprehension with soft evidence extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3916-3927, Online. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang,
Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035.
Hieu Pham, Qizhe Xie, Zihang Dai, and Quoc V Le. 2020. Meta pseudo labels. ArXiv preprint, abs/2003.10580.
Antti Rasmus, Mathias Berglund, Mikko Honkala, Harri Valpola, and Tapani Raiko. 2015. Semi-supervised learning with ladder networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 3546-3554.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009), pages 147-155, Boulder, Colorado. Association for Computational Linguistics.
Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. 2020. Denoising multi-source weak supervision for neural text classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3739-3754, Online. Association for Computational Linguistics.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005. Semi-supervised self-training of object detection models. In WACV/MOTION, pages 29-36.
Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 629-640, New Orleans, Louisiana. Association for Computational Linguistics.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. 2008. Collective classification in network data. AI magazine, 29(3).
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2054-2064, Brussels, Belgium. Association for Computational Linguistics.
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing
Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1195-1204.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Vikas Verma, Meng Qu, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. 2019. Graphmix: Regularized training of graph neural networks for semi-supervised learning. ArXiv preprint, abs/1909.11715.
Heinrich Von Stackelberg. 2010. Market structure and equilibrium. Springer Science & Business Media.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv preprint, abs/1910.03771.
Junyuan Xie, Ross B. Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 478-487. JMLR.org.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoc Le. 2020. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, and Chao Zhang. 2021. Fine-tuning pretrained language model with weak supervision: A contrastive-regularized self-training approach. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1063-1077, Online. Association for Computational Linguistics.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 649-657.
Yan Zhou, Murat Kantarcioglu, and Bhavani Thuraisingham. 2012. Self-training with selection-by-rejection. In ICDM.
Simiao Zuo, Chen Liang, Haoming Jiang, Xiaodong Liu, Pengcheng He, Jianfeng Gao, Weizhu Chen, and Tuo Zhao. 2021. Adversarial training as stackelberg game: An unrolled optimization approach. In EMNLP.
A Semi-Supervised Learning on Graphs
Datasets. We adopt three citation networks: Cora, Citeseer, and Pubmed (Sen et al., 2008) as benchmark datasets. Their statistics are summarized in Table 6. Similar to semi-supervised text classification tasks, for each dataset, we randomly sample  $N \in \{10, 20, 50, 100\}$  data points from each class and annotate them with clean labels, while the other data are treated as unlabeled. We use the same development and test sets for all the splits of a particular dataset.
Baselines. In addition to Self-training, we adopt four graph neural network methods as baselines. Note that Self-training uses GCN as its backbone.
$\diamond$  GCN (Kipf and Welling, 2017) adopts graph convolutions as an information propagation operator on graphs. The operator smooths label information over the graph, such that labeled nodes acknowledge features of unlabeled ones, and predictions are drawn accordingly.
$\diamond$  GraphVAT (Feng et al., 2019) leverages virtual adversarial training on graphs. The method generates perturbations to each data point, and promotes smooth predictions subject to the perturbations.
$\diamond$  GraphMix (Verma et al., 2019) is an interpolation-based regularization method. It uses a manifold mixup approach to learning more discriminative node representations.
$\diamond$  GRAND (Feng et al., 2020) performs data augmentation via a random propagation strategy. It also leverages a consistency regularization to encourage prediction consistency across different augmentations. GRAND uses a multi layer perception (MLP) as its backbone.
Settings. To demonstrate that differentiable self-training can be effectively combined with different models, we adopt DRIFT to two architectures: GCN, which is a graph convolution-based method; and GRAND, which is a MLP-based method that achieves state-of-the-art performance.
Results. Experimental results are summarized in Table 5. Notice that Self-training outperforms GCN. This is because while GCN only implicitly uses information of the unlabeled nodes, Self-training directly utilizes such information via the pseudo-labels. Furthermore, DRIFT+GCN enhances the performance of Self-training. The other baselines (e.g., GraphVAT, Graphmix, GRAND),
which are refinements and substitutions to the graph convolution operation, outperforms vanilla GCN. By equipping GRAND with differentiable self-training, DRIFT+GRAND achieves the best performance in 10 out of 12 experiments. The performance gain is more pronounced when there are only a few labeled samples, e.g., DRIFT+GRAND improves GRAND by more than  $11\%$  when there are 10 labeled samples per class.
Visualization of learned representations. Figure 5 visualizes the learned representations of Self-training and DRIFT. From Fig. 5a, we can see that Self-training mixes the representations of the red class and the blue class, as indicated in the red box. Such erroneous classification is alleviated by DRIFT (Fig. 5b). On CiteSeer, notice that Self-training generates a meaningless cluster (Fig. 5c), which is a sign that Self-training overfits on the label noise.
(a) Self-training on Cora.
(b) DRIFT on Cora.
(c) Self-training on Citeseer.
(d) DRIFT on Citeseer.
Figure 5: t-SNE plots of Self-training and DRIFT on Cora and Citeseer. Each color denotes a different class.
B Classification and Named Entity Recognition Datasets
Dataset statistics for the classification and named entity recognition tasks are presented in Table 7.
C Weak Supervision Sources
There are two types of semantic rules that we apply as weak supervisions:
- Keyword Rule: HAS  $(\mathbf{x}, \mathbf{L}) \to \mathbf{C}$ . If  $x$  matches one of the words in the list  $L$ , we label it as  $C$ .
Dataset	Cora	Citeseer	Pubmed
Labels per class	10	20	50	100	10	20	50	100	10	20	50	100
Baselines
GCN (Kipf and Welling, 2017)	74.5	77.4	81.6	85.1	67.1	69.5	71.9	74.9	71.0	75.1	81.8	84.8
Self-training (Lee, 2013)	74.4	79.1	83.5	85.1	70.5	73.1	75.1	76.2	71.8	75.2	82.5	84.6
GraphVAT (Feng et al., 2019)	75.2	78.6	83.1	85.3	67.6	70.5	72.6	75.8	71.8	75.5	82.1	85.0
GraphMix (Verma et al., 2019)	77.3	82.3	84.8	86.0	67.1	73.9	74.5	76.9	72.9	76.1	81.9	84.4
GRAND (Feng et al., 2020)	76.5	84.3	86.5	87.2	62.8	73.3	75.0	77.8	77.4	78.5	83.9	86.2
Ours
DRIFT+GCN	80.4	81.8	84.6	85.6	74.4	75.4	75.9	77.4	72.8	78.1	83.3	85.3
DRIFT+GRAND	82.1	85.4	87.3	87.9	74.1	76.0	75.7	78.5	79.2	79.3	85.2	86.8
Table 5: Accuracy (in %) of semi-supervised node classification on graphs. For all the splits of a particular dataset, we use the same development and test sets. We report the mean over ten runs. The best results are shown in bold.
Dataset	#Nodes	#Edges	#Class	#Dev	#Test	#Features
Cora	2,708	5,429	7	500	1,000	1,433
CiteSeer	3,327	4,732	6	500	1,000	3,703
Pubmed	19,717	44,338	3	500	1,000	500
Table 6: Statistics of datasets used in semi-supervised learning on graphs.
Dataset	Task	#Class	#Train	#Dev	#Test
AGNews	Topic	4	108k	12k	7.6k
IMDB	Sentiment	2	20k	2.5k	2.5k
Yelp	Sentiment	2	30.4k	3.8k	3.8k
Amazon	Sentiment	2	25k	2.5k	2.5k
MIT-R	Slot Filling	9	6.6k	1.0k	1.5k
CoNLL-03	NER	4	14.0k	3.2k	3.4k
Webpage	NER	4	385	99	135
Wikigold	NER	4	1.1k	280	274
BC5CDR	NER	2	4.5k	4.5k	4.7k
Table 7: Statistics of datasets used in text classification and named entity recognition tasks.
- Pattern Rule: MATCH(x, R)  $\rightarrow$  C. If  $x$  matches the regular expression  $R$ , we label it as  $C$ .
Two examples of semantic rules on AGNews and IMDB are given in Table 8 and Table 9.
All of the weak supervisions, i.e., linguistic rules, are from existing literature. The details are listed below:
- AGNews, IMDB, Yelp: We use the rules in Ren et al. (2020).
- MIT-R: We use the rules in Awasthi et al. (2020).
- CoNLL-03, WebPage, Wikigold: We use the keywords in Liang et al. (2020).
- BC5CDR: We use the keywords in Shang et al. (2018). Note that for simplicity, we do not use AutoPhrase to extract external keywords. Such an approach requires external corpus and extra parameter-tuning.
We use a validation set to tune DRIFT as well as all the baseline methods. We report the test result of the best model on the validation set. All the experimental results have passed a paired t-test with  $p < 0.05$ .
D.1 Baseline Settings
We implement the GraphVAT method by ourselves. For the other baselines, we follow the official release:
(1) MixText: https://github.com/GT-SALT/MixText/;
(2) BOND: https://github.com/ cliang1453/BOND;
(3) UAST: https://github.com/microsoft/UST;
(4) WeSTClass: https://github.com/yumeng5/WeSTClass;
(5) GCN: https://github.com/tkipf/pygcn;
(6) GRAND: https://github.com/THUDM/GRAND;
(7) GraphMix: https://github.com/vikasverma1077/GraphMix.
D.2 Weakly-Supervised Text Classification
Hyper-parameters are shown in Table 10.
D.3 Semi-Supervised Text Classification
We implement TextCNN with Pytorch (Paszke et al., 2019). We use the pre-trained 300 dimension FastText embeddings as the input vectors. Then, we set the filter window sizes to 2, 3, 4, 5 with 500
feature maps each. We train the model for 100 iterations as initialization, and set  $T = 1000$  during self-training. We use Stochastic Gradient Descent (SGD) with momentum  $m = 0.9$  and we set the learning rate to  $5 \times 10^{-4}$ . We set the dropout rate to 0.5 for the linear layers after the CNN. We tune the weight decay in  $[10^{-4}, 10^{-5}, 10^{-6}, 10^{-7}]$ .
Hyper-parameters are shown in Table 11.
D.4 Semi-Supervised Learning on Graphs
Our method serves as an efficient drop-in module to existing methods. There are only two parameters that we tune in the experiments: the exponential moving average rate  $\alpha$  and the temperature  $\tau$  of the soft pseudo-labels. For all the three datasets, we set  $\alpha = 0.99$ . For the temperature parameter, we use the following settings.
- Cora:  $1 / \tau = 3.0$  for GRAND and  $1 / \tau = 4.0$  for GCN.
- CiteSeer:  $1 / \tau = 3.0$  for GRAND and  $1 / \tau = 3.5$  for GCN.
- Pubmed:  $1 / \tau = 3.0$  for GRAND and  $1 / \tau = 4.0$  for GCN.
3We use the 1 million word vectors trained on Wikipedia 2017, UMBC webbase corpus and news dataset, which is available online: https://fasttext.cc/docs/en/english-vectors.html.
Hyper-parameter	AGNews	IMDB	Yelp	MIT-R	CoNLL-03	Webpage	Wikigold	BC5CDR
Dropout Ratio	0.1
Maximum Tokens	128	256	512	64	128	128	128	128
Batch Size	32	16	16	64	32	32	32	32
Weight Decay	\(10^{-4}\)
Learning Rate	\(10^{-5}\)
Initialization Steps	160	160	200	150	900	300	3500	1500
T	3000	2500	2500	1000	1800	200	700	1000
α	0.95	0.9	0.95	0.9	0.9	0.95	0.9	0.9
τ	0.5	0.5	0.5	0.5	0.5	0.5	0.5	0.5
Table 10: Hyper-parameter configurations for weakly-supervised text classification.
Hyper-parameter	AGNews	IMDB	Amazon
Dropout Ratio	0.1
Maximum Tokens	128	256	256
Batch Size	32	16	16
Weight Decay	10-4
Learning Rate	10-5
Initialization Steps	1200	1000	800
T	4000	3000	4000
α	0.95	0.99	0.9
τ	0.6	0.5	0.5
Table 11: Hyper-parameter configurations for semi-supervised text classification.
Other hyper-parameters and tricks used in training follow the corresponding works.