SlowGuess/NAACL_2021_OCR huggingface dataset:
Dataset	Task	Class	# Train	# Dev	# Test	Cover	Accuracy
AGNews	Topic	4	96k	12k	12k	56.4	83.1
IMDB	Sentiment	2	20k	2.5k	2.5k	87.5	74.5
Yelp	Sentiment	2	30.4k	3.8k	3.8k	82.8	71.5
MIT-R	Slot Filling	9	6.6k	1.0k	1.5k	13.5	80.7
TREC	Question	6	4.8k	0.6k	0.6k	95.0	63.8
Chemprot	Relation	10	12.6k	1.6k	1.6k	85.9	46.5
WiC	WSD	2	5.4k	0.6k	1.4k	63.4	58.8
Table 2: Dataset statistics. Here cover (in %) is the fraction of instances covered by weak supervision sources in the training set, and accuracy (in %) is the precision of weak supervision.
Method	AGNews	IMDB	Yelp	MIT-R	TREC	Chemprot	WiC (dev)
ExMatch	52.31	71.28	68.68	34.93	60.80	46.52	58.80
Fully-supervised Result
RoBERTa-CL° (Liu et al., 2019)	91.41	94.26	97.27	88.51	96.68	79.65	70.53
Baselines
RoBERTa-WL† (Liu et al., 2019)	82.25	72.60	74.89	70.95	62.25	44.80	59.36
Self-ensemble (Xu et al., 2020)	85.72	86.72	80.08	72.88	66.18	44.62	62.71
FreeLB (Zhu et al., 2020)	85.12	88.04	85.68	73.04	67.33	45.68	63.45
Mixup (Zhang et al., 2018)	85.40	86.92	92.05	73.68	66.83	51.59	64.88
SMART (Jiang et al., 2020)	86.12	86.98	88.58	73.66	68.17	48.26	63.55
Snorkel (Ratner et al., 2020)	62.91	73.22	69.21	20.63	58.60	37.50	-*
WeSTClass (Meng et al., 2018)	82.78	77.40	76.86	-⊗	37.31	-⊗	48.59
ImplyLoss (Awasthi et al., 2020)	68.50	63.85	76.29	74.30	80.20	53.48	54.48
Denoise (Ren et al., 2020)	85.71	82.90	87.53	70.58	69.20	50.56	62.38
UST (Mukherjee and Awadallah, 2020)	86.28	84.56	90.53	74.41	65.52	52.14	63.48
Our COSINE Framework
Init	84.63	83.58	81.76	72.97	65.67	51.34	63.46
COSINE	87.52	90.54	95.97	76.61	82.59	54.36	67.71
$\diamond$  : RoBERTa is trained with clean labels.  $\dagger$  : RoBERTa is trained with weak labels.  $*$  : unfair comparison.  $\otimes$  : not applicable.
Figure 3: Results of label corruption on TREC. When the corruption ratio is less than  $40\%$ , the performance is close to the fully supervised method.
AGNews, IMDB, Yelp, and WiC, our model achieves the same level of performance with models (RoBERTa-CL) trained with clean labels. This makes COSINE appealing in the scenario where only weak supervision is available.
4.2 Robustness Against Label Noise
Our model is robust against excessive label noise. We corrupt certain percentage of labels by randomly changing each one of them to another class. This is a common scenario in crowd-sourcing, where we assume human annotators mis-label each sample with the same probability. Figure 3 summarizes experiment results on the TREC dataset. Com
Table 3: Classification accuracy (in %) on various datasets. We report the mean over three runs.
Model	Dev	Test	#Params
Human Baseline	80.0	-
BERT (Devlin et al., 2019)	-	69.6	335M
RoBERTa (Liu et al., 2019)	70.5	69.9	356M
T5 (Raffel et al., 2019)	-	76.9	11,000M
Semi-Supervised Learning
SenseBERT (Levine et al., 2020)	-	72.1	370M
RoBERTa-WL† (Liu et al., 2019)	72.3	70.2	125M
w/ MT† (Tarvainen and Valpola, 2017)	73.5	70.9	125M
w/ VAT† (Miyato et al., 2018)	74.2	71.2	125M
w/ COSINE†	76.0	73.2	125M
Transductive Learning
Snorkel† (Ratner et al., 2020)	80.5	-	1M
RoBERTa-WL† (Liu et al., 2019)	81.3	76.8	125M
w/ MT† (Tarvainen and Valpola, 2017)	82.1	77.1	125M
w/ VAT† (Miyato et al., 2018)	84.9	79.5	125M
w/ COSINE†	89.5	85.3	125M
Table 4: Semi-supervised Learning on WiC. VAT (Virtual Adversarial Training) and MT (Mean Teacher) are semi-supervised methods.†: has access to weak labels.
pared with advanced fine-tuning and self-training methods (e.g. SMART and UST)<sup>11</sup>, our model consistently outperforms the baselines.
4.3 Semi-supervised Learning
We can naturally extend our model to semi-supervised learning, where clean labels are avail
<sup>11</sup>Note that some methods in Table 3, e.g., ImplyLoss and Denoise, are not applicable to this setting since they require weak supervision sources, but none exists in this setting.
able for a portion of the data. We conduct experiments on the WiC dataset. As a part of the SuperGLUE (Wang et al., 2019a) benchmark, this dataset proposes a challenging task: models need to determine whether the same word in different sentences has the same sense (meaning).
Different from previous tasks where the labels in the training set are noisy, in this part, we utilize the clean labels provided by the WiC dataset. We further augment the original training data of WiC with unlabeled sentence pairs obtained from lexical databases (e.g., WordNet, Wictionary). Note that part of the unlabeled data can be weakly-labeled by rule matching. This essentially creates a semi-supervised task, where we have labeled data, weakly-labeled data and unlabeled data.
Since the weak labels of WiC are generated by WordNet and partially reveal the true label information, Snorkel (Ratner et al., 2020) takes this unfair advantage by accessing the unlabeled sentences and weak labels of validation and test data. To make a fair comparison to Snorkel, we consider the transductive learning setting, where we are allowed access to the same information by integrating unlabeled validation and test data and their weak labels into the training set. As shown in Table 4, COSINE with transductive learning achieves better performance compared with Snorkel. Moreover, in comparison with semi-supervised baselines (i.e. VAT and MT) and fine-tuning methods with extra resources (i.e., SenseBERT), COSINE achieves better performance in both semi-supervised and transductive learning settings.
Error propagation mitigation and wrong-label correction. Figure 4 visualizes this process. Before training, the semantic rules make noisy predictions. After the initialization step, model predictions are less noisy but more biased, e.g., many samples are mis-labeled as "Amenity". These predictions are further refined by contrastive self-training. The rightmost figure demonstrates wrong-label correction. Samples are indicated by radii of the circle, and classification correctness is indicated by color, i.e., blue means correct and orange means incorrect. From inner to outer tori specify classification accuracy after the initialization stage, and the iteration 1,2,3. We can see that many incorrect predictions are corrected within three iterations. To illustrate: the right black dashed line means the corresponding sample is classified correctly after the first iteration,
and the left dashed line indicates the case where the sample is mis-classified after the second iteration but corrected after the third. These results demonstrate that our model can correct wrong predictions via contrastive self-training.
Better data representations. We visualize sample embeddings in Fig. 7. By incorporating the contrastive regularizer  $\mathcal{R}_1$ , our model learns more compact representations for data in the same class, e.g., the green class, and also extends the inter-class distances, e.g., the purple class is more separable from other classes in Fig. 7(b) than in Fig. 7(a).
Label efficiency. Figure 8 illustrates the number of clean labels needed for the supervised model to outperform COSINE. On both of the datasets, the supervised model requires a significant amount of clean labels (around 750 for Agnews and 120 for MIT-R) to reach the level of performance as ours, whereas our method assumes no clean sample.
Higher Confidence Indicates Better Accuracy. Figure 6 demonstrates the relation between prediction confidence and prediction accuracy on IMDB. We can see that in general, samples with higher prediction confidence yield higher prediction accuracy. With our sample reweighting method, we gradually filter out low-confidence samples and assign higher weights for others, which effectively mitigates error propagation.
Components of COSINE. We inspect the importance of various components, including the contrastive regularizer  $\mathcal{R}_1$ , the confidence regularizer  $\mathcal{R}_2$ , and the sample reweighting (SR) method, and the soft labels. Table 5 summarizes the results and Fig. 9 visualizes the learning curves. We remark that all the components jointly contribute to the model performance, and removing any of them hurts the classification accuracy. For example, sample reweighting is an effective tool to reduce error propagation, and removing it causes the model to eventually overfit to the label noise, e.g., the red bottom line in Fig. 9 illustrates that the classification accuracy increases and then drops rapidly. On the other hand, replacing the soft pseudo-labels (Eq. 3) with the hard counterparts (Eq. 2) causes drops in performance. This is because hard pseudo-labels lose prediction confidence information.
Hyper-parameters of COSINE. In Fig. 5, we examine the effects of different hyper-parameters, including the confidence threshold  $\xi$  (Eq. 9), the
Figure 4: Classification performance on MIT-R. From left to right: visualization of ExMatch, results after the initialization step, results after contrastive self-training, and wrong-label correction during self-training.
(a) Effect of  $\xi$
(b) Effect of  $T_{1}$ .
(c) Effect of  $T_{3}$ .
Figure 6: Accuracy vs. Confidence score.
Figure 5: Effects of different hyper-parameters.
(a) Embedding  $w / o \mathcal{R}_1$ .
(b) Embedding  $w / \mathcal{R}_1$
Figure 7: t-SNE (Maaten and Hinton, 2008) visualization on TREC. Each color denotes a different class.
(a) Results on Agnews.
Figure 8: Accuracy vs. Number of annotated labels.
(b) Results on MIT-R.
stopping time  $T_{1}$  in the initialization step, and the update period  $T_{3}$  for pseudo-labels. From Fig. 5(a), we can see that setting the confidence threshold too big hurts model performance, which is because an over-conservative selection strategy can result in insufficient number of training data. The stopping time  $T_{1}$  has drastic effects on the model. This is because fine-tuning COSINE with weak labels for excessive steps causes the model to unavoidably overfit to the label noise, such that the contrastive self-training procedure cannot correct the
Figure 9: Learning curves on TREC with different settings. Mean and variance are calculated over 3 runs.
Method	AGNews	IMDB	Yelp	MIT-R	TREC
Init	84.63	83.58	81.76	72.97	66.50
COSINE	87.52	90.54	95.97	76.61	82.59
w/o R1	86.04	88.32	94.64	74.11	78.28
w/o R2	85.91	89.32	93.96	75.21	77.11
w/o SR	86.72	87.10	93.08	74.29	79.77
w/o R1/R2	86.33	84.44	92.34	73.67	76.95
w/o R1/R2/SR	86.61	83.98	82.57	73.59	74.96
w/o Soft Label	86.07	89.72	93.73	73.05	71.91
Table 5: Effects of different components. Due to space limit we only show results for 5 representative datasets.
error. Also, with the increment of  $T_{3}$ , the update period of pseudo-labels, model performance first increases and then decreases. This is because if we update pseudo-labels too frequently, the contrastive self-training procedure cannot fully suppress the label noise, and if the updates are too infrequent, the pseudo-labels cannot capture the updated information well.
Fine-tuning Pre-trained Language Models. To improve the model's generalization power during fine-tuning stage, several methods are proposed (Peters et al., 2019; Dodge et al., 2020; Zhu et al., 2020; Jiang et al., 2020; Xu et al., 2020; Kong et al., 2020; Zhao et al., 2020; Gunel et al., 2021; Zhang et al., 2021; Aghajanyan et al., 2021; Wang et al., 2021), However, most of these methods focus on fully-supervised setting and rely heavily on large amounts of clean labels, which are not always available. To address this issue, we propose a contrastive self-training framework that fine-tunes pre-trained models with only weak labels. Compared with the existing fine-tuning approaches (Xu et al., 2020; Zhu et al., 2020; Jiang et al., 2020), our model effectively reduces the label noise, which achieves better performance on various NLP tasks with weak supervision.
Learning From Weak Supervision. In weakly-supervised learning, the training data are usually noisy and incomplete. Existing methods aim to denoise the sample labels or the labeling functions by, for example, aggregating multiple weak supervisions (Ratner et al., 2020; Lison et al., 2020; Ren et al., 2020), using clean samples (Awasthi et al., 2020), and leveraging contextual information (Mekala and Shang, 2020). However, most of them can only use specific type of weak supervision on specific task, e.g., keywords for text classification (Meng et al., 2020; Mekala and Shang, 2020), and they require prior knowledge on weak supervision sources (Awasthi et al., 2020; Lison et al., 2020; Ren et al., 2020), which somehow limits the scope of their applications. Our work is orthogonal to them since we do not denoise the labeling functions directly. Instead, we adopt contrastive self-training to leverage the power of pretrained language models for denoising, which is task-agnostic and applicable to various NLP tasks with minimal additional efforts.
Adaptation of LMs to Different Domains. When fine-tuning LMs on data from different domains, we can first continue pre-training on in-domain text data for better adaptation (Gururan et al., 2020). For some rare domains where BERT trained on general domains is not optimal, we can use LMs pretrained on those specific domains (e.g. BioBERT (Lee et al., 2020), SciBERT (Beltagy
et al., 2019)) to tackle this issue.
Scalability of Weak Supervision. COSINE can be applied to tasks with a large number of classes. This is because rules can be automatically generated beyond hand-crafting. For example, we can use label names/description as weak supervision signals (Meng et al., 2020). Such signals are easy to obtain and do not require hand-crafted rules. Once weak supervision is provided, we can create weak labels to further apply COSINE.
Flexibility. COSINE can handle tasks and weak supervision sources beyond our conducted experiments. For example, other than semantic rules, crowd-sourcing can be another weak supervision source to generate pseudo-labels (Wang et al., 2013). Moreover, we only conduct experiments on several representative tasks, but our framework can be applied to other tasks as well, e.g., namedentity recognition (token classification) and reading comprehension (sentence pair classification).
In this paper, we propose a contrastive regularized self-training framework, COSINE, for finetuning pre-trained language models with weak supervision. Our framework can learn better data representations to ease the classification task, and also efficiently reduce label noise propagation by confidence-based reweighting and regularization. We conduct experiments on various classification tasks, including sequence classification, token classification, and sentence pair classification, and the results demonstrate the efficacy of our model.
COSINE is a general framework that tackled the label scarcity issue via combining neural nets with weak supervision. The weak supervision provides a simple but flexible language to encode the domain knowledge and capture the correlations between features and labels. When combined with unlabeled data, our framework can largely tackle the label scarcity bottleneck for training DNNs, enabling them to be applied for downstream NLP classification tasks in a label efficient manner.
COSINE neither introduces any social/ethical bias to the model nor amplifies any bias in the data. In all the experiments, we use publicly available data, and we build our algorithms using public code bases. We do not foresee any direct social consequences or ethical issues.
We thank anonymous reviewers for their feedbacks. This work was supported in part by the National Science Foundation award III-2008334, Amazon Faculty Award, and Google Faculty Award.
Armen Aghajanyan, Akshit Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2021. Better fine-tuning by reducing representational collapse. In International Conference on Learning Representations.
Laura Aina, Kristina Gulordava, and Gemma Boleda. 2019. Putting words in context: LSTM language models and lexical ambiguity. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3342-3348.
Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules generalizing labeled exemplars. In International Conference on Learning Representations.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615-3620.
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages 539-546.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Jesse Dodge, Gabriel Ilharco, Roy Schwartz, Ali Farhadi, Hannaneh Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping. CoRR, abs/2002.06305.
Yarin Gal and Zoubin Ghahramani. 2015. Bayesian convolutional neural networks with bernoulli approximate variational inference. CoRR, abs/1506.02158.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. 2021. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Representations.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020. SMART: Robust and efficient fine-tuning for pretrained natural language models through principled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190.
Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. 2020. Calibrated language model fine-tuning for in- and out-of-distribution data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1326-1340.
Martin Krallinger, Obdulia Rabal, Saber A Akhondi, et al. 2017. Overview of the biocreative VI chemical-protein interaction track. In Proceedings of the sixth BioCreative challenge evaluation workshop, volume 1, pages 141-146.
Dong-Hyun Lee. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, page 2.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-1240.
Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, and Yoav Shoham. 2020. SenseBERT: Driving some sense into BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4656-4667.
Chen Liang, Yue Yu, Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. 2020. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, page 1054-1064.
Pierre Lison, Jeremy Barnes, Aliaksandr Hubin, and Samia Touileb. 2020. Named entity recognition without labelled data: A weak supervision approach. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1518-1533.
Jingjing Liu, Panupong Pasupat, Yining Wang, Scott Cyphers, and James R. Glass. 2013. Query understanding enhanced by hierarchical parsing structures. In 2013 IEEE Workshop on Automatic
Speech Recognition and Understanding, pages 72-77. IEEE.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Bingfeng Luo, Yansong Feng, Zheng Wang, Zhanxing Zhu, Songfang Huang, Rui Yan, and Dongyan Zhao. 2017. Learning with noise: Enhance distantly supervised relation extraction with dynamic transition matrix. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 430-439.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Dheeraj Mekala and Jingbo Shang. 2020. Contextualized weak supervision for text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 323-333.
Yu Meng, Jiaming Shen, Chao Zhang, and Jiawei Han. 2018. Weakly-supervised neural text classification. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, page 983-992.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9006-9017.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. 2018. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 41(8):1979-1993.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020. Uncertainty-aware self-training for text classification with few labels. CoRR, abs/2006.15315.
Gabriel Pereyra, George Tucker, Jan Chorowski, Lukasz Kaiser, and Geoffrey E. Hinton. 2017. Regularizing neural networks by penalizing confident output distributions. CoRR, abs/1701.06548.