SlowGuess/NAACL_2021_OCR huggingface dataset:
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7-14.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267-1273.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. CoRR, abs/1910.10683.
Alexander Ratner, Stephen H. Bach, Henry R. Ehrenberg, Jason A. Fries, Sen Wu, and Christopher Ré. 2020. Snorkel: rapid training data creation with weak supervision. VLDB Journal, 29(2):709-730.
Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. 2020. Denoising multi-source weak supervision for neural text classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3739-3754.
Chuck Rosenberg, Martial Hebert, and Henry Schneiderman. 2005. Semi-supervised self-training of object detection models. WACV/MOTION, 2.
Yaniv Taigman, Ming Yang, Marc'Aurelio Ranzato, and Lior Wolf. 2014. Deepface: Closing the gap to human-level performance in face verification. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems, pages 1195-1204.
Paroma Varma and Christopher Re. 2018. Snuba: Automating weak supervision to label training data. Proc. VLDB Endowment, 12(3):223-236.
Ellen M. Voorhees and Dawn M. Tice. 1999. The TREC-8 question answering track evaluation. In Proceedings of The Eighth Text REtrieval Conference, volume 500-246.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261-3275.
Aobo Wang, Cong Duy Vu Hoang, and Min-Yen Kan. 2013. Perspectives on crowdsourcing annotations for natural language processing. Language resources and evaluation, 47(1):9-31.
Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jia, Bo Li, and Jingjing Liu. 2021. InfoBERT: Improving robustness of language models from an information theoretic perspective. In International Conference on Learning Representations.
Hao Wang, Bing Liu, Chaozhuo Li, Yan Yang, and Tianrui Li. 2019b. Learning with noisy labels for sentence-level sentiment classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6286-6292.
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In Proceedings of The 33rd International Conference on Machine Learning, pages 478-487.
Qizhe Xie, Zihang Dai, Eduard H. Hovy, Minh-Thang Luong, and Quoc V. Le. 2019. Unsupervised data augmentation for consistency training. CoRR, abs/1904.12848.
Yige Xu, Xipeng Qiu, Ligao Zhou, and Xuanjing Huang. 2020. Improving BERT fine-tuning via self-ensemble and self-distillation. CoRR, abs/2002.10345.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. 2021. Revisiting few-sample BERT fine-tuning. In International Conference on Learning Representations.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems 28, pages 649-657.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226-2241.
Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang, Junyi Du, Leonardo Neves, and Xiang Ren. 2020. NERO: A neural rule grounding framework
for label-efficient relation extraction. In Proceedings of The Web Conference 2020, page 2166-2176.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2020. Freelb: Enhanced adversarial training for natural language understanding. In International Conference on Learning Representations.
A Weak Supervision Details
COSINE does not require any human annotated examples in the training process, and it only needs weak supervision sources such as keywords and semantic rules. According to some studies in existing works Awasthi et al. (2020); Zhou et al. (2020), such weak supervisions are cheap to obtain and are much efficient than collecting clean labels. In this way, we can obtain significantly more labeled examples using these weak supervision sources than human labor.
$\diamond$  Keyword Rule: HAS  $(\mathbf{x},\mathrm{L})\to \mathrm{C}$  .If  $x$  matches one of the words in the list  $L$  , we label it as  $C$
$\diamond$  Pattern Rule: MATCH  $(\mathbf{x},\mathbb{R})\to \mathbb{C}$  .If  $x$  matches the regular expression  $R$  , we label it as  $C$
In addition to the keyword rule and the pattern rule, we can also use third-party tools to obtain weak labels. These tools (e.g. TextBlob $^{12}$ ) are available online and can be obtained cheaply, but their prediction is not accurate enough (when directly use this tool to predict label for all training samples, the accuracy on Yelp dataset is around  $60\%$ ).
We now introduce the semantic rules on each dataset:
$\diamond$  AGNews, IMDB, Yelp: We use the rule in Ren et al. (2020). Please refer to the original paper for detailed information on rules.
$\diamond$  MIT-R, TREC: We use the rule in Awasthi et al. (2020). Please refer to the original paper for detailed information on rules.
$\diamond$  ChemProt: There are 26 rules. We show part of the rules in Table 6.
$\diamond$  WiC: Each sense of each word in WordNet has example sentences. For each sentence in the WiC dataset and its corresponding keyword, we collect the example sentences of that word from WordNet. Then for a pair of sentences, the corresponding weak label is "True" if their definitions are the same, otherwise the weak label is "False".
B Baseline Settings
We implement Self-ensemble, FreeLB, Mixup and UST based on their original paper. For other baselines, we use their official release:
WeSTClass (Meng et al., 2018): https://github.com/yumeng5/WeSTClass.
RoBERTa (Liu et al., 2019): https://github.com/huggingface/transformers.
SMART (Jiang et al., 2020): https://github.com/namisan/mt-dnn.
Snorkel (Ratner et al., 2020): https://www-snorkel.org/.
$\diamond$  ImplyLoss (Awasthi et al., 2020): https://github.com/awasthiabhijeet/Learning-From-Rules.
Denoise (Ren et al., 2020): https://github.com/weakrules/
Denoise-multi-weak-sources.
C Details on Experiment Setups
C.1 Computing Infrastructure
System: Ubuntu 18.04.3 LTS; Python 3.7; Pytorch 1.2. CPU: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz. GPU: GeForce GTX TITAN X.
C.2 Hyper-parameters
We use AdamW (Loshchilov and Hutter, 2019) as the optimizer, and the learning rate is chosen from  $1 \times 10^{-5}, 2 \times 10^{-5}, 3 \times 10^{-5}$ . A linear learning rate decay schedule with warm-up 0.1 is used, and the number of training epochs is 5.
Hyper-parameters are shown in Table 7. We use a grid search to find the optimal setting for each task. Specifically, we search  $T_{1}$  from 10 to 2000,  $T_{2}$  from 1000 to 5000,  $T_{3}$  from 10 to 500,  $\xi$  from 0 to 1, and  $\lambda$  from 0 to 0.5. All results are reported as the average over three runs.
C.3 Number of Parameters
COSINE and most of the baselines (RoBERTaWL / RoBERTa-CL / SMART / WeSTClass / SelfEnsemble / FreeLB / Mixup / UST) are built on the RoBERTa-base model with about 125M parameters. Snorkel is a generative model with only a few parameters. ApplyLoss and Denoise freezes the embedding and has less than 1M parameters. However, these models cannot achieve satisfactory performance in our experiments.
12https://textblob.readthedocs.io/en/ dev/index.html.
Rule	Example
HAS (x, [amino acid,mutant, mutat, replace]) → part_of	A major part of this processing requires endoproteolytic cleavage at spe-cific pairs of basic [CHEMICAL]amino acid[CHEMICAL] residues, an event necessary for the maturation of a variety of important bio-logically active proteins, such as insulin and [GENE]nerve growth factor[GENE].
HAS (x, [bind, interact, affinit]) → regulator	The interaction of [CHEMICAL]naloxone estrone azine[CHEMICAL](N-EH) with various [GENE]opioid receptor[GENE] types was studied in vitro.
HAS (x, [activat, increas, induc, stimulat, upregulat]) → upregulator/activator	The results of this study suggest that [CHEMI-CAL]noradrenaline[CHEMICAL] predominantly, but not exclusively, mediates contraction of rat aorta through the activation of an [GENE]alphalD-adrenoceptor[GENE].
HAS (x, [downregulat, inhibit, reduc, decreas]) → downregulator/inhibitor	These results suggest that [CHEMICAL]prostacyclin[CHEMICAL] may play a role in downregulating [GENE]tissue factor[GENE] expression in monocytes, at least in part via elevation of intracellular levels of cyclic AMP.
HAS (x, [agoni, tagoni]*) → agonist * (note the leading whitespace in both cases)	Alprenolol and BAAM also caused surmountable antagonism of [CHEMICAL]isoprenaline[CHEMICAL] responses, and this [GENE]beta 1-adrenoceptor[GENE] antagonism was slowly reversible.
HAS (x, [antagon]) → antagonist	It is concluded that [CHEMICAL]labetalol[CHEMICAL] and dilevalol are [GENE]beta 1-adrenoceptor[GENE] selective antagonists.
HAS (x, [modulat, allosteric]) → modulator	[CHEMICAL]Hydrogen sulfide[CHEMICAL] as an allosteric modu-lator of [GENE]ATP-sensitive potassium channels[GENE] in colonic inflammation.
HAS (x, [cofactor]) → cofactor	The activation appears to be due to an increase of [GENE]GAD[GENE] affinity for its cofactor, [CHEMICAL]pyridoxal phos-phate[CHEMICAL](PLP).
HAS (x, [substrate, catalyz, transport, produc, conver]) → substrate/product	Kinetic constants of the mutant [GENE]CrAT[GENE] showed modi-fication in favor of longer [CHEMICAL]acyl-CoAs[CHEMICAL] as substrates.
HAS (x, [not]) → not	[CHEMICAL]Nicotine[CHEMICAL] does not account for the CSE stimulation of [GENE]VEGF[GENE] in HFL-1.
Table 6: Examples of semantic rules on Chemprot.
Hyper-parameter	AGNews	IMDB	Yelp	MIT-R	TREC	Chemprot	WiC
Dropout Ratio	0.1
Maximum Tokens	128	256	512	64	64	400	256
Batch Size	32	16	16	64	16	24	32
Weight Decay	10-4
Learning Rate	10-5	10-5	10-5	10-5	10-5	10-5	10-5
T1	160	160	200	150	500	400	1700
T2	3000	2500	2500	1000	2500	1000	3000
T3	250	50	100	15	30	15	80
ξ	0.6	0.7	0.7	0.2	0.3	0.7	0.7
λ	0.1	0.05	0.05	0.1	0.05	0.05	0.05
Table 7: Hyper-parameter configurations. Note that we only keep certain number of tokens.
D Early Stopping and Earlier Stopping
Our model adopts the earlier stopping strategy during the initialization stage. Here we use "earlier stopping" to differentiate from "early stopping", which is standard in fine-tuning algorithms. Early stopping refers to the technique where we stop training when the evaluation score drops. Earlier stopping is self-explanatory, namely we fine-tune
the pre-trained LMs with only a few steps, even before the evaluation score starts dropping. This technique can efficiently prevent the model from overfitting. For example, as Figure 5(b) illustrates, on IMDB dataset, our model overfits after 240 iterations of initialization with weak labels. In contrast, the model achieves good performance even after 400 iterations of fine-tuning when using clean labels. This verifies the necessity of earlier stopping.
Distance d	Euclidean	Cos
Similarity W	Hard	KL-based	L2-based	Hard	KL-based	L2-based
AGNews	87.52	86.44	86.72	87.34	86.98	86.55
MIT-R	76.61	76.68	76.49	76.55	76.76	76.58
Table 8: Performance of COSINE under different settings.
E Comparison of Distance Measures in Contrastive Learning
The contrastive regularizer  $\mathcal{R}_1(\theta; \widetilde{\boldsymbol{y}})$  is related to two designs: the sample distance metric  $d_{ij}$  and the sample similarity measure  $W_{ij}$ . In our implementation, we use the scaled Euclidean distance as the default for  $d_{ij}$  and Eq. 5 as the default for  $W_{ij}$ [13]. Here we discuss other designs.
E.1 Sample distance metric  $d$
Given the encoded vectorized representations  $\boldsymbol{v}_i$  and  $\boldsymbol{v}_j$  for samples  $i$  and  $j$ , we consider two distance metrics as follows.
Scaled Euclidean distance (Euclidean): We calculate the distance between  $\mathbf{v}_i$  and  $\mathbf{v}_j$  as
$$
d _ {i j} = \frac {1}{d} \| \boldsymbol {v} _ {i} - \boldsymbol {v} _ {j} \| _ {2} ^ {2}. \tag {13}
$$
Cosine distance  $(\mathbf{COS})^{14}$ : Besides the scaled Euclidean distance, cosine distance is another widely-used distance metric:
$$
d _ {i j} = 1 - \cos (\boldsymbol {v} _ {i}, \boldsymbol {v} _ {j}) = 1 - \frac {\| \boldsymbol {v} _ {i} \cdot \boldsymbol {v} _ {j} \|}{\| \boldsymbol {v} _ {i} \| \| \boldsymbol {v} _ {j} \|}. \tag {14}
$$
E.2 Sample similarity measures  $W$
Given the soft pseudo-labels  $\widetilde{\mathbf{y}}_i$  and  $\widetilde{\mathbf{y}}_j$  for samples  $i$  and  $j$ , the following are some designs for  $W_{ij}$ . In all of the cases,  $W_{ij}$  is scaled into range  $[0,1]$  (we set  $\gamma = 1$  in Eq. 7 for the hard similarity).
Hard Similarity: The hard similarity between two samples is calculated as
$$
W _ {i j} = \left\{ \begin{array}{l l} 1, & \text {i f} \underset {k \in \mathcal {Y}} {\operatorname {a r g m a x}} [ \widetilde {\boldsymbol {y}} _ {i} ] _ {k} = \underset {k \in \mathcal {Y}} {\operatorname {a r g m a x}} [ \widetilde {\boldsymbol {y}} _ {j} ] _ {k}, \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {15}
$$
This is called a "hard" similarity because we obtain a binary label, i.e., we say two samples are similar if their corresponding hard pseudo-labels are the same, otherwise we say they are dissimilar.
Soft KL-based Similarity: We calculate the similarity based on KL distance as follows.
$$
W _ {i j} = \exp \left(- \frac {\beta}{2} \left(\mathcal {D} _ {K L} (\widetilde {\boldsymbol {y}} _ {i} \| \widetilde {\boldsymbol {y}} _ {j}) + \mathcal {D} _ {K L} (\widetilde {\boldsymbol {y}} _ {j} \| \widetilde {\boldsymbol {y}} _ {i})\right)\right), \tag {16}
$$
where  $\beta$  is a scaling factor, and we set  $\beta = 10$  by default.
Soft L2-based Similarity: We calculate the similarity based on L2 distance as follows.
$$
W _ {i j} = 1 - \frac {1}{2} \left\| \widetilde {\boldsymbol {y}} _ {i} - \widetilde {\boldsymbol {y}} _ {j} \right\| _ {2} ^ {2}, \tag {17}
$$
E.3 COSINE under different  $d$  and  $W$ .
We show the performance of COSINE with different choices of  $d$  and  $W$  on Agnews and MIT-R in Table 8. We can see that COSINE is robust to these choices. In our experiments, we use the scaled euclidean distance and the hard similarity by default.
To accelerate contrastive learning, we adopt the doubly stochastic sampling approximation to reduce the computational cost. Specifically, the high confidence samples  $\mathcal{C}$  in each batch  $\mathcal{B}$  yield  $\mathcal{O}(|\mathcal{C}|^2)$  sample pairs, and we sample  $|\mathcal{C}|$  pairs from them.
14We use Cos to distinguish from our model name CO-SINE.