SlowGuess/NAACL_2021_OCR huggingface dataset:
Yue Yu* Simiao Zuo* Haoming Jiang Wendi Ren Tuo Zhao Chao Zhang
Georgia Institute of Technology, Atlanta, GA, USA
{yueyu, simiaozuo, jianghm, wren44, tourzhao, chaozhang}@gatech.edu
Fine-tuned pre-trained language models (LMs) have achieved enormous success in many natural language processing (NLP) tasks, but they still require excessive labeled data in the fine-tuning stage. We study the problem of fine-tuning pre-trained LMs using only weak supervision, without any labeled data. This problem is challenging because the high capacity of LMs makes them prone to overfitting the noisy labels generated by weak supervision. To address this problem, we develop a contrastive self-training framework, COSINE, to enable fine-tuning LMs with weak supervision. Underpinned by contrastive regularization and confidence-based reweighting, our framework gradually improves model fitting while effectively suppressing error propagation. Experiments on sequence, token, and sentence pair classification tasks show that our model outperforms the strongest baseline by large margins and achieves competitive performance with fully-supervised fine-tuning methods. Our implementation is available on https://github.com/yueyu1030/COSINE.
Language model (LM) pre-training and fine-tuning achieve state-of-the-art performance in various natural language processing tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2019). Such approaches stack task-specific layers on top of pre-trained language models, e.g., BERT (Devlin et al., 2019), then fine-tune the models with task-specific data. During fine-tuning, the semantic and syntactic knowledge in the pre-trained LMs is adapted for the target task. Despite their success, one bottleneck for fine-tuning LMs is the requirement of labeled data. When labeled data are scarce, the fine-tuned models often suffer from degraded performance, and the large number of parameters can cause severe overfitting (Xie et al., 2019).
To relieve the label scarcity bottleneck, we fine-tune the pre-trained language models with only weak supervision. While collecting large amounts of clean labeled data is expensive for many NLP tasks, it is often cheap to obtain weakly labeled data from various weak supervision sources, such as semantic rules (Awasthi et al., 2020). For example, in sentiment analysis, we can use rules 'terrible'  $\rightarrow$  Negative (a keyword rule) and '  $\star$  not recommend  $*^{\prime}\rightarrow$  Negative (a pattern rule) to generate large amounts of weak labels.
Fine-tuning language models with weak supervision is nontrivial. Excessive label noise, e.g., wrong labels, and limited label coverage are common and inevitable in weak supervision. Although existing fine-tuning approaches (Xu et al., 2020; Zhu et al., 2020; Jiang et al., 2020) improve LMs' generalization ability, they are not designed for noisy data and are still easy to overfit on the noise. Moreover, existing works on tackling label noise are flawed and are not designed for fine-tuning LMs. For example, Ratner et al. (2020); Varma and Ré (2018) use probabilistic models to aggregate multiple weak supervisions for denoising, but they generate weak-labels in a context-free manner, without using LMs to encode contextual information of the training samples (Aina et al., 2019). Other works (Luo et al., 2017; Wang et al., 2019b) focus on noise transitions without explicitly conducting instance-level denoising, and they require clean training samples. Although some recent studies (Awasthi et al., 2020; Ren et al., 2020) design labeling function-guided neural modules to denoise each sample, they require prior knowledge on weak supervision, which is often infeasible in real practice.
Self-training (Rosenberg et al., 2005; Lee, 2013) is a proper tool for fine-tuning language models with weak supervision. It augments the training set with unlabeled data by generating pseudo-labels for them, which improves the models' generalization power. This resolves the limited coverage issue in
Equal Contribution.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1063-1077 June 6-11, 2021. ©2021 Association for Computational Linguistics
weak supervision. However, one major challenge of self-training is that the algorithm still suffers from error propagation—wrong pseudo-labels can cause model performance to gradually deteriorate.
We propose a new algorithm COSINE<sup>1</sup> that fine-tunes pre-trained LMs with only weak supervision. COSINE leverages both weakly labeled and unlabeled data, as well as suppresses label noise via contrastive self-training. Weakly-supervised learning enriches data with potentially noisy labels, and our contrastive self-training scheme fulfills the denoising purpose. Specifically, contrastive self-training regularizes the feature space by pulling samples with the same pseudo-labels close while pushing samples with different pseudo-labels apart. Such regularization enforces representations of samples from different classes to be more distinguishable, such that the classifier can make better decisions. To suppress label noise propagation during contrastive self-training, we propose confidence-based sample reweighting and regularization methods. The reweighting strategy emphasizes samples with high prediction confidence, which are more likely to be correctly classified, in order to reduce the effect of wrong predictions. Confidence regularization encourages smoothness over model predictions, such that no prediction can be over-confident, and therefore reduces the influence of wrong pseudo-labels.
Our model is flexible and can be naturally extended to semi-supervised learning, where a small set of clean labels is available. Moreover, since we do not make assumptions about the nature of the weak labels, COSINE can handle various types of label noise, including biased labels and randomly corrupted labels. Biased labels are usually generated by semantic rules, whereas corrupted labels are often produced by crowd-sourcing.
Our main contributions are: (1) A contrastive-regularized self-training framework that fine-tunes pre-trained LMs with only weak supervision. (2) Confidence-based reweighting and regularization techniques that reduce error propagation and prevent over-confident predictions. (3) Extensive experiments on 6 NLP classification tasks using 7 public benchmarks verifying the efficacy of CO-SINE. We highlight that our model achieves competitive performance in comparison with fully-supervised models on some datasets, e.g., on the
Yelp dataset, we obtain a  $97.2\%$  (fully-supervised) v.s.  $96.0\%$  (ours) accuracy comparison.
In this section, we introduce weak supervision and our problem formulation.
Weak Supervision. Instead of using human-annotated data, we obtain labels from weak supervision sources, including keywords and semantic rules $^2$ . From weak supervision sources, each of the input samples  $x \in \mathcal{X}$  is given a label  $y \in \mathcal{Y} \cup \{\emptyset\}$ , where  $\mathcal{Y}$  is the label set and  $\emptyset$  denotes the sample is not matched by any rules. For samples that are given multiple labels, e.g., matched by multiple rules, we determine their labels by majority voting. Problem Formulation. We focus on the weakly-supervised classification problems in natural language processing. We consider three types of tasks: sequence classification, token classification, and sentence pair classification. These tasks have a broad scope of applications in NLP, and some examples can be found in Table 1.
Formally, the weakly-supervised classification problem is defined as the following: Given weakly-labeled samples  $\mathcal{X}_l = \{(x_i,y_i)\}_{i = 1}^L$  and unlabeled samples  $\mathcal{X}_u = \{x_j\}_{j = 1}^U$ , we seek to learn a classifier  $f(x;\theta):\mathcal{X}\to \mathcal{Y}$ . Here  $\mathcal{X} = \mathcal{X}_l\cup \mathcal{X}_u$  denotes all the samples and  $\mathcal{V} = \{1,2,\dots ,C\}$  is the label set, where  $C$  is the number of classes.
Our classifier  $f = g \circ \mathrm{BERT}$  consists of two parts: BERT is a pre-trained language model that outputs hidden representations of input samples, and  $g$  is a task-specific classification head that outputs a  $C$ -dimensional vector, where each dimension corresponds to the prediction confidence of a specific class. In this paper, we use RoBERTa (Liu et al., 2019) as the realization of BERT.
The framework of COSINE is shown in Figure 1. First, COSINE initializes the LM with weak labels. In this step, the semantic and syntactic knowledge of the pre-trained LM are transferred to our model. Then, it uses contrastive self-training to suppress label noise propagation and continue training.
The training procedure of COSINE is as follows. Initialization with Weakly-labeled Data. We fine-tune  $f(\cdot ;\theta)$  with weakly-labeled data  $\mathcal{X}_l$  by
<sup>1</sup>Short for Contrastive Self-Training for Fine-Tuning Pre-trained Language Model.
Examples of weak supervisions are in Appendix A.
Figure 1: The framework of COSINE. We first fine-tune the pre-trained language model on weakly-labeled data with early stopping. Then, we conduct contrastive-regularized self-training to improve model generalization and reduce the label noise. During self-training, we calculate the confidence of the prediction and update the model with high confidence samples to reduce error propagation.
Formulation	Example Task	Input	Output
Sequence Classification	Sentiment Analysis, Topic Classification, Question Classification	[x1, ..., xN]	y
Token Classification	Slot Filling, Part-of-speech Tagging, Event Detection	[x1, ..., xN]	[y1, ..., yN]
Sentence Pair Classification	Word Sense Disambiguation, Textual Entailment, Reading Comprehension	[x1, x2]	y
Table 1: Comparison of different tasks. For sequence classification, input is a sequence of sentences, and we output a scalar label. For token classification, input is a sequence of tokens, and we output one scalar label for each token. For sentence pair classification, input is a pair of sentences, and we output a scalar label.
solving the optimization problem
$$
\min  _ {\theta} \frac {1}{| \mathcal {X} _ {l} |} \sum_ {\left(x _ {i}, y _ {i}\right) \in \mathcal {X} _ {l}} \operatorname {C E} \left(f \left(x _ {i}; \theta\right), y _ {i}\right), \tag {1}
$$
where  $\mathrm{CE}(\cdot ,\cdot)$  is the cross entropy loss. We adopt early stopping (Dodge et al., 2020) to prevent the model from overfitting to the label noise. However, early stopping causes underfitting, and we resolve this issue by contrastive self-training.
Contrastive Self-training with All Data. The goal of contrastive self-training is to leverage all data, both labeled and unlabeled, for fine-tuning, as well as to reduce the error propagation of wrongly labelled data. We generate pseudo-labels for the unlabeled data and incorporate them into the training set. To reduce error propagation, we introduce contrastive representation learning (Sec. 3.2) and confidence-based sample reweighting and regularization (Sec. 3.3). We update the pseudo-labels (denoted by  $\widetilde{\pmb{y}}$ ) and the model iteratively. The procedures are summarized in Algorithm 1.
$\diamond$  Update  $\widetilde{y}$  with the current  $\theta$ . To generate the pseudo-label for each sample  $x \in \mathcal{X}$ , one straightforward way is to use hard labels (Lee, 2013)
$$
\widetilde {y} _ {\text {h a r d}} = \underset {j \in \mathcal {Y}} {\operatorname {a r g m a x}} [ f (x; \theta) ] _ {j}. \tag {2}
$$
Notice that  $f(x;\theta)\in \mathbb{R}^{C}$  is a probability vector and  $[f(x;\theta)]_j$  indicates the  $j$ -th entry of it. How-
ever, these hard pseudo-labels only keep the most likely class for each sample and result in the propagation of labeling mistakes. For example, if a sample is mistakenly classified to a wrong class, assigning a 0/1 label complicates model updating (Eq. 4), in that the model is fitted on erroneous labels. To alleviate this issue, for each sample  $x$  in a batch  $\mathcal{B}$ , we generate soft pseudo-labels $^3$  (Xie et al., 2016, 2019; Meng et al., 2020; Liang et al., 2020)  $\widetilde{\boldsymbol{y}} \in \mathbb{R}^C$  based on the current model as
$$
\widetilde {\boldsymbol {y}} _ {j} = \frac {[ f (x ; \theta) ] _ {j} ^ {2} / f _ {j}}{\sum_ {j ^ {\prime} \in \mathcal {Y}} [ f (x ; \theta) ] _ {j ^ {\prime}} ^ {2} / f _ {j ^ {\prime}}}, \tag {3}
$$
where  $f_{j} = \sum_{x^{\prime}\in \mathcal{B}}[f(x^{\prime};\theta)]_{j}^{2}$  is the sum over soft frequencies of class  $j$  . The non-binary soft pseudolabels guarantee that, even if our prediction is inaccurate, the error propagated to the model update step will be smaller than using hard pseudo-labels.  $\diamond$  Update  $\theta$  with the current  $\widetilde{\pmb{y}}$  .We update the model parameters  $\theta$  by minimizing
$$
\mathcal {L} (\theta ; \widetilde {\boldsymbol {y}}) = \mathcal {L} _ {c} (\theta ; \widetilde {\boldsymbol {y}}) + \mathcal {R} _ {1} (\theta ; \widetilde {\boldsymbol {y}}) + \lambda \mathcal {R} _ {2} (\theta), \tag {4}
$$
where  $\mathcal{L}_c$  is the classification loss (Sec. 3.3),  $\mathcal{R}_1(\theta; \widetilde{\boldsymbol{y}})$  is the contrastive regularizer (Sec. 3.2),  $\mathcal{R}_2(\theta)$  is the confidence regularizer (Sec. 3.3), and  $\lambda$  is the hyper-parameter for the regularization.
3More discussions on hard vs柔软 are in Sec. 4.5.
Algorithm 1: Training Procedures of COSINE.
Input: Training samples  $\mathcal{X}$ ; Weakly labeled samples  $\mathcal{X}_l \subseteq \mathcal{X}$ ; Pre-trained LM  $f(\cdot; \theta)$ .
// Fine-tune the LM with weakly-labeled data.
for  $t = 1,2,\dots ,T_{1}$  do
Sample a minibatch  $\mathcal{B}$  from  $\mathcal{X}_l$ .
Update  $\theta$  by Eq. 1 using AdamW.
// Conduct contrastive self-training with all data.
for  $t = 1,2,\dots ,T_2$  do
Update pseudo-labels  $\widetilde{\pmb{y}}$  by Eq. 3 for all  $x\in \mathcal{X}$
for  $k = 1,2,\dots ,T_{3}$  do
Sample a minibatch  $\mathcal{B}$  from  $\mathcal{X}$ .
Select high confidence samples  $\mathcal{C}$  by Eq. 9.
Calculate  $\mathcal{L}_c$  by Eq. 10,  $\mathcal{R}_1$  by Eq. 6,  $\mathcal{R}_2$  by
Eq. 12, and  $\mathcal{L}$  by Eq. 4.
Update  $\theta$  using AdamW.
Output: Fine-tuned model  $f(\cdot ;\theta)$ .
High-confidence
Sample Pairs
Compact Clusters of
Samples
Figure 2: An illustration of contrastive learning. The black solid lines indicate similar sample pairs, and the red dashed lines indicate dissimilar pairs.
3.2 Contrastive Learning on Sample Pairs
The key ingredient of our contrastive self-training method is to learn representations that encourage data within the same class to have similar representations and keep data in different classes separated. Specifically, we first select high-confidence samples (Sec. 3.3)  $\mathcal{C}$  from  $\mathcal{X}$ . Then for each pair  $x_{i}, x_{j} \in \mathcal{C}$ , we define their similarity as
$$
W _ {i j} = \left\{ \begin{array}{l l} 1, & \text {i f} \underset {k \in \mathcal {Y}} {\operatorname {a r g m a x}} [ \widetilde {\boldsymbol {y}} _ {i} ] _ {k} = \underset {k \in \mathcal {Y}} {\operatorname {a r g m a x}} [ \widetilde {\boldsymbol {y}} _ {j} ] _ {k} \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {5}
$$
where  $\widetilde{\pmb{y}}_i,\widetilde{\pmb{y}}_j$  are the soft pseudo-labels (Eq. 3) for  $x_{i},x_{j}$ , respectively. For each  $x\in \mathcal{C}$ , we calculate its representation  $\pmb {v} = \mathrm{BERT}(x)\in \mathbb{R}^{d}$ , then we define the contrastive regularizer as
$$
\mathcal {R} _ {1} (\theta ; \widetilde {\boldsymbol {y}}) = \sum_ {(x _ {i}, x _ {j}) \in \mathcal {C} \times \mathcal {C}} \ell (\boldsymbol {v} _ {i}, \boldsymbol {v} _ {j}, W _ {i j}), \tag {6}
$$
$$
\ell = W _ {i j} d _ {i j} ^ {2} + (1 - W _ {i j}) \left[ \max  \left(0, \gamma - d _ {i j}\right) \right] ^ {2}. \tag {7}
$$
Here,  $\ell (\cdot ,\cdot ,\cdot)$  is the contrastive loss (Chopra et al., 2005; Taigman et al., 2014),  $d_{ij}$  is the distance between  $\pmb{v}_i$  and  $\pmb{v}_j$ , and  $\gamma$  is a pre-defined margin.
For samples from the same class, i.e.  $W_{ij} = 1$ , Eq. 6 penalizes the distance between them, and
for samples from different classes, the contrastive loss is large if their distance is small. In this way, the regularizer enforces similar samples to be close, while keeping dissimilar samples apart by at least  $\gamma$ . Figure 2 illustrates the contrastive representations. We can see that our method produces clear interclass boundaries and small intra-class distances, which eases the classification tasks.
3.3 Confidence-based Sample Reweighting and Regularization
While contrastive representations yield better decision boundaries, they require samples with high-quality pseudo-labels. In this section, we introduce reweighting and regularization methods to suppress error propagation and refine pseudo-label qualities.
Sample Reweighting. In the classification task, samples with high prediction confidence are more likely to be classified correctly than those with low confidence. Therefore, we further reduce label noise propagation by a confidence-based sample reweighting scheme. For each sample  $x$  with the soft pseudo-label  $\widetilde{\boldsymbol{y}}$ , we assign  $x$  with a weight  $\omega(x)$  defined by
$$
\omega = 1 - \frac {H (\widetilde {\boldsymbol {y}})}{\log (C)}, H (\widetilde {\boldsymbol {y}}) = - \sum_ {i = 1} ^ {C} \widetilde {\boldsymbol {y}} _ {i} \log \widetilde {\boldsymbol {y}} _ {i}, \tag {8}
$$
where  $0 \leq H(\widetilde{\boldsymbol{y}}) \leq \log(C)$  is the entropy of  $\widetilde{\boldsymbol{y}}$ . Notice that if the prediction confidence is low, then  $H(\widetilde{\boldsymbol{y}})$  will be large, and the sample weight  $\omega(x)$  will be small, and vice versa. We use a pre-defined threshold  $\xi$  to select high confidence samples  $\mathcal{C}$  from each batch  $\mathcal{B}$  as
$$
\mathcal {C} = \{x \in \mathcal {B} \mid \omega (x) \geq \xi \}. \tag {9}
$$
Then we define the loss function as
$$
\mathcal {L} _ {c} (\theta , \widetilde {\boldsymbol {y}}) = \frac {1}{| \mathcal {C} |} \sum_ {x \in \mathcal {C}} \omega (x) \mathcal {D} _ {\mathrm {K L}} (\widetilde {\boldsymbol {y}} \| f (x; \theta)), \tag {10}
$$
$$
\mathcal {D} _ {\mathrm {K L}} (P \| Q) = \sum_ {k} p _ {k} \log \frac {p _ {k}}{q _ {k}} \tag {11}
$$
is the Kullback-Leibler (KL) divergence.
Confidence regularization The sample reweighting approach promotes high confidence samples during contrastive self-training. However, this strategy relies on wrongly-labeled samples to have low confidence, which may not be true unless we prevent over-confident predictions. To this end, we propose a confidence-based regularizer that encourages smoothness over predictions, defined as
$$
\mathcal {R} _ {2} (\theta) = \frac {1}{| \mathcal {C} |} \sum_ {x \in \mathcal {C}} \mathcal {D} _ {\mathrm {K L}} \left(\mathbf {u} \| f (x; \theta)\right), \tag {12}
$$
4We use scaled Euclidean distance  $d_{ij} = \frac{1}{d}\| \pmb {v}_i - \pmb {v}_j\| _2^2$  by default. More discussions on  $W_{ij}$  and  $d_{ij}$  are in Appendix E.
where  $\mathcal{D}_{\mathrm{KL}}$  is the KL-divergence and  $\mathbf{u}_i = 1 / C$  for  $i = 1,2,\dots ,C$ . Such term constitutes a regularization to prevent over-confident predictions and leads to better generalization (Pereyra et al., 2017).
Datasets and Tasks. We conduct experiments on 6 NLP classification tasks using 7 public benchmarks: AGNews (Zhang et al., 2015) is a Topic Classification task; IMDB (Maas et al., 2011) and Yelp (Meng et al., 2018) are Sentiment Analysis tasks; TREC (Voorhees and Tice, 1999) is a Question Classification task; MIT-R (Liu et al., 2013) is a Slot Filling task; Chemprot (Krallinger et al., 2017) is a Relation Classification task; and WiC (Pilehvar and Camacho-Collados, 2019) is a Word Sense Disambiguation (WSD) task. The dataset statistics are summarized in Table 2. More details on datasets and weak supervision sources are in Appendix A<sup>5</sup>. Baselines. We compare our model with different groups of baseline methods:
(i) Exact Matching (ExMatch): The test set is directly labeled by weak supervision sources.
(ii) Fine-tuning Methods: The second group of baselines are fine-tuning methods for LMs:
$\diamond$  RoBERTa (Liu et al., 2019) uses the RoBERTa-base model with task-specific classification heads.
$\diamond$  Self-ensemble (Xu et al., 2020) uses self-ensemble and distillation to improve performances.
$\diamond$  FreeLB (Zhu et al., 2020) adopts adversarial training to enforce smooth outputs.
$\diamond$  Mixup (Zhang et al., 2018) creates virtual training samples by linear interpolations.
SMART (Jiang et al., 2020) adds adversarial and smoothness constraints to fine-tune LMs and achieves state-of-the-art result for many NLP tasks. (iii) Weakly-supervised Models: The third group of baselines are weakly-supervised models<sup>6</sup>:
$\diamond$  Snorkel (Ratner et al., 2020) aggregates different labeling functions based on their correlations.
$\diamond$  WeSTClass (Meng et al., 2018) trains a classifier with generated pseudo-documents and use self-training to bootstrap over all samples.
$\diamond$  ApplyLoss (Awasthi et al., 2020) co-trains a rule-based classifier and a neural classifier to denoise.
$\diamond$  Denoise (Ren et al., 2020) uses attention network to estimate reliability of weak supervisions, and then reduces the noise by aggregating weak labels.
UST (Mukherjee and Awadallah, 2020) is state-of-the-art for self-training with limited labels. It estimates uncertainties via MC-dropout (Gal and Ghahramani, 2015), and then select samples with low uncertainties for self-training.
Evaluation Metrics. We use classification accuracy on the test set as the evaluation metric for all datasets except MIT-R. MIT-R contains a large number of tokens that are labeled as "Others". We use the micro  $F_{1}$  score from other classes for this dataset.<sup>7</sup>
Auxiliary. We implement COSINE using PyTorch $^{8}$ , and we use RoBERTa-base as the pretrained LM. Datasets and weak supervision details are in Appendix A. Baseline settings are in Appendices B. Training details and setups are in Appendix C. Discussions on early-stopping are in Appendix D. Comparison of distance metrics and similarity measures are in Appendix E.
4.1 Learning From Weak Labels
We summarize the weakly-supervised leaning results in Table 3. In all the datasets, COSINE outperforms all the baseline models. A special case is the WiC dataset, where we use WordNet to generate weak labels. However, this enables Snorkel to access some labeled data in the development set, making it unfair to compete against other methods. We will discuss more about this dataset in Sec. 4.3.
In comparison with directly fine-tuning the pretrained LMs with weakly-labeled data, our model employs an "earlier stopping" technique<sup>10</sup> so that it does not overfit on the label noise. As shown, indeed "Init" achieves better performance, and it serves as a good initialization for our framework. Other fine-tuning methods and weakly-supervised models either cannot harness the power of pretrained language models, e.g., Snorkel, or rely on clean labels, e.g., other baselines. We highlight that although UST, the state-of-the-art method to date, achieves strong performance under few-shot settings, their approach cannot estimate confidence well with noisy labels, and this yields inferior performance. Our model can gradually correct wrong pseudo-labels and mitigate error propagation via contrastive self-training.
It is worth noticing that on some datasets, e.g.,
<sup>5</sup>Note that we use the same weak supervision signals/rules for both our method and all the baselines for fair comparison.
6All methods use RoBERTa-base as the backbone unless otherwise specified.
The Chemprot dataset also contains "Others" type, but such instances are few, so we still use accuracy as the metric.
<sup>8</sup>https://pytorch.org/
<sup>9</sup>https://wordnet.princeton.edu/
<sup>10</sup>We discuss this technique in Appendix D.