SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\begin{array}{l} \min  _ {\theta_ {t} ^ {S}} \mathcal {L} \left(\theta_ {t} ^ {S}\right) = \frac {1}{N} \sum_ {x _ {i} \in \mathcal {X} _ {u} \cup \mathcal {X} _ {l}} \ell_ {S} \left(x _ {i}, F, \theta_ {t} ^ {S}\right), \tag {9} \\ \mathrm {s . t .} F \left(\theta_ {t} ^ {T} \left(\theta_ {t} ^ {S}\right)\right) = \left[ \widetilde {\mathbf {y}} \left(\theta_ {t} ^ {T} \left(\theta_ {t} ^ {S}\right)\right), \omega \left(\theta_ {t} ^ {T} \left(\theta_ {t} ^ {S}\right)\right) \right]. \\ \end{array}
$$
Figure 1: Left: Learning curve. Right: The last epoch accuracy. The vertical lines signify run-to-run variance.
Figure 2: Demonstration of Self-training stuck at bad optima. The solid diamonds are labeled samples.
Here  $N = N_{l} + N_{u}$  is the total number of training samples. Note that in comparison with (3), we drop the supervised loss  $\mathcal{L}_{\mathrm{sup}}$ . Moreover, the teacher model now generates soft pseudo-labels for all the data, instead of only the data in  $\mathcal{X}_u$ .
We conduct two sets of experiments: weakly- and semi-supervised text classification. We also examine semi-supervised node classification on graphs (Appendix A). All the results have passed a paired t-test with  $p < 0.05$ . When using pre-trained language models, we employ a RoBERTa-base (Liu et al., 2019) model obtained from the HuggingFace (Wolf et al., 2019) codebase. We implement all the methods using PyTorch (Paszke et al., 2019), and experiments are run on NVIDIA 2080Ti GPUs. All the training details are deferred to the appendix.
4.1 Warmup: TwoMoon Experiments
To understand the efficacy of DRIFT, we conduct a semi-supervised classification experiment on a classic synthetic dataset "TwoMoon". The dataset contains two classes, and for each class we generate 12 labeled samples and 500 unlabeled ones.
We compare DRIFT with conventional Self-training. The only difference between the two
Dataset	AGNews	IMDB	Yelp	MIT-R	CoNLL-03	Webpage	BC5CDR	Wikipold
RoBERTa-Full	91.41	94.26	97.27	88.51	90.11 (89.14/91.10)	72.39 (66.29/79.73)	85.15 (83.74/86.61)	86.43 (85.33/87.56)
RoBERTa-Weak	82.25	72.60	79.91	70.95	75.61 (83.76/68.90)	59.11 (60.14/58.11)	78.51 (74.96/82.42)	51.55 (49.17/54.50)
WeSTClass	82.78	77.40	76.86	-	-	-	-	-
Self-training	86.07	85.72	89.95	73.59	77.28 (83.42/71.98)	56.90 (54.32/59.74)	79.92 (74.73/85.90)	56.90 (54.32/59.74)
UAST	86.28	84.56	90.53	74.41	77.92 (83.30/73.20)	58.18 (56.33/60.14)	81.50 (80.09/82.98)	57.79 (52.64/64.05)
BOND	86.19	88.36	93.18	75.90	81.48 (82.05/80.92)	65.74 (67.37/64.19)	81.53 (79.54/83.63)	60.07 (53.44/68.58)
DRIFT	87.80	91.56	96.24	77.15	81.74 (81.45/82.02)	66.04 (65.23/66.87)	82.62 (82.57/82.68)	60.66 (57.50/64.21)
Table 1: Accuracy (in %) of weakly-supervised text classification on various datasets. We report the mean over three runs. DRIFT is initialized from RoBERTa-Weak. For text classification tasks, we report the accuracy; and for NER tasks, we report  $F_{1}$  (precision/recall). The best results are shown in bold, except RoBERTa-Full, which is a fully-supervised model and is included here as a reference.
methods is that DRIFT adopts the differentiable strategies, while Self-training does not. In both methods, the teacher/student model is a two-layer feed-forward neural network, with hidden dimension 50 and tanh (hyperbolic tangent) as the nonlinearity. We first train the models for 50 epochs using the labeled samples. We then conduct self-training with learning rate 0.01 and Adam (Kingma and Ba, 2015) as the optimizer. We adopt an exponential moving average approach (2) with  $\alpha = 0.5$  and we set the temperature parameter  $\tau = 0.5$  for the soft pseudo-labels (6).
We conduct 10 trails, and Fig. 1 shows the accuracy and the variance during training. We can see that Self-training yields a much larger variance, indicating an unstable training process. Note that the performance gain of DRIFT to Self-training has passed a paired-student t-test with p-value  $< 0.05$ .
Moreover, by examining the experimental results, we find that Self-training at times gets stuck at suboptimal solutions. As an example, in Fig. 2, notice that the two methods behave equally well at epoch 20. However, Self-training gets stuck and does not improve at epoch 150. This is because the teacher generates hazardous labels that avert the student from improving. Meanwhile, by incorporating differentiable strategies, the performance of DRIFT improves at epoch 150 from epoch 20.
4.2 Weakly-Supervised Text Classification
We fine-tune a pre-trained RoBERTa model for weakly-supervised learning. In addition, we demonstrate that our method works well when trained-from-scratch and when using different backbones than the Transformer (Vaswani et al., 2017). See Section 4.3 and Table 3 for details.
Settings. We use the following datasets: Topic Classification on AGNews (Zhang et al., 2015); Sentiment Analysis on IMDB (Maas et al., 2011)
and Yelp (Meng et al., 2018); Slot Filling on MITR (Liu et al., 2013); and Named Entity Recognition (NER) on CoNLL-03 (Tjong Kim Sang, 2002), Webpage (Ratinov and Roth, 2009), Wikigold (Balasuriya et al., 2009), and BC5CDR (Li et al., 2016). The dataset statistics are summarized in Table 7. For each dataset, we generate weak labels using some pre-defined rules, after which the same data and generated weak labels are used by all the methods. More details about the weak supervision sources are in Appendix C.
We adopt several baselines:
- RoBERTa (Liu et al., 2019) uses the RoBERTa-base model with task-specific classification heads.
- Self-training (Lee, 2013; Rosenberg et al., 2005) uses the conventional teacher-student framework, where a teacher generates pseudolabels, and a student makes predictions.
- WeSTClass (Meng et al., 2018) leverages generated pseudo-documents and uses self-training to bootstrap over all the samples.
- BOND (Liang et al., 2020) uses a teacher-student framework for self-training. The teacher model is periodically updated to generate pseudo-labels when training the student.
- UAST (Mukherjee and Awadallah, 2020) estimates uncertainties of unlabeled data via MCdropout (Gal and Ghahramani, 2016) during self-training, and then selects samples with low uncertainties. It is the state-of-the-art self-training method for text data with few labels.
Recall that for weakly-supervised learning, we first fine-tune a RoBERTa model using the weakly-labeled data, and then we discard the weak labels and continue with self-training. This is an effective
strategy to reduce overfitting on label noise (Yu et al., 2021). We follow this procedure for both DRIFT and all the baseline methods.
Results. Experimental results are summarized in Table 1. We can see that DRIFT achieves the best performance in all the tasks. Notice that the baselines that adopt self-training, e.g., Class, Self-training, UAST, and BOND, outperform the vanilla RoBERTa-Weak method. This is because in weakly-supervised learning, a noticeable amount of labels are inaccurate. Therefore, without noise suppressing approaches such as self-training, models cannot behave well. However, without taking the teacher's strategy into account, these methods still suffer from training instabilities, such that they are not as effective as DRIFT.
We highlight that on some datasets, performance of our method is close to the fully-supervised model RoBERTa-Full, even though we do not use any clean labels. For example, DRIFT achieves  $91.6\%$  vs.  $94.3\%$  performance on IMDB,  $96.2\%$  vs.  $97.3\%$  on Yelp, and 82.6 vs. 85.1 on BC5CDR.
4.3 Semi-Supervised Text Classification
Datasets. We adopt AGNews, IMDB, and Amazon (McAuley and Leskovec, 2013) (see Table 7) in this set of experiments. For each dataset, we randomly sample  $N \in \{30,50,200,1000\}$  data points from each class and annotate them with clean labels, while the other data are treated as unlabeled. Note that for all the splits of a particular dataset, we use the same development and test sets.
Settings. Our differentiable self-training framework works well in both fine-tuning and training-from-scratch regimes. Moreover, our approach is flexible to accommodate different neural architectures. We conduct two sets of experiments. In the first set, we fine-tune a pre-trained RoBERTa model, which uses the Transformer (Vaswani et al., 2017) as its backbone. In the second set of experiments, we train a TextCNN (Kim, 2014) model from scratch, which employs a convolutional neural network as the foundation.
Baselines. Besides RoBERTa, Self-training, and UAST, which are used in weakly-supervised classification tasks, we adopt several new methods as baseline approaches.
VAMPIRE (Gururangan et al., 2019) pre-trains a unigram document model on unlabeled data
using a variational auto-encoder, and then uses its internal states as features for downstream applications.
- UDA (Xie et al., 2020) uses back translation and word replacement to augment unlabeled data, and forces the model to make consistent predictions on the augmented data to improve model performance.
- MixText (Chen et al., 2020) augments the training data by interpolation in the hidden space, and it exploits entropy and consistency regularization to further utilize unlabeled data during training.
Results. Experimental results are summarized in Table 2. We can see that DRIFT achieves the best performance across the three datasets under different setups. Notice that the performance of VAMPIRE is not satisfactory. This is because it does not use pre-trained models, unlike the other baselines. Pre-trained language models contain rich semantic knowledge, which can be effectively transferred to the target task and boost model performance. All the baselines do not explicitly consider the teacher's strategy, and thus, they suffer from training instabilities.
We remark that UDA, UAST and MixText leverage external sources or data augmentation methods to make full use of the unlabeled data. These methods can potentially combine with DRIFT, which is of separate interests.
Fine-tuning vs. Training-from-scratch. Table 3 shows the results of training a TextCNN model from scratch. We can see that the model trained from scratch performs worse than fine-tuning a pretrained model (Table 2). This is because TextCNN has significantly less parameters than RoBERTa, and is not pre-trained on massive text corpora. Therefore, we cannot take advantage of the semantic information from pre-trained models.
Nevertheless, under both weakly-supervised and semi-supervised learning settings, DRIFT consistently outperforms the baseline methods. This indicates that our method is architecture independent, and does not rely on transferring existing semantic information. As such, differentiable self-training serves as an effective plug-in module for existing models. We remark that DRIFT does not introduce any additional tuning parameter in comparison with conventional self-training.
Dataset	AGNews	IMDB	Amazon
Labels/class	30	50	200	1000	30	50	200	1000	30	50	200	1000
RoBERTa-Semi	83.98	87.44	88.01	90.91	86.64	88.37	89.25	90.54	88.21	89.66	92.31	93.65
VAMPIRE	-	-	83.90	85.80	-	-	82.20	85.40	-	-	-	-
UDA	85.92	88.09	88.33	91.22	89.30	89.42	89.72	90.87	-	-	-	-
MixText	88.50	88.85	89.20	91.55	84.34	88.72	89.45	91.20	-	-	-	-
Self-training	84.62	88.04	88.67	91.47	88.13	88.80	89.84	91.04	89.92	90.55	92.55	93.83
UAST	87.74	88.65	89.21	91.81	89.21	89.56	90.11	91.48	91.27	91.50	92.68	93.97
DRIFT	89.46	89.67	90.17	92.47	89.77	90.03	90.83	92.39	91.82	92.67	93.16	94.28
Table 2: Accuracy (in %) of semi-supervised text classification on various datasets. DRIFT is initialized from RoBERTa-Semi. The best results are shown in bold.
Dataset	AGNews	IMDB
Labels per class	weak	30	50	200	1000	weak	30	50	200	1000
TextCNN	79.45	78.81	79.98	85.46	86.78	82.44	63.32	66.61	73.22	78.29
Self-training	81.69	81.98	82.67	86.26	88.15	84.76	64.68	65.26	73.60	79.04
UAST	81.48	82.05	83.34	86.67	87.90	83.97	64.23	68.70	73.95	79.13
DRIFT	82.55	83.34	85.01	87.38	88.66	86.44	65.65	69.86	74.61	79.38
$\diamond$  Components of DRIFT. We inspect different components of DRIFT, including the differentiable pseudo labels (DrPL), differentiable sample weights (DrW), and the sample reweighting strategy  $(\mathrm{SR})^2$ . Experimental results are summarized in Table 4. We observe that both differentiable pseudolabels and differentiable sample weights contribute to model performance, as removing any of them hurts the classification accuracy. Also, DRIFT excels when the labels are noisy. We can see that our method brings  $2.41\%$  performance gain on average under the weakly-supervised setting, while it only promotes  $1.08\%$  average gain under the semi-supervised setting. Such results indicate that differentiable pseudo-labels and sample weight are effective in suppressing label noise.
$\diamond$  Sensitivity to hyper-parameters. We study models' sensitivity to the exponential moving average rate  $\alpha$  and the soft pseudo-label's temperature  $\tau$ . Figure 3 shows the results. We can see that model performance peaks when  $\alpha$  is around 0.9. The teacher model updates too aggressively with a smaller  $\alpha$  (e.g.,  $\alpha = 0.7$ ), and too conservatively with a larger alpha (e.g.,  $\alpha = 0.99$ ). In the
Table 3: Results of DRIFT and self-training baselines on AGNews and IMDB. We use TextCNN as the backbone and train the models from scratch. DRIFT is initialized from TextCNN. The best results are shown in bold. "Weak" means the weak-supervision setting.
Method	AGNews	IMDB
#labels	30	1000	weak	30	1000	weak
DRIFT	89.46	92.47	87.80	89.77	92.39	91.56
w/o DrPL	87.63	92.11	86.13	89.05	91.75	86.78
w/o DrW	88.51	91.22	86.62	88.47	91.49	90.24
w/o SR	88.84	91.20	86.19	88.15	91.95	87.76
Table 4: Effects of different components of DRIFT. Here "weak" means the weak-supervision setting.
(a) Effect of  $\alpha$  on AGNews.
(b) Effect of  $\tau$  on AGNews.
(c) Effect of  $\alpha$  on IMDB.
(d) Effect of  $\tau$  on IMDB.
Figure 3: Parameter study. Here "semi" means the semi-supervision setting with 30 labels per class, and "weak" means the weak-supervision setting.
For models without DrPL, we do not differentiate the pseudo-labels. For models without DrW, we still use (7) to perform sample rewriting, but we do not differentiate the weights, i.e., w/o DrW equals to w/ SR. For models without SR, we do not use sample reweighing.
first case, the generated pseudo-labels are not reliable; and in the second case, model improves too slow. Also notice that the semi-supervised model
Figure 4: Sample predictions under weak supervision. From inside to outside, the four rings correspond to the results at iteration 0 (initialization using RoBERTa), 100, 200, 300, respectively.
is not sensitive to the temperature parameter. The weakly-supervised model achieves the best performance when  $\tau = 0.5$ . Note that a smaller  $\tau$  essentially generates hard pseudo-labels, which drastically hurts model performance.
Figure 4 demonstrates error reduction. Samples are indicated by radii of the circle, and classification correctness is indicated by color. For example, if a radius has color orange, blue, blue, blue, then it is mis-classified at iteration 0, and correctly classified at iteration 100, 200, and 300. We can see that Self-training suffers from error accumulation, as around  $2\%$  more samples are mis-classified between iteration 200 and 300. In contrast, in DRIFT, a noticeable amount of incorrect predictions are rectified, and the accuracy improves by more than  $15\%$  after 300 iterations.
In this paper, we propose a differentiable self-training framework, DRIFT, which formulates the teacher-student framework in self-training as a Stackelberg game. The formulation treats the student as the leader, and the teacher as the follower. In DRIFT, the student is in an advantageous position by recognizing the follower's strategy. In this way, we can find a better descent direction for the student and can stabilize training. Empirical results on weakly- and semi-supervised natural language processing tasks suggest the superiority of DRIFT to conventional self-training.
Conventional self-training is a heuristic and does not pose a well-defined optimization problem. In conventional methods, the teacher optimizes an im
plicit function through different components, e.g., pseudo-labels. We follow this convention and formulate self-training as a Stackelberg game. Our formulation is a principle that can motivate follow-up works.
In our Stackelberg game formulation (3), the student's utility function is the objective function of the minimization problem. The teacher's utility function is an implicit function, which can be written as the following:
$$
\begin{array}{l} \text {U t i l i t y} (\text {t e a c h e r}) = \mathcal {D} (\text {t e a c h e r}, \text {s t u d e n t}) \\ + \mathcal {R} _ {1} (\text {t e a c h e r}, \text {c o n f i d e n c e}) \\ + \mathcal {R} _ {2} (\text {t e a c h e r}, \text {u n c e r t a i n t y}). \\ \end{array}
$$
Here, the first term  $\mathcal{D}$  is some divergence between the teacher and the student (i.e., the KL-divergence in (8)), and the two regularizers  $\mathcal{R}_1$  and  $\mathcal{R}_2$  are defined implicitly. That is,  $\mathcal{R}_1$  regularizes model confidence (realized by the soft pseudo-labels in (6)), and  $\mathcal{R}_2$  regularizes model uncertainty (realized by the sample weights in (7)). Even though the utility function of the teacher is implicit, the solution of it is explicitly given, namely the teacher's strategy  $F(\theta^T)$  in (3) is the solution to the teacher's implicit utility.
Because the strategy of the teacher is explicit (in contrast to implicitly defined by an optimization problem), the teacher's utility is maximized with such a strategy. Thus, equilibrium of the Stackelberg game exists, and every local optimum of the minimization problem is an equilibrium. In this work, we use off-the-shelf algorithms (Adam and AdamW) to find local minima of (3) (a.k.a. equilibria of the game).
We remark that in a Stackelberg game, the names "leader" and "follower" indicate the relative importance and priority of the two players. In our framework, we use the student model for prediction. Therefore, the student model is more important than the teacher, so that we grant it a higher priority and say it is the leader.
We remark that self-distillation (Furlanello et al., 2018) is a special case of self-training, which is a supervised learning method. We can also differentiate the teacher model in self-distillation, such that DRIFT can be extended to supervised learning.
This paper proposes Differentiable Self-Training (DRIFT), a self-training framework for NLP tasks. We demonstrate that the DRIFT framework can be used for text classification and named entity recognition tasks. Moreover, the framework is also demonstrated to be effective for semi-supervised classification on graphs. We use publicly available datasets to conduct all the experiments. And the proposed method is built using public code bases. We do not find any ethical concerns.
Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules generalizing labeled exemplars. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R. Curran. 2009. Named entity recognition in Wikipedia. In Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources (People's Web), pages 10-18, Suntec, Singapore. Association for Computational Linguistics.
David Berthelot, Nicholas Carlini, Ian J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. 2019. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5050-5060.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mix-Text: Linguistically-informed interpolation of hidden space for semi-supervised text classification. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jingfei Du, Edouard Grave, Belize Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021. Self-training improves pre-training for natural language understanding. In Proceedings of the 2021 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5408-5418, Online. Association for Computational Linguistics.
Fuli Feng, Xiangnan He, Jie Tang, and Tat-Seng Chua. 2019. Graph adversarial training: Dynamically regularizing based on graph structure. TKDE.
Wenzheng Feng, Jie Zhang, Yuxiao Dong, Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. 2020. Graph random neural networks for semi-supervised learning on graphs. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Yoav Freund and Robert E Schapire. 1997. A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci, 55(1).
Tommaso Furlanello, Zachary Chase Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born-again neural networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1602-1611. PMLR.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1050-1059. JMLR.org.
Garrett B. Goh, Charles Siegel, Abhinav Vishnu, and Nathan Oken Hodas. 2018. Using rule-based labels for weak supervised learning: A chemnet for transferable chemical property prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018, pages 302-310. ACM.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5880-5894, Florence, Italy. Association for Computational Linguistics.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 541-550, Portland, Oregon, USA. Association for Computational Linguistics.