SlowGuess/ACL_2020_OCR huggingface dataset:
Jiaao Chen  
Georgia Tech  
jchen896@gatech.edu
Zichao Yang CMU zichaoy@cs.cmu.edu
Diyi Yang  
Georgia Tech  
dyang888@gatech.edu
This paper presents MixText, a semi-supervised learning method for text classification, which uses our newly designed data augmentation method called TMix. TMix creates a large amount of augmented training samples by interpolating text in hidden space. Moreover, we leverage recent advances in data augmentation to guess low-entropy labels for unlabeled data, hence making them as easy to use as labeled data. By mixing labeled, unlabeled and augmented data, MixText significantly outperformed current pre-trained and fined-tuned models and other state-of-the-art semi-supervised learning methods on several text classification benchmarks. The improvement is especially prominent when supervision is extremely limited. We have publicly released our code at https://github.com/GT-SALT/MixText.
In the era of deep learning, research has achieved extremely good performance in most supervised learning settings (LeCun et al., 2015; Yang et al., 2016). However, when there is only limited labeled data, supervised deep learning models often suffer from over-fitting (Xie et al., 2019). This strong dependence on labeled data largely prevents neural network models from being applied to new settings or real-world situations due to the need of large amount of time, money, and expertise to obtain enough labeled data. As a result, semi-supervised learning has received much attention to utilize both labeled and unlabeled data for different learning tasks, as unlabeled data is always much easier and cheaper to collect (Chawla and Karakoulas, 2011).
This work takes a closer look at semi-supervised text classification, one of the most fundamental tasks in language technology communities. Prior research on semi-supervised text classification can
Figure 1: TMix takes in two text samples  $x$  and  $x'$  with labels  $y$  and  $y'$ , mixes their hidden states  $h$  and  $h'$  at layer  $m$  with weight  $\lambda$  into  $\tilde{h}$ , and then continues forward passing to predict the mixed labels  $\tilde{y}$ .
be categorized into several classes: (1) utilizing variational auto encoders (VAEs) to reconstruct the sentences and predicting sentence labels with latent variables learned from reconstruction such as (Chen et al., 2018; Yang et al., 2017; Gururangan et al., 2019); (2) encouraging models to output confident predictions on unlabeled data for self-training like (Lee, 2013; Grandvalet and Bengio, 2004; Meng et al., 2018); (3) performing consistency training after adding adversarial noise (Miyato et al., 2019, 2017) or data augmentations (Xie et al., 2019); (4) large scale pretraining with unlabeled data, then finetuning with labeled data (Devlin et al., 2019). Despite the huge success of those models, most prior work utilized labeled and unlabeled data separately in a way that no supervision can transit from labeled to unlabeled data or from unlabeled to labeled data. As a result, most semi-supervised models can easily still overfit on the very limited labeled data, despite unlabeled data is
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2147-2157 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
abundant.
To overcome the limitations, in this work, we introduce a new data augmentation method, called TMix (Section 3), inspired by the recent success of Mixup (Gururangan et al., 2019; Berthelot et al., 2019) on image classifications. TMix, as shown in Figure 1, takes in two text instances, and interpolates them in their corresponding hidden space. Since the combination is continuous, TMix has the potential to create infinite mount of new augmented data samples, thus can drastically avoid overfitting. Based on TMix, we then introduce a new semi-supervised learning method for text classification called MixText (Section 4) to explicitly model the relationships between labeled and unlabeled samples, thus overcoming the limitations of previous semi-supervised models stated above. In a nutshell, MixText first guesses low-entropy labels for unlabeled data, then uses TMix to interpolate the label and unlabeled data. MixText can facilitate mining implicit relations between sentences by encouraging models to behave linearly in-between training examples, and utilize information from unlabeled sentences while learning on labeled sentences. In the meanwhile, MixText exploits several semi-supervised learning techniques to further utilize unlabeled data including self-target-prediction (Laine and Aila, 2016), entropy minimization (Grandvalet and Bengio, 2004), and consistency regularization (Berthelot et al., 2019; Xie et al., 2019) after back translations.
To demonstrate the effectiveness of our method, we conducted experiments (Section 5) on four benchmark text classification datasets and compared our method with previous state-of-the-art semi-supervised method, including those built upon models pre-trained with large amount of unlabeled data, in terms of accuracy on test sets. We further performed ablation studies to demonstrate each component's influence on models' final performance. Results show that our MixText method significantly outperforms baselines especially when the given labeled training data is extremely limited.
2.1 Pre-training and Fine-tuning Framework
The pre-training and fine-tuning framework has achieved huge success on NLP applications in recent years, and has been applied to a variety of NLP tasks (Radford et al., 2018; Chen et al., 2019; Akbik et al., 2019). Howard and Ruder (2018)
proposed to pre-train a language model on a large general-domain corpus and fine-tune it on the target task using some novel techniques like discriminative fine-tuning, slanted triangular learning rates, and gradual unfreezing. In this manner, such pretrained models show excellent performance even with small amounts of labeled data. Pre-training methods are often designed with different objectives such as language modeling (Peters et al., 2018; Howard and Ruder, 2018; Yang et al., 2019b) and masked language modeling (Devlin et al., 2019; Lample and Conneau, 2019). Their performances are also improved with training larger models on more data (Yang et al., 2019b; Liu et al., 2019).
2.2 Semi-Supervised Learning on Text Data
Semi-supervised learning has received much attention in the NLP community (Gururangan et al., 2019; Clark et al., 2018; Yang et al., 2015), as unlabeled data is often plentiful compared to labeled data. For instance, Gururangan et al. (2019); Chen et al. (2018); Yang et al. (2017) leveraged variational auto encoders (VAEs) in a form of sequence-to-sequence modeling on text classification and sequential labeling. Miyato et al. (2017) utilized adversarial and virtual adversarial training to the text domain by applying perturbations to the word embeddings. Yang et al. (2019a) took advantage of hierarchy structures to utilize supervision from higher level labels to lower level labels. Xie et al. (2019) exploited consistency regularization on unlabeled data after back translations and tfidf word replacements. Clark et al. (2018) proposed cross-veiw training for unlabeled data, where they used an auxiliary prediction modules that see restricted views of the input (e.g., only part of a sentence) and match the predictions of the full model seeing the whole input.
2.3 Interpolation-based Regularizers
Interpolation-based regularizers (e.g., Mixup) have been recently proposed for supervised learning (Zhang et al., 2017; Verma et al., 2019a) and semi-supervised learning (Berthelot et al., 2019; Verma et al., 2019b) for image-format data by overlaying two input images and combining image labels as virtual training data and have achieved state-of-the-art performances across a variety of tasks like image classification and network architectures. Different variants of mixing methods have also been designed such as performing interpolations in the input space (Zhang et al., 2017), combining inter
polations and cutoff (Yun et al., 2019), and doing interpolations in the hidden space representations (Verma et al., 2019a,c). However, such interpolation techniques have not been explored in the NLP field because most input space in text is discrete, i.e., one-hot vectors instead of continues RGB values in images, and text is generally more complex in structures.
2.4 Data Augmentations for Text
When labeled data is limited, data augmentation has been a useful technique to increase the amount of training data. For instance, in computer vision, images are shifted, zoomed in/out, rotated, flipped, distorted, or shaded with a hue (Perez and Wang, 2017) for training data augmentation. But it is relatively challenging to augment text data because of its complex syntactic and semantic structures. Recently, Wei and Zou (2019) utilized synonym replacement, random insertion, random swap and random deletion for text data augmentation. Similarly, Kumar et al. (2019) proposed a new paraphrasing formulation in terms of monotone submodular function maximization to obtain highly diverse paraphrases, and Xie et al. (2019) and Chen et al. (2020) applied back translations (Sennrich et al., 2015) and word replacement to generate paraphrases on unlabeled data for consistency training. Other work which also investigates noise and its incorporation into semi-supervised named entity classification (Lakshmi Narayan et al., 2019; Nagesh and Surdeanu, 2018).
3 TMix
In this section, we extend Mixup-a data augmentation method originally proposed by (Zhang et al., 2017) for images-to text modeling. The main idea of Mixup is very simple: given two labeled data points  $(\mathbf{x}_i,\mathbf{y}_i)$  and  $(\mathbf{x}_j,\mathbf{y}_j)$ , where  $\mathbf{x}$  can be an image and  $\mathbf{y}$  is the one-hot representation of the label, the algorithm creates virtual training samples by linear interpolations:
$$
\tilde {\mathbf {x}} = \operatorname {m i x} \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}\right) = \lambda \mathbf {x} _ {i} + (1 - \lambda) \mathbf {x} _ {j}, \tag {1}
$$
$$
\tilde {\mathbf {y}} = \operatorname {m i x} \left(\mathbf {y} _ {i}, \mathbf {y} _ {j}\right) = \lambda \mathbf {y} _ {i} + (1 - \lambda) \mathbf {y} _ {j}, \tag {2}
$$
where  $\lambda \in [0,1]$ . The new virtual training samples are used to train a neural network model. Mixup can be interpreted in different ways. On one hand, Mixup can be viewed as data augmentation approach which creates new data samples based on the original training set. On the other
hand, it enforces a regularization on the model to behave linearly among the training data. Mixup was demonstrated to work well on continuous image data (Zhang et al., 2017). However, extending it to text seems challenging since it is infeasible to compute the interpolation of discrete tokens.
To this end, we propose a novel method to overcome this challenge — interpolation in textual hidden space. Given a sentence, we often use a multilayer model like BERT (Devlin et al., 2019) to encode the sentences to get the semantic representations, based on which final predictions are made. Some prior work (Bowman et al., 2016) has shown that decoding from an interpolation of two hidden vectors generates a new sentence with mixed meaning of two original sentences. Motivated by this, we propose to apply interpolations within hidden space as a data augment method for text. For an encoder with  $L$  layers, we choose to mixup the hidden representation at the  $m$ -th layer,  $m \in [0, L]$ .
As demonstrated in Figure 1, we first compute the hidden representations of two text samples separately in the bottom layers. Then we mix up the hidden representations at layer  $m$ , and feed the interpolated hidden representations to the upper layers. Mathematically, denote the  $l$ -th layer in the encoder network as  $g_{l}(:,\boldsymbol{\theta})$ , hence the hidden representation of the  $l$ -th layer can be computed as  $\mathbf{h}_l = g_l(\mathbf{h}_{l-1};\boldsymbol{\theta})$ . For two text samples  $\mathbf{x}_i$  and  $\mathbf{x}_j$ , define the 0-th layer as the embedding layer, i.e.,  $\mathbf{h}_0^i = \mathbf{W}_{\mathrm{E}}\mathbf{x}_i, \mathbf{h}_0^j = \mathbf{W}_{\mathrm{E}}\mathbf{x}_j$ , then the hidden representations of the two samples from the lower layers are:
$$
\mathbf {h} _ {l} ^ {i} = g _ {l} \left(\mathbf {h} _ {l - 1} ^ {i}; \boldsymbol {\theta}\right), l \in [ 1, m ],
$$
$$
\mathbf {h} _ {l} ^ {j} = g _ {l} \left(\mathbf {h} _ {l - 1} ^ {j}; \boldsymbol {\theta}\right), l \in [ 1, m ].
$$
The mixup at the  $m$ -th layer and continuing forward passing to upper layers are defined as:
$$
\tilde {\mathbf {h}} _ {m} = \lambda \mathbf {h} _ {m} ^ {i} + (1 - \lambda) \mathbf {h} _ {m} ^ {j},
$$
$$
\tilde {\mathbf {h}} _ {l} = g _ {l} \left(\tilde {\mathbf {h}} _ {l - 1}; \boldsymbol {\theta}\right), l \in [ m + 1, L ].
$$
We call the above method TMix and define the new mixup operation as the whole process to get  $\tilde{\mathbf{h}}_L$ :
$$
\operatorname {T M i x} \left(\mathbf {x} _ {i}, \mathbf {x} _ {j}; g (.; \boldsymbol {\theta}), \lambda , m\right) = \tilde {\mathbf {h}} _ {L}.
$$
By using an encoder model  $g(\cdot ;\theta)$  ,TMix interpolates textual semantic hidden representations as a type of data augmentation. In contrast with Mixup defined in the data space in Equation 1,
TMix depends on an encoder function, hence defines a much broader scope for computing interpolations. For ease of notation, we drop the explicit dependence on  $g(:, \theta)$ ,  $\lambda$  and  $m$  in notations and denote it simply as  $\mathrm{TMix}(\mathbf{x}_i, \mathbf{x}_j)$  in the following sections.
In our experiments, we sample the mix parameter  $\lambda$  from a Beta distribution for every batch to perform the interpolation:
$$
\lambda \sim \operatorname {B e t a} (\alpha , \alpha),
$$
$$
\lambda = \max  (\lambda , 1 - \lambda),
$$
in which  $\alpha$  is the hyper-parameter to control the distribution of  $\lambda$ . In TMix, we mix the labels in the same way as Equation 2 and then use the pairs  $(\tilde{\mathbf{h}}_L,\tilde{y})$  as inputs for downstream applications.
Instead of performing mixup at random input layers like Verma et al. (2019a), choosing which layer of the hidden representations to mixup is an interesting question to investigate. In our experiments, we use 12-layer BERT-base (Devlin et al., 2019) as our encoder model. Recent work (Jawahar et al., 2019) has studied what BERT learned at different layers. Specifically, the authors found  $\{3,4,5,6,7,9,12\}$  layers have the most representation power in BERT and each layer captures different types of information ranging from surface, syntactic to semantic level representation of text. For instance, the 9-th layer has predictive power in semantic tasks like checking random swapping of coordinated clausal conjuncts, while the 3-rd layer performs best in surface tasks like predicting sentence length.
Building on those findings, we choose the layers that contain both syntactic and semantic information as our mixing layers, namely  $\mathbf{M} = \{7,9,12\}$ . For every batch, we randomly sample  $m$ , the layer to mixup representations, from the set  $\mathbf{M}$  computing the interpolation. We also performed ablation study in Section 5.5 to show how TMix's performance changes with different choice of mix layer sets.
Text classification Note that TMix provides a general approach to augment text data, hence can be applied to any downstream tasks. In this paper, we focus on text classification and leave other applications as potential future work. In text classification, we minimize the KL-divergence between the mixed labels and the probability from the clas
sifier as the supervision loss:
$$
L _ {\mathrm {T M i x}} = \mathrm {K L} (\operatorname {m i x} (\mathbf {y} _ {i}, \mathbf {y} _ {j}) | | p (\operatorname {T M i x} (\mathbf {x} _ {i}, \mathbf {x} _ {j}); \phi)
$$
where  $p(\cdot ;\phi)$  is a classifier on top of the encoder model. In our experiments, we implement the classifier as a two-layer MLP, which takes the mixed representation  $\mathrm{TMix}(\mathbf{x}_i,\mathbf{x}_j)$  as input and returns a probability vector. We jointly optimize over the encoder parameters  $\theta$  and the classifier parameters  $\phi$  to train the whole model.
4 Semi-supervised MixText
In this section, we demonstrate how to utilize the TMix to help semi-supervised learning. Given a limited labeled text set  $\mathbf{X}_l = \{\mathbf{x}_1^l,\dots,\mathbf{x}_n^l\}$ , with their labels  $\mathbf{Y}_l = \{\mathbf{y}_1^l,\dots,\mathbf{y}_n^l\}$  and a large unlabeled set  $\mathbf{X}_u = \{\mathbf{x}_1^u,\dots,\mathbf{x}_m^u\}$ , where  $n$  and  $m$  are the number of data points in each set.  $\mathbf{y}_i^l\in \{0,1\} ^C$  is a one-hot vector and  $C$  is the number of classes. Our goal is to learn a classifier that efficiently utilizes both labeled data and unlabeled data.
We propose a new text semi-supervised learning framework called MixText<sup>1</sup>. The core idea behind our framework is to leverage TMix both on labeled and unlabeled data for semi-supervised learning. To fulfill this goal, we come up a label guessing method to generate labels for the unlabeled data in the training process. With the guessed labels, we can treat the unlabeled data as additional labeled data and perform TMix for training. Moreover, we combine TMix with additional data augmentation techniques to generate large amounts of augmented data, which is a key component that makes our algorithm work well in setting with extremely limited supervision. Finally, we introduce an entropy minimization loss that encourages the model to assign sharp probabilities on unlabeled data samples, which further helps to boost performance when the number of classes  $C$  is large. The overall architecture is shown in Figure 2. We will explain each component in detail.
Back translations (Edunov et al., 2018) is a common data augmentation technique and can generate diverse paraphrases while preserving the semantics of the original sentences. We utilize back translations to paraphrase the unlabeled data. For each  $\mathbf{x}_i^u$  in the unlabeled text set  $\mathbf{X}_u$ , we generate  $K$
<sup>1</sup>Note that MixText is a semi-supervised learning framework while TMix is a data augmentation approach.
Figure 2: Overall Architecture of MixText. MixText takes in labeled data and unlabeled data, conducts augmentations and predicts labels for unlabeled data, performs TMix over labeled and unlabeled data, and computes supervised loss, consistency loss and entropy minimization term.
augmentations  $\mathbf{x}_{i,k}^{a} = \mathrm{augment}_{k}(\mathbf{x}_{i}^{u}), k \in [1, K]$  by back translations with different intermediate languages. For example, we can translate original sentences from English to German and then translate them back to get the paraphrases. In the augmented text generation, we employ random sampling with a tunable temperature instead of beam search to ensure the diversity. The augmentations are then used for generating labels for the unlabeled data, which we describe below.
4.2 Label Guessing
For an unlabeled data sample  $\mathbf{x}_i^u$  and its  $K$  augmentations  $\mathbf{x}_{i,k}^{a}$ , we generate the label for them using weighted average of the predicted results from the current model:
$$
\begin{array}{l} \mathbf {y} _ {i} ^ {u} = \frac {1}{w _ {o r i} + \sum_ {k} w _ {k}} (w _ {o r i} p (\mathbf {x} _ {i} ^ {u}) \\ + \sum_ {k = 1} ^ {K} w _ {k} p (\mathbf {x} _ {i, k} ^ {a})) \\ \end{array}
$$
Note that  $\mathbf{y}_i^u$  is a probability vector. We expect the model to predict consistent labels for different augmentations. Hence, to enforce the constraint, we use the weighted average of all predictions, rather than the prediction of any single data sample, as the generated label. Moreover, by explicitly introducing the weight  $w_{ori}$  and  $w_k$ , we can control the contributions of different quality of augmentations to the generated labels. Our label guessing method improves over (Tarvainen and Valpola, 2017) which utilizes teacher and student models to predict labels for unlabeled data, and UDA (Xie et al., 2019) that just uses  $p(\mathbf{x}_i^u)$  as generated labels.
To avoid the weighted average being too uniform, we utilize a sharpening function over predicted labels. Given a temperature hyper-parameter  $T$ :
$$
\operatorname {S h a r p e n} \left(\mathbf {y} _ {i} ^ {u}, T\right) = \frac {\left(\mathbf {y} _ {i} ^ {u}\right) ^ {\frac {1}{T}}}{\left| \left(\mathbf {y} _ {i} ^ {u}\right) ^ {\frac {1}{T}} \right| | _ {1}},
$$
where  $||.||_1$  is  $l_{1}$ -norm of the vector. When  $T\to 0$  the generated label becomes a one-hot vector.
4.3 TMix on Labeled and Unlabeled Data
After getting the labels for unlabeled data, we merge the labeled text  $\mathbf{X}_l$ , unlabeled text  $\mathbf{X}_u$  and unlabeled augmentation text  $\mathbf{X}_a = \{x_{i,k}^a\}$  together to form a super set  $\mathbf{X} = \mathbf{X}_l \cup \mathbf{X}_u \cup \mathbf{X}_a$ . The corresponding labels are  $\mathbf{Y} = \mathbf{Y}_l \cup \mathbf{Y}_u \cup \mathbf{Y}_a$ , where  $\mathbf{Y}^a = \{\mathbf{y}_{i,k}^a\}$  and we define  $\mathbf{y}_{i,k}^a = \mathbf{y}_i^u$ , i.e., the all augmented samples share the same generated label as the original unlabeled sample.
In training, we randomly sample two data points  $\mathbf{x},\mathbf{x}^{\prime}\in \mathbf{X}$  , then we compute  $\mathrm{TMix}(\mathbf{x},\mathbf{x}^{\prime})$ $\mathrm{mix}(\mathbf{y},\mathbf{y}^{\prime})$  and use the KL-divergence as the loss:
$$
L _ {\mathrm {T M i x}} = \mathbb {E} _ {\mathbf {x}, \mathbf {x} ^ {\prime} \in \mathbf {X}} \mathrm {K L} (\operatorname {m i x} (\mathbf {y}, \mathbf {y} ^ {\prime}) | | p (\mathrm {T M i x} (\mathbf {x}, \mathbf {x} ^ {\prime}))
$$
Since  $\mathbf{x},\mathbf{x}^{\prime}$  are randomly sampled from  $\mathbf{X}$ , we interpolate text from many different categories: mixup among among labeled data, mixup of labeled and unlabeled data and mixup of unlabeled data. Based on the categories of the samples, the loss can be divided into two types:
Supervised loss When  $\mathbf{x} \in \mathbf{X}_l$ , the majority information we are actually using is from the labeled data, hence training the model with supervised loss.
Consistency loss When the samples are from unlabeled or augmentation set, i.e.,  $\mathbf{x} \in \mathbf{X}^u \cup \mathbf{X}^a$ , most information coming from unlabeled data, the KL-divergence is a type of consistency loss, constraining augmented samples to have the same labels with the original data sample.
4.4 Entropy Minimization
To encourage the model to produce confident labels on unlabeled data, we propose to minimize the entropy of prediction probability on unlabeled data as a self-training loss:
$$
L _ {\mathrm {m a r g i n}} = \mathbb {E} _ {\mathbf {x} \in \mathbf {X} _ {u}} \max  (0, \gamma - | | \mathbf {y} ^ {u} | | _ {2} ^ {2}),
$$
where  $\gamma$  is the margin hyper-parameter. We minimize the entropy of the probability vector if it is larger than  $\gamma$ .
Combining the two losses, we get the overall objective function of MixText:
$$
L _ {\mathrm {M i x T e x t}} = L _ {\mathrm {T M i x}} + \gamma_ {m} L _ {\mathrm {m a r g i n}}.
$$
5.1 Dataset and Pre-processing
We performed experiment with four English text classification benchmark datasets: AG News (Zhang et al., 2015), BPpedia (Mendes et al., 2012), Yahoo! Answers (Chang et al., 2008) and IMDB (Maas et al., 2011). We used the original test set as our test set and randomly sampled from the training set to form the training unlabeled set and development set. The dataset statistics and split information are presented in Table 1.
For unlabeled data, we selected German and Russian as intermediate languages for back translations using FairSeq², and the random sampling temperature was 0.9. Here is an example, for a news from AG News dataset: "Oil prices rallied to a record high above $55 a bar