SlowGuess/NAACL_2022_OCR huggingface dataset:
Hai-Ming Xu and Lingqiao Liu and Ehsan Abbasnejad
{hai-ming.xu, lingqiao.liu, ehsan.abbasnejad}@adelaide.edu.au
Semi-supervised learning is a promising way to reduce the annotation cost for text-classification. Combining with pre-trained language models (PLMs), e.g., BERT, recent semi-supervised learning methods achieved impressive performance. In this work, we further investigate the marriage between semi-supervised learning and a pre-trained language model. Unlike existing approaches that utilize PLMs only for model parameter initialization, we explore the inherent topic matching capability inside PLMs for building a more powerful semi-supervised learning approach. Specifically, we propose a joint semi-supervised learning process that can progressively build a standard  $K$ -way classifier and a matching network for the input text and the Class Semantic Representation (CSR). The CSR will be initialized from the given labeled sentences and progressively updated through the training process. By means of extensive experiments, we show that our method can not only bring remarkable improvement to baselines, but also overall be more stable, and achieves state-of-the-art performance in semi-supervised text classification. Code is available at: https://github.com/HeimingX/PCM.
Text classification is a fundamental task in natural language processing (NLP) and underpins various applications, e.g., spam detection (Jindal and Liu, 2007), sentiment analysis (Pang et al., 2002) and text summarization (Gambhir and Gupta, 2017). Supervised training of text classifiers often demands a large amount of annotation, which can be expensive for many applications. Semi-supervised learning (SSL) provides an economical way for alleviating this burden since it can make use of easy-accessible unlabeled samples to build a reasonably performed classifier with a limited amount of labeled data. Recently, SSL received increasing attention in both image classification (Tarvainen and
Valpola, 2017; Berthelot et al., 2019b; Sohn et al., 2020) and text classification (Xie et al., 2019b; Chen et al., 2020; Liu et al., 2021) areas.
Meanwhile, pre-trained language models (PLMs) (Yang et al., 2019a; Devlin et al., 2019; Radford et al., 2019) are developing rapidly and achieve impressive performance in various NLP tasks (Sun et al., 2019; Zhu et al., 2020) including text classification (Garg and Ramakrishnan, 2020). In the context of semi-supervised text classification, many existing methods achieve excellent performance by directly using a PLM as a sentence encoder and further fine-tuning it with a semi-supervised learning process (Xie et al., 2019b; Chen et al., 2020; Bhattacharjee et al., 2020; Sun et al., 2020).
In this paper, we further explore the usage of PLMs for SSL. We go beyond the strategy of using PLMs for encoder initialization and make full use of inner knowledge of PLMs. Concretely, we identify that some PLMs, e.g., BERT, have an inherent matching capability between sentence and class-related words thanks to its pre-training pretext task (Devlin et al., 2019) (as the examples shown in Fig. 1). We further propose to strengthen this capability through SSL on labeled and unlabeled data. Specifically, we develop a joint training process to update three components progressively, that is, a classifier that performs the standard  $K$ -way classification, a class semantic representation (CSR) that represents the semantic of each category, and a matching classifier that matches the input sentence against the CSR. Those three components can help each other during the training process, i.e., the  $K$ -way classifier will receive more accurate pseudo-labels by jointly generating pseudo-labels with the matching classifier; the matching classifier will also upgrade its matching capability with the guidance of the  $K$ -way classifier. The CSR will become more accurate and comprehensive with the improvement of the  $K$ -way classifier and matching
Human Language Technologies, pages 3003 - 3013
(a) Sentence example on class "family"
(b) Sentence example on class "sports"
Figure 1: Visualization of the inherent matching capability of BERT on examples from Yahoo! Answers. We append class semantic-related words (CSW) of all classes at the end of input sentence. Different colors denote different classes. The color on each token of input sentence represents the category of its most attended CSW (with color brightness indicating the attention value, please see Sec.3 for more details). The histograms on the right demonstrate the cosine similarity between the average features of sentence and features of each CSW.
classifier. This joint process leads to a more powerful semi-supervised learning algorithm for the text classification task. Throughout our experimental evaluation, we demonstrate that the proposed method achieves the state-of-the-art performance on text data, especially when the number of labeled sentences becomes extremely low, i.e., 3 or 5.
In this section, we briefly review the relevant research works.
2.1 General Semi-Supervised Learning
Semi-supervised learning is a longstanding research topic in machine learning. Existing methods adopt different ways of utilizing unlabeled samples, e.g., "transductive" models (Joachims, 2003; Gammerman et al., 2013), multi-view style approaches (Blum and Mitchell, 1998; Zhou and Li, 2005) and generative model-based methods (Kingma et al., 2014; Springenberg, 2016). With the renaissance of the deep neural net
work, consistency-regularization-based deep SSL approaches (Laine and Aila, 2017; Tarvainen and Valpola, 2017; Miyato et al., 2018) have achieved impressive performance on various tasks, and our work largely builds upon the method in this category. The key idea of these methods is to constrain the model to be consistent in the neighborhood of each sample in the input space. Specifically, II-Model (Laine and Aila, 2017), UDA (Xie et al., 2019b) and FixMatch (Sohn et al., 2020) directly add various perturbations to the input data, Mean-teacher (Tarvainen and Valpola, 2017) uses a teacher model to simulate sample perturbation, and Virtual Adversarial Training (Miyato et al., 2018) skillfully constructs an adversarial sample. More recently, mixup (Zhang et al., 2018) method proposed another kind of consistency constraint that requires the input and output of the model to satisfy an identical linear relationship. Based on this technique, many state-of-the-art methods are published, e.g., ICT (Verma et al., 2019b), MixMatch (Berthelot et al., 2019b) and ReMixMatch (Berthelot
et al., 2019a).
2.2 Semi-Supervised Text Classification
Semi-supervised learning has gained a lot of attention in the field of text classification. Many recent semi-supervised text classification methods focus on how to adapt the existing SSL methodologies to the sentence input. (Miyato et al., 2017) applied perturbations to word embeddings for constructing adversarial and virtual adversarial training. (Clark et al., 2018) designed auxiliary prediction modules with restricted views of the input to encourage consistency across views. With the development of PLMs, (Jo and Cinarel, 2019) performed self-training between two sets of classifiers which are initialized differently, one with pre-trained word embeddings and random values for the other. Both (Xie et al., 2019b) and (Chen et al., 2020) took the pre-trained BERT to initialize the sentence feature extractor, where the former conducted consistency-regularization between the original sentence and its back-translation generated one, and the latter further introduced the manifold mixup (Verma et al., 2019a) into text classification. Although these methods may achieve decent performances, we believe that they haven't fully explored the inherent knowledge in a PLM. Our work takes a step further in this direction.
3 Inherent matching capability of a PLM
In this section, we will demonstrate the inherent topic matching capability of BERT which motivates our method. Utilizing PLMs for a downstream task has become common since it often brings a significant performance boost (Zhu et al., 2020; Chen et al., 2020). In the context of semi-supervised learning, a PLM is usually employed for initializing the network before performing semi-supervised training. However, the value of a PLM can go beyond a good initial model or feature extractor. In particular, a PLM like BERT has already learned certain topic matching capabilities thanks to its pretext tasks. For example, BERT uses the next sentence prediction (NSP) as one of its pretext tasks. In this task, the network is asked to discern if two input sentences are two successive sentences in the original corpus. After training on this task, BERT can implicitly acquire topic matching capability since two successive sentences in a paragraph usually share the same topic.
Fig. 1 shows a concrete investigation of the in
herent matching capability of BERT. Following the NSP task, we concatenate the sentence and class semantic-related words  $C_k$ , e.g., "sports", via the format: "[CLS] sentence [SEP]  $C_1 \cdots C_k \cdots C_K$  [SEP]". Then we pass the input sequence to a pretrained BERT and calculate the attention value of each token with respect to each class name. Specifically, this attention value is calculated by averaging the last layer self-attention values across all heads between a token and the appended word  $C_k$ . For better visualization, we use different color to show the class that leads to the largest attention value (indicated by the color brightness).
From the visualization, we can see that BERT can automatically match keywords corresponding to the respective class names. Moreover, we find that if we replace the class names with words under the same topic, i.e., family  $\rightarrow$  boyfriend, sports  $\rightarrow$  football, the words related to the ground-truth class can still be attended, as shown in Fig. 1a and 1b.
Finally, we extract BERT last-layer's feature corresponding to each class word  $C_k$  and average features align with sentence tokens, and compare the cosine similarity between them. As histograms shown in Fig. 1a and 1b, we can find that the correct class leads to the highest matching score, although not always by a large margin.
4 Progressive Class-semantic Matching
To further strengthen the above topic matching capability and use it for classification, we propose to progressively build a sentence-class matching model through the framework of semisupervised learning. Formally, we aim to build a classifier from a few annotated samples  $\mathcal{L} = \{x_1,x_2,\dots ,x_{n_l}\}$ , whose labels are  $\mathcal{V} = \{y_1,y_2,\dots ,y_{n_l}\}, y_i\in \{1,\dots ,k,\dots ,K\}$  and a large amount of unlabeled samples  $\mathcal{U} = \{x_1,x_2,\dots ,x_{n_u}\}$  (where  $n_l\ll n_u$ ).
The idea is to construct a process that can jointly update three components: (1) a standard  $K$ -way classifier (2) a matching classifier which matches texts against class semantic representation (3) the class semantic representation (CSR) itself. The update of each component will help other components and thus can iteratively bootstrap classification performance. We call our method as Progressive Class-semantic Matching (PCM).
Figure 2: Overview of the proposed PCM model. Lines in the same color indicate how the information travels in our model.  $\{C_k\}$  denotes the set of class semantic-related words. "avg" means the average of word embeddings within the same class. "GAP" represents the global average pooling of the input text features. "concat" is a feature concatenate operation. We clarify the details of initializing and updating of  $\{C_k\}$  in Secs. 4.2 and 4.3.
4.1 Three components of PCM
Fig. 2 shows how we realize the three components. Similar to the example in Section 3, we construct the input to the BERT by concatenating sentence with class semantic-related words  $\{C_i\}, i \in \{1, \dots, k, \dots, K\}$ . Considering the size of  $\{C_i\}$  may vary and the computation cost may increase heavily when the number of classes grows, we calculate an average of embeddings of all words belonging to the same class before passing them to the pre-trained BERT encoder. This average embedding is called class semantic representation (CSR).
The last layer output features corresponding to tokens in the input text are averaged and treated as the sentence representation. On top of the sentence representation, we build a standard  $K$ -way classifier. We implement it by a two-layer MLP and it will output a set of logits  $\{o_i^s\}$  called semantic logits and posterior probabilities  $\{p_i^s\}$  after applying Softmax to  $\{o_i^s\}$ .
In addition to the  $K$ -way classifier, we also build a class-sentence matching classifier which is realized by another MLP applying to the concatenation between the sentence representation and the output features corresponding to each CSR. The output of this matching classifier is called matching logits
$\{o_i^m\}$  and Sigmoid function is applied to convert it into the probabilistic form, denoted as  $\{p_i^m\}$ . Note that the matching classifier is realized in a multi-label formulation, that is, the summation of  $\{p_i^m\}$  over all classes is not necessarily equal to 1. It allows the scenario that a sentence matches more than one class and the case that a sentence does not match any class. This design avoids the case that achieving high matching probability for one class merely because its matching score is higher than those of other classes (but it actually with low absolute matching logits for all classes). We empirically find that using this mechanism is helpful for the matching classifier (but not necessarily for the  $K$ -way predictor as discussed in Section 5.2).
4.2 Initialization of CSR
The proposed PCM model requires an initial CSR, i.e., the average word embedding of a set of class semantic-related words, to start the iteration. Although manually choosing a list of seed words (e.g., class names) can be an ideal way for the CSR initialization, it may suffer from leveraging prior knowledge and leads to an unfair comparison to existing SSL algorithms. An alternative approach is to automatically identify a set of class semantic-related words. This might be useful for the case
that class names in some corpora do not carry a clear semantic meaning, e.g., the rating of reviews.
In this paper, we use the following method to automatically collect the class semantic-related words: we start by fine-tuning a pre-trained BERT classifier on the labeled set. Then passing each labeled text into the fine-tuned model and calculate attention values for each token. The attention value of a token is calculated by averaging all the attention received for this token<sup>1</sup>. After removing stop words, we retain the top- $j$  e.g.,  $j = 75$ , attended words for each class to calculate the initial CSR.
4.3 Update of three components
The three components are progressively updated by seamlessly incorporating them into an SSL framework. In particular, our method is built upon UDA (Xie et al., 2019b), one of the state-of-the-art approaches in semi-supervised text classification. The idea is to first construct an augmented version of unlabeled data by back translation (Edunov et al., 2018) and then enforce the prediction to be consistent through a consistency-regularization loss for unlabeled data. The following describes the detailed updating process:
Update of the standard  $K$ -way classifier and the class-sentence matching classifier: The update is performed on labeled and unlabeled data at the same time. For labeled data, both classifiers are updated by performing stochastic gradient descent with the following objective function.
$$
\begin{array}{l} \mathcal {L} _ {l} = \frac {1}{n _ {l}} \sum_ {j = 1} ^ {n _ {l}} \sum_ {i = 1} ^ {K} \underbrace {- \mathbb {I} _ {i} ^ {j} \log p _ {i} ^ {s} (x _ {j})} _ {\text {c r o s s e n t r o p y (C E)}} + \\ \underbrace {- \mathbb {I} _ {i} ^ {j} \log p _ {i} ^ {m} \left(x _ {j}\right) - \left(1 - \mathbb {I} _ {i} ^ {j}\right) \log \left(1 - p _ {i} ^ {m} \left(x _ {j}\right)\right)} _ {\text {b i n a r y c r o s s e n t r o p y (B C E)}} \tag {1} \\ \end{array}
$$
where  $p_i^s (x_j)$  and  $p_i^m (x_j)$  are the probabilities of  $x_{j}$  belonging to class  $i$  from the view of the  $K$ -way classifier and the matching classifier, respectively. Since the matching classifier is designed in a multi-label style, we use binary cross-entropy loss for it.  $\mathbb{I}_i^j$  is an indicator whose value equals to 1 if  $y_{j} = i$ , and 0 otherwise.
For unlabeled data, we follow UDA to use a student-teacher alike training strategy, that is, we first use the original sentence input  $x_{j} \in \mathcal{U}$  to
obtain the prediction target (similar to a pseudo label) and then enforce the prediction of the back-translated version  $x_{j}^{a}$  of  $x_{j}$  being close to the prediction target. Formally, if the prediction of one unlabeled sample satisfies all the following rules, the prediction target will be generated:
$$
\left\{ \begin{array}{l} \max  _ {i} \left(p _ {i} ^ {s} \left(x _ {j}\right)\right) > = \operatorname {c o n f i d} 1 \\ \max  _ {i} \left(p _ {i} ^ {m} \left(x _ {j}\right)\right) > = \operatorname {c o n f i d} 2 \\ \arg \max  _ {i} \left(p _ {i} ^ {s} \left(x _ {j}\right)\right) = = \operatorname {a r g m a x} _ {i} \left(p _ {i} ^ {m} \left(x _ {j}\right)\right) \end{array} \right. \tag {2}
$$
where  $\text{confid1}$  and  $\text{confid2}$  are two pre-defined confidence thresholds and we empirically find  $\text{confid1} = 0.95$  and  $\text{confid2} = 0.7$  performs well in our experiments. For the  $K$ -way classifier, the pseudo prediction target is a sharpened posterior probability, i.e.,  $\hat{p}^s = \operatorname{Softmax}(o^s / T)$  with  $T \leq 1$ . For the matching classifier, we directly generate a pseudo-label by  $\hat{y}_i = \operatorname{argmax}_i p_i^m$ . The loss function for the unlabeled data is
$$
\begin{array}{l} \mathcal {L} _ {u} = \frac {1}{n _ {u}} \sum_ {j = 1} ^ {n _ {u}} \left(\mathrm {K L} \left(p ^ {s} \left(x _ {j} ^ {a}\right), \hat {p} ^ {s} \left(x _ {j}\right)\right) \right. \tag {3} \\ \left. + \operatorname {B C E} \left(p ^ {m} \left(x _ {j} ^ {a}\right), \hat {y} _ {i} \left(x _ {j}\right)\right)\right) \\ \end{array}
$$
where  $\mathrm{KL}(\cdot ,\cdot)$  denotes the KL divergence.
Update of CSR: The initialized CSR might not be accurate or comprehensive enough to represent the class semantics. Similar to the approach proposed in Section 4.2, we use the newly updated model to collect a better CSR. The collection process on labeled sentences is still as described in Section 4.2. While the same extraction operation is performed on unlabeled texts only when they satisfy the conditions in Eq. 2. We update the CSR whenever the number of validation set<sup>2</sup> samples meeting conditions in Eq. 2 increases. Generally, during the course of semi-supervised learning, the classifiers become stronger and the selected class-related words tend to become more accurate. Table 3 gives an example to show the difference of most attended words between initialization and after training.
In this section, we perform the experimental study of the PCM method on four text datasets.
Datasets Following MixText (Chen et al., 2020), we use four datasets, namely, AG News (Zhang
2Please note that we do not use any label information here.
$^{1}$ Magnitude of the attention value indicates the importance of this token.
Dataset	Label Type	# Classes	# Unlabeled	# Test
AG News	News Topic	4	20,000	7,600
DBpedia	Wikipedia Topic	14	70,000	70,000
Yahoo! Answers	QA Topic	10	50,000	60,000
IMDB	Review Sentiment	2	10,000	25,000
Table 1: Statistics of four text datasets.
Dataset	Method	Number of Labeled Example Per Clas
3	5	10	20	50
AG News	BERT-FT	76.70±4.72	79.90±2.34	83.46±2.73	84.97±1.73	87.35±0.56
UDA	78.25±7.61	82.97±2.87	86.75±0.88	86.77±0.10	88.23±0.49
MixText	81.60±9.04	85.84±1.32	85.56±2.95	87.60±0.48	88.14±0.75
PCM(ours)	84.85±0.86	87.20±0.42	88.31±0.47	88.34±0.27	88.85±0.27
DBpedia	BERT-FT	86.68±2.59	91.86±2.46	96.60±0.46	97.84±0.23	98.59±0.22
UDA	93.51±2.23	95.88±2.78	97.26±1.50	98.59±0.04	98.93±0.06
MixText	93.25±0.68	96.93±0.41	98.39±0.09	98.64±0.18	98.84±0.05
PCM(ours)	94.37±0.49	97.04±0.68	98.70±0.04	98.80±0.06	99.07±0.05
Yahoo!	BERT-FT	45.93±3.67	50.75±4.32	61.84±2.37	63.89±0.94	67.29±0.68
UDA	48.30±11.09	57.09±5.69	65.15±1.54	67.76±0.60	69.38±0.78
MixText	60.27±4.29	65.77±1.78	67.23±1.97	68.19±1.33	69.11±0.73
PCM(ours)	63.52±2.63	67.09±0.54	68.34±1.03	69.21±0.42	70.28±0.47
IMDB	BERT-FT	60.11±2.41	65.17±8.39	73.20±2.97	78.70±6.75†	83.91±1.13
UDA	63.01±1.07	71.90±10.80	89.05±1.70	90.20±0.54‡	90.41±0.45
MixText	56.27±3.46	71.89±4.89	83.38±3.35	86.27±1.36	88.30±1.24
PCM(ours)	73.86±1.04	86.06±0.74	89.94±0.44	91.10±0.28	91.15±0.15
† Single run accuracy (81.6%) is reported in UDA (Xie et al., 2019b) for a reference. ‡ This number is reasonable on one GPU card with 11GB memory. See experimental tutorial (Xie et al., 2019a) for details.
Table 2: Test accuracy (%) of all comparing methods on four datasets. Models are trained with 3/5/10/20/50 labeled data per class.  $\pm$  denotes the Standard Error of the Mean (S.E.M.) over three random sampled label sets. Best results are indicated as bold.
et al., 2015), DBpedia (Lehmann et al., 2015), Yahoo! Answers (Chang et al., 2008), and IMDB (Maas et al., 2011) for our experiments. We use the same data splits as in MixText (Chen et al., 2020). The detailed statistics of the four datasets are presented in Table 1.
Implementation details Same as MixText  $^{3}$ , we use back-translation to perform data augmentation. Two languages, German and Russian, are chosen as the intermediate language. The back-translation texts on Yahoo! Answers are provided by MixText, and we directly use them. For the other three datasets, we generate the back-translation data by ourselves (with Fairseq toolkit (Ott et al., 2019)).
We use the input format "[CLS] Sentence [SEP]"
for all the baseline methods. We empirically find this format leads to the overall best performance. Meanwhile, this format actually brings performance improvement to both UDA and MixText methods. So we are comparing against stronger baselines in our paper.
Due to BERT's length limit, we only kept the last 256 tokens for IMDB and the first 256 tokens for the other datasets during training. We use the same learning-rate setting for all methods: 5e-6 for the BERT encoder and 5e-4 for the classifier (i.e., a two-layer MLP with a 128 hidden size and tanh as its activation function). All our experiments were run on a GeForce RTX 2080 Ti GPU and each experiment takes around 5 hours.
Comparing methods We compare the proposed PCM method with three baselines: (1) fine-tuning the pre-trained BERT-based-uncased model on the
$^{3}$ https://github.com/GT-SALT/MixText (2-clause BSD License)
Figure 3: Accuracy on varying number of unlab. data.
labeled texts directly, denote as BERT-FT. (2) Unsupervised data augmentation method (UDA) (Xie et al., 2019b) and (3) the recently proposed MixText method (Chen et al., 2020). To make a fair comparison, we conduct all experiments based on the same codebase released by the authors of MixText (Chen et al., 2020).