SlowGuess/NAACL_2022_OCR huggingface dataset:
Minyi Zhao<sup>1</sup> Lu Zhang<sup>1</sup> Yi Xu<sup>1</sup>  
Jiandong Ding<sup>2</sup> Jihong Guan<sup>3</sup> Shuigeng Zhou<sup>1*</sup>
<sup>1</sup>Shanghai Key Lab of Intelligent Information Processing, and School of
Computer Science, Fudan University, China
2Alibaba Group Tongji University, China
$^{1,2}$  {zhaomy20, l_zhang19, yxu17, jdding, sgzhou} @fudan.edu.cn  $^{3}$ jhguan@tongji.edu.cn
Recent works have empirically shown the effectiveness of data augmentation (DA) for NLP tasks, especially for those suffering from data scarcity. Intuitively, given the size of generated data, their diversity and quality are crucial to the performance of targeted tasks. However, to the best of our knowledge, most existing methods consider only either the diversity or the quality of augmented data, thus cannot fully tap the potential of DA for NLP. In this paper, we present an easy and plug-in data augmentation framework EPiDA to support effective text classification. EPiDA employs two mechanisms: relative entropy maximization (REM) and conditional entropy minimization (CEM) to control data generation, where REM is designed to enhance the diversity of augmented data while CEM is exploited to ensure their semantic consistency. EPiDA can support efficient and continuous data generation for effective classifier training. Extensive experiments show that EPiDA outperforms existing SOTA methods in most cases, though not using any agent network or pre-trained generation network, and it works well with various DA algorithms and classification models. Code is available at https://github.com/zhaominyiz/EPiDA.
Data augmentation (DA) is widely-used in classification tasks (Shorten and Khoshgoftaar, 2019; Feng et al., 2021; Zhang et al., 2021). In computer vision (CV), (Krizhevsky et al., 2012; Chatfield et al., 2014; Szegedy et al., 2015) adopt strategies like flipping, cropping, tilting to perform DA. In natural language processing (NLP), (Xie et al., 2017; Coulombe, 2018; Niu and Bansal, 2018; Wei and Zou, 2019) find that native augmentation skills such as spelling errors, synonym replacement, deleting and swapping, can bring considerable performance improvement. All these methods use
various transformations for data augmentation, but they do not achieve equal success in different NLP tasks (Yang et al., 2020). Sometimes, they fail to guarantee semantic consistency, and may even bring semantic errors that are harmful to classification. The reason lies in that data augmentation for NLP is in discrete space, so it can easily incur large deviation of semantics (e.g. in sentiment classification task, deleting emotional words from a sentence will make its meaning completely different).
Generally, given the size of generated data, their diversity and quality are crucial to the performance of targeted tasks (Ash et al., 2019). Recent works have begun to emphasize the diversity or quality of augmented data. For example, in CV, AA (Cubuk et al., 2019), Fast-AA (Lim et al., 2019) and LTA (Luo et al., 2020) employ agent networks to learn how to enhance diversity. In NLP, language models are widely used to control generation quality, including Back-translation (Sennrich et al., 2016; Yu et al., 2018), Seq2seq models (Kobayashi, 2018; Kumar et al., 2019; Yang et al., 2020), GPT-2 (Radford et al., 2019; Anaby-Tavor et al., 2020; Quteineh et al., 2020; Liu et al., 2020) and T5 (Dong et al., 2021). In addition, some works (Morris et al., 2020) in NLP utilize adversarial augmentation to enrich the diversity of the samples. However, to the best of our knowledge, most existing works consider only either the quality or the diversity of augmented data, so cannot fully exploit the potential of data augmentation for NLP tasks. Besides, recent existing DA methods for NLP tasks usually resort to pre-trained language models, are extremely inefficient due to huge model complexity and tedious finetuning, which limits the scope of their applications.
In this paper, we propose a new data augmentation framework for text classification. This framework is called EPiDA (the abbreviation of Easy Plug-in Data Augmentation), which employs two mechanisms to control the diversity and quality
Human Language Technologies, pages 4742 - 4752
Figure 1: The pipeline of EPiDA. EPiDA first evaluates the candidate samples from the perspectives of diversity and quality, then takes the samples of high diversity and quality as training samples. It can work with various DA algorithms and classification models.
of augmented data: relative entropy maximization (REM) and conditional entropy minimization (CEM), where the former is for boosting diversity while the latter for ensuring quality. Fig. 1 shows the pipeline of EPiDA. EPiDA consists of a DA algorithm, a classifier, and a Sample Evaluation And Selection (SEAS) module. SEAS works with the DA algorithm and the classifier, and evaluates the candidate samples with the feedback of the classifier. With REM and CEM, SEAS can select samples of high diversity and quality to train the classifier continuously until the model converges.
1. We propose an easy plug-in data augmentation framework EPiDA for text classification. EPiDA can work with various existing DA algorithms and classification models, it is general, efficient, and easy-to-deploy.
2. We design two mechanisms relative entropy maximization (REM) and conditional entropy minimization (CEM) to boost the diversity and quality of augmented data simultaneously in an explicit and controllable way.
3. We conduct extensive experiments to evaluate EPiDA. Experimental results show that EPiDA outperforms existing DA methods, and works well with different DA algorithms and classification models.
The rest of this paper is organized as follows: Sec. 2 reviews related work and highlights the differences between our work and major existing methods. Sec. 3 introduce our method in details. Sec. 4 presents the results of performance evaluation, and Sec. 5 concludes the paper.
In this section, we first review the related work of DA for NLP, then expound the differences between our method and the major existing ones. According to the methodology of data generation, existing methods can be categorized into three types: rule-based, interpolation-based, and model-based, respectively.
2.1 Rule-Based Methods
These works use easy and predetermined transformations without model components. (Kolomiyets et al., 2011; Zhang et al., 2015; Wang and Yang, 2015) use synonyms to replace words. EDA (Wei and Zou, 2019) and AEDA (Karimi et al., 2021) introduce random insertions, swaps, and deletions. Xie et al. (2017) employed spelling errors to augment sentences. Sahin and Steedman (2018) conducted sentence rotating via dependency tree morphing. Wei et al. (2021) proposed a multi-task view of DA.  $\mathrm{SUB}^2$  (Shi et al., 2021) generates new examples by substituting substructures via constituency parse trees. Although these methods are easy to implement, they do not consider controlling data quality and diversity.
2.2 Interpolation-Based Methods
MIXUP (Zhang et al., 2017) pioneers this type of works by interpolating the input and labels of two or more real examples. Recently, many MIXUP strategies (Verma et al., 2019; Yun et al., 2019) were proposed in CV. Due to the discrete nature of inputs of NLP tasks, such methods can be applied to NLP tasks only via padding and mixing embeddings or higher hidden layers (Chen et al., 2020; Si et al., 2021).
2.3 Model-Based Methods
Seq2seq and language models have been used to generate high quality samples. Among these approaches, Back-translation (Sennrich et al., 2016; Yu et al., 2018) translates sentences into another language and then translates it back to the original language. RNNs and transformers are used to reconstruct sub-parts of real data with contextual information (Kobayashi, 2018; Gao et al., 2019; Yang et al., 2020). Recently, we have witnessed the great success of large-scale pre-trained language models (PLMs) such as BERT (Devlin et al., 2019), XLNet (Yang et al., 2019), GPT-2 (Radford et al., 2019) in NLP tasks. These state-of-the-art PLMs
Method	Div	Qua	LM	FB
AA (Cubuk et al., 2019)	✓	×	✓	✓
EDA (Wei and Zou, 2019)	✓	×	×	×
DataBoost (Liu et al., 2020)	×	✓	✓	×
LearnDA (Zuo et al., 2021)	✓	✓	✓	×
VDA (Zhou et al., 2021)	✓	✓	✓	×
Ours EPiDA	✓	✓	×	✓
Table 1: A qualitative comparison between EPiDA and major existing methods from four aspects: whether controlling the diversity and quality of the augmented data, whether using language model or agent network and whether using the feedback of the classifier.
are also widely used to augment samples (Ng et al., 2020; Nie et al., 2020; Anaby-Tavor et al., 2020; Quteineh et al., 2020; Liu et al., 2020; Dong et al., 2021). For example, DataBoost (Liu et al., 2020) develops a reinforcement learning strategy to guide the conditional generation without changing the architecture of GPT-2. Besides, adversarial augmentation (i.e., attack, GANs) are also used to enrich the diversity of the generated samples (Morris et al., 2020; Simoncini and Spanakis, 2021). Although model-based methods can control generation quality well via PLMs, they are computationally inefficient, which limits their applications.
2.4 Differences between EPiDA and Existing Methods
To expound the differences between EPiDA and typical existing methods, in Tab. 1 we present a qualitative comparison from four dimensions: whether controlling the diversity and quality of the augmented data, whether using pre-trained model (language model or agent network), and whether using the feedback from the classifier.
As shown in Tab. 1, among the existing methods, most control only either diversity (e.g. AA and EDA) or quality (e.g. DataBoost) of augmented data, thus cannot completely leverage the potential of data augmentation. And most use language model or agent network, which is beneficial to data quality but also inefficient. Only the recent LearnDA (Zuo et al., 2021) and VDA (Zhou et al., 2021) consider both diversity and quality, and only AA uses feedback of the classifier. Our EPiDA addresses both diversity and quality of augmented data via the feedback of the classifier in an explicit and controllable way, without the help of any additional model components, which makes it not only more effective but also more efficient.
Note that in addition to the differences listed in Tab. 1, our method EPiDA differs from LearnDA in at least three other aspects: 1) LearnDA employs perplexity score (PPL) and cosine similarity to measure diversity and quality respectively, while EPiDA adopts two mechanisms relative entropy maximization (REM) and conditional entropy minimization (CEM) to control diversity and quality, which is theoretically more rational and solid. 2) LearnDA is for event causality identification, while EPiDA is mainly for text classification. 3) LearnDA needs knowledge guidance, while EPiDA does not. These make it difficult to evaluate LearnDA in our experimental settings. Thus, we do not conduct performance comparison between EPiDA and LearnDA. Nevertheless, in our ablation study, we replace REM and CEM with PPL and cosine similarity in EPiDA, and our experimental results show that EPiDA with REM and CEM performs better than that with PPL and cosine similarity. Besides, comparing with VDA that requires PLM to provide substitution probability, EPiDA is free of PLMs, and is more effective, efficient and practical.
As shown in Fig. 1, EPiDA consists of three components: a DA algorithm  $T$ , a classifier or classification model  $C$ , and a Sample Evaluation and Selection (SEAS) module that is the core component of EPiDA. Generally, the DA algorithm and the classifier can be any of existing DA algorithms and classifiers. With the feedback of the classifier, SEAS evaluates candidate samples generated by the DA algorithm in terms of diversity and quality via the Relative Entropy Maximization (REM) mechanism and the Conditional Entropy Minimization (CEM) mechanism, and outputs the qualified samples to further train the classifier. So EPiDA can serve as a plug-in component to boost existing DA algorithms for training better target models.
3.1 The Rationale to Control DA
Consider a classification task with a dataset  $X$  of  $n$  samples:  $X = \{(x_{1},y_{1}),(x_{2},y_{2}),\dots,(x_{n},y_{n})\}$ . Here,  $x_{i}$  is a sample,  $y_{i}$  is its label. The loss function is
$$
L _ {o} (\omega) = \frac {1}{n} \sum_ {i = 1} ^ {n} l \left(\omega^ {\top} \phi \left(x _ {i}\right); y _ {i}\right). \tag {1}
$$
where  $\phi :\mathbb{R}^d\to \mathbb{R}^D$  is a finite-dimensional feature map,  $\omega \in \mathbb{R}^{D}$  means learnable parameters, and  $l$
can be a common loss function like cross-entropy.
Now we employ a DA algorithm  $T$  to conduct augmentation for each sample in  $X$ . Let  $t_i^j$  be the  $j$ -th sample generated by  $T$  with  $x_i$  as input, and  $m$  samples are generated from  $x_i$ , the loss function for the generated samples can be written as
$$
L _ {g} (\omega) = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {1}{m} \sum_ {j = 1} ^ {m} l \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right); y _ {i}\right). \tag {2}
$$
Here, we assume 1)  $t_i^j$  and  $x_{i}$  have the same label  $y_{i}$ , so we can use  $y_{i}$  to optimize the new loss function; 2) Data augmentation does not significantly change the feature map  $\phi$ , that is, augmentation can maintain semantic consistency of the sample space. Now we combine the augmented samples into the original samples, thus the total loss function of EPiDA can be written as follows:
$$
L (\omega) = L _ {o} (\omega) + L _ {g} (\omega). \tag {3}
$$
Recall that we use the feedback of the classifier  $C$  to select samples. Specifically, we use the original training samples  $X$  to pre-train the classifier  $C$ , and for each generated sample  $t_i^j$ , the feedback signal about  $t_i^j$  from the classifier is used for evaluating  $t_i^j$ . When the generation process is over, all generated samples  $\{t_i^j\}$  are used to train  $C$  again.
First, we consider how to generate samples of high diversity. Intuitively, generated samples should be different from the original samples. Recalling that the classifier  $C$  is pretrained by  $X$ , so for generated sample  $t_i^j$ , its loss  $l(\omega^\top \phi(t_i^j); y_i)$  should be large. In this sense, given the classifier  $C$  ( $\omega$  is fixed), we select samples that meet the following objective function:
$$
\max  _ {t _ {i} ^ {j}} L _ {g} (\omega , \phi (t _ {i} ^ {j})), \tag {4}
$$
which means that we are to generate "hard" samples for the classifier to cope with.
Second, we consider how to control the quality of augmented data. Recall that we assume for each augmented sample  $t_i^j$ , its label  $y_i$  keeps unchanged, so we can use the original label to evaluate the loss function. However, due to the discrete nature of language, it is nontrivial for augmented samples to meet this assumption. Taking the sentiment analysis task for example, suppose we use EDA (Wei and Zou, 2019) to augment  $x_i =$  "you'll probably love it", EDA may delete the word "love". Obviously,
the resulting sentence breaks the semantic consistency. To guarantee semantic consistency, we limit the semantic deviation of  $\phi (t_i^j)$  from  $\phi (x_{i})$ . Let  $M$  and  $\rho$  be a metric function to measure semantic difference between samples and a threshold respectively, we impose the following constraint on  $\phi (t_i^j)$ :
$$
\left| M \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right), \omega^ {\top} \phi \left(x _ {i}\right)\right) \right| \leq \rho . \tag {5}
$$
Thus, we can enhance data diversity by optimizing Eq. (4), and improve data quality using Eq. (5). The problem turns to solve Eq. (4) and Eq. (5).
3.2 Relative Entropy Maximization
We rewrite the objective function in Eq. (4) via:
$$
\begin{array}{l} L _ {g} \left(\omega , \phi \left(t _ {i} ^ {j}\right)\right) = \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} l \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right); y _ {i}\right) \tag {6} \\ = \frac {1}{n m} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {m} D (p (\omega^ {\top} \phi (t _ {i} ^ {j})), p (y _ {i})) + H (p (y _ {i})), \\ \end{array}
$$
where  $p, H, D$  indicate probability distribution, Shannon entropy, and relative entropy respectively, and actually  $H(p(y_i)) = 0$  since  $p(y_i)$  is a one-hot vector. According to Eq. (6), we try to augment samples with large relative entropy under the given labels. Thus, we call this method relative entropy maximization (REM) mechanism. As relative entropy measures the difference between the two distributions  $p(\omega^\top \phi(t_i^j))$  and  $p(y_i)$ , the larger the difference is, the more diverse the augmented sample is. Therefore, we define the diversity score  $s_{div}^{ij}$  of augmented sample  $t_i^j$  as follows:
$$
s _ {d i v} ^ {i j} = D \left(p \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right)\right), p \left(y _ {i}\right)\right). \tag {7}
$$
3.3 Conditional Entropy Minimization
We use conditional entropy as the metric function in Eq. (5) to constrain the semantic deviation of  $\phi(t_i^j)$  from  $\phi(x_i)$ , i.e.,  $M(\cdot, \cdot) \coloneqq H(\cdot|\cdot)$ . Then, Eq. (5) can be rewritten to
$$
H \left(p \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right)\right) \mid p \left(\omega^ {\top} \phi \left(x _ {i}\right)\right)\right) \leq \rho . \tag {8}
$$
where  $H(\cdot ,\cdot)$  is conditional entropy. Furthermore, to meet Eq. (8), we select samples  $\{t_i^j\}$  by solving the following optimization problem:
$$
\min  _ {t _ {i} ^ {j}} H \left(p \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right)\right) \mid p \left(\omega^ {\top} \phi \left(x _ {i}\right)\right)\right). \tag {9}
$$
We call this conditional entropy minimization (CEM) mechanism. The smallest value of
$H(p(\omega^{\top}\phi (t_{i}^{j}))|p(\omega^{\top}\phi (x_{i})))$  is 0, indicating that given  $p(\omega^{\top}\phi (x_i))$ ,  $p(\omega^{\top}\phi (t_i^j))$  is exactly predictable. Eq. (9) can also be expanded to the difference between Shannon entropy  $H$  and mutual information  $I$ , i.e.,  $H(X|Y) = H(X) - I(X,Y)$ . In other words, CEM minimizes the entropy of the selected sample  $t_i^j$  and maximizes the mutual information between  $t_i^j$  and the original sample  $x_{i}$  which means that CEM tries to augment samples of high prediction probability and high similarity with the original sample. As in REM, we define the quality score  $s_{qua}^{ij}$  of augmented sample  $t_i^j$  as
$$
s _ {q u a} ^ {i j} = - H \left(p \left(\omega^ {\top} \phi \left(t _ {i} ^ {j}\right)\right) \mid p \left(\omega^ {\top} \phi \left(x _ {i}\right)\right)\right). \tag {10}
$$
3.4 Algorithm and Implementation
The procedure of EPiDA is presented in Alg. 1. For each input sample  $x_{i}$ , EPiDA outputs  $m$  augmented samples. First, we employ  $T$  to generate  $K * m$  candidate augmented samples for  $x_{i}$ , where  $K$  is a hyperparameter to amplify the number of candidate samples, which is called amplification factor. Then, for each augmented sample, we use REM and CEM to evaluate its diversity score  $(s_{div})$  and quality score  $(s_{qua})$ , respectively. Next, we adopt Min_Max_Norm to make  $s_{div}$  and  $s_{qua}$  fall in [0,1]. After that, we add them together as the overall score of the sample, and sort all the augmented samples in descending order according to
Sentence	sdiv	squa	stot
Go Set a Watchman comes out Tuesday and I'm really excited for it	0.00	1.00	1.00
Go Set a Watchman comes out Tuesday and I'm really mad for it	0.96	0.03	0.99
Go Set a Watchman out Tuesday and I'm really excited for it	0.05	0.92	0.97
Go Set a security guard comes out Tues and I'm really excited for it	0.86	0.15	1.01
Table 2: An example from the sentiment analysis task to demonstrate the effect of REM and CEM. The first row is the original sample, the rest are the augmented samples. Underlined are salient words.
their scores. Finally, we take the top  $m$  samples from all the  $K * m$  candidate samples as the output, and utilize them to train the classifier.
By nature, the goals of REM and CEM are conflicting, i.e., a sample of high diversity is more probably of low quality, and vice versa. We give an example in Tab. 2 to demonstrate this point. REM encourages to change salient words, which is prone to break the semantic consistency (see the 3rd row, "excited" is changed to "mad", leading to large diversity score but small quality score). However, CEM tends to make the augmented samples keep semantic consistency, i.e., has large quality score but small diversity score (see the 4th row, "comes" is deleted). By jointly considering REM and CEM, satisfactory samples with balanced diversity and quality can be found (see the 5th row).
Besides, the calculation of  $s_{div}$  and  $s_{qua}$  requires the feedback of the classifier, so we first pre-train the classifier using the original samples, then with EPiDA we can generate samples of high diversity and quality for the classifier continuously.
In this section, we conduct extensive experiments to evaluate EPiDA, including performance comparison with SOTA methods, performance evaluation when working with different DA algorithms and classification models, ablation study, and qualitative visualization of samples augmented by EPiDA.
Datasets for five different tasks are used in our experiments: Question Classification (Li and Roth, 2002) (TREC,  $N = 5,452$ ), News Classification (Zhang et al., 2015) (AGNews,  $N = 120,000$ ), Tweets Sentiment Analysis (Rosenthal et al., 2017)
Method	Sentiment	Irony	Offense
10%	40%	PPL	10%	40%	PPL	10%	40%	PPL
EDA (randomly delete, swap etc.)	0.560	0.608	41.22	0.530	0.515	76.07	0.637	0.629	37.37
Contextual Word Embs Aug. (insert, replace using Bi-RNN LM)	0.610	0.627	1043.18	0.518	0.593	1146.40	0.663	0.713	1729.62
Back-translation Aug. (Eng. → Fr. → Eng. as aug. text)	0.617	0.620	474.29	0.520	0.541	423.32	0.655	0.724	345.23
Data Boost (RL-guided conditional generation)	0.591	0.642	56.23	0.591	0.639	77.40	0.695	0.784	35.18
Ours EPIDA: REM only (EDA as the DA algorithm)	0.619	0.650	66.86	0.624	0.665	77.09	0.662	0.673	81.13
Ours EPIDA: CEM only (EDA as the DA algorithm)	0.629	0.659	8.10	0.629	0.666	12.11	0.668	0.670	8.57
Ours EPIDA: REM+CEM (EDA as the DA algorithm)	0.639	0.659	25.40	0.651	0.687	53.17	0.680	0.687	32.56
Table 3: Performance comparison with existing augmentation methods.  $10\%$  :  $10\%$  original data  $+30\%$  augmented data  $(m = 3)$ ;  $40\%$  :  $40\%$  original data  $+40\%$  augmented data  $(m = 1)$ . We report the F1 score of the BERT classifier averaged over five repeated experiments on each dataset. We also report the perplexity score (PPL) of 10,000 randomly sampled data augmented by each method, where PPL is evaluated by the kenLM language model trained on the training data of each task.
(Sentiment,  $N = 20,631$ ), Tweets Irony Classification (Van Hee et al., 2018) (Irony,  $N = 3,817$ ) and Tweets Offense Detection (Founta et al., 2018) (Offense,  $N = 99,603$ ), where  $N$  is the number of training samples. To fully demonstrate the performance of data augmentation, we use only part of each dataset. In the following experiments, the percentage (\%) that follows the task name means the ratio of training data used from each dataset, e.g. Irony  $1\%$  means that  $1\%$  of the dataset is used. Macro-F1 (F1 for binary tasks) is used as performance metric, and all the experiments are repeated five times. The amplification factor  $K$  is set to 3.
4.2 Comparing with SOTA Methods
Here we carry out performance comparison with major SOTA methods to show the superiority of EPiDA on three datasets: Sentiment, Irony and Offense. For a fair comparison, we strictly follow the experimental setting of DataBoost (Liu et al., 2020): we do only one round of augmentation to ensure that the number of samples of our method is consistent with that of the other methods, and use BERT as the classifier. We use the widely used EDA as the DA algorithm of EPiDA. We do not use DataBoost because it is not yet open-sourced. The experimental results are presented in Tab. 3.