SlowGuess/NAACL_2022_OCR huggingface dataset:
From Tab. 3, we can see that 1) with the help of EPiDA, the performance of EDA is greatly improved. In particular, comparing with the original EDA, EPiDA gets performance improvement of  $14.1\%$ ,  $8.39\%$ ,  $22.83\%$ ,  $33.40\%$ ,  $6.75\%$  and  $9.22\%$  in six task settings, respectively. 2) Our method outperforms DataBoost in four settings. In particular,  $\mathrm{EPiDA + EDA}$  gets performance improvement of  $8.12\%$ ,  $2.65\%$ ,  $10.15\%$  and  $6.99\%$  in various settings of the Sentiment and Irony tasks. 3) The variants of EPiDA that utilize only REM or CEM to enhance diversity or quality are inferior to using both, which demonstrates the effectiveness of joint enhancement. 4) DataBoost performs better in the Offense task, the reason lies in that DataBoost can create novel sentences from Offense (a relatively huge corpus) via GPT-2, while EDA only conducts word-level augmentation, which limits EPiDA's performance. 5) We also present PPL as an auxiliary metric to measure the generation perplexity. Our method outperforms the others due to the high quality of data generation. We also provide experimental comparisons with other DA approaches  $(\mathrm{SUB}^2$  and VDA) and generation speed results in the supplementary file. In conclusion, EPiDA is a powerful and efficient technique.
Method	TREC	AGNews	Sentiment	Irony	Offense
1%	10%	0.05%	0.1%	1%	10%	1%	10%	0.1%	1%
CNN	0.722	0.806	0.745	0.826	0.446	0.584	0.534	0.616	0.479	0.548
+EPiDA with EDA	0.745	0.814	0.806	0.829	0.520	0.598	0.579	0.621	0.493	0.551
+EPiDA with CWE	0.737	0.817	0.806	0.826	0.524	0.598	0.578	0.623	0.488	0.556
+EPiDA with TextAttack	0.723	0.838	0.819	0.829	0.527	0.600	0.568	0.631	0.481	0.549
BERT	0.769	0.914	0.759	0.820	0.507	0.602	0.589	0.635	0.508	0.630
+EPiDA with EDA	0.786	0.931	0.813	0.832	0.538	0.641	0.598	0.652	0.525	0.629
+EPiDA with CWE	0.780	0.922	0.821	0.834	0.546	0.642	0.597	0.655	0.517	0.630
+EPiDA with TextAttack	0.762	0.930	0.816	0.839	0.551	0.621	0.595	0.634	0.505	0.626
XLNet	0.746	0.904	0.749	0.776	0.563	0.620	0.576	0.651	0.522	0.626
+EPiDA with EDA	0.756	0.906	0.768	0.790	0.556	0.609	0.588	0.651	0.536	0.625
+EPiDA with CWE	0.750	0.894	0.779	0.795	0.554	0.608	0.592	0.659	0.532	0.627
+EPiDA with TextAttack	0.758	0.909	0.790	0.798	0.555	0.591	0.570	0.654	0.528	0.618
4.3 Performance with Different DA Algorithms and Classifiers
EPiDA is a plug-in component that can work with different DA algorithms and classifiers. Here, to check how EPiDA performs with different DA algorithms and classifiers, we consider three frequently-used DA algorithms: rule-based EDA (Wei and Zou, 2019), model-based CWE (Kobayashi, 2018) and Attack-based TextAttack (Morris et al., 2020), and three different classifiers: CNN (Kim, 2014), BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019). And to show that EPiDA can cope with different NLP classification tasks, we present the results on five different tasks: TREC, AGNews, Sentiment, Irony and Offense. In order to fully evaluate the performance of DA algorithms, we use only a small part of the training data. The experimental results are presented in Tab. 4. Here, we remove the restriction of only one time augmentation so that EPiDA can continuously generate qualified samples. We call this online augmentation, to differentiate it from one-time augmentation. As shown in Tab. 4, EPiDA is applicable to various NLP classification tasks. Although these tasks have different forms of data (questions or tweets) and different degrees of classification difficulty, EPiDA boosts performance on these tasks in almost all cases. More details on how EPiDA controls the generation quality are discussed in ablation study. Besides, we can also see that EPiDA works well with the three DA algorithms EDA, CWE and TextAttack. All achieve improved performance in most
Table 4: Performance comparison with different DA algorithms and classification models on five classification tasks.
ID	DA	REM	CEM	OA	PT	TREC 1%	Irony 1%
1	-	-	-	-	-	0.722	0.534
2	✓	-	-	-	-	0.736	0.474
3	✓	-	-	✓	-	0.723	0.550
4	✓	✓	-	-	✓	0.729	0.557
5	✓	-	✓	-	✓	0.723	0.559
6	✓	✓	✓	-	-	0.734	0.548
7	✓	✓	✓	✓	-	0.739	0.575
8	✓	✓	✓	-	✓	0.740	0.576
9	✓	✓	✓	✓	✓	0.745	0.579
Table 5: Ablation study on TREC  $1\%$  and Irony  $1\%$ . Here, DA, REM, CEM, OA and PT indicate whether applying DA, REM, CEM, online DA (performing multiple DA operations) and pre-training, respectively.
cases. For the three different classification models: CNN, BERT and XLNet, with the help of EPiDA, they all but XLNet on Sentiment get classification performance improvement, which shows that EPiDA is insensitive to classification models.
Here we conduct ablation study to check the effectiveness of different EPiDA configurations. We take CNN as the classifier, EDA as the DA algorithm and report the Macro-F1 score over five repeated experiments on TREC  $1\%$  and Irony  $1\%$ . Tab. 5 shows the experimental results.
Effect of REM and CEM. The 4th and 5th rows show the results with only REM and CEM, respectively. Both of them perform better than the baseline (1st row), but not as good as the combined case
(the 8th row). On TREC (relatively simple task), REM outperforms CEM (0.729 vs. 0.723), while on Irony (relatively hard task), CEM outperforms REM (0.559 vs. 0.557). Using only REM can limitedly boost performance since REM promotes the generation of high diversity samples, which may have wrong labels. And using only CEM is also not enough to fully tap the performance as CEM tends to generate redundant samples.
We also compare our 'REM + CEM' with 'PPL + cosine similarity' used in LDA (Zuo et al., 2021). Our method achieves the performance of 0.740 and 0.576 on TREC  $1\%$  and Irony  $1\%$ , while the latter achieves 0.730 and 0.562. This shows that our 'REM + CEM' is more effective.
Effect of online augmentation. Comparing the results of the 2nd and the 3rd rows, the 6th and the 7th rows, the 8th and the 9th rows, we can see that generally online augmentation can boost performance, as online augmentation can generate sufficient qualified samples to train the model.
Effect of pre-training. As REM and CEM use the feedback of the classifier, a pre-trained classification model should be beneficial to REM and CEM. By comparing the results of the 6th and the 8th rows, the 7th and the 9th rows, it is obvious that pre-training can improve performance.
Effect of normalization. In Alg. 1, we normalize  $s_{div}$  and  $s_{qua}$ . Here, we check the effect of normalization. With the same experimental settings, the performance results on TREC  $1\%$  and Irony  $1\%$  without normalization are 0.732 and 0.568, lower than the normalized results 0.740 and 0.576. This shows that normalization is effective.
How to combine REM and CEM? How to combine REM and CEM is actually how to combine the values of  $s_{div}$  and  $s_{qua}$ . We consider three simple schemes: addition ( $s_{tot} = s_{div} + s_{qua}$ ), multiplication ( $s_{tot} = s_{div} * s_{qua}$ ) and weighted addition ( $s_{tot} = \alpha s_{div} + (1 - \alpha)s_{qua}$ ,  $\alpha$  is a hyperparameter to tradeoff REM and CEM). Note that for multiplication, there is possibly an extreme situation: after normalization,  $s_{div}$  or  $s_{qua}$  may be very small and even approaches 0, then the multiplication result is very small or even zero, which means that REM and CEM do not take effect in sample generation. In our experiments, the multiplication scheme achieves performance of 0.725 and 0.572 on TREC 1% and Irony 1%, lower than the addition scheme 0.740 and 0.576. As for weighted addition, we find that setting  $\alpha = 0.5$  can achieve satisfactory re
Metric	EDA	REM only	CEM only	EpiDA
Error Rate↓	3.05%	6.75%	0.64%	1.53%
Distance↑(×10-2)	0.54	1.21	0.25	0.78
Table 6: Error rate and distance on Sentiment dataset.
Figure 2: Hidden state visualization of the sentiment analysis task by t-SNE. Left: original and EDA augmented data; Right: original and EPiDA augmented data. Different colors mean different classes (Neutral, Negative and Positive), and different shapes mean different samples (original, generated by EPiDA and EDA).
sults (see the supplementary file). This is actually equal to the addition scheme. Therefore, in our experiments, we use only the addition scheme.
Quality and diversity metrics. Here, we provide another two metrics to verify EPiDA from the perspective of quality and diversity. For quality, we use the augmentation error rate. As for diversity, we calculate the average distance of samples before and after augmentation (ignoring wrong samples). From the perspective of quality and diversity, a good DA should has a small error rate but a large distance. Experimental results are given in Tab. 6. We can see that EPiDA gets better trade-off between error rate and distance.
Effect of the amplification factor  $K$ . The amplification factor  $K$  determines the size  $Km$  of candidate samples from which  $m$  samples are chosen. On the one hand, with a large  $K$ , we have more choices, which seems beneficial to diversity. On the other hand, more candidate samples make the selected samples more homogenous, not good for diversity. By grid search, we set  $K$  to 3 in our experiments, the experimental results are shown in the supplementary file.
4.5 Visualization Effect of EPiDA
Above we give comprehensive quantitative performance evaluation of EPiDA, here to intuitively illustrate the effectiveness of EPiDA, we visualize some augmented samples of EPiDA, and compare them with that of EDA. Specifically, we utilize BERT as the classifier and visualize its hidden state on the sentiment analysis task via t-SNE (Van der Maaten and Hinton, 2008). Fig. 2 shows the results. In terms of data quality, we find that two negative samples generated by EDA are located in Neural and Positive classes, while samples generated by EPiDA are generally properly located. And in the point of view of diversity, samples generated by EPiDA extend the distributed areas of the original data, while samples generated by EDA are mainly located in the areas of the original samples. This shows that samples generated by EPiDA are more diverse than those generated by EDA.
In this paper, we present an easy plug-in data augmentation technique EPiDA to control augmented data diversity and quality via two mechanisms: relative entropy maximization and conditional entropy minimization. Through extensive experiments, we show that EPiDA outperforms existing methods, and can work well with different DA algorithms and classification models. EPiDA is general, effective, efficient, and easy-to-deploy. In the future, more verification of our method is expected to be conducted on other classification tasks.
This work was partially supported by National Key R&D Program of China under grant No. 2021YFC3340302, and Alibaba Innovative Research (AIR) programme under contract No. SCCW802020046613.
Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020. Do not have enough data? deep learning to the rescue! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7383-7390.
Jordan T Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. 2019. Deep batch active learning by diverse, uncertain gradient lower bounds. arXiv preprint arXiv:1906.03671.
Ken Chatfield, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2014. Return of the devil in the details: Delving deep into convolutional nets. arXiv preprint arXiv:1405.3531.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification. arXiv preprint arXiv:2004.12239.
Claude Coulombe. 2018. Text data augmentation made simple by leveraging nlp cloud apis. arXiv preprint arXiv:1812.04718.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2019. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Xin Luna Dong, Yaxin Zhu, Zuohui Fu, Dongkuan Xu, and Gerard de Melo. 2021. Data augmentation with adversarial training for cross-lingual nli. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5158-5167.
Steven Y Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for nlp. arXiv preprint arXiv:2105.03075.
Antigoni Maria Founta, Constantinos Djouvas, Despoina Chatzakou, Ilias Leontiadis, Jeremy Blackburn, Gianluca Stringhini, Athena Vakali, Michael Sirivianos, and Nicolas Kourtellis. 2018. Large scale crowdsourcing and characterization of twitter abusive behavior. In Twelfth International AAAI Conference on Web and Social Media.
Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019. Soft contextual data augmentation for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5539-5544.
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. Aeda: An easier data augmentation technique for text classification. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2748-2754.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457.
Oleksandr Kolomiyets, Steven Bethard, and Marie Francine Moens. 2011. Model-portability experiments for textual temporal analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 271-276.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25:1097-1105.
Ashutosh Kumar, Satwik Bhattachamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609-3619.
Xin Li and Dan Roth. 2002. Learning question classifiers. In COLING 2002: The 19th International Conference on Computational Linguistics.
Sungbin Lim, Ildoo Kim, Taesup Kim, Chiheon Kim, and Sungwoong Kim. 2019. Fast autoaugment. arXiv preprint arXiv:1905.00397.
Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020. Data boost: Text data augmentation through reinforcement learning guided conditional generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9031-9041.
Canjie Luo, Yuanzhi Zhu, Lianwen Jin, and Yongpan Wang. 2020. Learn to augment: Joint data augmentation and network optimization for text recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13746-13755.
John X Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. Proceedings of the 2020 EMNLP.
Nathan Ng, Kyunghyun Cho, and Marzyeh Ghassemi. 2020. Ssmba: Self-supervised manifold based data augmentation for improving out-of-domain robustness. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1268-1283.
Yuyang Nie, Yuanhe Tian, Xiang Wan, Yan Song, and Bo Dai. 2020. Named entity recognition for social media texts with semantic augmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1383-1391.
Tong Niu and Mohit Bansal. 2018. Adversarial over-sensitivity and over-stability strategies for dialogue models. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 486-496.
Husam Quteineh, Spyridon Samothrakis, and Richard Sutcliffe. 2020. Textual data augmentation for efficient active learning on tiny datasets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7400-7410.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Sara Rosenthal, Noura Farra, and Preslav Nakov. 2017. Semeval-2017 task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-2017), pages 502-518.
Gözde Gül Şahin and Mark Steedman. 2018. Data augmentation via dependency tree morphing for low-resource languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5004-5009.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96.
Haoyue Shi, Karen Livescu, and Kevin Gimpel. 2021. Substructure substitution: Structured data augmentation for nlp. arXiv preprint arXiv:2101.00411.
Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data, 6(1):1-48.
Chenglei Si, Zhengyan Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2021. Better robustness by more coverage: Adversarial and mixup data augmentation for robust finetuning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1569-1576.
Walter Simoncini and Gerasimos Spanakis. 2021. Seqattack: On adversarial attacks for named entity recognition. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 308-318.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39-50.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. 2019. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR.
William Yang Wang and Diyi Yang. 2015. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# petpeeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557-2563.
Jason Wei, Chengyu Huang, Shiqi Xu, and Soroush Vosoughi. 2021. Text augmentation in a multi-task view. arXiv preprint arXiv:2101.05469.
Jason Wei and Kai Zou. 2019. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196.
Ziang Xie, Sida I Wang, Jiwei Li, Daniel Lévy, Aiming Nie, Dan Jurafsky, and Andrew Y Ng. 2017. Data-noising as smoothing in neural network language models. arXiv preprint arXiv:1703.02573.
Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji-Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. 2020. G-daug: Generative data augmentation for commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1008-1025.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. 2019. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
Lu Zhang, Jiandong Ding, Yi Xu, Yingyao Liu, and Shuigeng Zhou. 2021. Weakly-supervised text classification based on keyword graph. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2803-2813.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28:649-657.
Kun Zhou, Wayne Xin Zhao, Sirui Wang, Fuzheng Zhang, Wei Wu, and Ji-Rong Wen. 2021. Virtual data augmentation: A robust and general framework for fine-tuning pre-trained models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3875-3887.
Xinyu Zuo, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, Weihua Peng, and Yuguang Chen. 2021. Learnda: Learnable knowledge-guided data augmentation for event causality identification. arXiv preprint arXiv:2106.01649.