SlowGuess/ACL_2020_OCR huggingface dataset:
As shown in Table 3, the removal of label embeddings causes considerable performance drops, and the results are even worse than that on the original dataset. This is probably due to the poor Recall performance that can be explained as follows. When label sequence information is not present, the augmentation is prone to produce decayed examples in which some new aspect terms are generated in the positions of label  $O$ , or verse vice. The model trained with such decayed examples is misled not to extract these aspect terms in the test stage. As a result, the model makes many false-negative errors, leading to poor Recall scores. This indicates that label embeddings are helpful for generating qualified sentences for aspect term extraction.
5.3 Why Sequence-to-Sequence Generation?
As mentioned before, we formulate the data augmentation for aspect term extraction as a conditional generation problem that is solved by masked sequence-to-sequence learning. One may argue that other pre-trained language models like BERT and GPT-2 are also competent for this task as in (Wu et al., 2019; Sudhakar et al., 2019; Keskar et al., 2019). Here we compare them and demonstrate the superiority of our approach in this task.
Following some previous work (Wu et al., 2019; Sudhakar et al., 2019; Keskar et al., 2019), we modify the settings of BERT and GPT-2 to make them fit this task. Readers are recommended to refer to Appendix for more details. Moreover, a widely-used replacement-based method is implemented for comparison, in which half of the tokens are randomly replaced by their synonyms from Word
Net (Miller, 1995). We use fluency and  $\mathrm{BLEU}^7$  to evaluate the generated sentences. Note that these datasets do not contain the original training examples because we want to focus more on the generated ones. We employ BERT-FTC as the implementation model and train it on these datasets. The results on the test sets are presented in Table 4.
From the table, we note that the F1 scores of GPT-2 are the worst because of its low recall scores. This conforms with the architecture and the language modeling objective of GPT-2, which does not have an encoder to encode the label information. In this case, the decoding step is uncontrollable and cannot generate a sentence fitting the label sequence. In contrast, our framework contains an encoder to encode a sentence and the label sequence simultaneously, and a decoder to generate sentences conditional on the encoder output. That is, our decoder takes advantage of both context information and aspect label information, making the augmentation conditional and controllable.
BERT performs the worst in this task in fluency. This can be attributed to its independence assumption in the process of generation, which means that all masked tokens are independently reconstructed, likely leading to in-coherent word sequences. In contrast, our approach generates the sequence in an auto-regressive way, with each decoding step based on the result of its previous step, ensuring fluent new sentences.
The replacement-based method does not take into account the sentence context and leads to poor fluency scores. Also, there are limited words to choose for synonyms in such lexical databases as WordNet. Thus, such replacement-based methods can only produce sentences of limited diversity, which is confirmed by the BLEU scores.
Fluency is measured by the perplexity of sentence, and is calculated by OpenAI GPT. In this metric, sentences with lower perplexity scores are more fluent. Note that the GPT here is different from GPT-2 that we use to generate text data.
The original sentences are taken as reference.
7062
Dataset	Laptop	Restaurant
Precision/Recall/F1	BLEU	Fluency	Precision/Recall/F1	BLEU	Fluency
Source	77.36/81.53/79.39	100.00	203	74.86/74.64/74.75	100.00	158
DATA_synonym	74.91/79.06/76.93	76.68	558	71.23/76.43/73.74	78.22	438
DATA_BERT	80.06/78.60/79.32	69.93	461	72.34/76.37/74.30	68.57	435
DATA_GPT-2	71.97/61.10/66.09	54.86	328	70.53/57.42/63.30	58.60	314
DATA_Ours	80.27/80.98/80.61	69.70	242	73.95/76.95/75.41	68.42	257
Table 4: Results on datasets generated by different augmentation approaches. Source denotes the original datasets; DATA synonym denotes the datasets obtained by randomly replacing tokens with their synonyms. DATA_BERT, DATA_GPT-2 and DATA_Ours denote the datasets generated by BERT, GPT-2 and our augmentation approach, respectively. BERT-FTC is used as the implementation model, and lower Fluency scores mean more fluent sentences.
Source:
Augmented:	Also, the space bar makes a noisy click every time you use it.
Also, the space bar will get stuck there every time you use it.
Source:
Augmented:	The hinge design forced you to place various connections all around the computer, left right ...
The hinge design also allows you to adjust the angle around the computer, left right ...
Source:
Augmented:	Their pad penang is delicious and everything else is fantastic.
Their pad penang is mediocre but everything else is fantastic.
Source:
Augmented:	I am learning the finger options for the mousepad that allow for quicker browsing of web pages.
I also enjoy the fact that it has a mousepad that allow for quicker browsing of web pages.
Source:
Augmented:	I charge it at night and skip taking the cord with me because of the good battery life.
I don’t have to carry the cord with me because of the good battery life.
Table 5: Examples generated by our augmentation approach. Texts in bold, blue and purple represent aspect terms, masked fragments and generated fragments, respectively.
To sum up, our data augmentation model benefits considerably from its encoder-decoder architecture and the masked sequence-to-sequence generation mechanism, which is controllable to ensure qualified data augmentation for aspect term extraction. The results show that this sequence-to-sequence generation framework is non-replaceable by other language models such as BERT and GPT-2.
We finally present several augmented examples in Table 5 to illustrate the effect of our augmentation method more intuitively. We observe that the contents of the masked fragments can be dramatically changed from their original forms after augmentation. In some cases, the sentiment polarities are even reversed. Nevertheless, the new contexts are still appropriate for the aspect terms, making them qualified and also diversified new training examples for aspect term extraction.
In this paper, we have presented a conditional data augmentation approach for aspect term extraction.
We formulated it as a conditional generation problem and proposed a masked sequence-to-sequence generation model to implement it. Unlike existing augmentation approaches, ours is controllable to generate qualified sentences, and allows more diversified new sentences. Experimental results on two review datasets confirm its effectiveness in this conditional augmentation scenario. We also conducted qualitative studies to analyze how this augmentation approach works, and tested other language models to explain why our masked sequence-to-sequence generation framework is favored. Moreover, the proposed augmentation method tends not to be unique to the current task and could be applied to other low-resource sequence labeling tasks such as chunking and named entity recognition.
The work was partially supported by the Fundamental Research Funds for the Central Universities (No.19lgpy220) and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No.2017ZT07X355).
7063
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1724- 1734, Doha, Qatar.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota.
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489-500, Brussels, Belgium.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680.
Rahul Gupta. 2019. Data augmentation for low resource sentiment analysis using generative adversarial networks. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7380-7384. IEEE.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 504-515, Florence, Italy.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339, Melbourne, Australia.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, pages 1587-1596.
Wei Jin, Hung Hay Ho, and Rohini K Srihari. 2009. A novel lexicalized hmm-based learning framework for web opinion mining. In Proceedings of the 26th
annual international conference on machine learning, pages 465-472.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv:1909.05858.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. International Conference on Learning Representations.
Diederik P Kingma and Max Welling. 2013. Autoencoding variational bayes. arXiv preprint arXiv:1312.6114.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 452-457, New Orleans, Louisiana. Association for Computational Linguistics.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Fangtao Li, Chao Han, Minlie Huang, Xiaoyan Zhu, Ying-Ju Xia, Shu Zhang, and Hao Yu. 2010. Structure-aware review mining and summarization. In Proceedings of the 23rd international conference on computational linguistics, pages 653-661.
Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018. Aspect term extraction with history attention and selective transformation. In *IJCAI*, pages 4194-4200.
Bing Liu. 2012. Sentiment analysis and opinion mining. Synthesis lectures on human language technologies, 5(1):1-167.
Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Finegrained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1433-1443.
Huaishao Luo, Tianrui Li, Bing Liu, and Junbo Zhang. 2019. DOER: Dual cross-shared RNN for aspect term-polarity co-extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 591-601, Florence, Italy.
Dehong Ma, Sujian Li, Fangzhao Wu, Xing Xie, and Houfeng Wang. 2019. Exploring sequence-to-sequence learning in aspect term extraction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3538-3547.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
7064
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, AL-Smadi Mohammad, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clercq, et al. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 19-30.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27-35, Dublin, Ireland. Association for Computational Linguistics.
Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational Linguistics, 37(1):9-27.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2016. Improving language understanding by generative pre-training. 2018. URL https://s3-us-west-2. amazonaws.com/openai-assets/research-covers/language unsupervised/language-understanding_paper.pdf.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 86-96, Berlin, Germany.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In ICML.
Akhilesh Sudhakar, Bhargav Upadhyay, and Arjun Maheswaran. 2019. "transforming" delete, retrieve, generate approach for controlled text style transfer. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3260-3270, Hong Kong, China. Association for Computational Linguistics.
Zhiqiang Toh and Jian Su. 2016. Nlangp at semeval-2016 task 5: Improving aspect based sentiment analysis using neural network features. In Proceedings of the 10th international workshop on semantic evaluation (SemEval-2016), pages 282-288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
William Yang Wang and Diyi Yang. 2015. That's so annoying!!!: A lexical and frame-semantic embedding based data augmentation approach to automatic categorization of annoying behaviors using# pet peeve tweets. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2557-2563.
Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019. Conditional BERT contextual augmentation. In Computational Science - ICCS 2019 - 19th International Conference, Faro, Portugal, June 12-14, 2019, Proceedings, Part IV, pages 84-95.
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. 2019. BERT post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2324-2335, Minneapolis, Minnesota. Association for Computational Linguistics.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2018. Double embeddings and cnn-based sequence labeling for aspect extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649-657.
7065
We modify the settings of BERT and GPT-2 to make them fit our augmentation task as follows.
BERT We follow Wu et al. (2019) who applied the masked language model task to augment their training data for text classification. The segment embeddings in the input layer are replaced by label embeddings. The objective is to predict the masked token  $t_i$  based on the conditional probability distribution  $p(t_i|L, S \setminus \{t_i\})$ , where  $S$  and  $L$  denote the sentence and its label sequence, respectively, and  $S \setminus \{t_i\}$  means the context of  $t_i$ . To predict  $t_i$ , both the context  $S \setminus \{t_i\}$  and the label sequence  $L$  are considered. The word with the highest probability among all vocabulary words is chosen.
In our experiments,  $50\%$  of the tokens in a sentence are masked individually. These tokens are then reconstructed in the masked language modeling task. Finally, the predicted tokens and the un-masked tokens constitute an augmented sentence.
GPT-2 We refer to the work of Sudhakar et al. (2019) and Keskar et al. (2019), which aim at style transfer and controllable generation, respectively. In the training stage, the format of input is: [ASP] aspect $_1$  [ASP] aspect $_2$  [BOS] sentence [EOS], where [ASP] is a special symbol followed by an aspect term, and [BOS] is appended before the source sentence. For example, for a source sentence "I was disappointed with this restaurant", where "restaurant" is the aspect term, the input takes the following format: [ASP] restaurant [BOS] I was disappointed with this restaurant. [EOS]. Let  $S$  denote the source sentence. The autoregressive model learns to reconstruct  $S$  given  $C_X$ , which is the fragment of the input from the first token [ASP] to [BOS]. The objective is to maximize:
$$
L (\theta) = \log p \left(\boldsymbol {S} \mid C _ {X}; \theta\right) \tag {8}
$$
where  $\theta$  denotes the parameters of GPT-2.
In the augmentation stage, this model takes the fragment  $C_X$  and half of the source sentence as input, and tries to reconstruct the other half. An augmented sentence is formed by joining the un-masked half of the sentence and the reconstructed half. The beam size is set to 5 in our experiments.
7066