SlowGuess/NAACL_2022_OCR huggingface dataset:
Part 4 denotes the proposed Generative Cross-Domain Data Augmentation Framework, i.e., GCDDA, which designs a BART-based generative model (Lewis et al., 2020) to convert the labeled source-domain review to a target-domain review while predicting its corresponding token-level labels. We train the  $\mathrm{BERT_E + CRF}$  classifier on the generated target-domain data for aspect and opinion co-extraction.
The comparison results of all methods are shown in Table 2. We can clearly observe that our approach achieves the best performance in terms of the average Micro-F1. Specifically, GCDDA outperforms the baseline approach  $\mathrm{BERT_E + CRF}$  by  $4.90\%$  and  $0.36\%$  for aspect extraction and opinion extraction, respectively. Compared with the significant improvement on aspect extraction, we achieve a relatively low result on opinion term extraction. We attribute it to the following reasons: 1) the aspect terms across two domains tend to be quite different and have small overlaps; and
Model	R->L	R->D	L->R	L->D	D->R	D->L	AVE
AS	OP	AS	OP	AS	OP	AS	OP	AS	OP	AS	OP	AS	OP
CrossCRF	19.72	59.20	21.07	52.05	28.19	65.52	29.96	56.17	6.59	39.38	24.22	46.67	21.63	53.17
RAP	25.92	62.72	22.63	54.44	46.90	67.98	34.54	54.25	45.44	60.67	28.22	59.79	33.94	59.98
Hier-Joint	33.66	-	33.20	-	48.10	-	31.25	-	47.97	-	34.74	-	38.15	-
RNSCN	40.43	65.85	35.10	60.17	52.91	72.51	40.42	61.15	48.36	73.75	51.14	71.18	44.73	67.44
TRNN-GRU	40.15	65.63	37.33	60.32	53.78	73.40	41.19	60.20	51.17	74.37	51.66	68.79	45.88	67.12
TIMN	43.68	68.44	35.45	59.05	54.12	73.69	38.63	62.22	53.82	76.52	52.46	69.32	46.36	68.21
SemBridge	50.67	71.51	43.34	63.46	63.04	80.48	44.91	64.15	60.19	80.21	53.02	72.63	52.53	72.08
BERTB+CRF	40.71	74.96	40.28	64.73	40.31	80.74	44.24	59.22	59.35	81.17	53.46	75.05	46.39	72.65
SA-EXAL	47.59	75.79	40.50	63.33	54.67	80.05	42.19	60.19	54.54	71.57	47.72	63.98	47.87	69.15
BERTE+CRF	52.77	75.94	43.65	66.36	50.08	82.39	45.47	60.29	64.21	81.79	58.75	76.15	52.49	73.82
CDRG-Merge	58.23	76.08	37.96	62.19	72.88	82.34	40.62	59.04	66.79	82.23	54.26	76.42	55.12	73.05
GCDDA (Ours)	66.56	77.63	44.80	64.86	62.22	82.67	45.11	60.72	68.23	82.44	57.44	76.75	57.39	74.18
Table 2: Comparison results of different methods for aspect and opinion extraction on Micro-F1. The AVE above means averaged scores on all domain pairs. The best scores are in bold. All methods are grouped into four parts. The last part is our approach.
Model	F1	BLEU	Perplexity
ASP	OP
Source	52.49	73.82	100.00	27.48
CDDA-MLM	56.66	73.66	90.59	28.49
GCDDA	57.39	74.18	89.72	28.14
Table 3: Comparison results of two different Cross-Domain Data Augmentation methods on several evaluation metrics for generated data. Source denotes the source-domain labeled data. CDDA-MLM denotes the data generated by BERT. GCDDA is our method.
2) the opinion terms across two domains tend to have a significant overlap, e.g., great, good, terrible usually occurring in both domains. The similar trend can also be observed in the state-of-the-art approach SemBridge (Chen and Qian, 2021). Moreover, compared with SemBridge which is based on static word vectors, our method GCDDA improves its performance by  $4.86\%$  and  $2.10\%$ , which proves the effectiveness of the pre-trained language model. Lastly, GCDDA outperforms our recent data augmentation-based method CDRG-Merge by  $2.27\%$  and  $1.13\%$  on the two tasks, respectively. All these observations show the effectiveness of our proposed approach.
5.5 Evaluation on Generated Samples
In this section, we conduct several experiments to evaluate the quality of our generated target labeled data. For better comparison, we construct another cross-domain data augmentation approach (CDDA-MLM), which can be regarded as a variant of our GCDDA approach based on the masked language model BERT. CDDA-MLM adopts the same mask
strategy and post-processing as GCDDA, and we post-train BERT on the unlabeled data from the target domain with the standard MLM objective. Moreover, we employ it to replace the [mask] token with target words. Unlike GCDDA, there is no label embedding and label prediction in CDDA-MLM. In addition, the original labels can be directly transferred to the target sentence due to the word-to-word generation.
As shown in Table 3, we evaluate the generated data with three metrics including F1-score,  $\mathrm{BLEU}^1$ , and Perplexity², and report the average results on six cross-domain pairs. For F1-score, we observe that our proposed model GCDDA achieves the best result. It is due to the fact that we add the label embedding for GCDDA to capture the consistency between tokens and labels, which leads to more precise and controllable generation of target aspects or opinions in the appropriate place conditioned on the source-domain context and its labels. Additionally, compared with only using source labeled data, the significant improvement shows that our method GCDDA is indeed helpful for the domain adaptation problem.
Moreover, GCDDA obtains lower BLEU than CDDA-MLM, which indicates that our method generates more diverse sentences. This is because that GCDDA decodes the whole sentence while CDDA-MLM generates new words only on the [mask]
<sup>1</sup>We choose the source data as the reference and use BLEU-1 as the final result. The lower value means more differences from the source data.
2We calculate the perplexity with a pre-train language model GPT. The more fluent sentence has a lower value.
Source	CDDA-MLM	GCDDA
R->L	0.3502	0.1796	0.1128
R->D	0.3897	0.2284	0.1598
L->R	0.3306	0.2315	0.1488
L->D	0.0826	0.0667	0.0575
D->R	0.3888	0.2685	0.1429
D->L	0.1078	0.0812	0.0701
AVE	0.2749	0.1760	0.1153
Table 4: Maximum Mean Discrepancy (MMD) between the target-domain test set and the dataset generated from CDDA-MLM, GCDDA. The lower is better.
Source	the food was extremely tasty , creatively presented and the wine excellent .
CDDA-MLM	the screen is extremely well ##sty , creative ##ly presented and the are excellent .
GCDDA	the screen is extremely lightweight , creatively pre-sented and the battery excellent .
Source	the food is prepared quickly and efficiently .
CDDA-MLM	the received is up quickly and efficiently .
GCDDA	the computer runs quickly and efficiently .
Source	i recommend the jelly fish , drunken chicken and the soupy dumplings , certainly the stir fry blue crab .
CDDA-MLM	i recommend the color ##screen , other 8 , the really key ##ba also , certainly the anti the hybrid ##nes .
GCDDA	i recommend the backlit , wireless keyboard , the pre-loaded mbp applications , certainly the toshiba satellite .
Source	we had the lobster sand and it was fantastic .
CDDA-MLM	i had it head basement and it was fantastic .
GCDDA	i bought this netbook and it was fantastic .
Source	instead of wasting your time here : support restau-rants that care about food .
CDDA-MLM	instead of wasting your time here : call people that care about you .
GCDDA	instead of wasting your time here : buy computers that care about gaming .
position. Furthermore, each masked token is predicted independently in CDDA-MLM, which may generate incoherent collocation as the length of the mask segment grows. Instead, the auto-regressive model BART in GCDDA takes into account the coherence and the consistency of sentences. Thus, GCDDA obtains a better result than CDDA-MLM in terms of Perplexity.
Moreover, we measure the distribution distance between our generated target dataset and the test set from the target domain. Specifically, we use the BERT model to obtain the representation of each sentence in two datasets, and then calculate the distribution distance by Maximum Mean Discrepancy  $(\mathrm{MMD}^3)$ . The comparison results are shown in Table 4. The results, on the other hand, prove
Table 5: Comparison examples from source-domain labeled data and generated data by CDDA-MLM, GCDDA. Words in blue denote the aspect terms and words in red denote the opinion terms. All generated samples are extracted from domain pair  $(\mathsf{R} - > \mathsf{L})$
R->L	dvd games, wireless devices, wireless keyboard, tosibaspeakers, digital graphics, hp g73, hard drive applications,pre-loaded applications,light weight, precise, over weight,ingenious, well built
L->R	corn sauce, house vibe, wine quality, chili sauce, grilledmeat, outdoor dining atmosphere, service provider, internaldecor, heating system, steak platter, cleaner, much larger,softer
R->D	ipod plug, battery capacity, radio signal, sound quality ofmusic, built-in speakerphone, usb connection, sony camera,black card with fm transmitter, 2mb flash card, amazinggreat , user friendly
Table 6: New aspect and opinion terms generated by GCDDA which have not appeared in the training data of both domains. These examples are extracted from cross-domain pairs  $(\mathbb{R} - > \mathbb{L},\quad \mathbb{L} - > \mathbb{R},\quad \mathbb{R} - > \mathbb{D})$  with three different target domains.
that our method can be used to alleviate the domain discrepancy issue in domain adaptation.
Case Study: In Table 5, we show several examples generated by CDDA-MLM and GCDDA on the cross-domain pair  $\mathbb{R} - > \mathbb{L}$ . It can be obviously found that our method produces more precise aspects or opinions related to the domain laptop. We summarize this into two reasons: 1) integrating the BART model with label embedding has captured the consistency between token and its label; and 2) the other is owed to the domain prompt in the front of the decoder, which controls the domain where the generated words come from. Moreover, the label prediction from the decoder makes it possible to generate more flexible sentence while obtaining the adaptive token tags. We also present aspects and opinions extracted from our generated dataset in Table 6. These words or phrases have not appeared in the source and target training datasets, which shows that the generated dataset expands the vicinity distribution for aspect/opinion identification. To sum up, our method GCDDA not only exploits the rich contexts from the source domain, but also produces new aspects and opinions to generate more diverse target-domain samples. The two factors are integrated to train a more robust model for the target domain.
Parameter Study: As mentioned in Section 4.3, we mask sentences with different seeds to produce more target samples. Figure 2 shows the average F1 score of generating different sizes of augmented data from CDDA-MLM and GCDDA for aspect and opinion co-extraction. It can be seen that the two models consistently achieve the best result when the size is 3, i.e., generating three different target reviews from one labeled source review.
3Maximum Mean Discrepancy (MMD) reflects the similarity of distribution between two datasets.
Figure 2: Average F1-score on different size of samples generated by CDDA-MLM and GCDDA for aspect and opinion co-extraction.
Index	Method	F1
ASP	OP
0	GCDDA (ours)	57.39	74.18
1	W/O data augmentation	54.27	73.56
2	BART ⇒ Bi-LSTM	56.78	73.58
3	W/O data filter	55.50	72.71
4	1~4-gram ⇒ 1-gram	56.92	73.55
5	W/O source data	56.71	73.80
6	W/O encoder label embedding	55.88	73.04
Table 7: Ablation study of our Generative Cross-Domain Data Augmentation method.
Finally, to better analyze the effectiveness of each key component in our method, we conduct the ablation study and show the results in Table 7.
We first remove the whole cross-domain data augmentation framework in index 1, and only use the pseudo labeled data  $D_{TP}$  to train the main task classifier. It shows a considerable performance drop due to the label noise in  $D_{TP}$ . This indicates that cross-domain data augmentation needs to be applied, which can produce high-quality target samples by combining the contexts from  $D_{S}$  with the weak supervision signals in  $D_{TP}$ . In index 2, we replace the pre-trained model BART with BiLSTM. Compared with Bi-LSTM, the pre-trained BART model obtains a better result, because it possesses rich language knowledge and a more complex model structure for generation tasks.
Index 3 proves the importance of data filter in the post-processing step. As shown in Table 5, our generated samples usually have the same syntactic structure as the labeled source data. Therefore, the base classifier trained on  $D_{S}$  can be employed to filter out some noisy samples in our generated target-domain data. In index 4, compared with our  $1\sim 4$ -gram mask, the single word mask strategy limits the diversity of generated sentences, and thus decreases the model performance. Moreover, the performance drops when we remove the la
beled source data for the reconstruction task, which can capture the domain-invariant features for the BART model. Finally, removing the label embedding layer in the BART encoder also leads to the performance decline.
In this paper, we proposed a Generative Cross-Domain Data Augmentation Framework for unsupervised domain adaptation, which leverages the labeled source-domain data to directly generate labeled target-domain data based on a fine-tuned sequence-to-sequence model BART. Experiments on three benchmark datasets show that our generative approach generates high-quality target-domain data with fine-grained annotation and outperforms previous domain adaptation methods for aspect and opinion co-extraction.
The authors would like to thank the anonymous reviewers for their insightful comments. This work was supported by the Natural Science Foundation of China (62076133 and 62006117), and the Natural Science Foundation of Jiangsu Province for Young Scholars (BK20200463) and Distinguished Young Scholars (BK20200018).
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th annual meeting of the association of computational linguistics, pages 440-447.
Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020. Local additivity based data augmentation for semi-supervised ner. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1241-1251.
Zhuang Chen and Tieyun Qian. 2021. Bridge-based active domain adaptation for aspect term extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 317-327.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kru-engkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6045-6057, Online. Association for Computational Linguistics.
Ying Ding, Jianfei Yu, and Jing Jiang. 2017. Recurrent neural networks with auxiliary labels for cross-domain opinion target extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Fei Gao, Jinhua Zhu, Lijun Wu, Yingce Xia, Tao Qin, Xueqi Cheng, Wengang Zhou, and Tie-Yan Liu. 2019. Soft contextual data augmentation for neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5539-5544, Florence, Italy. Association for Computational Linguistics.
Deepanway Ghosal, Devamanyu Hazarika, Abhinaba Roy, Navonil Majumder, Rada Mihalcea, and Soujanya Poria. 2020. Kingdom: Knowledge-guided domain adaptation for sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3198-3210.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Domain adaptation for large-scale sentiment classification: A deep learning approach. In ICML.
Chenggong Gong, Jianfei Yu, and Rui Xia. 2020. Unified feature and instance based domain adaptation for end-to-end aspect-based sentiment analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7035-7045.
Hongyu Guo, Yongyi Mao, and Richong Zhang. 2019. Augmenting data with mixup for sentence classification: An empirical study. arXiv preprint arXiv:1905.08941.
Ting-Wei Hsu, Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2021. Semantics-preserved data augmentation for aspect-based sentiment analysis. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4417-4422.
Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168-177.
Niklas Jakob and Iryna Gurevych. 2010. Extracting opinion targets in a single and cross-domain setting with conditional random fields. In Proceedings of
the 2010 conference on empirical methods in natural language processing, pages 1035-1045.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880.
Fangtao Li, Sinno Jialin Pan, Ou Jin, Qiang Yang, and Xiaoyan Zhu. 2012. Cross-domain co-extraction of sentiment and topic lexicons. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 410-419.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018a. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1865-1874.
Kun Li, Chengbo Chen, Xiaojun Quan, Qing Ling, and Yan Song. 2020. Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7056-7066.
Zheng Li, Xin Li, Ying Wei, Lidong Bing, Yu Zhang, and Qiang Yang. 2019. Transferable end-to-end aspect-based sentiment analysis with selective adversarial learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Zheng Li, Ying Wei, Yu Zhang, and Qiang Yang. 2018b. Hierarchical attention transfer network for cross-domain sentiment classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Pengfei Liu, Shafiq Joty, and Helen Meng. 2015. Finegrained opinion mining with recurrent neural networks and word embeddings. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 1433-1443.
Junghyun Min, R Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339-2352.
Sinno Jialin Pan, Xiaochuan Ni, Jian-Tao Sun, Qiang Yang, and Zheng Chen. 2010. Cross-domain sentiment classification via spectral feature alignment. In Proceedings of the 19th international conference on World wide web, pages 751-760.
OrenPereg,Daniel Korat,and Moshe Wasserblat.2020. Syntactically aware cross-domain aspect and opinion terms extraction.InProceedings of the 28th International Conference on Computational Linguistics, pages 1772-1777.
Maria Pontiki, Dimitrios Galanis, Harris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th international workshop on semantic evaluation (SemEval 2015), pages 486-495.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27-35, Dublin, Ireland. Association for Computational Linguistics.
Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion word expansion and target extraction through double propagation. Computational linguistics, 37(1):9-27.
Wenya Wang and Sinno Jialin Pan. 2018. Recursive neural structural correspondence network for cross-domain aspect and opinion co-extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2171-2181.
Wenya Wang and Sinno Jialin Pan. 2019. Transferable interactive memory network for domain adaptation in fine-grained opinion extraction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7192-7199.
Wenya Wang and Sinno Jialin Pan. 2020. Syntactically meaningful and transferable recursive neural networks for aspect and opinion extraction. Computational Linguistics, 45(4):705-736.
Wenya Wang, Sinno Jialin Pan, Daniel Dahlmeier, and Xiaokui Xiao. 2017. Coupled multi-layer attentions for co-extraction of aspect and opinion terms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
Meixi Wu, Wenya Wang, and Sinno Jialin Pan. 2020. Deep weighted maxsat for aspect-based opinion extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5618-5628.
Hu Xu, Bing Liu, Lei Shu, and S Yu Philip. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. In Proceedings of
the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2324-2335.
Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), pages 2979-2985.
Jianfei Yu, Chenggong Gong, and Rui Xia. 2021. Cross-domain review generation for aspect-based sentiment analysis. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4767-4777.
Jianfei Yu and Jing Jiang. 2016. Learning sentence embeddings with auxiliary tasks for cross-domain sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 236-246.
Jianfei Yu, Jing Jiang, and Rui Xia. 2018. Global inference for aspect and opinion terms co-extraction based on multi-task neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(1):168-177.
Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2015. Neural networks for open domain targeted sentiment. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 612-621.
Yftah Ziser and Roi Reichart. 2018. Pivot based language modeling for improved neural domain adaptation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1241-1251.