Cross-Domain Data Augmentation with Domain-Adaptive Language Modeling for Aspect-Based Sentiment Analysis
Year: 2023

Abstract:
Cross-domain Aspect-Based Sentiment Analysis (ABSA) aims to leverage the useful knowledge from a source domain to identify aspectsentiment pairs in sentences from a target domain. To tackle the task, several recent works explore a new unsupervised domain adaptation framework, i.e., Cross-Domain Data Augmentation (CDDA), aiming to directly generate much labeled target-domain data based on the labeled source-domain data. However, these CDDA methods still suffer from several issues: 1) preserving many source-specific attributes such as syntactic structures; 2) lack of fluency and coherence; 3) limiting the diversity of generated data. To address these issues, we propose a new cross-domain Data Augmentation approach based on Domain-Adaptive Language Modeling named DALM, which contains three stages: 1) assigning pseudo labels to unlabeled target-domain data; 2) unifying the process of token generation and labeling with a Domain-Adaptive Language Model (DALM) to learn the shared context and annotation across domains; 3) using the trained DALM to generate labeled target-domain data. Experiments show that DALM consistently outperforms previous feature adaptation and CDDA methods on both ABSA and Aspect Extraction tasks. The source code is publicly released at https://github.com/NUSTM/DALM.

heading: 1 Introduction
text: As an important task in sentiment analysis, AspectBased Sentiment Analysis (ABSA) aims to extract aspect terms from sentences and predict the sentiment polarity towards each aspect term (Liu, 2012; Pontiki et al., 2016). For example, given a sentence “The screen is broken", the aspect term is screen and its sentiment polarity is Negative. With the advancements of deep learning techniques, a myriad of neural approaches have been proposed for ABSA
∗Equal contribution. † Corresponding author.
and achieved promising results on several benchmark datasets (Li et al., 2019a; He et al., 2019; Chen and Qian, 2020b). However, these methods heavily rely on labeled data with fine-grained annotation, which is often time-consuming and expensive to obtain for many emerging domains.
To alleviate the reliance on labeled data, many previous works resorted to unsupervised domain adaptation techniques, which aim to transfer knowledge from a resource-rich source domain to a target domain only with unlabeled data (Blitzer et al., 2007; Pan et al., 2010; Zhuang et al., 2015). Most existing domain adaptation methods on the ABSA task focus on learning shared feature representations across domains (Wang and Pan, 2018; Li et al., 2019c; Gong et al., 2020; Chen and Qian, 2021). Although these methods have obtained promising results, their models are only trained on the sourcedomain labeled data and thus insensitive to the important target-specific aspect and opinion terms.
To address this limitation, several recent studies have explored a new domain adaptation framework named Cross-Domain Data Augmentation (CDDA), which aims to directly generate much target-domain labeled data based on the labeled data from the source domain. These existing methods can be summarized into two groups: Masked
1456
Language Model (MLM)-based CDDA (Yu et al., 2021; Yang et al., 2022) and Sequence-to-Sequence (Seq2Seq)-based CDDA (Chen et al., 2021; Li et al., 2022). As shown in Fig. 1(a) and Fig. 1(b), the core idea behind existing CDDA methods is to first mask source-specific words in the sourcedomain labeled data, followed by using either the well-trained MLM or Seq2Seq models to automatically generate target-specific words and labels in the masked positions. Despite achieving significant improvements over previous feature adaptation methods, these CDDA approaches still have several shortcomings: 1) they only mask source-specific words or phrases but preserve other source-specific attributes such as syntactic structures, which make the distribution of the generated data different from that of the real target-domain data; 2) replacing source-specific words with target-specific words may destruct the semantic meaning of the original sentence, making the generated data lack of fluency and coherence; 3) existing CDDA methods regard each source-domain sentence as the template, thus limiting the diversity of the generated data.
To tackle these shortcomings, we propose a new cross-domain Data Augmentation approach based on Domain-Adaptive Language Modeling named DA2LM, which consists of three stages, including Domain-Adaptive Pseudo Labeling, DomainAdaptive Language Modeling, and Target-Domain Data Generation. Specifically, the labeled source data and unlabeled target data are first leveraged to train a base domain adaptation model, which is then used for predicting pseudo labels of unlabeled data in the target domain. Secondly, we design a novel Domain-Adaptive Language Model (DALM), and train it on the labeled source data and pseudo-labeled target data to learn the transferable context and label across domains. Different from most existing LMs, our DALM unifies the process of data generation and fine-grained annotation, aiming to simultaneously generate the next token and predict the label of the current token at each time step of the training stage. Finally, given the trained DALM, we employ it to generate many labeled target-domain data in an autoregressive manner with a probability-based generation strategy.
Our main contributions can be summarized as follows:
• We propose a three-stage framework named cross-domain Data Augmentation with Domain Adaptive Language Modeling (DA2LM), which
can generate a large amount of labeled targetdomain data for the cross-domain ABSA task.
• Under the framework, we devise a new domainadaptive language model, which unifies the process of data generation and labeling and captures the domain-invariant context and annotation for target-domain data generation.
• Experiments on four benchmark datasets demonstrate that our framework significantly outperforms a number of competitive domain adaptation methods on both ABSA and Aspect Extraction (AE) tasks. Further analysis on generated data shows the superiority of our framework in terms of data distribution, diversity, and fluency.

heading: 2 Related Work

heading: 2.1 Aspect-Based Sentiment Analysis (ABSA)
text: As an important task in sentiment analysis, ABSA has been extensively studied in the last decade. Earlier works mainly focus on two subtasks of ABSA, i.e., aspect extraction (AE) (Liu et al., 2015; Chen and Qian, 2020a) and aspect-based sentiment classification (ASC) (Zhang et al., 2016; Chen et al., 2017; Sun et al., 2019; Wang et al., 2020). Recently, many supervised methods are proposed to solve the two sub-tasks in an end-to-end manner, which either resort to multi-task learning to exploit the relations between AE and ASC (Luo et al., 2019; He et al., 2019; Chen and Qian, 2020b) or employ a collapsed tagging scheme to combine AE and ASC into a unified label space and formulate the task as a sequence labeling problem (Wang et al., 2018; Li et al., 2019a,b). Despite obtaining promising results on several benchmark datasets, these methods suffer from the lack of annotated data in many emerging domains. To alleviate this issue, we aim to propose an unsupervised domain adaptation method to generate sufficient labeled data for ABSA in any target domain.

heading: 2.2 Unsupervised Domain Adaptation
text: In the literature, a myriad of unsupervised domain adaptation methods have been proposed for coarsegrained sentiment analysis (Zhuang et al., 2020), including pivot-based methods (Blitzer et al., 2007; Yu and Jiang, 2016; Ziser and Reichart, 2018; Xi et al., 2020), auto-encoders (Glorot et al., 2011; Zhou et al., 2016), domain adversarial networks (Ganin and Lempitsky, 2015; Ganin et al., 2016; Li et al., 2018), and semi-supervised methods (He
et al., 2018; Ye et al., 2020). These methods primarily focus on learning domain-invariant representations to alleviate the distribution discrepancy across domains. Inspired by the success of these representation-based methods, a few recent studies have adapted them to the cross-domain ABSA task, in which the key idea is to learn a shared representation for each word or aspect term across domains (Ding et al., 2017; Wang and Pan, 2018, 2019, 2020; Li et al., 2019c; Zeng et al., 2022; Chen and Qian, 2022). Moreover, Lekhtman et al. (2021) proposed a customized pre-training approach with aspect category shift for the aspect extraction task.
Despite obtaining promising results, the major limitation of these aforementioned methods for cross-domain ABSA is that their models for the main ABSA task is solely trained on the sourcedomain labeled data. Thus, their models are insensitive to target-specific features. To address this issue, some studies have explored a Cross-Domain Data Augmentation framework (CDDA) to directly generate much target-domain labeled data, including MLM-based CDDA (Yu et al., 2021; Yang et al., 2022) and Seq2Seq-based CDDA (Chen et al., 2021; Li et al., 2022). However, the generated data by these methods has several limitations including 1) preserving many source-specific attributes such as syntactic structures; 2) lack of fluency and diversity. Thus, in this work, we aim to propose a new data augmentation framework that can generate fluent target-domain labeled data without any source-specific attributes.

heading: 3 Methodology

heading: 3.1 Problem Definition and Notations
text: Following previous studies (Li et al., 2019c), we formulate ABSA and AE as a sequence labeling problem. Given a sentence with n words x = {w1, w2, ..., wn}, the goal is to predict its corresponding label sequence y = {y1, y2, ..., yn}, where yj ∈ {B-POS, I-POS, B-NEG, I-NEG, B-NEU, I-NEU, O} for ABSA and yj ∈ {B, I, O} for AE.
In this work, we focus on the unsupervised domain adaptation setting, in which the source domain has enough labeled data and the target domain only has unlabeled data. Let DS = {(xsi ,ysi )}N s i=1 denote a set of source-domain labeled data, and DT = {xti}N t
i=1 a set of target-domain unlabeled data. The goal is to leverage DS and DT to predict the label sequences of test data from the target domain.

heading: 3.2 Overview
text: As illustrated in Figure 2, our Cross-Domain Data Augmentation framework contains three key stages, including 1) Domain-Adaptive Pseudo Labeling, 2) Domain-Adaptive Language Modeling, and 3) Target-Domain Data Generation. In the first stage, an aspect-aware domain adaptation model is trained to assign pseudo labels to unlabeled data in the target domain. In the second stage, the labeled source data and the pseudo-labeled target data are used to train a domain-adaptive language model, which integrates data generation and sequence labeling in a unified architecture to capture the transferable context and annotation across domains. After training the DALM, the last stage uses probabilitybased generation strategy to generate diverse targetdomain data with fine-grained annotations in an autoregressive manner.

heading: 3.3 Domain-Adaptive Pseudo Labeling
text: In this stage, our goal is to assign the pseudo labels to each unlabeled data in the target domain. Since the data distribution of the source domain is different from that of the target domain, directly training a classifier on the labeled source data to predict the pseudo labels of the unlabeled target data will bring much noise. Thus, it is necessary to alleviate the domain discrepancy to improve the quality of pseudo-labels. Since aspect terms are shown to play a crucial role in ABSA (Gong et al., 2020), we attempt to explicitly minimize the distance between source-domain and target-domain aspect term representations via Maximum Mean Discrepancy (MMD) (Gretton et al., 2012).
Specifically, given the labeled source data DS and the unlabeled target data DT , we first obtain the aspect terms in DS via the gold labels and extract the aspect terms in DT based on a rulebased algorithm named Double Propagation (Qiu et al., 2011). Let us use xd = {wd1 , wd2 , ..., wdn} to denote a source or target domain sentence and use ad = {wdi , ..., wdj } to denote one of the aspect terms in the sentence, where d ∈ {s, t}. We then employ a pre-trained BERT model to obtain the hidden representation of the sentence Hd = {hd1,hd2, ...,hdn} and the aspect term representation ad = g(hdi , ...,h d j ), where h
d ∈ Rr, r refers to the hidden dimension, and g(·) denotes the meanpooling operation. Next, we propose an aspectlevel MMD loss to alleviate the distribution dis-
crepancy across domains as follows:
Lmmd = d2k(DSa ,DTa ) = 1
(N sa) 2
Nsa∑
i,j
k(asi ,a s j)+
1
(N ta) 2
Nta∑
i,j
k(ati,a t j)−
2
N saN t a
Nsa∑
i
Nta∑
j
k(asi ,a t j),
where DSa and DTa respectively denote the sets of aspect term representations in the source domain and the target domain, N sa and N t a refer to the number of aspect terms in the two domains, and k(·) denotes the Gaussian Kernel function.
Meanwhile, for each source sample, the hidden representation Hs is fed into a Conditional Random Field (CRF) layer to predict the label sequence for the ABSA or AE task p(ys|Hs). The goal is to minimize the negative log-probability of the correct label sequence of each source-domain sample:
Lcrf = − Ns∑
i=1
log p(ysi |Hsi ). (1)
The CRF loss for the ABSA or AE task and the aspect-level MMD loss are combined to train the base model Cb:
L = Lcrf + αLmmd, (2) where α is the hyper-parameter.
Finally, we use Cb to assign pseudo labels to each sample in DT , and obtain DPT = {(xpti ,y pt i )}N t i=1.

heading: 3.4 Domain-Adaptive Language Modeling
text: To generate a large amount of target-domain labeled data with diverse syntactic structures, we propose a Domain-Adaptive Language Model (DALM), which leverages the labeled source data DS and the pseudo-labeled target data DPT to learn the shared distribution of words and labels across domains. Since our DALM unifies the process of word generation and sequence labeling, at each time step, we employ the current input token and the predicted label at the previous step to simultaneously maximize the probabilities of predicting the next token and the label of the current token.
Specifically, for each sample (x,y) ∈ DS ∪ DPT , we first construct an input token sequence, in which we insert a special token ⟨BOS⟩ to denote the sentence beginning, followed by a domain-specific token (i.e., [source] or [target]) to distinguish the domain that x belongs to. Let xin = {⟨BOS⟩, w0, w1, w2, ..., wn} denote the expanded input sentence, where w0 ∈ {[source], [target]}. Moreover, we construct another input label sequence, denoted by yin = {⟨BOL⟩, y⟨BOS⟩, y0, y1, y2, ..., yn−1}, where ⟨BOL⟩ denotes the initial state of the label sequence, y⟨BOS⟩ is O, and yj refers to the label of wj . According to the input, the output token sequence is xout = {w0, w1, w2, ..., wn, ⟨EOS⟩}. The output label sequence is yout = {y⟨BOS⟩, y0, y1, y2, ..., yn}. The top of Figure 2
shows an example of two input and two output sequences for a sample from the source domain.
Next, for the input token sequence xin, we employ a decoder such as LSTM and the pre-trained GPT-2 model (Radford et al., 2019) to get its hidden representation as follows:
ew−1, e w 0 , ..., e w n = Decoder(w−1, w0, w1, ..., wn),
where w−1 denotes ⟨BOS⟩, ewt ∈ Rd is the token representation, and d is the hidden dimension. For the input label sequence yin, a label embedding layer is used to get the label representation:
ey−2, ..., e y n−1 = LabelEmb(y−2, y−1, ..., yn−1),
where y−2 and y−1 denote ⟨BOL⟩ and y⟨BOS⟩, and eyt ∈ Rd. Next, at each time step t, we add ewt and eyt−1 to produce a token and label-aware representation (i.e., et = ewt + e y t−1), which is then fed into two different full-connected softmax layers to predict the probabilities of the next token wt+1 and the label yt as follows:
P (wt+1|w≤t, y≤t−1) = σ(Wwet + bw), (3) P (yt|w≤t, y≤t−1) = σ(Wyet + bw), (4)
where σ is the softmax function, and Wx ∈ R|Vx|×d, Wy ∈ R|Vy |×d, and |Vx| and |Vy| are the vocabulary size and the label size. For each sample (x,y) ∈ DS ∪ DPT , we optimize the parameters for DALM by minimizing the combination of cross entropy losses for the output token sequence and label sequence as follows:
L = Lw + Ly, (5) Lw = − n∑
t=−1 logP (wt+1|w≤t, y≤t−1), (6)
Ly = − n∑
t=−1 logP (yt|w≤t, y≤t−1). (7)

heading: 3.5 Target-Domain Data Generation
text: After training the DALM, we employ it to generate target-domain data with fine-grained annotations in an autoregressive manner. As shown in the bottom of Figure 2, the ⟨BOS⟩ token and the target-specific token [target] are fixed as the first two input tokens of the DALM, and ⟨BOL⟩ and O are fixed as the first two input labels. Next, we adopt a probabilitybased generation strategy to generate the following tokens and their corresponding labels.
At each time step t, we first rank all the tokens in Vx based on the probabilities computed by Eq. 3 and pick top-k tokens as a candidate set Ct+1. We then sample a token wt+1 from Ct+1 as the next token. As the candidate tokens in Ct+1 are predicted with higher probabilities, the generated data are generally fluent and close to the real target-domain data. Moreover, given the same context, the DALM can choose a synonym as the next token due to the randomness of sampling, which is conducive to diversifying the generated data.
Next, for the label generation at each time step t, we directly select the label with the highest probability computed by Eq. 4 as the label of the current token yt, which can ensure the quality of the generated label sequence.
The above process of token generation and labeling will be stopped when the next token is predicted as ⟨EOS⟩. Because of the randomness brought by sampling, the trained DALM can be used to generate any amount of labeled data. However, generating more data may lead to significant vocabulary redundancy of generated data. Thus, once the size of generated data equals to Ng, we will stop generating target-domain labeled data.

heading: 3.6 Generated Data Filtering
text: To mitigate the presence of low-quality labels in the target data generated from the probability-based generation strategy, we introduce the following steps for generated data filtering: 1) Delete data with the illogical labels that violate the prefix order of the BIO tagging schema (e.g., having O before I in the AE task and having B-Positive before INeutral in the ABSA task); 2) Delete repetitive data whose token and label sequences are the same, and only keep one of the duplicate samples; 3) Use the base model Cb in Section 3.3 to predict the label sequences of the generated sentences and delete data whose label sequences are different from those predicted by Cb.
Let us use Dg = {(xgi ,y g i )}N g
i=1 to denote the set of generated target-domain data. We then train a standard BERT-CRF model (Li et al., 2019b) on Dg, and use it to predict the label sequences of test data from the target domain.

heading: 4 Experiments

heading: 4.1 Experimental Settings
text: Datasets. To evaluate the effectiveness of the proposed DA2LM framework, we conduct experi-
ments on four benchmark datasets, namely Laptop (L), Restaurant (R), Device (D), and Service (S), as shown in Table 1. L contains data from the laptop domain in SemEval 2014 (Pontiki et al., 2014). R is the union set of the restaurant data from SemEval 2015 (Pontiki et al., 2015) and SemEval 2016 (Pontiki et al., 2016). D contains device data about 5 digital products (Hu and Liu, 2004). S contains data from web services (Toprak et al., 2010).
Evaluation. Following (Li et al., 2019c), we choose 10 different source → target domain pairs for experiments. L → D and D → L are removed since the two domains are very similar. For each cross-domain pair, DA2LM generates sufficient target-domain labeled data and then directly trains a BERT-CRF classifier on the generated targetdomain data. We evaluate the model predictions based on Micro-F1 under the exact match, which means that the predicted aspect-sentiment pairs are considered as correct only if they exactly match with the gold aspect-sentiment pairs.
Parameter Setting. For the BERT-CRF model used in DA2LM, we employ a domain-specific BERT-base model named BERT-Cross (Xu et al., 2019), which was post-trained on a large amount of Yelp and Amazon Electronic data (He and McAuley, 2016). For Domain-Adaptive Pseudo Labeling, the hyper-parameter α in Eq. 2 is set as 0.01, and we adopt the Adam algorithm with a learning rate of 3e-5 to optimize the parameters. For Domain-Adaptive Language Modeling, we finetune the LSTM and the pre-trained language model GPT-2 (Radford et al., 2019) on DS ∪ DPT , and using the Adam algorithm as the optimizer with a learning rate of 3e-3 and 3e-4 respectively. For Target-Domain Data Generation, we choose the top-k tokens (i.e., k=100) as the candidate set and the maximum number of generated data Ng is set to 10000 in token-sampling generation. All the experiments are run on a single Nvidia 1080Ti GPU.

heading: 4.2 Main Results
text: To show the effectiveness of our DA2LM approach, we consider the following competitive domain adaptation comparison systems for the cross-
domain ABSA task.
• BERT-NoDA (Kenton and Toutanova, 2019): a baseline system without domain adaptation, which directly fine-tunes a BERT-base model on labeled source-domain data.
• BERT-Cross (Xu et al., 2019): a domainadaptive BERT-CRF model, in which the BERTbase model was post-trained on a myriad of Ecommerce data and the full model is fine-tuned on labeled source-domain data.
• UDA (Gong et al., 2020): a unified domain adaptation approach that integrates feature-based and instance-based adaptation for cross-domain ABSA.
• FMIM (Chen and Wan, 2022): a featurebased domain adaptation method, using the finegrained mutual information maximization technique.
• CDRG (Yu et al., 2021): a cross-domain review generation approach that exploits each labeled source-domain review to generate a labeled target-domain review based on masked language models.
• GCDDA (Li et al., 2022): a generative crossdomain data augmentation framework that leverages a pre-trained sequence-to-sequence model BART to generate target-domain data with finegrained annotation.
The comparison results on the cross-domain ABSA and AE task are reported in Table 2. For our proposed framework, we present the results of both LSTM and GPT-2-based DA2LM. We can observe that our framework generally achieves the best performance on most cross-domain pairs and DA2LM outperforms the state-of-the-art method by 1.86% and 0.90% on average for the ABSA and AE task respectively. We conjecture the reasons as follows. First, DA2LM can directly generate numerous high-quality target domain labeled data, thereby overcoming the sensitivity to source data in feature-based domain adaptation methods. Second, there is still a considerable distribution discrepancy between the generated data in previous cross-domain data augmentation methods and the real target-domain data because these methods preserve source-specific attributes such as syntactic structures. Moreover, since previous cross-domain data augmentation methods are based on the word replacement technology, the fluency and diversity
Tasks Methods S→R S→L S→D R→S R→L R→D L→S L→R D→S D→R AVE
of generated data in these methods are inferior to our DA2LM approach.
In addition to the above observations, Table 2 shows that LSTM-based DA2LM is similar to GPT-2-based DA2LM and also outperforms previous domain adaptation methods on average, which implies that our cross-domain data augmentation framework is robust and does not rely on the pretrained language model.
Furthermore, as shown in Table 1 and Table 2, the proposed model underperforms several baseline systems when the source/target sample size ratio is larger than 1 (e.g., R → S, L → S, D → S, R → L). We believe the reason of the performance drop is as follows: when the number of targetdomain data is less than that of source-domain data, it will inevitably lead the Domain-Adaptive Language Model (DALM) to pay more attention to source-domain data instead of target-domain data. Hence, in the target-domain data generation process, the trained DALM may still generate sourcespecific words, and thus bring negative effects to the final performance.

heading: 4.3 Ablation Study
text: To explore the effects of each component in DA2LM, we show the results of our ablation study in Table 3.
Firstly, after removing the aspect-level MMD loss in the domain-adaptive pseudo labeling (DAPL) stage, the average performance on 10 cross-domain pairs drops dramatically, which indicates that it is important to alleviate the domain discrepancy via the MMD loss in DAPL. Secondly, removing the domain-adaptive language modeling
(DALM) and target-domain data-generation (DG) stages decreases the average F1 score by 2.71 absolute percentage points. This shows that automatically generating a large amount of target-domain labeled data plays an indispensable role in our DA2LM framework. Thirdly, for the training of DALM, the removal of source-domain labeled data also leads to a significant drop in the average F1 score. This implies that the source-domain data is indeed helpful for capturing domain-invariant context and annotation.
Moreover, we remove the malposed generation strategy, which means it does not take the current token into account when predicting the label of the current token. As shown in Table 3, the performance of DA2LM drops dramatically since it generates low-quality label sequences. Lastly, because a language model-based data augmentation method DAGA (Ding et al., 2020) has shown success in standard in-domain ABSA tasks, we propose to replace DALM in our DA2LM framework with a variant of DAGA, i.e., a language model trained on source and target-domain data with linearized
Criterion Methods S→R S→L S→D R→S R→L R→D L→S L→R D→S D→R AVE
labels before each aspect term. For fair comparison, we also employ GPT-2 (Radford et al., 2019) as the pre-trained language model. As shown at the bottom of Table 3, replacing DALM with DAGA leads to a moderate performance drop, which proves the importance of DALM in our DA2LM approach.

heading: 4.4 Evaluation on Generated Data
text: In this subsection, we conduct additional experiments to evaluate the quality of data generated by DA2LM and report the performance in Table 4.
Diversity. Diversity denotes the percentage of unique aspect terms in all aspect terms. The results in Table 4 clearly show that DA2LM can generate more aspect terms since other methods need to regard source-domain sample as the template. Moreover, our framework employs a probabilitybased sampling strategy to generate the next token, which can improve the diversity of generated aspect terms.
Perplexity. To evaluate the coherence of generated data, we further calculate the perplexity1 of data generated from each compared method based on a pre-trained language model GPT-2.2 In the fourth to sixth rows of Table 4, it is clear to see
1https://huggingface.co/spaces/ evaluate-measurement/perplexity
2Note that different from Li et al. (2022) which uses 2 as the base of the exponential function, we employ e as the base.
that the perplexity of our DA2LM framework is significantly lower than that of other methods. This shows that for MLM-based and Seq2Seq-based CDDA methods, simply replacing source-specific attributes with target-specific attributes may break the syntactic structure of the original sentence and thus the generated sentences are not coherent. In contrast, our DA2LM framework relies on language modeling to automatically generate tokens and their corresponding labels in an autoregressive manner.
Maximum Mean Discrepancy (MMD). MMD is used to measure the distribution distance between the generated data in different methods and the real target-domain test data. The results in the last four rows show that the generated data in DA2LM are much closer to the target domain than other methods, which indicates DA2LM can generate more authentic target-domain data and better alleviate the distribution discrepancy across domains.
Visualization. To visually verify the superiority of our DA2LM framework, we further utilize t-SNE (Van der Maaten and Hinton, 2008) to perform a visualization of the sentence representations obtained by a pre-trained language model BERT (Kenton and Toutanova, 2019). Figure 3 shows the visualization result on a cross-domain pair S → R. As shown in Figure 3, the distribution of generated data in CDRG and GCDDA is still similar to that of source-domain data because these methods still
preserve many source-domain attributes including contexts and syntactic structures. In contrast, there is almost no discrepancy between the generated data in DA2LM and the target-domain data, as shown in the right of Figure 3.
These observations demonstrate the advantage of DA2LM over previous CDDA methods in terms of diversity, fluency, and data distribution.

heading: 4.5 Compatibility with Existing DA Methods
text: To show the compatibility of our DA2LM framework, we replace the base model Cb in the first stage (i.e., domain-adaptive pseudo labeling) with other existing domain adaptation methods including UDA (Gong et al., 2020), FMIM (Chen and Wan, 2022) and CDRG (Yu et al., 2021).
Table 5 shows the average results of different base models with their DA2LM variants on 10 source → target domain pairs for the cross-domain ABSA task and the cross-domain AE task, respectively. Firstly, we can find that by using the targetdomain labeled data from our DA2LM framework, the performance of existing domain adaptation methods is generally boosted on average for crossdomain ABSA and AE, which demonstrates the usefulness of our DA2LM framework and the robustness of the generated target-domain data. Secondly, by comparing all DA2LM variants, we can observe that DA2LM-FMIM consistently obtains the best average performance on cross-domain ABSA and AE. This suggests that our DA2LM framework is compatible with any domain adaptation method, and it can generally achieve better results with better base models.

heading: 5 Conclusion
text: In this paper, we proposed a cross-domain Data Augmentation framework based on DomainAdaptive Language Modeling (DA2LM), which contains three key stages to automatically generate sufficient target-domain labeled data, including 1) Domain-Adaptive Pseudo Labeling, 2) Domain-Adaptive Language Modeling, and 3) Target-Domain Data Generation. Experiments on four benchmark datasets show that our DA2LM framework consistently outperforms the state-ofthe-art method for the cross-domain ABSA task. Moreover, further evaluation results demonstrate the superiority of the generated data in terms of diversity, fluency, and data distribution.
Limitations
Despite obtaining promising results, our proposed approach still has the following limitations.
First, although our DA2LM approach can generate a large amount of target-domain data with high diversity, the generated words are still limited by the source-domain labeled data and target-domain unlabeled data. How to make the model generate novel target-domain words is a challenging problem to explore in the future.
Second, our DA2LM model is primarily proposed for the ABSA and AE tasks, which are not directly applicable for the other information extraction tasks with more than two elements, such as Aspect Sentiment Triplet Extraction (ASTE). Therefore, cross-domain data augmentation for multiple-element information extraction tasks may be a promising followup direction.
Ethics Statement
We conduct experiments on four publicly available datasets, i.e., Laptop (L), Restaurant (R), Device (D), and Service (S). These datasets do not share personal information and do not contain sensitive content that can be harmful to any individual or community. Due to the lack of ethics and bias constraint in the data generation process, the generated data from our trained Domain-Adaptive Language Model may contain sensitive and misleading content. Therefore, it is necessary to manually check these generated data when applying them to realworld applications.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their insightful comments. This work was supported by the Natural Science Foundation of China (62076133 and 62006117), and the Natural Science Foundation of Jiangsu Province for Young Scholars (BK20200463) and Distinguished Young Scholars (BK20200018).

heading: A Appendix
text: A.1 Case Study and Error Analysis In this section, we select several representative examples generated by different methods to demonstrate the effectiveness of our DA2LM framework.
Case Study. Table 6 shows several examples of CDRG, GCDDA and DA2LM on a cross-domain pair L → R. Firstly, we can observe that the MLMbased approach CDRG and the Seq2Seq-based approach GCDDA fail to replace some sourcespecific words such as “laptop” and “Miscrosoft office” with target-specific words. Besides, it is clear that the generated target-domain data in CDRG and GCDDA are lack of fluency, coherence, and diversity, because they both generate target-domain data based on a source template sentence by replacing words. In contrast, our DA2LM approach can generate much more diverse target-domain data due to the randomness of sampling. Moreover, because the DALM in our framework is based on the language model, it is not surprising that the sentences generated in DA2LM are generally fluent and coherent.
Error Analysis. Furthermore, we also manually verify the label correctness of the target-domain data generated from our DA2LM framework, and show two generated samples with incorrect labels at the bottom of Table 6. We find that DA2LM is prone to identify a target-specific attribute as an aspect term, even if it is not the target of the sentiment expression (e.g., “restaurants”) or is an incomplete aspect term (e.g., “sake”). We conjecture the reason is our adoption of a rule-based algorithm to obtain the target-domain aspect terms to minimize the distance between source-domain and target-domain
aspect term representations in Section 3.3, which may result in the noise in the pseudo-labeled target data for Aspect Term Extraction. However, the results and analysis in Section 4.5 demonstrate that our DA2LM framework is generally compatible with various domain adaptation methods and has the potential to deliver better performance when employed in conjunction with more powerful base models.
A.2 Detailed Evaluation on the Compatibility with Existing DA Methods
Table 7 and Table 8 show the detailed comparison results of different base models with their DA2LM variants on all domain-pairs for the cross-domain ABSA task and the cross-domain AE task. We can observe that the variants of our DA2LM show consistent improvements over different base models on most domain pairs for both tasks.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and senction Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? We use the pre-trained language model GPT-2 as mentioned in Section 3.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: In Section 3 named Methodology.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? We use publicly available pretrained language models and datasets from previous works.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? In Section 3, we discuss in detail how to use the scientific artifact. And we introduce the intended use of our framework in Section 1.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data we use is based on publicly available datasets, which have been checked and preprocessedby previous works.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? We describe the key stages and settings in Section 3 and Section 4 in detail.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We describe the dataset we use in Section 4.
C 3 Did you run computational experiments?

heading: In Section 4.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? We describe the parameters setting and computing infrastructure in Section 4.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? We describe the experiment setup in Section 4.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We describe them in Section 4.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? We describe them in Section 4.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: How to make the model generate novel target-domain words is a challenging problem to explore in the future. Second, our DA2LM model is primarily proposed for the ABSA and AE tasks, which are not directly applicable for the other information extraction tasks with more than two elements, such as Aspect Sentiment Triplet Extraction (ASTE). Therefore, cross-domain data augmentation for multiple-element information extraction tasks may be a promising followup direction