SlowGuess/NAACL_2022_OCR huggingface dataset:
Junjie Li, Jianfei Yu*, and Rui Xia*
{jj_li, jfyu, rxia}@njust.edu.cn
As a fundamental task in opinion mining, aspect and opinion co-extraction aims to identify the aspect terms and opinion terms in reviews. However, due to the lack of fine-grained annotated resources, it is hard to train a robust model for many domains. To alleviate this issue, unsupervised domain adaptation is proposed to transfer knowledge from a labeled source domain to an unlabeled target domain. In this paper, we propose a new Generative Cross-Domain Data Augmentation framework for unsupervised domain adaptation. The proposed framework is aimed to generate target-domain data with fine-grained annotation by exploiting the labeled data in the source domain. Specifically, we remove the domain-specific segments in a source-domain labeled sentence, and then use this as input to a pre-trained sequence-to-sequence model BART to simultaneously generate a target-domain sentence and predict the corresponding label for each word. Experimental results on three datasets demonstrate that our approach is more effective than previous domain adaptation methods. The source code is publicly released at https://github.com/NUSTM/GCDDA.
Aspect and opinion co-extraction is a fundamental task in opinion mining. It aims to extract aspect terms and opinion terms from review sentences. Given a review "the keyboard is great", the aspect term is keyboard and the opinion term is great. Most existing works formulate aspect and opinion co-extraction as a sequence labeling task. With the support of deep learning, many supervised methods have achieved remarkable results (Liu et al., 2015; Yin et al., 2016; Wang et al., 2017; Wu et al., 2020). However, owing to the high cost of fine-grained annotations, the scarcity of labeled data in many new domains makes them fail to obtain a robust performance.
To alleviate the deficiency of labeled data, unsupervised domain adaptation is proposed to transfer knowledge from a labeled source domain to an unlabeled target domain. The main difficulty of unsupervised domain adaptation lies in the distribution discrepancy between data from different domains. Specifically, reviews from different domains may have distinct aspect terms, opinion terms, and expression patterns. To tackle the distribution discrepancy problem, many domain adaptation methods were proposed for coarse-grained sentiment classification (Blitzer et al., 2007; Yu and Jiang, 2016; Ziser and Reichart, 2018; Ghosal et al., 2020). However, due to the complexity of fine-grained domain adaptation for aspect and opinion extraction, only a handful of studies attempt to address this issue. Most of them are based on the following three paradigms:
- rule-based adaptation, which iteratively extracts aspects and opinions in sentences based on domain-independent syntactic rules and opinion dictionary (Li et al., 2012; Ding et al., 2017).
- feature-based adaptation, which incorporates the general syntactic information to learn a domain-invariant word representation across domains (Wang and Pan, 2018; Li et al., 2019; Pereg et al., 2020; Chen and Qian, 2021).
- data augmentation-based adaptation, which utilizes the labeled data in the source domain to directly generate high-quality target-domain data with fine-grained annotation (Yu et al., 2021).
However, the paradigms mentioned above suffer from the following problems. For rule-based adaptation methods, it is difficult to design high-quality manual rules and opinion set, which may bring low precision results. For feature-based adaptation methods, although they can bridge the domain gap by using the unlabeled data from both domains, the
Human Language Technologies, pages 4219 - 4229
main task classifier is only trained by the source labeled data, which fails to exploit the important supervision signals in the target domain. Although our recent work (Yu et al., 2021) has shown the superiority of data augmentation-based adaptation methods, its main limitation lies in that it only replaces source-specific aspects and opinions in the source domain review with target-specific aspects and opinions, but ignoring other domain-specific attributes such as collocations and expression styles, which limits the quality and diversity of generated target-domain data.
To this end, we propose a Generative Cross-Domain Data Augmentation framework for unsupervised domain adaptation, which generates target labeled data from source labeled data based on a pre-trained sequence-to-sequence model, i.e., BART (Lewis et al., 2020). Specifically, given the labeled samples in the source domain, we first train a classifier to assign pseudo labels for each unlabeled sample in the target domain. With the labeled and pseudo labeled samples from both domains, we then remove their domain-specific features to obtain domain-independent samples. Next, we employ the BART model by feeding the domain-independent samples and their token labels to the encoder and reconstructing their original texts and corresponding labels in the decoder. In the inference stage, given a source-domain sentence in which the domain-specific features are removed, the model can generate a target-domain sentence by integrating target-specific features into the source context while predicting its token-level labels. Compared with the previous method (Yu et al., 2021), our new approach generates high-quality target-domain data with more flexible syntactic patterns by making full use of the knowledge from the domain-invariant source contexts. With the label prediction, our approach can easily obtain the adaptive token labels for each generated target-domain sample.
Our main contributions in this work can be summarized as follows:
- We propose a Generative Cross-Domain Data Augmentation framework for unsupervised domain adaptation, which exploits the pre-trained sequence-to-sequence model BART to generate the target-domain data with fine-grained annotation based on the labeled source-domain data.
- Extensive experiments on three datasets of aspect and opinion co-extraction demonstrate the
effectiveness of our generative framework. Further analysis shows that our framework is able to generate more fluent and diversified target-domain samples than previous approaches.
2.1 Aspect and Opinion Extraction
Aspect and opinion extraction is an fundamental task in opinion mining, which is widely studied in the literature. Unsupervised learning methods have been proposed to extract aspect and opinion terms by association rule and extended opinions (Hu and Liu, 2004) or by manual syntactic rules and predefined opinion dictionary (Qiu et al., 2011). With the constant attention on deep learning, many supervised approaches have achieved remarkable results (Liu et al., 2015; Zhang et al., 2015; Yin et al., 2016; Wang et al., 2017; Yu et al., 2018; Wu et al., 2020). However, the deficiency of annotated data in many domains make them fail to achieve desired results.
To solve the issue mentioned above, many domain adaptation methods proposed for coarse-grained sentiment classification either align the domain-specific feature space with domain-independent pivot words (Blitzer et al., 2007; Pan et al., 2010; Yu and Jiang, 2016) or learn a domain-invariant representation based on auto-encoder (Glorot et al., 2011; Yin et al., 2016) and domain adversarial learning (Li et al., 2018b). However, only a few fine-grained domain adaptation approaches are proposed for ABSA (Li et al., 2019; Gong et al., 2020) or aspect and opinion extraction. Most of them can be categorized into two types: 1) rule-based methods (Li et al., 2012; Ding et al., 2017) exploit manual syntactic rules and opinion seeds to extract aspects and opinions, which usually obtains less satisfactory results due to the quality of rules; 2) feature representation-based methods focus on learning a domain-invariant word representation (Wang and Pan, 2018; Pereg et al., 2020; Chen and Qian, 2021) based on the unlabeled data. However, the main task classifier fails to exploit the important supervision signals in the target domain.
Data augmentation is another solution for the scarcity of annotation data. Many previous works mainly focus on the sentence-level task (Guo
Figure 1: Overview of our proposed Generative Cross-Domain Data Augmentation framework.
et al., 2019; Min et al., 2020) and the token-level task (Gao et al., 2019; Chen et al., 2020). For aspect/opinion extraction, Ding et al. (2020) designed a data augmentation method with language models trained on the linearized labeled sentences to generate considerable labeled data. Hsu et al. (2021) exploited a masked language model BERT to substitute unimportant words in sentences to enhance the data diversity. Li et al. (2020) proposed to generate a new review while preserving its original labels and aspect terms with a masked sequence-to-sequence model. However, all the studies above follow the in-domain setting, which fails to exploit the rich contexts from other domains. Our recent work (Yu et al., 2021) proposed a cross-domain review generation method, which replaces source-specific aspects and opinions in labeled source-domain reviews with target-specific aspect and opinions based on the masked language model BERT. However, it fails to consider other domain-specific attributes such as collocations and expression styles in cross-domain review generation. Based on our previous work, we propose a Generative Cross-Domain Data Augmentation framework for unsupervised domain adaptation, which can generate more flexible target data with fine-grained annotation in this paper.
In this paper, we focus on the aspect and opinion co-extraction task and formulate it as a sequence labeling problem. We denote a review
as a sequence of tokens  $x = [x_{1},x_{2},\dots,x_{n}]$ , and the task aims to predict the label sequence of the review  $y = [y_{1},y_{2},\dots,y_{n}]$ , where  $y_{i}\in \{B - ASP, I - ASP,B - OP,I - OP,O\}$ . For unsupervised domain adaptation, we are given a set of source-domain labeled reviews  $D_{S} = \{(x_{i}^{s},y_{i}^{s})\}_{i = 1}^{N^{s}}$  and a set of unlabeled data from the target domain  $D_{U} = \{x_{i}^{u}\}_{i = 1}^{N^{u}}$ . The goal is to predict token-level labels on the test set from the target domain  $D_{T} = \{x_{i}^{t}\}_{i = 1}^{N^{t}}$ .
In this paper, we propose a Generative Cross-Domain Data Augmentation framework, which generates target-domain reviews with fine-grained annotation from source-domain labeled reviews. Figure 1 illustrates the overall architecture of our generative framework. Specifically, we leverage the labeled samples in the source domain to train a classifier, and then assign pseudo labels for each unlabeled sample in the target domain. Given the labeled and pseudo labeled sentences from both domains, we employ a segment mask module to remove their domain-specific features while preserving their context and original labels. Next, a pre-trained sequence-to-sequence model BART concatenates the domain-invariant context and original labels as the input of the encoder, followed by recovering the original sentences and predicting their token-level labels via the decoder. During the inference stage, given a labeled source-domain sentence, the model removes its domain-specific
features, and then employs the fine-tuned BART model to generate a target-domain sentence with its token-level labels.
4.1 Pseudo Label Annotation
First, we train a base classifier on the labeled data from the source domain  $D_{S}$ , which employs a pretrained BERT model (Devlin et al., 2019) to obtain the contextualized word representation and a Conditional Random Field (CRF) layer for sequence labeling. The trained classifier is then applied to assign pseudo labels on the unlabeled data from the target domain  $D_{U}$  to achieve the pseudo labeled target-domain data  $D_{TP}$ . However, the label quality of  $D_{TP}$  tends to be relatively low due to the distribution discrepancy between the source and target domains. Therefore, it is crucial to generate high-quality target-domain data by combining the contexts from the labeled source-domain data  $D_{S}$  with the weak supervision in the pseudo labeled target-domain data  $D_{TP}$ .
4.2 Domain-Specific Feature Mask and Reconstruction
To automatically generate the target-domain data with fine-grained annotation, we propose a two-step approach as follows. First, a domain-specific segment set is extracted from the labeled and pseudo labeled datasets in both domains. Next, given a labeled sentence, we replace the domain-specific segments with [mask] tokens, followed by feeding the masked domain-independent review to a pre-trained BART model (Lewis et al., 2020) to reconstruct its original text, domain prompt, and token-level labels.
4.2.1 Domain-Specific Segment Mask
Because reviews in different domains contain different aspects, opinions, and expression patterns, we define the text segments occurring more frequently in one domain as domain-specific segments or features. To obtain these domain-specific features, we introduce a frequency-ratio method based on (Li et al., 2018a). Specifically, we segment all sentences into word segments of different lengths, and then calculate the relative frequency of the n-gram segment  $w$  in the dataset  $D_v$  as follows:
$$
s \left(w, D _ {v}\right) = \frac {\operatorname {c o u n t} \left(w , D _ {v}\right) + \lambda}{\left(\sum_ {v ^ {\prime} \in V , v ^ {\prime} \neq v} \operatorname {c o u n t} \left(w , D _ {v ^ {\prime}}\right)\right) + \lambda}, \tag {1}
$$
where  $\text{count}(w, D_v)$  denotes the frequency of an n-gram  $w$  in  $D_v$ ,  $v \in V$ ,  $V = \{S, TP\}$ , and  $\lambda$  is the smoothing parameter.
Next, we filter these n-gram  $w$  based on the relative frequency as follows:
$$
s (w, D _ {v}) \geq \delta , \tag {2}
$$
where  $\delta$  is a specified threshold.
We regard the filtered n-gram segments as the domain-specific segment set  $M$ . Given a review, we exploit the Forward Maximum Matching algorithm to match the segments that appear in  $M$ , and then replace each matched word with a special token [mask]. For example, in Figure 1, given a sentence "i like the spicy tuna roll", if the matched segment is "spicy tuna roll", we obtain a corresponding masked sentence "i like the [mask] [mask] [mask]". It is worth noting that as long as one word of a domain-specific aspect phrase or opinion phrase is masked, we will mask the whole aspect or opinion phrase.
4.2.2 Reconstruction and Label Generation
According to the above steps, for each sample  $(X,L)\in D_S\cup D_{TP}$ , a corresponding masked tuple  $(\tilde{X},L)$  can be obtained as the model input, where the masked sentence is  $\tilde{X} = [\tilde{x}_1,\tilde{x}_2,\dots,\tilde{x}_n]$  and the label sequence for each word is  $L = [l_1,l_2,\dots,l_n]$ . To reconstruct the original text as well as its corresponding labels, we implement several modification on BART.
Encoder: In addition to the word embedding and position embedding layers in BART, we also establish a label embedding layer:
$$
E _ {x} = \operatorname {T o k e n E m b} ([ \tilde {x} _ {1}, \tilde {x} _ {2}, \dots , \tilde {x} _ {n} ]), \tag {3}
$$
$$
E _ {l} = \operatorname {L a b e l E m b} ([ l _ {1}, l _ {2}, \dots , l _ {n} ]), \tag {4}
$$
where  $E_{x}\in R^{n\times d},E_{l}\in R^{n\times d}$  , and  $d$  is the dimension of the embedding. The output of the hidden state can be formulated as:
$$
H = \operatorname {B a r t E n c o d e r} \left(E _ {x} + E _ {l}\right), \tag {5}
$$
where  $H\in R^{n\times d^{\prime}}$ $d^{\prime}$  denotes the hidden dimension.
Decoder: To inform the BART model to distinguish the features from different domains, we insert a domain label tuple  $([\text{source}], O)$  or  $([\text{target}], O)$  at the beginning of the decoder, which can be regarded as a domain prompt. For each time step  $t$ , the decoder takes the previous decoder predictions
Algorithm 1 Training Procedure
Require:  $D_S$ : source labeled dataset;  $D_U$ : target unlabeled dataset;  $K$ : number of training iterations for BART;
1: Train a base classifier on dataset  $D_{S}$
2: Get the target pseudo labeled dataset  $D_{TP}$  by assigning pseudo labels on  $D_U$
3: Extract and construct domain-specific n-gram segments set  $M$  from  $D_S \cup D_{TP}$  using Eq.(1), Eq.(2)
4: for  $i \gets 1$  to  $K$  do
5: Select an example  $(X,L),d$  from  $D_{S}\cup D_{TP},$  where  $d\in \{[source],[target]\}$
6:  $(\hat{X},L)\gets SegmentMask(X,L,M)$
7:  $(X', L') \gets ([d] + X, [O] + L)$
8:  $\theta \gets BART((\tilde{X},L),(X',L'))$
9: end for
10: Return  $\theta$
$(x_{< t}, l_{< t})$  and the encoder output  $H$  as inputs to get the possibility of the next token and the next token label with two separate linear layers:
$$
P \left(x _ {t} \mid x _ {<   t}, l _ {<   t}, H\right) = \operatorname {S o f t m a x} \left(W _ {x} z _ {t} + b _ {x}\right), \tag {6}
$$
$$
P \left(l _ {t} \mid l _ {<   t}, x _ {<   t}, H\right) = \operatorname {S o f t m a x} \left(W _ {l} z _ {t} + b _ {l}\right), \tag {7}
$$
where  $W_{x} \in R^{|V_{x}| \times d}$ ,  $W_{l} \in R^{|V_{l}| \times d}$ , and  $|V_{x}|$  and  $|V_{l}|$  refer to the dictionary size and the number of label types, respectively. The hidden layer vector  $z_{t}$  for the time step  $t$  is as follows:
$$
z _ {t} = \operatorname {B a r t D e c o d e r} \left(E _ {t}\right), \tag {8}
$$
$$
E _ {t} = \operatorname {T o k e n E m b} \left(x _ {t - 1}\right) + \operatorname {L a b e l E m b} \left(l _ {t - 1}\right). \tag {9}
$$
For each sample, we calculate the negative log-likelihood loss for tokens and the label sequence, respectively:
$$
\mathcal {L} _ {x} = - \sum_ {t = 1} ^ {n + 1} \log \left(P \left(x _ {t} \mid x _ {<   t}, l _ {<   t}, H\right)\right), \tag {10}
$$
$$
\mathcal {L} _ {l} = - \sum_ {t = 1} ^ {n + 1} \log \left(P \left(l _ {t} \mid l _ {<   t}, x _ {<   t}, H\right)\right). \tag {11}
$$
The final training loss consists of the addition of two parts:
$$
\mathcal {L} = \mathcal {L} _ {x} + \mathcal {L} _ {l}. \tag {12}
$$
Algorithm 1 details the training procedure of our method.
4.3 Cross-Domain Data Generation
During the inference stage, given a source-domain labeled sentence  $(X,L)\in D_S$ , we employ the same mask strategy to remove its domain-specific features. Then, the masked tuple  $(\tilde{X},L)$  is fed to the BART encoder. Different from the decoder in
Dataset	Domain	Sentence	Train	Test
R	Restaurant	5841	4381	1460
L	Laptop	3845	2884	961
D	Device	3836	2877	959
Table 1: The statistics of our datasets.
the training phase, we only provide  $([target], O)$  as the domain prompt to decode a target-domain sentence based on the auto-regressive way and jointly predict its token-level labels. To generate more samples and further increase the diversity of generated samples, we introduce several strategies as follows. For each sample  $(X, L) \in D_S$ , we repeat the mask step three times to get three masked tuple  $(\tilde{X}, L)$  with different seeds. Specifically for the matched segments without any aspects or opinions, we randomly mask them with a probability of  $60\%$ . Thus, we can obtain three different target reviews from one source labeled review.
4.4 Post-Processing
We post-process our generated target labeled data with the following steps: 1) Delete sentences with incorrect labels that do not follow the BIO schema; 2) Use a base classifier trained on the labeled source-domain data to assign labels on the generated target-domain data. Delete sentences whose assigned tags are inconsistent with generated ones.
4.5 Training for the Main Task
Finally, we can obtain the high-quality target-domain data with fine-grained annotation generated from our approach. We then use these generated samples for the main task of aspect and opinion co-extraction. Similar to Section 4.1, we employ a pre-trained BERT model (Devlin et al., 2019) with a CRF layer as the main task classifier, and train it on the generated data only. Based on the trained classifier, we evaluate its performance on the test set of the target domain.
We use three benchmark datasets from different domains, namely Restaurant (R), Laptop (L) and Device (D). R and L are two combination datasets from SemEval 2014 and 2015 (Pontiki et al., 2014, 2015). D consists of reviews from digital device collected by (Hu and Liu, 2004). Following previous works (Wang and Pan, 2018; Pereg et al., 2020; Chen and Qian, 2021), we use three different seeds
to split the dataset from each domain into training set and testing set by a ratio of 3:1. The statistics are shown in Table 1. We report the average results on different splits.
5.2 Experimental Setting
5.2.1 Cross-Domain Data Augmentation
For domain-specific segment mask module, we set the length of  $n$ -gram segment  $w$  to  $n \in [1, 4]$  and the relative frequency threshold  $\delta$  to 10. A pretrained sequence-to-sequence model BART is finetuned on  $D_{S} \cup D_{TP}$  for 3 training epochs with batch size set to 8. We use Adam as the optimizer with a learning rate of 5e-5. The fine-tuned BART is used to generate triple target-domain labeled data conditioned on the source-domain labeled data.
5.2.2 Aspect and Opinion Co-Extraction
As mentioned in Section 3, we formulate aspect and opinion co-extraction as a sequence labeling task. We use  $\mathrm{BERT}_{\mathrm{E}} + \mathrm{CRF}$  as the base classifier to assign pseudo labels in Section 4.1 and Section 4.4, which consists of the uncased  $\mathrm{BERT}_{\mathrm{base}}$  (Devlin et al., 2019) model post-trained on a combined data of E-commerce reviews from the Amazon Electronics dataset and the Yelp Challenge (Xu et al., 2019) and a CRF layer (Lafferty et al., 2001). For the main task, we also use the  $\mathrm{BERT}_{\mathrm{E}} + \mathrm{CRF}$  classifier and train it on our generated data. We use the Adam optimizer fro both BERT and CRF with different learning rates of 3e-5 and 0.02, respectively. Moreover, we employ the early stopping strategy, and stop the training procedure of the  $\mathrm{BERT}_{\mathrm{E}} + \mathrm{CRF}$  classifier after 2 epochs in both pseudo label annotation and main task training stages. We finally report the average Micro-F1 of three different splits for aspect and opinion co-extraction.
We divide all comparison systems into four parts:
Part 1 denotes hand-built feature-based methods. CrossCRF (Jakob and Gurevych, 2010) uses a linear-chain CRF with some manual features including POS tags and dependency relations to bridge the domain gap. RAP (Li et al., 2012) proposes a Relational Adaptive bootstraPping method to extract aspects and opinions based on source labeled data and pre-defined syntactic rules.
Part 2 denotes word2vec-based methods. HierJoint (Ding et al., 2017) is a cross-domain RNN model, which exploits auxiliary labels consisting of aspect terms extracted by manually designed
syntactic rules and opinion seeds. RNSCN (Wang and Pan, 2018) and TRNN-GRU (Wang and Pan, 2020) integrate syntactic relations by constructing the dependency tree. TIMN (Wang and Pan, 2019) proposes a Transferable Interactive Memory Network that can learn shared representations across domains effectively. SemBridge (Chen and Qian, 2021) is a CNN-based model, which links the source and target domains with semantic bridges by retrieving transferable semantic prototypes based on syntactic roles.
Part 3 denotes pre-trained model-based methods.  $\mathrm{BERT_B + CRF}$  consists of the uncased  $\mathrm{BERT}_{\mathrm{base}}$  model and a CRF layer. SA-EXAL (Pereg et al., 2020) achieves domain adaptation by combining the pre-trained BERT model with a syntactic-aware attention mechanism.  $\mathrm{BERT_E + CRF}$  is the base classifier introduced in Section 5.2.2. Both the  $\mathrm{BERT_B + CRF}$  and  $\mathrm{BERT_E + CRF}$  are only trained on the labeled source-domain data. CDRG (Yu et al., 2021) is a data augmentation-based adaptation method, which generates labeled target-domain data by replacing source-specific aspects and opinions in source domain reviews with target-specific aspects and opinions. CDRG-Merge trains the  $\mathrm{BERT_E + CRF}$  classifier on the combination of labeled source-domain data and generated target-domain data for aspect and opinion co-extraction.