# Cross-Domain End-To-End Aspect-Based Sentiment Analysis with Domain-Dependent Embeddings

Authors: Yingjie Tian; Linrui Yang; Yunchuan Sun; Dalian. Liu
Journal: Complexity
 (2021)

---

## Abstract

With the development of sentiment analysis, studies have been gradually classified based on different researched candidates. Among them, aspect-based sentiment analysis plays an important role in subtle opinion mining for online reviews. It used to be treated as a group of pipeline tasks but has been proved to be analysed well in an end-to-end model recently. Due to less labelled resources, the need for cross-domain aspect-based sentiment analysis has started to get attention. However, challenges exist when seeking domain-invariant features and keeping domain-dependent features to achieve domain adaptation within a fine-grained task. This paper utilizes the domain-dependent embeddings and designs the model CD-E2EABSA to achieve cross-domain aspect-based sentiment analysis in an end-to-end fashion. The proposed model utilizes the domain-dependent embeddings with a multitask learning strategy to capture both domain-invariant and domain-dependent knowledge. Various experiments are conducted and show the effectiveness of all components on two public datasets. Also, it is also proved that as a cross-domain model, CD-E2EABSA can perform better than most of the in-domain ABSA methods.


---

## Body

## 1. Introduction

Sentiment analysis, as one of the most popular tools in the natural language processing (NLP) domain, has attracted enormous explorations [1]. Through the process of sentiment analysis, which is also named as opinion mining, the sentiment tendency of texts is extracted. Based on different research objects, the field of sentiment analysis can be classified into three levels, including document level, sentence level, and aspect level. Document-level and sentence-level sentiment analyses are based on the assumption that there is only one valuable topic in the document or sentence. However, when considering the practical opinion mining in user feedback related to products or services, one or more entities, the evaluation of which might be different, are mentioned in one sentence, which demands a more fine-grained analysis method. Aspect-level sentiment analysis, which focuses on mining and differentiating the sentiment polarity of various entities or aspects in one sentence, gradually ascends the stage [2]. This paper will focus on the aspect-based sentiment analysis and extracting the author’s feelings about certain entities or targets [3]. There might be one or more entities mentioned in one sentence. Aspect-based sentiment analysis (ABSA) was used to be treated as a task that composed of several steps or subtasks, including aspect extraction that consists of opinion target extraction (OTE) and aspect categories detection (ASD), as well as sentiment polarity clarification (SPC) [4]. Therefore, models were mainly designed, focusing on one or two subtasks. Methods that can identify entities where emotions are expressed in were designed for aspect term extraction (OTE) [5–8]. Moreover, some works aimed at distinguishing aspects with defined aspect categories [9, 10]. Furthermore, the research studies pay more attention on classifying known aspects’ sentiment polarity [11–13]. However, when considering the purpose of practicability, it is more convenient and user friendly to build an end-to-end model regarding aspect-level sentiment analysis as a whole task. But, the challenge exits when designing one model for two or three subtasks at the same time, especially when these tasks are usually solved diversely from the perspective of model designing. ASD and SPC are widely regarded as binary or multiclassification tasks, while OTE is solved with models designed for sequence-labelling tasks. The typical way of combining these subtasks was accomplishing them one by one with a pipeline solution, which resulted in a complex model structure. With the rise of deep learning, neural networks and pretrained frameworks have been designed and developed for NLP tasks including sentiment analysis [14, 15]. Neural networks were then designed for pipeline works that can achieve these subtasks in sequence [16]. Recently, End-to-End Aspect-Based Sentiment Analysis (E2E-ABSA) has also been proposed and become popular [17, 18]. Fine-grained sentiment analysis is considered as one sequence-labelling task with multiple tags representing both position information and sentiment polarity of targets, which achieves the complete task in an end-to-end fashion. Various measures have been taken to define the strategy and the set of labels for sequence labelling, including joint training [19], span-based extraction [20], and the unified model [17]. This paper will employ the strategy of unified model as a tagging scheme for ABSA, which only needs reviews as input and avoids the limitation of mined knowledge brought by defined aspects’ set.But, there are still issues for utilizing a deep-learning model in aspect-based sentiment analysis. First of all, although the volume of reviews has accumulated rapidly, thanks to the continuous development of computer technology, there is insufficient labelled data for machine learning. Moreover, contents of reviews have certain domain discrepancy, which leads to the challenge of cross-domain knowledge transfer. For instance, models which have trained based on comments for products such as mobile phones might get a bad result with comments for restaurants as the test data. In addition, from the perspective of seeking sentiment polarity towards aspects, same opinion words might bring opposite emotions towards aspects in different domains. As shown in Figure1, “long” time results in positive sentiment for a review of a restaurant, but different in laptop domain. As a consequence, transfer learning has been introduced to aspect-based sentiment analysis in order to achieve the knowledge transmission across fine-grained opinion mining in various fields. In the beginning, the analysis was also carried out based on a certain subtask model. Recently, transferable end-to-end aspect-based sentiment analysis [21] was also proposed combining the E2E-ABSA and selective transfer learning methods.Figure 1
Labeled aspects and their related opinion words in reviews of restaurant domain and laptop domain (“B-POS,” “I-POS,” and “E-POS” represent begin, inside, and end word of a positive aspect, respectively, and “S-POS” and “S-NEG” mean positive and negative aspect with a single word).This paper will follow the trend of cross-domain E2E-ABSA and design a new strategy by introducing Bert and domain-dependent embeddings. The proposed framework CD-E2EABSA contains the main network of the word embedding layer and the aspect-based sentiment analysis layer with a parameter generation module. Bert is utilized as the word embedding layer in the model, which learned contextual embeddings, and, rather than glove or word2vec, can better facilitate the following downstream task with specialized contextual information. Also, it can avoid the out-of-vocabulary (OOV), especially when reviews of one domain contain a considerable number of proper nouns, which used to result in the introduction of dictionaries and manual features. Besides, such a design ensures the recognition of each aspect with its sentiment polarity for two domains. Previous work of cross-domain ABSA mainly pursued mining domain-invariant features to minimize the distance of two domains and achieve transfer learning. A few attempted to take advantage of domain-dependent features for the task. But, their models were built with two or three subtasks [22], while CD-E2EABSA utilized it in a unified model inspired by a cross-domain NER model [23]. The main contributions of this paper are as follows:(1)
Model-based transfer learning is introduced to cross-domain end-to-end aspect-based sentiment analysis in this paper.(2)
In order to benefit the cross-domain ABSA without complex structure, Bert, as the pretrained embedding layer, is firstly introduced into transferable E2E-ABSA in this paper. Also, CD-ABSA makes use of domain-related knowledge as features to ensure the model’s cognition of different domains.(3)
In the experiments, components are shown effective, and the cross-domain model performance can finally be similar to that of the one-domain model in aspect-based sentiment analysis.


## 2. Related Work

### 2.1. Aspect-Based Sentiment Analysis

Aspect-based sentiment analysis is traditionally divided into three or two subtasks, in which rule-based [6], topic modeling [24], and traditional sequence labelling models (such as CRF) [25] were used for OTE and ACD, and classifiers [16] or neural networks with attentions [11, 12] were used for SPC. For instance, ILWAANet [26] extracted features from both data and lexicon by multiple attentions to support the more comprehensive capture of aspects.After the approval that using a more integrated solution can achieve a more effective result in fine-grained sentiment analysis [19], a unified model for target-based opinion mining was proposed in [17]. It promotes the development of ABSA and stimulates further research from different dimensions. First of all, Li et al. [18] combined the unified model with pretrained model Bert for E2E-ABSA. Furthermore, there were works to explore different labelling and modeling strategies. DOER [5] solves the E2E-ABSA by introducing the cross-shared unit to connect aspect term extraction and aspect sentiment classification and use the joint labels instead of collapsed labels. Also, Zeng et al. [27] trained a joint learning end-to-end neural network for ASD and SPC tasks, while Hu et al. [20] introduced the span-based labelling scheme to targeted opinion mining within an open-domain condition and argued the pipeline strategy can perform better. As for modeling strategies, Peng et al. [28] improved the unified model to accomplish the novel subtask they present named aspect sentiment triplet extraction (ASTE). Besides, other machine learning approaches have been employed to enrich the content of E2E-ABSA. Open-domain learning is also introduced to the unified model [29] recently. The main difference between CD-E2EABSA and the abovementioned models is the introduction of the domain adaptation and achieving cross-domain fine-grained sentiment analysis based on the development of end-to-end sentiment analysis.


### 2.2. Transfer Learning in ABSA

The incorporation of transfer learning and ABSA can result in the knowledge transfer among different subtasks of ABSA or cross-domain aspect-based sentiment analysis. Transfer learning is a mode that encourages the machine to autonomously acquire knowledge from data and apply it to new tasks with new data. According to Pan and Yang [30], the basic methods of transfer learning can be divided into 4 types, namely, sample-based, model-based, feature-based, and relation-based transfer learning. Among them, feature-based and model-based transfer learning are more often combined with fine-grained sentiment analysis. Also, Xu et al. [31] utilized Bert as the language model to learn domain knowledge and accomplish the ABSA task. Feature-based transfer learning refers to the method of transferring features in order to reduce the gap between the source and target domains and then feed features to classifiers. Hu et al. [22] proposed a model achieving the aspect-based sentiment analysis by focusing on distilling domain-invariant features. On the other hand, model-based transfer learning is based on the assumption that the data in the source domain and target domain can share some model parameters. In this field, in addition to directly using pretrained models [18, 32], researchers mainly tried to absorb task transfer to enhance the learning ability of the ABSA model. For instance, Li et al. [18] employed coarse-to-fine task transfer in order to leverage knowledge from the ASD with rich resources to OTE. Recently, using IKTN, Liang et al. [33] achieved the domain-specific and sentiment-related knowledge transfer among subtasks of aspect-level sentiment analysis. Also, Li et al [21] introduced selective adversarial learning (SAL) as the domain adaptation method to the unified model and achieved the cross-domain ABSA. However, most of them did not regard aspect-level sentiment analysis as an end-to-end problem solving, and they did not use contextual information in the feature generation to assist subsequent tasks. CD-E2EABSA will treat cross-domain ABSA as an end-to-end task and explore the model-based transfer learning that employs domain-specific knowledge learnt from the network.


### 2.3. Domain Embeddings for Sentiment Analysis

Most existing research studies focused on extracting domain-invariant features to assistant domain knowledge transfer in cross-domain ABSA [34]. Liang et al. [33] separated the domain-invariant embeddings from domain-dependent embeddings by using an adversarial multitask learning framework, while Hu et al. [22] used a simpler way to distilling domain-invariant features by regarding domain-independent and domain-dependent features as orthogonal information. On the other hand, some models focused on domain-dependent embeddings in multidomain sentence-level sentiment analysis, which is introduced to multidomain ABSA in this paper. For instance, domain-aware embeddings [35], which combine word embeddings and domain-dependent features, were generalized and enhance the efficiency of sentence-level sentiment analysis. Liu et al [36] introduced the domain embeddings for sentence-level multidomain sentiment analysis in order to address the issue of different sentiment words for different domains. The main difference between it and this paper is that CD-E2EABSA focused on aspect-level sentiment analysis and learned domain representation with a parameter generation network through training instead of using an attention mechanism. Furthermore, the multitask learning strategy is widely used when a model aims at extracting domain-related representations for sentiment analysis [36]. The multitask learning strategy can learn multiple related tasks at the same time and use sharing knowledge in the learning process to improve the performance and generalization ability of the model in each task. An interactive multitask learning network [37] has also been applied to the unified model for E2E-ABSA, which simultaneously trains for both aspect-level and document-level sentiment analysis. CD-E2EABSA will employ the multitask strategy to utilize domain-dependent embeddings in the cross-domain E2E-ABSA task.


## 2.1. Aspect-Based Sentiment Analysis

Aspect-based sentiment analysis is traditionally divided into three or two subtasks, in which rule-based [6], topic modeling [24], and traditional sequence labelling models (such as CRF) [25] were used for OTE and ACD, and classifiers [16] or neural networks with attentions [11, 12] were used for SPC. For instance, ILWAANet [26] extracted features from both data and lexicon by multiple attentions to support the more comprehensive capture of aspects.After the approval that using a more integrated solution can achieve a more effective result in fine-grained sentiment analysis [19], a unified model for target-based opinion mining was proposed in [17]. It promotes the development of ABSA and stimulates further research from different dimensions. First of all, Li et al. [18] combined the unified model with pretrained model Bert for E2E-ABSA. Furthermore, there were works to explore different labelling and modeling strategies. DOER [5] solves the E2E-ABSA by introducing the cross-shared unit to connect aspect term extraction and aspect sentiment classification and use the joint labels instead of collapsed labels. Also, Zeng et al. [27] trained a joint learning end-to-end neural network for ASD and SPC tasks, while Hu et al. [20] introduced the span-based labelling scheme to targeted opinion mining within an open-domain condition and argued the pipeline strategy can perform better. As for modeling strategies, Peng et al. [28] improved the unified model to accomplish the novel subtask they present named aspect sentiment triplet extraction (ASTE). Besides, other machine learning approaches have been employed to enrich the content of E2E-ABSA. Open-domain learning is also introduced to the unified model [29] recently. The main difference between CD-E2EABSA and the abovementioned models is the introduction of the domain adaptation and achieving cross-domain fine-grained sentiment analysis based on the development of end-to-end sentiment analysis.


## 2.2. Transfer Learning in ABSA

The incorporation of transfer learning and ABSA can result in the knowledge transfer among different subtasks of ABSA or cross-domain aspect-based sentiment analysis. Transfer learning is a mode that encourages the machine to autonomously acquire knowledge from data and apply it to new tasks with new data. According to Pan and Yang [30], the basic methods of transfer learning can be divided into 4 types, namely, sample-based, model-based, feature-based, and relation-based transfer learning. Among them, feature-based and model-based transfer learning are more often combined with fine-grained sentiment analysis. Also, Xu et al. [31] utilized Bert as the language model to learn domain knowledge and accomplish the ABSA task. Feature-based transfer learning refers to the method of transferring features in order to reduce the gap between the source and target domains and then feed features to classifiers. Hu et al. [22] proposed a model achieving the aspect-based sentiment analysis by focusing on distilling domain-invariant features. On the other hand, model-based transfer learning is based on the assumption that the data in the source domain and target domain can share some model parameters. In this field, in addition to directly using pretrained models [18, 32], researchers mainly tried to absorb task transfer to enhance the learning ability of the ABSA model. For instance, Li et al. [18] employed coarse-to-fine task transfer in order to leverage knowledge from the ASD with rich resources to OTE. Recently, using IKTN, Liang et al. [33] achieved the domain-specific and sentiment-related knowledge transfer among subtasks of aspect-level sentiment analysis. Also, Li et al [21] introduced selective adversarial learning (SAL) as the domain adaptation method to the unified model and achieved the cross-domain ABSA. However, most of them did not regard aspect-level sentiment analysis as an end-to-end problem solving, and they did not use contextual information in the feature generation to assist subsequent tasks. CD-E2EABSA will treat cross-domain ABSA as an end-to-end task and explore the model-based transfer learning that employs domain-specific knowledge learnt from the network.


## 2.3. Domain Embeddings for Sentiment Analysis

Most existing research studies focused on extracting domain-invariant features to assistant domain knowledge transfer in cross-domain ABSA [34]. Liang et al. [33] separated the domain-invariant embeddings from domain-dependent embeddings by using an adversarial multitask learning framework, while Hu et al. [22] used a simpler way to distilling domain-invariant features by regarding domain-independent and domain-dependent features as orthogonal information. On the other hand, some models focused on domain-dependent embeddings in multidomain sentence-level sentiment analysis, which is introduced to multidomain ABSA in this paper. For instance, domain-aware embeddings [35], which combine word embeddings and domain-dependent features, were generalized and enhance the efficiency of sentence-level sentiment analysis. Liu et al [36] introduced the domain embeddings for sentence-level multidomain sentiment analysis in order to address the issue of different sentiment words for different domains. The main difference between it and this paper is that CD-E2EABSA focused on aspect-level sentiment analysis and learned domain representation with a parameter generation network through training instead of using an attention mechanism. Furthermore, the multitask learning strategy is widely used when a model aims at extracting domain-related representations for sentiment analysis [36]. The multitask learning strategy can learn multiple related tasks at the same time and use sharing knowledge in the learning process to improve the performance and generalization ability of the model in each task. An interactive multitask learning network [37] has also been applied to the unified model for E2E-ABSA, which simultaneously trains for both aspect-level and document-level sentiment analysis. CD-E2EABSA will employ the multitask strategy to utilize domain-dependent embeddings in the cross-domain E2E-ABSA task.


## 3. Methodology

### 3.1. Problem Statement

This paper deals with the cross-domain end-to-end aspect-based sentiment analysis and treats it as a sequence-labelling task while solving the cross-domain aspect extraction and sentiment classification at the same time. The cross-domain end-to-end aspect-based sentiment analysis task is formulated mathematically as follows, givenSAn with n words from source domain A and TBm with m words from target domain B. In order to achieve recognizing aspect-level sentiment through the sequence labelling, a unified tagging scheme [17] is adopted for every word. Tags for every word contains {B-POS, B-NEG, B-NEU, I-POS, I-NEG, I-NEU, E-POS, E-NEG, E-NEU, S-POS, S-NEG, S-NEU, O}, where O represents other words and others are designed for aspects in the sentence. Among them, B-, I-, E-, and S- indicate beginning, inside, ending, and single words for one aspect, respectively, while POS, NEG, and NEU represent positive, negative, and neural polarity categories. The Figure 2 shows an example sentence and the unified tag of each word in the sentence from the laptop domain. Domain embeddings ED are then used to accomplish the cross-domain task. ED⊇ds,dt, where s and t represent the source domain and the target domain. The final goal of CD-E2EABSA is to get aspect-level sentiment labels for two domains’ texts with the help of domain embeddings in an end-to-end fashion.Figure 2
The sample tagging sentence in the laptop domain.


### 3.2. Methods

For the fine-grained sentiment analysis problem that is regarded as a sequence labelling task, this paper adopts a more advanced word vector generation layer and transforms the traditional BiLSTM-CRF network by introducing domain vectors to achieve cross-domain aspect-based sentiment analysis. This section will illustrate the structure of the model, which is named as CD-E2EABSA and shown in Figure3. The CD-E2EABSA mainly consists of a word embedding layer, shared-ABSA layer with a parameter generation network, and private CRF layer for each domain. Given an input sequence X, the network utilized Bert as the word embedding layer, which learns the contextual information and forms the embeddings for the following network. Embeddings are then fed into a Bi-LSTM layer with domain-dependent knowledge based on a parameter generation network. Finally, private CRFs are used for each domain and achieve the sequence labelling task for them. This section includes three sections depicting each layer of the model in turns and the multitask strategy we employ.Figure 3
The structure of CD-E2EABSA.


#### 3.2.1. Word Embedding Layer

The word embedding layer is the first layer in CD-E2EABSA that accepts input sequenceX and outputs embeddings for the following neural network, which is basic and essential as it vectorizes words from texts and makes the aspect-based sentiment analysis feasible. Most existing solutions for end-to-end ABSA used traditional word vector sets such as glove as model inputs, which are complicated to operate with data reprocessing for lexicon. Also, some words cannot be found in glove pretrained embeddings and need to be initialized immediately. It has been confirmed that when using Bert [14], generating token-level representations with sentences as input in the ABSA task can take contextual information into account and improve the effect of text mining models on the basis [18]. In this paper, Bert is also introduced as the word embedding layer and produces features that contain each tokens’ contextual information. L transformer layers are employed to refine the token-level features which combine token embeddings, position embeddings, and segment embeddings at first, and Hl is finally generated and transferred as contextualized representation to downstream tasks as equation (1). The word embedding layer is shared between source- and target-domain texts.(1)Hl=TransformerlHl−1.


#### 3.2.2. Cross-Domain ABSA Layer

Word embeddings are then fed into the cross-domain ABSA layer, where features are distilled and tags are acquired word by word for both domains. The Bidirectional Long Short-Term Memory network (Bi-LSTM) and Conditional Random Field (CRF) [38] are used as the main model structure of the cross-domain ABSA layer in CD-E2EABSA, among which Bi-LSTM can capture majority information from the past and future in one sequence and CRF can take the input sequence and output the target tag sequence. In detail, Bi-LSTM is introduced to accept word embeddings Hl from the word embedding layer and output hidden representations ht′ considering the sequence labelling formulation of E2E-ABSA task. Furthermore, traditional Bi-LSTM is transformed in CD-E2EABSA to combining with domain-dependent embeddings to realize the cross-domain task. Traditional hidden outputs ht of Bi-LSTM can be denoted as in equation (2) with bidirectional hidden states.(2)hft⟶=LSTMhft−1⟶,hl,θLSTM,hbt⟵=LSTMhbt+1⟵,hl,θLSTM,ht=hft⟶⊕hbt⟵,where hft⟶ and hbt⟵, respectively, represent the forward and backward hidden states and t represents the present moment. θLSTM indicates all parameters used in the Bi-LSTM network. Also, ⊕ represents concatenation.Following the work in [23], a parameter generation network is used to reform the Bi-LSTM by utilizing domain embeddings which can learn domain-dependent knowledge through training both domains. Parameters for Bi-LSTM is generated with learnt domain embeddings ds and dt. The resulting parameters θPGLSTM are calculated as follows:(3)θPGLSTM=W⊗ED,where W symbolizes the set of parameters for traditional Bi-LSTM. Also, ED∈RU represents the domain embeddings for each domain, where U means the embedding size for the feature and D contains source domain and target domain. ⊗ refers to tensor contraction. Domain embeddings are gradually trained during the process of training and test during the process of dev and test for each domain.After the generation of parameters, the module of Bi-LSTM is modified to accept the input of parameters for Bi-LSTM. Also, with the input for Bi-LSTM and parametersθPGLSTM, bidirectional hidden outputs ht′ of Bi-LSTM for one domain can finally be denoted as equation (4), where hft⟶′ and hbt⟵′, respectively, represent the forward and backward hidden states of Bi-LSTM, ht′ is finally calculated by concatenating hft⟶′ and hbt⟵′. By combining the parameter generation module and Bi-LSTM, CD-E2EABSA learns more contextual knowledge while completing the domain knowledge learning.(4)hft⟶′=LSTMhft−1⟶′,hl,θPGLSTM,hbt⟵′=LSTMhbt+1⟵′,hl,θPGLSTM,ht′=hft⟶⊕hbt⟵′.In addition, in order to enhance the results of the sequence labelling task, CRF is employed as the output layer for ABSA. CRF pursues the globally most probable tag sequence and is widely used as the output layer in models designed for sequence labelling. Two standard CRFs, including CRF (S) and CRF (T) are used for the source and target domains (Figure2), respectively. Under this setting, each CRF is trained for one domain and can handle the special case that the source domain and target domain have different label sets. Given are the hidden outputs of Bi-LSTM h′=h1′,…,hn′, and the label of the sequence is y=l1,…,ln. The probability of the label sequence of an input sequence X can be calculated as follows:(5)pY|X=exp∑i=0nwCRFli⋅hi′+bCRFli−1,li∑y′exp∑i=0nwCRFli⋅hi′+bCRFli−1,li,where wCRF and bCRF are parameters in CRF. y′ refers to an arbitrary label sequence. All possible tag sequences get through a softmax formulation to get the final likelihood pY|X. Also, Viterbi algorithm is used to find the highest scored label sequence as output.


#### 3.2.3. Multitask Learning Strategy

The multitask learning strategy is used to learn domain embeddings and assist the cross-domain ABSA task. The multitask learning strategy ensures the alternate use and learning of domain vectors, which is a prerequisite for realizing independent learning of domain knowledge and sharing of nondomain knowledge. As shown in detail in Algorithm1, mini batches of two domains’ training set take turns to be fed into the network CD-E2EABSA through the training process. The domain embedding ED, which contains es for the source domain and et for the target domain, and parameters of the network are firstly initialized randomly. Through training, each domain generates parameters with W and ED within Bi-LSTM of the cross-domain ABSA layer, and gradients are then computed for f=W,ED in equation (3). At the end of each training step, all the parameters are updated together by minimizing the loss. Also, the sentence-level negative log likelihood loss equation (6) is used for training one dataset DABSA=xn,ynn=1N.(6)LABSA=−1DABSA∑n=1Nlogpyn|xn.Algorithm 1:Multitask learning strategy: pseudocode for the multitask learning strategy in CD-E2EABSA.
Input: training dataParameters:W, ED, θCRFOutput: Target model(1)
While training steps not end do:(2)
Generating minibatches for each domain data:Bs, Bt(3)
θLsSTM ⟵ f(W, es) (Equation (3))(4)
∆W, ∆ds, ∆θCRFs  ⟵ train(Bs) (Equation (7))(5)
θLtSTM ⟵ f(W, et) (Equation (3))(6)
∆W, ∆dt, ∆θCRFt ⟵ train(Bt) (Equation (7))(7)
UpdateW, dD(8)
End WhileBut, in terms of the whole loss function for the training of CD-E2EABSA, both the source domain and target domain need to be considered. Also, the final loss function is denoted in equation (7), where λd indicates the weight of one domain and L2 regularization λ/2θ2 is added to reduce overfitting with parameters λ and θ, the parameter set of the model. Also, d∈S,T indicates the source and target domain in the loss function.(7)L=∑d∈S,TλdLABSAd+λ2θ2.


## 3.1. Problem Statement

This paper deals with the cross-domain end-to-end aspect-based sentiment analysis and treats it as a sequence-labelling task while solving the cross-domain aspect extraction and sentiment classification at the same time. The cross-domain end-to-end aspect-based sentiment analysis task is formulated mathematically as follows, givenSAn with n words from source domain A and TBm with m words from target domain B. In order to achieve recognizing aspect-level sentiment through the sequence labelling, a unified tagging scheme [17] is adopted for every word. Tags for every word contains {B-POS, B-NEG, B-NEU, I-POS, I-NEG, I-NEU, E-POS, E-NEG, E-NEU, S-POS, S-NEG, S-NEU, O}, where O represents other words and others are designed for aspects in the sentence. Among them, B-, I-, E-, and S- indicate beginning, inside, ending, and single words for one aspect, respectively, while POS, NEG, and NEU represent positive, negative, and neural polarity categories. The Figure 2 shows an example sentence and the unified tag of each word in the sentence from the laptop domain. Domain embeddings ED are then used to accomplish the cross-domain task. ED⊇ds,dt, where s and t represent the source domain and the target domain. The final goal of CD-E2EABSA is to get aspect-level sentiment labels for two domains’ texts with the help of domain embeddings in an end-to-end fashion.Figure 2
The sample tagging sentence in the laptop domain.


## 3.2. Methods

For the fine-grained sentiment analysis problem that is regarded as a sequence labelling task, this paper adopts a more advanced word vector generation layer and transforms the traditional BiLSTM-CRF network by introducing domain vectors to achieve cross-domain aspect-based sentiment analysis. This section will illustrate the structure of the model, which is named as CD-E2EABSA and shown in Figure3. The CD-E2EABSA mainly consists of a word embedding layer, shared-ABSA layer with a parameter generation network, and private CRF layer for each domain. Given an input sequence X, the network utilized Bert as the word embedding layer, which learns the contextual information and forms the embeddings for the following network. Embeddings are then fed into a Bi-LSTM layer with domain-dependent knowledge based on a parameter generation network. Finally, private CRFs are used for each domain and achieve the sequence labelling task for them. This section includes three sections depicting each layer of the model in turns and the multitask strategy we employ.Figure 3
The structure of CD-E2EABSA.


### 3.2.1. Word Embedding Layer

The word embedding layer is the first layer in CD-E2EABSA that accepts input sequenceX and outputs embeddings for the following neural network, which is basic and essential as it vectorizes words from texts and makes the aspect-based sentiment analysis feasible. Most existing solutions for end-to-end ABSA used traditional word vector sets such as glove as model inputs, which are complicated to operate with data reprocessing for lexicon. Also, some words cannot be found in glove pretrained embeddings and need to be initialized immediately. It has been confirmed that when using Bert [14], generating token-level representations with sentences as input in the ABSA task can take contextual information into account and improve the effect of text mining models on the basis [18]. In this paper, Bert is also introduced as the word embedding layer and produces features that contain each tokens’ contextual information. L transformer layers are employed to refine the token-level features which combine token embeddings, position embeddings, and segment embeddings at first, and Hl is finally generated and transferred as contextualized representation to downstream tasks as equation (1). The word embedding layer is shared between source- and target-domain texts.(1)Hl=TransformerlHl−1.


### 3.2.2. Cross-Domain ABSA Layer

Word embeddings are then fed into the cross-domain ABSA layer, where features are distilled and tags are acquired word by word for both domains. The Bidirectional Long Short-Term Memory network (Bi-LSTM) and Conditional Random Field (CRF) [38] are used as the main model structure of the cross-domain ABSA layer in CD-E2EABSA, among which Bi-LSTM can capture majority information from the past and future in one sequence and CRF can take the input sequence and output the target tag sequence. In detail, Bi-LSTM is introduced to accept word embeddings Hl from the word embedding layer and output hidden representations ht′ considering the sequence labelling formulation of E2E-ABSA task. Furthermore, traditional Bi-LSTM is transformed in CD-E2EABSA to combining with domain-dependent embeddings to realize the cross-domain task. Traditional hidden outputs ht of Bi-LSTM can be denoted as in equation (2) with bidirectional hidden states.(2)hft⟶=LSTMhft−1⟶,hl,θLSTM,hbt⟵=LSTMhbt+1⟵,hl,θLSTM,ht=hft⟶⊕hbt⟵,where hft⟶ and hbt⟵, respectively, represent the forward and backward hidden states and t represents the present moment. θLSTM indicates all parameters used in the Bi-LSTM network. Also, ⊕ represents concatenation.Following the work in [23], a parameter generation network is used to reform the Bi-LSTM by utilizing domain embeddings which can learn domain-dependent knowledge through training both domains. Parameters for Bi-LSTM is generated with learnt domain embeddings ds and dt. The resulting parameters θPGLSTM are calculated as follows:(3)θPGLSTM=W⊗ED,where W symbolizes the set of parameters for traditional Bi-LSTM. Also, ED∈RU represents the domain embeddings for each domain, where U means the embedding size for the feature and D contains source domain and target domain. ⊗ refers to tensor contraction. Domain embeddings are gradually trained during the process of training and test during the process of dev and test for each domain.After the generation of parameters, the module of Bi-LSTM is modified to accept the input of parameters for Bi-LSTM. Also, with the input for Bi-LSTM and parametersθPGLSTM, bidirectional hidden outputs ht′ of Bi-LSTM for one domain can finally be denoted as equation (4), where hft⟶′ and hbt⟵′, respectively, represent the forward and backward hidden states of Bi-LSTM, ht′ is finally calculated by concatenating hft⟶′ and hbt⟵′. By combining the parameter generation module and Bi-LSTM, CD-E2EABSA learns more contextual knowledge while completing the domain knowledge learning.(4)hft⟶′=LSTMhft−1⟶′,hl,θPGLSTM,hbt⟵′=LSTMhbt+1⟵′,hl,θPGLSTM,ht′=hft⟶⊕hbt⟵′.In addition, in order to enhance the results of the sequence labelling task, CRF is employed as the output layer for ABSA. CRF pursues the globally most probable tag sequence and is widely used as the output layer in models designed for sequence labelling. Two standard CRFs, including CRF (S) and CRF (T) are used for the source and target domains (Figure2), respectively. Under this setting, each CRF is trained for one domain and can handle the special case that the source domain and target domain have different label sets. Given are the hidden outputs of Bi-LSTM h′=h1′,…,hn′, and the label of the sequence is y=l1,…,ln. The probability of the label sequence of an input sequence X can be calculated as follows:(5)pY|X=exp∑i=0nwCRFli⋅hi′+bCRFli−1,li∑y′exp∑i=0nwCRFli⋅hi′+bCRFli−1,li,where wCRF and bCRF are parameters in CRF. y′ refers to an arbitrary label sequence. All possible tag sequences get through a softmax formulation to get the final likelihood pY|X. Also, Viterbi algorithm is used to find the highest scored label sequence as output.


### 3.2.3. Multitask Learning Strategy

The multitask learning strategy is used to learn domain embeddings and assist the cross-domain ABSA task. The multitask learning strategy ensures the alternate use and learning of domain vectors, which is a prerequisite for realizing independent learning of domain knowledge and sharing of nondomain knowledge. As shown in detail in Algorithm1, mini batches of two domains’ training set take turns to be fed into the network CD-E2EABSA through the training process. The domain embedding ED, which contains es for the source domain and et for the target domain, and parameters of the network are firstly initialized randomly. Through training, each domain generates parameters with W and ED within Bi-LSTM of the cross-domain ABSA layer, and gradients are then computed for f=W,ED in equation (3). At the end of each training step, all the parameters are updated together by minimizing the loss. Also, the sentence-level negative log likelihood loss equation (6) is used for training one dataset DABSA=xn,ynn=1N.(6)LABSA=−1DABSA∑n=1Nlogpyn|xn.Algorithm 1:Multitask learning strategy: pseudocode for the multitask learning strategy in CD-E2EABSA.
Input: training dataParameters:W, ED, θCRFOutput: Target model(1)
While training steps not end do:(2)
Generating minibatches for each domain data:Bs, Bt(3)
θLsSTM ⟵ f(W, es) (Equation (3))(4)
∆W, ∆ds, ∆θCRFs  ⟵ train(Bs) (Equation (7))(5)
θLtSTM ⟵ f(W, et) (Equation (3))(6)
∆W, ∆dt, ∆θCRFt ⟵ train(Bt) (Equation (7))(7)
UpdateW, dD(8)
End WhileBut, in terms of the whole loss function for the training of CD-E2EABSA, both the source domain and target domain need to be considered. Also, the final loss function is denoted in equation (7), where λd indicates the weight of one domain and L2 regularization λ/2θ2 is added to reduce overfitting with parameters λ and θ, the parameter set of the model. Also, d∈S,T indicates the source and target domain in the loss function.(7)L=∑d∈S,TλdLABSAd+λ2θ2.


## 3.2.1. Word Embedding Layer

The word embedding layer is the first layer in CD-E2EABSA that accepts input sequenceX and outputs embeddings for the following neural network, which is basic and essential as it vectorizes words from texts and makes the aspect-based sentiment analysis feasible. Most existing solutions for end-to-end ABSA used traditional word vector sets such as glove as model inputs, which are complicated to operate with data reprocessing for lexicon. Also, some words cannot be found in glove pretrained embeddings and need to be initialized immediately. It has been confirmed that when using Bert [14], generating token-level representations with sentences as input in the ABSA task can take contextual information into account and improve the effect of text mining models on the basis [18]. In this paper, Bert is also introduced as the word embedding layer and produces features that contain each tokens’ contextual information. L transformer layers are employed to refine the token-level features which combine token embeddings, position embeddings, and segment embeddings at first, and Hl is finally generated and transferred as contextualized representation to downstream tasks as equation (1). The word embedding layer is shared between source- and target-domain texts.(1)Hl=TransformerlHl−1.


## 3.2.2. Cross-Domain ABSA Layer

Word embeddings are then fed into the cross-domain ABSA layer, where features are distilled and tags are acquired word by word for both domains. The Bidirectional Long Short-Term Memory network (Bi-LSTM) and Conditional Random Field (CRF) [38] are used as the main model structure of the cross-domain ABSA layer in CD-E2EABSA, among which Bi-LSTM can capture majority information from the past and future in one sequence and CRF can take the input sequence and output the target tag sequence. In detail, Bi-LSTM is introduced to accept word embeddings Hl from the word embedding layer and output hidden representations ht′ considering the sequence labelling formulation of E2E-ABSA task. Furthermore, traditional Bi-LSTM is transformed in CD-E2EABSA to combining with domain-dependent embeddings to realize the cross-domain task. Traditional hidden outputs ht of Bi-LSTM can be denoted as in equation (2) with bidirectional hidden states.(2)hft⟶=LSTMhft−1⟶,hl,θLSTM,hbt⟵=LSTMhbt+1⟵,hl,θLSTM,ht=hft⟶⊕hbt⟵,where hft⟶ and hbt⟵, respectively, represent the forward and backward hidden states and t represents the present moment. θLSTM indicates all parameters used in the Bi-LSTM network. Also, ⊕ represents concatenation.Following the work in [23], a parameter generation network is used to reform the Bi-LSTM by utilizing domain embeddings which can learn domain-dependent knowledge through training both domains. Parameters for Bi-LSTM is generated with learnt domain embeddings ds and dt. The resulting parameters θPGLSTM are calculated as follows:(3)θPGLSTM=W⊗ED,where W symbolizes the set of parameters for traditional Bi-LSTM. Also, ED∈RU represents the domain embeddings for each domain, where U means the embedding size for the feature and D contains source domain and target domain. ⊗ refers to tensor contraction. Domain embeddings are gradually trained during the process of training and test during the process of dev and test for each domain.After the generation of parameters, the module of Bi-LSTM is modified to accept the input of parameters for Bi-LSTM. Also, with the input for Bi-LSTM and parametersθPGLSTM, bidirectional hidden outputs ht′ of Bi-LSTM for one domain can finally be denoted as equation (4), where hft⟶′ and hbt⟵′, respectively, represent the forward and backward hidden states of Bi-LSTM, ht′ is finally calculated by concatenating hft⟶′ and hbt⟵′. By combining the parameter generation module and Bi-LSTM, CD-E2EABSA learns more contextual knowledge while completing the domain knowledge learning.(4)hft⟶′=LSTMhft−1⟶′,hl,θPGLSTM,hbt⟵′=LSTMhbt+1⟵′,hl,θPGLSTM,ht′=hft⟶⊕hbt⟵′.In addition, in order to enhance the results of the sequence labelling task, CRF is employed as the output layer for ABSA. CRF pursues the globally most probable tag sequence and is widely used as the output layer in models designed for sequence labelling. Two standard CRFs, including CRF (S) and CRF (T) are used for the source and target domains (Figure2), respectively. Under this setting, each CRF is trained for one domain and can handle the special case that the source domain and target domain have different label sets. Given are the hidden outputs of Bi-LSTM h′=h1′,…,hn′, and the label of the sequence is y=l1,…,ln. The probability of the label sequence of an input sequence X can be calculated as follows:(5)pY|X=exp∑i=0nwCRFli⋅hi′+bCRFli−1,li∑y′exp∑i=0nwCRFli⋅hi′+bCRFli−1,li,where wCRF and bCRF are parameters in CRF. y′ refers to an arbitrary label sequence. All possible tag sequences get through a softmax formulation to get the final likelihood pY|X. Also, Viterbi algorithm is used to find the highest scored label sequence as output.


## 3.2.3. Multitask Learning Strategy

The multitask learning strategy is used to learn domain embeddings and assist the cross-domain ABSA task. The multitask learning strategy ensures the alternate use and learning of domain vectors, which is a prerequisite for realizing independent learning of domain knowledge and sharing of nondomain knowledge. As shown in detail in Algorithm1, mini batches of two domains’ training set take turns to be fed into the network CD-E2EABSA through the training process. The domain embedding ED, which contains es for the source domain and et for the target domain, and parameters of the network are firstly initialized randomly. Through training, each domain generates parameters with W and ED within Bi-LSTM of the cross-domain ABSA layer, and gradients are then computed for f=W,ED in equation (3). At the end of each training step, all the parameters are updated together by minimizing the loss. Also, the sentence-level negative log likelihood loss equation (6) is used for training one dataset DABSA=xn,ynn=1N.(6)LABSA=−1DABSA∑n=1Nlogpyn|xn.Algorithm 1:Multitask learning strategy: pseudocode for the multitask learning strategy in CD-E2EABSA.
Input: training dataParameters:W, ED, θCRFOutput: Target model(1)
While training steps not end do:(2)
Generating minibatches for each domain data:Bs, Bt(3)
θLsSTM ⟵ f(W, es) (Equation (3))(4)
∆W, ∆ds, ∆θCRFs  ⟵ train(Bs) (Equation (7))(5)
θLtSTM ⟵ f(W, et) (Equation (3))(6)
∆W, ∆dt, ∆θCRFt ⟵ train(Bt) (Equation (7))(7)
UpdateW, dD(8)
End WhileBut, in terms of the whole loss function for the training of CD-E2EABSA, both the source domain and target domain need to be considered. Also, the final loss function is denoted in equation (7), where λd indicates the weight of one domain and L2 regularization λ/2θ2 is added to reduce overfitting with parameters λ and θ, the parameter set of the model. Also, d∈S,T indicates the source and target domain in the loss function.(7)L=∑d∈S,TλdLABSAd+λ2θ2.


## 4. Experiments

In this section, the experiments that were conducted to show the performance of CD-E2EABSA and their results will be illustrated and analysed.


### 4.1. Datasets and Settings

Two widely used review datasets for laptops and restaurants are included in the experiments, the number of sentences and aspects for two datasets is shown in Table1. There are customers’ reviews related to restaurants and laptops, and both of them are from SemEval [39] and modified by Li et al. [17]. We utilize the two datasets as two cross-domain ABSA tasks, which are knowledge transferring from the restaurant dataset to the laptop dataset and from the laptop dataset to the restaurant dataset.Table 1
Statistics of datasets.
DatasetsTrainDevTestTotalRestaurantSentence27413043044245Aspect20412566342931LaptopSentence349038721586035Aspect389341322876593As for the model implementation, the transformers2.0 of Pytorch is chosen as the framework to generate the CD-E2EABSA. The pretrained “bert-base-uncased” model is utilized for the word embedding layer illustrated before, where the number of transformersL is 12 and the hidden size for each h is 768. The embedding size of domain-dependent features is set as 8 in the experiments for parameter generation. As for the cross-domain ABSA layer, single-layer Bi-LSTM is used, and the batch size is set as 8 to fit the GPU memory. Also, the learning rate is set as 2e − 5. The model is trained up to 1500 steps at first. After training 1000 steps, the model selection will be started for every 100 steps according to the microaveraged F1 score. Following these settings, bidirectional transfer learning among two datasets with different random seeds is recorded and the average results are reported.


### 4.2. Main Results

In this section, an ablation study, case study, and development study are used to present and discuss the performance of CD-E2EABSA with its components. Besides, a comparison study is conducted to show that CD-E2EABSA can be trained in two domains at the same time and has the same effect as the existing models which are trained in a single domain.


#### 4.2.1. Ablation Study

An ablation study is conducted to investigate the effectiveness of the main components we used in CD-E2EABSA. The main results are shown in Table2, and experiments are conducted on two cross-domain datasets. R-L and L-R indicate the knowledge transfer from the restaurant domain to the laptop domain and from the laptop domain to the restaurant domain, respectively. Dev and Tes represent the scores calculated from validation and test sets in the corresponding domain’s dataset. CD-E2EABSA is compared with four baselines, including CD-E2EABSA (glove), CD-E2E-ABSA (noPG), CD-E2EABSA (noCRF), and CD-E2EABSA (noPG + noCRF).Table 2
Results of models in two cross-domain ABSA tasks. Best scores are in bold.
ModelR-L (Dev)R-L (Tes)L-R (Dev)L-R (Tes)CD-E2EABSARest60.0462.2657.8260.18Laptop50.4953.1051.0852.51CD-E2EABSA (glove)Rest–53.44–53.60Laptop–40.56–42.13CD-E2EABSA (noPG)Rest51.1451.6952.8054.72Laptop47.1346.1147.1244.10CD-E2EABSA (noCRF)Rest53.5552.3733.4138.41Laptop37.5540.1543.6740.82CD-E2EABSA (noPG + noCRF)Rest40.5338.0940.3738.35Laptop22.5428.8417.0727.92CD-E2EABSA (glove) uses the 100-sized glove embeddings as the initialized word representations instead of using Bert as the embedding layer. Also, the following network contains the BiLSTM-CRF and parameter generation module, the same as the CD-E2EABSA.CD-E2EABSA (noPG) does not comprise the parameter generation module introduced before. Also, it uses Bert as the embedding layer, following by BiLSTM-CRF as the aspect-based sentiment layer. Although the multitask learning strategy is also employed, CD-E2EABSA (noPG) is trained without domain-dependent embeddings supporting the distinction between two domains.CD-E2EABSA (noCRF) does not have private CRF for each domain as the output layer. A softmax layer is followed to produce predictions after input sequence get through the word embedding layer and Bi-LSTM with a parameter generation module.CD-E2EABSA (noPG + noCRF) is also conducted, which has the word embedding layer and the ABSA layer including Bi-LSTM and a softmax layer.The main results of the ablation study are depicted in Table2, and CD-E2EABSA performs best in both cross-domain aspect-based sentiments. Micro-F1 scores are used to represents the performance of models. The comparison among all models indicates the effectiveness of all components in CD-E2EABSA. The worse result from CD-E2EABSA (glove) indicates that contextual information extracted by the word embedding layer can contribute to the result from the perspective of feature extraction in the model. Moreover, the result of CD-E2EABSA (noPG) shows that directly using the same LSTM parameters without domain-dependent vectors is not good, which implies that the parameter generation module can bring domain-related information to the network through training. Besides, it is shown that CRF, as a classical tool of sequence labelling, still contributes well in the E2E-ABSA task, as it seeks the best labelling sequence from global considerations.


#### 4.2.2. Case Study

In order to show the effect of the ablation study more intuitively, a case study is presented in this section to show the significance of CD-E2EABSA through tagging results. Sentences from the laptop-14 test dataset are used and presented in Figures4 and 5. Figure 4 shows the tags we get from those models, and Figure 5 transforms the tags to final recognition of aspects and their tagged sentiment polarity. It is obvious that CD-E2EABSA performs the best among them and recognizes both aspect and its sentiment polarity correctly. When there is no CRF, the model CD-E2EABSA (noCRF) cannot recognize the aspect in the sentence, nor its sentiment polarity. CD-E2EABSA can handle both issues with CRF. Besides, we found that domain embeddings can ensure the recognition of aspects’ boundary. CD-E2EABSA (noPG) fails to recognize the boundary of the aspect in the sentence. For instance, it treats “Tech support” as two aspects and tagged with diverse sentiment tags. Also, CD-E2EABSA (noPG + noCRF) can only identify a single word as an aspect, while CD-E2EABSA solves this with the parameter generation module and distinguishes more integrated aspects.Figure 4
Tagging examples output by different models.Figure 5
Performance of different models in the ABSA task.


#### 4.2.3. Development Study

A development study is conducted to further present the effectiveness of each component in CD-E2EABSA. First of all, it is found through the experiment that the OOV of CD-E2EABSA (glove) is higher than 0.14, which means there are 1279 of 8980 words that cannot be found in the file of glove embeddings and need to be initialized randomly at the start. How would it influence the efficiency of the model? After comparing the CD-E2EABSA and CD-E2EABSA (glove) in the ablation study, we explain this with Figure6, which shows the curve of micro-F1 scores for the test results of CD-E2EABSA and CD-E2EABSA (glove) in the cross-domain ABSA task from restaurant to laptop through 5-epoch training. CD-E2EABSA (S) and CD-E2EABSA (T) represent the results of CD-E2EABSA in the source (restaurant) domain and target (laptop) domain, respectively, while CD-E2EABSA (glove) (S) and CD-E2EABSA (Glove) (T) represent the results of CD-E2EABSA (glove) in two domains. It is obvious that the word embeddings produced by the word embedding layer in CD-E2EABSA work better than glove embeddings at the beginning and keep their superiority through the training process. This suggests that contextual knowledge learned in the word embedding layer for each token can generate more knowledge for the downstream tasks and improve model performance.Figure 6
The change of F1 scores using the word embedding layer and glove embeddings in CD-E2EABSA after 4 epochs. (i) CD-E2EABSA (S) uses Bert as the embedding layer for thw source domain; (ii) CD-E2EABSA (T) uses Bert as the embedding layer for the target domain; (iii) CD-E2EABSA (Glove) (S) uses glove embedding as input for the source domain; and (iv) CD-E2EABSA (Glove) (T) uses glove embedding as input for the target domain.After discussing the significance of the word embedding layer, we also analyse the parameter generator centred on the domain vector. Two domain-dependent embeddings are output and visualized. As it is shown in Figure7, the two 8-dimensional domain vectors learned though the training process of a cross-domain ABSA task are extracted and reduced to two-dimensional vectors through the t-SNE algorithm [40]. It is obvious that domain-dependent embeddings are totally different for the laptop and restaurant domain, suggesting CD-E2EABSA can learn and distil an embedding for every domain. They contain domain knowledge, which can help CD-E2EABSA to recognize texts from one domain and accomplish the cross-domain ABSA task from the perspective of parameters in Bi-LSTM. In the example in Figure 1, domain knowledge means long time represents different sentiment polarities in restaurant and laptop domains. Moreover, from Figure 5 of the ablation study, it is found that CD-E2EABSA can recognize boot time as a whole target in laptop reviews with the domain embedding.Figure 7
Final domain vectors for two domains.The impact of steps for training the model are also investigated. As it is demonstrated in Figure8, taking the result of CD-E2EABSA for cross-domain aspect-based sentiment analysis from the restaurant domain to laptop domain as an example, we found that convergence is still achieved after 1500 steps that we initialized before until 5000 steps and further steps do not provide considerable improvement.Figure 8
The comparison of different steps.


#### 4.2.4. Comparison Study

Comparison experiments are performed to compare CD-E2EABSA and existing models, the results in Table3 suggesting that CD-E2EABSA can perform nearly the same in two domains simultaneously as the models trained in one domain. We compare the proposed model with ATAE-LSTM [12], IAN [11], IMN [37], and UM [17]. Also, results are shown in Table 3 where F1-R and F1-L represent F1 scores of different models in the restaurant and laptop dataset from [39]. The proposed method is firstly compared with two LSTM-based ABSA models designed for the SPC task, including ATAE-LSTM [12] and IAN [11]. They get input sequences and their aspects and combine the attention mechanism with the LSTM network to identify aspect-level sentiment in the restaurant domain. CD-E2EABSA does not need the aspects as input and achieves the end-to-end ABSA task and performs better than them. CD-E2EABSA is also compared with IMN, an interactive multitask E2E-ABSA model which utilized document-level knowledge for one-domain aspect-based sentiment analysis. It is shown that IMN was conducted in a divided restaurant-domain dataset [37], while CD-E2EABSA is trained on the combination of both datasets. We found that, after 5000 steps of training, the F1 score for the targeted restaurant domain can exceed one of the performances of IMN, which is designed for document-aspect knowledge transfer in one domain. Although CD-E2EABSA cannot get better F1 score than IMN, it is essential to point out that IMN is trained in one domain for one time while CD-E2EABSA is trained with two domains’ datasets at the same time. Moreover, from the perspective of the dataset, IMN uses the document-level dataset additionally to assist the extraction of fine-grained emotions. Finally, the proposed method is compared with UM by Li et al. [17], who introduced the unified tagging scheme for a single-domain E2E-ABSA task. It is depicted that CD-E2EABSA can be trained in both domains and, at the same time, performs better than UM which is trained in a single domain. In other words, CD-E2EABSA can get a better result by training once, while UM needs to be trained twice.Table 3
The comparison results among different methods. Best scores are in bold, and second-best scores are underlined.
F1-RF1-LATAE-LSTM66.3663.24IAN67.7163.72IMN (Rest-14)75.6670.76IMN (Rest-15)71.76–UM69.8057.90CD-E2EABSA (L-R_5000)73.9663.85


## 4.1. Datasets and Settings

Two widely used review datasets for laptops and restaurants are included in the experiments, the number of sentences and aspects for two datasets is shown in Table1. There are customers’ reviews related to restaurants and laptops, and both of them are from SemEval [39] and modified by Li et al. [17]. We utilize the two datasets as two cross-domain ABSA tasks, which are knowledge transferring from the restaurant dataset to the laptop dataset and from the laptop dataset to the restaurant dataset.Table 1
Statistics of datasets.
DatasetsTrainDevTestTotalRestaurantSentence27413043044245Aspect20412566342931LaptopSentence349038721586035Aspect389341322876593As for the model implementation, the transformers2.0 of Pytorch is chosen as the framework to generate the CD-E2EABSA. The pretrained “bert-base-uncased” model is utilized for the word embedding layer illustrated before, where the number of transformersL is 12 and the hidden size for each h is 768. The embedding size of domain-dependent features is set as 8 in the experiments for parameter generation. As for the cross-domain ABSA layer, single-layer Bi-LSTM is used, and the batch size is set as 8 to fit the GPU memory. Also, the learning rate is set as 2e − 5. The model is trained up to 1500 steps at first. After training 1000 steps, the model selection will be started for every 100 steps according to the microaveraged F1 score. Following these settings, bidirectional transfer learning among two datasets with different random seeds is recorded and the average results are reported.


## 4.2. Main Results

In this section, an ablation study, case study, and development study are used to present and discuss the performance of CD-E2EABSA with its components. Besides, a comparison study is conducted to show that CD-E2EABSA can be trained in two domains at the same time and has the same effect as the existing models which are trained in a single domain.


### 4.2.1. Ablation Study

An ablation study is conducted to investigate the effectiveness of the main components we used in CD-E2EABSA. The main results are shown in Table2, and experiments are conducted on two cross-domain datasets. R-L and L-R indicate the knowledge transfer from the restaurant domain to the laptop domain and from the laptop domain to the restaurant domain, respectively. Dev and Tes represent the scores calculated from validation and test sets in the corresponding domain’s dataset. CD-E2EABSA is compared with four baselines, including CD-E2EABSA (glove), CD-E2E-ABSA (noPG), CD-E2EABSA (noCRF), and CD-E2EABSA (noPG + noCRF).Table 2
Results of models in two cross-domain ABSA tasks. Best scores are in bold.
ModelR-L (Dev)R-L (Tes)L-R (Dev)L-R (Tes)CD-E2EABSARest60.0462.2657.8260.18Laptop50.4953.1051.0852.51CD-E2EABSA (glove)Rest–53.44–53.60Laptop–40.56–42.13CD-E2EABSA (noPG)Rest51.1451.6952.8054.72Laptop47.1346.1147.1244.10CD-E2EABSA (noCRF)Rest53.5552.3733.4138.41Laptop37.5540.1543.6740.82CD-E2EABSA (noPG + noCRF)Rest40.5338.0940.3738.35Laptop22.5428.8417.0727.92CD-E2EABSA (glove) uses the 100-sized glove embeddings as the initialized word representations instead of using Bert as the embedding layer. Also, the following network contains the BiLSTM-CRF and parameter generation module, the same as the CD-E2EABSA.CD-E2EABSA (noPG) does not comprise the parameter generation module introduced before. Also, it uses Bert as the embedding layer, following by BiLSTM-CRF as the aspect-based sentiment layer. Although the multitask learning strategy is also employed, CD-E2EABSA (noPG) is trained without domain-dependent embeddings supporting the distinction between two domains.CD-E2EABSA (noCRF) does not have private CRF for each domain as the output layer. A softmax layer is followed to produce predictions after input sequence get through the word embedding layer and Bi-LSTM with a parameter generation module.CD-E2EABSA (noPG + noCRF) is also conducted, which has the word embedding layer and the ABSA layer including Bi-LSTM and a softmax layer.The main results of the ablation study are depicted in Table2, and CD-E2EABSA performs best in both cross-domain aspect-based sentiments. Micro-F1 scores are used to represents the performance of models. The comparison among all models indicates the effectiveness of all components in CD-E2EABSA. The worse result from CD-E2EABSA (glove) indicates that contextual information extracted by the word embedding layer can contribute to the result from the perspective of feature extraction in the model. Moreover, the result of CD-E2EABSA (noPG) shows that directly using the same LSTM parameters without domain-dependent vectors is not good, which implies that the parameter generation module can bring domain-related information to the network through training. Besides, it is shown that CRF, as a classical tool of sequence labelling, still contributes well in the E2E-ABSA task, as it seeks the best labelling sequence from global considerations.


### 4.2.2. Case Study

In order to show the effect of the ablation study more intuitively, a case study is presented in this section to show the significance of CD-E2EABSA through tagging results. Sentences from the laptop-14 test dataset are used and presented in Figures4 and 5. Figure 4 shows the tags we get from those models, and Figure 5 transforms the tags to final recognition of aspects and their tagged sentiment polarity. It is obvious that CD-E2EABSA performs the best among them and recognizes both aspect and its sentiment polarity correctly. When there is no CRF, the model CD-E2EABSA (noCRF) cannot recognize the aspect in the sentence, nor its sentiment polarity. CD-E2EABSA can handle both issues with CRF. Besides, we found that domain embeddings can ensure the recognition of aspects’ boundary. CD-E2EABSA (noPG) fails to recognize the boundary of the aspect in the sentence. For instance, it treats “Tech support” as two aspects and tagged with diverse sentiment tags. Also, CD-E2EABSA (noPG + noCRF) can only identify a single word as an aspect, while CD-E2EABSA solves this with the parameter generation module and distinguishes more integrated aspects.Figure 4
Tagging examples output by different models.Figure 5
Performance of different models in the ABSA task.


### 4.2.3. Development Study

A development study is conducted to further present the effectiveness of each component in CD-E2EABSA. First of all, it is found through the experiment that the OOV of CD-E2EABSA (glove) is higher than 0.14, which means there are 1279 of 8980 words that cannot be found in the file of glove embeddings and need to be initialized randomly at the start. How would it influence the efficiency of the model? After comparing the CD-E2EABSA and CD-E2EABSA (glove) in the ablation study, we explain this with Figure6, which shows the curve of micro-F1 scores for the test results of CD-E2EABSA and CD-E2EABSA (glove) in the cross-domain ABSA task from restaurant to laptop through 5-epoch training. CD-E2EABSA (S) and CD-E2EABSA (T) represent the results of CD-E2EABSA in the source (restaurant) domain and target (laptop) domain, respectively, while CD-E2EABSA (glove) (S) and CD-E2EABSA (Glove) (T) represent the results of CD-E2EABSA (glove) in two domains. It is obvious that the word embeddings produced by the word embedding layer in CD-E2EABSA work better than glove embeddings at the beginning and keep their superiority through the training process. This suggests that contextual knowledge learned in the word embedding layer for each token can generate more knowledge for the downstream tasks and improve model performance.Figure 6
The change of F1 scores using the word embedding layer and glove embeddings in CD-E2EABSA after 4 epochs. (i) CD-E2EABSA (S) uses Bert as the embedding layer for thw source domain; (ii) CD-E2EABSA (T) uses Bert as the embedding layer for the target domain; (iii) CD-E2EABSA (Glove) (S) uses glove embedding as input for the source domain; and (iv) CD-E2EABSA (Glove) (T) uses glove embedding as input for the target domain.After discussing the significance of the word embedding layer, we also analyse the parameter generator centred on the domain vector. Two domain-dependent embeddings are output and visualized. As it is shown in Figure7, the two 8-dimensional domain vectors learned though the training process of a cross-domain ABSA task are extracted and reduced to two-dimensional vectors through the t-SNE algorithm [40]. It is obvious that domain-dependent embeddings are totally different for the laptop and restaurant domain, suggesting CD-E2EABSA can learn and distil an embedding for every domain. They contain domain knowledge, which can help CD-E2EABSA to recognize texts from one domain and accomplish the cross-domain ABSA task from the perspective of parameters in Bi-LSTM. In the example in Figure 1, domain knowledge means long time represents different sentiment polarities in restaurant and laptop domains. Moreover, from Figure 5 of the ablation study, it is found that CD-E2EABSA can recognize boot time as a whole target in laptop reviews with the domain embedding.Figure 7
Final domain vectors for two domains.The impact of steps for training the model are also investigated. As it is demonstrated in Figure8, taking the result of CD-E2EABSA for cross-domain aspect-based sentiment analysis from the restaurant domain to laptop domain as an example, we found that convergence is still achieved after 1500 steps that we initialized before until 5000 steps and further steps do not provide considerable improvement.Figure 8
The comparison of different steps.


### 4.2.4. Comparison Study

Comparison experiments are performed to compare CD-E2EABSA and existing models, the results in Table3 suggesting that CD-E2EABSA can perform nearly the same in two domains simultaneously as the models trained in one domain. We compare the proposed model with ATAE-LSTM [12], IAN [11], IMN [37], and UM [17]. Also, results are shown in Table 3 where F1-R and F1-L represent F1 scores of different models in the restaurant and laptop dataset from [39]. The proposed method is firstly compared with two LSTM-based ABSA models designed for the SPC task, including ATAE-LSTM [12] and IAN [11]. They get input sequences and their aspects and combine the attention mechanism with the LSTM network to identify aspect-level sentiment in the restaurant domain. CD-E2EABSA does not need the aspects as input and achieves the end-to-end ABSA task and performs better than them. CD-E2EABSA is also compared with IMN, an interactive multitask E2E-ABSA model which utilized document-level knowledge for one-domain aspect-based sentiment analysis. It is shown that IMN was conducted in a divided restaurant-domain dataset [37], while CD-E2EABSA is trained on the combination of both datasets. We found that, after 5000 steps of training, the F1 score for the targeted restaurant domain can exceed one of the performances of IMN, which is designed for document-aspect knowledge transfer in one domain. Although CD-E2EABSA cannot get better F1 score than IMN, it is essential to point out that IMN is trained in one domain for one time while CD-E2EABSA is trained with two domains’ datasets at the same time. Moreover, from the perspective of the dataset, IMN uses the document-level dataset additionally to assist the extraction of fine-grained emotions. Finally, the proposed method is compared with UM by Li et al. [17], who introduced the unified tagging scheme for a single-domain E2E-ABSA task. It is depicted that CD-E2EABSA can be trained in both domains and, at the same time, performs better than UM which is trained in a single domain. In other words, CD-E2EABSA can get a better result by training once, while UM needs to be trained twice.Table 3
The comparison results among different methods. Best scores are in bold, and second-best scores are underlined.
F1-RF1-LATAE-LSTM66.3663.24IAN67.7163.72IMN (Rest-14)75.6670.76IMN (Rest-15)71.76–UM69.8057.90CD-E2EABSA (L-R_5000)73.9663.85


## 4.2.1. Ablation Study

An ablation study is conducted to investigate the effectiveness of the main components we used in CD-E2EABSA. The main results are shown in Table2, and experiments are conducted on two cross-domain datasets. R-L and L-R indicate the knowledge transfer from the restaurant domain to the laptop domain and from the laptop domain to the restaurant domain, respectively. Dev and Tes represent the scores calculated from validation and test sets in the corresponding domain’s dataset. CD-E2EABSA is compared with four baselines, including CD-E2EABSA (glove), CD-E2E-ABSA (noPG), CD-E2EABSA (noCRF), and CD-E2EABSA (noPG + noCRF).Table 2
Results of models in two cross-domain ABSA tasks. Best scores are in bold.
ModelR-L (Dev)R-L (Tes)L-R (Dev)L-R (Tes)CD-E2EABSARest60.0462.2657.8260.18Laptop50.4953.1051.0852.51CD-E2EABSA (glove)Rest–53.44–53.60Laptop–40.56–42.13CD-E2EABSA (noPG)Rest51.1451.6952.8054.72Laptop47.1346.1147.1244.10CD-E2EABSA (noCRF)Rest53.5552.3733.4138.41Laptop37.5540.1543.6740.82CD-E2EABSA (noPG + noCRF)Rest40.5338.0940.3738.35Laptop22.5428.8417.0727.92CD-E2EABSA (glove) uses the 100-sized glove embeddings as the initialized word representations instead of using Bert as the embedding layer. Also, the following network contains the BiLSTM-CRF and parameter generation module, the same as the CD-E2EABSA.CD-E2EABSA (noPG) does not comprise the parameter generation module introduced before. Also, it uses Bert as the embedding layer, following by BiLSTM-CRF as the aspect-based sentiment layer. Although the multitask learning strategy is also employed, CD-E2EABSA (noPG) is trained without domain-dependent embeddings supporting the distinction between two domains.CD-E2EABSA (noCRF) does not have private CRF for each domain as the output layer. A softmax layer is followed to produce predictions after input sequence get through the word embedding layer and Bi-LSTM with a parameter generation module.CD-E2EABSA (noPG + noCRF) is also conducted, which has the word embedding layer and the ABSA layer including Bi-LSTM and a softmax layer.The main results of the ablation study are depicted in Table2, and CD-E2EABSA performs best in both cross-domain aspect-based sentiments. Micro-F1 scores are used to represents the performance of models. The comparison among all models indicates the effectiveness of all components in CD-E2EABSA. The worse result from CD-E2EABSA (glove) indicates that contextual information extracted by the word embedding layer can contribute to the result from the perspective of feature extraction in the model. Moreover, the result of CD-E2EABSA (noPG) shows that directly using the same LSTM parameters without domain-dependent vectors is not good, which implies that the parameter generation module can bring domain-related information to the network through training. Besides, it is shown that CRF, as a classical tool of sequence labelling, still contributes well in the E2E-ABSA task, as it seeks the best labelling sequence from global considerations.


## 4.2.2. Case Study

In order to show the effect of the ablation study more intuitively, a case study is presented in this section to show the significance of CD-E2EABSA through tagging results. Sentences from the laptop-14 test dataset are used and presented in Figures4 and 5. Figure 4 shows the tags we get from those models, and Figure 5 transforms the tags to final recognition of aspects and their tagged sentiment polarity. It is obvious that CD-E2EABSA performs the best among them and recognizes both aspect and its sentiment polarity correctly. When there is no CRF, the model CD-E2EABSA (noCRF) cannot recognize the aspect in the sentence, nor its sentiment polarity. CD-E2EABSA can handle both issues with CRF. Besides, we found that domain embeddings can ensure the recognition of aspects’ boundary. CD-E2EABSA (noPG) fails to recognize the boundary of the aspect in the sentence. For instance, it treats “Tech support” as two aspects and tagged with diverse sentiment tags. Also, CD-E2EABSA (noPG + noCRF) can only identify a single word as an aspect, while CD-E2EABSA solves this with the parameter generation module and distinguishes more integrated aspects.Figure 4
Tagging examples output by different models.Figure 5
Performance of different models in the ABSA task.


## 4.2.3. Development Study

A development study is conducted to further present the effectiveness of each component in CD-E2EABSA. First of all, it is found through the experiment that the OOV of CD-E2EABSA (glove) is higher than 0.14, which means there are 1279 of 8980 words that cannot be found in the file of glove embeddings and need to be initialized randomly at the start. How would it influence the efficiency of the model? After comparing the CD-E2EABSA and CD-E2EABSA (glove) in the ablation study, we explain this with Figure6, which shows the curve of micro-F1 scores for the test results of CD-E2EABSA and CD-E2EABSA (glove) in the cross-domain ABSA task from restaurant to laptop through 5-epoch training. CD-E2EABSA (S) and CD-E2EABSA (T) represent the results of CD-E2EABSA in the source (restaurant) domain and target (laptop) domain, respectively, while CD-E2EABSA (glove) (S) and CD-E2EABSA (Glove) (T) represent the results of CD-E2EABSA (glove) in two domains. It is obvious that the word embeddings produced by the word embedding layer in CD-E2EABSA work better than glove embeddings at the beginning and keep their superiority through the training process. This suggests that contextual knowledge learned in the word embedding layer for each token can generate more knowledge for the downstream tasks and improve model performance.Figure 6
The change of F1 scores using the word embedding layer and glove embeddings in CD-E2EABSA after 4 epochs. (i) CD-E2EABSA (S) uses Bert as the embedding layer for thw source domain; (ii) CD-E2EABSA (T) uses Bert as the embedding layer for the target domain; (iii) CD-E2EABSA (Glove) (S) uses glove embedding as input for the source domain; and (iv) CD-E2EABSA (Glove) (T) uses glove embedding as input for the target domain.After discussing the significance of the word embedding layer, we also analyse the parameter generator centred on the domain vector. Two domain-dependent embeddings are output and visualized. As it is shown in Figure7, the two 8-dimensional domain vectors learned though the training process of a cross-domain ABSA task are extracted and reduced to two-dimensional vectors through the t-SNE algorithm [40]. It is obvious that domain-dependent embeddings are totally different for the laptop and restaurant domain, suggesting CD-E2EABSA can learn and distil an embedding for every domain. They contain domain knowledge, which can help CD-E2EABSA to recognize texts from one domain and accomplish the cross-domain ABSA task from the perspective of parameters in Bi-LSTM. In the example in Figure 1, domain knowledge means long time represents different sentiment polarities in restaurant and laptop domains. Moreover, from Figure 5 of the ablation study, it is found that CD-E2EABSA can recognize boot time as a whole target in laptop reviews with the domain embedding.Figure 7
Final domain vectors for two domains.The impact of steps for training the model are also investigated. As it is demonstrated in Figure8, taking the result of CD-E2EABSA for cross-domain aspect-based sentiment analysis from the restaurant domain to laptop domain as an example, we found that convergence is still achieved after 1500 steps that we initialized before until 5000 steps and further steps do not provide considerable improvement.Figure 8
The comparison of different steps.


## 4.2.4. Comparison Study

Comparison experiments are performed to compare CD-E2EABSA and existing models, the results in Table3 suggesting that CD-E2EABSA can perform nearly the same in two domains simultaneously as the models trained in one domain. We compare the proposed model with ATAE-LSTM [12], IAN [11], IMN [37], and UM [17]. Also, results are shown in Table 3 where F1-R and F1-L represent F1 scores of different models in the restaurant and laptop dataset from [39]. The proposed method is firstly compared with two LSTM-based ABSA models designed for the SPC task, including ATAE-LSTM [12] and IAN [11]. They get input sequences and their aspects and combine the attention mechanism with the LSTM network to identify aspect-level sentiment in the restaurant domain. CD-E2EABSA does not need the aspects as input and achieves the end-to-end ABSA task and performs better than them. CD-E2EABSA is also compared with IMN, an interactive multitask E2E-ABSA model which utilized document-level knowledge for one-domain aspect-based sentiment analysis. It is shown that IMN was conducted in a divided restaurant-domain dataset [37], while CD-E2EABSA is trained on the combination of both datasets. We found that, after 5000 steps of training, the F1 score for the targeted restaurant domain can exceed one of the performances of IMN, which is designed for document-aspect knowledge transfer in one domain. Although CD-E2EABSA cannot get better F1 score than IMN, it is essential to point out that IMN is trained in one domain for one time while CD-E2EABSA is trained with two domains’ datasets at the same time. Moreover, from the perspective of the dataset, IMN uses the document-level dataset additionally to assist the extraction of fine-grained emotions. Finally, the proposed method is compared with UM by Li et al. [17], who introduced the unified tagging scheme for a single-domain E2E-ABSA task. It is depicted that CD-E2EABSA can be trained in both domains and, at the same time, performs better than UM which is trained in a single domain. In other words, CD-E2EABSA can get a better result by training once, while UM needs to be trained twice.Table 3
The comparison results among different methods. Best scores are in bold, and second-best scores are underlined.
F1-RF1-LATAE-LSTM66.3663.24IAN67.7163.72IMN (Rest-14)75.6670.76IMN (Rest-15)71.76–UM69.8057.90CD-E2EABSA (L-R_5000)73.9663.85


## 5. Discussion

This section will discuss the limitations of CD-E2EABSA and future research prospects for the solutions proposed in this article based on main results. As it was shown before, this paper designs an end-to-end deep-learning network structure for cross-domain fine-grained sentiment analysis tasks. CD-E2EABSA regards the aspect-level sentiment analysis as a sequence labelling task and links it with the classic sequence labelling task of named entity recognition which draws inspiration from cross-domain named entity recognition. Also, the result shows that CD-E2EABSA can perform better for both domains than most methods trained and test in one domain. In the future, we will discuss whether task-level migration can also be introduced into fine-grained sentiment analysis to solve the problem of insufficient knowledge in this field. However, this requires the addition of other sufficient experimental data as a prerequisite, such as multidomain datasets, or other related sequence annotation datasets in exist domain such as the laptop field. In addition, in the training process of this model, especially the optimizer, there are many excellent optimization [41–43] methods that deserve to be introduced to improve the efficiency of model training, including the Swarm Optimizer [44–46].


## 6. Conclusions

A new end-to-end model is provided for cross-domain aspect-based sentiment analysis by learning of domain knowledge and sharing of nondomain knowledge. Bert is utilized as the embedding layer and generates contextual representations for words. Bi-LSTM and CRF are used as in the cross-domain ABSA layer for the output fine-grained sentiment analysis. Also, domain embeddings are introduced with a parameter generation network to achieve the mode-based transfer learning through a multitask learning strategy. From the perspective of experiments, we demonstrated the importance and effectiveness of all components in the cross-domain ABSA task. The case study and development study show the recognition of domain knowledge by the proposed model. Also, CD-E2EABSA can get the second-best results after the cross-domain training when compared with exited in-domain aspect-based sentiment analysis models.



---

*Source: 5529312-2021-03-15.xml*