An Empirical Study of Sentiment-Enhanced Pre-Training for Aspect-Based Sentiment Analysis
Year: 2023

Abstract:
Aspect-Based Sentiment Analysis (ABSA) aims to recognize fine-grained opinions and sentiments of users, which is an important problem in sentiment analysis. Recent work has shown that Sentiment-enhanced Pre-Training (SPT) can substantially improve the performance of various ABSA tasks. However, there is currently a lack of comprehensive evaluation and fair comparison of existing SPT approaches. Therefore, this paper performs an empirical study to investigate the effectiveness of different SPT approaches. First, we develop an effective knowledge-mining method and leverage it to build a large-scale knowledgeannotated SPT corpus. Second, we systematically analyze the impact of integrating sentiment knowledge and other linguistic knowledge in pre-training. For each type of sentiment knowledge, we also examine and compare multiple integration methods. Finally, we conduct extensive experiments on a wide range of ABSA tasks to see how much SPT can facilitate the understanding of aspect-level sentiments.1

heading: 1 Introduction
text: Aspect-Based Sentiment Analysis (ABSA) is an important problem in sentiment analysis (Pontiki et al., 2014). Its goal is to recognize opinions and sentiments towards specific aspects from usergenerated content (Zhang et al., 2022). Traditional ABSA approaches generally develop several separate models (Xu et al., 2018; Xue and Li, 2018; Fan et al., 2019) or a joint model (He et al., 2019; Chen and Qian, 2020), establishing interactions between different sentiment elements through specific model structures.
∗ Corresponding Authors 1We release our code, data, and pre-trained model weights
at https://github.com/HITSZ-HLT/SPT-ABSA.
In recent years, pre-trained models (Devlin et al., 2019; Liu et al., 2019; Raffel et al., 2020) have yielded excellent results in extensive NLP tasks. This inspires many research efforts that leverage pre-training techniques to learn sentiment-aware representations. Among them, Xu et al. (2019a) reveal that pre-training on the sentiment-dense corpus through masked language modeling alone could result in significant improvements on three downstream ABSA tasks. Further, researchers undertake many explorations on integrating sentiment knowledge (e.g., sentiment words) in the pre-training phase (Tian et al., 2020; Zhou et al., 2020; Ke et al., 2020; Li et al., 2021; Fan et al., 2022), as sentiment knowledge has been widely demonstrated to be helpful in various ABSA tasks (Li and Lam, 2017; Zeng et al., 2019; He et al., 2019; Xu et al., 2020a; Wu et al., 2020b; Liang et al., 2022).
Despite significant gains in various ABSA tasks, there has not been a comprehensive evaluation and fair comparison of existing Sentiment-enhanced Pre-Training (SPT) approaches. Therefore, this paper conducts an empirical study of SPT-ABSA to systematically investigate and analyze the effectiveness of the existing approaches. We mainly concentrate on the following questions: (a) what impact do different types of sentiment knowledge have on downstream ABSA tasks?; (b) which knowledge integration method is most effective?; and (c) does injecting non-sentiment-specific linguistic knowledge (e.g., part-of-speech tags and syntactic relations) into pre-training have positive impacts? Based on the experimental investigation of these questions, we eventually obtain a powerful sentiment-enhanced pre-trained model. We evaluate it on a wide range of ABSA tasks to see how much SPT can facilitate the understanding of aspect-level sentiments.
9633
To enable our study, we prepare a large-scale knowledge-annotated SPT corpus. We obtain and collate over 100 million user-generated reviews from Yelp and Amazon. Subsequently, we develop an effective semi-supervised method for sentiment knowledge mining and annotating. This method is driven by lexicons and syntactic rules, and we devise an Expectation-Maximization (EM) algorithm to estimate them. Experiments demonstrate that this method can mine more considerable and accurate sentiment knowledge than the existing methods.
Our contributions can be concluded as follows:
• We develop an effective sentiment knowledge mining method and leverage it to build a largescale knowledge-annotated SPT corpus.
• We systematically review and summarize the existing SPT approaches and empirically investigate and analyze their effectiveness.
• We conduct extensive experiments on ABSA tasks and illustrate how SPT can facilitate the understanding of aspect-level sentiments.

heading: 2 Analysis Setup

heading: 2.1 Pre-training Data
text: Following Xu et al. (2019a), we use user-generated reviews from Yelp datasets2 and Amazon reviews datasets3 (Ni et al., 2019) for pre-training. We remove those reviews that are too short (<50 characters) and too long (>500 characters) and end up with a corpus containing 140 million reviews in 28 domains. Its statistic is detailed in Appendix A.1.

heading: 2.1.1 Sentiment Knowledge Mining
text: In this paper, we mainly investigate four typical types of sentiment knowledge: reviews’ rating
2https://www.yelp.com/dataset 3https://nijianmo.github.io/amazon/
index.html
scores, sentiment words, word sentiment polarity, and aspect words. We illustrate them in Figure 1(a). Since only annotations of rating scores exist in the collected pre-training corpus, we develop an effective semi-supervised sentiment knowledge mining method.
Our method draws inspiration from the double propagation algorithm proposed by Qiu et al. (2011). They observe that there are some syntactic patterns linking aspect words and sentiment words, which is illustrated in Figure 1(b). Consequently, they define some syntactic rules to expand the aspect lexicon and sentiment lexicon iteratively. However, their method requires careful manual selection of syntactic rules. This limitation hinders the exploitation of complex syntactic patterns, such as (pizza, awful) in “we had a lamb pie pizza that was awful”.
To overcome the above limitation, we devise an Expectation-Maximization (EM) algorithm to learn syntactic rules. In our method, the annotations of sentiment words and aspect words in the reviews are treated as unobserved latent variables, and the lexicons and rules are treated as the parameters. We first initialize parameters using MPQA (Wilson et al., 2005) and several simple syntactic rules; E-step annotates the reviews through the current estimate for the parameters; M-step updates the parameters according to the expected annotations. This process can be formulated as:
initialize θ(0), (1)
repeat:
y (t) S ,y (t) A = E(x;θ (t)), (2) θ(t+1) = M(x,y(t)S ,y (t) A ), (3)
where θ = (LS ,LA,PSS ,PAA,PSA,PAS) denotes the lexicons and syntactic rules. For each mined sentiment word, we use Pointwise Mutual Information (PMI) to determine its polarity (Turney, 2002; Tian et al., 2020). See Appendix B for
more details.

heading: 2.1.2 Syntax Knowledge Acquisition
text: We annotate four types of syntax knowledge in the reviews using spaCy4. For each word, we annotate its part-of-speech tag. If there is a dependency relation between two words, we annotate its direction and type. If a word is the ancestor of another word, we annotate their dependency distance.

heading: 2.2 Downstream Tasks and Datasets
text: An aspect-level opinion can be defined as a triplet consisting of an aspect term, the corresponding opinion term, and the sentiment polarity (Peng et al., 2020). Therefore, we select Aspect term Extraction (AE), Aspect-oriented Opinion term Extraction (AOE), and Aspect-level Sentiment Classification (ASC) to measure a model’s understanding of these three sentiment elements, respectively. These downstream tasks are illustrated in Table 1. The datasets for these three ABSA tasks are derived from Wang et al. (2017); Fan et al. (2019). Their statistics are detailed in Appendix A.2.

heading: 3 Method
text: Given a review X of length T , a pre-trained model produces its word-level contextualized representations and review-level representation, which can be generally formulated as follows:
h[CLS],h1, · · · ,hT = f(x1, · · · , xT ;θPLM ).
General-purpose pre-training (Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; He et al., 2020) mostly learns parameters through Masked Language Modeling (MLM). In MLM, a certain proportion of words C in the review is masked, and the masked review X̃ is then input to the pre-trained
4The trained pipeline we use is en_core_web_sm 3.3.0.
model to recover the masked part:
P (x̃t) = softmax(FFNN(ht)), (4)
LMLM = − 1 |C| ∑
t∈C logP (x̃t = xt), (5)
where FFNN denotes a feed-forward neural network with non-linear activation, and for simplicity, we still use ht to denote the word-level representation of X̃ .
Existing SPT approaches integrate sentiment knowledge in two main ways: (1) knowledgeguided masking prioritizes masking sentiment knowledge in reviews and leverages MLM to increase the model’s awareness of sentiment knowledge; (2) knowledge supervision directly converts sentiment knowledge into labels and then predicts them by the word-level representations and the review-level representation.

heading: 3.1 Integrating Aspect & Sentiment Words
text: A common way to integrate aspect and sentiment words is to increase their masking probabilities (Tian et al., 2020; Zhou et al., 2020; Ke et al., 2020; Li et al., 2021). There are two implementations: (1) mask-by-probability masks these words with probability x% and masks other words with probability 15%; (2) mask-by-proportion randomly masks these words to y% of the total words and masks other words to (15− y)% of the total words. The main difference between these two implementations is that the former is more sensitive to the proportion of aspect and sentiment words in the review.
In addition to increasing their masking probability, we propose the strategy of masking their contexts. Our motivation stems from the observation sentiment expressions are often closer to aspect words, and thus we can leverage aspect words to locate sentiment-dense segments of a review. We assign a higher masking probability to words that are closer to aspect words. To achieve this, we
empirically choose the normal distribution and geometric distribution for the masking probability assignment, and the corresponding masking strategies are denoted as mask-context-norm and maskcontext-geo. Figure 2 provides an illustration of these two masking strategies, and detailed implementations can be found in Appendix C.
Moreover, an alternative way is to convert the aspect and sentiment words to pseudo-labels and then use the word-level representations to predict these pseudo-labels, which can be formulated as:
P (yt) = softmax(FFNN(ht)), (6)
LA/S-PL = − 1
T
∑
t
logP (y∗t ), (7)
where yt ∈ {AspW,Other} for integrating aspect words, and yt ∈ {SenW,Other} for integrating sentiment words.

heading: 3.2 Integrating Review Rating
text: The review’s rating score reflects its overall sentiment. To integrate it, Zhou et al. (2020); Ke et al. (2020) introduce rating prediction. They predict the rating score by the review-level representation and use the cross-entropy function to calculate the loss:
P (yRAT ) = softmax(Linear(h[CLS])), (8) LRAT -CE = − logP (y∗RAT ), (9)
where yRAT ∈ {1,2,3,4,5}. Besides, Li et al. (2021) adopt the supervisedcontrastive-learning objective (Khosla et al., 2020) to integrate review rating. With this objective, representations from the same sentiment are pulled closer together than representations from different sentiments. Specifically, the loss for a batch B is calculated as follows:
ℓ(i, j) = − log exp(sim(i, j)/τ)∑ k∈B\i exp(sim(i, k)/τ) , (10)
LRAT -SCL = 1 |B| ∑
i∈B
1 |P (i)| ∑
j∈P (i) ℓ(i, j), (11)
where P (i) = {j|j ∈ B\i, y(i)RAT = y (j) RAT } is the set of indices of all positives in batch B for i, and τ is the temperature. They use sim(i, j) = s⊤i sj on the normalized representations si = h (i) [CLS]/∥h (i) [CLS]∥ as similarity metric.

heading: 3.3 Integrating Other Sentiment Knowledge
text: Word Polarity. To integrate this knowledge, Tian et al. (2020); Zhou et al. (2020); Ke et al. (2020) introduce the objective of word polarity prediction. In this objective, word polarity is inferred based on word-level representations, similar to Equation 6. There are two variants: one only predicts the polarity of masked-sentiment-words, i.e., yt ∈ {POS,NEG}; the other predicts the polarity of allmasked-words, i.e., yt ∈ {POS,NEG,Other}. The difference is that the latter also includes the label of sentiment words in the supervision.
Aspect-Sentiment Pair. Tian et al. (2020) regard a sentiment word with its nearest noun (the maximum distance is 3) as an aspect-sentiment pair. They argue that aspect-sentiment pairs reveal more information than sentiment words do. Therefore, they propose aspect-sentiment pair prediction to capture the dependency between aspect and sentiment. They randomly mask at most 2 aspectsentiment pairs in each review and predict them through the review-level representation5:
P (x) = softmax(FFNN(h[CLS])), (12)
LPAIR = − 1 |P| ∑
t∈P logP (x = xt), (13)
whereP is the set of indices of words in the masked aspect-sentiment pairs.
5There are some differences in the loss calculation from the original paper (Tian et al., 2020). See our note on this in Appendix D.
Emoticons (e.g., “:-)” and “:-(”) are often inserted in the reviews to express emotions. Zhou et al. (2020) point out that integrating emoticons can capture more token-level sentiment knowledge. Consistent with Zhou et al. (2020), we treat emoticons as special tokens during the tokenization process and assign them a masking probability of 50% when masking.

heading: 3.4 Integrating Syntax Knowledge
text: Although syntax knowledge has been widely incorporated in fine-tuning various ABSA tasks (Zhang et al., 2019; Huang and Carley, 2019; Wang et al., 2020; Chen et al., 2022), few works explore its impact on SPT. In this paper, we cover four types of syntax knowledge and integrate them through knowledge supervision. We infer part-of-speech tags in the same way as in Equation 6 and trans-
form the predictions of dependency-direction, dependency-type, and dependency-distance into word-pair classification. The word-pair classification can be formulated as follows:
h̃i = FFNN(hi), h̃j = FFNN(hj), (14)
h̃i→j = [h̃i; h̃j ; h̃i − h̃j ; h̃i ∗ h̃j ], (15) P (yi→j) = softmax(Linear(h̃i→j)), (16) LDIR/TY P/DIS = − ∑
i,j
logP (y∗i→j). (17)

heading: 4 Experiment

heading: 4.1 Implementation Details
text: Following Xu et al. (2019a); Zhou et al. (2020); Li et al. (2021), we use BERT (Devlin et al., 2019) as the base framework and initialize the model
weights through BERT-base-uncased. We implement pre-training with a batch size of 1000, and an initial learning rate of 2e-4. See Appendix E for the detailed hyper-parameters. Our pre-training corpus covers 28 domains, such as Restaurant, Laptop, and Books. For most experiments, we only pre-train 10k steps on both Restaurant and Laptop. To mitigate the effect of randomness, we run each pre-training approach twice, evaluate each pre-trained model on three downstream tasks 10 times, and release the average results. Moreover, we also pre-train 400k steps on all domains to fully exploit the potential of SPT.

heading: 4.2 Main Results
text: We continue to pre-train BERT via the different SPT approaches and subsequently fine-tune them on three ABSA tasks. Their performance is reported in Table 2. We see that MLM alone yields notable improvements, where the maximum is achieved on the ASC task in Restaurant-14, nearly 3%. Integrating sentiment and syntax knowledge leads to a variety of impacts.
What impact do different types of sentiment knowledge have on downstream ABSA tasks? Most sentiment knowledge contributes to performance improvement on the ASC task, with sentiment words, review ratings, and aspect-sentiment pairs showing the highest potential. Integrating aspect words provides general benefits, and masking their contexts improves performance on nearly all downstream tasks. The impact of integrating
emoticons is minimal.
Which knowledge integration method is most effective? For aspect words, masking their contexts has a generally positive impact, while increasing their masking probabilities does not. This finding suggests that predicting their context is helpful as the context often contains the key cues of its sentiment. For sentiment words, mask-by-proportion is better than mask-by-probability. This is because the former can better balance the masking proportion of sentiment knowledge and general knowledge. For review ratings, we observe that using supervised contrastive learning does not show a salient advantage over cross-entropy, indicating that the application of contrastive learning on SPT still needs exploration.
Does integrating syntax knowledge have positive impacts? Aspect terms are often phrases, such as the orecchiette with sausage and chicken. These phrases tend to follow certain part-of-speech patterns. From Table 2, we observe that integrating part-of-speech helps the model extract aspect terms. Furthermore, Pouran Ben Veyseh et al. (2020) state that the dependency structure can provide useful information to improve the performance of AOE. Our experimental results align with their statement.
Whether integrating multiple knowledge simultaneously can lead to better results? According to the results in Table 2, we find that integrating multiple knowledge simultaneously does not necessarily lead to better performance. The most
obvious evidence lies in the combination of aspect words and aspect-sentiment pairs. In most scenarios, the introduction of an additional type of knowledge brings both benefits and drawbacks. Experimental results highlight that the combination of aspect words, review ratings, and syntax knowledge achieves the best trade-off, yielding an average improvement of 2.42% over BERT.
Further, we compare the best combination with previous SPT works and present the results in Table 3. These results show that under the same computational cost, this combination outperforms previous works in most cases. When pre-training 400k steps, we observe an average improvement of 3.25% over BERT. In addition, we note an anomaly in the model pre-trained on all domains. Specifically, its performance on the AOE task in Laptop-14 does not increase with the number of pre-training steps. This may be because the dependency between the aspect term and opinion term varies between domains. Further exploration of this phenomenon is warranted in future research.

heading: 4.3 Results on More Downstream Tasks
text: In addition to the three basic ABSA tasks, we further evaluate the SPT model on more ABSA tasks and datasets.
Aspect Sentiment Triplet Extraction (ASTE) aims to extract the aspect terms along with the corresponding opinion terms and the expressed sentiment (Peng et al., 2020). As a compound task, ASTE evaluates the model’s understanding of aspect-level sentiments comprehensively. We take the pre-trained model as the language encoder and select three classical methods for triplet extraction: GTS (Wu et al., 2020a), BMRC (Chen et al., 2021), and Span-ASTE (Xu et al., 2021). We conduct experiments on ASTE-Data-v2 (Xu et al., 2020b) and present the results in Table 4.
Experimental results show that SPT can generally improve the performance of ASTE. The best pre-trained model is our SPT (400k), which achieves an average improvement of 3.31% and 3.93% on GTS and Span-ASTE, respectively. Additionally, we observe that SPT has a smaller improvement on BMRC, suggesting that the paradigm of machine reading comprehension relies more on the model’s understanding of natural language statements than on the quality of the representations of partial words.
Cross-domain ABSA aims to transfer ABSA annotations from a resource-rich domain to a resourcepoor domain (Gong et al., 2020). This task requires the model to possess the ability to learn domain-
invariant sentiment knowledge. We leverage this task to evaluate the cross-domain capabilities of pre-trained models. We conduct experiments on the datasets released by Gong et al. (2020) and list the results in Table 5.
We find that SPT greatly boosts the performance of BERT on cross-domain ABSA. The best models are those pre-trained on a mixture of multiple domains (BERTREVIEW and Our SPT (400k)). Besides, we notice that removing syntax knowledge causes a significant drop in performance, highlighting the importance of syntax knowledge in crossdomain ABSA. Despite achieving notable improvements over BERT, SPT alone has not yet achieved satisfactory performance, suggesting that addressing cross-domain ABSA requires more than just employing SPT.
MAMS (Jiang et al., 2019) is a challenging benchmark dataset for ABSA, where each review contains at least two different aspects with different sentiments. We list experimental results on MAMS in Table 6. We find that SPT still shows performance gains on this dataset, but only at most 1%, which is relatively lower than the gains observed on other datasets. This suggests that SPT for multiaspect scenarios deserves further exploration.

heading: 4.4 Further Analysis
text: Effect of SPT on Data-scarce Scenarios. Data scarcity is a critical challenge in ABSA. We explore the effect of SPT under different amounts of training data. As depicted in Figure 3, the improvements from SPT become more obvious with less training data, with maximums of 5.11% and 6.65%. Furthermore, with SPT, the performance originally achieved using the entire training data can be attained using only 40% of it. This suggests that SPT is a feasible solution to alleviate the issue
of data scarcity in ABSA.
Evaluation for Knowledge-mining Methods. We utilize Aspect term Extraction (AE) and Opinion term Extraction (OE) to indirectly evaluate knowledge-mining methods. Since opinion terms and aspect terms are typically phrases while mining results are at the word level, we use overlap-F1 as the evaluation metric. The difference with the normal F1-score is that overlap-F1 recognizes a prediction as correct as long as it overlaps with any gold-truth term. We conduct experiments on the datasets provided by Wang et al. (2017) and list the results in Table 7. According to these results, our knowledge-mining method exhibits significant improvements over previous methods in both pre-
cision and recall. This superiority is particularly significant in AE. These observations demonstrate that our knowledge-mining method can mine more considerable and accurate sentiment knowledge.
Case Study. We present several representative examples in Table 8. The first three examples demonstrate that SPT can enable the model to discern sentiment polarities in complex semantics, such as comparisons and negations. However, the fourth and fifth examples highlight SPT’s limitations in more intricate contexts, such as statements containing factual information. Consequently, more advanced techniques are required to enhance the model’s understanding of sentiment.

heading: 5 Conclusion
text: In this paper, we perform an empirical study of Sentiment-enhanced Pre-Training (SPT). Our study investigates the impacts of integrating sentiment knowledge and other linguistic knowledge in pre-training on Aspect-Based Sentiment Analysis (ABSA). To enable our study, we first develop an effective knowledge-mining approach, leverage it to build a large-scale SPT corpus, and then select a range of ABSA tasks as the benchmark to systematically evaluate a pre-trained model’s understanding of aspect-level sentiments. Experimental results reveal the following findings: (1) integrating aspect words brings general benefits to downstream tasks; (2) integrating sentiment words, review ratings, or aspect-sentiment pairs significantly improves the performance on aspect-level sentiment classification; (3) integrating syntax knowledge can help the model extract aspect terms and aspectoriented opinion terms; and (4) the combination of aspect words, review ratings, and syntax knowledge achieves the best trade-off, yielding an average improvement of 3.25% over BERT. We further examine SPT’s effectiveness on more ABSA tasks and find that SPT can improve the performance of
a wide range of downstream tasks. Notably, SPT improves the model’s cross-domain capabilities. In addition, we also demonstrate the effectiveness of our knowledge-mining method.

heading: Acknowledgments
text: We thank the anonymous reviewers for their valuable suggestions to improve the overall quality of this manuscript. This work was partially supported by the National Natural Science Foundation of China (62006062, 62176074, 62176076), Natural Science Foundation of Guangdong 2023A1515012922, the Shenzhen Foundational Research Funding (JCYJ20220818102415032, JCYJ20200109113441941), the Major Key Project of PCL2021A06, Guangdong Provincial Key Laboratory of Novel Security Intelligence Technologies 2022B1212010005, and Key Technologies Research and Development Program of Shenzhen JSGG20210802154400001.
Limitations
While this paper provides a systematic investigation and analysis of existing Sentiment-enhanced Pre-Training (SPT) approaches, it is important to acknowledge the following limitations:
• The existing SPT approaches do not exhibit sufficient performance improvement in the multi-aspect scenario. Achieving significant performance enhancements in this particular scenario through SPT poses a challenging task and warrants further exploration.
• Compared to the SPT corpus, the existing downstream ABSA datasets cover fewer domains. This limits the in-depth analysis of aspect-level sentiments on different domains.
We believe that addressing the above limitations can facilitate the development of SPT.

heading: A Additional Data Statistics
text: A.1 Statistic of the SPT Corpus
We collect user-generated reviews from Yelp and Amazon to build the SPT corpus. We record the reviews from Yelp as the Restaurant domain and the reviews from Cell Phones and Accessories of Amazon as the Laptop domain. These two domains contain 11 million reviews in total. The detailed statistic is presented in Table 9.
Note for Laptop reviews. In the Amazon reviews dataset, Laptop is a subcategory under Electronics. However, this Laptop subcategory only contains 220k reviews, significantly less than the 4 million reviews on Yelp (Restaurant). To address this issue, some researchers have included reviews from other similar domains. For instance, Yin et al. (2016) added Cell Phone reviews to the Laptop corpus, while Wang et al. (2021) used Cell Phone reviews as the unlabeled data for semeval14lap. Cell Phone category contains 6 million reviews, which is large enough. Although the use of Cell Phone reviews as a substitute for Laptop reviews is not ideal, our analysis shows that the two domains share high similarity in aspect-level sentiment expression.
A.2 Statistic of the Downstream ABSA tasks
Dataset for the AE and ASC tasks is provided by Wang et al. (2017). We download it from https://github.com/yhcc/ BARTABSA/tree/main/data/wang. Since there is no explicit validation set, we randomly split 20% of its training data as the validation set. Its statistic is presented in Table 10. Following Xu et al. (2019b), for the ASC task, we only train and test the model on those aspect terms with polarity POS, NEU, and NEG.
Dataset for the AOE task is provided by Fan et al. (2019). We download it from https:// github.com/NJUNLP/TOWE. Following Fan et al. (2019), we randomly split 20% of its training set as the validation set. Its statistic is presented in Table 10.
Dataset for the ASTE task is ASTE-Data-v2, which is provided by Xu et al. (2020b). We derive
it from https://github.com/xuuuluuu/ Position-Aware-Tagging-for-ASTE. Its statistic is presented in Table 10.
Dataset for cross-domain ABSA is provided by Gong et al. (2020). We derive it from https:// github.com/NUSTM/BERT-UDA. Its statistic is presented in Table 11.
MAMS Dataset is presented by Jiang et al. (2019). We download it from https://github.com/ siat-nlp/MAMS-for-ABSA. Its statistic is presented in Table 11.

heading: B Supplementary Description of Our Sentiment Knowledge Mining Method
text: B.1 Mining Aspect & Sentiment Words
Existing SPT works have developed several knowledge-mining methods (Tian et al., 2020; Zhou et al., 2020; Ke et al., 2020; Li et al., 2021). They leverage part-of-speech patterns or sentiment lexicons to annotate sentiment words. Then, they
treat the nearest nouns to the sentiment words as aspect words or build an aspect lexicon based on the aspect annotations of the existing downstream datasets. However, these knowledge-mining methods either lack domain adaptability or are unscalable. Therefore, this paper develops an effective knowledge-mining method.
Hypothesis. Any two words in the same sentence are connected by a syntactic path. This paper denotes a syntactic path as a sequence of dependency relations and part-of-speech tags. For example, given the sentence “we had a lamb pie pizza that was awful”, the syntactic path from pizza to awful is denoted as (NOUN, relcl−−−→,VERB, acomp−−−−→,ADJ). Qiu et al. (2011) observe that there are some syntactic paths linking aspect words and sentiment words. Based on this observation, we assume that there exist lexicons LA,LS and aspect-sentiment path set PAS that satisfy:
If wi and wj are linked by path p, then
wi ∈ LA and p ∈ PAS =⇒ wj ∈ LS , (18) wi ∈ LA and wj ∈ LS =⇒ p ∈ PAS , (19)
We leverage this assumption to mine lexicons and path sets.
Initialization. We initialize the sentiment lexicon
with adjectives of MPQA6 and initialize the aspect lexicon as an empty set. We empirically initialize four path sets with several simple paths:
P(0)AA = {(NOUN, conj−−−→,NOUN),
(NOUN, conj←−−−,NOUN)},
P(0)SS = {(ADJ, conj−−−→,ADJ),
(ADJ, conj←−−−,ADJ)},
P(0)AS = {(NOUN, nsubj←−−−,AUX, acomp−−−−→,ADJ)}, P(0)SA = {(ADJ, acomp←−−−−,AUX, nsubj−−−→,NOUN)}.
Expectation. We first leverage the two lexicons to annotate aspect words and sentiment words in the sentences. Then, we treat these annotations as conditions to further annotate aspect and sentiment words in the sentences using the four path sets through Equation (18). We merge the two annotations as the final annotation of this step.
Maximization. For each word wi, we count its occurrences of being annotated as the aspect word in the E-step, denoted as #wi_asp. We also count its occurrences in the corpus, denoted as #wi. Thus, we calculate the aspect score of a word as follows:
aspect-score(wi) = #wi_asp
#wi . (20)
We add those words whose aspect scores are greater than the given threshold αA to the aspect lexicon LA. The expansion of the sentiment lexicon is similar.
For each path p, we count the annotations of all the word pairs (wi, wj) linked by this path. The aspect-sentiment score of the path is calculated by:
as-score(p) = #wi_asp ∗#wj_sen
#wi_asp , (21)
where #wj_sen denotes its occurrences of being annotated as the sentiment word. We add those paths whose aspect-sentiment score is greater than the given threshold αAS to PAS . The other three path sets are expanded similarly.
For each domain, we run the EM algorithm on 500k reviews to learn the parameters and then annotate all reviews.
6http://mpqa.cs.pitt.edu/lexicons/ subj_lexicon/
B.2 Polarity Assignment We leverage the reviews’ rating scores for polarity assignments. We empirically treat 5-star-rated reviews as positive reviews and (1,2,3)-star-rated as negative reviews. For a sentiment word wi, we count its occurrences in positive and negative reviews, denoted as #wi_pos and #wi_neg. Thus the polarity score of this word can be calculated by:
pmi_so(wi) = log #wi_pos ∗#neg #wi_neg ∗#pos , (22)
where #pos and #neg denote the total number of positive and negative reviews. Equation (22) is derived from Pointwise Mutual Information (PMI) (Turney, 2002).
For each domain, we calculate the polarity score of each sentiment word on all reviews. Then, we empirically assign those sentiment words whose polarity scores are greater than 0.2 as POS and those sentiment words whose polarity scores are less than -0.2 as NEG.

heading: C Detailed Implementation of Masking Context
text: We leverage aspect words to locate the sentimentdense segments of the review and improve the masking probability of their contexts.
Masking Context by Normal Distribution. Suppose there is only one aspect in the review, and its position is t. We use the normal distribution N (t, σ) for the masking probability assignment, where σ is a hyper-parameter. This could be formulated by:
fnorm(i; t, σ) = 1√ 2πσ exp
( −(i− t) 2
2σ2
) .
pnorm(i) = { fnorm(i; t, σ) i ̸= t, 0 i = t.
Actually, a review often contains more than one aspect word. Therefore, we sample k aspect words for each review, repeat the probability calculation k times, and pick up the maximum probability for each word. k is related to the length of the review:
k = Tz, (23)
where T is the length, and z is a hyper-parameter. Finally, we perform normalization to ensure that the masked part is 15% of the review.
Masking Context by Geometric Distribution. We also use geometric distribution for the masking probability assignment:
fgeo(i; p) = (1− p)i−1p. (24)
pgeo(i) =    fgeo(t− i; p) i < t, 0 i = t,
fgeo(i− t; p) i > t. (25)
where p is a hyper-parameter. We also sample k aspect words and perform normalization.
We find that both masking context methods are highly sensitive to hyper-parameters. In our experiment, we set σ = 6, p = 0.4, and z = 0.1.

heading: D Additional Notes for Existing SPT Approaches
text: Note for SEKP. Tian et al. (2020) propose aspectsentiment pair prediction to capture the dependency between aspect and sentiment. They regard aspectsentiment pair prediction as a multi-label classification task and further transform it into multiple binary classification tasks. However, their implementation only includes positive samples but ignores negative samples when calculating the loss. Therefore, we adopt a different implementation to correct this mistake. This implementation is described in Equation 12 and 13. Besides, since our pre-training corpus contains aspect annotation, we regard a sentiment word with its nearest aspect word as an aspect-sentiment pair.
Note for Pre-training Step. Existing SPT works have different setups. We estimate their computation based on batch size, maximum text length, model architecture, and training step. Then, we calculate the training step to reach this computation under our settings.

heading: E Hyper-Parameters
text: We list the detailed hyper-parameters of SPT in Table 12. In our SPT, we integrate ASPECTWORD, REVIEWRATING, and SYNTAX in pre-training. Therefore, the loss for SPT is calculated by:
LSPT = LMLM + α1LRAT + α2LSY N , (26) LSY N = α3LPoS + α4LDIR + α5LDIS . (27)
We set α1 = 1.2, α2 = 1.2, α3 = 0.8, α4 = 1.0, α5 = 0.3.

heading: F Additional Results
text: Effect of Pre-training Step. The pre-training step is the most critical hyper-parameter in SPT. We show its effect on the downstream ABSA tasks in Figure 4. We can see that the performance on AE and ASC generally increases with the number of pre-training steps. Although the increase is slowly decreasing, it can be expected that the performance will continue to increase if pre-training continues on the basis of SPT (400k). However, this trend is not obvious on AOE. In particular, the performance in Laptop-14 degrades as the pre-training step increases. We speculate that this is due to the differences between domains, which are more pronounced for the AOE task. Not only does the wording of the opinion term differ across domains, but the dependency between the aspect term and opinion term also differs across domains. These differences deserve further exploration and analysis in future works.
Full Results on Cross-domain ABSA. We present the full results on two cross-domain ABSA tasks in Table 13 and 14.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section of Limitations
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: 1. Grammarly. We use it to correct grammatical errors. 2. DeepL. We use it for translation between languages.
B 3 Did you use or create scientific artifacts?

heading: Appendix A.2.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Appendix A.2.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? The artifacts we use have no terms attached to them but can be used in various scientific studies by default when provided they are cited.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The artifacts we use do not specifically state their intended use
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.2.
C 3 Did you run computational experiments? Section 3

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix E.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix E.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2.1.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: These differences deserve further exploration and analysis in future works. Full Results on Cross-domain ABSA. We present the full results on two cross-domain ABSA tasks in Table 13 and 14. ACL 2023 Responsible NLP Checklist