Exploring Graph Pre-training for Aspect-based Sentiment Analysis
Authors: Xiaoyi Bao
Zhongqing Wang
Guodong Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: Aspect-based Sentiment Analysis; Generative model; Graph pre-train

Abstract: Existing studies tend to extract the sentiment elements in a generative manner in order to avoid complex modeling.
Despite their effectiveness, they ignore importance of the relationships between sentiment elements that could be crucial, making the large pre-trained generative models sub-optimal for modeling sentiment knowledge. 
Therefore, we introduce two pre-training paradigms to improve the generation model by exploring graph pre-training that targeting to strengthen the model in capturing the elements' relationships.
Specifically, We first employ an Element-level Graph Pre-training paradigm, which is designed to improve the structure awareness of the generative model. Then, we design a Task Decomposition Pre-training paradigm to make the generative model generalizable and robust against various irregular sentiment quadruples. 
Extensive experiments show the superiority of our proposed method, validate the correctness of our motivation.

Exploring Graph Pre-training for Aspect-based Sentiment Analysis
Xiaoyi Bao', Zhongqing Wang', Guodong Zhou'
Natural Language Processing Lab, Soochow University, Suzhou, China
p221354541 3@outlook, com
(wangzq, gdzzou edu. cn
Abstract
Existing studies tend to extract the sentiment elements generative manner in order to avoid complex modeling. Despite their effec- tiveness, they ignore importance of the relation- ships between sentiment elements that could be crucial, making the large pre-trained gener- ative models sub-optimal for modeling senti- ment knowledge. Therefore, introduce two pre-trainn pre-training paradigms to improve the gener- ation model by exploring graph pre-training that targeting to strengthen the model in cap- turing the elements' relationships. Specifically, We first employ an Element-level Graph Pre- training paradigm, which is designed to im- prove the structure awareness the generative model. Then, design Task-level Graph Pre-training paradigm make the generative model generalizable and robust against vari- irregular sentiment quadruples. Extensive experiments show the superiority of our pro- posed method, validate the correctness of our motivation. Our code be found in https://g8 thubtttucccoo hwt.uu...o//Hteee EGP4ABSA-EMNLP200
Kask
Element-level Graph Pre-train
o
8
Task-level Graph Pre-train
Aspect
Opinion
Category
Polarity
Figure 1:
Two proposed pre-train paradigms.
either sequence-level (Qiu et al., 2011; I; Peng et al., 2020; Cai et al., 2021) or token-level classification problems (Tang et al., 2016) in joint learning or pipeline manner. However, these methods not only require sophisticated and complex modeling of sen- timent elements but also suffer severely from ror propagation since the overall prediction perfor- mance hinges on the accuracy of every step (Peng et al., 2020).
1 Introduction
More recently, studies tend to tackle the ABSA problem with unified generative approach (Zhang et al., 2021b,a: Yan et al.. 2021: Bao et al.. 2022). They organize the target sequence in dif- ferent approaches, namely listing (Zhang al., 2021b); "(apps, hard, Software, Negative)", in- dexing Yan al., 2021): "(1,1,3,3)" paraphrasing (Zhang et al., 2021a): "(Software is good because apps are hard)" or opinion tree(Bao et al., 2022): "(Rooot, Aspect Software, apps J, Opinion ( NNegative hard })))))" However, they ignore the importance of the relationships among elements (e.g. sentiment polarity should be identified based on opinion words, like great identifies positive polarity and disappointing identifies negative po- larity).
Aspect-based sentiment analysis (ABSA) has drawn increasing attention in the community, which includes four fine-grained elements: aspect term, opinion term, aspect category, and opinion polarity. The first two terms exist as text span in the review sentence while the remaining two the classification result of aspect and opinion re- spectively. Each four mapped sentiment elements form an aspect-level sentiment quadruple. For in- stance, for the given review "The apps are hard to use." the corresponding quadruple is (apps, hard, Software, Negative).
The joint extraction of quadruples is the most complex and challenging subtask among all the ABSA tasks, previous work usually formulate it as
Corresponding author
In this situation, a natural question is how to

---

Element-leve level Graph Pre-train
Opinion Tree
Sabrask EPd
Input
Ourgut
ROOT
Subtask
[Tree]
handl
appe
Panaeee Mash
Pinetune
Quad
Review
Prompee
Review
Tree
Tree
Pre-train
Generative Model
Aspect
Opinion
Aspect
Opinion
Subtask Aspect
Subeask
Subtask Finetune
Input
Input
Input
Input
Design Postttive Software Negative
Fronge
[Aspas]
[opeet]
Troept
Diaewe]
Generate
coategory
Linearized
Dernde
surface
smooth
apps
hard
laad
Awdew
hard
Banion
hard
bard to
Tree
(aspect) (opinion) (aspect) (opinion)
Output
Output
Output
Output
Aspect
Optaion
Calcauey
Polarty
Task-level Graph Pre-train
Figure 2: Overview of joint pre-training, the subtasks will be introduced in the following section. We simplify the process of Task-level Graph Pre-training for brief reading, the detailed process will be introduced in the following section.
strengthen the generative model in modeling aspect- level sentiment structure. We believe the challenges locate in two aspects. First is structural modeling: the huge gap between the pre-training and finetun- ing phases makes it difficult to model its succinet yet distinctive structure: certain components ( (.g. aspect term) ) in sentiment structure obviously more important than others. Another challenge is the gen- eralization and robustness of the generative model: the generative model should be generalizable and robust against irregular sentiment quadruples. It is crucial since the structure is built depending on the quadruples and the challenging scenarios in real practice are usually brought by the irregular sentiment quadruples.
cedure to learn to handle the task in stages. Specifi- cally, we first decompose the quadruple extraction task into multiple subtasks. Each subtask corre- sponds to mapping the steps for manually building an opinion tree from scratch. Afterwards, fea- ture a prompt-based learning strategy separately acquire the knowledge of subtasks and finally em- ploy the learned knowledge to tackle the main task, i.e., generating the entire opinion tree. The decom- posed subtasks build fundamental knowledge of irregular sentiment quadruples for generation.
As shown in Figure 2, we then jointly pre-train the model with the two paradigms above and fine- tune the model with the F Finetune task. The ad- vantages of our pre-training method over previ- ous learning methods are threefold: 1) both the Element-level Graph Pre-training and Tesk-level Graph Pre-training are designed depending on the intrinsic characteristics of the opinion tree instead of treating it as a plain graph.2) the Element-level Graph Pre-training abandons the strategy of cap- turing the complex structure but focuses directly on the core elements. 3) the Task-level Graph Pre- training explicitly forces the model to learn the irregular quadruples with an casy-to-hard routine, making it easier for the model to learn the funda- mental knowledge required. The detailed evalua- tion shows that our model significantly advances the state-of oaae---f-hh-aas the-art performance on several bench- mark datasets.
In this study, proposed two novel graph pre- training paradigms address above challenges. As shown in Figure 1, we first introduce an op- timal self-encoding method called Element-level Graph Pre-training. We abandon the traditional indiscriminate masking strategy (equally random masking every node or edge and depending on the characteristics of the opinion tree, adopt sen- timent element level masking. Given the opinion tree of the review "The apps are hard use.". only sentiment nodes (namely apps, hard, Software Negative or the sub-trees they composed in the graph will be masked. In this case, this method can serve as an effective addition to structural modeling in opinion tree generation.
We then propose Task-level Graph Pre-training paradigm, which mimics the human learning pro-

---

Reform
elements, proposed a sentiment tree structure called opinion tree, and employed generative model extract the linearized tree. However, the genera- tive model is pre-trained to solve textual sequence tasks(e.g. masked language model) but finetuned for structure generation, between which exists a huge gap, making generative models sub-optimal for modeling structural knowledge.
(Root, (Quad, ( (Aspect (Software, apps ) Opinion
Output
Opinion Tree
Decoder
Encoder
Different from previous studies, we introduce two pre-training paradigms for opinion tree gener- ation without treating it as plain graph. To our knowledge, we are the first to consider designing methods depending on the intrinsic characteristics of the opinion tree.
The surface smooth, but
Aspect
Opinton
Input
the apps are hard to use.
Category
Polarity
Figure 3: Opinion tree generation model.
2
Related Work
3 Opinion Tree Generation Model
There are four aspect-level sentiment elements in ABSA, the various combination of these elements form the numerous sub-tasks of ABSA. The re- searches on ABSA generally follow route from handling single sub-task complex compositions of them. The starting point usually locates in the prediction of a single sentiment element, which is the target of fundamental sub-tasks, such as extract- ing the aspect term (Qiu et al.. 2011; Tang et al., 2016; Wang et al., 2021), classifing the aspect cate- gory mentioned in the sentence (Bu et al., 2021; Hu et al., 2019), and detecting the sentiment polarity for given aspect (Tang al., 2016; Chen al., 2022a; Liu et al., 2021; Scoh et al., 2021: Zhang et al., 2022).
In this section, introduce the basic opinion tree generation model we employed to generate in the pre-train and finetune phases, along with the objec- tive functions and training.
3.1 Opinion Tree Construction
For further strengthen the relationship between el- ements, we build a structure called opinion tree, which aims to jointly model all sentiment elements in tree for given review sentence. The opinion tree can be considered as semantic representa- tion in order better represent the structure of sentiment elements. Inside the opinion tree, cach sentiment element would be connected with an- other node as either the child or parent relation to represent the crucial relationship.
Since the sentiment elements are naturally cor- related, many studies further focus on exploring the co-extraction of sentiment elements, including aspect and opinion term extraction (Xu al., 2020; Li et al., 2022); aspect term extraction and its po- larity detection (Zhang and Qian, 2020); aspect category and polarity detection (Cai et al., 2020). Furthermore, recent studies also employed end-to- end models to extract all the sentiment elements in triplet or quadruple format (Peng et al., 2020; Wan et al., 2020; Cai et al., 2021; Zhang et al., 202 Chen et al., 2022b; Mukherjee al., 2021).
As shown in Figure 3, we construct the opinion tree using rooted directed acyelic graph, including nodes of aspect, opinion, category, and polarity, along with the semantic relations between them. After that, linearize the opinion tree to the target sequence via depth-first traversal.
3.2 Generation Model
We employ the pre-trained language model T5 (Raf- fel et al., 2020) to generate the linearized opin- ion tree. As shown in Figure 3, it is an encoder- decoder architecture model, the input would be the raw review and the output is linearized opinion tree. Given the token sequence x #1, .... T|a] input, the sequence- sequence-to-ssqueene io-sequence model outputs the lin- carized representation y y1 ... B/y To this end, the sequence- oo-sequence model first computes the hidden vector representation:
More recently, studies tend to design a uni- fied framework to extract quadruples at one stop with pre-trained encoder-decoder language models, achieving great improvements in ABSA (Zhang al., 2021a). The target sequence of them is formed by either class index (Yan et al., 2021) the desired sentiment element (Zhang et al., 2021b). OTG (Bao et al., 2022) addressed the im- portance semantic correlations among sentiment
= (I1,,.., 2..,,,, F()
(1)

---

Input
Subtask
Subtask
Prompt
Review
Tree
EGPI
[Tree]
The apps are hard
(Root,Quad, <Masks....
(Root,,Quad (Negative,
EGP2
[Sentence]
The apps <Mask> use.
<Masko>
The apps are hard to use.
EGP3
[Sentence]
The apps <Mask> use.
(Root. (Quad, ... (Negative, hard)
The apps are hard use. (Root, (Quad, ...NNegative. The apps are hard to use.
EGP4
[Tree]
The apps <Mask> use.
(Root,(Quad, Mask>.... (Root.(Quad, ...sMas......)
EGPS
[Sentence]
The apps <Mask> use.
Table Subtasks of Element-level Graph Pre-training.
ROOT
where 1 uf. Xo; 0) is calculated
Quad
by the decoder.
(Root,
Linearize (Quad,
Aspect
Opinion
Aspect (Software, apps).
Pre-training Paradigms
Software
Negative
Opinion (Negative, hard }))
In this study, we introduce two pre-training paradigms for opinion tree generation. As shown in Figure 2, the two paradigms and finetune task share the same input format with a joint input of prompt, encoded text and tree, each method con- sists of set of subtasks focus on respective train- ing targets. The combination subtasks forms the joint pre-training in our work, we will introduce the paradigms first in this section.
hard
Mask
(Roat,
(Software, apps)
(Quad,
/ legitimate structure / sentiment
(Aspect
Mask
(Opinion (Negative, hard JT))
Figure 4: Example of element-leve graph masking.
After the input token sequence is encoded, the decoder predicts the output sequence token-by- token with the sequential input tokens' hidden vectors. At the i-th step of generation, the self- attention decoder predicts the i-th token y: in the linearized form, and decoder state h as:
4.1 Element-level Graph Pre-training
The opinion tree directly composed of subtrees that represent respective quadruples, this naturally decides the noteworthy information must locate within the aspect-level sentiment element instead of the other parts of the opinion tree, which could be other structure nodes. For instance, for lin- earized opinion tree "(Root, (Aupec (Soft- ware, appsi..Opinion (Negative, hard)", the in- discriminate masking may mask a sub-sequence "(Opinion (" that: 1) logically can not be reform into a valid structure due to the non-closing brack- ets. 2) contains nodes (e.g." "Opeiioiii not included in the crucial sentiment elements.
ye: 1 ([H: h4 1. 1i
(2)
The conditional probability of the whole output sequence p(y(ar) is progressively combined by the probability of each step p(u (ull,,,,))
[y]
ply II p(uluce.z)
(3)
where y<i = y1. and p(ulle<), 2) are the probabilities over target vocabulary V.
On the other hand, Element-level Graph Pre-training paradigm masks aspect-level element nodes (including aspect term, opinion term, aspect category, and opinion polarity) the opinion tree, as shown in Figure 4, the masked sequence "(Soft- ware, apps )" represent legitimate struct and covers core sentiment element only. If continuous nodes are masked, the corresponding sub-graph will be masked whole. The method can not only make the masked node are crucial sentiment ele- ments but also guarantee the corresponding sub- sequence logically legitimate.
The objective functions is to maximize the out- put linearized opinion tree Xr probability given the review sentence Xo. Therefore, we optimize the negative log- likkliiihoodd loss function:
1
L
Σ
logp( Xo;
(4)
|
(Xr
where is the model parameters, and (Xo, Xr) is a (sentence, tree) pair in training set T, then
log p(Xpp((Xpxxp Xo:
log bg | 27.... "h..../11 1-1 Xoi
(5)
With the element-level graph mask strategy in- troduced above, we propose a set of pre-training

---

Input
Output
Prompt
Sentence
Aspect
[Aspect]
The apps are hard to use.
(Root, (Aspect, apps))
Opinion
[Opinion]
The apps hard The apps are hard to use.
Pre
(Root,(Opinion, hard))
crrrrtu L Tre-frmn
Category
[Category]
[Aspect]
(Root, Aspect (Software, apps))
Polarity
[Polarity] [Opinion]
The apps hard use.
(Root, Opinion (Negative, hard))
Loop
Pair
[Aspect] [Opinion]
The apps are hard to use.
(Root, Quad (Aspect ... hard))
Triple
[Aspect] [Opinion]
[Polarity]
The apps hard
(Root, Quad[Aspect )
Finetune
Asseet [Opinion] [Polarity][aategory
The apps are hard to use.
(Root,(Quad, ...Neaative, hurd)
Figure 5: Subtasks of Task-level Graph Pre-training paradigm. Note the finetune task has been added into the pre- gre-tranning phase.
Epoch range
Tree mask rate
Text mask rate
masked text from opinion tree is unreasonable since opinion tree contains limited information discussed before.
0% ~ 25%
0,25
0.15
25% 50%
0.3
0.15
500% 75%
0.35
0.15
75% 100%
0.4
0.15
4.2 Task-level Graph Pre-training
Table 2: Dynamic masking rate.
Inspired by the human-learning process we pro- pose a Task-level Graph Pre-training paradigm, whose subtasks follow the routine of human learn- ing procedure learn to build the opinion tree from scratch. Specifically, we first decompose the quadruple extraction task into multiple sub- tasks. Each subtask corresponds to mapping the steps for manually building an opinion tree from scratch. The paradigm consists of six subtasks, four (Aspect, Opinion, Calegory, Polarily) of which extract sentiment structure the fundamen- tal knowledge for building opinion tree, the rest (Paér, Triple) target the intermediate state of the procedure with co-extraction. The subtasks and the corresponding steps of building can be found Appendix A. In this case, we force the model to focus directly on irregular cases with a gradual pro- cess to build fundamental knowledge for OTG. The inputs of Task-level Graph Pre-training are similar to the previous paradigm, which would be con- cat of prompt and a sentence. Then the subtasks in Task-level Graph Pre-training paradigm can be given as shown in Figure 5.
subtasks. The inputs would concat a prompt, a sentence, and an opinion tree. The sentence and tree will be masked with different masking rates while the prompt illustrates the output tar- get, either the sentence or tree. For given re- view (I1, (. Rn) and lincarized tree == (4,,22... (n), We design the 5 subtasks in the Element-level Graph Pre-training paradigm, which can be found in Table 1. Among which, EPG1 and EPG4 are designed to help the model generate the complete tree by adding text infor- mation while EPG2, EPG3 and EPG5 help the model to generate the full review by adding the structural information.
To further emphasize the interaction between the pre-training and finetune phases, we designed a dynamic masking rate for Element-level Graph Pre-training paradigms: a small masking rate is used in the initial phase, and then the masking rate increases with training rounds, so that at the end of pre-training, all partially masked pre-training tasks be very close the finetune tasks (which can be considered 100% masking rate), the specific masking rate shown in Table 2. Note our mask- ing rate obviously lower than previous work (Bai et al., 2022), that is because recovering a nearly all-
4.3 Joint Pre-training
We use joint pre-training method to com- bine the advantages of the Flement-level Graph Pre-training paradigm and Task-level Graph Pre- training paradigms. In addition, we include the

---

60%
We employ T5 and fine-tune its parameters for our opinion tree generation model. We tune the parameters of our models by grid searching on the validation dataset. We select the best models by early stopping using the Accuracy results on the validation dataset. The model parameters are optimized by Adam (Kingma and Ba, 2015), the learning rate of pre-training and finetuning is 3e-5 and le-4 respectively. The batch size is 16. Our experiments are carried out with an Nvidia RTX 3090 GPU. The experimental results are obtained by averaging ten runs with random initialization.
40%
30%
20%
10%
0%
Basic
Bilmppicit One ToMany
Monoomplicit
Laplop
Figure 6: Statistic regular and irregular situations of opinion trees.
In evaluation, a quadruple is viewed as correct if and only if the four elements, well as their combination, exactly the same as those in the gold quadruple. On this basis, calculate the Precision and Recall, and use Fl score as the final evaluation metric for aspect sentiment quadruple extraction (Cai et al., 2021; Zhang et al., 2021a).
finetune task Finetune in the pre-train phase for narrowing the gap between two phases and avoid- ing overfitting. During pre-traine pre-training, the model will be cyclically trained in the order of a loop started with the subtasks of the Element-level Graph Pre- training, followed by Task-level Graph Pre-training, the gradient will be updated after accumulating the loss in each epoch. After that, save the model weights and finetune the model with finetune task Finetune.
5.2 Main Results
We compare the proposed method with several classification-based aspect-based sentiment anal- ysis models, including. DP (Qiu et al., 2011), JET (Xu et al., 2020), TAS-BERT (Wan et al., 2020) and Extract-Clsssif (Cai et al., 2021). In addi- tion, generative models are also compared, such as BARTABSA (Yan et al., 2021), GAS (Zhang et al., 2021b), Paraphrase (Zhang et al., 20DI (Hu et al., 2022), Seq2Path (Mao et al., 2022) and OTG (Bao et al., 2022).4.
5 Experiments
this section, we introduce the datasets used for evaluation and the baseline methods employed for comparison. We then report the experimental re- sults conducted from different perspectives, and analyze the effectiveness of the proposed model with different factors.
Particularly, build two Large Language Model (LLM) baselines: CharGPT is a sibling model to InstructGPT (Ouyang et al., 2022), which is trained to follow instruction in a prompt and pro- vide a detailed response. We ask it to generate all the sentiment elements from the input review sen- tences. LLaMA (Touvron et al., 2023) is a collec- tion of foundation language models, these models trained trillions of tokens, and have shown that it is possible train state-of-the-arr models ing publicly available datasets exclusively. We use LLaMA-7B, and fine-tune it on the ABSA dataset. As shown in Table 3, we find that generative models outperform previous classification-bart methods and the structural generative method sur-
5.1 Setting
In this study, we use ACOS dataset (Cai et al., 2021) for our experiments. Following the setting from (Cai et al., 2021), we divide the original dataset into training set, validation set, and testing set. In addition, we choose 20,000 sentences from Yelp'. and 20,000 sentences from the laptop domain in Amazon to pre-train the opinion tree generation model, the sentences are annotated by the OTG model without pre-training.
Following the setting of Bao et al. (2023), divide the quadruples into 4 types, apart from the basic situation, there are irregular situations: One- to-Many, Mono-Implice and Bi- Ii-mmicc The statistic can be found in Figure 6.
5 www.doo/.t.h.eeeee
co/ trrnssrrrrrrrs////
"We directly adopt the result from Bao et al. (2022) 'https://openal coo//tlo/chattpt.
httpps//wwww..lpp.oo///aaaaa hwwp://oyyyoooooodddd/a//aaaaaaa
"https://hugging co/ co/bocsstransformmrss
main/model_doc/ lamm.

---

Restaurant
Laptop
Method
P.
R.
F1.
P.
R.
Fl.
DP
0.3467
0.1508
0.2104
0.1304
0.0057
0.0800
JET
0.5981
0.2894
0.3901
0.4452
0.1625
0.2381
TAS-BERT
0.2629
0.4629
0.3353
0.4715
0.1922
0.2731
Extract-Classie
0.3854
0.5296
0.4461
0.4556
0.2948
0.3580
BARTABSA
0.5662
0.5535
0.5598
0.4165
0.4046
0,4105
GAS
0.6069
0.5852
0.5959
0.4160
0.4275
0.4217
Paraphrase
0.5898
0.5911
0.5904
0.4177
0.4504
0.4334
TODA
0.5904
0.6029
0.5966
0.4359
0.4367
0.4363
Seq2Path
0.6029
0.5961
0.5995
0.4448
0.4375
0.4411
ChatGPT
0.5014
0.3625
0.4207
0.4492
0.3123
0.3541
LLaMA
0.5963
0.6097
0.6029
0.4461
0.4392
0.4426
OTG
0.6138
0.6190
0.6164
0.4408
0.4381
0.4394
Ours
0.6486
0.6297
0.6390
0.4523
0.4523
0.4512
Table 3: Comparison with baselines.
passes non-structural methods, this indicates that semantic structure does contribute to quadruple extraction. Meanwhile, our proposed model out- performs all the previous studies significantly (p < 0.05), which has an advantage 2.36% and 0.92% in Restaurant and Laptop domain respectively. The result shows that the proposed joint pre-training is effective in modeling tree structural constraints for generative model, while the large gap between pre-training and finetuning significantly encumbers previous systems. Furthermore, the results also in- dicate the effectiveness of our Element-level Graph Pre-training and Task Decomposition paradigms, which are used to unify the pre-train and finetune task with special task designs depending on the intrinsic characteristics of the opinion tree instead of treating it as plain graph.
Method
Restaurant
Laptop
Ours
0.6390
0,4512
-GG
0.6339
0,4490
-TGP
0.6334
0.4463
-EGP& TGP
0.6164
0.4393
Table 4: Impact of pre-training paradigm.
Method
Restaurant
Laptop
OTG
0.6164
0.4394
Indiseriminate
0.6287
0.4423
+ +Finetune
0.6211
0.441
+ T&GPI, Finetune
0,6243
0,4413
+ EGPI, Finetune
0.6276
0.4421
EGP2, EGP3
+ + EGPs
0.6334
0.4463
Ours
0.6390
0.4512
6 Analysis and Discussion
Table Impact subtasks Element-level Graph Pre-training paradigm.
In this section, we first give some analysis and discussion to show the influence of Element-level Graph Pre-training (EGP) and Task-level Graph Pre-training (TGP) paradigms. After that, we will investigate our search over masking rate, the influ- ence of pre-training subtasks.
avgerage drop of 0.52% while EGP's cause 0.21 %, this may due to the generalization and robustness being more effective than the structural association.
6.2 Efffct of Element-level Graph Pre-training
6.1 Influence of Different Factors
We first investigate the difference between the two paradigms, from Table can find, all the paradigms are beneficial to extract the opinion tree. Among which TGP paradigm's 's contribution outper- forms EGP paradigm, the removal TGP cause an
Under the setting of our element-level masking de- sign for graph pre-train, previous graph-masking strategies can be classified into the indiscriminate paradigm, which means indiscriminately mask- ing random nodes and words in tree or text.

---

Method
OTG
Ours
the basic situation. Thus we mimic the human learning procedure for building an opinion tree from scratch with Task-level Graph Pre-training to strengthen its fundamental knowledge.
Basie
0.6517
0.6610
0.93%
OncTooMayy
0.4503
0.4720
+ 2.117%
Monolmplicit
0.4035
0.4261
2.26%
Bilmplicit 0.4184
0,4341
1..7%%
We investigate the paradigm's effect by com- paring the model's performance on each irregu- lar quadruple situation. As shown in Table 6 our model's improvement in all of the irregular classes surpasses the basic situation when com- pared with OTG. This result indicates that our pre- train method significantly improves the model's performance with a burst in generalization and robustness against irregular sentiment quadruples, which accomplish the foundation for building an opinion tree and should be taken into consideration apart from improving the structural awareness.
Table 6: The average performance of different situations in Restaurant and Laptop domain.
Method
Restaurant
Laptop
OTG
0.6164
0.4394
Asppect
0.6251
0.4439
Aspect, Opinion
0.6275
0,4447
Aspect, Opinton,
0.6294
0.4473
Puir, Triple
Asppect Opinion, category
0.6294
0.4473
polarity, Pair, Triple
Aspeet, Opanion, category
polarity, Pair. Triple
0.6339
0.4490
Simelume
We then investigate the impact of subtasks in TGP paradigm. We remove the subtasks in the paradigms gradually. Table 7 shows the result for Task Decomposition paradigm: the contributions of subtasks stay in similar scope, among which the Aspect surpasses others with tiny gap, this may due to the lower implicit rate of aspect terms?. In addition, all the subtasks are beneficial to ex- tract the opinion tree. It is worth noting that, the par- ticipation of finetune task Finetune demonstrates an obviously positive effect in both paradigms, which improves two domains with an average of 0.31%, 11, this phenomenon gives us conclusion that adding the finetune task in the pre-train phase is effective solution for narrowing the gap between them.
Table 7: Impact subtasks in Task-level Graph Pre- training paradigm.
this situation, there will be one intuitive question: Whether the element-level masking design does achieve a performance better than the indiscrini- nate paradigm as we expect?
We investigate this question by employing abla- tion experiments. We first design indiscriminate paradigm under similar settings, then we give the performance of using different paradigms in Ta- ble 5. As we can see, our element-level paradigm outperforms the indiscriminate paradigm, this re- sult shows the superiority of our element-level masking design, and also validated our motivation: for target graphs that contain limited knowledge like opinion tree, indiscriminate masking strategies would be sub-optimal and fine-grained masking should be adopted.
7 Conclusion
In this study, we propose two novel pre-train paradigms for opinion tree generation, which are designed depending on the intrinsic characteristics of the opinion tree. Specifically, the Element-level Graph Pre- Pre-trainin to--ttriiing paradigm abandons the strategy of capturing the complex structure but focuses di- rectly on the core elements. While the Task-level Graph Pre-training explicitly focuses on improving the generalization and robustness against irregu- lar quadruples with an easy-to-hard routine. Fur- thermore, we explore dynamic masking rate and cyclical train method for jointly combining the pre-training paradigms in order to bridge the gap between the pre-training and finetuning phases in modeling structural knowledge.
We then investigate the impact of subtasks in EGP paradigm. We add the subtasks in paradigm gradually. As we can see in Table 5, the sub- task pair of EPG5 and EPG4 (+All Ps)s) con- tributes the most to the performance (0.58% and 0.42% in each domain respectively), which aims to integrate the complementary information from both formations to generate text and tree respectively, indicating the significance of the complementary association.
6.3 Effeet of Task-level Graph Pre-training
As shown in Table 6, the OTG model obviously be short in its generalization and robustness against ir- regular sentiment quadruples when compared with
"The average implicit rate of aspect term and opinion term is 22.63% 33% and 24. 19% respectively

---

Experimental results show that our proposed model can achieve state-of-the-art performance in ABSA. In addition, the results also validate that, for target graphs that contain certain knowledge like opinion tree, the improving strategy should be made based on the intrinsic characteristics of the structure instead of treating it as plain graph.
of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 340-350, Online. Association for Computational Linguisties.
Chenhua Chen, Zhiyang Teng, Zhongqing Wang, and Yue Zhang. 2022a. Discrete opinion tree induction for aspect-based sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics (Volume I: Long Papers), pages 2051-2064, Dublin, Ireland. Association for Computational Linguistics.
Limitations
The limitations of our work can be stated from three perspectives. First, our pre-training method contains many subtasks that will consume vast com- putational cost during pre-train (the inference cost will not change). If possible, further work should try to explore time-saving pre-training method. Secondly, more tasks could be further explored, including cross-domain and cross-lingo sentiment analysis tasks. Finally, we focus on opinion tree generation in one major language. The perfor- mance of other languages remains unknown.
Hao Chen, Zepeng Zhai, Fangxiang Feng, Ruifan Li, and Xiaojie Wang. 2022b. Enhanced multi- channec graph convolutional network for aspect sentiment triplet extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers). pages 2974-2985, Dublin, Ireland. Association for Computational Lin- guistics.
Mengting Hu, Yike Wu, Hang Gao, Yinhao Bai, and Shiwan Zhao. 2022. Improving aspect sentiment quad prediction via template-order data augmenta- tion. In Proceedingy of the 2022 Conference on Em- pirical Methods in Natural Language Processing. pages 7889-7900, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
References
Xuefeng Bai, Yulong Chen, and Yue Zhang. 2022. Graph pre-training for AMR parsing and generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume
Mengting Hu, Shiwan Zhao, Li Zhang, Keke Cai, Zhong Su, Renhong Cheng, and Xiaowei Shen. 2019. CAN: Constrained attention networks for multi multi-aspect sen- timent analysis. In Proceedings of the 2019 Confer- ence Empirical Methods in Natural Language Pro- cessing and the 9th Internatiooal Joint Conference on Natural Language Processing (EMNLPP-ICNILP), pages 4601-4610, Hong Kong, China, Association for Computational Linguistics.
I: Long Papers). pages 6001-6015, Dublin, Ireland. Association for Computational Linguisties.
Xiaoyi Bao, Xiaotong Jiang. Zhongging Wang, Yue Zhang, and Guodong Zhou. 2023. Opinion tree pars- ing for aspect-based sentiment analysis.
Xiaoyi Bao. Zhongqing Wang, Xiaotong Jiang, Rong Xiao, and Shoushan Li. 2022. Aspect-based senti- ment analysis with opinion tree generation. In Pro- ceedings of the Thirty-First International Joint Con- ference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022, pages 4044-4050. ijcai.org.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings.
Junjie Li. Jianfei Yu, and Rui Xia. 2022. Genera- tive cross- doos-- data augmentation for aspect and opinion co-extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4219-4229, Seattle, United States. Association for Computational Lin- guistics.
Jiahao Bu, Lei Ren, Shuang Zheng, Yang Yang, Jingang Wang. Fuzheng Zhang, and Wci Wu. 2021. ASAP: A Chinese review dataset towards aspect category senti- ment analysis and rating prediction. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 2069-2079, Online. Association for Computational Linguistics.
Jian Liu, Zhiyang Teng, Leyang Cui, Hanmeng Liu, and Yue Zhang, 2021. Solving aspect category sentiment analysis as text generation task. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 4406-4416, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Hongjie Cai, Yaofeng Tu, Xiangsheng Zhou, Jianfei Yu. and Rui Xia. 2020. Aspect-category based senti- ment analysis with hierarchical graph convolutional network. In Proceedings of the 28th International Conference on Computational Linguistics, pages 833- 843, Barcelona, Spain (Online). International Com- mittee on Computational Linguistics.
Yue Mao, Yi Shen, Jingchao Yang, Xiaoying Zhu, and Longjun Cai. 2022. Seq2Path: Generating sentiment tuples as paths of a tree. In Findings of the Asso- ciation for Computational Linguistics: ACI. 2022,
Hongjie Cai, Rui Xia, and Jianfei Yu. 2021. Aspect- quadruple extraction with implicit aspects and opinions. In Proceedings

---

pages 2215-2225, Dublin, Ireland. Association for Computational Linguistics.
Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguistics.
Rajdeep Mukherjee, Tapas Nayak, Yash Butala, Sourangshu Bhattacharya, and Pawan Goyal. 2021. frree free decoding framework using pointer networks for aspect sentiment triplet extrac- tion. In Proceedings of the 202/ Conference on Em- pirical Methods in Natural Language Processing, pages 9279-9291, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020. Position-aware tagging for aspect sentiment triplet extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2339-2349, Online. Association for Computational Linguistics.
PASTE: A tagging-
Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021. A unified generative framework for aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computationall Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2416-2429, Online. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. CoRR, abs/2203.02155
instr
Mi Zhang and Tieyun Qian. 2020. Convolution over hi- erarchical syntactic and lexical graphs for aspect level sentiment analysis. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 3540-3549, Online. As- sociation for Computational Linguistics.
Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8600-8607.
Wenxuan Zhang. Yang Deng, Xin Li, Yifei Yuan, Li- dong Bing, and Wai Lam. 2021a. Aspect sentiment quad prediction as paraphrase generation. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 9209- 9219, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Guang Qiu, Bing Liu, Jiajun Bu, and Chun Chen. 2011. Opinion Word Expansion and Target Extrac- tion through Double Propagation. Computational Linguistics, 37(1):9-27.
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena. Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-texi transformer. Journal of Machine Learning Research,
Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021b. Towards generative aspect-based sentiment analysis. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers), pages 504-510, Online. Association for Computational Linguistics.
(1(10)):1667.
Ronald Scoh, Ian Birle, Mrinal Tak, Haw-Shiuan Chang, Brian Pinette, and Alfred Hough. 2021. Open aspect target sentiment classification with natural language prompts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 631 1-6322, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Zheng Zhang, Zili Zhou, and Yanna Wang. 2022. SSEGCN: Syntactic and semantic enhanced graph convolutional network for aspect-based sentiment analysis. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages
Tang,ang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016. Effective Istms for surget-eppndent sentiment classification. In COLING 2016, pages
Seattle, United States. Association for Computational Linguisties.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
A Building Procedure
In Task-level Graph Pre-training paradigm, the sub- tasks are set to follow the routine of building the opinion tree from scratch. For building an opinion tree manually, humans often learn to find funda- mental elements, such as aspects or opinions, fol- lowed by finding the corresponding classification result such as category and polarity to build a sin- gle quadruple unit, then composing multiple units to fulfill a more challenging goal, i.., writing the entire opinion tree.
Hai Wan, Yufei Yang, Jianfeng Du, Yanan Liu, Kunxun Qi. and Jeff Z. Pan. 2020. Targee-asppet--entinn joint detection for aspect-based sentiment analysis. In AAA/ 2020, pages 9122-9129.
Qianlong Wang, Zhiyuan Wen, Qin Zhao, Min Yang. and Ruifeng Xu. 2021. Progressive self-training with discriminator for aspect term extraction. In Proceed- ings of the 2021 Conference on Empiricul Methods in Natural Language Processing, pages 257-268,

---

Based on the process introduced, we design the subtasks in Task-level Graph Pre-training paradigm. Each subtask corresponds to mapping the steps for manually building an opinion tree from scratch. The paradigm consists of six subtasks: Aspect, Opimion, Category, Polarity.Pat Pair and Triple. Their prompts and target graph can be found in Fig- ure 7. Among which, Aspect and Opinion focus searching the basic elements of each quadruple:
Aspect: Extract all the aspect terms in the review in the form of tree, Figure 7 (a).
Opinion: Extract all the Opinion terms in the review in the form of tree, Figure 7 (b).
Category and Polarity further explore the clas- sification results with the corresponding basic ele- ments:
Category: On the base of Aspect, extract the category classification result of the aspect terms in the review in the form of a tree, Fig- ure 7 (c).
Polarity: On the base of Opinion, extract the polarity classification result of the opin- ion terms the review in the form of tree. Figure 7 (d).
Pair and Triple fulfill the mapping between quadruples.
Pair: On the base of Aspect and Opinion, map the corresponding aspect term and opin- ion term within a quadruple, Figure 7 (e).
Triple: On the base of Aspect and Polarily, map the corresponding aspect term and opin- ion term and polarity within a quadruple, Fig- ure (f).

---

[Aspect] ROOT
[Opinion]
ROOT
Aspect
Aspect apps
Opinion
Opinion
surface
smooth
hard
(a) Aspect Extraction
(b) Opinion Extraction
aspect
uspect
opinion
opinion
[ategory[[Aspecl
|Polarity| (atiity///ppiooniniii
ROOT
ROOT
Aspect
Aspect
Opinion Opinion
(c) Category Classification
Design
Software
Positive
Negative
category
category
polarity
polarity
(d) Polarity Classification
surface
surface aspect
smooth opinion
hard
aspect
opinion
[Aspect] [Opinion]
ROOT
Quad
Quad
Aspect Opinion
Aspect Opinion
Pair Mapping
surface smooth
apps aspect opinion
hard
aspect opinion
[Aspect][Opinion Pollarity
ROOT
Quad
Quad
Aspect
Opinion
Aspoct
Opinion
(f) Triple Mapping
surface
Positive
apps
Negative
aspect
polarity
aspect
polarity
smooth opinion
hard
opinion
[ree] ROOT
Quad
Quad
Aspect Opinion
Aspect
Opinion
gg) Finetune
Design Positive
Software Negative
category polarity category polarity
surface smooth apps hard
aspect
opinion
aspect
opinion
Figure Building procedure of subtasks in Task-level Graph Pre- Pre-training.


Reviews:
[{"review_id": "9FGLh5Sivd", "reviewer": "Reviewer_Jvv2", "summary": "", "questions": "1. In the main results, LLaMa-7B prominently performs better than ChatGPT. Generally, ChatGPT prominently outperforms LLaMa-7B, and 7B size is also not large enough. The main results show that after fine-tuning, LLaMa-7B can prominently performs better than ChatGPT. Could you introduce more details about the implementations and discuss more? e.g., the fine-tuning settings, the prompts used to these LLMs, should we use fine-tuned smaller LLMs rather than larger LLMs without fine-tuning?, etc.  ", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Dear reviewer,\n\n#### We really appreciate your valuable reviews, we will respond to your review one by one as follows: \n\n> 1. The techniques involved in the proposed framework have been commonly used for ABSA, including graph pre-training, opinion tree generation and so on, and it seems not surprising enough to combine them together. The experimental results only show the performance can be improved, but lack of the explanations of why the performance can be improved.\n\n#### **Response:** The motivation of our work is applying the pretraining method to different levels of ABSA graph while other graph pretraining works stick to treating the structured sequence as plain text, which is an obviously rough method that ignores the intrinsic characteristic of the graph itself. Hence, a more refined strategy regarding the target graph is the work we are motivated to do and contributes to the improvement of the performance. \n\n> 2. The experimental results are not exactly convincing, by comparing with the main results in [Bao IJCAI 2022] and [Bao ACL Finding 2023]. For example,\n> - For the scores of OTG method, [Bao ACL Finding 2023] < this paper < [Bao IJCAI 2022]. Note that this is a significant difference, for example, on the Restaurant dataset, for F1 score, [Bao ACL Finding 2023] 0.6040 < this paper 0.6164 < [Bao IJCAI 2022] 0.6283; on the laptop dataset, for F1 score, [Bao ACL Finding 2023] 0.3998 < this paper 0.4394 < [Bao IJCAI 2022] 0.4544\n> - On the laptop dataset, for F1 score, although the scores of OTG method in this paper 0.4394 < the scores of proposed method in this paper 0.4512; The scores of OTG method in [Bao IJCAI 2022] 0.4544 > the scores of proposed method in this paper 0.4512;\n\n#### **Response:** \n\n- The difference between the OTG in [Bao IJCAI 2022] and  [Bao ACL Finding 2023] is because of their datasets,  [Bao ACL Finding 2023] deletes part of the data(around 1.5%) in the ACOS dataset, which makes their results can not be compared directly.\n- As [Bao IJCAI 2022] utilizes external notation resources from other areas such as the trained AMR model, we directly adopt the performance of OTG without pretraining from [Bao IJCAI 2022] for a fair comparison, we will add the explanation into the paper once been accepted.\n\n> 3. It could be convincing to discuss case studies and error studies to highlight the effectiveness of each proposed component. For example, this paper mentions that the Element-level Graph Pre-training abandons the strategy of capturing the complex structure but focuses directly on the core elements. However, without case study, it is less convincing to figure it out. An example of case study can be found in “Graph pre-training for AMR parsing and generation”.\n\n#### **Response:** Due to page limitation, we finally decided not to put the case study on the paper. We believe your review's suggestion can surely help show the effectiveness of our method and we would like to add it to the appendix if accepted.\n\n\n> 3. In the main results, LLaMa-7B prominently performs better than ChatGPT. Generally, ChatGPT prominently outperforms LLaMa-7B, and 7B size is also not large enough. The main results show that after fine-tuning, LLaMa-7B can prominently performs better than ChatGPT. Could you introduce more details about the implementations and discuss more? e.g., the fine-tuning settings, the prompts used to these LLMs, should we use fine-tuned smaller LLMs rather than larger LLMs without fine-tuning?, etc.\n\n#### **Response:** We fine-tuned LLaMA -7B with Lora, and inferred opinion tree on ChatGPT with zero-shot learning. Based on our experiment result, we believe that in the NLP tasks whose targets are not close to natural language (e.g. opinion tree generation, AMR parsing), finetuning a smaller LLM might be better than direct inferring on the larges as they are not pre-trained to do so.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The main idea is reasonable, and the manuscript is well-organized and presented.\n\n2. The Element-level Graph Pre-training paradigm focuses directly on the core elements (aspect terms, opinion words, etc.) enabling the capture of the relationships between core sentiment elements.\n\n3. The Task-level Graph Pre-training paradigm decomposes the quadruple extraction task into multiple subtasks to help the main task.\n", "reasons_to_reject": "1. An incremental study of this topic based on previous works.\n- [Bao IJCAI 2022] Aspect-based Sentiment Analysis with Opinion Tree Generation, IJCAI 2022 (OTG method)\n- [Bao ACL Finding 2023] Opinion Tree Parsing for Aspect-based Sentiment Analysis, Findings of ACL 2023\n\nThe techniques involved in the proposed framework have been commonly used for ABSA, including graph pre-training, opinion tree generation and so on, and it seems not surprising enough to combine them together. The experimental results only show the performance can be improved, but lack of the explanations of why the performance can be improved. \n\n\n2. The experimental results are not exactly convincing, by comparing with the main results in [Bao IJCAI 2022] and [Bao ACL Finding 2023]. For example, \n\n- For the scores of OTG method, [Bao ACL Finding 2023] < this paper < [Bao IJCAI 2022]. Note that this is a significant difference, for example, on the Restaurant dataset, for F1 score, [Bao ACL Finding 2023] 0.6040 < this paper 0.6164 < [Bao IJCAI 2022] 0.6283; on the laptop dataset, for F1 score, [Bao ACL Finding 2023] 0.3998 < this paper 0.4394 < [Bao IJCAI 2022] 0.4544\n\n- On the laptop dataset, for F1 score, although the scores of OTG method in this paper 0.4394 < the scores of proposed method in this paper 0.4512; The scores of OTG method in [Bao IJCAI 2022] 0.4544 > the scores of proposed method in this paper 0.4512; \n\nThere are also other significant differences on the performance of the baseline methods in these papers. \n\n\n3. It could be convincing to discuss case studies and error studies to highlight the effectiveness of each proposed component. For example, this paper mentions that the Element-level Graph Pre-training abandons the strategy of capturing the complex structure but focuses directly on the core elements. However, without case study, it is less convincing to figure it out. An example of case study can be found in “Graph pre-training for AMR parsing and generation”. \n\n", "questions_to_authors": "1. In the main results, LLaMa-7B prominently performs better than ChatGPT. Generally, ChatGPT prominently outperforms LLaMa-7B, and 7B size is also not large enough. The main results show that after fine-tuning, LLaMa-7B can prominently performs better than ChatGPT. Could you introduce more details about the implementations and discuss more? e.g., the fine-tuning settings, the prompts used to these LLMs, should we use fine-tuned smaller LLMs rather than larger LLMs without fine-tuning?, etc.  ", "paper_topic_and_main_contributions": "This paper proposes two pre-training paradigms on the opinion tree to improve the generation model for aspect-based sentiment analysis (ABSA) tasks, i.e., element-level graph pre-training and task-level graph pre-training. The Element-level Graph Pre-training focuses directly on the core sentiment elements of the ABSA tasks, and it uses a sentiment element-level masking strategy. The Task-level Graph Pre-training decomposes the quadruple extraction task into multiple subtasks for building an opinion tree from scratch and employs the learned knowledge of each subtask for the main task. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- In L320, there is no explanation for the acronym \"OTG\". \n\n- In Section 6.1, a typo in the sentence \"Among which TGP paradigm’s contribution outperforms TGP paradigm,\" \n"}, "extra_text": {}}, {"review_id": "wDIZoZXZr9", "reviewer": "Reviewer_JuKD", "summary": "", "questions": "1. In Line 125, the paper claimed \"several benchmark datasets\", maybe the \"ACOS dataset\" is better.\n2. In lines 517-520, the authors claimed, \"the Task-level Graph Pre-training explicitly focuses on improving the generalization and robustness ...\" . Regarding experiments, generalization, and robustness have not been presented yet. Perhaps experiments with more tasks or more datasets could be added.\n3. In Fig. 5, for the prompt of different sub-task, is it still effective to use the numbers 0-5 as prompts for different subtasks? As the displayed prompt is just a tag.\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Dear reviewer,\n\n#### We appreciate your valuable reviews, we will respond to your review one by one as follows: \n\n> 1. Experiments were performed on the ACOS dataset only, and more datasets can be added.\n\n#### **Response:** For the aspect-level sentiment task, the ACOS dataset is the only dataset that contains the full elements(aspect, opinion, category, polarity) we are motivated to focus on with a refined element-level pretraining method. Besides, the ACOS dataset has already covered the previous SemEval Res/Lap 14/15/16 dataset, making it comprehensive for validating the method's performance.\n\n> 2. As claimed in the Limitation, the computational cost of so many tasks during pre-training is huge.\n\n#### **Response:** Compared with other pretraining methods such as the joint pre-training method used in [Bao IJCAI 2022], which involved not only ABSA annotations but also AMR and dependence trees, our method has already saved the computational cost as it only needs the in-task annotation cost.\n\n- [Bao IJCAI 2022] Aspect-based Sentiment Analysis with Opinion Tree Generation, IJCAI 2022 (OTG method)\n\n> In Line 125, the paper claimed \"several benchmark datasets\", maybe the \"ACOS dataset\" is better.\n#### **Response:** We appreciate your suggestion, and we will correct it once accepted.\n\n\n> In lines 517-520, the authors claimed, \"the Task-level Graph Pre-training explicitly focuses on improving the generalization and robustness ...\" . Regarding experiments, generalization, and robustness have not been presented yet. Perhaps experiments with more tasks or more datasets could be added.\n#### **Response:** \n- The improvement of the generalization and robustness against irregular quadruples has been shown in Table 6 and explained in Section 6.3. \n\n- The Task-level Graph Pre-training can explicitly improve the model's generalization and robustness when facing the irregular quadruples as it builds the opinion tree following the build-from-scratch procedure, which means every irregular composition has been emphasized during the pre-training, such as one-to-many will be focused on in the subtask shown in Fig 5 line 5. \n\n\n>In Fig. 5, for the prompt of different sub-task, is it still effective to use the numbers 0-5 as prompts for different subtasks? As the displayed prompt is just a tag.\n#### **Response:** We haven't done the experiment yet, we believe even if it works, it will still cause a slight decrease in performance since the pre-trained language model will be unable to catch the semantic relationship with the prompts and specific tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors present two graph-based pre-training paradigms in detail;\n2. This paper gives a new perspective on the ABSA task in a pre-trained generational manner.\n", "reasons_to_reject": "1. Experiments were performed on the ACOS dataset only, and more datasets can be added.\n2. As claimed in the Limitation, the computational cost of so many tasks during pre-training is huge.", "questions_to_authors": "1. In Line 125, the paper claimed \"several benchmark datasets\", maybe the \"ACOS dataset\" is better.\n2. In lines 517-520, the authors claimed, \"the Task-level Graph Pre-training explicitly focuses on improving the generalization and robustness ...\" . Regarding experiments, generalization, and robustness have not been presented yet. Perhaps experiments with more tasks or more datasets could be added.\n3. In Fig. 5, for the prompt of different sub-task, is it still effective to use the numbers 0-5 as prompts for different subtasks? As the displayed prompt is just a tag.\n", "paper_topic_and_main_contributions": "This paper focuses on the ABSA task in a generational way. The authors present two pre-training paradigms for opinion tree generation, including Element-level and Task-level Graph Pre-training. Experimental results validate the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. line 021, \"validate\" should be \"and validate\";\n2. line 164-165, \"propose\" and \"employ'\"should be \"proposed\" and \"employed\";\n3. line 174, \"without not ...\" is confusing;\n"}, "extra_text": {}}, {"review_id": "077QFIgF86", "reviewer": "Reviewer_LibN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "#### Dear reviewer,\n\n#### We appreciate your valuable reviews, we will respond to your review one by one as follows: \n\n> This paper only use ACOS dataset (Cai et al., 2021) for experiments. Experiments on more datasets will make this paper more valuable.\n\n#### **Response:** For the aspect-level sentiment task, the ACOS dataset is the only dataset that contains the full elements(aspect, opinion, category, polarity) we are motivated to focus on with a refined element-level pretraining method. Besides, the ACOS dataset has already covered the previous SemEval Res/Lap 14/15/16 dataset, making it comprehensive for validating the method's performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a reasonable pre-training method to improve the model, and the experimental results verify the effectiveness of the method. The implementation described by the author is rich in details and reproducible.", "reasons_to_reject": "This paper only use ACOS dataset (Cai et al., 2021) for experiments. Experiments on more datasets will make this paper more valuable.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduce two pre-training paradigms to improve the generation model for Aspect-based sentiment analysis. \n\n1. This paper employ an Element-level Graph Pretraining paradigm, which is designed to improve the structure awareness of the generative model. \n2. This paper design a Task-level Graph Pre-training paradigm to make the generative model generalizable and robust against various irregular sentiment quadruples.\n\nThe experiment results show the superiority of our proposed method, validate the correctness of our motivation.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes two pre-training approaches to improve opinion tree generation for aspect-based sentiment analysis. The proposed method is sound validated by experiments. However, the experiments have been performed only on one dataset (ACOS) while more extensive evaluation is needed. Morever, the reviews highlighted that lack of extensive explanations on why the performance can be improved with the proposed method.
Decision: Accept-Findings