SIR-ABSC: Incorporating Syntax into RoBERTa-based Sentiment Analysis Models with a Special Aggregator Token
Authors: IKHYUN CHO
Yoonhwa Jung
Julia Hockenmaier
Venue: EMNLP 2023 Findings (2023)
Keywords: Aspect-based sentiment analysis
Pre-trained Language Models
RoBERTa
TL;DR: A syntax-integrated RoBERTa-based model for aspect-based sentiment analysis.

Abstract: We present a simple, but effective method to incorporate syntactic dependency information directly into transformer-based language models (e.g. RoBERTa) for tasks such as Aspect-Based Sentiment Classification (ABSC), where the desired output depends on specific input tokens. In contrast to prior approaches to ABSC that capture syntax by combining language models with graph neural networks over dependency trees, our model, Syntax-Integrated RoBERTa for ABSC (SIR-ABSC) incorporates syntax directly into the language model by using a novel aggregator token. Yet, SIR-ABSC outperforms these more complex models, yielding new state-of-the-art results on ABSC.

SIR-ABSC: Incorporating Syntax into RoBERTaabased Sentiment Analysis Models with Special Aggregator Token
Ikhyun Cho and Yoonhwa Jung and Julia Hockenmaier University of Illinois Urbana- Sabana-Chammpain (ihcho2, yoonhwa2, juliahmr) ) @ @il@lllii alllioiis.... edu
Abstract
graph neural network (GNN) obtained from the dependency graph (Tang et al., 2020; Wang et al., 2020; Hou et al., 2021; Xiao et al., 2022). However, such combined models have two major limitations:
We present simple, but effective method to in- corporate syntactic dependency information di- rectly into transformer-based language models (e.g. RoBERTa) for tasks such as Aspect- Based Sentiment Classification (ABSC), where the de- sired output depends specific input tokens. In contrast to prior approaches to ABSC that capture syntax by combining language models with graph neural networks over dependency trees, our model, cyntax-Integrated RoBERTa for ABSC (SIR-ABSC) incorporates syntax directly into the language model by using novel aggregator token. SIR-ABSC' outper- forms these more complex models, yielding new state-of thee----fheee-eeeeeeee art results on ABSC.
1. Suboptimal Interaction: A &ypical challenge in combining PLMs and GNNs is to make the models effectively interact with each other. Some approaches (Tang et al., 2020; Lu et al., 2020) attempt to accomplish this through heavy model architecture engineering. However, the PLM and GNN still operate in an asynchronous manner (one after the other), limiting their interaction, and yielding only minor improvement in performance. We hypothesize that more integrated models yield larger boosts in performance.
1 Introduction
Espect-Based Sentiment Classification (ABSC, Pontiki et al. (2014), Figure 1) is a fine-grained sentiment analysis task that aims to handle the fact that even simple statements such as "The ambience was nice, but service wasn't 't so great." may express different sentiments towards different aspects (this reviewer is positive about the restaurant's "ambi- ence", but negative about its "service"). In ABSC, the aspect to be classified identified by a target string the input sentence (e.g. "ambience") "), and systems have to return the polarity (positive, neu- tral, negative) of the corresponding sentiment.
Suboptimal Aggregation: Existing (PLM GNN) models cannot emphasize information at specific distance(s) away from the target token. This is problematic since the models pre-define the number of GNN layers before- hand (Bai et al., 2020; Veyseh et al., 2020: Zhao et al., 2022). For example, the number GNN layers is fixed to three while an input sequence has the key sentiment word at a dis- tance of one, the redundant second and third GNN layers can introduce noise. The ability to identify the most important distance(s) (one in this case) and to focus on that specific dis- tance(s) based on the input crucial reduce such noise.
Pre-trained language models (PLMs) have been shown to work well for ABSC, presumably be- cause their attention mechanisms capture semantic connections between target and context words (Li al., 2019; Xu al., 2020; Karimi et al., 2021). Starting with Do al. (2019), PLMs have been supplemented with syntactic features, typically ex- tracted from dependency graphs. This is typically done by using the word embeddings obtained from the PLM to initialize the node embeddings of a
In order to alleviate these limitations, we propose syntax-integrated RoBERTa ABIR-A a novel framework for effectively augmenting PLMs with syntactic information. We chose RoBERTa (Liu et al., 2019) PLMM baseline due its notable performance on ABSC (Dai al., 2021).
Instead of stacking multiple GNN layers on top of a PLM as in most previous works, we insert a
'The code for SIR-ABSC is publicly availahle at https: //g thub. com/ hcchh RBAABC

---

Dependency tree
Aspect-Based
sutkee
Sentiment Clsssficatioon
Input: Aithough Vista compered to XP sucked
athough
(ABSC)
Targel sepect: Vista
Targel sejrct: e Senimmen: = positive
Syntactic distance to 'vista'
Sentiment negative
suckked
aitherugh
Surface distance from target
Syntactic Distance to 'PP"
compares
xp
although
compared
Ap
surbed
Syntactic distance target
3ttoouge
comppared
itthough
Sucked
although
Detanoe
Ditence:
Figure (Left Top) ABSC, sentiment to be predicted depends on desired target aspect (words from the input). (Left Bottom) For ABSC, syntactic distance can be more informative than surface distance. (Right) Syntactic distances as defined by dependency tree
new [g] token into the PLM that acts as syntax aggregator and effectively replaces the work of
or neutral) of the sentiment in input sentence [11 W22 ..., up.... Wp+m-1: Wn) towards given target aspect & (a substring of the input sentence: fj = (ppp---.-p4mm1))
GNN layers. Variable Distance Control (VDC) mechanism allows [g] to focus on capturing syntac- tic knowledge by using constrained attention masks that reflect the graph structure. A Dependency- Aware Aggregation (DAA) mechanism leverages dependency label information. We further improve model with Automatic VDC Learning (Auto- VDC), which enables SIR-ABSC capture and focus on important syntactic distances based on the input, an important advantage over GNNs. SIR- ABSC outperforms prior approaches, and estab- lishes new state of the art, on the most widely used ABSC datasets.
2.1 Language Models for ABSC
Pre-trained language models (PLMs) such as BERT (Devlin et al., 2019) and RoBERTa (Liu al., 2019) have gained predominance for many NLP tasks, including ABSC. RoBERTa, variant of BERT, is known to show notable performance on ABSC tasks (Dai et al., 2021), and forms the basis of the models explored in this paper. RoBERTa (and BERT) are (pre)trained on large amounts of raw text with masked language modeling objec- tive. Both models use Transformer (Vaswani et al., 2017) architecture in which each token's 's em- bedding is fed through multiple layers such that each token's 's embedding in a given layer can attend to tokens in the sequence (in the same layer). To adapt these models for classification tasks, special token ([CLS] for BERT, [s] for RoBERTa) whose output is fed into task- sseecicc feedforward layer is included in the input sequence. A separation to- ken ([SEP) or (/s]) can be used to separate the input sequence from other task-specific information.
Our main contributions are summarized below:
We present a novel approach to incorporating syntactic information into PLM through the use of a syntax aggregator token.
We highlight two limitations in current GNN- based approaches (suboptimal interaction and aggregation) and present effective methods (DAA and Auto-VDC) to address them.
To our knowledge, this is the first work to utilize dependency graph information without resorting GNNs in ABSC.
For the ABSC task, RoBERTa is typically used as follows: after tokenization, the input sentence is constructed as '[s] input sentence [/s] [/s] aspect sequence [/s]', where the aspect sequence includes the target aspect itself. The [s] token of the last layer is used for the final prediction and fine-tuning.
Our model achieves state-of-tte -fte--of-thcar results on two out the four widely used ABSC datasets and demonstrates competitive performance on the remaining ones.
2 Aspect-Basee Sentiment Classification
2.2 Combining PLMs with syntax
ABSC, illustrated Figure 1, the task is to predict the polarity (positive, negative
A common approach to ABSC is to supplement a PLM with syntactic information obtained from a

---

dependency parser. dependency graph (Figure 1) each word in the sentence corresponds node, with labeled edges indicating word-word depen- dencies. Note that the syntactic distance between words (e.g. sucked and vista) can be much smaller than their surface (string) distance (Figure 1).
2022a). What is common to all these approaches is that the PLM and GNN operate in a serial fashion, and are not tightly integrated.
Attention-mask based approaches Another promising approach to incorporate syntactic infor- mation into PLMs that is more related to this paper. is to manipulate the Transformer's 's self-attention masks. For example, Syntax-BERT (Bai et al., 2021) uses multiple masks induced from the syn- tactic trees (e.g., parent, children, sibling, pairwise masks) incorporate syntactic information into BERT. To do so, it requires multiple (usually more than 90) sub-networks, which share the same model parameters in a masked self-attention module, and then the outputs are aggregated through another attention layer, named a topical attention module.
Since the dependency parser and the PLM may use different tokenizers, tokenization needs to be broken into two stages to integrate both models seamlessly. The input sentence is first tokenized by the dependency parser, and then each token is again tokenized by RoBERTa's tokenizer, follow- ing previous work (Tang et al., 2020).
Graph Neural Network-based ABSC models To incorporate syntax into ABSC models, PLMs have been augmented with Graph Neural Networks (GNNs, Kipf and Welling (2016)) that capture the structure of the sentence's 's dependency tree. Al- though there are many variants (Trisna and Jie. 2022), the basic idea behind GNNs is to represent each node as a vector h that is updated via graph convolution in each layer (1 E 21. L)) of the GNN (Kipf and Welling, 2016) by aggregating its neighborhood information from the previous layer:
The key difference between Syntax-BERT and SIR-ABSS AE-ABBCC is that Syntax- alters all the in- put tokens' attention masks and aggregates them in a serial manner, while SIR-ABSC (which is specifically designed for tasks like ABSC, where the desired output depends on specific parts of the input) keeps the original input tokens intact while only modifying the attention-mask of the newly added [g] token.
h = a (As + bi)
SIR-ABSC
a is an activation function, W and b are learnable parameters, and Aly is the entry of the graphs adja- cency matrix that indicates whether nodes i and are connected (in which case Aij 1; otherwise Aij 0). The general framework for augment- ing PLMS with GNNs for ABSC is to initialize the GNN node embeddings with the PLM's output em- beddings and use the final embeddings of the target aspects in the last GNN layer for classification. Zhang et al. (2019) were the first to implement a GNN-based model for ABSC, adding multi- layered Graph Convolutional Network (GCN) to encode dependency graphs on top of the word em- bedding layer. Wang al. (2020) and Bai al. (2020) proposed a relational graph attention net- work (R-GAT), which computes an additional at- tention distribution using the dependency label em- beddings. Tang et al. (2020) strengthened interac- tions between contextual and graph representations through mutual biaffine module. Mei et al. (2023) incorporated part-of-speech, distance, and syntactic dependency supervised manner. Several recent research in ABSC has tried to revise dependency graphs due to their imperfections or add external resources (Xiao et al., 2021, 2022; Liang et al.,
We present SIR-ABSC (Figure 2), which incorpo- rates syntactic information into PLM designed for ABSC tasks. We accomplish this goal by augment- ing RoBERTa with four components: (1) a single additional input token, named [g]. whose attention masks depend on the structure of the input's 's depen- dency tree(s), paired with (2) "variable distance control" mechanism that specifies how the struc- ture of the dependency graph is reflected in [gl's attention masks, (3) dependency-awart aggrega- tion which incorporates dependency labels using embeddings, and (4) an VDC" mechanism that learns the ideal VDC for given input.
Input and output After tokenization with RoBERTa's tokenizer, the input to SIR-ABSC is "[s] [g] input sentence [/s] [/s] aspect sequence [/s]', where the aspect sequence is the token "Ig]" followed by the target aspect words. [g] uses its own independent dictionary embedding in the input layer but is initialized with the initial [s] embedding. We evaluate this choice in Section 5.
To obtain the output, the final layers of the [s] and [g] tokens are pooled and fed to a classification

---

Semartics eggregation
Sentimenc Polarity
[g]
token's layer-wise attention masks
Syntaa aggragation
(lower 12y
VDC
strwadh
corpcred
IN
GNN
Jayer
comprred
scce
IN]
P41
vite
Asynchronous Aggregation
Synchronous Aggregation
PLM
VDc Jape
6)
PLM
athoug
compared is no mutes
(b)
(S]
Vi
vote
VDc Beeer 80
aFhoogg
Vile
compured
to
ao aucked
W
wota
[+] Existing (PLM-GNN)
(b) Our approach
approaches
The overall architecture of our approach. Instead of stacking GNNs on top of a PLM (left), SIR-ABSC inserts syntax aggregator token, [g]. into the PLM for synchronous and deeper aggregation (right). heuristic VDC of
Figure 2:
is used in this example ("vista" is the target aspect). DAA is not explicitly represented for the sake of simplicity. Best viewed in color.
layer as follows. First, global vector is created by averaging [s] and [g] embeddings followed by weight We and an activation function f:
For our baseline model, we assume that each layer I of [g] is associated with specific dis- tance dj. This yields layer-specific attention mask d=d; m (a vector whose elements m are 0 if the distance D(j) between token j and the target aspect words is greater than d,, and otherwise). If distance is syntax-based, D(j) is the length of the shortest path between token and the target aspect (so, if the target aspect consists of multiple tokens, take the minimum distance to any of its compo- nent tokens). If the distance is surface-based, D(j) simply the token distance the target aspect. To summarize, all tokens except [g] are updated precisely as in the original PLM using the original attention mask m, and [g] is updated using its own attention mask m
c-5(2(x, X,) Then, the final pooled vector (h) is:
h a1 022
where,
(a1. 02) softmax(cX) cx,))
Then, hs is fed into a fully connected layer (Wrc)) to calculate the final sentiment polarity score E RC,
y Sooftmax(hWrre
d=d, mo- as follows:
where C is the number of sentiment polarity classes. We use the cross-entropy loss function as the ob- jective function.
OKKTT
x out softmax
+mm
Vl
(1)
Va 02K
The [g] token and distance-based attention masks The [g] token is inserted next the [s] token. Unlike the [s] token that attends to every token in the input, we allow each layer I of [g] to only attend to the subset of input tokens j that are at most a distance D(j)= S dj away from the target aspect, allowing us to capture the intuition that the relevance of each word in a sentence to ABSC de- pends on its distance to the target aspect words. We achieve this with bastacee-based attention masks (Figure that depend either on syntactic or surface distance, and are only used for the [g] token, and the VDC hyperparameters explained below that de- fine the parameter d,. We do not restrict how the input tokens can attend to [g].
x out softmax
+ m
mo=d
d=
vl (2)
Vd
where,
Q' - xt W. K² = xx w vl x! wi
if D()) & de inf otherwise
d=d;
m g.i
X and X are the input and output token embed- dings of the 1-th self-attention layer, m is the atten- tion mask that cancels out padding tokens, m md=d, the [g] token's 's attention mask, '', KKl, v are the I-th Query, Key, and Value weights, respectively,

---

is the token index, and 9 the [g] token's in- dex. For simplicity, we ignore the relatively minor details, such as the FFN and normalization layers.
formation as below:
X - f(x + Wopp) = =X Vi Wv
Unlike previous works where PLM and GNN operate one after the other, SIR-ABSC is a syn- chronous and fully-integrated model in that [s] (= semantic aggregator) and [g] (= syntactic aggre- gator) attend to and affect each other's em- beddings every layer. We verify that this fully- integrated characteristic is crucial for SIR-ABSC's performance in Section 5.
If is the activation function, Xdep is the dependency label embedding of the input tokens, Wdee is a learnable weight matrix that maps dependency la- bel embeddings to the size of the PLM's embed- ding. This allows SIR-ABSC leverage the de- pendency label information for better aggregation.
Variable Distance Control (VDC) To specify the attention masks used by the [g] token, we intro- named Variable Distance Control (VDC). VDC is list 12 non- negative integers where the /-th element represents the de value of the I-th layer of the [g] token: With VDC of 10.00000.000,111111111
Automatic VDC learning VDC) In Sec- tion 5, Table 2 shows the significance of the VDC configuration in SIR-ABSC, with up to 1.1 dif- ference in model performance. This result naturally gives rise to challenging question: How can we decide the ideal VDC configuration?
duce new set of hyper-parametto
the first six layers of the [g] token attend only to the target aspect, and the remaining six layers attend to tokens connected to the target via direct dependency link.
Given numerous possible configurations for VDC, it is inefficient to rely on heuristics such grid search. We therefore define learning-baa -based approach to finding the ideal VDC configuration by computing multiple [g] embeddings based on different VDC values and then merging them into one using a learnable weighted sum.
Note that increasing VDCs (e.g.. 10.0:.000.111111122222.211 (0.0...I.1.112222.33333333.) can be used to mimic how GNNs work. Through graph convolution, the i-th layer of GNN aggregates features of nodes up to length i away from each node in the graph, allowing the GNN to gradually aggregate information from more and more distant nodes in its upper layers. Empirical results in Section 5 show that increasing VDCs have indeed better performance than constant VDCs (e.g.. [2.222..22222222222222222. or decreasing Here, our
We first create multiple [g] candidates X where each uses the attention mask of VDC=i follows. First, the Qú vector is transformed Q (9.0) by weight W and an activation f:
[g.1}
Proo f(O'w!)
Then, X ((o3) is generated using attention mask of VDC of
VDCs (e.g. [3,3,3,,,2,,,11111111] baseline model uses heuristically defined VDCs, referred to as "heuristic-VDDe in this paper.
Q Clonn) Ret +
x X = softmax
+mg + v
vd
Dependency-aware aggregation (DAA) VDC allows the [g] token to distinguish tokens with dif- ferent syntactic distances. However, the model is unaware of the dependency edge labels which could contain valuable information. To consider re-formulate
Then, we use learnable weighted sum to create our final [gl vector X
x ob x [1g.0 +... of X(O
this information in SIR-ABSC, the update of the [g] token (Eq. 2) by including learnable embedding vectors corresponding to the dependency label to each token as follows:
where the weights are computed as follows. First, vector is computed by averaging X X followed by weight W and an activation function f:
t: x++1 (=1(1t1()
((mn)) Wt
c c=
OiR'T Vd
x xou = Softmax
+ m
Then, the weights are computed by applying the softmax over the dot products with c:
where x!
in: K and y in equation 2 are modified to X! K. and V to consider the dependency in- in:
(a/ ob) softmax(X) Xo,0) ,... x (o)))c))

---

Auto-VDC' is illustrated in Figures 3 and 4. this ap- proach has the following advantages: First, due to the attention pooling, the ideal VDC of each layer can be automatically learned by the attention scores. That is, x (g.*) with the ideal VDC = n for layer / will be automatically assigned higher weight. Fur- thermore, unlike heuristic-VDD where the model must attend tokens up distance of max (VDC), Auto-V VDC can adjust which distances to focus on based on the input, thereby reducing the noise from the redundant distances.
adjacency matrix generated based on the syntactic distance from the target; (7) MWM-GCN (Zhao et al., 2022) uses masking based on the syntactic distance from the target and additional multi-head self-attention layers for better performance; (8) SG- GCN (Veysch et al., 2020) applies a regulation gate on the tokens based on their similarity to the target; (9) MOBERTa-RGAII PWCN (Dai et al., 2021) produces the baseline RoBERTa model with using dependency graph in two different approaches (re- lational GAT and point-wise convolution network); (10) AM-RoBERTa (Feng et al., 2022b) applies target-relevant masking to the self-attention mech- anism in the PLM to focus more on target-related contexts; (11) AABERTa-DI.GM (Mei et al., 2023)
4 Experimental Results
Datasets and Experimental Settings We use the most widely used ABSC data sets: the Lap- top and Restaurant datasets from SemEval-2014 task (Pontiki et al., 2014), the Twitter dataset of Dong et al. (2014), and the MAMS-ATH A dataset (Jiang et al., 2019). Table 5 in Appendix A shows the statistics of the ABSC datasets. For STR-ABSC, we use the pre-trained RoBERTa-base model provided by huggingface. We use spaCy²'s en_core_weit www_wwbbb..... sm model version 3.3.0 the depen- dency parser. Finetuning uses batch size of 32, dropout rate of 0.1, learning rate of (1.0e-5, 1.5c-5, 2.0e-5) using the AdamW optimizer. and 300 for the dependency embedding dimension. We run the experiments with five random seeds and report av- erage accuracy and macro-F1. We used the same set of random seeds for each ablation experiment. All experiments are conducted on single Tesla A100 GPU.
proposes a supervision- based approach to incorpo- rate syntax information into PLM.
Note that baselines that use external knowledge sources in addition to dependency graphs (e.g., Sentic GCN-BERT (Liang et al., 2022b), KGAN (Zhong et al., 2023), SGAN (Yuan et al., 2022)) are not direct competitors to SIR-ABSC, although SIR- ABSC still outperforms all these methods except for KGAN on Twitter (which is not fair compari- son since KGAN relies on an external knowledge source, WordNet). BERT4GCN (Xiao et al., 2021) is excluded since it uses a different experiment set- ting while no public code is available. It is also worth noting that all these baseline models use GNN-based modules to aggregate syntactic infor- mation. SIR-ABSC, on the other hand, presents new aggregating approach via the syntax aggrega- tor token without resorting to GNNs.
Baselines We compare SIR-ABSC with previ- ous (PLM+Dependency graph) models in Table 1.
Overall Results Table compares SIR-ABSC against all competitive RoBERTa+GNN or BERT+GNN combination models that use depen- dency graphs extracted from parsers 234 (Tang al., 2020; Wang al., 2020; Xiao et al., 2022; Zhao et al., 2022; Liang al., 2022a: Veyseh al., 2020; Xiao et al., 2021; Dai et al., 2021; Feng et al., 2022a: Mei et al., 2023). Notably, SIR- ABSC outperforms all previous models on Lap- top14, Restaurant14, and MAMS-ATSA datasets, establishing a new sfate-of-the-aar record. On Twit- ter, SIR-ABSC clearly outperforms RoBERTa- based models and is competitive with the (overall better performing) BERT-based models. Signifi-
Specifically, (1) DGEDT-BERT (Tang et al., 2020) proposes a biaffine module that deepens the inter- action of semantic and syntactic representations; (2) kumaGCN (Chen et al., 2020) utilizes depen- dency graph along with a latent graph induced from self-attention neural networks; (3) dotGCN (Chen et al., 2022) uses GNN over induced tree trained by reinforcement learning: (4) CHGMAN (Niu et al., 2022) uses a three-channel multi-view learn- ing model on the dependency graph for better repre- sentation learning; (5) R-GAT (Wang et al., 2020) applies relational graph attention networks over the dependency graph to incorporate edge label in- formation into the model; (6) DGNN (Xiao et al., 2022) uses dependency graph and an additional EspaCy parsers:
Stanford CoreNLP: github. io/CoreeLP/ "Biaffine Parser (Dozat and Manning, 2016) from the Al- lenNLP package ht tps: / /aallenna wwl.enai..rgg/............ w:tg/aleennlllll...
https://stanfftt

---

layer 12-
202
011
012
layer 10
641
011
090
608
641
010
011
ltyer
layer
Inyer
038
322
616
853
0M
504
606
511
08
layer 5
6:1
031
001
013
849
090
805
841
03
031
1yyer
byer 621
221 038
845
031
501 030
Figure 3: With Auto-VDC, the [g] token automatically learns the ideal VDC configuration. For example, in the lower layers, the [g] token's 's attention scores focus on the target aspect ("touch screen functions") and the [s] token. As it moves to the upper layers, the [g] token shifts its focus to the critical phrase "did not enjoy" for the final prediction. This shows that Auto-VDC progressively consolidates information from increasingly distant nodes in the upper layers, just like heuristic VDCs and GNNs, but an automatic way, thereby diminishing the need for manual heuristic adjustments.
Base PLM Medels
External
Ast.
DOEDT-BERT' MIDGCC
Dop. Guaph Dkp kingh
79.8
75.6
96.3
50.0
77.9
15.4
67.00
76.3
19-4
1013
17.0
73.0
80:10
16-2
1K.1
Dep: Gruph
81.5
Grip)
Goog
Gingh
((Nezsichr))
SOKI SXGEN-BERI
(ingh
15.88
N2.2
12.5
Oune SIR-ABS (BERT))
Graph
K222220-1 31
20.1130.1
87.3/3107)
76N3672
KS KS
Bollutt REAAT!
Cinph
81.4
87.4
76.4
K5.
BoWN AMS-UGGEKIs"
kinph
54.11
87.4
75.6
83.0
1611
82.5
66.2
Dep. Graph
84.4
82.0
88.6
83.6
75.5
24.6
B4B
$4.3
KGAN
Dep. Grach
V(WeedNa)
63.9
91.1
28.5
84.1
S0.6
79.6
Oune SIR-ABSC"
Dap. Graps
35.1230.35
77,03500)
Table I: SIR-ABSC outperforms all prior works Laptop and Restaurant, and competitive on Twitter and MAMS.
cance tests following Dror al. (2019) result in minimum epsilon values of using p-value of p < 0.01, indicating that SIR-ABSC's performance is "stochastically greater" than the baselines (de- tails in Appendix E). We also conducted experi- ments using BERT (SIR-ABS (BERT)) instead of RoBERTa and it shows comparable performance with the state-of-the--at BERT models (details in Appendix G).
Analysis
We now examine the effect of the design decisions that distinguish SIR-ABSC from RoBERTa.
Does [g] require syntactic distances? To under- stand the impact of syntax SIR-ABSC, we now compare it to variant that uses surface distance instead of syntactic distance. The surface (or posi- tion) distance of token is computed simply by the number of tokens between the closest target aspect token and the corresponding token following previ- ous works (Zeng et al., 2019; Phan and Ogunbona. 2020). Focusing on words near the target aspect is known to be effective in the ABSC task (Zeng et al., 2019). But syntactic distance is often very different from surface distance (see Figures and 2.)
Twitter and multi-sentence items Table in Appendix shows that the Twitter dataset has a large proportion of multi-sentence items. Since multi-sentence items have multiple dependency graphs, it requires us to combine them by adding dummy root node that links to the heads of each sentence. This, well as RoBERTA's generally lower performance on Twitter, maybe one reason why we do not achieve state-of-the-an on Twitter. We have also not attempted to examine how parser accuracy contributes to performance differences.
Table 2 shows results for all three VDC types (decreasing, constant, and increasing) under both metrics that indicate that syntactic distances yield generally better performance than surface distances, especially in the increasing VDC configuration.

---

Variable Distance Control (VDC)
Lap14
Rest
Twitter
MAMs (VMa(Val)
Adc.
FT
Asc
FT
Acc.
F1
Acc.
FT
ROBFRTa-ASC
82.1(+0.9)
78.941111
87.6(+088)
81,7(+088)
75.6++0.6)
74.5(=077)
8444+044
83.0==0.5)
ANN-ABCC (Posiion Distiner))
Decrensing NIX.
(3.42+0.7)
804(4008)
$8.51+0.6)
63.2(4000)
76.50+0.8)
75.3c=07)
84.76004)
84.3==044
Constant-VDCC
83.7(+0.7)
80.7(+0.7)
(8.6(+B.6)
53.3(+0.5)
76-4++0.7)
15:4(=0.8)
849:+04)
84.(==0.5)
Inereaving-DCC
83.7(=0.9)
80.((+0.8)
88.6(#007)
63.2(+0.7)
76 76.9)#0099)
76.00=110)
848(2003)
84.4c=0.2)
SIR-ABSC (Dependency Graph)
Decressing-VIX:
83.8++0.6)
80.8(+0:7)
(8.2(+0.6)
82,9(+0.6)
76.5(+0.7)
(5.4(+008)
848(+044
84 (4.(4)
ConstantVDD
83 83.9)=0.66)
(0.7(+0.6)
88.9 (8.9(#0077)
63.70+00.))
76.4(++0.8
75. 75.21=00.7)
848(=0.4)
84 84.3))0.))
Incrasing-VDC
84.1(:00.88 81.1(200.8)
89.3(::1.6)
84.1(200.))
77.2(1:0.9)
76.3::0.9)
85.0k:0.3)
84.6(=03)
Table 2: Empirical results on the effectiveness of VDC. The results show that SIR-ABSC generally shows better performance in the order of decreasing < fixed < increasing VDCs. This result matches our intuition of [g] imitating GNN described in Section 3. A more detailed result table in the Appendix C.
SIR-ABSC
Lap14
Restl4
Twitter
MAMs (Val)
Auto-
FI
Avc.
FI
F1
Acc.
85.0(=0.55
82.1(=0.6)
89.7C+0.6)
84.8(=0.9)
77.5(=0.6)
76.4(=0.0)
85.4(::04)
84.9(=0.4)
84.7(+00.6
81.9(40.7)
09,5(+
84.4(+0.7)
77.3(=0.6)
76.2(=0.7)
85.2(+-03)
84.8(+0.2)
84.6(+00.8)
81.8(-08)
89.4C+0.4)
84.4(+088)
71.2(+0.6)
76.3(+0.6)
85.2(+0.5)
84.7(+0.5)
X
(4.1(+).8)
81.104088)
89.3(+0.6)
84.1(+0.8)
77.2(+0.9)
76.3(+0.)
85.0(+0.3)
84.6(+0.3)
Table 3: Ablation study results. The first row is equivalent to our final SIR-ABSC model and the last row is equivalent our baseline SIR-ABSC model using increasing heuristicVVDC.
Consistency with general GNNs SIR-ABSC is inspired by how GNNs gradually aggregate in- formation from nodes that are more and more distant in their upper layers. As mentioned in section 3, increasing VDC hyperparameters can be used to mimic this behavior. As men- tioned above, Table 2 summarizes experiments conducted on three different types of VDCs: in- creasing (e.g. [0.0.00.111112222333333333 (e.g.,
intentionally block the attention masks between [s] and [g] in every layer. Consequently, the [s] and [g] tokens cannot attend to each other during the self-attention layers, weakening their interaction. By comparing SIR-ABSC with Table 4 Variant (w/o s<*g interaction), we can observe that the drop in performance considerable, verifying our hypothesis that more integrated models can allevi- ate the suboptimal interaction issue and yield more significant performance improvements.
constant and decreasing (c.g.,
[3.3,3.222222111110000000
How does SIR-ABSC alleviate the suboptimal aggregation issue? We conducted an ablation study to verify the effectiveness of DAA and Auto- VDC, and the results are summarized in Table 3. From the table, we can observe that each method improves the performance over our baseline model (using increasing heuristicVVDCC) and both applied together further enhances the performance. These results verify that SIR-ABSC can effectively uti- lize the dependency labels and learns to identify and focus on specific distances based on the input, alleviating the suboptinal aggregation issue. We also include two case studies, Figures 3 and 4, for qualitative analysis. In Figure 3, SIR-ABSC" [g] focuses on the target aspect ("touchscreen func- tions") in the carly layers and then gradually cap- tures the key phrase "did not enjoy" in the later layers, showing some consistency with increasing heuristic-VDD results. Also, from Figure 4 we can see that SIR- AIR-ABSC learns to focus more on vDCs of 0,1, and 2 and less on 3 (see Figure 4
From the table, we can observe the followings. First, the VDC hyperparameter is crucial for SIR- ABSC's performance, verifying [g]'s effectiveness. Also, it can be seen that SIR-A ABC has the highest performance with increasing VDCs (i.e. when it is most similar to typical GNNs), and the lowest performance with decrcasing VDCs (i.e. when it is the least similar GNNs). More detailed experiment results are provided in Appendix C.
How does SIR-ABSC alleviate the suboptimal interaction issue? Existing (PLM+GNN) models in ABSC work in an asynchronous fashion (i.e., one after the other), which results in weak interaction between the semantic and syntactic information (Tang et al., 2020). SIR-ABSC, the other hand, is a fully-integrated model where [s] (=semantic aggregator) and [g] (=syntactic aggregator) inter- act with each other in every layer. To verify the strength of this fully-integrated characteristic, we conducted an ablation study (Table 4) where we

---

SPA
Seyer
Myarts
23
VOC-o tokkns: (resolution
Myar
layer
Mymm
VOC-I resolittion
laye
(sic has, highe; resoluttion, ane]
VDC-d tokern: (ei; has, higher resolution, but, lonts, are, small]
Heer
VDCs their corresponding ng tokens
[b] VDC weights
lsl
Figure 4: A case study the effect of the Auto-VDC mechanism. Target aspect is "resolution". Auto-VDC learns to focus more on the key distances (VDCs of and 2) and less on the redundant distance (VDC of 3). As a result, the model successfully focuses on the key phrase "has higher resolution" at the last layers.
[g] loken
Lap14
Rest14
Twitter
MAMs
Acc.
Acc.
Acc.
Acc.
SIR ABSC
85.0(:00.8)
82.1(2:99)
89.7(=0.6)
84.8(1999)
77.5(1:06)
76.4(20.))
85.2(20.4)
84.8(20.4)
Variants
1. wlo a ++ interaction
84.2(+:0.8)
81.3(#0.9)
89.3(204)
84.1(20.6)
17.2(40.9)
76.1(=099)
85.0(=0.3)
84.5(20.3)
2. [gl aspect embed.
84.7(1:0.8)
81.6(20.7)
89.3(=0.6)
84.361066)
77.2(10.88
76.01=0.9)
85.0(20.4)
84.6(10.4)
3. SIR-ABSC-[g) (Appendix F)
83.5(+:0.6)
80.5(+006)
88.3(=04)
82.9(+0.4)
75.6(+:08)
74.3(=0.)
84.6(+0.3)
84.20+0.3)
Table 4: Empirical results on the sullly-nttgratedness and the inherent strength of the pre-trained [s] token embedding.
b) where VDC=3 tokens are indeed redundant (in- cludes "fonts are small" where the target is "res- olution")), verifying that SIR-ABSC can capture important distances based on the input. We provide more detailed ablation and case study results in Appendix D.
Conclusion
This paper has proposed a novel framework, SIR- A ABSC, that effectively incorporates syntactic in- formation directly into pre-trained large language model (PLM) such as RoBERTa for tasks like Aspect-Based Sentiment Classification (ABSC), in which the desired output depends on specific words in the input, and where syntactic distance to the relevant input words may be important. In contrast to prior work, where separate GNN was added to the output of the PLM, in our model, attention masks for new [g] token capture syntactic informa- tion, and a new hyper-parameter, named variable distance control (VDC), captures graph structure in a similar fashion. Dependency-aware aggrega- tion (DAA) mechanism allows SIR-ABSC to use the dependency labels effectively, and finally, the Auto-VDC mechanism learns mixture of multiple distances, allowing the model to identify and focus on important syntactic distances for given input. To the best of our knowledge, SIR-ABSC is the first model to incorporate syntactic knowledge into RoBERTa for ABSC without resorting to GNNs. Experiments show we achieve state-of-the-art per- formance in SemEval-201 4 task 4. This demon- strates the efficiency of our approach and suggests a new paradigm for augmenting PLM with syntax.
Inherent strength of the pre-trained [s] token as an aggregator There seems be an inherent ad- vantage in using the pre-trained embedding of the [s] token also for Ig]. Table (Variant 2) compares SIR-ABSC (in which the dictionary embedding of [g] is initialized with [s]'s embedding), with a variant in which we use the actual aspect word's 's dictionary embeddings as the [g] embedding (if the aspect consists of several words, average their embeddings). Initializing [g] with the [s] embed- ding yields better performance, perhaps because the [s] embedding is better suited to aggregate in- formation than the embeddings of other tokens, providing a better starting point for a sequence ele- ment that is also intended to aggregate information (albeit of a slightly different nature). Erase later: (Chen et al., 2020). (Chen et al., 2022), (Niu et al., 2022), (Wang et al., 2020), (Xiao et al., 2022), (Zhao et al., 2022), (Liang al., 2022b), (Veyseh et al., 2020), (Dai et al., 2021), (Feng et al., 2022b),
Limitations
(Mei et al., 2023), (Xiao et al., 2021), (Liang et al., 2022a), (Yuan et al., 2022), (Zhong et al., 2023)
In this section, we summarize the limitations of our model that can be improved in the future. First, we

---

did not explore various types of dependency parsers as mentioned in Section 4, which could result in dif- ferent performances. Similarly, we did not examine how accurate the parser we used was and how much performance difference it could make, as men- tioned in Section 4. Second, we did not consider other types of syntactic information besides de- pendeney trees, such phrase dependency graphs, which could also be effective for ABSC. Lastly, our method might be limited to single-sentence items. Although we inserted dummy root node to merge multiple dependency graphs within a single input sentence, our model does not achieve state-of saee--ffthhe- thth-- art results on the Twitter dataset, which consists primarily of multi-sentence items. Therefore, it would be crucial to investigate more effective meth- ods for merging multiple dependency graphs in given input sentence.
deep bidirectional transformers for language under- standing. ArXiv, abs/18 111188800440555
Hai Ha Do, PWC Prasad, Angelika Maag, and Abeer Alsadoon. 2019. Deep learning for aspect-based sen- timent analysis: A comparative review. Expert Sys- tems with Applications, I 18:2222-299.
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target drgeeddependent Twitter sentiment clas- sification. In Proceedings of the 52nd Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 49-54, Baltimore, Maryland. Association for Computational Linguis- tics.
Timothy Dozat and Christopher D Mamning. 2016. Deep biaffine attention for neural dependency pars- ing. arXiv preprimt arXiv: 1.01734.
Rotem Dror, Segev Shlomov, and Roi Reichart. 2019. Deep dominance-hov to properly compare deep neu- ral models. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 1773-2785
References
Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang. Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax- BERT: Improving pre-trained transformers with syn- tax trees. In Proceedings of the 16th Conference of the European Chapter of the Association for Compu-
Ao Feng. Xuelei Zhang, and Xinyu Song. 2022a. Un- restricted attention may not be all you need-masked attention mechanism focuses better on relevant parts in aspect-based sentiment analysis. IEEE Access, 10:8518-88 (888188-528..
tational Linguistics: Main Volume, pages 301 1-3020, Online. Association for Computational Linguistics.
Ao Feng. Xuelei Zhang, and Xinyu Song. 2022b. Un- restricted attention may not be all you need-masked attention mechanism focuses better on relevant parts in aspect-based sentiment analysis. IEFE Access, 10:8518-8528.
Xuefeng Bai. Pengbo Liu, and Yue Zhang. 2020. In- vestigating typed syntactic dependencies for targeted sentiment classification using graph attention neural network. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 29:503-55 514.
Xiaochen Hou, Peng Qi, Guangtao Wang, Rex Ying. Jing Huang, Xiaodong He, and Bowen Zhou. 2021. Graph ensemble learning over multiple dependency trees for aspect-level sentiment classification. arXiv preprint arXiv:2 103. 11794.
Chenhua Chen, Zhiyang Teng. Zhongqing Wang, and Yue Zhang. 2022. Discrete opinion tree induction for aspect-based based sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2051-2064,
Qingnan Jiang. Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019. A challenge dataset and effec- tive models for aspect-basce sentiment analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNE pages 6280- 6285, Hong Kong, China. Association for Computa- tional Linguistics.
Chenhua Chen, Zhiyang Teng, and Yue Zhang. 2020. Inducing target-specific latent structures for aspect sentiment classification. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 5596-5600
Junqi Dai, Hang Yan, Tianxiang Sun, Pengfei Liu, and Xipeng Qiu. 2021. Does syntax matter? a strong baseline for aspect-based sentiment analysis with RoBERTa. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 1816-1829, Online, Association for Computational Linguistics.
Akbar Karimi, Leonardo Rossi, and Andrea Prati. 2021. Adversarial training for aspect-based sentiment anal- ysis with BERT. In 2020 25th International Con- ference on Pattern Recognition (ICPR), pages 8797- 8803.
Thomas N Kipf and Max Welling. 2016. Semi- supervised classification with graph convolutional networks. arXiv preprint
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of

---

Xin Li, Lidong Bing. Wenxuan Zhang, and Wai Lam. 2019. Exploiting BERT for end-to and aspect-based sentiment analysis. arXiv preprint 1rX.v:191..00883.
Komang Wahyu Trisna and Huang Jin Jie. 2022. Deep learning approach for aspect-based sentiment classi- fication: A comparative review. Applied Artificial Intelligence, pages 1-37.
Bin Liang, Hang Su, Lin Gui, Erik Cambria, and Ruifeng Xu. 2022a. Aspect-based sentiment anal- ysis via affective knowledge enhanced graph con- volutional networks. Knowledge-Rased Systems, 235:107643.
Ashish Vaswani. Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, ArXiv, 706///07633766
Amir Pouran Ben Veyseh, Nasim Nour, Franck Demon- court, Quan Hung Tran, Dejing Dou, and Thien Huu Nguyen. 2020. Improving aspect- based senti- ment analysis with gated graph convolutional net- works and syntax-based regulation. arXiv preprint arXiv: 1338.
Bin Liang. Hang Su. Lin Gui, Erik Cambria, and Ruifeng Xu. 2022b. Aspect-based sentiment anal- ysis via affective knowledge enhanced graph con- volutional networks. Knowledge BonseddgeBBased Systems, 235:107643.
Kai Wang. Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020. Relational graph attention network for espect-baset, sentiment analysis. arXiv preprint arX4v:2004. 12362..
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 11692.
Luwei Xiao, Yun Xue, Hua Wang, Xiaohui Hu,
Donghong Gu, and Yongsheng Zhu. 2022. Exploring fine-grained syntactic information for aspect-based sentiment classification with dual graph neural net- works. Neurocomputing, 471:48-55 59.
Zhibin Lu, Pan Du, and Jun Nie. 2020. VGCN- BERT: augmenting BERT with graph embedding for text classification. Advances in Information Re- trieval, 12035:369.
Zeguan Xiao, Jiarun Wu, Qingliang Chen, and Congjian Deng. 2021. BERT4GCN: Using BERT intermediate layers to augment GCN for aspect-based sentiment classification. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Process- ing, pages 9193-92000 Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Xiaoyong Mei, Yougen Zhou, Chenjing Zhu, Mengting Wu, Ming Li, and Shirui Pan. 2023. A disentan- gled linguistic graph model for explainable aspect- based sentiment analysis. Knowledge-Rasse Based Systems, 260:1 10150.
Hao Niu, Yun Xiong, Jian Gao, Zhongehen Miao, Xi- aosu Wang. Hongrun Ren, Yao Zhang, and Yangy- ong Zhu. 2022. Composition-based heterogeneous graph multi-channel attention network for multi- aspect multi- sentiment classification. In Proceedings of the 29th International Conference on Computa- tional Linguistics, pages 6827-6836.
Qiannan Xu, Li Zhu. Tao Dai, and Chengbing Yan. 2020. Aspect-based sentiment classification with multi-attention network. Neurocomputing, 388:135- 143.
Li Yuan, Jin Wang. Liang-Chip Yu, and Xuejie Zhang. 2022. syntactic graph attention network for aspect- level sentiment analysis. IEEE Transactions on Arti- ficial Intelligence.
Minh Hieu Phan and Philip O Ogunbona. 2020. Mod- elling context and syntactical features for aspect- based sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 321 1211-33 1-3220.
Biqing Zeng. Heng Yang. Ruyang Xu, Wu Zhou, and Xuli Han. 2019. Lef: A local context focus mecha- nism for aspect- based sentiment classification. Ap- plied Sciences, (16):3389.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Har- ris Papagcorgiou, Ion Androutsopoulos. and Suresh Manandhar. 2014. SemEval- 2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (Se- mEval 2014), pages 27-35, Dublin, Ireland. Associa- tion for Computational Linguistics.
Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect- based sentiment classification with aspect-specific graph convolutional networks. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP) pages 4568 4578, Hong Kong. China. Association for Computational Linguistics.
Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. 2020. Dependency graph enhanced dual transformer structure for aspect- based sentiment classification. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics. pages 6578- 6588, Online. Association for Computational Lin- guistics.
Ziguo Zhao, Mingwei Tang. Wei Tang. Chunhao Wang. and Xiaoliang Chen. 2022. Graph convolutional net- work with multiple weight mechanisms for aspect- based sentiment analysis. Neurocomputing 500:124- 134.

---

Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, Hua Jin, and Dacheng Tao. 2023. Knowledge graph aug- mented network towards multiview representation learning for aspect-based sentiment analysis. IEEE Transactions on Knowledge and Data Engineering.
A Details datasets
Our model SIR-ABSC is evaluated on four differ- ent datasets from SemEval-2014 Task 4, Twitter, and MAMS datasets. Table shows the statistics of the datasets. Table 6 shows the percentage of mul- tiple sentences per item and the average number of sentences per item each dataset.
Dataset
Train
Test
Val
Restaurant (SemEval-2014)
3608
1120
Laptop (SemEval-2014)
2328
638
Twitter
6248
692
MAMS-ATSA
11186
1336
1332
Table 5: Dataset Overview
Datasets
Distribation
Train
Tost
Val
Lap14
2 mulliple sent/fite
7,86
7.84
Avg. sent sent/item
1,09
1.09
Res14
% multiple /item
4,02
4.38
Avg. number of sent./itm
1,04
1.05
Twittrr
% of multiple sent /item
59.44
60.55
Avg. number sent./item
1.99
1.96
MAMS-ATSA
% multiple /itemm
2.12
0.82
0.60
Avg. number of sent./ite
1.02
1.01
1.01
Table 6: Prevalence of multi-sentence items the
ABSC datasets.
B Comparing different pooler types
The [s] and [g] token outputs combined after the last layer of STR-ABSC encoders as described in Section 3. We conduct experiments on three different types of poolers for combining [s] and [g] token embeddings at the final layer: average, max, and attention pooling. Table 7 summarizes the results of using different pooler types for SIR- A ABSC. The result shows that attention pooling shows better results in general.
C Detailed variable distance control results
Our variable distance control (VDC) is a unique hyper-parameter which consists of 12 non-negative integers, where each integer represents the dj value of the I-th layer. Theoretically there are exponen- tially many possible values for VDC but use three representative types: increasing, constant, and decreasing VDCs.
We heuristically chose specific values for each type of VDCs and the detailed results are summa- rized in Table 8. The table shows that SIR-ABSC

---

Lapl4
Rest14
Twitter
we can observe that the edge label information is relatively more important than edge direction infor- mation in DAA. Another interesting finding is that DAA is more effective on BERT than RoBERTa. We speculate that this is because RoBERTa already utilizes the information contained in the depen- dency edge label to some extent, which is probably one of the reasons why RoBERTa shows better overall baseline performance than BERT.
Pooler types
Acc.
Fl
Acc.
F1
Acc.
Fi
SIR-ABSC
w/ max pooling
83.2
80.0
88.7
83.3
74.8
73.7
w/ avg pooling
83.8
80.6
88.8
83.5
76.5
75.5
w/ att pooling
84.1
81.1
89.3
84.1
77.2
76.3
Table 7: Comparing different pooler types for SIR- ABSC. We used VDC = 10..0.0.1..1111.2222233333 with the default full-interaction for the experiment.
Additionally, we incorporate two case studies to illustrate the effectiveness of DAA in Figure 5, From the above examples, we can observe that the attention scores of words (pretentious, and, inappro- priate) and (anywhere, else) get much higher when DAA is applied. Our speculation is that this occurs because DAA learns the significance of "advel" and "advmod" labels, resulting in high attention scores being assigned to expressions such as (pretentious adocl claim, be adomod, else, and else adeat, any- where) thereby leading to correct predictions. This is also consistent with our intuition since adverbial typical GNN mechanism through increasing VDC configuration.
has the highest performance with increasing VDCs. Increasing VDCs are designed to work as the most similar to the typical GNN by aggregating informa- tion from the closest nodes to farther nodes based on the target aspect. On the other hand, decreasing VDCs has the lowest performance where the de- creasing VDCs are designed to work as least similar to a GNN in the opposite order (i.e., aggregating in- formation from farther nodes to closer nodes based on the target aspect). From these results, we can conclude that SIR-ABSC successfully imitates the
modifiers are related to adjectives which is highly likely to be important for deciding the sentiment.
D Detailed ablation and case study results on DAA
E Significance tests
Following the proposed significance test designed for comparing deep learning models from (Dror et al., 2019), we ran their official code with our SIR- ABSC and the bascline models. Specifically, we compared (1) SIR-ABSC with -abbselinee and (2) SIR-ABSC (BERT) with BaRT-baseline.
We provide more detailed ablation and case study results regarding DAA in Table 9 and Figure 5. We experiment two additional variants of the proposed DAA mechanism: DAA (w/o edge label) and DAA (w/o edge direction), to verify the effectiveness of different kinds of syntactic information. Specifi- cally, DAA (w/o edge direction) ignores the direc- tion of the edges and assign the embeddings solely based on the dependency label. DAA (w/o edge la- bel) ignores the edge labels (e.g., nsubj) and assign the embeddings solely based on the edge direction. By comparing #1 and #2 from Table 9, serve that employing the VDC mechanism with the [gl token to replace GNNs leads significant im- provement in performance (average accuracy gain of 1.475% and average fl-score gain of 1.775). By further supplementing the model with missing in- formation, such as edge direction and edge label information, DAA yields additional average gains of 0.2% on accuracy and 0.275 fl-score over [RoBERTa+ VDCJ (#2 #5). The DAA ablation results (#3 and #4) show that the performance de- creases for both DAA (w/ edge direction) and DAA (w/o edge label), indicating their importance in DAA performance. Consistent with our intuition,
This test takes three components (results from method A, results from method B, and a confidence level) as input, and the purpose of this test is to answer the question: "Is A almost stochastically greater than B?" This test is based on the relatively new concept of "Almost Stochastic Order" which is a relaxed version of "Stochastic Order". The test outputs minimum value where the result is interpreted as follows:
ob-
Iffe > 0.5: We can't It say A is almost stochas- tically greater than B.
If S 0.5: We can say A is almost stochasti- cally greater than B.
If = 0: We can say A is "stochastically greater" than B not just "almost stochastically greater" which the best result we can get in this test. Mathematically meaning inf (x < F(x)} 2 inf(x : t & G(x)} for Vt. E (0, 1).

---

advel
boo
p0by
Compoonnd
Coa
(abvmed
nesbi
(compunnn)
Sentiment: Negative
pretentious
imppprppiiale
mtj
grill
provides
power
dinners
Senriment: Neutral 0.01 1.84 0.39
0.08
0.05
0.04
0.40
X
X
0.2
1,58
2.22
5.21
13,65
14.76
17.82
32.86
3.71
Sentiment: Negative
5,56
1,50
23.
1.65
advmmol
nsubi
advmod
advmmod
det
attr
Sentiment: Positive
[8]
[g]
Anywhere
clse
the
prices
would
be
3x
high
Prediction: Negative 0.23 4.50
0.13
0.32
12.80
14.41
8.87
26.66
Prediction: Positive 0.49 9.40
4.01
5,56
9.08
4.44
13.38
11.07
15.44
10.59
Figure 5: Two case studies employing DAA. The targets (dinners, prices) are marked in bold. The first and second rows below sentence depict the attention scores the [g] token without and with DAA, respectively. Attention scores that increase more than ten-fold when DAA is applied also marked in bold. In these examples, DAA enables the [g] token to place greater emphasis on critical sentiment words like "pretentious," "inappropriate." "anywhere," and "else," which contributes to accurate predictions.
where F and G are the (empirical) CDIss of A and B.
and independent [g] token for the GNN role as in SIR-ABSC.
Please refer to the paper for further details. Applying the test on our models with confi- dence level (I-a) of 0.99 (in other words, signifi- cance level (a) of 0.01), both tests (1) and (2) re- sulted in minimum values of O. This indicates that SIR-ABSC and SIR-ABSC (BERT) are stochas- tically greater than RoBERTa- -oBelia-baseline and BERT- baseline with a confidence level of 0.99, respec- tively, verifying the superiority of our method.
G SIR-ABSC (BERT) results
We chose RoBERTa as our baseline PLM due to its better overall performance in ABSC tasks (Dai et al., 2021). However, to verify the generality of our method, we also conducted experiments on ABSC (BERT) which uses BERT instead of RoBERTa. The results summarized in Table 10. We that our method is effective on BERT well and the significance greateness" under significance level of a 0.01, verifying the general effectiveness. The amount improvement in performance could differ depend- ing on the PLM and we can notice that the amount of enhancement when using RoBERTa (+1.93 on average) is relatively larger than when applied to BERT (+ 1.60 on average).
results in "stochastic
F Additional analysis: does [g] need to be a separate token?
We now compare SIR-ABSC variant that does not use [g] token, but instead uses the target to- kens at the end of the input sequence (recall that the input sequence has the form of "[s] sentence [/s] [/s] aspect sequence [/s]'). We call this the STR-AB STR-ABSC-[g] variant. As Table 4 shows, the loss in performance is considerable compared to using independent [g] token in the original SIR- ABSC. We speculate the drop in performance is due to the original input sentence getting corrupted when we modify the aspect token's 's attention mask. This indicates the importance of using an additional

---

Lapl4
Rest14
Twitter
MAMS
Variable Distance Control (VDC)
Acc.
F1
Acc.
Fl
Acc.
F1
Acc.
F1
GoBERTA (Position Distance)
Decreasing-VDD VDC
83.4
80.4
88.5
83.2
76.5
75.3
84.7
84.3
VDC = [222211110000]
83.3
80.2
88.2
82.7
76.0
74.8
84.7
84.2
VDC = [333222111000]
82.0
78.8
88.4
82.8
76.5
75.3
84.7
84.3
VDC [444422220000]
83.4
80.4
88.4
83.1
75.6
74.4
84.7
84.2
VDC [554433221 100]
83.1
79.9
88.5
83.2
75.2
73.7
84.6
84.2
VDC [666444222000]
83.2
80.0
87.8
82.0
76.0
74.8
84.7
84.2
Constant-VDDC
83.7
80.7
88.6
83.3
76.4
75,4
84.9
84.4
Macreasing-VDC
83.7
80.5
88.6
83.2
76.9
76.0
84.8
84.4
VDC [00001 [00011111222]
83.5
80.3
87.8
82.1
76.9
76.0
84.7
84.3
VDC (000111222331)
83.5
80.5
88.5
83.2
75.5
74.4
84.7
84.3
VDC [000022224444]
83.6
80.4
87.9
82.2
75.7
74.4
84.8
84.4
VDC [001122334455
83.3
80.3
88.6
83.1
76.1
74.9
84.8
84.4
VDC [000222444666]
83.7
80.5
88.3
82.5
76.6
75.8
84.7
84.2
GoBERTA (Dependency Graph)
Decreasing-VDD VDC
83.7
80.6
88.2
82.8
76.5
75.4
84.8
84.3
VDC = 1122201110000)
83.5
80.4
88.1
82.7
75.4
74.2
84.7
84.3
VDC = [333222111000]
82.6
79.6
87.1
81.1
76.1
75.1
84.5
84.0
VDC [444422220000]
83.4
80.5
87.9
82.2
76.5
75.4
84.8
84.3
VDC (55443322110) 100]
83.8
80.8
88.2
82.9
75.3
74.2
84.6
84.1
VDC [666444222000]
83.2
80.0
88.2
82.7
75.6
74.3
84.5
84.0
Constant- VDC
83.7
80.4
88.9
83.7
76.4
75.2
85.0
84.5
Increasing-Vi
83.8
80.8
89.1
83.8
77.1
75.9
85.0
84.6
VDC (000011112222]
84.1
81.1
88.3
82.8
77.2
76.3
84.6
84.2
VDC [000111222333
83.8
80.8
89.1
83.8
76.7
75.5
84.7
84.3
VDC (000022224444)
83.5
80.5
88.9
83.5
75.7
74.6
84.8
84.3
VDC [001122334455]
83.2
80.2
89.3
84.1
74.7
75.9
85.0
84.6
VDC [000222444666]
82.5
79.4
88.9
83.8
76.9
75.9
84.8
84.5
Table 8: Detailed experimental results the effect of VDC. The results show that SIR-ABSC generally shows better performance in the order of decreasing < fixed < increasing VDCs. This result matches our intuition of [g] token imitating GNN as described Section

---

Lapl4
Rest14
Twitter
MAMS
Ablations (Validation)
F1
Acc.
Acc.
SIR-ABSC (RoBERTa)
#1: RoBERTa baseline
82.1
78.9
87.6
81.7
75.6
74.5
84.4
83.9
#2: #1 [[][ VDK
84.1
81.1
89.3
84.1
77.2
76.3
85.0
84.6
#3 (DAA Ablation #1): #2+DAA (w/o edge label)
84.2
81.3
89.3
84.3
77.2
76.0
84.8
84.3
(DAA Ablation #2): #2+DAA (w/o edge direction)
81.5
89.4
84.3
77.3
76.5
84.9
84.4
#5 (DAA Ablation #3): + (proposed)
84.6
81.8
89.4
84.4
77.2
76.3
85.2
84.7
#6 (SIR-ABSC): #5+Auto-VDDC
85.0
82.1
89.7
84.8
77.5
76.4
85.4
84.9
2. SIR-ABSC (BERT)
#1: BERT baseline
79.2
75.1
85.8
79.3
76.4
74.8
84.0
82.7
#2: [+[g] tokent VDC
80.8
77.1
86.7
80.6
77.1
75.9
84.3
83.1
#3 (DAA Ablation #1): #2+DAA (w/o edge label)
81.1
77.5
86.9
80.9
77.2
75.9
84.3
83.2
#4 (DAA Ablation #2): #2+DAA (w/o edge direction)
81.6
78,2
87.0
81.1
77.4
76.1
84.4
83.5
(DAA Ablation #3): #2+DAA (proposed)
78.6
87.0
81.2
77.6
76.4
83.5
#6 (SIR-ABSC (BERT)): A5touuo VDC
82.2
79.1
87.3
81.6
77.7
76.6
84.7
83.6
Table detailed ablation study results.
Lap14
SIR-ABSC (BERT)
Acc.
Fl
Acc.
F1
Acc.
BERT baseline
79.2(10.6)
75.1(20055
85.8(10.7)
79.3(20.7)
76.4(20.7)
74.8(207)
84.0(20.3)
82.7(100.4)
SIR-ABSC (BERT)
(9404)
87.3(40.7) 81.6(+0.8)
77.7(+0.8)
76.6(-0.7)
84.6(+0.4)
83.3(+0.5)
Table 10: Empirical results using BERT the PLM. Note that our technique is effective on BERT well (significance test resulting in "stochastic greateness") showing comparable results with BERT-based state-of-the-are art results. However, the improvement generally less than when applied RoBERTa.


Reviews:
[{"review_id": "bGmOeoiT6U", "reviewer": "Reviewer_HrVu", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer HrVu for the insightful comments. We are glad to hear that you found our work novel and interesting. We summarized how we have addressed your comments in our revised paper below. We hope these revisions make our paper stronger.\n\n---\n\n[Comment 1] “There are already many researches focus on employing graph-based models for ABSC, what is the difference between the proposed model and previous studies.”\n\n=>  This paper presents a new approach to incorporating syntactic information into PLMs using a new syntax-aggregator token ([g] token) instead of resorting to GNNs. To our knowledge, this is the first work to utilize dependency graph information without using a GNN in ABSC. We summarized the main advantages of SIR-ABSC over standard GNN-based methods below. \n\n\n1. Since the aggregator token is part of the input sequence, it can naturally make use of the existing modules in the PLM. Specifically,\n    * **Reusing existing parameters:** The [g] token can take advantage of the existing and pre-trained self-attention parameters in the PLM. Therefore, it can effectively aggregate syntactic information with only a minor additional work of manipulating [g] token’s attention mask (explained in Lines 271-274).\n    * **Fully-integratedness:** [s] (=semantic aggregator) and [g] (=syntactic aggregator) tokens interact with each other throughout the entire 12 layers of PLM, making them synchronously updated compared to the asynchronous behavior of standard GNN-based approaches (i.e., PLM operates first and then the GNN operates). Through ablation studies, we show that this synchronicity (or fully-integratedness) is crucial for SIR-ABSC’s performance, which is absent in standard GNN-based methods (explained in the section “How does SIR-ABSC alleviate the suboptimal interaction issue?” (Lines 474-494) and Table 4 variant 1).\n    * **Utilizing [s] token’s inherent strength:** We hypothesize that the [s] token implicitly learns a meaningful ability to summarize information from the input sentence as an aggregator during the heavy pre-training process. The [g] token can easily benefit from this inherent strength of the [s] token by initializing [g]’s embedding with [s]’s embedding, leading to a better performance (explained in the section “Inherent strength of the pre-trained [s] token as an aggregator” (Lines 520-535) and Table 4 variant 2).\n\n\n2. The DAA mechanism allows SIR-ABSC to utilize the dependency label information, which contains useful information for the ABSC task (explained in Lines 318-338, 495-506).\n3. The Auto-VDC mechanism allows SIR-ABSC to adjust and focus more on the key syntactic distance of the input data, a substantial advantage over previous GNN-based models (explained in section “How does SIR-ABSC alleviate the suboptimal aggregation issue?” (Lines 495-519) and Figure 4).\n\nIn the revised paper, we will include a bullet list of main contributions at the end of the Introduction section to highlight the differences in our approach more clearly, as below.\n\n\"\"\"\n\nOur main contributions are summarized below: \n* We present a novel approach to incorporating syntactic information into PLM with a syntax aggregator token.\n* We highlight two limitations of existing GNN-based approaches (suboptimal interaction and suboptimal aggregation) and provide methods (DAA and Auto-VDC) that effectively alleviate them. \n* To our knowledge, this is the first work to utilize dependency graph information without resorting to GNNs in ABSC. \n* Our model shows state-of-the-art results among RoBERTa-based models on all four widely used ABSC datasets and comparable results on BERT-based models.\n\n\"\"\"\n\n[Comment 2] The authors should compare the proposed model with previous studies.\n\n=> We compared with previous studies in Section 2.2 (Combining PLMs with syntax), but we apologize if the comparison was not exhaustive enough. Following your advice, we will include more details in Section 2.2 by including more references ([1], [2], [3], [4]). \n\nWe truly appreciate your valuable reviews and feedback.\n\n---\n\nReference:\n* [1] Li Yuan, Jin Wang, Liang-Chih Yu, Xuejie Zhang. 2020. Syntactic Graph Attention Network for Aspect-Level Sentiment Analysis, In IEEE Transactions on Artificial Intelligence, pages 1-15,\n* [2] Chenhua Chen, Zhiyang Teng, and Yue Zhang. 2020. Inducing Target-Specific Latent Structures for Aspect Sentiment Classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5596–5607, Online. Association for Computational Linguistics.\n* [3] Chenhua Chen, Zhiyang Teng, Zhongqing Wang, and Yue Zhang. 2022. Discrete Opinion Tree Induction for Aspect-based Sentiment Analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2064, Dublin, Ireland. Association for Computational Linguistics.\n* [4] Hao Niu, Yun Xiong, Jian Gao, Zhongchen Miao, Xiaosu Wang, Hongrun Ren, Yao Zhang, and Yangyong Zhu. 2022. Composition-based Heterogeneous Graph Multi-channel Attention Network for Multi-aspect Multi-sentiment Classification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6827–6836, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed model is novel, interest, and simple. Experiment results show effective of the proposed model.", "reasons_to_reject": "There are already many researches focus on employing graph-based models for ABSC, what is the difference between the proposed model and previous studies.\n\nThe authors should compare the proposed model with previous studies.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper present a simple, but effective method to incorporate syntactic dependency information directly into transformer-based language models for ABSC. Experiment results show effective of the proposed model.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2mJmp0Q0zk", "reviewer": "Reviewer_hL41", "summary": "", "questions": "1. What kinds of syntatic information are used? How do they affect the performance? It is encouraged to provide a detailed analysis in the experiments.\n2. The contribution should be highlighted at the end of Introduction.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank reviewer hL41 for the detailed and thoughtful review. We are delighted to hear that you found our work interesting. Each comment and how we have addressed them in our revised paper is summarized below. \n\n[Comment 1] What kinds of syntactic information are used? How do they affect the performance? It is encouraged to provide a detailed analysis in the experiments. \n\n=> We used two types of syntactic information: (1) Syntactic distance (i.e., the number of hops from a word to the target aspect based on the dependency graph) and (2) dependency labels (e.g., nsubj). We also tried using POS-tags but observed no significant improvement in performance, thus excluding POS-tags from our final model. Based on your feedback, we conducted an additional analysis on DAA focusing on how the dependency information (edge label and edge direction) affects the model’s performance. We attached the results and summarized our key findings below:\n\n| DAA ablations |      Laptop (Acc/F1-score)      |  Restaurant (Acc/F1-score)|\n|----------|:-------------:|------:|\n| DAA (proposed)|   **84.6/81.8** |  **89.4/84.4** |\n| DAA (w/o edge direction) |    84.4/81.5 |  89.4/84.3 |\n|DAA (w/o edge label) | 84.2/81.3 |   89.3/84.3 |\n\n* [1] DAA (w/o edge direction): we ignore the direction of the edges and assign the embeddings solely based on the dependency label.\n* [2] DAA (w/o edge label): we ignore the edge labels (e.g., nsubj) and assign the embeddings solely based on the edge direction.\n* The results show that the performance decreases for both DAA (w/o edge direction) and DAA (w/o edge label), indicating their importance in DAA performance. Consistent with our intuition, we can observe that the edge label information is relatively more important than edge direction information in DAA.\n\nIn addition to the above quantitative analysis, we conducted a qualitative analysis by choosing two examples for case studies. Due to OpenReview not supporting images, we wrote each word and its corresponding attention score next to it in parenthesis. Note that the attention score is the score attended by the [g] token at the PLM's last layer (DAA is only applied to the [g] token).\n\nCase study #1\n\n---\n\nText: How pretentious and inappropriate for mj grill to claim that it provides power lunch and **dinners**!\n\nAspect: dinners\n\nTrue sentiment: Negative\n\n```\ndependency tree:\n        lunch--conj-->dinner\n        lunch --cc--> and\n        lunch --compound--> power\n        provides--dobj--> lunch\n        provides--nsubj--> it\n        provides--mark--> that\n        claim--ccomp--> provides\n        claim--aux--> to\n        pretentious--advcl--> claim\n        pretentious--prep--> for\n        for --pobj-->grill\n        grill--compound-->mj\n        pretentious--conj--> inappropriate\n        pretentious--cc--> and\n        pretentious--advmod--> how\n```\n\n---\n\nWithout DAA: \n* [g] token's attention scores: \n    * [s] (0.0001) [g] (0.0184)  how (0.0039) pretentious (0.0008) and (0.0005) inappropriate (0.0004) for (0.0040) mj (X) grill (X) to (0.002) claim (0.0158) that (0.0222) it (0.0521) provides (0.1365) power (0.1476) lunch (0.1782) and (0.3286) dinners (0.0371) ! (0.0518)\n* Prediction: \n    * Neutral ([Negative: 0.0101, Neutral: 0.7712, Positive: 0.2188])\n\nWith DAA:\n* [g] token's attention scores (attentions scores with more than x10 increase are shown in bold case): \n    * **[s] (0.0022)** [g] (0.0614)  how (0.00391) **pretentious (0.0556)** **and (0.0150)** **inappropriate (0.0507)** for (0.0244) mj (X) grill (X) **to (0.0620)** claim (0.0590) that (0.0773) it (0.1067) provides (0.2374) power (0.0858) lunch (0.0165) and (0.0928) dinners (0.0008) ! (0.0132)\n* Prediction: \n    * Negative ([Negative: 0.7758, Neutral: 0.1160, Positive: 0.1082])\n\nCase study #2\n\n---\n\nText: Anywhere else, the **prices** would be 3x high\n\nAspect: prices\n\nTrue sentiment: Positive\n\n```  \ndependency tree:\n        be--aux-->would\n        be --attr--> 3x\n        3x --advmod--> high\n        high--advmod--> as\n        be --nsubj--> prices\n        be --advmod--> else\n        prices--det--> the\n        else --advmod--> anywhere\n       \n```\n\n---\n\nWithout DAA:\n* [g] token's attention scores: \n    * [s] (0.0023) [g] (0.045) anywhere (0.0013) else (0.0032) , (0.0116) the (0.1280) prices (0.0320) would (0.1441) be (0.0887) 3 (0.0552) x (0.123) high (0.2666) ! (0.099) \n* Prediction: \n    * Negative ([Negative: 0.9970,  Neutral: 0.0021,  Positivie: 0.0009])\n\nWith DAA:\n* [g] token's attention scores (attentions scores with more than x10 increase are shown in bold case): \n    * [s] (0.0049) [g] (0.094) **anywhere (0.0401) else (0.0678)**, (0.0316) the (0.0908) prices (0.0444) would (0.1338) be (0.1107) 3 (0.0604) x (0.094) high (0.1059) ! (0.1442)\n* Prediction: \n    * Positive ([Negative: 0.3558,  Neutral: 0.0021,  Positive: 0.6421])\n\nFrom the above examples, we can observe that the attention scores of words (pretentious, and, inappropriate) and (anywhere, else) get much higher when DAA is applied. We speculate that this is because the DAA learns the importance of \"advcl\" and \"advmod\" labels and therefore assign high attention scores on (pretentious--advcl--> claim, be --advmod--> else,  and else --advmod--> anywhere). This is also consistent with our intuition since adverbial modifiers are related to adjectives which is highly likely to be important for deciding the sentiment. \n\n[Comment 2] The contribution should be highlighted at the end of Introduction.\n\n=> Following your advice, we will add a bullet list of main contributions at the end of the introduction: \n\n“““\n\nOur main contributions are summarized below: \n* We present a novel approach to incorporating syntactic information into PLM with a syntax aggregator token.\n* We highlight two limitations of existing GNN-based approaches (suboptimal interaction and suboptimal aggregation) and provide methods (DAA and Auto-VDC) that effectively alleviate them. \n* To our knowledge, this is the first work to utilize dependency graph information without resorting to GNNs in ABSC. \n* Our model shows state-of-the-art results among RoBERTa-based models on all four widely used ABSC datasets and comparable results on BERT-based models.\n\n“““\n\n[Comment 3] Missing References: The proposed idea is similar to the following paper. The authors should discuss the difference. https://ieeexplore.ieee.org/document/9976197\n\n=> We will add the corresponding paper (SGAN) in Section 2.2. The main difference between our work and SGAN is the synchronicity. SGAN, similar to other general GNN-based methods, operates asynchronously (i.e., BiLSTM operates first and then the SGAN module), limiting their interaction, which is the central limitation we wanted to tackle in this paper. SIR-ABSC, on the other hand, is a synchronous model where the semantic and syntactic aggregator modules get updated simultaneously via the newly introduced [g] token inside the PLM. The DAA mechanism is also slightly different from SGAN. SGAN proposes a unique way that first expands dependency labels using the POS-tag of the source node and then uses a target embedding-oriented weighted sum for aggregating multi-head vectors. SIR-ABSC, on the other hand, does not use the POS-tag information and it reuses the multi-head self-attention mechanism in the PLM for utilizing the dependency labels.\n\nWe hope these revisions will make our paper more robust. Thank you again for your thoughtful feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a new method to address previous two limitations: suboptimal interaction and suboptimal aggregation for the ABSC task.", "reasons_to_reject": "None", "questions_to_authors": "1. What kinds of syntatic information are used? How do they affect the performance? It is encouraged to provide a detailed analysis in the experiments.\n2. The contribution should be highlighted at the end of Introduction.", "paper_topic_and_main_contributions": "This paper presents Syntax-Integrated RoBERTa for ABSC (SIR-ABSC) that incorporates syntax directly into the language model by using a novel aggregator token. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The proposed idea is similar to the following paper. The authors should discuss the difference.\nhttps://ieeexplore.ieee.org/document/9976197"}, "extra_text": {}}, {"review_id": "hADDRbLsFz", "reviewer": "Reviewer_XaBJ", "summary": "", "questions": "1. How effective is DAA?\n2. How well does this method work on BERT", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer XaBJ for the valuable comments and the recognition of our insightful and interesting new idea. We would like to provide more explanations to address your concerns one by one. \n\n[Comment 1] “The overall presentation is not clear enough, especially Figure 2 cannot show the main idea of ​​the work, which brings troubles to readers.”\n\n=> We apologize if our paper structure was not concise. In the revised paper, we will update Figure 2 and emphasize the key mechanisms (VDC, DAA, Auto-VDC) for better understanding. In the revised paper, we will also include a bullet list of main contributions at the end of the Introduction section to highlight the differences in our approach more clearly, as below.\n\n“““\n\nOur main contributions are summarized below: \n* We present a novel approach to incorporating syntactic information into PLM with a syntax aggregator token.\n* We highlight two limitations of existing GNN-based approaches (suboptimal interaction and suboptimal aggregation) and provide methods (DAA and Auto-VDC) that effectively alleviate them. \n* To our knowledge, this is the first work to utilize dependency graph information without resorting to GNNs in ABSC. \n* Our model shows state-of-the-art results among RoBERTa-based models on all four widely used ABSC datasets and comparable results on BERT-based models.\n\n“““\n\n[Comment 2] “In addition, the proposed DAA module uses the attention mechanism to fuse the labels of the dependency tree, but the subsequent ablation experiments did not sufficiently prove the effectiveness of the method, and no further experiments were conducted to analyze the method.”\n\n=> We have conducted an additional ablation experiment regarding DAA and summarized the results in comment 4. \n\n[Comment 3] There are also typos, and many illustrations are not clear enough.\n\n=> Thanks for pointing out typos. We will fix typos in the revised paper. For the illustrations, we’ve included additional figure descriptions for better understanding. Also, we will update Figure 2 in the revised paper. \nThe Figure descriptions (captions) will be updated as below: \n* Figure 3: With Auto-VDC, the [g] token automatically learns the ideal VDC configuration. For example, in the lower layers, the [g] token’s attention scores focus on the target aspect (\"touch screen functions\") and [s] token. In the last layers, the [g] token focuses on the key phrase ``did not enjoy’’ for the final prediction. This shows that Auto-VDC gradually aggregates information from more and more distant nodes in the upper layers, just like heuristic VDCs and GNNs, but in an automatic way, thereby reducing the manual heuristic works.\n\n* Figure 4: A case study on the effect of the Auto-VDC mechanism. The target aspect is “resolution”. Figure (a) shows the tokens corresponding to each VDC value. With Auto-VDC, our model learns to focus more on the key distances (VDCs of 1 and 2) and less on the redundant distance (VDC of 3), as in Figure (b). As a result, the model successfully focuses on the key phrase “has higher resolution” at the last layers instead of concentrating on \"fonts are small\" which is irrelevant to the target \"resolution\".\n\n\n[Comment 4] How effective is DAA?\n\n=> We used two types of syntactic information: (1) Syntactic distance (i.e., the number of hops from a word to the target aspect based on the dependency graph) and (2) dependency labels (e.g., nsubj). We also tried using POS-tags but observed no significant improvement in performance, thus excluding POS-tags from our final model. Based on your feedback, we conducted an additional analysis on DAA focusing on how the dependency information (edge label and edge direction) affects the model’s performance. We attached the results and summarized our key findings below:\n\n| DAA ablations |      Laptop (Acc/F1-score)      |  Restaurant (Acc/F1-score)|\n|----------|:-------------:|------:|\n| DAA (proposed)|   **84.6/81.8** |  **89.4/84.4** |\n| DAA (w/o edge direction) |    84.4/81.5 |  89.4/84.3 |\n|DAA (w/o edge label) | 84.2/81.3 |   89.3/84.3 |\n\n* [1] DAA (w/o edge direction): we ignore the direction of the edges and assign the embeddings solely based on the dependency label.\n* [2] DAA (w/o edge label): we ignore the edge labels (e.g., nsubj) and assign the embeddings solely based on the edge direction.\n* The results show that the performance decreases for both DAA (w/o edge direction) and DAA (w/o edge label), indicating their importance in DAA performance. Consistent with our intuition, we can observe that the edge label information is relatively more important than edge direction information in DAA.\n\nIn addition to the above quantitative analysis, we conducted a qualitative analysis by choosing two examples for case studies. Due to OpenReview not supporting images, we wrote each word and its corresponding attention score next to it in parenthesis. Note that the attention score is the score attended by the [g] token at the PLM's last layer (DAA is only applied to the [g] token).\n\nCase study #1\n\n---\n\nText: How pretentious and inappropriate for mj grill to claim that it provides power lunch and **dinners**!\n\nAspect: dinners\n\nTrue sentiment: Negative\n\n```\ndependency tree:\n        lunch--conj-->dinner\n        lunch --cc--> and\n        lunch --compound--> power\n        provides--dobj--> lunch\n        provides--nsubj--> it\n        provides--mark--> that\n        claim--ccomp--> provides\n        claim--aux--> to\n        pretentious--advcl--> claim\n        pretentious--prep--> for\n        for --pobj-->grill\n        grill--compound-->mj\n        pretentious--conj--> inappropriate\n        pretentious--cc--> and\n        pretentious--advmod--> how\n```\n\n---\n\nWithout DAA: \n* [g] token's attention scores: \n    * [s] (0.0001) [g] (0.0184)  how (0.0039) pretentious (0.0008) and (0.0005) inappropriate (0.0004) for (0.0040) mj (X) grill (X) to (0.002) claim (0.0158) that (0.0222) it (0.0521) provides (0.1365) power (0.1476) lunch (0.1782) and (0.3286) dinners (0.0371) ! (0.0518)\n* Prediction: \n    * Neutral ([Negative: 0.0101, Neutral: 0.7712, Positive: 0.2188])\n\nWith DAA:\n* [g] token's attention scores (attentions scores with more than x10 increase are shown in bold case): \n    * **[s] (0.0022)** [g] (0.0614)  how (0.00391) **pretentious (0.0556)** **and (0.0150)** **inappropriate (0.0507)** for (0.0244) mj (X) grill (X) **to (0.0620)** claim (0.0590) that (0.0773) it (0.1067) provides (0.2374) power (0.0858) lunch (0.0165) and (0.0928) dinners (0.0008) ! (0.0132)\n* Prediction: \n    * Negative ([Negative: 0.7758, Neutral: 0.1160, Positive: 0.1082])\n\nCase study #2\n\n---\n\nText: Anywhere else, the **prices** would be 3x high\n\nAspect: prices\n\nTrue sentiment: Positive\n\n```  \ndependency tree:\n        be--aux-->would\n        be --attr--> 3x\n        3x --advmod--> high\n        high--advmod--> as\n        be --nsubj--> prices\n        be --advmod--> else\n        prices--det--> the\n        else --advmod--> anywhere\n       \n```\n\n---\n\nWithout DAA:\n* [g] token's attention scores: \n    * [s] (0.0023) [g] (0.045) anywhere (0.0013) else (0.0032) , (0.0116) the (0.1280) prices (0.0320) would (0.1441) be (0.0887) 3 (0.0552) x (0.123) high (0.2666) ! (0.099) \n* Prediction: \n    * Negative ([Negative: 0.9970,  Neutral: 0.0021,  Positivie: 0.0009])\n\nWith DAA:\n* [g] token's attention scores (attentions scores with more than x10 increase are shown in bold case): \n    * [s] (0.0049) [g] (0.094) **anywhere (0.0401) else (0.0678)**, (0.0316) the (0.0908) prices (0.0444) would (0.1338) be (0.1107) 3 (0.0604) x (0.094) high (0.1059) ! (0.1442)\n* Prediction: \n    * Positive ([Negative: 0.3558,  Neutral: 0.0021,  Positive: 0.6421])\n\nFrom the above examples, we can observe that the attention scores of words (pretentious, and, inappropriate) and (anywhere, else) get much higher when DAA is applied. We speculate that this is because the DAA learns the importance of \"advcl\" and \"advmod\" labels and therefore assign high attention scores on (pretentious--advcl--> claim, be --advmod--> else,  and else --advmod--> anywhere). This is also consistent with our intuition since adverbial modifiers are related to adjectives which is highly likely to be important for deciding the sentiment. \n\n[Comment 5] How well does this method work on BERT?\n\n⇒ We conducted experiments using BERT (SIR-ABSC (BERT)) instead of RoBERTa (We already included the experiment in Lines 417 - 421). Our empirical results on using BERT as the PLM (Appendix F) show our method is effective on BERT as well (significant test resulting in ``stochastic greatness”). \n\nTo address your comment, we’ve conducted additional ablation experiments on the effectiveness of DAA on SIR-ABSC (BERT) (Will be added in Appendix F). As shown in the below table, we can observe that each method (DAA and Auto-VDC) improves the performance over the BERT baseline. These results indicate that leveraging dependency labels (DAA) and a mechanism to learn to focus on specific distances based on the input (Auto-VDC) effectively work to enhance model performance.  \n\n| SIR-ABSC (BERT) ablations |      Laptop (Acc/F1-score)      |  Restaurant (Acc/F1-score)|\n|----------|:-------------:|------:|\n| BERT baseline |   79.2/75.1 | 85.8/79.3 |\n|   $\\hspace{5pt}$  + DAA |    81.8/78.6 | 87.0/81.2 |\n|  $\\hspace{5pt}$ + Auto-VDC (=SIR-ABSC (BERT))| **82.2/79.1** |   **87.3/81.6** |\n\nDue to the time limit for rebuttal, we were only able to conduct the ablation experiments for SIR-ABSC (BERT) on the Laptop and Restaurant14 dataset. Experiments on the remaining datasets (MAMS and Twitter) will be included in the revised paper.\n\n[Comment 6] Missing References: Inducing target-specific latent structures for aspect sentiment classification. Discrete opinion tree induction for aspect-based sentiment analysis. Composition-based Heterogeneous Graph Multi-channel Attention Network for Multi-aspect Multi-sentiment Classification. and so forth.\n\n⇒ Thanks for your suggestions on references. We have summarized the main contributions of paper [1], [2], [3] below:\n* Paper [1] is to amalgamate a dependency tree and a target-specific latent graph via a GCN gating mechanism. The primary focus lies in inducing latent structures for aspect-level sentiment classification. They establish individual latent graphs for each aspect by leveraging the Kumaraswamy distribution alongside a multi-head attention approach. These graphs are then merged using the same graph convolutional matrix, thereby avoiding the introduction of extra parameters within the gating mechanism.     \n* Similarly, Paper [2] induces a discrete opinion tree structure for each aspect term. They train the attention-based tree inductor through a policy network and regularize attention weights using KL divergence. This regularization prompts the aspect term to attend contexts with shorter distances.\n* Paper [3] proposes a multi-channel aggregating mechanism (similar to multi-view learning) to utilize the syntactic dependency graphs. The approach involves a three-view aggregation strategy: (1) original view: original node embeddings initialized with the output of BERT, (2) node type view: distinguishes between aspect nodes and the rest (referred to as context nodes), and (3) edge type view: uses a composed dependency tag for aspect-context nodes and a sum of intermediate tokens along the dependency graph for aspect-aspect nodes. \n\nThe main difference between our work with these references is synchronicity. Existing GNN-based models in ABSC (including the above references) operate asynchronously, where GNN operates after PLM finishes operating. The core idea of our work is to remove the GNN module, which inevitably causes this asynchronous behavior. Instead, we inserted a novel aggregator token, the [g] token, inside the PLM. By manipulating the attention mask of the [g] token, we successfully replaced the work of GNN and presented a fully integrated and synchronous model for ABSC. To our knowledge, this is the first work to utilize the dependency graph information without any GNN modules in ABSC. \n\nFrom the syntax information perspective, papers [1] and [2] utilize undirected graphs. While paper [3] and our work both incorporate trainable embeddings for edge information, the consideration of edge direction information is absent in paper [3]. To effectively leverage both edge label and direction information, we introduce the DAA mechanism. In the context of DAA's ablation studies (comments 4 and 5), the utilization of edge label and direction information demonstrably enhances our model's performance. \n\nWe will include the above papers and discussions in Section 2.2 of our revised paper.\n\nWe hope these revisions will make our paper more robust. Thank you again for your thoughtful feedback.\n\n---\n\nReference:\n\n* [1] Chenhua Chen, Zhiyang Teng, and Yue Zhang. 2020. Inducing Target-Specific Latent Structures for Aspect Sentiment Classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5596–5607, Online. Association for Computational Linguistics.\n* [2] Chenhua Chen, Zhiyang Teng, Zhongqing Wang, and Yue Zhang. 2022. Discrete Opinion Tree Induction for Aspect-based Sentiment Analysis. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2064, Dublin, Ireland. Association for Computational Linguistics.\n* [3] Hao Niu, Yun Xiong, Jian Gao, Zhongchen Miao, Xiaosu Wang, Hongrun Ren, Yao Zhang, and Yangyong Zhu. 2022. Composition-based Heterogeneous Graph Multi-channel Attention Network for Multi-aspect Multi-sentiment Classification. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6827–6836, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is a new way of incorporating dependency information without resorting to GNNs, which is instructive.", "reasons_to_reject": "The overall presentation is not clear enough, especially Figure 2 cannot show the main idea of ​​the work, which brings troubles to readers. In addition, the proposed DAA module uses the attention mechanism to fuse the labels of the dependency tree, but the subsequent ablation experiments did not sufficiently prove the effectiveness of the method, and no further experiments were conducted to analyze the method. There are also typos, and many illustrations are not clear enough.", "questions_to_authors": "1. How effective is DAA?\n2. How well does this method work on BERT", "paper_topic_and_main_contributions": "This work proposes a new way of incorporating dependency information without resorting to GNNs, which is instructive. But the overall presentation is not clear enough, especially Figure 2 cannot show the main idea of ​​the work, which brings troubles to readers. In addition, the proposed DAA module uses the attention mechanism to fuse the labels of the dependency tree, but the subsequent ablation experiments did not sufficiently prove the effectiveness of the method, and no further experiments were conducted to analyze the method. There are also typos, and many illustrations are not clear enough.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are also typos, and many illustrations are not clear enough.  Figure 2 cannot show the main idea of ​​the work, which requires modifying the display.", "Justification_for_Ethical_Concerns": "not involving"}, "extra_text": {"Missing_References": "Inducing target-specific latent structures for aspect sentiment classification.\nDiscrete opinion tree induction for aspect-based sentiment analysis.\nComposition-based Heterogeneous Graph Multi-channel Attention Network for Multi-aspect Multi-sentiment Classification.\nand so forth."}}]

Author rebuttal:

Meta-review: This paper proposes a new method that incorporates syntactic dependency information into a Transformer model without using GNNs for Aspect-Based Sentiment Classification (ABSC) task. Concretely, the proposed method inserts a new token [g] to represent syntactic information. Moreover, the authors introduce VDC (Variable distance control) that specifies the attention mask used by [g], and further extend it from heuristic to learnable. They also introduce DAA (dependency-aware aggregation) to represent dependency labels.    

As reviewers mentioned, the authors present a new and interesting method for ABSC. Unlike the baselines, this method does not use GNNs therefore mitigates suboptimal interaction and aggregation for the task. 

I appreciated that the authors were highly engaged in the discussion period and provided a large amount of additional results and clarification. Addressing reviewers' feedback, the authors conducted ablation experiments to show the impact of DAA. Although DAA improves the results only by a small margin, when combined with Auto-VDC the model results in the best performance. Additionally, the authors provided another set of comparisons with the BERT-based baselines. Reflecting on the authors' comment, their method is only competitive in this setting, however, methodological novelty and achieving high results with Roberta model is an important contribution. Finally, I believe incorporating additional content provided during the discussion period, will improve the paper.
Decision: Accept-Findings