Meta Auxiliary Labels with Constituent-based Transformer for Aspect-based Sentiment Analysis
Authors: Ling Min Serena Khoo
Hai Leong Chieu
Venue:  (2021)
Keywords: Natural Language Processing
Sentiment Analysis

Abstract: Aspect based sentiment analysis (ABSA) is a challenging natural language processing task that could benefit from syntactic information. Previous work exploit dependency parses to improve performance on the task, but this requires the existence of good dependency parsers.  In this paper, we build a constituent-based transformer for ABSA that can induce constituents without constituent parsers. We also apply meta auxiliary learning to generate labels on edges between tokens, supervised by the objective of the ABSA task.  Without input from dependency parsers, our models outperform previous work on three Twitter data sets and match previous work closely on two review data sets.

Under review as a conference paper at ICLR 2021
META AUXILIARY LABELS WITH CONSTITUENT- BASED TRANSFORMER FOR ASPECT-BASED SENTI- MENT ANALYSIS
Anonymous authors Paper under double-bling blind review
ABSTRACT
Aspect based sentiment analysis (ABSA) is a challenging natural language pro- cessing task that could benefit from syntactic information. Previous work exploit dependency parses to improve performance on the task, but this requires the ex- istence of good dependency parsers. this paper, we build constituent-bas transformer for ABSA that can induce constituents without constituent parsers. We also apply meta auxiliary learning to generate labels on edges between tokens, supervised by the objective of the ABSA task. Without input from dependency parsers, our models outperform previous work on three Twitter data sets and match previous work closely two review data sets.
INTRODUCTION
Aspect-based Sentiment Analysis (ABSA) is the task of predicting sentiment polarity towards ob- served aspects in a sentence. Recent work (Bai et al., [0000, Huang & Carley] 2019); [Sun et al.] 2019 Wang et al. 2020) used syntactic information from dependency parses achieve new state- of-the-art results on benchmark ABSA data sets. However, these works (i) assumed the existence of good dependency parsers, and (ii) could not further optimize the pre-defined dependency labels for downstream performance of ABSA. Motivated by these limitations, we propose to induce syntactic information with supervision from the ABSA task.
To take syntax into account, we aim to induce the necessary syntactic information for the ABSA task with inductive biases. We first design Constituent-basee Transformer (ConsTrans)) to group tokens into constituents supervised by the ABSA objective. We argue that the formation of con- stituents provides a hierarchical structure of the sentence that is suitable for sentiment analysis. For example, in the sentence "Chinese dumplings in this restaurant taste very good" with the aspect term "Chinese dumplings", ", it is important to accurately assign the phrase "taste very good" to the aspect.
Next, as seen in Figure I even though the dependency graph structures for both sentences are identical, the sentiment towards "Chelsea" is positive for the input sentence on the left and negative for the one on the right. Therefore, the type of syntactic relationship between tokens would be useful identify the sentiment towards the aspect term. Hence, we further extend ConsTrans into a Relational Constituent-baa based Transformer (ReiConsTrans) to learn relation embeddings between every pair of tokens in the input sentence. We find that simply adding relation embedding fails to outperform ConsTrans. Inspired by Liu et al. (2019), we further extend RelConsTrans to supervise the relation embedding with an auxiliary label generator (RelConsTransLLG) In previous work (e.g. Bai et al. 2020, Huang & Carley] 2019), the dependency parser played the role of the auxiliary label generator. However, such dependency parsers were not trained to provide auxiliary labels meant to improve ABSA. RelConsTransLG enables us to train the auxiliary label generator alongside the primary task to generate auxiliary labels that could directly enhance the performance of ABSA.
We evaluate our models five data sets restaurant and laptop reviews (Pontiki al. 2014). ACL14 Twitter14 data (Dong et al. 2014), Twitter15 and Twitter17 from a multi-modal ABSA data set (Yu & Jiang 2019). Compared against previous work which used dependency parsers, our models outperform them on all the Twitter data sets and matched previous work closely on the review data sets even without the use of constituent dependency parser.

---

Under review as a conference paper at ICLR 2021
root
root
nsubj
dobj
nsub)
dobj
Chelsea
defeated
Arsenal
Arsenal
defeated
Chelsea
Figure I: Dependency parse labels as auxiliary labels that help sentiment disambiguation. Tokens in bold and underlined the aspect terms. Example taken from Bai et al. 2020)
Constiuen-eased Transformer Encoder Stack
Higher Layers
Trams fommer Encoder Layer
Acd Norm
Chinese Dumaings
In this restaurant
laste
Forward
ADJ
Norm
Layers
Adjusted Mulli-Head
Head Allention Scores
Chirese
Dumpings
this
restourant
taste
Very
2o0d
Muis-Head
Altention
Constituent Probabliity Soorer
(b) A lower ConsTruns layer: the shaded region is different from vanilla Transformer.
(a) ConsTruns Encoder Stack: dotted arrows refer to lower at- tention weights between tokens from different constituents.
MODEL FORMULATION
Given a sentence of m tokens, $ (wa, .... Wm-1], and a target aspect, t (wj...., W5+g-1] of length the objective of ABSA is to predict the sentiment polarity {negative, neutral, positive) towards the target aspect mentioned in sentence s. all our models, we use the pretrained BERT (Devlin et al. 2018) model (BERT-base-uncased] to obtain contextual embeddings model, and we fine tune it together with the model. We format the input to the BERT model as a sentence pair: [CLS]
inputs to our
[SEP] +t+ + [SEP]. We represent each token w; with the representation h& bert,12 obtained from the last layer of BERT as input to our model. Our base model is 4-layer transformer on this representation, similar to the baseline Transformer(() in Bai et al. (2020). In the of this section, we describe the modifications we make to this transformer build our three proposed models, ConsTrans, RelConsTrans and RelConsTranslar
2.1 CONSTITUENT-BASSI TRANSFORMER (ConsTrans)
ConsTrans contains a stack Transformer encoder layers (Vaswani et al. (017) with Multi-Hess Attention (MHA) and a point-wiss wise feed forward sub-layer in each layer. As illustrated in Figure 2a the encoder stack of ConsTrans is grouped into two parts the lower layers and the upper layers. In all experiments, between a vanilla Transformer network and ConsTrans is that the attention scores computed in the MHA layer between a pair of tokens are adjusted based on the probability that the two tokens belong to the same constituent. In the lower layers, attention weights are adjusted such that greater attention weights are assigned to tokens within the same constituent. This adjustment layers of the encoder to allow for longer range interactions.
have 2 layers each in both the lower and upper layers. The main difference
imposed at upper
2bjurossssswws single encoder layer from the lower layers of the encoder stack. The shaded region in the figure, which emphasizes the difference from vanilla Transformer encoder layer, contains three components: the MHA which provides the vanilla attention scores, the constituent probability scorer, and finally the adjusted MHA scorer that computes the final attention.
Constituent Probability Scorer Kim et al. (000b) found that tokens from the same constituent tend to exhibit similar attention distributions. Hence propose to determine the probability that pair tokens belong to the same constituent by the similarity of their attention distributions. We use the scaled dot- dro-product attention (Vaswani et al. 2017) in the MHA layer to first obtain the attention
2

---

Under review as a conference paper at ICLR 2021
distributions of token:
exp F (h1,
(W)n))(W))))) v
of Eg exp FI= h,))
(l)(
where W E Rddmodet and W E Remostidd are projection layers that project the query and key to the various attention heads with dimension d, and du respectively. The attention distribution for token i at layer and attention head z is then defined to be the vector of = [a[]]i-- To obtain the attention distribution similarity for two tokens, and j. concatenate the attention patterns of the pair before passing it through a projection layer:
sim! V o (Walat aff)) (Walaj a!:1)
where sim refers to the attention distribution similarity score for token and token j in layer for refers to the concatenation function and the sigmoid function so that sim E [0. We also note that this ensures sim = =simf..-
attention head Wo E Rd.1 is a linear projection,
We then use the attention distribution similarity scores to compute the probability that pair of tokens belong to the same constituent. The base probability d that tokens and belong to the same constituent, is computed as follows:
I sim k,1+ Th ++k.++++++ 1>3
j
(3)
This formulation considers the probability that tokens spanned by the two tokens and j form contiguous constituent. Moreover, since sim! E [0, 1]. the probability that two tokens are in same constituent would decrease monotonically with the distance between and j. To encourage the induced constituents to be consistent across layers, the final constituent probabilities obtained at the current layer would be the weighted sum of itself and constituent probabilities from the previous layer:
- = A
S. + (1 - A) G/
1-1,2
where À E [0, 1] is aaper-parameter that we tune.
Adjusted Attention in the Lower Constituent Layers Finally, we adjust the attention scores between pair of tokens according to the probability that the pair belongs to the same constituent, through a softmax layer:
exp
0.3
(5)
Σj exp 4 a 1.3'
't' where af denotes the adjusted attention score for token i and j for layer I and attention head
2.2 RELATIONAL. CONSTITUENT-BASER TRANSFORMER (RelConsTrans)
We extend ConsTrans with the objective of learning relation embedding between pairs of tokens. For each pair of tokens, the goal is to learn an embedding that represents the syntactic relation between the pair. To generate the relation embedding, we learn non-linear projection for the concatenation of the representation for the tokens:
Tij = FLU hz)))
where ri, is the learnt embedding for token and token j, ELU is the exponential linear unit, hy and hy are BERT embeddings for token and j respectively. The learnt embedding would be included in two ways during attention computation and during information propagation stage.
Relation-aware Attention Computation To perform relation-aware attention computation, make the following changes to the adjusted scaled dot-product attention formulation in Equation before adjusting the attention scores with constituent probability in Equation 5

---

Under review as a conference paper at ICLR 2021
Entrooy Luss for ABSA
Reiational Conetitunn--aaee Transformer
ABSA Prediction label, Ypred
rwation emooddinng
Inpul Layer (BERT)
Meta-Leartt
Reletion Label Generator
Reletion Label,
Figure 3: Oveview of Relational Constituent-basee Transformer and Relation Label Generator.
12 exp Flz (H,, Σ, exp F hy
(b),)) (Won)WWk + riy) dd
where W Kr E dRdd,dd projects the learnt relation embedding Faj and is shared across attention heads. The attention weights would be determined by both textual features, h/y, and syntactic relation represented by rij-
Relation- information propagation The attention weights obtained from Equation would then be used to weigh the contribution other tokens in updating the representation of a token, h with the following equation:
1+ = W ai (Wtth) + W
(8)
where Wit E Romoded is projection layer that project the value vector to various attention heads with dimension d,. Wi Rddosmm projects the concatenation of vectors from each attention head to size dmnodet for layer Wt,, E Rdd.dd is projection layer and is shared across attention heads. Therefore, both textual and syntactic and features would be propagated from one token to another.
2.3 RelConsTrans WITH LABEL GENERATOR (RelConsTrunss..G
We found that RelConsTrans fails to outperform ConsTrans, possibly due to a lack guidance on how the relation embedding should be learned. Previous work (e.g., (Bai et al. 2020) used the depen- dency parser as an auxiliary label generator to improve ABSA. To avoid the need for dependency parser, we propose to meta learn relation label generator that would be trained alongside the pri- mary task to generate auxiliary labels optimal for enhancing the performance of ABSA. An overview of the Relational Constituent-
Transformer with the Relation Generator (RelConsTransL.C) is shown in Figure 3 The relation label generator, trained in a self-supervised manner (Liu et al.) 2019), would produce relation labels to guide the learning of the relation embedding.
As syntax information has been shown to be useful for ABSA in previous work (e.g., Bai et al.) 2020), we design our relation label generator to encourage the generation of syntax related labels as relation labels with supervision from the ABSA task. Hewitt & Manning (209) showed that the I.2 distance of a linear projection of token embeddings obtained from BERT could recover the parse tree distances between the tokens. Therefore, we learn linear transformation of the word representation space with the intention of learning syntactic relatedness. The learned syntactic relatedness would then be used as the ground truth for the L2 norm of the relation embedding. Different from (Hewitt) & Manning (2019), we do not use ground truth labels to train the relation label generator. Instead, learn this linear projection in meta learning manner.
For a pair of tokens i and j. we learn a linear projection for the BERT representation:
by W1 Wi(h bert,n
h bert.n) + b1,
where by is a scalar relation label for token i and j. W2 and b2 are the weights and bias of the linear transformation layer. bert," represents the embedding of token from the th layer of the BERT
4

---

Under review as a conference paper at ICLR 2021
Data Set
Positive
Neutral
Negative
Train
Dev
Test
Train
Dev
Test
Train
Dev
Test
Restaurant
2164
728
637
196
807
Laptop
994
341
464
169
870
128
Twitter14
1561
173
3127
346
1560
173
Twitter15
928
303
317
1883
670
607
368
149
113
Twitter17
1508
515
1638
517
573
416
Twitter14 (AS)
1538
190
3300
173
1445
288
Table 1: Statisties of the 5 benchmark data sets. TS refers to splitting the data by aspect.
model. As different layers of BERT appear to represent different types of information as shown by Tenney et al. (2019), could recover different information by selecting different BERT layers (different n). our experiments, we fixed the value n to for all data sets.
Meta Training Relation Label Generator To guide the training of the embedding, we minimize the mean square error (MSE) of the generated label and the L2 norm of the relation embedding:
MSE(1,)
Σ (1rrsll2 1.j)2
(10)
Drawing inspiration from recent work by Liu al. (019). train our label generator using the loss from ABSA with the goal of generating relation labels lij to directly optimize for the performance of the main task.
Let meei be the parameters of our main model, RelConsTrans. To update the parameters of maain we aim to minimize multi-task loss cross-entropy loss, from the ABSA prediction task and the MSE loss described in Equation 10
arg min (L + MSE(Lr)))
(1I)
Omain
Let 0then be the weights of the RelConsTrans after one gradient update step of gradient descent:
o main Omain O'mann
V arg min moinn[T(s ((ū. + MSF((r)))
(12)
@ maain
where Oraai is the learning rate to train the RelConsTrans. Note that the MSE from the relation embedding would not be used to train the relation label generator. Therefore, the parameters of the relation label generator, Oewr should be updated by solely the loss from ABSA:
arg min min(L(Ú u));
(13)
duwe
To update the weights of the generator, a second order derivative computed. While this formulation was inspired by Liu et al. (2019), the second-order derivative trick used in our model was also used in number of other meta-learning frameworks such as Finn et al. (2017).
We train the two models in tandem, over few iterations. We found it useful to train Omain and Oawx with separate training sets. For each data set, we took subset the cases which contain two or more aspects (meta-train set) in the same sentence for training dauz- This subset is removed from the main training set (train set) used to train the main RelConsTrans. More details of the meta- train set would be provided in the appendix A.I
3 RESULTS AND ANALYSIS
We conducted experiments on 5 benchmark data sets restaurant reviews, laptop reviews from Se- mEval 2014 (Pontiki et al. 2014), ACLI4 Twitter14 data set (Dong et al. (0044 and Twitter15 and Twwtter17 from multi-modas ABSA data set by (Yu & Jiang 2019). For analysis,
additional
experiments on a split of the Twitter14 44 data set by aspect. We summarize the statistics of the data in
Table I For data sets with development sets, we perform model selection the development sets.
For Restaurant, Laptop and Twitter14, 14, we compare against published results from BERT-PT (Xu al. 2019) BERT-SPC (Song al. 2019), AEN-BERT (Song et al. 2019), SDGCN-BERT Zhao
5

---

Under review as a conference paper at ICLR 2021
Data Set
Restaurant
Laptop Ace
Twitter14
Model
Acc
F1
Acc
Data Set
TwitterIS
Twitter17
BERI-PT
85.0
77.0
78.1
75.1
Model
Acc
Acc
AEN-BERT
84.5
83.1
77.0
73.8
79.0
79.9
75.0
76.3
74.7
73.6
72.1
73.1
AE-L.TMM
MemNet
70.3
70.1
61.8
63.4
61.7
64.2
60.9
58.0
SDGCN-BERT
83.6
76.5
81.4
78.3
Transformer(B3
84,9
77.9
79.3
76.1
RAM
70.7
63.1
64.4
61.0
MGAN
71.2
64.2
RGAT-Bal
86.6
80.3
813
78.6
75.8
74.7
RGAI-Wang*
74.1
76.2
BERT+BL
BERT
74.2
74.3
70.0
68.9
68.2
68.9
65.2
66.1
RGAT-Wang (re-run)"
85.7
79.1
79.0
75.6
73.6
73.1
TomBERT
77.2
71.8
70.5
68.0
L.CSSASC-CDW
DGEDT-BEKT"
86,7
86.3
80.3
80.0
79.8
80.5
77.1
75.6
77.9
75.4
ReiConsTrons
ConsTrans
76.5
76.9
71.6
72.5
69.3
69.0
68.2
67.7
RelConsTrans
ConsTrans
85.8
85.4
79.3
80.8
80.6
76.4
77.2
75.9
76.6
75.0
RelConsTramsLLG
76.8
73.3
69.8
68.5
ReIConsTransI..G
86.7
81.4
81.0
78.1
76.9
75.5
Table 2: Accuracy and F-score (FI) for 5 data sets: In the left (resp. right) table, systems marked are those that used dependency parses (resp. multi-modal information). The best Macro Fl for each data set is in bold. For significance tests, compare against RGAT- Wang(re-run), TomBERT and BERT+BL. Our results are significant against (GAT-Wang(re-run and BERT+BL. Twitter17 was significant against TomBERT (which used image data in addition text data) but not Twitte115.
al. 2020). Transformer(B) (Bai et al., p. RGAT-Bai (Bai et 2020). RGAT- Wang (Wang
et al. 2020 DGEDT-BERT (Tagg et al. 2020) and
(Phan & Ogunbona, 2020
The Transformer(B) a baseline model used by Bai et al. (2020), and the baseline vanilla Trans-
former which ConsTrans is built upon. For Twitter 15 and Twitter17, compare against pub- lished results in
Yu & Jiang 2019): MemNet (Tang et al. 2016). RAM (Chen et al.
2017).
MGAN (Fan et al. 2018). BERT, BERT+BL (Yu & Jiang 2019 and TomBERT (Yu & Jiang
2019
For the Restaurant and Twitter14 4 data sets, we outperform all previous work that did not require de- pendency parsers by competitive margins (4.8 F-score for Restaurant and 2.4 F-score for Twitter14) 14). Our results on the Laptop data (78.1) is also close to the state-of-the-art results (78.3) achieved by SDGCN-BERT. Furthermore, comparing results with models that require dependency parser, also outperform a number of models while closely matching the results of others. For Twitter15 and Twitter17, we see in Table that our best model outperforms previous work that uses only textual content by a margin (3.3 F-score for Twitter15 and 2.4 F-Score for Twitter17). Our model also outperforms TomBERT, the multi-modal models for Twitter15 and Twitter17.
To conduct statistical significance tests, we attempt to reproduce the results published for RGAT- Wang, TomBERT and BERT+BL. For RGAT- Wang, we could not reproduce their published results (with their recommended settings), and hence we can only conduct the test against the results we obtain with their open source code, shown as RGAT Wang(re-run) ization test Ych 2000) with 100,000 shuffles. We found that RelConsTrans[G outperforms RGAT- Wang(re-run) and BERT+BL significantly (p- < 0.15). RelConsTransLG significantly outperforms TomBERT (which has additional access to image data) for Twitter17, but not Twitter15.
Table We run the random-
When comparing our proposed ConsTrans model to a vanilla Transformer, we observe that Con- sTrans outperforms the vanilla Transformer model for both the Restaurant and Laptop data sets. This suggests that it is indeed useful to induce constituents for ABSA. Lastly, comparing ConsTrans and RelConsTranstee the data sets. This suggests that our meta-learnt label generator is able to generate useful auxiliary labels for ConsTrans for the ABSA task.
observe that RelConsTrrans.G consistently outperforms ConsTrans for all
3.1 ANALYSIS
In this section, we provide findings from ablation studies and analysis of our proposed models.
Grammar Induction We derive constituent trees with the constituent probabilities to verify if the derived trees resemble ground truth constituent trees. In Figure 4 we show an example where our derived constituent tree shows similar structure the ground truth constituency tree. Notably, are able to accurately recall the aspect term, "jessica alba" as a constituent. The algorithm to derive constituent trees and more examples are provided in Appendix A.4 and (.6

---

Under review as a conference paper at ICLR 2021
IPSSENT
just 1iiben anl Fatuae of
Tnaest Lompan)
all
poot present Future comsery
(rroon and Fature
IvEEST Cousir
past present asd fsture of the lenestt congare
part and luture
the honeet
TUCENS
ETe past
present and Future
LTe Forest conpars
Figure 4: Example of derived constituent Tree by ConsTrans (Left) and constituent Tree from Berke- ley Neural Parser (Kitaev & Klein 2018) (Right).
We postulate that the ability of ConsTraas to group aspect terms into noun phrases should improve its ABSA accuracy. To study our hypothesis, we looked at ConsTrans's ability to recall the entire aspect term as noun phrase. For simplicity, we only look at records where aspect terms were not broken down into sub-word tokens. For Twitter17, we found that ConsTrans achieves 67.8 recall rate for correctly predicted instances and 62.2 for incorrect instances. The Pearson correlation coefficient between prediction accuracy and recall rate was significant (with p < 0.2), indicating the usefulness of being able to induce good constituents for ABSA.
Generalibility of RelConsTransLL. The key argument by Liu et al. (2019) for designing an addi- tional label generator is to increase the generalizability of the main model. To test the generalizabil- ity of IrCConnrrans...... we create a more challenging version of Twitter 14 by splitting the data such that the train and test set comprises of different aspect terms. The statistics of the data after splitting by aspect (denoted by AS) is shown in [ We further split the train set by aspect terms to create a meta-train set train the label generator ReIConsTransLG, Therefore, the relation label gener- ator is trained to generate relation labels that enhance the performance of data with foreign aspect terms. In this AS setting, RelConsTransLG achieves a F-score of 64.3 while ConsTrans achieved a F-score of 62.8. Our designed framework mimics the actual train and test setting and is therefore able to increase the generalizability of RelConsTransIL..
Different layers of BERT as input Tenney al. 2019) found that different BERT layers encap- sulate different information useful for various NLP tasks. Therefore, we experimented with using all 12 layers of BERT as input to the label generator to study the impact on the F-score. The graph for the F-score against BERT layer (n)) is provided in Appendix A.3 Using representation from the 6th layer of BERT yields the best results for the restaurant data set and we are able to consistently outperform models that do not use dependency parses for all value of n chosen. Furthermore. this is an indication that syntactic information indeed useful for ABSA since lower layers of BERT were found to encapsulate syntactic information.
Interpreting learnt relation labels Our relation label generator is designed to encourage the gen- eration of syntax related labels. To verify the hypothesis that generated relation label is related to syntax, we reconstruct the dependency parses using the learnt relation embedding. Interestingly, while Dozat & Manning (017) have found that the L2 norm of relation embedding resembles syn- tactic distance (i.c., a lower norm means stronger dependency), we found that our learned relation embedding exhibits an opposite phenomenon: higher L.2 norm indicates a stronger dependency. We hypothesize that relation embedding with higher L2 norm would influence attention weights to a greater extent. Therefore, the L2 norm of our learnt relation embedding would represent syntac- tic relatedness rather than syntactic distance. We then construct parse trees by linking tokens with highest L2 norm of their relation embedding as detailed in Appendix AA.5
Manual inspection of these records suggest that while full parse tree was not induced, are able to recover most of adjective-noun relations, As seen in Figure we are able to retrieve the relation of (" ("pleasans". "sttff"") and ("friendly", "staff"". This is expected since understanding adjective- noun relations would be most important to ABSA compared to other types of relations. Therefore, training GG with supervision from solely ABSA would yield this behaviour.
Furthermore, to look at the ability of ReIConsTransLG to link relevant adjective terms, we engaged two annotators to annotate the adjective terms relevant each aspect term for the test set for the
7

---

Under review as a conference paper at ICLR 2021
the
Staff
extrenely
friendly
and
pleasant
Figure 5: Examples of induced dependency parses Tree by RelConsTransl.. (Top arrows) and ground truth dependency parses (bottom arrows) from StanfordNLP Arrows in grey are for opinion terms accurately linked to the aspect term "staff".
run/
Restaurant data. The annotators reconciled their differing opinions and gave each record final label. Records with no clear opinion terms given a "None" label. There were 1,120 records annotated and 811 had annotated adjective terms. We rank the relatedness of tokens with the aspect term by the L2 norm of the relation embedding and compare it with ground truth ranks. Ground truth ranks were obtained by ranking tokens with their syntactic distance obtained from StanfordNL. dependeney parser (Chen Manning 2014) with tied rank taken into account. For records where the adjective term was more than syntactic distance away, we obtain equal or smaller rank than the syntactic distance in 63.5% of the cases. Compared against position offset ranks, we obtain an equal or small rank than the number of position offsets in 65.0% of the cases.
RELATED WORK
Sentiment analysis (Pang & Lee 2008) is well studied natural language processing problem. Early works applied sentiment analysis to product reviews as a text classification problem. However, a review social media post could express different sentiments to different aspects, and the task of aspect-based sentiment analysis aims at a finer classification of sentiment towards specific as- pects (Dong et al. 2014) or aspects (Pontiki et al., 2016).
Recent work on ABSA has shown that the use of dependency parses for ABSA helps to improve performance (Bai et al. 22200) Huang & Carley| [2r1 Sun al. 2019 Waan al.) 2020). However, supervised dependency parsers require substantial amount of annotated data, and might perform badly for out-of-domain (e.g., social media) or low-resource languages. On the other hand, it has been shown that contextual embeddings such BERT (Devlin al. 2018) contain significant in- formation that could be useful to parsers (Clark et al. 2019 Kim et al. 2020a). Previous work such as Hewitt & Manning (2019) have shown that linear projection is sufficient to recover syntactic information from BERT embedding. In this paper, we show that we can achieve similar ABSA performance without supervised parsers, by leveraging BERT which was trained with raw data.
Previous work on unsupervised grammar induction such as [hhnn et al. (2019); Kim et al. (2019) aims to induce grammar from data. Our primary objective not to induce grammar, but to encourage the model to learn to perform the ABSA task by learning the causal edge dependencies between constituents. We show that our approach is able to achieve results that rivals those obtained by models that have access to supervised dependency parsers.
In this work, applied meta auxiliary learning (Liu et al. 2019) which learns generate auxiliary labels, supervised by the primary task. While Liu et al. (2019) failed to interpret the auxiliary labels for the computer vision tasks they worked on, showed that in our case, the induced auxiliary labels can be interpreted as syntactic relatedness to certain extent.
CONCLUSION
In this paper, we apply a meta auxiliary learning approach to the ABSA task, and we show that the induced relations between phrases are interpretable and supports the primary task of sentiment analysis. We show that learning the auxiliary labels improve results data sets. Without using dependency parsers, our approach performs competitively compared to previous work that used dependency parses input.
baselines all five
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Xuefeng Bai, Pengbo Liu, and Yue Zhang. Exploiting typed syntactic dependencies for targeted sentiment classification using graph attention neural network. CoRR, abs/2002.09685,, 2020. URL
http://orrraa..//oooooas 09655
Danqi Chen and Christopher Manning. A fast and accurate dependency parser using neural net- works. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 740-750, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.31 1..1115v1/D14-11 1082. URL www.h:tpps//.......m http:///wwww.. a web. org/ hatt//anthllooooooo ogy/ D14-100 082
Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang. Recurrent attention network mem- ory for aspect sentiment analysis. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 452-461, Copenhagen, Denmark, September 2017. As- sociation for Computational Linguistics. doi: 10. 10.18663/11/D1--1077. URL https::/wwm
acl wel.ebb.rgg/ee http//nthhooggggg bccLebbb.rr//aanttoooooo//01--
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. CoRR, Abss/1906.04341, 2019. URL htllhttt:///xxiii org/
abs 190/ 04341
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805, 2018. URL
http: org/ abs/1810 wtw.zivvorrg//bs111100
04805
Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pp. 49-54, Baltimore, Maryland, June 2014. Association for Computational Linguistics. doi: 10.3115/v1/ P14-2009. URL https: www. aclweb. org. http:///aatooooorr//------0000
Timothy Dozat and Christopher D. Manning. Deep biaffine attention for neural dependency pars- ing. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: / //openreview.
Feifan Fan, Yansong Feng, and Dongyan Zhao Multi-grained attention network for aspect-level sentiment classification. In Proceedings the 2018 Conference on Empirical Methods Natural Language Processing, pp. 3433-3442, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/Di8.. 1380. URL wtttt:tts/// www. acl web.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. CoRR, abs/1703.0344l 2017. URL http::///axxv..rrrr/ abtgg//bbs/ii 03400
John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representa- tions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Pa- pers), pp. 4129-41331 Minneapolis, Minnesota, June 2019. Association for Computational Lin- guisties. doi: 10.18653/v1/111--444999 URL https: /www wwwweb. http://wwwwwbbb.//aart....tee wwtpe//.....g/aaaooogoooo.o. N19 1419
Binxuan Huang and Kathleen Carley. Syntax-aware aspect level sentiment classification with graph attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5469-5477, Hong Kong, China. November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D19. 1549. URL https: / www. aclweb. org/ antholoogyDD1915..

---

Under review as a conference paper at ICLR 2021
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Confer- ence on Learning Representations, 2020a. https:::
net/ forummidrmm
H 11IxPRNNtPR
Taeuk Kim, Jihun Choi, Daniel Edmiston, and Sang goo Lee. Are pre-trained language models aware of phrases? simple but strong baselines for grammar induction. In International Confer- ence on Learning Representations, 2020b. URL CRLLhtts:://// https:://penrevvew.... net. forum?id H1xPR3NtPB
Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. Unsu- pervised recurrent neural network grammars. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 1105-1117, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/1//11--1111 URL
https: / www.www..ccc.. www.aaclweb.org.. org wtt..orr//nooooggg/ge/eee.e
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. h
7/arxiv. orq/ abs/ 1112. 6980
Nikita Kitaev and Dan Klein. Constituency parsing with a self-attentive encoder. Abs/1805.01052 2018 URL
CoRR,
01052
Shikun Liu, Andrew Davison, and Edward Johns. genesuuuppriie with meta auxil- iary learning. CoRR, abs//900.089909 2019. URL/ttpp//aaxxvv.rr//bb///////
Bo Pang and Lillian Lee, Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval, 2(1 2(1-2):1-135, 2008.
Minh Hieu Phan and Philip O. Ogunbona. Modelling context and syntactical features for aspect- based sentiment analysis. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 321111112221 1-3220, Online, July 2020. Association for Computational Linguistics. doi: 10. 1/118/6///22200ccll actimm......mmmm.m92999... URL https: hittps:/wwit //wp:///wwwwwo
aclweb. awwp://wwwwwwwwwbbbbb.. org/
anthology/2220.
main.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 27-35, Dublin, Ireland, August 2014. Association for Computational Linguistics. doi: 10.3115/v1/$14-2004 URL https: / http://wwwwwllcl.lccc www.wb.wwwee................ www.a/wwwwww.....ooooooooooooo hwt..///w......attoooooog///////
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clereq, et al. Semeval-2016 task 5: Aspect based sentiment analysis. In 10th International Workshop Se- mantic Evaluation (SemEval 2016). 2016
Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. Ordered neurons: Integrating tree structures into recurrent neural networks. International Conference on Learning Repre- sentations, 2019, http::/t::////ttt http:///ppeeeevveeeeeee net/ mett/oorummdddsl11....
Youwei Song, Jiahai Wang, Tao Jiang, Zhiyue Liu, and Yanghui Rao. Attentional encoder network for targeted sentiment classification. CoRR, abs/1902.09314, 2019. URL http:://axxi:
prg/abs/1902. 09314
Kai Sun, Richong Zhang, Samuel Mensah, Yongyi Mao, and Xudong Liu. Aspect-level senti- ment analysis via convolution over dependency tree. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- Natural Language Processing (EMNLP-IJCNLP,, pp. 5679-5688, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v//D19-111 1569. URL https: // www. a web. www.org//atholllleoooooooooooooe htwp:/tthhoooooolllo/eegp
10

---

Under review as a conference paper at ICLR 2021
Duyu Tang, Bing Qin, and Ting Liu. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 214-224, Austin, Texas, November 2016. Association for Computational Lin- guistics. doi: 10. 10.18633/1/D111-00211 URL https:: /ttp://wwww//////o ac www..w..ww...................... www.rrrrr/ato...eeoeoooee ogy/ D16-1021
Hao Tang, Donghong Ji, Chenliang Li, and Qiji Zhou. Dependency graph enhanced dual- transformer structure for aspect-based sentiment classification. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 6578-6588 66788 Online, July 2020. Association for Computational Linguistics. doi: (0.18653/v1/2020.co main.588. mainmm. URL
Sattps:/vvet http://wwwwwwww...o.. /ttpo///aa.ooooogg//22000oooooo acl main. 588,
lan Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. CoRR, abs/1905.05950, 2019. URL httpa//////.///////aaaaooo/ao///
05950
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, abs/1706.03762, 2017.
URL htttp:/aaaaxi org/ http//aa/1111555353 03762
Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. Relational graph attention network for aspect-based sentiment analysis. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi: 10.18653/w/20220.a.. m95.i.... URL
http://dx.cr doi
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. BERT post-training for review reading compre- hension and aspect-based sentiment analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 2324-2335, Minneapolis. Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1/11--1222 URL
https: / www www.acllwa ac web. orgg barrg/antholoyy.. 9-
Alexander Yeh. More accurate tests for the statistical significance of result differences. In Pro- ceedings of the 18th Conference on Computational Linguistics Volume 2, COLING '00, pp. 947-953, USA, 2000. Association for Computational Linguistics. doi: 10.3115//92730.999777 URL
3111///011111/999333303 992783
Jianfei Yu and Jing Jiang. Adapting BERT for target-oriented multimodal sentiment classification. In Sarit Kraus (ed.), Proceedings of the Twenty-Eights International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pp. 1490855444, ijcai.org, 2019. doi: 10. 2e.22446jjaii20099999/ 444.0//0100..///////.///////2//0 20119//7755
Pinlong Zhao, Linlin Hou, and Wu. Modeling sentiment dependencies with graph convolutional networks for aspect-level sentiment classification. Knowledge-Basee Systems, 193:105443, 2020.

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 META-SPLITTING DATA SET
To generate the meta-train set, first obtain subset of samples with multiple aspects of differing sentiment from the full train set. 80% of this subset would then be used as the meta-train set. For Twitter2014 (AS), we sampled this meta-train set by aspects. The intention Label Generator to generate labels that can encourage the model to accurately link relevant opinion terms the aspect term through challenging samples. The statistics of the meta-train set is provided in T
train the Relation
Data set
Positive
Neutral
Negative
Total
Restaurant
195
162
520
Laptop
85
98
61
244
Twitter2014
265
567
301
1133
Twitter2015
123
161
340
Twitter2017
320
470
111
901
Twitter2014 (AS)
207
253
254
714
Table 3: Statistics of the meta-train sets. AS refers splitting the data by aspect.
A.2 IMPLEMENTATION DETAILS
In all experiments, we used the ADAM optimizer (Kingma & Ba, 2015) with a learning rate of 10-3 optimize the models (ConsTrans, RelConsTrans, generator). The BERT model was fine-tuned together with the main model built top of it. We used 6 multi-head attention heads and the size the hidden layer was set to be 384. We stacked encoder layers. Lower layers defined to be the first 2 layers in the encoder stack while the last layers are higher layers. Dropout was applied to all input embeddings. The L2 penalisation term for the model's parameters set to be le-5. The size of the learnt embedding for each relation label was set to be 786 and batch size of 4 was used to train all the models.
and the relation label
A.3 DIFFERENT LAYERS OF BERT
The graph for Macro F-score against BERT layer (n) is shown Figure 6 Using representation from the 6th layer of BERT yields the best results for the restaurant data set and we used the 6th layer of BERT input to the relation label generator for all the other data sets.
81.5
63.0
80.5
60.0
00.0
79
79.4
79.2
BERT Layer (n)
Figure 6: Macro F-score on Restaurant data set with different BERT layer as input to LG,
A.4 ALGORITHM TO DERIVE CONSTITUENT TREES
To derive constituent trees, we iteratively split the input constituents into 2 parts till each constituent only made up of only token. The breaking point is defined to be the point where the probability of being in the same constituent for token and i+1 1 the lowest. We use the constituent probability from the second layer of our encoder stack find the breaking point.
12

---

Under review as a conference paper at ICLR 2021
Algorithm Unsupervised constituent tree derivation
1: m < Constituent layer chosen
2: x < Input tokens
3: <- Constituent probability from layer m for current constituent
4: stack = [a]]
5: constituents 0
6:
7: procedure GETCONSTTREE(T, p)
8:
while len(stack) != 0 do
9:
current_consi stack[0]
10:
Po + constituent probabilities for current_const
b argmin(6g)
Find the breaking point
12:
left const x[:b]
right const x[b:]
14:
if len(left_const) > then
add left_const to constituents end if
16:
17:
len(right. coosst) then
18:
add right.const constituents
19:
end
20:
add left.const and right- const consliluents end while
21:
Return constituents
23:
end procedure
A.5 ALGORITHM TO INDUCE DEPENDENCY PARSES
To recover parse trees, we iteratively linked tokens to another token with the highest L2 norm for the learnt relation embedding. We do not allow tokens to be linked multiple times.
Algorithm 2 Unsupervised dependency parse tree induction
1: procedure GETDEPNTREE(C p)
2: depen_Jaeel: = 0]
3: relation_norm f- The L2 norm of learnt embedding
4: for + length. do
5: current_man max(relation.normg
6:
i, j ppwsttion(currenttmmaa.
> Get the position of max L2 norm
7:
add (i.j) to depen labels
8: relation norm[: j] = -CO;
D Do not link to token j
9: relation_normli,
OC;
Do not allow loops
10: end for
Return depen
12:
end procedure
A.6 EXAMPLES OF DERIVED CONSTITUENT TREES
We provide examples of constituent trees derived from ConsTrans. In general, while not able to fully replicate ground truth constituent trees, we noticed that the model was able to recall noun phrases reasonably.
13

---

Under review as a conference paper at ICLR 2021
gelting really wolk Use inmuriaa day parall with rain jumt
nith Piolie willar
Arthing ready to salt in the arsortal day parade with rain ruy
Icees
with Ptults veober
gotting ready to volk
the ecmortol day porode with By našn puy
ktti
fdu1t wetoon
getting resdy to salk
the nonoriol da porase wits
by wzin guy
rwadyy
cayy
Recorial
Figure 7: Example of derived constituent Tree by ConsTrans with aspect term "@ Jullia webber".
hba uestern cont anfenencc rinals
goldon state warriors shock oklahona city thunder, take
nbuu
jue's Lern conference finals
golden state warriors shock oklahoona city Uhunder, Lake
western confecence
Finals
goldon state warriors shack oklahona city thundor,
take ....
western
conterence
gelden state Narriors
shock oklahona city Uhunder
goldon
warriors
shock
oklahona city thundar,
goldon
state
oklahona city thundor
cklahona
city thunder
Figure& 8: Example of derived constituent Tree by ConsTrans with aspect term "@golden state war- riors".
sadiq kahn
was western like donald trump nigel farage
win was retuke to western nativists like donald trung
and nigel farage
win
Nas
western nativists donald trump
and
nigol fanage
was rebuke to western
nativists like donald trurp
Nas
a rebuike to western
nativists like donald
trump
rebuie
westerr
nativists
Like donald
a
rebuke to
Figure 9: Example of derived constituent Tree by ConsTrans with aspect term "sadiq Kahn's".
A.7 EXAMPLES OF INDUCED DEPENDENCY PARSE TREES
We provide more examples of induced dependency parses from RelConsTransLG: While in- duced parse trees were not identical to ground truth parse trees, we were able to link adjective terms noun phrases reasonably.
14

---

Under review as a conference paper at ICLR 2021
everything
about
place
adorable
even
bathroom
Figure 10: Examples of induced dependency parses Tree by RelConsTransLG (Top arrows) and ground truth dependency parses (bottom arrows) from StanfordNLP Stanfor htapp::/ccrree http///oreeep..u.r Arrows in grey for opinion terms accurately linked the aspect term "place".
the
host
owner
and
servers
personable
and
caring
Figure 11: Examples of induced dependency parses Tree by RelConsTransLG (Top arrows) and ground truth dependency parses (bottom arrows) from StanfordNLP Arrows in grey are for opinion terms accurately linked to the aspect term "owner".
run/
For
appetizer
their
calamari
is
winner
Examples of induced dependency parses Tree by RelConsTrauss..G (Top arrows) and ground truth dependency parses (bottom arrows) from Stanford StanforNNLPhtttss:/// Arrows in grey for opinion terms accurately linked the aspect term "calamari".
Figure 12:
15


Reviews:
[{"review_id": "wmp9BxX3dEV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a Transformer-based model for aspect-based sentiment analysis, intended to support the unsupervised induction of constituents within the Transformer forward pass. Their evaluations demonstrate that their model can match (and in some cases improve upon) models which depend upon explicit dependency parse information in the input, and reliably exceed parse-free models.  \n\nI strongly vote for rejection, largely on grounds of quality, elaborated below.  \n\nPros: The paper begins with an interesting idea and implementation, and the results support the claim that their architecture may replicate some of the contribution of dependency parse information.\n\nCons: The presentation is unclear on the concept of \"constituent\" and the motivation of the model. The later iterations of the model become rather complicated and don't seem well-motivated. The qualitative evaluations don't strongly support the claims of the paper.  \n\nQuality\n\n1. The design of the model and the use of the word \"constituent\" seems conceptually problematic to me. What do you take the word \"constituent\" to mean for your motivation and model design? It seems to me that it might be sufficient to call the ConsTrans model a \"spatial attention smoothing\" model. Why isn't this a sufficient description? What does the concept \"constituent\" add?This question is relevant because the later model iterations are developed on further syntactic ideas (typed/labeled relations, syntactic \"distance\"). But if the model doesn't have a necessary syntactic framing, it's not clear these are the correct model improvements to consider.\n2. The qualitative evaluations are far too light, testing only a small amount of the model performance.\n  a. Grammar induction: In particular, I would appreciate a far more in-depth evaluation of the inferred constituent structures. How do they compare to gold and silver dependency parses of within-domain sentences? The current evaluation checks for model inferences on just one short span of text (the aspect term). This is probably one of the easiest terms for the model to recognize as a constituent, too, since the aspect terms are known to be constituents and have verbatim copies in the input sentence. The current evaluation is also only performed on Twitter17 --- why?\n  b. Interpreting learnt relation labels: I found this evaluation extremely confusing, involving an ad-hoc dependency parsing algorithm built upon an a posteriori fact discovered in model analysis (that relation embedding L2 norm indicates inverse syntactic distance). The resulting parse in Figure 5 is almost entirely incorrect and commits many basic mistakes (for example, not linking the determiner \"the\" with its immediately adjacent noun). The claim about linking adjectives and nouns is not particularly interesting to me, since this is far less ambitious than the motivation of the model --- if it were, the model could have been quite a bit simpler, I think.\n3. Significance results are given (thanks!) but with a strangely high significance threshold (0.15 at one point and 0.2 at another). This is not a reasonable significance threshold in my view.  \n\nClarity\n\nSome minor comments:\n\n1. The constituent derivation algorithm is not clear. Eqn 3 suggests that constituent probabilities are a function of token pairs, but Algorithm 1 line 11 suggests that they can be indexed by a single token position. Is something missing from the algorithm presentation?\n2. Eqn 3 first condition should be i <= j, I think.\n3. Figure 4 is not a complete sentence. There's no verb. There's also no clear sentiment (aspect-based or otherwise) that we could talk about for this sentence. A different sentence could serve as both a motivating example for constituency and as a more revealing example of the model's syntactic knowledge.\n\n## Post-rebuttal response\n\nI have read the other reviews and the authors' responses, and do not wish to change my review. The proposed model seems quite complex with somewhat unclear conceptual motivations, and does not clearly demonstrate impressive performance gains despite the complexity. I would suggest that the authors attempt to change one of these things in a later paper, either by revisiting the model design, or task choice and evaluation (to better motivate the model).", "extra_scores": {}, "extra_text": {}}, {"review_id": "uJBRu69Qpa", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "According to two phenomena in Transformer's pre-trained language model: \n1. Tokens with similar self-attention distribution tend to be distributed in the same constituent; \n2. The L2 distance between token representations reveals the degree of syntactic dependency; \n\nan aspect-based sentimental analysis system is designed which does not rely on explicit syntactic information. It is not a surprising point to combine the so-called syntactic information learning with sentential analysis. Although the model has achieved comparable results with systems using explicit syntax, the meta-learning is somewhat overclaimed.\n\nThere are some defects in this work:\n1. Over dependence on the pre-trained language model: Although the author claims that explicit syntactic information is not used, the system relies on the syntactic information contained in the pre-trained language models such as BERT to promote the training of ConsTrans, RelConsTrans, and RelConsTransLG. I doubt whether the training of the model can converge without the pre-trained language model only relying on the randomly initialized transformer and still achieve comparable accuracy with systems with explicit syntax? What is the performance of using a weaker pre-trained language model like Distibert? How about the performance in stronger pre-trained models like RoBERTa?\n2. Model size and inference performance: the extra transformer layers added makes the result not directly comparable with that of BERT. Therefore, the author needs to report the effect of the BERT baseline under the same parameter scale. In addition, to what extent will the introduction of the constituent probability scorer and relation label generator affect the inference performance of the model? For this simple task of sentiment analysis, do we need such a complex model design and training method design after the already huge pre-trained language model such as BERT?\n3. Inappropriate description: \n(1) In RelConsTrans, the proposed model encodes the syntactic relation as a feature for constituent scoring based on the role of syntactic relation for sentiment analysis. But according to Eq. 6 and 7, the so-called relation (that could not be determined) act as a bias in scoring function, I think it just provides a more feature source (dependency syntax) for scoring. \n(2) In RelConsTransLG, the r_{i,j} and l_{i,j} is similar. I think the only conclusion that can be drawn is that the model is further enhanced with the features of dependency syntax recently, which has little relationship with meta-learning. \n\nGenerally speaking, this work can be regarded as integrating the implicit constituent and dependency syntactic features from the pre-trained language models into the training in the sentient analysis, so that explicit syntax is no longer needed. Appendix A.6 and A.7 also confirm this point. Although it has a positive effect on the need to remove explicit syntax, its reliance on pre-trained language models limits the extensibility of this approach and is likely to be ineffective on traditional unpretrained LSTM, CNN, and Transformer models. In addition, since the writing is pretended to make the model close to a concept of meta-learning, it ignores the important points that should be clarified.", "extra_scores": {}, "extra_text": {}}, {"review_id": "uDLoSKFBej_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a constituent-based transformer model with auxiliary relation embeddings and labels to enhance the performance of aspect-based sentiment analysis. The constituent-based transformer modifies the original transformer by re-weighting the attention weights with constituent-based similarities. The auxiliary relation predictions helps to differentiate the relations that are useful for sentiment prediction. \n\nIn general, this paper is readable and has a clear motivation. The strengths include:\n1. The idea of modifying the basic transformer to a constituent-based transformer by incorporating constituent similarities without any supervision is interesting. \n2. The authors propose three model variations progressively, namely ConsTrans, RelConsTrans and RelConsTransLG to clearly demonstrate the motivation and effect of each variation.\n\nHowever, the paper still lack the following aspects:\n1. Some parts of the description is unclear with details missing. From (9), the generated $l_{ij}$ is a continuous value, how do you make it a relation label? Why do you take the L2 norm on $r_{ij}$ in (10)? What is the intuition of such computation to obtain the MSE? What is $L(\\hat{y},y)$ in (11) and how to compute it? If $y$ is the sentiment label of the aspects, how it could be used to update the label generator $l_{ij}$? What is the intuition of training $\\theta_{main}$ and $\\theta_{aux}$ using separate datasets? Could you show what is the difference in terms of the performance?\n2. Given the complexity of the model, the improvement compared to the baseline models are relatively trivial. And it is somewhat insufficient to limit the application only to aspect-based sentiment analysis. The contribution is thus limited.\n3. How do you sample the meta-train set? Is the result over one meta-train set or averaged over different meta-train set?\n4. Minor points: caption is missing for figure 2. The first row in (3) should be $i\\leq j$.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a constituent-based transformer for aspect-based sentiment analysis. The approach allows conducting aspect-based sentiment analysis to leverage the syntactic information without pre-specified dependency parse trees. 

Overall, the idea is interesting. However, all the reviewers shared the following concerns: 

- Paper descriptions of methodology and experiments are not clear and require significant rewriting and reorganization. 
- The proposed approach is not well-justified by the empirical study presented in the paper. Especially, a more detailed ablation study is required to justify the design. 

We would suggest the authors addressing the feedback from the reviewers to improve the paper. 
Decision: Reject