Multi-Dimensional Explanation of Reviews
Authors: Diego Antognini
Claudiu Musat
Boi Faltings
Venue:  (2020)
Keywords: deep learning
explanation
interpretability
reviews
multi-aspect
sentiment analysis
mask

Abstract: Neural models achieved considerable improvement for many natural language processing tasks, but they offer little transparency, and interpretability comes at a cost. In some domains, automated predictions without justifications have limited applicability. Recently, progress has been made regarding single-aspect sentiment analysis for reviews, where the ambiguity of a justification is minimal. In this context, a justification, or mask, consists of (long) word sequences from the input text, which suffice to make the prediction. Existing models cannot handle more than one aspect in one training and induce binary masks that might be ambiguous. In our work, we propose a neural model for predicting multi-aspect sentiments for reviews and generates a probabilistic multi-dimensional mask (one per aspect) simultaneously, in an unsupervised and multi-task learning manner. Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable.

Under review as a conference paper at ICLR 2020
MULTI-DIMENSIONAL EXPLANATION N OF REVIEWS
Anonymous authors
Paper under double-blind review
ABSTRACT
Neural models achieved considerable improvement for many natural language processing tasks, but they offer little transparency, and interpretability comes at a cost. In some domains, automated predictions without justifications have limited applicability. Recently, progress has been made regarding single-aspect sentiment analysis for reviews, where the ambiguity of justification minimal. In this context, a justification, or mask, consists of (long) word sequences from the input text, which suffice make the prediction. Existing models cannot handle more than one aspect in one training and induce binary masks that might be ambiguous. In our work, we propose neural model for predicting multi-aspect sentiments for reviews and generates a probabilistic multi-dimensionl mask (one per aspect) si- multaneously, in an unsupervised and multi-task learning manner. Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaning- ful, and interpretable.
INTRODUCTION
Neural networks have become the standard for many natural language processing tasks. Despite the significant performance gains achieved by these complex models, they offer little transpurency con- cerning their inner workings. Thus, they come at the cost of interpretability (Jain & Wallace 2019).
In many domains, automated predictions have real impact on the final decision, such as treatment options in the field of medicine. Therefore, it is important to provide the underlying reasons for such a decision. We claim that integrating interpretability in a (neural) model should supply the reason of the prediction and should yield better performance. However, justifying a prediction might be ambiguous and challenging. Prior work includes various methods that find the justification in an input text - also called rationale or mask of target variable. The mask is defined as one or multiple pieces of text fragments from the input text [ Each should contain words that altogether short, coherent, and alone sufficient for the prediction as substitute of the input (Lei et al. 2016).
Many works have been applied single-aspect sentiment analysis for reviews, where the ambiguity of a justification is minimal. In this case, we define an aspect as an attribute of a product or ser- vice (Giannakopoulos al.) (017)), such Location or Cleanliness for the hotel domain. There are three different methods to generate masks: using reinforcement learning with a trained model (Li al. 2016b), generating rationales in an unsupervised manner and jointly with the objective func- tion (Lei et al. 2016), or including annotations during training (Bao et al. 2018 Zhang et al. 2016).
However, these models generate justifications that are 1) only tailored for one aspect, and 2) ex- pressed as a hard (binary) selection of words. A review text reflects opinions about multiple topics a user cares about (Musat et al., 2013). appears reasonable to analyze multiple aspects with a multi-task learning setting, but model must be trained as many times as the number of aspects. A hard assignment of words to aspects might lead to ambiguities that are difficult to capture with binary mask: in the text "The room was large, clean and close to the beach.". the word "room" refers to the aspects Room, Cleanliness and Location. Finally, collecting human provided rationales scale is expensive and thus impractical.
In this work, we study interpretable multi-aspect sentiment classification. We describe an architec- ture for predicting the sentiment of multiple aspects while generating probabilistic (soft) multi-
In the rest of the paper, we will the terms mask, justification and rationale interchangeahly.

---

Under review as a conference paper at ICLR 2020
dimensional mask (one dimension per aspect) jointly, in an unsupervised and multi-task learning manner. We show that the induced mask is beneficial for identifying simultaneously what parts of the review relate to what aspect, and capturing ambiguities of words belonging multiple aspects. Thus, the induced mask provides fine-grained interpretability and improves the final performance.
Traditionally interpretability came at cost of reduced accuracy. In contrast, evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable compared to attention-based methods and a single-aspect masker. We show that it can be a benefit to 1) guide the model to focus on different parts of the input text, and further improve the sentiment prediction for all aspects. Therefore, interpretabilty does not come at a cost anymore.
The contributions of this work can be summarized as follow:
We propose a Multi-Aspect Masker (MAM), an end-to-end neural model for multi-aspect sentiment classification that provides fine-grained interpretability in the same training. Given a text review as input, the model generates a probabilistic multi-dimensional mask, with one dimension per aspect. It predicts the sentiments of multiple aspects, and highlights long sequences of words justifying the current rating prediction for each aspect;
We show that interpretability does not come cost: our final model significantly out- performs strong baselines and attention models, both in terms of performance and mask coherence. Furthermore, the level of interpretability is controllable using two regularizers;
Finally, we release a new dataset for multi-aspect sentiment classification, which contains 140k reviews from TripAdvisor with five aspects, each with its corresponding rating
2
RELATED WORK
2.1
INTERPRETABILITY
Developing interpretable models is of considerable interest to the broader research community, even more pronounced with neural models (Kim et al.] [2015) Doshi- Velez & Kim 2017). Many works analyzed and visualized state activation (Karpathy et al. 2015; Li et al.) [2016a] Montavon et al. 2018), learned sparse and interpretable word vectors (Faruqui al. 2015b a Herbelot & Vecchi, 2015) or analyzed attention (Clark et al. 2019]]Jain & Wallace 2019 Our work differs from these terms of what is meant by an explanation. Our system identifies one multiple short and coherent text fragments that - as substitute of the input text - are sufficient for the prediction.
2.2 ATTENTION-BANE BASED MODELS
Attention models Vaaswani al. 2017 Yang al.) 2016 Lin et al. 2017) have been shown to improve prediction accuracy, risualization and interpretabillity The most popular and widely used attention mechanism is soft attention (Bahdanau al., 2015) over hard (Luong et al. 2015) and sparse ones (Martins & Astudillo, 2016). According to Jain & Wallace (019)): & Smith (2019), standard attention modules noisily predict imput importance; the weights cannot provide safe and meaningful explanations.
Our model differs two ways from attention mechanisms: our loss includes two regularizers to fa- vor long word sequences for interpretability; the normalization is not done over the sequence length.
2.3 MULETI-ASPECT SENTIMENT CLASSIFICATION
Review multi-asped sentiment classification is sometimes seen as a sub-problem (Wang al. 2010 McAuley et al. 2012 Pappas& & Popescu-Belle 2014), by utilizing heuristic-based methods or topic models. Recently neural models achieved significant improvements with less feature engineering. Yin et al. (0017) built a hierarchical attention model with aspect representations by using set of manually defined topics. [i et al. (2018) extended this work with user attention and additional features such overall rating, aspect, and user embeddings. The disadvantage of these methods is their limited interpretability, as they rely on many features in addition to the review text.
We will make the code and data available.
2

---

Under review as a conference paper at ICLR 2020
2.4 RATIONALE-BASE MODELS
The idea of including human rationales during training has been explored in (Zhang et al. 2016
Marshall et al. 2015 Bao al., 2018). Although they have been shown be beneficial, they expensive to collect and might vary across annotators. In our work, no annotation is used.
The work most closely related ours Li et al. (2016b) and Lei et (2016). Both generate hard rationales and address single-aspect sentiment classification. Their model must be trained separately for each aspect, which leads ambiguities, Li al. (2016b) developed post-training method that removes words from review text until another trained model changes its prediction. Lei et al. (2016) provides model that learns an aspect sentiment and its rationale jointly, but hinders the per- formance and relies on assumptions on the data, such as a small correlation between aspect ratings.
In contrast, our model: 1) supports multi-aspect sentiment classification, 2) generates soft multi- dimensional masks in single training; 3) the masks provide interpretability and improve the per- formance significantly.
METHOD: MULTI-ASPECH MASKER
Masker
Outout for Interpretability
Sugdwes ypeye
Probability distribution over aspects for each word
Embeddings
e e
NMO
MR
Text
Output Predictions
Shared Encoder
Classifier a
A
Boafna Maa
NOJ
has
Yes
Word embeddings multiplied by each aspect distribution
Encoded review per aspect 0;
FNN 04
Rating per aspect aq
Figure 1: The proposed Multi-Aspect Masker (MAM) model architecture for A aspects.
Let be review composed of L words x² x.2 ...,,.."""" xL and Y the target A-dimensional senti- ment vector, corresponding to the different rated aspects. Our proposed model, called Multi-Aspect Masker, is composed of three components: bution over aspects for each word, resulting in A+1 I different masks (including one for not-aspect); 2) an Encoder that learns representation
Masker module that computes a probability distri-
review conditioned on the induced masks; 3) a Clas- sifier that predicts the target variables. The overall model architecture is shown in Figure I Our framework generalizes for other tasks, and each neural module is interchangeable with other models.
The Masker first computes a hidden representation h for each word zf in the input sequence, re Many sequence models could realize this task, such as
ing their word embeddings el, .e.,...c'' c². recurrent, attention, convolution neural networks. In case, chose convolutional network because it led to a smaller model, faster training, and empirically, performed similarly to recurrent models. Let a; denote the 7t aspect for I,..., A, and an the not-aspect case, because many words can be irrelevant to every aspect. We define M R(++1), the aspect distribution of the input word of as:
L P(M = II P(M))) = IIII P(m ==1=
L A
(I)
Because we have categorical distributions, we cannot directly sample P(M* and backpropagate the gradient through this discrete generation process. Instead, we model the variable mf. using the Straight Through Gumbel Softmax (Jang al. 2017 Maddison et al. 2017). to approximate sampling from a categorical distribution. We model the parameters of each Gumbel Softmax dis- tribution M with single-layer feedforward neural network followed by applying log softmax, which induces the log probabilities of the ft distribution: W, = log (ooftmmaxxxWW" + 6)). W and shared across all tokens, to have constant number of parameters with respect the sequence
3

---

Under review as a conference paper at ICLR 2020
length. We control the sharpness of the distributions with the temperature parameter Com- pared to attention mechanisms, the word importance is probability distribution over the targets: En P(m., (z*) 1, instead of a normalization over the sequence length, Σ/= P(a7(=")=
Given soft multi-dimensional mask M [((+))xx we define each sub-mask Ma, E TR. as:
Ma: = P(m., [zl), P(m |=²), ,.., P(mm),
We weight the word embeddings by their importance towards an aspect a, with the induced sub- masks, such that Es, = Ma, о E = P(ma, |xt) ( P(m², lee),,,,... P(m (a*) (er).ez. Thereafter, each modified embedding Ea, is fed into the Encoder block. Note that Eas is ignored because Ma only serves to absorb probabilities of words that are insignificant to every aspect
The Encoder module includes convolutional neural network, for the same reasons earlier, fol- lowed by a max-over-time pooling to obtain a fixed-length feature vector. It produces the hidden representation ha; for each aspect dg. To exploit commonalities and differences among aspects, we share the weights of the encoders for all E,,- Finally, the Classifier block contains for each aspect a two-layer feedforward neural networks followed by softmax layer to predict the sentiment Da..
3.1 INTERPRETABLE MASKS
Attention model
Multi-Aspect Masker Trained on Exent with Xps Lee,, and loont
Trained on Esent and no constraint
stayed at daulsol september 2013 and could n't have asked for anymore for the price !! it is great location only 2 miuntes walk to jet space and sankeys with short drive to ushnaia the hotel is has sic but cleaned daily and i did nt have any problems at all with the bathroom or kitchen facilities the lady at reception was really helpful and explained every- thing we needed to know .... even when we managed to miss our flight She let us stay around and use the facilities until got later fight . there are loads of restaurants in the vicinity and supermarkets and shops right outside loved these apartments so much that booked to come back for septemher 2014 | can not wait g
stayed at daulsol september 2013 and could n't have asked for anymore for the price ! it is great only 2 minutes walk to jet space and sankeys with short drive to ushuaid the hotel is ba- sic but cleaned daily and i did nnt have any problems at all with the bachroom kitchen facilitieg the lady at reception was really helpful and explaimed every- thing we needed to know even when we managed to miss our flight she let stay around and use the facilitics until we got on later flight there are loads of restaurants in the vicinity and supermarkets and shops right outside loved these apartments so much that booked to come back for september 2014 !! can not wait !
location
Aspect Changes: 30
Aspect Changes:
Justifications obtained for a hotel review, with an attention model and Multi-Aspect Masker, where the colors represent the aspects: Service, Cleanliness, Value, Location, and Room. Masks lead to mostly long sequences describing clearly each aspect (one switch per aspect), while at- tention to many short and interleaving sequences (30 changes between aspects), where most relate to noise or multiple aspects. Highlighted words correspond to the highest aspect-attention scores above 1/L (i.e., more important than a uniform distribution), and the aspect a; maximizing P(m",, l2").
Figure 2:
The first objective to optimize is the sentiment loss, represented with the cross-entropy between the true aspect sentiment label ya, and the prediction Da.:
Esent Σ leross.entropy Da.)
(3)
Training Multi-Aspect Masker to optimize Event will lead to meaningless sub-masks Map. as the model tends to focus on certain key-words. Consequently, we guide the model to produce long and meaningful sequences of words, as shown in Figure [ We propose two regularizers: the first controls the number of selected words, and the second favors consecutive words belonging to the
ii P(mág (=') 1.0, it implies that ΣA, P(mm, and consequently, eú, Ö.
4

---

Under review as a conference paper at ICLR 2020
same aspect. For the first term Eeet, calculate the probability Pael of tagging word as aspect and then compute the cross- cross-entropy with a parameter Ap. The hyper-parameter Ap can be interpreted as the prior on the number of selected words among all aspects, which corresponds to the expectation of syse) as the optimizer will try to minimize the difference hetween psel and A
L - Psel T. (1- lx
(4)
see Ebinary Chinary.cross.co wtwpsrrossssiieee crossenttopy (Pael: Ap)
The second regularizer discourages aspect transition between two consecutive words, by minimizing the mean variation of two consecutive aspect distributions. We generalize the formulation in Lei et al. 2016), from a hard binary single-aspect selection, to soft probabilistic multi-aspect selection.
L Σ Pdis = T. t=1
A
Σ
P(ma
P(m
(1mm-11 (o (
(5)
lcont Ebinary cross entropy (Pdis:
Finally, we train our Multi-Aspect Masker an end-to-end manner, and optimize the final loss AMA Esent Avet - Esel + Acont beant, where Asel and Acont control the impact of each constraint.
4 EXPERIMENTS
In this section, assess model on two dimensions: the predictive performance and the quality of the induced mask. We first evaluate Multi-Aspect Masker on the multi-aspect sentiment classifi- cation task. In second experiment, measure the quality of the induced sub-masks using aspect sentence-level annotations, and an automatic topic model evaluation method
4.1 DATASETS
McAuley et al. 2012) provided 1.5 million beer reviews from BeerAdvocat. Each contains multiple sentences describing various beer aspects: Appearance. Smell, Palate, and Taste; users also provided a five-star rating for each aspect. Lei et al. (2016) modified the dataset to suit the requirements of their method As consequence, the obtained subset, composed of 280k reviews, does reflect the real data distribution: it contains only the first three aspects, and the sentiment correlation be- tween any pair of aspects is decreased significantly (27. 7.% on average, instead of 71.8% originally). We denote this version as the Filtered Beer dataset, and the original one as the Full Beer dataset.
To evaluate the robustness models across domains, we crawled 140k hotel reviews from TripAd- visor. Each review contains five-star rating for each aspect: Service, Cleanliness. Value, Location, and Room. The average correlation between aspects high (63.0% on average). Compared to beer reviews, hotel reviews are longer, noisier, and less structured, as shown in Appendix A.3
As in Bao al. (2018). we binarize the problem: ratings at three and above are labeled as positive and the rest as negative. We further divide the datasets into 80/ 10/10 for train, development, and test subsets (more details Appendix A.I).
4.2 BASELINES
We compared our Multi-Aspect Masker (MAM) with various baselines. We first used a simple baseline, Sentiment Majority, that reports the majority sentiment across aspects, as the sentiment correlation between aspects might be high (see Section 4.1). Because this information is not avail- able at testing, we trained a model to predict the majority sentiment of review using Wang & Manning (2012). The second bascline we used is a shared encoder followed by A classifiers, that denote Emb Enc Clf. This model does not offer any interpretability. We extended it with
*The detailed experimental setup descrihed in Appendix A.2
Foor the three first aspects, they trained a simple linear regression model predict the rating of an aspect given the others and then selected reviews with the largest prediction error.
5

---

Under review as a conference paper at ICLR 2020
Table Performance of the best models of each architecture for the Filtered Beer dataset.
F1 Score
Interp.
Model
Params
Macro
A1
A2
Aa
None
Sentiment Majority
426k
68.89
67.48
73.49
65.69
Emb20o + ENC + Clf
173k
78.23
78.38
80.86
75.47
Coarse-
Embzau
ENCENN + A Stuured ClI
196k
78.19
77.43
80.96
76.16
grained
Embz0o + LnCLSIM + A Shared
+ Clf
186k
78.16
75.88
81.25
77.36
NB-SVM (Wang & Manning 2012)
3 4266
74.60
73.50
77.32
72.99
SAM (Lei et al. 2016
3 641kk
77.06
77.36
78.99
74.83
Fine- grained
Emb20o
EnG,STM
Clf
458k
78.82
77.35
81.65
77.47
Aspact-wise
Emb200
EnG,
Aspast-wise +
458k:
78.96
78.54
81.56
76.79
Embz0o + Masker EnC CIf (Ours)
274k
79.32
78.58
81.71
77.66
Embgau-s EnceeN
Clf (Ours)
175k
79.66
78.74
82.02
78.
shared attention mechanism (Bahdanau et al. 2015) after the encoder, noted A Shared that provides coarse-grained interpretabiliy; for all aspects, the network focuses on the same words in the input.
Our final goal is to achieve the best performance and provide fine-grained interpretability; to visu- alize what sequences of words model focuses on and to predict the aspect sentiment predictions. To this end, we included other baselines: two trained separately for each aspect and two trained with a multi-axpect sentiment loss. We employed for the first ones: the well-known NB-SVM of Wang & Manning (2012) for sentiment analysis tasks, and the Single Aspect-Mask (SAM) model from Lei et al. (2016), each trained separately for each aspect. The two last methods are composed of separate encoder, attention mechanism, and classifier for each aspect. We utilized two types of attention mechanism: additive (Bahdanau et al. 2015), and sparse (Martins & Astudillo 2016). We call each variant Multi -spect-Attentions (MAA) and Multi Seaee-SSpare-AAttttiiin (MASA). Diagrams for the baselines can found in Appendix A.5
4.3 MULTI-ASPECT SENTIMENT CLASSIFICATION
In this section, we enquire whether fine-grained interpretability can become a benefit rather than cost. We group the models and baselines in three different levels of interpretability:
None: cannot identify what parts of the review are important for the prediction:
Coarse-grained: can identify what parts of the reviews important predict all aspect sentiments, without knowing what part corresponds to what aspect;
Fine-grained: for each aspect, we can identify what parts are used to predict its sentiment.
4.3.1 BEER REVIEWS
Overall Fl scores (macro and for each aspect A₁ for the comtrolled-envirommmntt Filtered Beer (where there are assumptions on the data distribution) and the real-world Full Beer dataset are shown in Tand Table 2
We find that our Multi-Aspect Masker (MAM) model 8, with 1.7 2.1 times fewer parameters than wsse attention models ( ①), performs better average than all other baselines on both datasets, and provides fine-grained interpretability. For the synthetic Filtered Beer dataset, MAM achieves significant improvement of least 0.36 macro F1 score, and 0.05 for the Full Beer one.
To demonstrate that the induced sub-masks Ma. .... Ma. are 1) meaningful for other models to improve final predictions, and 2) bring fine- -reeggr interpretability, we extracted and concatenated the masks to the word embeddings, resulting in contextualized embeddings (Peters al.) 2018).

---

Under review as a conference paper at ICLR 2020
Table Performance of the best models of each architecture for the Full Beer dataset.
F1 Score
Interp.
Model
Params
Macro
A1
A2
Ag
A4
None
Sentiment Majority
560k
73.01
71.83
75.65
71.26
73.31
Emb200
Enc Clf
188k
76.45
71.44
78.64
71.88
80.83
Coarse-
Emb200
ENCCNN
Shund Clf
226k:
77.06
73.44
78.68
75.79
80.32
grained
Emba0o
ENC Shwed
Clf
219h
78.03
74.25
79.53
75.76
82.57
NB-SVM Wang & Manning
2012
4 560k
72.11
72.03
74.95
68.11
73.35
SAM Lei al. 2016)
1 641k
76.62
72.93
77.94
75.70
79.91
Fine-
grained
Emb200 + EnCLSIM A Sparse
Aspect-wise Clf
611
77.62
72.75
79.62
75.81
82.28
Emb200 ESC Aspectowise
Clf
611k
78.50
74.58
79.84
77.06
82.53
Emb20o Masker + EnC + Clf (Ours)
289k
78.55
74.87
79.93
77.39
82.02
Emb20044 + Enc + Clf (Ours)
191k
78.94
75.02
80.17
77.86
82.71
We trained simple Encoder-Classifier with the contextualized embeddings ② which has approxi- mately 1.5 times fewer parameters than MAM. We achieved macro Fl score absolute improvement of 0.34 compared to MAM, and 1.43 compared the non-contextualized variant for the Filtered Beer dataset; for the Full Beer one, the performance increases by 0.39 and 2.49 respectively. Simi- larly, each individual aspect A, Fl score of MAM improved
similar extent.
We provide in Appendix A.3.I and A.3.2 visualizations of reviews with the computed sub-masks May...,, Ma, and attentions by different models. Not only do sub-masks enable the reach of higher performance; they better capture parts of reviews related to each aspect compared to other methods.
Both NB-SVM 4 and SAM S offering fine-grat tineggraieed interpretability and trained separately for each aspect, significantly underperform compared the Encoder-Class Claassffirr !. This result is expected: the goal SAM is to provide rationales at the price of performance (Lei et al. 2016), and NB-
SVM might not perform well because of its simplicity. Shared attention models (2 3 perform similarly to the Picooer-CClasiffer D. but provide only voarse-grainee interpretability.
However, in the Full Beer dataset, SAM 5 obtains better results than the Encoder-Classfie base- line I and NB-SVM ④ which is outperformed by all other models. It might be counterintuitive that SAM performs better, but claim that its behavior comes from the high correlation between aspects: SAM selects words that should belong to aspect aj to predict the sentiment of aspect aj (a: j az). Moreover, in Section 4.5, show that a single-aspect mask from SAM cannot be employed for interpretability.
Finally, Sentiment Majority O is outperformed by large margin by all other models in the Filtered Beer dataset, because of the low sentiment correlation between aspects. However, in the realistic dataset Full Beer, Sentiment Majority obtains higher score and performs better than NB-SVM ① These results emphasize the ease of the Filtered Beer dataset compared to the Full Beer one, because of the assumptions not holding in the real data distribution.
4.3.2 MODEL. ROBUSTNESS HOTEL REVIEWS
On the Hotel dataset, the learned mask M from Multi-Aspect Masker 8 again meaningful, by increasing the performance and adding interpretability. The Encoder-Classifer with contextual- ized embeddings 2 outperforms other models significantly, with an absolute macro Fl score improvement of 0.49. Moreover, it achieves the best individual F1 score for each aspect Ai,
Visualizations of reviews, with masks and attentions, are available in Appendix A.3.3 The inter- pretability comes from the long sequences that MAM identifies, unlike attention models. In com- parison, SAM 5 lacks coverage and suffers from ambiguity due to the high correlation between aspects. We observe that Multi-Aspect Masker O performs slightly worse than aspect-wise atten-
7

---

Under review as a conference paper at ICLR 2020
Table Performance of the best models of each architecture for the Hotel dataset.
F1 Score
Interp.
Model
Params
Macro
A1
A2
Ag
A4
As
None
Sentiment Majority
309
85.91
89.98
90.70
92.12
63.09
91.67
Emb300
Enc Clf
263k
90.30
92.91
93.55
91.12
76.65
91.29
Coarse-
Emb30o
EnC
Shaned Clf
301k
90.12
92.73
93.55
93.76
76.40
94.17
grained
Emba0o
ENCLSIM
Shared
Clf
270k
88.22
91.13
92.19
93.33
71.40
93.06
NB-SVM Wang & Manning
2012
5 309k
87.17
90.04
90.77
92.30
71.27
91.46
SAM Lei al. 2016
5 824kk
87.52
91.48
91.45
92.04
70.80
91.85
Fine-
Emb200
EnCLSIM
A Sparse
Aspect-wise Clf
1010k:
90.23
93.11
93.32
93.58
77.21
93.92
grained
Embs00
EnC Aspectowise
Clf
1010k
90.21
92.84
93.34
93.78
76.87
94.21
Emb300 Masker + EnC + Clf (Ours)
404k
92.84
92.95
93.91
76.27
93.71
Emb300+5 + Enc + Clf (Ours)
267k
90.79
93.38
93.82
94.55
77.47
94.71
Table 4: Precision of selected words for each aspect. Percentage of words corresponds to the ratio of the number of highlighted words to the full review. All models are trained on Filtered Beer.
Smell
Palate
Appearance
Interp.
Model
Prec.
% Words
Prec.
% Words
Prec.
% Words
SVM
Lei et al.
2016
21.6
7%
24.9
7%
38.3
13%
SAA
88.4
7%
65.3
7%
80.6
13%
Fine-
SAM
Lei et al.
2016
95.1
7%
80.2
7%
96.3
14%
grained
MASA
87.0
4%
42.8
5%
74.5
4%
MAA
51.3
7%
32.9
7%
44.9
14%
MAM (Ours)
96.6
7%
81.7
7%
96.7
14%
The model has been trained separately for each aspect.
tion models ? + K, with 2.5 times fewer parameters. We emphasize that using the induced masks in the Encoder- Encoder-Classifer 9 already achieves the best performance.
The Single Aspect-Mass ④ obtains the lowest relative macro Fl score the three datasets: a differ- ence of 3.227; 226 and 2..2 for the Filtered Beer and Full Beer dataset respectively. This proves that the model is not meant to provide rationales and increase the performance simultaneously.
Finally, show that learning soft multi dimensioal masks along training objectives achieves strong predictive results, and using these to create contextualized word embeddings and train a baseline model with, provides the best performance across the three datasets.
4.4 MASK INTERPRETABILITY
In these experiments, we verify that Multi-Aspect Masker generates induced masks Ma, .... Mas that, in addition to improving performance, are meaningful and can be interpreted by humans.
4.4.1 MASK PRECISION
Evaluating justifications that have short and coherent pieces of text challenging because there is no gold standard provided with reviews. McAuley et al. (2012) provided 994 beer reviews with aspect sentence-level annotations, although model computes masks at finer level. Each sentence of the
8

---

Under review as a conference paper at ICLR 2020
Table 5: Average Topic Coherence (NPMT) across different top-N words for each dataset. Each aspect is considered as a topic and the masks (or attentions) are used to compute P(wla.).
NPMI
Model
N 5
5
N
10
N = 15 N 20 N 25
N 30
Mean
SAM' (Lei et al. 2016)
0.123
0.149
0.134
0.169
0.219
0.248
0.174
Filtered
Beer
MASA
0.024
0.059
0.159
0.200
0.271
0.325
0.173
MAA
0.072
0.103
0.141
0.208
0.259
0.325
0.185
MAM (Ours)
0.042
0.114
0.171
0.216
0.276
0.329
0.192
SAM' (Lei et al. 2016)
0.046
0.120
0.129
0.243
0.308
0.396
0.207
Full
Beer
MASA
0.020
0.082
0.130
0.168
0.234
0.263
0.150
MAA
0.064
0.189
0.255
0.273
0.332
0.401
0.252
MAM (Ours)
0.083
0.187
0.264
0.348
0.410
0.477
0.295
SAM' (Lei et al.
2016
0.041
0.103
0.152
0.180
0.233
0.281
0.165
Hotel
MASA
0.043
0.127
0.166
0.295
0.323
0.458
0.235
MAA
0.128
0.218
0.352
0.415
0.494
0.553
0.360
MAM (Ours)
0.134
0.251
0.349
0.496
0.641
0.724
0.432
The model has been trained separately for each aspect.
Metric that correlates best with human judgment (Lau & Baldwin 2016).
dataset is annotated with one aspect label, indicating what aspect that sentence covers. We evaluate the precision of words highlighted by each model. For both, ours and Lei et (2016). we used trained models on beer reviews and extracted a similar number of selected words.
We show that the generated sub-masks Me Ma Mes obtained with Multi Aspect Masker (MAM) correlate best with human judgment. Table[I the precision of the masks and attentions com- puted on sentence-level aspect annotations. We reported results of the models in Lei al. (016):: SVM, the Single Aspect-Attention (SAA) and Single Aspect-Mast Mask (SAM) trained separately for each aspect because they find hard justifications for single aspect. In comparison to SAM, our MAM model obtains significant higher precisions with an average of +1.13. Interestingly, SVM and attention models perform poorly compared with mask models: especially MASA that focuses only couple of words due to the sparseness of the attention (examples in Appendix A.3.1 A.3.I
4.5 MASK COHERENCE
In addition to evaluating masks with human annotations, we computed their semantic interpretability for each dataset. According to Aletras & Stevenson (2013); Lau et al. (2014). NPMI (Bouma) 2009) is a good metric for qualitative evaluation of topics, because it matches human judgment most closely. However, the top-N topic words, used for evaluation, are often selected arbitrarily. To alleviate this problem, we followed Lau & Baldwin (2016): we computed the topic coherence over several cardinalities N, and report all the results, as well as the average; the authors claim the mean leads to more stable and robust evaluation. More details are available in Appendix A.4
We show that generated masks by MAM obtains the highest mean NPMI and, on average, superior results in all datasets (17 out of 21 cases), while only needing single training. Results are shown in Table S For the Hotel and Full Beer datasets, where reviews reflect the real data distribution, model significantly outperforms SAM and attention models for N > 20. For smaller N, MAM obtains higher scores in four out of six cases, and for these two, the difference is only below 0.003.
For the -eviroommneeniiieeeettt Filtered Beer dataset, MAM still performs better for N 15, al- though the differences are smaller, and is beat by SAM for < 10. However, SAM obtains poor results in all other cases of all datasets and must be trained as many times as the number of aspects.

---

Under review as a conference paper at ICLR 2020
We show the top words for each aspect and human evaluation in Appendix A.4 Generally, our model finds better sets of words among the three datasets compared with other methods.
CONCLUSION
In this work, we propose Multi- Multi-Aspect Masker, an end-to-end neural network architecture to per- form multi-aspect sentiment classification for reviews. Our model predicts aspect sentiments while generating a probabilistic (soft) multi-dimensional mask (one dimension per aspect) simultaneously, in unsupervised and multi-task learning manner. We showed that the induced mask is beneficial to guide the model to focus on different parts of the input text and to further improve the sentiment prediction for all aspects. Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable compared to attention- based methods and single- aspect masker.
REFERENCES
Nikolaos Aletras and Mark Stevenson. Evaluating topic coherence using distributional semantics. In Proceedings of the 10th International Conference on Computational Semantics (IWCS 2013)
Long Papers, pp. 13 22, 2013.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings (Baddanau et al. (2015). URL arxiv. org/ abs/ 1409.0477 0473
Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing, pp. 1903-1913, Brussels, Belgium, October-November 2018, Association for Com- putational Linguistics. doi: 10.18653/v1/D13 1/11//12..1
URL https: www.w.awwweb.org.
anthology/DI8-12L1
G. Bouma. Normalized (pointwise) mutual information in collocation extraction. In C. Chiarcos, E. de Castilho, and M. Stede (eds.), Von der Form zur Bedeutung: Texte automatisch verarbeiten / From Form to Meaning: Processing Texts Automatically, Proceedings of the Biennial GSCL Conference 2009, pp. 31 40-4,, Tübingen, 2009. Gunter Narr Verlag.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276-286, Florence, Italy, Au-
gust 2019. Association for Computational Linguistics. URL https: www. www.wb.wwwb............ orrg/
anthology//WI944228
Finale Doshi Velez and Been Kim. Towards rigorous science of interpretable machine learning.
arXiv preprint arXiv: 1702.1170.000000 086008 2017.
Manaal Faruqui, Jesse Dodge, Sujay Kumar Jauhar, Chris Dyer, Eduard Hovy, and Noah A. Smith. Retrofitting word vectors to semantic lexicons. In Proceediiggs of the 2015 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1606-1615, Denver, Colorado, May-June 2015a. Association for Computational Linguistics. doi: 10.3115/v1/115-111 1184, URL httpstt:::/ / /twp:///wwww.///.. acwweb..b......................b org/
anthology/N155111
Manaal Faruqui, Yulia Tsvetkov, Dani Yogatama, Chris Dyer, and Noah A Smith. Sparse overcom- plete word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( \Volume 1: Long Papers), pp. 1491-150M 2015b.
Athanasios Giannakopoulos, Diego Antognini, Claudiu Musat, Andreea Hossmann, and Michael Baeriswyl. Dataset construction via attention for aspect term extraction with distant supervision. In 2017 IEEE International Conference on Data Mining Workshops (ICDMW), pp. 370-380. IEEE, 2017.
10

---

Under review as a conference paper at ICLR 2020
Aurélie Herbelot and Eva Maria Vecchi. Building shared world: Mapping distributional to model- theoretic semantic spaces. In Proceedings of the 2015 Conference on Empirical Methods in Nat- ural Language Processing, pp. 22-32, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. 1735-1780) 1997.
Long short-term memory. Neural computation, 9(8):
Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3543-3556, 2019.
Eric Jang. Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings Jang et (2017). URL https:://ppeneeeee
net/
forum?id=rkE33
Andrej Karpathy, Justin Johnson, and Fei-Fei Li. Visualizing and understanding recurrent networks. CoRR, abs/ 506.02078. 2015. URL
02078
Been Kim, Julie A Shah, and Finale Doshi-Velee Mind the gap: A generative approach to inter- pretable feature selection and extraction. In Advances in Neural Information Processing Systems, pp. 2260-2268, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings Kingma & Ba (2015). URL http: org/ abs/ 1412 6980
Jey Han Lau and Timothy Baldwin. The sensitivity of topic coherence evaluation to topic cardinality. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483 9833487,11 2016.
Jey Han Lau, David Newman, and Timothy Baldwin. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pp. 530-539, 2014.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. Rationalizing neural predictions. In Proceedings of the 2016 Conference on Empirical Methods Natural Language Processing, pp. 107-117, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 18.18653/v1/ D16- 1011. URL htt.::ttss///e
aclweb org/ anthooogy/DDD- 1011
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nip. In Proceedings of the 2016 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 681-691, 201 6a.
Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding neural networks through representation erasure. arXiv preprint 1rXiv:1612.0822. 2016b.
Junjie Li, Haitong Yang, and Chengqing Zong. Document-level multi-aspect sentiment classification by jointly modeling users, aspects, and overall ratings. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 925 936,9366422 2018
Zhouhan Lin, Minwei Feng, Cícero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. In 5th International Confer- ence on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference www./eeete//id-eee...e
Track Proceedings Lin et 2017) URL
5uqx
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-based
neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412-1421, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 1tt/3/////D1111111111 1166. URL https:: /wwww. aclweb.
lorg/antho how.hhhh1ool1111116..11
11

---

Under review as a conference paper at ICLR 2020
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings Maddison et al. www.wweeewwwe............ net/ htt///orrmmmdd//// JE51551
(2017).
lain J Marshall, Joël Kuiper, and Byron C Wallace. Robotreviewer: evaluation of system for automatically assessing bias in clinical trials. Journal of the American Medical Informatics Asso- ciation, 23(1):193- 2015.
Andre Martins and Ramon Astudillo. From softmax to spansemax: A sparse model of attention and multi-label classification, In International Conference on Machine Learning, pp. 1614-1623, 2016.
Julian McAuley, Jure Leskovec, and Dan Jurafsky. Learning attitudes and attributes from multi- aspect reviews. In Proceedings of the 2012 IEEE 12th International Conference on Data Min- ing. ICDM 12. pp. 1020-1025, Washington, DC, USA, 2012. IEEE Computer Society. ISBN 978-0-7695-4905-7 doi: 10. 10.1109/1CDM.202..11 URL http: dx. dei. dww.oo.oo......ooooooooooo..1... 1109/ ICDM.2012.00
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed represen- tations of words and phrases and their compositionality. In Advances in neural information pro- cessing systems, pp. 3111-3119, 2013.
Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller. Methods for interpreting and un- derstanding deep neural networks. Digital Signal Processing, 73:1-15, 2018.
Claudiu Cristian Musat, Yizhong Liang, and Boi Faltings. Recommendation using textual opinions. In Twenty- -hind International Joint Conference on Artificial Intelligence, 2013.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Nikolaos Pappas and Andrei Popescu-Belis. Explaining the stars: Weighted multiple-instance learn- ing for aspect-based sentiment analysis. In Proceedingy of the 2014 Conference on Empirical Methods In Natural Language Processing (EMNLP), pp. 455-466, 2014.
Matthew Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Papers), pp. 2227-2237, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/1//N1-12222 1202. URL https: http://wwwww.o www ac www.wwwwwwbb......c............w orq/ Batpoooogyyy/111111
Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th An- nual Meeting of the Association for Computational Linguistics, pp. 2931-2951, Florence, Italy, July 2019. Association for Computational Linguistics. URL https: www. aclweb. orq/ anthology/E19-1222
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 1529):1999-198, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Hongning Wang, Yue Lu, and Chengxiang Zhai. Latent aspect rating analysis review text data: a rating regression approach. In Proceedings of the 16th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 783-792. ACm, 2010.
12

---

Under review as a conference paper at ICLR 2020
Sida Wang and Christopher Manning. Baselines and bigrams: Simple, good sentiment and topic classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 90 94, Jeju Island, Korea, July 2012. Association for Computational Linguistics. URI.tttps. / www. ac web wwwboorgg/attoo
ogy/plyyp////. 2018
Zichao Yang. Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. Hierarchical attention networks for document classification. In Proceedings of the 2016 conference the North American chapter of the association for computational linguisties: human language technologies, pp. 1480-1489, 2016.
Yichun Yin, Yangqiu Song, and Ming Zhang. Document-level multi-aspect sentiment classification machine comprehension. In Proceedings the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2044-2054, 2017.
Ye Zhang, lain Marshall, and Byron C. Wallace. Rationale-augmented convolutional neural net- works for text classification. In Proceedings of the 2016 Conference on Empirical Methods in Nat- ural Language Processing, pp. 795-804 Austin, Texas, November 2016. Association for Com- putational Linguistics. doi: 10.18653/v1/D16- 1076.
URL https:/ www. wwwweb...o....o.
antho Btccology/DD6-111 1076
13

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 DESCRIPTIVE STATISTICS OF THE DATASETS
Table 6: Statistics of the multi-aspect review datasets. Full Beer and Hotel represent real-world beer and hotel reviews respectively. Filtered Beer contains a subset of beer reviews with assumptions, leading to more straightforward and unrealistic dataset.
Dataset
Filtered Beer
Full Beer
Hotel
Number of reviews
280, 000
1,586, 259
140, 000
Average word-length of review
157.5 84.3
147.1 + 79.7
188.3 50.0
Average sentence-length of review
11.0 - 5.7
10.3 + 5.4
10.4 + 4.4
Number of aspects
3
4
5
Average ratio of over & reviews per aspect
3.29
12.89
1.02
Average correlation between aspects
27.2%
71.8%
63.0%
Max correlation between two aspects
29.8%
73.4%
86.5%
A.2 EXPERIMENTAL DETAILS
For each model, the review encoder was either bi-directional single-layer forward recurrent neural network using Long Short-Term Memory (Hochreiter & Schmidhuber 1997) with 64 hidden units or the multi-channel text convolutional neural network, similar Kim et al. (015). with 3, 5, width filters and 50 feature maps per filter. Each aspect classifier is a two-layer feedforward neural network with ReLU activation function (Nair Hinton (010)) We used the 300-dimensional pre- trained word embeddings of Lei et al. 2016) for beer reviews. For the hotel domain, trained word2vec (Mikolov al. 2013) on large collection of hotel reviews with an embedding size of 300.
We used dropout (Srivastava et al. 2014) of 0.1, clipped the gradient norm at 1.0 if higher, added L2-norm regularizer with regularization factor of patience of three iterations. We used Adam (Kingma & Ba 2015) for training with learning rate of 0.001, 31 = 0.9, and B2 = 0.999. The temperature т for Gumbel-Softme fixed at 0.8. The two regularizer terms and the prior of model are Agel 0.03, Acont 0.04, and Ap = 0.11 for the Filtered Beer dataset; Agel 0.03, Avont = 0.03, and Ap = 0.15 for the Full Beer dataseet; and Asel 0.02, Acont 0.02 and Ap 0.10 for the Hotel dataset. We all experiments for maximum of 50 epochs with a batch-size of 256 and Titan X GPU. For the model of Lei et al. (2016), we reused the code from the authors.
and trained using early stopping with
distributions was
A.3 VISUALIZATION THE MULTI-DIMENSIONAL FACETS OF REVIEWS
We randomly sampled reviews from each dataset and computed the masks and attentions of four models: our Multi-Aspect Masker model (MAM), the Single Aspect-Mask method (SAM) of Lei al. (006)) and two attention models with additive and sparse attention, called Multi Aspect- Attentions (MAA) and Multi Aspect-Sparse-Attennte Atteetionss (MASA) respectively (more details Sec- aspect and the shade its confidence. All models generate soft attentions or masks besides SAM, which does hard masking, Samples for the Filtered Beer, Full Beer and Hotel dataset shown below.
tion 4.2p. Each color represents
14

---

Under review as a conference paper at ICLR 2020
A.3.1 FILTERED BEER DATASET
Appearance Smell
Palate
Appearance
Smell
Palate
Multi Aspect-Masks (Ours)
Single Aspect- Mask (Lei et al., 2016)
ruby red brown color Huffy white single
ruby red brown color Buffy white single
finger head settles down thin cap coating
fnngrr head settles down thin ca coating
thin lacing over the sides the glass some
thin lacing all the sides the glass some
Taint burnt sweet malt smells but little else and
faint burnt sweet malt smells but little else and
very faint, t: taste is very solid brown malts
very faint taste is very solid brown malts
and some sweetness maybe some toffee biscuit
and some sweetness. maybe some toffee biscuit
and burnt flavors too m decent carbonation
and burnt flavors too im decent carbonation
followed by mediumi bodied feel Bavvr coats the tongne for a very satisfying and lasting finish [
followed by medium bodied feel Blavor poats the tougue for a very satisfying and lasting finish
an easy drinker as good brown should be
an easy drinker as good brown should be
my wife is a big brown fan so 'Il definitely be
my wife is a big brown fan 80 'Il definitely be
grabhing this one for her again a solid beer for any time the year.. served in a standard pint glass
grahhing this one for her again a solid beer for any time the year.. , served in a standard pint
glass
Appearance Smell
Palate
Appearance Smell Palaté
Multi Aspect-Attentions
Multi Aspect-Sparse
Atatettppu Attentions
ruby red brown in color fluffy off white single
ruby red brown in color Butfy off white single
finger head settles down to thiu cap coating
finger head settles down a thin cap coating thin lacing all over the sides the glass some faint burnt, sweet malt smells, but little else and very faint t: taste is very solid for brown malts and some sweetness maybe some toffee biscuit and burnt flavors too m decent carbonation is followed by a medium bodied feel. flavor coats the tongue for a very satisfying and lasting finish d : an easy drinker as good brown should be my wife is a big brown fan
thin lacing all over the sides the glass some faint burnt. sweet malt smells, but little else and
very faint taste is very solid for brown malts and some sweetness may some toffee biscuit and burnt flavors too m decent carbonation is followed by a medium bodied foel. flavor coats the tongue for very satisfying and lasting finish
: an easy drinker as a good brown should be
my wife is a big brown fan i 'Il definitely be grabbing this one for her again a solid beer for any time of the year served in standard pint glass
'll definitely be grabbing this one for her agnin a sulid beer for any time of the year . served in a standard pint
glass.
Figure 3: Our model MAM highlights all the words corresponding to aspects. SAM only highlights the most crucial information, but some words are missing out, and one is ambiguous. MAA and MASA fail to identify most of the words related to the aspect Appearance, and only a few words have high confidence, resulting in noisy labeling. Additionally. MAA considers words belonging to the aspect Taste whereas the Filtered Beer dataset does not include it in the aspect set.
15

---

Under review as a conference paper at ICLR 2020
Appearance Smell Palate
Appearance Smell
Palate
Multi Aspect-Mast (Ours)
Single Aspect- Mask (Lei et al., 2016)
crystal clear gold aunt Buffy three hnger white crystal clear gold tann Juffy three hinerr white
head that holds it own very well when falls it head that holds it own very well when it falls it
falls
ring full white lace glass clean
falls
ring full white
glass clean
crisp floral pine eitrie lemon crisp biscuit
emsp floral pine citric lemon t- crisp biscuit
malt up front hops all the way through grassy
malt up front hops all the way through, grassy
lemon tart yeast at finish, hop bitterness through
lemon tart yeast at finish. hop bitterness through finish m- dry bubbly coarse high carbonation light bodied hops leave impression on palette nice hop bitterness good flavor, sessionable. recommended. good brew
finish m- dry bubbly poarse high carbonation
light bodied hops leave impression on palette
nice hop bitterness good flavor, sessiouable recommended. good brew
Appearance Smell Palate
Appearance Smell Palate
Multi Aspect Attentions
Multi Aspect Aspect-Sparse Sparse- Attentions
crystal clear gold taunt Huffy three finger white head that holds own very well when falls falls to a 1/2 ring full white lace on glass clean crisp floral pine citric lemon crisp biscuit malt front hops all the way through grassy lemon, tart yeast fimish hop bitterness through finish dry bubbly coarse high carbonation light bodied hops leave impression on palette d- nice hop bitterness good flavor sessionable recommended good brew
crystal clear gold taunt Huffy three finger white head that holds own very well when falls falls to a 1/2 ring full white lace on glass clean crisp floral pine citric lemon crisp biscuit malt frout hops all the way through, grassy lemon, tart yeast finish. hop bitterness through finish m- dry bubbly coarse high carbonation light bodied hops leave impression on palette d- nice hop bitterness good flavor sessionable. recommended good brew
Figure 4: MAM finds the exact parts corresponding the aspect Appearance and Palate while covering most of the aspect Smell. SAM identifies key-information without any ambiguity, but lacks coverage. MAA highlights confidently nearly all the words while having some noise for the aspect Appearance. MASA selects confidently only most predictive words.
16

---

Under review as a conference paper at ICLR 2020
A.3.2 FULL BEER DATASET
Appearance
Smell
Palate
Taste
Appearance
Smell
Palate
Taste
Multi Aspect-Masks (Ours)
Single Aspect- Mask (Lei et al., 2016)
sa 's harvest pumpkin ale 2011 had this last year loved it and bought 6 harvest packs and saved the pumpkins and the dunkel 's ... not too sure why sa dropped the dunkel, i think it would make
sa 's harvest pumpkin ale 2011 had this last year > loved it and hought 6 harvest packs and saved the pumpkins and the dunkel 's ... not too sure why sa dropped the dunkel, i think it would make great standard to them pours a dark brown with 1 bone white head, that settles down to thin lace across the top of the brew.. smells of the typical pumpkin pie spice, , along with good squash note tastes just like last years, very subtle , nothing over the top damn good pumpkin ale that is worth seeking out when mean everything is subtle i mean everything nothing is overdone in this pumpkin ale and is great representation of the original style. mouthfeel is somewhat thick with a pleasant coating feel overall loved it last year- and i love it this year do n't get me wrong its no pumpking, but this is a damn fine pumpkin ale that could hold its own any day among all the others would rate this as my 4th favorite pumpkin ale to date i 'm not sure why the bros rated it so low, but do n't take their opinion, make your own !
great standard to them pours a dark brown with bone white head that settles down to thin lace across the top the brow smells of
the typical pumpkin pie spice slong with guod squasli note, tastes just like last years very subtle nothing over the top damn good pumpkin ale that is worth seeking out when mean everything is subtle mean everything nothing is overdone in this pumpkin ale, : and is great representation of the original style mouthfeel is somewhat thick witli à pleasant coating foel overall loved it last year, and i love it this year do n't get me wrong. its no pumpking. but this is a damn fine pumpkin ale that could hold its own any day among all the others. i would rate this as my 4th favorite pumpkin ale to date 'm not sure why the bros rated it so low, but do n't take their opinion, make your own! !
Appearance Smell
Palate
Tasté
Appearance Smell
Palate
Tasté
Multi Aspect- Atteentnttsns
Multi Sspect-Sparse Sparse-Ateentiooon Attentions
sa 's harvest pumpkin ale 2011 had this last year loved it and bought 6 harvest packs and saved the pumpkins and the dunkel 's ... not too sure why sa dropped the dunkel, i think it would make great standard to them pours a dark brown with 1 bone white head. that settles down to thin lace across the top of the brew smells of the typical pumpkin pie spice, along with a good squashi note tastes just like last years I very subtle nothing over the top damn good pumpkin ale that worth seeking out when mean everything is subtle i mean everything nothing is overdone in this pumpkin ale. and is great representation of the original style mouthieel is somewhat thick with pleasant coating feel I overall , loved it last year, and i love it this year do n't get me wrong its no pumpking.. but this is damn fine pumpkin ale that could hold its own any day among all the others i would rate this as my 4th favorite pumpkin ale to date 'm not sure why the bros rated it so low, but do n't take their opinion, make your own
sa 's harvest pumpkin ale 2011 had this last year loved it and bought 6 harvest packs and saved the pumpkins and the dunkel 's ... not too sure why sa dropped the dunkel, i think it would make great standard to them pours a dark brown with a 1 bone white head that settles down to thin lace across the top of the brew smells of the typical pumpkin pie spice, , along with a good squash note tastes just like last years very subtle , nothing over the top a damn good pumpkin ale that worth seeking out when mean everything is subtle i mean everything nothing is overdone in this pumpkin ale, , and is great representation of the original style.. mouthieel is somewhat thick with a pleasant coating feel overall , loved it last year,. ; and i love it this year do n't get me wrong. its no pumpking but this is damn fine pumpkin ale that could hold its own any day among all the others. i would rate this as my 4th favorite pumpkin ale to date i 'm not sure why the bros rated it so low, , but do n't take their opinion, make your own. !
Figure 5: MAM can identify accurately what parts of the review describe each aspect Due to the high imbalance and correlation, MAA provides very noisy labels, while MASA highlights only few important words. We can see that SAM is confused and performs poor selection.
17

---

Under review as a conference paper at ICLR 2020
Appearance
Smell
Palate
Taste
Appearance
Smell
Palate
Taste
Multi Sapect-Masks (Ours)
Single Aspect- Mask (Lei et al., 2016)
75el bottle shared with larrylsb, pre grad bright
75el bottle shared with larrylsb, , pre - grad bright hazy gold with a big white head the flavor has bursting fruit and funky yeast with tropical and peach standing out the flavor has the same intense fruitiness with a funky lightly tart edge and the
hazy gold with big white head the flavor has
bursting fruit and funky yeast with tropical and
peach standing out the flavor has the same intense fruitiness with a funky lightly tart edge and nice hop balance dry and refreshing
the tongue medium bodied with perfect carbonation that livens up the palate this was just beautiful stuff that i "m already craving more of.
nice hop balance dry and refreshing tongue medium bodied with perfect carbonation that livens up the palate this was just beautiful stuff that 'm already craving more of
Appearance
Smell
Palate
Taste
Appearance
Smell
Palate
Taste
Multi Aspect- Attentions
Multi
Attentions
75cl bottle shared with larrylsb, pre grad bright hazy gold with big white head the flavor has bursting fruit and funky yeast with tropical and peach standing out. the Hlavor has the intense fruitiness with a funky lightly tart edge and the
75cl bottle shared with larrylsb, pre grad bright hazy gold with big white head the flavor has bursting fruit and funky yeast with tropical and peach standing out the flavor has the same intense fruitiness with a funky lightly tart edge and nice hop balance dry and refreshing tongue medium bodied with perfect carbonation that livens up the palate this was just beautiful stuff that 'm already craving more of
nice hop balance dry and refreshing tongue medium bodied with perfect carbonation that livens up the palate this just beautiful stuff that m already craving more of
the
Figure 6: On short review, MAM achieves near-perfect annotations, while SAM highlights only two words where one is ambiguous with respect to four aspects. MAA mixes between the aspect Appearance and Smell. MASA identifies some words but lacks coverage.
18

---

Under review as a conference paper at ICLR 2020
A.3.3 HOTEL DATASET
Service Cleanliness
Value
Location
Room
Service Cleanliness
Value
Location
Room
Multi Aspect-Masks (Ours)
Single Aspect-Mask (Lei et al., 2016)
i stayed daulsol in september 2013 and could
i stayed at daulsol in september 2013 and could n't have asked for anymore for the price ! it is great location ... ooly minutes walk to jet, space and sankeys with short drive to ushuaia the hotel basic but cleaned daily and i did have any problems at all with the bathroom or kitchen facilities the lady reception was really helpful and explained everything we needed to know even when we managed miss our flight she lot us stay around and use the facilities until we got on a later flight there are loads of restaurants the vicinity and supermarkets and shops right outside loved these apartments so much that booked to come back for september 2014 ! can not wait :)
n't have asked for anymore for the price ! is
great location ... only minutes walk jet space
and sankeys with short drive to ushuaia the
hotel basic but cleaned daily and did have
any problems at all with the bathroom or kitchen
facilities the lady reception was really helpful
and explained everything we needed to know
even when we mamaged to miss our flight she let us stay ground and use the facilities uutil we gut on a later flight there are loads of restaurants in the vicinity and supermarkets and shops right outside loved these apartments so much that booked to come back for september 2014! !! can not wait
Service Cleanliness
Value Location Room
Service Cleanliness Value Location
Room
Multi Aspect-Attentious
Multi Aspect-Sparsed Sparctttpppeetttetiooos Attentions
i stayed at daulsol in september 2013 and could n't have asked for anymore for the price ! it is great location .... only 2 minutes walk to jet space and sankeys with a short drive to ushuaia the hotel is basic but cleaned daily and i did nt have any problems at all with the bathroom or kitchen facilities the lady reception was really helpful and explained everything we needed to know .... even when managed to miss our flight she let us stay sround and use the facilities until we gut on later flight. there are loads of restaurants in the vicinity and supermarkets and shops right outside loved these apartments so much that booked to come back for september 2014 !! can not wait :)
i stayed at daulsol in september 2013 and could n't have asked for anymore for the price it is a great location ... only 2 minutes walk to jet space and sankeys with a short drive to ushuaia the hotel is basic but eleaned daily and i did nt have any problems at all with the bathroom or kitchen facilities the lady reception was really helpful and explained everything we needed to know even when managed to miss our flight she let us stay around and use the facilities until we got on later flight.. there are loads of restaurants in the vicinity and supermarkets and shops right outside loved these apartments so much that booked to come back for september 2014 !! can not wait :)
Figure 7: MAM emphasizes consecutive words, identifies important spans while having small amount of noise. SAM focuses on certain specific words and spans, but labels are ambiguous. The MAA model highlights many words, ignores a few important key-phrases, and labels are noisy when the confidence is not high. MASA provides noisier tags than MAA.
19

---

Under review as a conference paper at ICLR 2020
Service Cleanliness Value Location Room
Service Cleanliness Value Location
Room
Multi- Aspeect Masker (Ours)
Single Aspect-Mask (Lei et al., 2016)
stayed at the parasio apartments early april 2011 reception staff absohtely bantastic great eustomer service M nt Fault at all ! we were on the 4th floor, facing the front of the hotel basic. but nice and clean good location | not too far away from the strip and beach 10 min walk) ) however
stayed at the parasio 10 apartments early april 2011 reception staff absolutely fantastic great eustomer service ca ut fauld st all we were on the 4th floor, facing the front of the hotel .. basic, but nice and clean good location, not too far away from the strip and beach 10 min walk ). however do not go out alone at night at all ! went to the end of the street one night and got mugged all my money camera everything got sratches on my chest which has now scarred me. and had bruises at the time just make sure you have got someone with you at all times , the local people are very renound for this . went to police station the next day [ in old town ) and there was many english in there reporting their muggings from the day before shocking ! apart from this incident ( on the first night we arrived :( ) we had a good time in the end plenty of langhs and everything very cheap beer - leuro! ! fryups 2enro would go back again, but maybe stay somewhere else closer to the beach sol pelicanos etc ) ... this hotel is next to an alley called muggers alley
do not go out alone at night at all i went to
the end of the street one night and got mugged.
all my money, camera... everything got sratches on my chest which has now scarred me. and had bruises at the time. just make sure you have got somcone with you at all times , the local people are very renound for this went to police station the next day old town ) and there was many english in there reporting their mugginng from the day before. shocking ! apart from this incident ( on the first night we arrived :() we had a good time in the end, plenty of laughs and everything
very cheap! beer Leuro! fryups 2euro] would go
back again. but maybe stay somewhere else closer
to the beach sol pelicanos etc ) this hotel is next
to an alley called] muuggrr alley
Service Cleanliness Value Location Room
Service Cleanliness
Value Locstion
Room
Multi Aspect-Sparse- Attentions
Multi Aspect-Attentionn
stayed at the parasio 10 apartments early april 2011 reception staff absolutely fantastic great customer service ca ut fault at all ! we were on the 4th floor, facing the front of the hotel basic, but nice and clean good location, not too far away from the strip and beach 10 min walk ) however do not go out alone at night at all i went to the end of the street one night and got mugged. all my money camera .. everything got sratches on my chest which has now scarred me, and had bruises at the time. just make sure you have got someone with you at all times , the local people are very renound for this . went to police station the next day { in old town ) and there was many english in there reporting their muggings from the day before shocking ! apart from this incident { on the first night we arrived o( ) we had a good time in the end, plenty of laughs and everything very cheap! ! beer - leuro! fryups 2euro. would go back again, but maybe stay somewhere else closer to the beach sol pelicanos etc ) ). this hotel is next to an alley called muggers alley
stayed at the parasio 10 spartments early april 2011 reception staff absolutely fantastic great customer service .. ca nt fault at all we were on the 4th floor, facing the front of the hotel basic, but nice and clean. good location, not too far away from the strip and beach ( 10 min walk) ). however do not go out alone at night at all i went to the end of the street one night and got mugged all my money, camera everything ! got statches on my chest which has now scarred me, and had bruises at the time. just make sure you have got somcone with you at all times, the local people are very renound for this went to police station the next day { in old town) ) and there was many english in there reporting their muggings from the day before shocking !! spart from this incident ( on the first night we arrived :() ) we had a good time in the end, plenty of laughs and everything very cheap! beer - leuro! fryups Zeuro. would go back again, , but maybe stay somewhere else closer to the beach ( sol pelicanos etc) .. this hatel is next to an alley called muggers alley"
Figure& 8: Our MAM model finds most of the important span of words with a small amount of noise. SAM lacks coverage but identifies words where half are correctly tags and the others ambiguous. MAA partially correctly highlights words for the aspects Service, Location, and Value while missing out the aspect Cleantiness. MASA confidently finds a few important words.
20

---

Under review as a conference paper at ICLR 2020
A.4 TOPIC WORDS PER ASPECT
For each model, we computed the probability distribution of words per aspect by using the induced sub-masks May...., Ma, or attention values. Given aspect a, and set of top-N words wa the Normalized Pointwise Mutual Information (Bouma 2009) coherence score is:
P( .012 P(wo, (P(w))))
1-1 log
NPMI w
(6)
log P( (ww way)
Top words of coherent topics (i.e., aspects) should share a similar semantic interpretation and thus, interpretability of topic can be estimated by measuring how many words are not related. For each aspect a; and word w having been highlighted at least once as belonging to aspect ag we computed the probability P(wla;) on each dataset and sorted them in decreasing order of P(wla:). Unsurprisingly, we found that the most common words are stop words such as "a" and "it", because masks mostly word sequences instead of individual words. To gain a better interpretation of the aspect words, we followed the procedure in McAuley et al. (2012): we first computed averages across all aspect words for each word w: bm [AT E P(wlas), which generates a general distribution that includes words common to all aspects. The final word distribution per aspect is computed by removing the general distribution: P(wla:) = P(wla,) byr-
After generating the final word distribution per aspect, we picked the top ten words and asked two human annotators identify intruder words, i.e., words not matching the corresponding aspect. We show in subsequent tables the top ten words for each aspect, where red denotes all words identified as unrelated the aspect by the two annotators. Generally, our model finds better sets of words across the three datasets compared with other methods. Additionally, we observe that the aspects can be easily recovered given its top words.
Table 7: Top ten words for each aspect from the Filtered Beer dataset, learned by various models. Red denotes intruders according to two human annotators. For the three aspects, MAM has only one word considered as an intruder, followed by MASA with SAM (two) and MAA (six).
Model
Top-10 Words
Appearance
SAM
head color white brown dark lacing pours amber clear black
MASA
head lacing lace retention glass foam color amber yellow cloudy
MAA
nice dark amber pours black hazy brown great cloudy clear head color lacing white brown clear amber glass black retention
MAM (Ours)
SAM
sweet malt hops coffee chocolate citrus hop strong smell aroma smell aroma nose smells sweet aromas scent hops malty roasted
Smell
MASA
MAA
taste smell aroma sweet chocolate lacing malt roasted hops nose smell aroma nose smells sweet malt citrus chocolate caramel aromas
MAM (Ours)
SAM
mouthfeel smooth medium carbonation bodied watery body thin creamy full mouthfeel medium smooth body nice m- feel bodied mouth beer
Palate
MASA
MAA
carbonation mouthfeel medium overall smooth finish body drinkability bodied watery
MAM (Ours)
mouthfeel carbonation medium smooth body bodied drinkability good mouth thin
21

---

Under review as a conference paper at ICLR 2020
Table 8: Top ten words for each aspect from the Full Beer dataset, learned by various models. Red denotes intruders according to two annotators. Found words are generally noisier due to the high correlation between Taste and other aspects. However, MAM provides better results than other methods.
Model
Top-10 Words
Apperance
SAM
nothing beautiful lager nice average macro lagers corn rich gorgeous
MASA
lacing head lace smell amber retention beer nice carbonation glass head lacing smell aroma color pours amber glass white retention head lacing smell white lace retention glass aroma fan thin
MAA
MAM (Ours)
SAM
faint nice mild light slight complex good wonderful grainy great aroma hops nose chocolate caramel malt citrus fruit smell fruits taste hints hint lots starts blend mix upfront malts taste malt aroma hops sweet citrus caramel nose malts chocolate
Smell
MASA
MAA
MAM (Ours)
SAM
thin bad light watery creamy silky medium body smooth perfect smooth light medium thin creamy bad watery full crisp clean
Pulate
MASA
MAA
good beer carbonation smooth drinkable medium bodied nice body overall
MAM (Ours)
carbonation medium mouthfeel body smooth bodied drinkability creamy light overall
SAM
decent great complex delicious tasty favorite pretty sweet well best good drinkable nice tasty great enjoyable decent solid balanced average
Taste
MASA
MAA
malt hops flavor hop flavors caramel malts bitterness bit chocolate
MAM (Ours)
malt sweet hops flavor bitterness finish chocolate bitter caramel sweetness
Table 9: Top ten words for each aspect from the Hotel dataset, learned by various models. Red denotes intruders according to human annotators. Besides SAM, all methods find similar words for most aspects except the aspect Value, where MAM does not have intruder.
Model
Top-10 Words
SAM
staff' service friendly nice told helpful good great lovely manager
Service
MASA
friendly helpful told rude nice good pleasant asked enjoyed worst staff service helpful friendly nice good rude excellent great desk staff friendly service desk helpful manager reception told rude asked
MAA
MAM (Ours)
Cleanliness
SAM
clean cleaned dirty toilet smell cleaning sheets comfortable nice hair clean dirty cleaning spotless stains cleaned cleanliness mold filthy bugs clean dirty cleaned filthy stained well spotless carpet sheets stains clean dirty bathroom room bed cleaned sheets smell carpet toilet
MASA
MAA
MAM (Ours)
SAM
good stay great well dirty recommend worth definitely friendly charged
Value
MASA
great good poor excellent terrible awful dirty horrible disgusting comfortable
MAA
night stayed stay nights 2 day price water 4
MAM (Ours)
good price expensive paid cheap worth better pay overall disappointed
SAM
location close far place walking definitely located stay short view location beach walk hotel town located restaurants walking close taxi
Location
MASA
MAA
location hotel place located close area beach view situated
MAM (Ours)
location great area walk beach hotel town close city street
SAM
dirty clean small best comfortable large worst modern smell spacious comfortable small spacious nice large dated well tiny modern basic room rooms bathroom bed spacious small beds large shower modern comfortable small spacious nice modern rooms large tiny walls
Room
MASA
MAA
MAM (Ours)
22

---

Under review as a conference paper at ICLR 2020
A.5 BASELINE ARCHITECTURES
Encoder
Classifier
FNN aj FNN d₂
Max Pooling
Review
Embeddings
e2 e2 e
CNN
Ya1
FNN aA
Encoded Review
Rating per aspect
Figure 9: Baseline model Emb + EnC Clf.
Encoder
Classifier
FNN a, FNN da
Review
Embeddings
eelee el
CNN
Pooling
Max
Attention
h
Ya
Encoded Review
FNN
Rating per aspect
Figure 10: Baseline model Emb Enc CNN + A Shared Clf.
Encoder
Classifier
FNN aj FNN a₂
Attention
Review
Embeddings
el, eb
LSTM
h
Ya
FNN a
Encoded Review
Rating per aspect
Figure 11: Baseline model Emb + Enc 1STM + Shared + Clf.
Encoders
Classifier
FNN FNN a
Attention
Review
Embeddings
el,e2 ,,,
LSTM
has
Ya1
Encoded
FNN a
Rating per aspect
Review
aspect
Figure 12: Baseline model Emb Enc
[Sparse]
+ Clf. Attention either additive sparse.
LSTM Aspect-wise
23


Reviews:
[{"review_id": "Syla5c10Kr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a neural model for multi-aspect sentiment of reviews. Apart from sentiment classification, the architecture is able to return a mask for each aspect highlighting the sequences of words used to make the sentiment prediction for that aspect. This improves the interpretability of the model. The aspects are fixed and the training set requires a multi-labeled dataset with one binary positive/negative label per aspect.\n\nThe network architecture is mostly standard: a text review is turned into a list of word embeddings which are fed through a trainable masker outputting one mask per aspect, essentially \"selecting\" a different set of words (embeddings) per aspect. From here on, a standard encoder can be used (the authors end up using a CNN with max-pooling but LSTM or other could be used), followed by a classification step (standard MLP) outputting a sentiment score per aspect. A couple tricks are used to make the masks converge to something meaningful by adding regularization terms to control the number of selected words and encouraging consecutive words to be selected.\n\nI'm leaning toward rejection partly due to the limited novelty in the approach but mainly due to the experimental evaluation section which requires improvements.\n- High aspect correlation baseline: the average aspect correlation across reviews in the training set is high (that is, if one aspect is positive, most aspects will be and viceversa). A natural baseline to include is the optimal \"no-aspect\" decision boundary: in other words, if for each review we were given the majority rating across aspects (is this review mostly positive or negative) and were able to predict this always correctly, what would the macro-aspect F1 score be if we were to simply predict that label to all aspect of that review?\n- Please quantify how much does your model rely on this aspect correlation. For example, if you were to sort the reviews in the validation set by their aspect correlation, and chart the F1 score as a function of the prefix of this sorted dataset what would we see? This would be especially telling when comparing the different baselines and classifiers again each other in this chart.\n- SVM: you mentioned not running it on the larger datasets due to lengthy training time. I would suggest including NB-SVM implemented with logistic regression rather than actual SVM since it's always a great baseline and fast to compute (in \"Baselines and Bigrams: Simple, Good Sentiment and Topic Classification\").\n- The final model that ends up giving the best results is really a 2-step pipeline (rather than end-to-end trainable) and requires training twice: once to learn the masks and once to actually train the rest of the network after the word embeddings are extended with the masks.\n\n \n\n\n\n\n\n\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "S1l3vd0DdS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is mostly concerned with multi-aspect sentiment classification.\n\nMulti-aspect sentiment analysis is an old problem (> 10 years), that usually operates on datasets with multidimensional sentiment labels (price, cleanliness, etc.). This paper has some variation on this old problem in the sense that it's more focused on generating explanations / justifications rather than purely on sentiment analysis.\n\nAlthough an interesting variant on a (somewhat niche) existing problem, I do feel that some aspects of this paper are a little dated -- claiming that the method is more useful for interpretability / justification doesn't seem so convincing these days unless you can make a stronger argument as to how it could actually be used to generate justifications that would be shown to users. I don't really see that in the current version. At the moment the output of the system appears to be attention weights over several aspects, showing which words in a review focus on which aspect. But this doesn't seem so different from how older methods (e.g. Bag-of-Words based techniques for multi-aspect sentiment classification), though their labels were possibly at the level of sentences rather than reviews.\n\nThe actual method itself seems appropriate, and is a nice update on previous techniques for multi-aspect sentiment analysis.\n\nPerhaps I missed some detail explaining why such a comparison is impossible, but there seems to be no direct comparison against traditional techniques for multi-aspect sentiment analysis. E.g. aren't even traditional models (like say Titov & McDonald) comparable to this? I understand the goal is somewhat different, but the input/output modality seems similar and such a comparison could be helpful.\n\nOverall, I think these days the standard is quite high for papers that make claims about generating convincing justifications/interpretations/explanations. There's little by way of human evaluation (there's a bit in the appendix, but it's not really focused on whether or not the model is useful for the claimed purpose of \"explanation\").\n\nMore positively, the authors have done *a lot* by way of experiments, in terms of ablations, and showing quantitative performance from different dimensions. The appendix is detailed and shows many examples. Overall I feel somewhat borderline about this paper, because there's a lot here, but I feel some of these key components mentioned above are missing.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper."}, "extra_text": {}}, {"review_id": "HkxCRSeCFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work addresses the problem of aspect-based sentiment analysis *with* rationale extraction that can serve as an explanation (interpretation) of the decision.\nCompared to previous work this paper models the problem as multi-aspect classification, therefore having one model for all aspects instead of one for each aspect. This is obviously useful, but in addition to having a smaller total model they show that they also have (slightly) better prediction AND AT THE SAME TIME \"better\" explanations (which is always hard to evaluate).\n\nThis is a useful paper, and I can clearly see it used. However, it might not get many ICLR attendants excited.\n\nI found the paper very hard to read, having to re-read many parts to understand exactly what is the goal of each section. The row names of the table 1-3 do not correspond to the names in the text which makes the understanding even harder. Also, I think that some plot that shows accuracy vs \"interpretability\", with model size as a third dimension (eg surface of the bubble) would convey the take-home of the paper much clearer than the many tables with small differences between them.\n\nA minor comment: the SVM has a huge parameter space, which is weird. Are you using *all* bigrams? A fairer comparison would be to use more n-grams, but keep only the most frequent ones.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a neural network model for predicting multi-aspect sentiment and generating masks that can justify the predictions. The positive aspects of the paper include improved results over the state-of-the-art.

Reviewers found the technical novelty limited, and the experiments short of being fully convincing. After the author rebuttal, there were discussions between the reviewers and the AC, and the reviewers still thought the paper is not fully convincing given these limitations.

I thank the authors for their submission and detailed responses to the reviewers and hope to see this research in a future venue.
Decision: Reject