Year: 2019

Abstract:
Neural models achieved considerable improvement for many natural language processing tasks, but they offer little transparency, and interpretability comes at a cost. In some domains, automated predictions without justifications have limited applicability. Recently, progress has been made regarding single-aspect sentiment analysis for reviews, where the ambiguity of a justification is minimal. In this context, a justification, or mask, consists of (long) word sequences from the input text, which suffice to make the prediction. Existing models cannot handle more than one aspect in one training and induce binary masks that might be ambiguous. In our work, we propose a neural model for predicting multi-aspect sentiments for reviews and generates a probabilistic multi-dimensional mask (one per aspect) simultaneously, in an unsupervised and multi-task learning manner. Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable.

heading: 1 INTRODUCTION
text: Neural networks have become the standard for many natural language processing tasks. Despite the significant performance gains achieved by these complex models, they offer little transparency concerning their inner workings. Thus, they come at the cost of interpretability (Jain & Wallace, 2019).
In many domains, automated predictions have a real impact on the final decision, such as treatment options in the field of medicine. Therefore, it is important to provide the underlying reasons for such a decision. We claim that integrating interpretability in a (neural) model should supply the reason of the prediction and should yield better performance. However, justifying a prediction might be ambiguous and challenging. Prior work includes various methods that find the justification in an input text — also called rationale or mask of a target variable. The mask is defined as one or multiple pieces of text fragments from the input text.1 Each should contain words that altogether are short, coherent, and alone sufficient for the prediction as a substitute of the input (Lei et al., 2016).
Many works have been applied to single-aspect sentiment analysis for reviews, where the ambiguity of a justification is minimal. In this case, we define an aspect as an attribute of a product or service (Giannakopoulos et al., 2017), such as Location or Cleanliness for the hotel domain. There are three different methods to generate masks: using reinforcement learning with a trained model (Li et al., 2016b), generating rationales in an unsupervised manner and jointly with the objective function (Lei et al., 2016), or including annotations during training (Bao et al., 2018; Zhang et al., 2016).
However, these models generate justifications that are 1) only tailored for one aspect, and 2) expressed as a hard (binary) selection of words. A review text reflects opinions about multiple topics a user cares about (Musat et al., 2013). It appears reasonable to analyze multiple aspects with a multi-task learning setting, but a model must be trained as many times as the number of aspects. A hard assignment of words to aspects might lead to ambiguities that are difficult to capture with a binary mask: in the text ”The room was large, clean and close to the beach.”, the word ”room” refers to the aspects Room, Cleanliness and Location. Finally, collecting human-provided rationales at scale is expensive and thus impractical.
In this work, we study interpretable multi-aspect sentiment classification. We describe an architecture for predicting the sentiment of multiple aspects while generating a probabilistic (soft) multi-
1In the rest of the paper, we will use the terms mask, justification and rationale interchangeably.
dimensional mask (one dimension per aspect) jointly, in an unsupervised and multi-task learning manner. We show that the induced mask is beneficial for identifying simultaneously what parts of the review relate to what aspect, and capturing ambiguities of words belonging to multiple aspects. Thus, the induced mask provides fine-grained interpretability and improves the final performance.
Traditionally interpretability came at a cost of reduced accuracy. In contrast, our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable compared to attention-based methods and a single-aspect masker. We show that it can be a benefit to 1) guide the model to focus on different parts of the input text, and 2) further improve the sentiment prediction for all aspects. Therefore, interpretabilty does not come at a cost anymore.
The contributions of this work can be summarized as follow:
• We propose a Multi-Aspect Masker (MAM), an end-to-end neural model for multi-aspect sentiment classification that provides fine-grained interpretability in the same training. Given a text review as input, the model generates a probabilistic multi-dimensional mask, with one dimension per aspect. It predicts the sentiments of multiple aspects, and highlights long sequences of words justifying the current rating prediction for each aspect;
• We show that interpretability does not come at a cost: our final model significantly outperforms strong baselines and attention models, both in terms of performance and mask coherence. Furthermore, the level of interpretability is controllable using two regularizers;
• Finally, we release a new dataset for multi-aspect sentiment classification, which contains 140k reviews from TripAdvisor with five aspects, each with its corresponding rating.2

heading: 2 RELATED WORK

heading: 2.1 INTERPRETABILITY
text: Developing interpretable models is of considerable interest to the broader research community, even more pronounced with neural models (Kim et al., 2015; Doshi-Velez & Kim, 2017). Many works analyzed and visualized state activation (Karpathy et al., 2015; Li et al., 2016a; Montavon et al., 2018), learned sparse and interpretable word vectors (Faruqui et al., 2015b;a; Herbelot & Vecchi, 2015) or analyzed attention (Clark et al., 2019; Jain & Wallace, 2019). Our work differs from these in terms of what is meant by an explanation. Our system identifies one or multiple short and coherent text fragments that — as a substitute of the input text — are sufficient for the prediction.

heading: 2.2 ATTENTION-BASED MODELS
text: Attention models (Vaswani et al., 2017; Yang et al., 2016; Lin et al., 2017) have been shown to improve prediction accuracy, visualization, and interpretability. The most popular and widely used attention mechanism is soft attention (Bahdanau et al., 2015) over hard (Luong et al., 2015) and sparse ones (Martins & Astudillo, 2016). According to Jain & Wallace (2019); Serrano & Smith (2019), standard attention modules noisily predict input importance; the weights cannot provide safe and meaningful explanations.
Our model differs in two ways from attention mechanisms: our loss includes two regularizers to favor long word sequences for interpretability; the normalization is not done over the sequence length.

heading: 2.3 MULTI-ASPECT SENTIMENT CLASSIFICATION
text: Review multi-aspect sentiment classification is sometimes seen as a sub-problem (Wang et al., 2010; McAuley et al., 2012; Pappas & Popescu-Belis, 2014), by utilizing heuristic-based methods or topic models. Recently, neural models achieved significant improvements with less feature engineering. Yin et al. (2017) built a hierarchical attention model with aspect representations by using a set of manually defined topics. Li et al. (2018) extended this work with user attention and additional features such as overall rating, aspect, and user embeddings. The disadvantage of these methods is their limited interpretability, as they rely on many features in addition to the review text.
2We will make the code and data available.

heading: 2.4 RATIONALE-BASED MODELS
text: The idea of including human rationales during training has been explored in (Zhang et al., 2016; Marshall et al., 2015; Bao et al., 2018). Although they have been shown to be beneficial, they are expensive to collect and might vary across annotators. In our work, no annotation is used.
The work most closely related to ours is Li et al. (2016b) and Lei et al. (2016). Both generate hard rationales and address single-aspect sentiment classification. Their model must be trained separately for each aspect, which leads to ambiguities. Li et al. (2016b) developed a post-training method that removes words from a review text until another trained model changes its prediction. Lei et al. (2016) provides a model that learns an aspect sentiment and its rationale jointly, but hinders the performance and relies on assumptions on the data, such as a small correlation between aspect ratings.
In contrast, our model: 1) supports multi-aspect sentiment classification, 2) generates soft multidimensional masks in a single training; 3) the masks provide interpretability and improve the performance significantly.

heading: 3 METHOD: MULTI-ASPECT MASKER

heading: Review Text
text: ECNN + Masker + Clf
Let X be a review composed of L words x1, x2, ..., xL and Y the target A-dimensional sentiment vector, corresponding to the different rated aspects. Our proposed model, called Multi-Aspect Masker, is composed of three components: 1) a Masker module that computes a probability distribution over aspects for each word, resulting in A+1 different masks (including one for not-aspect); 2) an Encoder that learns a representation of a review conditioned on the induced masks; 3) a Classifier that predicts the target variables. The overall model architecture is shown in Figure 1. Our framework generalizes for other tasks, and each neural module is interchangeable with other models.
The Masker first computes a hidden representation h` for each word x` in the input sequence, using their word embeddings e1, e2, ..., eL. Many sequence models could realize this task, such as recurrent, attention, or convolution neural networks. In our case, we chose a convolutional network because it led to a smaller model, faster training, and empirically, performed similarly to recurrent models. Let ai denote the ith aspect for i = 1, ..., A, and a0 the not-aspect case, because many words can be irrelevant to every aspect. We define M ` ∈ R(A+1), the aspect distribution of the input word x` as:
P (M|X) = L∏
`=1
P (M `|x`) = L∏
`=1 A∏ i=0 P (m`ai |x `) (1)
Because we have categorical distributions, we cannot directly sample P (M `|x`) and backpropagate the gradient through this discrete generation process. Instead, we model the variable m`ai using the Straight Through Gumbel Softmax (Jang et al., 2017; Maddison et al., 2017), to approximate sampling from a categorical distribution. We model the parameters of each Gumbel Softmax distribution M ` with a single-layer feedforward neural network followed by applying a log softmax, which induces the log-probabilities of the `th distribution: ω` = log(softmax(Wh` + b)). W and b are shared across all tokens, to have a constant number of parameters with respect to the sequence
length. We control the sharpness of the distributions with the temperature parameter τ . Compared to attention mechanisms, the word importance is a probability distribution over the targets:∑T
t=0 P (m ` at |x
`) = 1, instead of a normalization over the sequence length, ∑L
`=1 P (a `|x`) = 1.
Given a soft multi-dimensional mask M ∈ R(A+1)×L, we define each sub-mask Mai ∈ RL as:
Mai = P (m 1 ai |x 1), P (m2ai |x 2), ..., P (mLai |x L) (2)
We weight the word embeddings by their importance towards an aspect ai with the induced submasks, such thatEai =Mai E = P (m1ai |x 1) ·e1, P (m2ai |x 2) ·e2, ..., P (mLai |x
L) ·eL. Thereafter, each modified embedding Eai is fed into the Encoder block. Note that Ea0 is ignored because Ma0 only serves to absorb probabilities of words that are insignificant to every aspect.3
The Encoder module includes a convolutional neural network, for the same reasons as earlier, followed by a max-over-time pooling to obtain a fixed-length feature vector. It produces the hidden representation hai for each aspect ai. To exploit commonalities and differences among aspects, we share the weights of the encoders for allEai . Finally, the Classifier block contains for each aspect ai a two-layer feedforward neural networks followed by a softmax layer to predict the sentiment ŷai .

heading: 3.1 INTERPRETABLE MASKS
text: Attention model Multi-Aspect Masker Trained on `sent and no constraint Trained on `sent with λp, `sel, and `cont
Masker, where the colors represent the aspects: Service, Cleanliness, Value, Location, and Room. Masks lead to mostly long sequences describing clearly each aspect (one switch per aspect), while attention to many short and interleaving sequences (30 changes between aspects), where most relate to noise or multiple aspects. Highlighted words correspond to the highest aspect-attention scores above 1/L (i.e., more important than a uniform distribution), and the aspect ai maximizing P (m`ai |x `).
The first objective to optimize is the sentiment loss, represented with the cross-entropy between the true aspect sentiment label yai and the prediction ŷai :
`sent = A∑ i=1 `cross entropy(yai , ŷai) (3)
Training Multi-Aspect Masker to optimize `sent will lead to meaningless sub-masks Mai , as the model tends to focus on certain key-words. Consequently, we guide the model to produce long and meaningful sequences of words, as shown in Figure 2. We propose two regularizers: the first controls the number of selected words, and the second favors consecutive words belonging to the
3if P (m`a0 |x `) ≈ 1.0, it implies that ∑A i=1 P (m ` ai |x `) ≈ 0 and consequently, e`ai ≈ ~0.
same aspect. For the first term `sel, we calculate the probability psel of tagging a word as aspect and then compute the cross-entropy with a parameter λp. The hyper-parameter λp can be interpreted as the prior on the number of selected words among all aspects, which corresponds to the expectation of Binomial(psel), as the optimizer will try to minimize the difference between psel and λp.
psel = 1
L L∑ `=1 ( 1− P (m`a0 |x `) ) `sel = `binary cross entropy(psel, λp)
(4)
The second regularizer discourages aspect transition between two consecutive words, by minimizing the mean variation of two consecutive aspect distributions. We generalize the formulation in Lei et al. (2016), from a hard binary single-aspect selection, to a soft probabilistic multi-aspect selection.
pdis = 1
L L∑ `=1 [ 1 A+ 1 A∑ a=0 | P (m`ai |x `)− P (m`−1ai |x `−1) | ]
`cont = `binary cross entropy(pdis, 0)
(5)
Finally, we train our Multi-Aspect Masker in an end-to-end manner, and optimize the final loss `MAM = `sent+λsel ·`sel+λcont ·`cont, where λsel and λcont control the impact of each constraint.

heading: 4 EXPERIMENTS
text: In this section, we assess our model on two dimensions: the predictive performance and the quality of the induced mask. We first evaluate Multi-Aspect Masker on the multi-aspect sentiment classification task. In a second experiment, we measure the quality of the induced sub-masks using aspect sentence-level annotations, and an automatic topic model evaluation method.4

heading: 4.1 DATASETS
text: McAuley et al. (2012) provided 1.5 million beer reviews from BeerAdvocat. Each contains multiple sentences describing various beer aspects: Appearance, Smell, Palate, and Taste; users also provided a five-star rating for each aspect. Lei et al. (2016) modified the dataset to suit the requirements of their method.5 As a consequence, the obtained subset, composed of 280k reviews, does not reflect the real data distribution: it contains only the first three aspects, and the sentiment correlation between any pair of aspects is decreased significantly (27.2% on average, instead of 71.8% originally). We denote this version as the Filtered Beer dataset, and the original one as the Full Beer dataset.
To evaluate the robustness of models across domains, we crawled 140k hotel reviews from TripAdvisor. Each review contains a five-star rating for each aspect: Service, Cleanliness, Value, Location, and Room. The average correlation between aspects is high (63.0% on average). Compared to beer reviews, hotel reviews are longer, noisier, and less structured, as shown in Appendix A.3.
As in Bao et al. (2018), we binarize the problem: ratings at three and above are labeled as positive and the rest as negative. We further divide the datasets into 80/10/10 for train, development, and test subsets (more details in Appendix A.1).

heading: 4.2 BASELINES
text: We compared our Multi-Aspect Masker (MAM) with various baselines. We first used a simple baseline, Sentiment Majority, that reports the majority sentiment across aspects, as the sentiment correlation between aspects might be high (see Section 4.1). Because this information is not available at testing, we trained a model to predict the majority sentiment of a review using Wang & Manning (2012). The second baseline we used is a shared encoder followed by A classifiers, that we denote Emb + Enc + Clf. This model does not offer any interpretability. We extended it with a
4The detailed experimental setup is described in Appendix A.2. 5For the three first aspects, they trained a simple linear regression model to predict the rating of an aspect
given the others and then selected reviews with the largest prediction error.
shared attention mechanism (Bahdanau et al., 2015) after the encoder, noted AShared , that provides a coarse-grained interpretability: for all aspects, the network focuses on the same words in the input.
Our final goal is to achieve the best performance and provide fine-grained interpretability: to visualize what sequences of words a model focuses on and to predict the aspect sentiment predictions. To this end, we included other baselines: two trained separately for each aspect and two trained with a multi-aspect sentiment loss. We employed for the first ones: the well-known NB-SVM of Wang & Manning (2012) for sentiment analysis tasks, and the Single Aspect-Mask (SAM) model from Lei et al. (2016), each trained separately for each aspect. The two last methods are composed of a separate encoder, attention mechanism, and classifier for each aspect. We utilized two types of attention mechanism: additive (Bahdanau et al., 2015), and sparse (Martins & Astudillo, 2016). We call each variant Multi Aspect-Attentions (MAA) and Multi Aspect-Sparse-Attentions (MASA). Diagrams for the baselines can be found in Appendix A.5.

heading: 4.3 MULTI-ASPECT SENTIMENT CLASSIFICATION
text: In this section, we enquire whether fine-grained interpretability can become a benefit rather than a cost. We group the models and baselines in three different levels of interpretability:
• None: we cannot identify what parts of the review are important for the prediction; • Coarse-grained: we can identify what parts of the reviews were important to predict all
aspect sentiments, without knowing what part corresponds to what aspect;
• Fine-grained: for each aspect, we can identify what parts are used to predict its sentiment.

heading: 4.3.1 BEER REVIEWS
text: Overall F1 scores (macro and for each aspect Ai) for the controlled-environment Filtered Beer (where there are assumptions on the data distribution) and the real-world Full Beer dataset are shown in Table 1 and Table 2.
We find that our Multi-Aspect Masker (MAM) model 8 , with 1.7 to 2.1 times fewer parameters than aspect-wise attention models ( 6 + 7 ), performs better on average than all other baselines on both datasets, and provides fine-grained interpretability. For the synthetic Filtered Beer dataset, MAM achieves a significant improvement of at least 0.36 macro F1 score, and 0.05 for the Full Beer one.
To demonstrate that the induced sub-masks Ma1 , ...,MaA are 1) meaningful for other models to improve final predictions, and 2) bring fine-grained interpretability, we extracted and concatenated the masks to the word embeddings, resulting in contextualized embeddings (Peters et al., 2018).
Table 2: Performance of the best models of each architecture for the Full Beer dataset.
F1 Score

heading: Interp. Model Params Macro A1 A2 A3 A4
text: None 0 Sentiment Majority 560k 73.01 71.83 75.65 71.26 73.31
1 Emb200 + EncCNN + Clf 188k 76.45 71.44 78.64 74.88 80.83
Coarsegrained 2 Emb200 + EncCNN + AShared + Clf 226k 77.06 73.44 78.68 75.79 80.32 3 Emb200 + EncLSTM + AShared + Clf 219k 78.03 74.25 79.53 75.76 82.57
Finegrained 4 NB-SVM (Wang & Manning, 2012) 4 · 560k 72.11 72.03 74.95 68.11 73.35 5 SAM (Lei et al., 2016) 4 · 644k 76.62 72.93 77.94 75.70 79.91 6 Emb200 + EncLSTM + A Sparse Aspect-wise + Clf 611k 77.62 72.75 79.62 75.81 82.28
7 Emb200 + EncLSTM + AAspect-wise + Clf 611k 78.50 74.58 79.84 77.06 82.53 8 Emb200 + Masker + EncCNN + Clf (Ours) 289k 78.55 74.87 79.93 77.39 82.02 9 Emb200+4 + EncCNN + Clf (Ours) 191k 78.94 75.02 80.17 77.86 82.71
We trained a simple Encoder-Classifier with the contextualized embeddings 9 , which has approximately 1.5 times fewer parameters than MAM. We achieved a macro F1 score absolute improvement of 0.34 compared to MAM, and 1.43 compared to the non-contextualized variant for the Filtered Beer dataset; for the Full Beer one, the performance increases by 0.39 and 2.49 respectively. Similarly, each individual aspect Ai F1 score of MAM is improved to a similar extent.
We provide in Appendix A.3.1 and A.3.2 visualizations of reviews with the computed sub-masks Ma1 , ...,MaA and attentions by different models. Not only do sub-masks enable the reach of higher performance; they better capture parts of reviews related to each aspect compared to other methods.
Both NB-SVM 4 and SAM 5 , offering fine-grained interpretability and trained separately for each aspect, significantly underperform compared to the Encoder-Classifier 1 . This result is expected: the goal of SAM is to provide rationales at the price of performance (Lei et al., 2016), and NBSVM might not perform well because of its simplicity. Shared attention models ( 2 + 3 ) perform similarly to the Encoder-Classifier 1 , but provide only coarse-grained interpretability.
However, in the Full Beer dataset, SAM 5 obtains better results than the Encoder-Classifier baseline 1 and NB-SVM 4 , which is outperformed by all other models. It might be counterintuitive that SAM performs better, but we claim that its behavior comes from the high correlation between aspects: SAM selects words that should belong to aspect ai to predict the sentiment of aspect aj (ai 6= aj). Moreover, in Section 4.5, we show that a single-aspect mask from SAM cannot be employed for interpretability.
Finally, Sentiment Majority 0 is outperformed by a large margin by all other models in the Filtered Beer dataset, because of the low sentiment correlation between aspects. However, in the realistic dataset Full Beer, Sentiment Majority obtains higher score and performs better than NB-SVM 4 . These results emphasize the ease of the Filtered Beer dataset compared to the Full Beer one, because of the assumptions not holding in the real data distribution.

heading: 4.3.2 MODEL ROBUSTNESS - HOTEL REVIEWS
text: On the Hotel dataset, the learned mask M from Multi-Aspect Masker 8 is again meaningful, by increasing the performance and adding interpretability. The Encoder-Classifier with contextualized embeddings 9 outperforms all other models significantly, with an absolute macro F1 score improvement of 0.49. Moreover, it achieves the best individual F1 score for each aspect A1, ..., A5.
Visualizations of reviews, with masks and attentions, are available in Appendix A.3.3. The interpretability comes from the long sequences that MAM identifies, unlike attention models. In comparison, SAM 5 lacks coverage and suffers from ambiguity due to the high correlation between aspects. We observe that Multi-Aspect Masker 8 performs slightly worse than aspect-wise atten-

heading: Interp. Model Params Macro A1 A2 A3 A4 A5
text: Smell Palate Appearance

heading: Interp. Model Prec. %Words Prec. %Words Prec. %Words
text: Finegrained SVM* (Lei et al., 2016) 21.6 7% 24.9 7% 38.3 13% SAA* 88.4 7% 65.3 7% 80.6 13% SAM* (Lei et al., 2016) 95.1 7% 80.2 7% 96.3 14% MASA 87.0 4% 42.8 5% 74.5 4% MAA 51.3 7% 32.9 7% 44.9 14% MAM (Ours) 96.6 7% 81.7 7% 96.7 14%
* The model has been trained separately for each aspect.
tion models ( 7 + 8 ), with 2.5 times fewer parameters. We emphasize that using the induced masks in the Encoder-Classifier 9 already achieves the best performance.
The Single Aspect-Mask 5 obtains the lowest relative macro F1 score of the three datasets: a difference of−3.27;−2.6 and−2.32 for the Filtered Beer and Full Beer dataset respectively. This proves that the model is not meant to provide rationales and increase the performance simultaneously.
Finally, we show that learning soft multi-dimensional masks along training objectives achieves strong predictive results, and using these to create contextualized word embeddings and train a baseline model with, provides the best performance across the three datasets.

heading: 4.4 MASK INTERPRETABILITY
text: In these experiments, we verify that Multi-Aspect Masker generates induced masks Ma1 , ...,MaA that, in addition to improving performance, are meaningful and can be interpreted by humans.

heading: 4.4.1 MASK PRECISION
text: Evaluating justifications that have short and coherent pieces of text is challenging because there is no gold standard provided with reviews. McAuley et al. (2012) provided 994 beer reviews with aspect sentence-level annotations, although our model computes masks at a finer level. Each sentence of the
dataset is annotated with one aspect label, indicating what aspect that sentence covers. We evaluate the precision of words highlighted by each model. For both, ours and Lei et al. (2016), we used trained models on beer reviews and extracted a similar number of selected words.
We show that the generated sub-masksMa1 ,Ma2 ,Ma3 obtained with Multi-Aspect Masker (MAM) correlate best with human judgment. Table 4 presents the precision of the masks and attentions computed on sentence-level aspect annotations. We reported results of the models in Lei et al. (2016): SVM, the Single Aspect-Attention (SAA) and Single Aspect-Mask (SAM) — trained separately for each aspect because they find hard justifications for a single aspect. In comparison to SAM, our MAM model obtains significant higher precisions with an average of +1.13. Interestingly, SVM and attention models perform poorly compared with mask models: especially MASA that focuses only on a couple of words due to the sparseness of the attention (examples in Appendix A.3.1).

heading: 4.5 MASK COHERENCE
text: In addition to evaluating masks with human annotations, we computed their semantic interpretability for each dataset. According to Aletras & Stevenson (2013); Lau et al. (2014), NPMI (Bouma, 2009) is a good metric for qualitative evaluation of topics, because it matches human judgment most closely. However, the top-N topic words, used for evaluation, are often selected arbitrarily. To alleviate this problem, we followed Lau & Baldwin (2016): we computed the topic coherence over several cardinalities N , and report all the results, as well as the average; the authors claim the mean leads to a more stable and robust evaluation. More details are available in Appendix A.4.
We show that generated masks by MAM obtains the highest mean NPMI and, on average, superior results in all datasets (17 out of 21 cases), while only needing a single training. Results are shown in Table 5. For the Hotel and Full Beer datasets, where reviews reflect the real data distribution, our model significantly outperforms SAM and attention models for N ≥ 20. For smaller N , MAM obtains higher scores in four out of six cases, and for these two, the difference is only below 0.003.
For the controlled-environment Filtered Beer dataset, MAM still performs better for N ≥ 15, although the differences are smaller, and is beat by SAM for N ≤ 10. However, SAM obtains poor results in all other cases of all datasets and must be trained as many times as the number of aspects.
We show the top words for each aspect and a human evaluation in Appendix A.4. Generally, our model finds better sets of words among the three datasets compared with other methods.

heading: 5 CONCLUSION
text: In this work, we propose Multi-Aspect Masker, an end-to-end neural network architecture to perform multi-aspect sentiment classification for reviews. Our model predicts aspect sentiments while generating a probabilistic (soft) multi-dimensional mask (one dimension per aspect) simultaneously, in an unsupervised and multi-task learning manner. We showed that the induced mask is beneficial to guide the model to focus on different parts of the input text and to further improve the sentiment prediction for all aspects. Our evaluation shows that on three datasets, in the beer and hotel domain, our model outperforms strong baselines and generates masks that are: strong feature predictors, meaningful, and interpretable compared to attention-based methods and a single-aspect masker.

heading: A APPENDIX

heading: A.1 DESCRIPTIVE STATISTICS OF THE DATASETS

heading: A.2 EXPERIMENTAL DETAILS
text: For each model, the review encoder was either a bi-directional single-layer forward recurrent neural network using Long Short-Term Memory (Hochreiter & Schmidhuber, 1997) with 64 hidden units or the multi-channel text convolutional neural network, similar to Kim et al. (2015), with 3, 5, 7 width filters and 50 feature maps per filter. Each aspect classifier is a two-layer feedforward neural network with ReLU activation function (Nair & Hinton, 2010). We used the 200-dimensional pretrained word embeddings of Lei et al. (2016) for beer reviews. For the hotel domain, we trained word2vec (Mikolov et al., 2013) on a large collection of hotel reviews with an embedding size of 300.
We used dropout (Srivastava et al., 2014) of 0.1, clipped the gradient norm at 1.0 if higher, added L2-norm regularizer with a regularization factor of 10−6 and trained using early stopping with a patience of three iterations. We used Adam (Kingma & Ba, 2015) for training with a learning rate of 0.001, β1 = 0.9, and β2 = 0.999. The temperature τ for Gumbel-Softmax distributions was fixed at 0.8. The two regularizer terms and the prior of our model are λsel = 0.03, λcont = 0.04, and λp = 0.11 for the Filtered Beer dataset; λsel = 0.03, λcont = 0.03, and λp = 0.15 for the Full Beer dataseet; and λsel = 0.02, λcont = 0.02 and λp = 0.10 for the Hotel dataset. We ran all experiments for a maximum of 50 epochs with a batch-size of 256 and a Titan X GPU. For the model of Lei et al. (2016), we reused the code from the authors.
A.3 VISUALIZATION OF THE MULTI-DIMENSIONAL FACETS OF REVIEWS
We randomly sampled reviews from each dataset and computed the masks and attentions of four models: our Multi-Aspect Masker model (MAM), the Single Aspect-Mask method (SAM) of Lei et al. (2016) and two attention models with additive and sparse attention, called Multi AspectAttentions (MAA) and Multi Aspect-Sparse-Attentions (MASA) respectively (more details in Section 4.2). Each color represents an aspect and the shade its confidence. All models generate soft attentions or masks besides SAM, which does hard masking. Samples for the Filtered Beer, Full Beer and Hotel dataset are shown below.

heading: A.3.1 FILTERED BEER DATASET

heading: A.3.2 FULL BEER DATASET
text: Appearance Smell Palate Taste
Multi Aspect-Masks (Ours)
Appearance Smell Palate Taste
Single Aspect-Mask (Lei et al., 2016)

heading: A.3.3 HOTEL DATASET
text: Service Cleanliness Value Location Room
Multi Aspect-Masks (Ours)
Service Cleanliness Value Location Room
Single Aspect-Mask (Lei et al., 2016)
Service Cleanliness Value Location Room
Multi-Aspect Masker (Ours)
Service Cleanliness Value Location Room
Single Aspect-Mask (Lei et al., 2016)

heading: A.4 TOPIC WORDS PER ASPECT
text: For each model, we computed the probability distribution of words per aspect by using the induced sub-masks Ma1 , ...,MaA or attention values. Given an aspect ai and a set of top-N words w N ai , the Normalized Pointwise Mutual Information (Bouma, 2009) coherence score is:
NPMI(wNai) = N∑ j=2 j−1∑ k=1 log P (wkai ,wjai ) P (wkai )P (wjai ) − logP (wkai , w j ai)
(6)
Top words of coherent topics (i.e., aspects) should share a similar semantic interpretation and thus, interpretability of a topic can be estimated by measuring how many words are not related. For each aspect ai and word w having been highlighted at least once as belonging to aspect ai, we computed the probability P (w|ai) on each dataset and sorted them in decreasing order of P (w|ai). Unsurprisingly, we found that the most common words are stop words such as ”a” and ”it”, because masks are mostly word sequences instead of individual words. To gain a better interpretation of the aspect words, we followed the procedure in McAuley et al. (2012): we first computed averages across all aspect words for each word w: bw = 1|A| ∑|A| i=1 P (w|ai), which generates a general distribution that includes words common to all aspects. The final word distribution per aspect is computed by removing the general distribution: P̂ (w|ai) = P (w|ai)− bw. After generating the final word distribution per aspect, we picked the top ten words and asked two human annotators to identify intruder words, i.e., words not matching the corresponding aspect. We show in subsequent tables the top ten words for each aspect, where red denotes all words identified as unrelated to the aspect by the two annotators. Generally, our model finds better sets of words across the three datasets compared with other methods. Additionally, we observe that the aspects can be easily recovered given its top words.

heading: A.5 BASELINE ARCHITECTURES

heading: Encoder Classifier
text: ECNN + Clf

heading: Encoder
text: ECNN + AShared + Clf
Encoder
Classifier
ELSTM + AShared + Clf
ELSTM + AAspect-Wise + Clf

Review summaries:
This paper proposes a neural model for multi-aspect sentiment of reviews. Apart from sentiment classification, the architecture is able to return a mask for each aspect highlighting the sequences of words used to make the sentiment prediction for that aspect. This improves the interpretability of the model. The aspects are fixed and the training set requires a multi-labeled dataset with one binary positive/negative label per aspect.