Multi-Dimensional Explanation of Reviews

1 INTRODUCTION . Neural networks have become the standard for many natural language processing tasks . Despite the significant performance gains achieved by these complex models , they offer little transparency concerning their inner workings . Thus , they come at the cost of interpretability ( Jain & Wallace , 2019 ) . In many domains , automated predictions have a real impact on the final decision , such as treatment options in the field of medicine . Therefore , it is important to provide the underlying reasons for such a decision . We claim that integrating interpretability in a ( neural ) model should supply the reason of the prediction and should yield better performance . However , justifying a prediction might be ambiguous and challenging . Prior work includes various methods that find the justification in an input text — also called rationale or mask of a target variable . The mask is defined as one or multiple pieces of text fragments from the input text.1 Each should contain words that altogether are short , coherent , and alone sufficient for the prediction as a substitute of the input ( Lei et al. , 2016 ) . Many works have been applied to single-aspect sentiment analysis for reviews , where the ambiguity of a justification is minimal . In this case , we define an aspect as an attribute of a product or service ( Giannakopoulos et al. , 2017 ) , such as Location or Cleanliness for the hotel domain . There are three different methods to generate masks : using reinforcement learning with a trained model ( Li et al. , 2016b ) , generating rationales in an unsupervised manner and jointly with the objective function ( Lei et al. , 2016 ) , or including annotations during training ( Bao et al. , 2018 ; Zhang et al. , 2016 ) . However , these models generate justifications that are 1 ) only tailored for one aspect , and 2 ) expressed as a hard ( binary ) selection of words . A review text reflects opinions about multiple topics a user cares about ( Musat et al. , 2013 ) . It appears reasonable to analyze multiple aspects with a multi-task learning setting , but a model must be trained as many times as the number of aspects . A hard assignment of words to aspects might lead to ambiguities that are difficult to capture with a binary mask : in the text ” The room was large , clean and close to the beach. ” , the word ” room ” refers to the aspects Room , Cleanliness and Location . Finally , collecting human-provided rationales at scale is expensive and thus impractical . In this work , we study interpretable multi-aspect sentiment classification . We describe an architecture for predicting the sentiment of multiple aspects while generating a probabilistic ( soft ) multi- 1In the rest of the paper , we will use the terms mask , justification and rationale interchangeably . dimensional mask ( one dimension per aspect ) jointly , in an unsupervised and multi-task learning manner . We show that the induced mask is beneficial for identifying simultaneously what parts of the review relate to what aspect , and capturing ambiguities of words belonging to multiple aspects . Thus , the induced mask provides fine-grained interpretability and improves the final performance . Traditionally interpretability came at a cost of reduced accuracy . In contrast , our evaluation shows that on three datasets , in the beer and hotel domain , our model outperforms strong baselines and generates masks that are : strong feature predictors , meaningful , and interpretable compared to attention-based methods and a single-aspect masker . We show that it can be a benefit to 1 ) guide the model to focus on different parts of the input text , and 2 ) further improve the sentiment prediction for all aspects . Therefore , interpretabilty does not come at a cost anymore . The contributions of this work can be summarized as follow : • We propose a Multi-Aspect Masker ( MAM ) , an end-to-end neural model for multi-aspect sentiment classification that provides fine-grained interpretability in the same training . Given a text review as input , the model generates a probabilistic multi-dimensional mask , with one dimension per aspect . It predicts the sentiments of multiple aspects , and highlights long sequences of words justifying the current rating prediction for each aspect ; • We show that interpretability does not come at a cost : our final model significantly outperforms strong baselines and attention models , both in terms of performance and mask coherence . Furthermore , the level of interpretability is controllable using two regularizers ; • Finally , we release a new dataset for multi-aspect sentiment classification , which contains 140k reviews from TripAdvisor with five aspects , each with its corresponding rating.2 2 RELATED WORK . 2.1 INTERPRETABILITY . Developing interpretable models is of considerable interest to the broader research community , even more pronounced with neural models ( Kim et al. , 2015 ; Doshi-Velez & Kim , 2017 ) . Many works analyzed and visualized state activation ( Karpathy et al. , 2015 ; Li et al. , 2016a ; Montavon et al. , 2018 ) , learned sparse and interpretable word vectors ( Faruqui et al. , 2015b ; a ; Herbelot & Vecchi , 2015 ) or analyzed attention ( Clark et al. , 2019 ; Jain & Wallace , 2019 ) . Our work differs from these in terms of what is meant by an explanation . Our system identifies one or multiple short and coherent text fragments that — as a substitute of the input text — are sufficient for the prediction . 2.2 ATTENTION-BASED MODELS . Attention models ( Vaswani et al. , 2017 ; Yang et al. , 2016 ; Lin et al. , 2017 ) have been shown to improve prediction accuracy , visualization , and interpretability . The most popular and widely used attention mechanism is soft attention ( Bahdanau et al. , 2015 ) over hard ( Luong et al. , 2015 ) and sparse ones ( Martins & Astudillo , 2016 ) . According to Jain & Wallace ( 2019 ) ; Serrano & Smith ( 2019 ) , standard attention modules noisily predict input importance ; the weights can not provide safe and meaningful explanations . Our model differs in two ways from attention mechanisms : our loss includes two regularizers to favor long word sequences for interpretability ; the normalization is not done over the sequence length . 2.3 MULTI-ASPECT SENTIMENT CLASSIFICATION . Review multi-aspect sentiment classification is sometimes seen as a sub-problem ( Wang et al. , 2010 ; McAuley et al. , 2012 ; Pappas & Popescu-Belis , 2014 ) , by utilizing heuristic-based methods or topic models . Recently , neural models achieved significant improvements with less feature engineering . Yin et al . ( 2017 ) built a hierarchical attention model with aspect representations by using a set of manually defined topics . Li et al . ( 2018 ) extended this work with user attention and additional features such as overall rating , aspect , and user embeddings . The disadvantage of these methods is their limited interpretability , as they rely on many features in addition to the review text . 2We will make the code and data available . 2.4 RATIONALE-BASED MODELS . The idea of including human rationales during training has been explored in ( Zhang et al. , 2016 ; Marshall et al. , 2015 ; Bao et al. , 2018 ) . Although they have been shown to be beneficial , they are expensive to collect and might vary across annotators . In our work , no annotation is used . The work most closely related to ours is Li et al . ( 2016b ) and Lei et al . ( 2016 ) . Both generate hard rationales and address single-aspect sentiment classification . Their model must be trained separately for each aspect , which leads to ambiguities . Li et al . ( 2016b ) developed a post-training method that removes words from a review text until another trained model changes its prediction . Lei et al . ( 2016 ) provides a model that learns an aspect sentiment and its rationale jointly , but hinders the performance and relies on assumptions on the data , such as a small correlation between aspect ratings . In contrast , our model : 1 ) supports multi-aspect sentiment classification , 2 ) generates soft multidimensional masks in a single training ; 3 ) the masks provide interpretability and improve the performance significantly . 3 METHOD : MULTI-ASPECT MASKER . Review Text . ECNN + Masker + Clf Let X be a review composed of L words x1 , x2 , ... , xL and Y the target A-dimensional sentiment vector , corresponding to the different rated aspects . Our proposed model , called Multi-Aspect Masker , is composed of three components : 1 ) a Masker module that computes a probability distribution over aspects for each word , resulting in A+1 different masks ( including one for not-aspect ) ; 2 ) an Encoder that learns a representation of a review conditioned on the induced masks ; 3 ) a Classifier that predicts the target variables . The overall model architecture is shown in Figure 1 . Our framework generalizes for other tasks , and each neural module is interchangeable with other models . The Masker first computes a hidden representation h ` for each word x ` in the input sequence , using their word embeddings e1 , e2 , ... , eL . Many sequence models could realize this task , such as recurrent , attention , or convolution neural networks . In our case , we chose a convolutional network because it led to a smaller model , faster training , and empirically , performed similarly to recurrent models . Let ai denote the ith aspect for i = 1 , ... , A , and a0 the not-aspect case , because many words can be irrelevant to every aspect . We define M ` ∈ R ( A+1 ) , the aspect distribution of the input word x ` as : P ( M|X ) = L∏ ` =1 P ( M ` |x ` ) = L∏ ` =1 A∏ i=0 P ( m ` ai |x ` ) ( 1 ) Because we have categorical distributions , we can not directly sample P ( M ` |x ` ) and backpropagate the gradient through this discrete generation process . Instead , we model the variable m ` ai using the Straight Through Gumbel Softmax ( Jang et al. , 2017 ; Maddison et al. , 2017 ) , to approximate sampling from a categorical distribution . We model the parameters of each Gumbel Softmax distribution M ` with a single-layer feedforward neural network followed by applying a log softmax , which induces the log-probabilities of the ` th distribution : ω ` = log ( softmax ( Wh ` + b ) ) . W and b are shared across all tokens , to have a constant number of parameters with respect to the sequence length . We control the sharpness of the distributions with the temperature parameter τ . Compared to attention mechanisms , the word importance is a probability distribution over the targets : ∑T t=0 P ( m ` at |x ` ) = 1 , instead of a normalization over the sequence length , ∑L ` =1 P ( a ` |x ` ) = 1 . Given a soft multi-dimensional mask M ∈ R ( A+1 ) ×L , we define each sub-mask Mai ∈ RL as : Mai = P ( m 1 ai |x 1 ) , P ( m2ai |x 2 ) , ... , P ( mLai |x L ) ( 2 ) We weight the word embeddings by their importance towards an aspect ai with the induced submasks , such thatEai =Mai E = P ( m1ai |x 1 ) ·e1 , P ( m2ai |x 2 ) ·e2 , ... , P ( mLai |x L ) ·eL . Thereafter , each modified embedding Eai is fed into the Encoder block . Note that Ea0 is ignored because Ma0 only serves to absorb probabilities of words that are insignificant to every aspect.3 The Encoder module includes a convolutional neural network , for the same reasons as earlier , followed by a max-over-time pooling to obtain a fixed-length feature vector . It produces the hidden representation hai for each aspect ai . To exploit commonalities and differences among aspects , we share the weights of the encoders for allEai . Finally , the Classifier block contains for each aspect ai a two-layer feedforward neural networks followed by a softmax layer to predict the sentiment ŷai .

Summary: This paper proposes a neural model for multi-aspect sentiment of reviews. Apart from sentiment classification, the architecture is able to return a mask for each aspect highlighting the sequences of words used to make the sentiment prediction for that aspect. This improves the interpretability of the model. The aspects are fixed and the training set requires a multi-labeled dataset with one binary positive/negative label per aspect.