Experiments ::: Results	Table TABREF34 and Table TABREF35 show the final results. Our model outperforms all the baseline models and the top-performing models with both generated summary and golden summary, for all the three datasets. In the scenario where golden summaries are used, BiLSTM+self-attention performs the best among all the baselines, which shows that attention is a useful way to integrate summary and review information. Hard-attention receives more supervision information compared with soft-attention, by supervision signals from extractive summaries. However, it underperforms the soft attention model, which indicates that the most salient words for making sentiment classification may not strictly overlap with extractive summaries. This justifies the importance of user written or automatic-generated summary. A comparison between models that use summary information and those that do not use summary information shows that the review summary is useful for sentiment classification. In addition, the same models work consistently better when the user written gold summary is used compared to a system generated summary, which is intuitively reasonable since the current state-of-the-art abstractive summarization models are far from perfect.

Exploring Hierarchical Interaction Between Review and Summary for Better Sentiment Analysis	Sentiment analysis provides a useful overview of customer review contents. Many review websites allow a user to enter a summary in addition to a full review. It has been shown that jointly predicting the review summary and the sentiment rating benefits both tasks. However, these methods consider the integration of review and summary information in an implicit manner, which limits their performance to some extent. In this paper, we propose a hierarchically-refined attention network for better exploiting multi-interaction between a review and its summary for sentiment analysis. In particular, the representation of a review is layer-wise refined by attention over the summary representation. Empirical results show that our model can better make use of user-written summaries for review sentiment analysis, and is also more effective compared to existing methods when the user summary is replaced with summary generated by an automatic summarization system.

Introduction	Sentiment analysis ,  is a fundamental task in natural language processing. In particular, sentiment analysis of user reviews has wide applications, , , . In many review websites such as Amazon and IMDb, the user is allowed to give a summary in addition to their review. Summaries usually contain more abstract information about the review. As shown in Figure FIGREF3, two screenshots of reviews were taken from Amazon and IMDb websites, respectively. The user-written summaries of these reviews can be highly indicative of the final polarity. As a result, it is worth considering them together with the review itself for making sentiment classification. To this end, some recent work ,  exploits joint modeling. The model structure can be illustrated by Figure FIGREF4. In particular, given a review input, a model is trained to simultaneously predict the sentiment and summary. As a result, both summary information and review information are integrated in the review encoder through back-propagation training. However, one limitation of this method is that it does not explicitly encode a summary during test time. One solution, as shown in Figure FIGREF4, is to train a separate summary generator, which learns to predict a summary given a review.

Summary information is integrated into the representation of the review content at each attention layer, thus, a more abstract review representation is learned in subsequent layers based on a lower-layer representation. This mechanism allows the summary to better guide the representation of the review in a bottom-up manner for improved sentiment classification. We evaluate our proposed model on the SNAP (Stanford Network Analysis Project) Amazon review datasets , which contain not only reviews and ratings, but also golden summaries. In scenarios where there is no user-written summary for a review, we use pointer-generator network  to generate abstractive summaries. Empirical results show that our model significantly outperforms all strong baselines, including joint modeling, separate encoder and joint encoder methods. In addition, our model achieves new state-of-the-art performance, attaining 2.1% (with generated summary) and 4.8% (with golden summary) absolutely improvements compared to the previous best method on SNAP Amazon review benchmark.

Method ::: Hierarchically-Refined Review Encoder ::: Sequence Encoding Layer	Given a review $\mathbf {x}^w = \lbrace emb(x_1^w),...,emb(x_n^w)\rbrace $, another BiLSTM is adopted (the same equation with different parameters compared to the one used in the summary encoder), deriving a sequence of review hidden states $\mathbf {H}^w=\lbrace \mathbf {h}^w_1, \mathbf {h}^w_2,...,\mathbf {h}^s_n \rbrace $.

Experiments ::: Baselines ::: BiLSTM+Hard Attention	To demonstrate the efficiency of our model structure, we also adopt hard attention  for comparison, which is supervised using an extractive summarization objective. In particular, words in the original review that match to the corresponding summary are treated as the summary in their original order. In the case of Figure FIGREF3, the extractive summaries for the review are “James Cameron's Titanic is easily the most overrated film in history”, which corresponds to the user-written summary “James Cameron's 1997 Titanic is easily the most overrated film in history!”. The model also calculates another loss between attention weights and extractive summary labels, so that the hard attention weights are trained to strictly follow the extractive summary. For baselines that adopt the separate encoder structure, we generally calculate the representations of review and summary separately with two encoders that hold their own parameters, and then concatenate the two representations alongside the hidden-size dimension. For the joint encoder baselines, we first concatenate the review and summary text, and then encode the concatenated text with one single encoder.

This allows a sentiment classifier to simultaneously encode the review and its summary, before making a prediction using both representations. One further advantage of this model is that it can make use of a user-given summary if it is available with the review, which is the case for the review websites shown in Figure 1. We therefore investigate such a model. One limitation of this method, however, is that it does not capture interaction of review and summary information as thoroughly as the method shown in Figure FIGREF4, since the review and the summary are encoded using two separate encoders. To address this issue, we further investigate a joint encoder for review and summary, which is demonstrated in Figure FIGREF4. The model works by jointly encoding the review and the summary in a multi-layer structure, incrementally updating the representation of the review by consulting the summary representation at each layer. As shown in Figure FIGREF5, our model consists of a summary encoder, a hierarchically-refined review encoder and an output layer. The review encoder is composed of multiple attention layers, each consisting of a sequence encoding layer and an attention inference layer.

Experiments ::: Datasets	We empirically compare different methods using Amazon SNAP Review Dataset , which is a part of Stanford Network Analysis Project. The raw dataset consists of around 34 millions Amazon reviews in different domains, such as books, games, sports and movies. Each review mainly contains a product ID, a piece of user information, a plain text review, a review summary and an overall sentiment rating which ranges from 1 to 5. The statistics of our adopted dataset is shown in Table TABREF20. For fair comparison with previous work, we adopt the same partitions used by previous work , , which is, for each domain, the first 1000 samples are taken as the development set, the following 1000 samples as the test set, and the rest as the training set.

Experiments ::: Baselines ::: BiLSTM+Self-attention @!START@@!END@.	This baseline uses a BiLSTM with hidden size of 256 in both directions. On the top of BiLSTM, self-attention is used to provide a set of summation weight vectors for the final representation. This method is conceptually simple yet gives the state-of-the-art results for many classification and text matching tasks. Its main difference to our model lies in the fact that attention is performed only in the top hidden layer in this method, yet in every layer in ours.

Interestingly, as shown in the second section of the table, the gold summary itself does not lead to better sentiment accuracy compared with the review itself, which shows that summaries better serve as auxiliary information sources to review contents. With both gold summaries and automatic-generated summaries, our model gives better results as compared to BiLSTM+self-attention. The latter integrates information from reviews and summaries only in the top representation layer, which is also the standard practice in question answering  and machine translation  models. In contrast, our model integrates summary information into the review representation in each layer, thereby allowing the integrated representation to be hierarchically refined, leading to more abstract hidden states. Finally, the fact that with gold summary, our baseline and final models outperforms the state-of-the-art methods by jointly training shows the importance of making use of user written summaries when they are available. Even with system summary, out models still outperforms HSSC and SAHSSC, showing that our network is more effective than parameter sharing under the same setting without input summaries.

Experiments ::: Development Experiments	We use the Toys & Games development set to investigate different key configurations of our model. The results are shown in Table TABREF29.

Related Work	The majority of recent sentiment analysis models are based on either convolutional or recurrent neural networks to encode sequences , . In particular, attention-based models have been widely explored, which assign attention weights to hidden states to generate a representation of the input sequence. A hierarchical model with two levels of attention mechanisms was proposed for document classification . Self-attention mechanism has also been used in sentiment analysis , . However,  empirically showed that self-attention mechanism does not consistently agree with the most salient features, which means that self-attention models may suffer from attending on explicit but irrelevant sentimental words. Rationales were also introduced to sentiment analysis task.  proposed a unsupervised latent model that selects a rationale and then uses the rationale for sentiment analysis. A rationale-augmented CNN model  was proposed, which regards golden rationales as additional input and uses the probability as rationale-level attention weights to generate the final representation for text classification. There has also been work focusing on joint summarization and sentiment classification , , whose general structures are illustrated in Figure FIGREF4. These models can predict sentiment label and summary simultaneously. However, they do not encode summaries explicitly during test time, which makes their performance be limited to some extent.

Experiments ::: Experimental Settings	We use GloVe  300-dimensional embeddings as pretrained word vectors. A LSTM hidden size of 256 and four heads for multi-head attention mechanism are adopted. We use Adam  to optimize our model, with an initial learning rate of 0.0003, a decay rate of 0.97, momentum parameters $\beta _1 = 0.9$, $\beta _2 = 0.999$, and $\epsilon = 1 \times 10^{-8}$. The dropout rate is set depending on the size of each dataset, which is 0.5 for both Toys & Games and Sports & Outdoors and 0.2 for Movies & TV. We conduct experiments with both golden summaries and generated summaries. For generating automatic-decoded summaries, we train a pointer-generator network (PG-Net) with coverage mechanism , which is a specially designed sequence-to-sequence attention-based model that can generate the summary by copying words from the text document or generating words from a fixed vocabulary set at the same time. We generally follow the experimental settings in the original paper except for some minor adjustments specially made for our datasets. Noted that in our work PG-Net can be replaced by any other summarization model.

Experiments ::: Baselines ::: SAHSSC @!START@@!END@.	This work also adopts encoder parameter sharing for jointly sentiment classification and summarization. They use two separate BiLSTMs with self-attention mechanism for generating review and summary representations.

Following , we adopt a residual connection around each attention inference layer, followed by layer normalization  : $\mathbf {H}$ is then fed to the subsequent sequence encoding layer as input, if any. According to the equations of standard LSTM and Equation DISPLAY_FORM13, tokens of the original review that are the most relevant to the summary are focused on more by consulting summary representation. The hidden states $\mathbf {H}^{w,s}$ are thus a representation matrix of the review text that encompass key features of summary representation. Multi-head attention mechanism ensures that multi-faced semantic dependency features can be captured during the process, which is beneficial for scenarios where several key points exist in one review. Note also that our design of the review encoding part of the hierarchically-refined attention network is similar to the Transformer architecture in the use of multi-head attention, residual connection and layer normalization . However, our experiments show that bi-directional LSTM works better compared to self-attention network as a basic layer structure. This may result from the fact that Transformer requires a larger amount of training data for the most effectiveness.

Experiments ::: Development Experiments ::: Self-attention Baseline	We compare different numbers of BiLSTM layers and hidden sizes in BiLSTM self-attention. As can be seen, with more layers a stacked BiLSTM with larger hidden sizes does not give better results compared to a hidden size of 256 either.

Experiments ::: Development Experiments ::: Number of Layers	As Table TABREF29 shows, the accuracy increases when increasing layer numbers from 1 to 2. More layers do not increase the accuracy on development set. We thus set 2 as the number of review encoder layers in the experiments. The best performing model size is comparable to that of the BiLSTM self-attention, demonstrating that the number of parameters is not the key factor to models' performance.

Conclusion	We investigated a hierarchically-refined attention network for better sentiment prediction. Our model allows multi-interaction between summary and review representation in a hierarchical manner. Empirical results show that the proposed method outperforms all strong baselines and previous work and achieves new state-of-the-art performance on SNAP Amazon Review dataset.

Experiments ::: Results ::: Case Study	Our model has a natural advantage of interpretability thanks to the use of attention inference layer. We visualize the hierarchically-refined attention of two samples from the test set of Toys & Games. We also visualize self-attention distribution for fair comparison. To make the visualizations clear and to avoid confusion, we choose to visualize the most salient parts, by rescaling all attention weights into an interval of $[0, 100]$ and adopting 50 as a threshold for attention visualization, showing only attention weights $\ge 50$. As shown in Figure FIGREF38, the example with generated summary has 5 stars as its golden rating score. The summary text is “fun for the whole new game in all ages ! ! ! fun ! ! !", which suggests that the game is (1) fun (from word “fun") and (2) not difficult to learn (from phrase “all ages"). It can be seen that both the self-attention model and the first layer of our model attend to the strongly positive phrase “quite fun", which is relevant to the word “fun" in the summary.

Experiments ::: Baselines ::: BiLSTM+Pooling.	For this baseline, we use a BiLSTM with hidden sizes of 256 in both directions, and average pooling across all hidden states to form the representation. This method serves as a naive baseline for making use of both review and summary in sentiment classification. It can also be used to compare the effectiveness of the review itself, the summary itself and the combination of both when used as inputs to the problem.

Experiments ::: Baselines ::: HSSC @!START@@!END@.	This model adopts encoder parameter sharing for jointly sentiment classification and summarization. It predicts the sentiment label using a highway layer, concatenating the hidden state in summary decoder and the original text representation in encoder.

Experiments ::: Results ::: Review Length	Figure FIGREF37 consists of line graphs on the accuracy of BiLSTM+self-attention, BiLSTM+pooling and our model against the review length. As the review length increases, the performance of all models decreases. BiLSTM+self-attention does not outperform BiLSTM+pooling on long text. Our method gives better results compared to two baseline models for long reviews, demonstrating that our model is effective for capturing long-term dependency. This is likely because hierarchically-refined attention maintains the most salient information while ignoring the redundant parts of the original review text. Our model can thus be more robust when review has irrelevant sentimental words, which usually exists in larger reviews such as the example in Figure FIGREF3. The hierarchical architecture allows the lower layers to encode local information, while the higher layers can capture long-term dependency and thus better encode global information.

In comparisons the second layer attends to the phrase “much easier", which is relevant to the phrase “in all ages" in the summary. This verifies our model's effectiveness of leveraging abstractive summary information. Figure FIGREF38 illustrates a 5-star-rating example with golden summary. The summary text is “Favorite Game to Teach to Newbies". As shown in the heatmap, self-attention can only attend to some general sentimental words, such as “hard", “fun", “immensely" and “most", which deviates from the main idea of the document text. In comparison, the first layer of our model attends to phrases like “easy to teach", which is a perfect match of the phrase “teach to newbies" in the summary. This shows that the shallow sequence inference layer can learn direct similarity matching information under the supervision of summarization. In addition, the second layer of our model attends to phrases including “would recommend this to anyone", which links to “easy to teach" and “Teach to Newbies", showing that the deeper sequence inference layer of our model can learn potential connections between the review and the summary.

Method ::: Model Overview	Figure FIGREF5 gives the architecture of the proposed model, which consists of three modules: a summary encoder, a hierarchically-refined review encoder and an output layer. The summary encoder encodes the summary into a hidden state matrix. The review encoder consists of several layers for representing $\mathbf {x}^w$, each containing a sequence encoding sublayer and an attention inference sublayer. The sequence encoding sublayer encodes the review text as a word sequence. The attention inference layer acts as a key component, which takes the hidden states from both the original review and the summary as input calculating dot-product attention weights for original review under additional supervision from summary information. Multi-head attention  as well as residual connection are also adopted. The output layer predicts the potential sentiment label according to hidden states from the previous layer.

Method ::: Summary Encoder	Input for the summary encoder is a sequence of summary word representations $\mathbf {x}^s = \mathbf {x}^s_1, \mathbf {x}^s_2, ..., \mathbf {x}^s_m = \lbrace emb(x_1^s), ..., emb(x_m^s)\rbrace $, where $emb$ denotes a word embedding lookup table. Word representations are fed into a standard BiLSTM. We adopt a standard LSTM formulation, where a sequence of hidden states $\mathbf {h}_t$ are calculated from a sequence of $\mathbf {x}_t$($t \in [1,...,m]$).

Method ::: Problem Formulation	The input to our task is a pair $(X^w, X^s)$, where $X^w = x^w_1, x^w_2, ..., x^w_n$ is a summary and $X^s = x^s_1, x^s_2,...,x^s_m$ is a review, the task is to predict the sentiment label $y \in [1, 5]$, where 1 denotes the most negative sentiment and 5 denotes the most positive sentiment. $n$ and $m$ denote the size of the review and summary in the number of words, respectively. The training set is $D=\lbrace (X^w_i, X^s_i, y_i)\rbrace |_{i=1}^M$ where $M$ is the total number of training examples.

Method ::: Hierarchically-Refined Review Encoder ::: Attention Inference Layer	In the attention inference layer, we model the dependencies between the original review and the summary with multi-head dot-product attention.Each head produces an attention matrix $\mathbf {\alpha } \in \mathbb {R}^{d_h \times 1}$ consisting of a set of similarity scores between the hidden state of each token of the review text and the summary representation. The hidden state outputs are calculated by where $\mathbf {W}_i^Q \in \mathbb {R}^{d_{h} \times \frac{d_{h}}{k}}$, $\mathbf {W}_i^K \in \mathbb {R}^{d_{h} \times \frac{d_{h}}{k}}$ and $\mathbf {W}_i^V \in \mathbb {R}^{d_{h} \times \frac{d_{h}}{k}}$ are model parameters. $Q$, $K$ and $V$ represent Query, Key and Value, respectively. $k$ is the number of parallel heads and $i \in [1,k]$ indicates which head is being processed.

A forward left-to-right LSTM layer and a backward right-to-left LSTM yield a sequence of forward hidden states $\lbrace {\stackrel{\rightarrow }{\mathbf {h}_1^s}},...,{\stackrel{\rightarrow }{\mathbf {h}_n^s}}\rbrace $ and a sequence of backward hidden states $\lbrace {\stackrel{\leftarrow }{\mathbf {h}_1^s}},...,{\stackrel{\leftarrow }{\mathbf {h}_n^s}}\rbrace $, respectively. The two hidden states are concatenated to form a final representation: We then apply an average-pooling operation over the hidden and take $\mathbf {h}^s = avg\_pooling(\mathbf {h}^s_1, \mathbf {h}^s_2,...,\mathbf {h}^s_n)$ as the final representation of summary text.

Experiments	We compare our model with several strong baselines and previous state-of-the-art methods, investigating its main effects.

Method ::: Hierarchically-Refined Review Encoder	The hierarchically-refined review encoder consists of several review encoder layers, each of which is composed of a sequence encoding layer and an attention inference layer.

Method ::: Output Layer	Finally, global average pooling is applied after the previous layer, and then followed by a classifier layer: where $\hat{y}$ is the predicted sentiment label; $\mathbf {W}$ and $\mathbf {b}$ are parameters to be learned.

Method ::: Training	Given a dataset $D={\lbrace (X^w_t,X^s_t,y_t)\rbrace }|^{|T|}_{t=1}$, our model can be trained by minimizing the cross-entropy loss between where $\mathbf {p}^{y_t}$ denotes the value of the label in $\mathbf {p}$ that corresponds to $y_t$.

Experiments ::: Development Experiments ::: Hidden Size	We see an evident improvement of our model when the hidden size increases from 128 to 256. However, the improvement becomes relatively small compared to a large increase in the number of parameters when the hidden size is further increased to 360. Therefore, we adopt 256 as the hidden size in our experiments.

Method	In this section, we introduce our proposed model in details. We first give the problem formulation, followed by an overview of the proposed model, and explain each layer of our model in details, before finally giving the loss function and training methods.

Question: What is the performance difference of using a generated summary vs. a user-written one?
Answer: 2.7 accuracy points