Attentional Encoder Network for Targeted Sentiment Classification

Abstract: Targeted sentiment classification aims at determining the sentimental tendency towards specific targets. Most of the previous approaches model context and target words with RNN and attention. However, RNNs are difficult to parallelize and truncated backpropagation through time brings difficulty in remembering long-term patterns. To address this issue, this paper proposes an Attentional Encoder Network (AEN) which eschews recurrence and employs attention based encoders for the modeling between context and target. We raise the label unreliability issue and introduce label smoothing regularization. We also apply pre-trained BERT to this task and obtain new state-of-the-art results. Experiments and analysis demonstrate the effectiveness and lightweight of our model.

Targeted sentiment classification is a fine-grained sentiment analysis task, which aims at determining the sentiment polarities (e.g., negative, neutral, or positive) of a sentence over “opinion targets” that explicitly appear in the sentence. For example, given a sentence “I hated their service, but their food was great”, the sentiment polarities for the target “service” and “food” are negative and positive respectively. A target is usually an entity or an entity aspect.
In recent years, neural network models are designed to automatically learn useful low-dimensional representations from targets and contexts and obtain promising results BIBREF0 , BIBREF1 . However, these neural network models are still in infancy to deal with the fine-grained targeted sentiment classification task.
Attention mechanism, which has been successfully used in machine translation BIBREF2 , is incorporated to enforce the model to pay more attention to context words with closer semantic relations with the target. There are already some studies use attention to generate target-specific sentence representations BIBREF3 , BIBREF4 , BIBREF5 or to transform sentence representations according to target words BIBREF6 . However, these studies depend on complex recurrent neural networks (RNNs) as sequence encoder to compute hidden semantics of texts.
The first problem with previous works is that the modeling of text relies on RNNs. RNNs, such as LSTM, are very expressive, but they are hard to parallelize and backpropagation through time (BPTT) requires large amounts of memory and computation. Moreover, essentially every training algorithm of RNN is the truncated BPTT, which affects the model's ability to capture dependencies over longer time scales BIBREF7 . Although LSTM can alleviate the vanishing gradient problem to a certain extent and thus maintain long distance information, this usually requires a large amount of training data. Another problem that previous studies ignore is the label unreliability issue, since neutral sentiment is a fuzzy sentimental state and brings difficulty for model learning. As far as we know, we are the first to raise the label unreliability issue in the targeted sentiment classification task.
This paper propose an attention based model to solve the problems above. Specifically, our model eschews recurrence and employs attention as a competitive alternative to draw the introspective and interactive semantics between target and context words. To deal with the label unreliability issue, we employ a label smoothing regularization to encourage the model to be less confident with fuzzy labels. We also apply pre-trained BERT BIBREF8 to this task and show our model enhances the performance of basic BERT model. Experimental results on three benchmark datasets show that the proposed model achieves competitive performance and is a lightweight alternative of the best RNN based models.
The main contributions of this work are presented as follows:
The research approach of the targeted sentiment classification task including traditional machine learning methods and neural networks methods.
Traditional machine learning methods, including rule-based methods BIBREF9 and statistic-based methods BIBREF10 , mainly focus on extracting a set of features like sentiment lexicons features and bag-of-words features to train a sentiment classifier BIBREF11 . The performance of these methods highly depends on the effectiveness of the feature engineering works, which are labor intensive.
In recent years, neural network methods are getting more and more attention as they do not need handcrafted features and can encode sentences with low-dimensional word vectors where rich semantic information stained. In order to incorporate target words into a model, Tang et al. tang2016effective propose TD-LSTM to extend LSTM by using two single-directional LSTM to model the left context and right context of the target word respectively. Tang et al. tang2016aspect design MemNet which consists of a multi-hop attention mechanism with an external memory to capture the importance of each context word concerning the given target. Multiple attention is paid to the memory represented by word embeddings to build higher semantic information. Wang et al. wang2016attention propose ATAE-LSTM which concatenates target embeddings with word representations and let targets participate in computing attention weights. Chen et al. chen2017recurrent propose RAM which adopts multiple-attention mechanism on the memory built with bidirectional LSTM and nonlinearly combines the attention results with gated recurrent units (GRUs). Ma et al. ma2017interactive propose IAN which learns the representations of the target and context with two attention networks interactively.
Given a context sequence INLINEFORM0 and a target sequence INLINEFORM1 , where INLINEFORM2 is a sub-sequence of INLINEFORM3 . The goal of this model is to predict the sentiment polarity of the sentence INLINEFORM4 over the target INLINEFORM5 .
Figure FIGREF9 illustrates the overall architecture of the proposed Attentional Encoder Network (AEN), which mainly consists of an embedding layer, an attentional encoder layer, a target-specific attention layer, and an output layer. Embedding layer has two types: GloVe embedding and BERT embedding. Accordingly, the models are named AEN-GloVe and AEN-BERT.
Let INLINEFORM0 to be the pre-trained GloVe BIBREF12 embedding matrix, where INLINEFORM1 is the dimension of word vectors and INLINEFORM2 is the vocabulary size. Then we map each word INLINEFORM3 to its corresponding embedding vector INLINEFORM4 , which is a column in the embedding matrix INLINEFORM5 .
BERT embedding uses the pre-trained BERT to generate word vectors of sequence. In order to facilitate the training and fine-tuning of BERT model, we transform the given context and target to “[CLS] + context + [SEP]” and “[CLS] + target + [SEP]” respectively.
The attentional encoder layer is a parallelizable and interactive alternative of LSTM and is applied to compute the hidden states of the input embeddings. This layer consists of two submodules: the Multi-Head Attention (MHA) and the Point-wise Convolution Transformation (PCT).
Multi-Head Attention (MHA) is the attention that can perform multiple attention function in parallel. Different from Transformer BIBREF13 , we use Intra-MHA for introspective context words modeling and Inter-MHA for context-perceptive target words modeling, which is more lightweight and target is modeled according to a given context.
An attention function maps a key sequence INLINEFORM0 and a query sequence INLINEFORM1 to an output sequence INLINEFORM2 : DISPLAYFORM0 
 where INLINEFORM0 denotes the alignment function which learns the semantic relevance between INLINEFORM1 and INLINEFORM2 : DISPLAYFORM0 
 where INLINEFORM0 are learnable weights.
MHA can learn n_head different scores in parallel child spaces and is very powerful for alignments. The INLINEFORM0 outputs are concatenated and projected to the specified hidden dimension INLINEFORM1 , namely, DISPLAYFORM0 
 where “ INLINEFORM0 ” denotes vector concatenation, INLINEFORM1 , INLINEFORM2 is the output of the INLINEFORM3 -th head attention and INLINEFORM4 .
Intra-MHA, or multi-head self-attention, is a special situation for typical attention mechanism that INLINEFORM0 . Given a context embedding INLINEFORM1 , we can get the introspective context representation INLINEFORM2 by: DISPLAYFORM0 
 The learned context representation INLINEFORM0 is aware of long-term dependencies.
Inter-MHA is the generally used form of attention mechanism that INLINEFORM0 is different from INLINEFORM1 . Given a context embedding INLINEFORM2 and a target embedding INLINEFORM3 , we can get the context-perceptive target representation INLINEFORM4 by: DISPLAYFORM0 
After this interactive procedure, each given target word INLINEFORM0 will have a composed representation selected from context embeddings INLINEFORM1 . Then we get the context-perceptive target words modeling INLINEFORM2 .
A Point-wise Convolution T ransformation (PCT) can transform contextual information gathered by the MHA. Point-wise means that the kernel sizes are 1 and the same transformation is applied to every single token belonging to the input. Formally, given a input sequence INLINEFORM0 , PCT is defined as: DISPLAYFORM0 
 where INLINEFORM0 stands for the ELU activation, INLINEFORM1 is the convolution operator, INLINEFORM2 and INLINEFORM3 are the learnable weights of the two convolutional kernels, INLINEFORM4 and INLINEFORM5 are biases of the two convolutional kernels.
Given INLINEFORM0 and INLINEFORM1 , PCTs are applied to get the output hidden states of the attentional encoder layer INLINEFORM2 and INLINEFORM3 by: DISPLAYFORM0 
After we obtain the introspective context representation INLINEFORM0 and the context-perceptive target representation INLINEFORM1 , we employ another MHA to obtain the target-specific context representation INLINEFORM2 by: DISPLAYFORM0 
 The multi-head attention function here also has its independent parameters.
We get the final representations of the previous outputs by average pooling, concatenate them as the final comprehensive representation INLINEFORM0 , and use a full connected layer to project the concatenated vector into the space of the targeted INLINEFORM1 classes. DISPLAYFORM0 
 where INLINEFORM0 is the predicted sentiment polarity distribution, INLINEFORM1 and INLINEFORM2 are learnable parameters.
Since neutral sentiment is a very fuzzy sentimental state, training samples which labeled neutral are unreliable. We employ a Label Smoothing Regularization (LSR) term in the loss function. which penalizes low entropy output distributions BIBREF14 . LSR can reduce overfitting by preventing a network from assigning the full probability to each training example during training, replaces the 0 and 1 targets for a classifier with smoothed values like 0.1 or 0.9.
For a training sample INLINEFORM0 with the original ground-truth label distribution INLINEFORM1 , we replace INLINEFORM2 with DISPLAYFORM0 
 where INLINEFORM0 is the prior distribution over labels , and INLINEFORM1 is the smoothing parameter. In this paper, we set the prior label distribution to be uniform INLINEFORM2 .
LSR is equivalent to the KL divergence between the prior label distribution INLINEFORM0 and the network's predicted distribution INLINEFORM1 . Formally, LSR term is defined as: DISPLAYFORM0 
The objective function (loss function) to be optimized is the cross-entropy loss with INLINEFORM0 and INLINEFORM1 regularization, which is defined as: DISPLAYFORM0 
 where INLINEFORM0 is the ground truth represented as a one-hot vector, INLINEFORM1 is the predicted sentiment distribution vector given by the output layer, INLINEFORM2 is the coefficient for INLINEFORM3 regularization term, and INLINEFORM4 is the parameter set.
We conduct experiments on three datasets: SemEval 2014 Task 4 BIBREF15 dataset composed of Restaurant reviews and Laptop reviews, and ACL 14 Twitter dataset gathered by Dong et al. dong2014adaptive. These datasets are labeled with three sentiment polarities: positive, neutral and negative. Table TABREF31 shows the number of training and test instances in each category.
Word embeddings in AEN-GloVe do not get updated in the learning process, but we fine-tune pre-trained BERT in AEN-BERT. Embedding dimension INLINEFORM0 is 300 for GloVe and is 768 for pre-trained BERT. Dimension of hidden states INLINEFORM1 is set to 300. The weights of our model are initialized with Glorot initialization BIBREF16 . During training, we set label smoothing parameter INLINEFORM2 to 0.2 BIBREF14 , the coefficient INLINEFORM3 of INLINEFORM4 regularization item is INLINEFORM5 and dropout rate is 0.1. Adam optimizer BIBREF17 is applied to update all the parameters. We adopt the Accuracy and Macro-F1 metrics to evaluate the performance of the model.
In order to comprehensively evaluate and analysis the performance of AEN-GloVe, we list 7 baseline models and design 4 ablations of AEN-GloVe. We also design a basic BERT-based model to evaluate the performance of AEN-BERT.
 
Non-RNN based baselines:
 INLINEFORM0 Feature-based SVM BIBREF18 is a traditional support vector machine based model with extensive feature engineering.
 INLINEFORM0 Rec-NN BIBREF0 firstly uses rules to transform the dependency tree and put the opinion target at the root, and then learns the sentence representation toward target via semantic composition using Recursive NNs.
 INLINEFORM0 MemNet BIBREF19 uses multi-hops of attention layers on the context word embeddings for sentence representation to explicitly captures the importance of each context word.
 
RNN based baselines:
 INLINEFORM0 TD-LSTM BIBREF1 extends LSTM by using two LSTM networks to model the left context with target and the right context with target respectively. The left and right target-dependent representations are concatenated for predicting the sentiment polarity of the target.
 INLINEFORM0 ATAE-LSTM BIBREF3 strengthens the effect of target embeddings, which appends the target embeddings with each word embeddings and use LSTM with attention to get the final representation for classification.
 INLINEFORM0 IAN BIBREF4 learns the representations of the target and context with two LSTMs and attentions interactively, which generates the representations for targets and contexts with respect to each other.
 INLINEFORM0 RAM BIBREF5 strengthens MemNet by representing memory with bidirectional LSTM and using a gated recurrent unit network to combine the multiple attention outputs for sentence representation.
 
AEN-GloVe ablations:
 INLINEFORM0 AEN-GloVe w/o PCT ablates PCT module.
 INLINEFORM0 AEN-GloVe w/o MHA ablates MHA module.
 INLINEFORM0 AEN-GloVe w/o LSR ablates label smoothing regularization.
 INLINEFORM0 AEN-GloVe-BiLSTM replaces the attentional encoder layer with two bidirectional LSTM.
 
Basic BERT-based model:
 INLINEFORM0 BERT-SPC feeds sequence “[CLS] + context + [SEP] + target + [SEP]” into the basic BERT model for sentence pair classification task.
Table TABREF34 shows the performance comparison of AEN with other models. BERT-SPC and AEN-BERT obtain substantial accuracy improvements, which shows the power of pre-trained BERT on small-data task. The overall performance of AEN-BERT is better than BERT-SPC, which suggests that it is important to design a downstream network customized to a specific task. As the prior knowledge in the pre-trained BERT is not specific to any particular domain, further fine-tuning on the specific task is necessary for releasing the true power of BERT.
The overall performance of TD-LSTM is not good since it only makes a rough treatment of the target words. ATAE-LSTM, IAN and RAM are attention based models, they stably exceed the TD-LSTM method on Restaurant and Laptop datasets. RAM is better than other RNN based models, but it does not perform well on Twitter dataset, which might because bidirectional LSTM is not good at modeling small and ungrammatical text.
Feature-based SVM is still a competitive baseline, but relying on manually-designed features. Rec-NN gets the worst performances among all neural network baselines as dependency parsing is not guaranteed to work well on ungrammatical short texts such as tweets and comments. Like AEN, MemNet also eschews recurrence, but its overall performance is not good since it does not model the hidden semantic of embeddings, and the result of the last attention is essentially a linear combination of word embeddings.
As shown in Table TABREF34 , the performances of AEN-GloVe ablations are incomparable with AEN-GloVe in both accuracy and macro-F1 measure. This result shows that all of these discarded components are crucial for a good performance. Comparing the results of AEN-GloVe and AEN-GloVe w/o LSR, we observe that the accuracy of AEN-GloVe w/o LSR drops significantly on all three datasets. We could attribute this phenomenon to the unreliability of the training samples with neutral sentiment. The overall performance of AEN-GloVe and AEN-GloVe-BiLSTM is relatively close, AEN-GloVe performs better on the Restaurant dataset. More importantly, AEN-GloVe has fewer parameters and is easier to parallelize.
To figure out whether the proposed AEN-GloVe is a lightweight alternative of recurrent models, we study the model size of each model on the Restaurant dataset. Statistical results are reported in Table TABREF37 . We implement all the compared models base on the same source code infrastructure, use the same hyperparameters, and run them on the same GPU .
RNN-based and BERT-based models indeed have larger model size. ATAE-LSTM, IAN, RAM, and AEN-GloVe-BiLSTM are all attention based RNN models, memory optimization for these models will be more difficult as the encoded hidden states must be kept simultaneously in memory in order to perform attention mechanisms. MemNet has the lowest model size as it only has one shared attention layer and two linear layers, it does not calculate hidden states of word embeddings. AEN-GloVe's lightweight level ranks second, since it takes some more parameters than MemNet in modeling hidden states of sequences. As a comparison, the model size of AEN-GloVe-BiLSTM is more than twice that of AEN-GloVe, but does not bring any performance improvements.
In this work, we propose an attentional encoder network for the targeted sentiment classification task. which employs attention based encoders for the modeling between context and target. We raise the the label unreliability issue add a label smoothing regularization to encourage the model to be less confident with fuzzy labels. We also apply pre-trained BERT to this task and obtain new state-of-the-art results. Experiments and analysis demonstrate the effectiveness and lightweight of the proposed model.

Figure and table captions:
Figure 1: Overall architecture of the proposed AEN.
Table 1: Statistics of the datasets.
Table 2: Main results. The results of baseline models are retrieved from published papers. Top 2 scores are in bold.
Table 3: Model sizes. Memory footprints are evaluated on the Restaurant dataset. Lowest 2 are in bold.

Questions about this paper:
How big is their model?

Answers:
Proposed model has 1.16 million parameters and 11.04 MB.