Modeling Complex Interactions in Long Documents for Aspect-Based Sentiment Analysis (2024)

Abstract: The growing number of online articles and reviews necessitates innovative techniques for document-level aspect-based sentiment analysis. Capturing the context in which an aspect is mentioned is crucial. Existing models have focused on relatively short reviews and may fail to consider distant contextual information. This is especially so in longer documents where an aspect may be referred to in multiple ways across dispersed sentences. This work introduces a hierarchical Transformer-based architecture that encodes information at different level of granularities with attention aggregation mechanisms to learn the local and global aspectspecific document representations. For empirical validation, we curate two datasets of long documents: one on social issues, and another covering various topics involving trust-related issues. Experimental results show that the proposed architecture outperforms state-of-the-art methods for document-level aspect-based sentiment classification. We also demonstrate the potential applicability of our approach for long document trust prediction.

heading: 1 Introduction
text: As user-generated content on the web continues to multiply at an exponential rate, the need for automated sentiment in these documents has grown markedly. The ability to discover fine-grained sentiments can provide valuable insights as to how, why, and where an entity is liked and trusted1. Early works have focused on classifying the overall sentiment of a document (Yang et al., 2016; Turney, 2002; Diao et al., 2023), while subsequent research performs aspect-based sentiment analysis to identify the fine-grained sentiments concerning the different aspects of some target entity (Severyn and Moschitti, 2015; Pontiki et al., 2016; Nazir et al., 2020; Brauwers and Frasincar, 2021).
1https://www.edelmandxi.com/trustintelligence/measuring-trust-prerequisite-unlocking-growth
Aspect-based sentiment analysis can be performed at the sentence-level or document-level. Sentence-level aspect-based sentiment analysis focuses on independently classifying the sentiments associated with aspects in individual sentences (Peng et al., 2020; Yan et al., 2021). However, this approach fails to consider the context of the aspect, which can often be inferred from preceding or succeeding sentences or paragraphs. In Figure 1, the sentiment expressed toward the aspect "Location" is not clear just by looking at sentence S21. By examining the surrounding sentences S1, S20 and S22, which are all positive, one could infer that the phrase "in the middle of the tourist section" has a positive sentiment, demonstrating the importance of context in aspect-based sentiment analysis at the document level. Further, sentences in the same document may express conflicting sentiments towards the same aspect. For example, sentence S17, S18 and S19 express a positive sentiment towards the aspect "Room", but S4 and S28 convey a negative sentiment. Simply classifying the overall sentiment based on a single sentence or taking the majority vote may led to incorrect conclusions.
23
In this work, we design a hierarchical Transformer-based architecture called DART that leverages multiple layers of attention mechanisms. This allows us to capture the dependencies among sentences in long documents and learn aspectspecific document representations. DART performs attention aggregation on the learned representations to take into account both the local and global contexts. By employing learnable global aspect queries, our model aggregates sentiments that reflects the overall sentiment of the document, even in the presence of conflicting sentiments.
We curate two datasets, one focusing on social issues and another on trust-related issues. Initial experiments indicate that even GPT-4 has difficulty dealing with implicit aspects and often misinterprets sentiment due to insufficient aspect knowledge. Comprehensive experiments show that DART achieves state-of-the-art accuracy for document-level aspect-based sentiment classification, and is also effective in predicting trust and polarity in long complex documents.

heading: 2 Related Work
text: Research on aspect-based sentiment analysis can be broadly classified into sentence level and document level. Sentence-level aspect-based sentiment analysis includes using Long Short-Term Memory (LSTM) network to model aspects in sentences (Tang et al., 2016), attention-based LSTM to correlate aspects and sentiment polarities (Wang et al., 2016; Ma et al., 2017; Tay et al., 2018), deep memory networks to integrate aspect information (Tang et al., 2016; Chen et al., 2017), and gated networks to select aspect-specific sentiment in sentences (Zhang et al., 2016; Xue and Li, 2018). (Chen et al., 2020) introduce graph attention networks to improve sentence prediction by incorporating sentiment preference information from the document context. The work in (Yan et al., 2021) propose a unified framework for fine-grained sentiment analysis to identify the aspect and opinion terms as well as its sentiment polarity for each sentence.
Document-level aspect-based sentiment analysis predicts the sentiment polarity for each aspect mentioned in a document. Traditional approaches have largely relied on feature engineering. Latent rating regression (LRR) (Wang et al., 2010) is a probabilistic graphical model that generates document sentiment representation from a weighted sum of the latent aspect variables. (Lu et al., 2011)
use support vector regression model based on handcrafted features to predict aspect ratings. To handle correlation between aspects, (McAuley et al., 2012) add a dependency term that explicitly encodes relationships between aspects. These methods have strict assumptions about words and sentences such as whether a word is an aspect or sentiment towards an aspect, and typically use bag-of-words representations which are insufficient to capture the order of words and complex semantics.
Neural network methods for document-level aspect sentiment analysis include N-DMSC (Yin et al., 2017), VWS-DMSC (Zeng et al., 2019) and D-MILN (Ji et al., 2020). N-DMSC employs hierarchical LSTM to create aspect-aware document representations using question-answer pairs constructed from aspect-related keywords and aspect ratings. VWS-DMSC uses a multi-task learning framework with rules to extract target-opinion word pairs to guide the sentiment prediction towards document aspects in a weakly supervised manner. D-MILN is a multiple instance learning network that models the relation between aspect-level and document-level sentiment with document-level supervision. (Fei et al., 2021) model the latent targetopinion distribution as prior information and employ a two-layer BiLSTM to obtain the overall document-level sentiment classification.
Transformer models have been utilized for aspect sentiment analysis (Fei et al., 2022; Islam and Bhattacharya, 2022). However, they are limited to processing sequences of up to 512 tokens. To overcome this limitation, models such as Longformer (Beltagy et al., 2020), Big Bird (Zaheer et al., 2020), Hi-Transformer (Wu et al., 2021) and LongT5 (Guo et al., 2022) have been introduced. However, these models have not yet been specifically utilized for aspect-based sentiment analysis.

heading: 3 Proposed Framework
text: The proposed DART framework takes as input a document d and an aspect aj and outputs the predicted sentiment for aj . Figure 2 shows the architecture of DART which consists of four key blocks:
Sentence Encoding Block. This block focuses on transforming the document into individual sentences and using a pretrained language model to generate representations for every sentence-aspect combination.
Global Context Interaction Block. This block employs dual transformer encoders to model interac-
tions among sentences and generate context-aware sentence embeddings. This is a crucial component of DART as it captures essential aspect-specific information across long-range dependencies. Aspect Aggregation Block. This block aggregates the contextually enriched sentence embedding to produce an aspect-specific representation of the entire document. Sentiment Classification Block. With the document representation obtained, this block leverages a two-layered Multilayer Perceptron (MLP) to predict the sentiment for the aspect.

heading: 3.1 Sentence Encoding Block
text: Initially, the input document is divided into M sentences, denoted as s1, s2, ..., sM . This is achieved using the sentence splitter from the Natural Language Toolkit2. Then we construct fixed length sequences seq1, seq2, ..., seqM , adding right paddings if needed. Each sequence seqi is given by:
seqi = [CLS] aj [SEP] si [SEP]
2nltk.org
where [CLS] and [SEP] are the special tokens to denote the sentence-level information and separator respectively. The sequence seqi is fed into a BERTbased pretrained model to generate the embedding
[eCLSi , e 1 i , e 2 i , ..., e L i ]
where eki is the k th token in seqi and L is the fixed length of the sequence.

heading: 3.2 Global Context Interaction Block
text: This block captures dependencies among sentences so that a sentence can be understood in the broader context of the entire document, thus increasing the accuracy of sentiment prediction for a specific aspect. It incorporates two transformer encoders which serve different purposes.
The first transformer encoder focuses on the inter-sentence relationships. It uses the [CLS] tokens which are condensed representations of their respective sentences, and apply self-attention to these tokens across all sentences. This allows the encoder to obtain the context information, and produce a set of contextually enriched [CLS] tokens,
eCLSi , 1 ≤ i ≤ M , each representing its sentence in the context of the whole document. Positional information is retained by adding the standard learnable position embeddings. The output from this transformer is hCLSi .
After capturing the context information, the second transformer encoder further refines each sentence’s representation. It takes the context-enriched hCLSi token from the first encoder and combines it with the original embeddings of the sentence tokens. The combined input [hCLSi , e 1 i , ..., e L i ] undergoes another round of self-attention, producing the enriched sentence representation [rCLSi , r 1 i , ..., r L i ] where each ri is influenced both by its original context and the broader document context.

heading: 3.3 Aspect Aggregation Block
text: This block plays a pivotal role in the DART framework by generating a unified document representation that captures the overall sentiment of a document concerning a specific aspect. It serves as a bridge between understanding individual sentences and comprehending the document as a whole, especially concerning a specific aspect. Given that sentiment towards an aspect can be scattered throughout a document, this block ensures that all these sentiments are appropriately aggregated.
The key idea of this block is obtain a aspectspecific representation through a two-level aggregation process. The first level weighs the importance of each token in the sentence concerning the aspect and the broader context by performing a local attentive pooling. The enriched sentence representation [rCLSi , r 1 i , ..., r L i ] from the global context interaction block undergoes a local aggregation process to obtain the output li:
li = α0r CLS i +
L∑
k=1
αkr k i (1)
where αk is the attention weight for the kth token, determined based on its relevance to the aspect and the overall sentence context.
The second level takes the aggregated representations li of each sentence and performs a global attentive pooling to determine how much attention each sentence should receive when forming the overall document representation d̂j with respect to the aspect aj . This aggregation is given by
d̂j =
M∑
i=1 exp(e1i f(li))∑M i′=1 exp(e 1 i f(li′ )) li (2)
where f(·) is a linear projection followed by the tanh function.
The weighting coefficients ensure that sentences more relevant to the aspect have a greater influence on the final document representation d̂j .

heading: 3.4 Sentiment Classification Block
text: This block is the final stage in the DART framework. The goal of this block is to utilize the aggregated document representation, which has been enriched with context and focused on a particular aspect, to predict the sentiment associated with that aspect. The final document representation d̂j is passed through the two-layer MLP to obtain the probability distribution for the positive or negative sentiment towards the aspect aj .

heading: 4 Performance Study
text: We implement DART in PyTorch1.13.0 and carry out experiments on the A100-SXM4 GPUs with 40 GB. We use the following datasets:
BeerAdvocate. This dataset contains reviews and ratings on predefined beer aspects: feel, look, smell, and taste, each rated on a scale of 1 to 5. The ratings are binarized into positive and negative sentiment.
TripAdvisor. This dataset consists of hotel reviews with ratings of 1 to 5 stars for aspects value, room, location, cleanliness, check in/front desk, service, and business. Again, these ratings are binarized.
SocialNews. We curate this dataset from news articles related to social issues from the PerSenT dataset (Bastan et al., 2020). We identify six implicit aspects, namely crime-justice, digital-online, economic issues, health, human rights, and work. A group of labelling experts was trained using educational guideline pack and a series of face to face sessions so that they have a clear understanding of the definition of aspects and sentiment. An expert benchmarking assessment was performed where 100 verified labels were assigned to each prospective annotators and those who reached a 70% agreement with experts were selected. Finally, three annotators are asked to assess the sentiment towards these aspects and we use the majority vote as the ground truth sentiment. The Kappa interannotator agreement is 93.14%.
Table 1 summarizes the characteristics of these datasets. DART utilizes the pre-trained model bigbird-roberta-base (Zaheer et al., 2020) in the Sentence Encoding Block. For the Global Context Interaction Block, the first Transformer en-
coder has 4 layers, while the second Transformer encoder has 2 layers. Both have 12 self-attention heads with a hidden size of 768. We use AdamW optimizer with a dropout rate of 0.1, and a batch size of 16. Each experiment is repeated 5 times and we report the average results on three datasets.

heading: 4.1 Comparative Study
text: We first compare DART with non-transformer aspect-based sentiment classification methods: LRR (Wang et al., 2010) is a probabilistic graphical regression model. Guided by the overall rating and the aspect keywords, LRR infers the latent ratings for each aspect. A high rating indicates positive sentiment towards the aspect in the document. VWS-DMSC (Zeng et al., 2019) is a weakly supervised model that predicts the sentiment with respect to an aspect. Target-opinion word pairs are extracted as supervision signal to learn the sentiment without using aspect polarity annotations. D-MILN (Ji et al., 2020) is also a weakly supervised model for document-level aspect sentiment classification. It employs multiple instance learning to learn the relation between aspect-level and document-level sentiment. N-DMSC (Yin et al., 2017) is a supervised neural model for document aspect sentiment classification. It employs hierarchical LSTM to generate aspectaware document representations.
Table 2 shows the average accuracy for BeerAdvocate and TripAdvisor. We see that DART outperforms all the methods by a large margin. Using deep embedding features yields better results compared to traditional ngram features in LRR. Unlike N-DMSC, VWS-DMSC and D-MILN, DART does not require a pre-defined set of aspect-related keywords, reducing the complexity and pre-processing requirement in real-world scenarios.
Next, we compare the performance of DART with transformer-based models on long documents: InstructABSA (Scaria et al., 2023) uses the 11Bparameter T5 model, with a maximum input sequence length of 512, for sentence-level aspectbased sentiment analysis. As such, we truncate the
input when the length of instruction prompts and document exceeds 512 tokens.
MDABSA (Van Thin et al., 2022) is a joint multitask architecture that aims to perform both aspect category detection and sentiment polarity classification tasks simultaneously.
Longformer (Beltagy et al., 2020) employs sliding windows to enable long-range coverage for long document modelling. We adapt Longformer for the document aspect sentiment classification task by first obtaining an aspect-aware document representation through feeding the aspect and document pair separated by the [SEP] token. The representation is then fed to a two-layer multi-layer perceptron to make sentiment prediction.
Big Bird (Zaheer et al., 2020) This is an encoderonly model that extends the sparse attention pattern with random attention for longer sequences. We adapt Bird Bird for sentiment classification in the same way as we have done for Longformer.
LongT5 (Guo et al., 2022) is the state-of-theart transformer architecture for long inputs. The original LongT5 is an encoder-decoder structure with a new transient attention mechanism (TGlobal), which mimics ETC’s local/global mechanism(Ainslie et al., 2020). Here, we leverage its encoder pre-trained weights and adapt it in the same way for fair comparison.
GPT4 (OpenAI, 2023) large language models (LLMs) have shown impressive results across various tasks. Here, we select gpt-4-0613 as the representative LLM and perform the experiments under zero-shot and few-shot settings. For GPT4-
fewshot, we adopt the prompt from (Scaria et al., 2023) to perform aspect sentiment predictions.
Table 3 shows the performance of Transformerbased models on the SocialNews dataset, with details of their accuracy in handling different aspects. The results indicate that DART excels in five key aspects, particularly in the digital-online and health aspects, and is the second best model for the economic issues aspect. This demonstrates DART’s ability to handle diverse and complex aspects. Appendix A provides a visualization of the learned document representations via t-sne.
Figure 3 shows the accuracy achieved in SocialNews for documents that exceed a certain length, as specified on the x-axis. The gap in performance between DART and other models widens as the document length surpasses the 1024-token threshold. DART continues to demonstrate superior performance even with extremely long documents, exceeding 2048 tokens in length. This indicates DART’s proficiency in analyzing larger documents, which is an important aspect in real-world sentiment analysis scenarios.
Table 4 shows the average accuracy and macro F1 scores on the long documents in Beer Advocate and TripAdvisor over 5 runs. We see that DART
achieves the best performance, with marked improvements over existing models. Similar gains is observed for the F1 scores, confirming DART’s effectiveness in dealing with long documents for sentiment classification.
InstructABSA, which achieved state-of-the-art on SemEval 2014, 15, and 16 datasets for aspect sentiment classification, and MDABSA both perform worse than DART. This indicates that the methodologies developed for sentence-level aspectbased sentiment analysis or short texts do not extend well to longer documents. The results also reveal that our more compact, specialized DART model, which contains 687 million parameters, exceeded the performance of GPT4.

heading: 4.2 Ablation Study
text: We examine the effect of the various components in DART on its performance. We implemented two variants: (a) w/o Int. where the interaction component is bypassed and the outputs from the Sentence Encoding Block is fed directly to the Aspect Aggregation Block; and (b) w/o Agg. where the aggregation component is omitted and the average of the [CLS] vectors is used as the document representation for sentiment prediction.
Table 5 shows the results. Compared to BeerAdvocate and TripAdvisor, we see a significant drop in the accuracy for SocialNews when the Sentence Interaction block is removed because 22.9% of the documents are longer than 512 on SocialNews. Similar reduction in accuracy is observed when we do not incorporate the Aspect Aggregation block. This demonstrates the importance of capturing the interaction among sentences in long documents as well as aggregating aspects locally and globally.

heading: 4.3 Case Studies
text: Here, we present case studies to show DART’s ability to highlight phrases relevant to the target aspects. Figure 4 shows an article from SocialNews related to the aspect HEALTH. Only DART correctly predicts the negative sentiment towards this aspect while both Big Bird and Longformer give a positive sentiment. Phrases in purple are highlighted by DART as the basis for its negative prediction. In contrast, Big Bird and Longformer could not adequately capture the context, leading them to overlook the underlying negative sentiment.
Figure 5 shows two sample reviews from TripAdvisor. For the top review, DART focuses on phrases related to VALUE (highlighted in red) and correctly predicts a positive sentiment towards the aspect VALUE while Big Bird and Longformer give the wrong predictions. For the bottom review, DART predicts the correct negative sentiment towards the aspect CLEAN, with relevant phrases highlighted in green. We see that although DART attends to the phrase “Overall room be clean daily”, it is able to identify negative phrases such as “exotic huge dead cockroach”, “dingy bed and blanket” and “The shower stall do not close” to be associated to the aspect CLEAN and gives the correct prediction. In contrast, Big Bird and Longformer mistakenly interpret the sentiment as positive.

heading: 5 Application of DART to Trust and Polarity Prediction
text: While DART is originally conceptualized for sentiment analysis, the framework is versatile and can
be extended for trust analysis and polarity prediction. In this section, we show that DART’s ability to capture context information and aspect-specific attention aggregation makes it well-suited to evaluate trust-related aspects and gauge the degree of alignment or opposition on a topic.
We compile a dataset for trust prediction, comprising of 2925 documents, of which 60.5% are long documents with more than 512 tokens. This dataset emphasizes four trust-related aspects: ability, dependability, integrity and purpose. We enlist three independent annotators to assess the trust polarity for each aspect, and take the majority vote as the ground truth. The annotation labels are "trust", "distrust", "mixed", and "no indication". The Kappa inter-annotator agreement is 87.29%. We call this dataset TrustData.
For polarity prediction, we use the Hyperpartisan dataset (Kiesel et al., 2019) consisting of news articles which have been manually labelled as hyperpartisan or not. There are 645 articles, out of which 53.3% have more than 512 tokens.
In Table 6, we see that DART gives the best accuracy and F1 score for trust and polarity predictions. The improvements achieved by DART over the second best model are statistically significant with p-value < 0.05, indicating the effectiveness of the global context interaction block in DART to capture the context information in long documents.
Table 7 provides a detailed breakdown of model accuracies in predicting the polarity of the different aspects in the TrustData dataset. We see that DART gives the best performance in three key aspects, and
is competitive with GPT4-zeroshot for the Dependability aspect, possibly due to the fewer number of documents with this aspect. The improvements suggest that the aspect-specific attention aggregation block in DART significantly enhances its ability to focus on phrases relevant to the various aspects.

heading: 6 Conclusion
text: We have described DART, a hierarchical transformer-based framework for document-level aspect-based sentiment analysis. DART handles the complexities of longer text through its global
context interaction and two-level aspect aggregation blocks, which enhance the model’s ability to recognize and amplify aspect-specific content across long-range dependencies. This enables DART to focus on relevant phrases associated with the target aspect. Experiments on various datasets indicate DART’s effectiveness in handling long documents. We have also shown the applicability of DART for trust and polarity prediction and will make the curated SocialNews dataset publicly available. Future work includes extending DART’s capabilities to handle aspect-based sentiment analysis involving multiple entities.

heading: Acknowledgments
text: This work is supported by the Singapore Ministry of Education Academic Research Fund Tier 1. We would like to thank Katie Stuart for her comments.
Limitations
This work assumes each document contains only one entity. There is a need to develop a benchmark that can assess aspect based sentiments towards different entities in long documents.

heading: Future Work
text: Future work includes extending DART’s capabilities to handle aspect-based sentiment analysis involving multiple entities