LLaMA-Based Models for Aspect-Based Sentiment Analysis (2024)

Abstract: While large language models (LLMs) show promise for various tasks, their performance in compound aspect-based sentiment analysis (ABSA) tasks lags behind fine-tuned models. However, the potential of LLMs fine-tuned for ABSA remains unexplored. This paper examines the capabilities of open-source LLMs finetuned for ABSA, focusing on LLaMA-based models. We evaluate the performance across four tasks and eight English datasets, finding that the fine-tuned Orca 2 model surpasses stateof-the-art results in all tasks. However, all models struggle in zero-shot and few-shot scenarios compared to fully fine-tuned ones. Additionally, we conduct error analysis to identify challenges faced by fine-tuned models.

heading: 1 Introduction
text: Aspect-based sentiment analysis (ABSA) aims to extract detailed sentiment information from text (Zhang et al., 2022). ABSA includes four sentiment elements: aspect term (a), aspect category (c), opinion term (o), and sentiment polarity (p). Given the example review “The steak was delicious”, the elements are “steak”, “food quality”, “delicious” and “positive”, respectively.
Initially, ABSA research focused on extracting individual sentiment elements, e.g. aspect term extraction or aspect category detection (Pontiki et al., 2014). Recent research has transitioned towards compound tasks involving multiple sentiment elements, such as aspect sentiment triplet extraction (ASTE) (Peng et al., 2020), target aspect category detection (TASD) (Wan et al., 2020), aspect category opinion sentiment (ACOS) (Cai et al., 2021), and aspect sentiment quad prediction (ASQP) (Zhang et al., 2021a). Table 1 shows the output formats of these ABSA tasks.
Modern ABSA research often utilizes pretrained language models, mainly focusing on sequence-to-sequence models. Compound ABSA
tasks are typically formulated as text generation problems (Zhang et al., 2021b,a; Gao et al., 2022; Hu et al., 2022; Gou et al., 2023), which allows to solve compound ABSA tasks simultaneously.
Lately, large language models (LLMs), such as ChatGPT (OpenAI, 2022), LLaMA 2 (Touvron et al., 2023b) and Orca 2 (Mitra et al., 2023), have made significant progress across various natural language processing tasks. However, more traditional approaches that fine-tune Transformer-based models with sufficient data have shown superior performance over ChatGPT in compound ABSA tasks (Zhang et al., 2023; Gou et al., 2023). Additionally, fine-tuning LLMs on a single GPU is challenging due to their large number of parameters. Techniques like QLoRA (Dettmers et al., 2023) address this challenge using a quantized 4-bit frozen backbone LLM with a small set of learnable LoRA weights (Hu et al., 2021). However, studies have yet to explore the capabilities of fine-tuned opensource LLMs for ABSA.
This paper examines the unexplored potential of LLaMA-based models fine-tuned for English ABSA alongside their performance in zero-shot and few-shot scenarios. Our key contributions include: 1) Introducing the capabilities of fine-tuned LLaMA-based models for ABSA. 2) Conducting a comparative analysis of two LLaMA-based models against state-of-the-art results across four ABSA tasks and eight datasets. 3) Evaluating models’ performance in zero-shot, few-shot, and fine-tuning scenarios, demonstrating the superior performance
63
of the fine-tuned Orca 2 model, surpassing stateof-the-art results across all datasets and tasks. 4) Presenting error analysis of the top-performing model.1

heading: 2 Related Work
text: Early ABSA studies focused on predicting one or two sentiment elements (Liu et al., 2015; Zhou et al., 2015; He et al., 2019; Cai et al., 2020) before progressing to more complex tasks involving triplets and quadruplets, such as ASTE (Peng et al., 2020), TASD (Wan et al., 2020), ASQP (Zhang et al., 2021a) and ACOS (Cai et al., 2021).
Recent ABSA research focuses primarily on text generation initiated by GAS (Zhang et al., 2021b). PARAPHRASE (Zhang et al., 2021a) converts labels to natural language. LEGO-ABSA (Gao et al., 2022) explores multi-tasking, DLO (Hu et al., 2022) optimizes element ordering, MVP (Gou et al., 2023) combines differently ordered outputs, and Scaria et al. (2023) adopt instruction tuning.
Gou et al. (2023) and Zhang et al. (2023) show that ChatGPT struggles with compound ABSA tasks in zero-shot and few-shot settings. Simmering and Huoviala (2023) report promising results with close-source LLMs for a single simple ABSA task.

heading: 3 Experimental Setup
text: We employ the 7B and 13B versions of LLaMA 2 (Touvron et al., 2023b) and Orca 2 (Mitra et al., 2023) models from the HuggingFace Transformers library2 (Wolf et al., 2020). LLaMA 2 offers models of various sizes tailored for dialogue tasks, building upon the LLaMA framework (Touvron et al., 2023a). Orca 2 extends this collection with enhanced reasoning capabilities.

heading: 3.1 Experimental Details
text: For fine-tuning, we follow recommendations from Dettmers et al. (2023) and use QLoRA with the following settings: 4-bit NormalFloat (NF4) with double quantization and bf16 computation datatype, batch size of 16, constant learning rate of 2e-4, AdamW optimizer (Loshchilov and Hutter, 2019), LoRA adapters (Hu et al., 2021) on all linear Transformer block layers, and LoRA r = 64 and α = 16.
1Code and datasets are available at https://github.com/ biba10/LLaMA-ABSA.
2https://github.com/huggingface/transformers
We fine-tune the models for up to 5 epochs and choose the best-performing model based on validation loss. Following Mitra et al. (2023), we compute loss only on tokens generated by the model, excluding the prompt with instructions.
For zero-shot and few-shot experiments, we use 4-bit quantization of the models. Preliminary experiments indicated that 4-bit quantized models performed similarly to 8-bit quantized models and non-quantized models.
All experiments, including zero-shot and fewshot scenarios, employ greedy search decoding and are conducted on an NVIDIA A40 with 48 GB GPU memory.

heading: 3.2 Evaluation Metrics
text: We use micro F1-score as the primary evaluation metric, chosen based on related work, and report average results from 5 runs with different seeds. We consider a predicted sentiment tuple correct only if all its elements exactly match the gold tuple.

heading: 3.3 Tasks & Datasets
text: We evaluate the LLMs on four tasks: two involving quadruplets (ASQP and ACOS) and two involving triplets (TASD and ASTE). We select two datasets for each task and use the same data splits as previous works for a fair comparison. Table 1 displays the output targets for each task.
We use Rest15 and Rest16 datasets for ASQP in the restaurant domain, initially introduced in SemEval tasks (Pontiki et al., 2015, 2016), later aligned and supplemented by Zhang et al. (2021a). For ACOS, we employ ACOS-Rest and ACOSLap datasets from Cai et al. (2021), focusing on implicit aspects and opinions and providing comprehensive evaluation. We use the dataset from Xu et al. (2020) and Wan et al. (2020) for ASTE and TASD, respectively. Table 2 shows the detailed data statistics. ASTE datasets are the only ones that do not include implicit sentiment elements.

heading: 3.4 Prompting Strategy & Fine-Tuning
text: LLMs show varied responses despite similar prompts (Perez et al., 2021; Lu et al., 2022). Our goal is to design simple, clear, and straightforward prompts to standardize evaluations across datasets and ensure consistent assessment of LLMs.
Our prompts define sentiment elements and output format. Sentiment element definitions include the permitted label space, e.g. allowed sentiment
polarities and aspect categories. The output format describes the expected structure of model responses, allowing us to decode the responses into our desired format. We supplement the prompts with the first ten training examples for a given task for few-shot learning. We use the same prompts for fine-tuning as for zero-shot experiments. Figure 1 illustrates a prompt for quadruplet tasks. Appendix A presents the prompts for the triplet tasks.
During the fine-tuning experiments, we train the model to generate the output in the desired format, as shown in Figure 1.

heading: 4 Results
text: Table 3 shows the results of LLaMA-based models. The results demonstrate the remarkable potential of Orca 2, especially in its 13B version, which sur-
passes previous benchmarks across all four tasks and eight datasets. Notably, the TASD task shows the most significant improvement, with 6% and 8% enhancements for the Rest15 and Rest16 datasets, respectively. While improvements for other tasks are relatively smaller, they remain noteworthy. There are marginal enhancements, within 1%, for the ASQP and ASTE tasks and the ACOS-Lap dataset. However, the ACOS-Rest dataset sees a significant improvement exceeding 4%, indicating notable progress. The remarkable advancements in the TASD task suggest that predicting opinion terms not included in the TASD task presents the most significant challenge for these models. The larger Orca 2 achieves a substantial improvement of 2.87% on average.
The 7B version of Orca 2 performs similarly to
the state-of-the-art (SOTA) for most tasks. However, it falls behind by over 2% in the Rest15 dataset and ASTE task. Nonetheless, it notably exceeds previous SOTA results for the TASD task by 3–5%, highlighting the challenge of predicting opinion terms absent in the TASD task. Nevertheless, the smaller Orca 2 performs almost 1% better on average than the previous best results.
Orca 2 significantly outperforms LLaMA 2, with the smaller Orca 2 model even surpassing the larger LLaMA 2 model, underscoring the superior reasoning capabilities of Orca 2. Additionally, it suggests that opting for more advanced but smaller models may be more beneficial than using larger models with less sophistication. The TASD task is the only task LLaMA 2 outperforms previous SOTA results. Compared to previous SOTA results, on average, the larger version is more than 2% worse, and the smaller version is 5% worse.
In zero-shot and few-shot scenarios, both evaluated LLaMA-based models exhibit notably inferior performance compared to their fine-tuned counterparts, particularly in quadruplet tasks. ChatGPT, with significantly more parameters, notably outperforms these models across zero-shot and few-shot scenarios. However, ChatGPT notably underperforms compared to fine-tuned models.

heading: 4.1 Error Analysis
text: To gain insights into the challenges of sentiment prediction, we conduct an error analysis focusing on identifying the most difficult sentiment elements to predict. We manually investigate predictions of 100 random test samples from the best-performing run of Orca 2 with 13B parameters for each dataset. Figure 2 depicts the results of the error analysis.
In most cases, the most challenging element to predict is the opinion term, often comprising multiple words. The model frequently struggles to predict the text span precisely, for instance, predicting “mild” instead of “too mild”. Following closely in difficulty is typically the aspect term, which encounters similar mistakes as opinion terms, but aspect terms are more often just one word, making such errors less frequent. Sentiment polarity proves to be the easiest to predict. However, an exception arises in the ACOS-Lap dataset, where the aspect category emerges as the most challenging due to the extensive category variety of the dataset (81 categories in the test set, compared to only 12 in the restaurant datasets).
The model also occasionally confuses semantically similar aspect categories, such as “restaurant general” with “restaurant miscellaneous” or “keyboard usability” with “keyboard general”.
The most common error considering sentiment polarity is in predicting the “neutral” class, possibly due to imbalanced label distribution, since the “neutral” class is the least frequent in all datasets. In contrast to observations made by Zhang et al.
(2021a), we did not encounter errors related to text generation, such as generating words for aspect or opinion terms that are absent in the original text.
Additionally, we identified mistakes in the dataset labels. For example, in the ACOS-Rest dataset, the aspect “service” in the sentence “worst service i ever had” is labelled as “positive”, despite being clearly “negative”, a prediction the model also makes correctly. Similarly, we noticed inconsistencies in the datasets, such as in the sentence “One of the best hot dogs I have ever eaten”, where the expression “hot dogs” is not labelled as an aspect term for the “food quality” category; instead, it is labelled as an implicit aspect term (“NULL”), contrary to other examples. These labelling errors could potentially negatively impact the final scores of evaluated models.

heading: 5 Conclusion
text: This paper presents a comprehensive evaluation of LLaMA-based models for compound ABSA tasks. We show that these models underperform in zeroshot and few-shot scenarios compared to smaller models fine-tuned specifically for ABSA. However, we demonstrate that fine-tuning the LLaMA-based models for ABSA significantly improves their performance, and the best model outperforms previous state-of-the-art results on all eight datasets and four tasks. Error analysis reveals that predicting opinion terms is generally the most challenging for the evaluated models.

heading: Acknowledgements
text: This work has been partly supported by the OP JAC project DigiTech no. CZ.02.01.01/00/23_021/0008402 and by the Grant No. SGS-2022-016 Advanced methods of data processing and analysis. Computational resources were provided by the e-INFRA CZ project (ID:90254), supported by the Ministry of Education, Youth and Sports of the Czech Republic.
Limitations
Results highlight LLaMA-based models’ ineffectiveness in compound ABSA tasks in zero-shot and few-shot scenarios. Additionally, their performance in non-English languages remains unclear. Future work could also consider other open-source models based on a different architecture.
Ethics Statement
We experiment with well-known datasets used in prior scientific studies, ensuring fair and honest analysis while conducting our work ethically and without harming anybody.

heading: A Prompts
text: Figure 3 shows the prompt for the TASD task, while Figure 4 presents the prompts for the ASTE task. The prompts are also available in our code.

heading: Future Work
text: Future work could also consider other open-source models based on a different architecture