Venue: NAACL 2025

# MATO: A Model-Agnostic Training Optimization

for Aspect Sentiment Triplet Extraction

Shaopeng Tang1, Lin Li1,* Xiaohui Tao2, Leqi Zhong1, Qing Xie1

###### Abstract

As an important fine-grained sentiment analysis task, aspect sentiment triplet extraction (ASTE) aims to identify three elements, i.e., aspect, opinion and sentiment polarity as a triplet. Advanced ASTE researches have mostly explored triplet-wise ability to achieve superior improvement. However, existing models with strong in-house performances may struggle to generalize to the challenging cases with the diverse expression of inter-triplet and intra-triplet elements. To this end, we propose a **M**odel-**A**gnostic **T**raining **O**ptimization (**MATO**) to improve ASTE model inference consistent with expected results facing triplet element diversity. Specifically, to indicate the capacity to accommodate the diverse elements, we design interrupted and intra-triplet metamorphic relations (MRs), and calculate the violation rate (VR) on each element of one triplet through metamorphic testing (MT). Moreover, we propose an element-wise diversity-aware loss based on the VRs of aspect, opinion and sentiment, which can be jointly trained with existed ASTE models via uncertainty weighing. Conducted on four benchmark datasets and seven ASTE models, experimental results show that our MATO can enhance their diversity capacity, decreasing the average element-wise VMs by 3.28% to 15.36%. Meanwhile, our MATO is comparable to or better than those in terms of F1-score.

1Wuhan University of Technology, China, 2University of Southern Queensland, Australia

{karitown, cathylilin, zlq_lucky, felixxq}@whut.edu.cn, xiaohuti.tao@unisq.edu.au

## 1 Introduction

Aspect sentiment triplet extraction (ASTE) aims to identify three elements, i.e., aspect term, opinion term and sentiment polarity as a triplet. As in the example "The sound is nice and loud; I do n't have any problems with hearing anything." in Fig. 1 (c), its goal is to extract two triplets "(sound, nice, POS)" and "(sound, loud, POS)".

Many approaches to ASTE have been proposed successively. Peng et al. [2020] introduced the ASTE task at the first time and provided a two-stage framework in a pipeline approach to accomplish the extraction of aspect term and opinion term successively as well as the classification of sentiment. To overcome the error propagation problem, subsequent works adopted table filling representation to jointly model the ASTE task Wu et al. [2020b, Chen et al., 2022b, Zhang et al., 2022, Sun et al., 2024]. Some works used sequence tagging to enrich label representation to enhance representation learning Xu et al. [2020, 2021]; Liang et al. [2023]; Li et al. [2023]. Besides, there were some studies that tried to convert the ASTE task into a machine reading comprehension (_MRC_) task Mao et al. [2021]; Chen et al. [2021a, Zhai et al., 2022]. Recently, generative model has gained significant results on many tasks, and equally some works have addressed the ASTE task with generative manner Zhang et al. [2021a, Zhou and Qian, 2023].

The above methods explored the ability to prove the model's feature representation, feature learning and the ability to inference etc., and have obtained superior performance. However, they may struggle to generalize to the challenging cases with the diverse expression of inter-triplet and intra-triplet elements. As shown in Fig. 1 (a) and (b), the extraction of the triplet "(OS, happy, POS)" is affected just by changing the other triplet's opinion, which indicates that the resistance of ASTE models

Figure 1: Two sets of examples for some ASTE models. (a) and (c) are from LAP14 while inter-triplet and intra-triplet diverse expressions appear in (b) and (d), respectively. The underlined indicates the distinctions.



to **inter-triplet** diverse expression is not powerful enough. Similarly, as shown in Figs. 1 (c) and (d), simply making a synonym transformation 1 for the opinion of the triplet "(sound, nice, POS)" leads to an inversion of the output sentiment, which suggests that the perceptual ability of ASTE models to **intra-triplet** diverse expression is also insufficient.

Footnote 1: According to the Oxford English Dictionary ([https://www.oed.com](https://www.oed.com)), scrummy means excellent, marvelous, enjoyable and delicious etc., representing nonstandard London speech.

To address the aforementioned problem, we propose a **M**odel-**A**gnostic **T**raining **O**ptimization (**MATO**) to improve an ASTE model inference consistent with expected results facing diversity. Specifically, we firstly design **inter-triplet and intra-triplet metamorphic relations (MRs)** from the perspective of potential causes affecting the extraction result of one triplet. Based on these MRs, we introduce a metric for assessing the diversity and use it for training optimization. **The violation rate (VR) conducted with metamorphic testing (MT)**, can be calculated by comparing the target triplets between their originals and metamorphosis to check whether it follows the MR. For example, we can establish an inter-triplet MR based on (a) and (b) in Fig. 1. Return to MRs, we can analyze the element (i.e., aspect, opinion, and sentiment) of the diverse expression. And thus statistically obtain the VR on each element, indicating the capacity to diversity of triplet elements. And the paired Wilcoxon signed rank tests [23] are performed on the MT results to ensure that the MRs are highly qualified.

Secondly, in order to make an ASTE model more focused on the triplet itself and shield from other triplets when generating triplets, we introduce three discerners to identify aspect, opinion, and sentiment, and propose **an element-wise diversity-aware loss based on VRs**. In particular, we sum three losses from the three discerners with the weights from the element-wise VMs. Finally, to better simultaneously learn ASTE triplet extraction and element-wise diversity awareness, an uncertainty-based weighting is applied to jointly train **diversity-aware loss and ASTE loss**, that is, MATO can work with most ASTE models.

Extensive experiments are conducted with seven ASTE SOTA models on four benchmark datasets. Our finding is that the capacity of the seven models facing inter-triplet and intra-triplet diverse expression remains significantly weak. The addition of our MATO to those ASTE SOTA models can enhance their capacity, decreasing the average element-wise VMs by 3.28% to 15.36%. Meanwhile, our MATO is comparable to or better than those in terms of F1-score.

## 2 Related Work

### ASTE Models

Aspecent Sentiment Triplet Extraction (ASTE) is a typical task in current research of aspect-based sentiment analysis (ABSA), proposed by Peng et al. (2020). ABSA is a traditional fine-grained sentiment analysis [49, 7, 18, 19, 20, 17, 21]. The early work of ABSA involved three basic tasks, including aspect term extraction [18, 19, 20, 21, 22], opinion term extraction [20, 21] and aspect-level sentiment classification [23, 24, 25, 26].

Recent ASTE studies consider the integrity among the three elements and can be classified into five streams, that is, pipeline [20], table filling [20, 21, 22, 23, 24], sequence tagging [21, 22, 23], MRC-based [20, 21, 22, 23] and generative manner [21, 22, 23, 24, 25]. These methods explored the ability to improve the model's feature representation, feature learning and the ability to inference etc., and have obtained superior performance. But few focuses enough on the element-wise diversity capacity.

### Metamorphic Testing in NLP

In software engineering, metamorphic testing (MT) is the process of testing a program by examining the metamorphic relation (MR) between the results of multiple executions of the program to find and correct defects and errors in the software [21]. MR is task-specific, and many works designed specific MRs for different tasks. Jiang et al. (2021) identified six types of MRs for the machine translation task, covering a wide range of properties that most NLI tasks are expected to have. The experimental results could explain the capabilities 

of the NLI model in different dimensions. Manino et al. (2022) proposed three MRs, which addressed the properties of systematicity, compositionality and transitivity. Manino et al. (2022) tested the internal consistency of state-of-the-art NLP models, and they did not always behave according to their expected linguistic properties. Hyun et al. (2024) proposed a framework with MT for analyzing large language model to address the limited coverage of quality attributes. Pu et al. (2023) adopted MT to evaluate the robustness of hand pose estimation models and provided suggestions on the choice of HPE models for different applications. Recent studies have found that the **property-based** validation method (such as violation rate based on MT) is more flexible than the traditional **reference-based** validation method (precision, recall and F1-score etc.) in revealing the actual language understanding capability of the NLP models Chen et al. (2021); Aleti (2023); Wang et al. (2024).

To our literature review, our work is the first to consider ASTE models enhanced with MT. The major challenge is to design suitable MRs to reflect inter-triplet and intra-triplet diverse expressions and the violation rate based on MT can be used for training optimization.

## 3 Proposed Method

In this section, we present our MATO in details as shown in Fig. 2. ASTE model can generate the hidden representations for aspect, opinion and sentiment through feature representation, feature learning and inference stages. ASTE model computes ASTE loss by comparing the labels. We use MT to indicate the capacity to diversity of ASTE model and introduce diversity-aware loss to enhance the perception ability. Finally, an uncertainty weighing is applied to jointly train diversity-aware loss and ASTE loss. This process does not depend on the specific ASTE model (i.e., model-agnostic).

### Task Description

Given a sentence \(X=\{w_{1},w_{2},\ldots,w_{n}\}\) with \(n\) words, the goal of the ASTE model is to output all triplets \(\mathcal{T}=\{(a,o,s)_{i}\}^{m}_{i=1}\) in the sentence, where \(a\) and \(o\) denote aspect term and opinion term respectively, and they both come from the sentence \(X\). The sentiment polarity \(s\) belongs to the label set \(S=\{POS,NEU,NEG\}\), and \(m\) is the number of triplets in the sentence.

Figure 2: The overview of our MATO, a model-agnostic working with ASTE models





### Metamorphic Relations Design

The current mainstream ASTE models suffer from the problem of weak capacity when facing element-wise diverse expression, while MT based on task-specific MRs are able to cover such linguistic property. Therefore, we regard to design appropriate MRs to alleviate this diversity problem. As shown in the example in Table 1, we design inter-triplet and intra-triplet MRs from the viewpoint of possibly causing an unexpected change in the output for one triplet (denoted as target triplet).

#### 3.2.1 Inter-triplet MRs

The expression of triplets aside from the target triplet is individual to individual, i.e. there is inter-triplet diverse expression. The output result of target triplet may be affected by external diversity, and an ASTE model with diversity capacity should avoid such cases. We principally consider the influence of other triplets on target triplet and design inter-triplet MRs, and then determine if there is a violation against one MR by comparing the consistency of the real output with the expected output.

**MR1-1:** According to the relative independence among triplets, the synonym transformation 2 to the opinion of other triplets in the sentence should not affect the output of the target triplet, i.e., its expected output is consistent to its original output.

Footnote 2: The synonyms and aneurysms in this work are derived from NLTK ([https://www.nltk.org](https://www.nltk.org)) and an online dictionary ([https://github.com/meetDevveloper/freeDictionaryAPI](https://github.com/meetDevveloper/freeDictionaryAPI)).

**MR1-2:** When the sentiment of other triplets in the sentence is inverted (i.e., the opinion undergoes an funstorm transformation), the extraction of the target triplet should not be affected, i.e., its expected output is consistent to its original output.

Example for MR1-2:

Source Input: The screen is bright and the keyboard is nice.

Follow-up Input: The screen is bright and the keyboard is nice.

**MR1-3:** In order to bring in more factors that may affect the target triplet extraction results, we add to the sentences some phrases 3 consisting of the triplet with the opposite sentiment of the target triplet. This should not affect the extraction of the target triplet, i.e., its expected output is consistent to its original output.

Footnote 3: The generation of phrases from triplets in the datasets are automatically implemented with Parrot ([https://github.com/PrithivirajDemodaran/Parrot_Paraphraser](https://github.com/PrithivirajDemodaran/Parrot_Paraphraser)).

**MR1-4:** For a further analysis of the impact of other triplets on target triplet extraction in a sentence, we substitute aspect term and opinion term involved in other triplets with "[UNK]" to mask the semantic information brought by the other triplets. The expected output of the target triplet should be consistent to its original output without this part of the semantic information.

Example for MR1-4:

Source Input: The screen is bright and the keyboard is nice.

Follow-up Input: The screen is bright and the UNKG is UNKG.

#### 3.2.2 Intra-triplet MRs

ASTE models with diversity capacity should not only be able to extract target triplet easing the influence of other triplets, but more importantly focus on target triplet own diverse information. For such considerations, we check whether the ASTE model is able to respond correctly to its own changes by introducing diversity to the target triplet.

**MR2-1:** The sentiment is dependent, so we first consider making some changes to the holder of the sentiment, i.e., making a synonym/hypernym transformation 4 to aspect. The aspect in the target triplet output should change accordingly, the rest should be consistent.

Example for MR2-1:

Source Input: The screen is bright and the keyboard is nice.

Follow-up Input: The register is bright and the keyboard is nice.

**MR2-2:** In natural language, there are a variety of expressions that convey the approximate meaning. The model needs to be able to maintain a diversity-aware performance facing different opinon expressions, and understand the semantic information in the text. We apply a synonym transformation to the opinion of the target triplet. The opinion in the target triplet output should change accordingly, the rest should be consistent.





**MR2-3:** It is not sufficient to transform either aspect or opinion alone, so we consider synonym/hypernym transformations for both at the same time. The aspect and opinion in the target triplet output should change accordingly, the sentiment should be consistent.

Example for MR2-3: Source Input: The screen is bright and the keyword is nice. Follow-up Input: The emitter is client and the keyword is nice.

**MR2-4:** Antonym transformation leads to changes in the meaning of the source input, and the model needs to understand the new semantics and accurately capture the sentiment polarity corresponding to the opinion. The opinion in the target triplet output should change accordingly, the aspect should be consistent. Moreover, The sentiment should be reversed.

Example for MR2-4: Source Input: The screen is bright and the keyword is nice. Follow-up Input: The screen is unclear and the keyword is nice. We can generate a large amount of data to test the model based on the above MRs. According to the comparison of the output of the target triplet and the expected output, we can calculate the corresponding VRs: \(\{VR_{1},VR_{2},\ldots,VR_{8}\}\), indicating the capacity of the ASTE model facing inter-triplet and intra-triplet diverse expression.

### Diversity Awareness

Given an input sentence \(X=\{w_{1},w_{2},\dots,w_{n}\}\) with n tokens and the triplets \(\mathcal{T}=\{(a,o,s)_{i}\}^{m}_{i=1}\). The last hidden layer representation sequence for aspect, opinion and sentiment in the ASTE model are denoted separately as:

\[H_{l_{i}}=\{h_{1},h_{2},\dots,h_{l_{i}}\},i\in\{1,2,3\} \tag{1}\]

where \(l_{1},l_{2}\) and \(l_{3}\) denote the length of the last hidden layer representation sequence for aspect, opinion and sentiment, respectively. We acquire the hidden representations of which represent aspect, opinion and sentiment:

\[H_{e}=\{h_{e_{1}},h_{e_{2}},\dots,h_{e_{m}}\},e\in\{a,o,s\} \tag{2}\]

To make the ASTE model more focus on the above representations, we introduce three discerners (i.e., three linear layers) and sigmoid activation function to identify aspect, opinion and sentiment:

\[\hat{y}_{e}=Sigmoid(discerner_{e}(H_{e})),\\ e\in\{a,o,s\} \tag{3}\]

From these, three identification binary cross-entropy (BCE) losses can be obtained:

\[Loss_{e}=BCE(y_{e},\hat{y}_{e}),e\in\{a,o,s\} \tag{4}\]

where the shape of \(y_{e}\) is the same with \(\hat{y}_{e}\) and its values are fully 1. We can summarize to derive VMs on aspect, opinion and sentiment:

\[VR_{e}=\frac{1}{|map_{e}|}\sum_{i\in map_{e}}VR_{i},e\in\{a,o,s\} \tag{5}\]

where \(|map_{e}|\) denotes the length of \(map_{e}\). \(map_{a}\) is \(\{3,4,5,7\}\), \(map_{o}\) is \(\{1,2,3,4,6,7,8\}\) and \(map_{s}\) is \(\{2,3,4,8\}\) based on the type of the diverse element in the above MR

So far, we can obtain the diversity-aware loss:

\[Loss_{aware}=\sum_{e\in\{a,o,s\}}VR_{e}Loss_{e} \tag{6}\]

Diversity-aware loss makes the ASTE model per-cieve element-wise diverse expression.

### Model-Agnostic Autoweighted Training

Let \(f^{W}(X)\) be the output of the ASTE model with weights \(W\) on input \(X\). The loss of the ASTE model can be uniformly formulated as \(Loss_{aste}\).

Diversity-aware loss is model-agnostic and can be jointly trained with \(Loss_{aste}\). Kendall et al. (2018) proposed to use uncertainty to weigh multiple losses to achieve multi-task learning for scene geometry and semantics. According to the proof of Kendall et al. (2018). we can approximate the overall loss for ASTE task:

\[Loss_{overall}\approx \frac{1}{\sigma_{1}^{2}}Loss_{aste}+\frac{1}{\sigma_{2}^{2}} Loss_{aware}\] \[+log\sigma_{1}+log\sigma_{2} \tag{7}\]

where the positive scalars \(\sigma_{1}\) and \(\sigma_{2}\) can be learnt from triplet extraction output and diversity-aware output distributions, respectively, where the parameter determines how "uniform" (flat) the discrete distribution is. This relates to its uncertainty, as measured in entropy (Kendall et al., 2018).

Later, Liebel and Korner (2018) have improved to avoid the loss value becoming negative during training. In practical, the unified training of ASTE triplet extraction and triplet focusing for multiple objectives is based on AutomaticWeightedLoss 5.

Footnote 5: [https://github.com/Mikoto10032/AutomaticWeightedLoss](https://github.com/Mikoto10032/AutomaticWeightedLoss)

## 4 Experiments

This paper aims to address three research questions.

**RQ1**: Is the traditional reference-based validation method (i.g., F1-score) effective enough in revealing the diversity capacity of the ASTE model?

**RQ2**: Is the property-based validation method (i.e., VR) effective?

**RQ3**: Does our MATO significantly improve the capacity of the ASTE model when facing inter-triplet and intra-triplet diverse expression?

We conduct a thorough assessment of the quality of our designed MRs following Chen et al. (2021b), the results of which are shown in Appendix A.3.

### Datasets and Baselines

We conducted extensive experiments on four ASTE benchmark datasets refined by Xu et al. (2020). All datasets are sourced from SemEval Challenges (Pontiki et al., 2014, 2015, 2016), including reviews in the laptop and restaurant domains. The statistics of the four datasets are shown in Table 2.

For analyzing the capacity of various types of models in solving ASTE task as well as validating the effectiveness of our MATO, we select seven SOTA models as our baselines. (1) **Table filling**: EMCGCN (Chen et al., 2022b), BDTF (Zhang et al., 2022), MiniConGTS (Sun et al., 2024). (2) **Sequence tagging**: STAGE (Liang et al., 2023), SimSTAR (Li et al., 2023). (3) **MRC-based**: COM-MRC (Zhai et al., 2022). (4) **Generative manner**: SLGM (Zhou and Qian, 2023). Detailed descriptions for these baselines can be found in the Appendix A.1.

### Experimental Settings

We run the above seven models corresponding to the parameter settings in their papers. For each synonym/antonym/hypernym transformation, we obtain at most 10 synonyms/antonys/hypernyms from NLTK or the online dictionary. To minimize the influence of randomness, we ran run each model 5 times with different random initializations and then report the average results. All experiments are conducted on an NVIDIA TITAN XP GPU.

### RQ1: Overall Performance with F1-score

Table 3 lists the experimental comparisons of the seven models before and after the introduction of MATO. MATO can improve the F1-score (green background) of the six models on the four datasets, and slightly reduce the F1-score (red background) of the COM-MRC, which suggests that our MATO can improve the traditional performance of most of ASTE models. Another interesting finding is that the addition of MATO is able to reduce the gap between precision and recall while keeping the performance of the F1-score stable.

\begin{table}
\begin{tabular}{r r r r r r r} \hline \hline \multirow{2}{*}{Dataset} & \multicolumn{2}{c}{Train} & \multicolumn{2}{c}{Dev} & \multicolumn{2}{c}{Test} \\ \cline{2-9}  & \#S & \#T & \#S & \#T & \#S & \#T \\ \hline LAP14 & 906 & 1460 & 219 & 346 & 328 & 543 \\ RES14 & 1266 & 2338 & 310 & 577 & 492 & 994 \\ RES15 & 605 & 1013 & 148 & 249 & 322 & 485 \\ RES16 & 857 & 1394 & 210 & 339 & 326 & 514 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Statistics of four ASTE datasets (Xu et al., 2020). #S and #T represent the number of sentences and triplets, respectively.

 

In order to analyze whether it is appropriate to use the traditional reference-based metric (e.g., F1-score) directly to evaluate the diversity capacity of the model, we discuss the relation between VR and F1-score. Fig. 3 shows the average VR and F1-score of each model on LAP14, and it is not difficult to find out that the VR does not always decrease with the growth of F1-score, which suggests that it is inappropriate to use F1-score directly to assess the capacity of the ASTE model.

### RQ2: Diversity Capacity Comparison between F1-score and VR

To ensure the effectiveness of our designed MRs, evaluation for MRs is essential. Chen et al. (2021b) introduced multiple mutants to MRC models to evaluate MRs. They compared whether there was a significant difference (i.e., **better**, **similar**, or **worse**) between two VRs with the paired Wilcoxon signed-rank test. One VR was calculated from the output of the samples which the ASTE model without any mutants can pass under F1-score before and after the introduction of mutants. The other VR was calculated from the output of the metamorphosis data generated following the MRs before and after the introduction of mutants. Inspired by this, and considering that our baselines are transformer-based models, we adopt the four mutants: Gaussian Fuzzing (GF), Weight Shuffling (WS), Neuron Effect Blocking (NEB) and Neuron Switch (NS), and details are in Appendix A.2.

We execute the evaluation experiments following the methodology of Chen et al. (2021b) in both 2-tailed and 1-tailed manners to calculate \(p\) values, at the \(\sigma\) level of 0.05. Thus, we have:

* In the 2-tailed test, VR calculated by a particular MR is **similar** (denoted as \(\longleftrightarrow\)) to F1-score in revealing the mutants when \(p\geq\sigma\).
* In the 1-tailed test, VR calculated by a particular MR is **better** (denoted as \(\uparrow\)) than F1-score in revealing the mutants when \(p<\sigma\).
* Otherwise, VR calculated by a particular MR is **worse** (denoted as \(\downarrow\)) than F1-score in revealing the mutants.

Figure 4: Statistics of the paired Wilcoxon signed-rank test results on SLGM (Here red color is few and more results of other six models are at Appendix A.3).

Figure 3: The relation of average VR and F1-score of each model without MATO on LAP14



The paired Wilcoxon signed-rank test results are in the Appendix A.3. Taking the test on SLGM as an example, as shown in Fig. 4, the VRs calculated by MRs are better or similar in revealing the mutants, indicating that the property-based validation method VR based the inter-triplet and intra-triplet MRs are more effective than the traditional reference-based validation method F1-score.

### RQ3: Diversity Capacity Gain from MATO

The MT results on SLGM are in Table 4 and the MT results on the others are in the appendix A.4.

To start with, we can find that SLGM's VMs of for four datasets on MRs except MR1-2 is higher than 0.2, which indicates that there is still much room for improvement in the capacity of SLGM. We can find that the performance of these models varies in facing inter-triplet and intra-triplet diverse expression. For example, the VRs of SLGM in MR1-2 are all below 0.2 while the VRs of SLGM in MR2-4 are all above 0.4 in Table 4. This suggests that there is capacity difference on SLGM when facing inter-triplet and intra-triplet diversity.

Next, we can enhance the capacity of different dimensions of the ASTE model significantly by introducing diversity-aware loss. Take SLGM as an example on LAP14, as shown in Fig. 5, we can find that MATO significantly reduces the VR of MR1-1, MR1-2, MR1-3, MR2-1, MR2-2 and MR2-3 (all reduced by more than 10%). This suggests that MATO is able to keep the extraction process more capacity by having the ASTE model focus more on triplet representations. The introduction of MATO slightly increases the VR of MR2-4 onLAP14 (from 0.6186 to 0.6244, VR% below 1%), which does not affect the capacity of SLGM.

### Case Study

We perform a case study on the response of SLGM when facing inter-triplet and intra-triplet diverse expression for the target triplet. From the results of (1) and (2) in Table 5, SLGM suffers from impacts outside the target triplet when solving the ASTE task, whereas our MATO enables the model to be shielded from the impacts when extracting the target triplet. From the results of (3) and (4) in

\begin{table}
\begin{tabular}{c c c c c c c c c c c c} \hline Dataset & Model & MR1-1 & MRI-1 & MR1-3 & MR1-4 & MR2-1 & MR2-2 & MR2-3 & MR2-4 & Aspect & Opinion & Sentiment \\ \hline  & SLGM & 0.2503 & 0.1549 & 0.2866 & 0.4110 & 0.3273 & 0.3371 & 0.2548 & **0.6186** & 0.3199 & 0.3305 & 0.3678 \\  & +MATO & **0.2029** & **0.1240** & **0.2242** & **0.3821** & **0.2832** & **0.2657** & **0.1843** & 0.6244 & **0.2685** & **0.2868** & **0.3387** \\ \hline  & SLGM & 0.1807 & 0.1318 & 0.2838 & 0.2798 & 0.2423 & 0.2216 & 0.1855 & **0.5392** & 0.2479 & 0.2603 & 0.3087 \\  & +MATO & **0.1384** & **0.0973** & **0.1922** & **0.2413** & **0.1804** & **0.1618** & **0.1241** & 0.5581 & **0.1845** & **0.2162** & **0.2722** \\ \hline  & SLGM & 0.2274 & 0.1125 & 0.3148 & 0.3943 & 0.3435 & 0.2356 & 0.2247 & 0.4214 & 0.3193 & 0.2758 & 0.3108 \\  & +MATO & **0.1906** & **0.0650** & **0.2740** & **0.3549** & **0.3056** & **0.1933** & **0.1661** & **0.4201** & **0.2752** & **0.2377** & **0.2785** \\ \hline  & SLGM & 0.1632 & 0.1004 & 0.2526 & 0.3075 & 0.2604 & 0.1903 & 0.1739 & **0.5988** & 0.2486 & 0.2552 & 0.3148 \\  & +MATO & **0.1236** & **0.0838** & **0.1787** & **0.2332** & **0.1896** & **0.1503** & **0.1326** & 0.6029 & **0.1835** & **0.2150** & **0.2747** \\ \hline  & & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 4: The MT results (i.e., VRs) on SLGM (other six models see Appendix A.4). The value of VR ranges from 0 to 1, with smaller values indicating a greater capacity facing the inter-triplet and intra-triplet diverse expression. The score marked as **bold** means the better performance in pairwise comparison.

Figure 5: MATO reduces the VR magnitudes of all MRs on LAP for SLGM



Table 5, SLGM is not able to perceive the change of the target triplet and reacts incorrectly, while our MATO is able to enhance the model's ability to perceive the target triplet.

As shown in Table 6, we perform a case study on two representative large language models (LLMs). From the results of (1) and (2) in Table 6, the two LLMs can easily determine the sentiment polarity of "scrummy" based on its rich knowledge storage. However, from the results of (3) and (4) in Table 6, after masking some semantic information with "[UNK]", the two LLMs both fail to identify the aspect "use", and mistakes "It " as aspect, while SLGM+MATO can still identify aspect correctly.

## 5 Conclusions

In this paper, we design metamorphic relations to evaluate the capacity of seven SOTA ASTE models by metamorphic testing and find that most of the models have weak capacity in dealing with diverse expression in sentiment analysis. Meanwhile, we propose a model-agnostic diversity-aware method to improve ASTE model inference consistent with expected results facing triplet diversity. Experiments show that our MATO can significantly enhance the diversity capacity of ASTE models.

## Limitations

Although the MRs we design are able to expose problems regarding the diversity capacity of the ASTE model and MATO can improve the capacity, there are some limitations. On the one hand, the MRs we design only introduce diversity into triplets, while there are much diversity in the real world, such as changes in sentence structure and typos etc., and we will follow up with deeper work in this problem. On the other hand, from the results of MT, we can find that MATO does not bring considerable enhancement on MR2-4, and our subsequent work will explore how to make the model able to capture such intra-triplet sentiment inversion changes.

## Acknowledgments

This work has been partially supported by the National Natural Science Foundation of China (NSFC) with Grant No. 62276196.

## References

* Aleti (2023) Aldeida Aleti. 2023. Software testing of generative AI systems: Challenges and opportunities. In _IEEE/ACM International Conference on Software Engineering: Future of Software Engineering, ICSEC-FoSE 2023, Melbourne, Australia, May 14-20, 2023_, pages 4-14. IEEE.
* Brauwers and Frasincar (2023) Gianni Brauwers and Flavius Frasincar. 2023. A survey on aspect-based sentiment classification. _ACM Comput. Surv._, 55(4):65:1-65:37.
* Chen et al. (2022a) Chenhua Chen, Zhiyang Teng, Zhongqing Wang, and Yue Zhang. 2022a. Discrete opinion tree induction for aspect-based sentiment analysis. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 2051-2064. Association for Computational Linguistics.
* Chen et al. (2022b) Hao Chen, Zepeng Zhai, Fangxiang Feng, Ruifan Li, and Xiaojie Wang. 2022b. Enhanced multi-channel graph convolutional network for aspect sentiment triplet extraction. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 2974-2985. Association for Computational Linguistics.
* Chen et al. (2021a) Shaowei Chen, Yu Wang, Jie Liu, and Yuelin Wang. 2021a. Bidirectional machine reading comprehension for aspect sentiment triplet extraction. In _ThirtyFifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_, pages 12666-12674. AAAI Press.

\begin{table}
\begin{tabular}{c|l l|l c|l c c} \hline No. & Input & GPT-do & Violate? & Qwen2.5 & Violate? & SLGM+MATO & Violate? \\ \hline \multirow{4}{*}{(1)} & The sound is nice and loud; I do n’t have any & \multirow{4}{*}{No} & \multirow{4}{*}{(sound, nice, POS)} & \multirow{4}{*}{(sound, scrummy, POS)} & \multirow{4}{*}{No} & \multirow{4}{*}{(sound, scrummy, POS)} & \multirow{4}{*}{No} \\  & problems with hearing anything. & & & & & & \\ \cline{1-1} \cline{5-7}  & The sound is scrummy and loud; I do n’t have & \multirow{4}{*}{(sound, scrummy, POS)} & \multirow{4}{*}{Yes} & \multirow{4}{*}{(sound, scrummy, POS)} & \multirow{4}{*}{Yes} & \multirow{4}{*}{(sound, scrummy, POS)} & \multirow{4}{*}{No} \\ \cline{1-1} \cline{5-7}  & any problems with hearing anything. & & & & & & \\ \hline \end{tabular}
\end{table}
Table 6: Case study for the target triplets ”(sound, nice, POS)” and ”(use, light, POS)” when GPT-do and Qwen2.5 face inter-triplet and intra-triplet diverse expression, respectively. The underlined parts indicate the distinctions.

 

Songqiang Chen, Shuo Jin, and Xiaoyuan Xie. 2021b. Validation on machine reading comprehension software without annotated labels: a property-based method. In _ESEC/FSE '21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021_, pages 590-602. ACM.
* Chen et al. (2018) Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou. 2018. Metamorphic testing: A review of challenges and opportunities. _ACM Comput. Surv._, 51(1):4:1-4:27.
* Chen and Qian (2020) Zhuang Chen and Tieyun Qian. 2020. Enhancing aspect term extraction with soft prototypes. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 2107-2117. Association for Computational Linguistics.
* Corder and Foreman (2014) Gregory W Corder and Dale I Foreman. 2014. _Nonparametric statistics: A step-by-step approach_. John Wiley & Sons.
* Dai and Song (2019) Hongliang Dai and Yangqiu Song. 2019. Neural aspect and opinion term extraction with mined rules as weak supervision. In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_, pages 5268-5277. Association for Computational Linguistics.
* Hyun et al. (2024) Sangwon Hyun, Mingyu Guo, and M Ali Babar. 2024. Metal: Metamorphic testing framework for analyzing large-language model qualities. In _2024 IEEE Conference on Software Testing, Verification and Validation (ICST)_, pages 117-128. IEEE.
* Jiang et al. (2021) Mingyue Jiang, Houzhen Bao, Kaiyi Tu, Xiao-Yi Zhang, and Zuohua Ding. 2021. Evaluating natural language inference models: A metamorphic testing approach. In _32nd IEEE International Symposium on Software Reliability engineering, ISSRE 2021, Wuhan, China, October 25-28, 2021_, pages 220-230. IEEE.
* Kendall et al. (2018) Alex Kendall, Yarin Gal, and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_, pages 7482-7491. Computer Vision Foundation / IEEE Computer Society.
* Li et al. (2023) Dongxu Li, Zhihao Yang, Yuquan Lan, Yunqi Zhang, Hui Zhao, and Gang Zhao. 2023. Simple approach for aspect sentiment triplet extraction using span-based segment tagging and dual extractors. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2023, Taipei, Taiwan, July 23-27, 2023_, pages 2374-2378. ACM.
* Li et al. (2020) Kun Li, Chengbo Chen, Xiaojun Quan, Qing Ling, and Yan Song. 2020. Conditional augmentation for aspect term extraction via masked sequence-to-sequence generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 7056-7066. Association for Computational Linguistics.
* Li et al. (2024) Lin Li, Shaopeng Tang, and Renwei Wu. 2024. Majority rules guided aspect-category based sentiment analysis via label prior knowledge. In _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy_, pages 10952-10957. ELRA and ICCL.
* Li et al. (2021) Ruifan Li, Hao Chen, Fangxiang Feng, Zhanyu Ma, Xiaojie Wang, and Eduard H. Hovy. 2021. Dual graph convolutional networks for aspect-based sentiment analysis. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 6319-6329. Association for Computational Linguistics.
* Liang et al. (2023) Shuo Liang, Wei Wei, Xian-Ling Mao, Yuanyuan Fu, Rui Fang, and Dangyang Chen. 2023. STAGE: span tagging and greedy inference scheme for aspect sentiment triplet extraction. In _Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty- Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023_, pages 13174-13182. AAAI Press.
* Liebel and Korner (2018) Lukas Liebel and Marco Korner. 2018. Auxiliary tasks in multi-task learning. _CoRR_, abs/1805.06334.
* Liu et al. (2023) Dan Liu, Lin Li, Xiaohui Tao, Jian Cui, and Qing Xie. 2023. Descriptive prompt paraphrasing for target-oriented multimodal sentiment classification. In _Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023_, pages 4174-4186. Association for Computational Linguistics.
* Liu et al. (2020) Haoyue Liu, Ishani Chatterjee, MengChu Zhou, Xiaoyu Sean Lu, and Abdullah Abusorrah. 2020. Aspect-based sentiment analysis: A survey of deep learning methods. _IEEE Trans. Comput. Soc. Syst._, 7(6):1358-1375.
* Manino et al. (2022) Edoardo Manino, Julia Rozanova, Danilo S. Carvalho, Andre Freitas, and Lucas C. Cordeiro. 2022. Systematicity, compositionality and transitivity of deep NLP models: a metamorphic testing perspective. In _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_, pages 2355-2366. Association for Computational Linguistics.
* Mao et al. (2021) Yue Mao, Yi Shen, Chao Yu, and Longjun Cai. 2021. A joint training dual-mrc framework for aspect based 

sentiment analysis. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_, pages 13543-13551. AAAI Press.
* Peng et al. (2020) Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_, pages 8600-8607. AAAI Press.
* Pontiki et al. (2016) Maria Pontiki, Dimitris Galanis, Haris Papageorgiou, Ion Androutsopoulos, Suresh Manandhar, Mohammad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphee De Clercq, Veronique Hoste, Marianna Apidianaki, Xavier Tannier, Natalia V. Loukachevitch, Evgeniy V. Kotelnikov, Nuria Bel, Salud Maria Jimenez Zafra, and Gulsen Eryigit. 2016. Semeval-2016 task 5: Aspect based sentiment analysis. In Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016, San Diego, CA, USA, June 16-17, 2016_, pages 19-30. The Association for Computer Linguistics.
* Poutik et al. (2015) Maria Poutik, Dimitris Galanis, Haris Papageorgiou, Suresh Manandhar, and Ion Androutsopoulos. 2015. Semeval-2015 task 12: Aspect based sentiment analysis. In Proceedings of the 9th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2015, Denver, Colorado, USA, June 4-5, 2015_, pages 486-495. The Association for Computer Linguistics.
* Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. Semeval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014_, pages 27-35. The Association for Computer Linguistics.
* Pu et al. (2023) Muxin Pu, Chun Yong Chong, and Mei Kuan Lim. 2023. Robustness evaluation in hand pose estimation models using metamorphic testing. In 8th IEEE/ACM International Workshop on Metamorphic Testing, MET@ICSE 2023, Melbourne, Australia, May 14, 2023, pp. 31-38. IEEE.
* Schouten and Frasincar (2016) Kim Schouten and Flavius Frasincar. 2016. Survey on aspect-level sentiment analysis. _IEEE Trans. Knowl. Data Eng._, 28(3):813-830.
* Sun et al. (2024) Qiao Sun, Liujia Yang, Minghao Ma, Nanyang Ye, and Qingying Gu. 2024. MiniConGTS: A near ultimate minimalist contrastive grid tagging scheme for aspect sentiment triplet extraction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2817-2834, Miami, Florida, USA. Association for Computational Linguistics.
* Tang et al. (2016) Duyu Tang, Bing Qin, and Ting Liu. 2016. Aspect level sentiment classification with deep memory network. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, Online, pp. 214-224. The Association for Computational Linguistics.
* Wan et al. (2020) Hai Wan, Yufei Yang, Jianfeng Du, Yanan Liu, Kunxun Qi, and Jeff Z. Pan. 2020. Target-aspect-semtent joint detection for aspect-based sentiment analysis. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 9122-9129. AAAI Press.
* Wang et al. (2024) Jun Wang, Yanhui Li, Zhifei Chen, Lin Chen, Xiaofang Zhang, and Yuming Zhou. 2024. Knowledge graph driven inference testing for question answering software. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pages 119:1-119:13. ACM.
* Wang et al. (2016)Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pp. 606-615. The Association for Computational Linguistics.
* Wu et al. (2020a) Meixi Wu, Wenya Wang, and Sinno Jialin Pan. 2020a. Deep weighted maxsat for aspect-based opinion extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 5618-5628. Association for Computational Linguistics.
* Wu et al. (2020b) Zhen Wu, Chengcan Ying, Fei Zhao, Zhifang Fan, Xinyu Dai, and Rui Xia. 2020b. Grid tagging scheme for aspect-oriented fine-grained opinion extraction. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, Association for Computational Linguistics.
* Xu et al. (2018) Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2018. Double embeddings and cnn-based sequence labeling for aspect extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pp. 592-598. Association for Computational Linguistics.
* Xu et al. (2021) Lu Xu, Yew Ken Chia, and Lidong Bing. 2021. Learning span-level interactions for aspect sentiment triplet 

extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 4755-4766. Association for Computational Linguistics.
* Xu et al. (2020) Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020. Position-aware tagging for aspect sentiment triplet extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_, pages 2339-2349. Association for Computational Linguistics.
* Xue and Li (2018) Wei Xue and Tao Li. 2018. Aspect based sentiment analysis with gated convolutional networks. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_, pages 2514-2523. Association for Computational Linguistics.
* Yan et al. (2021) Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021. A unified generative framework for aspect-based sentiment analysis. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_, pages 2416-2429. Association for Computational Linguistics.
* Yin et al. (2016) Yichun Yin, Furu Wei, Li Dong, Kaimeng Xu, Ming Zhang, and Ming Zhou. 2016. Unsupervised word and dependency path embeddings for aspect term extraction. In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, New York, NY, USA, 9-15 July 2016_, pages 2979-2985. IJCAI/AAAI Press.
* Zhai et al. (2022) Zepeng Zhai, Hao Chen, Fangxiang Feng, Ruifan Li, and Xiaojie Wang. 2022. COM-MRC: A context-masked machine reading comprehension framework for aspect sentiment triplet extraction. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 3230-3241. Association for Computational Linguistics.
* Zhang et al. (2021a) Wenxuan Zhang, Yang Deng, Xin Li, Yifei Yuan, Lidong Bing, and Wai Lam. 2021a. Aspect sentiment quad prediction as paraphrase generation. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 9209-9219. Association for Computational Linguistics.
* Zhang et al. (2021b) Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. 2021b. Towards generative aspect-based sentiment analysis. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021_, pages 504-510. Association for Computational Linguistics.
* Zhang et al. (2022) Yicc Zhang, Yifain Yang, Yihui Li, Bin Liang, Shiwei Chen, Yixue Dang, Min Yang, and Ruifeng Xu. 2022. Boundary-driven table-filling for aspect sentiment triplet extraction. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 6485-6498. Association for Computational Linguistics.
* Zhou and Qian (2023) Shen Zhou and Tiesyun Qian. 2023. On the strength of sequence labeling and generative models for aspect sentiment triplet extraction. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 12038-12050. Association for Computational Linguistics.

## Appendix A Appendix

### The baselines

The seven baselines are described as follows.

**MiniConGTS:** It improves and utilizes pre-trained representations by integrating a minimalist tagging scheme and a novel token-level contrastive learning strategy Sun et al. (2024).

**SLGM:** It proposes a sequence labeling enhanced generative model to enhance the mutual informative clues between aspect and opinion terms Zhou and Qian (2023).

**STAGE:** It proposes span tagging and greedy inference to extract sentiment triplets in span-level, where each span might consist of multiple words and play different roles simultaneously Liang et al. (2023).

**SimSTAR:** It proposes a span-based solution with segment tagging and dual extractors to address inadequate representation of the information through previous tagging schemes and insufficient usage of all available sentiment data Li et al. (2023).

**BDTF:** It proposes boundary-driven table-filling, which represents each triplet as a relation region in the 2D table and transforms the ASTE task into detection and classification of relation regions Zhang et al. (2022).

**COM-MRC:** It proposes a novel context-masked MRC framework to address the failure of the MRC-based methods when facing multiple aspect terms, due to the interference from other aspect terms Zhai et al. (2022).

**EMCGCN:** It proposes an enhanced multi-channel graph convolutional network model to fully 

utilize the relations between words (Chen et al., 2022b).

### The mutants

The four mutants are described as follows.

**Gaussian Fuzzing:** Weights of the neurons act as the key to control the decision logic of the neural networks. This mutant fuzzes the values of the weights for all the target neurons to change the connection importance they represent through adding Gaussian noise.

**Weight Shuffling:** The output of a neuron is usually determined by the neurons in the previous layer through the connections with weights. This mutant shuffles the weights of randomly picked target neurons to disturb their connections with their previous layers.

**Neuron Effect Blocking:** Every neuron in a neural network contributes to its final decision to some extent. This operator removes the influence of randomly picked target neurons to the final decision by resetting their connection weights of the next layers to zeros to block the propagation of their effects.

**Neuron Switch:** Different neurons in one neural network layer usually play different roles on the connected neurons in the next layer. This operator switches the weights of two neurons in the same layer to exchange their effects for next layer.

### The statistics of the paired Wilcoxon signed-rank test results for the other six models

On the whole, the data generated by the inter-triplet and intra-triplet MRs we design are better than the original data in revealing the mutants (i.e., Gaussian Fuzzing, Weight Shuffling, Neuron Effect Blocking and Neuron Switch) as shown in Fig. 6. GF, WS, NEB and NS denote Gaussian Fuzzing, Weight Shuffling, Neuron Effect Blocking and Neuron Switch respectively. Better conclusion shows that the data generated by the corresponding MR is better than the original data in detecting the corresponding mutant, which indicates that the designed MR is effective in analyzing the model diversity capacity.

There are two significant findings. To begin with, most of the 2-tailed p values in the "better" test are 0.0020, and most of the 1-tailed p values are 0.0010 (both extreme borderline values), indicating that the generated data are significantly better than the original data in revealing the mutants (Corder and Freeman, 2014).

Furthermore, we find that almost all of the "worse" test results occur in Weight Shuffling and Neuron Effect Blocking, which may be due to the fact that these seven transformer-based models are the deep neural network.

### The MT results for the other six models

The MT results for the other six models are in Table 7. It can be observed that the introduction of our MATO is able to significantly reduce the VRs of the six models on all except MR2-4, as well as significantly reduce the VRs on aspect term, opinion term and sentiment polarity, which suggests that our MATO is able to enhance their capacity.

 

Figure 6: Statistics of the paired Wilcoxon signed-rank test results on other six models





\begin{table}
\begin{tabular}{c c l r r r r r r r r r r r} \hline No. & Dataset & Model & MR1-1 & MR1-2 & MR1-3 & MR1-4 & MR2-1 & MR2-2 & MR2-3 & MR2-4 & Aspect & Opinion & Sentiment \\ \hline \multirow{6}{*}{1} & LAP14 & MiniConGTS & 0.3782 & 0.2519 & 0.5505 & 0.6525 & 0.5123 & 0.4854 & 0.3734 & 0.6263 & 0.5222 & 0.4740 & 0.5203 \\  & & +MATO & **0.3161** & **0.1865** & **0.5284** & **0.6173** & **0.5039** & **0.4528** & **0.3510** & **0.5667** & **0.5002** & **0.4313** & **0.4747** \\ \cline{2-13}  & RES14 & MiniConGTS & 0.3764 & **0.1963** & 0.5944 & 0.6539 & 0.5889 & 0.4261 & 0.4012 & 0.5993 & 0.5596 & 0.4582 & 0.5010 \\  & & +MATO & **0.3494** & 0.2066 & **0.5561** & **0.6304** & **0.5650** & **0.4037** & **0.3792** & **0.5267** & **0.5327** & **0.4360** & **0.4800** \\ \cline{2-13}  & RES15 & MiniConGTS & 0.3656 & **0.1958** & 0.6828 & 0.7345 & 0.6061 & 0.4223 & 0.3558 & 0.4299 & 0.5948 & 0.4552 & 0.5108 \\  & RES15 & +MATO & **0.3449** & 0.1966 & **0.6509** & **0.6982** & **0.5496** & **0.3781** & **0.3193** & **0.4148** & **0.5545** & **0.4290** & **0.4901** \\ \cline{2-13}  & RES16 & MiniConGTS & 0.3634 & 0.2104 & 0.6046 & 0.6536 & 0.5542 & 0.4282 & 0.3884 & 0.5988 & 0.5502 & 0.4639 & 0.5169 \\  & & +MATO & **0.3372** & **0.1842** & **0.5792** & **0.6207** & **0.5182** & **0.4040** & **0.3626** & **0.5829** & **0.5202** & **0.4387** & **0.4918** \\ \hline \multirow{6}{*}{2} & LAP14 & STAGE & 0.2772 & 0.1606 & 0.3476 & 0.5341 & 0.3943 & 0.3819 & 0.2652 & **0.6006** & 0.3853 & 0.3667 & 0.4107 \\  & & +MATO & **0.2446** & **0.1396** & **0.3397** & **0.4668** & **0.3264** & **0.3433** & **0.2356** & 0.6224 & **0.3421** & **0.3417** & **0.3921** \\ \cline{2-13}  & RES14 & STAGE & 0.2087 & 0.1456 & 0.2899 & 0.3573 & 0.2849 & 0.2365 & 0.1944 & **0.5465** & 0.2816 & 0.2827 & 0.3348 \\  & & +MATO & **0.1754** & **0.1256** & **0.2569** & **0.3146** & **0.2542** & **0.2081** & **0.1726** & 0.5501 & **0.2496** & **0.2576** & **0.3118** \\ \cline{2-13}  & RES15 & STAGE & 0.2445 & 0.1073 & 0.4204 & 0.5330 & 0.4548 & 0.3260 & 0.2822 & 0.4332 & 0.4226 & 0.3352 & 0.3735 \\  & & +MATO & **0.2286** & **0.0936** & **0.3766** & **0.4784** & **0.3966** & **0.3026** & **0.2541** & **0.4283** & **0.3764** & **0.3089** & **0.3442** \\ \cline{2-13}  & RES16 & STAGE & 0.1744 & 0.1066 & 0.2946 & 0.3586 & 0.2701 & 0.2172 & **0.1848** & **0.5865** & 0.2770 & 0.2747 & 0.3366 \\  & & +MATO & **0.1644** & **0.0991** & **0.2766** & **0.3284** & **0.2542** & **0.2113** & 0.1891 & 0.5906 & **0.2621** & **0.2656** & **0.3237** \\ \hline \multirow{6}{*}{3} & LAP14 & SimSTAR & 0.3090 & 0.1878 & 0.3650 & 0.5565 & 0.4043 & 0.4116 & 0.2756 & 0.6282 & 0.4004 & 0.3905 & 0.4344 \\  & & +MATO & **0.2575** & **0.1485** & **0.3156** & **0.4786** & **0.3380** & **0.3693** & **0.2572** & **0.6128** & **0.3474** & **0.3485** & **0.3889** \\ \cline{2-13}  & RES14 & SimSTAR & 0.1847 & 0.1325 & 0.2618 & 0.3227 & 0.2527 & 0.2064 & 0.1678 & 0.5538 & 0.2513 & 0.2614 & 0.3177 \\  & & +MATO & **0.1745** & **0.1281** & **0.2467** & **0.3016** & **0.2367** & **0.1943** & **0.1603** & **0.5511** & **0.2363** & **0.2509** & **0.3069** \\ \cline{2-13}  & RES15 & SimSTAR & 0.2460 & 0.1318 & 0.4068 & 0.5037 & **0.3833** & 0.3008 & **0.2213** & **0.4136** & 0.3788 & 0.3177 & 0.3640 \\  & & +MATO & **0.2415** & **0.1170** & **0.3579** & **0.4638** & 0.3859 & **0.2738** & 0.2232 & 0.4283 & **0.3577** & **0.3008** & **0.3418** \\ \cline{2-13}  & RES16 & SimSTAR & 0.1626 & 0.1066 & **0.2616** & 0.3167 & 0.2521 & 0.1896 & 0.1688 & **0.5988** & 0.2498 & 0.2578 & 0.3209 \\  & & +MATO & **0.1570** & **0.0960** & 0.2699 & **0.2872** & **0.2250** & **0.1723** & **0.1420** & 0.6070 & **0.2310** & **0.2473** & **0.3150** \\ \hline \multirow{6}{*}{4} & LAP14 & BDTF & 0.3690 & 0.1807 & 0.4913 & 0.6497 & **0.4776** & 0.4848 & 0.3502 & 0.6263 & 0.4922 & 0.4503 & 0.4870 \\  & & +MATO & **0.3549** & **0.1759** & **0.4800** & **0.6368** & 0.4831 & **0.4694** & **0.3474** & **0.6224** & **0.4868** & **0.4410** & **0.4788** \\ \cline{2-13}  & RES14 & BBDTF & 0.3934 & 0.2621 & 0.5737 & 0.6749 & 0.6066 & 0.4406 & 0.4134 & **0.5593** & 0.567