# Interpretability in Sentiment Analysis: A Self-Supervised Approach to Sentiment Cue Extraction
## 5 Experiments
### Results and Analysis
#### 5.4.2 Main Performance Evaluation  
Given the absence of annotated data, it is challenging to directly apply traditional sequence labeling evaluation metrics to assess SS-SCE. According to the definition of the SCE task, the sentiment orientation of \(X^{Y}\), obtained by masking \(X\) with the pseudo-label \(Y\), should align with that of \(X\). Therefore, we can indirectly evaluate SS-SCE by comparing the performance metrics of instances in the test set when using \(X\) as input versus using \(X^{Y}\) as input in the sentiment classification model. Specifically, we calculate the accuracy, precision, recall, and F1 scores for the test set when using \(X\) and \(X^{Y}\) as inputs, respectively, and measure the performance loss caused by using \(X^{Y}\) as input. Additionally, to statistically assess the impact of our SS-SCE method on the performance of sentiment classification, we conduct a \(t\)-test comparing the predictions made by the sentiment classification model for both the original input \(X\) and the input with extracted sentiment cues \(X^{Y}\). The null hypothesis (H0) posits that the SS-SCE method does not significantly reduce the performance metrics of sentiment classification compared to the original input \(X\). The alternative hypothesis (H1), on the other hand, suggests a significant reduction in these performance metrics, which would indicate an effect of the SS-SCE method. We set the confidence level for this test at 0.01, meaning a \(p\)-value less than 0.01 is required to reject the null hypothesis. Rejecting H0 would imply that the SS-SCE method significantly impacts the performance of the model, whereas failing to reject H0 would suggest that the SS-SCE method can extract sentiment cues without substantially compromising classification accuracy. However, relying solely on this is not sufficient, as there could be special cases where all values of \(Y\) are 1, leading to \(X^{Y}=X\). To avoid this scenario, we also evaluate using RCT, which is the proportion of sentiment cues extracted by SS-SCE relative to the original input. To demonstrate the effectiveness and detailed impact of SS-SCE on sentiment classification accuracy, including any performance loss, Table 3 offers a comprehensive comparison. This table contrasts the performance metrics--accuracy, precision, recall, and F1 scores--for the original input (\(X\)) and the input with extracted sentiment cues (\(X^{Y}\)), across various datasets. It quantifies the performance loss incurred using \(X^{Y}\) as input and includes RCT to indicate the proportion of sentiment cues identified. Additionally, the table details the results of the \(t\)-test, providing statistical insight into the significance of the differences observed between the performances of \(X\) and \(X^{Y}\). For the SST-2 dataset, compared to the original input \(X\), the prediction results using \(X^{Y}\) as input show a decrease across all major metrics, but the decrease is within 0.1, and the \(p\)-value from the \(t\)-test is greater than 0.01. This indicates that our SS-SCE method effectively extracts the majority of sentiment cues from the SST-2 dataset, albeit with some minor losses. The Ratio of Cue Tokens (RCT) is 0.1682, which means that tokens identified as sentiment cues by SS-SCE constitute 16.82% of the total in the SST-2 test set. This performance suggests that SS-SCE can extract sentiment cues without significantly compromising the accuracy of sentiment classification. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Metric** & **SST-2** & **Yelp** & **IMdb** & **ChnSentiCorp** \\ \hline Average Length & 32.02 & 79.57 & 265.02 & 90.49 \\ ATPS (ms) & 0.70 & 3.71 & 9.84 & 6.03 \\ ATOMS (s) & 1.49 & 19.84 & 49.68 & 30.27 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: This table presents the average length of each instance in four datasets, the average time consumed per sampling, and the average time to obtain the optimal mask sequence. For the Yelp dataset, the decline in metrics for \(X^{Y}\) is notably subtle, with all reductions less than 0.02.