# Interpretability in Sentiment Analysis: A Self-Supervised Approach to Sentiment Cue Extraction
## 5 Experiments
### Case Study: Comparing SS-SCE with Established Interpretability Methods  
To evaluate the unique contributions and effectiveness of SS-SCE, we perform a comparative analysis with established interpretability methods in text classification, including LIME [5], LIG [43], OCC [44], SVS [45], and LDS [46]. For this comparison, we use the Thermostat tool ([[https://github.com/DFKI-NLP/thermostat](https://github.com/DFKI-NLP/thermostat)]([https://github.com/DFKI-NLP/thermostat](https://github.com/DFKI-NLP/thermostat)) (accessed on 19 March 2024)) [47], which integrates state-of-the-art interpretability methods, offering a unified platform for analysis. This tool allowed us to apply these methods in a standardized way, ensuring a fair and consistent comparison between different interpretability approaches. Our analysis aimed not to compare SS-SCE directly with these methods, but to showcase how SS-SCE's focused approach on sentiment cues provides a different, potentially more nuanced perspective in understanding model decisions, especially in the context of sentiment analysis. Using Thermostat, we applied interpretability models trained on various datasets, such as IMDb with pre-trained language models such as BERT and ALBERT [48]. For a fair comparison, we chose the interpretability model trained with BERT on the IMDb dataset. To facilitate a comparison with SOTA methods, we manually annotated two selected instances, a positive and a negative, from the IMDb test set. We then calculated the precision, recall, and F1 score for each method's sentiment cue extraction on these annotated instances. The results of this comparative analysis are presented in Tables 7 and 8. Table 7 shows that the SS-SCE models, particularly SS-SCE\({}_{SST-2}\), demonstrate superior performance in extracting sentiment cues from the positive text instance when compared with SOTA interpretability methods, SS-SCE\({}_{SST-2}\) achieved the highest precision of 0.7778, recall of 0.8235, and F1 score of 0.8000, indicating a robust capability in accurately identifying and recalling relevant sentiment cues. The SS-SCE models trained on Yelp and IMDb datasets showed varying degrees of effectiveness, with SS-SCE\({}_{Yelp}\) displaying moderate performance and SS-SCE\({}_{IMDb}\), showing decent accuracy but lower effectiveness compared to SS-SCE\({}_{SST-2}\). This variation suggests the influence of training data characteristics on the model's performance. In contrast, the standard interpretability methods, while useful in their own right, exhibited lower performance metrics in comparison. LIME, LIG, OCC, SVS, and LDS demonstrated lower precision, recall, and F1 scores, indicating a potential limitation in their ability to capture the nuanced sentiment cues as effectively as the SS-SCE approach. Table 8 presents the performance of different interpretability methods in extracting sentiment cues from the negative text instance. The results indicate that the SS-SCE models, particularly SS-SCE\({}_{IMDb}\) and SS-SCE\({}_{SST-2}\), perform effectively in this context, albeit with some variations in precision and recall. \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline
**Metric** & \(\mathbf{X}\) & \(\mathbf{X^{Y}_{SST-2}}\) & \(\mathbf{loss_{SST-2}}\) & \(\mathbf{X^{Y}_{Yelp}}\) & \(\mathbf{loss_{Yelp}}\) & \(\mathbf{X^{Y}_{IMDb}}\) & \(\mathbf{loss_{IMDb}}\) & \(\mathbf{X^{Y}_{combined}}\) & \(\mathbf{loss_{combined}}\) \\ \hline Accuracy & 0.9328 & 0.8925 & 0.0403 & 0.8709 & 0.0620 & 0.8798 & 0.0531 & 0.8934 & 0.0394 \\ Precision & 0.9305 & 0.9205 & 0.0099 & 0.8564 & 0.0761 & 0.8333 & 0.0971 & 0.9327 & \(-\)0.0020 \\ Recall & 0.9359 & 0.8598 & 0.0762 & 0.8918 & 0.0441 & 0.9501 & \(-\)0.014 & 0.8474 & 0.0921 \\ F1 & 0.9332 & 0.8891 & 0.0441 & 0.8737 & 0.0594 & 0.8879 & 0.0453 & 0.8880 & 0.0452 \\ RCT & - & 0.1153 & - & 0.1186 & - & 0.2858 & - & 0.2478 & - \\ \hline \hline \end{tabular}
\end{table} 
Table 6: This table shows the test results on the IMDb dataset for models trained on SST-2, Yelp, IMDb, and the combined dataset.