# Interpretability in Sentiment Analysis: A Self-Supervised Approach to Sentiment Cue Extraction
## 4 Methodology
### Generating Dataset for Sentiment Cue Extraction
#### 4.2.2 Sentiment Classification Model  
As demonstrated in Figure 1, selecting an optimal pseudo label from the array of candidate sequences involves scoring each candidate. Within the SS-SCE framework, this scoring process is facilitated by a sentiment classification model. Herein, we provide an overview of the sentiment classification model used in the SS-SCE context.  
In our research, the sentiment classification model is built on BERT as the encoding mechanism, mainly due to its ability to effectively capture contextual nuances within the text. BERT, a transformer-based model, stands out for its dynamic encoding capabilities, compared to static word vector methods, such as GloVe [41], which may not fully grasp the context-dependent aspects of language.  
Moreover, compared to GPT [25, 26], another transformer-based architecture, BERT is more aligned with our needs. While GPT excels in text generation tasks due to its unidirectional nature, BERT's bidirectional training strategy makes it particularly suitable for understanding the nuanced expressions of sentiment in texts. This bidirectionality allows BERT to gather context from both sides of a token, offering a richer representation of the input text and enhancing the model's ability to discern the underlying sentiment.  
Additionally, sentiment classification models within the academic community often leverage BERT-based architectures, facilitating straightforward comparisons with other models in the field.  
When encoding the input \(X\) using BERT for our sentiment classification model, it is necessary to prepend a [CLS] token at the beginning and append a [SEP] token at the end of \(X\). Thus, the actual sequence inputted into BERT becomes \(X=\{[\text{CLS}],x_{1},x_{2},\cdots,x_{n},[\text{SEP}]\}\). For text classification tasks, the encoding of the [CLS] token is typically utilized to represent the encoding of the entire sentence.  
To facilitate comparisons with other models, we have constructed a remarkably straightforward binary text classification model \(f_{\text{sc}}\) based on the base version of BERT. In this model, \(f_{\text{sc}}\), the enhanced input \(X\) is encoded using BERT, resulting in a 768-dimensional vector representation, \(h_{X}^{768}\). This vector, specifically derived from the encoding of the [CLS] token, is then transformed into a two-dimensional vector, \(h_{X}^{2}\), via a fully connected layer. Subsequently, a softmax function converts \(h_{X}^{2}\) into a pair of probabilities that indicate the likelihood of \(X\) belonging to categories 0 (negative sentiment) and 1 (positive sentiment), respectively. The sentiment classification model \(f_{\text{sc}}\) can thus be expressed as:  
\[f_{\text{sc}}(X)=\text{softmax}(\text{FC}^{768\times 2}(\text{BERT}(X)_{[ \text{CLS}]})) \tag{3}\]  
where \(\text{FC}^{768\times 2}\) denotes the fully connected layer mapping the 768-dimensional BERT encoding to a 2-dimensional output, and \(\text{BERT}(X)_{[\text{CLS}]}\) refers to the representation of the [CLS] token produced by BERT, which serves as the aggregate representation of the enhanced input text for classification purposes.  
Accordingly, for an input, the model yields the following probability pair:  
\[(p_{c0},p_{c1})=f_{\text{cls}}(X), \tag{4}\]  
where \(p_{c0}\) and \(p_{c1}\) correspond to the probabilities of \(X\) being classified under negative and positive sentiments, respectively. Consequently, the sentiment prediction for \(X\) by \(f_{\text{cls}}\) is determined as:  
\[C=\arg\max(p_{c0},p_{c1}), \tag{5}\]  
This procedure also facilitates the computation of the Probability Discrepancy between the categories:  
\[\Delta P=|p_{c0}-p_{c1}|. \tag{6}\]  
In this context, \(\Delta P\), referred to as \"Probability Discrepancy\", is utilized to assess the intensity of the sentiment inclination prediction made by \(f_{\text{sc}}\) for \(X\). A larger \(\Delta P\) value indicates a more pronounced sentiment inclination in \(X\), reflecting the model's confidence in its sentiment classification.