# The Impact of Cause and Filler Word Encoding on Dementia Detection with Contrastive Learning
## 2 Materials and Methods
### Modeling
#### 2.3.2 Contrastive Learning  
In this section, we propose using CL for classification. CL is a well-established approach that employs pairs of positive and negative examples to guide the deep learning model. Over time, CL has gained significant traction, surpassing earlier benchmarks and enhancing performance in fully supervised, semi-supervised, and self-supervised learning environments. The core of CL is the creation of meaningful representations through the comparison of positive and negative instance pairings. The essential principle is that in an embedded space, similar examples should be close together, while distinct instances should be farther apart [30]. CL helps models find relevant features and similarities in the dataset by approaching learning as a differentiation process [30]. In this paper, we use an approach that the authors in [31] proposed. The authors proposed the dual contrastive learning (DualCL) framework that is a method that concurrently trains on the features of input samples and the parameters of classifiers. Essentially, DualCL treats the classifier parameters as extended samples linked to various labels. It then leverages CL to draw comparisons between these input samples and the extended, or augmented, samples. This approach allows for a more integrated and holistic learning process, where both the sample features and classifier parameters are understood and developed in relation to each other. In this section, we address the modeling portion of the experiments. The notation from [31] is adopted. The focus is on a text classification task encompassing \(K\) different classes. The dataset under consideration, denoted as \(\{\mathbf{x}_{i},y_{i}\}_{i=1}^{N}\), comprises \(N\) individual training instances. Each instance consists of an input sentence \(\mathbf{x}_{i}\in\mathbb{R}^{L}\), comprising \(L\) words, alongside its corresponding label \(y_{i}\). For clarity, the study uses \(\mathcal{I}=\{1,2,\cdots,N\}\) to represent the index set of the training samples and \(\mathcal{K}=\{1,2,\cdots,K\}\) to denote the index set of the labels. We explore self-supervised contrastive learning. This technique involves a dataset of \(N\) training samples, each accompanied by at least one augmented version within the set. If \(j(i)\) represents the index of the augmented sample originating from the \(i\)-th sample, the standard formula for contrastive loss is as follows:  
\[\mathcal{L}_{\text{self}}=\frac{1}{N}\sum_{i\in\mathcal{I}}-\log\frac{\exp \Bigl{(}\mathbf{z}_{i}\cdot\mathbf{z}_{j(i)}/\tau\Bigr{)}}{\sum_{a\in\mathcal{A}_{i} }\exp(\mathbf{z}_{i}\cdot\mathbf{z}_{a}/\tau)} \tag{4}\]  
where \(\mathbf{z}_{i}\) signifies the normalized form of \(\mathbf{x}_{i}\). The set \(\mathcal{A}_{i}:=\mathcal{I}\backslash\{i\}\) is the contrastive samples' set. The dot product is represented by the symbol \(\cdot\), and \(\tau>0\) acts as the temperature factor. In this context, the \(i\)-th sample is labeled as the anchor, the \(j(i)\)-th sample is considered a positive sample, and the rest of the samples, totaling \(N-2\), are deemed negative in relation to the \(i\)-th sample. In the context of feature representation \(\mathbf{z}_{i}\) and classifier \(\mathbf{\theta}_{i}\) for a given input example \(x_{i}\), the goal is to align the softmax transformation of \(\mathbf{\theta}_{i}\cdot\mathbf{z}_{i}\) with the actual label of \(\mathbf{x}_{i}\). The column of \(\theta_{i}\) that corresponds to the true label of \(x_{i}\) is represented as \(\theta_{i}^{*}\). The objective is to maximize the dot product \(\theta_{i}^{*}\cdot z_{i}\), thereby enhancing the representation of both \(\theta_{i}\) and \(z_{i}\) with supervised learning. To achieve this, the dual contrastive loss is introduced. This loss function aims to maximize the dot product \(\theta_{i}^{*}\cdot z_{j}\) when \(x_{j}\) shares the same label as \(x_{i}\), and minimize it when \(x_{j}\) has a different label.