# The Impact of Cause and Filler Word Encoding on Dementia Detection with Contrastive Learning
## 3 Results
### Contrastive Learning  
In this section, the results of using CL to perform the classification are presented. For this, we used the DualCL framework as introduced in Section 2.3.2. For more clarity, samples in the same class are considered positive, while if they belong to different classes, they are considered negative samples. We used 20-fold cross-validation with 20 epochs for the evaluation of the model performance. We used the Adam optimizer [38] with default parameters. For hyperparameters \(\lambda\) and \(\alpha\), we chose 0.75 and 0.5 after performing a grid search in the range [0-1].  
The first two rows in Table 4 show the impact of CL on the base model \(B_{0}\). Compared to the baseline, \(B_{0}\), the CL model achieved a roughly 30% improvement across both metrics, which shows the effectiveness of the CL approach. Keeping the filler words (third row) also improves accuracy, and f1-scores by an additional 1%. Compared to the previous results, we achieved a new best accuracy of 86%, which is an improvement of 1%.  
Since the dataset for training is small, we explored augmentation techniques to improve model performance. Similar augmentation techniques as used in previous sections were used, but they did not improve the model's performance. So, they are not reported in this paper. In Table 4 (rows 4 and 5), a contextualized embedding augmentation [42] was used to generate more samples. In particular, we used the BERT model for this contextualized augmentation. The choice of augmentation is very important, given the fact that too many changes can affect performance negatively due to the nature of the dataset and task. For training, three augmented samples per ground-truth sample were generated. In this case, when uh/um was not included, the model had a better performance in accuracy, better than all the results before. This might be due to the fact that augmentation imposes some changes, which in combination with uh/um impact the classification negatively. Compared to the results where no augmentation was utilized, the accuracy improved by 1% and the f1-score stayed the same.  
Previously, we observed that combining pause and filler word encoding improved performance. It is only natural to combine these results with CL. For this, we chose the best result from Table 3 phase 3, namely, \(E_{I_{1},S_{0},F_{1},V_{1}}\) (\(Pause(E_{I_{1},S_{0},F_{1},V_{1}})\) and \(Filler(E_{I_{1},S_{1},F_{0},V_{1}})\)), to apply CL. Table 4 (last two rows) shows the results for this encoding. We observe a 2% increase in accuracy for this model.  
In Table 5, we take a closer look at the impact of CL on pause encoding by repeating the experiments for Table 2 with CL present. An immediate conclusion that can be drawn from this table compared to Table 2 is that average accuracy has increased by 3%, which shows improvement in performance over all the encodings. In terms of the f1-score, the average stayed the same, which is due to some improvements but also some drops in performance. We also note that the variances in Table 2 are lower than the variance in Table 5. This is an indication that adding CL decreases the stability of the model.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Input Type** & **Acc.** & **F1** \\ \hline \(B_{0}\) & 0.56 \(\pm\) 0.11 & 0.42 \(\pm\) 0.27 \\ \(B_{0}\) + CL & 0.85 \(\pm\) 0.04 & 0.84 \(\pm\) 0.04 \\ \(B_{0}\) with Fillers + CL & 0.86 \(\pm\) 0.04 & 0.85 \(\pm\) 0.04 \\ \hline \(B_{0}\) + CL + Aug & 0.87 \(\pm\) 0.05 & 0.85 \(\pm\) 0.05 \\ \(B_{0}\) with Fillers + CL + Aug & 0.85 \(\pm\) 0.05 & 0.84 \(\pm\) 0.05 \\ \hline \(Pause(E_{I_{1},S_{0},F_{1},V_{1}}),Filler(E_{I_{1},S_{1},F_{0},V_{1}})\) & 0.85 \(\pm\) 0.06 & 0.86 \(\pm\) 0.06 \\ \(Pause(E_{I_{1},S_{0},F_{1},V_{1}}),Filler(E_{I_{1},S_{0},F_{1},V_{1}})\) + CL & 0.87 \(\pm\) 0.08 & 0.86 \(\pm\) 0.09 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results showing impact of contrastive learning (CL) and data augmentation (Aug) on different models.