# The Impact of Cause and Filler Word Encoding on Dementia Detection with Contrastive Learning
###### Abstract  
Dementia is primarily caused by neurodegenerative diseases like Alzheimer's disease (AD). It affects millions worldwide, making detection and monitoring crucial. This study focuses on the detection of dementia from speech transcripts of controls and dementia groups. We propose encoding in-text pauses and filler words (e.g., \"uh\" and \"um\") in text-based language models and thoroughly evaluating their impact on performance (e.g., accuracy). Additionally, we suggest using contrastive learning to improve performance in a multi-task framework. Our results demonstrate the effectiveness of our approaches in enhancing the model's performance, achieving 87% accuracy and an 86% f1-score. Compared to the state of the art, our approach has similar performance despite having significantly fewer parameters. This highlights the importance of pause and filler word encoding on the detection of dementia.  
dementia; contrastive learning; deep learning; text classification; LLMs; NLP +
Footnote †: journal: Medical  
through LLMs has become very popular due to their powerful ability to capture complex language patterns and enhance model performance [12; 13; 14; 15]. The authors in [14] proposed a sentenced-based pipeline that integrates various augmentation techniques, pre-trained LLMs, and classifiers to perform classification tasks. They thoroughly evaluated each augmentation technique, explored a range of pre-trained models, and experimented with different classifiers, including CNNs and RNNs. Additionally, the authors implemented various voting mechanisms to refine their results, presenting a detailed analysis of each component's impact on the overall performance. Other studies have introduced innovative methods, such as contrastive learning (CL), which involves learning from the data themselves to separate the feature space for each class [16; 17]. In [16], the authors propose a multimodal approach which uses both text and audio using graph neural networks (GNNs) with CL, and study the impact of both modalities. Also, the authors in [18; 19] explore the use of Transformers and attention-based approaches combined with CNNs and RNNs. Due to the limited data available in this field, several authors have proposed various augmentation techniques to mitigate this issue [20; 21; 22]. In [21], the authors proposed a generative approach for data augmentation by using variational auto-encoders (VAEs). Instead of raw text or speech, they augmented the lingual and acoustic features. These augmentation methods aimed to increase the dataset size, enhancing the model's ability to generalize and perform better [23].  
In [12; 24], the authors proposed a technique where pause information is encoded within the text, a feature typically used in speech-based analysis. Pauses are introduced using special characters, enabling the language model to recognize these features, which has been shown to enhance model performance. In [12], the authors utilized the ADReSS dataset [25] for their experiments and employed temporal word alignment to implement their methodology. Building on this idea, we apply a similar concept to the Pitt Corpus Cookie Theft dataset [26].  
In this paper, we propose a new methodology using LLMs for the detection of dementia from transcriptions of speech by encoding in-text pauses and filler words, and study the impact on the performance of contrastive learning. We hypothesize and validate that the inclusion of these encodings and modeling schemes lead to a significant impact on performance, starting at 57% accuracy for a baseline model and increasing to 87% by our final model. This paper is organized as follows: In Section 2, the main materials and methods used are explained, which include dataset preparation, in-text pause encoding, and the contrastive learning scheme. In Section 3, the results for the experiments are presented. In Section 4, our results are discussed. Finally, in Section 5, the paper is concluded.