RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Methodology
This work proposes a novel methodology, guided by the intuition that one of the fundamental qualities of a therapist is empathy . This direction aims to synthesize empathetic responses based on a broader understanding of affective language, circumventing the need for sensitive, real-world conversational data, enabling the model to detect emotions, and create explainable emotional profiles that can be useful for mental health assessment.
Datasets
In this work, three main sources of open-source data were used: Empathetic Dialogues Dataset Rashkin et al. ( 2018 ) : it has been used to train the RACLETTE model to identify emotions and respond empathically. This dataset is a large-scale multi-turn empathetic dialogue dataset collected on the Amazon Mechanical Turk, containing 24,850 one-to-one open-domain conversations. This dataset was selected for this task because, other than its high quality and appropriate size, it considers a much wider range of emotions compared to all other available datasets, which usually consider only a very limited subset (5-8) of fundamental emotions Zahiri and Choi ( 2017 ); Li et al. ( 2017 ) . Reddit Mental Health Dataset Low et al. ( 2020 ) : a collection of posts from specific Reddit forums (also called subreddits, Table 3 shows all the subreddits considered) have been used to construct the discrete distributions associated with each mental disorder to extract emotion embeddings. DailyDialog Dataset Li et al. ( 2017 ) : a collection of posts used to establish a control group for the emotional profiles assessment. It contains 13,118 dialogues split into a training set with 11,118 dialogues and validation and test sets with 1,000 dialogues each.

Figure 3 illustrates the RACLETTE workflow pipeline, highlighting the specific use of each dataset at various stages of the process. This visual representation provides a clear overview of how the different datasets are integrated into the system’s architecture, from training the empathetic model to extracting emotion embeddings and conducting comparative analyses.
Tailoring an LLM to React Empathetically
For this study, we chose to fine-tune the Mistral 7B model Jiang et al. ( 2023 ) , a 7-billion-parameters state-of-the-art LLM, known for its great performance combined with both computational and memory efficiency. This approach aligns with recent findings from Sekulić et al. ( 2024 ) , which demonstrate that fine-tuning LLMs on task-oriented dialogue data can reduce hallucinations.

The Empathetic Dialogues Dataset is purposefully formatted using a specific structure that allows to fully leverage the causal attention mask of the transformer decoder model to generate tokens for both the empathic response prediction and dialogue emotion detection tasks. Placing prompts before emotion labels enforces the autoregressive property of the model during training and inference Sun et al. ( 2023 ) , allowing the generative model to be used both as a classifier and a conversational agent.
The model learns to predict the next tokens by only attending to previous positions in the sequence in order to generate predictions sequentially.
Let P(y_{<emotion>}|y_{1},y_{2},\ldots,y_{N}) be the probability of the model predicting the emotion token y_{<emotion>} given the sequence of previous tokens y_{1},y_{2},\ldots,y_{N} , then the model’s objective can be defined as:

\max\sum_{t=1}^{N}\log P(y_{<emotion>}|y_{1},y_{2},\ldots,y_{N})

where N is the length of the sequence.

When generating the prediction of an emotion, the model iteratively produces the tokens that are more likely, given the previous tokens (see Figure 2 ). Unlike deterministic methods, this process can be guided to generate a diverse set of emotions by iteratively sampling over the predicted probability distribution of all tokens in the vocabulary. In this implementation, Top-K Sampling is used Holtzman et al. ( 2019 ) , which limits the sampling pool to the top-K most probable tokens, in this case, top-10 , balancing diversity with relevance. Then to generate multiple emotions, this process is repeated 10 times independently for each prompt.
Let V be the vocabulary and K be the sampling parameter:

TopK(P(y_{t}|y_{<t}),K)={y_{i}\in V:P(y_{i}|y_{<t})}

is among top-k probabilities.

These empirical distributions are aggregated across the entire conversation to obtain the emotional profile of the speaker.
Let C be the set of all prompts in a conversation and e_{i,k} the sampled emotion ( K=10 samples in total) for prompt i :

EmotionalProfile=\frac{1}{|C|}\sum_{i\in C}\frac{1}{K}\sum_{k=1}^{K}e_{i,k}

For this study, an unconventional 3-turns structure was implemented (see Figure 1 for an example). It can be summarized as Prompt, Emotion, and Response, separated by the special tokens: 
,
,
 and <endoftext> .

When predicting empathetic responses, the model will attend to the previous tokens in its context, (1) the whole history of the conversation, (2) the current prompt followed by the emotion, and learn to generate the appropriate reply as seen in the training dataset.
Emotion Recognition and Empathetic Response
Table 1 shows the results of RACLETTE in detecting the correct emotion for each conversation both at prompt and conversation levels.
The results related to individual prompts refer to the correct classification of emotions for single conversation turns. Regarding the conversation, this approach progressively concatenates each prompt, its predicted emotion, and subsequent response, thus enriching the model’s contextual awareness with each conversational turn. The accumulated emotion distributions for each prompt contribute to a more precise classification, resulting in a 3% increase in accuracy (from 56% to 59%). Notably, this methodology enhances accuracy, as the expanding conversational context provides more information for discerning the speaker’s emotional profile.

Out of approximately 10.9k utterances present in the test set, the report categorizes 5,242, as the classification is made solely on the speaker’s contributions.
In addition, to evaluate empathic replies to each of the speaker prompts, this table includes the BERTSCORE Zhang et al. ( 2019 ) , an automatic evaluation metric for text generation. Unlike traditional metrics that rely on exact word matches or n-grams, BERTSCORE evaluates the similarity between predicted and target replies by analyzing contextual embeddings of tokens obtained with the BERT model. This approach allows for a semantical understanding of the model’s performance, capturing the comparison of empathetic responses beyond mere lexical matching. Notably, a BERTSCORE of 0.87
indicates high semantic similarity between the responses given by the model and the target replies contained in the test set that were given by the human listeners.

Table 2 compares the overall emotional accuracy of RACLETTE with the accuracy of CAiRE, as reported by Lin et al. ( 2020 ) .
For completeness, in addition to CAiRE, we present other baselines from the literature that have used the same dataset to evaluate their performance.
As benchmarks for emotion classification accuracy, we consider the following approaches: (1.) In Chen et al. ( 2024 ) , authors propose a cause-aware empathetic generation method using Chain-of-Thought fine-tuning on Large Language Models. (2.) In Li et al. ( 2022 ) , authors introduced a knowledge-enhanced empathetic dialogue generation method incorporating external knowledge and emotional signals. (3.) The approach from Gao et al. ( 2021 ) , who proposed incorporating emotion cause recognition into empathetic response generation using an emotion reasoner and gated attention mechanism.

We report the accuracy value as presented in works that have used the same dataset, as shown in their respective manuscripts.

Table 2 shows how RACLETTE outperforms the benchmarks considered.
Also in this case, it can be observed that the choice of fine-tuning a generative model, leveraging its autoregressive characteristics for classification, leads to the best results.
Reddit’s Emotion embeddings applied to the Detection of Suicide Risk
This experiment aims to evaluate the use of emotion to create mental state embedding as a mechanism for diagnosing the potential risk of suicide. For lack of a professionally labeled dataset, this experiment, like the previous one, focuses on Reddit’s users.
To compare our approach with other related tasks and methods, we have built a dataset for binary classification of general conversation text versus suicidal text. We used the two subreddits CasualConversation and SuicideWatch, where CasualConversation is a subreddit for general conversation, and has generally been used by other methods as data for a clinically healthy class in other works Haque et al. ( 2021 ); Shen and Rudzicz ( 2017 ) . This dataset is part of a larger collection available on Kaggle 1 1 1 https://www.kaggle.com/datasets/suicide-watch , which has been carefully cleaned to ensure the reliability of the data. We select 5% of the initial samples at random as a test set ( \approx 10,585 samples). After inspecting posts for anomalous length deviations from the average, those lacking informative content are removed.

To compute the sample embeddings, each post is divided into sentences, with 10 emotions predicted per sentence. The final embedding aggregates these emotions across all sentences in the post, following a similar approach as used in Section 5 to extract reference embeddings.

The methodology encompasses the use of three distinct metrics for comparing embeddings: Kullback–Leibler (KL) divergence, Jensen–Shannon (JS) divergence, and Cosine Similarity (CS).
Our approach is based on the use of emotional profiles embeddings that are most closely associated with an elevated risk of suicide and match them against the user’s emotional profile.
Table 3 compares various emotion embeddings, focusing on their differences from the suicide embedding, measured by KL, JS, and CS. The results show emotional proximity between Suicide and Depression, as well as Borderline Personality Disorder, followed by Bipolar Disorder, Addiction, PTSD, and Schizophrenia. This pattern aligns with psychological insights that these mental health conditions are often linked to an increased risk of suicidal tendencies (Song et al., 2020 ) .

For these reasons, this experiment will focus on the use of these specific emotion embeddings, considering the embedding that is most similar to what is obtained by processing the sample post, and mapping it to the predicted label, in an unsupervised fashion, as follows:

Positive labels: suicide, depression, borderline personality disorder (BPD), bipolar disorder, PTSD, addiction, and schizophrenia. By combining these particular embeddings, the study aims to capture a spectrum of characteristic emotional patterns that are potentially indicative of an elevated risk of suicide.

Negative labels: normal and uniform distributions, where normal is obtained from the Daily Dialogue dataset, and embedding obtained from Casual Conversation’s subreddit are also used as the control group.
Results for Mental Health Classification
Table 4 shows the performance metrics: precision, recall, F1 score and accuracy. For each similarity metrics — Kullback–Leibler divergence, Jensen–Shannon divergence, and Cosine Similarity — we also introduce a combined method, where if any of these methods detect a risk of suicide, the label is assigned as positive. This experiment is designed to maximize recall, a critical metric in scenarios where missing a positive instance has severe consequences, thus reducing the emphasis on precision and false positives. As the quantitative results show, this approach achieves high recall at the cost of other metrics.

Additionally, these methods are compared with state-of-the-art unsupervised approaches based on RoBERTa Liu et al. ( 2019 ) and BERT’s Devlin et al. ( 2019 ) embedding representations, before grouping them into two classes using a K-Means clustering approach, as done in Subakti et al. ( 2022 ) .

The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63. Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84. The KL Divergence variant of RACLETTE stands out for its balanced performance, maintaining strong scores across all metrics (precision: 0.71, recall: 0.90, F1: 0.79, accuracy: 0.77). Both JS Divergence and Cosine Similarity methods show similar patterns, with high recall (0.93) but lower precision. The color intensity (green shading) in Table 4 indicates better performance, visually highlighting that RACLETTE’s approaches generally outperform the benchmark models.

A key advantage of this method is that it generates explainable representations and emotion embeddings, which can be visually inspected, providing valuable insights into an individual’s emotional profile.
Conclusions and Future Works
This paper introduces the RACLETTE system, which addresses two critical challenges in mental health support: the need for empathetic AI-driven conversation and reliable assessment tools. By fine-tuning a LLM, we demonstrate that it’s possible to create effective conversational agents that can accurately recognize users’ emotional states while generating high-quality empathetic responses, all while avoiding the use of sensitive clinical data. The system not only achieves state-of-the-art performance in emotion recognition, but also introduces a novel methodology for creating emotional profiles. These profiles, generated by aggregating emotion distributions from user interactions, serve as interpretable markers that can be compared with characteristic patterns associated with various mental health conditions. Our experimental results demonstrate both the system’s effectiveness in maintaining empathetic conversations and its potential as a preliminary screening tool through the analysis of emotional embeddings.

Future work should extend RACLETTE’s capabilities beyond purely supportive interactions to incorporate a broader range of therapeutic approaches. While RACLETTE currently focuses on emotional markers, a more comprehensive model would incorporate cognitive symptoms (memory, attention, perception, etc.) alongside affective components. Our framework can be adapted to such extensions, potentially offering a more holistic assessment that better reflects the complex nature of mental health conditions. By expanding our emotional embeddings to include markers of cognitive functioning, future works could provide more comprehensive profiles that better align with the multifaceted nature of mental health assessment in clinical practice.
Figure 1: An example of how our 3-turns coversation structure has been implemented.; An example of how our 3-turns coversation structure has been implemented.
Figure 2: A visual explanation of how the emotional profile of a user is updated across a conversation and how to extract the final emotion embedding.; A visual explanation of how the emotional profile of a user is updated across a conversation and how to extract the final emotion embedding.
Figure 3: Overview of the main steps of RACLETTE pipeline.; Overview of the main steps of RACLETTE pipeline.
Figure 4: (A) 2-Dimensional representation of mental disorders distribution after applying t-SNE dimensionality reduction. (B) Sorted emotion embedding of depression. (C.) Sorted emotion embedding of DailyDialog.; (A) 2-Dimensional representation of mental disorders distribution after applying t-SNE dimensionality reduction. (B) Sorted emotion embedding of depression. (C.) Sorted emotion embedding of DailyDialog.
(a) r/suicidewatch (1000 samples); (b) r/depression (1000 samples); (c) r/bpd (1000 samples); (d) r/bipolar (1000 samples); (e) r/schizophrenia (1000 samples); (f) r/ptsd (1000 samples); (g) r/addiction (1000 samples); (h) r/alcoholism (1000 samples); Emotional embeddings of subreddits related to high risk of suicide.
(a) r/anxiety (1000 samples); (b) r/socialanxiety (1000 samples); (c) r/healthanxiety (1000 samples); (d) r/lonely (1000 samples); (e) r/adhd (1000 samples); (f) r/autism (1000 samples); Emotional embeddings of different subreddits.
Emotion Classification Report: Evaluated on Individual prompts and Conversations from the Empathetic Dialogues test set.
Emotional accuracy comparison between RACLETTE and other benchmarks (best results highlighted in green).
Comparison of Emotion Embeddings: KL Divergence (KL), JS Divergence (JS), and Cosine Similarity (CS) w.r.t. suicide embedding.
Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.