RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Accurate and detailed analysis of clinical interviews is essential for effective psychodynamic diagnostics. In particular, Operationalized Psychodynamic Diagnostics (OPD) interviews Force ( 2008 ) serve as a cornerstone in psychodynamic assessment.
A key aspect of OPD is the assessment of the patient’s life-determining, often unconscious inner conflicts, such as conflicts relating to Dominance or Submissiveness, or to Self-value/esteem.
Automated analysis of psychodynamic conflicts from clinical interviews has the potential to support clinicians, reduce manual work, enhance objectivity, and may even lay the groundwork for the delivery of diagnostic interviews by artificial agents.
However, due to their long duration, low level of standardisation, and richness of information, semi-structured interviews pose unique challenges Adams ( 2010 ); Magaldi and Berler ( 2020 ) .
Prior natural language processing (NLP) work has often focused on short interview excerpts and broad diagnostic categories Low et al. ( 2020 ); Milintsevich et al. ( 2023 ) .
To the best of our knowledge, no previous work has addressed the recognition of fine-grained psychodynamic concepts from long semi-structured diagnostic interviews.

In this work, we introduce a novel approach for recognising the presence and significance of psychodynamic conflicts as classified in the OPD from full-length interviews.
Our method combines recent advancements in parameter-efficient fine-tuning Hu et al. ( 2022 ) and Retrieval-Augmented Generation (RAG) Lewis et al. ( 2020 ) with a summarisation approach in order to process and classify long (90 min) semi-structured psychodynamic diagnostic interviews.
In particular, we make use of a RAG framework to let the LLM access full-length interviews.
To allow the model to effectively reason about the interview to be scored, we additionally prompt it with a summary of the interview.
The classification is performed by an ensemble of four models, each of which was fine-tuned to analyse a specific temporal portion of an interview.
To evaluate our approach, we make use of a dataset of 141 OPD interview recordings Bock et al. ( 2016 ) .
Our approach consistently improves over baselines and ablation conditions.
It is able to reach weighted F1 scores of 0.78 and 0.81 for the conflicts Self-dependency and Dependency on Others , and Dominance or Submissiveness .
For the more challenging conflicts Self-sufficiency and Self-value/esteem , it is able to reach 0.59 and 0.58 F1, respectively.

Our specific contributions are threefold:

We present AutoPsyC 1 1 1 Code available at https://git.opendfki.de/philipp.mueller/autopsyc , the first LLM-based method for the recognition of presence and severity of psychodynamic conflicts from full-length OPD interviews, thereby bridging the fields of psychodynamic diagnostics and advanced NLP.

We evaluate AutoPsyC on a dataset of 141 90 minute long OPD interviews, showing that AutoPsyC consistently outperforms all baselines and in-depth ablation comparisons.

We demonstrate that information contained in the middle sections of interviews is particularly informative for classifier training.
Diagnostic Interviews in Psychotherapy and Psychiatry
Structured interviews, using standardized questions and scoring, improve psychiatric diagnosis reliability by reducing clinician bias. Tools like the Structured Clinical Interview for DSM-5 (SCID-5, First et al. ( 2016 ) ) ensure DSM-aligned accuracy but require extensive training, while the Mini-International Neuropsychiatric Interview (MINI, Sheehan et al. ( 1998 ) ) offers efficient screening at the cost of some diagnostic precision.
The Structured Interview of Personality Organization (STIPO) is a structured interview designed to assess personality functioning based on Kernberg’s object relations theory Clarkin et al. ( 2007 ) .

Unstructured interviews emphasize patient narratives and clinical intuition, enabling the exploration of unique experiences Nordgaard et al. ( 2013 ) . While fostering rapport and uncovering insights, their lack of standardization introduces variability and reduces reliability Shea ( 2016 ); Corbin and Morse ( 2003 ); O’Brien and Tabaczynski ( 2007 ); Widiger ( 2008 ); Fava et al. ( 2024 ); Lenouvel et al. ( 2022 ) . In this context, the PDM-2 provides a multidimensional diagnostic framework that emphasizes psychological functioning and personality organization over categorical symptom-based diagnosis Lingiardi et al. ( 2015 ) . Likewise, Malan’s triangles offer a conceptual model for understanding intrapsychic conflict and resistance, rather than a formalized interview procedure Malan ( 1979 ) .

Semi-structured interviews blend the structure of standardized questions with the flexibility to address emergent themes Fava et al. ( 2024 ); Lenouvel et al. ( 2022 ); Adams ( 2010 ); Brinkmann ( 2014 ); Magaldi and Berler ( 2020 ); Adeoye-Olatunde and Olenik ( 2021 ) . They have been shown to be particularly useful in complex cases like major depressive disorder Dupuy et al. ( 2020 ) . One example for a sem-structured format is the Core Conflictual Relationship Theme (CCRT) method to identify recurring interpersonal themes Luborsky and Crits-Christoph ( 1998 ) .
Operationalized Psychodynamic Diagnosis (OPD) uses semi-structured methods rooted in psychodynamic theory to assess self-experience, interpersonal relationships, and unconscious conflicts Force ( 2008 ); Cierpka et al. ( 2007 ) .
Unlike symptom-focused tools, OPD provides in-depth insights into personality organization and internal dynamics, aiding personalized therapeutic interventions.
Research has demonstrated the clinical relevance of OPD within therapeutic settings Cierpka et al. ( 2007 ); Benecke ( 2024 ); Cierpka et al. ( 2001 ); Rudolf et al. ( 2004 ) .
Despite their importance, the automatic analysis of semi-structured interviews remains under-explored.
In particular no previous work has attempted to automatically score OPD interviews.
Large Language Models for Psychiatric Diagnosis
The integration of NLP and machine learning in different aspects of mental health is a rapidly growing field of research Le Glaz et al. ( 2021 ); Lindsay et al. ( 2021 ); Hossain et al. ( 2024 ) .
One particular focus of attention is the automated diagnosis of conditions such as depression or schizophrenia by analysing text, speech, and facial expressions Barzilay et al. ( 2019 ); Low et al. ( 2020 ); Kishimoto et al. ( 2022 ); Oh et al. ( 2024 ); Milintsevich et al. ( 2023 ); Ettore et al. ( 2023 ) .
Tools like Diagnostisches Expertensystem für psychische Störungen (DIA-X-5) are being tested for consistency Hoyer et al. ( 2020 ) , while patient involvement is emphasized to ensure ethical use Brederoo et al. ( 2021 ) .

Recent work has shown that LLMs can be utilised to analyse complex human affect expression in conversation Broekens et al. ( 2023 ); Müller et al. ( 2024 ) , making them a promising candidate for applications in psychiatric disorders.
Indeed, LLMs are increasingly applied in psychiatry, identifying linguistic markers of disorders from social media posts and clinical transcripts Farruque et al. ( 2024 ); Lan et al. ( 2024 ); Zhang et al. ( 2024b ) . These models also assist in parsing unstructured EHR notes for early diagnosis Zhang et al. ( 2024b ) .

However, current automatic methods mainly detect broad disorder categories without capturing
the more detailed and often unconscious factors explored in psychodynamic assessments.
In particular, we are not aware of any approach to automatically recognise psychodynamic conflicts from clinical interviews.
Integrating Domain Knowledge in Large Language Models
The integration of domain-specific knowledge into LLMs, particularly through techniques like Retrieval-Augmented Generation (RAG), is transforming psychiatric applications by enabling models to access and apply current, relevant external information in real time Lewis et al. ( 2020 ) . Building on RAG, RAFT (Retrieval-Augmented Facilitation for Text) further optimizes domain-specific knowledge integration by prioritizing the most relevant medical literature and clinical guidelines Zhang et al. ( 2024a ) .
Other methods, such as parameter-efficient domain knowledge integration and the use of human-annotated features, also improve LLM performance in biomedical contexts Ke et al. ( 2021 ); Kim et al. ( 2024 ) .
In our work, we utilise RAG techniques to present the first LLM-based system able to recognise psychodynamic conflicts expressed in full-length OPD interviews.
Method
A schematic overview of our method is shown in Figure 1 .
The classification of a test interview consists of three steps.
In the first step, we employ LLaMA 3.1 (8B) Grattafiori et al. ( 2024 ) to generate a summary of the interview.
In the second step, we use this summary to build a prompt for the final classification, which is performed by an ensemble of LLaMa 3.1 (8B) models that are fine-tuned to different interview segments.
The prompts for these specialised models contain 5 interview summaries from the training set, including ground truth (few-shot examples).
Via a RAG framework Lewis et al. ( 2020 ) , each specialised model also has access to relevant sections of the OPD manual, as well as to the full test interview and all full interviews from the training set.
In the third step, we combine the individual classifications obtained from the specialised models using a weighted voting scheme driven by a multinomial logistic regression.
Summarization Method
To obtain a focused representation of each interview, we first generate a summary using a LLaMA 3.1 8b model Grattafiori et al. ( 2024 ) . The summarization prompt includes an example summary excerpted from the OPD Manual, instructing the model to adhere to a consistent style that reflects the diagnostic criteria. In this way, the generated summaries capture the diagnostically relevant information while filtering out extraneous details.
Training Data Integration and RAG Setup
In addition to the test interview summary, the classification prompt also includes few-shot examples in the form of summaries of interviews from the training set with associated ground truth labels.
We include one interview summary per ground truth class.
To further ground the classification in a domain-specific context, we employ a Retrieval-Augmented Generation (RAG) framework.
In particular, we upload the following information into the RAG vector database.

Training Interviews: We upload the full transcripts of all interviews from the training set without ground truth into the RAG knowledge base, pointing the model to them for retrieval. Adding ground truth information did not lead to improvements in preliminary experiments.

Test Interview: We also upload the full transcript of the test interview.

OPD Manual: The OPD-2 manual Force ( 2008 ) is organized into chapters corresponding to its axes, providing detailed descriptions and examples of OPD tasks and classifications. For our purposes, we included excerpts from the chapter on conflicts (Axis III) and the introductory section where the axes are defined and explained. Preliminary tests showed that incorporating the entire OPD-2 manual into the retrieval-augmented generation (RAG) system did not improve model performance compared to using only the relevant excerpts.

This integration ensures that the classification model benefits from exemplars of each diagnostic class and explicit domain knowledge.
The model is also able to access detailed information present in the full interview transcripts in case the summaries are inconclusive.
Classification Stage: Interview Segmentation and Finetuning
Given that OPD interviews are semi-structured—with diagnostic cues distributed throughout—we split each interview (or its summary) into k segments, where in our implementation k=4 (each segment being roughly 5,000 words).
In this way, each segments represents one quarter of the interview.
For each segment, we fine-tune a separate Llama 3.1 (8B) model using LoRA Hu et al. ( 2022 ) , which allows for parameter-efficient adaptation. During training, we provide the model with a prompt including the segment summary, the RAG-augmented context (i.e., training interview summaries and manual excerpts), and an example for each of the five classes. The model is trained to output a probability distribution over the diagnostic classes. This process allows each fine-tuned model to capture the specific semantic and contextual nuances present in its corresponding interview segment.
Result Aggregation
After obtaining classification probabilities from each of the four fine-tuned models, we combine their outputs using a weighted voting scheme. Specifically, we train a multinomial logistic regression model that assigns a weight w_{i} to the prediction p_{i}(c) of the i -th segment for class c . The final predicted class \hat{y} is computed as:

\hat{y}=\arg\max_{c}\sum_{i=1}^{k}w_{i}\,p_{i}(c) (1)

where k=4 in our implementation.
Dataset
The Kassel dataset Bock et al. ( 2016 ) , utilized in this study, comprises 141 participants recorded during Operationalized Psychodynamic Diagnostics (OPD) interviews.
Participants
The dataset includes both male (n = 21) and female (n = 120) participants, aged between 18 and 57 years.
Among them, 64 were inpatients diagnosed with at least one DSM-IV Association ( 2000 ) disorder, while 20 were healthy controls. The remaining participants had diverse diagnostic categories, including somatoform disorders (n = 22), borderline personality disorder (n = 19), depression (n = 18), and eating disorders such as anorexia (n = 14) and bulimia (n = 14). Anxiety disorders were observed in 13 participants. The inclusion criteria required informed consent, age above 18 years, and the absence of acute psychosis or schizophrenia.
Data Collection
Each participant underwent a clinical interview based on the framework of Operationalized Psychodynamic Diagnostics Force ( 2008 ) . The interviews were carried out by a team of two male and two female interviewers Bock et al. ( 2016 ) , all of whom were certified and trained in OPD application. The sessions, with an average duration of approximately 90 minutes, were recorded using split-screen technology to capture both the participants and interviewers. Both the interviewer and the interviewee were equipped with microphones to ensure clear audio capture. The audio from each session was transcribed verbatim into text by research staff or trained transcribers Bock et al. ( 2016 ); Vierl et al. ( 2023 ) .
These recordings provided the foundational data for subsequent analyses of behavioural and contextual elements.
Clinical Ground Truth
The dataset includes scores for Axes I-V of the OPD system.
In the scope of this paper, we focus on Axis III, which captures the patient’s life-determining (un)conscious inner conflicts.
In particular, these are: Conflicts related to self-dependency and dependency on others , Conflicts associated with dominance or submissiveness , Conflicts revolving around self-sufficiency , Self-value and self-esteem conflicts , Oedipal conflicts , Identity-related conflicts .
Each of these conflicts is rated with a five-class classification scheme.
The classes are Not assessable, Not present, Of little significance, Significant, Very significant . Their detailed description can be found in A.3 .

We decided to omit the conflicts Oedipal Conflict and Conflicts Related to Identity from further analysis, as these conflicts were diagnosed in only a few instances, making a robust evaluation of predictions infeasible.
For instance, in the case of Oedipal Conflict , 120 out of 141 instances were labeled as not present (see Appendix for further details). Figure 2 illustrates the class distribution of the remaining conflicts.
While not present is the most prevalent class for every conflict except Self-sufficiency, in all cases a significant portion of participants exist for whom the respective conflict is at least present with little significance.
In the following we provide a concise explanation of the four conflicts included in our analysis.

Conflicts related to self-dependency and dependency on others refer to the tension between striving for autonomy and seeking support from others, often leading to struggles between independence and fear of isolation. Conflicts associated with dominance or submissiveness involve power dynamics in interpersonal relationships, where individuals may oscillate between asserting control and yielding to authority, potentially resulting in power struggles or passivity. Conflicts revolving around self-sufficiency pertain to the balance between the need for care and the desire for independence, with individuals experiencing inner turmoil when their reliance on others contradicts their self-reliance. Self-value and self-esteem conflicts center on an individual’s sense of worth, encompassing struggles with feelings of inadequacy or inferiority, often manifesting in compensatory behaviours aimed at reinforcing self-image. The conflicts were classified based on the overall interview.
When binarizing the conflicts by treating not assessable and not present as “No,” and all other categories as “Yes,” only three cases did not present with any of the conflicts.
Experiments
In this section, we describe the evaluation protocol and baselines.
Evaluation Protocol
The dataset was partitioned into five fixed folds using stratified 5-fold cross-validation to maintain the proportional representation of key demographic and diagnostic variables. This was achieved by utilizing the StratifiedKFold module from the scikit-learn library Pedregosa et al. ( 2011 ) , with stratification based on Gender and Diagnosis . The stratification process guarantees a fair distribution of these attributes across all folds. This consistency was preserved across all experiments involving the 5-fold cross-validation framework.

To evaluate all models and baselines, we make use of the weighted F1 score. The weighted F1 score accounts for class imbalance by weighting classes proportionally to their prevalence, ensuring robust evaluation of both frequent and rare diagnostic categories. It balances precision (avoiding overpathologizing) and recall (preventing missed conflicts), aligning with clinical priorities.
To robustly estimate performance, we repeated all experiments several times and report the average weighted F1 score across all runs.
In the case of experiments involving LLMs, we average across 100 runs, and in the case of the computationally less expensive baseline experiments, we average across 1000 runs.
In addition to the averages, we also report their 95% confidence interval.
Baselines
We implement two simple baselines: a Demographic Baseline and a Random Baseline .
Demographic Baseline.
This baseline employs a neural network classifier using demographic attributes such as gender, clincial diagnosis group, and binned age as input features. Numerical features were normalized by subtracting the mean and dividing by the standard deviation, while categorical features were converted into numerical representations. The neural network, implemented in PyTorch Paszke et al. ( 2019 ) , consists of three fully connected layers with ReLU activations. It was trained for 30 epochs using cross-entropy loss and the Adam optimizer.
Random Baseline.
The random baseline leverages the DummyClassifier module from scikit-learn , configured with the stratified strategy. This classifier generates predictions by randomly assigning labels based on the class distribution of the training set. This random classifier serves as a naive baseline, highlighting the minimum expected performance for the classification task.
Overview
Table 1 summarises our weighted F 1 scores across the four psychodynamic conflicts. To more easily navigate the table, we partition the different ablation conditions into three cases, based on whether unlabelled training data is encorporated in the RAG framework and based on whether fine-tuning is performed with labelled training data.
Ablations are always named relative to the partition their are in.
For example, the ablation w/o Manual in the partition No Training Data in VDB, No Fine-tuning describes an ablation condition without training data integration into the RAG framework, without fine-tuning, and without integration of the OPD Manual in the RAG framework.

We observe that our full method (AutoPsyC), which combines our summarisation strategy with weighted voting across fine-tuned, temporally specialised models, as well as domain knowledge integration into the RAG framework, consistently outperforms all baselines and ablation conditions.
As illustrated in Figure 3 , models fine-tuned on the middle segments of the interviews consistently outperform those focusing on earlier or later sections.
Moreover, Figure 4 indicates that deviating from four total models or partitioning the interviews into fewer or more than four segments leads to a noticeable drop in overall performance.
Finally, we present an analysis of gender fairness of our model.
Overall low values of Conditional Demographic Disparity (CDD) indicate no major gender-related biases ( Table 2 ).
Which Model Configuration Works Best?
Our experimental results demonstrate the effectiveness of combining our summarisation strategy with Retrieval-Augmented Generation (RAG), instruction tuning and section-wise model specialization for psychodynamic conflict classification in clinical interviews.
As can be seen in (Table 1 ), AutoPsyC achieves superior performance across all four conflict categories, with weighted F1-scores ranging from 0.58 to 0.81. This represents a substantial improvement over both naive baselines (Demographic: 0.26–0.46; Dummy: 0.20–0.33) and non-instruction-tuned variants (0.50–0.74).

Our detailed ablation experiments indicates that AutoPsyC effectively integrates all available information.
We can observe a large decrease in performance when the test interview transcript is removed from the vector database (0.50-0.73 F1).
This indicates that our model indeed makes use of the full test interview transcript that is provided via the RAG framework to fill in information missing in the interview summary.
At the same time, we see that it does profit from the test interview summary, with losses of up to 0.04 F1 when the summary is removed.
We furthermore observe a clear loss in performance when the OPD manual is removed from the vector data base (0.52-0.74 F1), and a slightly lower loss in performance when the training set interviews are removed from the database (0.55-0.77 F1).
This indicates that even when using fine-tuned classification model, domain knowledge integration via the RAG setup is still helpful.
The weighted voting mechanism using multinomial logistic regression provides moderate but consistent performance gains (0.58–0.81 F1 vs. 0.55–0.78 for unweighted aggregation), suggesting that different interview sections contribute asymmetrically to conflict identification.

One general observation we can make is that fine-tuning leads to greater robustness w.r.t. other ablation conditions.
E.g. removing the OPD Manual from the vector database leads only to a moderate loss in performance when fine-tuned classification models are used (0.58-0.81 F1 vs. 0.52-0.74 F1).
In contrast, for the case of no fine-tuning, the losses are more dramatic (0.50-0.74 F1 vs. 0.43-0.61 F1).
Which Interview Section is most useful?
To further investigate which sections of the interviews are most informative fine-tuning classification models, we investigate the performance of our four pretrained models singled out across all conflicts (see Figure 3 ).
The results indicate that the models fine-tuned using the middle sections of the interviews outperform those tuned with other sections. After a careful examination of the interviews, we observed that, in the quarter 2 & 3, the interviewees often share information more closely related to their condition and situation.
An excerpt of the interview can be found in Appendix A.1 .