RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Additional Experiments: How Fair is the Model?
Fairness concerning gender is a critical issue in psychiatric diagnoses when using machine learning algorithms, as biases in training data or model predictions can lead to systematic disparities in diagnostic outcomes.
For instance, a study by Mosteiro et al. ( 2022 ) found that gender played an unexpected role in predictions related to benzodiazepine administration, potentially biasing the model’s decisions.

We evaluate fairness with respect to the gender attribute using Conditional Demographic Disparity (CDD) Wachter et al. ( 2021 ) .
CDD quantifies the difference in expected outcomes across demographic groups, with values closer to zero indicating fairer conditions.

CDD=\mathbb{E}[\hat{y}\midmale,y]-\mathbb{E}[\hat{y}\midfemale,y] (2)

Since we have more than two classes we compute CDD in one-vs.-rest fashion. We compute

CDD_{k}=\mathbb{E}\bigl{[}\hat{p}^{k}\midmale,\,y\bigr{]}\;-\;\mathbb{E}\bigl{[}\hat{p}^{k}\midfemale,\,y\bigr{]} (3)

for class k . This formulation reduces the multi-class problem to a
“one vs. rest” scenario by focusing on a single predicted probability \hat{p}^{k} . If class k is deemed the “positive” class, the
binary-based fairness thresholds from Wachter et al. ( 2020 ) can be applied to \text{CDD}_{k} .

Overall, the CDD values reported in Table 2 are relatively low (all below 0.006), as indicated by prior studies Wachter et al. ( 2021 ); Koumeri et al. ( 2023 ); Wachter et al. ( 2020 ) , suggesting minimal gender-based disparity across the different classes. Wachter et al. ( 2021 ) suggest that CDD values below 0.01 indicate minimal demographic disparity, while Koumeri et al. ( 2023 ) and Wachter et al. ( 2020 ) provide empirical evidence that values above 0.02-0.05 often indicate fairness concerns.
It is important to note that this fairness evaluation is not able to account for potential biases that are already present in the ground truth annotations on the dataset.
Ethical Considerations and Impact
The automation of psychodynamic diagnostics using NLP and machine learning presents both opportunities and ethical challenges. While enhancing objectivity, efficiency, and accessibility, its implementation requires careful ethical scrutiny to ensure responsible use in mental health care. Psychodynamic interviews contain sensitive data, necessitating strong anonymization and compliance with privacy regulations such as GDPR and HIPAA. Additionally, automated diagnostics may reflect biases present in training data, leading to disparities across demographic groups. Continuous bias auditing and fairness assessments are essential to mitigate these risks and ensure equitable model performance.

Automated tools should complement, not replace, human expertise.
AutoPsyC could serve as a supplementary tool for therapists during psychodynamic interviews, acting as a "second-eye" to enhance clinical decision-making American Psychological Association ( 2025 ) .
Additionally, AutoPsyC could be integrated into social interactive agents, chat applications, and telepsychiatry platforms, providing complementary therapeutic support Smith et al. ( 2019 ) . Furthermore, it could be utilized in psychological training tools to enhance the proficiency of conducting psychodynamic interviews American Psychological Association ( 2023 ) .

Psychodynamic diagnostics involve complex interpretations that extend beyond text-based pattern recognition. Thus, model outputs must be interpretable, allowing clinicians to integrate them into their assessments. Future research should prioritize explainability and transparency in AI-driven diagnostics.
As AI applications in mental health expand, concerns arise regarding consent, misuse, and potential stigmatization in non-clinical settings. Interdisciplinary collaboration among clinicians, ethicists, and policymakers is needed to safeguard patient autonomy and well-being.
Conclusion
We present a novel framework for automated conflict classification in psychodynamic interviews, achieving clinically relevant performance through three key innovations: (1) domain-adapted instruction tuning using segmented interview data, (2) RAG-enhanced contextual understanding through OPD Manual and other interview integration, and (3) confidence-weighted aggregation of specialized section models.

These results suggest that LLMs can be effectively adapted for complex psychiatric coding tasks when combined with domain-specific knowledge retrieval and structured interview analysis. The demonstrated technique for identifying diagnostically salient interview segments (quarters 2 & 3) offers methodological insights for computational psychiatry research. Future work should explore applications to other OPD Axes and integration with multimodal clinical data.
Limitations
While promising, our approach has several limitations. First, the dataset size (n=141 interviews) may limit generalizability, particularly for rare conflict subtypes. Second, the complex pipeline (RAG, summarization, 4 specialized models) incurs significant computational costs compared to monolithic models. Third, performance variation across conflict categories (0.58–0.81 F1) suggests task-specific optimization needs, particularly for Self-sufficiency classification.

The reliance on manual OPD Manual examples for summarization introduces potential annotation bias, and the gender fairness analysis does not account for non-binary identities. Additionally, our stratified sampling based on diagnosis and gender may not fully capture all confounding demographic factors. We focus on a single summarization approach, as our primary goal is to establish a proof of concept for automated OPD scoring. While alternative summarization methods could be explored in future work, this choice allows us to maintain methodological consistency and provide a clear baseline for comparison.

We split the interview into four parts based on word counts, which does not fully account for the semi-structured nature of our interviews. Future work could focus on automatically detecting interview segments for the fine-tuning process.

Our fairness analysis showed minimal gender-based disparity in predictions.
There are however many other ways in which our model may be biased.
On the interview dataset we utilised, we did not have access to e.g. information on socioeconomic status or education.
Further variations are such as cultural background are not sufficiently covered by the dataset as it was recorded with German-speaking people in Europe.
This geographic and cultural constraint represents another key limitation of our study.
It remains unclear, whether our approach would also work in very different cultural contexts.

Future research should address these limitations through larger multicentre datasets König et al. ( 2022 ) , lightweight model architectures, and explicit modeling of clinician raters’ variance. The current implementation also requires further validation for real-time clinical deployment, including robustness testing against speech recognition errors and patient dialect variations.
A more detailed investigation of how AutoPsyC handles defensive processes Freud ( 1936 ) remains an area for future research.
Beginning
Interviewer: So, let us begin with the second part. I will ask you various questions from different areas.

Interviewee: Hmm.

Interviewer: About your present life, your past, relationships, work life, etc. Yes. I would like to start by asking you to describe what is currently the most burdensome for you.

Interviewee: At the moment?

Interviewer: Yes, it can be anything.

Interviewee: (Exhales) Ahm. (-) In general, I am actually doing quite well. However, I must say that things that have burdened me in the past, especially over the past three years, have now become less significant. What currently affects me the most is the situation at home. My parents are about to get divorced, and that has been the most difficult thing in my life so far. I must say, it has also been very stressful for me, but I am slowly managing it quite well.
Middle
Interviewee: The period was simply long. I was 20 when it all started, and I would say that only in the last few months have I truly felt lighter inside. For about a year, things have been steadily improving, but before that, I felt terrible. At home, it was a crisis. My mother was struggling—she barely ate, she was just existing. That made me very sad because I am someone who tries to keep everyone together. Given my age, I was able to grasp everything more clearly. I always spoke with everyone, tried to mediate, and made sure we somehow lived through it. But it was simply too much. (Claps hands on the table.)

I do not regret anything, or at least not much, except for the lingering aftereffects, which sometimes scare me. But otherwise, I would do it all over again. It just went too far. There were long periods where I barely met anyone or made any plans. If someone invited me out, I would always say no because I had to check on my mother to see if she was alone.

It was a responsibility that suddenly fell upon me. I would not say that it was forced upon me—I took it on willingly. That is simply the kind of person I am. If I see someone struggling, I cannot ignore it. I am very attached to my family.

There were times, for example, at Easter, when my mother just drove off. I could see in her eyes that she did not want to live anymore. She says the same thing even now. Back then, it was even stronger—she simply did not want to go on. She just got into the car and drove away. (Shocked and saddened.) It was simply terrible.

At first, I wanted to prevent the separation, of course. As a child, you never want your parents to separate. But later, it was just about minimizing the damage. I lost count of how many times I sat there listening, trying to mediate. I took on the role of always being there. But at some point, it was just too much.

I still managed to get through it, though sometimes I look back and wonder how I did it. I held up well, except for my university studies, where I had some setbacks. That was where the burden really showed. The emotional toll and the time commitment were simply too much.
End
Interviewee: I actually feel much better now. I have accepted everything as it is. I am a realistic person. I do not try to convince myself of things that do not exist. I walk through life with open eyes. I see what is happening around me. I know the divorce statistics.

Interviewer: But until recently, they did not matter to you.

Interviewee: No.

Interviewer: (Laughs.)

Interviewee: (Laughs as well.) Yes, because within those four walls, everything was fine. That was my foundation, my roots, where I came from. It was intact.

Interviewer: But now that has changed.

Interviewee: Yes. And I know that no matter how well things may seem to be going, there is always the risk that it could fall apart. That belief, that certainty I once had, is gone. I used to truly believe in lasting relationships. But now, if you ask me whether I think a relationship will last a lifetime, I no longer believe that. It is a sad realization.

Interviewer: It sounds as if a vision or a dream has been lost.

Interviewee: Yes, definitely. No doubt about it.
A.2 Example Prompt for Self-dependency and Dependency on Others
Context: Relationships and attachments are of existential importance in every person’s life. They span the opposing poles of striving for close relationships and symbiotic proximity (dependency) and striving for well-developed independence and clear distance (powerful individuation). Individuation and dependency are fundamental elements of human life and experience, present in all areas of life. A life-defining conflict arises when this fundamental bipolar tension turns into a conflictual polarization. An individuation-dependency conflict is present only if this constellation is of existential importance and formative for a person’s life history: This conflict involves the activation of experiences that either seek or avoid closeness, not the shaping of relationships in terms of caregiving or avoiding caregiving. The theme of individuation-dependency deals with the question of being alone or the ability to be with others. In its pathological conflict version, it concerns the necessity of being alone or being with others as an existential requirement.

Task: Based on this context, classify the following interview excerpt regarding the theme of "autonomy-dependency" into one of the following categories: "not present" , "not assessable" , "of little significance" , "significant" , "very significant" .
For tasks where the interviews were summarized prior to classification, the model was first instructed to generate a summary of the interview based on a provided example. This example was derived from the OPD Manual.
A.3 Classification Classes
Not assessable – The category cannot be determined due to insufficient or ambiguous information. There may be a lack of relevant content, unclear statements, or methodological limitations preventing a reliable assessment.

Not present – There is no indication of the characteristic or phenomenon being evaluated. The available information does not support its existence or relevance in the given context.

Of little significance – The characteristic or phenomenon is present but plays only a minor role. It appears occasionally but does not have a substantial influence on behavior, emotions, or interactions.

Significant – The characteristic or phenomenon is clearly identifiable and has a notable impact. It influences thoughts, emotions, or interactions and is relevant to the overall assessment.

Very significant – The characteristic or phenomenon is a dominant feature. It strongly shapes experiences, interactions, or coping mechanisms and is central to the evaluation.
Figure 1: Overview over AutoPsyC.; Overview over AutoPsyC.
Figure 2: Class distributions for the psychodynamic conflicts investigated in this study.; Class distributions for the psychodynamic conflicts investigated in this study.
Figure 3: Performance of the four models across all classes.; Performance of the four models across all classes.
Figure 4: Performance of Different Number of Models; Performance of Different Number of Models
Figure 5: Class distributions for Oedipal Conflict and Conflicts related to Identity; Class distributions for Oedipal Conflict and Conflicts related to Identity
Average Weighted F1-Scores with 95% Confidence Intervals.
One-vs.-rest CDD values for each class across four conflicts.