Multi-Session Client-Centered Treatment Outcome Evaluation in Psychotherapy

## Abstract

Abstract In psychotherapy, therapeutic outcome assessment, or treatment outcome evaluation, is essential for enhancing mental health care by systematically evaluating therapeutic processes and outcomes. Existing large language model approaches often focus on therapist-centered, single-session evaluations, neglecting the client’s subjective experience and longitudinal progress across multiple sessions. To address these limitations, we propose IPAEval, a client-Informed Psychological Assessment-based Evaluation framework that automates treatment outcome evaluations from the client’s perspective using clinical interviews. IPAEval integrates cross-session client-contextual assessment and session-focused client-dynamics assessment to provide a comprehensive understanding of therapeutic progress. Experiments on our newly developed TheraPhase dataset demonstrate that IPAEval effectively tracks symptom severity and treatment outcomes over multiple sessions, outperforming previous single-session models and validating the benefits of items-aware reasoning mechanisms.

## 1 Introduction

In psychotherapy, therapeutic outcome assessment, a.k.a treatment outcome evaluation under clinical settings, refers to the systematic evaluation of therapeutic processes and outcomes (Groth-Marnat, ),
focusing on factors such as therapist effectiveness (Johns et al., ) and treatment efficacy (Jensen-Doss et al., ) to improve mental health care delivery.
It plays a significant role in enhancing the quality and effectiveness of mental health care by providing actionable insights that guide therapists in refining their treatment approaches (Wampold & Imel, ), ultimately leading to better client outcomes and improved therapeutic relationships in real-world clinical practice (Maruish & Leahy, ).

Over the last couple of years, the emergence of large language models has demonstrated their effectiveness in automatic evaluations, showing a high degree of alignment with human judgment when provided with proper instruction and contextual guidance (Liu et al., ; Li et al., ; Kim et al., ).
This aligns with the “LLMs-as-a-judge” paradigm, where LLMs are employed to simulate human evaluators by providing assessments based on natural language input (Zheng et al., ; Wang et al., ).
This paradigm has been extended to therapeutic outcome assessment by harnessing LLMs’ ability to model complex therapeutic procedures and interactions, offering a novel pathway for automating the assessment of therapeutic efficacy (Chiu et al., ; Lee et al., ; Li et al., ).

In the assessment, compared to psychometric tests (Furr, ) that are often constrained by the limitations of self-reported data, susceptibility to social desirability biases (Braun et al., ; Paulhus, ),
clinical interviews not only provide richer, more nuanced insights into the client’s emotional and behavioral states but also offer data that is more readily obtainable through natural, conversational interactions.
Therefore, many recent works leverage clinical interviews, potentially enriched by the client’s profile (Lee et al., ), to evaluate therapists from multiple perspectives, including behavioral labels (Chiu et al., ), skills adherence (Lee et al., ), and therapeutic rapport (Li et al., ; Yosef et al., ), offering a holistic view of their effectiveness in psychotherapy.

While the above therapist-centered assessments focus on evaluating the therapist’s techniques and adherence to therapeutic models, they often overlook the subjective experience and evolving needs of the client, limiting the depth of the evaluation (Wang et al., ; Yosef et al., ). In contrast, client-centered assessments, such as treatment outcome evaluation in common practice, prioritize the client’s perspective, offering a more comprehensive understanding of therapy’s impact by capturing changes in the client’s emotional, cognitive, and behavioral states across sessions (Hatfield & Ogles, ; Rogers, ).
Although a concurrent work, ClientCAST (Wang et al., ), presents an LLM-based client simulator for treatment outcome evaluations, which focuses on reducing harmful outputs and improving answering consistency, we stand fundamentally apart and never fabricate client responses that could distort the evaluation of treatment outcomes.
What’s worse, almost all previous approaches focus on evaluating individual therapy sessions in isolation, without considering the broader context of the client’s journey across multiple sessions.
This narrow scope limits the ability to assess longitudinal progress or capture the dynamic shifts in a client’s mental state and therapeutic needs over time, which are crucial for a comprehensive treatment outcome evaluation (Hayes & Andrews, ).

**Table 1: A comparison of IPAEval with other treatment outcome evaluation methods. Perspective indicates whether the evaluation is conducted from the therapist’s or the client’s point of view. Theory Adherence signifies whether the method is grounded in established psychological theories. Reasoning denotes whether the method involves generating intermediate reasoning steps before arriving at the final evaluation results. Evaluation Target refers to whether the method evaluates a single session or multiple sessions.**
| Method | Perspective | Theory Adherence | Reasoning | Evaluation Target |
| --- | --- | --- | --- | --- |
| CPsyCoun (Zhang et al., ) | Therapist | ✗ | ✗ | Single Session |
| Cactus (Lee et al., ) | Therapist | ✓ | ✗ | Single Session |
| ClientCAST (Wang et al., ) | Client | ✓ | ✗ | Single Session |
| IPAEval (Ours) | Client | ✓ | ✓ | Multiple Sessions |

Motivated by the above therapist-centered and single-session limitations (please see Table 1 for comparisons), we design a new evaluation framework, dubbed client-Informed Psychological Assessment-based Evaluation (IPAEval), for treatment outcomes in the format of clinical interviews.

Specifically, to achieve treatment outcome evaluation, we formulate an information extraction task that leverages clinical interviews to automatically populate psychometric tests for psychological assessments, bridging the gap between subjective client dialogues and standardized metrics. As such, treatment outcomes are evaluated through these assessments of clients conducted both before and after therapy, allowing for a more comprehensive understanding of therapeutic progress.
Upon this new framework, we first propose a cross-session client-contextual assessment module that integrates client history and contextual information across multiple sessions to enhance the accuracy of psychological assessments. Then, we present a session-focused client-dynamics assessment module that evaluates the effectiveness of individual therapy sessions by tracking real-time client responses and treatment outcomes within each session.
In the meantime, to boost reasoning capability in the extraction, we also present an items-aware reasoning prompt technique for psychometric test-oriented rationale generation.

To evaluate the proposed framework,
we first develop a new dataset, called TheraPhase, based on CPsyCoun (Zhang et al., ), which includes transcripts from initial and final therapy sessions. This dataset offers valuable insights into therapy progress and serves as a key resource for evaluating psychological assessments and treatment outcomes across multiple sessions.
Then, we tested nine LLMs, including closed-source models. These models were evaluated for their performance in psychological assessments and treatment outcome prediction, particularly in multi-session evaluations. IPAEval consistently tracked symptom severity and treatment outcomes across multiple sessions, a capability lacking in previous single-session models. Our ablation study confirmed that the items-aware reasoning mechanism significantly boosts model performance in both symptom detection and outcome prediction.

## 2 Related Work

#### Therapist Assessment using LLMs.

LLMs’ role-playing capabilities have led to increased interest in developing Role-Play Therapists (Chen et al., ; Chiu et al., ; Lee et al., ), but the lack of automated metrics for evaluating t
herapist is a significant challenge.
CPsyCoun (Zhang et al., ) employs an LLM-based evaluation method from the therapist’s perspective to assess single session, specifically evaluating the therapist’s comprehensiveness, professionalism, authenticity, and safety.
Lee et al. (), Li et al. (), and Yosef et al. () similarly adopt a therapist’s perspective with LLM-based evaluation, but they address CPsyCoun’s lack of support from psychological theories by employing the Cognitive Therapy Rating Scale (Goldberg et al., ) for CBT skills assessment and the Working Alliance Inventory (Hatcher & Gillaspy, ) for evaluating the therapeutic relationship. Notably, BOLT (Chiu et al., ) applied LLMs to identify therapist behaviors, evaluating the quality of dialogue sessions based on the frequency and sequence of LLM therapist behaviors. Clinical evidence (Goodson et al., ; Mason et al., ) shows that better therapists are linked to improved outcomes, but evaluating therapists alone may miss how much the client is benefiting (Robinson, ). The treatment outcome evaluation based on client-centered psychological assessment focuses more on results, specifically determining whether the therapy has brought about meaningful changes in the client’s life, which is the ultimate goal of the treatment (Groth-Marnat, ).

#### Client-centered Psychological Assessment.

Client-centered psychological assessment combines psychometric tests and clinical interviews to provide a comprehensive understanding of the individual (Spoto et al., ). Psychometric tests offer standardized data on psychological traits, while clinical interviews give deeper insights into the client’s personal experiences (Groth-Marnat, ). While tests may overlook certain nuances, interviews address these gaps by exploring context and individual differences.

Figure: Figure 1: What is Treatment Outcome?
Refer to caption: x1.png

In clinical practice, the use of multiple assessment methods ensures a more complete understanding of the client (Meyer et al., ; Groth-Marnat, ). Leveraging the powerful general language processing capabilities (Luo et al., ; Zhao et al., ) of LLMs enables the realization of complex and diverse assessment tasks. This contrasts with earlier approaches that focused solely on detecting individual psychological symptoms (Ji et al., ; Zhai et al., ), and a substantial body of research (Galatzer-Levy et al., ; Arcan et al., ; Rosenman et al., ) supports this advancement. For instance, several studies have utilized LLMs to analyze interviews (Gratch et al., ), assessing depression and Post-Traumatic Stress Disorder scores based on widely used psychometric tests like  (Kroenke et al., ) and PCL-C (Weathers et al., ).
However, precise psychological assessments enable therapists to grasp the client’s psychological state, but a psychological assessment alone cannot determine whether the treatment has brought about positive changes for the client.

Treatment outcomes evaluation complements psychological assessment by measuring the effectiveness of interventions over time (Maruish & Leahy, ). While psychological assessments provide a snapshot of the client’s mental state, as shown in the Figure 1, treatment outcomes evaluation focuses on tracking changes in symptoms and overall well-being throughout the therapeutic process. This dynamic evaluation allows therapists to determine whether the treatment has been successful and adjust strategies as needed to improve results.

## 3 Methodology

In this section, starting with a formal task definition (§3.1), we elaborate on our evaluation framework, called client-Informed Psychological Assessment-based Evaluation (IPAEval), which is mainly composed of 1) cross-session client-contextual assessment module (§3.2) for client-tracking psychological assessment and a session-focused client-dynamics assessment module (§3.3) to derive session-informed treatment outcome evaluation. Please see Figure 2 for an overall illustration of our framework.
As there is no precursor in clinical interviews-based treatment outcome evaluation, we, therefore, curate a new dataset, called TheraPhase, as a testbed for our proposed IPAEval framework.

Figure: Figure 2: An illustration of client-informed psychological assessment-based evaluation (IPAEval).
Refer to caption: x2.png

### 3.1 Task Definition of IPAEval Framework

To deliver treatment outcome evaluations for a certain client with profile $p$, based on multiple sessions $[s_{1},s_{2},\dots]$, we aim to evaluate the efficacy of a certain session $s_{k}$ in clinical interviews as treatment outcome.
Without sacrificing generalization to one whole treatment composed of several sessions, $s_{k}$ here could be a combination of the sessions.
To achieve the above, we need to split the task into two sequential sub-tasks – psychological assessment (${\bm{a}}_{k}$) based on client information and treatment outcome evaluation (${\bm{e}}_{k}$) based on two or more assessments.
To derive ${\bm{a}}_{k}$ and ${\bm{e}}_{k}$, we first define client-informed input of an assessment after $k$-th session, i.e.,

$$ $\displaystyle c_{k}=p\oplus s_{k}\oplus h_{k},~{}\text{where}~{}h_{k}=[s_{1}, \dots,s_{i-1}]\oplus[{\bm{a}}_{1},\dots]\oplus[{\bm{e}}_{1},\dots].$ (1) $$

Here, $h_{k}$ denotes a set of meta client-contextual information from the past, e.g., past interviews $[s_{1},\dots,s_{i-1}]$, past assessments $[{\bm{a}}_{1},\dots]$, or/and past outcome evaluations $[{\bm{e}}_{1},\dots]$.
Upon this, we could easily define psychological assessment after $i$-th session as

$$ $\displaystyle{\bm{a}}_{k}=\operatorname*{M}\nolimits^{\text{(a)}}(c_{k},{ \mathbb{T}}),$ (2) $$

where ${\mathbb{T}}$ is a set of psychometric tests as the criteria to evaluate the client information, and $\operatorname*{M}$ denotes an approach to derive ${\bm{a}}_{k}$.
Then, the treatment outcome evaluation for $k$-th session would defined as

$$ $\displaystyle{\bm{e}}_{k}=\operatorname*{M}\nolimits^{\text{(e)}}(c_{k},{\bm{a }}_{k},h_{k}),$ (3) $$

In the remaining, we omit the index of session $k$ if no confusion is raised for clear annotations.

To handle the above two sequential sub-tasks, we detail our two modules, which aim to tackle the sub-tasks respectively, in the following.

### 3.2 Cross-session Client-contextual Assessment

Existing research using client information with LLMs for mental health assessment, particularly for depression and PTSD, shows promising results (Galatzer-Levy et al., ; Arcan et al., ; Rosenman et al., ). However, these studies typically focus on specific symptoms and lack broad coverage of psychological conditions and transparency in interpreting scale results, which may erode trust among clinicians and clients, limiting clinical applications (Martin & Rouas, ).

To address these gaps, we introduce a two-stage prompt scheme that populates information from clinical interviews to fill psychometric tests by making the best of LLMs’ capability in natural language understanding (Zhao et al., ; Hua et al., ).
It’s applicable to various psychometric tests, specifically designed to provide interpretable psychological assessments.
Without sacrificing generality, in this work we utilize the Symptom Checklist-90 (SCL-90) (Derogatis & Unger, ), a widely used and comprehensive psychometric test for screening psychological symptoms.

#### Items-Aware Reasoning.

This stage aims to generate detailed reasoning for psychometric test items using LLMs, leveraging client information. Here, the items of the SCL-90 represent psychological symptoms. It correlates specific client information with the corresponding symptoms and items from the SCL-90, determining their presence and providing an interpretation.
Given an out-of-the-box LLM (Dubey et al., ; Yang et al., ; Jiang et al., ) able to follow instructions, we first curate a prompt to steer the LLM to extract information from interviews to structured psychometric tests with thought augmentations.
Inspired by a recent work Schulhoff et al. (), our prompt design integrates several components: a psychologist role, denoted by $r^{\text{(pi)}}$, skilled at recognizing symptoms, the SCL-90 as additional information ${\mathbb{T}}$, output formatting $o^{\text{(pi)}}$, and specific directives $d^{\text{(pi)}}$. Based on these, we can define psychometric interpretation prompt $p^{\text{(pi)}}$ as follows:

$$ $\displaystyle p^{\text{(pi)}}=f(r^{\text{(pi)}},{\mathbb{T}},o^{\text{(pi)}},d ^{\text{(pi)}})$ (4) $$

Furthermore, given the client information $c$, an LLM is prompted to generate items-aware reasoning results $\hat{{\mathbb{X}}}$:

$$ $\displaystyle\hat{{\mathbb{X}}}=\underset{{\mathbb{X}}}{\text{argmax}}\,P_{ \text{LLM}}({\mathbb{X}}|c,p^{\text{(pi)}}),$ (5) $$

where $p^{\text{(pi)}}$ $\hat{{\mathbb{X}}}$ represents a set of predicted items-aware reasoning results, each element in the set consisting of extracted client information, symptom category, specific symptom, presence, and a detailed explanation. It is noteworthy that this approach helps clinicians quickly trace the source of evidence for assessments and offers a clear pathway to understanding the interconnections and relevance of various symptoms presented by the client. The detailed prompt and an example of Items-Aware Reasoning are provided in Appendix A and Appendix C, respectively.

#### Psychological Assessment.

The other stage is designed to harness the capabilities of LLMs in conducting psychological assessments based on client information and item-aware reasoning results. Similar to the items-aware reasoning stage, it comprises four main components: a psychologist role denoted by $r^{\text{(sa)}}$, skilled at symptom assessment, the SCL-90 psychometric test ${\mathbb{T}}$, alongside score criteria $s$ serving as additional information, output formatting implemented $o^{\text{(sa)}}$, and specific directives $d^{\text{(sa)}}$.
Considering the practical constraints of client information, where not all 90 questions from the SCL-90 are likely to be addressed, we have simplified the scoring criteria. Instead of scoring each of the 90 items individually, the assessment has been adapted to score across 10 symptom dimensions derived from these items. Based on these, we can define symptom assessment prompt $p^{\text{(sa)}}$ as follows:

$$ $\displaystyle p^{\text{(sa)}}=f(r^{\text{(sa)}},{\mathbb{T}},s,o^{\text{(sa)}} ,d^{\text{(sa)}})$ (6) $$

Formally, given the client information $c$ and items-aware reasoning result $\hat{{\mathbb{X}}}$ generated by LLM, an LLM is prompted to generate assessment scores $\hat{{\bm{a}}}$:

$$ $\displaystyle\hat{{\bm{a}}}=\underset{{\bm{a}}}{\text{argmax}}\,P_{\text{LLM}} ({\bm{a}}|c,\hat{{\mathbb{X}}},p^{\text{(sa)}})$ (7) $$

Where $\hat{{\bm{a}}}$ represents the estimated assessment scores for each of the 10 symptom dimensions. The detailed prompt is provided in Appendix B

#### Remark: Avoiding Excessive Speculation.

ClientCAST (Wang et al., ), which simulates the client’s estimation of psychometric test scores, our approach avoids excessive speculation. By adjusting the range of psychometric test scores to account for items not yet addressed by the client, our method more accurately reflects the gradual disclosure of information over time or across multiple sessions, preventing incomplete or biased assessments due to initially unmentioned items.

### 3.3 Session-focused Client-dynamics Assessment

Given the assessment scores $\hat{{\bm{a}}}$ of client information $c$, we use them to calculate Positive Symptom Distress Index (PSDI) (Derogatis & Unger, ), which quantifies the level of distress associated with positive psychological symptoms. The PSDI is calculated by averaging the distress scores assigned to each symptom, providing a numerical indication of the severity and impact of these symptoms on the client’s overall mental health. The PSDI is mathematically expressed by the formula:

$$ $\displaystyle\text{PSDI}=\frac{1}{N}\sum_{i\in{\mathbb{P}}}\hat{{\bm{a}}}_{i}$ (8) $$

Where $N$ is the number of positive symptoms, and $\hat{{\bm{a}}}_{i}$ is the distress score for the $i$-th symptom, and ${\mathbb{P}}$ is the set containing the indices of all positive symptoms.

Consider a client whose initial stage information is denoted as $c_{i}$ and final stage information after completing treatment as $c_{f}$. By applying Equation 5 and 7 to the client information at each stage, we can obtain the initial stage assessment scores $\hat{{\bm{a}}_{i}}$ and final stage assessment scores $\hat{{\bm{a}}_{f}}$. Further, we can calculate the PSDI for both the initial and post-treatment stages using Equation 8 to assess the impact of treatment on the client’s positive psychological symptoms. We define the change in symptoms as

$$ $\displaystyle{\bm{e}}\coloneqq\Delta\text{PSDI}=\text{PSDI}_{f}-\text{PSDI}_{i}$ (9) $$

Where $\Delta\text{PSDI}$, defined as treatment outcome evaluation ${\bm{e}}$ for the session $s$ in this work, represents the change in the PSDI from before to after treatment, quantifying the impact of the intervention on the client’s distress related to positive psychological symptoms.

#### Remark: Advantages and Versatility of PSDI.

Although PSDI is originally derived from the SCL-90, the method of calculating the average score of positive items offers the advantage of focusing directly on the relevant items of a psychometric test, leading to a more precise evaluation of treatment outcomes. This approach is not limited to the SCL-90 and can be easily extended to other psychometric tests, providing a flexible and reliable tool for assessing progress across different stages of treatment.

### 3.4 TheraPhase Dataset

Popular datasets such as High-Low Quality (Pérez-Rosas et al., ), and AnnoMI (Wu et al., ), which contain client information primarily in the form of a single session, originate from public video sharing sources. These datasets only include client information relevant to the current stage and do not provide data for subsequent stages. To assess the changes in clients across different stages, we have constructed the TheraPhase Dataset based on the CPsyCoun (Zhang et al., ), which exhibits significant changes during a single session. Our dataset includes 400 pairs of client information from both the initial and completion stages of treatment.

#### Construction Process.

To construct the TheraPhase Dataset, we utilize a 5-shot prompting approach with GPT-4 to extract the initial stage information from a client’s comprehensive information. This method isolates the beginning portion of the client’s data, forming a paired dataset where each pair consists of the initial client information and the corresponding full client information. This setup allows for an analytical comparison between the initial conditions and the outcomes after therapeutic interventions. The statistics of the resulting dataset are listed in Table 3.

## 4 Experiments

In this section, we first conduct a psychological assessment based on various LLMs, evaluating their capability to detect and assess symptoms. Subsequently, we investigate their performance in predicting treatment outcomes.

### 4.1 Experimental Settings

#### IPAEval Setting Up.

The IPAEval framework is capable of handling various forms of client information, such as user profiles and interaction histories. However, due to data acquisition limitations, we primarily utilized consultation dialogue data as the main source of client information. Furthermore, IPAEval supports a variety of symptom-based psychometric tests, such as the General Health Questionnaire (GHQ) series (Montazeri et al., ), the Symptom Checklist (SCL) series, and the Brief Symptom Inventory (BSI) (Derogatis & Melisaratos, ). In this experiment, we utilized the Symptom Checklist-90 (SCL-90) (Derogatis et al., ), a widely recognized and comprehensive tool for assessment a broad range of psychological symptoms. The scoring criteria for assessing symptoms, as set up and outlined in Table 2. Additionally, to ensure structured output, our code utilizes LangChain(^1^11[https://www.langchain.com/](https://www.langchain.com/)) and Pydantic(^2^22[https://docs.pydantic.dev/](https://docs.pydantic.dev/)) for better LLMs integration and data validation.

**Table 2: Scoring Criteria for Symptom Assessment**
| Score | Description |
| --- | --- |
| -1 | Symptom not addressed. |
| 0 | Symptom addressed, but no symptoms found; no signs of distress or dysfunction. |
| 1 | Minimal symptoms, minor indications of distress but no significant dysfunction. |
| 2 | Clear symptoms, clear indications of distress, and significant dysfunction. |

#### Datasets.

We have selected two datasets for psychological assessment, High-Low Quality Counseling (Pérez-Rosas et al., ) and AnnoMI (Wu et al., ), which consist of counseling therapy transcripts extracted from publicly available videos on online platforms such as YouTube and Vimeo. However, there are issues of data duplication between these two datasets. Given the higher quality of data in AnnoMI, we have chosen to retain the AnnoMI data from the same sources. Furthermore, considering the context window limitation of one of our test models, GPT-4, the maximum number of dialogue turns is set to 102. To increase the testing challenge and ensure the dialogues are sufficiently complex for evaluating the model’s capability in handling extended therapeutic conversations, the minimum number of turns is set at 25. Based on these criteria, we have selected 110 client dialogue entries as our test data.

For treatment outcomes, we have selected the TheraPhase Dataset. This dataset comprises treament session transcripts that encompass two distinct phases of client interactions. Its advantage lies in the clear changes observable in clients across these phases, which aids in observing the treatment outcomes. The statistics of the resulting datasets are listed in Table 3.

**Table 3: Summary of key characteristics of the selected datasets, including language, number of clients, sessions, average number of utterances per session, and the average word count per utterance.**
| Datasets | Language | # of Clients | # of Sessions | Avg. # of Utterances | Words per Utterance |
| --- | --- | --- | --- | --- | --- |
| High-Low Quality Counseling<br>AnnoMI | English | 110 | 110 | 79.8<br>(std = 26.1) | 22.2<br>(std = 27.1) |
| TheraPhase | Chinese | 400 | 800 | 11.5<br>(std = 6.3) | 41.7<br>(std = 20.9) |

#### Evaluation Metrics.

We conducted a psychological assessment of LLMs focusing on two main aspects, symptom detection and symptom severity assessment. For symptom detection, we evaluated the model’s ability to identify symptoms from a broad range of client information using classification metrics such as Accuracy, Precision, Recall, and F1 scores (Binary, Macro, and Weighted), based on scoring criteria from Table 2 where -1 indicates a negative class and 0, 1, and 2 represent positive classes. For assessing symptom severity, we calculated the PSDI score for each client and used error metrics such as Mean Squared Error (MSE) and Mean Absolute Error (MAE). To gauge the model’s reliability, we reported the mean and standard deviation of these evaluation metrics across three runs, providing insight into the model’s consistency in performance.

In evaluating treatment outcomes, we focus on the change in positive symptom severity, represented by $\Delta\text{PSDI}$, which reflects the difference in mean positive symptom scores between two assessments. A $\Delta\text{PSDI}$ greater than 0 indicates a worsening of symptoms or the emergence of new ones, while a value less than or equal to 0 suggests symptom maintenance or improvement. We further evaluated the accuracy of predicting the direction of treatment outcome changes using metrics such as Accuracy, Precision, Recall, and F1 scores (Binary, Macro, and Weighted).

#### References Generation.

To evaluate the performance of the models on psychological assessment tasks, we first required a set of reference scores for symptom detection and severity assessment. However, due to the lack of existing labeled data, we manually annotated 30 randomly selected client sessions. This manual annotation was carried out by two co-authors of this paper, both with significant expertise in natural language processing (NLP) and mental health research. The annotation process achieved a Cohen’s kappa coefficient of 0.73, indicating substantial agreement between annotators. Following the annotation, we tested the performance of four closed-source models: GPT-4, GPT-4o, GPT-4-turbo, and GPT-4o-mini. The results, as shown in Table 4, indicated that GPT-4o outperformed the other models in both symptom detection and severity assessment. Based on these findings, GPT-4o was selected as the Gold Model for generating reference scores in psychological assessment tasks.

A similar issue arose in the treatment outcomes evaluation task. To address this, we followed the same approach as in the psychological assessment task. We manually annotated 60 sessions corresponding to 30 clients, focusing on their treatment outcomes. This annotation was again conducted by the two co-authors, achieving a Cohen’s kappa coefficient of 0.81, reflecting a high level of agreement. The results, as presented in Table 5 shows that GPT-4 achieved the highest performance, thus it was chosen as the Gold Model for generating reference scores in treatment outcomes evaluation task.

**Table 4: Comparison of different models on various performance metrics using human-annotated data in psychological assessment. Metrics with an upward arrow $\uparrow$ indicate higher values are better, while metrics with a downward arrow $\downarrow$ indicate lower values are better. The results show mean values along with standard deviations for each metric. Cells highlighted in blue represent the best-performing results.**
| Models | Accuracy $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1Binary $\uparrow$ | F1Macro $\uparrow$ | F1Weighted $\uparrow$ | MSE $\downarrow$ | MAE $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4 | 0.7744$\pm$0.01 | 0.6792$\pm$0.01 | 0.7187$\pm$0.01 | 0.6984$\pm$0.01 | 0.7591$\pm$0.01 | 0.7757$\pm$0.01 | 0.1369$\pm$0.02 | 0.2398$\pm$0.01 |
| GPT-4o | 0.7833$\pm$0.02 | 0.6674$\pm$0.02 | 0.8043$\pm$0.03 | 0.7295$\pm$0.02 | 0.7744$\pm$0.02 | 0.7867$\pm$0.02 | 0.1207$\pm$0.01 | 0.2272$\pm$0.02 |
| GPT-4-turbo | 0.7800$\pm$0.01 | 0.7503$\pm$0.03 | 0.5933$\pm$0.01 | 0.6623$\pm$0.01 | 0.7495$\pm$0.01 | 0.7734$\pm$0.01 | 0.2379$\pm$0.03 | 0.3754$\pm$0.03 |
| GPT-4o-mini | 0.4844$\pm$0.04 | 0.4079$\pm$0.02 | 0.9144$\pm$0.04 | 0.5634$\pm$0.01 | 0.4641$\pm$0.05 | 0.4370$\pm$0.06 | 0.1962$\pm$0.03 | 0.3265$\pm$0.02 |

**Table 5: Comparison of different models on various performance metrics using human-annotated data in treatment outcomes.**
| Models | Accuracy $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1Binary $\uparrow$ | F1Macro $\uparrow$ | F1Weighted $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| GPT-4 | 0.7444$\pm$0.06 | 0.8285$\pm$0.04 | 0.8406$\pm$0.04 | 0.8344$\pm$0.04 | 0.6370$\pm$0.08 | 0.7423$\pm$0.06 |
| GPT-4o | 0.6778$\pm$0.06 | 0.8219$\pm$0.02 | 0.7391$\pm$0.07 | 0.7770$\pm$0.05 | 0.5939$\pm$0.05 | 0.6916$\pm$0.05 |
| GPT-4-turbo | 0.6778$\pm$0.06 | 0.8046$\pm$0.02 | 0.7681$\pm$0.11 | 0.7815$\pm$0.05 | 0.5660$\pm$0.04 | 0.6809$\pm$0.04 |
| GPT-4o-mini | 0.7222$\pm$0.04 | 0.8625$\pm$0.06 | 0.7681$\pm$0.05 | 0.8090$\pm$0.03 | 0.6410$\pm$0.08 | 0.7306$\pm$0.05 |

#### Models.

We conducted an investigation into the performance of several closed-source and open-source LLMs. The closed-source models we tested include GPT-4 (OpenAI et al., ), GPT4o, GPT-4-turbo, and GPT-4o-mini, which represent the latest advancements in proprietary LLMs developed by OpenAI(^3^33Specific versions of the OpenAI models used in the tests were gpt-4-0613, gpt-4o-2024-05-13, gpt-4-turbo-2024-04-09, gpt-4o-mini-2024-07-18.).
. Additionally, we tested a variety of open-source models, such as Llama3.1-405B (Dubey et al., ), Llama3.1-70B (Dubey et al., ), Qwen2-72B (Yang et al., ), Mistral-8X22B (Jiang et al., ), and Mistral-8X7B (Jiang et al., ). These models vary significantly in terms of architecture, parameter size, and training data, providing a comprehensive overview of both commercial and community-driven LLM development. All of these models were invoked through API platforms(^4^44For the OpenAI models, we invoked them via [https://platform.openai.com](https://platform.openai.com), Mistral models through [https://console.mistral.ai/](https://console.mistral.ai/), Llama3.1 models via [https://fireworks.ai/](https://fireworks.ai/), and Qwen2 through [https://www.together.ai/](https://www.together.ai/).)

### 4.2 Main Evaluation Results

#### Psychological Assessments.

As shown in Table 6, GPT-4 achieved the best performance in symptom detection, excelling in both accuracy and binary F1 score, highlighting its strong ability to accurately identify symptoms. GPT-4-turbo demonstrated a more conservative approach with higher precision but lower recall, indicating it was more cautious in detecting symptoms but missed more cases. GPT-4o-mini excelled in recall but had reduced overall reliability due to a higher rate of false positives. Among open-source models, Qwen2-72B and Llama3.1-70B showed the closest performance to GPT-4, though they still fell short. Notably, Mistral-8X7B’s extremely low recall was caused by a significant number of output formatting errors, leading to evaluation failures. We will further discuss these formatting issues in Appendix D.

In symptom severity assessment, GPT-4 once again stood out with the lowest MSE and MAE, making it the most accurate model. Although GPT-4o-mini and GPT-4-turbo showed more balanced results, they were less precise compared to GPT-4. Among open-source models, Llama3.1-70B performed the best, though the gap between open-source and closed-source models remained substantial. Furthermore, GPT-4 exhibited the greatest consistency and reliability, with minimal variance across runs, indicating robust performance. In contrast, GPT-4o-mini showed more variability in MAE and MSE, and open-source models generally exhibited less stability compared to their closed-source counterparts.

**Table 6: Performance comparison between closed-source and open-source models across various evaluation metrics in psychological assessment.**
| Models | Accuracy $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1Binary $\uparrow$ | F1Macro $\uparrow$ | F1Weighted $\uparrow$ | MSE $\downarrow$ | MAE $\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Closed-Source Models |  |  |  |  |  |  |  |  |
| GPT-4 | 0.7973$\pm$0.01 | 0.7852$\pm$0.01 | 0.7121$\pm$0.01 | 0.7469$\pm$0.01 | 0.7889$\pm$0.01 | 0.7956$\pm$0.01 | 0.2100$\pm$0.02 | 0.3292$\pm$0.03 |
| GPT-4-turbo | 0.7561$\pm$0.00 | 0.8726$\pm$0.02 | 0.4913$\pm$0.01 | 0.6285$\pm$0.01 | 0.7234$\pm$0.00 | 0.7386$\pm$0.00 | 0.4055$\pm$0.05 | 0.4490$\pm$0.03 |
| GPT-4o-mini | 0.4915$\pm$0.00 | 0.4467$\pm$0.02 | 0.8824$\pm$0.01 | 0.5931$\pm$0.00 | 0.4576$\pm$0.01 | 0.4359$\pm$0.01 | 0.2245$\pm$0.01 | 0.3329$\pm$0.02 |
| Open-Source Models |  |  |  |  |  |  |  |  |
| Llama3.1-405B | 0.7291$\pm$0.00 | 0.6960$\pm$0.01 | 0.6306$\pm$0.00 | 0.6616$\pm$0.00 | 0.7179$\pm$0.00 | 0.7269$\pm$0.00 | 0.3922$\pm$0.03 | 0.4476$\pm$0.01 |
| Qwen2-72B | 0.7385$\pm$0.00 | 0.7405$\pm$0.01 | 0.5815$\pm$0.01 | 0.6513$\pm$0.01 | 0.7210$\pm$0.00 | 0.7322$\pm$0.00 | 0.3962$\pm$0.01 | 0.4559$\pm$0.00 |
| Llama3.1-70B | 0.7333$\pm$0.01 | 0.7201$\pm$0.01 | 0.5974$\pm$0.01 | 0.6529$\pm$0.01 | 0.7182$\pm$0.01 | 0.7286$\pm$0.01 | 0.3379$\pm$0.01 | 0.4041$\pm$0.00 |
| Mistral-8X22B | 0.6215$\pm$0.00 | 0.5405$\pm$0.00 | 0.6616$\pm$0.02 | 0.5948$\pm$0.00 | 0.6198$\pm$0.00 | 0.6238$\pm$0.00 | 0.5205$\pm$0.03 | 0.5452$\pm$0.02 |
| Mistral-8X7B | 0.6070$\pm$0.00 | 0.6158$\pm$0.01 | 0.1710$\pm$0.02 | 0.2672$\pm$0.02 | 0.4993$\pm$0.01 | 0.5364$\pm$0.01 | 1.5711$\pm$0.02 | 1.0927$\pm$0.01 |

#### Treatment Outcomes.

Table 7 compares the performance of closed-source and open-source models on treatment outcome evaluation tasks. Among the closed-source models, GPT-4-turbo achieved the highest scores across multiple metrics, making it the most effective model in treatment outcome prediction. GPT-4o and GPT-4o-mini displayed competitive performance but lagged slightly behind GPT-4-turbo. For the open-source models, Llama3.1-405B led the group with the highest accuracy and macro F1, demonstrating superior performance in treatment outcome tasks. Qwen2-72B and Llama3.1-70B also performed well, while Mistral-8X7B had the highest recall but struggled with lower F1 scores, indicating higher sensitivity but less consistent overall performance. Overall, both closed-source and open-source models showed strong capabilities, with GPT-4-turbo and Llama3.1-405B emerging as the top performers in their respective categories.

**Table 7: Performance comparison between closed-source and open-source models across various evaluation metrics in treatment outcomes.**
| Models | Accuracy $\uparrow$ | Precision $\uparrow$ | Recall $\uparrow$ | F1Binary $\uparrow$ | F1Macro $\uparrow$ | F1Weighted $\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| Closed-Source Models |  |  |  |  |  |  |
| GPT-4o | 0.6375$\pm$0.02 | 0.7706$\pm$0.01 | 0.7356$\pm$0.02 | 0.7526$\pm$0.01 | 0.5370$\pm$0.02 | 0.6448$\pm$0.02 |
| GPT-4-turbo | 0.6800$\pm$0.01 | 0.7824$\pm$0.01 | 0.7944$\pm$0.01 | 0.7883$\pm$0.00 | 0.5660$\pm$0.01 | 0.6772$\pm$0.01 |
| GPT-4o-mini | 0.6317$\pm$0.01 | 0.7727$\pm$0.01 | 0.7211$\pm$0.01 | 0.7459$\pm$0.01 | 0.5380$\pm$0.01 | 0.6420$\pm$0.01 |
| Open-Source Models |  |  |  |  |  |  |
| Llama3.1-405B | 0.6958$\pm$0.01 | 0.7965$\pm$0.01 | 0.7989$\pm$0.01 | 0.7976$\pm$0.00 | 0.5925$\pm$0.02 | 0.6951$\pm$0.01 |
| Qwen2-72B | 0.6725$\pm$0.01 | 0.7747$\pm$0.01 | 0.7944$\pm$0.01 | 0.7844$\pm$0.01 | 0.5515$\pm$0.01 | 0.6679$\pm$0.01 |
| Llama3.1-70B | 0.6708$\pm$0.01 | 0.7796$\pm$0.01 | 0.7822$\pm$0.01 | 0.7809$\pm$0.01 | 0.5597$\pm$0.02 | 0.6703$\pm$0.01 |
| Mistral-8X22B | 0.6383$\pm$0.01 | 0.7544$\pm$0.00 | 0.7678$\pm$0.01 | 0.7610$\pm$0.01 | 0.5089$\pm$0.01 | 0.6350$\pm$0.01 |
| Mistral-8X7B | 0.6825$\pm$0.01 | 0.7469$\pm$0.00 | 0.8722$\pm$0.02 | 0.8046$\pm$0.01 | 0.4779$\pm$0.00 | 0.6413$\pm$0.00 |

#### Impact of Parameters on Performance.

As shown in Figure 3, model parameter size has a clear impact on performance across tasks such as symptom detection, symptom severity evaluation, and treatment outcome prediction. Larger models consistently outperform smaller models, exhibiting higher F1 (Weighted) scores and lower MAE. This trend indicates that increasing model size enhances the model’s ability to handle complex tasks (Wen et al., ), especially in identifying subtle patterns related to psychological symptoms and predicting treatment outcomes.

Figure: Figure 3: The impact of model parameters on symptom detection, symptom severity evaluation, and treatment outcome prediction. Dashed lines represent the best-performing closed-source models.
Refer to caption: x3.png

### 4.3 Ablation Study

#### Impact on Items-aware Reasoning.

The ablation study, as shown in Figure 4, demonstrates the significant impact of items-aware reasoning on both psychological assessment and treatment outcomes evaluation tasks. Removing this feature led to a substantial decline in performance across all models. For psychological assessment tasks, models like GPT-4o and GPT-4 experienced noticeable drops in their ability to accurately detect symptoms and assess severity, as reflected by decreases in F1 scores and increases in error metrics. Similarly, in treatment outcomes evaluation, the absence of items-aware reasoning resulted in reduced performance, though the impact was less pronounced compared to psychological assessment. These results underscore the importance of items-aware reasoning in improving the precision of the models in these tasks.

Figure: Figure 4: The impact of items-aware reasoning on psychological assessment and treatment outcomes evaluation using human-annotated data across four OpenAI models.
Refer to caption: x4.png

## 5 Conclusion

We introduced IPAEval, which can address the limitations of existing therapeutic outcome evaluation approaches by shifting the focus from therapist-centered, single-session assessments to a comprehensive, client-informed framework. By leveraging clinical interviews and integrating both cross-session client-contextual and session-focused client-dynamics assessments, IPAEval provides a more holistic evaluation of treatment outcomes. Experiments on the TheraPhase dataset validate its effectiveness in tracking symptom severity and therapeutic progress over multiple sessions, demonstrating significant improvements over previous single-session models. This advancement highlights the importance of client-centered, multi-session evaluations for enhancing mental health care and guiding treatment adjustments.

## Limitations

The limitations of this paper are as follows: (1) Due to the shortage of professional psychological annotators, only two individuals were involved in a limited amount of data labeling. This resulted in fewer human-aligned experimental data. Future research should focus on developing more multi-session datasets that include psychological assessment scores. (2) As the amount of client information increases, smaller models with fewer parameters struggle to follow instructions effectively. This limits the scalability and performance of these models in more complex scenarios. Future research should explore strategies to enhance model adaptability in handling larger client information inputs.

## Appendix A Items-Aware Reasoning Prompts in Experiments

## Appendix B Psychological Assessment Prompts in Experiments

## Appendix C Items-Aware Reasoning Output Example

**Table 8: Items-Aware Reasoning Output Example**
| Session: |
| --- |
| Therapist: So, thank you for coming in today. |
| Client: Yes. |
| Therapist: How are you feeling today? |
| Client: I feel great actually. |
| Therapist: Yeah? Good. |
| Client: Yeah. |
| Therapist: Good. |
| Client: I feel good. |
| Therapist: And so you did your clarifications, value clarifications- |
| Client: Yeah. |
| Therapist: -and what are your top five? |
| Client: Yes. It was a good, uh, experience for me. It was different. It was different than usual. There were several things that were different, and, uh, the number one value that I put was self-respect. And I-I don’t even know if self-respect has ever been in my top five let alone my number one. |
| Therapist: Really? |
| Client: Yeah. And, um— |
| Therapist: Do you have any idea why that is? |
| Client: I do have an idea, I think, why that is. Um, I think that there’s been a few things that have happened recently and something that really came to my awareness, when I visited with my family, is that I have consistently through my whole life, probably, put other people first. And I have consistently, uh, almost not even considered myself in the equation. It was, uh, kind of sad in a way, at the time that I realized it. Uh, I didn’t realize how severe it actually was, but I was kind of glad that I realized it because I feel like it’s never too late to change- |
| Therapist: True. |
| Client: -and I feel like I can- I can, uh, respect and value myself just as much as I have other people. I know that’s important. And I feel like when I do that, I’m a better person for other people as well. |
| Therapist: Mm-hmm. By not putting yourself on the back burner so much? |
| …… |
| Items-Aware Reasoning Result: |
| Client Statement: I have consistently through my whole life, probably, put other people first. And I have consistently, uh, almost not even considered myself in the equation. |
| Symptom Category: Interpersonal Sensibility |
| Specific Symptom: Feeling others do not understand the client or are unsympathetic. |
| Presence: Yes |
| Explanation: The client’s statement indicates that they have been prioritizing others over themselves, which could be a sign of feeling misunderstood or not receiving empathy from others. |
| …… |
| Assessment Score: |
| …….; Interpersonal Sensitivity: 1;…… |

## Appendix D Output Formatting Errors

In our two experiments, OpenAI series models produced no errors in output formatting, whereas open-source models encountered numerous issues. Specifically, the Figure 5 below shows the error statistics for open-source models during the Assessment task, with the main issue being incorrect output that did not follow the Pydantic-defined JSON format.

Figure: Figure 5: Error distribution across different models.
Refer to caption: x5.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Could you provide more detail on why only 30 sessions were annotated for psychological assessment and 60 for treatment outcomes, given that these numbers are quite low relative to the available dataset? While I understand that annotating the entire dataset may not be feasible, expanding beyond this limited sample would likely improve the robustness of IPAEval’s evaluation.

Additionally, have you considered using multiple LLMs to cross-annotate the dataset? Cross-annotation with multiple models could help improve labeling quality and consistency while reducing the manual effort required, potentially increasing the reliability of the labeled data.
soundness: 3
strengths: Originality: The paper offers a novel approach to psychotherapy outcome evaluation by shifting from a therapist-centered, single-session paradigm to a client-centered, multi-session framework. This approach, embodied in the IPAEval framework, introduces a unique perspective within mental health assessments by prioritizing the client’s evolving experience across sessions.

Quality: The paper's experimental design is comprehensive, testing multiple LLMs with various statistical metrics. The authors also conduct a detailed ablation study, which highlights the importance of the items-aware reasoning feature by isolating its impact on the framework’s overall performance. 

Clarity: The paper is well-organized and easy to follow. Each part of IPAEval—multi-session tracking, session-specific assessments, and items-aware reasoning—is clearly explained, and the figures and tables effectively illustrate IPAEval’s advantages and performance.

Significance: IPAEval addresses a key gap in mental health care by enabling evaluations of therapy outcomes from the client’s perspective, a shift that could substantially improve the quality of care. The framework provides practical tools that can streamline and enhance the outcome assessment process, helping practitioners to track treatment effectiveness over time and adjust interventions more responsively.
summary: The paper presents IPAEval, a new framework for evaluating therapy outcomes by focusing on clients’ experiences over multiple sessions rather than isolated single sessions. Unlike traditional methods that center on the therapist's role, IPAEval shifts the focus to clients, aiming to capture their mental health journey more fully.
Its key contributions include: 1) IPAEval assesses changes over time, using information from multiple therapy sessions to provide a big-picture view of clients’ progress. 2) It also evaluates each session individually, noting immediate shifts in the client’s state and responses. 3) The framework uses a sophisticated reasoning system to interpret client statements more accurately, particularly in aligning with psychometric test criteria.
Tested on the TheraPhase dataset, IPAEval showed improved performance through items-aware reasoning mechanisms in both symptom detection and outcome prediction tasks.
weaknesses: 1: Despite having 110 and 800 client sessions available, the authors only annotated 30 for psychological assessment and 60 for treatment outcomes, using this small subset to establish a “Gold Model” for generating reference scores. This approach may introduce bias, as basing the Gold Model on such a small sample could affect the reliability of the labeled data, especially in representing the full diversity of client interactions.

2. The paper presents IPAEval as an improvement over models like ClientCAST and CPsyCoun, emphasizing its client-centered, multi-session approach. However, it doesn’t provide direct performance comparisons to back up these claims. While Table 1 outlines conceptual differences (such as focus and theoretical grounding), there’s no side-by-side benchmarking on tasks like symptom detection or outcome prediction. Without this empirical evidence, IPAEval’s advantages remain mostly theoretical. Adding direct comparisons with existing models would make the practical impact of IPAEval clearer and more convincing.

3. Figure 2, which aims to show the whole IPAEval framework, doesn’t clearly illustrate the Session-Focused Client-Dynamics Assessment module, especially how it tracks changes within a session from start to end. This mismatch between the figure and the text can make it confusing for readers trying to understand how session-specific changes are actually captured. Adding markers for initial and final assessments within each session would make the figure align better with the explanation in the text and help clarify how this part of the framework works.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Questions:
1. I’m confused by several terms – psychological assessment, symptom detection, severity assessment. It sounds to me that symptom detection, severity assessment are parts of psychological assessment. Can you clarify these definitions better?
2. I noticed that there are contrasting results between models on symptom severity in Figure 4 – any reason/hypothesis to explain this?

Minor:
1. Confused by the figure 2: the PSDI<0 should mean the good treatment outcomes but the sad face there indicates negative results. Can you clarify this?
2. Appreciate the formal definition on section 3 but it seems too long (~3 pages) for a 10 pages paper. Recommend to add more analysis/ elaboration on the results.
3. The fonts of the Figure 3 and Figure 4 are very small and cannot see them clearly without zooming.

Some useful but undiscussed related works are [1], [2] and [3]

In general, I can see that having a real-time therapeutic outcome assessment could be important for mental health practitioners. However, the paper lacks analysis/experiments on the effect of time (within session and cross-session) and lacks validation on the dataset (TheraPhase) to ensure it is close to real-world. I think having these may strengthen the contributions of the work and better differentiate this work to current works (like ClientCAST). Apart from that, the (arguably stat. insignificant) results (Figure 4: error bars overlapping before and after the reasoning prompt) in tracking therapeutic outcomes suggests a deeper analysis on the therapeutic outcomes e.g. which part of the therapeutic outcomes can be tracked better.

[1] https://arxiv.org/pdf/2407.00870

[2] https://arxiv.org/pdf/2405.19660

[3] https://arxiv.org/pdf/2310.13132
soundness: 2
strengths: 1. Proposes a generalizable pipeline transforming from client information/dialogue to metrics (treatment outcomes)
2. Conducts an ablation study on the pipeline, specifically on removing the item-aware reasoning (items classified by psychometric test, explanation) to see if it affects the performance of tracking both psychological assessments and treatment outcomes.
3. Performs human annotations for three tasks (psychological assessments: symptom detection and severity assessment; treatment outcomes) to select the golden models as baselines
summary: This work presents a framework IPAEval, designed for evaluating the treatment outcome based on the cross-session clients’ context in clinical interview. It makes use of client information (including profile, session history), psychometric test before interview, client’s utterances, and “item-aware reasoning” (items classified by psychometric test, explanation) to obtain the score of PSDI (a metric adopted in Symptom Checklist-90). This work also develops a TheraPhase dataset to simulate cross-session interviews based on CPsyCoun dataset (multi-turn dialogue for chinese psychological counseling). Authors experimented on TheraPhase to evaluate the treatment outcomes between models across sessions, and further selected 2 datasets (High-low quality counseling and AnnoMI: English multi-turn) to evaluate the psychological assessment. They find that the IPAEval performs better than the single-session models on tracking the symptom severity and treatment outcomes.
weaknesses: 1. The main claim of the paper is to evaluate the treatment outcomes across sessions. However, there is no validation/exploration on the TheraPhase dataset to show that it is reflective of real world outcomes (i.e. to ensure the distributions of treatment outcomes across time is close to real-world clients). 
2. The TheraPhase dataset is based on a Chinese counseling dialogue dataset. The performance difference between models (e.g. Llama vs. Qwen) may be affected by the language's comprehension ability, rather than the tracking ability on treatment outcomes. The findings on this dataset is questionable.
3. The goal and contributions of the paper are unclear. The IPAEval is targeted to evaluate the treatment outcomes (by authors’ definition is the clients’ states etc) but the authors also evaluate psychological assessment (e.g. symptom detection and severity assessment).
4. The “item-aware reasoning” sounds misleading as it does not contain a multi-step thinking process on rationales but only one intermediate step of giving the rationale behind the item classification. Maybe consider changing the naming of reasoning or provide more concrete examples/analysis.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses above.
soundness: 2
strengths: 1. The paper focuses on evaluate therapeutic outcome via LLMs, aiming at solving a meaningful problem.
2. It integrates some psychology tests and metrics into LLM evaluation framework. It is very interesting.
summary: This paper focuses on evaluate therapeutic outcome via LLMs, aiming at solving a meaningful problem. It not only proposes an evaluation framework but also contributes a precious dataset. The method shifts the focus from therapist-centered, single-session assessments to a comprehensive, client-informed framework.
weaknesses: 1. The TheraPhase dataset is constructed via GPT-4. There is no human evaluation to guarantee the reliability of the dataset.
2. Experiments only conducted on one or two datasets. It would be better to evaluate on more datasets.
3. I think the proposed approach is not novel. Most parts in the approach are already existing, such as the psychometric test, client profile and PSDI.
4. In this paper, GPT-4o is selected as the Gold Model. However, as shown in Table 4, its performance is also poor. So there may be deviation in the results.

Meta-review: The paper introduces IPAEval, a framework designed to evaluate therapy outcomes by focusing on clients' experiences across multiple sessions rather than isolated, single-session evaluations. IPAEval captures changes in clients' mental health over multiple therapy sessions, providing a big-picture view of progress. It introduces an advanced reasoning mechanism aligned with psychometric test criteria, enabling accurate symptom detection and outcome prediction. The paper introduces the TheraPhase dataset developed to simulate cross-session therapy interviews, based on the CPsyCoun dataset (multi-turn dialogues for Chinese psychological counseling). It also includes evaluations on datasets like High-Low Quality Counseling and AnnoMI (English multi-turn) for broader validation. IPAEval outperforms single-session models in tracking symptom severity and predicting treatment outcomes.

**Strengths identified**:
1. The client-centered, multi-session framework introduced by IPAEval is a novel approach that shifts the focus from traditional therapist-centered, single-session evaluations to a more holistic view of the client’s mental health journey.

2. Comprehensive ablation studies have been conducted isolating the importance of the items-aware reasoning feature in symptom detection and treatment outcomes.

3. Human annotations for tasks such as symptom detection, severity assessment, and treatment outcomes to establish robust baselines.

**Weaknesses that need addressing**:
1. Although IPAEval is presented as an improvement over models like ClientCAST and CPsyCoun, the paper does not include direct performance comparisons with these models, making its advantages appear theoretical rather than empirical.

2. The dataset is constructed using GPT-4 without human evaluation, which raises concerns about its reliability and quality.

Decision: Reject