RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Automatic Detection of Therapeutic Elements in Clinical Text
The growing use of technology in psychotherapy has expanded the possibilities of automating text data collection, such as therapy transcripts. This has increased interest in using natural language processing and ML to automatically detect, classify, or score therapist behaviors and client responses. For instance, recent research has attempted to identify empathy-related cues in counseling dialogue (Tao et al., 2024 ; Tavabi et al., 2023 ) . Other studies have focused on classifying types of reflections or questions posed by therapists (Can et al., 2016 ; Pérez-Rosas et al., 2017 ) .
Despite this progress, several significant challenges remain. First, therapy transcripts inherently contain sensitive information, restricting the available data for model training. Second, counselor behaviors are highly context-dependent; the same microcounseling skill may evoke different CFs. For example, respect for the client’s autonomy generally conveys an attitude of collaboration and partnership but is also an inherent part of goal alignment. Third, most existing studies focus on one specific behavioral construct (e.g., identifying therapist empathy alone) rather than a broad taxonomy encompassing multiple change principles and a wide range of microcounseling skills.
Recent work explored fine-grained analysis of psychotherapy sessions. Mayer et al. ( 2024 ) developed models that predict client emotions and therapist interventions at the utterance level. Similarly, Gibson et al. ( 2019 ) introduced multi-label, multi-task deep learning methods that simultaneously predict multiple behavioral codes within therapy dialogues. Despite these advancements, most existing studies still emphasize isolated behavioral constructs (e.g., identifying therapist empathy alone) rather than addressing a broader taxonomy of multiple change principles and diverse microcounseling skills. These limitations motivate the need for more holistic, theory-driven computational approaches that can parse complex therapeutic interactions at multiple levels of granularity.
Taxonomies and Graph-Based Modeling Approaches
Researchers have explored structured representations like taxonomies or ontologies to capture the hierarchical and interconnected nature of therapeutic elements, such as broad CFs and more granular microcounseling skills, and have applied these frameworks to classify symptoms, diagnoses, and interventions in mental health research (Evans et al., 2021 ) . However, few existing taxonomies systematically link higher-level CFs (e.g., the therapeutic relationship) to actionable skills (e.g., reflective listening, validating) that instantiate those factors in practice.
Graph-based ML offers a robust avenue for modeling these relationships. With methods such as Graph Convolutional Networks (GCNs) (Kipf and Welling, 2017 ) or GraphSAGE (Hamilton et al., 2017 ) , one can encode both the textual features of nodes (e.g., descriptions of skills) and the relational structure (e.g., which skills evoke which CFs) into a unified embedding space. Graph ML has shown promise in diverse classification tasks—among others, in detecting suicidality Lee et al. ( 2022 ) —suggesting that a similar strategy could be applied to psychotherapy discourse.
The CFiCS Graph
We construct a structured knowledge graph of therapeutic practices centered around five main types of nodes: the root of our graph, the CF therapeutic alliance; the CF elements (i.e., Bond, Goal Alignment, and Task Agreement), which define integral components of the CF therapeutic alliance; intervention concepts (ICs) (i.e., Empathy, Acceptance, and Positive Regard; Collaboration and Partnership), which express specific approaches to fostering the CF elements; therapeutic skills (i.e., Open-Ended Questions, Reflective Listening, Affirmation, Validation, Genuineness, Respect for Autonomy, Asking for Permission), which are practical techniques therapists use; and examples , which are therapist statements that illustrate how specific skills and concepts are applied.
Figure 1 a visualizes the connection patterns between the node types. The graph is heterogeneous, incorporating different node types and relationships, and sparse. Most examples link to one CF or one IC and one or two skills. Hence, the average path length is short due to the triadic pattern: example \rightarrow skill \rightarrow CF or IC. The structure is hierarchical, with CFs at the top, ICs as an intermediate layer, and therapeutic skills and examples forming the practical, grounded components. Clusters naturally form around specific CFs and ICs, creating thematic groupings.
The graph’s relationships are multi-relational and include edges like fosters , linking examples to the CFs they develop; expresses , connecting skills to ICs or examples to concepts; and demonstrates , linking examples to the therapeutic skills they showcase. This semantic structure provides a foundation for our classification approach.
The CFiCS Classification Model
We propose a model shown in Figure 1 b to classify CFs, ICs, and associated therapeutic skills for textual input. We leverage graph ML to exploit the topology of the nodes in the CFiCS graph and combine it with the textual embeddings produced by a pretrained language model. CFiCS enables inductive classification of previously unseen nodes, which do not have explicit edges but can still leverage the structural patterns learned from the graph during training in addition to the textual features.
Input graph structure
Let \mathcal{G}=(\mathcal{V},\mathcal{E}) be a heterogeneous graph where: \mathcal{V}=\mathcal{V}_{r}\cup\mathcal{V}_{f}\cup\mathcal{V}_{c}\cup\mathcal{V}_{s}\cup\mathcal{V}_{e} represents the set of all nodes, where \mathcal{V}_{r} the root node, \mathcal{V}_{f} the CF nodes, \mathcal{V}_{c} the IC nodes, \mathcal{V}_{s} the skill nodes, and \mathcal{V}_{e} the example nodes. \mathcal{E}\subseteq{(v_{i},v_{j})|v_{i},v_{j}\in\mathcal{V}} represents the bidirectional edges between nodes in the graph. There exist six distinct edge types, also visualized in Figure 1 b:

Fosters relation : (v_{e},v_{f}) denotes a relationship between an example node and a CF node, indicating that the example fosters the development of a specific CF in therapy.

Expresses relation : (v_{e},v_{c}) connects an example node to an IC node, signifying that the example expresses the therapeutic intention of an IC (e.g., reflective listening expresses empathy, acceptance, and positive regard).

Demonstrates relation : (v_{e},v_{s}) links an example node to a skill node, showing that the example demonstrates the practical application of a specific therapeutic skill.

Includes relation : (v_{f},v_{c}) links a CF node to an IC node, indicating that the IC is a specific approach to operationalizing the broader CF.

Conveys relation : (v_{s},v_{c}) connects a skill node to an IC node, signifying that the skill conveys the therapeutic intention of an IC.

Supports relation : (v_{s},v_{f}) connects a skill node to a CF node, highlighting that a microcounseling skill supports a CF element.
Node features
For each node v\in\mathcal{V} , let \tau_{v} denote the associated textual input. For nodes representing CFs, ICs, or skills, \tau_{v} consists of the node’s name and detailed description. In contrast, for example nodes, \tau_{v} comprises solely the example text. A pretrained language model \mathcal{M} (e.g., BERT Devlin et al. ( 2019 ) or ClinicalBERT Alsentzer et al. ( 2019 ) ) computes the feature vector for each node.

\mathbf{x}_{v}=\mathcal{M}(\tau_{v})

where \mathbf{x}_{v}\in\mathbb{R}^{d} and d is the embedding dimension of the chosen model (e.g., d=768 for BERT).
Learning task
We seek to learn node embeddings via a message passing framework inspired by GraphSAGE Hamilton et al. ( 2017 ) . Given training node representations \{\mathbf{x}_{v}\mid v\in\mathcal{V}\} and graph structure \mathcal{E} , we iteratively update each node v ’s representation as

\mathbf{h}_{v}^{(l)}=\gamma^{(l)}\!\Bigl{(}\mathbf{h}_{v}^{(l-1)},\mathrm{AGG}\!\bigl{\{}\mathbf{h}_{u}^{(l-1)}\mid u\in\mathcal{N}(v)\bigr{\}}\Bigr{)}

where \mathbf{h}_{v}^{(l)} is v ’s embedding at layer l , \gamma^{(l)} is a learnable transformation (often a nonlinear MLP), \mathrm{AGG} is a neighborhood aggregation function (e.g., mean or pooling aggregator), and \mathcal{N}(v) denotes v ’s neighbors. Although we present this in a GraphSAGE-oriented formulation, the same learning task is fully model-agnostic : by substituting different forms of \mathrm{AGG} (such as attention-based aggregation in GAT Veličković et al. ( 2018 ) or weighted normalized sums in GCN Kipf and Welling ( 2017 ) ) and choosing a suitable \gamma^{(l)} , one can instantiate a variety of GNN variants without altering the underlying message passing framework.
Classification tasks
Given an example node v\in\mathcal{V}_{e} with embedding \mathbf{h}_{v} , we define a single linear classification layer that encompasses all labels (i.e., for CFs, ICs, and skills). Let \mathcal{T}=\{\text{CF},\text{IC},\text{Skill}\} denote the task types, and let \mathcal{V}_{t} be the set of labels for each task t\in\mathcal{T} . We construct a single parameter matrix W and bias vector \mathbf{b} such that the row segments of W and the corresponding portions of \mathbf{b} map to the different tasks. Formally, the probability of assigning label y from \mathcal{V}_{t} to node v\in\mathcal{V}_{e} is computed by slicing the relevant portion of the linear output and applying a softmax.

p_{t}(y\mid v)\;=\;\mathrm{softmax}\Bigl{(}\bigl{(}W_{slice(t)}\,\mathbf{h}_{v}+\mathbf{b}_{slice(t)}\bigr{)}\Bigr{)}

where y\in\{1,\dots,|\mathcal{V}_{t}|\} , \mathbf{h}_{v} is the shared node embedding (e.g., obtained from a graph neural network), W_{\text{slice}(t)} and \mathbf{b}_{\text{slice}(t)} refer to the rows in W and \mathbf{b} that correspond to the label set \mathcal{V}_{t} , and \mathrm{softmax}(\cdot) is applied to the sliced logits to form a probability distribution specific to the task t .
Optimization objective
We treat each task t\in\mathcal{T} as a separate multi-class classification problem and define a cross-entropy loss \mathcal{L}_{t} on the predictions p_{t}(y_{v}\mid v) . Formally,

\mathcal{L}_{t}\;=\;-\sum_{v\in\mathcal{V}_{e}}\log\,p_{t}(y_{v}\mid v),

where y_{v}\in\mathcal{V}_{t} denotes the ground truth label for node v in task t . Our overall multi-task objective is a linear combination of these losses

\mathcal{L}\;=\;\sum_{t\in\mathcal{T}}\lambda_{t}\,\mathcal{L}_{t},

with weights \{\lambda_{t}\} controlling the relative importance of each task. Intuitively, each \mathcal{L}_{t} measures how well the model performs on task t , and the hyperparameters \lambda_{t} balance their contributions to the total loss.
Inference
A new node is isolated during inference, meaning it has no edges and lacks direct neighbors in the graph. When the set of neighbors \mathcal{N}(v) is empty, the aggregator defaults to relying solely on h_{v}^{(l-1)} . However, the model’s learned weights still capture global patterns from the training graph. The aggregator, which processes node features, has learned the overall graph structure and label signals, allowing it to embed the new node in a graph-aware feature space. Even without access to neighbors, the aggregator’s learned MLP transforms the new node’s features with knowledge learned during training on graph edges.
Implementation Details
We use either bert-base-uncased or ClinicalBERT from Huggingface Transformers Wolf et al. ( 2020 ) to encode the node name and description by average-pooling the last hidden state. We implement the model in Python using PyTorch Geometric.
The model processes 2\times 768 input channels, 768 for the text embedding and 768 for the graph embedding, through hidden layers of 64 channels in a three-layer architecture with a 0.5 dropout rate. It handles three task components: three CFs, two ICs, seven skills, and a neutral class for each task. During training, a forward pass computes logits for these tasks, followed by slicing the output into separate components and computing the task-specific losses weighted by predefined task weights (defaulting to uniform). The Adam optimizer Kingma and Ba ( 2017 ) updates the parameters, with training configured for up to 400 epochs, a learning rate of 1e-3, and a weight decay of 1e-4. The model performs validation at each epoch, tracks the lowest loss, and stops early if it detects no improvement for 50 epochs.
Dataset
Our dataset consists solely of manually created and curated examples drawn from established psychotherapy literature, rather than real patient conversations.
The dataset is structured as a heterogeneous, undirected, and multi-relational CFiCS graph modeling therapeutic practices. It includes three CF elements (e.g., Bond), two ICs (e.g., Collaboration and Partnership), and seven therapeutic skills (e.g., Reflective Listening). The dataset contains 69 fully annotated examples, including CF, IC, and skill annotations, and an additional 112 examples annotated only for therapeutic skills, illustrating their application contextually. An expert selected and curated these examples from respected psychotherapy literature on therapeutic alliance. Specifically, we identified reference samples directly from Fuertes ( 2019 ); Miller and Moyers ( 2021 ); Bailey and Ogles ( 2023 ) as representative instances of therapeutic interaction for each class. These original excerpts served as a reference for generating new synthetic samples using ChatGPT, ensuring alignment with the themes, styles, and therapeutic concepts illustrated in the literature.
Examples not assigned to any class are designated as neutral . The dataset is split into training and testing subsets (80/20), with k-fold cross-validation applied to the training data for model evaluation.
Baselines
We compare our approach against two baselines: a Random Forest (RF) Breiman ( 2001 ) and a BERT-based architecture. Additionally, we evaluate and compare several graph ML methods, including GAT, GCN, and GraphSage, to assess their effectiveness in modeling the relationships within the CFiCS graph. For the RF baseline, we convert each utterance into TF-IDF features and create a multi-output target vector where CFs, ICs, and skills are multi-class tasks. We then train a MultiOutputClassifier , effectively training one RF per output dimension. For the BERT-based model, we finetune a pretrained encoder that extracts a pooled [CLS] representation and optimizes three classification heads using cross-entropy loss for the CF, IC, and skill prediction.
Metrics
We report macro- and micro-averaged F1 scores for the multi-class tasks.
The macro F1 treats each class equally, computing the mean F1 over classes, whereas the micro F1 aggregates contributions from all classes to compute precision and recall overall. The micro F1 tallies the total number of correctly predicted skill labels versus all predictions, while the macro F1 averages the F1 values per skill category.
In addition, we use Precision@k and Recall@k as indicators of cluster quality to evaluate how well the model ranks and groups related concepts. Precision@k measures the proportion of relevant samples among the top-k similar nodes, reflecting how accurately the model embedds input features. Recall@k quantifies the proportion of relevant samples retrieved within the top-k similar samples, indicating how well the model captures relevant clusters. Higher Precision@k and Recall@k scores suggest that similar concepts are embedded closely together, providing additional validation of the quality of learned representations.
Performance of non-graph baselines
Table 1 reports the micro and macro F1 scores for various models on the CF, IC, and skill classification. We compare several baselines, including traditional TF-IDF and BERT multi-task classifiers, graph-based models without BERT, and our proposed CFiCS variants that integrate clinical BERT features.
The TF-IDF multi-task Random Forest baseline achieves relatively modest performance, with micro F1 scores of 52.50, 74.59, and 53.02 for CF, IC, and Skill, respectively, and corresponding macro F1 scores of 20.43, 38.04, and 49.77. The BERT multi-task classifier improves these numbers considerably (e.g., obtaining 59.69 and 59.60 micro F1 for CF and Skill, respectively), indicating the benefit of richer contextual representations.
Effect of graph structure on performance
Graph-based models without BERT features exhibit mixed results. For example, the GCN and GAT models without BERT yield micro F1 scores in the range of 55.70–56.89 for CF and around 70 for IC. However, their performance on the skill task is markedly lower (with micro F1 scores of 19.37, 23.24, and 14.39 for GCN, GAT, and GraphSage, respectively). This suggests that relying solely on graph structure without contextual text representations can be detrimental, particularly for the more nuanced skill classification.
In contrast, our proposed methods that incorporate clinical BERT features within the CFiCS framework demonstrate substantial improvements. Both the CFiCS GCN and CFiCS GAT variants with ClinicalBERT improve performance across all tasks. In particular, the CFiCS GAT with ClinicalBERT variant achieves micro F1 scores of 91.98, 93.21, and 93.83 for CF, IC, and Skill, respectively, with corresponding macro F1 scores of 82.39, 89.35, and 92.18.
The CFiCS models that integrate BERT features achieve the best results. The model labeled as CFiCS with ClinicalBERT (ours) achieves a micro F1 of 95.04 on CF and 96.30 on Skill, with perfect performance on the IC task (100.00 in both micro and macro F1). Similarly, CFiCS with BERT (ours) shows competitive performance with micro F1 scores exceeding 95% for CF, IC, and Skill, and macro F1 scores that are consistently high.
Precision and recall at different k
We evaluate the embedding quality by comparing the precision and recall at varying k ranging from one to ten.
Figure 2 presents the aggregated precision and recall for values of k ranging from 1 to 10. As expected, recall increases as k grows since more relevant items are retrieved, while precision declines due to the broader set of top-k retrievals. Table 2 provides a more detailed breakdown of precision and recall across different class types.
GraphSage demonstrates superior recall and precision at higher k, suggesting that its embeddings create more cohesive clusters of relevant nodes, making it more effective for retrieving multiple correct labels. GCN lags behind both models in rank-1 precision but improves recall at k=10. While its embeddings do not strongly differentiate the best match, they still capture helpful information for broader retrieval. Overall, GAT is best for fine-grained differentiation, GraphSage generates well-structured clusters that enhance overall representation quality, and GCN provides moderate performance with embeddings that favor broader contextual generalization.
Qualitative Results
In Figure 3 , we visualize the learned embeddings in 3-dimensional space using the dimension reduction method t-SNE van der Maaten and Hinton ( 2008 ) . We color the samples based on their respective class for all three class types (CF, IC, and skill).
We observe that skills form well-separated clusters, indicating that the model effectively distinguishes between different microcounseling techniques.
Additionally, higher-level CF influence representation structure, as utterances containing a skill alone (e.g., Genuineness, ✖ in Figure 3 c) are embedded separately from those where the skill co-occurs with broader therapeutic elements (e.g., Bond (B) or Empathy, Acceptance, and Positive Regard (EAR) (see the ✩ and ✚ sign in Figure 3 a and b). This suggests that the model captures fine-grained skill differentiation and hierarchical relationships between skills and ICs. Moreover, higher-level features such as the CF and IC cluster more closely together, showing less separation. The context of the skills separates them at the lower, more fine-grained skill level. This finding indicates that while the model captures distinct skill representations, the broader context in which practitioners apply these skills is crucial for differentiation.
Ethical and Impact Considerations
Psychotherapy transcripts contain highly sensitive and personal information, and patients are particularly vulnerable data subjects. However, our dataset consists solely of manually created and curated examples from the literature, not real patient conversations, ensuring no private or identifiable data is used. Using this method with real-world therapy data would require strict attention to privacy and confidentiality, ensuring compliance with data protection regulations.
ML models trained on limited or biased datasets can inadvertently learn and propagate biases in the data. Since our dataset is relatively small and manually curated, there is a risk that certain features are over- or underrepresented, potentially impacting the generalizability of our results.
Furthermore, automated classification of psychotherapy content could be misused if applied without proper oversight. For instance, misclassification of therapeutic interactions could lead to inaccurate feedback for therapists, and reliance on imperfect AI-driven assessments might undermine professional judgment. Therefore, the model should be deployed as an assistive tool rather than replacing human expertise.

Our work has implications for both psychotherapy research and practice.
Firstly, traditional research methods often fail to capture the complexity of the patient-therapist interaction process (Lundh and Falkenström, 2019 ) . For example, studies on therapeutic alliance patterns typically rely on post-session evaluations, which may oversimplify evolving patient-therapist dynamics (Falkenström and Larsson, 2017 ) . Automatic assessment of in-session "microprocesses" (Lundh and Falkenström, 2019 ) could offer a more precise understanding of common factors development, identifying key therapist skills linked to treatment success across modalities and client profiles.

Secondly, psychotherapy quality in practice depends on research-driven training and performance-based feedback (Baldwin and Imel, 2013 ) . Yet, many clinicians receive little feedback after initial training (Moyers et al., 2005 ) . A system providing session-by-session feedback on common factor usage on various levels of granularity could help therapists set incremental improvement goals and track progress in real time (Rousmaniere, 2016 ) .
Thirdly, automating common factor feedback would enable integration with digital health tools, linking therapist skill use to broader treatment data, including symptom levels and session attendance.
Conclusion
We presented a graph ML classification method CFiCS to classify common factors, intervention concepts, and associated skill usage.
Overall, the results demonstrate that combining textual features from ClinicalBERT with graph-based ML in the CFiCS framework significantly enhances classification performance, particularly for the challenging skill prediction task, and outperforms conventional TF-IDF, BERT, and pure graph-based baselines.