RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Limitations
One primary limitation is the dataset size. We evaluate our method using a manually curated dataset alongside examples from the literature. While the model demonstrates promising performance, the relatively small sample size may limit generalizability and increase the risk of overfitting to specific linguistic patterns or annotation biases.
A second limitation is language dependence. Our study focuses exclusively on English-language data, and we do not assess whether the method generalizes to other languages or multilingual settings. Given that therapeutic discourse varies linguistically and culturally, future work should explore cross-lingual adaptations and assess whether pretrained multilingual models (e.g., XLM-R, mBERT) can extend classification performance to other languages. An additional challenge is linguistic ambiguity per se. Identical statements can have different meanings depending on the context. Prosodic features play a key role in language comprehension Dahan ( 2015 ) , and models trained on spoken language outperform text-based approaches (Singla et al., 2020 ) . Thus, CFiCS classification could benefit from incorporating auditory and visual cues.
Another limitation is the lack of external validation on out-of-distribution datasets. Our dataset consists solely of manually curated literature examples and synthetically generated examples, rather than real therapist-patient interactions. While this approach has ethical advantages by avoiding privacy concerns, it limits the clinical relevance of the dataset.
Additionally, the effectiveness of common factor usage depends on their thoughtful application rather than mere frequency. Therapist responsiveness, seen as a “metacompetency” integrating skills like executive functioning and reflection (Hatcher, 2015 ) , is more valuable than rigid technique use (Stiles and Horvath, 2017 ) .
Future Work
Future work should expand the dataset and use real therapy interactions in different settings and with therapists using different approaches. Additionally, it may be beneficial to explore multilingual extensions, expand the CFiCS graph structure, validate on external corpora, and consider integrating prosodic features.
Acknowledgments
This work is funded by the Research Council of Norway (grant 321561) through the Adaptive Level of Effective and Continuous Care (ALEC2) project and was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation.
The model training was enabled by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
Figure 1: (a) Edge types and connection patterns between the node types for the common factor Therapeutic Relationship and (b) Common factor and microcounseling skill prediction with our CFiCS classification model.; (a) Edge types and connection patterns between the node types for the common factor Therapeutic Relationship and (b) Common factor and microcounseling skill prediction with our CFiCS classification model.
Figure 2: Precision and recall at different values of k (ranging from 1 to 10) for graph-based models, including GCN, GAT, and GraphSage.; Precision and recall at different values of k (ranging from 1 to 10) for graph-based models, including GCN, GAT, and GraphSage.
Figure 3: Visualization of the 197 learned embeddings after aggregating contextual information from the graph. TSNE reduces the 832-dimensional vectors (768 text features + 64 graph features) to two dimensions. Each point represents a node embedding, with colors indicating the three different class types. The embeddings are extracted from the final hidden layer during the forward pass.; Visualization of the 197 learned embeddings after aggregating contextual information from the graph. TSNE reduces the 832-dimensional vectors (768 text features + 64 graph features) to two dimensions. Each point represents a node embedding, with colors indicating the three different class types. The embeddings are extracted from the final hidden layer during the forward pass.
Three-fold cross-validation micro and macro F1 scores.
Precision@ k and Recall@ k for GAT, GCN, and GraphSage with ClinicalBERT ( k\in\{1,5,10\} ).