Fidelity Assessment Tool for a Dementia Carers’ Group-Psychotherapy Intervention
Authors: Chiu Mary , Nelles Laura Jane , Wesson Virginia , Lawson Andrea , Sadavoy Joel 
Published: November 2024

Context: Systematic evaluation of a practitioner&rsquo;s adherence and competence in delivering psychotherapeutic interventions can be complex. This study describes the development of a fidelity assessment tool for the Reitman Centre CARERS Program (RCCP), a carer group psychotherapy intervention with multiple didactic and clinical components. The tool&rsquo;s value in informing psychotherapy training and best practice for practitioners from diverse professional settings is examined. Methods: The RCCP Fidelity Assessment Tool (RCCP-FAT) was developed following an iterative process of item writing and checking. Seven components of the RCCP &ndash; Group Structure, Dementia Education, Problem Solving Techniques, Therapeutic Simulation, Vertical/Horizontal Cohesion, Overall Global Rating &ndash; were assessed, with 3 to 8 items and a &ldquo;global score&rdquo; assigned to each. Fifteen trained raters were paired-up to rate 36 RCCP sessions using RCCP-FAT. Rater agreement, correlation between itemized and global scores, and correlation between global ratings and RCCP participants&rsquo; satisfaction were calculated. Results: 1188 RCCP-FAT items were rated by each of the 2 Rater Groups. Rater agreement was calculated to be 54.3% (&kappa; = 0.32; 95% CI, 0.02681- 0.3729). A positive correlation was found between itemized and global scoring for four RCCP components evaluated (R = 0.833 to 0.929; p &lt;0.01). Global score and participants&rsquo; satisfaction with &ldquo;Simulation&rdquo; was also positively correlated (R = 0.626, p &lt;0.01). Conclusions: The study provided evidence for fair rater agreement for all RCCP-FAT assessment items. More importantly, the process of developing the tool systematically crystallized the clinical elements of RCCP, and helped to standardize training methods by creating a framework for providing feedback to learners that match the items on the RCCP-FAT. The use of RCCP-FAT to guide training and mentoring of incoming group leaders is essential in the scaling and dissemination of a complex training method like the RCCP to ensure fidelity to the original evidence-based intervention.Context: Systematic evaluation of a practitioner&rsquo;s adherence and competence in delivering psychotherapeutic interventions can be complex. This study describes the development of a fidelity assessment tool for the Reitman Centre CARERS Program (RCCP), a carer group psychotherapy intervention with multiple didactic and clinical components. The tool&rsquo;s value in informing psychotherapy training and best practice for practitioners from diverse professional settings is examined. Methods: The RCCP Fidelity Assessment Tool (RCCP-FAT) was developed following an iterative process of item writing and checking. Seven components of the RCCP &ndash; Group Structure, Dementia Education, Problem Solving Techniques, Therapeutic Simulation, Vertical/Horizontal Cohesion, Overall Global Rating &ndash; were assessed, with 3 to 8 items and a &ldquo;global score&rdquo; assigned to each. Fifteen trained raters were paired-up to rate 36 RCCP sessions using RCCP-FAT. Rater agreement, correlation between itemized and global scores, and correlation between global ratings and RCCP participants&rsquo; satisfaction were calculated. Results: 1188 RCCP-FAT items were rated by each of the 2 Rater Groups. Rater agreement was calculated to be 54.3% (&kappa; = 0.32; 95% CI, 0.02681- 0.3729). A positive correlation was found between itemized and global scoring for four RCCP components evaluated (R = 0.833 to 0.929; p &lt;0.01). Global score and participants&rsquo; satisfaction with &ldquo;Simulation&rdquo; was also positively correlated (R = 0.626, p &lt;0.01). Conclusions: The study provided evidence for fair rater agreement for all RCCP-FAT assessment items. More importantly, the process of developing the tool systematically crystallized the clinical elements of RCCP, and helped to standardize training methods by creating a framework for providing feedback to learners that match the items on the RCCP-FAT. The use of RCCP-FAT to guide training and mentoring of incoming group leaders is essential in the scaling and dissemination of a complex training method like the RCCP to ensure fidelity to the original evidence-based intervention.