Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol.

The measurement of therapist adherence to evidence-based practices has become a central feature of efforts to improve implementation of these practices in substance abuse and mental health services (Schoenwald, Garland, Chapman et al., 2011). This, in part, is attributable to empirical findings linking therapist adherence and desirable clinical outcomes (e.g., Hogue, et al., 2008; Sexton & Turner, 2010). Broadly defined, therapist adherence reflects the degree to which a therapist implements specific treatment components as intended (Hogue, Liddle, & Rowe, 1996). Unfortunately, there is little consensus on the optimal approaches for measuring therapist adherence in research and practice settings. Observational coding with trained raters, the most common rating method in research trials, provides highly objective information (Hogue et al., 1996; Mowbray, Holter, Teague, & Bybee, 2003). However, this approach can require considerable time and expense. Therapist self-ratings also have been employed (e.g., Weersing, Weisz, & Donenberg, 2002) and are less costly than observational coding methods. However, findings indicate that therapists tend to over-report their use of treatment techniques (e.g., Carroll, Nich, & Rounsaville, 1998; Martino, Ball, Nich, Frankforter, & Carroll, 2009). Clients have also been used as raters of therapist adherence (see Schoenwald et al., 2011), serving to increase the feasibility, and reduce the cost, of measurement. However, these are “untrained raters” and researchers have yet to establish whether clients can accurately rate the occurrence of specific therapeutic techniques. A few studies of substance abuse treatments have examined the relative performance of different rating methods, and have found low correspondence between therapist and observer ratings (Carroll et al., 1998; Martino et al., 2009). Similarly, Hurlburt, Garland, Nguyen, and Brookman-Frazee (2010) found that therapists and observational coders differed significantly in their perceptions of what occurred during community-based mental health treatment. Neither study, however, evaluated clients as raters of therapist adherence. An on-going clinical trial provided a unique opportunity to compare the performance of four adherence rater types: clients (youth and caregivers), therapists, trained raters, and treatment experts. The trial is evaluating Contingency Management combined with a sexual risk reduction intervention (CM/SRR) for reducing substance use and risky sexual behavior among youth in juvenile drug court (JDC). Given budgetary constraints of the trial, youth and caregivers were initially selected as the raters of therapist adherence to the intervention protocol. Therapist ratings were later obtained to provide a check on these reports. Preliminary analyses found that therapists reported much lower levels of CM/SRR implementation relative to youth and caregivers (Chapman, McCart, Sheidow, & Letourneau, 2010). The findings, however, were inconclusive without objective ratings for the same cases. Thus, funding was obtained to supplement these ratings with observational ratings provided by trained raters and CM/SRR treatment experts. The goal of the current study was to evaluate the accuracy of youth and caregivers, therapists, and trained raters on ratings of therapist adherence, with “benchmark” ratings provided by treatment experts.