Measuring treatment differentiation for implementation research: the Therapy Process Observational Coding System for Child Psychotherapy Revised Strategies scale.

Over the past decade, researchers and service agencies have built active collaborations to implement evidence-based treatments (EBTs) in practice settings (Center for the Study and Prevention of Violence, 2010). Implementation science has gained momentum (Aarons, Hurlburt, & Horwitz, 2010; Weisz, Ng, & Bearman, 2014), spurred by data suggesting that it takes approximately 17 years for EBTs to be disseminated into practice settings (Institute of Medicine, 2001). With regard to youth mental health care, implementation research strives to increase the speed and quality of information transmission between the science and practice of EBTs. Implementation science addresses the factors that influence the translation of science to practice (Aarons et al., 2010), with a central goal being to ascertain how EBTs can produce beneficial, sustainable effects in practice settings. Central to this goal is determining how EBTs can be delivered with integrity across practice settings (Allen, Linnan, & Emmons, 2012; McLeod, Southam-Gerow, Tully, Rodriguez, & Smith, 2013; Schoenwald et al., 2011). Defined as the extent to which the elements of an EBT are delivered according to the treatment model (Schoenwald et al., 2011), implementation integrity is a critical measurement domain. Few measures exist for characterizing implementation integrity (Schoenwald et al., 2011). It has been proposed that treatment integrity methods (also called treatment fidelity; e.g., Allen et al., 2012; Bellg et al., 2004) developed for treatment and evaluation research can address this measurement need (McLeod et al., 2013; Schoenwald et al., 2011). Researchers have yet to agree on a single definition of treatment integrity (e.g., Bellg et al., 2004; Hagermoser Sanetti, & Kratochwill, 2009; Perepletchikova & Kazdin, 2005; Waltz, Addis, Koerner, & Jacobson, 1993), but efforts to define implementation integrity have contained the following three treatment integrity components (see Allen et al., 2012; McLeod et al., 2013; Schoenwald et al., 2011): (a) Treatment adherence – the extent to which interventions considered integral to the treatment model(s) are delivered, (b) Treatment differentiation – the extent to which interventions not part of the intended model are delivered, and (c) Competence - the quality and responsiveness of treatment delivery. In addition, some have proposed that a fourth treatment integrity component (e.g., Lichstein, Riedel, & Grieve, 1994; called patient receipt or relational elements), is relevant when assessing implementation integrity, as perfect adherence to a treatment manual may not produce optimal outcomes unless there is a strong alliance and client involvement (Allen et al., 2012; McLeod et al., 2013). Together, it has been proposed that these components can be used to characterize implementation integrity (McLeod et al., 2013). The lion’s share of research on treatment integrity has focused on treatment adherence (Perepletchikova, Treat, & Kazdin, 2007). Though assessing adherence to the treatment model is important, it is not fully sufficient for implementation research (McLeod et al., 2013). When EBTs are transported to practice settings, it is possible that interventions not found in the EBTs may be delivered, which may influence outcomes (Southam-Gerow et al., 2010; Weisz et al., 2009). Moreover, usual clinical care (UC), a common comparison group in implementation research, encompasses a wide range of therapeutic interventions (Garland et al., 2010; Hurlburt, Garland, Nguyen, & Brookman-Frazee, 2010; McLeod & Weisz, 2010), and may even include some EBTs. For these reasons, it is critical to perform treatment differentiation checks that assess for an array of interventions (McLeod et al., 2013). At present there are three observational measures that can be used to assess treatment differentiation in implementation research with children and adolescents. The first, the Child Therapy Process Rating System (CTPRS; Hurlburt et al., 2010), characterizes mental health care for youth with externalizing problems. Hurlburt et al. (2010) reported that the interrater reliability (ICC) of the CTPRS items considered together was .69. However, when interrater reliability was calculated at the item level only 76% of the items achieved an ICC above .50. The other measures represent variants of the Therapy Process Observational Coding System for Child Psychotherapy – Strategies scale (TPOCS-S; McLeod, 2001). Originally used to characterize UC (McLeod & Weisz, 2010), the TPOCS-S, along with a modification (Garland et al., 2010), has been used to gauge the contents of UC for youth with internalizing (McLeod & Weisz, 2010) and externalizing (Garland et al., 2010) problems. Thus, most of the research conducted to date with the TPOCS-S has focused on characterizing UC for clinically-referred youths. As the TPOCS-S encompasses therapeutic interventions from a range of theoretical orientations it may be well-suited to assessment of treatment differentiation in implementation research (McLeod et al., 2013). Preliminary evidence supports this application of the TPOCS-S. The TPOCS-S was used to characterize UC and assess treatment differentiation in two effectiveness trials: (a) Southam-Gerow et al. (2010) used the TPOCS-S to observe 77 sessions from a sample of clinically-referred youths diagnosed with anxiety disorders randomly assigned to cognitive behavioral therapy (CBT) or UC1, and (b) Weisz et al. (2009) used the TPOCS-S to observe 94 sessions from a sample of clinically-referred youths diagnosed with depressive disorders randomly assigned to CBT or UC. In both studies, the CBT groups scored significantly higher than UC on CBT interventions and significantly lower than UC on psychodynamic interventions. These studies indicate that the TPOCS-S may have applications beyond being a tool for simply characterizing UC by demonstrating potential for using the TPOCS-S to assess treatment differentiation in implementation research. Although the TPOCS-S has shown initial promise as a treatment differentiation measure, only one study has reported on the reliability and validity of the TPOCS-S scores. McLeod and Weisz (2010) evaluated the preliminary reliability and validity of the TPOCS-S scores in a sample of 43 clinically referred youths with internalizing problems observed in 166 UC sessions. The TPOCS-S demonstrated good to excellent interrater reliability (item ICCs ranged from .72– .94) and analyses supported the validity of TPOCS-S subscale scores. However, the study was designed to assess the potential of the TPOCS-S to characterize UC so the psychometric analyses were limited to intercorrelations among the TPOCS-S subscale scores. To examine the potential of the TPOCS-S for assessing treatment differentiation in implementation research, it will be important to (a) assess a broader array of interventions found in EBTs and UC for youth emotional and behavioral problems, and (b) conduct further assessment of psychometric properties in research and practice settings (the contexts most relevant to assessing implementation integrity). The present study addressed these goals. The TPOCS-S was originally developed to assess a manageable subset of therapeutic interventions, grouped within categories prominent in the youth therapy literature—cognitive, behavioral, psychodynamic, client-centered, and family. Subscale items were specific therapeutic interventions documented in the literature for each of the categories (McLeod & Weisz, 2010). However, subsequent research has suggested that some interventions commonly found in EBTs and/or UC were not included (Bearman, Weisz, & McLeod, 2010; Garland et al., 2010; Hurlburt et al., 2010). For example, adding an item focused on parenting skills (e.g., giving effective commands) to the Family subscale would increase relevance of the system to interventions commonly used with youth externalizing problems (Garland et al., 2010). So, the present study tested an enriched version of the TPOCS-S, called the TPOCS-Revised Strategies scale (TPOCS-RS; McLeod, 2010), which includes 11 additional interventions. We examined how the TPOCS-RS performs across domains relevant to implementation research. For the TPOCS-RS to be useful for implementation research, coders must be capable of achieving item level reliability across (a) manual-based and non-manualized treatments, and (b) an EBT delivered in different settings. Additionally, as client and therapist characteristics may influence implementation integrity (Barber, Foltz, Crits-Christoph, & Chittams, 2004; Boswell et al., 2013), we evaluated whether targets of measurement that might influence implementation integrity (child, therapist, study group, coder, time in treatment) accounted for systematic variation in scores on the TPOCS-RS subscales. Finally, we tested whether scores on the TPOCS-RS subscales could detect expected differences between an EBT and UC. We describe the revision of the TPOCS-S and examine the psychometric properties of the TPOCS-RS scores. To achieve study goals, the TPOCS-RS was used to code sessions for youth diagnosed with anxiety disorders from a manual-based CBT delivered in research and practice settings and non-manualized UC. Although previous research has examined the psychometric properties of the TPOCS-S scores (e.g., McLeod & Weisz, 2010), this is the first study to investigate the psychometric properties of the TPOCS-S scores in the context of implementation research with a large, diverse sample.