Validating indicators of treatment response: application to trichotillomania.

It is widely agreed that statistical significance of between-group mean differences is an insufficient way to determine whether a therapy has clinically meaningful effects (e.g., Lambert, Hansen, & Bauer, 2008). One treatment could be very powerful yet fail to show statistical significance, for instance if it is compared to a potent alternative treatment or tested in a small study, or both. Another could have modest effects yet show statistical significance because it is tested against no treatment, with low within-group variability of results, and high sample size. Accordingly, methodologists have advocated supplementing standard significance testing by reporting the proportion of patients who obtain practically meaningful benefits (e.g., Jacobson, Follette, & Revenstorf, 1984). Numerous methods for quantifying practically meaningful effects have been developed. For some disorders or conditions, the most suitable indicator could be complete eradication of the problem behavior. A prominent clinical practice guideline for smoking cessation, for instance, judges treatments mainly on the basis of the proportion of patients showing 7-day point prevalence abstinence from smoking at 6-month follow-up (Fiore et al., 2008). Smoking rate reduction is considered insufficient to mark treatment success, in part because of health considerations, as even a modest rate of daily smoking (1–4 cigarettes/day) significantly predicts heart disease mortality and all-cause mortality (Bjartveit, & Tverdal, 2005). In other areas of treatment, however, abstinence may be unrealistic (e.g., schizophrenia symptoms) or super-normal and atypical of even the general, non-clinical population (e.g., worry, depressive symptoms). Therefore, various statistical metrics other than complete abstinence are often used instead to index meaningful benefit. A systematic review of such formulae recommended the clinical significance definition developed by Jacobson and colleagues (e.g., Jacobson & Truax, 1991) as provisionally the best choice (Lambert, Hansen, & Bauer, 2008), while noting that few studies have been conducted on the comparative validity of alternate methods. In the Jacobson framework, a clinically significant response requires that the patient make statistically reliable change during treatment and recover normal-range functioning by the end of treatment. The research reported in this article evaluated alternate indicators of practically meaningful treatment response in the context of a stepped care protocol for treating trichotillomania (TTM). TTM is characterized by the recurrent pulling out of one’s own hair, resulting in hair loss. TTM is associated with considerable psychosocial impairment. Relative to healthy control samples, people with TTM report greater disability (Diefenbach, Tolin, Hannan, Crocetto, & Worhunsky, 2005), lower life satisfaction (Diefenbach, Tolin, Hannan, et al., 2005), lower self-esteem (Diefenbach, Tolin, Hannan, et al., 2005), and lower quality of life (Odlaugh, Kim, & Grant, 2010). They often spend a great deal of time on hair pulling and may experience interference with their work and social lives (Wetterneck, Woods, Norberg, & Begotka, 2006). TTM can also have medical consequences, particularly if the patient ingests pulled hairs (McDonald, 2012). Studies of therapy for TTM have employed a variety of methods for quantifying a clinically meaningful response. Some investigators report the proportion of patients who achieve greater than a particular percentage reduction in symptom severity (e.g., 25%; Mouton-Odum, Keuthen, Wagener, Stanley, & Debakey, 2006). Others report the proportion of patients who recover a normal level of functioning (e.g., van Minnen, Hoogduin, Keijers, Hellenbrand, & Hendriks, 2003). To our knowledge, no published studies have used the full Jacobson/Truax criteria, including requiring statistically reliable change. This is a potentially important distinction from recovery of normal functioning because someone could recover normal functioning without actually benefitting much from treatment, if their pretreatment symptom scores were just barely higher than the cutoff demarcating normal functioning. Each of the three foregoing standards (25% reduction, recovery of normal functioning, clinical significance per Jacobson) can, in turn, be based on self-report vs. interviewer-rated symptoms, which were examined separately in our study in light of prior research showing that different measures and alternate perspectives often yield varying proportions of clinically significant responding (Lambert et al., 2008). Finally, although previous treatment outcome studies have not generally required it, one could define success in terms of complete abstinence from hair pulling, for two reasons. First, while there are no systematic epidemiological studies of truly representative community samples, it seems likely that the majority of people do not pull their hair at all for non-cosmetic reasons, making abstinence the modal, statistically “normal” state. Second, post-treatment abstinence has predicted superior two-year follow-up results after behavior therapy for TTM (Keijsers et al., 2006). The mere existence and use of differing methods of reporting clinically meaningful response to TTM treatment is not a problem. Indeed, premature closure on a given approach would run the risk of fostering a mono-method bias (Cook & Campbell, 1979), leaving us unable to detect the flaws of this consensus method. However, it is a problem that we do not know (a) how the methods compare to one another, nor (b) to the extent that they differ, whether one is more valid in relation to external criteria than are the others. For reviewers or consumers of the treatment literature, a lack of information on overlap or relative leniency of the methods complicates the task of interpreting cross-study findings. For instance, if Treatment A was successful for 40% of TTM patients in reducing interviewer-rated symptoms by at least 25%, is this a better outcome than Treatment B’s success in another project for 28% of patients, with success defined as complete abstinence from hair pulling? To be sure, ultimately such evaluations should be based on within-study comparisons, but in the meantime patients are being treated, and provisional decisions about which treatments enjoy the most empirical support need to be made. Likewise, the absence of comparative validation data makes it difficult to select a standard for determining when a course of treatment has been sufficiently successful, which is important for practicing clinicians who need to decide when to terminate treatment vs. extend it vs. perhaps augment it with an additional treatment method. Attempting to validate metrics for quantifying clinically meaningful response raises the question of what criteria should be used. There is no obvious gold standard criterion to which a valid index of treatment success should relate. Our research was conducted in the context of a stepped care protocol for treatment of TTM in which patients were given the choice to proceed from step 1 (web-based self-help) to step 2 (in-person individual therapy with Habit Reversal Training). We treated as one external criterion the question of whether those who met a particular standard during step 1 were less likely to proceed to step 2. Although this is a fallible standard – a patient who had responded adequately might nevertheless be interested in receiving in-person care, and a patient who had not responded might become discouraged and give up – it seemed likely that a useful marker of clinically meaningful response should at least on average be related to the patient’s behavior in proceeding with further treatment or not. We also collected several kinds of predictive validity evidence for each of the various standards. Clinical response from baseline to the end of our stepped care protocol (post-step 2 assessment) was related to follow-up (3 months after post-step 2 assessment) data on treatment satisfaction, functional impairment related to TTM, and quality of life. We expected that a valid indicator of whether someone has responded well to treatment should be associated with low impairment, high quality of life, and high treatment satisfaction three months later. We included diagnosis as a final criterion. One of the few previous studies of the predictive validity of clinical significance estimation methods related multiple statistical methods for quantifying clinical significance to depression relapse over the next two years (McGlinchey, Atkins, & Jacobson, 2002). In the present study, we likewise used prediction of TTM diagnostic status at 3-month follow-up as one of the validation criteria. In summary, in a study of stepped care for TTM, we evaluated seven potential standards for judging clinically meaningful response (complete abstinence from pulling as well as (a) >=25% reduction in symptom scores, (b) recovery of normal functioning, or (c) clinical significance, crossed with (i) self-reported or (ii) interviewer-rated symptoms) with regard to (1) comparative sensitivity/lenience, (2) convergence with the other standards, (3) concurrent validity in relation to choice of an additional treatment step, and (4) predictive validity in relation to 3-month follow-up treatment satisfaction, functional impairment, quality of life, and diagnosis.