Editorial: An Author's Checklist for Measure Development and Validation Manuscripts

A recent special issue of the Journal of Pediatric Psychology included papers focused on evidence-based assessment across several broad domains of assessment in pediatric psychology (e.g., adherence, pediatric pain, and quality of life). In one of these papers, Holmbeck et al. (2008) reviewed strengths and limitations of existing measures of psychosocial adjustment and psychopathology, concluding that many measures lacked supporting psychometric data (e.g., basic indices of reliability and validity) that would permit a complete evaluation of these measures. Given that measure development and validation papers are frequently published in JPP (Brown, 2007), it is important that we attend to guiding psychometric principles when developing and disseminating data on new measures to be employed with pediatric populations (Nunnally & Bernstein, 1994). Thus, the purpose of this paper is to present and describe a checklist for authors to use when submitting measure development papers to JPP. This checklist is included in the Appendix and is also included at the following link on the JPP website: http://www.oxfordjournals.org/our_journals/jpepsy/for_authors/measure%20development%20checklist.pdf Findings presented by Holmbeck et al. (2008) indicated that 34 of the 37 measures reviewed met pre-established “evidence-based assessment” (EBA) criteria for “well-established” measures (Cohen et al., 2008). To be considered “well-established,” a measure had to have been presented in at least two peer-reviewed journal articles by different investigatory teams, have demonstrated adequate levels of reliability and validity, and be accompanied by supporting information (e.g., a measure manual). Although most measures that we reviewed met these criteria, we also found that most of the 34 “well-established” measures were hampered by at least one major psychometric flaw and/or lacked important psychometric data. We concluded that a more fine-grained EBA classification system is needed. One important distinction in this literature relates to differences between empirically supported assessment and evidence-based assessment. This type of distinction was first discussed in the literature on clinical interventions (e.g., Spring, 2007). An empirically supported intervention is one that has demonstrated efficacy in randomized clinical trials or clinic-based effectiveness trials. An evidence-based intervention has empirical support in the manner just described, but also “integrates research evidence, clinical expertise, and patient preferences and characteristics … empirically-supported treatments (ESTs) are an important component of evidence-based practice (EBP), but EBP cannot be reduced to ESTs” (Spring, 2007, p.611). Applying these terms to the field of assessment and measure development efforts, an empirically supported assessment measure would be one that demonstrates satisfactory psychometric characteristics, broadly defined. To be evidence based, the measure should also demonstrate utility in clinical settings, be useful in making diagnoses, be sensitive to treatment effects, and/or provide incremental validity above and beyond other similar measures. Although papers in the special issue of JPP frequently referred to “evidence-based assessment” (Cohen et al., 2009), the articles included in the issue tended to evaluate the degree to which the measures were empirically-supported rather than evidence based. To be “evidence-based,” our reviews would have needed to integrate an evaluation of clinical utility, diagnostic utility, and treatment sensitivity with the empirical psychometric data presented in each review. As noted, the published reviews were more likely to focus on the latter rather than on the former. As suggested by Mash and Hunsley (2005), detailed EBA profiles would provide a complete evaluation of evidence across each of several psychometric and clinically relevant dimensions, including: (a) internal consistency, (b) test–retest reliability, (c) the availability of normative data, (d) content validity, (e) construct validity, (f) convergent and discriminant validity, (g) criterion-related validity, (h) incremental validity, (i) clinical utility, (j) diagnostic utility, and (k) treatment sensitivity. The focus on incremental validity and clinical and diagnostic utility raises the bar from a focus on “empirical support” (i.e., where the focus would tend to be primarily on psychometric data) to a broad focus on the “evidence base” for a measure. In developing the checklist that is the focus of this article, we attempted to provide a list of criteria relevant to establishing the evidence base (and not just empirical support) for a measure. In addition to shifting the focus from providing “empirical support” for a measure to providing an “evidence base” for our instruments, a checklist for measure development papers would permit JPP reviewers to evaluate such papers in the same way that reviewers of randomized clinical trials make use of the Consolidated Standards of Reporting Trials (CONSORT) checklist and flowchart (Altman et al., 2001). The CONSORT checklist contains reporting standards with respect to methodological features of and the manner in which results are reported in clinical trials. Moreover, authors are required to provide a flowchart that describes details of sample recruitment and attrition during the course of the study. Thus, a checklist for measure development papers would serve two interrelated purposes: (a) it would provide guidance to authors as they embark on the measure development process and would provide a list of criteria authors can use as they develop an evidence base for their measures, and (b) it would begin to standardize the manner in which psychometric and other assessment-related data are presented in measure development papers for this journal. Before providing a more detailed overview of the checklist, it is important to note that this checklist is rather exhaustive (see Appendix). As such, it represents what would “ideally” be expected for a measure development or validation manuscript rather than minimal criteria for such papers. No one paper can provide a complete evaluation of all important psychometric and clinically relevant dimensions that will establish once-and-for-all the evidence base for a measure. Instrument refinement is part of a measure development process that gradually builds an evidence base for a scale (see Smith & McCarthy, 1995, for suggestions on measure refinement). Indeed, the validation of any measure is a cumulative process that occurs across many different types of research studies and across research programs.