Face Validity of Meta-Analyses in Emotional or Behavioral Disorders:

Reliance on the power of mcta-analysis for unimpeachable evidence related to complex research questions in emotional or behavioral disorders (E/BD) can be potentially misleading if consumers automatically assume that metaanalytic results are always definitive. Specifically, published meta-analytic results rely heavily on several pivotal interpretive aspects, including (a) the definition and relationships between the primary study independent variables, (b) the manner in which the independent variables are coded, and (c) how these key variables are interrelated and reported (Cooper & Lindsay, 1998). Issues of face validity are especially germane when there are research examples of meta-analyses conducted on the same body of primary studies yielding dissimilar findings (e.g., debates over Dunn's research on learning styles [see Kavale & Forness, 1987, 1990; Kavale, Hirshoren, & Forness, 1998, versus Dunn, 1990; Dunn, Griggs, Olson, Beasley, & Gorman, 1995] and around social skills instruction in learning disabilities [see Swanson, 1996; Swanson & Malone, 1992, versus Ochoa & Olivarez, 1995]). These aspects, therefore, drive an imperative for careful and systematic reporting of published meta-analyses to increase the accuracy and reliability on their summative answers. This review focuses on the face validity of published meta-analyses in E/BD.PurposeThe purpose of this review is fourfold: (a) to review the content of meta-analyses in E/BD; (b) to identify the characteristics of reported E/BD meta-analyses via a set of face validity criteria; (c) using the same criteria set, to explain how this information may affect the overall quality of meta-analyses in E/BD; and (d) to compare results of the analysis to findings in other areas of the special education meta-analytical literature.Analytical Narrative Review of Meta-Analyses in E/BDMostert and Kavale (2001) noted that where quantitative review of a data set is not possible, it can be examined through a rigorous narrative analytical review. Jackson (1980) offered a methodological framework for researchers reporting such integrative reviews, stipulating that the review should embody (a) a selected research question or hypothesis, (b) sampling information, (c) representative characteristics of the review, (d) analysis of the primary studies, (e) interpretation of results, and (f) reporting of the integrative review. I use these perspectives to review 36 published E/BD meta-analyses.Selection of Review QuestionsIntegrative reviews might present either a straightforward examination of a research phenomenon in a research area or explanations of methodological variation among individual studies (Jackson, 1980). I combined these aspects to generate the following questions:1. What content is reported in published E/BD meta-analyses?2. Is there variation in the reported detail that can affect the face validity of published E/BD meta-analyses?3. What is the overall quality of the metaanalyses in E/BD?4. How do these results compare with previous work in this area with published metaanalyses in learning disabilities (ED; Mostert, 1996), mental retardation (MR; Mostert, 2003), and a comparative but smaller set of ED, MR, and BD meta-analyses (Mostert, 2001)?Sampling Information: The Primary E/BD Meta-AnalysesIn this review, I examined 36 published metaanalyses (marked in the reference section by an asterisk) containing, either entirely or in part, subject populations with E/BD. These meta-analyses comprise the primary studies for this reviewRepresentative Characteristics of the Primary Studies in the Integrative ReviewTo describe how the primary characteristics are analyzed in this review, I developed a set of criteria across six domains clarifying megaanalytical study characteristics to judge the face validity of any published meta-analysis (Mostert, 1996). The domains (i. …