Executive summary: Developing early literacy: Report of the National Early Literacy Panel.

and Article Review Project staff screened the 7,313 records, which included abstracts, using these selection criteria and categorized them according to their relevance to the four research questions. Ulrich’s International Periodicals Directory was consulted to identify whether the journals were refereed. If staff were unable to determine whether a journal was refereed through this source, each journal editor was contacted and queried about that journal’s article-selection process. The results of this initial screening of citations produced a set of 1,824 studies from which panel members, in pairs, reviewed article abstracts to ascertain the study’s relevance to question 1. To be relevant to this research question, a study had to focus on a child skill or ability measured any time from birth through the beginning of kindergarten, and this skill or ability had to be statistically linked to a reading, writing, or spelling outcome measured from the conclusion of kindergarten to any time later, usually through a correlational procedure. If either of the two panelists determined that an article, as described in the title and abstract, was potentially relevant to the research question, it was added to the list of articles to be retrieved. On the basis of the abstract review, 685 full-text articles were retrieved for a second round of panel screening to determine whether the articles met the initial selection criteria and were relevant to question 1. Again, pairs of panelists reviewed articles independently. When the panelists’ ratings were in disagreement concerning an article’s status, they were contacted and asked to review their decisions and come to agreement about its inclusion. Occasionally, the two panelists would request a third panelist’s opinion about the disposition of a particular article. At any stage of the process, if an article was excluded, the reason for the rejection was documented. This process resulted in a final set of 275 studies identified for research question 1. Of these 275 articles, 41 were rejected during the coding process for a variety of reasons (usually the data were reported inadequately and did not allow for coding of the needed information from the study), resulting in a final set of 234 articles for analysis. Coding Scheme and Coding Studies NELP identified seven categories for classifying study characteristics: report identification (ID number, citation, study coder, whether study rejected and reason); setting (program type, ages or grades, country of sample, population density); demographics (information about developiNg eaRly liteRacy: RepoRt of the NatioNal eaRly liteRacy paNel 5 subjects, languages, maternal education, family structure); research design (experimental, quasiexperimental, correlational); experimental design features (subject assignment, components of treatment, treatment fidelity); nonexperimental design features (correlational information, sample selection, measurement issues); and measurement information—all research designs (test names and categories, reliability, measures, means and standard deviations, ES). A coding instrument was developed using this coding scheme. A first reliability check on the coding instrument resulted in too many problems across coders to complete. To ensure consistency across coders, a coding manual was created with definitions of terms and operational definitions for coding procedures. Several panelists also made suggestions for revising the coding instrument. Using the revised coding instrument, a second coding trial was conducted with project coders. Results of an intercoder reliability check resulted in an agreement rate of 0.80. Instances of disagreement across coders were discussed and resolved. The coding manual was revised to reflect these coding problems. A final coding check involved a random selection of 10 percent of the already-coded articles that were then sent to panel members for an accuracy check. About half of these articles were returned with comments, suggestions, or feedback. As the coding of articles continued, when coders were confronted with especially difficult-to-code articles, panelists were consulted to provide coding assistance. The panel decided to aggregate various measures used in the original studies into more general conceptual categories for the purposes of coding. For example, many articles were coded that measured aspects of reading that occurred in kindergarten or later. These measures were grouped into the larger constructs of (1) reading, including measures of individual word identification, decoding of nonsense words, and any other measure that tapped the accuracy of reading words; (2) reading comprehension, measures of students’ understanding of a written passage; and (3) reading fluency, which was represented by assessments that measure a student’s ability to accurately and quickly read a series of words or sentences. Many constructs were defined in this manner. (See Appendix 1.B for the complete coding instrument.) Once studies were coded and ESs were compared, the panel classified outcome measures based on descriptions of variables, content of published measures, or a combination of those. A summary of the outcome domains into which variables were classified, including their definitions, is included in Appendix 1.C. Missing Data As articles were coded, it became apparent that there were numerous instances of missing data, especially data that were critical to conducting a meta-analysis. In these instances, project staff attempted to contact article authors to retrieve the missing data. These attempts usually were not successful, as authors did not respond or no longer had the needed data. As a result, articles with missing data were excluded if the data that were missing were essential to carrying out the analyses. Data Entry and Cleaning After all the studies were identified and coded onto code sheets, they were entered into a database designed by the National Center for Family Literacy (NCFL). Two coders were in charge of developiNg eaRly liteRacy: RepoRt of the NatioNal eaRly liteRacy paNel 6 entering the data, and these coders double-entered approximately 10 percent of the information from the code sheets to allow an examination of the accuracy of the coding; the agreement was found to be above 0.95. Additionally, after the data were entered, two panelists reviewed approximately 20 percent of the original articles to ensure the accuracy of the data. Finally, any ES that was found to be an outlier (correlations below –0.30 or Cohen’s d values greater than 3.0 or less than –3.0) was investigated for accuracy. Unit of Analysis Because many of the statistical procedures employed in a meta-analysis are sensitive to violations of the assumption of independence of observation, NELP decided to define the unit of analysis as the ESs obtained from independent groups. Frequently, a single article represented one group. However, in some cases, a particular group of participants was used in more than one study (longitudinal studies with multiple assessments across time were the bulk of these studies). When this occurred, results were grouped from across these articles and treated as a single group. Additionally, some articles reported data from multiple independent groups. If the groups were defined as distinct from one another, they were treated as independent groups. Analysis Studies for research question 1 consisted primarily of correlational data. These studies often reported correlations of measures that the panelists considered to be measuring the same relationship (i.e., multiple measures of PA correlated with multiple measures of reading). When this occurred, the panel averaged the Fisher z-transformed correlations so that each study contributed only one ES per relationship. The formula used for converting the correlations into z-scores is ESs obtained from each study were aggregated across studies by weighting the individual investigations by the inverse of the conditional variance of each z-transformed correlation: developiNg eaRly liteRacy: RepoRt of the NatioNal eaRly liteRacy paNel 7 And the weight (w i ) as Weighted average ESs were computed based on a fixed-effect model estimation procedure that weighted each ES by the inverse of the estimated variance (w i ). These weighted mean Fisher z-correlations were then computed for each predictor-outcome relationship. The standard error (SE) of this estimate was computed by taking the square root of the inverse of the sum of the individual variance estimates that comprised that effect and was used to estimate 95 percent confidence intervals (CIs) for each Fisher z-transformed correlation. Finally, the estimate and the lower and upper bounds of the CI were converted back into Pearson correlations. Additionally, an estimate of the homogeneity of the combined ESs (Q) was estimated using the following formula: Finally, the panel determined that, to report an ES, a minimum of three studies must have contributed to the estimation of that effect. In addition to the meta-analyses of simple correlations, the panel examined multivariate studies that were retrieved as part of the search for the first research question. Multivariate studies typically use multiple regression or similar analytic techniques to examine a variable’s predictive utility in the context of other variables. These multivariate studies were coded for each of the primary predictor variables identified by the meta-analyses of simple correlations. Coding included a list of all additional variables included as predictors in the multivariate analyses and whether the primary predictor variable continued to account for unique variance in the decoding, reading comprehension, or spelling outcome measure. Methodology for Interventions Research question 2 asked about the effectiveness of instructional procedures, interventions, or other intentional actions that adults could take to improv