ethanporcaro/eric-fulltext-meta huggingface dataset:
Estimating Statistical Power When Making Adjustments for Multiple Tests
Author: Porter, Kristin E.
Date: 2016-00-00
Type: Reports - Research
Subjects: Hypothesis Testing,Intervention,Error Patterns,Evaluation Methods,Testing,Statistical Analysis,Computation,Probability,Simulation

In recent years, there has been increasing focus on the issue of multiple hypotheses testing in education evaluation studies. In these studies, researchers are typically interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time or across multiple treatment groups. When multiple hypotheses are tested, the probability of committing at least some Type I errors increases and more dramatically with a greater number of tests. Multiple testing procedures (MTPs) adjust p-values for statistical estimates upwards to counteract this problem. MTPs are being increasingly applied in impact evaluations in education. For example, the IES technical methods report, &quot;Guidelines for Multiple Testing in Impact Evaluations,&quot; (Schochet, 2008) recommends multiple testing procedures as one of several strategies for dealing with the multiple hypotheses issue. In addition, the What Works Clearinghouse applies a particular procedure, the Benjamini-Hochberg procedure (Benjamini &amp; Hochberg, 1995) to statistically significant findings in studies under review that have estimated effects for multiple measures and/or groups (U.S. Department of Education, 2013). However, an important consequence of making adjustments for multiplicity is a change in the statistical power for detecting true effects. This paper provides critical alternatives to current practice in two ways. First, it presents alternatives to how power is typically defined in studies with multiple tests. Second, for multiple definitions of statistical power under multiplicity, this paper presents methods for estimating power while accounting for p-value adjustments using one of five common multiple testing procedures (MTPs)--Bonferroni (Dunn, 1959, 1961), Holm (Holm, 1979), single-step and step-down versions of Westfall-Young (Westfall and Young, 1995), and Benjamini-Hochberg (Benjamini and Hochberg, 1995) procedures.
Examining the Role of Early Academic and Non-Cognitive Skills as Mediators of the Effects of City Connects on Middle School Academic Outcomes
Author: O'Dwyer, Laura M.
Date: 2016-00-00
Type: Reports - Research
Subjects: Public Schools,Academic Achievement,Standardized Tests,Scores,Program Effectiveness,Intervention,Comparative Analysis,Control Groups,Experimental Groups,Longitudinal Studies,Readiness,Report Cards,Grade 5,Grade 8,Regression (Statistics),Hypothesis Testing,Elementary School Students,Middle School Students,Poverty,Study Habits

Out-of-school factors can significantly impact students' readiness to learn and thrive in school. Research confirms that larger social structures and contexts beyond the school are critical, accounting for up to two-thirds of the variance in student achievement (Coleman et al., 1966; Rothstein, 2010; Phillips, Brooks-Gunn, Duncan, Klebanov, &amp; Crane, 1998; Leventhal &amp; Brooks-Gunn, 2000). These out-of-school factors can be particularly pernicious in the context of poverty. For children living in poverty, limited resources and chronic stressors can result in poor attendance, high mobility, social-emotional dysfunction, and lack of readiness for school (Dearing, 2008). Even in the face of significant family strengths (Strauss, 2013), poverty can limit families' abilities to invest money, time and energy in children's growth and expose children to chaos and environmental contagions (Brooks-Gunn &amp; Duncan, 1997; Evans, 2004). Many have argued that schools cannot hope to close the achievement gap without addressing out-of-school factors (Berliner 2013; Bryk, et al., 2010). Historically, schools have addressed some of these factors through the work of school nurses, counselors, social workers, and psychologists, but student support has varied widely, without a standardized set of practices (Lean &amp; Colucci, 2010). The research described here was conducted as part of the evaluation of City Connects, a theoretically-guided, evidence-based approach to addressing out-of-school barriers to achievement and thriving in high-poverty urban elementary schools. This study extends prior research by examining the mechanisms that lead to the positive impact of City Connects on later academic achievement. The research question is: Are the positive effects of City Connects on student academic achievement (standardized test scores) mediated by students' early academic and non-cognitive skills (effort, behavior, work habits)? Although the mediation hypothesis was not confirmed by the data at hand, the study provides evidence that with high-quality elementary school student support, early academic and noncognitive skills lead to later achievement. Tables and figures are appended.
How Methodological Features Affect Effect Sizes in Education
Author: Cheung, Alan
Date: 2016-00-00
Type: Reports - Research
Subjects: Effect Size,Evidence Based Practice,Educational Change,Educational Policy,Research Design,Educational Experiments,Program Evaluation,Publications,Sample Size,Literature Reviews,Best Practices,Educational Research,Elementary Secondary Education,Mathematics Education,Science Education,Reading Instruction,Reading Programs,Early Childhood Education,Technology Uses in Education,Meta Analysis,Computer Software,Statistical Analysis,Q Methodology,Quasiexperimental Design,Randomized Controlled Trials

As evidence-based reform becomes increasingly important in educational policy, it is becoming essential to understand how research design might contribute to reported effect sizes in experiments evaluating educational programs. The purpose of this study was to examine how methodological features such as types of publication, sample sizes, and research designs affect effect sizes in experiments. In order to investigate the relationships between study methodological features and effect sizes, the authors analyzed 645 studies that met the standards of inclusion for any of 12 reviews written for the Best Evidence Encyclopedia and (in most cases) published in review journals. The reviews cover programs in elementary and secondary math, elementary and secondary science, and elementary and secondary reading, as well as a review of elementary reading programs for struggling readers and a review of early childhood education. Studies included in reviews focusing on technology applications in reading and math were also included. Comprehensive Meta-Analysis software Version 2 (Borenstein, Hedges, Higgins, &amp; Rothstein, 2005) was used to carry out all statistical analyses such as Q statistics and overall effect sizes. The findings suggest that effect sizes are roughly twice as large for published articles, small-scale trials, and experimenter-made measures, than for unpublished reports, large-scale studies, and independent measures, respectively. In addition, effect sizes are significantly higher in quasi-experiments than in randomized experiments. Based on the findings of the analyses, it is clear that researchers as well as policy makers need to take into account research design, sample size, measures, and type of publication before comparing effect sizes from program evaluations. Some specific recommendations are provided.
Implementation Work at Scale: An Examination of the Fidelity of Implementation Study of the Scale-Up Effectiveness Trial of Open Court Reading
Author: Sullivan, Kate
Date: 2016-00-00
Type: Reports - Research
Subjects: Program Implementation,Fidelity,Reading Programs,Student Experience,Classroom Techniques,Institutional Characteristics,Teaching Styles,Control Groups,Experimental Groups,Randomized Controlled Trials,Archives,Teacher Surveys,Classroom Observation Techniques,Interviews,Transcripts (Written Records),Program Effectiveness,Program Evaluation,Elementary School Students

The Fidelity of Implementation (FOI) study that is the focus of this report was conducted as a component of a scale-up effectiveness trial of the SRA/McGraw-Hill Open Court Reading program. The overall purpose of the FOI study was to support and provide context for findings from the larger experimental impact study of Open Court Reading (OCR). To accomplish this task, the FOI study addressed the following research questions: (1) In what ways were OCR students' experiences similar or different to those of students receiving other reading curricula; and (2) Was there significant variation in implementation fidelity of OCR among classrooms/teachers, schools, and districts? The first question addresses the issue of general fidelity (i.e., comparing treatment and control group implementation), and the second examines specific fidelity (i.e., variation of implementation within the treatment condition). General and specific fidelity are both critical areas of investigation to include in FOI study designs intended to inform scale-up effectiveness trials. The FOI study was conducted during two years of OCR implementation in the 49 elementary impact study schools (kindergarten through 5th grade) in seven districts across the country. This FOI study was a component of the cluster randomized controlled effectiveness trial of the OCR program. Overall, findings from specific fidelity analyses revealed that implementation of the OCR program among treatment teachers was relatively high. Two tables are appended.
Executive Function and Mathematics Achievement: Are Effects Construct- and Time-General or Specific?
Author: Duncan, Robert
Date: 2016-00-00
Type: Reports - Research
Subjects: Executive Function,Mathematics Achievement,Preschool Children,Child Development,Early Childhood Education,Scores,Predictor Variables,Inhibition,Self Control,Short Term Memory,Attention Control,Kindergarten,Mathematics Tests,Abstract Reasoning,Cognitive Ability,Cognitive Tests,Achievement Tests,Data Analysis,Intervention

Executive function (EF) is considered a set of interrelated cognitive processes, including inhibitory control, working memory, and attentional shifting, that are connected to the development of the prefrontal cortex and contribute to children's problem solving skills and self regulatory behavior (Best &amp; Miller, 2010; Garon, Bryson, &amp; Smith, 2008). EF skills are argued to be foundational for children to thrive in academic domains (Morrison, Cameron Ponitz, &amp; McClelland, 2010), particularly for mathematics (Blair, Ursache, Greenberg, Veron-Feagans, &amp; The Family Life Project Investigators, 2015). Consistent with this theory, EF measures are consistently found to correlate with children's mathematics achievement (Bull &amp; Lee, 2014; Friso-van den Bos, van der Ven, Kroesbergen, &amp; van Luit, 2013), and predict growth in mathematics using a variety of samples and analytic strategies (Blair et al., 2015; Fuhs, Nesbitt, Farran, &amp; Dong, 2014; McClelland et al., 2014). The current study attempts to answer two complementary research questions to better understand associations between executive function (EF) and mathematics in early childhood. First, the study examines whether children's mathematics scores are better predicted by a single EF factor or specific EF components (i.e., inhibitory control, working memory, attentional shifting). Given that EF tasks have been found to tap a single underlying EF factor in early childhood (e.g., Wiebe et al., 2008), the authors hypothesize that the EF factor will account for most of the task-specific EF associations with mathematics. Second, the study examines if associations between EF and mathematics are a function of time-general inter-individual differences or time-specific variation (i.e., cross-lagged effects). Large associations between EF and mathematics have been found (e.g., Fuhs et al., 2014); however, these estimates may be upwardly biased due to persistent interindividual differences not fully statistically controlled in cross-lagged panel models (Hamaker et al., 2015). Prior work on children's mathematics achievement suggests that the factors that affect achievement similarly across development contribute more to the longitudinal stability of individual differences in children's mathematics achievement than the direct effects of children's previous achievement on their later learning (Bailey, Watts, Littlefield, &amp; Geary, 2014). Thus, the authors hypothesize that the associations between EF and mathematics will be larger for the time general factors than for cross-lagged, time-specific factors. Tables and figures are appended.
Latent Class Models for Teacher Observation Data
Author: Halpin, Peter F.
Date: 2016-00-00
Type: Reports - Research
Subjects: Models,Multivariate Analysis,Scoring Rubrics,Teacher Evaluation,Evaluation Methods,Observation,Teaching Skills,Performance,Statistical Analysis,Measurement Techniques,Middle School Teachers,Language Arts,Teacher Effectiveness,Reliability

Recent research on multiple measures of teaching effectiveness has redefined the role of in-classroom observations in teacher evaluation systems. In particular, most states now mandate that teachers are observed on multiple occasions during the school year, and it is increasingly common that multiple raters are utilized across the different rating occasions. In-classroom observations are typically conducted using a rating rubric. However, related research has found that many rubrics measure multiple dimensions of instructional quality, suggesting that teachers' practices are not well described in terms of a total score. Halpin &amp; Kieffer (2015) argued for the use of latent class analysis (LCA) as a means of capturing the multidimensional features of rating rubrics, while also providing the standard error of measurement for each teacher, and item-level diagnostic information that can be used as the basis of feedback to educators and for professional development. The main purpose of the present research is to develop a multilevel extension of the LCA methodology described by Halpin &amp; Kieffer (2015). For a given rating rubric, the multilevel LCA approach is specifically intended to answer the following questions: (1) How reliably (precisely) is a teacher's teaching ability measured during any single observation session?; (2) How consistently does a teacher perform over observation sessions?; and (3) For a given teacher, how many observation sessions are required before his/her teaching ability has been measured with a desired level of precision? The last question in particular has relevance for policy, in that multi-rater systems can place heavy financial demands on school districts in terms of deploying a sufficient number of trained raters to meet required number of observation sessions per teacher. The proposed methodology allows for decisions about the required number of observations to be made on a teacher-by-teacher basis, and to be informed by the data collected from each teacher. Two figures are appended.
Focusing on the Whole Student: An Evaluation of Massachusetts' Wraparound Zones Initiative
Author: Gandhi, Allison Gruner
Date: 2016-00-00
Type: Reports - Research
Subjects: Holistic Approach,Academic Achievement,Attendance Patterns,Academic Persistence,Suspension,Mixed Methods Research,Formative Evaluation,Summative Evaluation,Low Achievement,Intervention,Elementary Schools,Middle Schools,Comparative Analysis,Program Effectiveness,Language Arts,Mathematics,School Culture,Student Needs,School Community Relationship,School Districts,Grants,Scores

Over the past twenty years, efforts to turn around low-performing schools have increasingly become a central component of federal and state education policy agendas. The purpose of the study was to evaluate the impact of the Wraparound Zones Initiative (WAZ), a program supported by the Massachusetts Department of Elementary and Secondary Education (ESE), on student outcomes including achievement, attendance, retention, and suspension. The study was conducted as part of a multi-year mixed-methods, formative and summative evaluation of WAZ that ESE commissioned from American Institutes for Research (AIR). The setting consisted of districts and schools in Massachusetts that had been identified by the state as chronically underperforming and in need of state intervention. The sample for this study was drawn from students in Cohort 1 and Cohort 2 WAZ schools serving elementary and/or middle grades, plus students in a set of matched non-WAZ comparison schools. The WAZ Initiative is designed to create coordinated district systems that allow schools to proactively and systematically address students' nonacademic needs. Comparative interrupted time series (CITS) design was used to measure the impact of receiving a WAZ grant on student outcomes, including student achievement, attendance, retention, and suspension. Overall, students in WAZ schools performed better on the Massachusetts Comprehensive Assessment System (MCAS) English language arts (ELA) and mathematics assessments as compared with students in comparison schools, when considering prior achievement trends. Effects were statistically significant after the second and third years of WAZ implementation for ELA, and after the second year for mathematics. This research study is significant in that it demonstrates that a program focused on student support and social-emotional learning can have an impact on student achievement, and can be an integral component of overall school turnaround strategy.
Examining Multidimensional Middle Grade Outcomes after Early Elementary School Grade Retention
Author: Hwang, Sophia
Date: 2016-00-00
Type: Reports - Research
Subjects: Early Childhood Education,Academic Achievement,Outcomes of Education,Early Adolescents,Grade Repetition,Probability,Scores,Grade 1,Grade 2,Grade 8,Program Effectiveness,Intervention,Longitudinal Studies,Children,Surveys,Cohort Analysis,Quasiexperimental Design,Middle School Students,Self Concept Measures,Self Esteem

Recently, researchers have begun to employ rigorous statistical methods and developmentally-informed theories to evaluate outcomes for students retained in non-kindergarten early elementary school. However, the majority of this research focuses on academic outcomes. Gaps remain regarding retention's effects on psychosocial outcomes important to later school and life outcomes. Additionally, some psychosocial and academic outcomes and changes may be evident in the short-term, but others may have a delayed presentation. These &quot;sleeper effects&quot; are less well-understood, creating a need for new research focusing on longer-term outcomes. In response, the current study uses causal methods and a national dataset to investigate both academic achievement and psychosocial outcomes in early adolescence after early elementary grade retention. Using propensity score methodological approaches and informed by developmental systems theory, this study estimates the effect of early (first and second) grade retention on academic achievement and psychosocial outcomes in eighth grade: six or seven years later. The goal is to determine whether retention is an effective intervention for students struggling to meet academic standards early in their elementary schooling, and to inform the education policy and practice debate regarding the costs and benefits associated with grade retention. Data for this study are from the Early Childhood Longitudinal Study, Kindergarten Class 1998-1999 (ECLS-K; U.S. Department of Education, 2009), a nationally representative sample following a longitudinal cohort of 21,260 kindergartners until the eighth grade. The dataset includes multiple sources of data including administrator surveys, teacher assessments, direct child assessments, child questionnaires, and parent surveys. The sample is comprised of 5,586 students. The current study utilizes a quasi-experimental, longitudinal approach, and propensity score methodology to estimate the effect of early grade retention on academic and psychosocial outcomes in the middle grades. At the time of this report, analyses are being conducted, and formal results are forthcoming. Tables are appended.
High School Career and Technical Education Participation and Initial College Enrollment: Evidence from Arkansas
Author: Dougherty, Shaun M.
Date: 2016-00-00
Type: Reports - Research
Subjects: Vocational Education,High School Students,Enrollment,Postsecondary Education,Probability,Influences,Student Participation,Dual Enrollment,Rural Schools,Urban Schools,Suburban Schools

This paper uses rich administrative data from Arkansas to understand whether and how high school career and technical education (CTE) programs are related to initial enrollment in college after high school. This descriptive work is designed to inform how other state and local policymakers understand the potential role of high school CTE participation in building human capital pathways to contribute to an informed society and a skilled workforce that reflects labor market demands. This paper addresses the following research questions: (1) Which high school CTE programs are associated with students enrolling in post-secondary education just after high school?; (2) Are the probabilities that a student who participated in CTE in high school any different from an otherwise similar student who did not participate in CTE, or who participated to a different extent?; and (3) What student, program, or enrollment factors are associated with initial college going? These factors include: gender, family income, urbanicity, prior work experience, or dual enrollment. This study is set in Arkansas and uses data that were made available through the Arkansas Research Center, which coordinates data for K-12, higher education, career and technical education, and the department of labor. The dataset is supplemented with data from the National Center for Education Statistics. Researchers followed three cohorts of ninth graders who started high school in 2008, 2009, and 2010. They observed what classes the students took in high school, and followed them through at least one year, and up to three years, past graduation. The dataset shows whether they graduated from high school, then whether they enrolled in a two- or four-year college or went straight into the workforce (and what their wages are). The cohorts include over 100,000 individual students, and more than 350,000 student-year observations. Students who participate in more CTE in high school or who are CTE concentrators are not, on average, substantially different in their probability of attending college just after high school relative to their peers who are not CTE concentrators or who take fewer CTE courses in high school. There is clear evidence, however, that among CTE concentrators, programs like health services have a much higher rate of enrolling in college. Tables are appended.