bluuebunny/crossref_metadata_2025 huggingface dataset:
High School Size: Effects on Budgets and Performance in New York City
Authors: Stiefel Leanna , Berne Robert , Iatarola Patrice , Fruchter Norm
Published: June 2007

Given the large numbers of competing proposals for improving urban schools so that students perform better, policymakers need to know budget as well as performance implications of proposed reforms. One reform group with a large constituency suggests that high schools should enroll small numbers of students, but some policymakers worry that the costs of small schools are excessive. This paper contributes to the school size policy debate by using methods and data that combine budget and performance information, with the school as the unit of analysis. With data on budgets, graduates, and characteristics of students in New York City public high schools, we estimate 4-year budgets per graduate. We find that small academic and large high schools are similar in terms of budgets per graduate and that some vocational and "transfer" high schools have the highest budgets per graduate. Because the literature on school size indicates that small high schools are more effective for minority and poor students, the similarity in outcomes that small and large high schools produce suggests that policymakers might do well to support the creation of more small high schools.
Achieving Scale and Quality in School-to-Work Internships: Findings From Two Employer Surveys
Authors: Bailey Thomas , Hughes Katherine , Barr Tavis
Published: June 2007

Work-based learning is a central component of the school-to-work strategy. Yet, a widespread system of work-based learning in the form of internships or apprenticeships will need to involve thousands of employers willing to provide placements. This paper examines the issue of employer involvement in the school-to-work strategy through a survey of employers participating in five school-to-work programs and a survey of a comparison group of nonparticipating employers in those same labor markets. The motivations and potential motivations of employers are identified; the data suggest that participants are motivated by philanthropy, while firms in the nonparticipating sample indicate that they would need more bottom-line-oriented arguments to convince them to join up. We also explore the quality of work-based learning placements provided to try to identify the relationship between the characteristics and motivations of employers and the quality of the internships that they provide.
From the Party/State to Multiethnic Democracy: Education and Social Cohesion in Europe and Central Asia
Authors: Heyneman Stephen P.
Published: June 2007

Can educational mechanisms lower social tension and help achieve social cohesion ? If so, how are these mechanisms defined and measured? What is the experience to date with the social utility of education mechanisms? How can one differentiate between an education system that is doing a good job of rein- forcing social cohesion and one that is doing a poor job? In this paper, I attempt to respond to these questions by (a) briefly reviewing some concepts of institutional and organizational economics so that the economic implications of education's social cohesion functions can be more clear, (b) reviewing the origins of public schooling so that the reader may place today's educational challenges in historical context, (c) reviewing the anecdotal and field experience to date in the European and Central Asian (ECA ) region in meeting the challenges of social cohesion, and (d) drawing some comparisons between the social cohesion performance of education systems in the ECA region and that of the U.S. In sum, I argue that social cohesion has significant economic benefits; that since its invention in the 17th century, public education has been one of the main contributions to social cohesion in the west; but that countries of the ECA region are having a difficult time replicating the western education experience. In fact, when compared to other parts of the world, the U.S. school system seems to perform rather well with respect to its social cohesion functions.
Worldwide Shadow Education: Outside-School Learning, Institutional Quality of Schooling, and Cross-National Mathematics Achievement
Authors: Baker David P. , Motoko Akiba  , LeTendre Gerald K. , Wiseman Alexander W.
Published: June 2007

The growth of structured, outside-school activities for improving students’ mathematics achievement is an enduring feature of modern schooling with major policy implications. These "shadow education " activities mimic, or shadow, formal schooling processes and requirements. Using extensive cross-national data from the Third International Mathematics and Science Study, we examine shadow education as a macro-phenomenon of modern schooling through its (a) prevalence, (b) strategies for use, and (c) associated national characteristics. We find that shadow education is prevalent worldwide, but that there is consider­able cross-national variation in its use. Contrary to findings from single country studies, we find most shadow education is remedial in nature. We then test hypotheses concerning the national origins of shadow education and its impact on nations’ production of mathematics achievement. Our results show that institutional factors of education, including limited access and lower levels of funding, drive the use of shadow education, instead of high-stakes testing and national achievement incentives. We conclude by discussing implications for both educational policy and theory regarding the degree to which institutional­ization of mass schooling increasingly dominates contexts of schooling.
Class Size and Eighth-Grade Math Achievement in the United States and Abroad
Authors: Pong Suet-ling , Pallas Aaron
Published: June 2007

Using data from the Third International Math and Science Study (TIMSS), we examine the relationship between class size and eighth-grade math achievement in Australia, Canada, France, Germany, Hong Kong, Korea, Iceland, Singapore, and the United States. Class sizes tend to be greater and more homogenous in centralized education systems compared with those in decentralized systems. The United States seems to be unique among the countries in our study. After controlling for possible confounding characteristics of the teacher, school, and classroom, in no other country than the United States did we find a beneficial effect of small classes. Contrary to our expectations, we also found little evidence that smaller or larger classes differ in the amount of curriculum taught or in the instructional practices of teachers. Except for the case of Hong Kong, neither curricular coverage nor instructional practices mediates the relationship between class size and math achievement.
Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study
Authors: Desimone Laura M , Porter Andrew C. , Garet Michael S. , Yoon Kwang Suk , Birman Beatrice F.
Published: June 2007

This article examines the effects of professional development on teachers’ instruction. Using a purposefully selected sample of about 207 teachers in 30 schools, in 10 districts in five states, we examine features of teachers’ professional development and its effects on changing teaching practice in mathematics and science from 1996–1999. We found that professional development focused on specific instructional practices increases teachers’ use of those practices in the classroom. Furthermore, we found that specific features, such as active learning opportunities, increase the effect of the professional development on teacher’s instruction.
Below the Bar: Profiles of Students who Fail State Reading Assessments
Authors: Buly Marsha Riddle , Valencia Sheila W.
Published: June 2007

State and school districts are looking for policies they believe will improve student performance. As a result, assessments have proliferated, stakes have increased, and specific curriculum and instructional approaches are being mandated. In this study, we probed beneath students' failing scores on a state reading assessment to investigate the needs of struggling students and implications for policy. We found that scores on state tests mask distinctive and multifaceted patterns of students' reading abilities that require dramatically different instructional emphases. We explore the implications of this complexity for state and local reform efforts that target improved teaching and learning.
The Long-Term Effects and Cost-Effectiveness of Success for All
Authors: Borman Geoffrey D. , Hewes Gina M.
Published: June 2007

Several renowned early interventions have compelling evidence of enduring achievement effects for at-risk children: Perry Preschool; the Abecedarian Project; and the Tennessee Class-Size Experiment. The costs and potential for national dissemination of such model programs, though, represent key practical concerns. This article examines the long-term outcomes and costs of another popular early intervention: Success for All. Relative to controls, Success for All students completed 8th grade at a younger age, with better achievement outcomes, fewer special education placements, fewer retentions, and at the same educational expense. Further cost-effectiveness comparisons to the three prominent interventions suggest that Success for All is deserving of similar recognition as a sound educational investment that provides strong and lasting educational benefits. None of these exemplary programs, though, can be expected to be the “great equalizer.”
Does External Accountability Affect Student Outcomes? A Cross-State Analysis
Authors: Carnoy Martin , Loeb Susanna
Published: June 2007

We developed a zero-to-five index of the strength of accountability in 50 states based on the use of high-stakes testing to sanction and reward schools, and analyzed whether that index is related to student gains on the NAEP mathematics test in 1996–2000. The study also relates the index to changes in student retention in the 9th grade and to changes in high school completion rates over the same period. The results show that students in high-accountability states averaged significantly greater gains on the NAEP 8th-grade math test than students in states with little or no state measures to improve student performance. Furthermore, students in high-accountability states do not have significantly higher retention or lower high school completion rates.
The Impact of High-Stakes Testing in Chicago on Student Achievement in Promotional Gate Grades
Authors: Roderick Melissa , Jacob Brian A. , Bryk Anthony S.
Published: June 2007

This article analyzes the impact of high-stakes testing in Chicago on student achievement in grades targeted for promotional decisions. Using a three-level Hierarchical Linear Model, we estimate achievement value added in gate grades (test-score increases over and above that predicted from a student’s prior growth trajectory) for successive cohorts of students and derive policy effects by comparing value added pre- and postpolicy. Test scores in these grades increased substantially following the introduction of high-stakes testing. The effects are larger in the 6th and 8th grades and smaller in the 3rd grade in reading. Effects are also larger in previously low-achieving schools. In reading, students with low skills experienced the largest improvement in learning gains in the year prior to testing, while students with skills closer to their grade level experienced the greatest benefits in mathematics.
Studying Large-Scale Reforms of Instructional Practice: An Example from Mathematics and Science
Authors: Hamilton Laura S. , McCaffrey Daniel F. , Stecher Brian M. , Klein Stephen P. , Robyn Abby , Bugliari Delia
Published: June 2007

A number of challenges are encountered when evaluating a large-scale, multisite educational reform aimed at changing classroom practice. The challenges include substantial variability in implementation with little information on actual practice, lack of common, appropriate outcome measures, and the need to synthesize evaluation results across multiple study sites. This article describes an approach to addressing these challenges in the context of a study of the relationships between student achievement and instructional practices in the National Science Foundation’s Systemic Initiatives (SI) program. We gathered data from eleven SI sites and investigated relationships at the site level and pooled across sites using a planned meta-analytic approach. We found small but consistent positive relationships between teachers’ reported use of standards-based instruction and student achievement. The article also describes the ways in which we addressed the challenges discussed, as well as a number of additional obstacles that need to be addressed to improve future evaluations of large-scale reforms.
Methodological Thoughts on Measuring the Impact of Private Sector Competition on the Educational Marketplace
Authors: Goldhaber Dan D. , Eide Eric R.
Published: June 2007

Numerous researchers have speculated on and attempted to quantify the impact of policies designed to increase school choice and competition between schools, and they have reached widely differing conclusions. In this article, we provide an overview of the research that focuses on the potential achievement effects of greater private sector competition on K-12 schooling. In particular, we explore the various methodologies that have been used to assess the relative effectiveness of public and private schools. We discuss the strengths and shortcomings of different approaches, and based on this, we ponder what is known and not known about the impact of increased choice and competition. In conclusion, we maintain that although many of the methodologies used to assess the effects of particular interventions, such as educational vouchers, are sound, they likely fail to capture the general equilibrium consequences of enhanced choice. As a result, many of the questions we would want answered to inform policymaking remain unknown.
Counterfeit Social Capital and Mexican-American Underachievement
Authors: Ream Robert K.
Published: June 2007

A critical issue facing U.S. schools and one with broad social implication is the persistent disparity in educational achievement between racial/ethnic groups. The achievement gap may be particularly pronounced for Mexican-Americans who constitute the vast majority of U.S. Latinos and are among the most educationally at risk of all Latino subgroups. By employing mixed-methods research techniques, this study shows that social network instability accompanying high mobility rates may contribute to Mexican-American underachievement. Moreover, this investigation challenges wholly beneficial and ecumenical notions of social capital. Mexican-origin youth in possession of what on the surface appears to be a valued form of social currency may actually be the unwitting recipients of a form of counterfeit social capital that impinges on their school success. Efforts to reduce unnecessary and reactive student mobility and increase the stock of beneficial forms of social capital (while rooting out its impostors) may deserve policy consideration.
Teacher Effects and Teacher Effectiveness: A Validity Investigation of the Tennessee Value Added Assessment System
Authors: Kupermintz Haggai
Published: June 2007

This article addresses the validity of teacher evaluation measures produced by the Tennessee Value Added Assessment System (TVAAS). The system analyzes student test score data and estimates the effects of individual teachers on score gains. These effects are used to construct teacher value-added measures of teaching effectiveness. We describe the process of generating teacher effectiveness estimates in TVAAS and discuss policy implications of using these estimates for accountability purposes. Specifically, the article examines the TVAAS definition of teacher effectiveness, the mechanism employed in calculating numerical estimates of teacher effectiveness, and the relationships between these estimates and student ability and socioeconomic background characteristics. Our validity analyses point to several logical and empirical weaknesses of the system, and underscore the need for a strong validation research program on TVAAS.
A Nation at Rest: The American Way of Homework
Authors: Gill Brian P. , Schlossman Steven L.
Published: June 2007

We use several national surveys to provide a 50-year perspective on time spent on homework. The great majority of American children at all grade levels now spend less than one hour studying on a typical day—an amount that has not changed substantially in at least 20 years. Moreover, high school students in the late 1940s and early 1950s studied no more than their counterparts did in the 1970s, 1980s, and 1990s. Changes in educational opinion on homework over the last half century have had little effect on student behavior, with only two notable exceptions: a temporary increase in homework time in the decade following Sputnik, and a new willingness in the last two decades to assign small amounts to primary-grade students.
Distributed Leadership in Schools: The Case of Elementary Schools Adopting Comprehensive School Reform Models
Authors: Camburn Eric , Rowan Brian , Taylor James E.
Published: June 2007

This is a study of distributed leadership in the context of elementary schools' adoption of comprehensive school reforms (CSR). Most CSRs are designed to configure school leadership by defining formal roles, and we hypothesized that such programs activate those roles by defining expectations for and socializing (e.g., through professional development) role incumbents. Configuration and activation were further hypothesized to influence the performance of leadership functions in schools. Using data from a study of three of the most widely adopted CSR models, support was found for the configuration and activation hypotheses. Leadership configuration in CSR schools differed from that of nonCSR schools in part because of the addition of model-specific roles. Model participation was also related to the performance of leadership functions as principals in CSR schools and CSR-related role incumbents were found to provide significant amounts of instructional leadership. Further support for the activation hypothesis is suggested by positive relationships between leaders' professional development experiences and their performance of instructional leadership.
Making Sense of Distributed Leadership: The Case of Peer Assistance and Review
Authors: Goldstein Jennifer
Published: June 2007

This article explores a case of shifting leadership responsibility for teacher evaluation. Peer Assistance and Review (PAR) formally involves teachers in the summative evaluation of other teachers—although the boundaries of the involvement are often vague. Since teacher evaluation has traditionally been the domain of school principals, involving teachers in teacher evaluation raises questions about how those faced with the new role make sense of it and enact it. The article draws on theories of professions, organizations, and institutions to examine the implementation of PAR in one large urban school district. Findings suggest that, despite positive sentiments about the policy across stakeholder groups, those involved wanted principals to remain a central figure in the evaluation of teachers in PAR. Education's hierarchical norms, the difficulty of conducting evaluations, district leadership, and program ambiguity are identified as challenges to distributing leadership.
Leadership Content Knowledge
Authors: Stein Mary Kay , Nelson Barbara S.
Published: June 2007

Drawing inspiration from Shulman’s (1986) construct of pedagogical content knowledge, we propose that leadership content knowledge is a missing paradigm in the analysis of school and district leadership. After defining leadership content knowledge as that knowledge of academic subjects that is used by administrators when they function as instructional leaders, we present three cases of instructional leadership—situated at different school and district levels—and examine each for evidence of leadership content knowledge in use. Based on a cross-case analysis, we argue that as administrative levels increase and functions become broader, leadership content knowledge becomes less fine-grained, though always anchored in knowledge of the subject, how it is learned (by adults as well as students), and how it is taught. We go on to suggest that all administrators have solid mastery of at least one subject (and the learning and teaching of it) and that they develop expertise in other subjects by “postholing,” that is, conducting in-depth explorations of an important but bounded slice of the subject, how it is learned, and how it is taught. We conclude with an exploration of how content knowledge and leadership knowledge might be intertwined and suggestions for further research.
A Meta-Analysis of Reading Recovery in United States Schools
Authors: D’Agostino Jerome V. , Murphy Judith A.
Published: June 2007

We conducted a meta-analysis of 36 studies of Reading Recovery (RR), an intensive tutorial intervention designed to develop the literacy skills of low-performing first-grade students. Few individual studies of the program have yielded conclusive evidence regarding the program’s effectiveness due to various methodological limitations. We relied on specific meta-analytic strategies to combine as much available evidence as possible to study overall program effects. We also analyzed the results from the few more rigorously designed studies separately. In general, we found positive program effects for both discontinued and not discontinued students on outcomes tailored to the program and standardized achievement measures. RR effects were most pronounced, however, for discontinued students on measures designed for the program. Contrary to conventional belief, we found no evidence suggesting that prior observed effects could be explained completely by factors resulting from methodological flaws (e.g., regression artifacts).
Why Do School District Budget Referenda Fail?
Authors: Ehrenberg Ronald G. , Ehrenberg Randy A. , Smith Christopher L. , Zhang Liang
Published: June 2007

Our article analyzes historical data for New York State on the percentage of school board budget proposals that are defeated each year and panel data that we have collected on budget vote success for individual school districts in the state. We find that changes in state aid have little impact on budget vote success. Defeating a budget in one year increases the likelihood that voters will defeat a budget the next year. Finally, districts have a lower probability of having their budget proposals defeated when their school board members have longer terms.
Relationships Between School Drug Searches and Student Substance Use in U.S. Schools
Authors: Yamaguchi Ryoko , O’Malley Patrick M. , Johnston Lloyd D.
Published: June 2007

The purposes of this article are (a) to describe drug search policies in nationally representative samples of schools from 1998 to 2001 and (b) to examine the association between drug searches by schools and reported drug use by students. Most schools (86%) reported searching students for cause or suspicion, while 40% of schools reported randomly searching students. The use of drug searches differed according to school characteristics. For-cause drug searches were positively associated with 8th-, 10th-, and 12th-grade students’ 12-month illicit drug use, while random drug searches were associated (positively) only with 10th-grade students’ 12-month drug use.
“Highly Qualified” to Do What? The Relationship Between NCLB Teacher Quality Mandates and the Use of Reform-Oriented Instruction in Middle School Mathematics
Authors: Smith Thomas M. , Desimone Laura M. , Ueno Koji
Published: June 2007

The federal No Child Left Behind Act of 2001 (NCLB) calls for a highly qualified teacher in every classroom. According to the legislation, “highly qualified” is defined as full certification, a bachelor’s degree, and demonstrated content knowledge in all core subjects taught. States, district, and schools are spending considerable resources addressing these provisions, but to what degree are the readily available measures of teacher qualifications associated with the type of teaching advocated by standards-based reform? The authors examine the relationships among educational credentials, preparedness to teach content, participation in professional development, and use of reform-oriented instruction by middle school mathematics teachers using data from the 2000 National Assessment of Educational Progress (NAEP). Their analyses suggest that preparedness to teach mathematics content and participation in content-related professional development activities are associated with reform-oriented teaching, measured here as increased emphases on conceptual learning goals for students and increased use of reform-oriented teaching strategies.
The Returns of a Community College Education: Evidence From the National Education Longitudinal Survey
Authors: Marcotte Dave E. , Bailey Thomas , Borkoski Carey , Kienzl Greg S.
Published: June 2007

Using the 2000 follow-up of the National Education Longitudinal Survey, the authors estimated earnings effects of a community college education. Previous research relied on data collected from students enrolled 20 or 30 years ago. Because the labor market and community colleges have changed dramatically since then, the authors provide an update by studying students enrolled in the 1990s. They found substantial evidence that a community college education has positive effects on earnings among young workers. This effect was larger for annual earnings than for hourly wages. Earnings benefits accrued both to those who failed to earn a credential and to those who earned an associate degree. The current results are similar to estimates for earlier cohorts. The stability of the wage advantage of a community college education during a period of growing enrollment provides evidence of a growing relative demand for a community college over a high school education.
State-Mandated Accountability in High Schools: Teachers’ Interpretations of a New Era
Authors: Louis Karen Seashore , Febey Karen , Schroeder Roger
Published: June 2007

This article addresses conflicting statements about the impact of accountability policies in which some argue that testing undermines good teaching, while others claim that it stimulates improvement. The authors begin with the assumption that it is important to explore implementers’ cognitive perspectives in order to understand a policy’s effects. Using a sensemaking perspective, they examine teachers’ responses to accountability in three high schools selected because they were deemed to have a strong teacher culture that supported quality education. The schools were located in states with diverse histories of accountability legislation, but in districts with well-established local accountability policies. Findings suggested that sensemaking activity occurred in all three schools and was associated with teachers’ interpretation of accountability policies and with their efforts to change classroom practice to be consistent with accountability policies. Active efforts by district-level administrators to mediate sensemaking affected teachers’ attitudes toward accountability policies and standards-driven reform.
Effects of Kindergarten Retention Policy on Children’s Cognitive Growth in Reading and Mathematics
Authors: Hong Guanglei , Raudenbush Stephen W.
Published: June 2007

Grade retention has been controversial for many years, and current calls to end social promotion have lent new urgency to this issue. On the one hand, a policy of retaining in grade those students making slow progress might facilitate instruction by making classrooms more homogeneous academically. On the other hand, grade retention might harm high-risk students by limiting their learning opportunities. Analyzing data from the US Early Childhood Longitudinal Study Kindergarten cohort with the technique of multilevel propensity score stratification, we find no evidence that a policy of grade retention in kindergarten improves average achievement in mathematics or reading. Nor do we find evidence that the policy benefits children who would be promoted under the policy. However, the evidence does suggest that children who are retained learn less than they would have had they instead been promoted. The negative effect of grade retention on those retained has little influence on the overall mean achievement of children attending schools with a retention policy because the fraction of children retained in those schools is quite small. Nevertheless, the effect of retention on the retainees is considerably large.
Called to Account: Analyzing the Origins and Spread of State Performance-Accountability Policies for Higher Education
Authors: McLendon Michael K. , Hearn James C. , Deaton Russ
Published: June 2007

Employing a theoretical framework derived from the policy innovation and diffusion literature, this research examines how variations over time and across state sociopolitical systems influence states’ adoption of accountability policies in higher education. Specifically, factors influencing the adoption of three kinds of performance-accountability policies for public higher education in the period 1979–2002 were investigated. Findings from the event history analysis supported the authors’ original hypotheses only in part; the primary drivers of policy adoption were legislative party strength and higher-education governance arrangements, but the direction of these influences varied across the policies studied.
Implications of Income-Based School Assignment Policies for Racial School Segregation
Authors: Reardon Sean F. , Yun John T. , Kurlaender Michal
Published: June 2007

A number of public school districts in the United States have adopted income-based integration policies—policies that use measures of family income or socioeconomic status—in determining school assignment. Some scholars and policymakers contend that such policies will also reduce racial segregation. In this article this assumption is explored by computing upper and lower bounds on the possible and probable levels of racial segregation that would result from race-neutral income-based school assignment policies. The article finds that, in general, income integration is no guarantee of even modest racial desegregation. In particular, the extent of ancillary racial integration produced by an income-integration policy will depend on the size of racial income disparities within a given district, the specifics of an income-integration policy, and the patterns of racial and socioeconomic residential segregation in a school district. Data on racial income inequality and income segregation in urban districts throughout the United States indicate that very high levels of racial segregation are possible under any practical income-integration policy. The authors conclude that, given the extent of residential racial segregation in the United States, it is unlikely that race-neutral income-integration policies will significantly reduce school racial segregation, although there is reason to believe that such policies are likely to have other beneficial effects on schooling.
Early Education Policy Alternatives: Comparing Quality and Outcomes of Head Start and State Prekindergarten
Authors: Henry Gary T. , Gordon Craig S. , Rickman Dana K.
Published: June 2007

The debates over the 2003 reauthorization of Head Start highlighted a controversy about the devolution of federal early education policy. At the center of the debate is the concern that state control of early education programs will reduce the quality and effectiveness of federal support for children living in poverty, and their families. The current fragmentation of early education policy, with both federal Head Start programs and state-subsidized prekindergarten programs operating in close proximity, presents an opportunity to compare the programs’ quality and effectiveness within a region of common support. In this study, propensity score techniques were used to match a probability sample of Head Start participants in Georgia with a group of children who were eligible for Head Start but who attended the state prekindergarten program in Georgia. The two groups were statistically similar at the beginning of their preschool year on three of four direct assessments (p < .05), but by the beginning of kindergarten the children attending the state prekindergarten program posted higher developmental outcomes on five of six direct assessments (p < .05) and 14 of 17 ratings by kindergarten teachers (p < .05). This study indicates that economically disadvantaged children who attended Georgia’s universal prekindergarten entered kindergarten at least as well prepared as similar children who attended the Head Start program.
Comparing and Contrasting the National Research Council Report On Evaluating Curricular Effectiveness With the What Works Clearinghouse Approach
Authors: Confrey Jere
Published: June 2007

This article summarizes the findings of the National Research Council (NRC) report On Evaluating Curricular Effectiveness and examines the reviews in middle grades mathematics undertaken by the What Works Clearinghouse (WWC). The NRC report reviewed and assessed 147 key evaluations of 13 National Science Foundation–supported K–12 mathematics curricula and six commercially generated curricula. The report found that the evaluations overall were not sufficiently robust to permit confident judgments on individual programs, so it instead focused on how to define effectiveness in conducting future evaluations. Effectiveness was defined as “an integrated judgment based on interpretation of a number of scientifically valid evaluations that combine social values, empirical evidence, and theoretical rationales” ( NRC, 2004 , p. 4). The report introduced a model for curricular evaluation that includes program theory, implementation, and outcome measures, and reviewed three major methodologies found in the literature: content analysis, comparative analysis, and case study. This article then examines the What Works Clearinghouse’s exclusive emphasis on experimental and quasi-experimental designs from the perspective of an author of the NRC report. The two reports agree in recognizing the need for significant improvement in evaluation quality; however, they differ in four areas: standards for individual studies, need for multiple methods, the way to accumulate information across a set of studies, and how to communicate results with the public. This article concludes with a call for focused efforts to address several shared targets needed to make further progress on how to establish curricular effectiveness.
Adoption of Merit-Based Student Grant Programs: An Event History Analysis
Authors: Doyle William R.
Published: June 2007

As of 2005, 15 states had adopted a broad-based merit aid program, providing a combined $1.2 billion for college students on the basis of academic qualifications. This represents a shift away from a long tradition of need-based aid at the state and federal levels. This article utilizes a Cox proportional hazards model to analyze states’ characteristics that are associated with their probability of adopting a merit aid program. Lower levels of college continuation and educational attainment are found to increase the hazard for a state’s adoption of a merit aid program, whereas higher levels of student out-migration decrease the proportional hazard for a state’s adoption of such a program.
Controlling Error in Multiple Comparisons, with Examples from State-to-State Differences in Educational Achievement
Authors: Williams Valerie S. L. , Jones Lyle V. , Tukey John W.
Published: June 2007

Three alternative procedures to adjust significance levels for multiplicity are the traditional Bonferroni technique, a sequential Bonferroni technique developed by Hochberg (1988) , and a sequential approach for controlling the false discovery rate proposed by Benjamini and Hochberg (1995). These procedures are illustrated and compared using examples from the National Assessment of Educational Progress (NAEP). A prominent advantage of the Benjamini and Hochberg (B-H) procedure, as demonstrated in these examples, is the greater invariance of statistical significance for given comparisons over alternative family sizes. Simulation studies show that all three procedures maintain a false discovery rate bounded above, often grossly, by α (or α/2). For both uncorrelated and pairwise families of comparisons, the B-H technique is shown to have greater power than the Hochberg or Bonferroni procedures, and its power remains relatively stable as the number of comparisons becomes large, giving it an increasing advantage when many comparisons are involved. We recommend that results from NAEP State Assessments be reported using the B-H technique rather than the Bonferroni procedure.
A Straightforward Approach to Markov Chain Monte Carlo Methods for Item Response Models
Authors: Patz Richard J. , Junker Brian W.
Published: June 2007

This paper demonstrates Markov chain Monte Carlo (MCMC) techniques that are particularly well-suited to complex models with item response theory (IRT) assumptions. MCMC may be thought of as a successor to the standard practice of first calibrating the items using E-M methods and then taking the item parameters to be known and fixed at their calibrated values when proceeding with inference regarding the latent trait, in contrast to this two-stage E-M approach, MCMC methods treat item and subject parameters at the same time; this allows us to incorporate standard errors of item estimates into trait inferences, and vice versa. We develop a MCMC methodology, based on Metropolis-Hastings sampling, that can be routinely implemented to fit novel IRT models, and we compare the algorithmic features of the Metropolis- Hastings approach to other approaches based on Gibbs sampling. For concreteness we illustrate the methodology using the familiar two-parameter logistic (2PL) IRT model; more complex models are treated in a subsequent paper (Patz & Junker, in press).
Nonparametric Tests for Analyzing Interactions Among Intra-Block Ranks in Multiple Group Repeated Measures Designs
Authors: Beasley T. Mark
Published: June 2007

This study developed an extension of the Hollander and Sethuraman (1978) statistic (B2 ) for testing discordance among intra-block rankings of K elements for multiple groups (J ≥ 2) of raters. B2 was demonstrated to be equivalent to the Pillai-Bartlett trace (V ) from a multivariate profile analysis performed on the ranks such that B2 = V (N - 1) Results confirmed the utility of B2 as an omnibus test of interaction (i.e., discordance) among intra-block ranks and demonstrated that it was more powerful than the multivariate approach to ranked data suggested by Serlin and Marascuilo (1983) . An extension of the Friedman (1937) two-way ANOVA for intra-block ranks was also developed. The adequacy of these procedures for testing interactions in multiple group repeated measures designs was investigated. The Friedman model demonstrated adequate statistical properties only when covariance matrices were spherical. Results also demonstrated that the Hollander-Sethuraman model was useful in testing interaction contrasts.
Some Class-Participation Demonstrations for Introductory Probability and Statistics
Authors: Gelman Andrew , Glickman Mark E.
Published: June 2007

We present several classroom demonstrations, based on well-known statistical ideas, that have sparked student involvement in our introductory undergraduate courses in probability and statistics.
Note on the Odds Ratio and the Probability Ratio
Authors: Cohen Michael P.
Published: June 2007

This note compares the odds ratio with the probability ratio (relative risk). These quantities arise, for example, in the analysis of educational and social science data by means of logistic regression.
Design Issues for Experiments in Multilevel Populations
Authors: Moerbeek Mirjam , van Breukelen Gerard J. P. , Berger Martijn P. F.
Published: June 2007

For the design of experiments in multilevel populations the following questions may arise: What is the optimal level of randomization? Given a certain budget for sampling and measuring, what is the optimal allocation of units? What is the required budget for obtaining a certain power on the test of no treatment effect? In this article these questions will be dealt with for populations with two or three levels of nesting and continuous outcomes. Multilevel models are used to model the relationship between experimental condition and the outcome variable. The estimator of the regression, coefficient associated with treatment condition, a parameter assumed to be fixed in this paper; is of main interest and should be estimated as efficiently as possible. Therefore, its variance is used as a criterion for optimizing the level of randomization and the allocation of units.
A Hierarchical IRT Model for Criterion-Referenced Measurement
Authors: Janssen Rianne , Tuerlinckx Francis , Meulders Michel , De Boeck Paul
Published: June 2007

A hierarchical IRT model is proposed for mastery classification in criterion-referenced measurement. In this model, items measuring the same criterion are grouped, and a difficulty and discrimination parameter of the criterion is estimated on the same scale as the person and item parameters. The level of proficiency of a student with respect to the criterion is determined by the probability of success on the criterion. Cutoff points on the probability scale can be used to classify respondents into masters and nonmasters. The hierarchical IRT model is estimated using the Gibbs sampler and tested using posterior predictive checks. The model is illustrated with a test measuring the attainment targets of reading comprehension (in Dutch) at the end of primary education.
Taylor Approximations to Logistic IRT Models and Their Use in Adaptive Testing
Authors: Veerkamp Wim J. J.
Published: June 2007

Taylor approximation can be used to generate a linear approximation to a logistic ICC and a linear ability estimator. For a specific situation it will be shown to result in a special case of a Robbins-Monro item selection procedure for adaptive testing. The linear estimator can be used for the situation of zero and perfect scores when maximum likelihood estimation fails to come up with a finite estimate. It is also possible to use this estimator to generate starting values for maximum likelihood and weighted likelihood estimation. Approximations to the expectation and variance of the linear estimator for a sequence of Robbins-Monro item selections can be determined analytically.
Detection of Known Items in Adaptive Testing with a Statistical Quality Control Method
Authors: Veerkamp Wim J. J. , Glas Cees A. W.
Published: June 2007

Due to previous exposure of items in adaptive testing, items may become known to a substantial portion of examinees. A disclosed item is bound to show drift in the item parameter values. In this paper it is suggested to use a statistical quality control method for the detection of known items. The method is worked out in detail for the 1-PL and 3-PL models. Adaptive test data are used to re-estimate the item parameters, and these estimates are used in a test of parameter drift. The method is illustrated in a number of simulation studies, including a power study.
A Taxonomy of IRT Models for Ordering Persons and Items Using Simple Sum Scores
Authors: Sijtsma Klaas , Hemker Bas T.
Published: June 2007

The stochastic ordering of the latent trait by means of the unweighted total score is considered for 10 dichotomous IRT models and 10 polytomous IRT models. The conclusion is that the stochastic ordering property holds for all dichotomous IRT models and for two polytomous IRT models. Also, the invariant item ordering property is considered for the same 20 IRT models. It is concluded that invariant item ordering holds for three dichotomous IRT models and three polytomous IRT models. The person and item ordering results are summarized in a taxonomy of IRT models. Some consequences far practical test construction are briefly discussed.
Weighted Simplex Procedures for Determining Boundary Points and Constants for the Univariate and Multivariate Power Methods
Authors: Headrick Todd C. , Sawilowsky Shlomo S.
Published: June 2007

The power methods are simple and efficient algorithms used to generate either univariate or multivariate nonnormal distributions with specified values of (marginal) mean, standard deviation, skew, and kurtosis. The power methods are bounded as are other transformation techniques. Given an exogenous value of skew, there is an associated lower bound of kurtosis. Previous approximations of the boundary for the power methods are either incorrect or inadequate. Data sets from education and psychology can be found to lie within, near, or outside tile boundary of the power methods. In view of this, we derived necessary and sufficient conditions using the Lagrange multiplier method to determine the boundary of the power methods. The conditions for locating and classifying modes for distributions on the boundary were also derived. Self-contained interactive Fortran programs using a Weighted Simplex Procedure were employed to generate tabled values of minimum kurtosis for a given value of skew and power constants for various (non)normal distributions.
A New Computer Algorithm for Simultaneous Test Construction of Two-Stage and Multistage Testing
Authors: Wu Ing-Long
Published: June 2007

This paper discusses the simultaneous approach for test generation of two-stage and multistage tests from an item bank. However, most previous test generation problems involved binary programming models, and the efficiency of available solution algorithms is the major concern for these problems. Therefore, this study considers two important concepts on the solution process, i.e., alternative ways to formulate mathematical models and alternative solution algorithms. Based on these two concepts, this paper is two-fold. First, two binary programming models with a special network structure that would be explored computationally are presented for modeling these problems. The first model maximizes test information function at one specified ability point and the second model matches the target test information function at several specified ability points as closely as possible. Second, an efficient special-purpose network algorithm is then used to solve these two models. Hence, the test construction process tends to be improved in terms of both computational efforts and quality of tests. An empirical study shows the results in line with the above two criteria.
CUSUM-Based Person-Fit Statistics for Adaptive Testing
Authors: van Krimpen-Stoop Edith M.L.A. , Meijer Rob R.
Published: June 2007

Item scores that do not fit an assumed item response theory model may cause the latent trait value to be inaccurately estimated. Several person-fit statistics for detecting nonfitting score patterns for paper-and-pencil tests have been proposed. In the context of computerized adaptive tests (CAT), the use of person-fit analysis has hardly been explored. Because it has been shown that the distribution of existing person-fit statistics is not applicable in a CAT, in this study new person-fit statistics are proposed and critical values for these statistics are derived from existing statistical theory. Statistics are proposed that are sensitive to runs of correct or incorrect item scores and are based on all items administered in a CAT or based on subsets of items, using observed and expected item scores and using cumulative sum (CUSUM) procedures. The theoretical and empirical distributions of the statistics are compared and detection rates are investigated. Results showed that the nominal and empirical Type I error rates were comparable for CUSUM procedures when the number of items in each subset and the number of measurement points were not too small. Detection rates of CUSUM procedures were superior to other fit statis­tics. Applications of the statistics are discussed.