bluuebunny/crossref_metadata_2025 huggingface dataset:
Reconstituting Schools: “Testing” the “Theory of Action”
Authors: Malen Betty , Croninger Robert , Muncey Donna , Redmond-Jones Donna
Published: June 2007

This article identifies key elements of the “theory of action” embodied in reconstitution reforms and examines them in light of findings acquired from a two-year study that documents what happened when a particular rendition of reconstitution was enacted and implemented. The evidence from this study suggests that the “theory of action” embedded in reconstitution reforms may be seriously, if not fatally flawed. On every critical count, the dominant patterns of implementation ran counter to the major premises (and promises) of the policy. This article considers alternative interpretations of the data and suggests directions for future research.
What Do Parents Want From Schools? Evidence From the Internet
Authors: Schneider Mark , Buckley Jack
Published: June 2007

One of the most contentious policy areas in the United States today is the expansion of school choice. While many dimensions of parental-choice behavior have been analyzed, many of the most enduring questions center on the aspects of schools parents prefer and how these preferences will affect the socioeconomic and racial composition of schools. Using Internet-based methodological tools, we study parental preferences revealed through information search patterns and compare these findings to the standard ones in the literature, which are based largely on telephone interviews. Based on this evidence we suggest that unfettered choice may lead to undesirable outcomes in the distribution of students, and it may also lead to reduced pressure on schools to improve academic performance.
Creaming Versus Cropping: Charter School Enrollment Practices in Response to Market Incentives
Authors: Lacireno-Paquet Natalie , Holyoke Thomas T. , Moser Michele , Henig Jeffrey R.
Published: June 2007

Proponents of school choice present market-based competition as a means of leveling disparities between race, class and performance in public school systems. Opponents see school choice as threatening to exacerbate this problem because competition for students will pressure individual schools into targeting students with the highest performance and the least encumbered with personal and social disadvantages. We suggest that some charter schools, by background and affiliation, are likely to be more market-oriented in their behavior than others, and test the proposition that market-oriented charter schools engage in cream-skimming while others disproportionately serve highly disadvantaged students. Comparing student composition in market-oriented charter schools, nonmarket-oriented charter schools, and traditional public schools in Washington, DC, we find little evidence that market-oriented charters are focusing on an elite clientele, but they are less likely than the other two types of schools to serve some high need populations. Rather than skimming the cream off the top of the potential student population, market-oriented charter schools may be “cropping off” service to students whose language or special education needs make them more costly to educate.
Randomized Experiments in Educational Policy Research: A Critical Examination of the Reasons the Educational Evaluation Community has Offered for not Doing Them
Authors: Cook Thomas D.
Published: June 2007

This article notes the paucity with which reform efforts in education have been evaluated experimentally, despite well-nigh universal acknowledgement that experiments provide the best justification for causal conclusions. And of the few experiments that have been completed, most were done by groups outside of the community of educational evaluators working in schools and colleges of education. The reasons educational evaluators cite for not doing experiments are critically appraised. Some are shown to be wrong or over-generalized; and others point to the need for future experiments that depend more heavily on program theory and the assessment of implementation and other intervening processes. The alternatives to experiments that educational evaluators prefer are briefly appraised and found wanting whenever a high standard is needed for justifying causal conclusions.
Age 21 Cost-Benefit Analysis of the Title I Chicago Child-Parent Centers
Authors: Reynolds Arthur J. , Temple Judy A. , Robertson Dylan L. , Mann Emily A.
Published: June 2007

We conducted the first cost-benefit analysis of a federally financed, comprehensive early childhood program. The Title I Chicago Child-Parent Centers are located in public schools and provide educational and family support services to low-income children from ages 3 to 9. Using data from a cohort of 1,539 program and comparison-group children born in 1980 who participate in the Chicago Longitudinal Study, measures of program participation were significantly associated with greater school achievement, higher rates of high school completion, and with significantly lower rates of remedial education services, juvenile delinquency, and child maltreatment. Economic analyses indicated that the measured and projected economic benefits of preschool participation, school-age participation, and extended program participation exceeded costs. In present-value 1998 dollars, the preschool program provided a return to society of $7.14 per dollar invested by increasing economic well-being and tax revenues, and by reducing public expenditures for remedial education, criminal justice treatment, and crime victims. The extended intervention program (4 to 6 years of participation) provided a return to society of $6.11 per dollar invested while the school-age program yielded a return of $1.66 per dollar invested. Findings demonstrate that an established public program can provide benefits that far exceed costs. Key elements of CPC program effectiveness include an instructional focus on literacy, opportunities for intensive parent involvement, and implementation by well-trained staff within a single administrative system.
The Influence of High School Graduation Requirement Policies in Mathematics and Science on Student Course-Taking Patterns and Achievement
Authors: Teitelbaum Peter
Published: June 2007

One of the most widely implemented educational reform efforts of the last two decades has been the adoption of increased high school graduation requirements, especially in mathematics and science. The present study examines the effect of that reform. More particularly, this article first investigates the extent to which a commonly adopted requirement that students complete three credits in mathematics and science was implemented at schools. Using multilevel regression analysis, the study then examines the relationship between three-course requirements in mathematics and science and three expected outcomes: increases in the number of credits students earned in mathematics and science, increases in the level of mathematics and science classes completed by students, and increases in student achievement in math and science, as measured by 8th- to 12th-grade test score gains. To test this relationship, the author drew a nationally representative sample of 1992 public high school graduates from the National Educational Longitudinal Study conducted in 1988. Analysis of these data indicates that many schools that required students to complete three courses in math or science in order to graduate did not strictly enforce this policy, allowing a substantial percentage of the students to graduate without earning three credits in these subjects. Nevertheless, three-course requirements influenced students to earn more credits in mathematics and science; however, only some students completed this additional coursework in advanced classes. In addition, student test score gains did not vary by high school graduation requirement policy. These findings suggest that increasing the number of credits students have to earn in mathematics and science to graduate from high school by itself may not be sufficient to improve student proficiency in these subjects.
Resources, Instruction, and Research
Authors: Cohen David K. , Raudenbush Stephen W. , Ball Deborah Loewenberg
Published: June 2007

Many researchers who study the relations between school resources and student achievement have worked from a causal model, which typically is implicit. In this model, some resource or set of resources is the causal variable and student achievement is the outcome. In a few recent, more nuanced versions, resource effects depend on intervening influences on their use. We argue for a model in which the key causal agents are situated in instruction; achievement is their outcome. Conventional resources can enable or constrain the causal agents in instruction, thus moderating their impact on student achievement. Because these causal agents interact in ways that are unlikely to be sorted out by multivariate analysis of naturalistic data, experimental trials of distinctive instructional systems are more likely to offer solid evidence on instructional effects.
Comprehensive School Reform in Culturally and Linguistically Diverse Contexts: Implementation and Outcomes from a Four-Year Study
Authors: Datnow Amanda , Borman Geoffrey D. , Stringfield Sam , Overman Laura T. , Castellano Marisa
Published: June 2007

This article presents findings from a 4-year study of 13 culturally and linguistically diverse elementary schools implementing comprehensive school reform (CSR) models. The study focused on: (a) the actions at the state and district levels that facilitated or inhibited reform implementation; (b) the adaptability of the various reforms in multicultural, multilingual contexts; and (c) the student achievement outcomes associated with reform, for schools as a whole and for language minority students in particular. Some schools implemented reforms and bilingual education programs in mutually supportive ways; others had difficulty adapting reforms to suit the needs of Limited English Proficient (LEP) students. Reforms generally helped educators meet goals for multicultural education, but in some cases, educators’ beliefs about student ability, race, and language served as constraints to reform. Students from CSR schools had achievement outcomes that were generally equivalent to those for students from matched comparison schools. Under some circumstances, though, LEP students and their English-speaking peers from CSR schools outperformed their comparison school counterparts.
Schools in Transition: Reform Efforts and School Capacity in Washington State
Authors: Borko Hilda , Wolf Shelby A. , Simone Genet , Uchiyama Kay Pippin
Published: June 2007

This article features case studies of two elementary schools, identified as exemplary by Washington educators, as they worked to enact Washington’s Essential Academic Learning Requirements and help students achieve proficiency on the Washington Assessment of Student Learning. We describe school- and classroom-level practices and analyze the schools’ progress toward successful enactment of Washington’s reform vision by considering six dimensions of school capacity: principal leadership; professional community; program coherence; technical resources; knowledge, skills, and dispositions of individual teachers; and learning opportunities for teachers. Supporting the claim that large-scale reform takes time, we argue that for a young reform effort “exemplary” is best understood in terms of capacity rather than outcome.
Making the Grade: Comparing DC Charter Schools to Other DC Public Schools
Authors: Schneider Mark , Buckley Jack
Published: June 2007

Across the United States, charter schools have become one of the most frequently used means of increasing choice among educational alternatives. In this article we use data from a recent telephone survey of Washington D.C. parents to evaluate the success of the District’s large and growing charter school program. We find that parents with children in charter schools rate their teachers, principals, facilities and schools higher than their traditional public counterparts. This finding is robust even when controlling for self-selection into charter schools. Based on these empirical results, we argue that the greater satisfaction with charter schools reflected in these differences in grades is not simply the result of the act of choosing.
Examining Relationships Between Where Students Start and how Rapidly they Progress: Using New Developments in Growth Modeling to Gain Insight into the Distribution of Achievement Within Schools
Authors: Seltzer Michael , Choi Kilchan , Thum Yeow Meng
Published: June 2007

Studying change in student achievement is of central importance in numerous areas of educational research, including efforts to monitor school performance, investigations of the effects of educational interventions over time, and school effects studies focusing on how differences in school policies and practices relate to differences in student progress. In this article, we argue that in studying patterns of change, it is often important to consider the relationship between where students start (i.e., their initial status) and how rapidly they progress (i.e., their rates of change). Drawing on recent advances in growth modeling methodology, we illustrate the potential value of such an approach in the context of monitoring school performance. In particular, we highlight the ways in which attending to initial status in analyses of student progress can help draw attention to possible concerns regarding the distribution of achievement within schools. To convey the logic of our approach and illustrate various analysis possibilities, we fit a series of growth models to the time series data for students in several schools in the Longitudinal Study of American Youth (LSAY) sample. In a final section, we discuss some of the possibilities that arise in employing a modeling approach of this kind in evaluating educational programs and in conducting school effects research.
Raising the Bar and Equity? Effects of State High School Graduation Requirements and Accountability Policies on Students' Mathematics Course Taking
Authors: Schiller Kathryn S. , Muller Chandra
Published: June 2007

In response to the national push to raise academic performance of all students, most states have adopted policies designed to raise academic standards, monitor progress toward those standards, and hold schools and students responsible for attaining them. Given the complex nature of the educational process, these policies are likely to have mixed effects on both general levels of attainment and stratification based on race or ethnicity and social class. Using nationally representative longitudinal data and hierarchical linear modeling, this article explored the association between students' mathematics course work and states' high school graduation requirements and assessment or accountability policies. We found that students in states with more graduation requirements tended to enroll in higher level mathematics courses as freshmen and persist to take more advanced level courses. Similar trends were also found for students in states that link test performance to consequences for schools. Extensive testing, however, had little effect on course taking except to increase differences based on socioeconomic status. In contrast, differences between racial or ethnic groups tended to be smaller in states where test performance was linked to consequences for students.
Leaders as Learners: An Inquiry into the Formation and Transformation of Principals’ Professional Perspectives
Authors: McGough David J.
Published: June 2007

This article reports on a research project that explored the manner in which veteran school principals become aware of, and committed to, new perspectives and practices. Personal stories of formative and transformative change from the careers of 23 school principals were collected and analyzed comparatively. Four analytic tools were devised that can support future studies of principal learning. The findings indicate that these principals’ professional learning is influenced by (a) impressions of teachers and schooling formed in early childhood, (b) progression through a common perspective development sequence with patterned influences, (c) a personal orientation to learning, which can be seen as a combination of four dynamics—connections, context, characteristics, and concepts—with one dominating the others, and (d) a story about oneself as a learner—the learning story—that serves as both a guide for approaching novelty and a catalyst for change. Implications for scholars and policymakers are discussed.
Rethinking Expert Testimony in Education Rights Litigation
Authors: Welner Kevin G. , Kupermintz Haggai
Published: June 2007

Courts often rely on the testimony of experts to understand arguments and implications in education rights litigation. But expert testimony, and statistical testimony in particular, can offer a false sense of security for the unwary. This article uses expert testimony offered in two recent desegregation cases to consider whether sufficient protections are presently in place to protect judges, who are usually statistical novices, from being confused or misled by experts. These case studies illustrate how, without the use of additional protections, courts can be misled. Following this examination, we offer suggestions intended to improve judges’ comprehension of expert testimony. At its most general level, this article addresses the role of researchers in presenting important educational issues in ways that speak clearly to policy-makers.
The National Board for Professional Teaching Standards (NBPTS) Process: Who Applies and What Factors Are Associated with NBPTS Certification?
Authors: Goldhaber Dan , Perry David , Anthony Emily
Published: June 2007

National Board Certification represents one of the most significant reform efforts in the area of teacher quality in the last two decades. Since the National Board for Professional Teaching Standards (NBPTS) certified its first round of teachers in 1995, approximately 32,000 teachers have become certified at a cost to the country of well over $300 million, yet no large-scale quantitative research exists on the candidates of the program. In this article, we describe the results of a study of teachers in North Carolina assessing the factors associated with the decision to apply to NBPTS and those associated with certification of candidates. We find that, all else equal, teachers who are African-American, female, score higher on standardized tests, and/or are younger are more likely to apply for certification. Among applicants, we find that African-American and male teachers are less likely to be certified, and teachers who score higher on standardized tests are far more likely to be certified.
Success for All: First-Year Results From the National Randomized Field Trial
Authors: Borman Geoffrey D. , Slavin Robert E. , Cheung Alan , Chamberlain Anne M. , Madden Nancy A. , Chambers Bette
Published: June 2007

This article reports first-year achievement outcomes of a national randomized evaluation of Success for All, a comprehensive reading reform model. Forty-one schools were recruited for the study and were randomly assigned to implement Success for All or control methods. No statistically significant differences between experimental and control groups were found in regard to pretests or demographic characteristics. Hierarchical linear model analyses revealed a statistically significant school-level effect of assignment to Success for All of nearly one quarter of a standard deviation—or more than 2 months of additional learning—on individual Word Attack test scores, but there were no school-level differences on the three other posttest measures assessed. These results are similar to those of earlier matched experiments and correspond with the Success for All program theory.
Updating the Economic Impacts of the High/Scope Perry Preschool Program
Authors: Nores Milagros , Belfield Clive R. , Barnett W. Steven , Schweinhart Lawrence
Published: June 2007

This article derives an updated cost-benefit ratio for the High/Scope Perry Preschool Program, an intensive preschool intervention delivered during the 1960s to at-risk children in Ypsilanti, Michigan. Because children were randomly assigned to the program or a control group, differences in outcomes are probably attributable to program status. Data on outcome differences is now available on participants as they reached the age of 40; outcomes include educational attainment, earnings, criminal activity, and welfare receipt. These outcomes are rendered in money terms and compared to the costs of delivering the program to calculate the net present value of the program both for participants and for society. The data show strong advantages for the treatment group in terms of higher lifetime earnings and lower criminal activity. For the general public, gains in tax revenues, lower expenditures on criminal justice, lower victim costs, and lower welfare payments easily outweigh program costs. At a 3% discount rate the program repays $12.90 for every $1 invested from the perspective of the general public; with a 7% discount rate, the repayment per dollar is $5.67. Returns are even higher if the total benefits—both public and private—are counted. However, there are strong differences by gender: a large proportion of the gains from the program come from lower criminal activity rates by the treatment group, almost all of which is undertaken by the males in the sample. The implications of these findings for public policy on early childhood education are considered.
Racial and Ethnic Differences in the Impact of Work-First Policies on College Access
Authors: Goldrick-Rab Sara , Shaw Kathleen M.
Published: June 2007

The college participation rates of African Americans and Latinos continue to lag behind those of other racial and ethnic groups in the United States, despite the efforts of financial aid and affirmative action policies. Two recent federal policies that are “work-first” in nature threaten to further exacerbate racial and ethnic disparities in college access. This article examines the complex ways in which the 1996 welfare reform and the 1998 Workforce Investment Act differentially affect opportunities for college enrollment among disadvantaged adults. Utilizing national and state-level data, the authors argue that both policies restrict access to postsecondary education through the implementation of their guiding philosophy, “work-first,” which emphasizes rapid job placement as the strategy of choice in achieving stable employment and moving out of poverty. These policies have reduced the size of the clientele receiving welfare and restricted access to education and training for those who remain on the rolls. Moreover, this reduction in access is particularly acute among African Americans and Latinos. Thus, the findings indicate that these work-first federal policies serve to limit higher education opportunities available to these already disadvantaged populations.
Are Charter School Students Harder to Educate? Evidence From Washington, D.C.
Authors: Buckley Jack , Schneider Mark
Published: June 2007

One point of debate in the recent controversy in the media and among policy analysts over the academic achievement of charter school students is whether the charter students are in some way harder to educate than their counterparts enrolled in traditional public schools. This article examines this question using data from the 2002–2003 school year in Washington, D.C. It begins by examining a simple binomial model of the proportion of students in key demographic and programmatic categories linked to educability. It then turns to the estimation of a more theoretically appropriate mixture model that assumes two latent categories of charter schools. It concludes with an analysis that moves beyond simple demographic/programmatic factors to consider measures of educability using individual-level survey data from charter and traditional public school students. Overall, there is mixed evidence of differences in the educability of students in the two sectors.
Effects of Class Size and Instruction on Kindergarten Achievement
Authors: Milesi Carolina , Gamoran Adam
Published: June 2007

Although experimental results indicate that smaller classes promote higher achievement in early elementary school, the broader literature on class-size effects is inconclusive. This seeming contradiction raises questions about the generalizability of experimental evidence, an issue that this article addresses by examining the effects of class size on achievement in kindergarten with data from a nationwide survey, the Early Childhood Longitudinal Study–Kindergarten Class of 1998–99. To distinguish class-level from individual-level effects, this analysis utilizes hierarchical linear models. In response to concerns about selectivity, teacher fixed-effects models are also estimated. In an effort to understand the inconsistent findings of the past, the authors examine classroom conditions that may affect the link between class size and academic achievement, and also consider whether class size has different effects for different groups of students. The authors find no evidence of class-size effects on student achievement in either reading or mathematics, and results indicate that class size is equally insignificant for students from different race/ethnic, economic, and academic backgrounds. Teacher fixed-effects analyses also yield null findings for class size. Instructional activities offer significant boosts to achievement, but the effects of instruction do not differ between small and large classes. The authors discuss why the small class size advantage evidenced by experimental research might not generalize to nonexperimental, naturally occurring settings throughout the nation.
Meeting Expectations? An Empirical Investigation of a Standards-Based Assessment of Reading Comprehension
Authors: Rupp André A. , Lesaux Nonie K.
Published: June 2007

The present study was designed to examine the relationship between performance on a standards-based assessment of reading comprehension in fourth grade and performance on a diagnostic battery of component skills of reading for a cohort of children who were followed from kindergarten through fourth grade. The findings demonstrate that the relationship between performance on component skills of reading and the proficiency classifications from the standards-based assessment is generally weak. They also demonstrate that the relationship between the proficiency classifications and a norm-referenced diagnostic measure of reading comprehension is only moderate. Furthermore, the study shows that the broad proficiency-level classifications of the standards-based assessment mask significant heterogeneity in children’s performance on some of the component skills assessed, both within and across proficiency classifications. In turn, the ability of the standards-based assessment results to inform instructional interventions is low. The implications of these findings for instructional planning within standards-based accountability systems are discussed.
Effects of a Voluntary Summer Reading Intervention on Reading Achievement: Results From a Randomized Field Trial
Authors: Kim James S.
Published: June 2007

The effects of a voluntary summer reading intervention were assessed in a randomized field trial involving 552 students in 10 schools. In this study, fourth-grade children received eight books to read during their summer vacation and were encouraged by their teachers to practice oral reading at home with a family member and to use comprehension strategies during independent, silent reading. Reading lessons occurred during the last month of school in June, and eight books were mailed to students biweekly during July and August. The estimated treatment effects on a standardized test of reading achievement (Iowa Test of Basic Skills) were largest for Black students (ES = .22), Latino students (ES = .14), less fluent readers (ES = .17), and students who reported owning fewer than 50 children’s books (ES = .13). The main findings suggest that a voluntary summer reading intervention may represent a scalable policy for improving reading achievement among lower performing students.
Street-Level Bureaucracy Revisited: Frontline District Central-Office Administrators as Boundary Spanners in Education Policy Implementation
Authors: Honig Meredith I.
Published: June 2007

The designation of district central-office administrators to operate as boundary spanners among the central office, schools, and community agencies can help with the implementation of challenging policy demands. However, educational research teaches little about central-office boundary spanners in practice. This article addresses that gap with findings from an embedded, comparative case study of boundary spanners in the implementation of collaborative education policy. The study’s conceptual framework draws on public management and sociological literature on boundary spanning and neo-institutional theories of decision making. Findings reveal that the boundary spanners in this case initially were particularly well suited to help with implementation in part because they brought non-traditional experiences to the central office. However, over time, many of the resources that aided them initially became liabilities that frustrated their work. This article documents the importance of examining boundary-spanning roles in implementation and suggests how central offices might provide supports to boundary spanners to increase their potential as levers of bureaucratic change.
Sample Size Estimation for Longitudinal Designs with Attrition: Comparing Time-Related Contrasts Between Two Groups
Authors: Hedeker Donald , Gibbons Robert D. , Waternaux Christine
Published: June 2007

Formulas for estimating sample sizes are presented to provide specified levels of power for tests of significance from a longitudinal design allowing for subject attrition. These formulas are derived for a comparison of two groups in terms of single degree-of-freedom contrasts of population means across the study timepoints. Contrasts of this type can often capture the main and interaction effects in a two-group repeated measures design. For example, a two-group comparison of either an average across time or a specific trend across time (e.g., linear or quadratic) can be considered. Since longitudinal data with attrition are often analyzed using an unbalanced repeated measures model (with a structured variance-covariance matrix for the repeated measures) or a random-effects model for incomplete longitudinal data, the variance-covariance matrix of the repeated measures is allowed to assume a variety of forms. Tables are presented listing sample size determinations assuming compound symmetry, a first-order autoregressive structure, and a non-stationary random-effects structure. Examples are provided to illustrate use of the formulas, and a computer program implementing the procedure is available from the first author.
Discrete-Time Discrete-State Latent Markov Models with Time-Constant and Time-Varying Covariates
Authors: Vermunt Jeroen K. , Langeheine Rolf , Bockenholt Ulf
Published: June 2007

Discrete-time discrete-state Markov chain models can be used to describe individual change in categorical variables. But when the observed states are subject to measurement error, the observed transitions between two points in time will be partially spurious. Latent Markov models make it possible to separate true change from measurement error The standard latent Markov model is, however, rather limited when the aim is to explain individual differences in the probability of occupying a particular state at a particular point in time. This paper presents a flexible logit regression approach which allows to regress the latent states occupied at the various points in time on both time- constant and time-varying covariates. The regression approach combines features of causal log-linear models and latent class models with explanatory variables. In an application pupils' interest in physics at different points in time is explained by the time-constant covariate sex and the time-varying covariate physics grade. Results of both the complete and partially observed data are presented.
Applications of Bayesian Decision Theory to Sequential Mastery Testing
Authors: Vos Hans J.
Published: June 2007

The purpose of this paper is to formulate optimal sequential rules for mastery tests. The framework for the approach is derived from Bayesian sequential decision theory. Both a threshold and linear loss structure are considered. The binomial probability distribution is adopted as the psychometric model involved. Conditions sufficient for sequentially setting optimal cutting scores are presented. Optimal sequential rules will be derived for the case of a subjective beta distribution representing prior true level of functioning. An empirical example of sequential mastery esting for concept-learning in medicine concludes the paper.
A Generalized Formula for the Mantel-Haenszel Differential Item Functioning Parameter
Authors: Roussos Louis A. , Schnipke Deborah L. , Pashley Peter J.
Published: June 2007

The present study derives a general formula for the population parameter being estimated by the Mantel-Haenszel (MH) differential item functioning (DIF) statistic. Because the formula is general, it is appropriate for either uniform DIF (defined as a difference in item response theory item difficulty values) or nonuniform DIF; and it can be used regardless of the form of the item response function. In the case of uniform DIF modeled with two-parameter-logistic response functions, the parameter is well known to be linearly related to the difference in item difficulty between the focal and reference groups. Even though this relationship is known to not strictly hold true in the case of three-parameter-logistic (3PL) uniform DIE the degree of the departure from this relationship has not been known and has been generally believed to be small By evaluating the MH DIF parameter, we show that for items of medium or high difficulty, the parameter is much smaller in absolute value than expected based on the difference in item difficulty between the two groups. These results shed new light on results from previous simulation studies that showed the MH DIF statistic has a tendency to shrink toward zero with increasing difficulty level when used with 3PL data.
Applications and Extensions of MCMC in IRT: Multiple Item Types, Missing Data, and Rated Responses
Authors: Patz Richard J. , Junker Brian W.
Published: June 2007

Patz and Junker (1999) describe a general Markov chain Monte Carlo (MCMC) strategy, based on Metropolis-Hastings sampling, for Bayesian inference in complex item response theory (IRT) settings. They demonstrate the basic methodology using the two-parameter logistic (2PL) model. In this paper we extend their basic MCMC methodology to address issues such as non-response, designed missingness, multiple raters, guessing behavior and partial credit (polytomous) test items. We apply the basic MCMC methodology to two examples from the National Assessment of Educational Progress 1992 Trial State Assessment in Reading: (a) a multiple item format (2PL, 3PL, and generalized partial credit) subtest with missing response data; and (b) a sequence of rated, dichotomous short-response items, using a new IRT model called the generalized linear logistic test model (GLLTM).
Methods for the Analysis of Pair-Matched Binary Data from School-based Intervention Studies
Authors: Vaughan Roger D. , Begg Melissa D.
Published: June 2007

In the evaluation of school-based intervention programs, students' knowledge, behavior, and attitudes about a particular issue are typically assessed before and after the intervention. The effectiveness of the intervention can then be gauged by comparing these "pre-treatment" and "post-treatment" responses. In the most rigorous evaluations, students are randomized to the intervention or control group. However, instead of randomizing individual students to treatment, most school-based studies rely on clustered randomization schemes. This is generally operationalized as the assignment of schools to treatment condition, although students within schools serve as the units of observation. Because there tends to be positive correlation between responses from students at the same school, the assumption of statistical independence is violated; hence, application of statistical tests that ignore this correlation can result in biased significance levels. This paper explores two statistical methods that account for this correlation in analyzing binary data. A proposal for adapting these methods for application to matched pairs data is presented. The performance of the methods is evaluated via simulation study.
Multidimensional Adaptive Testing with a Minimum Error-Variance Criterion
Authors: van der Linden Wim J.
Published: June 2007

Adaptive testing under a multidimensional logistic response model is addressed. An algorithm is proposed that minimizes the (asymptotic) variance of the maximum-likelihood estimator of a linear combination of abilities of interest. The criterion results in a closed-form expression that is easy to evaluate. In addition, it is shown how the algorithm can be modified if the interest is in a test with a "simple ability structure". The statistical properties of the adaptive ML estimator are demonstrated for a two-dimensional item pool with several linear combinations of the abilities.
Estimators of Random Effects Variance Components in Meta-Analysis
Authors: Friedman Lynn
Published: June 2007

In meta-analyses, groups of study effect sizes often do not fit the model of a single population with only sampling, or estimation, variance differentiating the estimates. If the effect sizes in a group of studies are not homogeneous, a random effects model should be calculated, and a variance component for the random effect estimated. This estimate can be made in several ways, but two closed form estimators are in common use. The comparative efficiency of the two is the focus of this report. We show here that these estimators vary in relative efficiency with the actual size of the random effects model variance component. The latter depends on the study effect sizes. The closed form estimators are linear functions of quadratic forms whose moments can be calculated according to a well-known theorem in linear models. We use this theorem to derive the variances of the estimators, and show that one of them is smaller when the random effects model variance is near zero; however, the variance of the other is smaller when the model variance is larger. This leads to conclusions about their relative efficiency.
Scheffé’s More Powerful F-Protected Post Hoc Procedure
Authors: Klockars Alan J. , Hancock Gregory R.
Published: June 2007

In 1970 Henry Scheffé proposed a more powerful version of his well known post hoc multiple comparison procedure, only to fail to recommend it by the paper's end. The point of the current paper is to bring this simple modification to a wider audience, complete with an original derivation, in hopes that the method will be embraced by researchers despite its creator's hesitations. Specifically, whereas Scheffé's original (1953) procedure advocates testing any exploratory post hoc contrast or comparison using a critical value assuming k - 1 between-group degrees of freedom, Scheffé's later modification (1970) will be demonstrated here showing that a more liberal critical value assuming k - 2 between-group degrees of freedom may be used if an omnibus null hypothesis across all means has been rejected. 
Rescuing Computerized Testing by Breaking Zipf's Law
Authors: Wainer Howard
Published: June 2007

For many reasons, both economic and psychometric, computerized tests are typically administered continuously throughout the year rather than at only a few dates spread out over the year. Continuous test administration can yield serious security problems, since examinees could remember some of the items administered and communicate them to other potential examinees. In this essay I argue that because of the nonlinear relationship between item usage and item security, this problem cannot be resolved merely by increasing the size of the item pool. I found that, with the then current methods of exposure control, in order for an item pool's security to increase linearly the size of the pool must increase exponentially. Such a strategy is not economically feasible. In addition to offering evidence of the failure of obtaining a secure item pool merely through increases in the item pool size I provide two alternative strategies with a much better chance of success.
A Comparison of Graded Response and Rasch Partial Credit Models with Subjective Well-Being
Authors: Baker John G. , Rounds James B. , Zevon Michael A.
Published: June 2007

Two multiple category item response theory models are compared using a data set of 52 mood terms with 713 subjects. Tellegen’s (1985) model of mood with two independent, unipolar dimensions of positive and negative affect provided a theoretical basis for the assumption of unidimensionality. Principle components analysis and item parameter tests supported the unidimensionality assumption. Comparative model data fit for the Samejima (1969) logistic model for graded responses and the Masters (1982) partial credit model favored the former model for this particular data set. Theoretical and practical aspects of the comparative application of multiple category models in the measurement of subjective well-being or mood are discussed.
Assessing Model Sensitivity of the Imputation Methods Used in the National Assessment of Educational Progress
Authors: Thomas Neal
Published: June 2007

The National Assessment of Educational Progress (NAEP) uses latent trait item response models to summarize performance of students on assessments of educational proficiency in different subject areas such as mathematics and reading. Because of limited examination time and concerns about student motivation. NAEP employs sparse matrix sampling designs that assign a small number of examination items to each sampled student to measure broad curriculums. As a consequence, each sampled student’s latent trait is not accurately measured, and NAEP uses multiple imputation missing data statistical methods to account for the uncertainty about the latent traits. The sensitivity of these model-based estimation and reporting procedures to statistical and psychometric assumptions is assessed. Estimation of the mean of the latent trait train different subpopulations was very robust to the modeling assumptions. Many of the other currently reported summaries, however; may depend on the modeling assumptions underlying the estimation procedures; these assumptions, motivated primarily by analytic tractability, are unlikely to attain, raising concerns about current reporting practices. The results indicate that more conservative criteria should be considered when forming intervals about estimates, and when assessing significance. A possible expansion of the imputation model is suggested that may improve its performance.
Closeness to Identity: A Conceptual Framework for the Definition and Interpretation of Correlations
Authors: Cahan Sorel
Published: June 2007

This paper introduces a way of understanding correlation coefficients based on the notion of scaled-score similarity (i.e., closeness to identity). It is suggested that the proposed framework has intuitive appeal. It points to the equivalence among various correlation coefficients by showing that correlation problems involve the notion of agreement between interval or ordinal measures expressed on different scales.
The Robustness of the Likelihood Ratio Chi-Square Test for Structural Equation Models: A Meta-Analysis
Authors: Powell Douglas A. , Schafer William D.
Published: June 2007

The robustness literature for the structural equation model was synthesized following the method of Harwell which employs meta-analysis as developed by Hedges and Vevea. The study focused on the explanation of empirical Type I error rates for six principal classes of estimators: two that assume multivariate normality (maximum likelihood and generalized least squares), elliptical estimators, two distribution-free estimators (asymptotic and others), and latent projection. Generally, the chi-square tests for overall model fit were found to be sensitive to non-normality and the size of the model for all estimators (with the possible exception of the elliptical estimators with respect to model size and the latent projection techniques with respect to non-normality). The asymptotic distribution-free (ADF) and latent projection techniques were also found to be sensitive to sample sizes. Distribution-free methods other than ADF showed, in general, much less sensitivity to all factors considered.
Variance Component Testing in Multilevel Models
Authors: Berkhof Johannes , Snijders Tom A. B.
Published: June 2007

Available variance component tests are reviewed and three new score tests are presented. In the first score test, the asymptotic normal distribution of the test statistic is used as a reference distribution. In the other two score tests, a Satterthwaite approximation is used for the null distribution of the test statistic. We evaluate the performance of the score tests and other available tests by means of a Monte Carlo study. The new tests are computationally relatively cheap and have good power properties.
Thresholds and Intransitivities in Pairwise Judgments: A Multilevel Analysis
Authors: Böckenholt Ulf
Published: June 2007

The method of paired comparisons belongs to a small class of models that simultaneously allow for consistency checks of the response behavior of a person and the estimation of item parameters. As a result, the paired comparison technique is of much importance when there are reasons to expect that respondents have difficulties appraising their values or preferences about an issue of interest. This article presents a hierarchical framework for the analysis of paired comparison data with three response categories that allow judges to be indifferent or undecided. The approach can be viewed as a stochastic representation of Luce’s (1956) semiorder. It facilitates statistical analyses of different types of inconsistent responses and yields new tests of the underlying judgmental process. An extensive analysis of a survey study illustrates the usefulness of a multilevel approach for modeling multiple trinary judgments.
The Rater Bundle Model
Authors: Wilson Mark , Hoskens Machteld
Published: June 2007

In this article an item response model is introduced for repeated ratings of student work, which we have called the Rater Bundle Model (RBM). Development of this model was motivated by the observation that when repeated ratings occur, the assumption of conditional independence is violated, and hence current state-of-the-art item response models, such as the rater facets model, that ignore this violation, underestimate measurement error, and overestimate reliability. In the rater bundle model these dependencies are explicitly parameterized. The model is applied to both real and simulated data to illustrate the approach.
Kappa as a Parameter of a Symmetry Model for Rater Agreement
Authors: Schuster Christof
Published: June 2007

If two raters assign targets to categories, the ratings can be arranged in a two-dimensional contingency table. A model for the frequencies in such a contingency table is presented for which Cohen’s kappa is a parameter. The model is based on two assumptions. First, the joint classification probabilities for the raters satisfy symmetry; second, the ratio of observed agreement to chance agreement is constant across categories. The model is illustrated using data from a study of the psychobiology of depression.