bluuebunny/crossref_metadata_2025 huggingface dataset:
The Child-Care and Preschool Workforce: Demographics, Earnings, and Unequal Distribution
Authors: Fuller Bruce , Strath Annelie
Published: June 2007

America's early education sector remains so radically decentralized—a far flung archipelago of pre-schools, family child-care homes, and subsidized individuals providing services-that basic information on local organizations and staff members remains scarce. This, despite rising policy interest in, and skyrocketing appropriations for, preschool programs which are aimed at boosting children's school readiness. Working from a social ecology framework, this study aimed to learn more about local populations of early education organizations. This paper uses 1990 household census data aggregated to the zip-code level to report on features of the early education workforce nationwide. Teachers and other staff in preschools and center-based programs reported low wages, averaging about $7,300 per year ($10,700 in 2000 dollars), with most working less than full time. The median center-based teacher was 34 years of age, reported having completed some college, and was married. The median worker in family child-care homes earned even less and only had a high school diploma. About 15% of all preschool teachers in urban areas were African-American; about 8% were Latina. Twice as many preschool and center teachers per 1,000 young children resided in affluent zip codes, relative to poor and lower middle-class areas. Preschools and centers located in blue-collar and middle-income zip codes displayed the lowest level of organizational formalization, compared to those operating in poor or affluent areas. We discuss the utility of 2000 census data to assess inequalities in the supply and quality of early education organizations and their staff, and modeling how economic and policy forces may shape organizational variability.
Evaluating the Evidence on Teacher Certification: A Rejoinder
Authors: Goldhaber Dan D. , Brewer Dominic J.
Published: June 2007

In "Does Teacher Certification Matter? Evaluating the Evidence " (appearing in this issue of Educational Evaluation and Policy Analysis ), Linda Darling-Hammond, Barnett Berry, and Amy Thoreson, comment on our earlier piece, "Does Teacher Certification Matter? High School Teacher Certification Status and Student Achievement" (Educational Evaluation and Policy Analysis, Summer 2000). Unfortunately, the Darling-Hammond et al. piece does not accurately portray our original article, which is one of the first studies of the effects of teacher certification on student achievement that uses a national student level dataset in a multivariate framework. Here we discuss the portrayal of our paper, and argue that there are important policy questions about certification, and the teacher labor market in general, that ought to be addressed by researchers in a dispassionate way that focuses on rigorous, high quality empirical findings. As yet we do not believe there is enough information to draw strong conclusions about the impacts of certification on the teacher applicant pool or about the overall level of quality of the teacher workforce. Thus, it is important to ask questions about these issues so that research can be used to guide the development of sound public policy.
Teacher Aides and Students’ Academic Achievement
Authors: Gerber Susan B. , Finn Jeremy D. , Achilles Charles M. , Boyd-Zaharias Jayne
Published: June 2007

Despite more than 600,000 teacher aides in American schools today, research provides little information about their classroom activities, their qualifications for carrying out their duties, or their impact on student achievement and behavior. This investigation asked whether the presence of a teacher aide in the classroom has any noticeable impact on pupils’ learning. Three primary questions were addressed: (1) In Grades K through 3, does the presence of a full-time teacher aide in the classroom affect students’ academic achievement? (2) If teacher aides have a positive effect on students’ performance, does the effect depend on the number of years the student attends classes with a teacher aide? (3) Do some functions of aides (i.e., clerical tasks, instructional tasks, noninstructional tasks) have a greater impact on student achievement than others? This investigation showed that the teacher aide movement in the United States has created a state of affairs that requires many aides to perform tasks for which they are ill-prepared. In addition, teacher aide data were analyzed from Tennessee’s Project STAR, a longitudinal experiment in which students were assigned at random to small classes, regular-size classes without an aide, or regular-size classes with a full-time teacher aide. The analyses reported here extend previous investigations, examining the functions and effects of teacher aides in depth. The results showed that teacher aides have little, if any, positive effect on students’ academic achievement. The only positive effect was an improvement in reading scores for students who attended a class with a teacher aide for 2 or 3 years. These results were the only exceptions to a plethora of negative findings. The study also showed that the types of duties aides performed had no bearing on student achievement. Because teacher aides are called upon increasingly to provide instruction to pupils, policies and research must help us select and prepare aides to perform effectively.
Schools, Achievement, and Inequality: A Seasonal Perspective
Authors: Alexander Karl L. , Entwisle Doris R. , Olson Linda S.
Published: June 2007

Are there socioeconomic differences in the seasonality of children’s learning over the school year and summer months? The achievement gap across social lines increases during the primary grades, as much research indicates, but descriptive analyses and HLM within-person growth models for a representative panel of Baltimore school children demonstrate that the increase can be traced mainly to the out-of-school environment (i.e., influences situated in home and community). School-year verbal and quantitative achievement gains are comparable for upper socioeconomic status (SES) and lower SES children, but summer gains, when children are out of school, evidence large disparities. During the summer, upper SES children’s skills continue to advance (albeit at a slower rate than during the school year), but lower SES children’s gains, on average, are flat. This seasonal pattern of achievement gains implies that schooling plays an important compensatory role, one that is obscured when achievement is compared on an annual basis, as is typical. Policy implications of the seasonality of learning are discussed, including support for preventive measures over the preschool years and for programs, possibly including calendar reforms and summer school, to support disadvantaged children’s learning year-round.
The Grasshopper and the Ant: Motivational Responses of Low-Achieving Students to High-Stakes Testing
Authors: Roderick Melissa , Engel Mimi
Published: June 2007

Advocates of high-stakes testing argue that providing strong incentives for students to take learning more seriously will result in greater student effort and motivation. Opponents argue that these policies set up low-achieving students to fail, looking to research on motivation for evidence that extrinsic and negative incentives such as the threat of retention will undermine students' engagement in school. This article seeks to evaluate these claims by examining the responses of 102 low-achieving sixth- and eighth-grade students to Chicago's highly publicized effort to end social promotion. Does this policy lead students to work harder? If so, to what extent does hard work pay off? The majority of these students described increased work effort under the policy. They reported greater attention to class work, increased academic press and support from teachers, and more time spent studying outside school. These efforts were confirmed by teachers' reports. Students with high levels of work effort generally had greater-than-average learning gains and positive promotional outcomes. Approximately one third, however, showed little work effort despite a desire not to be retained. These students faced significantly larger skill gaps and barriers to learning both within and outside school than did their peers with high work effort. How teachers manage high-stakes testing policies—whether they create environments that make low-achieving students feel supported and efficacious in responding to new demands and whether they direct students' efforts in productive ways—has an important impact on student motivation and passing rates.
Direct and Indirect Writing Assessments: Examining Issues of Equity and Utility
Authors: Heck Ronald H. , Crislip Marian
Published: June 2007

Performance tests are increasingly used as alternatives to, or in connection with, standardized multiple-choice tests as a means of assessing student learning and school accountability. Besides their proposed equity advantages over multiple-choice tests in measuring student learning across groups of students, performance assessments have also been viewed as having greater utility for monitoring school progress because of their proposed closer correspondence to the curriculum that is actually taught. We examined these assumptions by comparing third-grade student performance on a performance-based writing test and a multiple-choice test of language skills. We observed smaller differences in achievement on the writing performance assessment for some groups of students (e.g., low socioeconomic status, various ethnic backgrounds) than are commonly observed on multiple-choice tests. Girls, however, had higher mean scores than boys on both types of assessments. Moreover, the school's identification and commitment over time to improving its students' writing skills positively related to its students' outcomes on the writing performance test. Overall, our examination of performance-based writing assessment is encouraging with respect to providing a relatively fair assessment and measuring learning tasks that are related to the school's curricular practices.
Instructional Program Coherence: What It Is and Why It Should Guide School Improvement Policy
Authors: Newmann Fred M. , Smith BetsAnn , Allensworth Elaine , Bryk Anthony S.
Published: June 2007

We present the concept of instructional program coherence and explain why school improvement frameworks that incorporate instructional program coherence are more likely to advance student achievement than multiple, unrelated efforts. We present evidence that Chicago elementary schools with stronger instructional program coherence make higher gains in student achievement. We also share observations on how, in specific schools, principals and external partners directed key school resources toward the development of instructional program coherence. In closing, we discuss factors within the educational system that discourage instructional program coherence and suggest ways that school leaders, school improvement partners, and policymakers can support greater instructional program coherence.
Who Takes What Math and in Which Track? Using TIMSS to Characterize U.S. Students’ Eighth-Grade Mathematics Learning Opportunities
Authors: Cogan Leland S. , Schmidt William H. , Wiley David E.
Published: June 2007

This article examines the range of eighth-grade mathematics learning opportunities in the United States, drawing on data gathered for the Third International Mathematics and Science Study (TIMSS). Sources of variation in the provision of learning opportunities are identified, and patterns in eighth-grade mathematics course offerings are compared across schools. Comparison of students’ learning opportunities includes consideration of the specific course in which they were enrolled, the type of textbook employed for the course, and the proportion of time teachers devoted to teaching specific topics. Analyses revealed a mismatch between the mathematics course title and the textbook employed in the course for nearly 30% of U.S. eighth-grade students. Course-textbook combinations demonstrated significant relationships with the time teachers devoted to specific topics and the international topic difficulty score. Some differences in mathematics learning opportunities were found on the basis of a school’s location (urban, rural, suburban), size, and percentage of minority enrollment. We contend that the significant relationships found were not consistent enough to provide a satisfactory explanation for the observed variation and that individual student differences, which often provide an implicit rationale for tracking, also represent an inadequate explanation for the observed diversity in curricular opportunities. We conclude that recognizing the multiple definitions for students’ classroom mathematics learning experiences is an important step in reform and policy discussions.
Peer Review in the Funding of Research in Higher Education: The Australian Experience
Authors: Jayasinghe Upali W. , Marsh Herbert W. , Bond Nigel
Published: June 2007

In this article we evaluate the peer review process used to fund Australian university research across all disciplines. Peer reviews of research proposals (2,989 proposals, 6,233 external reviewers) submitted to the Australian Research Council (ARC) are related to characteristics of the researchers and of external reviewers. The reliability of the peer reviews was disappointingly low (interrater agreement of .53 for researcher ratings based on an average of 4.3 external reviewers per proposal). The gender and age of a researcher and the number of researchers on a research team did not affect the probability that funding would be granted, but professors were more likely to be funded than nonprofessors. Australian external reviewers gave lower ratings than did non-Australian reviewers, particularly those from North America. The number of external reviewers for each proposal and the number of proposals assessed by each external reviewer had small negative effects on ratings. Researcher-nominated external reviewers (those chosen by the authors of a research proposal) gave higher, less-reliable ratings than did panel-nominated external reviewers chosen by the ARC. To improve the reliability of peer reviews, we offer the following recommendations: (a) Researcher-nominated reviewers should not be used; (b) there should be more reviews per proposal; and (c) a smaller number of more highly selected reviewers should perform most of the reviews within each subdiscipline, thereby providing greater control over error associated with individual reviewers.
Mathematics and Sciences Course Taking Among Arab Students in Israel: A Case of Unexpected Gender Equality
Authors: Ayalon Hanna
Published: June 2007

Using multilevel analysis on a sample of academic-track 12th graders in 1989, this article compares gender inequality in course taking of mathematics and sciences in Arab and Jewish high schools. The findings show that gender inequality is almost nonexistent in Arab schools, whereas it is very prominent in Jewish, particularly secular, schools. The differences in gender inequality are explained by the differences in the curriculum. In contrast with Jewish high schools that offer a rich curriculum, Arab schools offer few advanced courses in humanities and social sciences. Consequently, male and female students who wish to take advanced courses have to choose mathematics and sciences. The findings point to a sociological paradox; the poor curriculum of Arab high schools enhances the chances of Arab female students to be exposed to highly valued knowledge. These findings shed an additional light on the beneficial effect that a restricted curriculum can have for members of disadvantaged groups.
A Comprehensive Framework for Evaluating Educational Vouchers
Authors: Levin Henry M.
Published: June 2007

Major policy debates have arisen around the subject of educational vouchers as an alternative for financing and organizing elementary and secondary education. To a large degree, comparisons between vouchers and the traditional system of educational finance and school operations have been limited to one or two dimensions of education such as the relative impact of a particular system on achievement test scores. This article describes a comprehensive, evaluative framework that draws upon a larger range of goals that have been posed for education in a democratic and free society. These criteria include: (a) freedom of choice, (b) productive efficiency, (c) equity, and (d) social cohesion. The framework demonstrates the importance of and tradeoffs among these four criteria in evaluating specific educational voucher plans and enables comparisons with other alternatives such as charter schools as well as the more traditional public school arrangements.
Do Low-Achieving Students Benefit More from Small Classes? Evidence from the Tennessee Class Size Experiment
Authors: Nye Barbara , Hedges Larry V. , Konstantopoulos Spyros
Published: June 2007

Recent evidence about the effects of class size on academic achievement from randomized experiments points to positive effects of small classes. However, the evidence about the mechanism producing these effects is less clear. Some scholars have argued for mechanisms that would imply greater effects of small classes for low-achieving students. This article investigates possible differential effects of small classes on achievement using data from Project STAR, a four-year, large-scale randomized experiment on the effects of class size. We examined the differential effects of small classes for students in the bottom half and bottom quarter, respectively, of their class's achievement distribution in kindergarten. Although small class effects are somewhat larger for low-achieving students in reading, the differential effects (interactions) are not statistically significant. Moreover, the small class effects for low-achieving students in mathematics are actually smaller than those for higher achieving students. Thus while there are unambiguous positive effects of small classes on achievement, there is no evidence for differentially larger effects of small classes for lower achieving students.
Performance, Graduation, and Transfer of Immigrants and Natives in City University of New York Community Colleges
Authors: Bailey Thomas , Weininger Elliot B.
Published: June 2007

The City University of New York (CUNY) plays an essential role in educating the burgeoning immigrant population of New York City. During the 1990s, the foreign-born share of CUNY’s undergraduate population rose from one third to almost one half. Nevertheless, little research has been carried out on this population. Focusing on foreign-born and native minority community college entrants, we compare these groups in terms of the number of credits earned and the likelihood of transferring to a four-year program and of completing an associate degree and a bachelor’s degree. We find that nativity, race, and ethnicity are all related to these outcomes. Moreover, whether a foreign-born student attended high school in the United States or abroad is an important determinant of educational outcomes.
Relational Networks, Social Trust, and Norms: A Social Capital Perspective on Students’ Chances of Academic Success
Authors: Goddard Roger D.
Published: June 2007

This study elaborates a theoretical rationale for relational networks, norms, and trust as structural and functional forms of social capital that can facilitate student achievement. The results of hierarchical generalized linear modeling show that 4th-grade students’ odds of passing state-mandated mathematics and writing assessments are modestly increased in urban schools characterized by high levels of social capital. The results suggest the need for more research investigating the extent to which social capital is independent from socioeconomic status and whether social capital may be developed in schools serving high concentrations of poor and minority students.
Does Competition Improve Teaching and Learning? Evidence from New Zealand
Authors: Ladd Helen F. , Fiske Edward B.
Published: June 2007

Central to the argument for more competition in education is that it will induce schools to provide higher quality education at no greater cost. This article sheds new light on this issue by measuring how competition among New Zealand’s schools affected student learning as perceived by teachers and principals. The analysis builds on the fact that New Zealand’s introduction of full parental choice in 1992 increased competitive pressures more for some schools than for others. With careful attention to various potential threats to validity, we conclude that competition—as perceived by teachers—generated negative effects on the quality of student learning and other aspects of schooling in New Zealand’s elementary schools.
The New Middle Management: Intermediary Organizations in Education Policy Implementation
Authors: Honig Meredith I.
Published: June 2007

Intermediary organizations have become increasingly prominent participants in education policy implementation despite limited knowledge about their distinctive functions and the conditions that constrain and enable those functions. This article addresses that research-practice gap by drawing on theories of organizational ecology and findings from a comparative case study of four intermediary organizations that helped with collaborative policy implementation in Oakland, California. I define intermediaries as organizations that operate between policymakers and implementers to affect changes in roles and practices for both parties and show that such organizations typically vary along at least five dimensions. Oakland’s intermediary organizations all provided new implementation resources—knowledge, political/social ties, and an administrative infrastructure—but faced different constraining and enabling conditions. Using insights from this strategic case study, this article begins to build theory about intermediary organizations as important participants in contemporary policy implementation.
District Policy and Beginning Teachers: A Lens on Teacher Learning
Authors: Grossman Pamela , Thompson Clarissa
Published: June 2007

This analysis considers the role district policy environments play in the lives of beginning teachers. As part of a larger longitudinal study of teacher learning in the language arts, the authors analyzed the experiences of three first-year teachers in two contrasting school districts. This article assesses the role of policies concerning curriculum, professional development, and mentoring in teachers’ opportunities in learning to teach language arts. The ways in which districts were organized had consequences for what these beginning teachers learned about teaching; district structures either encouraged or deflected conversations about teaching English. In addition, the authors found that districts served powerful roles as teacher educators. The tasks the districts assigned the teachers, the resources they provided, the learning environments they created, and the conversations they provoked proved to be consequential in shaping both teachers’ concerns and their opportunities for learning about teaching language arts.
Immigrants and the Distribution of Resources Within an Urban School District
Authors: Schwartz Amy Ellen , Stiefel Leanna
Published: June 2007

In New York City, where almost 14% of elementary school pupils are foreign-born and roughly half of these are “recent immigrants,” the impact of immigrant students on school resources may be important. While immigrant advocates worry about inequitable treatment of immigrant students, others worry that immigrants drain resources from native-born students. In this article, we explore the variation in school resources and the relationship to the representation of immigrant students. To what extent are variations in school resources explained by the presence of immigrants per se rather than by differences in student educational needs, such as poverty or language skills, or differences in other characteristics, such as race? Our results indicate that, while schools resources decrease with the representation of immigrants, this relationship largely reflects differences in the educational needs of immigrant students. Although analyses that link resources to the representation of foreign-born students in 12 geographic regions of origin find some disparities, these are again largely driven by differences in educational need. Finally, we find that some resources increase over time when there are large increases in the percentage of immigrants in a school, but these results are less precisely estimated. Thus, elementary schools appear not to be biased either against or for immigrants per se, although differences in the needs of particular groups of immigrant students may lead to more (or fewer) school resources.
High School Exit Examinations and High School Completion: Evidence From the Early 1990s
Authors: Warren John Robert , Edwards Melanie R.
Published: June 2007

For more than two decades, many states have required students to pass exit examinations to obtain high school diplomas. Despite these sweeping policy initiatives, we know relatively little about whether such policies are related to the chances that high school students will obtain diplomas. The authors estimate the association between high school exit examination requirements and students’ chances of obtaining diplomas, acquiring general educational development (GED), or leaving school with neither credential. They also assess whether these associations vary according to socioeconomic status (SES) or prior academic achievement. Using data from the National Education Longitudinal Study of 1988, they found that high school exit examination requirements—at least as they existed in the early 1990s—are not associated with increased chances of obtaining a GED or of leaving school with neither a GED nor a diploma, even among low-SES and low-achieving students.
The Effects of Affirmative Action Programs: Evidence From the University of California at San Diego
Authors: Rose Heather
Published: June 2007

Using administrative data from the University of California at San Diego, the author explicitly identifies and studies students admitted under affirmative action programs. On average, these students earned grade point averages (GPAs) 0.30 points lower than those of nonaffirmative students. The difference in graduation rates is larger, with 57% of affirmative action students graduating compared to 73% of their nonaffirmative action peers. When compared to students just above the regular admissions cutoff, the differences are smaller—the difference in graduation rates is only 8 percentage points, and the difference in GPAs is only 0.20 points. A student’s family, school, and neighborhood characteristics can explain a small part of these differences, but academic preparation explains most of the difference.
Dropout Rates After High-Stakes Testing in Elementary School: A Study of the Contradictory Effects of Chicago’s Efforts to End Social Promotion
Authors: Allensworth Elaine M.
Published: June 2007

Across the country, grade promotion is tied increasingly to performance on standardized tests. One concern about such policies is that they might increase dropout rates. Policy proponents counter that adverse effects of grade retention should be more than offset by beneficial effects from rising achievement. Using data from Chicago, this study examines dropout rates after implementation of an eighth-grade promotion standard. The results indicate that retention by the policy did have adverse effects on dropping out, but the relationship was smaller than seen with traditional teacher-initiated retention and was unrelated to the timing of dropping out. Systemwide, slight decreases in dropout rates among the 90% of students who were not retained counterbalanced the higher dropout rates among those retained.
Longitudinal Achievement Effects of Multiyear Summer School: Evidence From the Teach Baltimore Randomized Field Trial
Authors: Borman Geoffrey D. , Dowling N. Maritza
Published: June 2007

Employing a randomized field trial, this 3-year study explored the effects of a multiyear summer school program in preventing the cumulative effect of summer learning losses and promoting longitudinal achievement growth, for a total treatment group of 438 students from high-poverty schools. Longitudinal outcomes for the participants were contrasted to those for 248 children randomized into a no-treatment control condition. Multilevel growth models revealed no intention-to-treat effects of assignment to the multiyear summer school program. However, student attendance patterns at the voluntary program were variable across the 3 years that the intervention was offered. Maximum likelihood mixture models, which estimated the effects of the treatment for compliers, revealed statistically significant effects on learning across all three literacy domains tested for those students who attended the Summer Academy at an above average rate across two or more of the three summers that it was offered. Relative to their control-group counterparts, treatment compliers held advantages of 40% to 50% of one grade level on the final posttests.
Using Structured Classroom Vignettes to Measure Instructional Practices in Mathematics
Authors: Stecher Brian , Le Vi-Nhuan , Hamilton Laura , Ryan Gery , Robyn Abby , Lockwood J. R.
Published: June 2007

Large-scale educational studies frequently require accurate descriptions of classroom practices to judge implementation and impact. However, it can be difficult to obtain these descriptions in a timely, efficient manner. To address this problem, the authors developed a vignette-based measure of one aspect of mathematics instructional practice, reform-oriented instruction. Teachers read contextualized descriptions of teaching practices that varied in terms of reform-oriented instruction, and rated the degree to which the options corresponded to their own likely behaviors. Responses from 80 fourth-grade teachers yielded fairly consistent responses across two parallel vignettes and moderate correlations with other scales of reform-oriented instruction derived from classroom observations, surveys, and logs. The results suggested that the vignettes measure important aspects of reform-oriented instruction that are not captured by other measurement methods. Based on this work, it appears that vignettes can be a useful tool for research on instructional practice, but cognitive interviews with participating teachers provided insight into possible improvements to the items.
Multivariate Tests for Correlated Data in Completely Randomized Designs
Authors: Mielke Paul W. , Berry Kenneth J.
Published: June 2007

Power comparisons are provided for three permutation tests and the Bartlett-Nanda-Pillai trace test in completely randomized experimental designs with correlated multivariate dependent variables. Five bivariate distributions, five levels of correlation, equal and unequal sample sizes, two variations of location shifts, and two types of commensuration are considered in the power analyses. With very few exceptions, the power of the Bartlett-Nanda-Pillai trace test is found to be less than the corresponding power for at least one of the permutation tests.
Approximations of Normal IRT Models for Change
Authors: Tan E.S. , Ambergen A.W. , Does R.J.M.M. , Imbos Tj.
Published: June 2007

In this paper the one parameter Item Response Theory (IRT) model with normal Item Characteristic Curves (ICC) in longitudinal context has been studied. The abilities are structured according to a general mixed effects linear regression model. The items are supposed to be a sample from a large bank of items with constant mean difficulty. If the number of repeated measures is large, then commonly used simultaneous estimation procedures often lead to practical problems with respect to multidimensional numerical integrations. In this article, an approximation of the normal ICC is introduced that leads to simple ability and difficulty estimators with nice asymptotic properties. The relative efficiency and bias of the ability estimator are studied. An illustration with real data shows high relative efficiency within an acceptable range of the domain of the ICC. Moreover, the bias is very small. A simulation study shows the effect of non-normal item parameters on the regression estimates. The results suggest that the proposed procedure is rather robust against departures from normality. However, the estimation of the correlations between regression parameters can be seriously biased.
Estimation of Unattenuated Factor Loadings
Authors: Woodward Todd S. , Hunter Michael A.
Published: June 2007

It is demonstrated that traditional exploratory factor analytic methods, when applied to correlation matrices, cannot be used to estimate unattenuated factor loadings. However, these values can be accurately estimated when the disattenuated correlation matrix, or the covariance matrix, is used as input. A mathematical basis for the advantage of this application of factor analysis is presented in this paper, as is an explanation of how these equations apply differentially to common factor analysis (CFA) and principal component analysis (PCA). Graphic displays which describe the comparative performance of CFA and PCA when extracting factors from the correlation matrix, the covariance matrix, and the disattenuated correlation matrix are provided. It is concluded that although the most accurate estimates of the unattenuated factor loadings can be achieved when CFA is used to decompose the covariance matrix or the disattenuated correlation matrix as the percentage of measurement error decreases, and the number of indicators per factor increases, the impact of methodology choice diminishes.
Synthesizing Results from the Trial State Assessment
Authors: Raudenbush Stephen W. , Fotiu Randall P. , Cheong Yuk Fai
Published: June 2007

Using data collected under the Trial State Assessment (TSA) of the National Assessment of Educational Progress (NAEP), this article describes and illustrates a two-stage statistical model for investigating state-to-state variation in mathematics achievement. At the first stage, within each state, a two-level hierarchical linear model is estimated via maximum likelihood. At the second stage, results are combined across states using Bayesian estimation implemented via the Gibbs sampler. The results reveal considerable state-to-state heterogeneity in mathematics proficiency, but most heterogeneity is explainable on the basis of covariates defined on students, teachers, and schools. The findings suggest that interest in state comparisons might productively focus on state differences in policy-relevant correlates of proficiency rather than on state differences in mean proficiency. The analytical approach can be applied in other cases where data are dense at the lower level of a hierarchy but thin at the higher level.
Using Loss Functions for DIF Detection: An Empirical Bayes Approach
Authors: Zwick Rebecca , Thayer Dorothy T. , Lewis Charles
Published: June 2007

We investigated a DIF flagging method based on loss functions. The approach builds on earlier research that involved the development of an empirical Bayes (EB) enhancement to Mantel-Haenszel (MH) DIF analysis. The posterior distribution of DIF parameters was estimated and used to obtain the posterior expected loss for the proposed approach and for competing classification rules. Under reasonable assumptions about the relative seriousness of Type I and Type II errors, the loss-function-based DIF detection rule was found to perform better than the commonly used "A, " "B, " and "C" DIF classification system, especially in small samples.
Generalized Appended Product Indicator Procedure for Nonlinear Structural Equation Analysis
Authors: Wall Melanie M. , Amemiya Yasuo
Published: June 2007

Interest in considering nonlinear structural equation models is well documented in the behavioral and social sciences as well as in the education and marketing literature. This article considers estimation of polynomial structural models. An existing method is shown to have a limitation that the produced estimator is inconsistent for most practical situations. A new procedure is introduced and defined for a general model using products of observed indicators. The resulting estimator is consistent without assuming any distributional form for the underlying factors or errors. Identification assessment and standard error estimation are discussed. A simulation study addresses statistical issues including comparisons of discrepancy functions and the choice of appended product indicators. Application of the new procedure in a substance abuse prevention study is also reported.
Detecting Nonlinear Associations, Plus Comments on Testing Hypotheses About the Correlation Coefficient
Authors: Wilcox Rand R.
Published: June 2007

Let (Yi,Xi ), i = 1, . . . , n, be a random sample from some p + 1 variate distribution where Xi is a vector of length p. In the social sciences, the most common strategy for detecting an association between Y and the marginal distributions is to test the hypothesis that the corresponding correlations are zero using a standard Student’s t test. There are two practical problems with this strategy. First, for reasons described in the article, there are situations where the correlation between two random variables is zero, but Student’s t test is not even asymptotically correct. In fact, the probability of rejecting can approach one as the sample size gets large, even though the hypothesis of a zero correlation is true. Of course, one can also apply standard methods based on a linear regression model and the least squares estimator, but the same practical problems arise. Second, Student’s t test can miss nonlinear associations. This latter problem is the main motivation for this article. Results of a former study suggest an approach that avoids both of the difficulties just described. Based on simulations, it is found that the Cramér-von Mises form of the test statistic is generally better than the Kolmogorov-Smirnov form. Situations arise where this method has less power than Student’s t test, but this is due in part to t test’s use of an incorrect estimate of the standard error.
Outlier Measures and Norming Methods for Computerized Adaptive Tests
Authors: Bradlow Eric T. , Weiss Robert E.
Published: June 2007

The problem of identifying outliers has two important aspects: the choice of outlier measures and the method to assess the degree of outlyingness (norming) of those measures. Several classes of measures for identifying outliers in Computerized Adaptive Tests (CATs) are introduced. Some of these measures are new and are constructed to take advantage of CATs’ sequential choice of items; other measures are taken directly from paper and pencil (P&P) tests and are used for baseline comparisons. Assessing the degree of outlyingness of CAT responses, however, can not be applied directly from P&P tests because stopping rules associated with CATs yield examinee responses of varying lengths. Standard outlier measures are highly correlated with the varying lengths which makes comparison across examinees impossible. Therefore, four methods are presented and compared which map outlier statistics to a familiar probability scale (a p value). The application of these methods to CAT data is new. The methods are explored in the context of CAT data from a 1995 Nationally Administered Computerized Examination (NACE).
Bayesian Inference with Probability Matrix Decomposition Models
Authors: Meulders Michel , De Boeck Paul , Van Mechelen Iven , Gelman Andrew , Maris Eric
Published: June 2007

Probability Matrix Decomposition models may be used to model observed binary associations between two sets of elements. More specifically, to explain observed associations between two elements, it is assumed that B latent Bernoulli variables are realized for each element and that these variables are subsequently mapped into an observed data point according to a prespecified deterministic rule. In this paper, we present a fully Bayesian analysis for the PMD model making use of the Gibbs sampler. This approach is shown to yield three distinct advantages: (a) in addition to posterior mean estimates it yields (1 — α)% posterior intervals for the parameters, (b) it allows for an investigation of hypothesized indeterminacies in the model's parameters and for the visualization of the best possible reduction of the posterior distribution in a low-dimensional space, and (c) it allows for a broad range of goodness-of-fit tests, making use of the technique of posterior predictive checks. To illustrate the approach, we applied the PMD model to opinions of respondents of different countries concerning the possibility of contracting AIDS in a specific situation.
Nonparametric Item Response Function Estimates with the EM Algorithm
Authors: Rossi Natasha , Wang Xiaohui , Ramsay James O.
Published: June 2007

The methods of functional data analysis are used to estimate item response functions (IRFs) nonparametrically. The EM algorithm is used to maximize the penalized marginal likelihood of the data. The penalty controls the smoothness of the estimated IRFs, and is chosen so that, as the penalty is increased, the estimates converge to shapes closely represented by the three-parameter logistic family. The one-dimensional latent trait model is recast as a problem of estimating a space curve or manifold, and, expressed in this way, the model no longer involves any latent constructs, and is invariant with respect to choice of latent variable. Some results from differential geometry are used to develop a data-anchored measure of ability and a new technique for assessing item discriminability. Functional data-analytic techniques are used to explore the functional variation in the estimated IRFs. Applications involving simulated and actual data are included.
Cross-Classification Multilevel Logistic Models in Psychometrics
Authors: Van den Noortgate Wim , De Boeck Paul , Meulders Michel
Published: June 2007

In IRT models, responses are explained on the basis of person and item effects. Person effects are usually defined as a random sample from a population distribution. Regular IRT models therefore can be formulated as multilevel models, including a within-person part and a between-person part. In a similar way, the effects of the items can be studied as random parameters, yielding multilevel models with a within-item part and a between-item part. The combination of a multilevel model with random person effects and one with random item effects leads to a cross-classification multilevel model, which can be of interest for IRT applications. The use of cross-classification multilevel logistic models will be illustrated with an educational measurement application.
Interactive Profiler: An Intuitive, Web-Based Statistical Application in Visualizing Educational and Marketing Databases
Authors: Ip Edward H. , Leung Phillip , Johnson Joseph
Published: June 2007

We describe the design and implementation of a web-based statistical program—the Interactive Profiler (IP). The prototypical program, developed in Java, was motivated by the need for the general public to query against data collected from the National Assessment of Educational Progress (NAEP), a large-scale US survey of the academic state of American students. The emphasis of the program requirements is on bringing the NAEP data to a broader and not necessarily technically prepared audience. We show that the IP is an intuitive tool for visualizing students’ profiles. Because the same principles for visualization apply to other databases, in this article we also demonstrate how IP can be used to visualize a large marketing database. Besides NAEP, our statistical visualization tool should be pertinent to other federally maintained databases and large-scale marketing databases.
Evaluation of the CATSIB DIF Procedure in a Pretest Setting
Authors: Nandakumar Ratna , Roussos Louis
Published: June 2007

A new procedure, CATSIB, for assessing differential item functioning (DIF) on computerized adaptive tests (CATs) is proposed. CATSIB, a modified SIBTEST procedure, matches test takers on estimated ability and controls for impact-induced Type 1 error inflation by employing a CAT version of the SIBTEST “regression correction.” The performance of CATSIB in terms of detection of DIF in pretest items was evaluated in a simulation study. Simulated test takers were adaptively administered 25 operational items from a pool of 1,000 and were linearly administered 16 pretest items that were evaluated for DIF. Sample size varied from 250 to 500 in each group. Simulated impact levels ranged from a 0- to 1-standard-deviation difference in mean ability levels. The results showed that CATSIB with the regression correction displayed good control over Type 1 error, whereas CATSIB without the regression correction displayed impact-induced Type 1 error inflation. With 500 test takers in each group, power rates were exceptionally high (84% to 99%) for values of DIF at the boundary between moderate and large DIF. For smaller samples of 250 test takers in each group, the corresponding power rates ranged from 47% to 95%. In addition, in all cases, CATSIB was very accurate in estimating the true values of DIF, displaying at most only minor estimation bias.
Asymptotic Effect of Misspecification in the Random Part of the Multilevel Model
Authors: Berkhof Johannes , Kampen Jarl Kennard
Published: June 2007

The authors examine the asymptotic effect of omitting a random coefficient in the multilevel model and derive expressions for the change in (a) the variance components estimator and (b) the estimated variance of the fixed effects estimator. They apply the method of moments, which yields a closed form expression for the omission effect. In practice, the model parameters are estimated by maximum likelihood; however, since the moment estimator and the maximum likelihood estimator are both consistent, the presented expression for the change in the variance components estimator asymptotically holds for the maximum likelihood estimator as well. The results are illustrated with an analysis of mathematics performance data.
Teaching Statistical Inference for Causal Effects in Experiments and Observational Studies
Authors: Rubin Donald B.
Published: June 2007

Inference for causal effects is a critical activity in many branches of science and public policy. The field of statistics is the one field most suited to address such problems, whether from designed experiments or observational studies. Consequently, it is arguably essential that departments of statistics teach courses in causal inference to both graduate and undergraduate students. This article discusses an outline of such courses based on repeated experience over more than a decade.
Interpreting Regression Diagnostics
Authors: Karakostas KX
Published: June 2007

This article presents a technique that will help teachers, researchers, and other people who use the linear regression models, especially those in education and social sciences, to understand and interpret the residuals graphics better.
Maximum Likelihood Analysis of a Two-Level Nonlinear Structural Equation Model With Fixed Covariates
Authors: Lee Sik-Yum , Song Xin-Yuan
Published: June 2007

In this article, a maximum likelihood (ML) approach for analyzing a rather general two-level structural equation model is developed for hierarchically structured data that are very common in educational and/or behavioral research. The proposed two-level model can accommodate nonlinear causal relations among latent variables as well as effects of fixed covariate in its various components. Methods for computing the ML estimates, and the Bayesian information criterion (BIC) for model comparison are established on the basis of powerful tools in statistical computing such as the Monte Carlo EM algorithm, Gibbs sampler, Metropolis–Hastings algorithm, conditional maximization, bridge sampling, and path sampling. The newly developed procedures are illustrated by results obtained from a simulation study and analysis of a real data set in education.