bluuebunny/crossref_metadata_2025 huggingface dataset:
The Case of the Failed Diversity Rule: A Multiple Streams Analysis
Authors: Stout Karen Evans , Stevens Byron
Published: June 2007

In 1988 Minnesota passed a Multicultural and Gender Fair Curriculum Rule ("diversity" rule) that held districts accountable through submitting a local inclusive education plan to the state department of education. Compliance was slow; thus, between 1991–1997 the State Board of Education added more "teeth" to the rule. When the revised diversity rule entered the policy arena, it precipitated a contentious battle about multicultural curriculum. Using a multiple streams framework ( Kingdon, 1995 ) and a case study methodology, we sought to understand the events and to test the explanatory power of the framework. Results show that the diversity rule became pitted against a less controversial, resource-rich graduation standards approach in the solution stream. Ultimately, a vitriolic commentary in the Minneapolis Star Tribune resulted in power politics primarily from the governor. The rule was tabled, and in the fallout from the conflict, the legislature eliminated the State Board of Education as part of an omnibus education bill. The framework was helpful in understanding the events of the case but less explanatory about the role of media and power and in determining the "window of opportunity." The outcome of the case raises questions about progress toward multicultural curriculum and the ability to write nonpartisan education policy in an increasingly politicized policymaking environment.
Direct and Indirect Writing Assessments: Examining Issues of Equity and Utility
Authors: Heck Ronald H. , Crislip Marian
Published: June 2007

Performance tests are increasingly used as alternatives to, or in connection with, standardized multiple-choice tests as a means of assessing student learning and school accountability. Besides their proposed equity advantages over multiple-choice tests in measuring student learning across groups of students, performance assessments have also been viewed as having greater utility for monitoring school progress because of their proposed closer correspondence to the curriculum that is actually taught. We examined these assumptions by comparing third-grade student performance on a performance-based writing test and a multiple-choice test of language skills. We observed smaller differences in achievement on the writing performance assessment for some groups of students (e.g., low socioeconomic status, various ethnic backgrounds) than are commonly observed on multiple-choice tests. Girls, however, had higher mean scores than boys on both types of assessments. Moreover, the school’s identification and commitment over time to improving its students’ writing skills positively related to its students’ outcomes on the writing performance test. Overall, our examination of performance-based writing assessment is encouraging with respect to providing a relatively fair assessment and measuring learning tasks that are related to the school’s curricular practices.
Collective Sensemaking about Reading: How Teachers Mediate Reading Policy in Their Professional Communities
Authors: Coburn Cynthia E.
Published: June 2007

Recent research on the relationship between instructional policy and classroom practice suggests that teachers interpret, adapt, and even transform policies as they put them into place. This paper extends this line of research, using an in-depth case study of one California elementary school to examine the processes by which teachers construct and reconstruct multiple policy messages about reading instruction in the context of their professional communities. Drawing primarily on institutional and sensemaking theory, this paper puts forth a model of collective sensemaking that focuses on the ways teachers co-construct understandings of policy messages, make decisions about which messages to pursue in their classrooms, and negotiate the technical and practical details of implementation in conversations with their colleagues. It also argues that the nature and structure of formal networks and informal alliances among teachers shape the process, with implications for ways in which messages from the policy environment influence classroom practice. Finally, the paper explores the role school leaders play in shaping the sensemaking process.
Gifted Children in the Current Policy and Fiscal Context of Public Education: A National Snapshot and State-Level Equity Analysis of Texas
Authors: Baker Bruce D.
Published: June 2007

Should we be concerned if educational resources for gifted and talented children vary widely from school to school, district to district, or state to state? Does it matter whether those resources are distributed unevenly by race or social class? This article begins by addressing the basic underlying question: Do gifted and talented children require supplemental resources at all? Two alternate theoretical perspectives are discussed. Under one, the standards-based cost function, there is no need to provide supplemental resources to gifted children, whereas under the alternative resource-cost model, there may be a reasonable need. Accepting the resource-cost model assumption that gifted children do require supplemental resources, this article then explores the distribution of gifted and talented programming opportunities across a national sample of students(The National Educational Longitudinal Study of 1988) and the distribution of fiscal and human resources to gifted education within the Texas, a state long considered a national leader in gifted education. National results show that Hispanic and Native American students are less likely to have access to gifted programs in the eighth grade than Asian/Pacific Island students. Also, students in the lowest socioeconomic status (SES) quartile are far less likely than students in the highest two SES quartiles to have access to eighth-grade gifted and talented programs. Students in districts that are large or suburban, or both, are more likely than those in urban districts to have access, and students in Southern and Western states far more likely to have access than those in the Northeast. Texas results show substantial variance (CV = 125%) in spending per gifted pupil across districts. They further show that district level fiscal resources and community economic characteristics influence spending and the availability of specialized personnel.
The Difficulty of Identifying Rare Samples to Study: The Case of High Schools Divided Into Schools-Within-Schools
Authors: Lee Valerie E. , Ready Douglas D. , Johnson David J.
Published: June 2007

This note has two purposes: (1) to describe a process used to identify a rare sample of U.S. public secondary schools, those that are completely divided into smaller subunits that we call “schools-within-schools” (SWS); and (2) to provide descriptive information about the schools of that kind that we identified in our national search in Fall 1998. The procedures that we used to locate “full-model” SWS high schools in the United States involved snowball sampling and telephone interviews, in which we used a two-stage screening strategy. Although we searched extensively, we located a modest number (n = 55) of full-model public SWS high schools. Beyond spelling out our methodology for locating this unusual sample, we provide descriptive information about the schools along several dimensions: their regional location, the type of communities in which they are located, the size of their school districts, how many students they serve, their racial or ethnic compositions, and the themes on the basis of which their subunits are organized. We offer tentative conclusions about the SWS reform based on our research.
The Effect of Early Childhood Intervention and Subsequent Special Education Services: Findings from the Chicago Child-Parent Centers
Authors: Conyers Liza M. , Reynolds Arthur J. , Ou Suh-Ruu
Published: June 2007

This article explores patterns of special education services during the elementary grades among children who participated in either the Child-Parent Center (CPC) Preschool Program or other early childhood programs in the Chicago Public Schools. The study sample included 1,377 low-income, racial minority children in the Chicago Longitudinal Study. Controlling for family background characteristics that might affect educational performance, children who participated in Child-Parent Center preschool had a significantly lower rate of special education placement (12.5%) than the comparison group (18.4%), who participated in an alternative all-day kindergarten program. The estimated impact of CPC preschool intervention was best explained by the cognitive advantage hypothesis. This article provides support for the long-term impact of the CPC preschool intervention on special education outcomes.
Transition Leadership in a Shifting Policy Environment
Authors: Goldring Ellen , Crowson Robert , Laird David , Berk Robert
Published: June 2007

While transformational leadership has received much attention in recent years, the purpose of this article is to explicate the concept of transition leadership and its centrality to understanding policy implementation and school change through a case study of one district and its school principals, transitioning from court-ordered desegregation to unitary status. The article begins with a discussion of the concept of transition leadership and then examines school principals’ transitional leadership, that is, the process of helping to guide an organization’s response to a very significant change in citywide policy. Data reported in this article are based on a series of open-ended, in-depth interviews with a sample of the district’s elementary and middle school principals. Analyses suggest that transition leadership entails an unmaking of past policy while simultaneously remaking new policies and a heavy reliance on establishing a social fabric and sense of community. The conclusion calls for added attention to the balancing of continuity and change that may be necessary to the leadership of a policy transition.
State Policies and Equal Opportunity: The Example of Gifted Education
Authors: Baker Bruce D. , Friedman-Nimz Reva
Published: June 2007

This study explores the relationship between state policies, including state mandates and state aid allocations, and the distribution of educational opportunities. Specifically, we analyze the availability of and participation rates in programs for gifted and talented students using data from the Common Core of Data 1993–94 and the Schools and Staffing Survey 1993–94. Analyses herein suggest that program mandates and funding may be effective tools for increasing the distribution of opportunities for gifted children. However, models of both aid distribution and opportunity distribution indicate a tendency of states more significantly involved in gifted education, as indicated by mandates and funding, to promote regressive distributions of opportunities (greater availability in schools with fewer low-income students) through regressive distributions of aid (higher levels of aid to districts with fewer children in poverty). More specific case analyses, however, reveal that some states like Virginia may be taking steps to promote more neutral distributions of opportunities through more progressive allocations of state aid.
Passing Reading First: Prominence and Processes in an Elite Policy Network
Authors: Miskel Cecil , Song Mengli
Published: June 2007

Reading became a hotbed of policy activity during the 1990s with both state and federal policymakers launching major initiatives to raise reading achievement. When President George W. Bush introduced his Reading First initiative in early 2001, we saw a unique opportunity to investigate the policy processes, antecedents, and unplanned events surrounding an important and potentially controversial initiative. We began with the assumption that major policy changes are typically made by powerful actors operating in relatively open issue networks. In regard to the Reading First legislation, however, we found a small clique of inside policy entrepreneurs who fashioned major changes in a short period. Upon recognizing this condition, we expanded the research purpose to describe the actions of this insider policy group.
Making Sense of Distributed Leadership: The Case of Peer Assistance and Review
Authors: Goldstein Jennifer
Published: June 2007

This article explores a case of shifting leadership responsibility for teacher evaluation. Peer Assistance and Review (PAR) formally involves teachers in the summative evaluation of other teachers—although the boundaries of the involvement are often vague. Since teacher evaluation has traditionally been the domain of school principals, involving teachers in teacher evaluation raises questions about how those faced with the new role make sense of it and enact it. The article draws on theories of professions, organizations, and institutions to examine the implementation of PAR in one large urban school district. Findings suggest that, despite positive sentiments about the policy across stakeholder groups, those involved wanted principals to remain a central figure in the evaluation of teachers in PAR. Education’s hierarchical norms, the difficulty of conducting evaluations, district leadership, and program ambiguity are identified as challenges to distributing leadership.
Retention Under Chicago’s High-Stakes Testing Program: Helpful, Harmful, or Harmless?
Authors: Roderick Melissa , Nagaoka Jenny
Published: June 2007

In the mid-1990s, the Chicago Public Schools declared an end to social promotion and instituted promotional requirements based on standardized test scores in the third, sixth, and eighth grades. This article examines the experience of third and sixth graders who were retained under Chicago’s policy from 1997 to 2000. The authors examine the progress of these students for 2 years after they were retained and estimate the short-term effects of retention on reading achievement. Students who were retained under Chicago’s high-stakes testing policy continued to struggle during their retained year and faced significantly increased rates of special education placement. Among third graders, there is no evidence that retention led to greater achievement growth 2 years after the promotional gate. Among sixth graders, there is evidence that retention was associated with lower achievement growth. The effects of retention were estimated by using a growth curve analysis. Comparison groups were constructed by using variation across time in the administration of the policy, and by comparing the achievement growth of a group of low-achieving students who just missed passing the promotional cutoff to a comparison group of students who narrowly met the promotional cutoff at the end of the summer. The robustness of the findings was tested using an instrumental variable approach to address selection effects in estimates.
High School Exit Examinations and State-Level Completion and GED Rates, 1975 Through 2002
Authors: Warren John Robert , Jenkins Krista N. , Kulick Rachael B.
Published: June 2007

This article investigates the extent to which state-mandated high school exit examinations are associated with state-level public high school completion rates in the United States. The authors estimate a series of state and year fixed effects models using a new measure of state-level public high school completion rates and archival information about states’ policies on high school exit examinations from 1975 through 2002. The study finds that state high school exit examinations—particularly the “more difficult” examinations that have been implemented recently in some states—are associated with lower public high school completion rates and higher rates of General Educational Development test taking. Furthermore, the study finds that the association between state policies on high school exit examinations and public high school completion grows stronger as states become more racially and ethnically diverse and as poverty rates increase.
Does the Age That Children Start Kindergarten Matter? Evidence of Long-Term Educational and Social Outcomes
Authors: Lincove Jane Arnold , Painter Gary
Published: June 2007

The appropriate age for students to begin school is an issue of debate for educators, administrators, and parents. Parents worry that young children may not be able to compete with older classmates; schools worry that young students will not be able to meet rigorous academic standards associated with school accountability. Past literature is inconclusive as to the overall effect of age at school entry. Some research suggests that younger students have lower average achievement in early elementary school, while others find that students with summer birthdates, who are assumed to be younger at school entry, gain more education on average. At present, little is known about the impact of age at school entry on education attainment as students transition from high school into college and the labor market. This study uses data from the National Education Longitudinal Survey to examine long-term effects of age at school entry on both educational and social outcomes, with special attention to those students who enter kindergarten a year later than their peers. The results of this study suggest that delaying kindergarten does not create any long-term advantages for students.
The Influence of Access to General Education Curriculum on Alternate Assessment Performance of Students With Significant Cognitive Disabilities
Authors: Roach Andrew T. , Elliott Stephen N.
Published: June 2007

The primary purpose of this investigation was to understand the influence of access to the general curriculum on the performance of students with significant cognitive disabilities, as measured by the Wisconsin Alternate Assessment (WAA) for Students with Disabilities. Special education teachers (N = 113) submitted case materials for students with significant disabilities who were assessed using the WAA. Cases included WAA rating scales, students’ individualized education programs (IEPs), and a curricular access questionnaire. Structural equation modeling was used to examine the influence of curricular access on students’ WAA performance. Students’ scores on the WAA Reading, Language Arts, and Mathematics scales were included in the model as the indicators for the latent factor—student performance. Student grade level, teacher reports of students’ curricular access, percentage of academic focused IEP goals, and time spent in general education settings were included as predictors of students’ performance. Results indicated the model accounted for 41% of the variance in the latent factor of student performance. Implications of these results are discussed for the validity of WAA scores and the design of professional development and teacher support materials.
A Note on Testing for Multivariate Effect Sizes
Authors: Timm Neil H.
Published: June 2007

This note provides the investigator with an overall test procedure for testing the significance of multivariate effect sizes, assuming a regression model for the vector of effects, using Wald's statistic. Given a regression model of the form δ = Xβ, the step down Finite Intersection Test (FIT) procedure is used to test that all effect sizes are zero. Finally, an overall study effect size is proposed. Large sample tests are developed that involve the multivariate chi-square distribution.
F Tests for Mean and Covariance Structure Analysis
Authors: Yuan Ke-Hai , Bentler Peter M.
Published: June 2007

Covariance structure analysis is used for inference and for dimension reduction with multivariate data. When data are not normally distributed, the asymptotic distribution free (ADF) method is often used to fit a proposed model. The ADF test statistic is asymptotically distributed as a chi-square variate. Experience with real data indicates that the ADF statistic tends to reject theoretically meaningful models. Empirical simulation shows that the ADF statistic rejects correct models too often for all but impractically large sample sizes. By comparing mean and covariance structure analysis with its analogue in the multivariate linear model, we propose some modified ADF test statistics whose distributions are approximated by F distributions. Empirical studies show that the distributions of the new statistics are more closely approximated by F distributions than are the original ADF statistics when referred to chi-square distributions. Detailed analysis indicates why the ADF statistic fails on large models and why F tests and corrections give better results. Implications for power analysis and model tests in other areas are discussed.
Conditionally Linear Mixed-Effects Models With Latent Variable Covariates
Authors: Blozis Shelley A. , Cudeck Robert
Published: June 2007

A version of the nonlinear mixed-effects model is presented that allows random effects only on the linear coefficients. Nonlinear parameters are not stochastic. In nonlinear regression, this kind of model has been called conditionally linear. As a mixed-effects model, this structure is more flexible than the popular linear mixed-effects model, while being nearly as straightforward to estimate. In addition to the structure for the repeated measures, a latent variable model ( Browne, 1993 ) is specified for a distinct set of covariates that are related to the random effects in the second level. Unbalanced data are allowed on the repeated measures, and data that are missing at random are allowed on the repeated measures or on the observed variables of the factor analysis sub-model. Features of the model are illustrated by two examples.
A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong
Authors: Vargha András , Delaney Harold D.
Published: June 2007

McGraw and Wong (1992) described an appealing index of effect size, called CL, which measures the difference between two populations in terms of the probability that a score sampled at random from the first population will be greater than a score sampled at random from the second. McGraw and Wong introduced this "common language effect size statistic" for normal distributions and then proposed an approximate estimation for any continuous distribution. In addition, they generalized CL to the n-group case, the correlated samples case, and the discrete values case.

In the current paper a different generalization of CL, called the A measure of stochastic superiority, is proposed, which may be directly applied for any discrete or continuous variable that is at least ordinally scaled. Exact methods for point and interval estimation as well as the significance tests of the A = .5 hypothesis are provided. New generalizations ofCL are provided for the multi-group and correlated samples cases.
Univariate and Bivariate Loglinear Models for Discrete Test Score Distributions
Authors: Holland Paul W. , Thayer Dorothy T.
Published: June 2007

The well-developed theory of exponential families of distributions is applied to the problem of fitting the univariate histograms and discrete bivariate frequency distributions that often arise in the analysis of test scores. These models are powerful tools for many forms of parametric data smoothing and are particularly well-suited to problems in which there is little or no theory to guide a choice of probability models, e.g., smoothing a distribution to eliminate roughness and zero frequencies in order to equate scores from different tests. Attention is given to efficient computation of the maximum likelihood estimates of the parameters using Newton's Method and to computationally efficient methods for obtaining the asymptotic standard errors of the fitted frequencies and proportions. We discuss tools that can be used to diagnose the quality of the fitted frequencies for both the univariate and the bivariate cases. Five examples, using real data, are used to illustrate the methods of this paper.
Bayesian Procedures for Prediction Analysis of Implication Hypotheses in 2 × 2 Contingency Tables
Authors: Lecoutre Bruno , Charron Camilo
Published: June 2007

Procedures for prediction analysis in 2 x 2 contingency tables are illustrated by the analysis of successes to six types of problems associated with the acquisition of fractions. According to Hildebrand, Laing, and Rosenthal (1977), hypotheses such as "success to problem type A implies in most cases success to problem type B" can be evaluated from a numerical index. This index has been considered in various other frameworks and can be interpreted in terms of a measure of predictive efficiency of implication hypotheses. Confidence interval procedures previously proposed for this index are reviewed and extended. Then, under a multinomial model with a conjugate Dirichlet prior distribution, the Bayesian posterior distribution of this index is characterized, leading to straightforward numerical methods.1 The choices of "noninformative" priors for discrete data are shown to be no more arbitrary or subjective than the choices involved in the frequentist approach. Moreover, a simulation study of frequentist coverage probabilities favorably compares Bayesian credibility intervals with conditional confidence intervals.
Dispersion of Responses Over Cells
Authors: Klingsporn M. J.
Published: June 2007

The usual requirement for testing hypotheses regarding the dispersion of responses distributed over taxa uses a Pearson's goodness of fit chi-square. More recently maximum likelihood statistics have also come into vogue. Standard recommendations for either technique is to have at least five observations expected under each taxon. An alternative procedure proposed in this paper is to use the distribution of the number of cells that are either empty or are singly occupied. Recursive formula were developed and rationalizations provided for analyzing either of these situations. A table showing number of cases needed to achieve. 10, .05, and .01 significance for excessive numbers of empty cells was developed.
Item Response Theory True Score Equatings and Their Standard Errors
Authors: Ogasawara Haruhiko
Published: June 2007

The asymptotic standard errors of the estimates of the equated scores by several types of item response theory (IRT) true score equatings are provided. The first group of equatings do not use IRT equating coefficients. The second group of equatings use the IRT equating coefficients given by the moment or characteristic curve methods. The equating designs considered in this article cover those with internal or external common items and the methods with separate or simultaneous estimation of item parameters of associated tests. For the estimates of the asymptotic standard errors of the equated true scores, the method of marginal maximum likelihood estimation is employed for estimation of item parameters.
Multidimensional Item Response Theory Modeling of Binary Data: Large Sample Properties of NOHARM Estimates
Authors: Maydeu-Olivares Albert
Published: June 2007

NOHARM is a program that performs factor analysis for dichotomous variables assuming that these arise from an underlying multinormal distribution. Parameter estimates are obtained by minimizing an unweighted least squares function of the first- and second-order marginal proportions. Here, large sample standard errors for restricted as well as rotated unrestricted factor solutions are given. Also a test of the goodness of fit of the model to the first- and second-order marginals of the contingency table is proposed. In a simulation study, it was found that for small models, accurate parameter estimates, standard errors, and goodness-of-fit tests can be obtained with as few as 100 observations. Furthermore, NOHARM estimates, standard errors, and goodness-of-fit tests are comparable to those obtained using a related LISREL procedure.
Handling Item Nonresponse in the U.S. Component of the IEA Reading Literacy Study
Authors: Winglee Marianne , Kalton Graham , Rust Keith , Kasprzyk Daniel
Published: June 2007

The U.S. component of the International Reading Literacy Study provides a data set where nonresponses to the background questionnaire items were filled in using imputation methods (mainly hot-deck). This study uses the completed data set for analyses and compares the results with those from other methods of handling missing data. Analyses conducted include regression and hierarchical linear models. The imputed data set yields results similar to those produced by available case analyses (pairwise deletion) and by the estimation and maximization algorithm analyses. The results, however, are different from those produced by complete case analyses (casewise deletion). For most analyses of the Reading Literacy Study, the data set completed by imputation is a convenient option.
Optimal Calibration Designs for Tests of Polytomously Scored Items Described by Item Response Theory Models
Authors: Holman Rebecca , Berger Martijn P. F.
Published: June 2007

This article examines calibration designs, which maximize the determinant of Fisher’s information matrix on the item parameters (D-optimal), for sets of polytomously scored items. These items were analyzed using a number of item response theory (IRT) models, which are members of the “divide-by-total” family, including the nominal categories model, the rating scale model, the unidimensional polytomous Rasch model and the partial credit model. We extend the known results for dichotomous items, both singly and in tests to polytomous items. The structure of Fisher’s information matrix is examined in order to gain insights into the structure of D-optimal calibration designs for IRT models. A theorem giving an upper bound for the number of support points for such models is proved. A lower bound is also given.

Finally, we examine a set of items, which have been analyzed using a number of different models. The locally D-optimal calibration design for each analysis is calculated using an exact numerical and a sequential procedure. The results are discussed both in general and in relation to each other.
Efficiency and Robustness of Alternative Estimators for Two- and Three-level Models: The Case of NAEP
Authors: Cheong Yuk Fai , Fotiu Randall P. , Raudenbush Stephen W.
Published: June 2007

This article investigates the efficiency and robustness of alternative estimators of regression coefficients for three-level data. To study student achievement, researchers might formulate a standard regression model or a hierarchical model with a two- or three-level structure. Having chosen the model, the researchers might employ either a model-based or a robust estimator of the standard errors. A simulation study showed that, as expected, the hierarchical model analyses produced more efficient point estimates than did analyses that ignored the covariance structure in the data, even when the normality assumption was violated. When samples were fairly large, the three-level analyses produced sound standard errors. In contrast, single-level analysis yielded seriously biased standard errors for coefficients defined at level 3 and level 2; and two-level analysis yielded biased standard errors for coefficients defined at level 2. These biases in standard error estimates were largely corrected by robust variance estimation. Implications of the results for analyzing NAEP and other large-scale surveys such as the Early Childhood Longitudinal Study (ECLS) and the Third International Mathematics and Science Study (TIMSS) are discussed.
Quick and Easy Implementation of the Benjamini-Hochberg Procedure for Controlling the False Positive Rate in Multiple Comparisons
Authors: Thissen David , Steinberg Lynne , Kuang Daniel
Published: June 2007

Williams, Jones, and Tukey (1999) showed that a sequential approach to controlling the false discovery rate in multiple comparisons, due to Benjamini and Hochberg (1995), yields much greater power than the widely used Bonferroni technique that limits the familywise Type I error rate. The Benjamini-Hochberg (B-H) procedure has since been adopted for use in reporting results from the National Assessment of Educational Progress (NAEP), as well as in other research applications. This short note illustrates that the B-H procedure is extremely simple to implement using widely available spreadsheet software. Given its easy implementation, it is feasible to include the B-H procedure in introductory instruction in inferential statistics, augmenting or replacing the Bonferroni technique.
Constrained Principal Component Analysis: Various Applications
Authors: Hunter Michael A. , Takane Yoshio
Published: June 2007

Constrained Principal Component Analysis (CPCA) is a method for structural analysis of multivariate data. It combines regression analysis and principal component analysis into a unified framework. This article provides example applications of CPCA that illustrate the method in a variety of contexts common to psychological research. We begin with a straightforward situation in which the structure of a set of criterion variables is explored using a set of predictor variables as row (subjects) constraints. We then illustrate the use of CPCA using constraints on the columns of a set of dependent variables. Two new analyses, decompositions into finer components and fitting higher order structures, are presented next, followed by an illustration of CPCA on contingency tables, and CPCA of residuals that includes assessing reliability using the bootstrap method.
An Assessment of Basic Computer Proficiency Among Active Internet Users: Test Construction, Calibration, Antecedents and Consequences
Authors: Bradlow Eric T. , Hoch Stephen J. , Wesley Hutchinson J.
Published: June 2007

The purpose of this article is to describe our efforts to create a test of basic computer proficiency, examine its properties using parametric test scoring methods, and identify some antecedents and consequences that accompany differences in performance. We also consider how much insight people have into their level of knowledge by examining the relationship between our tested measure of computer knowledge and self-rated knowledge scores collected at the same time. This research also adds to the large body of existing empirical work on computer literacy in the student population, by looking at computer literacy in a more general sample of the Internet-using population. A further purpose of this research, as a result, is to make our dataset available for future research.
Testing Non-nested Multivariate Effect Size Models in Meta-Analysis
Authors: Timm Neil H.
Published: June 2007

In modeling multiple effect sizes in meta-analysis, one often wants to evaluate two linear models for multivariate outcomes by including different sets of non-nested predictors for each outcome. This results in two non-nested multivariate effect size models. In this article, we show how to test the hypothesis that a non-nested model fits a set of predictors.
Combining Aggregate and Individual Level Data to Estimate an Individual Level Correlation Coefficient
Authors: Raghunathan Trivellore E. , Diehr Paula K. , Cheadle Allen D.
Published: June 2007

Methods are developed that use aggregate data, possibly based on a large number of individuals, and individual level data, from a small fraction of individuals from the same or similar population, to eliminate ecological bias inherent in the analysis of aggregate data. The primary focus is on estimating the individual level correlation coefficient but the proposed methodology can be extended to estimate regression coefficients. Two approaches, the method of moments and the maximum likelihood, are developed for a bivariate distribution, but can be extended to a multivariate distribution. The method of moments develops a corrected estimate of the within-group covariance matrix, which is then used to estimate the individual level correlation and regression coefficients. The second method assumes bivariate normality and maximizes the combined likelihood function based on the two data sets. The maximum likelihood estimates are obtained using the EM-algorithm. A simulation study investigates the repeated sampling properties of these procedures in terms of bias and the mean square error of the point estimates and the actual coverage of the confidence intervals. The maximum likelihood estimates are almost unbiased and the confidence intervals are well calibrated for simulation conditions considered. The method of moments estimates have the same desirable properties for some simulation conditions. Under all conditions, the correlation coefficient between aggregate variables is severely biased as an estimate of the individual level correlation coefficient.
An Optimal Design Approach to Criterion-Referenced Computerized Testing
Authors: Wiberg Marie
Published: June 2007

A criterion-referenced computerized test is expressed as a statistical hypothesis problem. This admits that it can be studied by using the theory of optimal design. The power function of the statistical test is used as a criterion function when designing the test. A formal proof is provided showing that all items should have the same item characteristics, i.e. items that have high discrimination, low guessing and difficulty near the cut-off score give the most powerful statistical test. An efficiency study shows how many times more items are needed if nonoptimal items are used instead of optimal items in order to get the same power in the test.
Statistical Power and Optimum Sample Allocation Ratio for Treatment and Control Having Unequal Costs per Unit of Randomization
Authors: Liu Xiaofeng
Published: June 2007

This article considers optimal sample allocation between the treatment and control condition in multilevel designs when the costs per sampling unit vary due to treatment assignment. Optimal unequal allocation may reduce the cost from that of a balanced design without sacrificing any power. The optimum sample allocation ratio depends only on the cost ratio between the treatment and control regardless of whether the randomization of sampling units occurs at levels 1, 2, or 3. Power functions for the exact tests for the main effect of treatment are derived for prototypical multilevel designs with unequal sample sizes in the treatment and control condition.
Calibrating Item Families and Summarizing the Results Using Family Expected Response Functions
Authors: Sinharay Sandip , Johnson Matthew S. , Williamson David M.
Published: June 2007

Item families, which are groups of related items, are becoming increasingly popular in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from each of a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families with multiple-choice items. We fit the model using the Markov Chain Monte Carlo (MCMC) algorithm, introduce the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family, and suggest a way to estimate the FERFs. This work is thus a step towards creating a tool that can save significant amount of resources in educational testing, by allowing proper analysis and summarization of data from tests involving item families.
A Probability Index of the Robustness of a Causal Inference
Authors: Pan Wei , Frank Kenneth A.
Published: June 2007

Causal inference is an important, controversial topic in the social sciences, where it is difficult to conduct experiments or measure and control for all confounding variables. To address this concern, the present study presents a probability index to assess the robustness of a causal inference to the impact of a confounding variable. The information from existing covariates is used to develop a reference distribution for gauging the likelihood of observing a given value of the impact of a confounding variable. Applications are illustrated with an empirical example pertaining to educational attainment. The methodology discussed in this study allows for multiple partial causes in the complex social phenomena that we study, and informs the controversy about causal inference that arises from the use of statistical models in the social sciences.
Constraining Item Exposure in Computerized Adaptive Testing With Shadow Tests
Authors: van der Linden Wim J , Veldkamp Bernard P
Published: June 2007

Item-exposure control in computerized adaptive testing is implemented by imposing item-ineligibility constraints on the assembly process of the shadow tests. The method resembles Sympson and Hetter’s (1985) method of item-exposure control in that the decisions to impose the constraints are probabilistic. The method does not, however, require time-consuming simulation studies to set values for control parameters before the operational use of the test. Instead, it can set the probabilities of item ineligibility adaptively during the test using the actual item-exposure rates. An empirical study using an item pool from the Law School Admission Test showed that application of the method yielded perfect control of the item-exposure rates and had negligible impact on the bias and mean-squared error functions of the ability estimator.
An Interesting Problem in the Estimation of Scoring Reliability
Authors: Livingston Samuel A.
Published: June 2007

A performance assessment consisting of 10 separate exercises was scored with a randomized scoring procedure. All responses to each exercise were rated once; in addition, a randomly selected subset of the responses to each exercise received an independent second rating. Each second rating was averaged with the corresponding first rating before the scores were computed. This article presents a method for estimating the scoring reliability (interrater reliability) coefficient and the standard error of scoring for the resulting scores. The report concludes with some numerical examples showing how the reliability estimation procedure can be used to estimate the effect of varying the proportions of responses that are double-scored.
Finite Mixture Dynamic Regression Modeling of Panel Data With Implications for Dynamic Response Analysis
Authors: Kaplan David
Published: June 2007

This article considers the problem of estimating dynamic linear regression models when the data are generated from finite mixture probability density function where the mixture components are characterized by different dynamic regression model parameters. Specifically, conventional linear models assume that the data are generated by a single probability density function characterized by a single set of regression model parameters. However, when the true generating model is finite mixture density function, then estimation of conventional linear models under the assumption of a single density function may lead to erroneous conclusions. Instead, it may be desirable to estimate the regression model under the assumption that the data are derived from a finite mixture density function and to examine differences in the parameters of the model within each mixture component. Dynamic regression models and subsequent dynamic response analysis using dynamic multipliers are also likely to be affected by the existence of a finite mixture density because dynamic multipliers are functions of the regression model parameters. Utilizing finite mixture modeling applied to two real data examples, this article shows that dynamic responses to changes in exogenous variables can be quite different depending on the number and nature of underlying mixture components. Implications for substantive conclusions based on the use of dynamic multipliers is discussed.
A Growth Model for Multilevel Ordinal Data
Authors: Segawa Eisuke
Published: June 2007

Multi-indicator growth models were formulated as special three-level hierarchical generalized linear models to analyze growth of a trait latent variable measured by ordinal items. Items are nested within a time-point, and time-points are nested within subject. These models are special because they include factor analytic structure. This model can analyze not only data with item- and time-level missing observations, but also data with time points freely specified over subjects. Furthermore, features useful for longitudinal analyses, “autoregressive error degree one” structure for the trait residuals and estimated time-scores, were included. The approach is Bayesian with Markov Chain and Monte Carlo, and the model is implemented in WinBUGS. They are illustrated with two simulated data sets and one real data set with planned missing items within a scale.
Estimating Causal Effects From Multilevel Group-Allocation Data
Authors: Gitelman Alix I.
Published: June 2007

In group-allocation studies for comparing behavioral, social, or educational interventions, subjects in the same group necessarily receive the same treatment, whereby a group and/or group-dynamic effect can confound the treatment effect. General counterfactual outcomes that depend on group characteristics, group membership, and treatment are developed to provide a structure for specifying causal effects of treatment in the multilevel setting. An average causal effect of treatment cannot be specified, however, without a simplifying assumption of group-membership invariance (i.e., no group-dynamic effect). Under group-membership invariance and ignorability assumptions, the average causal effect is then connected to estimable quantities of the hierarchical linear model (HLM). Furthermore, it is shown that the typical specification of the HLM involves conditional independence assumptions that actually preclude the group-dynamic effect.
Model Diagnostics for Bayesian Networks
Authors: Sinharay Sandip
Published: June 2007

Bayesian networks are frequently used in educational assessments primarily for learning about students’ knowledge and skills. There is a lack of works on assessing fit of Bayesian networks. This article employs the posterior predictive model checking method, a popular Bayesian model checking tool, to assess fit of simple Bayesian networks. A number of aspects of model fit, those of usual interest to practitioners, are assessed using various diagnostic tools. This article suggests a direct data display for assessing overall fit, suggests several diagnostics for assessing item fit, suggests a graphical approach to examine if the model can explain the association among the items, and suggests a version of the Mantel–Haenszel statistic for assessing differential item functioning. Limited simulation studies and a real data application demonstrate the effectiveness of the suggested model diagnostics.
Double P-P Plots for Comparing Differences Between Two Groups
Authors: Livingston Samuel A.
Published: June 2007

This article suggests a graphic technique that uses P-P plots to show the extent to which two groups differ on two variables. It can be used even if the variables are measured in completely different, noncomparable units. The comparison is symmetric with respect to the variables and the groups. It reflects the differences between the groups over the full range of values of each variable. Two examples from educational testing are included; the technique is also applicable to other fields of inquiry.