bluuebunny/crossref_metadata_2025 huggingface dataset:
Distribution of the Determinant of the Sample Correlation Matrix: Monte Carlo Type One Error Rates
Authors: Reddon John R. , Jackson Douglas N. , Schopflocher Donald
Published: March 2008

Computer sampling from a multivariate normal spherical population was used to evaluate the type one error rates for a test of sphericity based on the distribution of the determinant of the sample correlation matrix. The range of variates considered was 5 to 25, and the range of observations considered was 10 to 1,000. The type one error rates were estimated in each condition with 5,000 replications. The c.d.f. obtained from the asymptotic χ2 resulted in excessive type one errors in the conditions where the ratio of the sample size to the number of variates was small. The c.d.f. with finite sample corrections involving terms up to ο( N−3) performed much better in small samples but resulted in a biased test when the sample size was less than twice the number of variates.
Recent Developments in the Factor Analysis of Categorical Variables
Authors: Mislevy Robert J.
Published: March 2008

Despite known shortcomings of the procedure, exploratory factor analysis of dichotomous test items has been limited, until recently, to unweighted analyses of matrices of tetrachoric correlations. Superior methods have begun to appear in the literature, in professional symposia, and in computer programs. This paper places these developments in a unified framework, from a review of the classical common factor model for measured variables through generalized least squares and marginal maximum likelihood solutions for dichotomous data. Further extensions of the model are also reported as work in progress.
Maximum Likelihood Estimation in Generalized Rasch Models
Authors: De Leeuw Jan , Verhelst Norman
Published: March 2008

We review various models and techniques that have been proposed for item analysis according to the ideas of Rasch. A general model is proposed that unifies them, and maximum likelihood procedures are discussed for this general model. We show that unconditional maximum likelihood estimation in the functional Rasch model, as proposed by Wright and Haberman, is an important special case. Conditional maximum likelihood estimation, as proposed by Rasch and Andersen, is another important special case. Both procedures are related to marginal maximum likelihood estimation in the structural Rasch model, which has been studied by Sanathanan, Andersen, Tjur, Thissen, and others. Our theoretical results lead to suggestions for alternative computational algorithms.
Dropping out of High School in the United States: An Observational Study
Authors: Rosenbaum Paul R.
Published: March 2008

Using data from a two-stage probability sample of U.S. high school students, an attempt is made to estimate the effect that dropping out has on cognitive achievement test scores. Each sampled dropout from a school is matched by a multivariate procedure to a student who remained in the same school. The matched pair differences are then adjusted using analysis of covariance. The possibility that important covariates have been omitted from the analysis is addressed through tests of ignorable treatment assignment and through sensitivity analyses.
Maximal Correlation Between Normal and Dichotomous Variables
Authors: Gradstein Mark
Published: March 2008

The purpose of this paper is to calculate the upper limit of the correlation between normal and dichotomous variables. An empirically obtained correlation should be evaluated in view of this limit, instead of the usual limit of Pearson correlation.
The Use of Test Scores for Classification Decisions with Threshold Utility
Authors: van der Linden Wim J.
Published: March 2008

The classification problem consists of assigning subjects to one of several available treatments on the basis of their test scores, where the success of each treatment is measured by a different criterion. It is indicated how this problem can be formulated as an (empirical) Bayes decision problem. As an example, the case of classification with a threshold utility function is analyzed, and optimal assignment rules are derived. The results are illustrated empirically with data from a classification problem in which achievement test data are used to assign students to appropriate continuation schools.
The Inadequacy of ANOVA for Detecting Test Bias
Authors: Camilli Gregory , Shepard Lorrie A.
Published: March 2008

The inadequacy of ANOVA for detecting bias in test items should already be well understood, yet it persists as a popular method. Here, previous arguments are extended to explain why ANOVA may obscure test bias when it exists, as well as create false impressions of bias. In fact, it is demonstrated in this paper that ANOVA will fail to detect even absurdly large amounts of bias. More specifically, it is shown that bias contributes relatively more to the group main effect than to the group-by-item interaction.
The Journal of Educational Statistics: The First Decade in Review
Authors: Prosser Robert J. , Traub Ross E. , Vegeris Sandra L.
Published: March 2008

The objectives of this review were to assess the quality of articles published in JES, to find out whether or not JES is perceived to occupy a distinct niche among the quantitative journals that serve educational research, and to obtain suggestions for improvement. These objectives were addressed by one or another of the five components to the review: an assessment by experts, a questionnaire survey, a citation study, a content analysis of the articles in selected volumes of JES, and a study of keywords. The accumulated evidence indicates that JES is seen to publish good quality articles that make original contributions to knowledge. The questionable relevance of some articles to the study of education and the emphasis in some articles on measurement issues perhaps reflects the fact that the niche occupied by JES is not wholly congruent with the objective stated on the journal’s boilerplate. Suggestions are offered for ways of increasing the number of appropriate articles and increasing circulation.
A Comparison of Two Methods of Decomposing Item Difficulties
Authors: Green Kathy E. , Smith Richard M.
Published: March 2008

This paper compares two methods of estimating component difficulties for dichotomous test data. Simulated data are used to study the effects of sample size, collinearity, a measurement disturbance, and multidimensionality on the estimation of component difficulties. The two methods of estimation used in this study were conditional maximum likelihood estimation of parameters specified by the linear logistic test model (LLTM) and estimated Rasch item difficulties regressed on component frequencies. The results of the analysis indicate that both methods produce similar results in all comparisons. Neither of the methods worked well in the presence of an incorrectly specified structure or collinearity in the component frequencies. However, both methods appear to be fairly robust in the presence of measurement disturbances as long as there is a large number of cases (n = 1,000). For the case of fitting data with uncorrelated component frequencies, 30 cases were sufficient to recover the generating parameters accurately.
When Is it Preferable to Estimate Population Percentiles From a Set of Classes Rather Than From the Raw Data?
Authors: Châtillon Guy , Gélinas Réal , Martin Louise , Laurencelle Louis
Published: March 2008

For estimating percentiles of a distribution from a sample, it turns out that in many cases the observed percentiles provide better estimates when they are based on grouped data than when based on the raw data. This is the case especially when the data set is relatively small and the population distribution is not too skewed or heavy-tailed. This fact is surprising and contrary to the general feeling. Applied statisticians and producers of calculators could take advantage of it by implementing a small program for estimating percentiles from grouped data within their calculators. Such a program is generally impracticable if one uses raw data, due to lack of memory space and length of computing time.
Understanding Scoring Reliability: Experiments in Calibrating Essay Readers
Authors: Braun Henry I.
Published: March 2008

Scoring reliability of essays and other free-response questions is of considerable concern, especially in large, national administrations. This report describes a statistically designed experiment that was carried out in an operational setting to determine the contributions of different sources of variation to the unreliability of scoring. The experiment made novel use of partially balanced incomplete block designs that facilitated the unbiased estimation of certain main effects without requiring readers to assess the same paper several times. In addition, estimates were obtained of the improvement in reliability that results from removing variability from systematic sources of variation by an appropriate adjustment of the raw scores. This statistical calibration appears to be a cost-effective approach to enhancing scoring reliability when compared to simply increasing the number of readings per paper. The results of the experiment also provide a framework for examining other, simpler calibration strategies. One such strategy is briefly considered.
An Assessment of the Dimensionality of Three SAT-Verbal Test Editions
Authors: Cook Linda L. , Dorans Neil J. , Eignor Daniel R.
Published: March 2008

A strong assumption made by most commonly used item response theory (IRT) models is that the data are unidimensional, that is, statistical dependence among item scores can be explained by a single ability dimension. First-order and second-order factor analyses were conducted on correlation matrices among item parcels of SAT-Verbal items. The item parcels were constructed to yield correlation matrices that were amenable to linear factor analyses. The first-order analyses were employed to assess the effective dimensionality of the item parcel data. Second-order analyses were employed to test meaningful hypotheses about the structure of the data. Parcels were constructed for three SAT-Verbal editions. The dimensionality analyses revealed that one SAT-Verbal test edition was less parallel to the other two editions than these other editions were to each other. Refinements in the dimensionality methodology and a more systematic dimensionality assessment are logical extensions of the present research.
Evaluating the Normal Approximation to the Binomial Test
Authors: Ramsey Philip H. , Ramsey Patricia P.
Published: March 2008

The normal approximation to the binomial test with and without a continuity correction is evaluated in terms of control of Type I errors and power. The normal approximations are evaluated as robust for a given sample size, N, and at a given level α if the true Type I error rate never exceeds 1.5 α. The uncorrected normal test is found to be less robust than is implied by the currently applied guidelines. The most stringent currently used guideline of requiring σ2≥10 is adequate at α = .05 but must be increased to σ2 ≥35 at α = .01. The corrected test is shown to be robust but not conservative. Both tests are shown to have substantial power loss in comparison to the exact binomial test.
Type I Error Rates for Yao’s and James Tests of Equality of Mean Vectors Under VarianceCovariance Heteroscedasticity
Authors: Algina James , Tang Kezhen L.
Published: March 2008

For Yao’s and James’ tests, Type I error rates were estimated for various combinations of the number of variables (p), samplesize ratio (n1: n2), sample-size-to-variables ratio, and degree of heteroscedasticity. These tests are alternatives to Hotelling’s T2 and are intended for use when the variance-covariance matrices are not equal in a study using two independent samples. The performance of Yao’s test was superior to that of James’. Yao’s test had appropriate Type I error rates when p ≥ 10, (n1 + n2)/p ≥ 10, and 1:2 ≤ n1:n2 ≤ 2:1. When (n1 + n2)/p = 20, Yao’s test was robust when n1: n2 was 5:1, 3:1, and 4:1 and p was 2, 6, and 10, respectively.
Retention or Attrition Models
Authors: Guttman Irwin , Olkin Ingram
Published: March 2008

A model for retention and its counterpart, attrition, is presented. In a prototype example, students enter a program in each of k terms; some of the students complete the program, and the remainder leave. A key feature in the models proposed is that there is a dampening effect from term to term because the probability of leaving the program diminishes as the terms progress. The focus of this paper is the study of alternative models for the dampening in attrition rates. A number of alternative dampening effects are proposed that provide for different rates of attrition. Approximate maximum likelihood estimates for the underlying parameters in each model and a Bayesian analysis are provided.
Empirical Bayes Ranking Methods
Authors: Laird Nan M. , Louis Thomas A.
Published: March 2008

Ranking problems arise in setting priorities for investigations, in providing a simple summary of performance, in comparing objects in a manner robust to measurement scale, and in a wide variety of other applications. Commonly, rankings are computed from measurements that depend on the true attribute. Using the Gaussian model, we propose and compare methods for using these measurements to estimate the ranks of the underlying attributes and show that those based on an empirical Bayes model produce estimates that differ from ranking observed data. These differences result both from the effect of shrinking posterior means towards a common value by an amount that depends on the precision of individual measurements and from the Bayes processing of the posterior distribution to produce estimates that account for the uncertainty in the distribution of the ranks. We illustrate different ranking methods using data on school achievement reported by Aitkin and Longford (1986) . Mathematical and empirical results highlight the importance of using appropriate ranking methods and identify issues requiring further research.
Testing Independence in 2 × 2 Contingency Tables
Authors: Kroll Neal E. A.
Published: March 2008

Three of the potential models for the 2 × 2 contingency table are discussed: the hyper geometric Independence Trial, the double-binomial Comparative Trial, and the multinomial Double Dichotomy Trial. Then the critical regions of seven statistical tests are evaluated within each of these models. The uncorrected Pearson’s Chi-Square test and Upton’s correction are found to be overly liberal in all three models, particularly for the hypergeometric. A continuity correction, when properly applied, is shown to be an extremely good approximation to Fisher’s Exact test, which employs the hypergeometric distribution to evaluate outcome probabilities—but is an extremely conservative approximation to the double binomial and multinomial models. Haber’s test is recommended for these two cases, but the amount of work required may be prohibitive for many investigators.
An Evaluation of Woodruff’s Technique for Variance Estimation in Educational Surveys
Authors: Wilson Mark
Published: March 2008

An empirical sampling approach was used to assess the accuracy of a Taylor approximation for the estimation of sampling errors in the statistics involved in estimating a path model based on medium-sized samples gathered using five sample designs commonly used in educational research. The technique was found to give a reasonable approximation for the simplest of the designs but to deteriorate as clustering, stratification, and weighting became more pronounced.
Mediated Models for the Analysis of Confounded Variables and Self-Selected Samples
Authors: Birnbaum Michael H. , Mellers Barbara A.
Published: March 2008

This paper addresses the interpretation of data that are contaminated by self-selected samples and/or lack of experimental control. Wainer (1989) reviewed different methods for treating self-selected samples and concluded that the most defensible approach is to model the process that caused the data to be missing. In concert with Wainer, the present paper emphasizes the value of specifying models; however, the theses of the present paper are that (a) any analysis should be interpreted in the context of a set of rival theoretical models, (b) these models should allow latent mediating variables that are imperfectly measured by observed variables, and (c) modeling brings clarity to the conclusion that confounded data can be misleading. Simple models clarify the limitations on conclusions one might otherwise attempt to draw from tainted data.
Critical Values for Spearman’s Rank Order Correlation
Authors: Ramsey Philip H.
Published: March 2008

Introductory statistics texts have been noted to have inaccuracies in the tables of critical values for Spearman’s correlation. Even the best texts currently available use critical values from the exact distribution only for N≥ 11. Zar’s table gives critical values for N ≥ 100 but does not use the most accurate approximation procedure available. This paper provides a table of critical values based on the exact distribution for 3 ≥ N ≥ 18 and very accurate critical values for 19 ≥ N ≥ 100 estimated using the Edgeworth approximation.
An Investigation of the Type I Error and Power Properties of the Rank Transform Procedure in Factorial ANOVA
Authors: Sawilowsky Shlomo S. , Blair R. Clifford , Higgins James J.
Published: March 2008

This study examined the Type I error and power properties of the rank transform test when employed in the context of a balanced 2x2x2 fixed effects ANOVA. The results showed the rank transform procedure to be erratic with respect to both Type I error and power. Under some circumstances the test was both robust and powerful, whereas in other circumstances it was decidedly nonrobust and manifested power considerably below that of the usual ANOVA F test. It is recommended that researchers avoid this test except in those specific circumstances where its properties are well understood.
Error Rates in Competency Testing When Test Retaking Is Permitted
Authors: Huynh Huynh
Published: March 2008

False positive and false negative error rates are studied for competency testing where examinees are permitted to retake the test if they fail to pass. Formulae are provided for the beta-binomial and Rasch models, and estimates based on these two models are compared for several typical situations. Although Rasch estimates are expected to be more accurate than beta-binomial estimates, differences among them are found not to be substantial in a number of practical situations. Under relatively general conditions and when test retaking is permitted, the probability of making a false negative error is zero. Under the same situation, and given that an examinee is a true nonmaster, the conditional probability of making a false positive error for this examinee is one.
Comparing the Variances of Two Dependent Groups
Authors: Wilcox Rand R.
Published: March 2008

Let X and Y be dependent random variables with variances σ2x and σ2y. Recently, McCulloch (1987) suggested a modification of the Morgan-Pitman test of Ho: σ2x=σ2y But, as this paper describes, there are situations where McCulloch’s procedure is not robust. A subsample approach, similar to the Box-Scheffe test, is also considered and found to give conservative results, in terms of Type I errors, for all situations considered, but it yields relatively low power. New results on the Sandvik-Olsson procedure are also described, but the procedure is found to be nonrobust in situations not previously considered, and its power can be low relative to the two other techniques considered here. A modification of the Morgan-Pitman test based on the modified maximum likelihood estimate of a correlation is also considered. This last procedure appears to be robust in situations where the Sandvik-Olsson (1982) and McCulloch procedures are robust, and it can have more power than the Sandvik-Olsson. But it too gives unsatisfactory results in certain situations. Thus, in terms of power, McCulloch’s procedure is found to be best, with the advantage of being simple to use. But, it is concluded that, in terms of controlling both Type I and Type II errors, a satisfactory solution does not yet exist.
Critical Values for Two Multiple Comparison Procedures Based on the Studentized Range Distribution
Authors: Ramsey Philip P. , Ramsey Patricia P.
Published: March 2008

A table of critical values is provided for simple and accurate testing of the Welsch step-down procedure and the Shaffer-Welsch procedure. The advantages and use of the table are presented and discussed
Noniterative Factor Analysis Estimators, With Algorithms for Subset and Instrumental Variable Selection
Authors: Cudeck Robert
Published: March 2008

Noniterative estimators of the unrestricted factor analysis model have been developed by, among others, Hägglund (1982) and Ihara and Kano (1986) that are consistent and very efficient computationally. Whereas each of these methods has several desirable properties, both require a subjective decision regarding the selection of subsets of variables that are needed to compute estimates of the parameters. An algorithm called PACE, based on an application of the sweep operator, is presented that automatically selects subsets of variables used for the Ihara-Kano estimator. A second algorithm initially presented by Du Toit (1986) is also described that automatically selects reference variables used in Hägglund’s Fabin estimators. A Monte Carlo experiment is reviewed that compares the relative performance of these estimators in addition to several others. Both new methods performed well in this experiment. Their relative merits on other criteria are discussed.
The Minimally Important Difference Significant Criterion for Sample Size
Authors: Harris Richard J. , Quade Dana
Published: March 2008

For a wide range of tests of single-df hypotheses, the sample size needed to achieve 50% power is readily approximated by setting N such that a significance test conducted on data that fit one’s assumptions perfectly just barely achieves statistical significance at one’s chosen alpha level. If the effect size assumed in establishing one’s N is the minimally important effect size (i.e., that effect size such that population differences or correlations smaller than that are not of any practical or theoretical significance, whether statistically significant or not), then 50% power is optimal, because the probability of rejecting the null hypothesis should be greater than .5 when the population difference is of practical or theoretical significance but lower than .5 when it is not. Moreover, the actual power of the test in this case will be considerably higher than .5, exceeding .95 for a population difference two or more times as large as the minimally important difference (MID). This minimally important difference significant (MIDS) criterion extends naturally to specific comparisons following (or substituting for) overall tests such as the ANOVA F and chi-square for contingency tables, although the power of the overall test (i.e., the probability of finding some statistically significant specific comparison) is considerably greater than .5 when the MIDS criterion is applied to the overall test. However, the proper focus for power computations is one or more specific comparisons (rather than the omnibus test), and the MIDS criterion is well suited to setting sample size on this basis. Whereas Nmids(the sample size specified by the MIDS criterion) is much too small for the case in which we wish to prove the modified H0 that there is no important population effect, it nonetheless provides a useful metric for specifying the necessary sample size. In particular, the sample size needed to have a 1 – α probability that the (1 − α)-level confidence interval around one’s population parameter includes no important departure from H0 is four times Nmids when H0 is true and approximately [4/(1 – b)2].NMIDS when b (the ratio between the actual population difference and the minimally important difference) is between zero and unity. The MIDS criterion for sample size provides a useful alternative to the methods currently most commonly employed and taught.
Statistical Methods for Analyzing Collapsibility in Regression Models
Authors: Clogg Clifford C. , Petkova Eva , Shihadeh Edward S.
Published: March 2008

We give a unified treatment of statistical methods for assessing collapsibility in regression problems, including some possible extensions to the class of generalized linear models. Terminology is borrowed from the contingency table area where various methods for assessing collapsibility have been proposed. Our procedures, however, can be motivated by considering extensions, and alternative derivations, of common procedures for omitted-variable bias in linear regression. Exact tests and interval estimates with optimal properties are available for linear regression with normal errors, and asymptotic procedures follow for models with estimated weights. The methods given here can be used to compareβ1 and β2 in the common setting where the response function is first modeled asXβ1(reduced model) and then asXβ2+Zγ(full model), withZ a vector of covariates omitted from the reduced model. These procedures can be used in experimental settings (X= randomly asigned treatments,Z= covariates) or in nonexperimental settings where two models viewed as alternative behavioral or structural explanations are compared (one model withX only, another model withX andZ).
Chapter 3: Scaling Procedures in NAEP
Authors: Mislevy Robert J. , Johnson Eugene G. , Muraki Eiji
Published: March 2008

Scale-score reporting is a recent innovation in the National Assessment of Educational Progress (NAEP). With scaling methods, the performance of a sample of students in a subject area or subarea can be summarized on a single scale even when different students have been administered different exercises. This article presents an overview of the scaling methodologies employed in the analyses of NAEP surveys beginning with 1984. The first section discusses the perspective on scaling from which the procedures were conceived and applied. The plausible values methodology developed for use in NAEP scale-score analyses is then described, in the contexts of item response theory and average response method scaling. The concluding section lists milestones in the evolution of the plausible values approach in NAEP and directions for further improvement.
Chapter 4: Item Response Theory Scale Linking in NAEP
Authors: Yamamoto Kentaro , Mazzeo John
Published: March 2008

In educational assessments, it is often necessary to compare the performance of groups of individuals who have been administered different forms of a test. If these groups are to be validly compared, all results need to be expressed on a common scale. When assessment results are to be reported using an item response theory (IRT) proficiency metric, as is done for the National Assessment of Educational Progress (NAEP), establishing a common metric becomes synonymous with expressing IRT item parameter estimates on a common scale. Procedures that accomplish this are referred to here as scale linking procedures. This chapter discusses the need for scale linking in NAEP and illustrates the specific procedures used to carry out the linking in the context of the major analyses conducted for the 1990 NAEP mathematics assessment.
Chapter 5: Population Inferences and Variance Estimation for NAEP Data
Authors: Johnson Eugene G. , Rust Keith F.
Published: March 2008

In the National Assessment of Educational Progress (NAEP), population inferences and variance estimation are based on a randomization-based perspective where the link between the observed data and the population quantities of interest is given by the distribution of potential values of estimates over repeated samples from the same population using the identical sample design. Because NAEP uses a complex sample design, many of the assumptions underlying traditional statistical analyses are violated, and, consequently, analysis procedures must be adjusted to appropriately handle the structure of the sample. In this article, we discuss the use of sampling weights in deriving population estimates and consider the effect of nonresponse and undercoverage on those estimates. We also discuss the estimation of sampling variability from complex sample surveys, concentrating on the jackknife repeated replication procedure—the variance estimation procedure used by NAEP—and address the use of a simple approximation to sampling variability. Finally, we discuss measures of the stability of variance estimates.
Chapter 8: Applications and Extensions of NAEP Concepts and Technology
Authors: Rock Donald A. , Nelson Jennifer
Published: March 2008

The National Assessment of Educational Progress (NAEP) has consistently pioneered new assessment methods in conjunction with developing the psychometric methodologies underlying them. Several NAEP developments—such as complex matrix item sampling designs, the introduction of performance-based items in large-scale assessments, vertical scaling, and an intelligent computer system that produces unique assessment reports for participating jurisdictions in the NAEP Trial State Assessment program—are presented in this chapter along with a discussion of their extensions and applications to other current and future assessment projects.
Summarizing Monte Carlo Results in Methodological Research
Authors: Harwell Michael R.
Published: March 2008

Monte Carlo studies provide information that can assist researchers in selecting a statistical test when underlying assumptions of the test are violated. Effective use of this literature is hampered by the lack of an overarching theory to guide the interpretation of Monte Carlo studies. The problem is exacerbated by the impressionistic nature of the studies, which can lead different readers to different conclusions. These shortcomings can be addressed using meta-analytic methods to integrate the results of Monte Carlo studies. Quantitative summaries of the effects of assumption violations on the Type I error rate and power of a test can assist researchers in selecting the best test for their data. Such summaries can also be used to evaluate the validity of previously published statistical results. This article provides a methodological framework for quantitatively integrating Type I error rates and power values for Monte Carlo studies. An example is provided using Monte Carlo studies of Bartlett’s (1937) test of equality of variances. The importance of relating meta-analytic results to exact statistical theory is emphasized.
Meta-Analysis: Literature Synthesis or Effect-Size Surface Estimation?
Authors: Rubin Donald B.
Published: March 2008

A traditional meta-analysis can be thought of as a literature synthesis, in which a collection of observed studies is analyzed to obtain summary judgments about overall significance and size of effects. Many aspects of the current set of statistical tools for meta-analysis are highly useful—for example, the development of clear and concise effect-size indicators with associated standard errors. I am less happy, however, with more esoteric statistical techniques and their implied objects of estimation (i.e., their estimands) which are tied to the conceptualization of average effect sizes, weighted or otherwise, in a population of studies. In contrast to these average effect sizes of literature synthesis, I believe that the proper estimand is an effect-size surface, which is a function only of scientifically relevant factors, and which can only be estimated by extrapolating a response surface of observed effect sizes to a region of ideal studies. This effect-size surface perspective is presented and contrasted with the literature synthesis perspective. The presentation is entirely conceptual. Moreover, it is designed to be provocative, thereby prodding researchers to rethink traditional meta-analysis and ideally stimulating meta-analysts to attempt effect-surface estimations.
Refinements of Stout’s Procedure for Assessing Latent Trait Unidimensionality
Authors: Nandakumar Ratna , Stout William
Published: March 2008

This article provides a detailed investigation of Stout’s statistical procedure (the computer program DIMTEST) for testing the hypothesis that an essentially unidimensional latent trait model fits observed binary item response data from a psychological test. One finding was that DIMTEST may fail to perform as desired in the presence of guessing when coupled with many high-discriminating items. A revision of DIMTEST is proposed to overcome this limitation. Also, an automatic approach is devised to determine the size of the assessment subtests. Further, an adjustment is made on the estimated standard error of the statistic on which DIMTEST depends. These three refinements have led to an improved procedure that is shown in simulation studies to adhere closely to the nominal level of signficance while achieving considerably greater power. Finally, DIMTEST is validated on a selection of real data sets.
Marginal Maximum Likelihood Estimation for the Ordered Partition Model
Authors: Wilson Mark , Adams Raymond J.
Published: March 2008

This article describes a marginal maximum likelihood (MML) estimation algorithm for Wilson’s (1990) ordered partition model (OPM), a measurement model that does not require the set of available responses to assessment tasks to be fully ordered. The model and its estimation algorithm are illustrated through the analysis of an example data set. In the example, we use the ordered partition model to compare a set of alternative scoring schemes for open-ended science items.
Sensitivity Analysis for Fixed Effects in the Hierarchical Model: A Gibbs Sampling Approach
Authors: Seltzer Michael H.
Published: March 2008

Many recent applications of the two-level hierarchical model (HM) have focused on drawing inferences concerning fixed effects—that is, structural parameters in the Level 2 model that capture the way Level 1 parameters (e.g., children’s rates of cognitive growth, within-school regression coefficients) vary as a function of Level 2 characteristics (e.g., children’s home environments and educational experiences; school policies, practices, and compositional characteristics). Under standard assumptions of normality in the HM, point estimates and intervals for fixed effects may be sensitive to outlying Level 2 units (e.g., a child whose rate of cognitive growth is unusually slow or rapid, a school at which students achieve at an unusually high level given their background characteristics, etc.). A Bayesian approach to studying the sensitivity of inferences to possible outliers involves recalculating the marginal posterior distributions of parameters of interest under assumptions of heavy tails, which has the effect of downweighting extreme cases. The goal is to study the extent to which posterior means and intervals change as the degree of heavy-tailedness assumed increases. This strategy is implemented in the HM setting via a new Monte Carlo technique termed Gibbs sampling ( Gelfand & Smith, 1990 ; cf Tanner & Wong, 1987 ) and is illustrated through reanalyses of the data from a study of vocabulary growth in children ( Huttenlocher, Haight, Bryk, Seltzer, & Lyons, 1991 ). The Gibbs sampling approach presented builds on the work of Dempster, Laird, and Rubin (1980) ; Little and Rubin (1987) ; and Lange, Little, and Taylor (1989) concerning the use of the t distribution in robust statistical estimation. Extensions of this approach are discussed in the final section of the article.
On a Criterion for Combining Correlational Data
Authors: Viana Marlos A. G.
Published: March 2008

To test the hypothesis of a positive correlation, consider the following plans: Plan A consists of combining k >1 independent sample correlations, each one based on N bivariate observations and a common correlation parameter. Plan B consists of observing a single correlation based on the same total kN bivariate observations. Plan C also requires the total of kN observations and differs from A only in that the sample sizes are not homogeneous. It is shown that B is the relatively optimal plan, whether the corresponding testing procedures are compared in terms of relative large sample power of the test, based on a weighted sum of the corresponding Fisher’s z transformations, or in terms of their Bayes relative risks; the interpretation is that correlations “avoid” averaging. In contrast, it is also shown that certain statistics “prefer” averaging, while others are “indifferent.” A characterization of these classes was obtained in terms of the order of magnitude of the rate of change in the signal-to-noise ratio of the corresponding statistic, with respect to[Formula: see text] as the sample size N increases.
Empirical Bayes Analysis of Families of Survival Curves: Applications to the Analysis of Degree Attainment
Authors: Braun Henry I. , Zwick Rebecca
Published: March 2008

We present a novel approach to the empirical Bayes analysis of aggregated survival data from different groups of subjects. The method is based on a contingency table representation of the data and employs transformations to permit the use of normal priors. In contrast to the case of a single survival curve, the empirical Bayes analysis of families of such curves leads to estimates which offer a qualitative improvement over classical estimates based on the ratio of occurrence to exposure rates. This method is illustrated with data on the attainment of the doctoral degree from three major universities.
Testing Repeated Measures Hypotheses When Covariance Matrices are Heterogeneous
Authors: Keselman H. J. , Carriere Keumhee Chough , Lix Lisa M.
Published: March 2008

For balanced designs, degrees of freedom-adjusted univariate F tests or multivariate test statistics can be used to obtain a robust test of repeated measures main and interaction effect hypotheses even when the assumption of equality of the covariance matrices is not satisfied. For unbalanced designs, however, covariance heterogeneity can seriously distort the rates of Type I error of either of these approaches. This article shows how a multivariate approximate degrees of freedom procedure based on Welch (1947 , 1951 )- James (1951 , 1954) , as simplified by Johansen (1980) , can be applied to the analysis of unbalanced repeated measures designs without assuming covariance homogeneity. Through Monte Carlo methods, we demonstrate that this approach provides a robust test of the repeated measures main effect hypothesis even when the data are obtained from a skewed distribution. The Welch-James approach also provides a robust test of the interaction effect, provided that the smallest of the unequal group sizes is five to six times the number of repeated measurements minus one or provided that a reduced level of significance is employed.
A Crossed Random Effects Model for Unbalanced Data With Applications in Cross-Sectional and Longitudinal Research
Authors: Raudenbush Stephen W.
Published: March 2008

Hierarchical linear models have found widespread application when the data have a nested structure—for example, when students are nested within classrooms (a two-level nested structure) or students are nested within classrooms and classrooms are nested within schools (a three-level nested structure). Often, however, the data will have a more complex nested structure. In Example 1, students are nested within both neighborhoods and schools; however, a school can draw students from multiple neighborhoods, and a neighborhood can send students to multiple schools. In Example 2, children are nested within classrooms during the first year of the study; however, each child finds himself or herself with a new teacher and a new set of classmates during each subsequent year of the study. By combining Lindley and Smith’s (1972) concepts of exchangeability between and within regressions, this article formulates a crossed random effects model that applies to such data, provides maximum likelihood estimates via the EM algorithm, and illustrates application to study (a) neighborhood and school effects on educational attainment in Scotland and (b) classroom effects on mathematics learning during the primary years in the United States.
D-Optimal Sequential Sampling Designs for Item Response Theory Models
Authors: Berger Martijn P.F.
Published: March 2008

The selection of optimal designs in IRT models encounters at least two problems. The first problem is that Fisher’s information matrix is generally not independent of the values of the IRT parameters, and the second problem is that the design points are unknown parameters and have to be estimated together with the other parameters. In this study, these two problems are taken care of by a sequential design procedure. This procedure is a modification of a D-optimality procedure proposed by Wynn (1970) . The results show that this algorithm leads to consistent estimates and that errors in selecting the abilities generally do not affect optimality very much.
A Monte Carlo Comparison of Seven ε-Adjustment Procedures in Repeated Measures Designs With Small Sample Sizes
Authors: Quintana Stephen M. , Maxwell Scott E.
Published: March 2008

The purpose of this study was to evaluate seven univariate procedures for testing omnibus null hypotheses for data gathered from repeated measures designs. Five alternate approaches are compared to the two more traditional adjustment procedures (Geisser and Greenhouse’s ε̂ and Huynh and Feldt’s ε̃), neither of which may be entirely adequate when sample sizes are small and the number of levels of the repeated factors is large. Empirical Type I error rates and power levels were obtained by simulation for conditions where small samples occur in combination with many levels of the repeated factor. Results suggested that alternate univariate approaches were improvements to the traditional approaches. One alternate approach in particular was found to be most effective in controlling Type I error rates without unduly sacrificing power.
The Descriptive Use of Absolute Differences Between Pairs of Scores With a Common Mean and Variance
Authors: McGraw Kenneth O. , Wong S. P.
Published: March 2008

Researchers concerned with the degree of similarity in pairs of scores sharing a common mean and variance can express similarity in terms of the absolute differences between the scores. This article discusses the distribution of absolute differences between pairs of normally distributed scores having a common mean and variance and illustrates procedures for calculating moments and areas within this distribution.
A Monte Carlo Study on the Performance of a Corrected Formula for ɛ̃ Suggested by Lecoutre
Authors: Chen Ru San , Dunlap William P.
Published: March 2008

Lecoutre (1991) has pointed out an error in the Huynh and Feldt (1976) formula for ɛ̃ used to adjust the degree of freedom for an approximate test in repeated measures designs with two or more independent groups. The present simulation study confirms that Lecoutre’s corrected ɛ̃ yields less biased estimation of population ɛ and reduces Type I error rates when compared to Huynh and Feldt’s (1976) ɛ̃. The increased accuracy in Type I errors for group-treatment interactions may become substantial when sample sizes are close to the number of treatment levels.
Teacher’s Corner: The One-Sample Runs Test: A Category of Exception
Authors: Mogull Robert G.
Published: March 2008

The popular One-Sample Runs Test is used to identify a nonrandom pattern in a sequence of dichotomous elements. Although the test is generally effective in the identification of patterns, it is demonstrated to be incapable of signaling departures from randomness with run lengths of two. Further-more, with run lengths of two, increasing the sample size reduces the power of the test. Run lengths strictly of two, therefore, generate a unique category of anomaly in the test’s overall performance.
Test Equating From Biased Samples, With Application to the Armed Services Vocational Aptitude Battery
Authors: Little Roderick J. A. , Rubin Donald B.
Published: March 2008

The problem of equating a new standardized test to an old reference test is considered when the samples for equating are not randomly selected from the target population of test takers. Two problems with equating from biased samples are distinguished: (a) bias in the equating function arising from nonrandom selection of the equating sample, and (b) excessive variance in the equating function at scores that are relatively underrepresented in the equating sample relative to the target population. A theorem is presented that suggests that bias may not be a major problem for equating, even when the marginal distributions of scores are distorted by selection. Empirical analysis of data for equating the Armed Services Vocational Aptitude Battery (ASVAB) based on samples of recruits and applicants supports this contention. Analysis of ASVAB data also indicates that excessive variance in the equating function is a more serious issue. Variance-reducing methods, which smooth the test score distributions using extended beta binomial and loglinear polynomial models before equating by the equipercentile method, are presented. Empirical evidence suggests that these smoothing models are successful and yield equating functions that improve on both equipercentile and linear equating of the raw scores.
Longitudinal Data Analysis Examples With Random Coefficient Models
Authors: Rogosa David , Saner Hilary
Published: March 2008

Longitudinal panel data examples are used to illustrate estimation methods for individual growth curve models. These examples constitute one of the basic multilevel analysis settings, and they are used to illustrate issues and concerns in the application of hierarchical modeling estimation methods, specifically, the widely advertised HLM procedures of Bryk and Raudenbush. One main expository purpose is to demystify these analyses by showing equivalences with simpler approaches. Perhaps more importantly, these equivalences indicate useful data analytic checks and diagnostics to supplement the multilevel estimation procedures. In addition, we recommend the general use of standardized canonical examples for the checking and exposition of the various multilevel procedures; as part of this effort, methods for the construction of longitudinal data examples with known structure are described.
Hierarchical Models for Educational Data: An Overview
Authors: Morris Carl N.
Published: March 2008

The use of hierarchical models in statistical applications, and for educational data, is a promising but still underutilized approach. However, because these models are more complicated than many standard methods, it is important that we, as users and developers, not rush to use them before we understand them. We emphasize here, in support of the views on hierarchical models expressed in the 3 preceding papers by Draper, by Rogosa and Saner, and by de Leeuw and Kreft, the need to not diminish hard thinking about data and iterative model checking when fitting hierarchical models, the need for more and better software, the need to test methods to assure their proper calibration, and the need to produce supporting materials to aid analysts and users of hierarchical modeling methods.
Hierarchical Models and Social Sciences
Authors: Longford Nicholas T.
Published: March 2008

The view is presented that multilevel methods are just one element in a hypothetical complete analysis of observational data on human subjects. In most contexts several sources of uncertainty, in addition to those captured by a multilevel analysis, are present, and so the confidence placed in the results of a typical multilevel analysis is unrealistically optimistic. A “soft-ware-free” analysis of longitudinal data with rectangular design is outlined. Questions posed by the National Center for Education Statistics and elaborated by de Leeuw and Kreft are briefly discussed.