bluuebunny/crossref_metadata_2025 huggingface dataset:
Information Matrices in Latent-Variable Models
Authors: Mislevy Robert J. , Sheehan Kathleen M.
Published: March 2008

The Fisher, or expected, information matrix for the parameters in a latent-variable model is bounded from above by the information that would be obtained if the values of the latent variables could also be observed. The difference between this upper bound and the information in the observed data is the “missing information.” This paper explicates the structure of the expected information matrix and related information matrices, and characterizes the degree to which missing information can be recovered by exploiting collateral variables for respondents. The results are illustrated in the context of item response theory models, and practical implications are discussed.
A Nonparametric Test Statistic for the General Linear Model
Authors: Harwell Michael R. , Serlin Ronald C.
Published: March 2008

Puri and Sen (1969Puri and Sen (1985) presented a nonparametric test statistic based on a general linear model approach that is appropriate for testing a wide class of hypotheses. The two forms of this statistic, pure- and mixed-rank, differ according to whether the original predictor values or their ranks are used. Both forms permit the use of standard statistical packages to perform the analyses. The applicability of these statistics in testing a number of hypotheses is highlighted, and an example of their use is given. A simulation study for the multivariate-multiple-regression case is used to examine the distributional behavior of the pure- and mixed-rank statistics and an important competitor, the rank transformation of Conover and Iman (1981). The results suggest that the pure- and mixed-rank statistics are superior with respect to minimizing liberal Type I error rates, whereas the Conover and Iman statistic produces larger power values.
The Construction of Parallel Tests From IRT-Based Item Banks
Authors: Boekkooi-Timminga Ellen
Published: March 2008

The construction of parallel tests from IRT-based item banks is discussed. Tests are considered to be parallel whenever their information functions are identical. Simultaneous and sequential parallel test construction methods based on the use of 0–1 programming are examined for the Rasch and 3-parameter logistic model. Sequential methods construct the tests one after another; simultaneous methods construct them all at the same time. A heuristic procedure is used for solving the 0–1 programming problems. Satisfactory results are obtained, both in terms of the CPU-time needed and differences between the information functions of the parallel tests selected.
Comparing Expert and Trainee Assessments Using Regression Techniques
Authors: Sparks Ross S. , Ballantyne Roy
Published: March 2008

Models are constructed to compare trainee and expert assessment strategies. These models detect trainee differences in assessment in terms of location and scale, and detect the presence of bias towards particular groups of a population. Although the assessment may be evaluated by means of hypothesis tests, emphasis is placed upon the use of a criterion ( Mallows, 1973 ). The use of these procedures in assessment training programs is demonstrated in an application.
Simultaneous Optimization of Decisions Using a Linear Utility Function
Authors: Vos Hans J.
Published: March 2008

The purpose of this article is to simultaneously optimize decision rules for combinations of elementary decisions. With this approach, rules are found that make more efficient use of the data than could be achieved by optimizing these decisions separately. The framework for the approach is derived from Bayesian decision theory. To illustrate the approach, two elementary decisions (selection and mastery decisions) are combined into a simple decision network. A linear utility structure is assumed. Decision rules are derived both for quota-free and quota-restricted selection–mastery decisions in case of several subpopulations. An empirical example of instructional decision making in an individual study system concludes the article.
Box, Line, and Midgap Plots: Effects of Display Characteristics on the Accuracy and Bias of Estimates of Whisker Length
Authors: Stock William A. , Behrens John T.
Published: March 2008

Examined here were the accuracy and bias of estimates of whisker length based on box, line, and midgap plots. For each type of graph, a different sample of undergraduates viewed 48 single-plot graphs. For each plot, subjects were given the length of an interquartile spread and asked to estimate the length of a whisker. Plots varied in spatial orientation (horizontal or vertical), interquartile spread, the ratio of whisker length to interquartile spread, and whisker judged. Overall, estimates of whisker length for box and line plots were more accurate and less biased than those for midgap plots. Interquartile spread, the ratio of whisker length to interquartile spread, and the interaction of these two factors significantly influenced both accuracy and bias. Boxplots displayed a predicted pattern of over- and underestimation. We discuss how the present results complement related work of others (e.g., Cleveland & McGill, 1984 ) on depiction of quantity. We conclude that midgap plots are less optimal displays than box and line plots.
Sampling Probabilities for Aggregations, With Application to NELS:88 and Other Educational Longitudinal Surveys
Authors: Spencer Bruce D. , Foran Wendene
Published: March 2008

This article considers surveys where one can observe, after the sample is selected, that each member of the sample belongs to one or more aggregations. The population of aggregations is of interest, and we consider the probability that a given aggregation contains at least one sample member. An expression for the probability is derived as a function of population parameters, many of which can be known only if additional, costly data collection is undertaken. A variety of model-based estimators of those parameters is discussed, and their relative advantages and disadvantages are noted. The estimators are applied to a particular case, the National Educational Longitudinal Study of 1988 (NELS:88), in which the aggregations are tenth-grade schools. Evaluations of the estimators are presented. In NELS:88, a sample of eighth-grade schools and eighth-grade students in those schools was surveyed and 2 years later, when most of the students were in tenth grade, the students were resurveyed. If the tenth-grade school sample (i.e., the set of tenth-grade schools enrolling one or more sampled students) is a probabiliy sample, then we can make inferences to the population of tenth-grade schools. A requirement for a sample to be a probability sample is that the selection probabilities be nonzero for all units and be known for the selected units. This article discusses how those probabilities can be estimated from complete and incomplete data. Various estimators based on incomplete data are derived and empirically evaluated with data from a special test sample of schools and data from the Houston Independent School District.
Robustness of Yao’s, James’, and Johansen’s Tests Under Variance-Covariance Heteroscedasticity and Nonnormality
Authors: Algina James , Oshima Takako C. , Tang K. Linda
Published: March 2008

Type I error rates for Yao’s, James’ first order, James’ second order, and Johansen’s tests of equality of mean vectors for two independent samples were estimated for various conditions defined by the degree of heteroscedasticity and nonnormality (uniform, Laplace, t(5), beta (5, 1.5), exponential, and lognormal distributions). For these alternatives to Hotelling’s T2, variance-covariance homogeneity is not an assumption. Although the four procedures can be seriously nonrobust with exponential and lognormal distributions, they were fairly robust with the rest of the distributions. The performance of Yao’s test, James’ second order test, and Johansen’s test was slightly superior to the performance of James’ first order test.
An Omnibus Test for Systematic Changes in Judges’ Rankings
Authors: Critchlow Douglas E. , Verducci Joseph S.
Published: March 2008

Paired rankings arise when each subject in a study independently ranks a set of items, undergoes a treatment, and afterwards ranks the same set of items. For such data, a statistical test is proposed to detect if the subjects’ posttreatment rankings have moved systematically toward some unknown ranking or set of rankings. The null hypothesis for this test is that each subject’s post-treatment ranking is symmetrically distributed about his pretreatment ranking. The exact and asymptotic null distributions of the test statistic are simulated and compared, and the power of the test is studied. Using paired rankings from an experimental course in literary criticism, we also offer some graphical methods for representing such data that help us to interpret the test results.
Variable Importance in Multivariate Group Comparisons
Authors: Huberty Carl J. , Wisenbaker Joseph M.
Published: March 2008

Three views of relative variable contribution in a multivariate analysis of variance context are discussed: contribution to (a) latent construct definition, (b) linear discriminant function scores, and (c) grouping variable effects. It is argued that the third view is most meaningful. The problem of assessing the relative importance of a set of outcome variables is reviewed. Two numerical ranking methods are proposed. The bootstrap approach is used to compare the methods utilizing two real data sets. No definitive preference is advanced. The variable-ordering problem in two other multivariate analysis contexts is briefly discussed.
Chapter 7: Statistical and Psychometric Issues in the Measurement of Educational Achievement Trends: Examples From the National Assessment of Educational Progress
Authors: Zwick Rebecca
Published: March 2008

Like all studies involved in the assessment of trends in educational performance, the National Assessment of Educational Progress (NAEP) is confronted with an array of unresolved methodological and philosophical issues. One of the basic dilemmas faced by NAEP is how to measure performance change while remaining responsive to advances in curriculum and the technology of assessment. NAEP has become much more cautious about making seemingly insubstantial changes in the assessment because of the so-called NAEP reading anomaly—an apparently steep drop between 1984 and 1986 in estimated reading proficiency that was found to have resulted in part from changes in the order and context in which items appeared. Other issues that NAEP must consider in reporting performance trends are the effect of measurement scale indeterminacies and the ways in which interpretation of trend results can depend on the statistics that are selected for comparing proficiency distributions over time.
Bayesian Estimation of Normal Ogive Item Response Curves Using Gibbs Sampling
Authors: Albert James H.
Published: March 2008

The problem of estimating item parameters from a two-parameter normal ogive model is considered. Gibbs sampling (Gelfand & Smith, 1990) is used to simulate draws from the joint posterior distribution of the ability and item parameters. This method gives marginal posterior density estimates for any parameter of interest; these density estimates can be used to judge the accuracy of normal approximations based on maximum likelihood estimates. This simulation technique is illustrated using data from a mathematics placement exam.
The Analysis of Two-Factor Interactions in Fixed Effects Linear Models
Authors: Boik Robert J.
Published: March 2008

This article considers two related issues concerning the analysis of interactions in complex linear models. The first issue concerns the omnibus test for interaction. Apparently, it is not well known that the usual F test for interaction can be replaced, in many applications, by a test that is more powerful against a certain class of alternatives. The competing test is based on the maximal product interaction contrast F statistic and achieves its power advantage by focusing solely on product contrasts. The maximal product interaction F test is reviewed and three new results are reported: (a) An extended table of exact critical values is computed, (b) a table of moment functions useful for approximating the p-value corresponding to an observed maximal F statistic is computed, and (c) a simulation study concerning the null distribution of the maximal F statistic when data are unbalanced or covariates are present is reported. It is conjectured that lack of balance or presence of covariates has no effect on the null distribution. The simulation results support the conjecture. The second issue concerns follow-up tests when the omnibus test is significant. It appears that researchers, in general, do not perform coherent follow-up tests on interactions. To make it easier for researchers to do so, an exposition on the use of product interaction contrasts and partial interactions in complex fixed-effects models is provided. The recommended omnibus and follow-up tests are illustrated on an educational data set analyzed using SAS ( SAS Institute, 1988 ) and SPSS (1990) .
The Validity of the GMAT for the Prediction of Grades in Doctoral Study in Business and Management: An Empirical Bayes Approach
Authors: Zwick Rebecca
Published: March 2008

A validity study was conducted to examine the degree to which GMAT scores and undergraduate grade-point average (UGPA) could predict first-year average (FYA) and final grade-point average in doctoral programs in business. A variety of empirical Bayes regression models, some of which took into account possible differences in regressions across schools and cohorts, were investigated for this purpose. Indexes of model fit showed that the most parsimonious model, which did not allow for school or cohort effects, was just as useful for prediction as the more complex models. The three preadmissions measures were found to be associated with graduate school grades, though to a lesser degree than in MBA programs. The prediction achieved using UGPA alone as a predictor tended to be more accurate than that obtained using GMAT verbal (GMATV) and GMAT quantitative (GMATQ) scores together. Including all three predictors was more effective than using only UGPA. The most likely explanation for the lower levels of prediction than in MBA programs is that doctoral programs tend to be more selective. Within-school means on GMATV, GMATQ, UGPA, and FYA were higher than those found in MBA validity studies; within-school standard deviations on FYA tended to be smaller. Among these very select, academically competent doctoral students, highly accurate prediction of grades may not be possible.
Maximizing Power in Generalizability Studies Under Budget Constraints
Authors: Marcoulides George A.
Published: March 2008

Generalizability theory provides a framework for examining the dependability of behavioral measurements. When designing generalizability studies, two important statistical issues are generally considered: power and measurement error. Control over power and error of measurement can be obtained by manipulation of sample size and/or test reliability. In generalizability theory, the mean error variance is an estimate that takes into account both these statistical issues. When limited resources are available, determining an optimal measurement design is not a simple task. This article presents a methodology for minimizing mean error variance in generalizability studies when resource constraints are imposed.
Standard Errors and Sample Sizes for Two-Level Research
Authors: Snijders Tom A. B. , Bosker Roel J.
Published: March 2008

The hierarchical linear model approach to a two-level design is considered, some variables at the lower level having fixed and others having random regression coefficients. An approximation is derived to the covariance matrix of the estimators of the fixed regression coefficients (for variables at the lower and the higher level) under the assumption that the sample sizes at either level are large enough. This covariance matrix is expressed as a function of parameters occurring in the model. If a research planner can make a reasonable guess as to these parameters, this approximation can be used as a guide to the choice of sample sizes at either level.
Testing Variances in Psychological and Educational Research
Authors: Ramsey Philip H.
Published: March 2008

A review of the literature has shown that robust procedures for testing variances have become available. The two best procedures are one proposed by O’Brien (1981) and another by Brown and Forsythe (1974). An examination of these procedures for a wide variety of populations confirms their robustness and indicates that optimal power can usually be obtained by applying a test for kurtosis to aid in the decision between these two procedures.
A New and Simpler Approximation for ANOVA Under Variance Heterogeneity
Authors: Alexander Ralph A. , Govern Diane M.
Published: March 2008

A new approximation is proposed for testing the equality of k independent means in the face of heterogeneity of variance. Monte Carlo simulations show that the new procedure has Type I error rates that are very nearly nominal and Type II error rates that are quite close to those produced by James’s (1951) second-order approximation. In addition, it is computationally the simplest approximation yet to appear, and it is easily applied to Scheffé (1959) -type multiple contrasts and to the calculation of approximate tail probabilities.
Stepwise and Simultaneous Multiple Comparison Procedures of Repeated Measures’ Means
Authors: Keselman H. J.
Published: March 2008

Stepwise multiple comparison procedures (MCPs) for repeated measures’ means based on the methods of Hayter (1986) , Hochberg (1988) , Peritz (1970) , Ryan (1960) - Welsch (1977a) , Shaffer (1979 , 1986) , and Welsch (1977a) were compared for their overall familywise rates of Type I error when multisample sphericity and multivariate normality were not satisfied. Robust stepwise procedures were identified by Keselman, Keselman, and Shaffer (1991) with respect to three definitions of power. On average, Welsh’s (1977a) step-up procedure was found to be the most powerful MCP.
Two Aspects of the Simplex Model: Goodness of Fit to Linear Growth Curve Structures and the Analysis of Mean Trends
Authors: Mandys Frantisek , Dolan Conor V. , Molenaar Peter C. M.
Published: March 2008

This article has two objectives. The first is to investigate in greater detail the finding of Rogosa and Willett that the quasi-Markov simplex model fits a linear growth curve covariance structure. It is found that under various circumstances the quasi-Markov simplex model is rejected. Furthermore, the procedure is reversed by fitting the linear growth curve to quasi-Markov simplex covariance structure. It is found that the linear growth curve, like the quasi-Markov simplex, is not always rejected even though the model is formally incorrect. The second objective of this article is to present a quasi-Markov simplex model with structured means. This model, like the linear growth curve model with structured means, is based on the assumption that the variation in means and individual differences are attributable to the same causal agents. We argue that this assumption should be tested explicitly. An example is given.
Nonparametric Competitors to the Two-Way ANOVA
Authors: Toothaker Larry E. , Newman De
Published: March 2008

The ANOVA F and several nonparametric competitors for two-way designs were compared for empirical α and power. Simulation of 2 × 2, 2 × 4, and 4 × 4 designs was done with cell sizes of 5 and 10 when sampling from normal, exponential, and mixed normal distributions. Conservatism of both α and power in the presence of other nonnull effects was seen in the tests due to Puri and Sen (1985) and, to a lesser degree, in the rank transform tests ( Conover & Iman, 1981 ). Tests by McSweeney (1967) and Hettmansperger (1984) had liberal α for some designs and distributions, especially for small n. The ANOVA F suffers from conservative α and power for the mixed normal distribution, but it is generally recommended.
Type I Error Rates for Welch’s Test and James’s Second-Order Test Under Nonnormality and Inequality of Variance When There Are Two Groups
Authors: Algina James , Oshima T. C. , Lin Wen-Ying
Published: March 2008

Type I error rates were estimated for three tests that compare means by using data from two independent samples: the independent samples t test, Welch’s approximate degrees of freedom test, and James’s second-order test. Type I error rates were estimated for skewed distributions, equal and unequal variances, equal and unequal sample sizes, and a range of total sample sizes. Welch’s test and James’s test have very similar Type I error rates and tend to control the Type I error rate as well or better than the independent samples t test does. The results provide guidance about the total sample sizes required for controlling Type I error rates.
Teacher’s Corner: Origin of the Scaling Constant d = 1.7 in Item Response Theory
Authors: Camilli Gregory
Published: March 2008

The scaling constant d = 1.702 used in item response theory minimizes the maximum difference between the normal and logistic distribution functions. The theoretical and numerical derivation of d given by Haley (1952) is briefly recapitulated to provide access to curious researchers, instructors, and students.
Teacher’s Corner: A Note on Interpreting an R2 Value
Authors: Huberty Carl J.
Published: March 2008

It is suggested that the standard traditional null hypothesis of ρ2 = 0 be reconsidered in favor of ρ2 = E(R2). The test of the latter hypothesis is discussed. Also discussed is an effect-size index that is the difference between an adjusted R2 and E(R2).
Teacher’s Corner: A Note on Modified Rank Correlation
Authors: Zimmerman Donald W.
Published: March 2008

The Spearman rank correlation method separately assigns ranks to sample values of each of two variables, X and Y, and substitutes squared differences between the ranks into a computational formula derived from Pearson correlation. The present note examines an alternate procedure which separately transforms the X and Y values to standard scores, ranks the combined standard scores in a single sequence, and calculates the Pearson correlation between the ranks corresponding to the initial scores. A simulation study reveals that significance tests of correlation based on this method effectively control Type I and Type II error probabilities and are slightly more powerful than the Spearman method for normal and various nonnormal distributions and for sample sizes ranging from 8 to 30. This note discusses some advantages of the modified procedure.
A Maximum Test for Scale: Type I Error Rates and Power
Authors: Algina James , Blair R. Clifford , Coombs William T.
Published: March 2008

A maximum test in which the test statistic is the more extreme of the Brown-Forsythe and O’Brien’s test statistics is developed. Estimated Type I error rates and power are presented for the Brown-Forsythe test, O’Brien’s test, and the maximum test. For the conditions included in the study, Type I error rates for the maximum test are near the nominal level. In all conditions, the power of the maximum test tended to be equal to or greater than that of the test—O’Brien or Brown-Forsythe—that had the larger power.
The Impact of BIB Spiraling-Induced Missing Data Patterns on Goodness-of-Fit Tests in Factor Analysis
Authors: Kaplan David
Published: March 2008

This article considers the impact of missing data arising from balanced incomplete block (BIB) spiraled designs on the chi-square goodness-of-fit test in factor analysis. Specifically, data arising from BIB designs possess a unique pattern of missing data that can be characterized as missing completely at random (MCAR). Standard approaches to factor analyzing such data rest on forming pairwise available case (PAC) covariance matrices. Developments in statistical theory for missing data show that PAC covariance matrices may not satisfy Wishart distribution assumptions underlying factor analysis, thus impacting tests of model fit. One approach, advocated by Muthén, Kaplan, and Hollis (1987) for handling missing data in structural equation modeling, is proposed as a possible solution to these problems. This study compares the new approach to the standard PAC approach in a Monte Carlo framework. Results show that tests of goodness-of-fit are very sensitive to PAC approaches even when data are MCAR, as is the case for BIB designs. The new approach is shown to outperform the PAC approach for continuous variables and is comparatively better for dichotomous variables.
Improved Repeated Measures Stepwise Multiple Comparison Procedures
Authors: Keselman H. J. , Lix Lisa M.
Published: March 2008

Approximate degrees of freedom omnibus and pairwise test statistics of Johansen (1980) and Keselman, Keselman, and Shaffer (1991) , respectively, were used with numerous stepwise multiple comparison procedures (MCPs) to perform pairwise contrasts on repeated measures means. The MCPs were compared for their overall familywise rates of Type I error and for their sensitivity to detect true pairwise differences among means when multisample sphericity and multivariate normality assumptions were not satisfied. Results indicated that multiple range procedures which were modified according to the method described by Duncan (1957) were always robust with respect to Type I errors and were at least as powerful as the unmodified range procedures, and could result in increases in power as large as 22%. Overall, the Welsch (1977a) step-up, Peritz-Duncan ( Peritz, 1970 ), and Ryan-Welsch-Duncan ( Ryan, 1960 ; Welsch, 1977a ) multiple range procedures were found to be most powerful.
Asymptotic Standard Error of Equipercentile Equating
Authors: Liou Michelle , Cheng Philip E.
Published: March 2008

We propose simplified formulas for computing the standard errors of equiper-centile equating for continuous and discrete test scores. The suggested formulas are conceptually simple and easily extended to more complicated equating designs such as chained equipercentile equating, smoothed equipercentile equating, and equating using the frequency estimation method. Results from an empirical study indicate that the derived formulas work reasonably well for samples with moderate sizes (e.g., 1,000 examinees).
The Use of Fisher’s Z in Schmidt-Hunter–type Meta-Analyses
Authors: Law Kenneth S.
Published: March 2008

Over the past few years, there has been some dispute as to whether Fisher’s Z or the Pearson correlation (r) should be used in Schmidt-Hunter–type meta-analyses. The two major reasons that Z has not been recommended are the possibly larger positive bias in estimating mean population correlation (Mρ) and the problem of estimating the standard deviation of population correlations (SDρ). In this study, two new methods of estimating Mρ and SDρ by using Z are suggested and tested by Monte Carlo simulations. The results show no consistent advantage to using r instead of Z or vice versa in estimating both Mρ and SDρ In fact, the estimated Mρ and SDρ from all three methods were highly similar and were almost identical when rounded off.
The Estimation of School Effects
Authors: Raudenbush Stephen W. , Willms JDouglas
Published: March 2008

The increasing public demand to hold schools accountable for their effects on student outcomes lends urgency to the task of clarifying statistical issues pertaining to studies of school effects. This article considers the specification and estimation of school effects, the variability of effects across schools, and the proportion of variation in student outcomes attributable to differences in school context and practice. We present a statistical model that defines two different types of school effect: one appropriate for parents choosing schools for their children, the second for agencies evaluating school practice. Studies of both types of effect are viewed as quasi-experiments posing formidable obstacles to valid causal inference. A multilevel decomposition of variance within and between schools has important and perhaps counterintuitive implications for school evaluation. The potential for unbiased estimation depends on the type of effect under consideration because the two types of school effect have markedly different data requirements. Commonly used estimators of each effect are shown to be biased and, in some cases, inconsistent. Analyses of survey data from Scotland illustrate the recommended techniques. We conclude with a brief discussion of the role of school evaluation in a broader agenda of research in support of school improvement.
Bayesian Analysis in Applications of Hierarchical Models: Issues and Methods
Authors: Seltzer Michael H. , Wong Wing Hung , Bryk Anthony S.
Published: March 2008

In applications of hierarchical models (HMs), a potential weakness of empirical Bayes estimation approaches is that they do not to take into account uncertainty in the estimation of the variance components (see, e.g., Dempster, 1987 ). One possible solution entails employing a fully Bayesian approach, which involves specifying a prior probability distribution for the variance components and then integrating over the variance components as well as other unknowns in the HM to obtain a marginal posterior distribution of interest (see, e.g., Draper, 1995 ; Rubin, 1981 ). Though the required integrations are often exceedingly complex, Markov-chain Monte Carlo techniques (e.g., the Gibbs sampler) provide a viable means of obtaining marginal posteriors of interest in many complex settings. In this article, we fully generalize the Gibbs sampling algorithms presented in Seltzer (1993) to a broad range of settings in which vectors of random regression parameters in the HM (e.g., school means and slopes) are assumed multivariate normally or multivariate t distributed across groups. Through analyses of the data from an innovative mathematics curriculum, we examine when and why it becomes important to employ a fully Bayesian approach and discuss the need to study the sensitivity of results to alternative prior distributional assumptions for the variance components and for the random regression parameters.
The Effects of Display Characteristics on the Bias of Estimates of Whisker Length of Regular and Notched Boxplots
Authors: Wells James B. , Layne Ben H.
Published: March 2008

This study investigated the effects of display characteristics on the bias of estimates of whisker length of two different types of box-and-whisker plots. To test the hypothesis that the bias means of estimates of whisker length were significantly different for the experimental and control groups, a 2.234 mixed design was utilized. Notched box-and-whisker plots and regular box-and-whisker plots were levels of the between-subjects factor, while within-subjects factors were the interquartile spread, whisker length, and spatial orientation of the graphs. Results indicate that although there was no significant difference in the bias means associated with the two types of plots, bias seems to be the product of a three-way interaction between spatial orientation, interquartile spread, and whisker/spread ratio.
Teacher’s Corner: What to Do With the Upward Bias in R2: A Comment on Huberty
Authors: Snijders Tom A. B.
Published: March 2008

A recent article by Huberty (1994) discusses significance testing of R2 in linear regression and the definition of a corresponding effect size index. It recommends an adjustment to the standard null hypothesis, p2 = 0, in order to adjust for an upward bias in the statistic R2. This note suggests that the adjustment proposed by Huberty has some conceptual shortcomings. Existing improvements on R2 are described in some detail.
Estimating Effect Size Under Publication Bias: Small Sample Properties and Robustness of a Random Effects Selection Model
Authors: Hedges Larry V. , Vevea Jack L.
Published: March 2008

When there is publication bias, studies yielding large p values, and hence small effect estimates, are less likely to be published, which leads to biased estimates of effects in meta-analysis. We investigate a selection model based on one-tailed p values in the context of a random effects model. The procedure both models the selection process and corrects for the consequences of selection on estimates of the mean and variance of effect parameters. A test of the statistical significance of selection is also provided. The small sample properties of the method are evaluated by means of simulations, and the asymptotic theory is found to be reasonably accurate under correct model specification and plausible conditions. The method substantially reduces bias due to selection when model specification is correct, but the variance of estimates is increased; thus mean squared error is reduced only when selection produces substantial bias. The robustness of the method to violations of assumptions about the form of the distribution of the random effects is also investigated via simulation, and the model-corrected estimates of the mean effect are generally found to be much less biased than the uncorrected estimates. The significance test for selection bias, however, is found to be highly nonrobust, rejecting at up to 10 times the nominal rate when there is no selection but the distribution of the effects is incorrectly specified.
Improving Tabular Displays, With NAEP Tables as Examples and Inspirations
Authors: Wainer Howard
Published: March 2008

The modern world is rich with data; an inability to effectively utilize these data is a real handicap. One common mode of data communication is the printed data table. In this article we provide four guidelines the use of which can make tables more effective and evocative data displays. We use the National Assessment of Educational Progress both to provide inspiration for the development of these guidelines and to illustrate their operation. We also discuss a theoretical structure to aid in the development of test items to tap students’ proficiency in extracting information from tables.
Multilevel Item Response Models: An Approach to Errors in Variables Regression
Authors: Adams Raymond J. , Wilson Mark , Wu Margaret
Published: March 2008

In this article we show how certain analytic problems that arise when one attempts to use latent variables as outcomes in regression analyses can be addressed by taking a multilevel perspective on item response modeling. Under a multilevel, or hierarchical, perspective we cast the item response model as a within-student model and the student population distribution as a between-student model. Taking this perspective leads naturally to an extension of the student population model to include a range of student-level variables, and it invites the possibility of further extending the models to additional levels so that multilevel models can be applied with latent outcome variables. In the two-level case, the model that we employ is formally equivalent to the plausible value procedures that are used as part of the National Assessment of Educational Progress (NAEP), but we present the method for a different class of measurement models, and we use a simultaneous estimation method rather than two-step estimation. In our application of the models to the appropriate treatment of measurement error in the dependent variable of a between-student regression, we also illustrate the adequacy of some approximate procedures that are used in NAEP.
The Rasch Model for Speed Tests and Some Extensions With Applications to Incomplete Designs
Authors: Jansen Margo G. H.
Published: March 2008

In psychological measurement a distinction can be made between speed and power tests. Although most tests are partially speeded, the speed element is usually neglected. Here, the focus will be on latent trait models for pure speed tests. A particularly simple model has been developed by Rasch for the total response time on a (set of) pure speed test(s), based on the assumption that the test response times are approximately gamma distributed, with known index parameters and scale parameters depending on subject ability and test difficulty parameters. In the approach presented here the subject parameters are treated as random variables having a common gamma distribution. From this, maximum marginal likelihood estimators are derived for the test difficulties and the parameters of the latent subject distribution. This basic model can be extended in a number of ways. In a numerical example, an application of the Rasch model to reading data, which were incomplete by design, will be discussed.
Analysis of Repeated Measures Under Second-Stage Sphericity: An Empirical Bayes Approach
Authors: Boik Robert J.
Published: March 2008

The conventional multivariate analysis of repeated measures is applicable in a wide variety of circumstances, in part, because assumptions regarding the pattern of covariances among the repeated measures are not required. If sample sizes are small, however, then the estimators of the covariance parameters lack precision and, as a result, the power of the multivariate analysis is low. If the covariance matrix associated with estimators of orthogonal contrasts is spherical, then the conventional univariate analysis of repeated measures is applicable and has greater power than the multivariate analysis. If sphericity is not satisfied, an adjusted univariate analysis can be conducted, and this adjusted analysis may still be more powerful than the multivariate analysis. As sample size increases, the power advantage of the adjusted univariate test decreases, and, for moderate sample sizes, the multivariate test can be more powerful. This article proposes a hybrid analysis that takes advantage of the strengths of each of the two procedures. The proposed analysis employs an empirical Bayes estimator of the covariance matrix. Existing software for conventional multivariate analyses can, with minor modifications, be used to perform the proposed analysis. The new analysis behaves like the univariate analysis when samples size is small or sphericity is nearly satisfied. When sample size is large or sphericity is strongly violated, then the proposed analysis behaves like the multivariate analysis. Simulation results suggest that the proposed analysis controls test size adequately and can be more powerful than either of the other two analyses under a wide range of non-null conditions.
Analysis of Levene’s Test Under Design Imbalance
Authors: Keyes Tim K. , Levy Martin S.
Published: March 2008

By means of three analyses, the operating characteristics of Levene‘s test of homoscedasticity in a 1 -way layout are shown to be systematically affected by design imbalance when the error terms are normal. We consider a formal general linear hypothesis analysis, an empirical analysis, and an analysis of expected mean squares, employing the heuristic justification for an ordinary ANOVA. We propose a simple correction involving leverage and analyze its effect, as well as that of another adjustment, on power and level.
A Note on Huberty and Wisenbaker’s “Views of Variable Importance”
Authors: Thomas D. Roland
Published: March 2008

This article investigates criteria for assessing variable importance in MANOVA and descriptive discriminant analysis. Two criteria suggested by Huberty and Wisenbaker (1992) are examined, namely, (a) contribution to linear discriminant function scores and (b) contribution to grouping variable effects. Thomas and Zumbo (1996) have shown that the first criterion can be operationalized using discriminant ratio coefficients (DRCs). It is shown in this article that DRCs also provide an operational definition of grouping variable effects. Thus, it is proposed that the two criteria be amalgamated and called the contribution to grouping effects and discriminant scores. The F-to-remove indexes used by Huberty and Wisenbaker can then be regarded as operational definitions of a separate criterion, namely, the amount of additional information contributed to group discrimination.
Teacher’s Corner: A Note on Interpretation of the Paired-Samples t Test
Authors: Zimmerman Donald W.
Published: March 2008

Explanations of advantages and disadvantages of paired-samples experimental designs in textbooks in education and psychology frequently overlook the change in the Type I error probability which occurs when an independent-samples t test is performed on correlated observations. This alteration of the significance level can be extreme even if the correlation is small. By comparison, the loss of power of the paired-samples t test on difference scores due to reduction of degrees of freedom, which typically is emphasized, is relatively slight. Although paired-samples designs are appropriate and widely used when there is a natural correspondence or pairing of scores, researchers have not often considered the implications of undetected correlation between supposedly independent samples in the absence of explicit pairing.
Defining Johnson-Neyman Regions of Significance in the Three-Covariate ANCOVA Using Mathematica
Authors: Hunka Steve , Leighton Jacqueline
Published: March 2008

The Johnson-Neyman ANCOVA, used when nonhomogeneity of within-group regressions is present, poses special computational and plotting problems when three covariates are used. These problems can be overcome by using (a) an appropriate design and contrast matrix for the general linear model and (b) the Mathematica software system of computation to handle the symbolic and graphical processing requirements. Four-dimensional graphical representation of the polynomials which result are contour plotted in a three-dimensional space in order to define the regions of significance for contrasts. It is also shown that for some values of the covariate orthogonal contrasts are produced.
Generating Multiple Imputations for Matrix Sampling Data Analyzed With Item Response Models
Authors: Thomas Neal , Gan Nianci
Published: March 2008

Sample survey designs in which each participant is administered a subset of the items contained in a complete survey instrument are becoming an increasingly popular method of reducing respondent burden ( Mislevy, Beaton, Kaplan, Sheehan 1992 ; Raghunathan & Grizzle, 1995 ; Wacholder, Carroll, Pee, Gail 1994 ). Data from these survey designs can be analyzed using multiple imputation methodology that generates several imputed values for the missing data and thus yields several complete data sets. These data sets are then analyzed using complete data estimators and their standard errors ( Rubin, 1987b ). Generating the imputed data sets, however, can be very difficult. We describe improvements to the methods currently used to generate the imputed data sets for item response models summarizing educational data collected by the National Assessment of Educational Progress (NAEP), an ongoing collection of samples of 4th, 8th, and 12th grade students in the United States. The improved approximations produce small to moderate changes in commonly reported estimates, with the larger changes associated with an increasing amount of missing data. The improved approximations produce larger standard errors.
Application of Gibbs Sampling to Nested Variance Components Models With Heterogeneous Within-Group Variance
Authors: Kasim Rafa M. , Raudenbush Stephen W.
Published: March 2008

Bayesian analysis of hierarchically structured data with random intercept and heterogeneous within-group (Level-1) variance is presented. Inferences about all parameters, including the Level-1 variance and intercept for each group, are based on their marginal posterior distributions approximated via the Gibbs sampler Analysis of artificial data with varying degrees of heterogeneity and varying Level-2 sample sizes illustrates the likely benefits of using a Bayesian approach to model heterogeneity of variance (Bayes/Het). Results are compared to those based on now-standard restricted maximum likelihood with homogeneous Level-1 variance (RML/Hom). Bayes/Het provides sensible interval estimates for Level-1 variances and their heterogeneity, and, relatedly, for each group’s intercept. RML/Hom inferences about Level-2 regression coefficients appear surprisingly robust to heterogeneity, and conditions under which such robustness can be expected are discussed. Application is illustrated in a reanalysis of High School and Beyond data. It appears informative and practically feasible to obtain approximate marginal posterior distributions for all Level-1 and Level-2 parameters when analyzing large- or small-scale survey data. A key advantage of the Bayes approach is that inferences about any parameter appropriately reflect uncertainty about all remaining parameters.
Multilevel Modeling of Educational Data With Cross-Classification and Missing Identification for Units
Authors: Hill Peter W. , Goldstein Harvey
Published: March 2008

This paper presents a method for handling educational data in which students belong to more than one unit at a given level, but there is missing information on the identification of the units to which students belong. For example, a student might be classified as belonging sequentially to a particular combination of primary and secondary school, but for some students the identify of either the primary or the secondary school may be unknown. Similar situations arise in longitudinal studies in which students change school or class from one year to the next. The method involves setting up a cross-classified model, but replacing (0, 1) values for unit membership with weights reflecting probabilities of unit membership in cases where membership information is randomly missing. The method is illustrated with reference to longitudinal data on students’ progress in English.
The Kruskal-Wallis Test and Stochastic Homogeneity
Authors: Vargha András , Delaney Harold D.
Published: March 2008

For the comparison of more than two independent samples the Kruskal-Wallis H test is a preferred procedure in many situations. However, the exact null and alternative hypotheses, as well as the assumptions of this test, do not seem to be very clear among behavioral scientists. This article attempts to bring some order to the inconsistent, sometimes controversial treatments of the Kruskal-Wallis test. First we clarify that the H test cannot detect with consistently increasing power any alternative hypothesis other than exceptions to stochastic homogeneity. It is then shown by a mathematical derivation that stochastic homogeneity is equivalent to the equality of the expected values of the rank sample means. This finding implies that the null hypothesis of stochastic homogeneity can be tested by an ANOVA performed on the rank transforms, which is essentially equivalent to doing a Kruskal-Wallis H test. If the variance homogeneity condition does not hold then it is suggested that robust ANOVA alternatives performed on ranks be used for testing stochastic homogeneity. Generalizations are also made with respect to Friedman’s G test.
Item Response Theory Models Applied to Data Allowing Examinee Choice
Authors: Bradlow Eric T. , Thomas Neal
Published: March 2008

Examinations that permit students to choose a subset of the items are popular despite the potential that students may take examinations of varying difficulty as a result of their choices. We provide a set of conditions for the validity of inference for Item Response Theory (IRT) models applied to data collected from choice-based examinations. Valid likelihood and Bayesian inference using standard estimation methods require (except in extraordinary circumstances) that there is no dependence, after conditioning on the observed item responses, between the examinees choices and their (potential but unobserved) responses to omitted items, as well as their latent abilities. These independence assumptions are typical of those required in much more general settings. Common low-dimensional IRT models estimated by standard methods, though potentially useful tools for educational data, do not resolve the difficult problems posed by choice-based data.