bluuebunny/crossref_metadata_2025_split huggingface dataset:
VALIDITY OF GRE® GENERAL TEST SCORES FOR ADMISSION TO COLLEGES OF VETERINARY MEDICINE
Authors: Powers Donald E. 
Published: August 2014

ABSTRACTThis paper documents a case study of the validity of the Graduate Record Examinations (GRE®) General Test in a comprehensive sample of veterinary medical colleges. Extensive and complete data available from these schools allowed us to control several artifacts that are typical of “one‐shot” validation studies and thus enabled a relatively definitive assessment of the test's validity. Overall undergraduate grade point average (GPA), undergraduate GPA in the last 45 hours of courses, and GRE verbal, quantitative, and analytical scores were examined individually and in combination for their ability to predict success in veterinary school. For each of 16 veterinary medical colleges, statistical methods were applied to correct for the effects of range restriction in the predictors and unreliability of the criterion, and the results were summarized across all schools. When corrections were made for both range restriction and unreliability, the resulting validity coefficients (median multiple correlations) were .53 for three GRE scores, when used together; .56 for overall undergraduate GPA; and .72 for GRE scores and overall undergraduate GPA considered together. Adding GRE scores to undergraduate GPA increased the amount of variance explained, on average, by about 18% – a proportion that can be regarded as being “medium” to “large.”
PREPARING FOR THE PRE‐PROFESSIONAL SKILLS TESTS
Authors: Stricker Lawrence J. , Wilder Gita Z. 
Published: August 2014

ABSTRACTThis study investigated the extent and nature of preparation for the Pre‐Professional Skills Tests (PPST®), the reasons for preparing or not preparing, and differences in these results for White and minority‐group test takers and for middle‐class and working‐class test takers. Recent PPST test takers were surveyed. Preparation for the PPST was limited and mainly involved activities that were free or inexpensive, such as taking a sample test. The reported reasons for not preparing and the empirical correlates of measures of preparation were primarily attitudinal. Ethnic‐group and social‐class differences in the extent and nature of test preparation were minimal, but there were some differences in reported reasons and correlates of preparation, primarily less awareness of test preparation resources by White and middle‐class test takers and few correlates of test preparation for Black test takers.
REFINING A SYSTEM FOR COMPUTERIZED ADAPTIVE TESTING POOL CREATION1
Authors: Way Walter D. , Swanson Len , Steffen Manfred , Stocking Martha L. 
Published: August 2014

ABSTRACTThe purpose of this paper is to describe the development and features of a revised computerized adaptive testing (CAT) item pool creation system. This system refines and upgrades the system that is currently in use for the ongoing creation of CAT pools for several large‐scale, high‐stakes testing programs administered by Educational Testing Service (ETS®). Using data from a large‐scale CAT program administered by ETS, an illustrative application of the software is provided to compare pools created using simultaneous partitioning and sequential partitioning. The results of the comparison indicated that simultaneous partitioning was preferable to sequential partitioning in the conditions studied. The paper concludes by identifying and discussing several issues that remain to be investigated in utilizing the revised pool creation system with ETS CAT programs.
A METHOD FOR BUILDING A REALISTIC MODEL OF TEST TAKER BEHAVIOR FOR COMPUTERIZED ADAPTIVE TESTING
Authors: Stocking Martha L. , Steffen Manfred , Eignor Daniel R. 
Published: August 2014

ABSTRACTStatistical models of test taker behavior underlie many modern psychometric methodologies. An advantage of this approach is that it supports studies where simulated test taker responses are generated according to the particular statistical model in order to explore model properties. To the extent that such models fail to capture important aspects of real test takers' responses, the predictive utility of such simulation studies may be compromised. In this paper a methodology is explored that builds simulated test taker responses using information distilled from the responses of real test takers. This approach proved effective in capturing some aspects of test taker behavior that simpler models failed to incorporate.
ON THE PAST AND FUTURE OF NULL HYPOTHESIS SIGNIFICANCE TESTING1
Authors: Robinson Daniel H. , Wainer Howard 
Published: August 2014

ABSTRACTCriticisms of null hypothesis significance testing (NHST) have appeared recently in wildlife research journals (Anderson, Burnham, & Thompson, 2000; Anderson, Link, Johnson, & Burnham, 2001; Cherry, 1998; Guthery, Lusk, & Peterson, 2001; Johnson, 1999). In this essay we discuss these criticisms with regard to both current usage of NHST and plausible future use. We suggest that the historical usage of such procedures was not unreasonable and hence that current users might spend time profitably reading some of Fisher's applied work. However, we also believe that modifications to NHST and to the interpretations of its outcomes might better suit the needs of modern science. Our primary conclusion is that NHST is most often useful as an adjunct to other results (e.g., effect sizes) rather than as a stand‐alone result. We cite some examples, however, where NHST can be profitably used alone. Last, we find considerable experimental support for a less slavish attitude toward the precise value of the probability yielded from such procedures.
AN APPLICATION OF A BAYESIAN HIERARCHICAL MODEL FOR ITEM FAMILY CALIBRATION
Authors: Sinharay Sandip , Johnson Matthew S. , Williamson David M. 
Published: August 2014

ABSTRACTItem families, which are groups of items related to each other in some way, are increasingly used in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families. We fit that hierarchical model using the Markov chain Monte Carlo (MCMC) algorithm. Formulating the MCMC algorithm provides little additional difficulty (compared to fitting a simple item response theory model) even with the additional complexity in the form of hierarchy in the model. We show that the model can take into account the dependence structure inherent among the items and hence is an improvement over the models currently used in similar situations. We introduce the notion of the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family and suggest a way to estimate the FERFs. Our work is a step towards creating a tool that can save significant amount of resources for tests with item families, because calibrating only the item families might be enough rather than calibrating each item in the families separately.
IMPROVING THE STATISTICAL ASPECTS OF E‐RATER®: EXPLORING ALTERNATIVE FEATURE REDUCTION AND COMBINATION RULES
Authors: Feng Xin , Dorans Neil J. , Patsula Liane N. , Kaplan Bruce 
Published: August 2014

ABSTRACTThis study explores alternative ways of reducing the number of variables/features and additional ways of combining information across features to produce more stable and accurate e‐rater® scores. Following an explanation of the statistical aspects of the process is a description of alternatives to the process. Our explorations resulted in certain conclusions and directions for future research. We have examined enough e‐rater data to conclude that stepwise regression seems to be effective as a feature reduction procedure. However, this may be attributed to the consistently strong relationship with essay score that is observed for the content vector analysis (CVA) variables and the two variables used to approximate word length (number of auxiliary verbs and the ratio of the number of auxiliary verbs to the number of words). To yield better validation results, we also suggest that the hold‐out method for evaluating validity should replace the current two‐stage approach of first developing a model in a quasi‐uniform training sample and then validating these results in a target cross‐validation sample. More research is needed in several areas. First, explicit modeling of the part of essay scores that is unrelated to word length is warranted. The POM (Proportional Odds Model) approach should be investigated in greater depth. Also needed is a statistical justification for using essay scores to score CVA variables. Algorithmic approaches to prediction/classification problem, such as boosting, may prove fruitful. Further investigation of quantile regression and ridge regression should be conducted.
LARGE SAMPLE TESTS FOR COMPARING REGRESSION COEFFICIENTS IN MODELS WITH NORMALLY DISTRIBUTED VARIABLES
Authors: von Davier Alina A. 
Published: August 2014

ABSTRACTThe analysis of regression coefficients is an important issue in different scientific areas, mostly because conclusions about the relationship between variables, like causal interpretations, are drawn based on these coefficients. This paper focuses on the description of the null hypothesis of invariance of regression coefficients for multidimensional stochastic regressors. In this study, it is assumed that the variables have a joint normal distribution with unknown expectation and unknown positive definite covariance matrix. In this context, it is shown that the null hypothesis contains special parameter points, called singular and stationary parameter points, that influence the distribution of the commonly used test statistics under the null hypothesis. Three large sample statistics—the Wald test, the likelihood ratio test, and the efficient score test—are compared when testing this nonlinear null hypothesis. The results of a simulation study are presented. The goal of the simulations is to check the distributions of the three statistics for finite sample sizes and at a stationary point of the null hypothesis. Another aim is to compare the empirical values of the three statistics to one another, for different parameter constellations. It is shown that all three statistics present deviations from the expected chi‐squared distribution at this special parameter point. However, any of the three statistical tests might be used for testing the hypothesis of the invariance of the regression coefficients since they remain asymptotically conservative at the stationary points of the null hypothesis.
LINKING SCORES FROM TESTS OF SIMILAR CONTENT GIVEN IN DIFFERENT LANGUAGES: SPANISH LANGUAGE PAA™ AND ENGLISH LANGUAGE SAT® I
Authors: Cascallar Alicia S. , Dorans Neil J. 
Published: August 2014

ABSTRACTScore linkages between the Verbal and Math sections of the SAT® I: Reasoning Test and the corresponding sections of the new version of a Spanish‐language admissions test, the Prueba de Aptitud Académica® (PAA™), were investigated. A bilingual group design was employed. A language proficiency measure (ESLAT) was used to define the bilingual group and as a predictor variable. Prediction and scaling for concordance results were compared. Results indicated that for both single (PAA Verbal or PAA Math to the corresponding SAT I scores) and composite (PAA−V+M to SAT I−V+M and PAA−V+M+ESLAT to SAT I−V+M) score linkage, prediction is preferable to concordance. Comparison of prediction and concordance results for composite scores versus single construct scores indicates that when PAA Verbal is combined with PAA Math to form a composite, predictions of this composite are better than for Verbal alone but worse than predictions for Math alone.
A VERTICAL SCHEME OF BUNDLING NAEP BOOKLETS
Authors: Deng Hui , Ferris Jim , Hombo Catherine 
Published: August 2014

ABSTRACTA new scheme of bundling National Assessment of Educational Progress (NAEP) booklets, vertical bundling, was proposed in this study to overcome the limitations of the traditional methodology on booklet pairing balance and bundle length choice. This study compared the vertical scheme with the horizontal scheme through simulations using the 2002 NAEP spiral plans for Reading and Writing assessments. The results showed that the vertical scheme not only had flexibility with respect to bundle length, it also demonstrated potential for reducing the required number of different bundles, decreasing book wastage, and improving balance of within‐session booklet pairings. On the other hand, it inflated within‐session booklet duplication to some extent compared to the horizontal approach, a relatively small trade‐off for improved balance of booklet pairings.
SIMULATION STUDIES APPLYING POSTERIOR PREDICTIVE MODEL CHECKING FOR ASSESSING FIT OF THE COMMON ITEM RESPONSE THEORY MODELS
Authors: Sinharay Sandip , Johnson Matthew S. 
Published: August 2014

ABSTRACTModel checking in item response theory (IRT) is an underdeveloped area. No universal model‐checking tool exists in this field. Theposterior predictive model‐checkingmethod (Guttman, 1967; Rubin, 1981, 1984) is a popular Bayesian model‐checking tool because of its simplicity, strong theoretical basis, obvious intuitive appeal, and ability to provide graphical evidence. This study applies the posterior predictive model‐checking method to assess the fit of the popular IRT models (Rasch model, 2PL model, and 3PL model) for tests with dichotomous items. A series of simulation studies applying the method yields promising results. An important issue with the application of the posterior predictive model‐checking method is the choice ofdiscrepancy measures(which are like thetest statisticsin classical hypothesis testing). This article examines the performance of a number of discrepancy measures for assessing different aspects of fit of the common IRT models and suggests simple summarization of the results. Simple graphical summaries provide useful insight about the fit of the models. Theodds ratios, corresponding to the responses of the examinees to pairs of items, appear to be powerful discrepancy measures in detecting several types of misfits of the IRT models. The observed score distribution and biserial correlation coefficients are also found useful as discrepancy measures on a number of occasions. A companion article applying the posterior predictive method to real data examples shows that the method has the potential to become very useful in psychometrics.
DESIGN AND ANALYSIS IN A COGNITIVE ASSESSMENT
Authors: Yan Duanli , Mislevy Robert J. , Almond Russell G. 
Published: August 2014

ABSTRACTThere is growing interest in educational assessments that coordinate substantive considerations, learning psychology, task design, and measurement models. This paper concerns an analysis of responses from an assessment of mixed‐number subtraction that was created by Kikumi Tatsuoka (1983) in light of cognitive analyses of students' problem solutions. In particular, we fit a binary‐skills multivariate latent class model to the data and compare results to those obtained with an item response theory model and a modified latent class model suggested by model criticism indices. Markov chain Monte Carlo (MCMC) techniques are used to estimate the parameters in the model in a Bayesian framework that integrates information from substantive theory, expert judgment, and empirical data.
BEYOND ESSAY LENGTH: EVALUATING E‐RATER®'S PERFORMANCE ON TOEFL® ESSAYS
Authors: Chodorow Martin , Burstein Jill 
Published: August 2014

ABSTRACTThis study examines the relation between essay length and holistic scores assigned to Test of English as a Foreign Language™ (TOEFL®) essays by e‐rater®, the automated essay scoring system developed by ETS. Results show that an early version of the system, e‐rater99, accounted for little variance in human reader scores beyond that which could be predicted by essay length. A later version of the system, e‐rater01, performs significantly better than its predecessor and is less dependent on length due to its greater reliance on measures of topical content and of complexity and diversity of vocabulary. Essay length was also examined as a possible explanation for differences in scores among examinees with native languages of Spanish, Arabic, and Japanese. Human readers and e‐rater01 show the same pattern of differences for these groups, even when effects of length are controlled.
EXPLORING ITEM CHARACTERISTICS THAT ARE RELATED TO THE DIFFICULTY OF TOEFL DIALOGUE ITEMS
Authors: Kostin Irene 
Published: August 2014

ABSTRACTThe purpose of this study is to explore the relationship between a set of item characteristics and the difficulty of TOEFL® dialogue items. Identifying characteristics that are related to item difficulty has the potential to improve the efficiency of the item‐writing process The study employed 365 TOEFL dialogue items, which were coded on 49 variables, including 5 significant variables reported in Nissan, DeVincenzi, and Tang (1996). Of the 5 significant variables in Nissan et al., 3 correlated significantly with item difficulty in this study. Another 11 met a critical probability criterion. These 11 included representatives from three broad categories of variables: 2 in the category of word‐level factors, 1 in the category of discourse‐level factors, and 8 in the category of task‐processing factors. Multiple regression analyses indicate that the variables in this study account for about 40% of the variance in item difficulty.
ACCOMMODATIONS ON HIGH‐STAKES WRITING TESTS FOR STUDENTS WITH DISABILITIES
Authors: Cahalan‐Laitusis Cara 
Published: August 2014

ABSTRACTWith the addition of a writing component to many high‐stakes tests (e.g., college admission tests and high school exit exams), a review of writing accommodations for students with disabilities becomes critical. This paper reviews the accommodation policies of 20 states with high school exit exams of writing and three testing organizations that administer high‐stakes tests that assess writing. In addition, this paper reviews existing research on testing accommodations, specifically those that focus on writing assessments. A section on assistive technologies for writing is also included, because these technologies are becoming more widely used and are likely to be requested as testing accommodations. Finally, recommendations for needed research are provided.
JOINT AND CONDITIONAL MAXIMUM LIKELIHOOD ESTIMATION FOR THE RASCH MODEL FOR BINARY RESPONSES
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTThe usefulness of joint and conditional maximum‐likelihood is considered for the Rasch model under realistic testing conditions in which the number of examinees is very large and the number is items is relatively large. Conditions for consistency and asymptotic normality are explored, effects of model error are investigated, measures of prediction are estimated, and generalized residuals are developed.
APPLICATION OF THE STOCHASTIC EM METHOD TO LATENT REGRESSION MODELS
Authors: von Davier Matthias , Sinharay Sandip 
Published: August 2014

ABSTRACTThe reporting methods used in large scale assessments such as the National Assessment of Educational Progress (NAEP) rely on alatent regression model.The first component of the model consists of ap‐scale IRT measurement model that defines the response probabilities on a set of cognitive items inpscales depending on ap‐dimensional latent trait variableθ= (θ1, …θp). In the second component, the conditional distribution of this latent trait variableθis modeled by a multivariate, multiple regression on a set of predictor variables, which are usually based on student, school and teacher variables in assessments such as NAEP.In order to fit the latent regression model using the maximum (marginal) likelihood estimation technique, multivariate integrals have to be evaluated. In the computer program MGROUP used by ETS for fitting the latent regression model to data from NAEP and other sources, the integration is currently done either by numerical quadrature (for problems up to two dimensions) or by an approximation of the integral. CGROUP, the current operational version of the MGROUP program used in NAEP and other assessments since 1993, is based on Laplace approximation that may not provide fully satisfactory results, especially if the number of items per scale is small.This paper examines the application of stochastic expectation‐maximization (EM) methods (where an integral is approximated by an average over a random sample) to NAEP‐like settings. We present a comparison of CGROUP with a promising implementation of the stochastic EM algorithm that utilizes importance sampling. Simulation studies and real data analysis show that the stochastic EM method provides a viable alternative to CGROUP for fitting multivariate latent regression models.
MULTIPLE PERSONALITY DOMAINS IN RELATION TO OCCUPATIONAL CHOICE AND PERFORMANCE AMONG ESTABLISHED TEACHERS
Authors: Emmerich Walter , Rock Donald A. , Trapani Catherine S. 
Published: August 2014

ABSTRACTDoes personality impact differently on occupational choice and occupational performance? In a study of established teachers, interests, the Five‐Factor Model, and attributions were examined in relation to occupational choice and performance. Attributions were assessed using a new instrument designed for teachers. Choice of teaching specialty was found to be most strongly associated with interest in the arts and sciences, openness to experience, and internal attributions in response to a positive classroom event. On the other hand, quality of teaching assessed by the National Board of Professional Teaching Standards (NBPTS) was associated (negatively) with business interests and with a tendency to externalize blame in response to a problematic teaching situation. Study findings suggested the hypothesis that broad traits located relatively high in a personality hierarchy influence trait matching in selecting an occupation, whereas more specific motivational/dynamic factors located lower in the personality hierarchy influence performance within the chosen niche. The report includes supplemental evidence on the construct validity of the personality measures for assessing teachers.
IDENTIFYING THE WRITING TASKS IMPORTANT FOR ACADEMIC SUCCESS AT THE UNDERGRADUATE AND GRADUATE LEVELS
Authors: Rosenfeld Michael , Courtney Rosalea , Fowles Mary 
Published: August 2014

ABSTRACTThe authors conducted a large‐scale survey to confirm that the writing skills being assessed in the GRE® General Test can be linked to writing tasks that were judged to be important by graduate faculty from a variety of subject areas and across a wide range of institutions at both the graduate and undergraduate levels. The results obtained in this study provide an additional source of validity evidence for using the GRE Analytical Writing Assessment when making admission decisions for graduate school and are also useful in evaluating its relevance for use as an outcomes measure for upper‐division undergraduates.