bluuebunny/crossref_metadata_2025 huggingface dataset:
PSYCHOMETRIC EVALUATION OF THE NEW GRE® WRITING ASSESSMENT
Authors: Schaeffer Gary A. , Briel Jacqueline B. , Fowles Mary E. , Boodoo Gwyneth , Braun Henry , Lewis Charles , Thayer Dorothy
Published: August 2014

This study analyzed examinee responses to two essay prompts being considered for the Graduate Record Examinations (GRE®) Writing Assessment: “Present your Perspective on an Issue” and “Analyze an Argument.” Forty prompts (20 issue and 20 argument) were administered to over 2,300 students at 26 participating U.S. colleges and universities. Each student wrote two essays in response to either two issue prompts, two argument prompts, or one of each. Results show that the issue and argument writing tasks appear to assess relatively similar constructs, supporting the decision to include both types of prompts in the operational GRE Writing Assessment and to report a single, combined score. The results also support the random administration of prompts without equating adjustments, because within each task type, most of the prompts were comparable in difficulty and no important subgroup interactions with prompt classifications were detected. Finally, from a fairness perspective, the results show there were advantages to administering the issue prompt first and the argument prompt second. The GRE Program used the psychometric information provided by this study to make final design, delivery, and scoring decisions before introducing the operational assessment in the fall of 1999.
AN INQUIRY INTO THE NATURE OF THE SENTENCE‐COMPLETION TASK: IMPLICATIONS FOR ITEM GENERATION
Authors: Sheehan Kathleen M. , Mislevy Robert J.
Published: August 2014

The purpose of this two‐part study was to examine the cognitive skills underlying performance on Graduate Record Examinations (GRE®) sentence‐completion items. In the first investigation, a database of 112 operationally administered items was used to generate validity information for a subset of hypothesized skills that had been shown to be useful in previous research. Surprisingly, very few of the hypothesized skills performed as expected. In the second investigation, a subset of 45 item variants was used to (a) verify the results of the first investigation; (b) generate new hypotheses about the combinations of item‐stimulus features that signal particular required skills; and (c) provide an initial test of those hypotheses. This second investigation verified the findings of the first investigation and provided additional insights into the nature of the sentence completion task In particular, the analysis yielded detailed information about (a) the mental representations constructed by students during the process of item solving, (b) procedures for determining the subsets of item words that are actually considered by students during the process of item solving, and (c) approaches for predicting the operating characteristics of items when multiple solution strategies are possible. These findings will facilitate subsequent efforts to write new test items that target specific difficulty levels and to create performance summaries that are indicative of students' strengths and weaknesses.
AN ANALYSIS OF ADVANCED PLACEMENT (AP®) EXAMINATIONS IN ECONOMICS AND COMPARATIVE GOVERNMENT AND POLITICS
Authors: Breland Hunter M. , Oltman Philip K.
Published: August 2014

Advanced Placement Program® (AP®) Examinations in Macroeconomics, Microeconomics, and Comparative Government and Politics were studied to examine college course performance and gender differences. It was concluded that students who had received college credit for AP performed as well or better in higher‐level college courses in Macroeconomics and Microeconomics than students who had not taken AP courses. Gender differences in performance were observed in all three examinations, but the greatest gender differences, favoring male students, were observed for Comparative Government and Politics. A survey of instructors of Comparative Government and Politics indicated that only a small percentage of instructors had observed gender differences in performance in their courses, however.
AUTOMATIC ASSESSMENT OF VOCABULARY USAGE WITHOUT NEGATIVE EVIDENCE
Authors: Leacock Claudia , Chodorow Martin
Published: August 2014

This report describes the implementation and evaluation of an automated statistical method for assessing an examinee's use of vocabulary words in constructed responses. The grammatical error‐detection system, ALEK (Assessing Lexical Knowledge), infers negative evidence from the low frequency or absence of constructions in 30 million words of well‐formed, copy‐edited text from North American newspapers. ALEK detects two types of errors: those that violate basic principles of English syntax (e.g., agreement errors as in a desks) and those that show a lack of information about a specific word (e.g., treating a mass noun as a count noun in a pollution). The system evaluated word usage in essay‐length responses to Test of English as a Foreign Language (TOEFL®) prompts. ALEK was developed using three words and was evaluated on an additional 20 words that appeared frequently in TOEFL essays and in a university word list. System accuracy was evaluated to investigate its potential for scoring performance‐based measures of communicative competence. It performed with about 80% precision and 20% recall. False positives (correct usages that ALEK identified as errors) and misses (usage errors that were not recognized by ALEK) were analyzed, and methods for improving system performance were outlined.
DEVELOPMENT OF A SIBTEST BUNDLE METHODOLOGY FOR IMPROVING TEST EQUITY, WITH APPLICATIONS FOR GRE TEST DEVELOPMENT
Authors: Stout William , Bolt Dan , Froelich Amy Goodwin , Habing Brian , Hartz Sarah , Roussos Louis
Published: August 2014

The purpose of this study was to develop a differential‐item‐functioning (DIF)/impact methodology capable of accurately isolating underlying, conceptually‐based causes of DIF (differing item performances across examinee groups that are matched on ability) and impact (differing item performances across examinee groups that are not matched on ability) using data from the quantitative section of two administrations of the Graduate Record Examinations (GRE®) General Test. Analyses indicate that the “SIBTEST bundle methodology” that was developed in the study for GRE quantitative forms is effective and, as such, is exportable to various Educational Testing Service (ETS®) settings. This methodology should help improve test equity of future GRE quantitative forms, as well as test development and standardized tests in general. The developed methodology elevates statistically based DIF analyses from the mere screening of already manufactured items to the modification of test specifications and subsequent test construction processes.
USING CONFUSION INFUSION AND CONFUSION REDUCTION INDICES TO COMPARE ALTERNATIVE ESSAY SCORING RULES
Authors: Dorans Neil J. , Patsula Liane N.
Published: August 2014

Observed proportion agreement as a measure of association between two ratings of essay performance can be inflated when the number of rating categories is small. Cohen's Kappa adjusts observed agreement by subtracting out what one might expect if ratings were assigned independently of each other. The matrix of proportion agreements between two sets of assignment rules can be recast as a confusion matrix in which zero confusion is the equivalent of perfect agreement. Kappa can be viewed then as a measure of confusion reduction. A complementary measure, confusion infusion is defined. Its usefulness is illustrated with live data from a large‐scale testing program where e‐rater®, an automatic essay‐scoring algorithm, is used in place of a second reader. The confusion reduction and confusion infusion indices help make comparisons among the relative efficacy of two versions of e‐rater, and two other methods of assigning scores, a second reader and assigning all candidates the mode of the first reading.
VALIDATING LANGUEDGE™ COURSEWARE SCORES AGAINST FACULTY RATINGS AND STUDENT SELF‐ASSESSMENTS
Authors: Powers Donald E. , Roever Carsten , Huff Kristin L. , Trapani Catherine S.
Published: August 2014

LanguEdge™ Courseware is a software tool that is designed to help teachers of English as a second language (ESL) build and assess the communicative skills of their students. The purpose of this study was to generate information to help LanguEdge Courseware users understand better the meaning (or validity) of the assessment scores bases on the LanguEdge Courseware. Specifically, the objective was to describe, for each of the four sections of the LanguEdge assessment, relevant characteristics of test takers at various test score levels. To accomplish this objective, we gathered data that represent two different perspectives—those of instructors and those of students themselves.

Approximately 3,000 students each took one of two parallel forms of the LanguEdge assessment at domestic and international testing sites. Participants also completed a number of self‐assessment questions about their English language skills. In addition, for some study participants, instructors rated selected language skills.

LanguEdge test scores related moderately (correlations mostly in the .30s and .40s) with student self‐assessments. Of the four LanguEdge tests, Listening exhibited the strongest relationships to self‐assessments; Speaking, the next strongest; Reading, the next; and Writing, the least.

The correlations of faculty ratings with each of the LanguEdge section test scores were generally in the .40s, with some reaching the .50s. The correlations between the various student self‐assessment scales and faculty ratings were modest, mostly in the .30s. These correlations suggest that students and faculty had different perspectives on students' English language skills.
A BRIEF INTRODUCTION TO EVIDENCE‐CENTERED DESIGN
Authors: Mislevy Robert J. , Almond Russell G. , Lukas Janice F.
Published: August 2014

Evidence‐centered assessment design (ECD) is an approach to constructing educational assessments in terms of evidentiary arguments. This paper provides an introduction to the basic ideas of ECD, as well as some of the terminology and models that have been developed to implement the approach. In particular, it presents the high‐level models of the Conceptual Assessment Framework (CAF) and the Four‐process Delivery Architecture for assessment delivery systems. Special attention is given to the role of probability‐based reasoning in accumulating evidence across task performances, in terms of belief about unobservable variables that characterize the knowledge, skills, and/or abilities of students. This is the role traditionally associated with psychometric models, such as those of item response theory (IRT) and latent class models. To unify the ideas and to provide a foundation for extending probability‐based reasoning in assessment applications more broadly, however, a more general expression in terms of graphical models is indicated. This brief overview of evidence‐centered design provides the reader with a feel for where and how graphical models fit into the larger enterprise of educational and psychological assessment. A simple example based on familiar large‐scale standardized tests such as the Graduate Record Examinations® (GRE®) is used to fix ideas.
INVESTIGATING THE VALIDITY OF TOEFL: A FEASIBILITY STUDY USING CONTENT AND CRITERION‐RELATED STRATEGIES
Authors: Rosenfeld Michael , Oltman Philip K. , Sheppard Ken
Published: August 2014

The purpose of this study was to investigate the feasibility of two complementary approaches to assessing the validity of the TOEFL examination. One approach used evidence based on test content. In the context described in this report, evidence based on test content refers to the degree to which the items on the TOEFL examination are representative of the knowledge and skills required to demonstrate English proficiency in undergraduate and graduate programs throughout the United States and Canada. The content‐oriented approach used in this pilot study involved item rating procedures that were designed to evaluate and document the relationship between the language tasks or behaviors previously identified as important for academic success and the test items used to measure them. The second approach used a criterion‐related validation strategy. In this aspect of the study, experimental rating scales were developed for use by faculty to evaluate students' current levels of English language proficiency. These scales were designed to sample the domain of behaviors previously identified as important.
PARTIALLY OBSERVED MIXTURES OF IRT MODELS: AN EXTENSION OF THE GENERALIZED PARTIAL CREDIT MODEL
Authors: von Davier Matthias , Yamamoto Kentaro
Published: August 2014

The generalized partial credit model (GPCM) is used frequently in educational testing and in large‐scale assessments for analyzing polytomous data. Special cases of the generalized partial credit model are the partial credit model—or Rasch model for ordinal data—on the one hand and the 2‐parameter logistic (2PL) model on the other hand. The extension proposed here enables the use of the GPCM in discrete mixture item response theory (IRT) models with partially observed mixture information. Partially observed mixtures are discrete mixture distributions with missing grouping information for some, but not all, observations. This concept of partially observed mixtures integrates multigroup and mixture distribution IRT models. The proposed model includes the latent class analysis (LCA) and (multigroup) IRT models as well as mixture distribution IRT models as submodels. An application of the proposed partially observed mixture IRT model is presented. This application classifies about 80% of data from a large‐scale mathematics assessment into three school types and estimates a mixture IRT model for the whole sample in which the grouping information is unknown for 20% of the sample. The aim of the method is twofold: first, to estimate the latent distributions by school type and, second, to unmix the data with missing school types into the groups established by the portion of the sample with known school‐type information.
PRACTICAL APPLICATIONS OF POSTERIOR PREDICTIVE MODEL CHECKING FOR ASSESSING FIT OF COMMON ITEM RESPONSE THEORY MODELS
Authors: Sinharay Sandip
Published: August 2014

Model checking in item response theory (IRT) is an underdeveloped area. No unanimous model checking tool exists in this field even for the simple models, not to speak of the recently suggested complicated models. Theposterior predictive model checkingmethod (Guttman, 1967; Rubin, 1981, 1984) is a popular Bayesian model checking tool because of its simplicity, strong theoretical basis, intuitive appeal, and ability to provide graphical evidence. This paper applies the method to a number of real data examples. An important issue with the application of the posterior predictive model checking method is the choice ofdiscrepancy measures(which are like thetest statisticsin classical hypothesis testing). This paper also uses the discrepancy measures that are found useful in Sinharay and Johnson (2003). The posterior predictive model checking method seems to be promising in detecting different types of misfit of the common IRT models in real applications.
NEW SAT® WRITING PROMPT STUDY: ANALYSES OF GROUP IMPACT AND RELIABILITY
Authors: Breland Hunter , Kubota Melvin , Nickerson Kristine , Trapani Catherine , Walker Michael
Published: August 2014

This study investigated the impact on ethnic, language, and gender groups of a new kind of essay prompt type intended for use with the new SAT®. The study also generated estimates of the reliability of scores obtained using the prompts examined. To examine the impact of a new prompt type, random samples of eleventh‐grade students in 49 participating high schools were administered writing tests using four different prompts, two of an old type and two of a new type. To obtain estimates of the reliability of scores for the old and new types of prompts, schools were asked to participate in a second round of testing to occur four months after the initial testing. Results of the impact analyses revealed no significant prompt type effects for ethnic, gender, or language groups, although there were significant differences in mean scores for ethnic and gender groups for all prompts. The score reliability estimates obtained were similar to those obtained in previous studies.
ASSESSING FIT OF MODELS WITH DISCRETE PROFICIENCY VARIABLES IN EDUCATIONAL ASSESSMENT
Authors: Sinharay Sandip , Almond Russell , Yan Duanli
Published: August 2014

Model checking is a crucial part of any statistical analysis. As educators tie models for testing to cognitive theory of the domains, there is a natural tendency to represent participant proficiencies with latent variables representing the presence or absence of the knowledge, skills, and proficiencies to be tested (Mislevy, Almond, Yan, & Steinberg, 2001). Model checking for these models is not straightforward, mainly because traditional χ2‐type tests do not apply except for assessments with a small number of items. Williamson, Mislevy, and Almond (2000) note a lack of published diagnostic tools for these models.

This paper suggests a number of graphics and statistics for diagnosing problems with models with discrete proficiency variables. A small diagnostic assessment first analyzed by Tatsuoka (1990) serves as a test bed for these tools. This work is a continuation of the recent work by Yan, Mislevy, and Almond (2003) on this data set. Two diagnostic tools that prove useful are Bayesian residual plots and an analog of the item characteristic curve (ICC) plots. A χ2‐type statistic based on the latter plot shows some promise, but more work is required to establish the null distribution of the statistic. On the basis of the identified problems with the model used by Mislevy (1995), the suggested diagnostics are helpful to hypothesize an improved model that seems to fit better.
THE BIOGRAPHICAL INVENTORY IN NAVAL AVIATION SELECTION: INSIDE THE BLACK BOX
Authors: Stricker Lawrence J.
Published: August 2014

A biographical inventory has been used in the selection of students for naval aviation training since World War II, and its validity in predicting their retention‐attrition in this training has been well established. This study investigated the constructs underlying the inventory and their relations to students' retention‐attrition. A factor analysis of the items on the inventory for student pilots identified five factors. One factor, being a commissioned officer, appeared to account for the inventory's validity.
RELIABILITY OF SPEEDED NUMBER‐RIGHT MULTIPLE‐CHOICE TESTS
Authors: Attali Yigal
Published: August 2014

Contrary to common belief, reliability estimates of number‐right multiple‐choice tests are not inflated by speededness. Because examinees guess on questions when they run out of time, the responses to these questions show less consistency with the responses of other questions, and the reliability of the test will be decreased. The surprising implication is that adding questions to a multiple‐choice test may lower its reliability when the test is speeded. This paper develops the mathematical derivations and shows the effects of speededness on reliability in simulations.
MODEL DIAGNOSTICS FOR BAYESIAN NETWORKS
Authors: Sinharay Sandip
Published: August 2014

Assessing fit of psychometric models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. Bayesian networks, frequently used in educational assessments (see, for example, Mislevy, Almond, Yan, & Steinberg, 2001) primarily for learning about students' knowledge and skills, are no exception. This paper employs theposterior predictive model checkingmethod (Guttman, 1967; Rubin, 1984), a popular Bayesian model checking tool, to assess fit of simple Bayesian networks. A number of aspects of model fit, those of usual interest to practitioners, are assessed in this paper using various diagnostic tools. The first diagnostic used is direct data display—a visual comparison of the observed data set and a number of the posterior predictive data sets (that are predicted by the model). The second aspect examined here is item fit. Examinees are grouped into a number of equivalence classes, based on the generated values of their skill variables, and the observed and expected proportion correct scores on an item for the classes are combined to provide a χ2‐type and aG2‐type test statistic for each item. Another (similar) set of χ2‐type andG2‐type test statistic is obtained by grouping the examinees by their raw scores and then comparing their observed and expected proportion correct scores on an item. This paper also suggests how to obtain posterior predictive p‐values, natural candidate p‐values from a Bayesian viewpoint, for the χ2‐type andG2‐type test statistics. The paper further examines the association among the items, especially if the model can explain the odds ratios corresponding to the responses to pairs of items. Finally, in an effort to examine the issue of differential item functioning (DIF), this paper suggests a version of the Mantel‐Haenszel statistic (Holland, 1985), which uses “matched groups” based on equivalence classes, as a discrepancy measure with posterior predictive model checking. Limited simulation studies and a real data application examine the effectiveness of the suggested model diagnostics.
INVESTIGATING DIFFERENCES IN EXAMINEE PERFORMANCE BETWEEN COMPUTER‐BASED AND HANDWRITTEN ESSAYS
Authors: Yu Lei , Livingston Samuel A. , Larkin Kevin C. , Bonett John
Published: August 2014

This study compared essay scores from paper‐based and computer‐based versions of a writing test for prospective teachers. Scores for essays in the paper‐based version averaged nearly half a standard deviation higher than those in the computer‐based version, after applying a statistical control for demographic differences between the groups of examinees taking the two versions. The statistical control was implemented by means of a propensity score, applying weights to members of one group to match the propensity score distribution of the other group. The score difference between the groups did not change substantially when the analysis was restricted to examinees taking the same mix of essay topics or to examinees taking one particular essay.
PROBABILITY PREDICTION AND CLASSIFICATION
Authors: Haberman Shelby J.
Published: August 2014

Criteria for prediction of multinomial responses are examined in terms of estimation bias. Logarithmic penalty and least squares are quite similar in behavior but quite different from maximum probability. The differences ultimately reflect deficiencies in the behavior of the criterion of maximum probability.
TOWARD ACCESSIBLE COMPUTER‐BASED TESTS: PROTOTYPES FOR VISUAL AND OTHER DISABILITIES
Authors: Hansen Eric G. , Forer Douglas C. , Lee Moon J.
Published: August 2014

There is a great need to explore approaches for developing computer‐based testing systems that are more accessible for people with disabilities. This report explores three prototype test delivery approaches, describing their development and formative evaluations. Fifteen adults, 2 to 4 from each of the six disability statuses—blindness, low vision, deafness, deaf‐blindness, learning disability, and no disability—participated in a formative evaluation of the systems. Each participant was administered from 2 to 15 items in each of one or two of the systems. The study found that although all systems had weaknesses that should be addressed, almost all of the participants (13 of 15) would recommend at least one of the delivery methods for high‐stakes tests, such as those for college or graduate admissions. The report concludes with recommendations for additional research that testing organizations seeking to develop accessible computer‐based testing systems can consider.
STEREOTYPE THREAT, THE TEST‐CENTER ENVIRONMENT, AND PERFORMANCE ON THE GRE GENERAL TEST
Authors: Walters Alyssa M. , Lee Soonmook , Trapani Catherine
Published: August 2014

The study investigated the applicability of previous experimental research on stereotype threat to operational Graduate Record Examinations® (GRE®) General Test testing centers. The goal was to document any relationships between features of the testing environment that might cue stereotype threat as well as any impact on GRE test scores among African American, Hispanic, Asian American, and female test‐takers. Among such features were the gender and ethnicity of test proctors and more general factors, such as the size, activity level, and social atmosphere of test centers. Our analyses revealed several relationships among environmental factors and several variations in test performance for all groups. However, we found no direct support for stereotype threat and, in fact, found some effects for proctor ethnicity that ran counter to a stereotype‐threat explanation.
A BAYESIAN HIERARCHICAL MODEL FOR LARGE‐SCALE EDUCATIONAL SURVEYS: AN APPLICATION TO THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Authors: Johnson Matthew S. , Jenkins Frank
Published: August 2014

Large‐scale educational assessments such as the National Assessment of Educational Progress (NAEP) sample examinees to whom an exam will be administered. In most situations the sampling design is not a simple random sample and must be accounted for in the estimating model. After reviewing the current operational estimation procedure for NAEP, this paper describes a Bayesian hierarchical model for the analysis of complex large‐scale assessments. The model clusters students within schools and schools within primary sampling units. The paper discusses an estimation procedure that utilizes a Markov chain Monte Carlo algorithm to approximate the posterior distribution of the model parameters. Results from two Bayesian models, one treating item parameters as known and one treating them as unknown, are compared to results from the current operational method on a simulated data set and on a subset of data from the 1998 NAEP reading assessment. The point estimates from the Bayesian model and the operational method are quite similar in most cases, but there does seem to be systematic differences in measures of uncertainty (e.g., standard errors, confidence intervals).
UNDERSTANDING WHAT SAT REASONING TEST™ SCORES ADD TO HIGH SCHOOL GRADES: A STRAIGHTFORWARD APPROACH
Authors: Bridgeman Brent , Pollack Judy , Burton Nancy
Published: August 2014

Although a number of large‐scale studies have demonstrated an increase in predictive validity when SAT® scores are added to high school grades, this increment is often described in terms of a seemingly very small increase in explained variance. This may have led to the erroneous belief among test critics and others that students with low SAT scores would perform, on average, almost as well in college as students with high SAT scores. Using a sample of 41 colleges, the current study shows substantial differences in the percent of students who succeed (defined by a 2.5 or 3.5 college grade point average [CGPA] at the end of one year or four years in college) by SAT score level, even when intensity of the high school curriculum and high school grades are taken into account.
PERFORMANCE OF MEN AND WOMEN ON MULTIPLE‐CHOICE AND CONSTRUCTED‐RESPONSE TESTS FOR BEGINNING TEACHERS
Authors: Livingston Samuel A. , Rupp Stacie L.
Published: August 2014

Some previous research results imply that women tend to perform better, relative to men, on constructed‐response (CR) tests than on multiple‐choice (MC) tests in the same subjects. An analysis of data from several tests used in the licensing of beginning teachers supported this hypothesis, to varying degrees, in most of the tests investigated. The hypothesis was strongly supported in Praxis™ Principles of Learning and Teaching tests for secondary school teachers and in subject‐knowledge tests for social studies teachers, science teachers, and middle school mathematics teachers. Evidence for the hypothesis was weak in subject‐knowledge tests for middle school English teachers and for secondary school mathematics teachers. Subject‐knowledge tests for secondary school English teachers did not show the hypothesized relationship. The analysis was based on plots showing the cumulative percentages of men and women attaining each possible score on the MC and CR tests.
CONDITIONAL COVARIANCE THEORY AND DETECT FOR POLYTOMOUS ITEMS
Authors: Zhang Jinming
Published: August 2014

This paper extends the theory of conditional covariances to polytomous items. It has been mathematically proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, is positive if the two items are dimensionally homogeneous and negative otherwise. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as NAEP data. As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or by random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade. The results also indicate that the degree of multidimensionality of the NAEP reading data is weak.
THE CORRELATION BETWEEN THE SCORES OF A TEST AND AN ANCHOR TEST
Authors: Sinharay Sandip , Holland Paul
Published: August 2014

It is a widely held belief that an anchor test used in equating should be a miniature version (orminitest) of the tests to be equated; that is, the anchor test should be proportionally representative of the two tests in content and statistical characteristics. This paper examines the scientific foundation of this belief, especially regarding statistical characteristics. In particular, we study, through several simulation studies and a real data example, the correlation of the raw score of the total test with several anchor tests, including those having statistical characteristics that differ from those of the total test. Anchor tests with spread of item difficulties less than that of a total test are shown to consistently have higher correlations than a minitest. Because of the general view that higher correlations of the scores of an anchor test and the total test lead to better equating, our results suggest that requiring an anchor test to mimic the statistical characteristics of the total test may be too restrictive and may not be optimal.
MODEL ANALYSIS AND MODEL CREATION: CAPTURING THE TASK‐MODEL STRUCTURE OF QUANTITATIVE ITEM DOMAINS
Authors: Deane Paul , Graf Edith Aurora , Higgins Derrick , Futagi Yoko , Lawless René
Published: August 2014

This study focuses on the relationship between item modeling and evidence‐centered design (ECD); it considers how an appropriately generalized item modeling software tool can support systematic identification and exploitation of task‐model variables, and then examines the feasibility of this goal, using linear‐equation items as a test case. The first half of the study examines task‐model structures for linear equations and their relevance to item difficulty within ECD. The second half of the study presents prototype software, a Model Creator system for pure math items, designed to partially automate the creation of variant item models reflecting different combinations of task‐model variables. The prototype is applied to linear equations but is designed to generalize over a range of pure mathematical content types.
TENSIONS, TRENDS, TOOLS, AND TECHNOLOGIES: TIME FOR AN EDUCATIONAL SEA CHANGE
Authors: Shute Valerie J.
Published: August 2014

This report outlines three educational approaches: (a) traditional, the currently dominant approach, a largely lecture‐oriented, authoritarian style that makes heavy use of assessments of learning, which are useful for accountability purposes but only marginally useful for guiding day‐to‐day instruction; (b) progressive, a highly student‐centered approach that relies on assessments for learning, which can be very useful in guiding day‐to‐day instruction; and (c) unified, a new, integrated approach that uses the best of both kinds of assessments—for and of learning—and which leverages computer technology, educational measurement, and cognitive science to address factors that undermined earlier attempts to implement the progressive approach. This report examines some of the research, trends, and factors that should be considered, understood, and, in some cases, leveraged, in order to move toward the unified approach. Further, this report presents examples of how ETS projects are moving toward the new approach to harness assessment in the service of learning.
THE IMPACT OF CHANGES IN THE TOEFL EXAMINATION ON TEACHING AND LEARNING IN CENTRAL AND EASTERN EUROPE: PHASE 1, THE BASELINE STUDY
Authors: Wall Dianne , Horák Tania
Published: August 2014

The purpose of this report is to present the findings of the first phase in a longitudinal study of the impact of changes in the TOEFL® test on teaching and learning in test preparation classrooms. Observations were carried out and interviews conducted with teachers, students, and directors of studies at 10 institutions in six countries in Central and Eastern Europe. The findings constitute a baseline against which finding from future phases can be compared. The report includes an analysis of the types of impact originally envisaged by advisers to the new TOEFL and a discussion of factors in the educational context that may influence the types of impact that may appear once the new test is operational.
ADAPTIVE NUMERICAL INTEGRATION FOR ITEM RESPONSE THEORY
Authors: Antal Tamás , Oranje Andreas
Published: August 2014

Well‐known numerical integration methods are applied to item response theory (IRT) with special emphasis on the estimation of the latent regression model of NAEP. An argument is made that the Gauss‐Hermite rule enhanced with Cholesky decomposition and normal approximation of the response likelihood is a fast, precise, and reliable alternative for the numerical integration in NAEP and in IRT in general.
THE IMPACT OF SHORT‐TERM USE OF CRITERIONSM ON WRITING SKILLS IN NINTH GRADE
Authors: Rock JoAnn Leah
Published: August 2014

In this study, a hierarchical linear model (HLM) was used within a cluster random assignment design to test the impact of Criterionsm on students' writing skill when Criterion was used as a supplementary instructional tool in ninth grade English classrooms over a 4 week period. The study was conducted in three states: Indiana, Pennsylvania, and New Jersey. The sample consisted of 1,464 ninth graders aggregated across 10 schools, 34 teachers, and 61 classes. Findings indicate that over a short time period, Criterion use improved the mechanical aspects of student essays as measured by the analytic scores given to their essays. Students in ninth grade English classes who used Criterion several times a week over a 1 month period received higher analytic scores on their essays written at the end of the study period than those who were in classes that did not. The effect was small, but statistically significant at the .05 level using a one‐tailed test. These findings suggest a role for automated feedback systems in supporting writing instruction.
CONSTRUCT VALIDITY OF E‐RATER® IN SCORING TOEFL® ESSAYS
Authors: Attali Yigal
Published: August 2014

This study examined the construct validity of the e‐rater® automated essay scoring engine as an alternative to human scoring in the context of TOEFL® essay writing. Analyses were based on a sample of students who repeated the TOEFL within a short time period. Two e‐rater scores were investigated in this study, the first based on optimally predicting the human essay score and the second based on equal weights for the different features of e‐rater.

Within a multitrait‐multimethod approach, the correlations and reliabilities of human and e‐rater scores were analyzed together with TOEFL subscores (structured writing, reading, and listening) and with essay length. Possible biases between human and e‐rater scores were examined with respect to differences in performance across countries of origin and differences in difficulty across prompts. Finally, a factor analysis was conducted on the e‐rater features to investigate the interpretability of their internal structure and determine which of the two e‐rater scores reflects this structure more closely.

Results showed that the e‐rater score based on optimally predicting the human score measures essentially the same construct as human‐based essay scores with significantly higher reliability and consequently higher correlations with related language scores. The equal‐weights e‐rater score showed the same high reliability but significantly lower correlation with essay length. It is also aligned with the 3‐factor hierarchical (word use, grammar, and discourse) structure that was discovered in the factor analysis. Both e‐rater scores also successfully replicate human score differences between countries and prompts.
DISCLOSURE RISK IN EDUCATIONAL SURVEYS: AN APPLICATION TO THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Authors: Oranje Andreas , Freund David , Lin Mei‐jang , Tang Yuxin
Published: August 2014

In this paper, a data perturbation method for minimizing the possibility of disclosure of participants' identities on a survey is described in the context of the National Assessment of Educational Progress (NAEP). The method distinguishes itself from most approaches because of the presence of cognitive tasks. Hence, a data edit should have minimal impact on both relations among demographic variables and relations between demographic and proficiency variables. Furthermore, since only a few students are at risk to be disclosed in a typical sampling setting common to educational surveys, the proposed data perturbation is governed by a nonuniform probabilistic process. The method is applied to data from NAEP and impact is computed using proficiency averages, demographic proportions, statistical inference results, and loglinear models. Results show that the proposed perturbation method has very little impact on NAEP results, even at relatively large editing rates. Some data coarsening results are reported as well. While the univariate results are relatively unaffected from the coarsening, loglinear models from higher order contingency tables are affected. It is recommended to restrict disclosure limitation techniques to perturbation methods in the case of NAEP.
SUPPORTING EFFICIENT, EVIDENCE‐CENTERED ITEM DEVELOPMENT FOR THE GRE VERBAL MEASURE
Authors: Sheehan Kathleen M. , Kostin Irene , Futagi Yoko
Published: August 2014

This paper explores alternative approaches for facilitating efficient, evidence‐centered item development for a new type of verbal reasoning item developed for use on the GRE® General Test. Results obtained in two separate studies are reported. The first study documented the development and validation of a fully automated approach for locating the types of stimulus materials needed to support more efficient generation of the studied item type. The validation results confirmed that the proposed filtering technique can help item writers increase the percentage of acceptable stimulus paragraphs located per unit time interval from the current level of about 10% to nearly 30%. The second study documented the development and validation of a set of task models designed to help item writers generate new items that are optimally constructed to provide high‐quality evidence about targeted skills. The proposed models were validated by considering the percentage of difficulty variance accounted for by the specified item classifications. That amount ranged from slightly more than 30% for items designed to test vocabulary skills to slightly more than 40% for items designed to test additional verbal reasoning skills, such as generating near and far inferences and understanding complex oppositional reasoning.
THE IMPACT OF THE NATIONAL BOARD FOR PROFESSIONAL TEACHING STANDARDS: A REVIEW OF THE RESEARCH
Authors: Gitomer ew H.
Published: August 2014

Interest in the impact of the teacher advanced certification system instituted by the National Board for Professional Teaching Standards (NBPTS) has resulted in a quickly expanding body of literature. This review examines five critical issues covered in the literature: the extent to which National Board certified teachers (NBCTs) engage in stronger instructional practice, are associated with stronger learning gains in their classrooms and schools, benefit from participating in the assessment process, influence the teaching quality of their peers, and are present in schools and classrooms with different characteristics. While most comparisons provide evidence that support the incremental effectiveness of NBCTs, the effects are generally modest. However, all findings must be interpreted with caution, given the limitations of research designs, methods, and tools used to explore teacher effects.
THE CORRELATION BETWEEN ITEM PARAMETERS AND ITEM FIT STATISTICS
Authors: Sinharay Sandip , Lu Ying
Published: August 2014

Dodeen (2004) studied the correlation between the item parameters of the three‐parameter logistic model and two item fit statistics, and found some linear relationships (e.g., a positive correlation between item discrimination parameters and item fit statistics) that have the potential for influencing the work of practitioners who employ item response theory. This paper examines the same type of linear relationships as studied in Dodeen. However, this paper adds to the literature by employing item fit statistics not considered in Dodeen, which have been recently suggested and whose Type I error rates have been demonstrated to be generally close to the nominal level. Detailed simulations show that if one uses certain of the recently suggested item fit statistics, there is no need to worry about any linear relationships between the item parameters and item fit statistics.
INVESTIGATING THE EFFECTIVENESS OF A SYNTHETIC LINKING FUNCTION ON SMALL SAMPLE EQUATING
Authors: Kim Sooyeon , von Davier Alina A. , Haberman Shelby
Published: August 2014

The synthetic function, which is a weighted average of the identity (the trivial linking function for forms that are known to be completely parallel) and a traditional equating method, has been proposed as an alternative for performing linking with very small samples (Kim, von Davier, & Haberman, 2006). The purpose of the present study was to investigate the benefits of the synthetic function using various real data sets gathered from different administrations of tests from a licensure testing program. We investigated the chained linear, Tucker, Levine, and mean equating methods, along with the identity and the synthetic functions with small samples (N = 19 to 70). Neither the identity nor the synthetic functions worked as well as did other linear equating methods, because test forms differed markedly in difficulty. The synthetic function cannot be used as a solution or methodological fix to a problem that is caused by poor data collection design.
THE IMPACT OF ANCHOR TEST LENGTH ON EQUATING RESULTS IN A NONEQUIVALENT GROUPS DESIGN
Authors: Ricker Kathryn L. , von Davier Alina A.
Published: August 2014

This study explored the effects of external anchor test length on final equating results of several equating methods, including equipercentile (frequency estimation), chained equipercentile, kernel equating (KE) poststratification PSE with optimal bandwidths, and KE PSE linear (large bandwidths) when using the nonequivalent groups anchor test (NEAT) design. This study used pseudotests constructed of item responses from a real operational test. The equating methods were evaluated using an equating criterion. Conditional differences between the criterion scores and equated scores, and root mean square error of the differences (RMSE) were used as measures to compare the methods to the criterion equating, which in this study is an equivalent groups (EG) equipercentile equating function. The results indicate that bias tended to increase in the conversions as the anchor test length decreased, but the KE PSE with optimal bandwidths and equipercentile (frequency estimation) methods were less sensitive to this change than the other methods. The KE PSE linear method with large bandwidths performed poorly compared to the criterion across all anchor test lengths.
AN INITIAL FIELD TRIAL OF AN INSTRUMENT FOR MEASURING LEARNING STRATEGIES OF MIDDLE SCHOOL STUDENTS
Authors: Liu Ou Lydia , Jackson Teresa , Ling Guangming
Published: August 2014

Learning strategies have been increasingly recognized as a useful tool to promote effective learning. In response to the lack of available learning strategies measures for middle school students, this study designed an instrument for these students, assessing behavioral, cognitive, and metacognitive strategies. This instrument, the Middle School Learning Strategies (MSLS) scale, is examined in terms of factorial structure, reliability, and correlates. Three factors emerge from the analyses: effective strategies, help seeking, and bad habits. The subscales displayed a reasonable reliability, ranging from .70 to .87. Student grades in language arts, social studies, math, and science were collected as criterion variables. As expected, grades in these four subjects correlated positively with both effective strategies and help seeking, yet negatively with bad habits. As a pilot measure, this instrument has demonstrated promising features as a useful tool for students to evaluate and enhance their learning strategies.
ASYMPTOTIC LIMITS OF ITEM PARAMETERS IN JOINT MAXIMUM‐LIKELIHOOD ESTIMATION FOR THE RASCH MODEL
Authors: Haberman Shelby J.
Published: August 2014

Techniques are developed for approximation and exact computation of the asymptotic limit of the item parameter estimates obtained by application of joint maximum‐likelihood estimation to the Rasch model.
CONTINUOUS EXPONENTIAL FAMILIES: AN EQUATING TOOL
Authors: Haberman Shelby J.
Published: August 2014

Continuous exponential families may be employed to find continuous distributions with the same initial moments as the discrete distributions encountered in typical applications of classical equating. These continuous distributions provide distribution functions and quantile functions that may be employed in equating. To illustrate, an application is considered for a randomly equivalent groups design.
APPROACHES TO THE DESIGN OF DIAGNOSTIC ITEM MODELS
Authors: Graf Edith Aurora
Published: August 2014

Quantitative item models are item structures that may be expressed in terms of mathematical variables and constraints. An item model may be developed as a computer program from which large numbers of items are automatically generated. Item models can be used to produce large numbers of items for use in traditional, large‐scale assessments. But they have potential for use in other areas as well, including diagnostic assessment. In this report, I first review research on diagnostic assessment and then discuss how approaches to diagnostic assessment can inform the design of diagnostic item models.