bluuebunny/crossref_metadata_2025_split huggingface dataset:
AN INQUIRY INTO THE NATURE OF THE SENTENCE‐COMPLETION TASK: IMPLICATIONS FOR ITEM GENERATION
Authors: Sheehan Kathleen M. , Mislevy Robert J. 
Published: August 2014

ABSTRACTThe purpose of this two‐part study was to examine the cognitive skills underlying performance on Graduate Record Examinations (GRE®) sentence‐completion items. In the first investigation, a database of 112 operationally administered items was used to generate validity information for a subset of hypothesized skills that had been shown to be useful in previous research. Surprisingly, very few of the hypothesized skills performed as expected. In the second investigation, a subset of 45 item variants was used to (a) verify the results of the first investigation; (b) generate new hypotheses about the combinations of item‐stimulus features that signal particular required skills; and (c) provide an initial test of those hypotheses. This second investigation verified the findings of the first investigation and provided additional insights into the nature of the sentence completion task In particular, the analysis yielded detailed information about (a) the mental representations constructed by students during the process of item solving, (b) procedures for determining the subsets of item words that are actually considered by students during the process of item solving, and (c) approaches for predicting the operating characteristics of items when multiple solution strategies are possible. These findings will facilitate subsequent efforts to write new test items that target specific difficulty levels and to create performance summaries that are indicative of students' strengths and weaknesses.
AN ANALYSIS OF ADVANCED PLACEMENT (AP®) EXAMINATIONS IN ECONOMICS AND COMPARATIVE GOVERNMENT AND POLITICS
Authors: Breland Hunter M. , Oltman Philip K. 
Published: August 2014

ABSTRACTAdvanced Placement Program® (AP®) Examinations in Macroeconomics, Microeconomics, and Comparative Government and Politics were studied to examine college course performance and gender differences. It was concluded that students who had received college credit for AP performed as well or better in higher‐level college courses in Macroeconomics and Microeconomics than students who had not taken AP courses. Gender differences in performance were observed in all three examinations, but the greatest gender differences, favoring male students, were observed for Comparative Government and Politics. A survey of instructors of Comparative Government and Politics indicated that only a small percentage of instructors had observed gender differences in performance in their courses, however.
AUTOMATIC ASSESSMENT OF VOCABULARY USAGE WITHOUT NEGATIVE EVIDENCE
Authors: Leacock Claudia , Chodorow Martin 
Published: August 2014

ABSTRACTThis report describes the implementation and evaluation of an automated statistical method for assessing an examinee's use of vocabulary words in constructed responses. The grammatical error‐detection system, ALEK (Assessing Lexical Knowledge), infers negative evidence from the low frequency or absence of constructions in 30 million words of well‐formed, copy‐edited text from North American newspapers. ALEK detects two types of errors: those that violate basic principles of English syntax (e.g., agreement errors as in a desks) and those that show a lack of information about a specific word (e.g., treating a mass noun as a count noun in a pollution). The system evaluated word usage in essay‐length responses to Test of English as a Foreign Language (TOEFL®) prompts. ALEK was developed using three words and was evaluated on an additional 20 words that appeared frequently in TOEFL essays and in a university word list. System accuracy was evaluated to investigate its potential for scoring performance‐based measures of communicative competence. It performed with about 80% precision and 20% recall. False positives (correct usages that ALEK identified as errors) and misses (usage errors that were not recognized by ALEK) were analyzed, and methods for improving system performance were outlined.
DEVELOPMENT OF A SIBTEST BUNDLE METHODOLOGY FOR IMPROVING TEST EQUITY, WITH APPLICATIONS FOR GRE TEST DEVELOPMENT
Authors: Stout William , Bolt Dan , Froelich Amy Goodwin , Habing Brian , Hartz Sarah , Roussos Louis 
Published: August 2014

ABSTRACTThe purpose of this study was to develop a differential‐item‐functioning (DIF)/impact methodology capable of accurately isolating underlying, conceptually‐based causes of DIF (differing item performances across examinee groups that are matched on ability) and impact (differing item performances across examinee groups that are not matched on ability) using data from the quantitative section of two administrations of the Graduate Record Examinations (GRE®) General Test. Analyses indicate that the “SIBTEST bundle methodology” that was developed in the study for GRE quantitative forms is effective and, as such, is exportable to various Educational Testing Service (ETS®) settings. This methodology should help improve test equity of future GRE quantitative forms, as well as test development and standardized tests in general. The developed methodology elevates statistically based DIF analyses from the mere screening of already manufactured items to the modification of test specifications and subsequent test construction processes.
USING CONFUSION INFUSION AND CONFUSION REDUCTION INDICES TO COMPARE ALTERNATIVE ESSAY SCORING RULES
Authors: Dorans Neil J. , Patsula Liane N. 
Published: August 2014

ABSTRACTObserved proportion agreement as a measure of association between two ratings of essay performance can be inflated when the number of rating categories is small. Cohen's Kappa adjusts observed agreement by subtracting out what one might expect if ratings were assigned independently of each other. The matrix of proportion agreements between two sets of assignment rules can be recast as a confusion matrix in which zero confusion is the equivalent of perfect agreement. Kappa can be viewed then as a measure of confusion reduction. A complementary measure, confusion infusion is defined. Its usefulness is illustrated with live data from a large‐scale testing program where e‐rater®, an automatic essay‐scoring algorithm, is used in place of a second reader. The confusion reduction and confusion infusion indices help make comparisons among the relative efficacy of two versions of e‐rater, and two other methods of assigning scores, a second reader and assigning all candidates the mode of the first reading.
VALIDATING LANGUEDGE™ COURSEWARE SCORES AGAINST FACULTY RATINGS AND STUDENT SELF‐ASSESSMENTS
Authors: Powers Donald E. , Roever Carsten , Huff Kristin L. , Trapani Catherine S. 
Published: August 2014

ABSTRACTLanguEdge™ Courseware is a software tool that is designed to help teachers of English as a second language (ESL) build and assess the communicative skills of their students. The purpose of this study was to generate information to help LanguEdge Courseware users understand better the meaning (or validity) of the assessment scores bases on the LanguEdge Courseware. Specifically, the objective was to describe, for each of the four sections of the LanguEdge assessment, relevant characteristics of test takers at various test score levels. To accomplish this objective, we gathered data that represent two different perspectives—those of instructors and those of students themselves.Approximately 3,000 students each took one of two parallel forms of the LanguEdge assessment at domestic and international testing sites. Participants also completed a number of self‐assessment questions about their English language skills. In addition, for some study participants, instructors rated selected language skills.LanguEdge test scores related moderately (correlations mostly in the .30s and .40s) with student self‐assessments. Of the four LanguEdge tests, Listening exhibited the strongest relationships to self‐assessments; Speaking, the next strongest; Reading, the next; and Writing, the least.The correlations of faculty ratings with each of the LanguEdge section test scores were generally in the .40s, with some reaching the .50s. The correlations between the various student self‐assessment scales and faculty ratings were modest, mostly in the .30s. These correlations suggest that students and faculty had different perspectives on students' English language skills.
A BRIEF INTRODUCTION TO EVIDENCE‐CENTERED DESIGN
Authors: Mislevy Robert J. , Almond Russell G. , Lukas Janice F. 
Published: August 2014

ABSTRACTEvidence‐centered assessment design (ECD) is an approach to constructing educational assessments in terms of evidentiary arguments. This paper provides an introduction to the basic ideas of ECD, as well as some of the terminology and models that have been developed to implement the approach. In particular, it presents the high‐level models of the Conceptual Assessment Framework (CAF) and the Four‐process Delivery Architecture for assessment delivery systems. Special attention is given to the role of probability‐based reasoning in accumulating evidence across task performances, in terms of belief about unobservable variables that characterize the knowledge, skills, and/or abilities of students. This is the role traditionally associated with psychometric models, such as those of item response theory (IRT) and latent class models. To unify the ideas and to provide a foundation for extending probability‐based reasoning in assessment applications more broadly, however, a more general expression in terms of graphical models is indicated. This brief overview of evidence‐centered design provides the reader with a feel for where and how graphical models fit into the larger enterprise of educational and psychological assessment. A simple example based on familiar large‐scale standardized tests such as the Graduate Record Examinations® (GRE®) is used to fix ideas.
INVESTIGATING THE VALIDITY OF TOEFL: A FEASIBILITY STUDY USING CONTENT AND CRITERION‐RELATED STRATEGIES
Authors: Rosenfeld Michael , Oltman Philip K. , Sheppard Ken 
Published: August 2014

ABSTRACTThe purpose of this study was to investigate the feasibility of two complementary approaches to assessing the validity of the TOEFL examination. One approach used evidence based on test content. In the context described in this report, evidence based on test content refers to the degree to which the items on the TOEFL examination are representative of the knowledge and skills required to demonstrate English proficiency in undergraduate and graduate programs throughout the United States and Canada. The content‐oriented approach used in this pilot study involved item rating procedures that were designed to evaluate and document the relationship between the language tasks or behaviors previously identified as important for academic success and the test items used to measure them. The second approach used a criterion‐related validation strategy. In this aspect of the study, experimental rating scales were developed for use by faculty to evaluate students' current levels of English language proficiency. These scales were designed to sample the domain of behaviors previously identified as important.
PARTIALLY OBSERVED MIXTURES OF IRT MODELS: AN EXTENSION OF THE GENERALIZED PARTIAL CREDIT MODEL
Authors: von Davier Matthias , Yamamoto Kentaro 
Published: August 2014

ABSTRACTThe generalized partial credit model (GPCM) is used frequently in educational testing and in large‐scale assessments for analyzing polytomous data. Special cases of the generalized partial credit model are the partial credit model—or Rasch model for ordinal data—on the one hand and the 2‐parameter logistic (2PL) model on the other hand. The extension proposed here enables the use of the GPCM in discrete mixture item response theory (IRT) models with partially observed mixture information. Partially observed mixtures are discrete mixture distributions with missing grouping information for some, but not all, observations. This concept of partially observed mixtures integrates multigroup and mixture distribution IRT models. The proposed model includes the latent class analysis (LCA) and (multigroup) IRT models as well as mixture distribution IRT models as submodels. An application of the proposed partially observed mixture IRT model is presented. This application classifies about 80% of data from a large‐scale mathematics assessment into three school types and estimates a mixture IRT model for the whole sample in which the grouping information is unknown for 20% of the sample. The aim of the method is twofold: first, to estimate the latent distributions by school type and, second, to unmix the data with missing school types into the groups established by the portion of the sample with known school‐type information.
PRACTICAL APPLICATIONS OF POSTERIOR PREDICTIVE MODEL CHECKING FOR ASSESSING FIT OF COMMON ITEM RESPONSE THEORY MODELS
Authors: Sinharay Sandip 
Published: August 2014

ABSTRACTModel checking in item response theory (IRT) is an underdeveloped area. No unanimous model checking tool exists in this field even for the simple models, not to speak of the recently suggested complicated models. Theposterior predictive model checkingmethod (Guttman, 1967; Rubin, 1981, 1984) is a popular Bayesian model checking tool because of its simplicity, strong theoretical basis, intuitive appeal, and ability to provide graphical evidence. This paper applies the method to a number of real data examples. An important issue with the application of the posterior predictive model checking method is the choice ofdiscrepancy measures(which are like thetest statisticsin classical hypothesis testing). This paper also uses the discrepancy measures that are found useful in Sinharay and Johnson (2003). The posterior predictive model checking method seems to be promising in detecting different types of misfit of the common IRT models in real applications.
NEW SAT® WRITING PROMPT STUDY: ANALYSES OF GROUP IMPACT AND RELIABILITY
Authors: Breland Hunter , Kubota Melvin , Nickerson Kristine , Trapani Catherine , Walker Michael 
Published: August 2014

ABSTRACTThis study investigated the impact on ethnic, language, and gender groups of a new kind of essay prompt type intended for use with the new SAT®. The study also generated estimates of the reliability of scores obtained using the prompts examined. To examine the impact of a new prompt type, random samples of eleventh‐grade students in 49 participating high schools were administered writing tests using four different prompts, two of an old type and two of a new type. To obtain estimates of the reliability of scores for the old and new types of prompts, schools were asked to participate in a second round of testing to occur four months after the initial testing. Results of the impact analyses revealed no significant prompt type effects for ethnic, gender, or language groups, although there were significant differences in mean scores for ethnic and gender groups for all prompts. The score reliability estimates obtained were similar to those obtained in previous studies.
ASSESSING FIT OF MODELS WITH DISCRETE PROFICIENCY VARIABLES IN EDUCATIONAL ASSESSMENT
Authors: Sinharay Sandip , Almond Russell , Yan Duanli 
Published: August 2014

ABSTRACTModel checking is a crucial part of any statistical analysis. As educators tie models for testing to cognitive theory of the domains, there is a natural tendency to represent participant proficiencies with latent variables representing the presence or absence of the knowledge, skills, and proficiencies to be tested (Mislevy, Almond, Yan, & Steinberg, 2001). Model checking for these models is not straightforward, mainly because traditional χ2‐type tests do not apply except for assessments with a small number of items. Williamson, Mislevy, and Almond (2000) note a lack of published diagnostic tools for these models.This paper suggests a number of graphics and statistics for diagnosing problems with models with discrete proficiency variables. A small diagnostic assessment first analyzed by Tatsuoka (1990) serves as a test bed for these tools. This work is a continuation of the recent work by Yan, Mislevy, and Almond (2003) on this data set. Two diagnostic tools that prove useful are Bayesian residual plots and an analog of the item characteristic curve (ICC) plots. A χ2‐type statistic based on the latter plot shows some promise, but more work is required to establish the null distribution of the statistic. On the basis of the identified problems with the model used by Mislevy (1995), the suggested diagnostics are helpful to hypothesize an improved model that seems to fit better.
THE BIOGRAPHICAL INVENTORY IN NAVAL AVIATION SELECTION: INSIDE THE BLACK BOX
Authors: Stricker Lawrence J. 
Published: August 2014

ABSTRACTA biographical inventory has been used in the selection of students for naval aviation training since World War II, and its validity in predicting their retention‐attrition in this training has been well established. This study investigated the constructs underlying the inventory and their relations to students' retention‐attrition. A factor analysis of the items on the inventory for student pilots identified five factors. One factor, being a commissioned officer, appeared to account for the inventory's validity.
RELIABILITY OF SPEEDED NUMBER‐RIGHT MULTIPLE‐CHOICE TESTS
Authors: Attali Yigal 
Published: August 2014

ABSTRACTContrary to common belief, reliability estimates of number‐right multiple‐choice tests are not inflated by speededness. Because examinees guess on questions when they run out of time, the responses to these questions show less consistency with the responses of other questions, and the reliability of the test will be decreased. The surprising implication is that adding questions to a multiple‐choice test may lower its reliability when the test is speeded. This paper develops the mathematical derivations and shows the effects of speededness on reliability in simulations.
MODEL DIAGNOSTICS FOR BAYESIAN NETWORKS
Authors: Sinharay Sandip 
Published: August 2014

ABSTRACTAssessing fit of psychometric models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. Bayesian networks, frequently used in educational assessments (see, for example, Mislevy, Almond, Yan, & Steinberg, 2001) primarily for learning about students' knowledge and skills, are no exception. This paper employs theposterior predictive model checkingmethod (Guttman, 1967; Rubin, 1984), a popular Bayesian model checking tool, to assess fit of simple Bayesian networks. A number of aspects of model fit, those of usual interest to practitioners, are assessed in this paper using various diagnostic tools. The first diagnostic used is direct data display—a visual comparison of the observed data set and a number of the posterior predictive data sets (that are predicted by the model). The second aspect examined here is item fit. Examinees are grouped into a number of equivalence classes, based on the generated values of their skill variables, and the observed and expected proportion correct scores on an item for the classes are combined to provide a χ2‐type and aG2‐type test statistic for each item. Another (similar) set of χ2‐type andG2‐type test statistic is obtained by grouping the examinees by their raw scores and then comparing their observed and expected proportion correct scores on an item. This paper also suggests how to obtain posterior predictive p‐values, natural candidate p‐values from a Bayesian viewpoint, for the χ2‐type andG2‐type test statistics. The paper further examines the association among the items, especially if the model can explain the odds ratios corresponding to the responses to pairs of items. Finally, in an effort to examine the issue of differential item functioning (DIF), this paper suggests a version of the Mantel‐Haenszel statistic (Holland, 1985), which uses “matched groups” based on equivalence classes, as a discrepancy measure with posterior predictive model checking. Limited simulation studies and a real data application examine the effectiveness of the suggested model diagnostics.
INVESTIGATING DIFFERENCES IN EXAMINEE PERFORMANCE BETWEEN COMPUTER‐BASED AND HANDWRITTEN ESSAYS
Authors: Yu Lei , Livingston Samuel A. , Larkin Kevin C. , Bonett John 
Published: August 2014

ABSTRACTThis study compared essay scores from paper‐based and computer‐based versions of a writing test for prospective teachers. Scores for essays in the paper‐based version averaged nearly half a standard deviation higher than those in the computer‐based version, after applying a statistical control for demographic differences between the groups of examinees taking the two versions. The statistical control was implemented by means of a propensity score, applying weights to members of one group to match the propensity score distribution of the other group. The score difference between the groups did not change substantially when the analysis was restricted to examinees taking the same mix of essay topics or to examinees taking one particular essay.
TOWARD ACCESSIBLE COMPUTER‐BASED TESTS: PROTOTYPES FOR VISUAL AND OTHER DISABILITIES
Authors: Hansen Eric G. , Forer Douglas C. , Lee Moon J. 
Published: August 2014

ABSTRACTThere is a great need to explore approaches for developing computer‐based testing systems that are more accessible for people with disabilities. This report explores three prototype test delivery approaches, describing their development and formative evaluations. Fifteen adults, 2 to 4 from each of the six disability statuses—blindness, low vision, deafness, deaf‐blindness, learning disability, and no disability—participated in a formative evaluation of the systems. Each participant was administered from 2 to 15 items in each of one or two of the systems. The study found that although all systems had weaknesses that should be addressed, almost all of the participants (13 of 15) would recommend at least one of the delivery methods for high‐stakes tests, such as those for college or graduate admissions. The report concludes with recommendations for additional research that testing organizations seeking to develop accessible computer‐based testing systems can consider.
STEREOTYPE THREAT, THE TEST‐CENTER ENVIRONMENT, AND PERFORMANCE ON THE GRE GENERAL TEST
Authors: Walters Alyssa M. , Lee Soonmook , Trapani Catherine 
Published: August 2014

ABSTRACTThe study investigated the applicability of previous experimental research on stereotype threat to operational Graduate Record Examinations® (GRE®) General Test testing centers. The goal was to document any relationships between features of the testing environment that might cue stereotype threat as well as any impact on GRE test scores among African American, Hispanic, Asian American, and female test‐takers. Among such features were the gender and ethnicity of test proctors and more general factors, such as the size, activity level, and social atmosphere of test centers. Our analyses revealed several relationships among environmental factors and several variations in test performance for all groups. However, we found no direct support for stereotype threat and, in fact, found some effects for proctor ethnicity that ran counter to a stereotype‐threat explanation.