bluuebunny/crossref_metadata_2025_split huggingface dataset:
EDUCATIONAL ASSESSMENT USING INTELLIGENT SYSTEMS
Authors: Shute Valerie J. , Zapata‐Rivera Diego 
Published: August 2014

ABSTRACTRecent advances in educational assessment, cognitive science, and artificial intelligence have made it possible to integrate valid assessment and instruction in the form of modern computer‐based intelligent systems. These intelligent systems leverage assessment information that is gathered from various sources (e.g., summative and formative). This paper analyzes the role of educational assessment in intelligent systems, summarizes the characteristics of successfully deployed intelligent systems, and describes an evidence‐based approach to incorporating valid and reliable assessments into enhanced intelligent systems.
THE FUSION MODEL FOR SKILLS DIAGNOSIS: BLENDING THEORY WITH PRACTICALITY
Authors: Hartz Sarah , Roussos Louis 
Published: August 2014

ABSTRACTThis paper presents the development of the fusion model skills diagnosis system (fusion model system), which can help integrate standardized testing into the learning process with both skills‐level examinee parameters for modeling examinee skill mastery and skills‐level item parameters, giving information about the diagnostic power of the test. The development of the fusion model system involves advancements in modeling, parameter estimation, model‐fitting methods, and model‐fit evaluation procedures, which are described in detail in the paper. To document the accuracy of the estimation procedure and the effectiveness of the model‐fitting and model‐fit evaluation procedures, this paper also presents a series of simulation studies. Special attention is given to evaluating the robustness of the fusion model system to violations of various modeling assumptions. The results demonstrate that the fusion model system is a promising tool for skills diagnosis that merits further research and development.
EVALUATING THE CONSTRUCT‐COVERAGE OF THE E‐RATER® SCORING ENGINE
Authors: Quinlan Thomas , Higgins Derrick , Wolff Susanne 
Published: August 2014

ABSTRACTThis report evaluates the construct coverage of the e‐rater® scoring engine. The matter of construct coverage depends on whether one defines writing skill, in terms of process or product. Originally, the e‐rater engine consisted of a large set of components with a proven ability to predict human holistic scores. By organizing these capabilities into features, e‐rater researchers organized the e‐rater engine along the lines of trait scoring, which recognizes that essay quality has several dimensions. Some traits of essay quality cut across different methods for scoring essay quality, such as the rubrics employed by the GRE® and TOEFL® assessments, as well as the 6‐trait scoring model. Factor analyses conducted by Attali and Powers (2008) suggest that e‐rater features capture low‐level aspects of essay quality, such as sentence complexity, vocabulary, and conventions. Future e‐rater development should focus on (a) deepening and expanding coverage of the construct, such as by developing measures of essay content and organization, as well as on (b) addressing accuracy issues in existing features.
EFFICIENT FULL INFORMATION MAXIMUM LIKELIHOOD ESTIMATION FOR MULTIDIMENSIONAL IRT MODELS
Authors: Rijmen Frank 
Published: August 2014

ABSTRACTMaximum marginal likelihood estimation of multidimensional item response theory (IRT) models has been hampered by the calculation of the multidimensional integral over the ability distribution. However, the researcher often has a specific hypothesis about the conditional (in)dependence relations among the latent variables. Exploiting these relations may result in more efficient estimation algorithms. A well‐known example is the bi‐factor model, in which each item measures a general dimension and one ofKother dimensions, for which Gibbons and Hedeker (1992) showed that full information maximum likelihood estimation only requires the integration over two‐dimensional integrals. In this paper, it is shown how the approach of Gibbons and Hedeker (1992) can be placed into a graphical model framework. The advantage of the graphical model framework is that efficient estimation schemes can be derived in a fully automatic way by applying algorithms to the graphical representation of a statistical model. This renders the approach fairly generally applicable, and tedious derivations by hand are no longer involved. The generality of the approach is demonstrated by applying it to a multidimensional IRT model with a second order dimension. It turns out that full information maximum likelihood estimation for such a model also requires the evaluation of two‐dimensional integrals only.
SELECTION STRATEGIES FOR BIVARIATE LOGLINEAR SMOOTHING MODELS AND THEIR EFFECTS ON NEAT EQUATING FUNCTIONS
Authors: Moses Tim , Holland Paul 
Published: August 2014

ABSTRACTThis study evaluated 8 statistical strategies for selecting the parameterizations of loglinear models for bivariate test score distributions and for using these strategies in equating. Four of the strategies were based on significance tests of chi‐square statistics (likelihood ratio, Pearson, Freeman‐Tukey, and Cressie‐Read chi‐squares) and 4 additional strategies were based on different evaluations of the likelihood ratio chi‐square statistic (Akaike information criterion [AIC], Bayesian information criterion [BIC], consistent Akaike information criterion [CAIC], and Goodman). The focus was on the selection strategies' accuracies in selecting bivariate parameterizations and the implications of using each selection strategy for chained and poststratification equating functions. The results showed that selection strategies differ in terms of their tendencies to select complex versus simple bivariate parameterizations and that the strategies that tended to select more complex parameterizations (i.e., Pearson and Cressie‐Read chi‐squares) usually produced the least biased and most variable equating functions.
STOCHASTIC APPROXIMATION METHODS FOR LATENT REGRESSION ITEM RESPONSE MODELS
Authors: von Davier Matthias , Sinharay Sandip 
Published: August 2014

ABSTRACTThis paper presents an application of a stochastic approximation EM‐algorithm using a Metropolis‐Hastings sampler to estimate the parameters of an item response latent regression model. Latent regression models are extensions of item response theory (IRT) to a 2‐level latent variable model in which covariates serve as predictors of the conditional distribution of ability. Applications for estimating latent regression models for data from the 2000 National Assessment of Educational Progress (NAEP) grade 4 math assessment and the 2002 grade 8 reading assessment are presented and results of the proposed method are compared to results obtained using current operational procedures.
AN EXPLORATORY STUDY INTO INTERLANGUAGE PRAGMATICS OF REQUESTS: A GAME OF PERSUASION
Authors: Yang Hui‐Chun , Zapata‐Rivera Diego 
Published: August 2014

ABSTRACTTeachability and assessment of interlanguage pragmatics are two crucial but still relatively unexplored areas in second language research. Several instruments, such as multiple‐choice discourse completion tasks, written discourse completion tasks, role‐play tasks, and discourse self‐assessment tasks (J. D. Brown, 2001), have been used to assess students' interlanguage pragmatic skills. This paper describes a new assessment‐based gaming environment of English as a second language (ESL) pragmalinguistics called A Game of Persuasion that focuses on the speech act of request. This game allows users to engage in interactive written dialogue with an artificial professor (or pedagogical agent) in multiple academic contexts. Students explore contextually and socially appropriate request strategies while the system scores each attempt, assigns points, and provides formative and summative feedback. This paper presents (a) related research on instruments that have been used in the area of interlanguage pragmatics (i.e., request), (b) our assessment‐based gaming environment, and (c) reports on results from a study aimed at exploring usability aspects of the game and usage of request strategies employed by 30 English language learners (ELLs) who completed 10 written interactive request tasks with and without the presence of a pedagogical agent.
WHAT EFFECT DOES THE INCLUSION OR EXCLUSION OF REPEATERS HAVE ON TEST EQUATING?
Authors: Puhan Gautam 
Published: August 2014

ABSTRACTThis study investigated the effect of inclusion or exclusion of test repeaters on equating results. New forms of two tests were equated to their respective old forms using either all examinees or only the first‐time examinees in the new form sample. Since the old forms were the first forms administered for these tests, there were no repeaters and therefore the old form samples were unaltered. Results showed that for both tests, inclusion or exclusion of repeaters in the new form sample had very little impact on the final equating results. These findings do not provide enough evidence to support routine exclusion of repeaters from the equating sample before conducting the equating. Finally, some examples of simple analyses to assist the testing practitioner when making a judgment about whether including or excluding repeaters in the equating sample is reasonable are provided.
ENGLISH LANGUAGE LEARNERS WITH READING DISABILITIES: A REVIEW OF THE LITERATURE AND THE FOUNDATION FOR A RESEARCH AGENDA
Authors: Shore Jane R. , Sabatini John 
Published: August 2014

ABSTRACTThe issue of identifying reading difficulties and disabilities in English language learners (ELLs) is a complex one. It is an area that draws on the diverse disciplines of first‐ and second‐language acquisition, literacy, English language learning, and reading, including differences and disabilities research. This literature review aims to synthesize the research that aims to address the topic, focusing on the following three questions:
How does one identify reading difficulties/disabilities (RD) in individuals who are learning the English language?
What needs to be done to aid in identifying RD in ELLs?
What does one do with this information to support the English reading instruction of ELLs with RD?
The first section of the report covers basic terminology and gives an overview of the problem. The second section presents some of the challenges in distinguishing between what might be normal ELL language development from what might be identified as a disability or difference. The third section reviews reading difficulties in various languages and then focuses on reading difficulties when learning a second language. The fourth section provides an overview of the issues surrounding the process of identifying reading difficulties or differences in ELLs. The challenges begin at referral to services and span to measurement of disabilities.1 The fifth section provides information on interventions and the application of information gained through assessment. The sixth section, the conclusion, points to key considerations related to the identification of RD in ELLs on the K‐12 level, especially in the area of the definition, measurement, and instruction of ELLs identified with RD. It also examines the potential for further development and research of assessment and effective instructional programs. Key findings and implications are summarized in the areas of RDs, referral, current assessment, assessments in development, instructional practices, and teacher preparation.
THE INFLUENCE OF NONCOGNITIVE DOMAINS ON ACADEMIC ACHIEVEMENT IN K‐12
Authors: Lee Jihyun , Shute Valerie J. 
Published: August 2014

ABSTRACTThis paper identifies three noncognitive domains relevant for academic achievement in K‐12—student engagement, behavioral learning strategies, and school climate. The paper also documents empirical findings that show relationships between these three noncognitive domains and academic achievement, especially in the areas of reading and mathematics.
COMPARISON OF THE EFFECTS OF DISCRETE ANCHOR ITEMS AND PASSAGE‐BASED ANCHOR ITEMS ON OBSERVED‐SCORE EQUATING RESULTS
Authors: Zu Jiyun , Liu Jinghua 
Published: August 2014

ABSTRACTEquating of tests composed of both discrete and passage‐based items using the nonequivalent groups with anchor test (NEAT) design is popular in practice. This study investigated the impact of discrete anchor items and passage‐based anchor items on observed score equating via simulation. Results suggested that an anchor with a larger proportion of passage‐based items and/or a larger degree of local dependence among passage‐based items produces larger equating errors, especially when group ability differences are not minimal. Our findings challenge the common belief that an anchor should be a miniature version of the tests to be equated.
AN EMPIRICAL COMPARISON OF METHODS FOR EQUATING WITH RANDOMLY EQUIVALENT GROUPS OF 50 TO 400 TEST TAKERS
Authors: Livingston Samuel A. , Kim Sooyeon 
Published: August 2014

ABSTRACTA series of resampling studies investigated the accuracy of equating by four different methods in a random groups equating design with samples of 400, 200, 100, and 50 test takers taking each form. Six pairs of forms were constructed. Each pair was constructed by assigning items from an existing test taken by 9,000 or more test takers. The criterion equating was the direct equipercentile equating in the full group. Accuracy was described in terms of the root‐mean‐squared deviation (over 1,000 replications) of the sample equatings from the criterion equating. The equating methods investigated were equipercentile equating of smoothed distributions, linear equating, mean equating, and circle‐arc equating; they were compared with each other and with the identity. Circle‐arc equating produced the most accurate results for all sample sizes investigated, particularly in the upper half of the score distribution.
HOW CAN MULTIDIMENSIONAL ITEM RESPONSE THEORY BE USED IN REPORTING OF SUBSCORES?
Authors: Haberman Shelby J. , Sinharay Sandip 
Published: August 2014

ABSTRACTRecently, there has been increasing interest in reporting diagnostic scores. This paper examines reporting of subscores using multidimensional item response theory (MIRT) models. An MIRT model is fitted using a stabilized Newton‐Raphson algorithm (Haberman, 1974, 1988) with adaptive Gauss‐Hermite quadrature (Haberman, von Davier, & Lee, 2008). A new statistical approach is proposed to assess when subscores using the MIRT model have any added value over (a) the total score or (b) subscores based on classical test theory (Haberman, 2008; Haberman, Sinharay, & Puhan, 2006). The MIRT‐based methods are applied to several operational data sets. The results show that the subscores based on MIRT are slightly more accurate than subscore estimates derived by classical test theory.
INFORMATION AND COMMUNICATION TECHNOLOGY LITERACY: WHAT DO BUSINESSES EXPECT AND WHAT DO BUSINESS SCHOOLS TEACH?
Authors: Ali Radwan , Katz Irvin R. 
Published: August 2014

ABSTRACTToday's information and communications technology (ICT) provides unprecedented amounts of information to organizations and their employees. This overabundance challenges workers, placing an increasing premium on skills of sifting through information of sometimes dubious quality, integrating information critically, and producing well‐reasoned conclusions. This paper explores the hypothesis that employers want new hires skilled in ICT literacy, which includes navigating information effectively and using technology efficiently. Based on an ICT literacy framework for higher education, two surveys were created to assess the opinions of human resources (HR) consultants and business school faculty. HR consultants endorsed practically all elements of the framework as being important or essential for new hires. Especially valued were skills associated with information security, confidentiality, and ethical behavior. However, business school faculty did not report a corresponding focus on ICT literacy generally, or ethical/legal issues in particular, in their instruction. The results suggest that while ICT literacy does indeed appear to be among the key skills for today's workforce, much work must still be done to integrate these skills throughout the business school undergraduate curriculum.
MEASUREMENT ISSUES IN STATE ACHIEVEMENT COMPARISONS
Authors: Zwick Rebecca 
Published: August 2014

ABSTRACTPolitical perspectives on the advisability of state achievement comparisons have changed substantially over the last half‐century. As Selden (2004) noted regarding the National Assessment of Educational Progress (NAEP), “[w]hen it began in the 1960s, NAEP specifically was designed not to provide comparisons among states. This was done for political reasons, so that the assessment program would be palatable to the educational community and to the states” (p. 195). During the No Child Left Behind Era, state comparisons were complicated by the use of tests and proficiency standards that differed across states. Because of today's Common Core Standards Initiative, state comparisons are once again at the forefront of educational policy discussions. The initiative brings with it both significant opportunities and substantial psychometric challenges.
THE USE OF TWO ANCHORS IN NONEQUIVALENT GROUPS WITH ANCHOR TEST (NEAT) EQUATING
Authors: Moses Tim , Deng Weiling , Zhang Yu‐Li 
Published: August 2014

ABSTRACTIn the equating literature, a recurring concern is that equating functions that utilize a single anchor to account for examinee groups' nonequivalence are biased when the groups are extremely different and/or when the anchor only weakly measures what the tests measure. Several proposals have been made to address this equating bias by incorporating more than one anchor into nonequivalent groups with anchor test (NEAT) equating functions. These proposals have not been extensively considered or comparatively evaluated. This study evaluates three methods for incorporating more than one anchor into NEAT equating functions, including poststratification, imputation, and propensity score matching. The three methods are studied and compared in two examples. The implications for using the three equating approaches in practice and for developing alternative strategies to incorporate two anchors are discussed.
RESEARCH ON STANDARD ERRORS OF EQUATING DIFFERENCES
Authors: Moses Tim , Zhang Wenmin 
Published: August 2014

ABSTRACTIn this paper, thestandard error of equating difference(SEED) is described in terms of originally proposed kernel equating functions (von Davier, Holland & Thayer, 2004) and extended to incorporate traditional linear and equipercentile functions. These derivations expand on prior developments of SEEDs and standard errors of equating and provide additional insight about the relationships of kernel and traditional equating functions. Simulations are used to evaluate the SEEDs' accuracies.
AN EXAMINATION OF THE LINK BETWEEN RATER CALIBRATION PERFORMANCE AND SUBSEQUENT SCORING ACCURACY IN GRADUATE RECORD EXAMINATIONS® (GRE®) WRITING
Authors: Ricker‐Pedley Kathryn L. 
Published: August 2014

ABSTRACTA pseudo‐experimental study was conducted to examine the link between rater accuracy calibration performances and subsequent accuracy during operational scoring. The study asked 45 raters to score a 75‐response calibration set and then a 100‐response (operational) set of responses from a retired Graduate Record Examinations® (GRE®) writing prompt that had been vetted to determine correct scores prior to the study. The study found a positive relationship between calibration accuracy and scoring accuracy. Results suggest that a longer calibration test is more reliable and has a stronger correlation with operational scoring exact agreement; however, even tests as short as 10 responses had a reasonable correlation with operational accuracy, and the shorter test classified raters accurately 87% of the time when compared to the 75‐response criterion. The additional requirement of no discrepant scores during calibration had better correlation with exact agreement, but much poorer classification accuracy in the shorter tests. This research suggests that calibration is useful for screening raters in its current form, but that further investigation of passing standards and of less controlled operational scoring conditions is required.
WRITING ASSESSMENT AND COGNITION
Authors: Deane Paul 
Published: August 2014

ABSTRACTThis paper presents a socio‐cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writing assessment, and activity and genre theories; and presents a specific proposal about how writing assessment can be organized to promote best practices in writing instruction.
SMOOTHING AND EQUATING METHODS APPLIED TO DIFFERENT TYPES OF TEST SCORE DISTRIBUTIONS AND EVALUATED WITH RESPECT TO MULTIPLE EQUATING CRITERIA
Authors: Moses Tim , Liu Jinghua 
Published: August 2014

ABSTRACTIn equating research and practice, equating functions that are smooth are typically assumed to be more accurate than equating functions with irregularities. This assumption presumes that population test score distributions are relatively smooth. In this study, two examples were used to reconsider common beliefs about smoothing and equating. The first example involves a relatively smooth population test score distribution and the second example involves a population test score distribution with systematic irregularities. Various smoothing and equating methods (presmoothing, equipercentile, kernel, and postsmoothing) were compared across the two examples with respect to how well the test score distributions were reflected in the equating functions, the smoothness of the equating functions, and the standard errors of equating. The smoothing and equating methods performed more similarly in the first example than in the second example. The results of the second example illustrate that when dealing with systematically irregular test score distributions, smoothing and equating methods can be used in different ways to satisfy different equating criteria.
VALIDATION OF AUTOMATED SCORES OF TOEFL IBT®TASKS AGAINST NONTEST INDICATORS OF WRITING ABILITY
Authors: Weigle Sara Cushing 
Published: August 2014

ABSTRACTAutomated scoring has the potential to dramatically reduce the time and costs associated with the assessment of complex skills such as writing, but its use must be validated against a variety of criteria for it to be accepted by test users and stakeholders. This study addresses two validity‐related issues regarding the use of e‐rater®with the independent writing task on the TOEFL iBT®(Internet‐based test). First, relationships between automated scores of iBT tasks and nontest indicators of writing ability were examined. This was followed by exploration of prompt‐related differences in automated scores of essays written by the same examinees. Correlations between both human and e‐rater scores and nontest indicators were moderate but consistent, with few differences between e‐rater and human rater scores. E‐rater was more consistent across prompts than individual human raters, although there were differences in scores across prompts for the individual features used to generate total e‐rater scores.