bluuebunny/crossref_metadata_2025 huggingface dataset:
FACTOR STRUCTURE OF THE TOEFL® INTERNET‐BASED TEST (IBT): EXPLORATION IN A FIELD TRIAL SAMPLE
Authors: Sawaki Yasuyo , Stricker Lawrence , Oranje Andreas
Published: August 2014

The present study investigated the factor structure of a field trial sample of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT). An item‐level confirmatory factor analysis (CFA) was conducted for a polychoric correlation matrix of items on a test form completed by 2,720 participants in the 2003–2004 TOEFL iBT Field Study. CFA‐based multitrait‐multimethod (MTMM) analyses for the Reading and Listening sections showed that the language abilities assessed in each section were essentially unidimensional, while the factor structure of the entire test was best represented by a higher‐order factor model with a general factor (English as a second language/English as a foreign language ability) and four group factors for reading, listening, speaking, and writing. The integrated Speaking and Writing tasks, which require language processing in multiple modalities, well defined the target modalities (speaking and writing). These results broadly support the current reporting of four scores corresponding to the modalities and a total score, as well as the test design where the integrated tasks contribute only to the scores for the target modalities.
COMPARING DIFFERENT APPROACHES OF BIAS CORRECTION FOR ABILITY ESTIMATION IN IRT MODELS
Authors: Lee Yi‐Hsuan , Zhang Jinming
Published: August 2014

The method of maximum‐likelihood is typically applied to item response theory (IRT) models when the ability parameter is estimated while conditioning on the true item parameters. In practice, the item parameters are unknown and need to be estimated first from a calibration sample. Lewis (1985) and Zhang and Lu (2007) proposed the expected response functions (ERFs) and the corrected weighted‐likelihood estimator (CWLE), respectively, to take into account the uncertainty regarding item parameters for purposes of ability estimation. In this paper, we investigate the performance of ERFs and of the CWLE in different situations, such as various test lengths and levels of measurement error in item parameter estimation. Our empirical results indicate that ERFs can cause the bias in ability estimation to fall within [‐0.2, 0.2] for all conditions, whereas the CWLE can effectively reduce the bias in ability estimation provided that it has a good foundation to start from.
THEORETICAL AND EMPIRICAL STANDARD ERRORS FOR TWO POPULATION INVARIANCE MEASURES IN THE LINEAR EQUATING CASE
Authors: von Davier Alina A. , Manalo Jonathan R. , Rijmen Frank
Published: August 2014

The standard errors of the 2 most widely used population‐invariance measures of equating functions, root mean square difference (RMSD) and root expected mean square difference (REMSD), are not derived for common equating methods such as linear equating. Consequently, it is unknown how much noise is contained in these estimates. This paper describes 2 methods for obtaining the standard errors for RMSD and REMSD. The delta method relies on an analytical approximation and provides asymptotic standard errors. The grouped jackknife method is a sampling‐based method. Both methods were applied to a real data application. The results showed that there was very little difference between the standard errors found by the 2 methods.
EQUATING OF MIXED‐FORMAT TESTS IN LARGE‐SCALE ASSESSMENTS
Authors: Kim Sooyeon , Walker Michael E. , McHale Frederick
Published: August 2014

This study examined variations of the nonequivalent‐groups equating design for mixed‐format tests—tests containing both multiple‐choice (MC) and constructed‐response (CR) items—to determine which design was most effective in producing equivalent scores across the two tests to be equated. Four linking designs were examined: (a) an anchor with only MC items; (b) a mixed‐format anchor containing both MC and CR items; (c) a mixed‐format anchor incorporating CR item rescoring; and (d) a hybrid combining single‐group and equivalent‐groups designs, thereby avoiding the need for an anchor test. Designs using MC items alone or those using a mixed anchor without CR item rescoring resulted in much larger bias than the other two design approaches. The hybrid design yielded the smallest root mean squared error value.
FITTING THE STRUCTURED GENERAL DIAGNOSTIC MODEL TO NAEP DATA
Authors: Xu Xueli , von Davier Matthias
Published: August 2014

Xu and von Davier (2006) demonstrated the feasibility of using the general diagnostic model (GDM) to analyze National Assessment of Educational Progress (NAEP) proficiency data. Their work showed that the GDM analysis not only led to conclusions for gender and race groups similar to those published in the NAEP Report Card, but also allowed flexibility in estimating multidimensional skills simultaneously. However, Xu and von Davier noticed that estimating the latent skill distributions will be much more challenging with this model when there is a large number of subgroups to estimate. To make the GDM more applicable to NAEP data analysis, which requires a fairly large subgroups analysis, this study developed a log‐linear model to reduce the number of parameters in the latent skill distribution without sacrificing the accuracy of inferences. This paper describes such a model and applies the model in the analysis of NAEP reading assessments for 2003 and 2005. The comparisons between using this model and the unstructured model were made through the use of various results, such as the differences between item parameter estimates and the differences between estimated latent class distributions. The results in general show that using the log‐linear model is efficient.
STEREOTYPE THREAT SPILLOVER AND SAT® SCORES
Authors: Walker Michael E. , Bridgeman Brent
Published: August 2014

A recent study by Beilock, Rydell, and McConnell (2007) suggested that stereotype threat experienced in one domain (e.g., math) triggered by knowledge of a negative stereotype about a social group in that particular domain can spill over into subsequent tasks in totally unrelated domains (e.g., reading). The authors suggested that these findings might have implications for how the ordering of sections on standardized tests such as the SAT® or GRE could affect examinee performance. To test the authors' assertions, this study used data from a recent SAT administration in which either a reading, a math, or a writing section preceded a reading section. Performance on the subsequent reading section of members of a stereotype threatened group (i.e., women) who took the math section first was compared to performance of those who took the reading or writing section first. Results were inconsistent with the stereotype threat spillover hypothesis and serve to justify the warning of Cullen, Hardison, and Sackett (2004) to exercise caution in generalizing lab findings on stereotype threat to operational testing situations.
EVIDENCE‐CENTERED ASSESSMENT DESIGN FOR REASONING ABOUT ACCOMMODATIONS FOR INDIVIDUALS WITH DISABILITIES IN NAEP READING AND MATHEMATICS
Authors: Hansen Eric G. , Mislevy Robert J. , Steinberg Linda S.
Published: August 2014

Accommodations play a key role in enabling individuals with disabilities to participate in the National Assessment of Educational Progress (NAEP) and other large‐scale assessments. However, it can be difficult to know how accommodations affect the validity of results, thus making it difficult to determine which accommodations should be allowed. This study describes recent extension of evidence‐centered assessment design (ECD) for reasoning about the impact of accommodations and other accessibility features (e.g., universal design features) on the validity of assessment results, using examples from NAEP reading and mathematics. The study found that the ECD‐based techniques were useful in analyzing the effects of accommodations and other accessibility features on validity. Such design capabilities may increase assessment designers' capacity to employ accessibility features without undermining validity.
SMALL‐SAMPLE EQUATING BY THE CIRCLE‐ARC METHOD
Authors: Livingston Samuel A. , Kim Sooyeon
Published: August 2014

This paper suggests two new, related methods for estimating a test‐score equating relationship from small samples of test takers. These methods do not require the estimated equating transformation to be linear. Instead, they constrain the estimated equating curve to pass through 2 prespecified end‐points and a middle point determined from the data. Some preliminary results indicate that these methods outperform mean equating and other methods used for equating in small samples.
L‐BIVARIATE AND L‐MULTIVARIATE ASSOCIATION COEFFICIENTS
Authors: Kong Nan , Lewis Charles
Published: August 2014

Given a system of multiple random variables, a new measure called theL‐multivariate association coefficient is defined using (conditional) entropy. Unlike traditional correlation measures, theL‐multivariate association coefficient measures the multiassociations or multirelations among the multiple variables in the given system; that is, theL‐multivariate association coefficient measures the degree of the association for the given system. TheL‐multivariate association coefficient for the system of two random variables is also called theL‐bivariate association coefficient. The association measured by theL‐multivariate association coefficient is a general type of association, not any specific type of a linear or nonlinear association. Unlike theK‐dependence coefficient, which is an asymmetrical measure, theL‐multivariate association coefficient is a symmetrical measure. A direct application of theL‐multivariate association coefficient is in variables selection or variables reduction. This paper also explores the relationship between theL‐multivariate association coefficient and theK‐dependence coefficient.
A STUDY OF CONFIDENCE AND ACCURACY USING THE RASCH MODELING PROCEDURES
Authors: Paek Insu , Lee Jihyun , Stankov Lazar , Wilson Mark
Published: August 2014

This study investigated the relationship between students' actual performance (accuracy) and their subjective judgments of accuracy (confidence) on selected English language proficiency tests. The unidimensional and multidimensional IRT Rasch approaches were used to model the discrepancy between confidence and accuracy at the item and test level and to assess disattenuated strength of association between accuracy and confidence. The analysis results indicate a pattern of overconfidence bias (i.e., overestimation of success rate), which was related to item difficulty. In addition, the strength of association between accuracy and confidence dimension was relatively high: The confidence dimension explained 45% and 52% of the variability in the accuracy dimension for the two tests employed in this study.
A REVIEW OF RECENT DEVELOPMENTS IN DIFFERENTIAL ITEM FUNCTIONING
Authors: Mapuranga Raymond , Dorans Neil J. , Middleton Kyndra
Published: August 2014

In many practical settings, essentially the same differential item functioning (DIF) procedures have been in use since the late 1980s. Since then, examinee populations have become more heterogeneous, and tests have included more polytomously scored items. This paper summarizes and classifies new DIF methods and procedures that have appeared since the early 1990s and assesses their appropriateness for practical use. Widely used DIF methods are evaluated alongside these new methods for completeness, clarity, and comparability.
DEVELOPMENT AND VALIDITY EVIDENCE SUPPORTING A TEAMWORK AND COLLABORATION ASSESSMENT FOR HIGH SCHOOL STUDENTS
Authors: Zhuang Xiaohua , MacCann Carolyn , Wang Lijuan , Liu Lydia , Roberts Richard D.
Published: August 2014

Various policy papers and research studies assert that teamwork is one of the most important skills for students to learn if they are to become meaningful contributors to the 21st century workforce. However, outside of organizational psychology and adult populations, few reliable assessments of this construct exist, with suitable validity evidence scant or nonexistent. To redress this imbalance, teamwork assessments for high school students were developed using multiple methods: self‐report ratings, situational judgment testing, and teacher reports. Exploratory factor, confirmatory factor, and latent class analyses were used to determine the structure of the scales. Measures showed reasonable reliability and promising validity evidence, relating to each other and to academic achievement, while remaining relatively independent from personality. The advantages and disadvantages of each methodology and the potential applications for identification and intervention, selection, and evaluation of training programs are discussed. This report also serves as an archival document for the teamwork and collaboration assessments that have been developed at ETS for high school students.
THE REDESIGNED TOEIC® (LISTENING AND READING) TEST: RELATIONS TO TEST‐TAKER PERCEPTIONS OF PROFICIENCY IN ENGLISH
Authors: Powers Donald E. , Kim Hae‐Jin , Weng Vincent Z.
Published: August 2014

To facilitate the interpretation of test scores from the redesigned TOEIC® (listening and reading) test as a measure of English language proficiency, we administered a self‐assessment inventory to TOEIC examinees in Japan and Korea that gathered perceptions of their ability to perform a variety of everyday English language tasks. TOEIC scores related relatively strongly to test‐taker self‐reports for both reading and listening tasks. The results were, with few exceptions, extraordinarily consistent, with examinees at each higher TOEIC score level being more likely to report that they could successfully accomplish each of the everyday language tasks in English. The pattern of correlations also showed modest discriminant validity of the listening and reading components of the redesigned TOEIC, suggesting that both sections contribute to the measurement of English language skills.
NOTES ON A GENERAL FRAMEWORK FOR OBSERVED SCORE EQUATING
Authors: Moses Tim , Holland Paul
Published: August 2014

The purpose of this paper is to extend von Davier, Holland, and Thayer's (2004b) framework of kernel equating so that it can incorporate raw data and traditional equipercentile equating methods. One result of this more general framework is that previous equating methodology research can be viewed more comprehensively. Another result is that the standard error of equated score difference (SEED) has a wider application than originally proposed. The methods described in this paper are empirically evaluated in an accompanying simulation study (Moses & Holland, 2007).
FACTOR STRUCTURE OF THE TOEFL INTERNET‐BASED TEST ACROSS SUBGROUPS
Authors: Stricker Lawrence J. , Rock Donald A.
Published: August 2014

This study assessed the invariance in the factor structure of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT) across subgroups of test takers who differed in native language and exposure to the English language. The subgroups were defined by (a) Indo‐European and Non‐Indo‐European language family, (b) Kachru's classification of outer and expanding circles of countries (based on prevalence of English use in educational and business contexts), and (c) years of classroom instruction in the English language. The same factor structure (four first‐order factors corresponding to the test sections and a single higher‐order factor encompassing these factors) was identified in each subgroup. The results support the present scoring scheme for the TOEFL iBT assessment and suggest that the test functions the same way for diverse subgroups of test takers.
CONSISTENCY OF SAT® I: REASONING TEST SCORE CONVERSIONS
Authors: Haberman Shelby J. , Guo Hongwen , Liu Jinghua , Dorans Neil J.
Published: August 2014

This study uses historical data to explore the consistency of SAT® I: Reasoning Test score conversions and to examine trends in scaled score means. During the period from April 1995 to December 2003, both Verbal (V) and Math (M) means display substantial seasonality, and a slight increasing trend for both is observed. SAT Math means increase more than SAT Verbal means. Several statistical indices indicate that, during the period under study, raw‐to‐scale conversions are very stable, although conversions for extreme raw score points are less stable than are other conversions.
EDUCATIONAL ASSESSMENT USING INTELLIGENT SYSTEMS
Authors: Shute Valerie J. , Zapata‐Rivera Diego
Published: August 2014

Recent advances in educational assessment, cognitive science, and artificial intelligence have made it possible to integrate valid assessment and instruction in the form of modern computer‐based intelligent systems. These intelligent systems leverage assessment information that is gathered from various sources (e.g., summative and formative). This paper analyzes the role of educational assessment in intelligent systems, summarizes the characteristics of successfully deployed intelligent systems, and describes an evidence‐based approach to incorporating valid and reliable assessments into enhanced intelligent systems.
THE FUSION MODEL FOR SKILLS DIAGNOSIS: BLENDING THEORY WITH PRACTICALITY
Authors: Hartz Sarah , Roussos Louis
Published: August 2014

This paper presents the development of the fusion model skills diagnosis system (fusion model system), which can help integrate standardized testing into the learning process with both skills‐level examinee parameters for modeling examinee skill mastery and skills‐level item parameters, giving information about the diagnostic power of the test. The development of the fusion model system involves advancements in modeling, parameter estimation, model‐fitting methods, and model‐fit evaluation procedures, which are described in detail in the paper. To document the accuracy of the estimation procedure and the effectiveness of the model‐fitting and model‐fit evaluation procedures, this paper also presents a series of simulation studies. Special attention is given to evaluating the robustness of the fusion model system to violations of various modeling assumptions. The results demonstrate that the fusion model system is a promising tool for skills diagnosis that merits further research and development.
EVALUATING THE CONSTRUCT‐COVERAGE OF THE E‐RATER® SCORING ENGINE
Authors: Quinlan Thomas , Higgins Derrick , Wolff Susanne
Published: August 2014

This report evaluates the construct coverage of the e‐rater® scoring engine. The matter of construct coverage depends on whether one defines writing skill, in terms of process or product. Originally, the e‐rater engine consisted of a large set of components with a proven ability to predict human holistic scores. By organizing these capabilities into features, e‐rater researchers organized the e‐rater engine along the lines of trait scoring, which recognizes that essay quality has several dimensions. Some traits of essay quality cut across different methods for scoring essay quality, such as the rubrics employed by the GRE® and TOEFL® assessments, as well as the 6‐trait scoring model. Factor analyses conducted by Attali and Powers (2008) suggest that e‐rater features capture low‐level aspects of essay quality, such as sentence complexity, vocabulary, and conventions. Future e‐rater development should focus on (a) deepening and expanding coverage of the construct, such as by developing measures of essay content and organization, as well as on (b) addressing accuracy issues in existing features.
EFFICIENT FULL INFORMATION MAXIMUM LIKELIHOOD ESTIMATION FOR MULTIDIMENSIONAL IRT MODELS
Authors: Rijmen Frank
Published: August 2014

Maximum marginal likelihood estimation of multidimensional item response theory (IRT) models has been hampered by the calculation of the multidimensional integral over the ability distribution. However, the researcher often has a specific hypothesis about the conditional (in)dependence relations among the latent variables. Exploiting these relations may result in more efficient estimation algorithms. A well‐known example is the bi‐factor model, in which each item measures a general dimension and one ofKother dimensions, for which Gibbons and Hedeker (1992) showed that full information maximum likelihood estimation only requires the integration over two‐dimensional integrals. In this paper, it is shown how the approach of Gibbons and Hedeker (1992) can be placed into a graphical model framework. The advantage of the graphical model framework is that efficient estimation schemes can be derived in a fully automatic way by applying algorithms to the graphical representation of a statistical model. This renders the approach fairly generally applicable, and tedious derivations by hand are no longer involved. The generality of the approach is demonstrated by applying it to a multidimensional IRT model with a second order dimension. It turns out that full information maximum likelihood estimation for such a model also requires the evaluation of two‐dimensional integrals only.
SELECTION STRATEGIES FOR BIVARIATE LOGLINEAR SMOOTHING MODELS AND THEIR EFFECTS ON NEAT EQUATING FUNCTIONS
Authors: Moses Tim , Holland Paul
Published: August 2014

This study evaluated 8 statistical strategies for selecting the parameterizations of loglinear models for bivariate test score distributions and for using these strategies in equating. Four of the strategies were based on significance tests of chi‐square statistics (likelihood ratio, Pearson, Freeman‐Tukey, and Cressie‐Read chi‐squares) and 4 additional strategies were based on different evaluations of the likelihood ratio chi‐square statistic (Akaike information criterion [AIC], Bayesian information criterion [BIC], consistent Akaike information criterion [CAIC], and Goodman). The focus was on the selection strategies' accuracies in selecting bivariate parameterizations and the implications of using each selection strategy for chained and poststratification equating functions. The results showed that selection strategies differ in terms of their tendencies to select complex versus simple bivariate parameterizations and that the strategies that tended to select more complex parameterizations (i.e., Pearson and Cressie‐Read chi‐squares) usually produced the least biased and most variable equating functions.
STOCHASTIC APPROXIMATION METHODS FOR LATENT REGRESSION ITEM RESPONSE MODELS
Authors: von Davier Matthias , Sinharay Sandip
Published: August 2014

This paper presents an application of a stochastic approximation EM‐algorithm using a Metropolis‐Hastings sampler to estimate the parameters of an item response latent regression model. Latent regression models are extensions of item response theory (IRT) to a 2‐level latent variable model in which covariates serve as predictors of the conditional distribution of ability. Applications for estimating latent regression models for data from the 2000 National Assessment of Educational Progress (NAEP) grade 4 math assessment and the 2002 grade 8 reading assessment are presented and results of the proposed method are compared to results obtained using current operational procedures.
AN EXPLORATORY STUDY INTO INTERLANGUAGE PRAGMATICS OF REQUESTS: A GAME OF PERSUASION
Authors: Yang Hui‐Chun , Zapata‐Rivera Diego
Published: August 2014

Teachability and assessment of interlanguage pragmatics are two crucial but still relatively unexplored areas in second language research. Several instruments, such as multiple‐choice discourse completion tasks, written discourse completion tasks, role‐play tasks, and discourse self‐assessment tasks (J. D. Brown, 2001), have been used to assess students' interlanguage pragmatic skills. This paper describes a new assessment‐based gaming environment of English as a second language (ESL) pragmalinguistics called A Game of Persuasion that focuses on the speech act of request. This game allows users to engage in interactive written dialogue with an artificial professor (or pedagogical agent) in multiple academic contexts. Students explore contextually and socially appropriate request strategies while the system scores each attempt, assigns points, and provides formative and summative feedback. This paper presents (a) related research on instruments that have been used in the area of interlanguage pragmatics (i.e., request), (b) our assessment‐based gaming environment, and (c) reports on results from a study aimed at exploring usability aspects of the game and usage of request strategies employed by 30 English language learners (ELLs) who completed 10 written interactive request tasks with and without the presence of a pedagogical agent.
WHAT EFFECT DOES THE INCLUSION OR EXCLUSION OF REPEATERS HAVE ON TEST EQUATING?
Authors: Puhan Gautam
Published: August 2014

This study investigated the effect of inclusion or exclusion of test repeaters on equating results. New forms of two tests were equated to their respective old forms using either all examinees or only the first‐time examinees in the new form sample. Since the old forms were the first forms administered for these tests, there were no repeaters and therefore the old form samples were unaltered. Results showed that for both tests, inclusion or exclusion of repeaters in the new form sample had very little impact on the final equating results. These findings do not provide enough evidence to support routine exclusion of repeaters from the equating sample before conducting the equating. Finally, some examples of simple analyses to assist the testing practitioner when making a judgment about whether including or excluding repeaters in the equating sample is reasonable are provided.
ENGLISH LANGUAGE LEARNERS WITH READING DISABILITIES: A REVIEW OF THE LITERATURE AND THE FOUNDATION FOR A RESEARCH AGENDA
Authors: Shore Jane R. , Sabatini John
Published: August 2014

The issue of identifying reading difficulties and disabilities in English language learners (ELLs) is a complex one. It is an area that draws on the diverse disciplines of first‐ and second‐language acquisition, literacy, English language learning, and reading, including differences and disabilities research. This literature review aims to synthesize the research that aims to address the topic, focusing on the following three questions:

How does one identify reading difficulties/disabilities (RD) in individuals who are learning the English language?

What needs to be done to aid in identifying RD in ELLs?

What does one do with this information to support the English reading instruction of ELLs with RD?

The first section of the report covers basic terminology and gives an overview of the problem. The second section presents some of the challenges in distinguishing between what might be normal ELL language development from what might be identified as a disability or difference. The third section reviews reading difficulties in various languages and then focuses on reading difficulties when learning a second language. The fourth section provides an overview of the issues surrounding the process of identifying reading difficulties or differences in ELLs. The challenges begin at referral to services and span to measurement of disabilities.1 The fifth section provides information on interventions and the application of information gained through assessment. The sixth section, the conclusion, points to key considerations related to the identification of RD in ELLs on the K‐12 level, especially in the area of the definition, measurement, and instruction of ELLs identified with RD. It also examines the potential for further development and research of assessment and effective instructional programs. Key findings and implications are summarized in the areas of RDs, referral, current assessment, assessments in development, instructional practices, and teacher preparation.
THE INFLUENCE OF NONCOGNITIVE DOMAINS ON ACADEMIC ACHIEVEMENT IN K‐12
Authors: Lee Jihyun , Shute Valerie J.
Published: August 2014

This paper identifies three noncognitive domains relevant for academic achievement in K‐12—student engagement, behavioral learning strategies, and school climate. The paper also documents empirical findings that show relationships between these three noncognitive domains and academic achievement, especially in the areas of reading and mathematics.
COMPARISON OF THE EFFECTS OF DISCRETE ANCHOR ITEMS AND PASSAGE‐BASED ANCHOR ITEMS ON OBSERVED‐SCORE EQUATING RESULTS
Authors: Zu Jiyun , Liu Jinghua
Published: August 2014

Equating of tests composed of both discrete and passage‐based items using the nonequivalent groups with anchor test (NEAT) design is popular in practice. This study investigated the impact of discrete anchor items and passage‐based anchor items on observed score equating via simulation. Results suggested that an anchor with a larger proportion of passage‐based items and/or a larger degree of local dependence among passage‐based items produces larger equating errors, especially when group ability differences are not minimal. Our findings challenge the common belief that an anchor should be a miniature version of the tests to be equated.
AN EMPIRICAL COMPARISON OF METHODS FOR EQUATING WITH RANDOMLY EQUIVALENT GROUPS OF 50 TO 400 TEST TAKERS
Authors: Livingston Samuel A. , Kim Sooyeon
Published: August 2014

A series of resampling studies investigated the accuracy of equating by four different methods in a random groups equating design with samples of 400, 200, 100, and 50 test takers taking each form. Six pairs of forms were constructed. Each pair was constructed by assigning items from an existing test taken by 9,000 or more test takers. The criterion equating was the direct equipercentile equating in the full group. Accuracy was described in terms of the root‐mean‐squared deviation (over 1,000 replications) of the sample equatings from the criterion equating. The equating methods investigated were equipercentile equating of smoothed distributions, linear equating, mean equating, and circle‐arc equating; they were compared with each other and with the identity. Circle‐arc equating produced the most accurate results for all sample sizes investigated, particularly in the upper half of the score distribution.
HOW CAN MULTIDIMENSIONAL ITEM RESPONSE THEORY BE USED IN REPORTING OF SUBSCORES?
Authors: Haberman Shelby J. , Sinharay Sandip
Published: August 2014

Recently, there has been increasing interest in reporting diagnostic scores. This paper examines reporting of subscores using multidimensional item response theory (MIRT) models. An MIRT model is fitted using a stabilized Newton‐Raphson algorithm (Haberman, 1974, 1988) with adaptive Gauss‐Hermite quadrature (Haberman, von Davier, & Lee, 2008). A new statistical approach is proposed to assess when subscores using the MIRT model have any added value over (a) the total score or (b) subscores based on classical test theory (Haberman, 2008; Haberman, Sinharay, & Puhan, 2006). The MIRT‐based methods are applied to several operational data sets. The results show that the subscores based on MIRT are slightly more accurate than subscore estimates derived by classical test theory.
INFORMATION AND COMMUNICATION TECHNOLOGY LITERACY: WHAT DO BUSINESSES EXPECT AND WHAT DO BUSINESS SCHOOLS TEACH?
Authors: Ali Radwan , Katz Irvin R.
Published: August 2014

Today's information and communications technology (ICT) provides unprecedented amounts of information to organizations and their employees. This overabundance challenges workers, placing an increasing premium on skills of sifting through information of sometimes dubious quality, integrating information critically, and producing well‐reasoned conclusions. This paper explores the hypothesis that employers want new hires skilled in ICT literacy, which includes navigating information effectively and using technology efficiently. Based on an ICT literacy framework for higher education, two surveys were created to assess the opinions of human resources (HR) consultants and business school faculty. HR consultants endorsed practically all elements of the framework as being important or essential for new hires. Especially valued were skills associated with information security, confidentiality, and ethical behavior. However, business school faculty did not report a corresponding focus on ICT literacy generally, or ethical/legal issues in particular, in their instruction. The results suggest that while ICT literacy does indeed appear to be among the key skills for today's workforce, much work must still be done to integrate these skills throughout the business school undergraduate curriculum.
MEASUREMENT ISSUES IN STATE ACHIEVEMENT COMPARISONS
Authors: Zwick Rebecca
Published: August 2014

Political perspectives on the advisability of state achievement comparisons have changed substantially over the last half‐century. As Selden (2004) noted regarding the National Assessment of Educational Progress (NAEP), “[w]hen it began in the 1960s, NAEP specifically was designed not to provide comparisons among states. This was done for political reasons, so that the assessment program would be palatable to the educational community and to the states” (p. 195). During the No Child Left Behind Era, state comparisons were complicated by the use of tests and proficiency standards that differed across states. Because of today's Common Core Standards Initiative, state comparisons are once again at the forefront of educational policy discussions. The initiative brings with it both significant opportunities and substantial psychometric challenges.
THE USE OF TWO ANCHORS IN NONEQUIVALENT GROUPS WITH ANCHOR TEST (NEAT) EQUATING
Authors: Moses Tim , Deng Weiling , Zhang Yu‐Li
Published: August 2014

In the equating literature, a recurring concern is that equating functions that utilize a single anchor to account for examinee groups' nonequivalence are biased when the groups are extremely different and/or when the anchor only weakly measures what the tests measure. Several proposals have been made to address this equating bias by incorporating more than one anchor into nonequivalent groups with anchor test (NEAT) equating functions. These proposals have not been extensively considered or comparatively evaluated. This study evaluates three methods for incorporating more than one anchor into NEAT equating functions, including poststratification, imputation, and propensity score matching. The three methods are studied and compared in two examples. The implications for using the three equating approaches in practice and for developing alternative strategies to incorporate two anchors are discussed.
RESEARCH ON STANDARD ERRORS OF EQUATING DIFFERENCES
Authors: Moses Tim , Zhang Wenmin
Published: August 2014

In this paper, thestandard error of equating difference(SEED) is described in terms of originally proposed kernel equating functions (von Davier, Holland & Thayer, 2004) and extended to incorporate traditional linear and equipercentile functions. These derivations expand on prior developments of SEEDs and standard errors of equating and provide additional insight about the relationships of kernel and traditional equating functions. Simulations are used to evaluate the SEEDs' accuracies.
AN EXAMINATION OF THE LINK BETWEEN RATER CALIBRATION PERFORMANCE AND SUBSEQUENT SCORING ACCURACY IN GRADUATE RECORD EXAMINATIONS® (GRE®) WRITING
Authors: Ricker‐Pedley Kathryn L.
Published: August 2014

A pseudo‐experimental study was conducted to examine the link between rater accuracy calibration performances and subsequent accuracy during operational scoring. The study asked 45 raters to score a 75‐response calibration set and then a 100‐response (operational) set of responses from a retired Graduate Record Examinations® (GRE®) writing prompt that had been vetted to determine correct scores prior to the study. The study found a positive relationship between calibration accuracy and scoring accuracy. Results suggest that a longer calibration test is more reliable and has a stronger correlation with operational scoring exact agreement; however, even tests as short as 10 responses had a reasonable correlation with operational accuracy, and the shorter test classified raters accurately 87% of the time when compared to the 75‐response criterion. The additional requirement of no discrepant scores during calibration had better correlation with exact agreement, but much poorer classification accuracy in the shorter tests. This research suggests that calibration is useful for screening raters in its current form, but that further investigation of passing standards and of less controlled operational scoring conditions is required.
WRITING ASSESSMENT AND COGNITION
Authors: Deane Paul
Published: August 2014

This paper presents a socio‐cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writing assessment, and activity and genre theories; and presents a specific proposal about how writing assessment can be organized to promote best practices in writing instruction.
SMOOTHING AND EQUATING METHODS APPLIED TO DIFFERENT TYPES OF TEST SCORE DISTRIBUTIONS AND EVALUATED WITH RESPECT TO MULTIPLE EQUATING CRITERIA
Authors: Moses Tim , Liu Jinghua
Published: August 2014

In equating research and practice, equating functions that are smooth are typically assumed to be more accurate than equating functions with irregularities. This assumption presumes that population test score distributions are relatively smooth. In this study, two examples were used to reconsider common beliefs about smoothing and equating. The first example involves a relatively smooth population test score distribution and the second example involves a population test score distribution with systematic irregularities. Various smoothing and equating methods (presmoothing, equipercentile, kernel, and postsmoothing) were compared across the two examples with respect to how well the test score distributions were reflected in the equating functions, the smoothness of the equating functions, and the standard errors of equating. The smoothing and equating methods performed more similarly in the first example than in the second example. The results of the second example illustrate that when dealing with systematically irregular test score distributions, smoothing and equating methods can be used in different ways to satisfy different equating criteria.
VALIDATION OF AUTOMATED SCORES OF TOEFL IBT®TASKS AGAINST NONTEST INDICATORS OF WRITING ABILITY
Authors: Weigle Sara Cushing
Published: August 2014

Automated scoring has the potential to dramatically reduce the time and costs associated with the assessment of complex skills such as writing, but its use must be validated against a variety of criteria for it to be accepted by test users and stakeholders. This study addresses two validity‐related issues regarding the use of e‐rater®with the independent writing task on the TOEFL iBT®(Internet‐based test). First, relationships between automated scores of iBT tasks and nontest indicators of writing ability were examined. This was followed by exploration of prompt‐related differences in automated scores of essays written by the same examinees. Correlations between both human and e‐rater scores and nontest indicators were moderate but consistent, with few differences between e‐rater and human rater scores. E‐rater was more consistent across prompts than individual human raters, although there were differences in scores across prompts for the individual features used to generate total e‐rater scores.
SCORE COMPARABILITY FOR LANGUAGE MINORITY STUDENTS ON THE CONTENT ASSESSMENTS USED BY TWO STATES
Authors: Young John W. , Holtzman Steven , Steinberg Jonathan
Published: August 2014

In this research investigation of score comparability for language minority students (English language learners [ELLs] and former English language learners), we examined 3 indicators of score comparability (reliability, internal test structure, and differential item functioning) for 4th and 8th grade students who took the NCLB‐mandated content assessments in English‐language arts and mathematics in 2 different U. S. states. Overall, for the 8 assessments we examined, a high degree of score comparability was found for ELLs and former ELLs, when compared with native English speakers. The results from this study showed that although the assessments from the 2 states differed somewhat with respect to the 3 indicators, a high degree of score comparability was found for both states' content assessments.
FIT OF ITEM RESPONSE THEORY MODELS: A SURVEY OF DATA FROM SEVERAL OPERATIONAL TESTS
Authors: Sinharay Sandip , Haberman Shelby J. , Jia Helena
Published: August 2014

Standard 3.9 of theStandards for Educational and Psychological Testing(American Educational Research Association, American Psychological Association, & National Council for Measurement in Education, 1999) demands evidence of model fit when an item response theory (IRT) model is used to make inferences from a data set. We applied two recently suggested methods for assessing goodness of fit of IRT models—generalized residual analysis (Haberman, 2009) and residual analysis for assessing item fit (Bock & Haberman, 2009)—to several operational data sets. We assessed the practical significance of misfit whenever possible. This report summarizes our findings. Though evidence of misfit of the IRT model was found for all the data sets, the misfit was not always practically significant.
THE RELATIONSHIP BETWEEN RATERS' PRIOR LANGUAGE STUDY AND THE EVALUATION OF FOREIGN LANGUAGE SPEECH SAMPLES
Authors: Winke Paula , Gass Susan , Myford Carol
Published: August 2014

This study investigated whether raters' second language (L2) background and the first language (L1) of test takers taking the TOEFL iBT® Speaking test were related through scoring. After an initial 4‐hour training period, a group of 107 raters (mostly of learners of Chinese, Korean, and Spanish), listened to a selection of 432 speech samples that 72 test takers (native speakers of Chinese, Korean, and Spanish) produced. We analyzed the rating data using a multifaceted Rasch measurement approach to uncover potential biases in the rating process. In addition, 26 of the raters participated in stimulated recall sessions, during which they watched videos of themselves rating. Using the video as a prompt, we asked them to discuss and explain their rating processes at the time of rating. The results from our bias interaction analyses revealed that matches between the raters' L2 and the test takers' L1 resulted in some of the raters assigning ratings that were significantly higher than expected. As a whole, raters with Spanish as an L2 were significantly more lenient toward test takers who had Spanish as an L1, and raters with Chinese as an L2 were significantly more lenient toward test takers who had Chinese as an L1. Analyses of the qualitative data, assisted by the program QSR NVivo 8, revealed information concerning the raters' awareness of their biases.
THE SINGLE GROUP WITH NEARLY EQUIVALENT TESTS (SIGNET) DESIGN FOR EQUATING VERY SMALL VOLUME MULTIPLE‐CHOICE TESTS
Authors: Grant Mary C.
Published: August 2014

The single group with nearly equivalent tests (SiGNET) design proposed here was developed to address the problem of equating scores on multiple‐choice test forms with very small single‐administration samples. In this design, the majority of items in each new test form consist of items from the previous form, and the new items that were administered as unscored items in the previous form. Each form is equated using data from examinees who took the previous form. As the equating is a single‐group design, with the 2 forms having a large number of overlapping items, the size of the equating sample can be much smaller than in other designs.
CONSTRUCTED‐RESPONSE MATHEMATICS TASKS STUDY
Authors: Fife James H. , Graf Edith Aurora , Ohls Sarah
Published: August 2014

Six tasks, selected from assessments administered in 2007 as part of the Cognitively Based Assessment of, for, and as Learning (CBAL) project, were revised in an effort to remove difficulties with the tasks that were unrelated to the construct being assessed. Because the revised tasks were piloted on a different population from the original tasks, it was not possible to make direct comparisons between the performance of the revised tasks and that of the original tasks, other than to make a qualitative assessment of whether or not the nonconstruct difficulties had, in fact, been removed. But we were able to pilot between 2 and 4 versions of each revised task, and we could compare the performance of our pilot sample on the various versions of each task. For Mix It Up, we prepared 2 nonparallel versions—the first attempted to preserve the construct‐related difficulty of the original while removing the nonconstruct‐related ambiguities, and the second was intended to be an easier task that measured the same skills and abilities. For Fruit Drink and Paste we created 4 versions, carefully varying different aspects of the language while keeping other aspects constant. For the 2 tasks from Bigfoot, we varied 2 features independently, creating 2 versions of each feature and therefore 4 versions of each task. Finally, for Forest Carbon, we created 4 versions, varying from unscaffolded to carefully scaffolded. Because the revision of each task was its own experiment, the analysis of each task, and our conclusions from that analysis, are described separately.
EQUIVALENCY OF THE DINA MODEL AND A CONSTRAINED GENERAL DIAGNOSTIC MODEL
Authors: von Davier Matthias
Published: August 2014

This report shows that the deterministic‐input noisy‐AND (DINA) model is a special case of more general compensatory diagnostic models by means of a reparameterization of the skill space and the design (Q‐) matrix of item by skills associations. This reparameterization produces a compensatory model that is equivalent to the (conjunctive) DINA model, and is valid for all types of complex structure Q‐matrices, not only for trivial cases. This equivalency uses the GDM as the basis, is not based on recent developments of diagnosis models such as G‐DINA or LCDM. Model equivalency is a topic of some relevance as soon as researchers want to draw conclusions derived from any particular model‐based estimates. It can be shown that for multidimensional models, there are often multiple ways to specify different sets of latent variables and their relationships to observed variables. This report goes beyond showing that multiple versions of a design matrix lead to the same model‐based conditional probability space; it shows that a conjunctive diagnostic classification model can be expressed as a constrained special case of a compensatory diagnostic modeling framework.