bluuebunny/crossref_metadata_2025_split huggingface dataset:
EXAMINING TEST ITEMS FOR DIFFERENTIAL DISTRACTOR FUNCTIONING AMONG STUDENTS WITH LEARNING DISABILITIES
Authors: Middleton Kyndra , Laitusis Cara Cahalan 
Published: August 2014

ABSTRACTThis study examined whether distractor choices functioned differently for students without learning disabilities than they functioned for students with learning disabilities who received no accommodation, students with learning disabilities who received a read‐aloud accommodation, and students with learning disabilities who received some form of accommodation other than read‐aloud. The study's purpose was twofold: (a) to examine the results of the DDF analysis to determine whether the distractors functioned differently for the various groups of students and (b) to aid in determining whether the test may be modified for those students with learning disabilities by removing a distractor while maintaining adequate test validity and information.
EVALUATING THE EFFECTIVENESS OF A FULL‐POPULATION ESTIMATION METHOD
Authors: Braun Henry , Zhang Jinming , Vezzu Sailesh 
Published: August 2014

ABSTRACTAt present, although the percentages of students with disabilities (SDs) and/or students who are English language learners (ELL) excluded from a NAEP administration are reported, no statistical adjustment is made for these excluded students in the calculation of NAEP results. However, the exclusion rates for both SD and ELL students vary substantially across jurisdictions at a given administration, and, in some cases, have changed substantially over time within a jurisdiction. Consequently, comparisons of performance based on reported NAEP scores may indeed be biased by differential exclusion and identification practices.Using only NAEP data, this report investigates plausible explanations for the observed heterogeneity among jurisdictions in exclusion rates. It also examines the operating characteristics of a particular class of methods that carry out statistical adjustments to NAEP's reported scores to address the possible bias due to differential exclusion rates. The final results of such adjustments are termed full‐population estimates (FPEs). The conclusions are that there is both a strong likelihood of bias and that neither the current NAEP procedure nor the FPE methodologies constitutes an ideal solution. The former because it assumes that all excluded students could not meaningfully participate in NAEP, and the latter because they implicitly assume that all students could obtain a proper NAEP score.
COMPARISON OF MULTIDIMENSIONAL ITEM RESPONSE MODELS: MULTIVARIATE NORMAL ABILITY DISTRIBUTIONS VERSUS MULTIVARIATE POLYTOMOUS ABILITY DISTRIBUTIONS
Authors: Haberman Shelby J. , von Davier Matthias , Lee Yi‐Hsuan 
Published: August 2014

ABSTRACTMultidimensional item response models can be based on multivariate normal ability distributions or on multivariate polytomous ability distributions. For the case of simple structure in which each item corresponds to a unique dimension of the ability vector, some applications of the two‐parameter logistic model to empirical data are employed to illustrate how, at least for the example under study, comparable results can be achieved with either approach. Comparability involves quality of model fit as well as similarity in terms of parameter estimates and computational time required. In both cases, numerical work can be performed quite efficiently. In the case of the multivariate normal ability distribution, multivariate adaptive Gauss‐Hermite quadrature can be employed to greatly reduce computational labor. In the case of a polytomous ability distribution, use of log‐linear models permits efficient computations.
AN EVALUATION OF STATISTICAL STRATEGIES FOR MAKING EQUATING FUNCTION SELECTIONS
Authors: Moses Tim 
Published: August 2014

ABSTRACTNine statistical strategies for selecting equating functions in an equivalent groups design were evaluated. The strategies of interest were likelihood ratio chi‐square tests, regression tests, Kolmogorov‐Smirnov tests, and significance tests for equated score differences. The most accurate strategies in the study were the likelihood ratio tests and the significance tests for equated score differences.
COGNITIVELY BASED ASSESSMENT OF, FOR, AND AS LEARNING: A FRAMEWORK FOR ASSESSING READING COMPETENCY
Authors: O'Reilly Tenaha , Sheehan Kathleen M. 
Published: August 2014

ABSTRACTThis paper presents the rationale and research base for a reading competency model designed to guide the development of cognitively based assessment of reading comprehension. The model was developed from a detailed review of the cognitive research on reading and learning and a review of state standards for language arts. A survey of the literature revealed three key areas of reading competency: prerequisite reading skill, model building skill, and applied comprehension skill. Prerequisite reading skill is the ability to read text accurately and fluently. Model building skill is the ability to construct meaning from either decoded text or spoken language. Applied comprehension skill is the ability to use and apply the information contained in text for some particular purpose. The framework is discussed in terms of 7 key principles that have implications for the design of a modern assessment of reading.
HOW DO RATERS FROM INDIA PERFORM IN SCORING THE TOEFL IBT™ SPEAKING SECTION AND WHAT KIND OF TRAINING HELPS?
Authors: Xi Xiaoming , Mollaun Pam 
Published: August 2014

ABSTRACTThis study investigated the scoring of the Test of English as a Foreign Language™ Internet‐based Test (TOEFL iBT™) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on‐site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.‐based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non‐Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees.
TOWARD A DEFINITION OF VERBAL REASONING IN HIGHER EDUCATION
Authors: Burton Nancy W. , Welsh Cynthia , Kostin Irene , van Essen Thomas 
Published: August 2014

ABSTRACTThis paper briefly summarizes the literatures of reading and reasoning in the last quarter century, focusing mainly on the disciplines of cognitive science, cognitive developmental psychology, linguistics, and educational psychology. These literatures were synthesized to create a framework for defining verbal reasoning in higher education. Eight general cognitive and metacognitive operations were identified (including, for example, evaluating discourse, seeking and solving problems, and monitoring one's comprehension). Several dimensions underlying these operations on which individual skills may vary were identified (such as breadth of understanding, precision of understanding, or familiarity and facility). Finally, these ideal descriptions of verbal reasoning are applied to the assessment of verbal reasoning for selection in higher education. Problems in measurement and unanticipated consequences of measurement are discussed.
JACKKNIFING TECHNIQUES FOR EVALUATION OF EQUATING ACCURACY
Authors: Haberman Shelby J. , Lee Yi‐Hsuan , Qian Jiahe 
Published: August 2014

ABSTRACTGrouped jackknifing may be used to evaluate the stability of equating procedures with respect to sampling error and with respect to changes in anchor selection. Properties of grouped jackknifing are reviewed for simple‐random and stratified sampling, and its use is described for comparisons of anchor sets. Application is made to examples of item response theory (IRT) true‐score equating in which two‐parameter logistic and general partial credit models are employed.
LINKING PARAMETER ESTIMATES DERIVED FROM AN ITEM RESPONSE MODEL THROUGH SEPARATE CALIBRATIONS
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTA regression procedure is developed to link simultaneously a very large number of item response theory (IRT) parameter estimates obtained from a large number of test forms, where each form has been separately calibrated and where forms can be linked on a pairwise basis by means of common items. An application is made to forms in which a two‐parameter logistic model is applied to dichotomous items and a general partial credit model is applied to polytomous items.
DEFINING MATHEMATICS COMPETENCY IN THE SERVICE OF COGNITIVELY BASED ASSESSMENT FOR GRADES 6 THROUGH 8
Authors: Graf Edith Aurora 
Published: August 2014

ABSTRACTThis report makes recommendations for the development of middle‐school assessment in mathematics, based on a synthesis of scientific findings in cognitive psychology and mathematics education. The focus is on background research, rather than test specifications or example tasks. Readers interested in early development and pilot efforts associated with the Cognitively Based Assessment of, for, and as Learning (CBAL) project in mathematics (for which this review helped provide a theoretical foundation) should consult Graf, Harris, Marquez, Fife, and Redman (2009). The organization of the report is motivated by the evidence‐centered design (ECD) approach to assessment developed by Mislevy and colleagues (e.g., see Mislevy, Steinberg, & Almond, 2003). The first section consists of a broad literature review that characterizes mathematical competency with respect to both content and process. Subsequent sections discuss: how to model mathematical competency at the middle school level, the kinds of evidence that reflect the level of student competency and support future learning, and how to design tasks that elicit the target evidence.
LINKING ERRORS IN TREND ESTIMATION IN LARGE‐SCALE SURVEYS: A CASE STUDY
Authors: Xu Xueli , von Davier Matthias 
Published: August 2014

ABSTRACTOne of the major objectives of large‐scale educational surveys is reporting trends in academic achievement. For this purpose, a substantial number of items are carried from one assessment cycle to the next. The linking process that places academic abilities measured in different assessments on a common scale is usually based on a concurrent calibration of adjacent assessments using item response theory (IRT) models. It can be conjectured that the selection of common items has a direct effect on the estimation error of academic abilities due to item misfit, small changes in the common items, position effect, and other sources of constructirrelevant changes between measurement occasions. Hence, the error due to the common‐item sampling could be a major source of error for the ability estimates. In operational analyses, generally two sources of error are accounted for in variance estimation: student sampling error and measurement error. A double jackknifing procedure is proposed to include a third source of the estimation error, the error due to common‐item sampling. Three different versions of the double jackknifing were implemented and compared. The data used in this study were item responses from Grade 4 students who took the NAEP 2004 and 2008 math long‐term trend (LTT) assessments. These student samples used in this study are representative samples of Grade 4 student population in 2004 and in 2008 across the US. The results showed that these three double jackknifing approaches resulted in similar standard error estimates that were slightly higher than the estimates from the traditional approach, regardless of whether an item sampling scheme was used or items were dropped at random.
MODELING NONIGNORABLE MISSING DATA WITH ITEM RESPONSE THEORY (IRT)
Authors: Rose Norman , von Davier Matthias , Xu Xueli 
Published: August 2014

ABSTRACTLarge‐scale educational surveys are low‐stakes assessments of educational outcomes conducted using nationally representative samples. In these surveys, students do not receive individual scores, and the outcome of the assessment is inconsequential for respondents. The low‐stakes nature of these surveys, as well as variations in average performance across countries and other factors such as different testing traditions, are contributing factors to the amount of omitted responses in these assessments. While underlying reasons for omissions are not completely understood, common practice in international assessments is to employ a deterministic treatment of omissions, either as missing observations or as responses that are considered wrong. Both approaches appear problematic. In this project, we analyzed the effects of treating omitted responses either as missing or as wrong, as is done in the majority of international studies, and compared these data‐treatment solutions to model‐based approaches to treating omitted responses. The two types of model‐based approaches used in this study are: (a) extensions of multidimensional item response theory (IRT) with an additional dimension based on response indicator variables defined and calibrated together with the set of items containing the observed responses and (b) multidimensional, multiple‐group IRT models with a grouping variable representing the within‐country stratification of respondents by the amount of omitted responses. These two model‐based approaches were compared on the basis of simulated data and data from about 250,000 students from 30 Organisation for Economic Co‐operation and Development (OECD) Member countries participating in an international large‐scale assessment.
THE VALUE OF THE STUDIED ITEM IN THE MATCHING CRITERION IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSIS
Authors: Tan Xuan , Xiang Bihua , Dorans Neil J. , Qu Yanxuan 
Published: August 2014

ABSTRACTThe nature of the matching criterion (usually the total score) in the study of differential item functioning (DIF) has been shown to impact the accuracy of different DIF detection procedures. One of the topics related to the nature of the matching criterion is whether the studied item should be included. Although many studies exist that suggest the studied item should always be included in the criterion, the validity of this statement for models other than the Rasch model has not been studied. This study evaluates the effect of including/excluding the studied item in the matching criterion for situations that mimic real testing situations where the assumptions of the Rasch model are violated. A simulation study was conducted where the effect of including/excluding the studied item in the matching criterion was studied relative to different magnitudes of DIF and different group ability distributions, for data that follow the two‐parameter logistic (2PL) item response theory (IRT) and multidimensional item response theory (MIRT) models. Results from the study show that including the studied item leads to less biased DIF estimates and more appropriate Type I error rate, especially when the group ability distributions are different. Systematic bias positioning DIF estimation in favor of the high ability group was consistently found across all simulated conditions when the studied item was excluded from the matching criterion.
PRINCIPLES AND PRACTICES OF TEST SCORE EQUATING
Authors: Dorans Neil J. , Moses Tim P. , Eignor Daniel R. 
Published: August 2014

ABSTRACTScore equating is essential for any testing program that continually produces new editions of a test and for which the expectation is that scores from these editions have the same meaning over time. Particularly in testing programs that help make high‐stakes decisions, it is extremely important that test equating be done carefully and accurately. An error in the equating function or score conversion can affect the scores for all examinees, which is both a fairness and a validity concern. Because the reported score is so visible, the credibility of a testing organization hinges on activities associated with producing, equating, and reporting scores. This paper addresses the practical implications of score equating by describing aspects of equating and best practices associated with the equating process.
THE EFFECTIVENESS OF FEEDBACK FOR L1‐ENGLISH AND L2‐WRITING DEVELOPMENT: A META‐ANALYSIS
Authors: Biber Douglas , Nekrasova Tatiana , Horn Brad 
Published: August 2014

ABSTRACTThis research project undertook a review and synthesis of previous research on the effectiveness of feedback for individual writing development. The work plan was divided into two main phases. First, we surveyed all available studies that have investigated the effectiveness of writing feedback, including both quantitative and qualitative research, for students who have learned English as a first language (L1‐English), students who have learned English as a second language (L2‐English), and students who have learned second languages other than English. The results of this survey are described in a narrative overview of previous research pertaining to the role of feedback in the development of writing proficiency. The survey also identified the major theoretical constructs used in this research domain, providing the basis for subsequent statistical analysis.Second, we built on this survey to carry out a meta‐analysis of empirical studies in this research area. The goal of the meta‐analysis was to provide a quantitative investigation of the extent and ways in which feedback has been effective, summarizing the findings of previous quantitative studies that have employed suitable statistical measures. Several analytical steps were required for the meta‐analysis: developing a coding rubric; analyzing the research design and adequacy of reporting in studies to determine if they were suitable for inclusion; coding each study for all relevant research design factors; computing effect sizes for each study; and analyzing and interpreting the general patterns that hold across this set of studies.The meta‐analysis compared the gains in writing development with respect to several different kinds of feedback. Overall, feedback was found to result in gains in writing development. Beyond that, there were several predictable findings (e.g., that written feedback is more effective than oral feedback for writing development) and several other more noteworthy trends (e.g., that peer feedback is more effective than teacher feedback for L2‐English students; commenting is more effective than error location; and in general, focus on form and content seems to be more effective than an exclusive focus on form).
CAN SMOOTHING HELP WHEN EQUATING WITH UNREPRESENTATIVE SMALL SAMPLES?
Authors: Puhan Gautam 
Published: August 2014

ABSTRACTThe study evaluated the effectiveness of log‐linear presmoothing (Holland & Thayer, 1987) on the accuracy of small sample chained equipercentile equatings under two conditions (i.e., using small samples that differed randomly in ability from the target population versus using small samples that were distinctly different from the target population). Results showed that equating with small samples (e.g., N < 50) using either raw or smoothed score distributions can result in a substantial amount of random equating error (although smoothing reduced random equating error). Even with samples sizes of 100, the random equating error was quite large (greater than the difference that matters or DTM) for almost all score points. Moreover, when the small samples were unrepresentative of the target population, which is quite likely for small samples, the amount of equating bias (in addition to random equating error) was considerably large for both the raw and smoothed equatings. It was concluded that although presmoothing helped reduce random equating error, it is unlikely to reduce equating bias caused by using an unrepresentative sample. Other alternatives to the small sample equating problem that focus more on improving data collection than on improving existing equating methods are discussed.
AUTOMATED SUBSCORES FOR TOEFL IBT® INDEPENDENT ESSAYS
Authors: Attali Yigal 
Published: August 2014

ABSTRACTThe e‐rater® automated essay scoring system is used operationally in the scoring of TOEFL iBT® independent essays. Previous research has found support for a 3‐factor structure of the e‐rater features. This 3‐factor structure has an attractive hierarchical linguistic interpretation with a word choice factor, a grammatical convention within a sentence factor, and a fluency factor. The purpose of this study was to explore the feasibility of developing automated subscores for the TOEFL iBT independent task based on this 3‐factor structure. First, using a multiple‐group confirmatory factor analysis, the 3‐factor structure was found to be quite stable across major language groups. Next, the subscores based on these 3 factors were found to have added value in the context of repeater examinees by comparing the ability to predict a subscore on one test from the subscore on the other test or from the total e‐rater score on the other test. The results of this study could be used to develop and report subscores for the performance of examinees on TOEFL iBT independent prompts. Reporting subscores could have instructional and remedial value, both at the individual and institutional level.
DOES LINKING MIXED‐FORMAT TESTS USING A MULTIPLE‐CHOICE ANCHOR PRODUCE COMPARABLE RESULTS FOR MALE AND FEMALE SUBGROUPS?
Authors: Kim Sooyeon , Walker Michael E. 
Published: August 2014

ABSTRACTThis study examines the use of subpopulation invariance indices to evaluate the appropriateness of using a multiple‐choice (MC) item anchor in mixed‐format tests, which include both MC and constructed‐response (CR) items. Linking functions were derived in the nonequivalent groups with anchor test (NEAT) design using an MC‐only anchor set for 4 mixed‐format licensure tests. For each of those licensure tests, the linking functions were also derived separately for males and females, and those subpopulation functions were compared to the total group function. The mathematics, social studies, and science tests each produced acceptable differences between each of the subpopulation functions and the total group function within the cut‐score region, leading to consistent pass/fail designations for the examinees. The English test, which had a low correlation between MC and CR components (indicative of multidimensionality), produced the largest differences, casting doubt on the effectiveness of the MC‐only anchor.