bluuebunny/crossref_metadata_2025_split huggingface dataset:
LINKING ENGLISH‐LANGUAGE TEST SCORES ONTO THE COMMON EUROPEAN FRAMEWORK OF REFERENCE: AN APPLICATION OF STANDARD‐SETTING METHODOLOGY
Authors: Tannenbaum Richard J. , Wylie E. Caroline 
Published: August 2014

ABSTRACTThe Common European Framework of Reference (CEFR) describes language proficiency in reading, writing, speaking, and listening on a 6‐level scale. In this study, English‐language experts from across Europe linked CEFR levels to scores on three tests: the TOEFL® iBT test, the TOEIC® assessment, and the TOEIC Bridge™ test. Standard‐setting methodology (a modified Angoff approach and a modified examinee paper selection approach) was used to construct the linkages. Linkages were established for TOEFL iBT at levels B1, B2, and C1. Linkages were established for TOEIC at levels A1 through C1, with the exception of Reading at the C1 level. The TOEIC Bridge test was linked to its three targeted levels of the CEFR. The report details the methods, procedures, and results of the study.
COMPARISONS AMONG DESIGNS FOR EQUATING CONSTRUCTED‐RESPONSE TESTS
Authors: Kim Sooyeon , Walker Michael E. , McHale Frederick 
Published: August 2014

ABSTRACTThis study examined variations of a nonequivalent groups equating design used with constructed‐response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large‐scale exam, the study investigated the use of anchor CR item rescoring in the context of classical equating methods. Four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple‐choice (MC) anchor test, and (d) an equivalent groups design incorporating CR items rescored (no anchor test). The use of CR items without rescoring or the use of an external MC anchor resulted in much larger bias than the other two designs. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.
MAPPING STATE STANDARDS TO THE NAEP SCALE
Authors: Braun Henry , Qian Jiahe 
Published: August 2014

ABSTRACTThis report describes the derivation and evaluation of a method for comparing the performance standards for public school students set by different states. It is based on an approach proposed by McLaughlin and associates, which constituted an innovative attempt to resolve the confusion and concern that occurs when very different proportions of students in various states are declared to have met a standard with the same label. Our method, like McLaughlin's, employs equipercentile methods to map state standards on to a common scale, that associated with the National Assessment of Educational Progress (NAEP). We have also derived error estimates that take into account both NAEP's complex sampling design and measurement errors. The method was applied to two data sets, and the results were qualitatively similar to those obtained by McLaughlin's method. The paper notes the superior statistical properties of the proposed method and presents evidence that supports the viability and general utility of this approach.
ESTABLISHING THE VALIDITY OF TOEIC BRIDGE™ TEST SCORES FOR STUDENTS IN COLOMBIA, CHILE, AND ECUADOR
Authors: Sinharay Sandip , Feng Ying , Saldivia Luis , Powers Donald E. , Ginuta Anthony , Simpson Annabelle , Weng Vincent 
Published: August 2014

ABSTRACTThe validity of TOEIC Bridge™ scores as a measure of English language skill was examined from the standpoint of a unified concept of test validity. In this study, more than 6,000 test takers in 3 Latin American countries (Chile, Colombia, and Ecuador) took 1 form of the TOEIC Bridge test, and their scores were compared to additional information about the students (teacher judgments, self‐assessments, and performance on other academic achievement measures). The evidence collected was generally quite consistent with the interpretation of TOEIC Bridge scores as indicators of the English language competencies for the students examined.
AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATERSM V1.0
Authors: Xi Xiaoming , Higgins Derrick , Zechner Klaus , Williamson David M. 
Published: August 2014

ABSTRACTThis report presents the results of a research and development effort for SpeechRaterSM Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL®) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short‐ and long‐term goals for iterative improvements to SpeechRater scoring.
CRITERION‐RELATED VALIDITY OF THE TOEFL IBT LISTENING SECTION
Authors: Sawaki Yasuyo , Nissan Susan 
Published: August 2014

ABSTRACTThe study investigated the criterion‐related validity of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT) Listening section by examining its relationship to a criterion measure designed to reflect language‐use tasks that university students encounter in everyday academic life: listening to academic lectures. The design of the criterion measure was informed by students' responses to a survey on the frequency and importance of various classroom tasks that require academic listening, and the relationship of these tasks to successful course completion. The criterion measure consisted of three videotaped lectures (in physics, history, and psychology) and included tasks created by content experts who are former university professors of the relevant content area. These tasks reflected what the content experts expected students to have comprehended during the lecture.The criterion measure and the TOEFL iBT Listening section were administered to nonnative speakers of English who were enrolled in undergraduate and graduate programs. Data from 221 participants were analyzed. Substantial correlations were observed between the criterion measure and the TOEFL iBT Listening section score for the entire sample and for subgroups (Pearson correlation coefficients ranging from .56 to .74 and disattenuated correlations ranging from .62 to .82). Moreover, the analysis of the mean scores on the criterion measure for different ability groups indicated that participants who scored at or above typical cut scores for international student admission to academic programs (i.e., TOEFL iBT Listening section score of 14 or above) scored, on average, nearly 50% or more on the criterion measure, demonstrating reasonable comprehension of the academic lectures.
EVALUATING SOURCES OF COLLATERAL INFORMATION ON SMALL‐SAMPLE EQUATING
Authors: Kim Sooyeon , Livingston Samuel A. , Lewis Charles 
Published: August 2014

ABSTRACTThe purpose of this study was to examine the effectiveness of using collateral information to improve the accuracy of small sample equating. Collateral information from the equating of other tests was incorporated into an empirical Bayes (EB) estimate of the reference‐form score corresponding to each new‐form raw score. The evaluation consisted of resampling procedures using data from large‐group equatings of 10 different test forms, representing 9 different tests. Each large‐group equating provided the data for repeated small‐sample equatings based on resampling. The small‐sample equatings were done by 5 methods, including 2 EB methods that used the equating of the other 9 test forms as collateral information. The small‐sample equating methods were evaluated for agreement with the large‐group equating results. The new‐form sample size in the small‐sample equatings was systematically varied (10, 25, 50, 100, 200) as was the sample size in the equatings used as collateral information (100, 200, all available examinees). The results indicated that the use of collateral information tended to improve the accuracy of the equating when the new‐form sample size was 25 or fewer.
AN INVESTIGATION OF VOCABULARY LEARNING THROUGH VOCABULARY BUILDER
Authors: Ventura Matthew , Sabatini John 
Published: August 2014

ABSTRACTVocabulary Builder (VB) is an interactive, gamelike, educational task where students are presented with unknown words in natural reading environments and required to make word meaning inferences. VB supplies sentences through an automated text selection tool that finds text proposed to promote word learning. The students' goal is to try to supply the meaning of an unknown word based on contextual cues given in the text. Feedback is given indicating how close the students' responses are to the meaning of the target word using natural language processing tools. Results from a pilot study of 164 students revealed significant learning gains between pre‐ and posttest sessions using VB. VB appears to aid in the learning of new vocabulary terms.
THREE STATISTICAL TESTING PROCEDURES IN LOGISTIC REGRESSION: THEIR PERFORMANCE IN DIFFERENTIAL ITEM FUNCTIONING (DIF) INVESTIGATION
Authors: Paek Insu 
Published: August 2014

ABSTRACTThree statistical testing procedures well‐known in the maximum likelihood approach are the Wald, likelihood ratio (LR), and score tests. Although well‐known, the application of these three testing procedures in the logistic regression method to investigate differential item function (DIF) has not been rigorously made yet. Employing a variety of simulation conditions, this research (a) assessed the three tests' performance for DIF detection and (b) compared DIF detection in different DIF testing modes (targeted vs. general DIF testing). Simulation results showed small differences between the three tests and different testing modes. However, targeted DIF testing consistently performed better than general DIF testing; the three tests differed more in performance in general DIF testing and nonuniform DIF conditions than in targeted DIF testing and uniform DIF conditions; and the LR and score tests consistently performed better than the Wald test.
IMPROVED RELIABILITY ESTIMATES FOR SMALL SAMPLES USING EMPIRICAL BAYES TECHNIQUES
Authors: Oh Hyeonjoo J. , Guo Hongwen , Walker Michael E. 
Published: August 2014

ABSTRACTIssues of equity and fairness across subgroups of the population (e.g., gender or ethnicity) must be seriously considered in any standardized testing program. For this reason, many testing programs require some means for assessing test characteristics, such as reliability, for subgroups of the population. However, often only small sample sizes are available for the subgroups of interest. Traditionally used reliability estimates (e.g., Cronbach's alpha) can have low precision for small samples. This study investigated whether an empirical Bayes (EB) technique could produce more precise reliability estimates than traditional methods in the presence of small samples. Several Bayesian estimates were compared to estimates obtained by other methods (e.g., the traditionally and currently used Cronbach's alpha coefficient), in terms of both bias and variance. A secondary purpose of this study was to compare the various EB approaches across different sample sizes. This paper also discusses EB estimates of standard error of measurement (SEM), their accuracy and precision, and how they compare with SEM estimates derived from the alpha.
EVALUATION OF METHODS TO COMPUTE COMPLEX SAMPLE STANDARD ERRORS IN LATENT REGRESSION MODELS
Authors: Oranje Andreas , Li Deping , Kandathil Mathew 
Published: August 2014

ABSTRACTSeveral complex sample standard error estimators based on linearization and resampling for the latent regression model of the National Assessment of Educational Progress (NAEP) are studied with respect to design choices such as number of items, number of regressors, and the efficiency of the sample. This paper provides an evaluation of the extent these estimators are appropriate for the models and test lengths often encountered in NAEP and what the effect is on the NAEP imputation model. It is shown that in general the resampling method used in this study provides the most accurate standard errors. However, the differences with the linearization method chosen in this study are relatively small if only small models are used with respect to the independent variables of the latent regression. Illustration is provided through several small simulation studies and NAEP data analysis.
WHEN CAN SUBSCORES BE EXPECTED TO HAVE ADDED VALUE? RESULTS FROM OPERATIONAL AND SIMULATED DATA
Authors: Sinharay Sandip 
Published: August 2014

ABSTRACTRecently, there has been an increasing level of interest in using subscores for their potential diagnostic value. Haberman (2008) suggested a method based on classical test theory to determine whether subscores have added value over total scores. This paper provides a literature review and reports when subscores were found to have added value for several operational data sets. Then this paper provides results from a detailed simulation study that examines what properties subscores should possess in order to have added value. The results indicate that subscores have to satisfy strict standards of reliability and correlation to have added value. Augmented subscores (Haberman, 2008; Wainer et al., 2001) were found to have added value more often.
APPLICATION OF A GENERAL POLYTOMOUS TESTLET MODEL TO THE READING SECTION OF A LARGE‐SCALE ENGLISH LANGUAGE ASSESSMENT
Authors: Li Yanmei , Li Shuhong , Wang Lin 
Published: August 2014

ABSTRACTMany standardized educational tests include groups of items based on a common stimulus, known as testlets. Standard unidimensional item response theory (IRT) models are commonly used to model examinees' responses to testlet items. However, it is known that local dependence among testlet items can lead to biased item parameter estimates when using standard IRT models, and to overestimated reliability. In this study, a general polytomous testlet model was proposed to account for local dependence in testlet‐based tests that contain both dichotomously and polytomously scored items. The proposed model and a standard IRT model were fit to simulated data and several real data sets from the reading sections of a large‐scale English‐language test, and model fit was evaluated. Item parameters and test information obtained from the two models were compared to check the impact of local item dependence. In addition, a multidimensional IRT model with simple structure was also fit to the real data sets. Results based on both simulated and real data suggested that local dependence had a small impact on item parameter estimates and a relatively larger impact on test information and reliability. It was also found that the multidimensional IRT model with simple structure fit the real data sets better than the general polytomous testlet model and the standard IRT model did.
PERFORMANCE AND PASSING RATE DIFFERENCES OF AFRICAN AMERICAN AND WHITE PROSPECTIVE TEACHERS ON PRAXISTM EXAMINATIONS: A JOINT PROJECT OF THE NATIONAL EDUCATION ASSOCIATION (NEA) AND EDUCATIONAL TESTING SERVICE (ETS)
Authors: Nettles Michael T. , Scatton Linda H. , Steinberg Jonathan H. , Tyler Linda L. 
Published: August 2014

ABSTRACTThis report focuses on two aspects of teacher supply and quality: (a) the under‐representation of racial/ethnic minorities, especially African Americans in the teaching pool, and (b) teacher candidates' performance on licensure assessments, including general skills tests in reading, writing, and mathematics (known as Praxis I®) and selected tests covering areas of content and pedagogy (known as Praxis II®). The report presents trends in the overall and minority teaching workforce, utilizes some of the available PraxisTM program data, combined with interviews of faculty and students involved in college and university teacher preparation programs, to address supply, achievement, and performance gaps between prospective minority and White teachers. African American candidates are the focal group in this report, in order to gauge the effect that Praxis has on the supply of prospective African American teachers and to generate ideas for closing gaps. Other minority Praxis test‐taking populations in the selected states are not included because these are not large enough to produce sufficient data from which to make valid inferences.The research was collaboratively planned and guided by National Education Association and Educational Testing Service to use Praxis data for addressing the following questions: (a) What are the trends in minority representation among teacher candidates, (b) What are the differences between majority and minority candidates' performance on both Praxis I and Praxis II, (c) Are the performance differences reflected in the state licensing pass rates of majority and minority candidates, and (d) What factors should be targeted to reduce racial/ethnic group performance differences on Praxis I and Praxis II.
SOURCES OF SCORE SCALE INCONSISTENCY
Authors: Haberman Shelby J. , Dorans Neil J. 
Published: August 2014

ABSTRACTFor testing programs that administer multiple forms within a year and across years, score equating is used to ensure that scores can be used interchangeably. In an ideal world, samples sizes are large and representative of populations that hardly change over time, and very reliable alternate test forms are built with nearly identical psychometric properties. Under these conditions, most equating methods produce score conversions close to the identity function. Unfortunately, equating is sometimes performed on small non‐representative samples with variable distributions of ability, and administered tests are built to vague specifications. Here, different equating methods produce different results because they are based on different assumptions. In the nearly ideal case, there are smaller deviations from the identity function because great effort is taken to control variation. Even when equating is conducted under these desirable conditions, the random variation in form‐to‐form equating, when concatenated over time, can produce substantial shifts in score conversions, that is, scale drift. In this paper, we make distinctions among different sources of variation that may contribute to score‐scale inconsistency, and identify practices that are likely to contribute to it.
USE OF CONTINUOUS EXPONENTIAL FAMILIES TO LINK FORMS VIA ANCHOR TESTS
Authors: Haberman Shelby J. , Yan Duanli 
Published: August 2014

ABSTRACTContinuous exponential families are applied to linking test forms via an internal anchor. This application combines work on continuous exponential families for single‐group designs and work on continuous exponential families for equivalent‐group designs. Results are compared to those for kernel and equipercentile equating in the case of chained equating. The conversions produced by all methods are quite similar.
THE EVALUATION OF BIAS OF THE WEIGHTED RANDOM EFFECTS MODEL ESTIMATORS
Authors: Jia Yue , Stokes Lynne , Harris Ian , Wang Yan 
Published: August 2014

ABSTRACTEstimation of parameters of random effects models from samples collected via complex multistage designs is considered. One way to reduce estimation bias due to unequal probabilities of selection is to incorporate sampling weights. Many researchers have been proposed various weighting methods (Korn, & Graubard, 2003; Pfeffermann, Skinner, Holmes, Goldstein, & Rasbash, 1998) in estimating the parameters of hierarchical models, including random effects models as a special case. In this paper, the bias of the weighted analysis of variance (ANOVA) estimators of the variance components for a two‐level, one‐way random effects model is evaluated. For these estimators, analytic bias expressions are first developed, the expressions are then used to examine the impact of sample size, intraclass correlation coefficient (ICC), and the sampling design on the bias of the estimators. In addition, two‐stage sampling designs are considered, with a general probability design at the first stage (Level 2) and simple random sampling without replacement (SRS) at the second stage (Level 1). The study shows that firstorder weighted variance component estimators perform well when for moderate cluster sizes and ICC values. However, noticeable estimation bias can be found with this weighting method for small cluster sizes (less than 20), particularly when ICC is small (less than 0.2). In such scenarios, scaled first‐order weighted estimators can be an alternative. This paper is discussed in the context of National Assessment of Educational Progress (NAEP) 2003 4th Grade Reading National and State Assessment data, with Level 1 being the student level and Level 2 being the school level.
REPEATER EFFECTS ON SCORE EQUATING FOR A GRADUATE ADMISSIONS EXAM
Authors: Yang Wen‐Ling , Bontya Andrea M. , Moses Tim P. 
Published: August 2014

ABSTRACTUsing self‐reported but empirically verified repeater groups, we analyzed vast amounts of real test data across a wide range of administrations from a graduate admissions examination that was administered in a non‐English language to investigate repeater effects on score equating using the nonequivalent groups with anchor test (NEAT) design. Both linear and nonlinear equating models were considered in deriving the equating functions for various study groups. We evaluated scaled score differences between equating in the total group, the repeater group, and the first‐timer group using statistics of simple differences and subpopulation invariance measures developed and used widely in the last 10 years. Standard errors of statistics summarizing scaled score differences were estimated using a simulation approach to provide statistical criteria for evaluating the significance of equating differences. In addition, we used scaled score differences that were critical to admissions screening as criteria for evaluating the practical significance of equating differences. To put the investigation of repeater effects in proper perspective, we analyzed the repeater data for an in‐depth understanding of repeater performance trends. Overall, we found no significant effects of repeater performance on score equating for the exam being studied. Although many of the equating differences were practically significant, most of the practically significant differences were not statistically significant. However, further research with larger repeater samples is recommended to help explain the practical significance of equating differences consistently observed in this study for the repeater group. Potential problems associated with small repeater study sample sizes, issues of the practical criterion for evaluating the significance of equating differences, and study limitations are also discussed.
AN INVESTIGATION OF THE FIT OF LINEAR REGRESSION MODELS TO DATA FROM AN SAT® VALIDITY STUDY
Authors: Kobrin Jennifer L. , Sinharay Sandip , Haberman Shelby J. , Chajewski Michael 
Published: August 2014

ABSTRACTThis study examined the adequacy of a multiple linear regression model for predicting first‐year college grade point average (FYGPA) using SAT® scores and high school grade point average (HSGPA). A variety of techniques, both graphical and statistical, were used to examine if it is possible to improve on the linear regression model. The results suggest that the linear regression model mostly provides an adequate fit to the data and that more complicated models do not significantly improve the prediction of FYGPA from SAT scores and HSGPA.