bluuebunny/crossref_metadata_2025 huggingface dataset:
MONOTONE PROPERTIES OF A GENERAL DIAGNOSTIC MODEL
Authors: Xu Xueli
Published: August 2014

Monotonicity properties of a general diagnostic model (GDM) are considered in this paper. Simple data summaries are identified to inform about the ordered categories of latent traits. The findings are very much in accordance with the statements made about the GPCM (Hemker, Sijtsma, Molenaar, & Junker, 1996, 1997). On the one hand, by fitting a GDM with equal slopes across items, the observed total score X+ = Σj:qjk=1 xj demonstrates the monotone likelihood ratio (MLR) property in individual coordinate at the cost of losing model fit. On the other hand, fitting a GDM without slope restriction increases the model fit but by sacrificing MLR property. Trade‐offs between these two situations should be considered in practice.
AN ASSESSMENT FOR LEARNING SYSTEM CALLED ACED: DESIGNING FOR LEARNING EFFECTIVENESS AND ACCESSIBILITY
Authors: Shute Valerie J. , Hansen Eric G. , Almond Russell G.
Published: August 2014

This paper reports on a 3‐year, NSF‐funded research and development project called ACED: Adaptive Content with Evidence‐based Diagnosis. The purpose of the project was to design, develop, and evaluate an assessment for learning (AfL) system for diverse students, using Algebra I content related to geometric sequences (i.e., successive numbers linked by a common ratio). A key feature of the project was its dual focus on both learning effectiveness and accessibility. The project consisted of two studies. Study 1 (N = 268) experimentally evaluated the impact on learning of two ACED features: (a) elaborated task feedback and (b) adaptive sequencing of tasks. Study 2 examined the accessibility of ACED for students with visual disabilities (N = 4; two blind and two low‐vision)—in a condition involving both adaptive sequencing and elaborated feedback—using a variety of accessibility features. Available features included voicing of test content (via synthesized and prerecorded audio), talking tactile graphics, font enlargement, screen magnification, and so on. Together these studies lay groundwork for the design of AfL systems that can enhance the learning of individuals with and without disabilities.
MIXTURE DISTRIBUTION DIAGNOSTIC MODELS
Authors: von Davier Matthias
Published: August 2014

This paper introduces the mixture general diagnostic model (MGDM), an extension of the general diagnostic model (GDM). The MGDM extension allows one to estimate diagnostic models for multiple known populations as well as discrete unknown, or not directly observed mixtures of populations. The GDM is based on developments that integrate located latent class models; multiple classification latent class models; and discrete, multidimensional item response models into a common framework. Models of this type express the probability of a response vector as a function of parameters that describe the individual item response variables in terms of required skills and of indirectly observed (latent) skill profiles of respondents. The skills required for solving the items are, as in most diagnostic models, represented as a design matrix that is often referred to as a Q‐matrix. This Q‐matrix consists of rows describing, for each item response, what combination of skills is needed to succeed or to obtain partial or full credit. The hypothesized Q‐matrix is either the result of experts rating items of an existing assessment (retrofitting) or comes directly out of the design of the assessment instrument, in which it served as a tool to design the items.

The MGDM takes the GDM and integrates it into the framework of discrete mixture distribution models for item response data (see von Davier & Rost, 2006). This increases the utility of the GDM by allowing the estimation and testing of models for multiple populations. The MGDM allows for complex scale linkages that make assessments comparable across populations and makes it possible to test whether items function the same in different subpopulations. This can be done with known subpopulations (defined by grade levels, cohorts, etc.), as well as with unknown subpopulations that need to be identified by the model. In both cases, MGDMs make it possible to determine whether different sets of item‐by‐skill parameters and/or different skill distributions have to be assumed for different subpopulations. This amounts to a generalized procedure that can be used to test for differential item functioning (DIF) on one item or on multiple‐response variables using multiple‐group or mixture models. This procedure enables testing DIF models against models that allow additional skills for certain items in order to account for differences between subpopulations.
TIME REQUIREMENTS FOR THE DIFFERENT ITEM TYPES PROPOSED FOR USE IN THE REVISED SAT®
Authors: Bridgeman Brent , Laitusis Cara Cahalan , Cline Frederick
Published: August 2014

The current study used three data sources to estimate time requirements for different item types on the now current SAT Reasoning Test™. First, we estimated times from a computer‐adaptive version of the SAT® (SAT CAT) that automatically recorded item times. Second, we observed students as they answered SAT questions under strict time limits and recorded the amount of time taken for each question. Finally, we asked high school students to record the amount of time taken for test subsections that were composed of items of a single type. The rules of thumb used by test developers were quite accurate in rank ordering the item types from least to most time‐consuming, but the time actually spent was generally higher than assumed in the rules of thumb.
GROWTH IN READING PERFORMANCE DURING THE FIRST FOUR YEARS IN SCHOOL
Authors: Rock Donald A
Published: August 2014

This study addressed concerns about the potential for differential gains in reading during the first 2 years of formal schooling (K‐1) versus the next 2 years of schooling (1s‐3rd grade). A multilevel piecewise regression with a node at spring 1st grade was used in order to define separate regressions for the two time periods. Empirical Bayes estimates of individual growth curves were developed, and smoothed estimates were plotted for selected demographic groups.
AN ALTERNATIVE DATA COLLECTION DESIGN FOR EQUATING WITH VERY SMALL SAMPLES
Authors: Puhan Gautam , Moses Tim , Grant Mary , McHale Fred
Published: August 2014

A single group (SG) equating design with nearly equivalent test forms (SiGNET) design was developed by Grant (2006) to equate small volume tests. The basis of this design is that examinees take two largely overlapping test forms within a single administration. The scored items for the operational form are divided into mini‐tests called testlets. An additional testlet is created but not scored for the first form. If the scored testlets are Testlets 1‐6 and the unscored testlet is Testlet 7, then the first form is composed of Testlets 1‐6, the second form is composed of Testlets 2‐7, and Testlets 2‐6 are common to both test forms. They are administered as a single administered form, and when a sufficient number of examinees have taken the administered form for an SG equating, the second form (Testlets 2‐7) is equated to the first form (Testlets 1‐6) using SG equating. As evident, there are at least two merits of the SiGNET design over the nonequivalent groups with anchor test (NEAT) design. First, it facilitates the use of an SG equating design, which has the least random equating error, and second, it allows for the accumulation of sufficient data to equate the second form. Since the examinees scores are based on only the first form (i.e., the operational form), the two forms can be administered until sufficient data are collected to equate the second form. This study compared equatings under the SiGNET and NEAT designs and found reduced bias and error for the SiGNET design in very small sample size situations (e.g., N = 10 or 15). Implications for practice using the SiGNET design are also discussed.
MODEL‐BASED WEIGHTING AND COMPARISONS
Authors: Qian Jiahe
Published: August 2014

In survey research, sometimes the formation of groupings, or aggregations of cases on which to make an inference, are of importance. Of particular interest are the situations where the cases aggregated carry useful information that has been transferred from a sample employed in a previous study. For example, a school to be included in the sample of the High School Effectiveness (HSES) study must contain one or more cases transferred from the National Educational Longitudinal Study of 1988 (NELS:88). To calculate the aggregation inclusion probabilities, this study investigated three statistical models and, based on these models, derived the school weights for the HSES study. This study also assessed the effects of weighting by comparing the statistics yielded from different sets of weights: (a) those from an empirical population database and (b) those from data generated from simulation based on the principles of a superpopulation. Both categorical data and continuous variables were analyzed in the comparison.
AUTOMATED SCORING OF SHORT‐ANSWER OPEN‐ENDED GRE® SUBJECT TEST ITEMS
Authors: Attali Yigal , Powers Don , Freedman Marshall , Harrison Marissa , Obetz Susan
Published: August 2014

This report describes the development, administration, and scoring of open‐ended variants of GRE® Subject Test items in biology and psychology. These questions were administered in a Web‐based experiment to registered examinees of the respective Subject Tests. The questions required a short answer of 1‐3 sentences, and responses were automatically scored by natural language processing methods, using the c‐rater™ scoring engine, immediately after participants submitted their responses. Participants received immediate feedback on the correctness of their answers, and an opportunity to revise their answers. Subsequent human scoring of the responses allowed an evaluation of the quality of automated scoring. This report focuses on the success of the automated scoring process. A separate report describes the feedback and revision results.
LINKING ENGLISH‐LANGUAGE TEST SCORES ONTO THE COMMON EUROPEAN FRAMEWORK OF REFERENCE: AN APPLICATION OF STANDARD‐SETTING METHODOLOGY
Authors: Tannenbaum Richard J. , Wylie E. Caroline
Published: August 2014

The Common European Framework of Reference (CEFR) describes language proficiency in reading, writing, speaking, and listening on a 6‐level scale. In this study, English‐language experts from across Europe linked CEFR levels to scores on three tests: the TOEFL® iBT test, the TOEIC® assessment, and the TOEIC Bridge™ test. Standard‐setting methodology (a modified Angoff approach and a modified examinee paper selection approach) was used to construct the linkages. Linkages were established for TOEFL iBT at levels B1, B2, and C1. Linkages were established for TOEIC at levels A1 through C1, with the exception of Reading at the C1 level. The TOEIC Bridge test was linked to its three targeted levels of the CEFR. The report details the methods, procedures, and results of the study.
OUTLIERS IN ASSESSMENTS
Authors: Haberman Shelby J.
Published: August 2014

Outliers in assessments are often treated as a nuisance for data analysis; however, they can also assist in quality assurance. Their frequency can suggest problems with form codes, scanning accuracy, ability of examinees to enter responses as they intend, or exposure of items.
COMPARISONS AMONG DESIGNS FOR EQUATING CONSTRUCTED‐RESPONSE TESTS
Authors: Kim Sooyeon , Walker Michael E. , McHale Frederick
Published: August 2014

This study examined variations of a nonequivalent groups equating design used with constructed‐response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large‐scale exam, the study investigated the use of anchor CR item rescoring in the context of classical equating methods. Four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple‐choice (MC) anchor test, and (d) an equivalent groups design incorporating CR items rescored (no anchor test). The use of CR items without rescoring or the use of an external MC anchor resulted in much larger bias than the other two designs. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.
MAPPING STATE STANDARDS TO THE NAEP SCALE
Authors: Braun Henry , Qian Jiahe
Published: August 2014

This report describes the derivation and evaluation of a method for comparing the performance standards for public school students set by different states. It is based on an approach proposed by McLaughlin and associates, which constituted an innovative attempt to resolve the confusion and concern that occurs when very different proportions of students in various states are declared to have met a standard with the same label. Our method, like McLaughlin's, employs equipercentile methods to map state standards on to a common scale, that associated with the National Assessment of Educational Progress (NAEP). We have also derived error estimates that take into account both NAEP's complex sampling design and measurement errors. The method was applied to two data sets, and the results were qualitatively similar to those obtained by McLaughlin's method. The paper notes the superior statistical properties of the proposed method and presents evidence that supports the viability and general utility of this approach.
ESTABLISHING THE VALIDITY OF TOEIC BRIDGE™ TEST SCORES FOR STUDENTS IN COLOMBIA, CHILE, AND ECUADOR
Authors: Sinharay Sandip , Feng Ying , Saldivia Luis , Powers Donald E. , Ginuta Anthony , Simpson Annabelle , Weng Vincent
Published: August 2014

The validity of TOEIC Bridge™ scores as a measure of English language skill was examined from the standpoint of a unified concept of test validity. In this study, more than 6,000 test takers in 3 Latin American countries (Chile, Colombia, and Ecuador) took 1 form of the TOEIC Bridge test, and their scores were compared to additional information about the students (teacher judgments, self‐assessments, and performance on other academic achievement measures). The evidence collected was generally quite consistent with the interpretation of TOEIC Bridge scores as indicators of the English language competencies for the students examined.
AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATERSM V1.0
Authors: Xi Xiaoming , Higgins Derrick , Zechner Klaus , Williamson David M.
Published: August 2014

This report presents the results of a research and development effort for SpeechRaterSM Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL®) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short‐ and long‐term goals for iterative improvements to SpeechRater scoring.
RELIABILITY OF SCALED SCORES
Authors: Haberman Shelby J.
Published: August 2014

The reliability of a scaled score can be computed by use of item response theory. Estimated reliability can be obtained even if the item response model selected is not valid.
CRITERION‐RELATED VALIDITY OF THE TOEFL IBT LISTENING SECTION
Authors: Sawaki Yasuyo , Nissan Susan
Published: August 2014

The study investigated the criterion‐related validity of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT) Listening section by examining its relationship to a criterion measure designed to reflect language‐use tasks that university students encounter in everyday academic life: listening to academic lectures. The design of the criterion measure was informed by students' responses to a survey on the frequency and importance of various classroom tasks that require academic listening, and the relationship of these tasks to successful course completion. The criterion measure consisted of three videotaped lectures (in physics, history, and psychology) and included tasks created by content experts who are former university professors of the relevant content area. These tasks reflected what the content experts expected students to have comprehended during the lecture.

The criterion measure and the TOEFL iBT Listening section were administered to nonnative speakers of English who were enrolled in undergraduate and graduate programs. Data from 221 participants were analyzed. Substantial correlations were observed between the criterion measure and the TOEFL iBT Listening section score for the entire sample and for subgroups (Pearson correlation coefficients ranging from .56 to .74 and disattenuated correlations ranging from .62 to .82). Moreover, the analysis of the mean scores on the criterion measure for different ability groups indicated that participants who scored at or above typical cut scores for international student admission to academic programs (i.e., TOEFL iBT Listening section score of 14 or above) scored, on average, nearly 50% or more on the criterion measure, demonstrating reasonable comprehension of the academic lectures.
EVALUATING SOURCES OF COLLATERAL INFORMATION ON SMALL‐SAMPLE EQUATING
Authors: Kim Sooyeon , Livingston Samuel A. , Lewis Charles
Published: August 2014

The purpose of this study was to examine the effectiveness of using collateral information to improve the accuracy of small sample equating. Collateral information from the equating of other tests was incorporated into an empirical Bayes (EB) estimate of the reference‐form score corresponding to each new‐form raw score. The evaluation consisted of resampling procedures using data from large‐group equatings of 10 different test forms, representing 9 different tests. Each large‐group equating provided the data for repeated small‐sample equatings based on resampling. The small‐sample equatings were done by 5 methods, including 2 EB methods that used the equating of the other 9 test forms as collateral information. The small‐sample equating methods were evaluated for agreement with the large‐group equating results. The new‐form sample size in the small‐sample equatings was systematically varied (10, 25, 50, 100, 200) as was the sample size in the equatings used as collateral information (100, 200, all available examinees). The results indicated that the use of collateral information tended to improve the accuracy of the equating when the new‐form sample size was 25 or fewer.
AN INVESTIGATION OF VOCABULARY LEARNING THROUGH VOCABULARY BUILDER
Authors: Ventura Matthew , Sabatini John
Published: August 2014

Vocabulary Builder (VB) is an interactive, gamelike, educational task where students are presented with unknown words in natural reading environments and required to make word meaning inferences. VB supplies sentences through an automated text selection tool that finds text proposed to promote word learning. The students' goal is to try to supply the meaning of an unknown word based on contextual cues given in the text. Feedback is given indicating how close the students' responses are to the meaning of the target word using natural language processing tools. Results from a pilot study of 164 students revealed significant learning gains between pre‐ and posttest sessions using VB. VB appears to aid in the learning of new vocabulary terms.
THREE STATISTICAL TESTING PROCEDURES IN LOGISTIC REGRESSION: THEIR PERFORMANCE IN DIFFERENTIAL ITEM FUNCTIONING (DIF) INVESTIGATION
Authors: Paek Insu
Published: August 2014

Three statistical testing procedures well‐known in the maximum likelihood approach are the Wald, likelihood ratio (LR), and score tests. Although well‐known, the application of these three testing procedures in the logistic regression method to investigate differential item function (DIF) has not been rigorously made yet. Employing a variety of simulation conditions, this research (a) assessed the three tests' performance for DIF detection and (b) compared DIF detection in different DIF testing modes (targeted vs. general DIF testing). Simulation results showed small differences between the three tests and different testing modes. However, targeted DIF testing consistently performed better than general DIF testing; the three tests differed more in performance in general DIF testing and nonuniform DIF conditions than in targeted DIF testing and uniform DIF conditions; and the LR and score tests consistently performed better than the Wald test.
IMPROVED RELIABILITY ESTIMATES FOR SMALL SAMPLES USING EMPIRICAL BAYES TECHNIQUES
Authors: Oh Hyeonjoo J. , Guo Hongwen , Walker Michael E.
Published: August 2014

Issues of equity and fairness across subgroups of the population (e.g., gender or ethnicity) must be seriously considered in any standardized testing program. For this reason, many testing programs require some means for assessing test characteristics, such as reliability, for subgroups of the population. However, often only small sample sizes are available for the subgroups of interest. Traditionally used reliability estimates (e.g., Cronbach's alpha) can have low precision for small samples. This study investigated whether an empirical Bayes (EB) technique could produce more precise reliability estimates than traditional methods in the presence of small samples. Several Bayesian estimates were compared to estimates obtained by other methods (e.g., the traditionally and currently used Cronbach's alpha coefficient), in terms of both bias and variance. A secondary purpose of this study was to compare the various EB approaches across different sample sizes. This paper also discusses EB estimates of standard error of measurement (SEM), their accuracy and precision, and how they compare with SEM estimates derived from the alpha.
EVALUATION OF METHODS TO COMPUTE COMPLEX SAMPLE STANDARD ERRORS IN LATENT REGRESSION MODELS
Authors: Oranje Andreas , Li Deping , Kandathil Mathew
Published: August 2014

Several complex sample standard error estimators based on linearization and resampling for the latent regression model of the National Assessment of Educational Progress (NAEP) are studied with respect to design choices such as number of items, number of regressors, and the efficiency of the sample. This paper provides an evaluation of the extent these estimators are appropriate for the models and test lengths often encountered in NAEP and what the effect is on the NAEP imputation model. It is shown that in general the resampling method used in this study provides the most accurate standard errors. However, the differences with the linearization method chosen in this study are relatively small if only small models are used with respect to the independent variables of the latent regression. Illustration is provided through several small simulation studies and NAEP data analysis.
WHEN CAN SUBSCORES BE EXPECTED TO HAVE ADDED VALUE? RESULTS FROM OPERATIONAL AND SIMULATED DATA
Authors: Sinharay Sandip
Published: August 2014

Recently, there has been an increasing level of interest in using subscores for their potential diagnostic value. Haberman (2008) suggested a method based on classical test theory to determine whether subscores have added value over total scores. This paper provides a literature review and reports when subscores were found to have added value for several operational data sets. Then this paper provides results from a detailed simulation study that examines what properties subscores should possess in order to have added value. The results indicate that subscores have to satisfy strict standards of reliability and correlation to have added value. Augmented subscores (Haberman, 2008; Wainer et al., 2001) were found to have added value more often.
APPLICATION OF A GENERAL POLYTOMOUS TESTLET MODEL TO THE READING SECTION OF A LARGE‐SCALE ENGLISH LANGUAGE ASSESSMENT
Authors: Li Yanmei , Li Shuhong , Wang Lin
Published: August 2014

Many standardized educational tests include groups of items based on a common stimulus, known as testlets. Standard unidimensional item response theory (IRT) models are commonly used to model examinees' responses to testlet items. However, it is known that local dependence among testlet items can lead to biased item parameter estimates when using standard IRT models, and to overestimated reliability. In this study, a general polytomous testlet model was proposed to account for local dependence in testlet‐based tests that contain both dichotomously and polytomously scored items. The proposed model and a standard IRT model were fit to simulated data and several real data sets from the reading sections of a large‐scale English‐language test, and model fit was evaluated. Item parameters and test information obtained from the two models were compared to check the impact of local item dependence. In addition, a multidimensional IRT model with simple structure was also fit to the real data sets. Results based on both simulated and real data suggested that local dependence had a small impact on item parameter estimates and a relatively larger impact on test information and reliability. It was also found that the multidimensional IRT model with simple structure fit the real data sets better than the general polytomous testlet model and the standard IRT model did.
PERFORMANCE AND PASSING RATE DIFFERENCES OF AFRICAN AMERICAN AND WHITE PROSPECTIVE TEACHERS ON PRAXISTM EXAMINATIONS: A JOINT PROJECT OF THE NATIONAL EDUCATION ASSOCIATION (NEA) AND EDUCATIONAL TESTING SERVICE (ETS)
Authors: Nettles Michael T. , Scatton Linda H. , Steinberg Jonathan H. , Tyler Linda L.
Published: August 2014

This report focuses on two aspects of teacher supply and quality: (a) the under‐representation of racial/ethnic minorities, especially African Americans in the teaching pool, and (b) teacher candidates' performance on licensure assessments, including general skills tests in reading, writing, and mathematics (known as Praxis I®) and selected tests covering areas of content and pedagogy (known as Praxis II®). The report presents trends in the overall and minority teaching workforce, utilizes some of the available PraxisTM program data, combined with interviews of faculty and students involved in college and university teacher preparation programs, to address supply, achievement, and performance gaps between prospective minority and White teachers. African American candidates are the focal group in this report, in order to gauge the effect that Praxis has on the supply of prospective African American teachers and to generate ideas for closing gaps. Other minority Praxis test‐taking populations in the selected states are not included because these are not large enough to produce sufficient data from which to make valid inferences.

The research was collaboratively planned and guided by National Education Association and Educational Testing Service to use Praxis data for addressing the following questions: (a) What are the trends in minority representation among teacher candidates, (b) What are the differences between majority and minority candidates' performance on both Praxis I and Praxis II, (c) Are the performance differences reflected in the state licensing pass rates of majority and minority candidates, and (d) What factors should be targeted to reduce racial/ethnic group performance differences on Praxis I and Praxis II.
SOURCES OF SCORE SCALE INCONSISTENCY
Authors: Haberman Shelby J. , Dorans Neil J.
Published: August 2014

For testing programs that administer multiple forms within a year and across years, score equating is used to ensure that scores can be used interchangeably. In an ideal world, samples sizes are large and representative of populations that hardly change over time, and very reliable alternate test forms are built with nearly identical psychometric properties. Under these conditions, most equating methods produce score conversions close to the identity function. Unfortunately, equating is sometimes performed on small non‐representative samples with variable distributions of ability, and administered tests are built to vague specifications. Here, different equating methods produce different results because they are based on different assumptions. In the nearly ideal case, there are smaller deviations from the identity function because great effort is taken to control variation. Even when equating is conducted under these desirable conditions, the random variation in form‐to‐form equating, when concatenated over time, can produce substantial shifts in score conversions, that is, scale drift. In this paper, we make distinctions among different sources of variation that may contribute to score‐scale inconsistency, and identify practices that are likely to contribute to it.
USE OF CONTINUOUS EXPONENTIAL FAMILIES TO LINK FORMS VIA ANCHOR TESTS
Authors: Haberman Shelby J. , Yan Duanli
Published: August 2014

Continuous exponential families are applied to linking test forms via an internal anchor. This application combines work on continuous exponential families for single‐group designs and work on continuous exponential families for equivalent‐group designs. Results are compared to those for kernel and equipercentile equating in the case of chained equating. The conversions produced by all methods are quite similar.
THE EVALUATION OF BIAS OF THE WEIGHTED RANDOM EFFECTS MODEL ESTIMATORS
Authors: Jia Yue , Stokes Lynne , Harris Ian , Wang Yan
Published: August 2014

Estimation of parameters of random effects models from samples collected via complex multistage designs is considered. One way to reduce estimation bias due to unequal probabilities of selection is to incorporate sampling weights. Many researchers have been proposed various weighting methods (Korn, & Graubard, 2003; Pfeffermann, Skinner, Holmes, Goldstein, & Rasbash, 1998) in estimating the parameters of hierarchical models, including random effects models as a special case. In this paper, the bias of the weighted analysis of variance (ANOVA) estimators of the variance components for a two‐level, one‐way random effects model is evaluated. For these estimators, analytic bias expressions are first developed, the expressions are then used to examine the impact of sample size, intraclass correlation coefficient (ICC), and the sampling design on the bias of the estimators. In addition, two‐stage sampling designs are considered, with a general probability design at the first stage (Level 2) and simple random sampling without replacement (SRS) at the second stage (Level 1). The study shows that firstorder weighted variance component estimators perform well when for moderate cluster sizes and ICC values. However, noticeable estimation bias can be found with this weighting method for small cluster sizes (less than 20), particularly when ICC is small (less than 0.2). In such scenarios, scaled first‐order weighted estimators can be an alternative. This paper is discussed in the context of National Assessment of Educational Progress (NAEP) 2003 4th Grade Reading National and State Assessment data, with Level 1 being the student level and Level 2 being the school level.
REPEATER EFFECTS ON SCORE EQUATING FOR A GRADUATE ADMISSIONS EXAM
Authors: Yang Wen‐Ling , Bontya Andrea M. , Moses Tim P.
Published: August 2014

Using self‐reported but empirically verified repeater groups, we analyzed vast amounts of real test data across a wide range of administrations from a graduate admissions examination that was administered in a non‐English language to investigate repeater effects on score equating using the nonequivalent groups with anchor test (NEAT) design. Both linear and nonlinear equating models were considered in deriving the equating functions for various study groups. We evaluated scaled score differences between equating in the total group, the repeater group, and the first‐timer group using statistics of simple differences and subpopulation invariance measures developed and used widely in the last 10 years. Standard errors of statistics summarizing scaled score differences were estimated using a simulation approach to provide statistical criteria for evaluating the significance of equating differences. In addition, we used scaled score differences that were critical to admissions screening as criteria for evaluating the practical significance of equating differences. To put the investigation of repeater effects in proper perspective, we analyzed the repeater data for an in‐depth understanding of repeater performance trends. Overall, we found no significant effects of repeater performance on score equating for the exam being studied. Although many of the equating differences were practically significant, most of the practically significant differences were not statistically significant. However, further research with larger repeater samples is recommended to help explain the practical significance of equating differences consistently observed in this study for the repeater group. Potential problems associated with small repeater study sample sizes, issues of the practical criterion for evaluating the significance of equating differences, and study limitations are also discussed.
AN INVESTIGATION OF THE FIT OF LINEAR REGRESSION MODELS TO DATA FROM AN SAT® VALIDITY STUDY
Authors: Kobrin Jennifer L. , Sinharay Sandip , Haberman Shelby J. , Chajewski Michael
Published: August 2014

This study examined the adequacy of a multiple linear regression model for predicting first‐year college grade point average (FYGPA) using SAT® scores and high school grade point average (HSGPA). A variety of techniques, both graphical and statistical, were used to examine if it is possible to improve on the linear regression model. The results suggest that the linear regression model mostly provides an adequate fit to the data and that more complicated models do not significantly improve the prediction of FYGPA from SAT scores and HSGPA.
THE CBAL READING ASSESSMENT: AN APPROACH FOR BALANCING MEASUREMENT AND LEARNING GOALS
Authors: Sheehan Kathleen M. , O'Reilly Tenaha
Published: August 2014

No Child Left Behind has highlighted the need for new types of assessments that not only provide high‐quality evidence about what students know and can do, but also help to move learning forward. This paper describes a linked set of formative and summative reading assessments designed to address the tradeoffs inherent in these two goals. Targeted skills include the full range of competencies underlying proficient reading at the middle‐school level, including both lower‐order skills such as oral reading fluency and decoding, and higher‐order skills such as the ability to integrate and synthesize information from multiple texts. Data collected in pilot administrations of two prototype test forms are presented. Analyses suggest that this new approach yields acceptable measurement properties while simultaneously addressing crucial learning outcomes. This paper was presented June 23, 2009 at the National Conference on Student Assessment, Los Angeles, CA.
CBAL: RESULTS FROM PILOTING INNOVATIVE K–12 ASSESSMENTS
Authors: Bennett Randy Elliot
Published: August 2014

CBAL, an acronym for Cognitively Based Assessment of, for, and as Learning, is a research initiative intended to create a model for an innovative K–12 assessment system that provides summative information for policy makers, as well as formative information for classroom instructional purposes. This paper summarizes empirical results from 16 CBAL summative assessment pilots involving almost 10,000 online administrations conducted between 2007 and 2010. The results suggest that, on average, the CBAL initiative was successful in building innovative assessments in reading, writing, and mathematics that worked as intended. However, there was considerable variation in the functioning of test forms, especially in writing and math. That variation might suggest that the knowledge needed to produce high‐quality forms of these innovative tests in a replicable and scalable manner is not yet at hand. Further, although the results described offer a significant start, many critical questions must be answered before CBAL assessments (or ones like them) are ready for high‐stakes operational use.
COMPUTER‐ADAPTIVE TESTING FOR STUDENTS WITH DISABILITIES: A REVIEW OF THE LITERATURE
Authors: Stone Elizabeth , Davey Tim
Published: August 2014

There has been an increased interest in developing computer‐adaptive testing (CAT) and multistage assessments for K‐12 accountability assessments. The move to adaptive testing has been met with some resistance by those in the field of special education who express concern about routing of students with divergent profiles (e.g., some students with math‐based learning disabilities may have difficulty with basic computation but not high level problem solving) and poor performance on early test questions. This paper consists of a literature review focusing on adaptive testing issues for students with disabilities in the K‐12 sector. While it is clear that there are issues that will present obstacles to administering accountability tests adaptively to students with disabilities, this synthesis of research and policy developments with respect to this topic will be useful both for development of research agendas and to inform states that are currently using or are considering moving to CAT.
A DIFFERENTIAL WORD USE MEASURE FOR CONTENT ANALYSIS IN AUTOMATED ESSAY SCORING
Authors: Attali Yigal
Published: August 2014

This paper proposes an alternative content measure for essay scoring, based on the difference in the relative frequency of a word in high‐scored versus low‐scored essays. The differential word use (DWU) measure is the average of these differences across all words in the essay. A positive value indicates the essay is using vocabulary more typical of high‐scoring essays in this task, and vice versa. In addition to the traditional prompt level, this measure is also computed at the generic task level, where the content of an essay is evaluated in the context of the general vocabulary of the writing task (e.g., GRE® issue), across different prompts. Evaluation results across four GRE and TOEFL® tasks are presented. Factor analyses show that both the prompt and task DWU measures load on the same factor as the prompt‐specific content analysis measures of e‐rater®. Regression results on the human essay scores show that the generic task DWU measure is a strong predictor of human scores, second only to essay length among noncontent e‐rater features. This measure provides a way to conduct a task‐level content analysis that is related to the prompt‐specific content analysis of e‐rater but does not require prompt‐specific training. Regression results for the prompt‐level DWU show that it can be used as a replacement to prompt‐specific content‐vector analysis (CVA) features.
IMPROVING TEST SCORE REPORTING: PERSPECTIVES FROM THE ETS SCORE REPORTING CONFERENCE
Authors: Zapata‐Rivera Diego , Zwick Rebecca , Hullman Jessica , Rhodes Rebecca , Rodriguez Fernando , Shah Priti , Zwick Rebecca , Sklar Jeffrey C. , Zapata‐Rivera Diego
Published: August 2014

This volume includes 3 papers based on presentations at a workshop on communicating assessment information to particular audiences, held at Educational Testing Service (ETS) on November 4th, 2010, to explore some issues that influence score reports and new advances that contribute to the effectiveness of these reports. Jessica Hullman, Rebecca Rhodes, Fernando Rodriguez, and Priti Shah present the results of recent research on graph comprehension and data interpretation, especially the role of presentation format, the impact of prior quantitative literacy and domain knowledge, the trade‐off between reducing cognitive load and increasing active processing of data, and the affective influence of graphical displays. Rebecca Zwick and Jeffrey Sklar present the results of the Instructional Tools in Educational Measurement and Statistics for School Personnel (ITEMS) project, funded by the National Science Foundation and conducted at the University of California, Santa Barbara to develop and evaluate 3 web‐based instructional modules intended to help educators interpret test scores. Zwick and Sklar discuss the modules and the procedures used to evaluate their effectiveness. Diego Zapata‐Rivera presents a new framework for designing and evaluating score reports, based on work on designing and evaluating score reports for particular audiences in the context of the CBAL (Cognitively Based Assessment of, for, and as Learning) project (Bennett & Gitomer, 2009), which has been applied in the development and evaluation of reports for various audiences including teachers, administrators and students.
ADVANCING THE QUALITY AND EQUITY OF EDUCATION FOR LATINO STUDENTS: A WHITE PAPER
Authors: Young John W. , Lakin Joni , Courtney Rosalea , Martiniello María , Adler Rachel , Blood Ian , Burrus Jeremy , DiCrecchio Nicole , Elliott Diane , Miller Susanne
Published: August 2014

This white paper provides an overview of the issues that affect the quality and equity of education in grades K–16 for Latino students in the United States. This paper is organized chronologically to reflect the typical educational timeline for students in the United States, and we focused on several key transition points in the educational timeline: from elementary school to middle school, from middle school to high school, from high school to college, and from a two‐year college to a four‐year college. Each of these transition points can pose challenges for every student, but in this paper, we focused specifically on the unique challenges that Latino students encounter. It is our hope that this paper will provide an impetus to action and will serve as a guide for future directions for collective research efforts on the critical issues regarding the education of Latino students in the 21st century.
MODELING CHANGE IN LARGE‐SCALE LONGITUDINAL STUDIES OF EDUCATIONAL GROWTH: FOUR DECADES OF CONTRIBUTIONS TO THE ASSESSMENT OF EDUCATIONAL GROWTH
Authors: Rock Donald A.
Published: August 2014

This paper provides a history of ETS's role in developing assessment instruments and psychometric procedures for measuring change in large‐scale national assessments funded by the Longitudinal Studies branch of the National Center for Education Statistics. It documents the innovations developed during more than 30 years of working with longitudinal studies based on national samples. In addition to describing these innovations, it provides their rationale and presents some empirical results in their support.
A REVIEW OF ETS DIFFERENTIAL ITEM FUNCTIONING ASSESSMENT PROCEDURES: FLAGGING RULES, MINIMUM SAMPLE SIZE REQUIREMENTS, AND CRITERION REFINEMENT
Authors: Zwick Rebecca
Published: August 2014

Differential item functioning (DIF) analysis is a key component in the evaluation of the fairness and validity of educational tests. The goal of this project was to review the status of ETS DIF analysis procedures, focusing on three aspects: (a) the nature and stringency of the statistical rules used to flag items, (b) the minimum sample size requirements that are currently in place for DIF analysis, and (c) the efficacy of criterion refinement. The main findings of the review are as follows:

• The ETS C rule often displays low DIF detection rates even when samples are large.

• With improved flagging rules in place, minimum sample size requirements could probably be relaxed. In addition, updated rules for combining data across administrations could allow DIF analyses to be performed in a broader range of situations.

• Refinement of the matching criterion improves detection rates when DIF is primarily in one direction but can depress detection rates when DIF is balanced. If nothing is known about the likely pattern of DIF, refinement is advisable.

Each of these findings is discussed in detail, focusing on the case of dichotomous items.
DOES PREEQUATING WORK? AN INVESTIGATION INTO A PREEQUATED TESTLET‐BASED COLLEGE PLACEMENT EXAM USING POSTADMINISTRATION DATA
Authors: Gao Rui , He Wei , Ruan Chunyi
Published: August 2014

In this study, we investigated whether preequating results agree with equating results that are based on observed operational data (postequating) for a college placement program. Specifically, we examined the degree to which item response theory (IRT) true score preequating results agreed with those from IRT true score postequating and from observed score equating. Three academic subjects were examined in this study: analyzing and interpreting literature, American government, and college algebra. The findings suggested that differences between equating results from IRT true score preequating and postequating varied from subject to subject. In general, IRT true score postequating agreed with IRT true score preequating for most of the forms for a test subject. Any difference among the equating results can be attributed to the way through which items were pretested, contextual/order effects, or the violation of IRT assumptions.
A NOTE ON THE CHOICE OF AN ANCHOR TEST IN EQUATING
Authors: Sinharay Sandip , Haberman Shelby , Holland Paul , Lewis Charles
Published: August 2014

Anchor tests play a key role in test score equating. We attempt to find, through theoretical derivations, an anchor test with optimal item characteristics. The correlation between the scores on a total test and on an anchor test is maximized with respect to the item parameters for data satisfying several item response theory models. Results suggest that under these models, the minitest, the traditionally used anchor test, is not optimal with respect to anchor‐test‐to‐total‐test correlation; instead, an anchor test with items of medium difficulty, the miditest, seems to be the optimum anchor test. This finding agrees with the empirical findings of Sinharay and colleagues that the miditest mostly has higher anchor‐test‐to‐total‐test correlation compared to the minitest and mostly performs as well as the minitest in equating.
IDENTIFYING SPEECH ACTS IN E‐MAILS: TOWARD AUTOMATED SCORING OF THE TOEIC® E‐MAIL TASK
Authors: Felice Rachele De , Deane Paul
Published: August 2014

This study proposes an approach to automatically score the TOEIC® Writing e‐mail task. We focus on one component of the scoring rubric, which notes whether the test‐takers have used particular speech acts such as requests, orders, or commitments. We developed a computational model for automated speech act identification and tested it on a corpus of TOEIC responses, achieving up to 79.28% accuracy. This model represents a positive first step toward the development of a more comprehensive scoring model. We also created a corpus of speech act‐annotated native English workplace e‐mails. Comparisons between these and the TOEIC data allow us to assess whether English learners are approximating native models and whether differences between native and non‐native data can have negative consequences in the global workplace.
EXAMINING LINGUISTIC CHARACTERISTICS OF PARAPHRASE IN TEST‐TAKER SUMMARIES
Authors: Burstein Jill , Flor Michael , Tetreault Joel , Madnani Nitin , Holtzman Steven
Published: August 2014

This annotation study is designed to help us gain an increased understanding of paraphrase strategies used by native and nonnative English speakers and how these strategies might affect test takers' essay scores. Toward that end, this study aims to examine and analyze the paraphrase and the types of linguistic modifications used in paraphrase in test‐taker responses and differences that may exist between native and nonnative English speakers. We are also interested in how these factors might influence final essay score. Outcomes discussed in this report can be used to inform the development of new e‐rater® scoring engine features that capture information related to paraphrase, specifically in nonnative speaker responses to the TOEFL® exam integrated writing task.