bluuebunny/crossref_metadata_2025 huggingface dataset:
LINKING WITH CONTINUOUS EXPONENTIAL FAMILIES: SINGLE‐GROUP DESIGNS
Authors: Haberman Shelby J.
Published: August 2014

Continuous exponential families are applied to linking forms via a single‐group design. In this application, a distribution from the continuous bivariate exponential family is used that has selected moments that match those of the bivariate distribution of scores on the forms to be linked. The selected continuous bivariate distribution then yields continuous univariate marginal distributions for the two forms. These marginal distributions then provide distribution functions and quantile functions that may be employed in equating. Normal approximations are obtained for the sample distributions of the conversion functions.
SUBSCORES AND VALIDITY
Authors: Haberman Shelby J.
Published: August 2014

In educational testing, subscores may be provided based on a portion of the items from a larger test. One consideration in evaluation of such subscores is their ability to predict a criterion score. Two limitations on prediction exist. The first, which is well known, is that the coefficient of determination for linear prediction of the criterion score by the subscore cannot exceed the reliability coefficient of the subscore. The second limitation is on incremental validity. The coefficient of determination for linear prediction of the criterion score by both the total score and the subscore is at least as great as the coefficient of determination for linear prediction of the criterion score by only the total score. Incremental validity may be measured by the difference between these two coefficients of determination. This difference is no greater than the reliability of the residual from linear prediction of the subscore by the total score.
MONITORING AND FOSTERING LEARNING THROUGH GAMES AND EMBEDDED ASSESSMENTS
Authors: Shute Valerie J. , Ventura Matthew , Bauer Malcolm , Zapata‐Rivera Diego
Published: August 2014

To reveal what is being learned during the gaming experience, this report proposes an approach for embedding assessments in immersive games, drawing on recent advances in assessment design. Key to this approach are formative assessment to guide instructional experiences and evidence‐centered design to systematically analyze the assessment argument (including the claims about the learner and the evidence that supports or fails to support those claims). Elements of this approach that have been applied in a nongame setting are shown and ideas are discussed for applying the approach to an existing immersive game setting. Finally, the report offers suggestions for extending and applying this approach for existing games and the design of new ones.
FIRST LANGUAGE OF EXAMINEES AND ITS RELATIONSHIP TO EQUATING
Authors: Liang Longjuan , Dorans Neil J. , Sinharay Sandip
Published: August 2014

To ensure fairness, it is important to better understand the relationship of language proficiency with the standard procedures of psychometric analysis. This paper examines how equating results are affected by an increase in the proportion of examinees who report that English is not their first language, using the analysis samples for a large‐scale reading and mathematics test. The results indicate that equating is not affected much by an increase in the proportion of those examinees.
MODELS FOR EFFECTIVE AND SCALABLE TEACHER PROFESSIONAL DEVELOPMENT
Authors: Thompson Marnie , Goe Laura
Published: August 2014

The minute‐to‐minute and day‐by‐day use of assessment for learning holds great potential to change the trajectory of student learning in U.S. classrooms. But without effective and scalable systems of professional development that actually lead teachers to adopt these practices, the utility and impact of assessment for learning will be quite limited. Drawing on learning theory, expertise research, and research on effective professional development, this paper presents the theoretical and empirical basis behind an evolving program of professional development that employs school‐embedded teacher learning communities as a central component, called Keeping Learning on Track®. Implementations of the Keeping Learning on Track program in two districts and a consortium of districts are described in detail.
TEACHER PROFESSIONAL DEVELOPMENT FOCUSED ON FORMATIVE ASSESSMENT: CHANGING TEACHERS, CHANGING SCHOOLS
Authors: Wylie E. Caroline , Lyon Christine J. , Goe Laura
Published: August 2014

This paper outlines an approach to improving learning and teaching that combines two strong research bases: The research on formative assessment or assessment for learning provides information about what to change; research on teacher learning communities guides decisions about how to change. In this paper we describe the content and process for one model of teacher professional development. Two case studies for two school districts engaged in this process of teacher change are then presented along with descriptions of the observed effects on both the participating teachers and their larger school contexts.
FIRST LANGUAGE OF EXAMINEES AND ITS RELATIONSHIP TO DIFFERENTIAL ITEM FUNCTIONING
Authors: Sinharay Sandip , Dorans Neil J. , Liang Longjuan
Published: August 2014

To ensure fairness, it is important to better understand the relationship of language proficiency to standard psychometric analysis procedures. This paper examines how results of differential item functioning (DIF) analysis are affected by an increase in the proportion of examinees who report that English is not their first language in the analysis sample of a large‐scale assessment. The results vary by group. In some combinations of focal/reference groups, the magnitude of DIF is not appreciably affected by whether the DIF is performed on examinees whose first language is not English. In other groups, the first language status matters. The results vary by type of test as well. In addition, the magnitude of DIF for some items is substantially affected by whether the DIF is performed on examinees whose first language is not English.
SELF‐CONSTRUCTS AND ANXIETY ACROSS CULTURES
Authors: Lee Jihyun
Published: August 2014

This study examined the factorial structure of three related constructs, math self‐concept, math self‐efficacy, and math anxiety, across 41 countries. One factorial structure was achieved at both between‐ and within‐country levels. Within‐country variations of the self‐constructs were also noted in relation to math performance: Self‐concept showed the strongest correlations in some Western European countries, but it was self‐efficacy that demonstrated the strongest correlations among Asian and Eastern European countries. Findings also indicate that students in some Asian countries have low math self‐concept and math self‐efficacy and high math anxiety despite high math performance, while students in some Western European countries have high math performance and low levels of math anxiety.
USE OF GENERALIZED RESIDUALS TO EXAMINE GOODNESS OF FIT OF ITEM RESPONSE MODELS
Authors: Haberman Shelby J.
Published: August 2014

Generalized residuals are a tool employed in the analysis of contingency tables to examine goodness of fit. They may be applied to item response models with little complication. Their use is illustrated with testing data from operational programs. Models considered include the Rasch model and the two‐parameter logistic model.
THE MISSING DATA ASSUMPTIONS OF THE NONEQUIVALENT GROUPS WITH ANCHOR TEST (NEAT) DESIGN AND THEIR IMPLICATIONS FOR TEST EQUATING
Authors: Sinharay Sandip , Holland Paul W.
Published: August 2014

The nonequivalent groups with anchor test (NEAT) design involves missing data that are missing by design. Three popular equating methods that can be used with a NEAT design are the poststratification equating method, the chain equipercentile equating method, and the item‐response‐theory observed‐score‐equating method. These three methods each make different assumptions about the missing data in the NEAT design. Though studies have compared the equating performance of the three methods under the NEAT design, none has examined the missing data assumptions and their implications for such comparisons. The missing data assumptions can affect equating studies because it is necessary to fill in the missing data or their distribution in some way in order to have a true, or criterion, equating function to compare the accuracy and bias of the different methods. If the missing data or their distribution are filled in using missing data assumptions that correspond to a given method, that may favor that method in any comparison with the others. This paper first describes the missing data assumptions of the three equating methods and then performs a fair comparison of the 3 methods using data from 3 different operational tests. For each data set, we examine how the 3 equating methods perform when the missing data satisfy the assumptions made by only 1 of these equating methods. The chain equating method is somewhat more satisfactory overall than the other methods in our fair comparison of the methods; hence, we recommend that equating practitioners seriously consider the chain equating method when using the NEAT design. In addition, we conclude that the results from the different equating methods will tend to agree with each other when proper equating conditions are in place. Moreover, to uncover problems that might not reveal themselves otherwise, it is important for operational testing programs to apply multiple equating methods and study the differences among their results.
THE TOEIC® SPEAKING AND WRITING TESTS: RELATIONS TO TEST‐TAKER PERCEPTIONS OF PROFICIENCY IN ENGLISH
Authors: Powers Donald E. , Kim Hae‐Jin , Yu Feng , Weng Vincent Z. , VanWinkle Waverely
Published: August 2014

To facilitate the interpretation of test scores from the new TOEIC® speaking and writing tests as measures of English‐language proficiency, we administered a self‐assessment inventory to TOEIC examinees in Japan and Korea, to gather their perceptions of their ability to perform a variety of everyday English‐language tasks. TOEIC scores related relatively strongly to test taker self‐reports for both speaking and writing tasks. The results were extraordinarily consistent, with examinees at each higher TOEIC score level being more likely to report that they could successfully accomplish each of the everyday language tasks in English. The pattern of correlations also suggested modest discriminant validity of the new TOEIC speaking and writing measures, suggesting that both measures contribute uniquely to the assessment of English‐language proficiency.
UNTYING THE KNOT: REVIEW OF RESEARCH ON THE INFLUENCE OF PARENTAL INVOLVEMENT ON STUDENTS' ACADEMIC ACHIEVEMENT AT THE SECONDARY SCHOOL LEVEL
Authors: Shute Valerie J. , Hansen Eric G. , Underwood Jody
Published: August 2014

The idea that parental involvement engenders students' academic achievement is so intuitively appealing that society in general, and educators in particular, have considered parental involvement an important ingredient for the remedy of many ills in education today. In the 1980s and early 1990s, a number of studies were published that suggested the importance of parental involvement. In response, in the mid‐1990s, the popular press, policy makers, and school administrators promoted parental involvement. Legislation was enacted, such as the Goals 2000: Educate America Act and the reauthorized Elementary and Secondary Education Act (ESEA). However, in the late 1990s, research began appearing that questioned the preeminence of the parental role. A landmark in this debate was Judith Harris's 1998 book, The Nurture Assumption, in which Harris argued that parents have little or no influence over the long‐term development of their children's personality or academic success. Other studies have followed, suggesting that parental involvement may not matter as much or in the ways that had been believed. This paper reviews the research literature on the influence of parental involvement on academic achievement, with particular attention to the secondary school level.
CONSTRUCTION OF CHAINED TRUE SCORE EQUIPERCENTILE EQUATINGS UNDER THE KERNEL EQUATING (KE) FRAMEWORK AND THEIR RELATIONSHIP TO LEVINE TRUE SCORE EQUATING
Authors: Chen Haiwen (Henry) , Holland Paul
Published: August 2014

In this paper, we develop a new chained equipercentile equating procedure for the nonequivalent groups with anchor test (NEAT) design under the assumptions of the classical test theory model. This new equating is named chained true score equipercentile equating. We also apply the kernel equating framework to this equating design, resulting in a family of chained true score equipercentile equating functions, which include the Levine true score equating model as a special case.
EFFECT OF REPEATERS ON SCORE EQUATING IN A LARGE‐SCALE LICENSURE TEST
Authors: Kim Sooyeon , Walker Michael E.
Published: August 2014

This study investigated the subgroup invariance of equating functions for a licensure test in the context of a nonequivalent groups with anchor test (NEAT) design. Examinees who had taken a new, to‐be‐equated form of the test were divided into three subgroups according to their previous testing experience: (a) repeaters who previously took the reference form, to which the new form would be equated; (b) repeaters who previously took any form other than the reference form; and (c) first‐time test takers for whom the new form was the first exposure to the test. Equating functions obtained with two subgroups, all repeaters, and first‐time test takers were similar to those obtained with the total group, supporting score equatability of the two forms. However, when the repeater subgroup was divided into two groups based on the particular form examinees took previously, subgroup equating functions substantially differed from the total‐group equating function, indicating subgroup dependency of score equating. The results indicate that repeater membership needs to be more clearly specified to assess the impact of repeaters on score equating. Such clarification may be especially necessary for high‐stakes licensure tests because repeaters tend to perform more poorly on such tests than first‐time test takers.
USING THE GENERAL DIAGNOSTIC MODEL TO MEASURE LEARNING AND CHANGE IN A LONGITUDINAL LARGE‐SCALE ASSESSMENT
Authors: von Davier Matthias , Xu Xueli , Carstensen Claus H.
Published: August 2014

A general diagnostic model was used to specify and compare two multidimensional item‐response‐theory (MIRT) models for longitudinal data: (a) a model that handles repeated measurements as multiple, correlated variables over time (Andersen, 1985) and (b) a model that assumes one common variable over time and additional orthogonal variables that quantify the change (Embretson, 1991). Using MIRT‐model ability distributions that we allowed to vary across subpopulations defined by type of school, we also compared (a) a model with a single two‐dimensional ability distribution to (b) extensions of the Andersen and Embretson approaches, assuming multiple populations. In addition, we specified a hierarchical‐mixture distribution variant of the (Andersen and Embretson) MIRT models and compared it to all four of the above alternatives. These four types of models are growth‐mixture models that allow for variation of the mixing proportions across clusters in a hierarchically organized sample. To illustrate the models presented in this paper, we applied the models to the PISA‐I‐Plus data for assessing learning and change across multiple subpopulations. The results indicate that (a) the Embretson‐type model with multiple‐group assumptions fits the data better than the other models investigated, and (b) the higher performing group shows larger improvement at Time Point 2 than the lower performing group.
DOES CONTENT KNOWLEDGE AFFECT TOEFL IBT™ READING PERFORMANCE? A CONFIRMATORY APPROACH TO DIFFERENTIAL ITEM FUNCTIONING
Authors: Liu Ou Lydia , Schedl Mary , Malloy Jeanne , Kong Nan
Published: August 2014

The TOEFL iBT™ has increased the length of the reading passages in the reading section compared to the passages on the TOEFL® computer‐based test (CBT) to better approximate academic reading in North American universities, resulting in a reduced number of passages in the reading test. A concern arising from this change is whether the decrease in topic variety increases the likelihood that an examinee's familiarity with the particular content of a given passage will influence the examinee's reading performance. This study investigated differential item functioning and differential bundle functioning for six TOEFL iBT reading passages, three involving physical science and three involving cultural topics. The majority of items displayed little or no differential item functioning (DIF). When all of the items in a passage were examined, none of the passages showed differential functioning at the passage level. Hypotheses are provided for the DIF occurrences. Implications for fairness issues in test development are also discussed.
THE SPEAKING SECTION OF THE TOEFL IBT™ (SSTIBT): TEST‐TAKERS' REPORTED STRATEGIC BEHAVIORS
Authors: Swain Merrill , Huang Li‐Shih , Barkaoui Khaled , Brooks Lindsay , Lapkin Sharon
Published: August 2014

This study responds to the Test of English as a Foreign Language™ (TOEFL®) research agenda concerning the need to understand the processes and knowledge that test‐takers utilize. Specifically, it investigates the strategic behaviors test‐takers reported using when taking the Speaking section of the TOEFL iBT™ (SSTiBT). It also investigates how the reported strategic behaviors differed across integrated and independent tasks in the SSTiBT, as well as the relationship between test‐takers' reported strategic behaviors and their performance on the tasks as determined by their test scores. The participating students were 14 graduate and 16 undergraduate engineering students whose first language was Chinese.

The results indicate that test‐takers reported using 49 separate strategies when completing the SSTiBT tasks. Of the five strategy categories, the metacognitive, communication, and cognitive strategies were proportionally the most frequently reported. The interrelationships among these three categories were negative. Undergraduates reported using significantly more communication strategies, whereas graduates reported using significantly more cognitive and affective strategies. No statistically significant differences were found in reported strategy use across proficiency levels. The integrated tasks were more alike with respect to reported strategy use than were the independent and integrated tasks. Furthermore, the integrated tasks elicited a wider variety of reported strategy use than the independent tasks. Overall, we found no relationship between the total number of reported strategic behaviors and total test score on the SSTiBT.

We conclude that strategy use is integral to performing SSTiBT tasks and should therefore be considered as part of the construct of communicative performance. However, the relationship between strategy use and test performance is varied and is due to complex interactions among test‐taker characteristics, tasks, and contexts.
STRATEGIES FOR TESTING SLOPE DIFFERENCES
Authors: Moses Tim , Klockars Alan
Published: August 2014

The robustness and power of 9 strategies for testing the differences in groups' regression slopes were assessed under nonnormality and residual variance heterogeneity. For the conditions considered, the most robust strategies were the trimmed and Winsorized slope estimates used with the James second‐order test, the Theil‐Sen slope estimates used with James, and the Theil‐Sen estimates used with percentile bootstrapping. The use of Theil‐Sen slope estimates produced more powerful tests than the use of trimmed and Winsorized slopes.
EVALUATING SUBPOPULATION INVARIANCE OF LINKING FUNCTIONS TO DETERMINE THE ANCHOR COMPOSITION FOR A MIXED‐FORMAT TEST
Authors: Kim Sooyeon , Walker Michael E.
Published: August 2014

We examined the appropriateness of the anchor composition in a mixed‐format test, which includes both multiple‐choice (MC) and constructed‐response (CR) items, using subpopulation invariance indices. We derived linking functions in the nonequivalent groups with anchor test (NEAT) design using two types of anchor sets: (a) MC only and (b) a mix of MC and CR. In each anchor condition, we derived the linking functions separately for males and females and then compared those subpopulation functions to the total group function. In the MC‐only condition, the difference between the subpopulation functions and the total group function was not trivial in a score region that included cut scores, leading to inconsistent pass/fail decisions for low‐performing examinees in particular. Overall, the mixed anchor was a better choice than the MC‐only anchor to achieve subpopulation invariance between males and females. The research reinforces subpopulation invariance indices as a means of determining the adequacy of the anchor.
THREE MULTIDIMENSIONAL MODELS FOR TESTLET‐BASED TESTS: FORMAL RELATIONS AND AN EMPIRICAL COMPARISON
Authors: Rijmen Frank
Published: August 2014

Three multidimensional item response theory (IRT) models for testlet‐based tests are described. In the bifactor model (Gibbons & Hedeker, 1992), each item measures a general dimension in addition to a testlet‐specific dimension. The testlet model (Bradlow, Wainer, & Wang, 1999) is a bifactor model in which the loadings on the specific dimensions are restricted to be proportional to the loadings on the general dimension within each testlet (Li, Bolt, & Fu, 2006). In the second‐order model, the items only load on the testlet‐specific factors. The correlations between the testlet‐specific factors are modeled through a second‐order factor. The second‐order model is formally equivalent to the testlet model. The models were applied to a testlet‐based international English assessment test.
METHODS OF LINKING WITH SMALL SAMPLES IN A COMMON‐ITEM DESIGN: AN EMPIRICAL COMPARISON
Authors: Kim Sooyeon , Livingston Samuel A.
Published: August 2014

A series of resampling studies was conducted to compare the accuracy of equating in a common‐item design using four different methods: chained equipercentile equating of smoothed distributions, chained linear equating, chained mean equating, and the circle‐arc method. Four operational test forms, each containing more than 100 items, were used for the equating, with new‐form samples of 100, 50, 25, and 10 examinees and reference‐form samples three times as large. Accuracy was assessed in terms of the root‐mean‐squared difference (over 1,000 replications) of the sample equatings from the criterion equating. Overall, chained mean equating produced the most accurate results for low scores, but the circle‐arc method produced the most accurate results, particularly in the upper half of the score distribution.
ALTERNATIVE LOGLINEAR SMOOTHING MODELS AND THEIR EFFECT ON EQUATING FUNCTION ACCURACY
Authors: Moses Tim , Holland Paul
Published: August 2014

This simulation study evaluated the potential of alternative loglinear smoothing strategies for improving equipercentile equating function accuracy. These alternative strategies use cues from the sample data to make automatable and efficient improvements to model fit, either through the use of indicator functions for fitting large residuals or by averaging raw and smoothed frequencies. The strategies were studied across equating conditions based on rights‐scored and formula‐scored test data. Sample sizes were also manipulated. The results showed that the considered strategies produced equating functions with improved on‐average accuracy but with added random variability. Of the considered alternative strategies, the frequency averaging strategy produced the most accurate equating functions for most of the evaluations done in the study. The frequency averaging strategy is recommended for circumstances where the desired loglinear model appears to fit the data poorly and where time constraints and/or data conditions make traditional modeling approaches unrealistic.
ASSESSING FIT OF LATENT REGRESSION MODELS
Authors: Sinharay Sandip , Guo Zhumei , von Davier Matthias , Veldkamp Bernard P.
Published: August 2014

The reporting methods used in large‐scale educational assessments such as the National Assessment of Educational Progress (NAEP) rely on a latent regression model. There is a lack of research on the assessment of fit of latent regression models. This paper suggests a simulation‐based model‐fit technique to assess the fit of such models. The technique consists of investigating whether basic statistical summaries are predicted adequately by the latent regression model. Application of the suggested technique to an operational NAEP data set reveals important information regarding the fit of the latent regression model to the data.
AN EVIDENCE‐CENTERED APPROACH TO USING ASSESSMENT DATA FOR POLICYMAKERS
Authors: Underwood Jody S. , Zapata‐Rivera Diego , VanWinkle Waverely
Published: August 2014

District‐level policymakers are challenged to use evidence of student achievement to make policy decisions, such as professional development and other school improvement plans. They currently receive reports of student achievement data that are complex, difficult to read, and even harder to interpret. Using the research literature on policymakers' use of data and conducting focus groups and interviews, we elicited information on their roles and responsibilities, as well as questions these people would like to have answered from achievement data. We propose an evidence‐centered reporting framework to help policymakers determine which data they need, in order to design a series of reports that will answer their questions and to help them make sense of the data in support of policy decisions.
ALIGNING SCALES OF CERTIFICATION TESTS
Authors: Dorans Neil J. , Liang Longjuan , Puhan Gautam
Published: August 2014

Scores are the most visible and widely used products of a testing program. The choice of score scale has implications for test specifications, equating, and test reliability and validity, as well as for test interpretation. At the same time, the score scale should be viewed as infrastructure likely to require repair at some point. In this report we examine the issue of scale fit—how well the scale fits the intended uses of its scores—for certification tests. Two examples of scale fit are considered: one in which the test has a single threshold that separates the candidate population into pass‐fail groups, and one in which the test is required to support a restricted range of multiple thresholds.
LIMITS ON THE ACCURACY OF LINKING
Authors: Haberman Shelby J.
Published: August 2014

Sampling errors limit the accuracy with which forms can be linked. Limitations on accuracy are especially important in testing programs in which a very large number of forms are employed. Standard inequalities in mathematical statistics may be used to establish lower bounds on the achievable inking accuracy. To illustrate results, a variety of equating problems are considered.
MEASURING MULTIDIMENSIONAL LATENT GROWTH
Authors: Rijmen Frank
Published: August 2014

As is the case for any statistical model, a multidimensional latent growth model comes with certain requirements with respect to the data collection design. In order to measure growth, repeated measurements of the same set of individuals are required. Furthermore, the data collection design should be specified such that no individual is given the same item twice, while at the same time allowing for common items over time so that different measurement occasions can be linked. An example of such a data collection design is presented.

A computational challenge arises due to the high dimensional nature of a multidimensional latent growth model. Not only are there multiple dimensions within each measurement occasion, but insofar not all individuals change at the same rate for a given construct, that construct will also give rise to multiple dimensions over time. Fortunately, the computational burden can be overcome insofar as one is willing to incorporate assumptions on the latent structure, such as a bifactor or higher order structure within measurement occasions, and the assumption that the construct at a particular time point is independent of the construct at all previous time points given the construct at the immediately preceding time point (first order Markov assumption).
VARIANCE ESTIMATION FOR NAEP DATA USING A RESAMPLING‐BASED APPROACH: AN APPLICATION OF COGNITIVE DIAGNOSTIC MODELS
Authors: Hsieh Chueh‐an , Xu Xueli , von Davier Matthias
Published: August 2014

This paper presents an application of a jackknifing approach to variance estimation of ability inferences for groups of students, using a multidimensional discrete model for item response data. The data utilized to demonstrate the approach come from the National Assessment of Educational Progress (NAEP). In contrast to the operational approach used in NAEP, where plausible values are used to make ability inferences, the approach presented in this paper reestimates all parameters of the model, and makes ability inferences based on replicate samples of the jackknife without using plausible values.

Results of the standard errors are presented for estimates of group means, total means, and other statistics used in official reporting by NAEP. Differences in results between this approach and the operational approach are discussed.
GENERATING AUTOMATED TEXT COMPLEXITY CLASSIFICATIONS THAT ARE ALIGNED WITH TARGETED TEXT COMPLEXITY STANDARDS
Authors: Sheehan Kathleen M. , Kostin Irene , Futagi Yoko , Flor Michael
Published: August 2014

The Common Core Standards call for students to be exposed to a much greater level of text complexity than has been the norm in schools for the past forty years. Textbook publishers, teachers, and assessment developers are being asked to refocus materials and methods to ensure that students are challenged to read texts at steadily increasing complexity levels as they progress through school so that all students remain on track to achieve college and career readiness by the end of 12th grade. Although automated text analysis tools have been proposed as one method for helping educators achieve this goal, research suggests that existing tools are subject to three limitations: inadequate construct coverage; overly narrow criterion variables; and inappropriate treatment of genre effects. Modeling approaches developed to address these limitations are described. Recommended approaches are incorporated into a new text analysis system called SourceRater. Validity analyses implemented on an independent sample of texts suggest that, compared to existing approaches, SourceRater's estimates of text complexity are more reflective of the complexity classifications given in the new Standards. Implications for the development of learning progressions designed to help educators organize curriculum, instruction and assessment in reading are discussed.
EQUATING SUBSCORES USING TOTAL SCALED SCORES AS AN ANCHOR
Authors: Puhan Gautam , Liang Longjuan
Published: August 2014

Because the demand for subscores is ever increasing, this study examined two different approaches for equating subscores: (a) equating a subscore on the new form to the same subscore in the old form using internal common items as the anchor to conduct the equating, and (b) equating a subscore on the new form to the same subscore in the old form using equated total scores as the anchor to conduct the equating. Equated total scores can be used as an anchor to equate the subscores because the total equated scores are comparable across both the new and the old forms. Data from 2 tests (Tests X and Y) were used to conduct the study, and results showed that when the number of internal common items was large (approximately 50% of the total subscore), then using common items to equate the subscores was preferable. However, when the number of common items was small (approximately 25% of the total subscore, which is common practice), then using total scaled scores (TSS) to equate the subscores was preferable. Using raw subscores (not equating) resulted in a considerable amount of bias for both tests.
THE ROLE OF NONCOGNITIVE CONSTRUCTS AND OTHER BACKGROUND VARIABLES IN GRADUATE EDUCATION
Authors: Kyllonen Patrick C. , Walters Alyssa M. , Kaufman James C.
Published: August 2014

This report reviews the literature on noncognitive and other background predictors (e.g., personality, attitudes, and interests) as it pertains to graduate education. The first section reviews measures typically used in studies of graduate school outcomes, such as attrition and time to degree. A review of qualities faculty members and administrators say they desire and cultivate in graduate programs is conducted. There appears to be a divergence between the qualities faculty members say are important and the measures researchers typically use in validity studies.

The second section reviews three categories of noncognitive variables that might predict outcomes (general personality factors, quasi‐cognitive factors, and attitudinal factors) and the definitions, measures, correlates, and the validity of those measures. The role of background factors, both environmental and group, is part of this review.
EXAMINING AMERICAN POST‐SECONDARY EDUCATION
Authors: Liu Ou Lydia
Published: August 2014

The purpose of this report is to identify the most prominent issues in U.S. higher education and to develop strategic research plans to address the issues that are most relevant to ETS's capabilities in measurement and assessment through the ETS's higher education research initiative. In the United States, issues related to higher education such as improved performance and effective accountability have received unprecedented attention from stakeholders at many levels. At the national level, President Obama has set forth an ambitious agenda for American postsecondary education such that by 2020, the United States should once again have the largest concentration of citizens with a postsecondary degree. At the corporate level, ETS, as the world's largest educational research and testing organization, is ready to move beyond testing program‐based research in higher education and has the capability to deal with some of the most thorny issues in higher education. By strategically expanding post‐secondary research, ETS will establish itself as a pioneer and thought leader in the field of higher education.

The first part of the research report identifies four key issues existing in American higher education: enrollment and performance, retention and degree attainment, student learning and experience, and learning outcomes and accountability. The second part of the research report develops an ETS research agenda with short‐term and long‐term plans to address these issues. The agenda specifies short‐term and long‐term research goals that are specific, attainable, and measurable. Research findings from the studies proposed in this agenda have a potential for advancing understanding of the current situation and future needs of American higher education and also contributing to enhanced student learning at postsecondary institutions. Reaffirming and strengthening American higher education is critical to this country's success in the 21st century.
MEASUREMENT ERROR IN NONPARAMETRIC ITEM RESPONSE CURVE ESTIMATION
Authors: Guo Hongwen , Sinharay Sandip
Published: August 2014

Nonparametric, or kernel, estimation of item response curve (IRC) is a concern theoretically and operationally. Accuracy of this estimation, often used in item analysis in testing programs, is biased when the observed scores are used as the regressor because the observed scores are contaminated by measurement error. In this study, we investigate the deconvolution kernel estimation of IRC, correcting for the measurement error in the regressor variable. Using item response theory (IRT) simulated data and some real data, we compared the traditional kernel estimation and the deconvolution estimation of IRC. Results show that in capturing important features of the IRC, the traditional kernel estimation is comparable to the deconvolution kernel estimation in item analysis.
RECOMMENDATIONS FOR CONDUCTING DIFFERENTIAL ITEM FUNCTIONING (DIF) ANALYSES FOR STUDENTS WITH DISABILITIES BASED ON PREVIOUS DIF STUDIES
Authors: Buzick Heather , Stone Elizabeth
Published: August 2014

The purpose of this study is to help ensure that strategies for differential item functioning (DIF) detection for students with disabilities are appropriate and lead to meaningful results. We surveyed existing DIF studies for students with disabilities and describe them in terms of study design, statistical approach, sample characteristics, and DIF results. Based on descriptive and graphical summaries of previous DIF studies, we make recommendations for future studies of DIF for students with disabilities.
EXAMINING THE FACTOR STRUCTURE OF A STATE STANDARDS‐BASED SCIENCE ASSESSMENT FOR STUDENTS WITH LEARNING DISABILITIES
Authors: Steinberg Jonathan , Cline Frederick , Sawaki Yasuyo
Published: August 2014

This study examined the scores on a state standards‐based Grade 5 Science assessment obtained by a group of students without learning disabilities who took the standard form of the test and by three groups of students with learning disabilities: one taking the standard form of the test without accommodations or modifications, a second taking the test with accommodations, and a third group taking the test with modifications. The groups received accommodations or modifications that were specified in their 504 or IEP plans. Unlimited time was granted to complete the test. A series of item‐level, then parcel‐level, exploratory and confirmatory factor analyses investigated whether or not the assessment demonstrated factorial invariance for the four groups of students studied. The results of this study help substantiate the validity of test scores for students with disabilities who take the test with the particular set of accommodations or modifications that were used in this study. In addition, the results lend support to modifying this state's policy in order to aggregate scores obtained by students without learning disabilities and by students with learning disabilities who have taken the assessment with accommodations or modifications required by their 504 plans or IEPs for AYP purposes.
EVALUATION OF THE E‐RATER® SCORING ENGINE FOR THE TOEFL® INDEPENDENT AND INTEGRATED PROMPTS
Authors: Ramineni Chaitanya , Trapani Catherine S. , Williamson David M. , Davey Tim , Bridgeman Brent
Published: August 2014

Scoring models for the e‐rater® system were built and evaluated for the TOEFL® exam's independent and integrated writing prompts. Prompt‐specific and generic scoring models were built, and evaluation statistics, such as weighted kappas, Pearson correlations, standardized differences in mean scores, and correlations with external measures, were examined to evaluate the e‐rater model performance against human scores. Performance was also evaluated across different demographic subgroups. Additional analyses were performed to establish appropriate agreement thresholds between human and e‐rater scores for unusual essays and the impact of using e‐rater on operational scores. Generic e‐rater scoring models were recommended for operational use for both independent and integrated writing tasks. The two automated scoring models were recommended for operational use to produce contributory scores within a discrepancy threshold of 1.5 and 1.0 with a human score for independent and integrated prompts respectively.
ASSESSING QUALITY IN THE TEACHING OF CONTENT TO ENGLISH LANGUAGE LEARNERS
Authors: Turkan Sultan , Croft Andrew , Bicknell Jerome , Barnes Andwatta
Published: August 2014

This paper offers a chain of evidence collected to identify and verify the domain of teacher knowledge and skills required to teach ELLs content effectively in mathematics, science, reading/English language arts, and social studies. We initially assessed this domain through a review of literature and state standards for nonspecialized teacher certification. We developed a set of 67 evidence statements in 2 categories: pedagogical knowledge and linguistic knowledge. We conducted a national survey of practitioners and teacher educators to validate these statements, receiving 269 responses. A panel of 14 teacher educators and teachers further validated the statements by reaching consensus that the statements support the claims of the assessment under development. We found that the domain of knowledge necessary to teach ELLs in the content areas is insufficiently defined in the standards and literature. However, we identified that across the content areas, teachers should have knowledge and understanding of the register of the particular discipline. Beyond merely understanding the linguistic demands of a given discipline, teachers should also be able to raise ELLs' metalinguistic awareness of effective oral and written communication in the language of the content area.

Also, teachers' use of multimodal instructional materials was deemed commonly essential in assessing teacher knowledge. Finally, we acknowledge that the chain of evidence collected in this line of research is a small step taken forward to understanding what should be assessed and how. With that, we offer future directions for research.
THE STABILITY OF THE SCORE SCALES FOR THE SAT REASONING TEST™ FROM 2005 TO 2010
Authors: Guo Hongwen , Liu Jinghua , Curley Edward , Dorans Neil
Published: August 2014

This study examines the stability of the SAT Reasoning Test™ score scales from 2005 to 2010. A 2005 old form (OF) was administered along with a 2010 new form (NF). A new conversion for OF was derived through direct equipercentile equating. A comparison of the newly derived and the original OF conversions showed that Critical Reading and Mathematics score scales have experienced, at most, a moderate upward scale drift (no greater than 5 points on average), and the drift may be explained by an accumulation of random equating errors. The Writing score scale has experienced a significant upward scale drift (11 points on average), which may be caused by sources other than random equating errors.
A PRELIMINARY ANALYSIS OF KEYSTROKE LOG DATA FROM A TIMED WRITING TASK
Authors: Almond Russell , Deane Paul , Quinlan Thomas , Wagner Michael , Sydorenko Tetyana
Published: August 2014

The Fall 2007 and Spring 2008 pilot tests for theCBAL™ Writing assessment included experimental keystroke logging capabilities. This report documents the approaches used to capture the keystroke logs and the algorithms used to process the outputs. It also includes some preliminary findings based on the pilot data. In particular, it notes that the distribution of most of the pause length is consistent with data generated from a mixture of lognormal distributions. This corresponds to a cognitive model in which some pauses are merely part of the transcription (i.e., typing) process and some are part of more involved cognitive process (e.g., attention to writing conventions, word choice, and planning). In the pilot data, many of the features extracted from the keystroke logs were correlated with human scores. Due to the small sample sizes of the pilot studies, these findings are suggestive, not conclusive; however, they suggest a line of analysis for a large sample containing keystroke logging gathered in the fall of 2009.
CHOICE OF TARGET POPULATION WEIGHTS IN RATER COMPARABILITY SCORING AND EQUATING
Authors: Puhan Gautam
Published: August 2014

The purpose of this study was to demonstrate that the choice of sample weights when defining the target population under poststratification equating can be a critical factor in determining the accuracy of the equating results under a unique equating scenario, known as rater comparability scoring and equating. The nature of data collection under rater comparability scoring is such that it results in a very high correlation between the anchor and total score in the new form but only a moderate correlation in the reference form. I demonstrated, using data collected under a rater comparability scoring situation, that this difference in the anchor‐total correlation in the new and reference forms can have a predictable impact on the equating results based on different sample weights (i.e., the equating results are most accurate when the reference form sample is defined as the target population, least accurate when the new form sample is defined as the target population, and somewhere in the middle when the new and reference form samples are equally weighed when defining the target population).
CONTRIBUTIONS OF A NONPROFIT EDUCATIONAL MEASUREMENT ORGANIZATION TO EDUCATION POLICY RESEARCH
Authors: Coley Richard J. , Goertz Margaret E. , Wilder Gita Z.
Published: August 2014

As part of its mission, Educational Testing Service (ETS) has conducted education policy research focused on promoting equal opportunity for all individuals from early childhood through adulthood. This report provides a sampler of this extensive work in three areas chosen to represent major historical foci:

• Analyzing, evaluating, and informing public policy in educational governance, including school finance; teacher policy; and federal, state, and local education policy;

• Examining differential access to educational opportunity in three areas of longstanding interest to ETS: the gender gap, advanced placement programs, and graduate education; and

• Reporting on the educational outcomes of the U.S. population and describing the contexts for these outcomes and for the gaps in outcomes that exist among segments of the population.
SARA READING COMPONENTS TESTS, RISE FORM: TEST DESIGN AND TECHNICAL ADEQUACY
Authors: Sabatini John , Bruce Kelly , Steinberg Jonathan
Published: August 2014

This paper describes the design, development, and technical adequacy of the Reading Inventory and Student Evaluation (RISE) form of the Study Aid and Reading Assessment (SARA) reading components computer‐delivered assessment. The RISE form, designed for middle‐school students, began through a joint project between the Strategic Education Research Partnership (SERP), ETS, and a large urban school district, where middle‐school literacy had been identified as an area needing improvement. Educators were interested in understanding more about the component skills of their middle‐school students, particularly their struggling readers, using an efficient and reliable assessment. To date, results from our piloting of the RISE form with middle‐school students have established its technical adequacy. In the future, we plan to expand the RISE to create parallel forms for 6th‐to‐8th graders as well as to explore development of new test forms at other points in the grade continuum.
DEVELOPING ANALYTIC RATING GUIDES FOR TOEFL IBT'S INTEGRATED SPEAKING TASKS
Authors: Jamieson Joan , Poonpon Kornwipa
Published: August 2014

Research and development of a new type of scoring rubric for the integrated speaking tasks of TOEFL iBT® are described. These analytic rating guides could be helpful if tasks modeled after those in TOEFL iBT were used for formative assessment, a purpose which is different from TOEFL iBT's primary use for admission decisions. Two questions motivated the project: What can be done to make the criteria and standards for good performance clear not only to English language teachers and their students but also to novice raters? How can test‐takers be guided in the steps necessary to improve performance? Previous research, quantitative results of linguistic features present in spoken responses, and qualitative themes of raters' judgments were analyzed. Salient features associated with performance were then synthesized to develop three analytic rating guides, each using a series of yes/no questions. The rating guides expanded the current scoring scales for the three dimensions of delivery, language use, and topic development, so that key features of increasingly proficient speaking performance were described in more detail. Suggestions for future validation studies are provided in terms of an iterative process of trials and revisions.