bluuebunny/crossref_metadata_2025_split huggingface dataset:
EFFECT OF REPEATERS ON SCORE EQUATING IN A LARGE‐SCALE LICENSURE TEST
Authors: Kim Sooyeon , Walker Michael E. 
Published: August 2014

ABSTRACTThis study investigated the subgroup invariance of equating functions for a licensure test in the context of a nonequivalent groups with anchor test (NEAT) design. Examinees who had taken a new, to‐be‐equated form of the test were divided into three subgroups according to their previous testing experience: (a) repeaters who previously took the reference form, to which the new form would be equated; (b) repeaters who previously took any form other than the reference form; and (c) first‐time test takers for whom the new form was the first exposure to the test. Equating functions obtained with two subgroups, all repeaters, and first‐time test takers were similar to those obtained with the total group, supporting score equatability of the two forms. However, when the repeater subgroup was divided into two groups based on the particular form examinees took previously, subgroup equating functions substantially differed from the total‐group equating function, indicating subgroup dependency of score equating. The results indicate that repeater membership needs to be more clearly specified to assess the impact of repeaters on score equating. Such clarification may be especially necessary for high‐stakes licensure tests because repeaters tend to perform more poorly on such tests than first‐time test takers.
USING THE GENERAL DIAGNOSTIC MODEL TO MEASURE LEARNING AND CHANGE IN A LONGITUDINAL LARGE‐SCALE ASSESSMENT
Authors: von Davier Matthias , Xu Xueli , Carstensen Claus H. 
Published: August 2014

ABSTRACTA general diagnostic model was used to specify and compare two multidimensional item‐response‐theory (MIRT) models for longitudinal data: (a) a model that handles repeated measurements as multiple, correlated variables over time (Andersen, 1985) and (b) a model that assumes one common variable over time and additional orthogonal variables that quantify the change (Embretson, 1991). Using MIRT‐model ability distributions that we allowed to vary across subpopulations defined by type of school, we also compared (a) a model with a single two‐dimensional ability distribution to (b) extensions of the Andersen and Embretson approaches, assuming multiple populations. In addition, we specified a hierarchical‐mixture distribution variant of the (Andersen and Embretson) MIRT models and compared it to all four of the above alternatives. These four types of models are growth‐mixture models that allow for variation of the mixing proportions across clusters in a hierarchically organized sample. To illustrate the models presented in this paper, we applied the models to the PISA‐I‐Plus data for assessing learning and change across multiple subpopulations. The results indicate that (a) the Embretson‐type model with multiple‐group assumptions fits the data better than the other models investigated, and (b) the higher performing group shows larger improvement at Time Point 2 than the lower performing group.
DOES CONTENT KNOWLEDGE AFFECT TOEFL IBT™ READING PERFORMANCE? A CONFIRMATORY APPROACH TO DIFFERENTIAL ITEM FUNCTIONING
Authors: Liu Ou Lydia , Schedl Mary , Malloy Jeanne , Kong Nan 
Published: August 2014

ABSTRACTThe TOEFL iBT™ has increased the length of the reading passages in the reading section compared to the passages on the TOEFL® computer‐based test (CBT) to better approximate academic reading in North American universities, resulting in a reduced number of passages in the reading test. A concern arising from this change is whether the decrease in topic variety increases the likelihood that an examinee's familiarity with the particular content of a given passage will influence the examinee's reading performance. This study investigated differential item functioning and differential bundle functioning for six TOEFL iBT reading passages, three involving physical science and three involving cultural topics. The majority of items displayed little or no differential item functioning (DIF). When all of the items in a passage were examined, none of the passages showed differential functioning at the passage level. Hypotheses are provided for the DIF occurrences. Implications for fairness issues in test development are also discussed.
THE SPEAKING SECTION OF THE TOEFL IBT™ (SSTIBT): TEST‐TAKERS' REPORTED STRATEGIC BEHAVIORS
Authors: Swain Merrill , Huang Li‐Shih , Barkaoui Khaled , Brooks Lindsay , Lapkin Sharon 
Published: August 2014

ABSTRACTThis study responds to the Test of English as a Foreign Language™ (TOEFL®) research agenda concerning the need to understand the processes and knowledge that test‐takers utilize. Specifically, it investigates the strategic behaviors test‐takers reported using when taking the Speaking section of the TOEFL iBT™ (SSTiBT). It also investigates how the reported strategic behaviors differed across integrated and independent tasks in the SSTiBT, as well as the relationship between test‐takers' reported strategic behaviors and their performance on the tasks as determined by their test scores. The participating students were 14 graduate and 16 undergraduate engineering students whose first language was Chinese.The results indicate that test‐takers reported using 49 separate strategies when completing the SSTiBT tasks. Of the five strategy categories, the metacognitive, communication, and cognitive strategies were proportionally the most frequently reported. The interrelationships among these three categories were negative. Undergraduates reported using significantly more communication strategies, whereas graduates reported using significantly more cognitive and affective strategies. No statistically significant differences were found in reported strategy use across proficiency levels. The integrated tasks were more alike with respect to reported strategy use than were the independent and integrated tasks. Furthermore, the integrated tasks elicited a wider variety of reported strategy use than the independent tasks. Overall, we found no relationship between the total number of reported strategic behaviors and total test score on the SSTiBT.We conclude that strategy use is integral to performing SSTiBT tasks and should therefore be considered as part of the construct of communicative performance. However, the relationship between strategy use and test performance is varied and is due to complex interactions among test‐taker characteristics, tasks, and contexts.
STRATEGIES FOR TESTING SLOPE DIFFERENCES
Authors: Moses Tim , Klockars Alan 
Published: August 2014

ABSTRACTThe robustness and power of 9 strategies for testing the differences in groups' regression slopes were assessed under nonnormality and residual variance heterogeneity. For the conditions considered, the most robust strategies were the trimmed and Winsorized slope estimates used with the James second‐order test, the Theil‐Sen slope estimates used with James, and the Theil‐Sen estimates used with percentile bootstrapping. The use of Theil‐Sen slope estimates produced more powerful tests than the use of trimmed and Winsorized slopes.
EVALUATING SUBPOPULATION INVARIANCE OF LINKING FUNCTIONS TO DETERMINE THE ANCHOR COMPOSITION FOR A MIXED‐FORMAT TEST
Authors: Kim Sooyeon , Walker Michael E. 
Published: August 2014

ABSTRACTWe examined the appropriateness of the anchor composition in a mixed‐format test, which includes both multiple‐choice (MC) and constructed‐response (CR) items, using subpopulation invariance indices. We derived linking functions in the nonequivalent groups with anchor test (NEAT) design using two types of anchor sets: (a) MC only and (b) a mix of MC and CR. In each anchor condition, we derived the linking functions separately for males and females and then compared those subpopulation functions to the total group function. In the MC‐only condition, the difference between the subpopulation functions and the total group function was not trivial in a score region that included cut scores, leading to inconsistent pass/fail decisions for low‐performing examinees in particular. Overall, the mixed anchor was a better choice than the MC‐only anchor to achieve subpopulation invariance between males and females. The research reinforces subpopulation invariance indices as a means of determining the adequacy of the anchor.
THREE MULTIDIMENSIONAL MODELS FOR TESTLET‐BASED TESTS: FORMAL RELATIONS AND AN EMPIRICAL COMPARISON
Authors: Rijmen Frank 
Published: August 2014

ABSTRACTThree multidimensional item response theory (IRT) models for testlet‐based tests are described. In the bifactor model (Gibbons & Hedeker, 1992), each item measures a general dimension in addition to a testlet‐specific dimension. The testlet model (Bradlow, Wainer, & Wang, 1999) is a bifactor model in which the loadings on the specific dimensions are restricted to be proportional to the loadings on the general dimension within each testlet (Li, Bolt, & Fu, 2006). In the second‐order model, the items only load on the testlet‐specific factors. The correlations between the testlet‐specific factors are modeled through a second‐order factor. The second‐order model is formally equivalent to the testlet model. The models were applied to a testlet‐based international English assessment test.
METHODS OF LINKING WITH SMALL SAMPLES IN A COMMON‐ITEM DESIGN: AN EMPIRICAL COMPARISON
Authors: Kim Sooyeon , Livingston Samuel A. 
Published: August 2014

ABSTRACTA series of resampling studies was conducted to compare the accuracy of equating in a common‐item design using four different methods: chained equipercentile equating of smoothed distributions, chained linear equating, chained mean equating, and the circle‐arc method. Four operational test forms, each containing more than 100 items, were used for the equating, with new‐form samples of 100, 50, 25, and 10 examinees and reference‐form samples three times as large. Accuracy was assessed in terms of the root‐mean‐squared difference (over 1,000 replications) of the sample equatings from the criterion equating. Overall, chained mean equating produced the most accurate results for low scores, but the circle‐arc method produced the most accurate results, particularly in the upper half of the score distribution.
ALTERNATIVE LOGLINEAR SMOOTHING MODELS AND THEIR EFFECT ON EQUATING FUNCTION ACCURACY
Authors: Moses Tim , Holland Paul 
Published: August 2014

ABSTRACTThis simulation study evaluated the potential of alternative loglinear smoothing strategies for improving equipercentile equating function accuracy. These alternative strategies use cues from the sample data to make automatable and efficient improvements to model fit, either through the use of indicator functions for fitting large residuals or by averaging raw and smoothed frequencies. The strategies were studied across equating conditions based on rights‐scored and formula‐scored test data. Sample sizes were also manipulated. The results showed that the considered strategies produced equating functions with improved on‐average accuracy but with added random variability. Of the considered alternative strategies, the frequency averaging strategy produced the most accurate equating functions for most of the evaluations done in the study. The frequency averaging strategy is recommended for circumstances where the desired loglinear model appears to fit the data poorly and where time constraints and/or data conditions make traditional modeling approaches unrealistic.
ASSESSING FIT OF LATENT REGRESSION MODELS
Authors: Sinharay Sandip , Guo Zhumei , von Davier Matthias , Veldkamp Bernard P. 
Published: August 2014

ABSTRACTThe reporting methods used in large‐scale educational assessments such as the National Assessment of Educational Progress (NAEP) rely on a latent regression model. There is a lack of research on the assessment of fit of latent regression models. This paper suggests a simulation‐based model‐fit technique to assess the fit of such models. The technique consists of investigating whether basic statistical summaries are predicted adequately by the latent regression model. Application of the suggested technique to an operational NAEP data set reveals important information regarding the fit of the latent regression model to the data.
AN EVIDENCE‐CENTERED APPROACH TO USING ASSESSMENT DATA FOR POLICYMAKERS
Authors: Underwood Jody S. , Zapata‐Rivera Diego , VanWinkle Waverely 
Published: August 2014

ABSTRACTDistrict‐level policymakers are challenged to use evidence of student achievement to make policy decisions, such as professional development and other school improvement plans. They currently receive reports of student achievement data that are complex, difficult to read, and even harder to interpret. Using the research literature on policymakers' use of data and conducting focus groups and interviews, we elicited information on their roles and responsibilities, as well as questions these people would like to have answered from achievement data. We propose an evidence‐centered reporting framework to help policymakers determine which data they need, in order to design a series of reports that will answer their questions and to help them make sense of the data in support of policy decisions.
ALIGNING SCALES OF CERTIFICATION TESTS
Authors: Dorans Neil J. , Liang Longjuan , Puhan Gautam 
Published: August 2014

ABSTRACTScores are the most visible and widely used products of a testing program. The choice of score scale has implications for test specifications, equating, and test reliability and validity, as well as for test interpretation. At the same time, the score scale should be viewed as infrastructure likely to require repair at some point. In this report we examine the issue of scale fit—how well the scale fits the intended uses of its scores—for certification tests. Two examples of scale fit are considered: one in which the test has a single threshold that separates the candidate population into pass‐fail groups, and one in which the test is required to support a restricted range of multiple thresholds.
LIMITS ON THE ACCURACY OF LINKING
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTSampling errors limit the accuracy with which forms can be linked. Limitations on accuracy are especially important in testing programs in which a very large number of forms are employed. Standard inequalities in mathematical statistics may be used to establish lower bounds on the achievable inking accuracy. To illustrate results, a variety of equating problems are considered.
MEASURING MULTIDIMENSIONAL LATENT GROWTH
Authors: Rijmen Frank 
Published: August 2014

ABSTRACTAs is the case for any statistical model, a multidimensional latent growth model comes with certain requirements with respect to the data collection design. In order to measure growth, repeated measurements of the same set of individuals are required. Furthermore, the data collection design should be specified such that no individual is given the same item twice, while at the same time allowing for common items over time so that different measurement occasions can be linked. An example of such a data collection design is presented.A computational challenge arises due to the high dimensional nature of a multidimensional latent growth model. Not only are there multiple dimensions within each measurement occasion, but insofar not all individuals change at the same rate for a given construct, that construct will also give rise to multiple dimensions over time. Fortunately, the computational burden can be overcome insofar as one is willing to incorporate assumptions on the latent structure, such as a bifactor or higher order structure within measurement occasions, and the assumption that the construct at a particular time point is independent of the construct at all previous time points given the construct at the immediately preceding time point (first order Markov assumption).
VARIANCE ESTIMATION FOR NAEP DATA USING A RESAMPLING‐BASED APPROACH: AN APPLICATION OF COGNITIVE DIAGNOSTIC MODELS
Authors: Hsieh Chueh‐an , Xu Xueli , von Davier Matthias 
Published: August 2014

ABSTRACTThis paper presents an application of a jackknifing approach to variance estimation of ability inferences for groups of students, using a multidimensional discrete model for item response data. The data utilized to demonstrate the approach come from the National Assessment of Educational Progress (NAEP). In contrast to the operational approach used in NAEP, where plausible values are used to make ability inferences, the approach presented in this paper reestimates all parameters of the model, and makes ability inferences based on replicate samples of the jackknife without using plausible values.Results of the standard errors are presented for estimates of group means, total means, and other statistics used in official reporting by NAEP. Differences in results between this approach and the operational approach are discussed.
GENERATING AUTOMATED TEXT COMPLEXITY CLASSIFICATIONS THAT ARE ALIGNED WITH TARGETED TEXT COMPLEXITY STANDARDS
Authors: Sheehan Kathleen M. , Kostin Irene , Futagi Yoko , Flor Michael 
Published: August 2014

ABSTRACTThe Common Core Standards call for students to be exposed to a much greater level of text complexity than has been the norm in schools for the past forty years. Textbook publishers, teachers, and assessment developers are being asked to refocus materials and methods to ensure that students are challenged to read texts at steadily increasing complexity levels as they progress through school so that all students remain on track to achieve college and career readiness by the end of 12th grade. Although automated text analysis tools have been proposed as one method for helping educators achieve this goal, research suggests that existing tools are subject to three limitations: inadequate construct coverage; overly narrow criterion variables; and inappropriate treatment of genre effects. Modeling approaches developed to address these limitations are described. Recommended approaches are incorporated into a new text analysis system called SourceRater. Validity analyses implemented on an independent sample of texts suggest that, compared to existing approaches, SourceRater's estimates of text complexity are more reflective of the complexity classifications given in the new Standards. Implications for the development of learning progressions designed to help educators organize curriculum, instruction and assessment in reading are discussed.
EQUATING SUBSCORES USING TOTAL SCALED SCORES AS AN ANCHOR
Authors: Puhan Gautam , Liang Longjuan 
Published: August 2014

ABSTRACTBecause the demand for subscores is ever increasing, this study examined two different approaches for equating subscores: (a) equating a subscore on the new form to the same subscore in the old form using internal common items as the anchor to conduct the equating, and (b) equating a subscore on the new form to the same subscore in the old form using equated total scores as the anchor to conduct the equating. Equated total scores can be used as an anchor to equate the subscores because the total equated scores are comparable across both the new and the old forms. Data from 2 tests (Tests X and Y) were used to conduct the study, and results showed that when the number of internal common items was large (approximately 50% of the total subscore), then using common items to equate the subscores was preferable. However, when the number of common items was small (approximately 25% of the total subscore, which is common practice), then using total scaled scores (TSS) to equate the subscores was preferable. Using raw subscores (not equating) resulted in a considerable amount of bias for both tests.
THE ROLE OF NONCOGNITIVE CONSTRUCTS AND OTHER BACKGROUND VARIABLES IN GRADUATE EDUCATION
Authors: Kyllonen Patrick C. , Walters Alyssa M. , Kaufman James C. 
Published: August 2014

ABSTRACTThis report reviews the literature on noncognitive and other background predictors (e.g., personality, attitudes, and interests) as it pertains to graduate education. The first section reviews measures typically used in studies of graduate school outcomes, such as attrition and time to degree. A review of qualities faculty members and administrators say they desire and cultivate in graduate programs is conducted. There appears to be a divergence between the qualities faculty members say are important and the measures researchers typically use in validity studies.The second section reviews three categories of noncognitive variables that might predict outcomes (general personality factors, quasi‐cognitive factors, and attitudinal factors) and the definitions, measures, correlates, and the validity of those measures. The role of background factors, both environmental and group, is part of this review.
EXAMINING AMERICAN POST‐SECONDARY EDUCATION
Authors: Liu Ou Lydia 
Published: August 2014

ABSTRACTThe purpose of this report is to identify the most prominent issues in U.S. higher education and to develop strategic research plans to address the issues that are most relevant to ETS's capabilities in measurement and assessment through the ETS's higher education research initiative. In the United States, issues related to higher education such as improved performance and effective accountability have received unprecedented attention from stakeholders at many levels. At the national level, President Obama has set forth an ambitious agenda for American postsecondary education such that by 2020, the United States should once again have the largest concentration of citizens with a postsecondary degree. At the corporate level, ETS, as the world's largest educational research and testing organization, is ready to move beyond testing program‐based research in higher education and has the capability to deal with some of the most thorny issues in higher education. By strategically expanding post‐secondary research, ETS will establish itself as a pioneer and thought leader in the field of higher education.The first part of the research report identifies four key issues existing in American higher education: enrollment and performance, retention and degree attainment, student learning and experience, and learning outcomes and accountability. The second part of the research report develops an ETS research agenda with short‐term and long‐term plans to address these issues. The agenda specifies short‐term and long‐term research goals that are specific, attainable, and measurable. Research findings from the studies proposed in this agenda have a potential for advancing understanding of the current situation and future needs of American higher education and also contributing to enhanced student learning at postsecondary institutions. Reaffirming and strengthening American higher education is critical to this country's success in the 21st century.
MEASUREMENT ERROR IN NONPARAMETRIC ITEM RESPONSE CURVE ESTIMATION
Authors: Guo Hongwen , Sinharay Sandip 
Published: August 2014

ABSTRACTNonparametric, or kernel, estimation of item response curve (IRC) is a concern theoretically and operationally. Accuracy of this estimation, often used in item analysis in testing programs, is biased when the observed scores are used as the regressor because the observed scores are contaminated by measurement error. In this study, we investigate the deconvolution kernel estimation of IRC, correcting for the measurement error in the regressor variable. Using item response theory (IRT) simulated data and some real data, we compared the traditional kernel estimation and the deconvolution estimation of IRC. Results show that in capturing important features of the IRC, the traditional kernel estimation is comparable to the deconvolution kernel estimation in item analysis.