bluuebunny/crossref_metadata_2025_split huggingface dataset:
MULTIMETHOD CONSTRUCT VALIDATION OF THE TEST OF SPOKEN ENGLISH
Authors: Boldt R. F. , Oltman P. K. 
Published: August 2014

ABSTRACTAdministration of the Test of Spoken English (TSE) yields tapes of oral performance on items within six sections of the test. Trained scorers subsequently rate responses using four proficiency scales: pronunciation, grammar, fluency, and overall comprehensibility. This project examined the consistency of statistical relations among TSE scores with the measurement constructs these scores purport to reflect.Analyses included factor analysis and multidimensional scaling, which examined dimensions underlying the scores. These dimensional methods were applied to the 18 scores yielded by the combinations of section and scale. Another analysis was applied to scale scores averaged over sections. This analysis compared the ranking of pairs of scale scores obtained during the original scoring of selected taped performances with the ranking resulting from a rescoring by different raters.Multidimensional scaling analyses revealed that three‐dimensional solutions fit the scale intercorrelations with low stress values, and that the coordinates of the scales fell into three clusters in three‐dimensional space, those clusters being defined primarily by test section rather than by proficiency scales.An exploratory factor analysis revealed that a single dimension dominated the variation of the 18 section‐scale scores. However, when tapes for scale scores with substantial discrepancies were rerated, agreement between the order of original and rerated scale scores far exceeded chance. This indicated that raters were able to modify their judgements according to the scale being rated. Subsequent exploratory factor analysis indicated that both section and scale factors contribute to score variation. The factors were highly correlated, and the predominance of the single factor in the exploratory analysis was seen as arising from those correlations.
A TRANSPORTABILITY STUDY OF THE PRAXIS I: ACADEMIC SKILLS ASSESSMENTS COMPUTER‐BASED TEST (CBT) AND THE PRAXIS I: PRE‐PROFESSIONAL SKILLS TESTS (PPST) SPECIFICATION STATEMENTS FOR ENTRY‐LEVEL VOCATIONAL EDUCATION TEACHERS
Authors: Tannenbaum Richard J. , Rosenfeld Michael , Teryek Charles J. 
Published: August 2014

ABSTRACTThe purpose of this study was to determine if the 55 test specification statements for the Praxis I: Academic Skills Assessments computer‐based tests (CBT) and the Praxis I: Pre‐Professional Skills Tests (PPST) were judged to be appropriate for entry‐level vocational education teachers regardless of specialty area. The test specification statements were mailed to a random sample of 5,610 vocational education teachers representing 11 specialty areas. Useable surveys were returned by 2,347 vocational education teachers. Judgments of importance were made using a 5‐point rating scale. A test specification statement was considered to be appropriate for entry‐level vocational education teachers if respondents from at least 9 of 11 specialty areas judged it to have a mean importance greater than 3.00 (moderately important). The results indicated that all 55 test specifications received mean ratings above 3.00 by respondents from all 11 specialty areas. It may be concluded from the results that the test specification statements for the Praxis I: CBT and the Praxis I: PPST are appropriate for entry‐level vocational education teachers regardless of the specialty area in which they teach.
DEALING WITH UNCERTAINTY ABOUT ITEM PARAMETERS: EXPECTED RESPONSE FUNCTIONS
Authors: Mislevy Robert J. , Wingersky Marilyn S. , Sheehan Kathleen M. 
Published: August 2014

ABSTRACTIt is a common practice in item response theory (IRT) to treat estimates of item parameters, say B, as if they were the known, true quantities, B. However, ignoring the uncertainty associated with item parameters can lead to biases and over‐confidence in subsequent inferences such as ability estimation, especially when item‐calibration samples are small. This paper demonstrates how to incorporate uncertainty about B with Lewis's “expected response functions” (ERFs), pointwise expected values of item response conditional on examinee proficiency averaged over posterior distributions of item parameters. This paper presents ERFs, outlines procedures for computing them and using them in practical work, and gives an illustration with data from the National Assessment of Educational Progress. Advantages of approximating ERFs response curves with members of familiar parametric families of IRT curves are noted.
THE EFFECTS OF SAMPLE RESTRICTION ON GENDER DIFFERENCES
Authors: Lewis Charles , Willingham Warren W. 
Published: August 2014

ABSTRACTAs strongly suggested by recent work, patterns of gender difference can change because of changes in the selectivity of the sample itself. This is a statistical influence connected with the distributions of female and male scores, rather than a substantive influence related to demographic characteristics of the sample such as age or ethnicity. It is, nonetheless, an important influence because gender differences that are partly statistical in origin can easily confuse possible implications regarding education and assessment.This report proposes a general model to account for the effects of sample restriction on gender differences. Simulations showed the model to be quite accurate in reproducing standard mean differences and other statistics in a restricted sample. Three primary contributing factors were identified: the range‐restricting effects of sample selection, differential variability of female and male scores in the original sample, and the representation of females and males in the restricted sample. A test with actual data showed reasonably good consistency between trends predicted by the model and trends in gender differences that have been widely observed in advanced tests administered to select samples.
PRACTICAL ISSUES IN LARGE‐SCALE HIGH‐STAKES COMPUTERIZED ADAPTIVE TESTING
Authors: Mills Craig N. , Stocking Martha L. 
Published: August 2014

ABSTRACTComputerized adaptive testing (CAT), while well‐grounded in psychometric theory, has had few large‐scale applications for high‐stakes, secure tests in the past. This is now changing as the cost of computing has declined rapidly. As is always true at such junctures where theory is translated into practice, many practical issues arise that must now be addressed. This paper discusses a number of such issues and sketches out potential problems and potential solutions. Our purpose is to encourage further development of solutions to the issues presented as well as other practical issues facing measurement professionals involved with the implementation of adaptive testing.
A NEW METHOD OF CONTROLLING ITEM EXPOSURE IN COMPUTERIZED ADAPTIVE TESTING
Authors: Stocking Martha L. , Lewis Charles 
Published: August 2014

ABSTRACTIn the periodic testing environment associated with conventional paper‐and‐pencil tests, the frequency with which items are seen by test‐takers is tightly controlled in advance of testing by policies that regulate both the reuse of test forms and the frequency with which candidates may retake the test. In the continuous testing environment associated with more novel testing paradigms such as computerized adaptive testing (CAT), the computer itself can be used to control the frequency with which items are administered. This paper discusses previous methods of controlling item security in the continuous adaptive testing environment and presents a new method that overcomes some (but not all) of the disadvantages of previous attempts. An extensive example with this new method and a particular adaptive testing algorithm illustrates how concerns about test efficiency, parallelism, and security can be balanced.
ADJUSTMENT FOR READER RATING BEHAVIOR IN THE TEST OF WRITTEN ENGLISH
Authors: Longford Nicholas T. 
Published: August 2014

ABSTRACTThis report evaluates the impact of a potential scheme for score adjustment using data from the administrations of the Test of Written English in 1994. It is shown that, assuming non‐informative assignment of readers to essays, the adjustment due to reader differences reduces the mean squared error for all essays except those graded by readers with small workloads. The quality of the rating process, as described by the variances due to true scores, severity, and inconsistency, as well as the distribution of workloads, are similar across the administrations. This would make a reliable prediction of the optimal score adjustment in future administrations possible. Two approximations to the optimal adjustment are proposed.
REVISING ANSWERS TO ITEMS IN COMPUTERIZED ADAPTIVE TESTS: A COMPARISON OF THREE MODELS1,2
Authors: Stocking Martha L. 
Published: August 2014

ABSTRACTThe interest in the application of large‐scale computerized adaptive testing has served to focus attention on issues that arise when theoretical advances are made operational. Some of these issues stem less from changes in testing conditions and more from changes in testing paradigms. One such issue is that of the order in which questions are answered within a test or separately timed test section. In linear testing, this order of responses is entirely under the control of the test‐taker, who can omit questions, look ahead at questions, and return and revise answers to previous questions. The attempt to permit the same, or even reasonably restricted, control in adaptive testing can unintentionally result in transferring to the test‐taker control over which items are chosen for administration, threatening both test fairness and accuracy. This paper explores, using simulations, three models of permitting restricted test‐taker control over revising previous answers in the context of adaptive testing. Even under a worst‐case model of test‐taker revising behavior, two of the models of permitting item revisions work well in preserving test fairness and accuracy and one model studied may also preserve some cognitive processing styles developed by test‐takers for a linear testing environment.
MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING
Authors: Mislevy Robert J. , Wu Pao‐Kuei 
Published: August 2014

ABSTRACTThe basic equations of item response theory (IRT) provide a foundation for inferring examinees' abilities from responses to different test items. In practice, examinees do not generally provide a response to all items—for reasons that may or may not have been intended by the test administrator, and that may or may not be related to their ability. The mechanisms that produce missingness must be taken into account if correct inferences are to be drawn. Using concepts introduced by Rubin (1976), we discuss the implications for Bayesian and direct likelihood ability parameter estimation that are entailed by alternate test forms, targeted testing, adaptive testing, time limits, omitted responses, and examinee choice of tasks. Attention is focused on whether, in each case, the mechanism for missingness is “ignorable,” and, in those cases in which it is not, how it can be modeled.
HOW RELIABLE IS THE TOEFL TEST?
Authors: Wainer Howard , Lukhele Robert 
Published: August 2014

ABSTRACTWe estimated the reliability of scores on four forms of the Test of English as a Foreign Language (TOEFL®) using a hybrid IRT model. Very little difference between their overall reliability was found when the testlet items were assumed to be independent and their dependence was modeled. A larger difference in reliability was found when test sections were analyzed individually. Then we found as much as a 40 percent overestimate in reading comprehension testlets, with the longer testlets of the newest form of the TOEFL test showing the most local dependence. The listening comprehension testlets exhibited much less local dependence. We also found that the test was unidimensional enough for the use of univariate IRT to be efficacious, and that the reading comprehension testlets showed essentially no differential functioning by sex.
SPECULATIONS ON THE FUTURE OF LARGE‐SCALE EDUCATIONAL ASSESSMENT
Authors: Bennett Randy Elliot 
Published: August 2014

ABSTRACTIn this paper, I offer one scenario for the future of large‐scale educational assessment. I argue that, although large‐scale assessment has altered relatively little in recent years, this situation is about to change. The same competitive forces driving U.S. industry will compel test makers to (1) satisfy new market needs through continuous innovation, (2) improve productivity, (3) enhance customer‐service, and (4) address population diversity. In response, large‐scale assessment will reinvent itself to serve both summative and formative purposes, become curriculum‐embedded and performance‐based, occur at a distance, and incorporate constructs newly valued by society. This reinvention will be enabled chiefly by new technology and by advances in cognitive and measurement science.
TOWARD A COHERENT VIEW OF RELIABILITY IN TEST THEORY
Authors: Li Heng , Wainer Howard 
Published: August 2014

ABSTRACTReliability of test scores, as estimated through measures of internal consistency, has been characterized mathematically in many ways that appear, on the surface at least, to be very dissimilar to one another. In this essay we provide a general mathematical framework that specializes to four different reliability coefficients. Through consideration of this general framework it becomes easier to convey to students both the individual character of the different formulations of reliability and the extent of their underlying similarity. In addition to providing a coherent view of reliability, the unified formula is also found to be a convenient vehicle for introducing more specialized topics, such as the Kaiser‐Guttman rule.
THE RELATIONSHIP BETWEEN COMPUTER FAMILIARITY AND PERFORMANCE ON COMPUTER‐BASED TOEFL TEST TASKS
Authors: Taylor Carol , Jamieson Joan , Eignor Daniel , Kirsch Irwin 
Published: August 2014

ABSTRACTThe increasing use of computer‐based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 will confound language proficiency with computer proficiency and thus bring construct‐irrelevant variance to the measurement of examinees' English‐language abilities.In a Phase I study (Kirsch, Jamieson, Taylor, & Eignor, 1998), TOEFL examinees were surveyed regarding their computer familiarity and classified into one of three computer familiarity groups: low, moderate, and high. In this study, Phase II, more than 1,100 “low‐computer‐familiar” and “high‐computer‐familiar” examinees from 12 international sites were identified from the Phase I survey and administered a computer tutorial and a set of 60 computer‐based TOEFL test items. The relationship between level of computer familiarity and performance on the computer‐based items was then examined. The examinees in Phase II were largely representative of those in Phase I, who were representative of the general TOEFL test‐taking population. Thus, results from this phase of the study are considered generalizable to the current TOEFL examinee population.The effect of computer familiarity after adjustments for language ability was examined by performing a series of analyses of covariance (ANCOVAs), using TOEFL paper‐and‐pencil test score as the covariate. These analyses were followed by a series of ANCOVAs involving the computer familiarity variable and a number of other variables: gender, reason for taking the TOEFL test, times the TOEFL test had been taken, and location where the TOEFL test was taken. In a final set of analyses, the TOEFL paper‐and‐pencil test scores of the low‐ and high‐computer‐familiar examinees were weighted such that the groups had identical distributions on the covariate.After controlling for language ability, the researchers found no meaningful relationship between level of computer familiarity and level of performance on computerized language tasks among TOEFL examinees who had completed the computer tutorial. This finding was consistent for all but one of the subgroups considered. A small but practically significant interaction between computer familiarity and reason for taking the test was found on the set of computerized reading items. Researchers concluded that there was no evidence of adverse effects on the computer‐based TOEFL performance due to lack of prior computer experience.
COMPUTER ANALYSIS OF ESSAY CONTENT FOR AUTOMATED SCORE PREDICTION: A PROTOTYPE AUTOMATED SCORING SYSTEM FOR GMAT ANALYTICAL WRITING ASSESSMENT ESSAYS
Authors: Burstein Jill , Braden‐Harder Lisa , Chodorow Martin , Hua Shuyi , Kaplan Bruce , Kukich Karen , Lu Chi , Nolan James , Rock Don , Wolff Susanne 
Published: August 2014

ABSTRACTThis report discusses the development and evaluation of a research prototype system designed to automatically score essay responses to the GMAT Analytical Writing Assessments: (a). Analysis of an Argument (Argument essays) and (b). Analysis of an Issue (Issue essays) item types. The system, Electronic Essay Rater (e‐rater), was designed to automatically analyze several features of an essay and score the essay based on the features of writing as specified in holistic rubrics. E‐rater uses a hybrid feature methodology. It incorporates several variables that are derived statistically, extracted through NLP techniques, or achieved by simple “counting” procedures. The version of the e‐rater described in this report uses five sets of critical feature variables to build the final linear regression model used for predicting scores. The same set of critical variables was used to fit models for the issue and argument training essays and the following results were achieved. For the set of 275 cross‐validation data, exact or adjacent agreement with human rater scores reached 95%. For the 282 cross‐validation issue essays exact or adjacent agreement with human rater scores achieved 93%. The rich feature variables used as score predictors in e‐rater could potentially be used to generate explanation of score predictions, and diagnostic and instructional information.
DEVELOPMENT, SELECTION AND VALIDATION OF A SET OF COGNITIVE AND LINGUISTIC ATTRIBUTES FOR THE SAT I VERBAL: ANALOGY SECTION1
Authors: Buck Gary , VanEssen Tom , Tatsuoka Kikumi , Kostin Irene , Lutz Donna , Phelps Matthew 
Published: August 2014

ABSTRACTThe purpose of this study is to use the rule space procedure to develop a general set of cognitive and linguistic attributes that account for performance on the Analogy Section of the SAT I Verbal. The overall research strategy was to develop a set of cognitive and linguistic attributes on one form of the Analogy Section, the experimental form, and then to examine the generalizability of these by applying the same set of attributes to a different form of the test, the generalizability form. Results produced a set 13 cognitive and linguistic attributes, with 6 interaction attributes, that were able to classify 97% of test‐takers into their latent knowledge states on the experimental form; the same set of attributes were able to classify 90% of test takers into their latent knowledge states on the generalizability form. The analysis provides a strong indication of the important determinants of success on the SAT Analogy task, and provides evidence that these are relatively stable across forms.
GRE ANALYTICAL REASONING ITEM STATISTICS PREDICTION STUDY
Authors: Boldt R. F. 
Published: August 2014

ABSTRACTChalifour and Powers (1989) noted that the ability to predict item statistics might be used to reduce the volume of item pretesting (Mislevy, Sheehan, & Wingersky, 1993). It could also lead to better control of test statistics, such as item difficulty distributions, and to improved test specifications. Research, such as the previously cited study, is needed to examine the prospects for attaining these benefits. That study examined prediction of GRE analytical reasoning item statistics. Linear regression was used by these authors, but they surmised that non‐linear techniques might provide somewhat better prediction of item statistics.Chalifour and Powers had amassed item statistics on a very large sample of analytical reasoning items. That sample was used in the present study.For the present study, predictions were generated using a type of neural net, a technique that can accommodate a wide variety of non‐linear relationships, though at a cost of requiring the calculation of many constants in the prediction function. This technique did indeed provide more accurate predictions of item difficulty and item discrimination in an estimation sample. However, when the functions developed in the estimation sample were cross validated in a fresh sample, the advantages noted in the estimation sample disappeared. Only by including expert judgements of items difficulty with other predictors was the accuracy of prediction improved.The variables that produced the above results had been selected by Chalifour and Powers (1989) for their efficacy in linear prediction. The study reported here used a “genetic algorithm”, which provides a quasi‐random search of the predictor set to find an optimal set of predictors. The search used the validity of neural nets computed during operation of the algorithm to evaluate the efficacy of prediction. Hence, this technique sought variables sets that were optimal when a neural net was to be used.The genetic algorithm accomplished some improved prediction of item difficulties in the estimation, but no improvement with regard to discrimination. When the variable sets and the related nets were evaluated in the validation sample, any advantage gained by the search was lost. The validities of predictions in the validation sample were approximately equal whether or not the genetic algorithm had been used in developing the prediction functions.Examination of the root‐mean squares of the discrepancies between predicted and actual values for item difficulties (and discriminations) revealed no advantage for the neural net. This examination used the validation sample.In sum, application of the complex and computer‐intensive neural nets and genetic algorithms revealed no advantage over linear methods for predicting item difficulty and item discrimination statistics for GRE analytical reasoning items.