bluuebunny/crossref_metadata_2025_split huggingface dataset:
CONFIRMATORY FACTOR ANALYSIS OF THE TEST OF ENGLISH AS A FOREIGN LANGUAGE
Authors: Hale Gordon A. , Rock Donald A. , Jirele Thomas 
Published: August 2014

ABSTRACTIn a recent study, confirmatory factor analyses indicated that, for each of several language groups, TOEFL performance can be characterized by two factors, associated with (a) the Listening Comprehension section and (b) the other sections of the test (Hale, Stansfield, Rock, Hicks, Butler, & Oiler, 1988). This conclusion was inconsistent with that drawn in the most comprehensive previous factor‐analytic study of the TOEFL, which suggested a three‐factor solution for each of several languages (Swinton & Powers, 1980). The present study investigated the inconsistency in conclusions drawn from these two studies and provided further information about the factor structure of the TOEFL.It was hypothesized that the inconsistency between studies was related to the populations under investigation, as the earlier study used TOEFL examinees in both domestic and overseas test centers, whereas the more recent study used domestic examinees only. The present data did not support this hypothesis, however. Confirmatory factor analyses were conducted for each of several language groups, using data from the test form used in the more recent study (a 1984 TOEFL), and these analyses yielded essentially similar results for both domestic and overseas populations, as well as for the combined population. In all cases, the data supported a two‐factor interpretation, with the two factors related to the Listening Comprehension section and to the nonlistening sections.Other hypotheses were that the inconsistency between studies was due to differences in factor‐analytic methodologies used or to changes in the test over time. Confirmatory factor analyses were conducted using the data from the earlier study (taken from a 1976 TOEFL) as well as data from the more recent study. These analyses supported a two‐factor interpretation in both cases, for each of several language groups, with the two factors associated with Listening Comprehension and with the other sections of the test. Thus, the use of different factor‐analytic methodologies in the two previous studies undoubtedly contributed to the inconsistency, although further work would be needed to determine exactly what aspects of the methodology played a role. The data also provided tentative evidence that the basic factor structure of the test may not have changed substantially between 1976 and 1984.The role of examinee proficiency in determining the TOEFL's factor structure was also examined. For each of several language groups, low‐ and high‐proficiency groups were defined on the basis of approximately one third of the items in the TOEFL, drawn from all sections of the test, and factor analyses were then performed on an abbreviated TOEFL, which consisted of the remaining items. For both proficiency levels within each language group a two‐factor structure appeared to underlie performance, with the factors once again linked to Listening Comprehension and to the other, nonlistening sections of the test.
NONRESPONSE IN DECLARED ETHNICITY AND THE IDENTIFICATION OF DIFFERENTIALLY FUNCTIONING ITEMS
Authors: Allen Nancy L. , Wainer Howard 
Published: August 2014

ABSTRACTThe accuracy of procedures that are used to compare the performance of different groups of examinees on test items obviously depends upon the correct classification of members in each examinee group. The significance of this dependence is determined by the sensitivity of the statistical procedure and the proportion of examinees who are unidentified. Since the number of nonrespondents to questions of ethnicity is often of the same order of magnitude as the number of identified members of most minority groups, understanding the effect of nonresponse is crucial to evaluating the validity of a procedure which is used to study differential item functioning (DIF).In this study, we examined the effect of nonresponse to questions of ethnic identity on the measurement of DIF for SAT verbal items using a commonly used modern method, the Mantel‐Haenszel procedure (Holland and Thayer, 1988). We found that efforts to obtain more complete ethnic identifications from the examinees would be rewarded with more accurate DIF analyses.
DIFFERENTIAL TESTLET FUNCTIONING DEFINITIONS AND DETECTION
Authors: Wainer Howard , Sireci Stephen G. , Thissen David 
Published: August 2014

ABSTRACTIt is sometimes sensible to think of the fundamental unit of test construction as being larger than an individual item. This unit, dubbed the testlet, must pass muster in the same way that items do. One criterion of a good item is the absence of DIF—the item must function in the same way in all important subpopulations of examinees. In this paper we define what we mean by testlet DIF and provide a statistical methodology to detect it. This methodology parallels the IRT‐based likelihood ratio procedures explored previously by Thissen, Steinberg & Wainer (1988, in press). We illustrate this methodology with analyses of data from a testlet‐based experimental version of the Scholastic Aptitude Test.
A COMPARISON OF OPEN‐ENDED AND MULTIPLE‐CHOICE QUESTION FORMATS FOR THE QUANTITATIVE SECTION OF THE GRADUATE RECORD EXAMINATIONS GENERAL TEST
Authors: Bridgeman Brent 
Published: August 2014

ABSTRACTFree‐response counterparts to a set of items from the quantitative section of the Graduate Record Examinations General Test were developed. Examinees responded to these items by gridding numerical answers on a machine‐readable answer sheet in essentially the same way social security numbers are now gridded. In addition, the answer grid included special symbols for a negative sign (‐), a decimal point (.), a division sign (/), and a variable (k).The test section with the special answer sheet was administered at the end of a regular GRE administration. Test forms were spiraled so that random groups received either the grid‐in questions or the same questions in a multiple‐choice format. Because the amount of extra time required to grid in responses was unknown, both a long and a short form were developed, with students randomly receiving one or the other. Sample size for these comparisons was 12,465. In a separate data collection effort, 364 paid volunteers who had recently taken the General Test used a computer keyboard to enter answers to the same set of questions.Correct and incorrect answers chosen in the multiple‐choice format were compared with the answers generated in the free‐response format. Three‐parameter item characteristic curves were compared, as were correlational patterns with other test scores and undergraduate grade point average. Format by gender and format by ethnicity (Asian American, Black, Hispanic, and White) interactions were explored with analyses of covariance.Despite the format differences noted for individual items, total scores for the multiple‐choice and free‐response tests demonstrated remarkably similar correlational patterns, and there were no significant interactions of test format with either gender or ethnicity.
ITEM CONSTRUCTION AND PSYCHOMETRIC MODELS APPROPRIATE FOR CONSTRUCTED RESPONSES
Authors: Tatsuoka Kikumi K. 
Published: August 2014

ABSTRACTConstructed‐response formats are desired for measuring complex and dynamic response processes which require the examinee to understand the structures of problems and micro‐level cognitive tasks. These micro‐level tasks and their organized structures are usually unobservable. This study shows that elementary graph theory is useful for organizing these micro‐level tasks and for exploring their properties and relations. Moreover, this approach enables us to better understand macro‐level performances on test items. Then, an attempt to develop a general theory of item construction is described briefly and illustrated with the domains of fraction addition problems and adult literacy. Psychometric models appropriate for various scoring rubrics are discussed.
A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHM
Authors: Muraki Eiji 
Published: August 2014

ABSTRACTThe Partial Credit model with a varying slope parameter has been developed, and it is called the Generalized Partial Credit model. The item step parameter of this model is decomposed to a location and a threshold parameter, following Andrich's Rating Scale formulation. The EM algorithm for estimating the model parameters was derived. The performance of this generalized model is compared with a Rasch family of polytomous item response models based on both simulated and real data.Simulated data were generated and then analyzed by the various polytomous item response models. The results obtained demonstrate that the rating formulation of the Generalized Partial Credit model is quite adaptable to the analysis of polytomous item responses. The real data used in this study consisted of NAEP Mathematics data which was made up of both dichotomous and polytomous item types. The Partial Credit model was applied to this data using both constant and varying slope parameters. The Generalized Partial Credit model, which provides for varying slope parameters, yielded better fit to data than the Partial Credit model without such a provision.Index terms: item response modelpolytomous item response modelthe Partial Credit modelthe Rating Scale modelthe Nominal Response modelNAEP
SOME PRACTICAL CONSIDERATIONS WHEN CONVERTING A LINEARLY ADMINISTERED TEST TO AN ADAPTIVE FORMAT§
Authors: Wainer Howard 
Published: August 2014

ABSTRACTComputerized adaptive testing (CAT) is a set of techniques developed to increase the efficiency of testing. Its use, while still in its childhood, is well past its infancy. We now know enough to be able to use it in operational programs. Despite attempts to make CAT as compatible as possible with existing testing practices and procedures, there remain enough subtle differences between an adaptively and a linearly administered test that some loud cautions must be sounded. This note is meant to serve that purpose. We will discuss four practices which are commonplace in traditional testing but which are less sensible within CAT. These practices represent possible potholes on the road to the conversion of a linearly administered test to its adaptively administered successor. We also suggest some ways to avoid these potholes and hence smooth the transition to CAT.
THE FEASIBILITY OF MODELING SECONDARY TOEFL ABILITY DIMENSIONS USING MULTIDIMENSIONAL IRT MODELS
Authors: McKinley Robert L. , Way Walter D. 
Published: August 2014

ABSTRACTAn analysis of the skills necessary for performance on the TOEFL® test tends to support the view that there are important, although perhaps subtle, secondary dimensions present in the test. Given that these subtle secondary ability dimensions may be present in examinee response data, and that they do represent meaningful psychological variables, the purpose of this research was to explore the feasibility of an IRT‐based method of modeling examinee performance on these secondary ability dimensions. The procedure investigated is based on a multidimensional extension of the IRT model currently used for equating the TOEFL test. Both exploratory multidimensional IRT (MIRT) and confirmatory multidimensional IRT (CMIRT) models were investigated in the study. The work performed included the application of unidimensional IRT, MIRT, and CMIRT models to two TOEFL forms to evaluate the extent to which model fit is enhanced by using a multidimensional model, and to determine to what extent the additional fitted ability dimensions correspond to meaningful cognitive processes or content areas.The results of this study indicate that the MIRT and CMIRT procedures were successful in modeling secondary ability dimensions on TOEFL. The two procedures provided corroborative evidence in interpreting the structure of the test that was consistent with previous interpretations of the test's structure. The data presented in this study also provide an illustration of how a particular criterion for assessing model fit—the consistent Akaike information criterion—can be utilized to identify the best of several competing models of test structure.
AN ANALYSIS OF ENGLISH COMPOSITION TEST ESSAY PROMPTS FOR DIFFERENTIAL DIFFICULTY
Authors: Pomplun Mark , Wright David , Oleka Napoleon , Sudlow Marilyn 
Published: August 2014

ABSTRACTThe purpose of this study was to conduct a detailed analysis of the difficulty over time of the essay prompts for the English Composition Achievement Test (ECT) with Essay of the College Board. Differential difficulty was explored by considering whether the relationship between the reference groups—male and white students—and the focal groups—female, American Indian, Asian American, Asian American ESL (English Second Language), black, Hispanic, and Hispanic ESL students—had remained constant over the seven years studied. The study used ECT data for seven of the last eight years: 1983, 1985, 1986, 1987, 1988, 1989, and 1990. In order to study the relationship between the ECT objective and the ECT with Essay, linear regressions and conditional means of essay scores on objective scores were computed. For each comparison, difference lines for the regressions and conditional means were plotted in order to identify any years with aberrant differences. These differences were very consistent across the years, indicating that test developers had been successful in using essay prompts that did not generally cause groups to perform differentially. Two essay prompts, however, were identified that contained characteristics that may have been related to differential performance. It is recommended that these findings be followed by analysis of more essay prompts and by more research into the nature of the essay performance of minority and ESL groups.
A MONTE CARLO STUDY OF THE EFFECTS OF MODEL MISSPECIFICATION ON HLM ESTIMATES
Authors: Donoghue John R. , Jenkins Frank 
Published: August 2014

ABSTRACTMonte Carlo methods were used to investigate the effect of misspecification of the second‐level in a two‐level hierarchical linear model. Sample composition, heterogeneity of the group size, level of intraclass correlation, and correlation between second‐level predictors were manipulated. Twenty data sets were generated for each condition. Each data set was analyzed nine times with the HLM program, corresponding to the correct model and eight types of misspecification.The error variance σ2was estimated accurately for all model specifications. For the correct second‐level model, regression parameters γ and the covariance matrix of the second‐level errors τ were estimated accurately, although estimates were slightly biased for some levels of the sample composition manipulation. Misspecifications that failed to include a predictor had a larger effect than those which erroneously included an effect.Reactivity was assessed by examining parameters of equations which were correctly specified, although another equation in the model was misspecified. HLM showed little reactivity. Only one of the misspecified models yielded γ estimates that were significantly farther, on average, from the generating parameters than were those of the correctly specified model. Similar results were found for the root mean square residual between the generating and estimated τ.
DIFFERENTIAL ITEM FUNCTIONING ON THE GRADUATE MANAGEMENT ADMISSION TEST
Authors: O'Neill Kathleen A. , McPeek W. Miles , Wild Cheryl L. 
Published: August 2014

ABSTRACTThe purpose of this study was to identify differentially functioning items on operational administrations of the Graduate Management Admission Test (GMAT) through the use of the Mantel‐Haenszel statistic. Retrospective analyses of data collected over a three‐year period are reported for Black/White and female/male comparisons. Two tests are included in the analyses–Verbal and Quantitative. Data concerning the number of items identified as differentially difficult, the characteristics of the test questions associated with differential item difficulty, and the relationship of item difficulty to differential item functioning are discussed. In general, one percent to six percent of the items were identified as being differentially difficult per comparison with a greater number of items flagged in the female/male analyses than in the Black/White analyses. There was little overlap among the items flagged in the two analyses. Although the analyses suggested some content characteristics that may be related to differential item functioning, these findings about the GMAT items should be considered tentative since only a small number of items was studied and all investigations were post hoc analyses. Correlations between item difficulty and differential item functioning were generally low, with the exception of Quantitative items in the Black/White analyses. For these items, a moderately positive relationship existed between item difficulty and the differential item functioning statistic, showing that Black examinees performed differentially better than matched White examinees as item difficulty increased.
HUMAN ABILITIES AND MODES OF ATTENTION: THE ISSUE OF STYLISTIC CONSISTENCIES IN COGNITION
Authors: Messick Samuel 
Published: August 2014

ABSTRACTSpearman's notions of mental energy and mental span presage modern conceptions of attentional resources and working memory as fundamental to intelligence. Viewing attention as the conative directing of the intellect, as “the application of intellectual energy,” Spearman's quantitative law of mental span deals with limits on the allocation of attention. Because attentional resources are salient in both historical and current conceptions of intelligence, the occurrence of multiple and alternative modes of attention complicates these theories. Moreover, such consistent individual differences in attentional mode have important implications for the theory and measurement of cognitive processing more generally. Specifically, two broad bipolar factors have been identified that contrast sharp‐focus versus broad‐focus scanning and signal versus information scanning. These stylistic factors are linked to personality and reflect not only the enhancement of information processing in the focus of attention, but also the possibility of parallel processing in the fringe invoking the potential need for active inhibition of distracting or competing processes – points that were also anticipated by Spearman.
ESTIMATING THE CONSISTENCY AND ACCURACY OF CLASSIFICATIONS BASED ON TEST SCORES
Authors: Livingston Samuel A. , Lewis Charles 
Published: August 2014

ABSTRACTThis paper presents a method for estimating the accuracy and consistency of classifications based on test scores. The scores can be produced by any scoring method, including the formation of a weighted composite. The estimates use data from a single form. The reliability of the score is used to estimate its effective test length in terms of discrete items. The true‐score distribution is estimated by fitting a four‐parameter beta model. The conditional distribution of scores on an alternate form, given the true score, is estimated from a binomial distribution based on the estimated effective test length. The agreement between classifications on two alternate forms is estimated by assuming conditional independence, given the true score.An evaluation of the method showed that the estimates of the percent of test‐takers correctly classified and the percent consistently classified were within one percentage point of the actual values in most cases. Although the estimated effective test length and the estimates of the conditional standard error of measurement are sensitive to changes in the specified minimum and maximum possible scores, the estimates of the decision accuracy and decision consistency statistics are not.
THE EFFECT OF TEST SPEEDEDNESS ON SUBGROUP PERFORMANCE
Authors: Lawrence Ida M. 
Published: August 2014

ABSTRACTThe purpose of this study was to gauge the disadvantage, if any, of reaching difficult items that appear earlier in a test, at the expense of not reaching easier items that appear later in a test. The study focused on SAT and GMAT data and examined the effect for the following focal groups: females, Asian American, Blacks, and Hispanics. The impact of differential test speededness on subgroup differences in test scores was found to be minimal for SAT Mathematical and GMAT Quantitative tests. However, there appears to be a relationship between the presence of differential speededness on the SAT Verbal test and subgroup differences in test performance; after matching on total score, Black and Hispanic examinees receive credit for more difficult sets of items in the SAT Verbal test, relative to White examinees. Due to smaller sample sizes, findings for the GMAT Verbal tests were not as clear‐cut as the findings for the SAT Verbal tests. This research attempted to provide a technique for assessing the circumstances under which the effect of test speededness has a differential effect on the performance of subgroups, when differences in ability are taken into consideration. As is the case with all non‐experimental research on test speededness, the results of this study cannot predict how examinees would have performed with a shorter test, or more liberal time limits, because test strategies might have been different under those conditions.
THE UNIQUE CORRESPONDENCE OF ITEM RESPONSE FUNCTIONS AND ITEM CATEGORY RESPONSE FUNCTIONS IN POLYTOMOUSLY SCORED ITEM RESPONSE MODELS1
Authors: Chang Hua‐Hua , Mazzeo John 
Published: August 2014

ABSTRACTThe item response function (IRF) for a polytomously scored item is defined as a weighted sum of the item category response functions (ICRF, the probability of getting a particular score for a randomly sampled examinee of ability θ). This paper establishes the correspondence between an IRF and a unique set of ICRFs for two of the most commonly used polytomous IRT models (the partial credit models and the graded response model). Specifically, a proof of the following assertion is provided for these models: If two items have the same IRF, then they must have the same number of categories; moreover, they must consist of the same ICRFs. As a corollary, for the Rasch dichotomous model, if two tests have the same test characteristic function (TCF), then they must have the same number of items. Morover, for each item in one of the tests, an item in the other test with an identical IRF must be exist. Theoretical as well as practical implications of these results are discussed.