bluuebunny/crossref_metadata_2025 huggingface dataset:
FIELD DEPENDENCE REVISITED
Authors: Witkin Herman A. , Goodenough Donald R.
Published: August 2014

Results of the large number of recent studies bearing on the field‐dependence‐independence component of psychological differentiation are reviewed, and revisions and extensions of field‐dependence theory suggested by the newer evidence are identified. The field‐independent and field‐dependent cognitive styles now seem best conceived as tendencies to function with greater or less autonomy of external referents, manifested in both the cognitive and social domains. Associated with these contrasting tendencies, field‐independent people do well in cognitive restructuring tasks, at least in the spatial domain, whereas field‐dependent people show more of the interpersonal competencies involved in getting along with others. The field‐dependence‐independence cognitive‐style dimension is thus bipolar with regard to level, having no clear high or low ends; and it is neutral with regard to value, each pole having qualities adaptive in particular circumstances. Possible causal connections among field dependence‐independence, cognitive restructuring skills, and interpersonal competencies are considered. Finally, a distinction is drawn between people who are fixed with regard to the characteristics associated with a field‐dependent or field‐independent style and those who are mobile, having access to the characteristics of both styles.
THE INVENTORY OF DOCUMENTED ACCOMPLISHMENTS FOR GRADUATE ADMISSIONS: RESULTS OF A FIELD TRIAL STUDY OF ITS RELIABILITY, SHORT‐TERM CORRELATES, AND EVALUATION
Authors: Baird Leonard L. , Knapp Joan E.
Published: August 2014

This is the report of the third phase of a multi‐phase project designed to develop an inventory of documented accomplishments for graduate student selection. The material developed in earlier stages was pilot‐tested in Phase Three in 26 departments that represented the fields of English, biology, and psychology.

The major purpose of this study was to investigate the instrument developed in Stage Two in terms of: 1) technical soundness, and 2) the feasibility of its use in the admissions process. This study represents the transition of the use of documented accomplishments to predict graduate school achievement from a research phase to a more operational phase during which the instrument would be tested in actual admissions situations.

The departments were asked to administer the instrument to their newly enrolled graduate students. Straightforward descriptive and correlational analyses of the responses to the inventory were conducted to: (1) describe characteristics of the students and the instruments; and (2) identify the most reliable clusters of items and indices of accomplishment. Students were followed up at the end of their first year to assess their graduate school accomplishments and their relationship to previous attainments. This information provided data for analyses of the short‐term correlates of the measures. In addition, faculty were interviewed to determine the plausibility of the content of the instrument and the most accurate and operationally simple instructions and reporting formats. Students' reactions to the inventory were obtained by interview and by additional questions at the end of the inventory.
A STUDY OF THE RELIABILITY OF NEDELSKY'S METHOD FOR CHOOSING A PASSING SCORE
Authors: Livingston Samuel A. , Kastrinos William
Published: August 2014

Licensing and certification boards have begun to use Nedelsky's method of choosing passing scores. Alternate forms of a test, containing 25 questions in common, were used to investigate the reliability of passing scores chosen by this method. Seven content experts judged both forms of the test. The differences between these judges in their passing scores for the 25 common questions were large and consistent. The judges tended to set higher passing scores the second time they judged the questions, and to leave different wrong answer choices unmarked. These results suggest that users of Nedelsky's method should consider a two‐stage judgment procedure.
ANALYSIS OF CICS RESPONSE TIME
Authors: Braun Henry I. , Szatrowski Ted H.
Published: August 2014

A small sample of the IBM CICS system performance data routinely collected by the ETS performance management group was analyzed using regression techniques. Preliminary studies indicated that control records based on ten minute intervals obscured the relationship of interest, so a new data base consisting of constructed one minute control records was built. A model was then developed which accurately predicted the improvement that would be obtained with a change in the hardware configuration. The model is quite parsimonious, requiring one or two transaction variables and only one system control variable, the total number of page‐ins in the one minute interval. The methodology seems general enough to apply in other situations to aid in determining system bottlenecks and the consequences of their removal.
HOW TO DISPLAY DATA BADLY
Authors: Wainer Howard
Published: August 2014

Methods for displaying data badly have been under development for many years, and a wide variety of interesting and inventive schemes have emerged. Presented here is a synthesis yielding the twelve most powerful techniques that seem to underlie many of the realizations found in the literature. These twelve (the dirty dozen) are identified and illustrated.
THE DEAN'S LETTER OF EVALUATION AS A BASIS FOR THE SELECTION OF MEDICAL STUDENTS
Authors: Carlson Sybil B. , Ward William C. , Frederiksen Norman
Published: August 2014

The dean's letter of evaluation, as a summary of student performance in undergraduate medical education, contributes information that is potentially valuable for use in selection for graduate medical education. The dean's letters for 113 fourth‐year students from ten medical schools were coded to summarize the evaluations of performance provided in the letters. The performance dimensions that were reported most frequently and consistently in this sample of letters were factor analyzed; three discriminable dimensions were found. Despite the difficulties involved in using subjective evaluations, the dean's letter appears to yield information regarding various aspects of undergraduate performance that are used with some degree of consistency across medical schools in recommending students as candidates for residencies.
CONFIRMATORY FACTOR ANALYSIS OF THE TEST OF ENGLISH AS A FOREIGN LANGUAGE
Authors: Hale Gordon A. , Rock Donald A. , Jirele Thomas
Published: August 2014

In a recent study, confirmatory factor analyses indicated that, for each of several language groups, TOEFL performance can be characterized by two factors, associated with (a) the Listening Comprehension section and (b) the other sections of the test (Hale, Stansfield, Rock, Hicks, Butler, & Oiler, 1988). This conclusion was inconsistent with that drawn in the most comprehensive previous factor‐analytic study of the TOEFL, which suggested a three‐factor solution for each of several languages (Swinton & Powers, 1980). The present study investigated the inconsistency in conclusions drawn from these two studies and provided further information about the factor structure of the TOEFL.

It was hypothesized that the inconsistency between studies was related to the populations under investigation, as the earlier study used TOEFL examinees in both domestic and overseas test centers, whereas the more recent study used domestic examinees only. The present data did not support this hypothesis, however. Confirmatory factor analyses were conducted for each of several language groups, using data from the test form used in the more recent study (a 1984 TOEFL), and these analyses yielded essentially similar results for both domestic and overseas populations, as well as for the combined population. In all cases, the data supported a two‐factor interpretation, with the two factors related to the Listening Comprehension section and to the nonlistening sections.

Other hypotheses were that the inconsistency between studies was due to differences in factor‐analytic methodologies used or to changes in the test over time. Confirmatory factor analyses were conducted using the data from the earlier study (taken from a 1976 TOEFL) as well as data from the more recent study. These analyses supported a two‐factor interpretation in both cases, for each of several language groups, with the two factors associated with Listening Comprehension and with the other sections of the test. Thus, the use of different factor‐analytic methodologies in the two previous studies undoubtedly contributed to the inconsistency, although further work would be needed to determine exactly what aspects of the methodology played a role. The data also provided tentative evidence that the basic factor structure of the test may not have changed substantially between 1976 and 1984.

The role of examinee proficiency in determining the TOEFL's factor structure was also examined. For each of several language groups, low‐ and high‐proficiency groups were defined on the basis of approximately one third of the items in the TOEFL, drawn from all sections of the test, and factor analyses were then performed on an abbreviated TOEFL, which consisted of the remaining items. For both proficiency levels within each language group a two‐factor structure appeared to underlie performance, with the factors once again linked to Listening Comprehension and to the other, nonlistening sections of the test.
NONRESPONSE IN DECLARED ETHNICITY AND THE IDENTIFICATION OF DIFFERENTIALLY FUNCTIONING ITEMS
Authors: Allen Nancy L. , Wainer Howard
Published: August 2014

The accuracy of procedures that are used to compare the performance of different groups of examinees on test items obviously depends upon the correct classification of members in each examinee group. The significance of this dependence is determined by the sensitivity of the statistical procedure and the proportion of examinees who are unidentified. Since the number of nonrespondents to questions of ethnicity is often of the same order of magnitude as the number of identified members of most minority groups, understanding the effect of nonresponse is crucial to evaluating the validity of a procedure which is used to study differential item functioning (DIF).

In this study, we examined the effect of nonresponse to questions of ethnic identity on the measurement of DIF for SAT verbal items using a commonly used modern method, the Mantel‐Haenszel procedure (Holland and Thayer, 1988). We found that efforts to obtain more complete ethnic identifications from the examinees would be rewarded with more accurate DIF analyses.
DIFFERENTIAL TESTLET FUNCTIONING DEFINITIONS AND DETECTION
Authors: Wainer Howard , Sireci Stephen G. , Thissen David
Published: August 2014

It is sometimes sensible to think of the fundamental unit of test construction as being larger than an individual item. This unit, dubbed the testlet, must pass muster in the same way that items do. One criterion of a good item is the absence of DIF—the item must function in the same way in all important subpopulations of examinees. In this paper we define what we mean by testlet DIF and provide a statistical methodology to detect it. This methodology parallels the IRT‐based likelihood ratio procedures explored previously by Thissen, Steinberg & Wainer (1988, in press). We illustrate this methodology with analyses of data from a testlet‐based experimental version of the Scholastic Aptitude Test.
A COMPARISON OF OPEN‐ENDED AND MULTIPLE‐CHOICE QUESTION FORMATS FOR THE QUANTITATIVE SECTION OF THE GRADUATE RECORD EXAMINATIONS GENERAL TEST
Authors: Bridgeman Brent
Published: August 2014

Free‐response counterparts to a set of items from the quantitative section of the Graduate Record Examinations General Test were developed. Examinees responded to these items by gridding numerical answers on a machine‐readable answer sheet in essentially the same way social security numbers are now gridded. In addition, the answer grid included special symbols for a negative sign (‐), a decimal point (.), a division sign (/), and a variable (k).

The test section with the special answer sheet was administered at the end of a regular GRE administration. Test forms were spiraled so that random groups received either the grid‐in questions or the same questions in a multiple‐choice format. Because the amount of extra time required to grid in responses was unknown, both a long and a short form were developed, with students randomly receiving one or the other. Sample size for these comparisons was 12,465. In a separate data collection effort, 364 paid volunteers who had recently taken the General Test used a computer keyboard to enter answers to the same set of questions.

Correct and incorrect answers chosen in the multiple‐choice format were compared with the answers generated in the free‐response format. Three‐parameter item characteristic curves were compared, as were correlational patterns with other test scores and undergraduate grade point average. Format by gender and format by ethnicity (Asian American, Black, Hispanic, and White) interactions were explored with analyses of covariance.

Despite the format differences noted for individual items, total scores for the multiple‐choice and free‐response tests demonstrated remarkably similar correlational patterns, and there were no significant interactions of test format with either gender or ethnicity.
ITEM CONSTRUCTION AND PSYCHOMETRIC MODELS APPROPRIATE FOR CONSTRUCTED RESPONSES
Authors: Tatsuoka Kikumi K.
Published: August 2014

Constructed‐response formats are desired for measuring complex and dynamic response processes which require the examinee to understand the structures of problems and micro‐level cognitive tasks. These micro‐level tasks and their organized structures are usually unobservable. This study shows that elementary graph theory is useful for organizing these micro‐level tasks and for exploring their properties and relations. Moreover, this approach enables us to better understand macro‐level performances on test items. Then, an attempt to develop a general theory of item construction is described briefly and illustrated with the domains of fraction addition problems and adult literacy. Psychometric models appropriate for various scoring rubrics are discussed.
A GENERALIZED PARTIAL CREDIT MODEL: APPLICATION OF AN EM ALGORITHM
Authors: Muraki Eiji
Published: August 2014

The Partial Credit model with a varying slope parameter has been developed, and it is called the Generalized Partial Credit model. The item step parameter of this model is decomposed to a location and a threshold parameter, following Andrich's Rating Scale formulation. The EM algorithm for estimating the model parameters was derived. The performance of this generalized model is compared with a Rasch family of polytomous item response models based on both simulated and real data.

Simulated data were generated and then analyzed by the various polytomous item response models. The results obtained demonstrate that the rating formulation of the Generalized Partial Credit model is quite adaptable to the analysis of polytomous item responses. The real data used in this study consisted of NAEP Mathematics data which was made up of both dichotomous and polytomous item types. The Partial Credit model was applied to this data using both constant and varying slope parameters. The Generalized Partial Credit model, which provides for varying slope parameters, yielded better fit to data than the Partial Credit model without such a provision.

Index terms: item response model

polytomous item response model

the Partial Credit model

the Rating Scale model

the Nominal Response model

NAEP
SOME PRACTICAL CONSIDERATIONS WHEN CONVERTING A LINEARLY ADMINISTERED TEST TO AN ADAPTIVE FORMAT§
Authors: Wainer Howard
Published: August 2014

Computerized adaptive testing (CAT) is a set of techniques developed to increase the efficiency of testing. Its use, while still in its childhood, is well past its infancy. We now know enough to be able to use it in operational programs. Despite attempts to make CAT as compatible as possible with existing testing practices and procedures, there remain enough subtle differences between an adaptively and a linearly administered test that some loud cautions must be sounded. This note is meant to serve that purpose. We will discuss four practices which are commonplace in traditional testing but which are less sensible within CAT. These practices represent possible potholes on the road to the conversion of a linearly administered test to its adaptively administered successor. We also suggest some ways to avoid these potholes and hence smooth the transition to CAT.
THE FEASIBILITY OF MODELING SECONDARY TOEFL ABILITY DIMENSIONS USING MULTIDIMENSIONAL IRT MODELS
Authors: McKinley Robert L. , Way Walter D.
Published: August 2014

An analysis of the skills necessary for performance on the TOEFL® test tends to support the view that there are important, although perhaps subtle, secondary dimensions present in the test. Given that these subtle secondary ability dimensions may be present in examinee response data, and that they do represent meaningful psychological variables, the purpose of this research was to explore the feasibility of an IRT‐based method of modeling examinee performance on these secondary ability dimensions. The procedure investigated is based on a multidimensional extension of the IRT model currently used for equating the TOEFL test. Both exploratory multidimensional IRT (MIRT) and confirmatory multidimensional IRT (CMIRT) models were investigated in the study. The work performed included the application of unidimensional IRT, MIRT, and CMIRT models to two TOEFL forms to evaluate the extent to which model fit is enhanced by using a multidimensional model, and to determine to what extent the additional fitted ability dimensions correspond to meaningful cognitive processes or content areas.

The results of this study indicate that the MIRT and CMIRT procedures were successful in modeling secondary ability dimensions on TOEFL. The two procedures provided corroborative evidence in interpreting the structure of the test that was consistent with previous interpretations of the test's structure. The data presented in this study also provide an illustration of how a particular criterion for assessing model fit—the consistent Akaike information criterion—can be utilized to identify the best of several competing models of test structure.
AN ANALYSIS OF ENGLISH COMPOSITION TEST ESSAY PROMPTS FOR DIFFERENTIAL DIFFICULTY
Authors: Pomplun Mark , Wright David , Oleka Napoleon , Sudlow Marilyn
Published: August 2014

The purpose of this study was to conduct a detailed analysis of the difficulty over time of the essay prompts for the English Composition Achievement Test (ECT) with Essay of the College Board. Differential difficulty was explored by considering whether the relationship between the reference groups—male and white students—and the focal groups—female, American Indian, Asian American, Asian American ESL (English Second Language), black, Hispanic, and Hispanic ESL students—had remained constant over the seven years studied. The study used ECT data for seven of the last eight years: 1983, 1985, 1986, 1987, 1988, 1989, and 1990. In order to study the relationship between the ECT objective and the ECT with Essay, linear regressions and conditional means of essay scores on objective scores were computed. For each comparison, difference lines for the regressions and conditional means were plotted in order to identify any years with aberrant differences. These differences were very consistent across the years, indicating that test developers had been successful in using essay prompts that did not generally cause groups to perform differentially. Two essay prompts, however, were identified that contained characteristics that may have been related to differential performance. It is recommended that these findings be followed by analysis of more essay prompts and by more research into the nature of the essay performance of minority and ESL groups.
A MONTE CARLO STUDY OF THE EFFECTS OF MODEL MISSPECIFICATION ON HLM ESTIMATES
Authors: Donoghue John R. , Jenkins Frank
Published: August 2014

Monte Carlo methods were used to investigate the effect of misspecification of the second‐level in a two‐level hierarchical linear model. Sample composition, heterogeneity of the group size, level of intraclass correlation, and correlation between second‐level predictors were manipulated. Twenty data sets were generated for each condition. Each data set was analyzed nine times with the HLM program, corresponding to the correct model and eight types of misspecification.

The error variance σ2was estimated accurately for all model specifications. For the correct second‐level model, regression parameters γ and the covariance matrix of the second‐level errors τ were estimated accurately, although estimates were slightly biased for some levels of the sample composition manipulation. Misspecifications that failed to include a predictor had a larger effect than those which erroneously included an effect.

Reactivity was assessed by examining parameters of equations which were correctly specified, although another equation in the model was misspecified. HLM showed little reactivity. Only one of the misspecified models yielded γ estimates that were significantly farther, on average, from the generating parameters than were those of the correctly specified model. Similar results were found for the root mean square residual between the generating and estimated τ.
DIFFERENTIAL ITEM FUNCTIONING ON THE GRADUATE MANAGEMENT ADMISSION TEST
Authors: O'Neill Kathleen A. , McPeek W. Miles , Wild Cheryl L.
Published: August 2014

The purpose of this study was to identify differentially functioning items on operational administrations of the Graduate Management Admission Test (GMAT) through the use of the Mantel‐Haenszel statistic. Retrospective analyses of data collected over a three‐year period are reported for Black/White and female/male comparisons. Two tests are included in the analyses–Verbal and Quantitative. Data concerning the number of items identified as differentially difficult, the characteristics of the test questions associated with differential item difficulty, and the relationship of item difficulty to differential item functioning are discussed. In general, one percent to six percent of the items were identified as being differentially difficult per comparison with a greater number of items flagged in the female/male analyses than in the Black/White analyses. There was little overlap among the items flagged in the two analyses. Although the analyses suggested some content characteristics that may be related to differential item functioning, these findings about the GMAT items should be considered tentative since only a small number of items was studied and all investigations were post hoc analyses. Correlations between item difficulty and differential item functioning were generally low, with the exception of Quantitative items in the Black/White analyses. For these items, a moderately positive relationship existed between item difficulty and the differential item functioning statistic, showing that Black examinees performed differentially better than matched White examinees as item difficulty increased.
HUMAN ABILITIES AND MODES OF ATTENTION: THE ISSUE OF STYLISTIC CONSISTENCIES IN COGNITION
Authors: Messick Samuel
Published: August 2014

Spearman's notions of mental energy and mental span presage modern conceptions of attentional resources and working memory as fundamental to intelligence. Viewing attention as the conative directing of the intellect, as “the application of intellectual energy,” Spearman's quantitative law of mental span deals with limits on the allocation of attention. Because attentional resources are salient in both historical and current conceptions of intelligence, the occurrence of multiple and alternative modes of attention complicates these theories. Moreover, such consistent individual differences in attentional mode have important implications for the theory and measurement of cognitive processing more generally. Specifically, two broad bipolar factors have been identified that contrast sharp‐focus versus broad‐focus scanning and signal versus information scanning. These stylistic factors are linked to personality and reflect not only the enhancement of information processing in the focus of attention, but also the possibility of parallel processing in the fringe invoking the potential need for active inhibition of distracting or competing processes – points that were also anticipated by Spearman.
ESTIMATING THE CONSISTENCY AND ACCURACY OF CLASSIFICATIONS BASED ON TEST SCORES
Authors: Livingston Samuel A. , Lewis Charles
Published: August 2014

This paper presents a method for estimating the accuracy and consistency of classifications based on test scores. The scores can be produced by any scoring method, including the formation of a weighted composite. The estimates use data from a single form. The reliability of the score is used to estimate its effective test length in terms of discrete items. The true‐score distribution is estimated by fitting a four‐parameter beta model. The conditional distribution of scores on an alternate form, given the true score, is estimated from a binomial distribution based on the estimated effective test length. The agreement between classifications on two alternate forms is estimated by assuming conditional independence, given the true score.

An evaluation of the method showed that the estimates of the percent of test‐takers correctly classified and the percent consistently classified were within one percentage point of the actual values in most cases. Although the estimated effective test length and the estimates of the conditional standard error of measurement are sensitive to changes in the specified minimum and maximum possible scores, the estimates of the decision accuracy and decision consistency statistics are not.
THE EFFECT OF TEST SPEEDEDNESS ON SUBGROUP PERFORMANCE
Authors: Lawrence Ida M.
Published: August 2014

The purpose of this study was to gauge the disadvantage, if any, of reaching difficult items that appear earlier in a test, at the expense of not reaching easier items that appear later in a test. The study focused on SAT and GMAT data and examined the effect for the following focal groups: females, Asian American, Blacks, and Hispanics. The impact of differential test speededness on subgroup differences in test scores was found to be minimal for SAT Mathematical and GMAT Quantitative tests. However, there appears to be a relationship between the presence of differential speededness on the SAT Verbal test and subgroup differences in test performance; after matching on total score, Black and Hispanic examinees receive credit for more difficult sets of items in the SAT Verbal test, relative to White examinees. Due to smaller sample sizes, findings for the GMAT Verbal tests were not as clear‐cut as the findings for the SAT Verbal tests. This research attempted to provide a technique for assessing the circumstances under which the effect of test speededness has a differential effect on the performance of subgroups, when differences in ability are taken into consideration. As is the case with all non‐experimental research on test speededness, the results of this study cannot predict how examinees would have performed with a shorter test, or more liberal time limits, because test strategies might have been different under those conditions.
THE UNIQUE CORRESPONDENCE OF ITEM RESPONSE FUNCTIONS AND ITEM CATEGORY RESPONSE FUNCTIONS IN POLYTOMOUSLY SCORED ITEM RESPONSE MODELS1
Authors: Chang Hua‐Hua , Mazzeo John
Published: August 2014

The item response function (IRF) for a polytomously scored item is defined as a weighted sum of the item category response functions (ICRF, the probability of getting a particular score for a randomly sampled examinee of ability θ). This paper establishes the correspondence between an IRF and a unique set of ICRFs for two of the most commonly used polytomous IRT models (the partial credit models and the graded response model). Specifically, a proof of the following assertion is provided for these models: If two items have the same IRF, then they must have the same number of categories; moreover, they must consist of the same ICRFs. As a corollary, for the Rasch dichotomous model, if two tests have the same test characteristic function (TCF), then they must have the same number of items. Morover, for each item in one of the tests, an item in the other test with an identical IRF must be exist. Theoretical as well as practical implications of these results are discussed.
MULTIMETHOD CONSTRUCT VALIDATION OF THE TEST OF SPOKEN ENGLISH
Authors: Boldt R. F. , Oltman P. K.
Published: August 2014

Administration of the Test of Spoken English (TSE) yields tapes of oral performance on items within six sections of the test. Trained scorers subsequently rate responses using four proficiency scales: pronunciation, grammar, fluency, and overall comprehensibility. This project examined the consistency of statistical relations among TSE scores with the measurement constructs these scores purport to reflect.

Analyses included factor analysis and multidimensional scaling, which examined dimensions underlying the scores. These dimensional methods were applied to the 18 scores yielded by the combinations of section and scale. Another analysis was applied to scale scores averaged over sections. This analysis compared the ranking of pairs of scale scores obtained during the original scoring of selected taped performances with the ranking resulting from a rescoring by different raters.

Multidimensional scaling analyses revealed that three‐dimensional solutions fit the scale intercorrelations with low stress values, and that the coordinates of the scales fell into three clusters in three‐dimensional space, those clusters being defined primarily by test section rather than by proficiency scales.

An exploratory factor analysis revealed that a single dimension dominated the variation of the 18 section‐scale scores. However, when tapes for scale scores with substantial discrepancies were rerated, agreement between the order of original and rerated scale scores far exceeded chance. This indicated that raters were able to modify their judgements according to the scale being rated. Subsequent exploratory factor analysis indicated that both section and scale factors contribute to score variation. The factors were highly correlated, and the predominance of the single factor in the exploratory analysis was seen as arising from those correlations.
A TRANSPORTABILITY STUDY OF THE PRAXIS I: ACADEMIC SKILLS ASSESSMENTS COMPUTER‐BASED TEST (CBT) AND THE PRAXIS I: PRE‐PROFESSIONAL SKILLS TESTS (PPST) SPECIFICATION STATEMENTS FOR ENTRY‐LEVEL VOCATIONAL EDUCATION TEACHERS
Authors: Tannenbaum Richard J. , Rosenfeld Michael , Teryek Charles J.
Published: August 2014

The purpose of this study was to determine if the 55 test specification statements for the Praxis I: Academic Skills Assessments computer‐based tests (CBT) and the Praxis I: Pre‐Professional Skills Tests (PPST) were judged to be appropriate for entry‐level vocational education teachers regardless of specialty area. The test specification statements were mailed to a random sample of 5,610 vocational education teachers representing 11 specialty areas. Useable surveys were returned by 2,347 vocational education teachers. Judgments of importance were made using a 5‐point rating scale. A test specification statement was considered to be appropriate for entry‐level vocational education teachers if respondents from at least 9 of 11 specialty areas judged it to have a mean importance greater than 3.00 (moderately important). The results indicated that all 55 test specifications received mean ratings above 3.00 by respondents from all 11 specialty areas. It may be concluded from the results that the test specification statements for the Praxis I: CBT and the Praxis I: PPST are appropriate for entry‐level vocational education teachers regardless of the specialty area in which they teach.
DEALING WITH UNCERTAINTY ABOUT ITEM PARAMETERS: EXPECTED RESPONSE FUNCTIONS
Authors: Mislevy Robert J. , Wingersky Marilyn S. , Sheehan Kathleen M.
Published: August 2014

It is a common practice in item response theory (IRT) to treat estimates of item parameters, say B, as if they were the known, true quantities, B. However, ignoring the uncertainty associated with item parameters can lead to biases and over‐confidence in subsequent inferences such as ability estimation, especially when item‐calibration samples are small. This paper demonstrates how to incorporate uncertainty about B with Lewis's “expected response functions” (ERFs), pointwise expected values of item response conditional on examinee proficiency averaged over posterior distributions of item parameters. This paper presents ERFs, outlines procedures for computing them and using them in practical work, and gives an illustration with data from the National Assessment of Educational Progress. Advantages of approximating ERFs response curves with members of familiar parametric families of IRT curves are noted.
THE EFFECTS OF SAMPLE RESTRICTION ON GENDER DIFFERENCES
Authors: Lewis Charles , Willingham Warren W.
Published: August 2014

As strongly suggested by recent work, patterns of gender difference can change because of changes in the selectivity of the sample itself. This is a statistical influence connected with the distributions of female and male scores, rather than a substantive influence related to demographic characteristics of the sample such as age or ethnicity. It is, nonetheless, an important influence because gender differences that are partly statistical in origin can easily confuse possible implications regarding education and assessment.

This report proposes a general model to account for the effects of sample restriction on gender differences. Simulations showed the model to be quite accurate in reproducing standard mean differences and other statistics in a restricted sample. Three primary contributing factors were identified: the range‐restricting effects of sample selection, differential variability of female and male scores in the original sample, and the representation of females and males in the restricted sample. A test with actual data showed reasonably good consistency between trends predicted by the model and trends in gender differences that have been widely observed in advanced tests administered to select samples.
PRACTICAL ISSUES IN LARGE‐SCALE HIGH‐STAKES COMPUTERIZED ADAPTIVE TESTING
Authors: Mills Craig N. , Stocking Martha L.
Published: August 2014

Computerized adaptive testing (CAT), while well‐grounded in psychometric theory, has had few large‐scale applications for high‐stakes, secure tests in the past. This is now changing as the cost of computing has declined rapidly. As is always true at such junctures where theory is translated into practice, many practical issues arise that must now be addressed. This paper discusses a number of such issues and sketches out potential problems and potential solutions. Our purpose is to encourage further development of solutions to the issues presented as well as other practical issues facing measurement professionals involved with the implementation of adaptive testing.
A NEW METHOD OF CONTROLLING ITEM EXPOSURE IN COMPUTERIZED ADAPTIVE TESTING
Authors: Stocking Martha L. , Lewis Charles
Published: August 2014

In the periodic testing environment associated with conventional paper‐and‐pencil tests, the frequency with which items are seen by test‐takers is tightly controlled in advance of testing by policies that regulate both the reuse of test forms and the frequency with which candidates may retake the test. In the continuous testing environment associated with more novel testing paradigms such as computerized adaptive testing (CAT), the computer itself can be used to control the frequency with which items are administered. This paper discusses previous methods of controlling item security in the continuous adaptive testing environment and presents a new method that overcomes some (but not all) of the disadvantages of previous attempts. An extensive example with this new method and a particular adaptive testing algorithm illustrates how concerns about test efficiency, parallelism, and security can be balanced.
ADJUSTMENT FOR READER RATING BEHAVIOR IN THE TEST OF WRITTEN ENGLISH
Authors: Longford Nicholas T.
Published: August 2014

This report evaluates the impact of a potential scheme for score adjustment using data from the administrations of the Test of Written English in 1994. It is shown that, assuming non‐informative assignment of readers to essays, the adjustment due to reader differences reduces the mean squared error for all essays except those graded by readers with small workloads. The quality of the rating process, as described by the variances due to true scores, severity, and inconsistency, as well as the distribution of workloads, are similar across the administrations. This would make a reliable prediction of the optimal score adjustment in future administrations possible. Two approximations to the optimal adjustment are proposed.
REVISING ANSWERS TO ITEMS IN COMPUTERIZED ADAPTIVE TESTS: A COMPARISON OF THREE MODELS1,2
Authors: Stocking Martha L.
Published: August 2014

The interest in the application of large‐scale computerized adaptive testing has served to focus attention on issues that arise when theoretical advances are made operational. Some of these issues stem less from changes in testing conditions and more from changes in testing paradigms. One such issue is that of the order in which questions are answered within a test or separately timed test section. In linear testing, this order of responses is entirely under the control of the test‐taker, who can omit questions, look ahead at questions, and return and revise answers to previous questions. The attempt to permit the same, or even reasonably restricted, control in adaptive testing can unintentionally result in transferring to the test‐taker control over which items are chosen for administration, threatening both test fairness and accuracy. This paper explores, using simulations, three models of permitting restricted test‐taker control over revising previous answers in the context of adaptive testing. Even under a worst‐case model of test‐taker revising behavior, two of the models of permitting item revisions work well in preserving test fairness and accuracy and one model studied may also preserve some cognitive processing styles developed by test‐takers for a linear testing environment.
INSOLATION AND SHADOW
Authors: Reid‐Green Keith S.
Published: August 2014

The following document describes a mathematical method for locating a new building so that it will not cast shadows on existing buildings during business hours. This and other questions dealing with sunlight and shadows can be determined from the apparent motion of the Sun on the solstices, approximately June 22nd and December 22nd.
MISSING RESPONSES AND IRT ABILITY ESTIMATION: OMITS, CHOICE, TIME LIMITS, AND ADAPTIVE TESTING
Authors: Mislevy Robert J. , Wu Pao‐Kuei
Published: August 2014

The basic equations of item response theory (IRT) provide a foundation for inferring examinees' abilities from responses to different test items. In practice, examinees do not generally provide a response to all items—for reasons that may or may not have been intended by the test administrator, and that may or may not be related to their ability. The mechanisms that produce missingness must be taken into account if correct inferences are to be drawn. Using concepts introduced by Rubin (1976), we discuss the implications for Bayesian and direct likelihood ability parameter estimation that are entailed by alternate test forms, targeted testing, adaptive testing, time limits, omitted responses, and examinee choice of tasks. Attention is focused on whether, in each case, the mechanism for missingness is “ignorable,” and, in those cases in which it is not, how it can be modeled.
HOW RELIABLE IS THE TOEFL TEST?
Authors: Wainer Howard , Lukhele Robert
Published: August 2014

We estimated the reliability of scores on four forms of the Test of English as a Foreign Language (TOEFL®) using a hybrid IRT model. Very little difference between their overall reliability was found when the testlet items were assumed to be independent and their dependence was modeled. A larger difference in reliability was found when test sections were analyzed individually. Then we found as much as a 40 percent overestimate in reading comprehension testlets, with the longer testlets of the newest form of the TOEFL test showing the most local dependence. The listening comprehension testlets exhibited much less local dependence. We also found that the test was unidimensional enough for the use of univariate IRT to be efficacious, and that the reading comprehension testlets showed essentially no differential functioning by sex.
SPECULATIONS ON THE FUTURE OF LARGE‐SCALE EDUCATIONAL ASSESSMENT
Authors: Bennett Randy Elliot
Published: August 2014

In this paper, I offer one scenario for the future of large‐scale educational assessment. I argue that, although large‐scale assessment has altered relatively little in recent years, this situation is about to change. The same competitive forces driving U.S. industry will compel test makers to (1) satisfy new market needs through continuous innovation, (2) improve productivity, (3) enhance customer‐service, and (4) address population diversity. In response, large‐scale assessment will reinvent itself to serve both summative and formative purposes, become curriculum‐embedded and performance‐based, occur at a distance, and incorporate constructs newly valued by society. This reinvention will be enabled chiefly by new technology and by advances in cognitive and measurement science.
TOWARD A COHERENT VIEW OF RELIABILITY IN TEST THEORY
Authors: Li Heng , Wainer Howard
Published: August 2014

Reliability of test scores, as estimated through measures of internal consistency, has been characterized mathematically in many ways that appear, on the surface at least, to be very dissimilar to one another. In this essay we provide a general mathematical framework that specializes to four different reliability coefficients. Through consideration of this general framework it becomes easier to convey to students both the individual character of the different formulations of reliability and the extent of their underlying similarity. In addition to providing a coherent view of reliability, the unified formula is also found to be a convenient vehicle for introducing more specialized topics, such as the Kaiser‐Guttman rule.
THE RELATIONSHIP BETWEEN COMPUTER FAMILIARITY AND PERFORMANCE ON COMPUTER‐BASED TOEFL TEST TASKS
Authors: Taylor Carol , Jamieson Joan , Eignor Daniel , Kirsch Irwin
Published: August 2014

The increasing use of computer‐based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 will confound language proficiency with computer proficiency and thus bring construct‐irrelevant variance to the measurement of examinees' English‐language abilities.

In a Phase I study (Kirsch, Jamieson, Taylor, & Eignor, 1998), TOEFL examinees were surveyed regarding their computer familiarity and classified into one of three computer familiarity groups: low, moderate, and high. In this study, Phase II, more than 1,100 “low‐computer‐familiar” and “high‐computer‐familiar” examinees from 12 international sites were identified from the Phase I survey and administered a computer tutorial and a set of 60 computer‐based TOEFL test items. The relationship between level of computer familiarity and performance on the computer‐based items was then examined. The examinees in Phase II were largely representative of those in Phase I, who were representative of the general TOEFL test‐taking population. Thus, results from this phase of the study are considered generalizable to the current TOEFL examinee population.

The effect of computer familiarity after adjustments for language ability was examined by performing a series of analyses of covariance (ANCOVAs), using TOEFL paper‐and‐pencil test score as the covariate. These analyses were followed by a series of ANCOVAs involving the computer familiarity variable and a number of other variables: gender, reason for taking the TOEFL test, times the TOEFL test had been taken, and location where the TOEFL test was taken. In a final set of analyses, the TOEFL paper‐and‐pencil test scores of the low‐ and high‐computer‐familiar examinees were weighted such that the groups had identical distributions on the covariate.

After controlling for language ability, the researchers found no meaningful relationship between level of computer familiarity and level of performance on computerized language tasks among TOEFL examinees who had completed the computer tutorial. This finding was consistent for all but one of the subgroups considered. A small but practically significant interaction between computer familiarity and reason for taking the test was found on the set of computerized reading items. Researchers concluded that there was no evidence of adverse effects on the computer‐based TOEFL performance due to lack of prior computer experience.
COMPUTER ANALYSIS OF ESSAY CONTENT FOR AUTOMATED SCORE PREDICTION: A PROTOTYPE AUTOMATED SCORING SYSTEM FOR GMAT ANALYTICAL WRITING ASSESSMENT ESSAYS
Authors: Burstein Jill , Braden‐Harder Lisa , Chodorow Martin , Hua Shuyi , Kaplan Bruce , Kukich Karen , Lu Chi , Nolan James , Rock Don , Wolff Susanne
Published: August 2014

This report discusses the development and evaluation of a research prototype system designed to automatically score essay responses to the GMAT Analytical Writing Assessments: (a). Analysis of an Argument (Argument essays) and (b). Analysis of an Issue (Issue essays) item types. The system, Electronic Essay Rater (e‐rater), was designed to automatically analyze several features of an essay and score the essay based on the features of writing as specified in holistic rubrics. E‐rater uses a hybrid feature methodology. It incorporates several variables that are derived statistically, extracted through NLP techniques, or achieved by simple “counting” procedures. The version of the e‐rater described in this report uses five sets of critical feature variables to build the final linear regression model used for predicting scores. The same set of critical variables was used to fit models for the issue and argument training essays and the following results were achieved. For the set of 275 cross‐validation data, exact or adjacent agreement with human rater scores reached 95%. For the 282 cross‐validation issue essays exact or adjacent agreement with human rater scores achieved 93%. The rich feature variables used as score predictors in e‐rater could potentially be used to generate explanation of score predictions, and diagnostic and instructional information.