bluuebunny/crossref_metadata_2025 huggingface dataset:
PROFICIENCY SCALING BASED ON CONDITIONAL PROBABILITY FUNCTIONS FOR ATTRIBUTES
Authors: Tatsuoka Kikumi K. , Birenbaum Menucha , Lewis Charles , Sheehan Kathleen M.
Published: August 2014

This study introduces procedures for constructing a proficiency scale for a large‐scale test by applying Tatsuoka's Rule Space Model. The SAT Mathematics (SAT M), Section 2, is used for illustrating the process and the results. A task analysis is summarized in a mapping sentence, and then 14 processes and content attributes are identified for explaining the underlying cognitive aspects of the examinees' performance on the SAT M. Analysis results show that almost 98% of 2334 examinees are successfully classified into one of 468 cognitive states. The cognitive states are characterized by mastery or non‐mastery of the 14 attributes. Attribute Characteristic Curves, which are conditional probability functions defined on the SAT Scale, are introduced and used for interpreting an examinees' proficiency. Prototypes of a student's performance report and a group performance report are given as examples of possible ways for summarizing the analysis results.
RELIABILITY OF ESSAY RATING AND SCORE ADJUSTMENT
Authors: Longford N. T.
Published: August 2014

A model‐based approach to rater reliability for essays read by multiple readers is presented. Variation of rater severity (between‐rater variation) and rater inconsistency (within‐rater variation) is considered in presence of between‐examinee variation. An additive variance component model is posited and the method of moments for its estimation described. The model involves no distributional assumptions. Minimum mean squared error estimators of examinees' true scores and readers' severities are derived. Model diagnostic procedures are an integral component of the approach. The methods are illustrated on data from standardized educational tests.
DERIVING COMPARABLE SCORES FOR COMPUTER ADAPTIVE AND CONVENTIONAL TESTS: AN EXAMPLE USING THE SAT1,2
Authors: Eignor Daniel R.
Published: August 2014

Procedures used to establish the comparability of scores derived from the College Board Admissions Testing Program (ATP) computer adaptive SAT prototype and the paper‐and‐pencil SAT are described in this report. Both the prototype, which is made up of Verbal and Mathematical computer adaptive tests (CATs), and a form of the paper‐and‐pencil test were administered to just greater than 500 examinees using a random groups counterbalanced design. Both linear and equipercentile procedures were used for equating in each of the separate testing orders (paper‐and‐pencil then CAT or CAT then paper‐and‐pencil). Data were not pooled across the orders because the groups were not randomly equivalent due to administrative problems. The linear procedure was chosen for each test (Verbal or Mathematical) for each order, and results from the two orders were averaged. The final Verbal and Mathematical CAT conversions were quite similar to the paper‐and‐pencil conversions, although the two conversions for Verbal and two conversions for Mathematical did differ by as much as 20 scaled score points in certain regions of the scale.
CASE STUDIES IN COMPUTER ADAPTIVE TEST DESIGN THROUGH SIMULATION1,2
Authors: Eignor Daniel R. , Stocking Martha L. , Way Walter D. , Steffen Manfred
Published: August 2014

The extensive computer simulation work done in developing the computer adaptive versions of the Graduate Record Examinations (GRE) Board General Test and the College Board Admissions Testing Program (ATP) SAT is described in this report. Both the GRE General and SAT computer adaptive tests (CATs), which are fixed length in nature, were developed from pools of items that were calibrated using the three‐parameter logistic IRT model and item selection was based on the recently developed weighted deviations algorithm (see Swanson and Stocking, 1992), which simultaneously deals with content, statistical, and other constraints in the item selection process. For the GRE General CATs (Verbal, Quantitative, and Analytical), item exposure was controlled by using an extension of an approach originally developed by Sympson and Hetter (1988). For the SAT CATs (Verbal and Mathematical), item exposure was controlled by using a less complex randomization approach. Lengths of the CATs were determined so that CAT reliabilities matched or exceeded comparable full length paper‐and‐pencil test reliabilities.
PASSAGE DEPENDENCE OF THE NEW SAT® READING COMPREHENSION QUESTIONS
Authors: Powers Donald E. , Wilson Susan T.
Published: August 2014

It has been reasonably well established that test takers can, to varying degrees, answer some reading comprehension questions correctly without reading the passages on which the questions are based. This is true even for carefully constructed measures such as the College Board's SAT. As a result, the use of reading scores, including those from the SAT, as valid indicators of reading comprehension has been challenged.

The major aim of this study was to determine the strategies by which examinees are able to achieve better‐than‐chance performances without reading the passages. The focus of the research was a sample of reading comprehension questions similar to those that will be used in the revised SAT, to be introduced in 1994. Sets of reading comprehension questions were administered, without the passages, to a sample of verbally able students in eight secondary schools across the country. After attempting the task, these students were asked to complete a questionnaire describing the strategies they had used. The most often cited strategies involved choosing answers on the basis of consistency with other questions and reconstructing the main theme of a missing passage from all the questions and answers in a set. These strategies were also more likely to result in successful performance on individual test items than were any of the many other possible (and less construct‐relevant) strategies. Implications are discussed with regard to the construct validity of the new SAT reading comprehension questions and the advice that should be given to prospective SAT takers.
THREE PRACTICAL ISSUES FOR MODERN ADAPTIVE TESTING ITEM POOLS1
Authors: Stocking Martha L.
Published: August 2014

As adaptive testing moves towards operational implementation in large scale testing programs, where it is important that adaptive tests be as parallel as possible to existing linear tests, a number of practical issues arise. This paper concerns three such issues. First, optimum item pool size is difficult to determine in advance of pool construction. Retrospective results are analyzed for five operational pools; these analyses indicate that item pools of the size and quality of six to eight linear tests are adequate to support adaptive tests of roughly half the length of a parallel linear test. Second, item pools may not support sufficiently low exposure rates for items or a sufficiently small amount of test overlap to maintain test security when testing is conducted on a continuous basis. Various simulations suggest that multiple pools which can be chosen randomly before adaptive testing begins provide a satisfactory solution. Finally, over time it will be necessary to refresh or replace operational item pools. Issues that must be considered in refreshing or replacing item pools are discussed, and guidance is given for establishing benchmark values from initial pools as standards that must be met for adaptive tests from new pools to be considered parallel to adaptive tests from existing pools.
ASSESSING THE CLASSROOM PERFORMANCE OF BEGINNING TEACHERS: TEACHERS' JUDGMENTS OF EVALUATION CRITERIA
Authors: Wesley Scott , Rosenfeld Michael , Sims‐Gunzenhauser Alice
Published: August 2014

A central component of Praxis III: Classroom Performance Assessments is a set of proposed criteria that have been developed to assess the classroom performance of beginning teachers. This study surveyed 678 teachers from all grade levels to obtain their perceptions of the importance of these criteria. It was a follow‐up of Powers (1992). The results support his finding that the aspects of teaching embodied in the criteria are important for the competent performance of beginning teachers. These two studies substantiate the relevance of the criteria for assessing beginning teacher classroom performance.
ON EXAMINEE CHOICE IN EDUCATIONAL TESTING
Authors: Wainer Howard , Thissen David
Published: August 2014

Throughout our society, individuals are compared on summaries of diverse measures. Often the measures are neither the same for all competitors nor selected randomly from a universe of measures. In fact, comparisons often are based on a mixture of measures in which the competitors choose some or all of the elements of the mixture; applicants to colleges choose some of the courses they take in high school, which extracurricular activities they participate in, and (to some extent) which entrance exams they take. What are the consequences of any selection procedure, if such choices are allowed? How can the adverse consequences be minimized? This article summarizes results from tests that have allowed examinee choice heretofore and provides a basis for a fuller discussion of these issues.
EFFECT OF RASCH CALIBRATION ON ABILITY AND DIF ESTIMATION IN COMPUTER‐ADAPTIVE TESTS
Authors: Zwick Rebecca , Thayer Dorothy T. , Wingersky Marilyn
Published: August 2014

A simulation study of methods of assessing differential item functioning (DIF) in computer‐adaptive tests (CATs) was conducted by Zwick, Thayer and Wingersky (in press; 1993). Results showed that modified versions of the Mantel‐Haenszel and standardization methods work well with CAT data. In that study, data were generated using the three‐parameter logistic (3PL) model and this same model was assumed in obtaining item parameter estimates. In the current study, 3PL item response data were used, but the Rasch model was assumed in obtaining item parameter estimates, which, in turn, determined the information table to be used in the item selection algorithm. New Rasch‐based expected true scores were obtained for each examinee, based on responses to the CAT items. As in the previous study, the DIF statistics were highly correlated with the generating DIF, and the means and standard deviations of these statistics across items were close to their nominal values. There was, however, a tendency for DIF statistics to be slightly smaller in magnitude than in the 3PL analysis, resulting in a lower probability of detecting items with extreme DIF. This reduced sensitivity appeared to be related to a degradation in the accuracy of matching. Expected true scores from the Rasch‐based CAT tended to be biased downward, particularly for lower‐ability examinees. Unlike the Rasch CAT scores, Rasch expected true scores based on nonadaptive administration of all pool items behaved quite well, as did the nonadaptive and CAT‐based expected true scores obtained using the 3PL model.
DIF ANALYSIS FOR PRETEST ITEMS IN COMPUTER‐ADAPTIVE TESTING
Authors: Zwick Rebecca , Thayer Dorothy T. , Wingersky Marilyn
Published: August 2014

A simulation study of methods of assessing differential item functioning (DIF) in computer‐adaptive tests (CATs) was conducted by Zwick, Thayer and Wingersky (in press; 1993). Results showed that modified versions of the Mantel‐Haenszel and standardization methods work well with CAT data. DIF methods were also investigated for nonadaptive “pretest” items, for which item parameter estimates were assumed unavailable. The pretest DIF statistics were generally well‐behaved, but the Mantel‐Haenszel DIF statistics tended to have larger standard errors for the pretest items than for the CAT items. The current extension of the earlier work addressed the effect of using alternative matching methods for pretest items. Using a more elegant matching procedure did not lead to a reduction of the Mantel‐Haenszel standard errors and produced DIF measures that were nearly identical to those from the earlier study. Further investigation showed that the Mantel‐Haenszel standard errors tended to be larger when items were administered to examinees with a wide ability range, whereas the opposite was true of the standard errors of the standardization DIF statistic. Some theoretical findings were obtained that appear to explain this phenomenon.
COMPUTERIZED MASTERY TESTING USING FUZZY SET DECISION THEORY
Authors: Du Yi , Lewis Charles , Pashley Peter J.
Published: August 2014

Lewis and Sheehan (1990) developed a computerized sequential mastery test procedure that utilizes Bayesian decision theory to make “master,” “nonmaster,” or “continue testing” decisions. While maintaining their general framework of administering sequential testlets, a fuzzy set approach was used to develop an alternative computerized mastery test. This new procedure differs from Lewis and Sheehan's in that the decision rule is determined using fuzzy set decision theory, and ability estimates are obtained using the Rasch model rather than a three‐parameter logistic model. This article describes this new approach and illustrates the differences between the fuzzy set and Bayesian methods by way of an example.
AN INVESTIGATION OF IRT‐BASED ASSEMBLY OF THE TOEFL TEST
Authors: Chyn Susan , Tang K. Linda , Way Walter D.
Published: August 2014

The current study, carried out jointly by Test Development and Statistical Analysis staff at Educational Testing Service (ETS), investigated the feasibility of the Automated Item Selection (AIS) procedure for the Test of English as a Foreign Language (TOEFL). IRT‐based statistical specifications were developed. Two TOEFL® test forms were assembled using AIS and statistical and content‐related properties were evaluated. The results of this study show that the statistical consistency (parallelism) of the tests assembled using AIS would appear to be superior to the consistency of tests assembled using traditional test‐assembly procedures. The results of the study also provided strong evidence that AIS‐assembled TOEFL tests can successfully meet the IRT‐based specifications. Test Development staff observed visible gains in efficiency in item selection for Section 1 and Section 2 and the potential for time gains in Section 3. The implications of AIS for pool management were explored.
PASS‐FAIL RELIABILITY FOR TESTS WITH CUT SCORES: A SIMPLIFIED METHOD
Authors: Breyer F. Jay , Lewis Charles
Published: August 2014

A single‐administration classification reliability index is described that estimates the probability of consistently classifying examinees to mastery or nonmastery states as if those examinees had been tested with two alternate forms. The procedure is applicable to any test used for classification purposes, subdividing that test into two half‐tests, each with a cut score, where the sum of the two half‐test cut scores is equal to the cut score for the total test. The application of this pass‐fail consistency index to binary scored objective tests, nonbinary scored performance tests, and tests containing both binary and nonbinary scored questions is presented. A calculation example is provided together with look‐up tables.
RELATIONSHIPS BETWEEN DIFFERENTIAL PERFORMANCE ON MULTIPLE‐CHOICE AND ESSAY SECTIONS OF SELECTED AP® EXAMS AND MEASURES OF PERFORMANCE IN HIGH SCHOOL AND COLLEGE
Authors: Bridgeman Brent , Morgan Rick
Published: August 2014

Students with high scores (top third) on the essay portion of an Advanced Placement Examination and low scores (bottom third) on the multiple‐choice portion of the same examination were compared with students whose performance showed the opposite pattern (top third on the multiple‐choice questions and bottom third on the essay questions). Across examinations in different subject areas (history, English, and biology), students who were relatively strong in the essay format and weak in the multiple‐choice format were about as successful in their college courses as students whose performance showed the opposite pattern, especially in those courses where grades are typically not determined by multiple‐choice tests. Students who scored high on the multiple‐choice portion and low on the essay portion performed relatively well on other multiple‐choice tests, especially the verbal section of the SAT. Across several ethnic/racial groups, males tended to receive relatively high scores on the multiple‐choice portion of the AP United States History Examination while females received higher scores on the essays than on the multiple‐choice questions. Among females whose best language was not English, scores were substantially higher on the essay portion of the history examination; among males in this group, scores were slightly higher for the multiple‐choice questions. Because the population of students who take Advanced Placement Examinations is exceptionally able, generalizations to less able populations are not warranted.
AN INVESTIGATION OF PROPOSED REVISIONS TO SECTION 3 OF THE TOEFL TEST
Authors: Schedl Mary A. , Thomas Neal , Way Walter D.
Published: August 2014

The TOEFL testing program is currently exploring a change in Section 3 of the TOEFL® test that would replace the vocabulary subpart with additional reading comprehension questions. This change has been proposed by internal test development specialists and is supported by external experts in the field of English as a second language. The purpose of this study was to investigate the proposed revision to Section 3 in terms of the length and timing that would be necessary to address concerns of test speededness of the section. The study was carried out using an experimental design with test length and testing time defined as independent variables, and examinee test performance defined as the dependent variable. In addition, several psychometric issues relating to the proposed revision to Section 3 were investigated as part of the study.

The results of the study supported the implementation of a revised TOEFL Section 3 consisting of five reading passages with a total of 50 items. The results of the study also suggested that a total testing time of no less than 55 minutes should be allowed for the revised TOEFL Section 3. Additional psychometric analyses indicated that the current TOEFL score scale can be maintained with the revised Section 3, and that the proposed revisions will not appreciably affect the reliability and validity of Section 3 of the TOEFL test.
EQUITY ASSESSMENT FOR POLYTOMOUSLY SCORED ITEMS: A TAXONOMY OF PROCEDURES FOR ASSESSING DIFFERENTIAL ITEM FUNCTIONING1
Authors: Dorans Neil J. , Potenza Maria T.
Published: August 2014

Educational reform efforts have led to increased use of alternatives to the traditional binary–scored multiple choice item. Many of the stimuli employed by these alternative assessments yield complex responses that require complex scoring rules. Some of these new item types can be polytomously–scored. Differential item functioning (DIF) assessment is a form of equity assessment. DIF assessment attempts to identify items for which subpopulations of examinees exhibit performance differentials that are inconsistent with the performance differentials typically seen for those subpopulations on collections of items that purport to measure a common construct. Any DIF technique can be evaluated in terms of how well it meets certain statistical and practical criteria. These DIF assessment criteria need to be attended to before we can conclude that the items associated with alternative forms of assessment can be adequately tested for DIF. DIF methodology is well–defined for traditional, binary–scored multiple–choice items. This paper provides a classification scheme of DIF procedures for binary–scored items that is applicable to new DIF procedures for polytomously–scored items. In the process, a formal development of a polytomous version of a binary DIF technique is presented. Finally, several polytomous DIF techniques are evaluated in terms of statistical and practical criteria.
A CASE FOR ADJUSTING SUBJECTIVELY RATED SCORES IN THE ADVANCED PLACEMENT TESTS*
Authors: Longford Nicholas T.
Published: August 2014

The purpose of this report is to present a case for adjusting the scores for free response items in the Advanced Placement tests. Using information about the rating process from the reliability studies, administrations of the Advanced Placement test for three subject areas are analyzed, and it is shown that the minimum squared error score adjustments proposed by Longford (1993) result in changed AP grades for an appreciable percentage of the examinees.
ALTERNATIVE MODES OF ASSESSMENT, UNIFORM STANDARDS OF VALIDITY1
Authors: Messick Samuel
Published: August 2014

In contrast to multiple choice, alternative modes of assessment afford varying degrees of openness in the allowable responses. Prominent among the alternatives is the assessment of performance, sometimes in its own right where the issue is the quality of the particular performance per se, but more often as a vehicle for the assessment of knowledge, skill, or other attributes. Because inferences about score meaning in construct terms and about the action implications of that meaning are fundamentally similar in the alternative assessment modes (despite surface differences), the same standards of validity apply to all educational and psychological measurement. These standards are addressed in terms of content, substantive, structural, generalizability, external, and consequential aspects of construct validity.
ESTIMATING THE EFFECTS OF TEST LENGTH AND TEST TIME ON PARAMETER ESTIMATION USING THE HYBRID MODEL
Authors: Yamamoto Kentaro
Published: August 2014

When individuals perform tasks, they differ from each other not only in their ability to perform the tasks correctly, but also in their speed. Even though the traditional indicator of test speededness, missing responses, clearly indicates a lack of time to respond (thereby indicating the speededness of the test), it is inadequate for evaluating speededness in a multiple‐choice test scored as number correct and underestimates test speededness. Conventional IRT parameter estimation ignores the mixture of random responses during calibration; consequently, estimated parameters are biased.

The HYBRID model (Yamamoto, 1989) was extended (Yamamoto, 1990) to characterize when each examinee switches from an ability‐based response strategy to a strategy of responding randomly. The model has allowed us to evaluate test speededness by estimating the proportions of examinees who switch strategies at any possible point in the test. The estimated IRT parameters based on the HYBRID model were more accurate than the ordinary IRT‐only analysis.

With the extended HYBRID model applied to the data taken from an experimental form of TOEFL®, we found that 1) the test length had a small impact on the proportion of the examinees affected by the speededness of the test, 2) a greater proportion of examinees were affected by speededness of a test with a 50‐minute time limit than a test with a 55‐ or 60‐minute time limit, and 3) the difference in the proportions of examinees affected by speededness of tests under 55‐ and 60‐minute time limits was small. However, nearly 20 percent of the examinees were affected by speededness after completing 80 percent of the test. In other words, the last 20 percent of the responses of 20 percent of the examinees did not represent their true ability.
APPLICATION OF THE MANTEL‐HAENSZEL PROCEDURE TO COMPLEX SAMPLES OF ITEMS
Authors: Allen Nancy L. , Donoghue John R.
Published: August 2014

This Monte Carlo study examined the effect of complex sampling of items on the measurement of differential item functioning (DIF) using the Mantel‐Haenszel procedure. Data were generated using a 3PL IRT model according to the balanced incomplete block design used in the National Assessment of Educational Progress (NAEP). The length of each block of items and the number of DIF items in the matching variable were varied, as was the difficulty, discrimination, and presence of DIF in the studied item. Block, booklet, pooled booklet and extra‐information analyses were compared to a complete data analysis using the transformed log‐odds on the delta scale. The pooled booklet approach is recommended for use when items are selected for examinees according to a BIB design. This study has implications for DIF analyses of other complex samples of items, such as computer administered testing or another complex assessment design.
A COMPARISON OF THE PERFORMANCE OF GRADUATE AND UNDERGRADUATE SCHOOL APPLICANTS ON THE TEST OF WRITTEN ENGLISH
Authors: Zwick Rebecca , Thayer Dorothy T.
Published: August 2014

The performance of graduate and undergraduate school applicants on the Test of Written English (TWE®) was compared for each of 66 data sets, dating from 1988 to 1993. The analyses compared the average TWE score for graduates and undergraduates after matching examinees on the TOEFL® total score. The main finding was that, for matched examinees, undergraduate TWE means were higher than graduate means in 63 of the 66 data sets. Although these standardized mean differences (SMDs) never exceeded 0.3 of a TWE score point (with standard errors that were typically between 0.01 and 0.02), the results are noteworthy because they give a different picture than do simple comparisons of means for unmatched graduates and undergraduates, which show higher mean TWE scores for graduate applicants in the majority of cases. Of the nine SMDs exceeding 0.2, eight were in Region 1 (Asia and the Pacific) between October 1988 and May 1992. In evaluating these findings, the effects of the examinees' intended field of graduate study were investigated. For groups matched on TOEFL score, applicants to graduate programs in the physical and biological sciences tended to have lower TWE means than undergraduates and graduates in the social sciences. Graduate students in the physical sciences tended to have lower TWE means than matched graduates in the humanities. Graduate‐undergraduate differences in TWE performance may result in part from a greater concentration of graduate applicants in scientific fields. Another hypothesis is that undergraduates are more likely to have recently participated in intense English writing instruction. Region 1 may show larger graduate‐undergraduate differences because of greater demographic disparities in that region between these two groups of examinees.
MODELING THE MIXTURE OF IRT AND PATTERN RESPONSES BY A MODIFIED HYBRID MODEL1
Authors: Yamamoto Kentaro , Everson Howard T.
Published: August 2014

This study demonstrates the utility of a HYBRID psychometric model, which incorporates both item response theoretic and latent class models, for detecting test speededness. The model isolates where in a sequence of test items examinee response patterns shift from ones providing reasonable estimates of ability to those best characterized by a random response pattern. The study applied the HYBRID model to three distinct data sets: (1) simulated data representing the performance of 3,000 examinees on a 70‐item test; (2) data from a statewide field test (n=5997 of a 40 item reading comprehension test with a fixed time limits; and (3) data from a study of urban university students (n=752) who took a similar 40‐item reading comprehension test under varying time limits. The HYBRID model successfully identified “switch points” in examinee's item response patterns in all three data sets. This paper discusses applications of the model used to detect speededness and to provide adjusted estimates of item parameters and examinee abilities.
GENDER DIFFERENCES IN COLLEGE MATHEMATICS GRADES AND SAT‐M SCORES: A REANALYSIS OF WAINER AND STEINBERG
Authors: Bridgeman Brent , Lewis Charles
Published: August 2014

Wainer and Steinberg (1992) showed that within broad categories of first‐year college mathematics courses (e.g., calculus) men had substantially higher average scores on the mathematics section of the SAT (SAT‐M) than women who earned the same letter grade. In calculus courses, the median difference over the five letter grades was about 38 points on the 200‐800 SAT‐M scale. However, three aspects of their analyses may lead to unwarranted conclusions. First, they focused primarily on differences in SAT‐M scores given course grades when the more important question for admissions officers is the difference in course grades given scores on the predictor. Second, they failed to account for differences among calculus courses. Because different calculus courses may be differentially selected by men and women (e.g., calculus for engineers vs calculus for liberal arts students), and because these courses may have different grading standards, the way Wainer and Steinberg aggregated information without regard to the specific course taken could exaggerate gender differences. The reanalysis, taking distinctions among courses into consideration, suggests that their estimates were too large by about 10 points in calculus courses, although their estimates for pre‐calculus courses were confirmed. Most important, Wainer and Steinberg focused on the use of SAT‐M as an isolated indicator; such use is contrary to professional recommendations. The reanalysis indicated that a more appropriate composite indicator made up of both SAT‐M and high school grade point average demonstrated minuscule gender differences for both calculus and pre‐calculus courses.
CONTROLLING ITEM EXPOSURE CONDITIONAL ON ABILITY IN COMPUTERIZED ADAPTIVE TESTING
Authors: Stocking Martha L. , Lewis Charles
Published: August 2014

The interest in the application of large‐scale adaptive testing for secure tests has served to focus attention on issues that arise when theoretical advances are made operational. Many such issues have more to do with changes in testing conditions rather than testing paradigms. One such issue is that of insuring item and pool security in the continuous testing environment made possible by the computerized administration of a test, as opposed to the more periodic testing environment typically used for linear paper‐and‐pencil tests. In the continuous testing environment of adaptive testing, methods have been developed in the past to use the computer to control the rate at which particular items are exposed to test‐takers. These methods have typically employed randomization schemes, sometimes in reference to a particular target distribution of test‐taker ability. This paper presents a new method of controlling the exposure rate of items conditional on the ability level of an individual test‐taker. The properties of such conditional control on the exposure rates of items, when used in conjunction with a particular adaptive testing algorithm, are explored through five studies with simulated data.
PERSISTENCE IN SCIENCE OF HIGH‐ABILITY MINORITY STUDENTS, PHASE V: COMPREHENSIVE DATA ANALYSIS
Authors: Grandy Jerilee
Published: August 2014

Concern over the underrepresentation of minorities in mathematics, science, and engineering led to this longitudinal study begun by Thomas Hilton in 1986 under a grant from the National Science Foundation. The purpose of the study was to investigate why some high‐ability minority students follow through with their plans to become scientists or engineers, while others with the same plans do not.

Data came from three sources: (1) 1985 SAT files of a sample of minority students planning to major in math, science, and engineering and scoring above 550 on the SAT mathematics test, (2) a detailed survey questionnaire completed in 1987, and (3) a status survey in 1990.

Hilton reported results of the first four phases of the project, including multiple regression analyses. This report of Phase V describes the results of causal modeling using LISREL to determine the direct and indirect effects of gender, SES, high school variables, and college variables on student status five years after high school.

Most important to persistence in science and engineering were the type of college attended (2‐year or 4‐year), minority support systems early in college, and commitment to science or engineering by the sophomore year. The role of other variables and their effects on commitment and persistence are discussed in technical detail. Important gender differences are also discussed. The report concludes with a list of practical recommendations for counselors and heads of intervention programs both at the high school and college levels.
AN ANALYSIS OF FACTORS AFFECTING THE DIFFICULTY OF DIALOGUE ITEMS IN TOEFL LISTENING COMPREHENSION
Authors: Nissan Susan , DeVincenzi Felicia , Tang K. Linda
Published: August 2014

One of the item types in the Listening Comprehension Section of the TOEFL® test is the Dialogue. Because the Dialogue item pool needs to have an appropriate balance of items at a range of difficulty levels, test developers have examined items at various difficulty levels in an attempt to identify their features. In this study, the authors created a classification system for certain item features, classified a sample of the current Dialogue item pool, and conducted data analyses in an attempt to characterize the features of easy and difficult Dialogue items. They results of the analyses indicate that, of the features studied, five were significant: the presence of infrequent oral vocabulary, the sentence pattern of the utterances in the stimulus, the presence of negatives in the stimulus, the necessity of making an inference to answer the item, and the roles of the speakers in the stimulus.
SMOOTHED STANDARDIZATION ASSESSMENT OF TESTLET LEVEL DIF ON A MATH FREE‐RESPONSE ITEM TYPE1
Authors: Lyu C. Felicia , Dorans Neil J. , Ramsay James O.
Published: August 2014

A smoothed version of standardization, which merges kernel smoothing with the traditional standardization DIF approach, was used to examine DIF for student produced response (SPR) items on the SAT I Math at both the item and testlet levels. This nonparametric technique avoids model misspecification problems. It also has less sampling errors because of smoothing. Results from the smoothed item‐level DIF analysis showed that regular multiple choice items have more variability in DIF values than SPRs. The testlet DIF analysis indicated that White examinees may exceed comparable African American examinees by 14‐19 scaled score units on average. Differences between the smoothed standardization and the traditional standardization are small and most likely due to the use of the true score as a matching variable in the smoothed standardization approach.
A STUDY OF WRITING TASKS ASSIGNED IN ACADEMIC DEGREE PROGRAMS
Authors: Hale Gordon , Taylor Carol , Bridgeman Brent , Carson Joan , Kroll Barbara , Kantor Robert
Published: August 2014

Writing tasks assigned in 162 undergraduate and graduate courses in several disciplines at eight universities were collected. Using a sample of the assignments, key dimensions of difference were identified, and a classification scheme based on those dimensions was developed. Application of the classification scheme provided data on the prevalence of various types of assignments and, for essay tasks, showed the degree to which the assignments were characterized by each of several features. Differences in the kinds of writing tasks assigned in different groups of disciplines were examined.
STANDARD ERRORS OF THE KERNEL EQUATING METHODS UNDER THE COMMON‐ITEM DESIGN
Authors: Liou Michelle , Cheng Philip E. , Johnson Eugene G.
Published: August 2014

The present research derives simplified formulae for computing the standard error of the frequency estimation method for equating score distributions that are continuized using a uniform or Gaussian kernel function (Holland, King & Thayer, 1989; Holland & Thayer, 1987). The simplified formulae are applicable to equating both the observed‐ and smoothed‐score distributions (Rosenbaum & Thayer, 1987). Results from two empirical studies indicate that the simplified formulae work reasonably well for samples with moderate sizes, say one thousand examinees.
THE GRADED UNFOLDING MODEL: A UNIDIMENSIONAL ITEM RESPONSE MODEL FOR UNFOLDING GRADED RESPONSES
Authors: Roberts James S. , Laughlin James E.
Published: August 2014

Binary or graded disagree‐agree responses to attitude items are often collected for the purpose of attitude measurement. Although such data are sometimes analyzed with cumulative measurement models, recent investigations suggest that unfolding models are more appropriate (Roberts, 1995; Van Schuur & Kiers, 1994). Advances in item response theory (IRT) have led to the development of several parametric unfolding models for binary data (Andrich, 1988; Andrich & Luo, 1993; Hoijtink, 1991), but IRT models for unfolding graded responses have not been addressed in the psychometric literature. A parametric IRT model for unfolding either binary or graded responses was developed in this study. The model, called the graded unfolding model (GUM), is a generalization of Andrich & Luo's (1993) hyperbolic cosine model for binary data. A joint maximum likelihood procedure was implemented to estimate GUM parameters, and a subsequent recovery simulation showed that reasonably accurate estimates could be obtained with minimal data demands (e.g., as few as 100 subjects and 15 to 20 6‐category items). The applicability of the GUM to common attitude testing situations was illustrated with real data on student attitudes toward capital punishment. Index terms: attitude measurement, graded unfolding model, hyperbolic cosine model, ideal point process, item response theory, Likert scale, Thurstone scale, unidimensional scaling, unfolding model.
BRIDGING COGNITION AND PERSONALITY IN EDUCATION: THE ROLE OF STYLE IN PERFORMANCE AND DEVELOPMENT
Authors: Messick Samuel
Published: August 2014

In an attempt to illuminate the interrelatedness of noncognitive and cognitive domains – or, more pointedly, of affection, conation, and cognition – emphasis was placed on variables that not only operate across these domains but are also potentially integrative. Prominent among such variables are personal styles, particularly cognitive styles. The properties and problems of cognitive styles are examined, with special emphasis on field independence versus field sensitivity and on two stylistic dimensions of attentional scanning. The role of cognitive styles as both competence variables and performance variables is addressed, along with the difficulty of disentangling style from ability because of reciprocal determinism in their development. The educational implications of cognitive styles are explored, especially as they bear on the problem of the match between student characteristics and educational experiences and on the value‐laden nature of style‐based pedagogical decisions.
DIFFERENCES IN THE SURVEY RESPONSES OF ASIAN AMERICAN AND WHITE SCIENCE AND ENGINEERING STUDENTS
Authors: Grandy Jerilee
Published: August 2014

A recent survey of gender and ethnic differences among science and engineering majors suggested that Asian American students were less content with their undergraduate experiences and with their field of study than were White students or students of other ethnic groups. Asian Americans appeared to enjoy their studies less and to feel more pressured by family to continue their science or engineering studies. Their responses indicated that they were less encouraged by professors to continue in their field, and they gave lower ratings to their relationships with professors. This study reports the results of additional analyses aimed at better understanding the responses of Asian Americans.

Results of the additional analyses indicated that Asian American students tended to endorse the middle response on a 5‐point Likert scale more than White students did, and to avoid extreme responses. This response style accounted, in part, for the lower ratings Asian Americans gave to their undergraduate experiences. When items were rescored, collapsing the 5‐point scale to a 3‐point scale, Asian American ratings were still lower. These results indicated that there was at least one type of Asian American response set affecting the comparison of Asian Americans with Whites. Furthermore, Asian Americans were either less content with their studies, or, the tendency to respond more neutrally to questions of judgment was another type of response set.

A hypothesis was posed that students in large research universities would be less content than students in other institutions, and that because a greater proportion of Asian Americans than Whites attend large research universities, the relative discontent of Asian Americans might be attributable to the type of institution attended. This hypothesis was not supported by the data.

Response styles that are unique to specific ethnic groups may lead researcher's to draw erroneous conclusions from survey data. A number of recommendations are made for further research to determine how general the Asian American response set is, and to attempt to account for the lower satisfaction ratings of Asian Americans even when a correction is made for a response set.
OPTIMAL DESIGN OF ITEM POOLS FOR COMPUTERIZED ADAPTIVE TESTS
Authors: Stocking Martha L. , Swanson Len
Published: August 2014

Methods of optimal test assembly have served in the past as the foundation for the development of methods of assembling adaptive tests on‐the‐fly, A particular model and heuristic that facilitate the assembly of adaptive tests of the same complex structure as linear tests have already been developed (Swanson & Stocking, 1993; Stocking & Swanson, 1993). Similar methods can be used in the assembly of item pools designed to support adaptive testing. In the continuous testing environment of adaptive testing, optimal item pool design must simultaneously take into account the competing goals of reducing item exposure to enhance item security and maximizing this exposure to make the greatest possible use of every item. An extension of previous methods of optimal design to the issues of item pool design in the context of adaptive testing is developed in this paper, with examples.
DESCRIBING AND CATEGORIZING DIF IN POLYTOMOUS ITEMS
Authors: Zwick Rebecca , Thayer Dorothy T. , Mazzeo John
Published: August 2014

The purpose of this project was to evaluate statistical procedures for assessing differential item functioning (DIF) in polytomous items (items with more than two score categories). Three descriptive statistics—the Standardized Mean Difference, or SMD (Dorans & Schmitt, 1991), and two procedures based on SIBTEST (Shealy & Stout, 1993) were considered, along with five inferential procedures—two based on SMD, two based on SIBTEST, and the Mantel (1963) method. The DIF procedures were evaluated through applications to simulated data, as well as data from ETS tests.

The simulation included conditions in which the two groups of examinees had the same ability distribution and conditions in which the group means differed by one standard deviation. When the two groups had the same distribution, the descriptive index that performed best was the SMD. When the two groups had different distributions, a modified form of the SIBTEST DIF effect size measure tended to perform best. The five inferential procedures performed almost indistinguishably when the two groups had identical distributions. When the two groups had different distributions and the studied item was highly discriminating, the SIBTEST procedures showed much better Type I error control than did the SMD and Mantel methods, particularly in short tests. The power ranking of the five procedures was inconsistent; it depended on the direction of DIF and other factors.

Routine application of these polytomous DIF methods at ETS seems feasible in cases where a reliable test is available for matching examinees. For the Mantel and SMD methods, Type I error control may be a concern under certain conditions. In the case of SIBTEST, the current version cannot easily accommodate matching tests that do not use number‐right scoring. Additional research in these areas is likely to be useful.
WORK VALUES OF COLLEGE BOUND AND OTHER HIGH SCHOOL STUDENTS
Authors: Stricker Lawrence J.
Published: August 2014

This study identified the salient work values of college bound and noncollege bound high school students from their perceptions of occupations. Multidimensional scaling analyses of the students' judgments about the similarity of occupations extracted three factors in the college bound sample and seven in the noncollege bound sample. The same blue collar dimension appeared in both samples; additional dimensions in the noncollege bound sample were fragments of the broader dimensions in the college bound sample. These results indicate that the perceptions were more differentiated for the noncollege bound students than for the other students, but did not reflect different aspects of occupations, implying that the same work values were salient for both groups.
AN INVESTIGATION OF THE ROLE OF EDUCATIONAL BACKGROUND ON PERFORMANCE ON THE GRE® ANALYTICAL MEASURE
Authors: Enright Mary K. , Bejar Isaac I.
Published: August 2014

There have been a number of reports that educational background is related to performance on reasoning problems similar to those used on the GRE analytical measure. The purpose of this research was to evaluate whether educational background as defined by broad undergraduate major fields was related to differential difficulty for specific items on the GRE analytical measure. An index of differential difficulty, MH‐DDIF, was calculated for the reasoning items from 13 GRE forms. We found only low to moderate levels of MH‐DDIF for the two item types on the analytical measure. A higher proportion of logical reasoning items than analytical reasoning items had moderate MH‐DDIF. The relationship of item classification attributes to MH‐DDIF was evaluated. In addition, other attributes potentially related to MH‐DDIF for logical reasoning items were identified and their ability to predict MH‐DDIF was evaluated with a cross‐validation design. The attributes identified as potentially related to MH‐DDIF for items from one subset of GRE forms did not predict MH‐DDIF for items in a second subset. Thus, a strong link between item performance and educational background was not observed within item types. Because of the failure to cross‐validate the potential predictors of DIF, other factors that contribute to MH‐DDIF are briefly considered. The relationship of these results to other research on the effect of educational background is discussed. We also discuss implications for test design and how test development practices are effective in minimizing the impact of educational background within item types.
AUTOMATIC SCORING OF ADVANCED PLACEMENT BIOLOGY ESSAYS
Authors: Burstein Jill , Kaplan Randy , Wolff Susanne , Lu Chi
Published: August 2014

The Natural Language Processing Initiative (NLPI) at ETS currently does research and development for computer‐assisted scoring of natural language constructed‐responses. The purpose of NLPI's work is to develop computer‐assisted methods for scoring so that computer‐administered natural language constructed‐response items can be integrated into ETS' tests which can be scored efficiently with regard to time and cost. Until recently, NLPI's automated scoring efforts were primarily devoted to the development of computer programs used to score short‐answer responses constructed‐responses of up to 15 words. In previous studies, we have developed scoring systems for main idea item types (e.g., GUIDES items) and divergent thinking item types (e.g., the Formulating‐Hypotheses item (FH)). In the current study, we applied similar techniques to those used to score short‐answer responses in order to implement a prototype to automatically score AP Biology essays. This paper reports on the development process involved in building a linguistically‐based, computer scoring prototype for AP Biology essays and the results of the scoring system. The prototype was used to score essays that had been assigned an Excellent rating (9 or 10 points) or a Poor rating (0 ‐ 3 points) by human graders.1 The results reported in the paper illustrate agreement between human rater scores and computer‐based scores. The results of the study are promising and would encourage continued research in automated scoring of essays with highly analytical rubrics.