bluuebunny/crossref_metadata_2025 huggingface dataset:
DEVELOPMENT AND EVALUATION OF COMPUTER‐ADMINISTERED ANALYTICAL QUESTIONS FOR THE GRADUATE RECORD EXAMINATIONS GENERAL TEST
Authors: Bridgeman Brent , Rock Donald A.
Published: August 2014

Three new computer‐administered items types for the analytical scale of the Graduate Record Examination General Test were developed and evaluated. One item type was a free‐response version of the current analytical reasoning item type. The second item type was a somewhat constrained free‐response version of the pattern identification (or number series) item type in which the student had to state the rule that generated the series. The third item type used the computer to administer yes/no analysis of explanations questions with a limited branching strategy. The computer tests were administered at four ETS regional offices to a sample of students who had previously taken the GRE General Test. Scores from the regular GRE administration and the special computer administration were matched for a sample of 349 students. A number of test administration design issues were identified, including the need to provide adequate practice exercises, design of an interface comfortable for computer‐literate students, and problems with item‐level timing. The pattern identification items were too difficult (or the practice was inadequate), but the other items appeared to function well. There was no evidence that the open‐ended analytical reasoning items were measuring anything beyond what is measured by the current multiple‐choice version of these items.
AN APPROACH TO CLASSIFYING TEACHERS ACCORDING TO THEIR ORIENTATIONS TO TEACHING
Authors: Powers Donald E.
Published: August 2014

A small, representative sample of U.S. teachers was asked to indicate the extent to which their teaching corresponded to each of five orientations to (“models” of) teaching. Respondents were then clustered according to the similarity of their responses. Four categories emerged. A major perspective that distinguished the groups was whether or not they perceived their teaching as corresponding to an “additive model,” i.e., an approach in which content coverage is emphasized. The clusters also differed with respect to subject matter taught and teachers' assessments of the importance of several teaching functions.
DIAGNOSING KNOWLEDGE STATES IN ALGEBRA USING THE RULE SPACE MODEL
Authors: Birenbaum Menucha , Kelly Anthony E. , Tatsuoka Kikumi K.
Published: August 2014

This paper illustrates the use of rule space as a tool to support cognitive analyses of students' mathematical behavior. The rule space approach is explained and is then used to classify students into one of two methods for solving linear algebraic equations in one unknown and to diagnose their knowledge states in this topic. A 32‐item test with open‐ended questions was administered to 231 eighth and ninth graders. The following outcomes of the rule space model were presented: (a) a classification of examinees into knowledge states resulting from the two solution approaches at the group level along with individual examples; (b) tree‐diagrams of the transitional relationships among the states for each strategy. Implications for using the feedback provided by the rule space model in the context of instruction and assessment are discussed.
ADJUSTING COLLEGE GRADE‐POINT AVERAGE FOR VARIATIONS IN GRADING STANDARDS
Authors: Stricker Lawrence J. , Rock Donald A. , Burton Nancy W. , Muraki Eiji , Jirele Thomas J.
Published: August 2014

This study compared the effectiveness of several existing and proposed methods for statistically adjusting college GPAs for course and departmental differences in grading standards, using first‐semester grades from an entire entering class at a large state university. Most of the adjusted GPAs produced by these methods functioned similarly and, despite high correlations with actual GPA, had greater internal‐consistency reliability than actual GPA and were more predictable from SAT scores and high school rank (HSR). Most of the adjusted GPAs also functioned similarly with regard to sex differences in over‐underprediction. The adjusted GPAs and actual GPA exhibited the same small but significant sex differences in over‐underprediction by SAT scores, but the adjusted GPAs displayed smaller differences than actual GPA in over‐underprediction by SAT scores and HSR.
GRAPHICAL IRT‐BASED DIF ANALYSES
Authors: Pashley Peter J.
Published: August 2014

The detection of differential item functioning (DIF) has become an important psychometric research topic in recent years. A number of item response theory (IRT) methods for solving this problem have been suggested. A common approach is to calculate some function of the area between item response curves estimated from the subpopulations of interest. While these methods relay overall item level DIF information, they do not indicate the location and magnitude of DIF along the ability continuum. In order to provide these important details, this paper presents a method for producing simultaneous confidence bands for the difference between item response curves. After these bands have been plotted, the size and regions of DIF are easily identified. Implementation considerations and illustrative examples are also given.
CHOOSING: A TEST1
Authors: Wainer Howard , Thissen David
Published: August 2014

If examinees are permitted to choose to answer a subset of the questions on a test, by knowing only which questions were chosen we can obtain a measure of proficiency that may be as reliable as would have been obtained from the test graded traditionally. This new method of scoring is much less time consuming and expensive for both the examinee and the testing organization. Moreover, because of the decreased response burden we would expect that its use may reduce the nonresponse rate in such low impact educational assessments as the NAEP. We recommend that in assessments that allow the examinee to choose among a set of items, an attempt be made to encourage the examinee to record the item that will be answered before they are actually attempted. In this way we may be able to obtain much of the information that would have been contained in the actual responses, even if the examinee chooses finally not to answer.
INEXPERIENCED AND ANXIOUS COMPUTER USERS: COPING WITH A COMPUTER‐ADMINISTERED TEST OF ACADEMIC SKILLS
Authors: Powers Donald E. , O'Neill Kathleen
Published: August 2014

The objective of this study was to assess the degree to which the mode of administration of the computer‐based Academic Skills Assessments of The Praxis Series: Professional Assessments for Beginning Teachers™ contributes to performance differences among test takers. To make this determination, inexperienced or anxious computer users were recruited to take the assessments. The degree to which test design and test familiarization procedures effectively minimized variation due to comfort and familiarity with computers was examined from three perspectives:

the extent to which the availability of a personal, information‐providing test center supervisor influenced test performances, beyond the help provided by a computerized test familiarization tutorial,

the effect of within‐test practice on later performance on a subsequent section of the test, and

the relationship of computer‐based test performance to attitudes toward computers and experience in using them.

The conclusion was that performance on the tests is not unduly affected by computer administration.
“THIN” VERSUS “THICK” MATCHING IN THE MANTEL‐HAENSZEL PROCEDURE FOR DETECTING DIF
Authors: Donoghue John R. , Allen Nancy L.
Published: August 2014

This Monte Carlo study examined strategies for forming the matching variable for the Mantel‐Haenszel DIF procedure; “thin” matching on total test score was compared to forms of “thick” matching, pooling levels of the matching variable. Data were generated using a three‐parameter item response theory model with common guessing parameter. Number of subjects and test length were manipulated, as were the difficulty, discrimination, and presence/absence of DIF in the studied item. Outcome measures were the transformed log‐odds Δ̂MH, its standard error, and the MH chi‐square statistic.

For short tests (5 or 10 items), thin matching yielded very poor results, with a tendency to falsely identify items as possessing DIF against the reference group. The best methods of thick matching yielded outcome measure values closer to the expected value for non‐DIF items, and a larger value than thin matching when the studied item possessed DIF. Intermediate length tests yielded similar results for thin matching and the best methods of thick matching. The method of thick matching that performed best depended upon the measure used to detect DIF. Both difficulty and discrimination of the studied item were found to have a strong effect on the value of ΔMH.
A DIAGNOSTIC CLASSIFICATION MODEL FOR DOCUMENT PROCESSING SKILLS
Authors: Sheehan Kathleen , Tatsuoka Kikumi , Lewis Charles
Published: August 2014

This paper introduces a modification to the Rule Space diagnostic classification procedure which allows for processing of response vectors containing missing data. Rule Space is an approach to diagnostic classification which involves characterizing examinees' performances in terms of an underlying cognitive model of generalized problem‐solving skills. It has two components: (1) a procedure for determining a comprehensive set of knowledge states, where each state is characterized in terms of a unique subset of mastered skills; and (2) a procedure for classifying examinees into one or another of the specified states. The procedure for determining a comprehensive set of knowledge states is based on the Boolean descriptive function given in Tatsuoka (1991). The procedure for classifying examinees involves comparing examinees' scored response vectors to the patterns expected within each of the specified knowledge states (Tatsuoka, 1983, 1985, and 1987). Missing data is expected to be a common problem for this approach because, although the procedure for determining the comprehensive set of knowledge states requires a large pool of items, the procedure for examinee classification can be performed with smaller (less expensive) item subsets. This approach to diagnostic classification is illustrated with data collected in the Survey of Young Adult Literacy, a nationwide survey of literacy skills conducted by the National Assessment of Educational Progress (NAEP) in 1985.
DIRECT ASSESSMENT, DIRECT VALIDATION? AN EXAMPLE FROM THE ASSESSMENT OF WRITING
Authors: Powers Donald E. , Fowles Mary E. , Willard Ann E.
Published: August 2014

Complex performance assessments are becoming increasingly popular, at least in part because of the directness with which these assessments are accomplished. As Messick (1992) has argued, however, the apparent directness of such measures does not exempt them from the need to marshal validity evidence in support of their use.

An example is given of one kind of evidence – assessment users' satisfaction with the performances generated by examinees – that may contribute to the validation of performance assessments. The example is from the realm of the direct assessment of writing in the context of graduate school admissions.
REVISING SAT®‐VERBAL ITEMS TO ELIMINATE DIFFERENTIAL ITEM FUNCTIONING
Authors: Curley W. Edward , Schmitt Alicia P.
Published: August 2014

Based on initial SAT‐Verbal pretest data and/or hypotheses advanced in the research literature, the authors selected 7 sentence completions and 16 analogies with extreme levels of differential item functioning (DIF) and then systematically revised and readministered the items in an attempt to reduce or eliminate DIF. Several diverse conclusions can be drawn from the data. First, because of the apparent success in reducing extreme levels of DIF in SAT‐Verbal items, the authors recommend that such efforts be continued. Second, the particular terminology used in stems and keys (rather than the underlying reasoning skill being measured) seems to be a recurring source of DIF in SAT‐Verbal items. Third, larger sample sizes, particularly for minority focal groups, would help to stabilize the DIF categories used by Educational Testing Service (ETS) test developers. Fourth, because the ETS delta metric is unbounded at the extremes, the use of both the Standardization (p‐metric) and Mantel‐Haenszel (delta‐metric) methodologies is recommended for classifying the level of DIF for very easy and very difficult items. Finally, the paper concludes with a suggestion for further research concerning the possible relationship between DIF and predictive validity.
THE ACCURACY OF AUTOMATIC QUALITATIVE ANALYSES OF CONSTRUCTED‐RESPONSE SOLUTIONS TO ALGEBRA WORD PROBLEMS
Authors: Bennett Randy Elliot , Sebrechts Marc M.
Published: August 2014

This study evaluated expert system diagnoses of examinees' solutions to complex constructed‐response algebra word problems. Problems were presented to three samples, each of which had taken the GRE General Test. One sample took the problems in paper‐and‐pencil form and the other two on computer. Responses were then diagnostically analyzed by an expert system, GIDE, and by four ETS mathematics test developers using a fine‐grained categorization of error types. Results were highly consistent across the samples. Human judges agreed among themselves almost perfectly in describing responses as right or wrong but concurred at much lower levels (37% to 64% agreement) in categorizing the specific bugs they detected in incorrect solutions. The expert system agreed highly with the judges' right/wrong decisions (95% to 97% concurrence) and somewhat less closely (71% to 74%) with the bug categorizations that judges, themselves, agreed on. Seven principal causes of machine‐rater disagreement were detected, most of which could be remedied by making adjustments to GIDE, modifying the test presentation interface to constrain the form of examinee solutions, and working with test developers to specify rules for automatically dealing with special cases. These results suggest that highly accurate diagnostic analysis through knowledge‐based understanding of complex responses may be difficult to achieve at the fine‐grained level used by GIDE. The accuracy of qualitative judgments might be increased by using a smaller set of more general diagnostic categories and by integrating information from other sources, including performance on diverse item types.
FLAWED ITEMS IN COMPUTERIZED ADAPTIVE TESTING1,2
Authors: Potenza Maria T. , Stocking Martha L.
Published: August 2014

A (multiple‐choice) test item is identified as flawed if it has no single best answer. In spite of extensive quality control procedures, the administration of flawed items to test‐takers is inevitable. Common strategies for dealing with flawed items in conventional testing, grounded in the principle of fairness to test‐takers, are reexamined in the context of adaptive testing. An additional strategy, available for adaptive testing, of retesting from a pool cleansed of flawed items, is compared to the existing strategies. Retesting was found to be no practical improvement over current strategies.
USING THE FREE‐RESPONSE SCORING TOOL TO AUTOMATICALLY SCORE THE FORMULATING‐HYPOTHESES ITEM
Authors: Kaplan Randy M. , Bennett Randy Elliot
Published: August 2014

Large‐scale institutional testing, and testing in general, are in a period of rapid change. Among the more obvious dimensions is the growing use of constructed‐response items and of computer‐based testing. This study explores the potential for using a computer‐based scoring procedure for the formulating‐hypotheses item. This item type presents a situation and asks the examinee to generate explanations for it. Each explanation is judged right or wrong and the number of creditable explanations summed to produce an item score. Scores were generated for 30 examinees' responses to each of eight items by a semantic pattern‐matching program and independently by five human raters. On its initial scoring run, the program agreed highly with the raters' mean item scores for some questions and improved its concurrence substantially as modifications to the automatic scoring process were made. By the final run, correlations between the program and the raters on item scores ranged from .89 to .97, and mean human‐machine discrepancies ran from .6 to 1.1 on a 16‐point scale. At the individual‐hypothesis level, the proportion agreement ranged from .80 to .94, which, given the large disproportion of correct responses in the sample, was little better than chance. Also detected was a tendency on the part of the program to erroneously classify wrong responses as correct. We conclude that F‐H items might be more effectively scored by a semiautomatic system that combines machine processing with a small number of human judges, and we present a preliminary configuration for such a process.
BACKGROUND CHARACTERISTICS AND FUTURE PLANS OF HIGH‐SCORING GRE GENERAL TEST EXAMINEES
Authors: Pennock‐Román María
Published: August 2014

The purpose of this study was to describe the background characteristics and future plans of average and above average Graduate Record Examinations (GRE) General Test takers in the various racial/ethnic groups. In this report, the characteristics included percentage of females, undergraduate college major and grades, intended field of graduate study, graduate degree objective, year of receipt of baccalaureate degree, location of high school, family socioeconomic background, citizenship status, and language spoken in the home. The distribution of these characteristics is given in tables for students classified according to level of GRE Verbal (GRE‐V), GRE Quantitative (GRE‐Q), or GRE Analytic (GRE‐A) scores in intervals of 50 points beginning with scores of 500‐549 until 750‐800.

It was found that high parental socioeconomic indicators were associated with higher GRE scores, but a substantial proportion (17%) of the high‐scoring students came from homes where fathers had less than a high school education. This figure was particularly large for the Mexican American group (36%). The distribution of parental income showed that, for all groups, there was a sizable percentage who would be unable to afford graduate school without financial aid.

Most examinees with very high GRE‐Q scores tended to take the GRE soon after they had graduated from college, but the opposite tended to be true for very high GRE‐V scorers in the non‐Hispanic White group. This pattern is discussed in light of choice of major (humanities versus science) and the availability of financial aid for graduate study in various fields.

One surprising finding was that the pattern of choice of major for racial/ethnic groups in the high scoring levels was substantially different from that in the overall GRE population. Holding GRE‐Q level constant, high‐scoring non‐Hispanic Black, Hispanic American, and Asian American students were more likely to major in engineering and computer science than were non‐Hispanic White test takers.
THE EFFECT OF COMPLEXITY OF MOUSE MANIPULATION ON PERFORMANCE IN COMPUTERIZED TESTING
Authors: Oltman Philip K.
Published: August 2014

The purpose of the study was to explore the possibility that requirements for complex mouse manipulation in computer‐based testing might make more of a negative impact on certain groups than on others. Very little evidence of such differential effects was observed in this study.
OPTIONAL USE OF CALCULATORS ON A MATHEMATICAL TEST: EFFECT ON ITEM DIFFICULTY AND SCORE EQUATING
Authors: Lawrence Ida M. , Dorans Neil J.
Published: August 2014

This paper describes findings from two studies involving optional use of calculators on SAT mathematical items. The first study looked at the effects of calculator use on estimates of item difficulty. The second study looked at the effect of calculator use on equating results from an anchor test design. Study 1 looked at data on specific items that become inappropriate for a test that permits calculators because the skills measured by the item administered with a calculator are quite different when the item is administered without a calculator. Study 2 showed that, because the use of a calculator sometimes makes items easier, any equating design that utilizes an anchor test design needs to ensure that the anchor test is administered under the same condition, i.e., with a calculator or without a calculator. A solution for making this adjustment with a special equating study is described.
THE EFFECT OF THE PROBABILITY OF CORRECT RESPONSE ON THE VARIABILITY OF MEASURES OF DIFFERENTIAL ITEM FUNCTIONING
Authors: Zwick Rebecca
Published: August 2014

The Mantel‐Haenszel (MH; 1959) approach of Holland and Thayer (1988) is a well‐established method for assessing differential item functioning (DIF). The formula for the variance of the MH DIF statistic is based on work by Phillips and Holland (1987) and Robins, Breslow and Greenland (1986). Recent simulation studies showed that the MH variances tended to be larger when items were administered to “examinees” who were randomly selected from a population than when items were administered adaptively. An analytic perspective shed some light on this puzzling result. Although the general form of the MH variance is complex and does not provide an intuitive understanding of the phenomenon, application of certain Rasch model assumptions yields a simple expression that appears to explain the difference in variances for adaptive versus nonadaptive administration.
AN ALTERNATIVE METHOD FOR SCORING ADAPTIVE TESTS1
Authors: Stocking Martha L.
Published: August 2014

Modern applications of computerized adaptive testing (CAT) are typically grounded in Item Response Theory (IRT; Lord, 1980). While the IRT foundations of adaptive testing provide a number of approaches to adaptive test scoring that may seem natural and efficient to psychometricians, these approaches may be more demanding for test‐takers, test score users, interested regulatory institutions, and so forth, to comprehend. An alternative method, based on more familiar equated number‐correct scores and identical to that used to score and equate many conventional tests, is explored and compared with one that relies more directly on IRT. The conclusion is reached that scoring adaptive tests using the familiar number‐correct score, accompanied by the necessary equating to adjust for the intentional differences in adaptive test difficulty, is a statistically viable, although slightly less efficient, method of adaptive test scoring. To enhance the prospects for enlightened public debate about adaptive testing, it may be preferable to use this more familiar approach. Public attention would then likely be focussed on issues more central to adaptive testing, namely the adaptive nature of the test.
SETTING STANDARDS ON THE ASSESSOR PROFICIENCY TEST FOR THE PRAXIS III: CLASSROOM PERFORMANCE ASSESSMENT
Authors: Livingston Samuel A. , Sims‐Gunzenhauser Alice
Published: August 2014

The Praxis III Assessor's job is to interview the beginning teacher, observe the teacher in the classroom, score the teacher's performance on nineteen criteria, and summarize the evidence for each score. The Assessor Proficiency Test (APT) consists of performing the assessor's task for a videotaped lesson. It yields two scores: accuracy and documentation. In this study, five judges judged the accuracy and documentation of APT records produced by fifteen assessor trainees. Documentation judgments were made by group consensus; accuracy judgments were individual. Logistic regression analysis showed that both types of judgments were strongly related to the corresponding APT scores, but judges' individual standards for accuracy varied greatly.
A TRANSPORTABILITY STUDY OF THE READING, WRITING, AND MATHEMATICS SKILLS IMPORTANT FOR TEACHERS OF DEAF AND HARD OF HEARING STUDENTS
Authors: Rosenfeld Michael , Mounty Judith L. , Ehringhaus Michael E.
Published: August 2014

The purpose of this study was to determine if the 57 reading, writing, and mathematics skill statements derived from the test specifications for the Praxis I Basic Skills Assessments and the Pre‐Professional Skills Test (PPST) are considered to be important for entry‐level teachers of deaf and hard of hearing students. The skill statements were placed in a survey format, and judgments of importance were made using a 5‐point rating scale. Two thousand surveys were mailed to administrators at schools and programs for deaf and hard of hearing students and to program directors at teacher training institutions throughout the United States. The administrators were asked to distribute the survey to teachers, administrators, and teacher educators at their institution and to give special consideration to the inclusion of deaf and hard of hearing faculty and staff members. In selecting the sample of institutions, every attempt was made to represent the full range of perspectives and practices in the education of deaf and hard of hearing students with respect to language and communication and program type.

Usable surveys were returned by 833 teachers, administrators, and teacher educators. A skill statement was judged to be appropriate for entry‐level teachers of deaf and hard of hearing students if it was judged to have a mean importance rating of 3.00 or greater (moderately important) from the total group of respondents as well as from each of the 44 subgroup comparisons that were conducted. All 57 skill statements received mean ratings above 3.00 from the total group as well as from all relevant subgroups of respondents. It may be concluded from the survey results that the skill statements used to build test specifications for the Praxis I Basic Skills Assessments and the PPST are appropriate for entry‐level teachers of deaf and hard of hearing students. That is, the content of these assessments was judged to be important. However, it cannot be overemphasized that this finding does not imply that the items or modes of assessment being used to measure these skills have been judged to be appropriate for deaf and hard of hearing candidates. Additional studies need to be conducted to determine the impact of these measures on deaf and hard of hearing candidates as well as the need for any special accommodations to ensure that candidates with this disability have an adequate opportunity to demonstrate their level of skill on these measures.
AN ASSESSMENT OF SELECTED VALIDITY‐RELATED PROPERTIES OF A SHORTENED VERSION OF THE SECONDARY LEVEL ENGLISH PROFICIENCY TEST AND LOCALLY DEVELOPED WRITING TESTS IN THE LACCD CONTEXT
Authors: Wilson Kenneth M.
Published: August 2014

This is the report of a study that was underaken to obtain direct empirical evidence regarding aspects of the validity and usefulness for ESL placement of (a) a shortened version of the Secondary Level English Proficiency (SLEP) Test, being used for ESL placement by colleges in the Los Angeles Community College District (LACCD), and (b) locally developed and scored writing tests. The LACCD Central Office provided scores on the shortened version of the SLEP and the writing samples, grades in ESL courses, and background data (gender, language, educational status, and so on) for over 10,000 students. This report documents and evaluates

• patterns of performance on the shortened SLEP and the writing tests in the general ESL population and in selected demographic subpopulations,

• concurrent relationships among scores on the components of the LACCD placement battery,

• observed levels of correlation between scores on the shortened SLEP test, the writing test, and the placement composite, on the one hand, and student performance in ESL courses, as indexed by grade earned (a grade on the “A‐F” scale, or a Pass/Fail grade), on the other, by course and by college, and in various subgroups (e.g., gender, educational level, age, language), and

• the extent to which observed relationships in placed samples are influenced by non‐validity‐related factors (for example, differential restriction of range on the tests that were used to place students, sample size, and type of grading system).

The findings of this collaborative undertaking provide direct empirical evidence that the shortened SLEP and locally developed writing tests are providing valid information regarding related aspects of ESL proficiency in the demographically diverse ESL student population being served by the LACCD; and the findings logically extend available evidence supportive of the validity of the Secondary Level English Proficiency Test for ESL assessment purposes in the LACCD and elsewhere. Based on study findings, the local writing tests and the shortened SLEP appear to be providing an effective basis for placing students, within time constraints that appear to be considered necessary, from an administrative perspective. Research is needed to address questions regarding the extent to which use of the full‐length version of the SLEP would enhance the overall validity of placement. Some pertinent lines of inquiry are suggested.
USING TRILINEAR PLOTS FOR NAEP STATE DATA1
Authors: Wainer Howard
Published: August 2014

Understanding the distribution of achievement levels of students' performance on NAEP is aided through the use of the trilinear chart. This chart is described and its use illustrated with data from the 1992 state NAEP mathematics assessment. It is shown that one can see readily the trends in performance for different demographic groups for all of the 44 participating jurisdictions simultaneously.
ANALYSIS OF PROPOSED REVISIONS OF THE TEST OF SPOKEN ENGLISH
Authors: Henning Grant , Schedl Mary , Suomi Barbara K.
Published: August 2014

This research was conducted to compare a prototype revised form of the Test of Spoken English (TSE®) with the current version of the same test. The study compared interrater reliability, frequency of rater discrepancy at all score levels, component task adequacy, scoring efficacy, and other concurrent and construct validity evidence, including oral proficiency interview correlations for a subset of the examinee sample. The study employed a representative examinee sample of 342 nonnative speakers of English, purposely sampled from among the two professional domains of prospective university teaching assistants (N = 184) and prospective licensed medical professionals (N = 158).

One somewhat unusual component of the study was the attempt to involve persons most at risk in the judgment process. Thus, in addition to employing the usual group of trained raters for the scoring of the current and prototype versions of the test, 16 naive adult raters were purposely selected (eight first‐year university students from four broad academic disciplines and eight nondegreed prospective medical outpatients within four broad age levels) for having limited exposure to foreign languages and cultures. These 16 naive raters (eight females and eight males) provided concurrent judgments of the comprehensibility and communicative effectiveness of a subset of 40 recorded prototype examinations.

In general, the comparative evidence gathered appeared to underscore the psychometric quality of the prototype revised TSE and to support conclusions of its adequacy as an instrument used to make judgments of the oral English language proficiency of nonnative speakers in the targeted populations. Some additional suggestions are provided on ways to implement the scoring of the prototype version of the test.
A COGNITIVE ANALYSIS OF SOLUTIONS FOR VERBAL, INFORMAL, AND FORMAL‐DEDUCTIVE REASONING PROBLEMS
Authors: Enright Mary K. , Tucker Carol B. , Katz Irvin R.
Published: August 2014

A previous study of new item types for the analytical measure of the GRE General Test found that the items loaded on three of four separable factors that were labeled verbal reasoning, informal reasoning, formal‐deductive reasoning, and quantitative reasoning. The present study examined the issue of how processing differed for these item types in the context of a problem‐space framework. Protocols of examinees solving a small set of problems aloud were collected. These protocols were examined with respect to two phases of the problem‐solving process: problem representation and problem solution. For formal‐deductive items, all the information necessary to solve the problem was provided in the problem statement. The representation of formal‐deductive items involved the use of meaning‐reduced tokens and spatial diagrams. The units involved in the representation of informal reasoning and verbal reasoning item types included meaningful propositions and meaning‐emphasizing paraphrases. Reference to common background knowledge occurred. The analysis of the problem‐solution phase focused on the processes of evaluation, (judgments of the correctness of an option) and justification (statements of an argument or of evidence for why an option was or was not correct). First, the order of these processes was found to differ for formal‐deductive items and other item types. Secondly, item solutions varied in terms of the kinds of justifications that were offered by the examinees for accepting or rejecting options. These results illustrate how the addition of some item types to the GRE analytical measure will expand the variety of reasoning skills assessed. Implications of these results for cognitive models of reasoning are also discussed.
A TASK TYPE FOR MEASURING THE REPRESENTATIONAL COMPONENT OF QUANTITATIVE PROFICIENCY
Authors: Bennett Randy Elliot , Sebrechts Marc M. , Rock Donald A.
Published: August 2014

Two computer‐based categorization tasks were developed and pilot tested. For Study I, the task asked examinees to sort mathematical word problem stems according to prototypes. Results showed that those who sorted well tended to have higher GRE General Test scores and college grades than did examinees who sorted less proficiently. Examinees generally preferred this task to multiple‐choice items like those found on General Test quantitative section and felt the task was a fairer measure of their ability to succeed in graduate school. For study II, the task involved rating the similarity of item pairs. Both mathematics test developers and students participated, with the results analyzed by individual differences multidimensional scaling. Experts produced more scaleable ratings overall and primarily attended to two dimensions. Students used the same two dimensions with the addition of a third. Students who rated like the experts in terms of the dimensions used tended to have higher admissions test scores than those who used other criteria. Finally, examinees preferred multiple‐choice questions to the rating task and felt that the former was a fairer indicator of their scholastic abilities. The major implication of this work is in identifying a new task type for admissions tests, as well as for instructional assessment products that might help lower scoring examinees localize and remediate problem‐solving difficulties.
ASSESSING SOME OF THE PROPERTIES OF LONGER BLOCKS IN THE 1992 NAEP READING ASSESSMENT
Authors: Donoghue John R. , Mazzeo John
Published: August 2014

At Grades 8 and 12, the 1992 NAEP reading assessment contained a small number of 50 minute blocks, in addition to the usual 25 minute blocks. In order to determine whether to incorporate the 50 minute blocks into the operational scaling, this study sought to determine whether longer blocks measure a different construct from that assessed by the 25 minute blocks. Structural equation modeling tested the hypothesis that the structural parameters relating reading ability to demographic variables do not differ across block type. A multiple group analysis, where type of block (25 minute or 50 minute) defined the two groups, was used. The null hypothesis for this comparison was that the two types of blocks measure the same trait but may differ in observed mean and variance.

Results of the main analysis did not reject the hypothesis of invariant structural parameters, and so the 50 minute blocks were incorporated in the 1992 NAEP scales. Sensitivity analyses indicated that this conclusion was moderately robust to assumptions made about missing data for items which were not reached, although Grade 8 results were more robust than those for Grade 12. Analyses using other measures of fit yielded the same pattern of results, although the magnitude of the χ2 statistics were affected by the fit measure chosen, particularly the asymptotically distribution free method.

Attempts to replicate the main analysis in independent samples yielded similar χ2 values at Grade 8, but Grade 12 yielded χ2 values which were substantially higher for some of the samples. The Grade 12 results raise questions as to the generalizability of the main analysis. Alternatives to the reliance on χ2 measures are discussed for future research.
GE FRST EVALUATION REPORT: HOW WELL DOES A STATISTICALLY‐BASED NATURAL LANGUAGE PROCESSING SYSTEM SCORE NATURAL LANGUAGE CONSTRUCTED‐RESPONSES?
Authors: Burstein Jill C. , Kaplan Randy M.
Published: August 2014

There is a considerable interest at Educational Testing Service (ETS) to include performance‐based, natural language constructed‐response items on standarized tests. Such items can be developed, but the projected time and costs required to have these items scored by human graders would be prohibitive. In order for ETS to include these types of items on standardized tests, automated scoring systems need to be developed and evaluated. Automated scoring systems could decrease the time and costs required for human graders to score these items. This report details the evaluation of a statistically‐based scoring system, the General Electric Free‐Response Scoring Tool (GE FRST). GE FRST was designed to score short‐answer, constructed‐responses of up to 17 words. The report describes how the system performs for responses on three different item types. For the sake of efficiency, it is important to evaluate systems on a number of item types to see if the system's scoring method can generalize to a number of item types.
TOWARD COMPUTER‐BASED PERFORMANCE ASSESSMENT IN MATHEMATICS
Authors: Singley Mark K. , Bennett Randy Elliot
Published: August 2014

One of the main limitations of the current generation of computer‐based tests is its dependency on the multiple‐choice item. Our work is aimed at extending computer‐based testing by bringing limited forms of performance assessment to it in the domain of mathematics. This endeavor involves not only building task types that better reflect valued problem solving, but also creating an integrated set of supports including easy‐to‐use interfaces, tutorials to teach novice computer users how to negotiate those interfaces, tools that help test developers create items quickly, and mechanisms for scoring constructed responses more efficiently.
A PRELIMINARY COMPARISON OF THE EFFECTIVENESS OF CLUSTER ANALYSIS WEIGHTING PROCEDURES FOR WITHIN‐GROUP COVARIANCE STRUCTURE
Authors: Donoghue John R.
Published: August 2014

A Monte Carlo study compared the usefulness of six variable weighting methods for cluster analysis. Datasets were 100 bivariate observations from two subgroups, generated according to a finite normal mixture model. Subgroup size, within‐group correlation, within‐group variance, and distance between subgroup centroids were manipulated.

Of the clustering methods examined, the flexible average clustering algorithm with β = ‐. 15 or ‐.20 gave the best recovery. Of the remaining methods, Ward's method yielded the best recovery, followed closely by beta‐flexible linkage (β = ‐.50) and SAS's EML algorithm.

In the absence of variable weights, negative within‐group correlation resulted in much poorer recovery for all clustering algorithms. The ACE weighting method of Art, Gnanadesikan, and Kettenring provided a net improvement in 17‐24% of the datasets when used with better clustering algorithms. When used with the same clustering alogrithms, De Soete's ultrametric weighting yielded improved recovery 16‐22% of the time. However, although ultrametric weighting was more sensitive than ACE to negative within‐subgroup correlation. Clustering based on principal components was less effective. Therefore, the ACE method is preferred overall.

There is still room for improvement, however. Clustering with Mahalanobis distance based on the pooled within‐group covariance matrix indicated that knowing the correct covariance matrix would yield improved recovery (over ACE) approximately 10% of the time.
READER CALIBRATION AND ITS POTENTIAL ROLE IN EQUATING FOR THE TEST OF WRITTEN ENGLISH
Authors: Myford Carol M. , Marr Diana B. , Linacre J. Michael
Published: August 2014

When judges use a rating scale to rate performances, some judges may rate more severely than others, giving lower ratings. Judges may also differ in the consistency with which they apply rating criteria. In this study, we pilot tested a quality control procedure that provides a means for monitoring and adjusting for differences in reader performance. We employed FACETS, a Rasch‐based rating scale analysis procedure, to calibrate readers within and across two Test of Written English (TWE®) administrations. Our study had four goals: (1) to determine the extent to which individual readers can be considered interchangeable, both within and across TWE administrations; (2) to investigate reader characteristics and their relationships to the volume and quality of ratings; (3) to examine the effectiveness of third readings to adjudicate rating discrepancies; and (4) to make a preliminary determination of the feasibility of using FACETS Reader Severity Measures as a first step toward equating TWE scores across different topics and readers.
FACTORS IN PERFORMANCE ON BRIEF, IMPROMPTU ESSAY EXAMINATIONS
Authors: BRELAND HUNTER M. , BONNER MARILYN W. , KUBOTA MELVIN Y.
Published: August 2014

Brief, impromptu essays written for the 1990 administration of the College Board's English Composition Test (ECT) were randomly sampled for four different groups of examinees. These essays were subjected to further holistic ratings beyond those conducted for the ECT administration, and analytical ratings were also obtained. The holistic scores were correlated with the analytical scores to determine which essay characteristics were most closely associated with high holistic scores. The results indicated that overall organization, use of supporting materials, noteworthy ideas, rhetorical strategy, and thesis statement were the strongest correlates. Comparisons with the results of a previous study of 1979 ECT essays, written on a different topic, showed that these same characteristics were also most closely associated with high holistic scores in that study. These characteristics correlated strongly with high holistic scores for all subgroups randomly sampled, and for males and females.

Additional factors associated with better holistic scores but studied only for the 1990 topic included following instructions, completion of the task, use of a standard essay format, use of literary references, and proficiency in the English language.

Certain types of essay content were also associated with higher scores on the 1990 topic. Essays combining current affairs with literature and history or combining literature and history received slightly higher scores on average than essays based only on current affairs, literature, history, or personal experience. Nevertheless, some essays receiving high scores were based solely on personal experience, did not contain literary references, and did not use a standard essay format. The analyses suggest that some practice with this type of brief, impromptu essay, particularly under strict time constraints, may be useful as preparation for taking such essay examinations.
THE DIMENSIONALITY OF RESPONSES TO SAT ANALOGY ITEMS
Authors: Diones Ruth , Bejar Isaac I. , Chaffin Roger
Published: August 2014

This study continued the research on analogy problem‐solving on psychometric tests pursued by Bejar, Chaffin and Embretson (1991). In specific, characteristics of a semantic taxonomy and a cognitively and empirically motivated intensional/pragmatic (I/P) dichotomy were explored. There were two research questions: (1) Could Bejar et al.'s results be replicated with SAT items? and (2) Would factor analyses support the bidimensional processing structure suggested by the I/P distinction? A specially constructed test of disclosed SAT analogies was administered to a group of 189 undergraduate students. Though factor analyses did not support the expected bidimensionality, a better understanding of both the semantic taxonomy and the I/P dichotomy was achieved. Suggestions for future work were given.
GENDER AND ETHNIC GROUP DIFFERENCES ON THE GMAT ANALYTICAL WRITING ASSESSMENT
Authors: Bridgeman Brent , McHale Frederick
Published: August 2014

Gender and ethnic group differences on the Analytical Writing Assessment that is part of the Graduate Management Admissions Test were evaluated. Data from the first operational administration in October of 1994 were used. Standardized differences from the White male reference group were computed separately for men and women in four ethnic groups: White, Asian American, African American, and Hispanic/Latino. Within the White, African American, and Hispanic/Latino groups, women received higher scores than men on the Analytical Writing Assessment; in the Asian American group, men received higher writing scores, but the difference was not as great as on the Verbal score. Examinees whose best language was not English scored relatively higher on the Analytical Writing Assessment than on the Verbal measure. Simulations of eligibility for an admissions pool suggested that the addition of the Analytical Writing score would noticeably increase the number of the women in the pool, but would have virtually no impact on ethnic minorities. Rater and score reliability were reasonably consistent across ethnic and gender groups.
THE RELIABILITY OF DOCUMENT‐BASED ESSAY QUESTIONS ON ADVANCED PLACEMENT HISTORY EXAMINATIONS
Authors: Bridgeman Brent , Morgan Rick , Wang Ming‐mei
Published: August 2014

If a student writes two essays, the score reliability can be estimated from the correlation between the essays. However, if the essays are in different modes or require different skills, the reliability may be underestimated from the correlation. In Advanced Placement history examinations, students wrote one standard essay and one essay that required analysis and synthesis of historical documents that were included with the question statement. If these document‐based questions (DBQ) were assessing substantially different skills from the standard essays, then the reliability of the DBQ scores would be underestimated from their correlation with a standard essay score. A sample of 1045 U. S. history students and 891 European history students participated in a special study in which they wrote essays for either 2 DBQ questions and one standard essay question or 3 standard essay questions. The DBQ correlated as highly with a standard essay as with another DBQ, suggesting that the simple correlation of the two types of scores did not underestimate the reliability of the essay scores.
ASSESSING UNUSUAL AGREEMENT BETWEEN THE INCORRECT ANSWERS OF TWO EXAMINEES USING THE K‐INDEX: STATISTICAL THEORY AND EMPIRICAL SUPPORT
Authors: Holland Paul W.
Published: August 2014

Test security and other concerns can lead to an interest in the assessment of how unusual it is for the answers of two different examinees to agree as much as they do. At Educational Testing Service, a measure called the K‐index is used to assess ‘unusual agreement’ between the incorrect answers of two examinees on a multiple‐choice test. Here, I describe the K‐index and give the results of an empirical study of some of the assumptions that underlie it and its use. The results of this study show that the K‐index can be expected to give a conservative estimate of the probability of chance agreement in the typical situations for which it is used, and that several important assumptions underlying the K‐index are supported by relevant data. In addition, the results presented here suggest a minor modification of the current (as of 1993) application of K‐index to part of the SAT to better insure that it is a conservative measure of chance agreement.