bluuebunny/crossref_metadata_2025 huggingface dataset:
DIFFERENTIAL SPEEDEDNESS AND ITEM OMIT PATTERNS ON THE SAT
Authors: Schmitt Alicia P. , Dorans Neil J. , Crone Carolyn R. , Maneckshana Behroz T.
Published: August 2014

Two editions of the Verbal and Mathematical portions of the Scholastic Aptitude Test (SAT) were used to study differential speededness and differential omission and the relationships among differential item functioning (DIF), differential omission, and item difficulty for Asian‐Americans, Blacks, Hispanics, and females. Consistent and replicable evidence of differential speededness was found for Blacks and Hispanics. Use of an unspeeded criterion for matching in place of the traditional total score, which contains speeded items, does not affect the DIF analyses of the speeded items. A strong artifactual negative relationship between DIF and differential omission was found. The relationship between differential omission and difficulty was consistently positive on the Verbal sections for all comparison groups except the Asian‐American group, for whom it was consistently negative. On the Mathematical sections, this relationship was only consistently found for the female/male comparison, for whom it was negative. Finally, the relationship between difficulty and DIF was negative but smaller than previously observed.
THE PREDICTION OF GRE READING COMPREHENSION ITEM DIFFICULTY FOR EXPOSITORY PROSE PASSAGES FOR EACH OF THREE ITEM TYPES: MAIN IDEAS, INFERENCES AND EXPLICIT STATEMENTS
Authors: Freedle Roy , Kostin Irene
Published: August 2014

This study examines the predictability of GRE reading item difficulty (equated delta) for three major reading item types: main idea, inference and explicit statement items. Each item type is analyzed separately, using 110 GRE reading passages and their associated 244 reading items; selective analyses of 285 SAT reading items are also presented. Stepwise regression analyses indicates that the percentage of GRE delta variance accounted for varied from 20% to 52% depending upon the item type.

Details of item predictability were explored by evaluating several hypotheses. Results indicated that (1) multiple‐choice reading items are sensitive to variables similar to those reported in the experimental literature on comprehension, (2) many of these variables provide independent predictive information in regression analyses, and (3) substantial agreement between GRE and SAT reading predictability was found.
COGNITIVE PROCESSING REQUIREMENTS OF CONSTRUCTED FIGURAL RESPONSE AND MULTIPLE‐CHOICE ITEMS IN ARCHITECTURE ASSESSMENT
Authors: Martinez Michael E. , Katz Irvin R.
Published: August 2014

Contrasts between constructed response items and stem‐equivalent multiple‐choice counterparts have yielded only a few weak generalizations. Such comparisons typically have involved averaging item characteristics, and this aggregation has masked differences in statistical properties at the item level. Moreover, even aggregated format differences have not been explained in terms of differential cognitive processing demands of the items. In this paper, we examine item‐level differences between figural response items and their multiple‐choice counterparts in the domain of architecture. The figural response item format is an assessment form that uses figural material (such as, graphs, illustrations, and diagrams) as item stimuli and the medium through which knowledge and skill are demonstrated. We first examine item‐level format differences in difficulty and then ask whether there are corresponding differences in the cognitive processing requirements of the items that can account for the psychometric differences. After finding evidence for these connections, we propose that differences in processing requirements and concomitant psychometric properties might be systematic and predictable. These analyses are important in a larger sense in that they shed light on aspects of construct validity that are frequently neglected and they touch the interface of the usually segregated psychometric and cognitive methodologies.
THE VARIANCE OF RASCH ABILITY ESTIMATES FROM PARTIALLY‐KNOWN ITEM PARAMETERS
Authors: Mislevy Robert J.
Published: August 2014

A closed‐form approximation is given for the variance of examinee proficiency estimates in the Rasch model for dichotomous items, under the condition that only estimates, rather than true values, of item difficulty parameters are available. The term that must be added to the usual response‐sampling variance is inversely proportional to both the number of examinees in the item calibration sample and the length of the test. Illustrative numerical values suggest that the impact of uncertainty about Rasch item parameters on subsequent estimates of examinee proficiencies is less than has been observed in the 2‐ and 3‐parameter logistic IRT models.
DIF DETECTION AND DESCRIPTION: MANTEL‐HAENSZEL AND STANDARDIZATION1,2
Authors: Dorans Neil J. , Holland Paul W.
Published: August 2014

At the Educational Testing Service, the Mantel‐Haenszel procedure is used for differential item functioning (DIF) detection and the standardization procedure is used to describe DIF. This report describes these procedures. First, an important distinction is made between DIF and Impact, pointing the need to compare the comparable. Then, these two contingency table DIF procedures are described in some detail, first in terms of their own origins as DIF procedures, and then from a common framework that points out similarities and differences. The relationship between the Mantel‐Haenszel procedure and IRT models in general and the Rasch model, in particular, is discussed. The utility of the standardization approach for assessing differential distractor functioning is described. Several issues in applied DIF analyses are discussed including inclusion of the studied item in the matching variable, and refinement of the matching variable. Future research topics dealing with the matching variable, the studied variable and the group variable are also discussed.
COMBINING MULTIPLE‐CHOICE AND CONSTRUCTED RESPONSE TEST SCORES: TOWARD A MARXIST THEORY OF TEST CONSTRUCTION
Authors: Wainer Howard , Thissen David
Published: August 2014

Future assessment instruments will probably be composed of a combination of different types of questions. Even though different kinds of questions require different scoring procedures there may be a need, for some purposes, to have those different scores combined as a composite. In this account we describe how mixtures of such scores might be efficaciously combined. Or, if no post hoc adjustment is desired, we provide two characterizations of measurement effectiveness to aid in making unadjusted score combinations efficient.
EFFECTS OF AMOUNT OF TIME ALLOWED ON THE TEST OF WRITTEN ENGLISH
Authors: Hale Gordon A.
Published: August 2014

This study examined students' essay performance on topics from the Test of Written English (TWE®) under two time limits—30 minutes, as on the current TWE, and 45 minutes. In the main groups of the study, each student wrote an essay on one topic under one time limit and on another topic under the other time limit (with orders counterbalanced). The correlation between scores for 30‐ versus 45‐minute conditions was relatively high and approached the parallel‐form reliability of the task, as indicated by data from students who wrote essays on separate topics under the same time limits. Thus, the provision of additional time apparently had little effect on the standing of the students in relation to each other. Both parallel‐form reliability and interrater reliability were approximately the same for the 30‐ and 45‐minute conditions. Mean scores on the 6‐point TWE scale were found to be higher by about 1/4 to 1/3 point under the 45‐minute condition than the 30‐minute condition, indicating that provision of additional time produced a modest but reliable increase in scores. The magnitude of the effect was roughly comparable for students of low versus high proficiency, and for students in intensive English programs versus students in academic coursework. Responses to a questionnaire indicated that the students regarded 45 minutes as more sufficient for accomplishing the task than 30 minutes. The results are discussed in relation to the literature on time effects and to practical implications for the Test of Written English.
SCALAR ANALYSIS OF THE TEST OF WRITTEN ENGLISH
Authors: Henning Grant
Published: August 2014

The present research was conducted to explore the psychometric characteristics of the Test of Written English (TWE) rating scale. Rasch model scalar analysis methodology was employed with more than 4,000 scored essays across two elicitation prompts to gather the following information about the TWE rating scale and rating process:

the position and size of the interval on the overall latent trait that could be attributed to behavioral descriptors accompanying each possible integer scoring step on the TWE scale

the standard error of estimate associated with each possible transformed integer rating

the fit of rating scale steps and individual rated essays to a unidimensional model of writing ability and, concurrently, the adequacy of such a model, including the proportion of misfitting essays as a portion of all essays analyzed

the fit of individual readers to a unidimensional model of writing ability and to the expectations of a chi‐square contingency test of independence of readers and ratings assigned, along with information on some characteristics of misfitting readers

comparative scalar information for two distinct TWE elicitation prompts, including nonparametric tests of the independence of readers and scale steps assigned and the feasibility of equating of scales.

Results suggested that the intervals between TWE scale steps were surprisingly uniform and that the size of the intervals was appropriately larger than the error associated with assignment of individual ratings. The proportion of positively misfitting essays was small (approximately 1% of all essays analyzed) and was approximately equal to the proportion of essays that required adjudication by a third reader. This latter finding, along with the low proportion of misfitting readers detected, provided preliminary evidence of the feasibility of employing Rasch rating scale analysis methodology for the equating of TWE essays prepared across prompts. Some information on characteristics of misfitting readers was presented that could be useful in the reader training process.
SEX DIFFERENCES IN PROBLEM‐SOLVING STRATEGIES USED BY HIGH‐SCORING EXAMINEES ON THE SAT‐M
Authors: Gallagher Ann M.
Published: August 2014

An item classification scheme developed by Gallagher (1990) was refined, resulting in a more accurate prediction of sex differences in performance on the mathematical sections of the College Board's Scholastic Aptitude Test (SAT). When Differential Item Functioning (DIF) procedures were performed on item data for examinees scoring above 650, the majority of items that were flagged as favoring males required the use of mathematical insight, whereas all the items flagged as favoring females required standard algorithmic solutions.

Structured interviews were conducted with students (25 males and 22 females) in this score range to determine the nature of differences in strategy use. A classification scheme was developed for strategies that paralleled the item classification categories. There was substantial overlap in strategies used by males and females; however, analyses of strategy types used across all items indicated that females were more likely than males to use algorithmic strategies and males were more likely than females to use insightful strategies. It should be noted that these findings constitute a generalization across a group of subjects and items, and that there were several individual instances of males using more algorithmic strategies than females and females using more insightful strategies than males.

Questionnaire data gathered from students who participated in interviews indicated a positive relationship for both males and females between SAT‐mathematical performance and positive attitudes toward mathematics (e.g., liking mathematics as a subject and recognizing its usefulness to their adult lives). The use of algorithmic strategies, however, was correlated with negative attitudes toward mathematics (e.g., mathematics being difficult and not being relevant to their lives). Implications of these and other findings are discussed.
ELIGIBILITY ISSUES AND COMPARABLE TIME LIMITS FOR DISABLED AND NONDISABLED SAT® EXAMINEES
Authors: Ragosta Marjorie , Wendler Cathy
Published: August 2014

This project used data from test administration timing records, the College Board's Scholastic Aptitude Test (SAT) history file, and a survey questionnaire to investigate two issues: comparable testing time and eligibility for special test accommodations for SAT examinees with disabilities.

Comparable testing time for disabled examinees was found to be between 1½ and 2 times that for nondisabled examinees. Time limits in that range would assure that approximately equal percentages of disabled and nondisabled students would complete each section of the SAT The exception was blind students using braille or cassette versions of the test, who required between double and triple time.

Eligibility for special test accommodations is tied to severity of disability and documentation of disability. Some levels of disability could be distinguished for students with sensory disabilities, based on self‐reports and confirmed by the presence or absence of individualized education programs (IEPs). For example, IEPs were reported for 80 percent of totally blind candidates, 50 percent of legally blind candidates, and only 28 percent of visually impaired examinees who were not considered legally blind. Similarly, 90 percent of hearing‐impaired students using American Sign Language (ASL) reported having IEPs, compared to 76 percent of those using total communication (English and sign) and 42 percent of those for whom English was the preferred communication mode. Severity of disability could not be defined for physically disabled or learning‐disabled examinees, except on the basis of IEPs. However, it was difficult to isolate the need for test accommodations based on school practices (e.g., IEPs or special schools or classes), since examinees eligible for special test administrations were often tested under standard conditions.

Alternatives to the current eligibility policy are discussed, including a change to school‐based criteria and the use of individualized testing programs. A change to empirically derived testing times is also discussed.
A METHOD FOR SEVERELY CONSTRAINED ITEM SELECTION IN ADAPTIVE TESTING
Authors: Stocking Martha L. , Swanson Len
Published: August 2014

This paper examines the current state of attempts to incorporate expert test construction practices into a computerized adaptive testing paradigm. A new methodology for incorporating large numbers of constraints on adaptive test item selection is described. This methodology builds on the conventional, that is, linear, test construction practices employed by expert test specialists. The incorporation of such practices is necessary if computerized adaptive testing is to become a serious competitor to conventional paper‐and‐pencil testing in the future. The successful application of this new methodology to the design of an adaptive test in a real‐life setting is demonstrated by an example.
A PSYCHOMETRICALLY SOUND COGNITIVE DIAGNOSTIC MODEL: EFFECT OF REMEDIATION AS EMPIRICAL VALIDITY
Authors: Tatsuoka Kikumi K. , Tatsuoka Maurice M.
Published: August 2014

The purpose of this study was to validate the results of cognitive diagnoses using the rule‐space model and to demonstrate the usefulness of cognitive diagnoses for instruction. The results of the study strongly indicated that the rule‐space model can effectively diagnose students' knowledge states and can point out ways for remediating their errors quickly with minimum effort.

It was also found that the designing of instructional units for remediation can be effectively guided by the rule‐space model, because the determination of all possible ideal item‐score patterns, given an incidence matrix, is based on a tree structure of cognitive attributes, knowledge states, and items.
PREPUBLISHING THE ESSAY TOPICS FOR A TEACHER LICENSING TEST: A SMALL‐SCALE SIMULATION
Authors: Powers Donald E. , Fowles Mary E. , Farnum Marisa , Aberle Dale J. , Collins Grace E. , Bly Mary , Palmquist Michael , Lytle Richard , Mather Susan , Pollock Nancy
Published: August 2014

The computer‐based Praxis I: Academic Skills Assessment component of The Praxis Series: Professional Assessments for Beginning Teachers™ will include a measure of prospective teachers' writing proficiency. Current plans call for making available to test takers (before they take the exam) a pool of approximately 80 essay topics, from which the actual test topics will be drawn initially.

The objectives of the study described here were to predict the impact of this policy on the ways in which test takers will prepare for the test and to estimate the effect of this new practice on examinees' eventual test performance. To make these determinations, a small‐scale study was undertaken in collaboration with writing instructors at four different colleges or universities. Within the context of undergraduate writing courses at these institutions, instructors designed and conducted brief “coaching” programs in response to a challenge to provide any advice, instruction, tricks, etc., that might help their students achieve a minimally acceptable score on the test. A pool of 10 topics was disclosed and used in the coaching. Upon completion of the coaching, all students wrote two essays—one on a disclosed topic and one on a comparable, previously unseen topic.

Comparisons revealed very little difference between students' subsequent performances on these two types of topics. Performances on the two topics did not correlate differentially with any of three available indicators of academic ability/achievement. However, students believed that their writing skills were better revealed by writing on disclosed topics than on topics that had not been previously seen. In short, the study uncovered little to suggest that plans for disclosing a pool of Praxis I essay topics should be modified.
RELIABILITY OF THE TEST OF SPOKEN ENGLISH REVISITED
Authors: Boldt R. F.
Published: August 2014

The Test of Spoken English (TSE) is an internationally administered instrument for assessing nonnative speakers' proficiency in speaking English. The research foundation of the TSE® examination is currently described in the Manual for Score Users. This publication describes technical characteristics of the test, including such psychometric characteristics as level of difficulty, reliability, and validity. Consistent with the ETS Standards for Quality and Fairness, the Manual refers to two sources of variation other than the achievement being measured: interrater reliability and internal consistency. Because the reported data on both issues were drawn from a 1980 study of reliabilities based on two raters, newer data and more extensive analyses were needed.

The present study uses data from recent administrations of the TSE examination. Analysis of variance examined the effects of scale, section, examinee, and rater, as well as the interactions of these factors. Reliabilities were reported for item, section, and scale scores. Common and unique variance estimates were developed for each scale score for which a section was rated. Estimates of the effects of altering section lengths suggested that some sections should be lengthened and others omitted if reliability were to be maximized. Other suggestions were offered for improving reliability.
STRATEGY USE ON MULTIPLE‐CHOICE AND FREE‐RESPONSE ITEMS: AN ANALYSIS OF SEX DIFFERENCES AMONG HIGH SCORING EXAMINEES ON THE SAT‐M
Authors: Gallagher Ann , Mandinach Ellen
Published: August 2014

Twenty‐four students who scored 650 or more on the SAT‐M were asked to think aloud while solving 13 mathematics items in either multiple‐choice or free‐response format. Strategies students used to solve the items were classified as either algorithmic or insightful. Data analyses indicated that items in the free‐response format were significantly more difficult for females than multiple choice items. No significant difference was found for males. Females were more likely than males to use algorithmic strategies in both the multiple‐choice and free‐response formats, with no significant difference in types of strategies used in either format. Males were more likely to use insightful strategies in the multiple‐choice format than in the free‐response format. Finally, on multiple‐choice items, both males and females used options to detect calculation errors, however with females using this strategy somewhat more than males.
TOWARD A STABLE DIAGNOSTIC REPRESENTATION OF STUDENTS' ERRORS IN ALGEBRA
Authors: Birenbaum Menucha , Kelly Anthony E. , Tatsuoka Kikumi K. , Gutvirtz Yaffa
Published: August 2014

Diagnoses of students' performance on procedural mathematical tasks need to display a certain level of stability and robustness if they are to be used as the basis for remediation, particularly with computer‐delivered instruction. The purpose of this study was to compare two diagnostic approaches for describing students' errors in algebra – a bug analysis and a rule‐space analysis – with the goal of investigating the relative stability of the diagnoses derived from these approaches. Consistent with the findings of recent studies, a relatively large number of bugs were unstable; stable bugs tended to be infrequent. In contrast, the results of the rule‐space analysis yielded relatively more stable diagnoses. The results were discussed in light of their consequences for designing remediation.
DISTRIBUTIONS OF ACTFL RATINGS BY TOEFL SCORE RANGES
Authors: Boldt R. F. , Larsen‐Freeman D. , Reed M. S. , Courtney R. G.
Published: August 2014

The purpose of this study was to align verbal descriptions of test takers' language performance with distributions of the numerical scores they received on the three sections (Listening Comprehension, Structure and Written Expression, and Reading Comprehension and Vocabulary) of the Test of English as a Foreign Language (TOEFL).

The study used the American Council of Teaching Foreign Languages (ACTFL) descriptors as anchors for the TOEFL scores. The ACTFL Proficiency Guidelines, from which the descriptors were taken, are widely used.

Qualified English as a Second Language (ESL) instructors rated their students' listening, reading, and writing proficiency using the ACTFL descriptors. Very shortly thereafter, the ratees took the TOEFL. Students' ACTFL ratings were quantified and cross‐tabulated with TOEFL section scores.

Results were analyzed as follows: First, the need to adjust for rater difficulty was investigated. Second, typical descriptive data were developed. Third, several schemes were used to quantify the verbal descriptors from the ACTFL Guidelines. Fourth, the ratings and test scores were correlated. Fifth, the issue of the differential validity of the measures of speaking, listening, and reading was examined. Sixth, several estimates of the reliability of the ratings were obtained. Finally, the distributions of ratings at levels of the TOEFL section scores were developed. These latter distributions can be helpful in interpreting TOEFL scores in terms of language performance. Although there was no one‐to‐one correspondence between the TOEFL score level and the ACTFL rating level, the correlations between ratings and scores were substantial.
COMBINATION OF CONDITIONAL LOG‐LINEAR STRUCTURES
Authors: Kim Sung‐Ho
Published: August 2014

There has been increasing attention to the fine structure of abilities underlying task performance (Haertel and Wiley (in press)). One of the useful approaches for this is by use of graphical models to represent relationships among abilities and test items. Building a large graphical model is always an issue. Restrictions upon experiments and data collection, among others, may result in parts of the large model. Or it may be convenient for us to build parts of the large models first, and then try to combine those parts into a larger model. This paper derives, confined to categorical variables only, a theory which may be useful in combining conditional graphical models into a larger one. The main result of the paper is that we can see partial information about a true log‐linear structure (LLS) from its conditional LLSs and use the information in trying to guess the true LLS, assuming that the true LLS is graphical. An application of the result is illustrated using a simulated data set.
FAMILY CHARACTERISTICS AS PREDICTORS OF INDIVIDUAL DIFFERENCES IN CHICANO CHILDREN'S EMERGENT SCHOOL READINESS
Authors: Laosa Luis M.
Published: August 2014

This study of normal young children in 100 two‐parent Chicano households of widely varied socioeconomic levels assessed the ability of selected family sociodemographic and intellectual characteristics to predict individual differences in the children's performance on a measure of emergent school readiness. Data were collected longitudinally in the home at ages 30, 42, and 48 months. Emergent school readiness was measured using the Preschool Inventory, administered at each of the last two longitudinal points. The family characteristics examined were (a) specific sociodemographic variables (household financial income, mother's and father's schooling attainment levels, occupational status, and the family size, sibling constellation, and home language) and (b) the mother's and father's performance on the Culture Fair Intelligence Test. Contemporaneous and time‐lagged zero‐order and partial correlations and multiple regressions assessed the unique, shared, and shifting strengths of these hypothesized family predictors of child performance. Together these predictors accounted for 50% of the variance in child performance.
EXTENDING THE RULE SPACE MODEL TO A SEMANTICALLY‐RICH DOMAIN: DIAGNOSTIC ASSESSMENT IN ARCHITECTURE
Authors: Katz Irvin R. , Martinez Michael E. , Sheehan Kathleen , Tatsuoka Kikumi K.
Published: August 2014

This paper presents a technique for applying the Rule Space model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically‐rich domain. Responses to 22 architecture test items, developed to assess a range of architectural knowledge, were analyzed using Rule Space. Verbal protocol analyses guided the construction of a model of examinee performance, consisting of processes for constructing an initial representation of an item (labeled understand), forming goals and performing actions based on those goals (solve), and determining whether goals have been attempted and satisfied (check). Item attributes, derived from these processes, formed the basis for diagnosis. Our technique extends Rule Space's applicability by defining attributes in terms of item characteristics and the causal relations between characteristics and the problem‐solving model.

Data were collected from 122 architects of various ability levels (students, architecture interns, and professional architects). Rule Space successfully classified approximately 65%, 90%, and 40% of examinees based, respectively, on attributes associated with the understand, solve, and check processes of the problem‐solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels.
TEST THEORY RECONCEIVED
Authors: Mislevy Robert J.
Published: August 2014

Educational test theory consists of statistical and methodological tools to support inference about examinees' knowledge, skills, and accomplishments. The evolution of test theory has been shaped by the nature of users' inferences, which, until recently, have been framed almost exclusively in terms of trait and behavioral psychology. Progress in the methodology of test theory enabled users to extend the range of inference, sharpen the logic, and ground their interpretations more solidly within these psychological paradigms. In particular, the focus remained on students' overall tendency to perform in prespecified ways in prespecified domains of tasks; for example, to make correct answers to mixed‐number subtraction problems. Developments in cognitive and developmental psychology broaden the range of desired inferences, especially to conjectures about the nature and acquisition of students' knowledge. Commensurately broader ranges of data‐types and student models are entertained. The same underlying principles of inference that led to standard test theory can be applied to support inference in this broader universe of discourse. Familiar models and methods—sometimes extended, sometimes reinterpreted, sometimes applied to problems wholly different from those for which they were first devised—can play a useful role to this end.
MODELS FOR SCORING MISSING RESPONSES TO MULTIPLE‐CHOICE ITEMS
Authors: Longford Nicholas T.
Published: August 2014

This is a critical evaluation of the rules for coding and scoring of missing responses to multiple‐choice items in educational tests. The focus is on tests in which the test‐takers have little or no motivation; in such tests omitting and not reaching (as classified by the currently adopted operational rules) is quite frequent. Data from the 1991 NAEP Reading Assessment of 17‐year‐olds are used in analyses and illustrative examples. Alternative rules for scoring based on hypothesized behaviour of the test‐takers are proposed.
EVALUATION OF THE MAGNITUDE OF DIFFERENTIAL ITEM FUNCTIONING IN POLYTOMOUS ITEMS
Authors: Zwick Rebecca , Thayer Dorothy T.
Published: August 2014

Several recent studies have investigated the application of statistical inference procedures to the analysis of differential item functioning (DIF) in test items that are scored on an ordinal scale. Mantel's extension of the Mantel‐Haenszel test is a possible hypothesis‐testing method for this purpose. The development of descriptive statistics for characterizing DIF in polytomous test items has received less attention. A statistic that appears well‐suited as a summary index was proposed by Dorans and Schmitt. In this paper, two possible standard error formulas for this statistic are derived and evaluated, hypothesis testing procedures based directly on this descriptive statistic are outlined, and the results of applications to simulated data are presented.
APPROACHES TO NONIGNORABLE NONRESPONSE WITH APPLICATION TO SELECTION BIAS*
Authors: Allen Nancy L. , Holland Paul W. , Thayer Dorothy T.
Published: August 2014

The main purpose of this paper is to illuminate different approaches to nonignorable nonresponse. Previously literature in educational settings has emphasized bias due to using techniques that are appropriate in the case of ignorable nonresponse when nonignorable nonresponse has actually occurred. The literature in biometrics, econometrics, and statistics contains work that more specifically applicable in this setting. Methods introduced by Pearson, Rubin, Heckman, and Tukey are described and contrasted for two different cases. The two cases include: (1) the simple case where there is one variable of interest and the sampling rule depends on that variable, and (2) the more complicated case where there is one variable of interest and a covariate. In this second case, as in the first, the sampling rule depends on the variable of interest. Finally, the methods, as applied in case 1, are used in an example. The goal of the example is to estimate the total distribution of scores for an essay, when the essay is answered by only some of the test‐takers. Data from an actual test administration where examinees were given an optional essay question were used.
A MISSING DATA APPROACH TO ESTIMATING DISTRIBUTIONS OF SCORES FOR OPTIONAL TEST SECTIONS*
Authors: Allen Nancy L. , Hollanda Paul W. , Thayer Dorothy T.
Published: August 2014

Many testing programs include a section of optional questions, in addition to mandatory parts of a test. For instance, some testing programs have examinees select one of several essays or performance tasks as part of the test. These optional parts of a test are not often truly parallel to one another, making direct equating impossible. Groups of examinees selecting each optional test section are not equivalent to one another, so statistical moderation is impossible unless explicit assumptions are made. The purpose of this paper is to provide a general method based on missing‐data methods for nonignorable nonresponse (Little & Rubin, 1987) to estimate distributions of scores on an optional part of a test. The distributions for each selected test part can then be linked to one another to develop comparable scales. The method is compared with other methods used to put optional test sections on comparable scales. If issues about the validity of assumption are resolved, this method could eventually provide a basis for the combination of scores based on optional portions of a test with those based on common sections of a test.

This paper is the second in a series of three papers. In the first, “Approaches to Nonignorable Nonresponse with Application to Selection Bias” (Allen, Holland, and Thayer, in press, a), we apply missing‐data methods from settings external to education to this setting. A method from the econometric literature (Heckman, 1976) is contrasted to methods proposed by Little and Rubin (1987) and Tukey (Holland, 1986). In this paper, the perspective of measurement specialists who proposed methods to transform scores for optional questions to comparable scales is contrasted with the approach based on the work of Little and Rubin. In the final paper, “Estimating Scores for an Optional Test Section Using Information from a Common Section” (Allen, Holland, & Thayer, in press, b), the methods of Rubin and Tukey are extended to include common scores for the mandatory part of the test. Data from an example are actually transformed to comparable scales.
COLLEGE MAJOR AND GENDER DIFFERENCES IN THE PREDICTION OF COLLEGE GRADES
Authors: Pennock‐RomÁn MarÍa
Published: August 2014

Recent studies have found substantial reductions in gender differences in the prediction of academic achievement in college when variations in grading standards among courses were taken into account. The purpose of this project was to examine gender differences in the prediction of freshman grades after controlling for differential course grading based on college majors. This method involved deriving a variable that measured grading leniency using residual scores from the within‐gender regressions of freshman grades on high school grades and scores on the SAT for the non‐Latino white group. The procedure worked quite well and generalized to other groups not involved in the derivation of the grading‐leniency scale. Nevertheless, there were modest, sometimes statistically significant, gender differences in prediction that remained after this control variable was introduced into the regressions. The largest and smallest differences for females between actual grades and grades predicted from the males' regressions tended to be found in the African American and Asian American groups, respectively. The results imply that the use of information on college majors is a reasonable, practical procedure for controlling for grading leniency.
THE COLLEGE BOARD VOCABULARY STUDY
Authors: Breland Hunter M. , Jones Robert J. , Jenkins Laura , Paynter Marion , Pollack Judith , Fong Y. Fai
Published: August 2014

This study was conducted to provide an up‐to‐date source of word frequency information based on the kinds of reading materials to which high school and first‐year college students are exposed. It began with a comprehensive listing of reading materials from curriculum surveys, state curriculum guides, private school reading lists, research surveys, federal reports, recommended reading lists, and other sources. Materials mentioned most often were sampled or entire documents were obtained when they were available in electronic form. Included in the sample of reading materials were American and British novels, poetry, drama, essays, biographies, autobiographies, current periodicals of various types, historical documents, and text from an encyclopedia.

A corpus of 14,360,884 words of running text was assembled. This corpus was analyzed using the most sophisticated lexicographic methods available and the following statistics were generated: the overall frequency of occurrence of each word in the corpus, an index of dispersion for each word over 27 text categories, an estimate of the number of occurrences per one million words of running text for each word that would be expected in a similar but different corpus, and a standard frequency index developed from a logarithmic transformation.

This report describes the development of the corpus and the computation of the word frequency indexes. It also compares the corpus with other existing corpora and demonstrates the importance of up‐to‐date word frequency information. The comprehensive listing of reading materials examined and a list of sampled materials are included in the Appendixes.
DELIMITING THE VERBAL DOMAIN
Authors: DeMauro Gerald E. , Merritt Ave , Adams Richard
Published: August 2014

A functional analysis was made of the verbal domain as it is defined by the test development process at Educational Testing Service. The thesis of the study is that the verbal domain consists of semantic relations that examinees must interpret. Test developers use a system of rules to sample stimulus materials from among infinite possibilities that elicit an interpretive response. Many of these rules are codified in a body of memoranda and corporate publications, and others are in different stages of formation, beginning in the use by individual test development professionals, and eventually becoming part of generally accepted practice. Because the process of test development is central to the formation of the verbal domain, a job analysis approach was used to examine how rules are employed to present the desired semantic relations to the examinees for their interpretation. This approach explains the process of delimiting the verbal domain through a comprehensive review of the uses of the rule system that defines when the items are functioning properly and when they are not. Test developers from two major divisions of ETS were observed and interviewed. They also rated the importance of each of the rules, suggested rules that they use in their work that were not listed, and delineated the professional activities for which these rules are most important. The study indicates that the rule system for sampling semantic relations is an evolving body, both for the individual test developer and for test development as an institution.
A PRELIMINARY STUDY OF THE EFFECTS OF WITHIN‐GROUP COVARIANCE STRUCTURE ON RECOVERY IN CLUSTER ANALYSIS
Authors: Donoghue John R.
Published: August 2014

Two Monte Carlo studies investigated the effects of within‐group covariance structure on subgroup recovery by several widely used hierarchical clustering methods. Data sets were 100 bivariate observations from two subgroups, generated according to a finite normal mixture model. In Study 1, subgroup size, within‐group correlation, within‐group variance, and distance between subgroup centroids were manipulated. All clustering methods were strongly affected by within‐group correlation; negative correlation yielded much poorer recovery. Smaller effects were found for the interaction of clustering method with within‐group variance.

Study 2 separated the effects of direction of correlation from the direction of differences in the subgroup centroids. Subgroup size, within‐group correlation, direction of the vector separating subgroup centroids, and distance between subgroup centroids were manipulated. Superior recovery was associated with within‐group correlation that matched the direction of subgroup separation. Overall, the EML algorithm of SAS yielded best recovery, followed closely by Ward's method, average linkage, and a version of the beta‐flexible algorithm, although several interactions were noted.

The results are interpreted according to the weakness of the (squared) Euclidean distance as a measure of (dis)similarity for cluster analysis. Several alternative measures are discussed, and promising alternatives are identified for future investigation.
DEFINING THE CONTENT DOMAIN FOR THE PRAXIS II SUBJECT ASSESSMENT IN FRENCH: A JOB ANALYSIS FOCUSING ON KNOWLEDGE AND ABILITIES
Authors: Tannenbaum Richard J.
Published: August 2014

The purpose of this study was to conduct a job analysis focusing on the knowledge and abilities important for beginning French teachers. The results of the job analysis will be used to define the content domain of the Subject Assessment in French for the Praxis Series: Professional Assessments for Beginning Teachers®.

A domain of 212 knowledge statements and ability statements was developed by subject‐matter experts. These statements were then incorporated into a survey that was administered to a large number of French teachers, teacher educators, and state administrators. These professionals rated the importance of the knowledge statements and ability statements using a 5‐point scale.

A cut point of a mean of 2.50 was established to differentiate important statements (M ≥ 2.50) and unimportant statements (M < 2.50). The results indicated that a total of 19 knowledge statements and ability statements failed to pass the cut point. It is recommended that these 19 statements not be included in the development of the Subject Assessment in French; emphasis, instead, should be put on the 193 statements that did pass the cut point.
DEFINING THE CONTENT DOMAIN FOR THE PRAXIS II SUBJECT ASSESSMENT IN BUSINESS EDUCATION: A JOB ANALYSIS FOCUSING ON KNOWLEDGE AND SKILLS
Authors: Tannenbaum Richard J.
Published: August 2014

The purpose of this study was to conduct a job analysis of the knowledge and skills important for the competent performance of beginning business education teachers. The results of the job analysis will be used to define the content domain of the Subject Assessment in Business Education for the Praxis Series: Professional Assessments for Beginning Teachers®.

A domain of 207 important knowledge statements and skill statements was developed by subject‐matter experts. These statements were then incorporated into a survey that was administered to a large number of business education teachers, administrators, and teacher educators. These business education professionals rated the importance of the knowledge statements and skill statements using a 5‐point scale. The survey component functioned as an independent verification or refutation of the importance of the statements defined by the subject‐matter experts. Statements that were judged to have a mean importance rating equal to or greater than 2.50 (midpoint between moderately important and important) by all groups (teachers, administrators, and teacher educators) and all subgroups (race or ethnicity, sex, geographic region, and teaching experience) of survey respondents were considered eligible for inclusion in the development of the Subject Assessment in Business Education.

The results indicated that of the 207 knowledge statements and skill statements, 17 (8%) had a mean rating of less than 2.50 across all groups and subgroups of survey respondents. It is recommended that these 17 statements not be included in the development of the Subject Assessment in Business Education. Primary emphasis, in the test develop process, should be placed on the 190 knowledge or skill statements that did pass the 2.50 cut point.
AN ELECTRONIC INFRASTRUCTURE FOR A FUTURE GENERATION OF TESTS
Authors: Bennett Randy Elliot
Published: August 2014

ETS is moving rapidly to computerize its tests for admissions to post‐secondary education and occupational licensure/certification. Computerized tests offer important advantages, including immediate score reporting, the convenience of testing when the examinee wishes, and for adaptive tests, equal accuracy throughout the score scale and a shorter test with no loss in measurement precision. There is much more that technology can achieve, however. This paper describes an electronic infrastructure for integrating the best of traditional testing approaches with new technology. This multi‐organizational infrastructure has the potential to help assessment contribute more positively to learning and decision making. It can do this by making it easier to deliver tests that employ performance tasks, include important skills not well‐measured by current examinations, sample behavior frequently over a student's school career, and give instructionally useful feedback to individuals.
EVALUATING A PROTOTYPE ESSAY SCORING PROCEDURE USING OFF‐THE‐SHELF SOFTWARE
Authors: Kaplan Randy M. , Burstein Jill , Trenholm Harriet , Lu Chi , Rock Donald , Kaplan Bruce , Wolff Susanne
Published: August 2014

Constructed‐response items, whose responses consist of words, phrases, sentences, paragraphs, and essays are among the most difficult and costly to score. The increased use of constructed‐response items like essays creates a need for tools to partially or fully automatically score these responses. This study explores one approach to analyzing essay‐length natural language constructed‐responses.

In this study we develop and evaluate a decision model for scoring essays. The decision model uses off‐the‐shelf software for grammar and style checking of the English language. The first part of this study consisted of an evaluation of several commercial grammar checking programs. From this evaluation we select the best performing grammar checking programs to construct a decision model for scoring the essays. The second part of the study uses data produced from the selected grammar checking program(s) to make a decision about the score for an essay. Through statistical and linguistic methods, we analyze the performance of the decision model in an effort to understand its usefulness and practicality in a production scoring setting.
AN ANALYSIS OF THE DIMENSIONALITY OF TOEFL READING COMPREHENSION ITEMS
Authors: Schedl Mary , Gordon Ann , Carey Patricia A. , Tang K. Linda
Published: August 2014

The issue of exactly what is measured by different types of reading items has been a matter of interest in the field of reading research for many years. Language teaching and testing specialists have raised the question of whether a reading test for foreign students wishing to enter university in the United States should include questions testing abilities beyond linguistic and very general discourse competencies, or indeed whether it is possible to separate these language competencies from other competencies. The purpose of this study was to investigate the dimensionality of the TOEFL® reading test, based on the specifications in use as of April 1991. Of particular interest was whether four item types identified in the test specifications as “reasoning items” could be shown to measure, in addition to general reading ability, any abilities not measured by the other item types in the TOEFL reading test. Two techniques, Stout's procedure and NOHARM analyses, were employed to investigate the hypothesized two‐factor model. In both cases the data failed to fit the model, indicating that TOEFL “reasoning items” cannot be shown to measure a unique construct. However, the follow‐up exploratory analyses indicated that all 10 test forms used in the study violated the assumption of essential unidimensionality, and all of the forms appeared to fit a two‐factor model where the second factor may be related to passage content or position.