bluuebunny/crossref_metadata_2025 huggingface dataset:
THE ROLE OF PROBABILITY‐BASED INFERENCE IN AN INTELLIGENT TUTORING SYSTEM
Authors: Mislevy Robert J. , Gitomer Drew H.
Published: August 2014

Probability‐based inference in complex networks of interdependent variables is an active topic in statistical research, spurred by such diverse applications as forecasting, pedigree analysis, troubleshooting, and medical diagnosis. This paper concerns the role of Bayesian inference networks for updating student models in intelligent tutoring systems (ITSs). Basic concepts of the approach are briefly reviewed, but the emphasis is on the considerations that arise when one attempts to operationalize the abstract framework of probability‐based reasoning in a practical ITS context. The discussion revolves around HYDRIVE, an ITS for learning to troubleshoot an aircraft hydraulics system. HYDRIVE supports generalized claims about aspects of student proficiency through probability‐based combination of rule‐based evaluations of specific actions. The paper highlights the interplay among inferential issues, the psychology of learning in the domain, and the instructional approach upon which the ITS is based.
RELIABILITY OF ADVANCED PLACEMENT EXAMINATIONS
Authors: Bridgeman Brent , Morgan Rick , Wang Ming‐mei
Published: August 2014

The various methods for computing the reliability of scores on Advanced Placement examinations are summarized. For the free response portion of the examinations, raters can contribute to score unreliability through both systematic severity errors (in which some raters consistently rate more severely than other raters) and through inconsistency. Inconsistency appears to be a much greater problem then systematic severity errors. Question to question variation (or score reliability) is seen as a greater problem than rater inconsistencies. The impact of increasing or decreasing the number of topics is demonstrated by showing the proportion of students correctly classified as the number of topics changes. Procedures to enhance both rater and score reliability are discussed.
MANAGING THE INFLUENCE OF DIF FROM BIG ITEMS: THE 1988 ADVANCED PLACEMENT HISTORY TEST AS AN EXAMPLE
Authors: Wainer Howard , Lukhele Robert
Published: August 2014

Building tests out of large items brings with it a number of problems. One major problem is that it is often too difficult and too expensive to extensively pretest large items. Thus the sorts of screening for flaws that are pro forma for multiple choice items is not often done for large items. In addition, since there are so few large items on an operational test, not counting an entire item that is found to be flawed in an operational administration may be tantamount to aborting that administration. In this paper we examine the efficacy of the alternative of continuous item weighting. This alternative is illustrated on data from the 1988 administration of the College Board's Advanced Placement History Test.
CORRELATES OF SATISFACTION WITH GRADUATE SCHOOL APPLICANTS' PERFORMANCE ON THE GRE WRITING MEASURE
Authors: Powers Donald E. , Fowles Mary E.
Published: August 2014

The focus of the study was the new GRE writing measure and the proposed GRE scoring guide. The objective was to determine the degree to which the features of student essays upon which scores will be based are the same features that graduate educators use at their institutions to evaluate students' writing. A sample of essays, which had been previously reviewed and judged by graduate deans and faculty, were rescored several times by trained readers. Each rescoring was based on a specific trait in the GRE scoring guide – development of ideas, sentence structure, and so on. The influence of each feature on the judgments made by faculty/deans and on the scores assigned by trained readers was compared. No evidence was uncovered to suggest any differences between graduate deans/faculty and GRE essay readers with respect to the bases on which they judge essay quality.
EFFECTS OF DISCLOSING ESSAY TOPICS FOR A NEW GRE WRITING TEST
Authors: Powers Donald E. , Fowles Mary E.
Published: August 2014

To determine the effects on test performance and test validity of releasing GRE essay topics before an examination, some 300 prospective graduate students each wrote two essays, one on a previously unseen topic and one on a topic that was made available about two weeks before the test. Although for each of the four essay topics studied, performance was slightly better when a topic had been disclosed, analyses did not reveal any statistically significant effect of disclosure on test performance. Nor was there any detectable effect of disclosure on the meaning of test scores, as reflected in the relationship of test performance to several independent indicators of writing skill.

Limitations of the study are discussed in terms of its generalizability to a high‐stakes testing situation and the extent to which it addressed both the positive and the negative effects of disclosure.
USING A NEURAL NET TO PREDICT ITEM DIFFICULTY
Authors: Boldt R. F. , Freedle Roy
Published: August 2014

Several studies that attempted to predict item difficulty using linear weighted composites have been done at ETS and elsewhere. The variables in the composites were coded item characteristics. The validity of these predictions was evaluated using multiple correlations between predicted and actual item difficulties. Results of these studies indicate that the validities of the linear predictions were not zero, but neither were they high. Also, success varied with the particular prediction task attempted, with validities varying from .45 to .77. The figures were promising, but more precision would be helpful for item design and test specification. The present study sought to improve on the validity of the linear predictions by using a “neural net,” a technique that is widely used for nonlinear prediction and is borrowed from artificial intelligence applications. Predictions made using the neural net might be more accurate than those made using linear prediction for two reasons: (a) the latter is a special case of neural net prediction, one in which the variables do not interact, and (b) it is thought that item characteristics do interact to produce item difficulty. Unfortunately, it was found that using the neural net did not lead to substantial improvement in prediction when using the same characteristics as arguments that had proven superior for linear prediction. Perhaps the performance of the neural net did not compare more favorably with that of linear prediction because the variables used had already been chosen to be optimal for linear prediction. For this reason, in an extension of the original project, variables that were optimal for the neural net were sought using a search procedure known as “genetic algorithm.” As used in this study, the genetic algorithm is a constrained search procedure intended to find the best of a fixed number of predictors, that is, the same number used in linear prediction. Applying the genetic algorithm did, indeed, lead to improved prediction. And as expected, most of the variables that were effective in linear prediction did not survive the search. However, it proved to be useful to use the variables that were effective in linear prediction as the starting set for the genetic algorithm, rather than using a randomly selected set. Only two variables were found that were useful for prediction in both the linear system and a neural net; several other variables were found that might be useful in a neural net application but were not effective in a linear application. Substantial capitalization on chance was noted in this study, which can weaken the substantive inferences. Finally, results of this study indicate that the performance of the genetic algorithm could be improved by including chromosomes that contain variables found to be effective in a linear prediction system.
JOB ANALYSIS OF THE KNOWLEDGE IMPORTANT FOR NEWLY LICENSED SOCIAL STUDIES TEACHERS
Authors: Norback Judith Shaul , Bukatko Patricia A. , Wesley Scott , Rosenfeld Michael
Published: August 2014

The purpose of this study was to conduct a job analysis focusing on the knowledge important for newly licensed (certified) Social Studies teachers. The results of the job analysis will be used to develop test specifications for the Subject Assessment in Social Studies of the Praxis Series: Professional Assessments for Beginning Teachers™.

A domain consisting of 10 major knowledge areas (United States History, World History, Skills/Themes, Government/Civics/Political Science, Sociology, Anthropology, Psychology, Geography, Economics, and Pedagogy Specific to Social Studies) and 483 knowledge statements was developed by subject‐matter experts.

Due to the large size of the content domain, two survey forms were developed. Form 1 consisted of the major areas of U.S. History, World History, and Skills/Themes and contained 249 specific knowledge statements. Form 2 consisted of the areas of World History, Government/Civics/Political Science, Sociology, Anthropology, Psychology, Geography, Economics, and Pedagogy Specific to Social Stuies and contained 234 statements.

These survey forms were then administered to a national sample of social studies teachers, administrators, and college faculty whose names were obtained from the membership of the National Council of the Social Studies (NCSS). In addition, a supplementary group of teachers from schools with high levels of minority students (80–100%) and teachers with five or less years of teaching experience were included in order to increase the likelihood that a sufficient number of responses from these groups would be available to reflect their perspectives on the job of a newly licensed social studies teacher. These professionals rated the importance of each knowledge area and statement using a 5‐point rating scale that ranged from 0 (of no importance) to 4 (very important).

A cut point of a mean importance rating of 2.50 (the midpoint between moderately important [scale value 2] and important [scale value 3] was established to identify the core of important statements. The results of the mean analysis conducted for teachers and college faculty in the primary sample and for teachers in the supplementary sample for Form 1 showed that 19 of the 249 knowledge statements were rated less then 2.50 by one or more of the three groups. This represents 7.6% of the content domain. The corresponding results for Form 2 yielded 21 (9.0%) statements with mean ratings below the cut point. In the mean analysis for subgroups, 34 (13.7%) statements were rated below 2.50 for Form 1, while in Form 2, 29 (12.4%) were rated below the cut point.

The knowledge statements that were verified to be important by the surveyed teachers, college faculty, and the demographic subgroups should be used as the foundation for the development of test specifications.
USING MULTIMEDIA IN LARGE‐SCALE COMPUTER‐BASED TESTING PROGRAMS
Authors: Bennett Randy Elliot , Goodman Madeline , Hessinger James , Ligget John , Marshall Geoff , Kahn Helen , Zack June
Published: August 2014

To make substantial improvements in how and what we measure, future generations of tests will need to tap nontraditional constructs, base test designs on cognitive principles, and increase the diversity of problem types. The advent of computer‐based tests makes some of these improvements more feasible, especially diversifying problem types. One means of diversification is by including dynamic stimuli such as audio, video, and animation. We give examples of how such stimuli might be employed to assess skill in history, physical education, and the sciences. In addition, we pose an initial set of issues for consideration by operational testing programs.
A TREE‐BASED APPROACH TO PROFICIENCY SCALING & DIAGNOSTIC ASSESSMENT
Authors: Sheehan Kathleen M.
Published: August 2014

A new procedure for generating instructionally‐relevant diagnostic feedback is proposed. The approach involves first constructing a strong model of student proficiency and then testing whether individual students' observed item response vectors are consistent with that model. Diagnoses are specified in terms of the combinations of skills needed to score at increasingly higher levels on a tests's reported score scale. The approach is applied to the problem of developing diagnostic feedback for the SAT I Verbal Reasoning Test. Using a variation of Wright's (1977) person‐fit statistic, it is shown that the estimated proficiency model accounts for 91% of the “explainable” variation is students' observed item response vectors.
SIMULATING THE USE OF DISCLOSED ITEMS IN COMPUTERIZED ADAPTIVE TESTING1,2
Authors: Stocking Martha L. , Ward William C. , Potenza Maria T.
Published: August 2014

Regular use of questions previously made available to the public (disclosed) may provide one way to meet the requirement for large numbers of questions in a continuous testing environment, that is, an environment in which testing is offered at test‐taker convenience throughout the year rather than on a few prespecified test dates. First it must be shown that such use has effects on test scores small enough to be acceptable. In this study simulations are used to explore the use of disclosed items under a worst case scenario which assumes that disclosed items are always answered correctly. Some item pool and test designs were identified in which the use of disclosed items produces effects on test scores that may be viewed as negligible.
VALIDITY AND AUTOMATED SCORING: IT'S NOT ONLY THE SCORING
Authors: Bennett Randy Elliot , Bejar Isaac I.
Published: August 2014

Early work on automated scoring predated the ready availability of mechanisms for inexpensively delivering computer‐based tests and collecting responses. Hence, this work used responses to conventionally delivered tasks that had somehow been translated to machine‐readable form. The necessity of operating in this manner focused attention initially on the empirical characteristics of automated scores. As the availability of computer‐based testing environments grew it became possible to implement entire operational exams and, thus, to think broadly about the implications of automated scoring for validity. In this paper we argue that a comprehensive discussion of validity and automated scoring includes the interplay among construct definition and test and task design; examinee interface; tutorial; test development tools; automated scoring; and reporting—for in the development process these components affect one another. As modern validity theory postulates, the validation argument must, therefore, ideally provide not only empirical evidence of score relationships but also theoretical rationales to support a variety of design decisions. We further argue that the interdependency among computer‐based test components provides a unique opportunity to greatly improve educational and occupational assessment.
A SOFTWARE TOOL FOR RAPIDLY PROTOTYPING NEW FORMS OF COMPUTER‐BASED ASSESSMENTS
Authors: Katz Irvin R.
Published: August 2014

This report describes the Free‐Response Authoring, Delivery, and Scoring System (FRADSS). FRADSS allows test developers, who might have no programming experience, to put their ideas for computer‐based test items directly onto computer. The system was used for the GRE Mathematical Reasoning pre‐pilot and is currently being used for the GRE Engineering New Testing Initiative. The report begins with a description of the “niche” in the test development process that FRADSS is intended to fill, then presents several items authored using FRADSS. After describing the various components of the system (authoring, delivery, and scoring), the report discusses benefits of using FRADSS in the early stages of test development. These benefits were gleaned from interviews with item writers who have used FRADSS in their work.
TOWARD ASSESSMENT OF DESIGN SKILL IN ENGINEERING
Authors: Katz Irvin R. , James Carolyn M.
Published: August 2014

Design is an important part of engineering education, but cannot be assessed easily through current multiple‐choice tests. In this report, we explore the possibility of including design problems in assessments of engineering students. We report a study comparing constructed‐response design problems and constructed‐response versions of typical engineering assessments, labeled “analysis” problems. Through analyses of verbal protocols, we argue that design problems can elicit qualitatively different behaviors compared with analysis problems, and therefore may assess different underlying constructs. While design problems may be a worthwhile addition to assessments of engineering students, such problems present several challenges, including issues of automatic scoring, the amount of information about candidate skill contained in responses, and the potential for construct‐irrelevant variance. These challenges are surmountable, but require careful construction of design assessments to avoid potential pitfalls. This work represents one part of a research program aimed at informing the redesign of the GRE Engineering Subject Test.
EXAMINING THE VALIDITY OF A COMPUTER‐BASED GENERATING‐EXPLANATIONS TEST IN AN OPERATIONAL SETTING
Authors: Bennett Randy Elliot , Rock Donald A.
Published: August 2014

Generating Explanations (GE) is a computer‐delivered item type that presents a situation and asks the examinee to pose as many plausible reasons for it as possible. Previous research suggests that GE measures a divergent thinking ability largely independent of the convergent skills tapped by the GRE General Test. This study was conducted to determine if prior GE validity results generalized to the GRE candidate population, how population groups performed, what effects partial‐credit modeling might have for validity, and what problems were associated with operational administration. Validity results showed that earlier findings were generally supported: GE was found to be reliable but only marginally related to the General Test and to make significant (but small) independent contributions to the explanation of relevant criteria. With respect to population groups, GE produced smaller gender and ethnic group differences than did the General Test and showed the same relations to outside criteria across groups, suggesting it was measuring similar skills in each population. Attempts to model GE responses on a partial‐credit IRT scale succeeded but produced no improvement in relations with external criteria over those obtained by summing raw item scores. Finally, interviews conducted with examinees to detect potential delivery problems suggested that the directions needed to be shortened.
AN INVESTIGATION OF THE VALIDITY OF AN EMPIRICAL BAYES APPROACH TO MANTEL‐HAENSZEL DIF ANALYSIS
Authors: Zwick Rebecca , Thayer Dorothy T. , Lewis Charles
Published: August 2014

We investigated the utility of an empirical Bayes (EB) enhancement to Mantel‐Haenszel (MH) DIF analysis. We assume that the MH statistics are normally distributed and that the prior distribution of underlying DIF parameters is also normal. We use the posterior distribution of DIF parameters to make inferences about the item's true DIF status (the True DIF method) and the posterior predictive distribution to predict the item's future observed status (the Future DIF method). DIF status is expressed in terms of the probabilities associated with each of the five DIF levels defined by the ETS classification system: C‐, B‐, A, B+, and C+. The proposed methods have several possible advantages. First, the use of prior information allows for more stable DIF estimation, especially in small samples‐a feature that should be advantageous in computer‐adaptive testing. The EB approach may convey information about DIF stability in a more useful way. The results represent the state of knowledge about an item's true DIF status as probabilistic, not deterministic. The EB DIF results lend themselves well to graphic display, which may be useful in conveying findings to test developers and DIF committees. Also, results can be projected for future administration conditions that may differ from the present ones. As an alternative method of DIF flagging, we also investigated the use of a loss function to identify potential DIF items; this approach is very general and is not linked to the ETS DIF classification system. A possible advantage of the loss function procedure over the ETS classification system is its greater flexibility.
A BAYESIAN RANDOM EFFECTS MODEL FOR TESTLETS
Authors: Bradlow Eric T. , Wainer Howard , Wang Xiaohui
Published: August 2014

Standard item response theory (IRT) models fit to dichotomous examination responses ignore the fact that sets of items (testlets) often come from a single common stimuli (e.g. a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences such as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g an adaptive test). To model examinations that may be a mixture of independent items and testlets, we modified one standard IRT model to include an additional random effect for items nested within the same testlet. We use a Bayesian framework to facilitate posterior inference via a Data Augmented Gibbs Sampler (DAGS, Tanner and Wong 1987). The modified and standard IRT models are both applied to a data set from a disclosed form of the SAT. We also provide simulation results that indicates that the degree of precision bias is a function of the variability of the testlet effects, as well as the testlet design.
INQUIRING ABOUT EXAMINEES' ETHNICITY AND SEX: EFFECTS ON AP® CALCULUS AB EXAMINATION PERFORMANCE
Authors: Stricker Lawrence J.
Published: August 2014

Steele and Aronson (1995) found that the performance of African‐American subjects on test items portrayed as a problem‐solving task, in a laboratory experiment, was adversely affected when they were asked about their ethnicity. This outcome was attributed to “stereotype threat”: performance was disrupted by the subjects' concerns about fulfilling the negative stereotype concerning African Americans' intellectual ability. Extending that research, this field experiment evaluated the effects of inquiring about ethnicity and sex on the performance of examinees taking the Advanced Placement (AP) Calculus AB Examination in an actual test administration. With a minor exception, this inquiry had no statistically and practically significant effects on the test performance of African‐American, female, or other subgroups of examinees.
COMPUTER FAMILIARITY AMONG TOEFL EXAMINEES
Authors: Kirsch Irwin , Jamieson Joan , Taylor Carol , Eignor Daniel
Published: August 2014

The growing use of computer‐based testing raises concerns about access and equity. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 may introduce bias by confounding the measurement of English‐language proficiency with computer familiarity. As the first study in a series of research efforts to address this question, a 23‐item questionnaire focusing on examinees' access to, attitude toward, and experience using computers as well as related technologies was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration.

The examinee response rate for the survey was over 90%, and the sample of respondents was determined to be highly similar to the annual TOEFL examinee population in terms of gender, native language, test center region, TOEFL test score ranges, and reason for taking the test.

Factor analysis procedures (specifically, a principal factors approach with varimax rotation) were used to determine the dimensionality of the survey questionnaire. The first factor from a two‐factor common factor solution consisted of 11 items and had a reliability of .91. This factor was used to classify TOEFL examinees into one of three groups according to their level of familiarity with computers: low, moderate, or high. Overall, some 16% of the TOEFL population was judged to have low computer familiarity, another 34% to have moderate familiarity, and approximately 50% to have high familiarity. In terms of background characteristics, computer familiarity was found to be unrelated to age, but was related to gender, native language, region of the world where the examinee was born, and test‐center region. Computer familiarity was also shown to be related to individuals' TOEFL test scores and their reason for taking the test but unrelated to whether or not they had taken the test previously.

Analyses of variance were conducted to investigate the relationships among proficiency as measured by the paper‐and‐pencil TOEFL test, computer familiarity, and selected background characteristics, including gender, age, native language, and reason for taking the test. Effect sizes for each variable as well as for the interactions among the variables are discussed herein. The study results are also considered in light of existing research relating to various aspects of computer familiarity as well as in terms of their implications for professionals in universities and intensive language programs who will need to serve the needs of this diverse set of learners.
AUTOMATIC SCORING OF GENERALIZABLE FIGURAL RESPONSES
Authors: Reid‐Green Keith S.
Published: August 2014

Classes of figural responses such as bar graphs, number lines, fractions of spaces, etc., can be scored by a single program for each class. Item writers can therefore change item parameters without the need for a change to the scoring program.
QUALIFYING ESSAY READERS FOR AN ONLINE SCORING NETWORK (OSN)
Authors: Powers Don , Kubota Mel , Bentley Jill , Farnum Marisa , Swartz Rich , Willard Ann
Published: August 2014

Currently, personnel who evaluate examinee‐constructed responses for ETS‐administered testing programs must possess certain academic credentials. This study was designed to determine the extent to which current prerequisites for essay readers might be relaxed without sacrificing quality with regard to the accuracy with which essay scores are awarded. The specific interest was in a particular class of potential readers, namely, those who are not currently involved in postsecondary teaching and who therefore do not meet current standards for ETS essay readers. Secondary objectives of the study were to (1) provide information that would facilitate the determination of a passing score on the certification test that all ETS essay readers will be required to pass in the future and (2) evaluate, in a preliminary fashion, the effects of training on the performance of essay readers.

To accomplish these objectives, both experienced and inexperienced readers were recruited for the study. All study participants evaluated a set of essays before any training and another set after they had undergone standard training for scoring essays.

The results showed that training did affect the accuracy with which readers (especially those who were previously inexperienced) scored essays. Moreover, after training, a significant proportion of inexperienced readers exhibited a level of accuracy that was commensurate with that shown by experienced readers. A comparison of the performances of experienced and inexperienced readers suggested that a defensible passing score of about 90% accuracy could be set on the reader certification test when the standard of accuracy is agreement within one point of previously established essay scores.
INTERCULTURAL TRANSITIONS IN HUMAN DEVELOPMENT AND EDUCATION
Authors: Laosa Luis M.
Published: August 2014

Intercultural transitions are alterations that occur in individuals and in their environments as a consequence of contact between cultures. This paper focuses on the life events that children (and the adults in their lives) may face in bridging cultures, the options they are given, the choices they make‐or that adults make for them‐the circumstances and experiences that accompany these choices, and their consequences. The paper identifies and discusses issues in need of research concerning ways to foster scholastic success and socioemotional well‐being in populations that contemporary policies and practices may be putting at risk for school failure and for other forms of marginality. A better understanding of these issues can contribute to the improvement of intergroup relations, and it can be used to enhance the educational development and life chances of individuals from all ethnocultural and linguistic backgrounds.
UNIVARIATE AND BIVARIATE LOGLINEAR MODELS FOR DISCRETE TEST SCORE DISTRIBUTIONS
Authors: Holland Paul W. , Thayer Dorothy T.
Published: August 2014

The well‐developed theory of exponential families of distributions is applied to the problem of fitting the univariate histograms and discrete bivariate frequency distributions that often arise in the analysis of test scores. These models are powerful tools for many forms of parametric data smoothing and are particularly well‐suited to problems where there is little or no theory to guide a choice of probability models, e.g. smoothing a distribution to eliminate roughness and zero frequencies in order to equate scores from different tests. Attention is given to efficient computation of the maximum likelihood estimates of the parameters using Newton's Method and to computationally efficient methods for obtaining the asymptotic standard errors of the fitted frequencies and proportions. We discuss tools that can be used to diagnose the quality of the fitted frequencies for both the univariate and the bivariate cases. Five examples, using real data, are used to illustrate the methods of this paper.
SCIENCE AND ENGINEERING GRADUATES: CAREER ADVANCEMENT AND CAREER CHANGE
Authors: Grandy Jerilee
Published: August 2014

This project studied science and engineering (S/E) graduates who, after having been out of college some years, planned to return to graduate school either to advance their careers or to change careers. Two samples were studied.

First, from the 1990–91 GRE files, there were more than 10,000 examinees who had bachelor's degrees in mathematics, natural sciences, computer sciences, or engineering (S/E) and had been out of college at least 5 years. All were either applying to graduate school or had just begun graduate work. About 40% of engineers planned to change fields, and more than half of science graduates planned to switch. This great a loss of talent from the science/engineering talent pool motivated the second part of the study.

The second part of the study consisted of a questionnaire survey of a stratified sample drawn from the 1991–92 GRE files of science and engineering graduates who had been out of college at least five years. Women and minorities were oversampled so that separate analyses could be done for members of these groups. The sample was surveyed by mail to learn what factors might explain why some graduates were planning to continue in S/E while others were switching to different fields. Responses were obtained from 2,201 (82% of the original sample).

People changing fields, on average, were older and had been out of college longer. More important than any other variable was that they placed a greater value on making a contribution to society and working directly with other people than on the satisfaction of technical challenge. There was no evidence that people switched fields because they were unemployed.

Field switching was more frequent for females than males, for African Americans more than other ethnic groups, and for graduates in biological sciences more than graduates in other fields. Correlates of field switching were studied, and differences by gender and ethnic group were noted. There was very little gender difference in ratings of satisfaction with the most recent job in S/E. Gender was associated with many other variables, such as the importance of making a contribution to society. In a path analysis model using LISREL, the entire association between gender and continuation in S/E was accounted for by other variables.
EFFECTS OF COACHING ON SAT® I: REASONING SCORES
Authors: Powers Donald E. , Rock Donald A.
Published: August 2014

A College Board‐sponsored survey of a nationally representative sample of 1995–96 SAT® takers yielded a data base for more than 4,000 examinees, about 500 of whom had attended formal coaching programs outside their schools. Several alternative analytical methods were used to estimate the effects of coaching on SAT I: Reasoning scores. The various analyses produced somewhat different estimates. All of the estimates, however, suggested that the effects of coaching are far less than is claimed by major commercial test preparation companies. The revised SAT I does not appear to be any more coachable than its predecessor.
COMPARABILITY OF PAPER‐AND‐PENCIL AND COMPUTER ADAPTIVE TEST SCORES ON THE GRE® GENERAL TEST
Authors: Schaeffer Gary A. , Bridgeman Brent , Golub‐Smith Marna L. , Lewis Charles , Potenza Maria T. , Steffen Manfred
Published: August 2014

This report summarizes a study conducted to assess the comparability of paper‐and‐pencil (P&P) and computer adaptive test (CAT) scores on the GRE® General Test. Volunteer examinees from around the country were randomly assigned to take the test in either CAT mode or P&P mode. Results indicated that for each measure (verbal, quantitative, analytical), mean scores on the CAT were higher than mean scores on P&P. It was hypothesized that CAT examinee test‐taking behavior with regard to the CAT scoring method that was in place at the time of the study (allowing student to receive a score by answering only 80% or more of the questions) may have affected CAT scores. Investigation of this hypothesis indicated that CAT examinees who did not complete their CATs obtained higher mean scores than would be predicted. However, mean scores for examinees who completed their CATs were similar to mean scores for P&P examinees. A new psychometrically‐defensible CAT scoring method, called proportional scoring, was developed in which it is to the examinees' advantage, in terms of maximizing their scores, to carefully consider and answer as many items as they can. The proportional scoring method has been implemented in the operational GRE CAT program, and it is believed that this scoring method will result in CAT scores that are comparable to P&P scores. GRE CAT scores will continue to be closely monitored.
MODELS OF THE RELATIONSHIP BETWEEN EDUCATION SPENDING AND THE SOCIAL DISTRIBUTION OF ACHIEVEMENT
Authors: Wenglinsky Harold
Published: August 2014

This study relates various types of educational spending to mean mathematics achievement and its social distribution among students. A nationally representative database of 7,217 twelfth graders was developed by linking the 1992 National Assessment of Educational Progress in mathematics to the U.S. Common Core of Data and the School District Data Book. Through applying Hierarchical Linear Modeling to the database, the study found that spending on instruction and capital expenditures, while not related to mean achievement levels, were related to differences in achievement between SES groups; lower spending levels are associated with greater achievement gaps within schools.
THREE RESPONSE TYPES FOR BROADENING THE CONCEPTION OF MATHEMATICAL PROBLEM SOLVING IN COMPUTERIZED‐ADAPTIVE TESTS
Authors: Bennett Randy Elliot , Morley Mary , Quardt Dennis
Published: August 2014

We describe three open‐ended response types that should broaden the conception of mathematical problem solving embodied in computerized‐adaptive admissions tests. The Mathematical Expressions response type presents single‐best answer problems that call for an algebraic formalism, the correct rendition of which may take an infinite number of surface forms. Generating Examples presents loosely structured problems which may have many good answers, taking the form of a value, letter pattern, expression, equation, or list. Graphical Modeling problems ask the examinee to represent a given situation by plotting points on a grid, where there may be a single‐best answer or multiple correct responses. Finally, these three basic types can be combined to form extended constructed‐response problems. For each of the three basic types, this paper gives example items, describes the examinee interface and approach to automatic scoring, and reports available findings.
AN INVESTIGATION OF THE SIMULTANEOUS MODERATION OF AVERAGE GENDER AND AFRICAN‐AMERICAN SCORE DIFFERENCES ON A TEST OF MATHEMATICAL REASONING
Authors: Stocking Martha L. , Jirele Thomas , Lewis Charles , Swanson Len
Published: August 2014

A pool of items from operational tests of mathematical reasoning was constructed to investigate the feasibility of using automated test assembly methods to simultaneously moderate possibly irrelevant differences between the performance of women and men and African‐American and White test takers. None of the artificial tests investigated exhibited substantial impact moderation, although the estimated mean scaled score differences for the relevant population indicated a modest move in the intended direction: the difference between scaled score means was reduced by about 20% for women and men and about 9% for African‐American and White test takers. Although many issues in the implementation of this methodology remain to be solved, the consideration of impact in automated test assembly along with the maintenance of the detailed test plan appears to be a potential method of moderating possibly irrelevant mean test score differences.
THE POWER OF THE K‐INDEX (OR PMIR) TO DETECT COPYING
Authors: Lewis Charles , Thayer Dorothy T.
Published: August 2014

At Educational Testing Service, the K‐Index is used to assess unusual agreement between the incorrect responses of two test takers on a multiple‐choice test. Since the Fall of 1996, the value of the K‐Index used for this assessment has been adjusted, based on the Bonferroni inequality. The resulting value is referred to as the Probability of Matching Incorrect Responses (PMIR). If a test taker copies from someone (a Source) with no incorrect responses, the PMIR is useless (i.e., has zero power) for detecting unusual agreement. Similarly, if a Source has relatively few incorrect responses, the power of the PMIR will be low. We propose a framework within which to study the power of the PMIR. The power function derived within this framework exhibits quite complex behavior; however, we conclude that the power of the PMIR to detect substantial amounts of copying is quite low, even when the Source has a relatively large number of incorrect responses.
EXPLORATION OF AN AUTOMATED EDITING TASK AS A GRE® WRITING MEASURE1
Authors: Breland Hunter M.
Published: August 2014

Two editing tasks were developed and programmed for the computer to explore the possibility that such tasks might be useful as measures of writing skill. An informal data collection was then conducted with 52 prospective graduate students. These students completed the editing tasks with no time limit, as well as a writing experience questionnaire. Scores obtained on the two editing tasks were correlated with variables developed from the questionnaire. The total score for the two editing tasks correlated .52 with student self‐assessments of their writing ability, .46 with grade‐point average (GPA) based on courses requiring at least some writing, and .30 with writing accomplishments. The correlation with GPA, however, was only .14. The reliability of the total editing score was estimated at .84.
VALIDITY OF THE SECONDARY LEVEL ENGLISH PROFICIENCY TEST AT TEMPLE UNIVERSITY‐JAPAN
Authors: Wilson Kenneth M. , Graves Kimberly
Published: August 2014

The study reported herein assessed levels and patterns of concurrent correlations for Listening Comprehension (LC) and Reading Comprehension (RC) scores provided by the Secondary Level English Proficiency (SLEP) test with direct measures of ESL speaking proficiency (interview rating) and writing proficiency (essay rating), respectively, and (b) the internal consistency and dimensionality of the SLEP, by analyzing intercorrelations of scores on SLEP item‐type “parcels” (subsets of several items of each type included in the SLEP). Data for some 1,600 native‐Japanese speakers (recent secondary‐school graduates) applying for admission to Temple University‐Japan (TU‐J) were analyzed. The findings tended to confirm and extend previous research findings suggesting that the SLEP – which was originally developed for use with secondary‐school students–also permits valid inferences regarding ESL listening‐ and reading‐comprehension skills in postsecondary‐level samples.
THE ROLE OF THE UNIT OF ANALYSIS IN DIMENSIONALITY ASSESSMENT
Authors: Dorans Neil J. , Lawrence Ida M.
Published: August 2014

A variety of techniques exist for addressing the fundamental question: What does a test measure? The answer to that question often depends on how one asks the question of the data. At one extreme, we see correlations of a test with other editions of that test (e.g., parallel form reliability estimates). At the other extreme, we see exploratory factor analyses of item‐level data. In this paper, we argue that item‐level and test‐score dimensionality are not necessarily identical, define mathematical relationships among item‐level factors, test‐score factors and parcel‐level factors, and advocate that the choice of dimensionality technique should be based on the purpose of the dimensionality analysis. Finally, we illustrate how different approaches yield different answers with SAT‐Verbal data.
ITEMS BY DESIGN: THE IMPACT OF SYSTEMATIC FEATURE VARIATION ON ITEM STATISTICAL CHARACTERISTICS
Authors: Enright Mary K. , Morley Mary , Sheehan Kathleen M.
Published: August 2014

This study investigated the impact of systematic item feature variation on item statistical characteristics and the degree to which such information could be used as collateral information to supplement examinee performance data and reduce pretest sample size. Two families of word problem variants for the quantitative section of the Graduate Record Examination (GRE®) General Test were generated by systematically manipulating item features. For rate problems, the item design features affected item difficulty (Adj. R2 = .90), item discrimination (Adj. R2 = .50), and guessing (Adj. R2 = .41). For probability problems the item design features affected difficulty (Adj. R2 = .61), but not discrimination or guessing. The results demonstrate the enormous potential of systematically creating item variants. However, questions of how best to manage variants in item pools and to implement statistical procedures that use collateral information must still be resolved.
GRAPHICAL MODELING: A NEW RESPONSE TYPE FOR MEASURING THE QUALITATIVE COMPONENT OF MATHEMATICAL REASONING
Authors: Bennett Randy Elliot , Morley Mary , Quardt Dennis , Rock Donald A.
Published: August 2014

This study investigated the psychometric functioning of Graphical Modeling (GM) – a new computer‐delivered response type for assessing mathematical reasoning that asks candidates to respond to a problem situation by creating a graphical representation. GM problems can be like the single‐best‐answer items currently found on the General Test or they can be more loosely defined, allowing for multiple correct responses. Two GM tests differing from one another in the manipulation of specific item features were randomly spiraled among study participants. Analyses were performed relating to internal consistency reliability, relations with external criteria, features that contribute to item difficulty, adverse gender impact, and examinee perceptions. Results showed that GM scores were very reliable and moderately related to the General Test's quantitative section, suggesting that the introduction of GM items on the General Test might help broaden the GRE quantitative construct. In exploratory analyses of difficulty, one of three manipulated item features, problem structure, had a significant effect. Our impact analyses detected no significant gender differences independent of those associated with the GRE quantitative section. Finally, while more participants preferred regular multiple‐choice graphical reasoning questions to GM items, more also thought GM was the fairer indicator of their ability to undertake graduate study.
ASSESSING TIME TRENDS IN SEX DIFFERENCES IN SWIMMING AND RUNNING
Authors: Wainer Howard , Njue Catherine , Palmer Samuel J.
Published: August 2014

Men and women perform at different levels in athletics. In swimming and running races men are almost always faster. Over the past century the differences observed between men and women have been shrinking. This shrinkage seems to have been more dramatic at longer distances. In this article we examine the time trends in these differences by event through examination of world records and Olympic performances. We show that in running, using one sensible measure, women seem to be nearer men's performances in sprints than in distance races; in swimming the reverse is true. This measure provides better predictive inferences than the typical alternatives.
A SIMULATION STUDY OF THE EFFECT OF RATER DESIGNS ON ABILITY ESTIMATION
Authors: Hombo Catherine M. , Donoghue John R. , Thayer Dorothy T.
Published: August 2014

As more assessment programs move towards the use of constructed response and performance assessment items, the use of human raters to score these items necessarily will increase. Different designs can be created to assign raters to score examinee responses, and in this study some of these designs are evaluated in terms of their impact on the accuracy of examinee ability estimation. As expected, the optimum design where every rater judges every performance by every examinee results in ability estimates with minimum bias and small standard error. As this design is rarely, if ever, practical in real assessment situations, the results for nested and spiral rater designs are of more interest to practitioners. The nested rater designs result in biased ability estimates for the examinees judged by the most extreme raters, but the spiral rater designs examined prove surprisingly robust to rater tendencies.
SUBSTITUTING SAT® II: SUBJECT TESTS FOR SAT I: REASONING TEST: IMPACT ON ADMITTED CLASS COMPOSITION AND QUALITY
Authors: Bridgeman Brent , Burton Nancy , Cline Frederick
Published: August 2014

Using data from a sample of 10 colleges at which most students had taken both SAT® I: Reasoning Test and SAT II: Subject Tests, we simulated the effects of making selection decisions using SAT II scores in place of SAT I scores. Specifically, we treated the students in each college as forming the applicant pool for a more select college, and then selected the top two‐thirds (and top one‐third) of the students using high school grade point average (HSGPA) combined with either SAT I scores or the average of SAT II scores. Success rates, in terms of freshman grade point averages, were virtually identical for students selected by the different models. The percent of African American, Asian American, and White students selected varied only slightly across models. Appreciably more Mexican American and Other Latino students were selected with the model that used SAT II scores in place of SAT I scores because these students submitted Subject Test scores for the Spanish Test on which they had high scores.