bluuebunny/crossref_metadata_2025 huggingface dataset:
DIF DATA ON FREE‐RESPONSE SAT I MATHEMATICAL ITEMS1
Authors: Lawrence Ida M. , Lyu C. Felicia , Feigenbaum Miriam D.
Published: August 2014

The present research summarized Differential Item Functioning (DIF) results for Student Produced Response (SPR) items, a non‐multiple choice mathematical item type in the SAT I. DIF data from four field trial pretest administrations (620 SPR items) and ten final forms (100 SPR items) were examined. For the Male/Female comparison, the SPR item type exhibits negligible DIF. The Male/Female DIF results for the SPR items resemble DIF results for the multiple choice items in the SAT. For the African American/White comparison, SPR items exhibit more DIF than the multiple choice items. (Moderate or sizable DIF on multiple choice items in the old or new SAT‐M is rare.) In particular, when African American test takers are compared to matched groups of White test takers, they tend to perform less well than White test takers on the SPR items. Although the level of African American/White DIF on SPR items is elevated, relative to the multiple choice items, none of the SPR items in any of the final forms were flagged for exhibiting extreme DIF. The relationship between DIF and discrimination is discussed as a possible explanation for the elevated level of DIF on the SPR items.
EFFECTS OF MODE OF ITEM PRESENTATION ON STANDARD SETTING
Authors: Faggen Jane , Melican Gerald , Powers Don
Published: August 2014

The objective of this study was to determine the degree to which recommendations for passing scores, calculated on the basis of a traditional standard‐setting methodology, might be affected by the mode (paper vs. computer screen prints) in which test items were presented to standard setting panelists. Results were based on the judgments of 31 panelists for items in each of three parts of a computer‐based academic skills assessment designed for use as part of the teacher licensing process. None of the evidence collected–either the analysis of panelists' estimates of item difficulty or their perceptions of effects–revealed any differences in recommended passing standards resulting from the mode of item presentation.
INTELLIGENCE TESTING AND SOCIAL POLICY
Authors: Laosa Luis M.
Published: August 2014

There is a resurgence of scientific and public interest and controversy centering on four interrelated themes: intelligence testing; racial, ethnic, and socioeconomic differences in measured IQ; genetic and environmental influences on abilities; and the role of scientific research in social policy. Given the polarization of views, dispassionate discussion is needed to help inform the ongoing debate. The aim of this paper is to contribute to this discussion by helping to clarify certain issues, assumptions, and concerns that, because they are often misunderstood or ignored, tend to obfuscate the debate.
THE PERSONAL STATEMENT AS AN INDICATOR OF WRITING SKILL: A CAUTIONARY NOTE
Authors: Powers Donald E. , Fowles Mary E.
Published: August 2014

The objective of this study was to evaluate the personal statement as an indicator of writing skill. The evaluation was based on a comparison of specially‐evaluated personal statements with a standardized measure of writing ability requiring examinees to write timed expository essays. A sample of prospective graduate students wrote essays and provided copies of personal statements they had submitted for application to graduate school. A majority of the sample acknowledged receiving help in either drafting or revising their statements. Correlations of the two indicators (test essay and personal statement) with each of several non‐test indicators of writing skill revealed that the traditional expository essay was significantly more highly related to nearly all of the indicators. It is suggested that, while the personal statement may provide certain unique and important information about applicants (and thus be valid for this purpose), its validity as an indicator of writing skill is suspect.
CHOICE AMONG ESSAY TOPICS: IMPACT ON PERFORMANCE AND VALIDITY
Authors: Bridgeman Brent , Morgan Rick , Wang Ming‐mei
Published: August 2014

This study assessed the ability of history students to choose the essay topic on which they can get the highest score. A second, equally important, question was whether the score on the chosen topic was more highly related to other indicators of proficiency in history than the score on the unchosen topic. Overall, for both US and European history, scores were about one‐third of a standard deviation higher for the preferred topic than for the other topic. For US history, about 32% of the students made the wrong choice, that is 32% got a higher score on the other topic than on the preferred topic. In European history, 29% made the wrong choice. In the US history sample, the preferred essay correlated .40 with an external criterion score vs .34 for the other essay; in the European history sample, the preferred essay correlated .52 with the external criterion compared to .44 for the other topic.
RESEARCH PERSPECTIVES ON CONSTRUCTS OF CHANGE: INTERCULTURAL MIGRATION AND DEVELOPMENTAL TRANSITIONS
Authors: Laosa Luis M.
Published: August 2014

In every society there are certain important life events that are predictable and normative therein and that demand from the individual some major and lasting personal change; they are often–but not always–connected with a biological change of life (e.g., puberty, graduations, maturity, parenthood, retirement). In contrast to these transitions, there also exist other critical life events, which are less prevalent and less predictable, and which place extreme demands on the individual for personal change and adaptation. These extraordinary experiences include intercultural transitions, particularly those associated with international migration. In recent years, the need for research attention to this type of life transition in individual development has increased, largely as a result of significant changes in immigration trends and in the ethnocultural mix of the nation's population. A multifaceted, complex, and consequential event, intercultural migration poses special challenges as a subject of scientific inquiry. At the same time, it provides a unique window for viewing processes of change and adaptation. In this paper some of these needs, challenges, and opportunities are identified and discussed in the context of varied research perspectives and constructs of change.
MEASURING ACCOMPLISHMENTS: PSEUDOIPSATIVITY, QUANTITY VS. QUALITY, AND DIMENSIONALITY
Authors: Stricker Lawrence J. , Rock Donald A.
Published: August 2014

This study examined three questions about measures of accomplishments–notable attainments that have been publicly recognized: their pseudoipsativity, the correspondence between quantity and quality scores, and their dimensionality. Comparable samples of graduate students described their accomplishments on a questionnaire or judged the similarity of the same accomplishments. Accomplishments in the same field were positively correlated, while accomplishments in different fields varied in their correlations, some being positive and others being negative; these results are inconsistent with the predominantly negative correlations that would occur with ipsative measures. Measures of the quantity and quality (importance, rarity) of accomplishments correlated highly, after correction for attenuation, and appeared to assess the same thing. And analyses of the self‐report data found that accomplishments were factorially complex, with many of the same factors identified in other kinds of measures appearing; the factors observed in the judgment data largely corresponded to the semantic features of the accomplishment items.
DIFFERENCES IN STRATEGIES USED TO SOLVE STEM‐EQUIVALENT CONSTRUCTED‐RESPONSE AND MULTIPLE‐CHOICE SAT®‐MATHEMATICS ITEMS
Authors: KATZ IRVIN R. , FRIEDMAN DEBRA E. , BENNETT RANDY ELLIOT , BERGER ALIZA E.
Published: August 2014

This study investigated the strategies subjects adopted to solve stem‐equivalent SAT‐Mathematics (SAT‐M) word problems in constructed‐response (CR) and multiple‐choice (MC) formats. Parallel test forms of CR and MC items were administered to subjects representing a range of mathematical abilities. Format‐related differences in difficulty were more prominent at the item level than for the test as a whole. At the item level, analyses of subjects' problem‐solving processes appeared to explain difficulty differences as well as similarities.

Differences in difficulty derived more from test‐development than from cognitive factors: On items in which large format effects were observed, the MG response options often did not include the erroneous answers initially generated by subjects. Thus, the MC options may have given unintended feedback when a subject's initial answer was not an option or allowed a subject to choose the correct answer based on an estimate.

Similarities between formats occurred because subjects used similar methods to solve both CR and MC items. Surprisingly, when solving CR items, subjects often adopted strategies commonly associated with MC problem solving. For example, subjects appeared adept at estimating plausible answers to CR items and checking those answers against the demands of the item stem.

Although there may be good reasons for using constructed‐response items in large‐scale testing programs, multiple‐choice questions' of the sort studied here should provide measurement that is generally comparable to stem‐equivalent constructed‐response items.
COMPARING THE EFFECTIVENESS OF PROCEDURES TO DETECT ITEM PARAMETER DRIFT
Authors: Donoghue John R. , Isham Steven P.
Published: August 2014

This study used Monte Carlo methods to examine the use of several methods to identify item parameter drift. Number of examinees, number of items, and number of drift items in the test were manipulated. Overall, Lord's χ2 measure was the most effective in identifying items that exhibited drift. However, the measure was only accurate when the c‐parameter was constrained to be equal across calibrations. Of the remaining methods, the best (z‐test based on Raju's exact unsigned integral, the PARSCALE computer program's χ2 by subgroup, and Kim and Cohen's closed interval signed area measure) required empirical estimates of critical values for the test statistics in order to function well. This aspect detracts from their usefulness.
ON THE CONSEQUENCES OF SOME TEST RESCORING POLICIES
Authors: Wainer Howard , Bradlow Eric T.
Published: August 2014

When tests or portions of tests are scored subjectively by raters, a rescoring will yield a change in the ratings of some examinees. In a high stakes test with a fixed passing score a rescoring will result in the change of some pass/fail decisions. The number of changes depends on three things: (1) the reliability of the rating system, (2) the proportion of examinees that are expected to pass, and (3) the policy used to incorporate the rescore into the pass/fail decision. In this study, we provide a model that facilitates the evaluation of various rescoring strategies. We illustrate the use of this model for rescoring schemes that concatenate all ratings, old and new. The estimation of rescoring effects are computed by direct simulation. A further generalization of the basic model is also considered in which a test is comprised of a mixture of objectively and subjectively scored items. Last, we note that rescoring is merely a special case of the more general problem of the optimal allocation of judges and demonstrate an algorithm that accomplishes this.
MODELING THE RELATIONSHIP BETWEEN SCHOOL DISTRICT SPENDING AND ACADEMIC ACHIEVEMENT: A MULTIVARIATE ANALYSIS OF THE 1992 NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS AND THE COMMON CORE OF DATA
Authors: Wenglinsky Harold
Published: August 2014

In the wake of the Coleman Report, which found student socio‐economic status (SES) to play a large role in student performance and the economic resources of the school to play a small or non‐existent one, sociologists of education have tended to assume that inequalities in the economic resources of school districts do not reinforce inequalities in achievement between high SES and low SES students. Yet there is in fact a lack of consensus among educational researchers as to whether or not economic resources are associated with student achievement. This study attempts to test the hypothesis that money does matter, by applying the statistical technique of linear structural modeling to a nationally representative sample of eighth graders from data drawn from the National Assessment of Educational Progress and the Common Core of Data. The analysis finds that the data support the hypothesis that per‐pupil expenditures on instructional and central office administration are positively related to small class size, which in turn is positively related to academic achievement.
A TREE‐BASED APPROACH TO PROFICIENCY SCALING
Authors: Sheehan Kathleen M.
Published: August 2014

The process of translating examinees' observed test results into probabilistic statements about the patterns of skill mastery associated with performance at successive points on a test's reported score scale has been termed proficiency scaling. This paper introduces a new approach to proficiency scaling which was specifically developed to provide accurate, instructionally‐relevant score interpretations, even when the test under consideration is a non‐diagnostic, broad‐based ability test such as the SAT. The approach employs techniques derived from the realm of tree‐based regression to classify test items into nonoverlapping clusters such that all of the items in each cluster have similar values of item difficulty and require similar combinations of skills. A nonparametric smoothing procedure is then used to summarize cluster performance conditional on underlying test score. The resulting cluster characteristic curves provide the probability of responding correctly to items with specified skill requirements, expressed as a function of the underlying test score. This new approach is described in the context of a specific application: developing proficiency interpretations for the reading comprehension subsection of the SAT I Verbal Reasoning Test. It is shown that the generated proficiency interpretations account for 91% of the variation which would be accounted for by an optimal clustering solution. These results suggest that the approach is capable of extracting useful information about the specific combinations of skills needed to perform at increasingly higher levels on a test's reported score scale.
USING JUST NOTICEABLE DIFFERENCES TO INTERPRET TEST OF SPOKEN ENGLISH SCORES
Authors: Stricker Lawrence J.
Published: August 2014

This study explored the value of obtaining a Just Noticeable Difference (JND) — the difference in scores needed before observers discern a difference in examinees' English proficiency – for the current Test of Spoken English (TSE®) as a means of interpreting scores in practical terms, using college students' ratings of their international teaching assistants' English proficiency and adapting classical psychophysical methods. The test's concurrent validity against these ratings was also appraised. Three estimates of the JND were obtained. They varied considerably in size, but all were substantial when compared with the standard deviation of the TSE scores, the test's standard error of measurement, and guidelines for the effect size for mean differences. The TSE test correlated moderately with the rating criterion. The JND estimates appear to be meaningful and useful in interpreting the practical significance of TSE scores, and the test has some concurrent validity.
LINKING SCORES FOR COMPUTER‐ADAPTIVE AND PAPER‐AND‐PENCIL ADMINISTRATIONS OF THE SAT
Authors: Lawrence Ida , Feigenbaum Miriam
Published: August 2014

In October 1996, a computer‐adaptive version of the SAT was administered in an operational setting for the first time. This version of the test, the Computerized SAT, was administered to students who are seeking placement into various talent search programs. Because many students seeking placement into talent search programs will take the paper‐and‐pencil (P&P) SAT I, and scores between the traditional P&P test and the Computerized SAT (CAT) will be treated as interchangeable, it was necessary to conduct a field trial to insure that reported scores for the two modes of testing were on the same scale. In addition, because students seeking placement into talent search programs are typically in the 7th‐ or 8th‐grade and not part of the SAT I target population (i.e., high school juniors and seniors), it was necessary that the field trial use samples taken from the target population. The data collected in the field trial were used investigate the relationship between CAT and P&P scores in the target population for the SAT I. An anchor test design was used to scale the scores from the CAT to scores on the P&P measure. In addition to providing scaling data for the CAT, this study provided an opportunity to gather information about examinees' perceptions of the test‐taking experience in comparison to their experience with the paper‐based version of the SAT I.
POSSIBLE DETERMINANTS OF GENDER DIF ON THE ADVANCED PLACEMENT PSYCHOLOGY TEST: FAMILIARITY, INTEREST, AND EMOTIONAL REACTION
Authors: Stricker Lawrence J. , Emmerich Walter
Published: August 2014

This study evaluated the connection between gender differences in Advanced Placement (AP) Psychology students' familiarity, interest, and negative emotional reaction to items on the AP Psychology test and the items' gender differential item functioning (DDF). Gender DDF and gender differences in interest and emotional reaction varied appreciably with the content of the items. Gender differences in the three variables were substantially related to the items' gender DDF (e.g., R = .50). Much of the gender DDF on this test may be attributable to gender differences in these variables.
GRAPHICAL MODELS AND COMPUTERIZED ADAPTIVE TESTING
Authors: Almond Russell G. , Mislevy Robert J.
Published: August 2014

This paper synthesizes ideas from the fields of graphical modeling and educational testing, particularly Item Response Theory (IRT) applied to Computerized Adaptive Testing (CAT). Graphical modeling can offer IRT a language for describing multifaceted skills and knowledge, and disentangling evidence from complex performances. IRT‐CAT can offer graphical modelers several ways of treating sources of variability other than including more variables in the model. In particular, variables can enter into the modeling process at five levels: (1) in validity studies (but not in the ordinarily used model), (2) in task construction (in particular, in defining link parameters), (3) in test or model assembly (blocking and randomization constraints in selecting tasks or other model pieces), (4) in response characterization (that is, as part of task models that characterize a response), or (5) in the main (student) model. The Graduate Record Examinations®(GRE®) are used to illustrate ideas in the context of IRT‐CAT, and extensions are discussed in the context of language proficiency testing.
SCHOOL SEGREGATION OF CHILDREN WHO MIGRATE TO THE UNITED STATES FROM PUERTO RICO
Authors: Laosa Luis M.
Published: August 2014

Examined are ecological attributes of the schools that preadolescents migrating from Puerto Rico to the United States (New Jersey) attend in this country during the first two years following their arrival (N= 89 schools). The data show that Hispanics/Latinos are the majority of the student body in 43% of the schools; African Americans, in 30%; and European Americans, in 12%. Native speakers of Spanish are the majority of the student body in 29% of the schools. Approximately one half of the schools are located in economically depressed, highly urbanized areas. Although the schools are on average large, 44% of them enroll above capacity. In most schools the majority of the student body is from economically impoverished families with low levels of parental education. There are, however, wide differences among the schools on each of these variables. Correlations show that the higher a student body's proportion of Hispanics/Latinos or native speakers of Spanish, the higher is the student body's proportion of pupils from economically impoverished households with low levels of parental education, and the higher the school's likelihood of being crowded and of being located in a poor inner‐city area. Similarly, the higher a student body's proportion of African Americans, the higher is the student body's proportion of pupils from low‐income families, and the higher the school's likelihood of being in a poor inner‐city area. Discussed are implications for policy and for students' academic, linguistic, social, and emotional development; and hypotheses in need of research. Also presented is a historical overview, to the present, and discussion of U.S. policies and judicial decisions concerning school segregation.
DOES ADAPTIVE TESTING VIOLATE LOCAL INDEPENDENCE?
Authors: Mislevy Robert J. , Chang Hua‐Hua
Published: August 2014

Item response theory posits “local independence”, or conditional independence of item responses given item parameters and examinee proficiency parameters. The usual definition of local independence, however, addresses the context of fixed tests, and initially appears to yield incorrect response‐pattern probabilities in the context of adaptive testing. The paradox is resolved by introducing additional notation to deal with the item selection mechanism. The probability of a response vector can then be expressed as the product of two factors. One concerns item selection, and does not depend on the examinee ability parameter. The other concerns item response given item selection, and can be written as a product over items in a manner analogous to the expression for local independence in fixed tests.
SCORING ESSAYS AUTOMATICALLY USING SURFACE FEATURES
Authors: Kaplan Randy M. , Wolff Susanne , Burstein Jill C. , Lu Chi , Rock Don , Kaplan Bruce
Published: August 2014

Surface features of essays include nonlinguistic characteristics ‐ such as total number of words per essay (essay length), sentence length, and word length ‐ as well as linguistic characteristics ‐ such as the total number of grammatical errors, the types of grammatical errors, or the kinds of grammatical constructions (e.g., passive or active) that appear in an essay. This study examines the feasibility of using linguistic and nonlinguistic surface characteristics in an automatic procedure for predicting essay scores. It builds on earlier work by R.M. Kaplan, Burstein, Lu, B. Kaplan, Wolff (1995), Page (1966), and Page and Petersen (1995).
IMPLEMENTING SHAFFER'S MULTIPLE COMPARISON PROCEDURE FOR LARGE NUMBERS OF GROUPS
Authors: Donoghue John R.
Published: August 2014

Shaffer (1986) presented a multiple comparison procedure for paired comparisons that had superior power to many available alternatives. Unfortunately, use of the procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer's procedure to be used in a much larger class of problems.

Theoretical results concerning an aspect of Shaffer's procedure are derived. The use of these results in implementing the procedure is the described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer's procedure is outlined.

The present work allows Shaffer's procedure for multiple comparisons to be applied to large numbers of groups. Unlike earlier work, no assumptions about the testing situation need be made. These methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.
VALIDATING THE REVISED TEST OF SPOKEN ENGLISH AGAINST A CRITERION OF COMMUNICATIVE SUCCESS
Authors: Powers Donald E. , Schedl Mary A. , Wilson‐Leung Susan , Butler Frances A.
Published: August 2014

A communicative competence orientation was adopted to study the validity of test score inferences derived from the revised Test of Spoken English (TSE). To implement the approach, a sample of undergraduate students, primarily native speakers of English, provided a variety of reactions to, and judgments of, the test responses of a sample of TSE examinees. The TSE scores of these examinees, previously determined by official TSE raters, spanned the full range of TSE score levels. Undergraduate students were selected as “evaluators” because they, more than most other groups, are likely to interact with TSE examinees, many of whom become teaching assistants.

Student evaluations were captured by devising and administering a secondary listening test (SLT) to assess students' understanding of TSE examinees' speech, as represented by their taped responses to tasks on the Test of Spoken English. The objective was to determine the degree to which official TSE scores are predictive of listeners' ability to understand the messages conveyed by TSE examinees. Analyses revealed a strong association between TSE score levels and the judgments, reactions, and understanding of listeners. This finding applied to all TSE tasks and to nearly all of the several different kinds of evaluations made by listeners. Along with other information, the evidence gathered here should help the TSE program meet professional standards for test validation. The procedures may also prove useful in future test development efforts as a way of determining the difficulty of speaking tasks (and possibly writing tasks).
USING NEW TECHNOLOGY TO IMPROVE ASSESSMENT
Authors: Bennett Randy Elliot
Published: August 2014

Together with advances in cognitive and measurement science, new technology will permit the transformation of assessment in fundamental ways. Five assessment areas are discussed: test design, item generation, task presentation, scoring, and testing purpose and location. For each area, I identify a key assessment innovation, say why it's important, call attention to issues it raises, speculate about the innovation's eventual effects, and identify whom to watch as the innovation moves forward.
THE EFFECTS OF WITHIN‐GROUP COVARIANCE STRUCTURE ON RECOVERY IN CLUSTER ANALYSIS: II. EXTENSION TO THE P‐DIMENSIONAL CASE
Authors: Donoghue John R.
Published: August 2014

A previous paper (Donoghue, 1995a) found that failure to account for within‐group covariance structure can greatly affect the ability of commonly used cluster analysis procedures to recover known subgroup structure. This follow‐up study used Monte Carlo methods to extend those results to the more general case of more than two groups and higher dimensional data.

Data were generated according to a finite normal mixture model. Distance between group centroids, number of groups, relative group sizes, number of variables, and within‐group covariance structure were varied. Each dataset was analyzed using 11 hierarchical clustering algorithms.

The effects of within‐group covariance structure were generalized, and proved even more vexing than had been found in the two‐group, bivariate case. Of the clustering algorithms, flexible average clustering (β = ‐.20 or β = ‐.25) gave best cluster recovery, followed by Ward's method.

These results are interpreted in terms of the weakness of Euclidean distance as a measure of inter‐entity similarity. Based on preliminary results, some possible alternate measures of similarity are identified. However, more work is clearly needed before clear recommendations about the choice of similarity measure can be given.
FURTHER VALIDATION OF A WRITING ASSESSMENT FOR GRADUATE ADMISSIONS
Authors: Powers Donald E. , Fowles Mary E. , Welsh Cynthia K.
Published: August 2014

The objective of the study reported here was to collect validity evidence for a proposed writing assessment for graduate admissions – the Graduate Record Examinations (GRE®) Writing Assessment. In particular, the objective was to investigate the relationship between student performance on each of two exercises being considered for the assessment and several nontest indicators of writing skill and achievement, and to thereby establish the degree to which performance on the GRE Writing Assessment is related to writing performance in academic settings. A variety of nontest indicators were examined, but a particular focus was the quality of students' course‐related writing samples. Two such writing samples were collected for each participant in the study, as was considerable information about the nature of these samples. These data enabled an estimate of the reliability and generalizability of writing samples as a validity criterion. The data also permitted an analysis of the conditions and circumstances under which performance on course‐related writing assignments and performance on the two new writing exercises relate to one another, thus facilitating the interpretation of scores derived from the new writing assessment. The results revealed modest relationships between performance on the writing assessment essays and various nontest indicators of writing ability. Performance on the GRE Writing Assessment exhibited the strongest relationship with course‐related writing samples, arguably the most compelling of the nontest indicators. There was no indication that the relationship between the essays and course‐related writing samples may depend on particular characteristics of the sample.
TEST DIFFICULTY AND STEREOTYPE THREAT ON THE GRE® GENERAL TEST
Authors: Stricker Lawrence J. , Bejar Isaac I.
Published: August 2014

Recent research suggests that stereotype threat may adversely affect the performance of Black and female examinees on the Graduate Record Examination (GRE®) General Test, but that this threat may be minimized by using easier test items. The present study investigated the ability to reduce or eliminate stereotype threat by manipulating the difficulty of items administered via a computer‐adaptive version of the General Test; the generalizability of these findings for Black examinees as well as women, and for Verbal as well as Quantitative sections of the test; and the processes that may mediate the effects of stereotype threat on test performance. The standard version of the computer‐adaptive General Test or a modified version that presents examinees with items that are easier than usual, and a battery of measures of stereotype threat and possible symptoms or consequences of stereotype threat were administered to college seniors bound to graduate school and to first‐year graduate students. Reducing test difficulty did not affect test performance or explicit indexes of stereotype threat for any group, but it lowered the anxiety of White students and women and raised the self‐esteem of White students.
EVALUATING AN UNDERDETERMINED RESPONSE TYPE FOR THE COMPUTERIZED SAT
Authors: Bennett Randy Elliot , Morley Mary , Quardt Dennis , Rock Donald A. , Katz Irvin R.
Published: August 2014

We evaluated a machine‐scorable, computer‐delivered response type for measuring quantitative reasoning skill. “Generating Examples” (GE) is built around items that present constraints and ask candidates to give one or more answers that meet those constraints. These items are attractive because, like many real‐world problems, GE items can have multiple correct answers. In addition, they appear to tap cognitive processes somewhat distinct from those measured by conventional quantitative questions. Nine GE forms were spiraled among a sample of academically precocious youth taking the Computerized SAT in association with a national talent search program. The forms differed in item manipulations designed to affect difficulty and in the expected time per item needed for solution. Results showed that across item lengths, the insertion of certain constraints increased difficulty. In addition, after correcting for attenuation, GE items similar in time requirements to SAT Mathematical items correlated in the mid‐eighties to mid‐nineties with SAT Mathematical scores, indicating that GE items might fit reasonably well with the SAT.
THE PERFORMANCE ASSESSMENT STUDY IN WRITING: ANALYSIS OF THE SAT®II: WRITING SUBJECT TEST
Authors: BRELAND HUNTER M. , KUBOTA MELVIN Y. , BONNER MARILYN W.
Published: August 2014

This study examined the SAT®II: Writing Subject Test as a predictor of writing performance in college English courses. Writing performance was based on eight writing samples submitted as part of regular course work by students in eight colleges. The samples consisted of drafts and final papers submitted in response to four take‐home writing assignments. The samples submitted were evaluated by 20 experienced readers of student writing using a common scoring scale. Each sample was read and scored independently by two different readers. In addition to this writing performance criterion, two supplemental criteria were used: English course grades and students' self‐assessments of their own writing ability and experience. In evaluating the SAT II: Writing Test as a predictor of these three criteria, special attention was given to comparisons of the predictive effectiveness of the essay and multiple‐choice components of the test. It was concluded that both components were good predictors of the three criteria; however, the 40‐minute multiple‐choice component tended to produce higher predictive correlations than the 20‐minute essay component. The best predictions were obtained when the two components were combined.
PREDICTIONS OF FRESHMAN GRADE‐POINT AVERAGE FROM THE REVISED AND RECENTERED SAT® I: REASONING TEST
Authors: Bridgeman Brent , McCamley‐Jenkins Laura , Ervin Nancy
Published: August 2014

The impact of revisions in the content of the SAT® and changes in the score scale on the predictive validity of the SAT were examined. Predictions of freshman grade‐point average (FGPA) for the entering class of 1994 (who had taken the old SAT) were compared with predictions for the class of 1995 (who had taken the new SAT I: Reasoning Test). The 1995 scores were evaluated both on the original SAT Program scale and on the recentered scale introduced that year. The changes in the test content and recentering of the score scale had virtually no impact on predictive validity. Other analyses indicated that the SAT I predicts FGPA about equally well across different ethnic groups. Correlations were slightly higher for higher levels of parental education and family income, and grades were more predictable for students with intended majors in math/science (mathematics, engineering, and biological or physical sciences) than for students with other intended majors. Correlations of the SAT I and the composite of SAT I scores and high school grade‐point average (HSGPA) with FGPA were generally higher for women than for men, although this pattern was reversed at colleges with very high mean SAT I scores. When a single prediction equation was used for all students, men tended to get lower grades than predicted and women got higher grades than predicted. African‐American and Hispanic/Latino men received lower grades than predicted, but women in these groups performed as predicted by the composite. Both men and women with intended majors in math/science got lower grades than would be predicted by an equation based on scores for all enrolled students.
A PRACTICE ANALYSIS OF CERTIFIED SCHOOL PSYCHOLOGISTS
Authors: Rosenfeld Michael , Leung Susan Wilson , Oltman Philip K.
Published: August 2014

This report describes the practice analysis study of newly certified school psychologists. This project was conducted in cooperation with the National Association of School Psychologists (NASP). The report documents the methods used to define the performance domain of school psychologists, describes the types of statistical analyses conducted, reports the results of these analyses, and discusses the implications of these results for test development. The examination that results from this study will be used by NASP and many states to license and/or certify school psychologists.
TRENDS AND PROFILES: STATISTICS ABOUT GRE® GENERAL TEST INTERNATIONAL EXAMINEES SECOND EDITION
Authors: Grandy Jerilee
Published: August 2014

This supplement to Trends and Profiles presents statistics on international students from the Graduate Record Examinations (GRE) files from 1982 through 1996. In question‐and‐answer format with many tables and graphs, the report addresses frequently asked questions about noncitizens who take the GRE General Test. It covers demographics, undergraduate studies, graduate school plans and, GRE preparation and test scores. Statistical profiles for 1996 and trends from 1982 to 1996 are presented by gender and age group, and comparisons are made with statistics for U.S. citizens.
AN INVESTIGATION OF APPROACHES TO COMPUTERIZING THE GRE® SUBJECT TESTS
Authors: Stocking Martha L. , Smith Robert , Swanson Len
Published: August 2014

In this project, we explored the application of two common, computer‐based test (CBT) designs, computerized adaptive testing (CAT) and linear‐on‐the‐fly testing (LOFT) to two Graduate Record Examination (GRE) Subject Tests chosen to represent the range of different test structures: the GRE Mathematics Test and the GRE Biology Test. The results were mixed. Some variations proved adequate in meeting minimum psychometric requirements. However, no variation proved adequate in the implementation of a complete test blueprint, nor did any variation provide strong support for meeting test security requirements in an environment of unrestricted continuous testing. The conversion of the GRE Subject Tests from paper‐and‐pencil administration to a computer‐based testing format seems viable only if pretesting is implemented, item writing is expanded, test length is reduced, classification schemes are improved, and the roles of test specialists and the Committees of Examiners are substantially revised.
THE EFFECT OF COMPUTER‐BASED TESTS ON RACIAL/ETHNIC, GENDER, AND LANGUAGE GROUPS
Authors: Gallagher Ann , Bridgeman Brent , Cahalan Cara
Published: August 2014

This study examined data from several national testing programs to determine whether the change from paper‐based administration to computer‐based tests (CBTs) influences group differences in performance. Performance by gender, racial/ethnic, and language groups on the Graduate Record Examination (GRE®) General Test, the Graduate Management Admissions Test (GMAT®), the SAT® I: Reasoning (SAT) test, the Praxis®: Professional Assessment for Beginning Teachers (Praxis), and the Test of English as a Foreign Language (TOEFL) was analyzed to ensure that the change to CBTs does not pose a disadvantage to any of these subgroups, beyond that already identified for paper‐based tests. Although all differences were quite small, some consistent patterns were found for some racial/ethnic and gender groups. African American examinees and, to a lesser degree, Hispanic examinees appear to benefit from the CBT format. However, for some tests, the CBT version negatively impacted female examinees. Analyses by gender within race/ethnicity revealed a similar pattern, though only for White females. Analyses for groups based on language showed no consistent patterns, but results indicate that the computer‐based TOEFL has increased impact for some language groups – especially Chinese and Korean groups.
AN EMPIRICAL INVESTIGATION OF IMPACT MODERATION IN TEST CONSTRUCTION
Authors: Stocking Martha L. , Lawrence Ida , Feigenbaum Miriam , Jirele Thomas , Lewis Charles , Essen Thomas Van
Published: August 2014

This investigation constructed four different kinds of test sections using three methods of test assembly that incorporate the goals of simultaneous moderation of three different kinds of impact—gender impact, African American impact, and Hispanic American impact. The test sections were administered undetectably to random samples from the appropriate population. The results were evaluated by comparison of the characteristics of moderated sections with those of parallel operational sections. Almost all methods of test assembly produced either moderation of impact in the appropriate direction or no change in impact. Taking impact into account in test assembly tended to lower reliability slightly, reduce the relative efficiency for test takers in the middle score range while increasing efficiency for those with more extreme scores, raise concurrent validity slightly and maintain the construct measured by the parallel operational section.
A STUDY OF THE USE OF COLLATERAL STATISTICAL INFORMATION IN ATTEMPTING TO REDUCE TOEFL IRT ITEM PARAMETER ESTIMATION SAMPLE SIZES
Authors: Tang K. Linda , Eignor Daniel R.
Published: August 2014

The purpose of this study was to investigate whether classical item statistics could be used as collateral information in the IRT calibration of pretest items for the computer‐based Test of English as a Foreign Language® using BILOG, and thus lead to a reduction in examinee sample sizes. The development and maintenance of item pools to support computer‐based testing (CBT) programs have placed much greater demands on the item pretesting process than was the case with paper‐and‐pencil testing, and this study attempted to show whether, using this procedure, more items than might be expected could be pretested, given a fixed overall examinee volume. Data from three TOEFL pretest item pools used in implementing CBT were used to simulate conditions under which the effects of reduced sample sizes, augmented by statistical collateral information from classical item statistics, could be investigated. Results indicated that classical statistics alone do not provide a sufficient level of collateral information to allow a reduction in pretest sample sizes. Reasons for these results are offered and suggestions for further research are provided.
OVERESTIMATION OF LPI RATINGS FOR NATIVE‐KOREAN SPEAKERS IN THE TOEIC TESTING CONTEXT: SEARCH FOR EXPLANATION
Authors: Wilson Kenneth M. , Stupak Steven A.
Published: August 2014

Regression‐based guidelines have been developed for predicting ratings of speaking proficiency in English as a foreign language (EFL), as assessed using the Language Proficiency Interview (LPI) procedure, from scores on the TOEIC® (Test of English for International Communication) Test. These guidelines, which reflect the regression of LPI rating on TOEIC scores in a combined sample composed of native speakers of Japanese, French, Spanish, and Arabic, respectively, were found to overestimate LPI rating when applied to data for native speakers of Korean. It was hypothesized that this outcome would be expected if development of EFL speaking proficiency (as assessed by the LPI procedure) tends to lag relatively more behind development of the proficiencies that are assessed by the TOEIC Test, for native‐Korean‐speaking EFL learners than for demographically comparable EFL learners in other national/linguistic settings (at least those represented in the study sample). An indirect assessment of this hypothesis was undertaken in a series of exploratory analyses involving section‐ and total‐score means for native‐language groups on the familiar, three‐part version of the Test of English as a Foreign Language™ (TOEFL®). Findings were interpreted as being generally, albeit indirectly, supportive of the developmental lag hypothesis, as well as of working assumptions underlying introduction and analysis of TOEFL native‐language reference group means as group analytic variables. Needed lines of inquiry are suggested.
COMPARING CONDITIONAL AND MARGINAL DIRECT ESTIMATION OF SUBGROUP DISTRIBUTIONS
Authors: von Davier Matthias
Published: August 2014

Many large‐scale assessment programs in education utilize “conditioning models” that incorporate both cognitive item responses and additional respondent background variables relevant for the population of interest. The set of respondent background variables serves as a predictor for the latent traits (proficiencies/abilities) and is used to obtain a conditional prior distribution for these traits. This is done by estimating a linear regression, assuming normality of the conditional trait distributions given the set of background variables. Multiple imputations, or plausible values, of trait parameter estimates are used in addition to or, better, on top of the conditioning model—as a computationally convenient approach to generating consistent estimates of the trait distribution characteristics for subgroups in complex assessments. This report compares, on the basis of simulated and real data, the conditioning method with a recently proposed method of estimating subgroup distribution statistics that assumes marginal normality. Study I presents simulated data examples where the marginal normality assumption leads to a model that produces appropriate estimates only if subgroup differences are small. In the presence of larger subgroup differences that cannot be fitted by the marginal normality assumption, however, the proposed method produces subgroup mean and variance estimates that differ strongly from the true values. Study II extends the findings on the marginal normality estimates to real data from large‐scale assessment programs such as the National Assessment of Educational Progress (NAEP) and the National Adult Literacy Survey (NALS). The research presented in Study II shows differences between the two methods that are similar to the differences found in Study I. The consequences of relying upon the assumption of marginal normality in direct estimation are discussed.
EFFECT OF FEWER QUESTIONS PER SECTION ON SAT® I SCORES
Authors: Bridgeman Brent , Trapani Catherine , Curley Edward
Published: August 2014

The impact of allowing more time for each question on SAT® I: Reasoning Test scores was estimated by embedding sections with a reduced number of questions into the standard 30‐minute equating section of two national test administrations. Thus, for example, questions were deleted from a verbal section that contained 35 questions to produce forms that contained 27 or 23 questions. Scores on the 23‐question section could then be compared to scores on the same 23 questions when they were embedded in a section that contained 27 or 35 questions. Similarly, questions were deleted from a 25‐question math section to form sections of 20 and 17 questions. Allowing more time per question had a minimal impact on verbal scores, producing gains of less than 10 points on the 200–800 SAT scale. Gains for the math score were less than 30 points. High‐scoring students tended to benefit more than lower‐scoring students, with extra time creating no increase in scores for students with SAT scores of 400 or lower. Ethnic/racial and gender differences were neither increased nor reduced with extra time.
A HISTORICAL PERSPECTIVE ON THE CONTENT OF THE SAT®
Authors: Lawrence Ida M. , Rigol Gretchen W. , Essen Thomas Van , Jackson Carol A.
Published: August 2014

This paper provides a historical perspective on the content of the SAT®. The review begins at the beginning, when the first College Board SAT (the “Scholastic Aptitude Test”) was administered to 8,040 students on June 23, 1926. At that time, the SAT consisted of nine subtests: definitions, arithmetical problems, classification, artificial language, antonyms, number series, analogies, logical inference, and paragraph reading. Over the years, the SAT has evolved in the way it measures what is now referred to as verbal and mathematical “reasoning.” With each redesign of the SAT, a variety of considerations were taken into account, including fairness issues, scaling issues, cost, public perception, face validity, changes in the test‐taking population, changes in patterns of test preparation, and changes in the college admissions process. This paper describes the reasons for the various changes while emphasizing that the value of SAT scores rests on the test's high technical quality, and on the assumption that scores would maintain their meaning over time.