bluuebunny/crossref_metadata_2025_split huggingface dataset:
GRAPHICAL MODELS AND COMPUTERIZED ADAPTIVE TESTING
Authors: Almond Russell G. , Mislevy Robert J. 
Published: August 2014

ABSTRACTThis paper synthesizes ideas from the fields of graphical modeling and educational testing, particularly Item Response Theory (IRT) applied to Computerized Adaptive Testing (CAT). Graphical modeling can offer IRT a language for describing multifaceted skills and knowledge, and disentangling evidence from complex performances. IRT‐CAT can offer graphical modelers several ways of treating sources of variability other than including more variables in the model. In particular, variables can enter into the modeling process at five levels: (1) in validity studies (but not in the ordinarily used model), (2) in task construction (in particular, in defining link parameters), (3) in test or model assembly (blocking and randomization constraints in selecting tasks or other model pieces), (4) in response characterization (that is, as part of task models that characterize a response), or (5) in the main (student) model. The Graduate Record Examinations®(GRE®) are used to illustrate ideas in the context of IRT‐CAT, and extensions are discussed in the context of language proficiency testing.
SCHOOL SEGREGATION OF CHILDREN WHO MIGRATE TO THE UNITED STATES FROM PUERTO RICO
Authors: Laosa Luis M. 
Published: August 2014

ABSTRACTExamined are ecological attributes of the schools that preadolescents migrating from Puerto Rico to the United States (New Jersey) attend in this country during the first two years following their arrival (N= 89 schools). The data show that Hispanics/Latinos are the majority of the student body in 43% of the schools; African Americans, in 30%; and European Americans, in 12%. Native speakers of Spanish are the majority of the student body in 29% of the schools. Approximately one half of the schools are located in economically depressed, highly urbanized areas. Although the schools are on average large, 44% of them enroll above capacity. In most schools the majority of the student body is from economically impoverished families with low levels of parental education. There are, however, wide differences among the schools on each of these variables. Correlations show that the higher a student body's proportion of Hispanics/Latinos or native speakers of Spanish, the higher is the student body's proportion of pupils from economically impoverished households with low levels of parental education, and the higher the school's likelihood of being crowded and of being located in a poor inner‐city area. Similarly, the higher a student body's proportion of African Americans, the higher is the student body's proportion of pupils from low‐income families, and the higher the school's likelihood of being in a poor inner‐city area. Discussed are implications for policy and for students' academic, linguistic, social, and emotional development; and hypotheses in need of research. Also presented is a historical overview, to the present, and discussion of U.S. policies and judicial decisions concerning school segregation.
DOES ADAPTIVE TESTING VIOLATE LOCAL INDEPENDENCE?
Authors: Mislevy Robert J. , Chang Hua‐Hua 
Published: August 2014

ABSTRACTItem response theory posits “local independence”, or conditional independence of item responses given item parameters and examinee proficiency parameters. The usual definition of local independence, however, addresses the context of fixed tests, and initially appears to yield incorrect response‐pattern probabilities in the context of adaptive testing. The paradox is resolved by introducing additional notation to deal with the item selection mechanism. The probability of a response vector can then be expressed as the product of two factors. One concerns item selection, and does not depend on the examinee ability parameter. The other concerns item response given item selection, and can be written as a product over items in a manner analogous to the expression for local independence in fixed tests.
SCORING ESSAYS AUTOMATICALLY USING SURFACE FEATURES
Authors: Kaplan Randy M. , Wolff Susanne , Burstein Jill C. , Lu Chi , Rock Don , Kaplan Bruce 
Published: August 2014

ABSTRACTSurface features of essays include nonlinguistic characteristics ‐ such as total number of words per essay (essay length), sentence length, and word length ‐ as well as linguistic characteristics ‐ such as the total number of grammatical errors, the types of grammatical errors, or the kinds of grammatical constructions (e.g., passive or active) that appear in an essay. This study examines the feasibility of using linguistic and nonlinguistic surface characteristics in an automatic procedure for predicting essay scores. It builds on earlier work by R.M. Kaplan, Burstein, Lu, B. Kaplan, Wolff (1995), Page (1966), and Page and Petersen (1995).
IMPLEMENTING SHAFFER'S MULTIPLE COMPARISON PROCEDURE FOR LARGE NUMBERS OF GROUPS
Authors: Donoghue John R. 
Published: August 2014

ABSTRACTShaffer (1986) presented a multiple comparison procedure for paired comparisons that had superior power to many available alternatives. Unfortunately, use of the procedure has been severely limited by the complexity of its implementation. This paper presents a method of allowing Shaffer's procedure to be used in a much larger class of problems.Theoretical results concerning an aspect of Shaffer's procedure are derived. The use of these results in implementing the procedure is the described. Next, two heuristics are described which greatly enhance the efficiency of the method. Finally, an efficient algorithmic method of implementing Shaffer's procedure is outlined.The present work allows Shaffer's procedure for multiple comparisons to be applied to large numbers of groups. Unlike earlier work, no assumptions about the testing situation need be made. These methods are illustrated by application to two data sets; comparisons of the order of 11 clustering methods, and comparison of the means of 44 states from the 1994 National Assessment of Educational Progress Trial State Assessment of Reading at Grade 4.
VALIDATING THE REVISED TEST OF SPOKEN ENGLISH AGAINST A CRITERION OF COMMUNICATIVE SUCCESS
Authors: Powers Donald E. , Schedl Mary A. , Wilson‐Leung Susan , Butler Frances A. 
Published: August 2014

ABSTRACTA communicative competence orientation was adopted to study the validity of test score inferences derived from the revised Test of Spoken English (TSE). To implement the approach, a sample of undergraduate students, primarily native speakers of English, provided a variety of reactions to, and judgments of, the test responses of a sample of TSE examinees. The TSE scores of these examinees, previously determined by official TSE raters, spanned the full range of TSE score levels. Undergraduate students were selected as “evaluators” because they, more than most other groups, are likely to interact with TSE examinees, many of whom become teaching assistants.Student evaluations were captured by devising and administering a secondary listening test (SLT) to assess students' understanding of TSE examinees' speech, as represented by their taped responses to tasks on the Test of Spoken English. The objective was to determine the degree to which official TSE scores are predictive of listeners' ability to understand the messages conveyed by TSE examinees. Analyses revealed a strong association between TSE score levels and the judgments, reactions, and understanding of listeners. This finding applied to all TSE tasks and to nearly all of the several different kinds of evaluations made by listeners. Along with other information, the evidence gathered here should help the TSE program meet professional standards for test validation. The procedures may also prove useful in future test development efforts as a way of determining the difficulty of speaking tasks (and possibly writing tasks).
USING NEW TECHNOLOGY TO IMPROVE ASSESSMENT
Authors: Bennett Randy Elliot 
Published: August 2014

ABSTRACTTogether with advances in cognitive and measurement science, new technology will permit the transformation of assessment in fundamental ways. Five assessment areas are discussed: test design, item generation, task presentation, scoring, and testing purpose and location. For each area, I identify a key assessment innovation, say why it's important, call attention to issues it raises, speculate about the innovation's eventual effects, and identify whom to watch as the innovation moves forward.
THE EFFECTS OF WITHIN‐GROUP COVARIANCE STRUCTURE ON RECOVERY IN CLUSTER ANALYSIS: II. EXTENSION TO THE P‐DIMENSIONAL CASE
Authors: Donoghue John R. 
Published: August 2014

ABSTRACTA previous paper (Donoghue, 1995a) found that failure to account for within‐group covariance structure can greatly affect the ability of commonly used cluster analysis procedures to recover known subgroup structure. This follow‐up study used Monte Carlo methods to extend those results to the more general case of more than two groups and higher dimensional data.Data were generated according to a finite normal mixture model. Distance between group centroids, number of groups, relative group sizes, number of variables, and within‐group covariance structure were varied. Each dataset was analyzed using 11 hierarchical clustering algorithms.The effects of within‐group covariance structure were generalized, and proved even more vexing than had been found in the two‐group, bivariate case. Of the clustering algorithms, flexible average clustering (β = ‐.20 or β = ‐.25) gave best cluster recovery, followed by Ward's method.These results are interpreted in terms of the weakness of Euclidean distance as a measure of inter‐entity similarity. Based on preliminary results, some possible alternate measures of similarity are identified. However, more work is clearly needed before clear recommendations about the choice of similarity measure can be given.
FURTHER VALIDATION OF A WRITING ASSESSMENT FOR GRADUATE ADMISSIONS
Authors: Powers Donald E. , Fowles Mary E. , Welsh Cynthia K. 
Published: August 2014

ABSTRACTThe objective of the study reported here was to collect validity evidence for a proposed writing assessment for graduate admissions – the Graduate Record Examinations (GRE®) Writing Assessment. In particular, the objective was to investigate the relationship between student performance on each of two exercises being considered for the assessment and several nontest indicators of writing skill and achievement, and to thereby establish the degree to which performance on the GRE Writing Assessment is related to writing performance in academic settings. A variety of nontest indicators were examined, but a particular focus was the quality of students' course‐related writing samples. Two such writing samples were collected for each participant in the study, as was considerable information about the nature of these samples. These data enabled an estimate of the reliability and generalizability of writing samples as a validity criterion. The data also permitted an analysis of the conditions and circumstances under which performance on course‐related writing assignments and performance on the two new writing exercises relate to one another, thus facilitating the interpretation of scores derived from the new writing assessment. The results revealed modest relationships between performance on the writing assessment essays and various nontest indicators of writing ability. Performance on the GRE Writing Assessment exhibited the strongest relationship with course‐related writing samples, arguably the most compelling of the nontest indicators. There was no indication that the relationship between the essays and course‐related writing samples may depend on particular characteristics of the sample.
TEST DIFFICULTY AND STEREOTYPE THREAT ON THE GRE® GENERAL TEST
Authors: Stricker Lawrence J. , Bejar Isaac I. 
Published: August 2014

ABSTRACTRecent research suggests that stereotype threat may adversely affect the performance of Black and female examinees on the Graduate Record Examination (GRE®) General Test, but that this threat may be minimized by using easier test items. The present study investigated the ability to reduce or eliminate stereotype threat by manipulating the difficulty of items administered via a computer‐adaptive version of the General Test; the generalizability of these findings for Black examinees as well as women, and for Verbal as well as Quantitative sections of the test; and the processes that may mediate the effects of stereotype threat on test performance. The standard version of the computer‐adaptive General Test or a modified version that presents examinees with items that are easier than usual, and a battery of measures of stereotype threat and possible symptoms or consequences of stereotype threat were administered to college seniors bound to graduate school and to first‐year graduate students. Reducing test difficulty did not affect test performance or explicit indexes of stereotype threat for any group, but it lowered the anxiety of White students and women and raised the self‐esteem of White students.
EVALUATING AN UNDERDETERMINED RESPONSE TYPE FOR THE COMPUTERIZED SAT
Authors: Bennett Randy Elliot , Morley Mary , Quardt Dennis , Rock Donald A. , Katz Irvin R. 
Published: August 2014

ABSTRACTWe evaluated a machine‐scorable, computer‐delivered response type for measuring quantitative reasoning skill. “Generating Examples” (GE) is built around items that present constraints and ask candidates to give one or more answers that meet those constraints. These items are attractive because, like many real‐world problems, GE items can have multiple correct answers. In addition, they appear to tap cognitive processes somewhat distinct from those measured by conventional quantitative questions. Nine GE forms were spiraled among a sample of academically precocious youth taking the Computerized SAT in association with a national talent search program. The forms differed in item manipulations designed to affect difficulty and in the expected time per item needed for solution. Results showed that across item lengths, the insertion of certain constraints increased difficulty. In addition, after correcting for attenuation, GE items similar in time requirements to SAT Mathematical items correlated in the mid‐eighties to mid‐nineties with SAT Mathematical scores, indicating that GE items might fit reasonably well with the SAT.
THE PERFORMANCE ASSESSMENT STUDY IN WRITING: ANALYSIS OF THE SAT®II: WRITING SUBJECT TEST
Authors: BRELAND HUNTER M. , KUBOTA MELVIN Y. , BONNER MARILYN W. 
Published: August 2014

ABSTRACTThis study examined the SAT®II: Writing Subject Test as a predictor of writing performance in college English courses. Writing performance was based on eight writing samples submitted as part of regular course work by students in eight colleges. The samples consisted of drafts and final papers submitted in response to four take‐home writing assignments. The samples submitted were evaluated by 20 experienced readers of student writing using a common scoring scale. Each sample was read and scored independently by two different readers. In addition to this writing performance criterion, two supplemental criteria were used: English course grades and students' self‐assessments of their own writing ability and experience. In evaluating the SAT II: Writing Test as a predictor of these three criteria, special attention was given to comparisons of the predictive effectiveness of the essay and multiple‐choice components of the test. It was concluded that both components were good predictors of the three criteria; however, the 40‐minute multiple‐choice component tended to produce higher predictive correlations than the 20‐minute essay component. The best predictions were obtained when the two components were combined.
PREDICTIONS OF FRESHMAN GRADE‐POINT AVERAGE FROM THE REVISED AND RECENTERED SAT® I: REASONING TEST
Authors: Bridgeman Brent , McCamley‐Jenkins Laura , Ervin Nancy 
Published: August 2014

ABSTRACTThe impact of revisions in the content of the SAT® and changes in the score scale on the predictive validity of the SAT were examined. Predictions of freshman grade‐point average (FGPA) for the entering class of 1994 (who had taken the old SAT) were compared with predictions for the class of 1995 (who had taken the new SAT I: Reasoning Test). The 1995 scores were evaluated both on the original SAT Program scale and on the recentered scale introduced that year. The changes in the test content and recentering of the score scale had virtually no impact on predictive validity. Other analyses indicated that the SAT I predicts FGPA about equally well across different ethnic groups. Correlations were slightly higher for higher levels of parental education and family income, and grades were more predictable for students with intended majors in math/science (mathematics, engineering, and biological or physical sciences) than for students with other intended majors. Correlations of the SAT I and the composite of SAT I scores and high school grade‐point average (HSGPA) with FGPA were generally higher for women than for men, although this pattern was reversed at colleges with very high mean SAT I scores. When a single prediction equation was used for all students, men tended to get lower grades than predicted and women got higher grades than predicted. African‐American and Hispanic/Latino men received lower grades than predicted, but women in these groups performed as predicted by the composite. Both men and women with intended majors in math/science got lower grades than would be predicted by an equation based on scores for all enrolled students.
A PRACTICE ANALYSIS OF CERTIFIED SCHOOL PSYCHOLOGISTS
Authors: Rosenfeld Michael , Leung Susan Wilson , Oltman Philip K. 
Published: August 2014

ABSTRACTThis report describes the practice analysis study of newly certified school psychologists. This project was conducted in cooperation with the National Association of School Psychologists (NASP). The report documents the methods used to define the performance domain of school psychologists, describes the types of statistical analyses conducted, reports the results of these analyses, and discusses the implications of these results for test development. The examination that results from this study will be used by NASP and many states to license and/or certify school psychologists.
TRENDS AND PROFILES: STATISTICS ABOUT GRE® GENERAL TEST INTERNATIONAL EXAMINEES SECOND EDITION
Authors: Grandy Jerilee 
Published: August 2014

ABSTRACTThis supplement to Trends and Profiles presents statistics on international students from the Graduate Record Examinations (GRE) files from 1982 through 1996. In question‐and‐answer format with many tables and graphs, the report addresses frequently asked questions about noncitizens who take the GRE General Test. It covers demographics, undergraduate studies, graduate school plans and, GRE preparation and test scores. Statistical profiles for 1996 and trends from 1982 to 1996 are presented by gender and age group, and comparisons are made with statistics for U.S. citizens.
AN INVESTIGATION OF APPROACHES TO COMPUTERIZING THE GRE® SUBJECT TESTS
Authors: Stocking Martha L. , Smith Robert , Swanson Len 
Published: August 2014

ABSTRACTIn this project, we explored the application of two common, computer‐based test (CBT) designs, computerized adaptive testing (CAT) and linear‐on‐the‐fly testing (LOFT) to two Graduate Record Examination (GRE) Subject Tests chosen to represent the range of different test structures: the GRE Mathematics Test and the GRE Biology Test. The results were mixed. Some variations proved adequate in meeting minimum psychometric requirements. However, no variation proved adequate in the implementation of a complete test blueprint, nor did any variation provide strong support for meeting test security requirements in an environment of unrestricted continuous testing. The conversion of the GRE Subject Tests from paper‐and‐pencil administration to a computer‐based testing format seems viable only if pretesting is implemented, item writing is expanded, test length is reduced, classification schemes are improved, and the roles of test specialists and the Committees of Examiners are substantially revised.
THE EFFECT OF COMPUTER‐BASED TESTS ON RACIAL/ETHNIC, GENDER, AND LANGUAGE GROUPS
Authors: Gallagher Ann , Bridgeman Brent , Cahalan Cara 
Published: August 2014

ABSTRACTThis study examined data from several national testing programs to determine whether the change from paper‐based administration to computer‐based tests (CBTs) influences group differences in performance. Performance by gender, racial/ethnic, and language groups on the Graduate Record Examination (GRE®) General Test, the Graduate Management Admissions Test (GMAT®), the SAT® I: Reasoning (SAT) test, the Praxis®: Professional Assessment for Beginning Teachers (Praxis), and the Test of English as a Foreign Language (TOEFL) was analyzed to ensure that the change to CBTs does not pose a disadvantage to any of these subgroups, beyond that already identified for paper‐based tests. Although all differences were quite small, some consistent patterns were found for some racial/ethnic and gender groups. African American examinees and, to a lesser degree, Hispanic examinees appear to benefit from the CBT format. However, for some tests, the CBT version negatively impacted female examinees. Analyses by gender within race/ethnicity revealed a similar pattern, though only for White females. Analyses for groups based on language showed no consistent patterns, but results indicate that the computer‐based TOEFL has increased impact for some language groups – especially Chinese and Korean groups.
AN EMPIRICAL INVESTIGATION OF IMPACT MODERATION IN TEST CONSTRUCTION
Authors: Stocking Martha L. , Lawrence Ida , Feigenbaum Miriam , Jirele Thomas , Lewis Charles , Essen Thomas Van 
Published: August 2014

ABSTRACTThis investigation constructed four different kinds of test sections using three methods of test assembly that incorporate the goals of simultaneous moderation of three different kinds of impact—gender impact, African American impact, and Hispanic American impact. The test sections were administered undetectably to random samples from the appropriate population. The results were evaluated by comparison of the characteristics of moderated sections with those of parallel operational sections. Almost all methods of test assembly produced either moderation of impact in the appropriate direction or no change in impact. Taking impact into account in test assembly tended to lower reliability slightly, reduce the relative efficiency for test takers in the middle score range while increasing efficiency for those with more extreme scores, raise concurrent validity slightly and maintain the construct measured by the parallel operational section.
A STUDY OF THE USE OF COLLATERAL STATISTICAL INFORMATION IN ATTEMPTING TO REDUCE TOEFL IRT ITEM PARAMETER ESTIMATION SAMPLE SIZES
Authors: Tang K. Linda , Eignor Daniel R. 
Published: August 2014

ABSTRACTThe purpose of this study was to investigate whether classical item statistics could be used as collateral information in the IRT calibration of pretest items for the computer‐based Test of English as a Foreign Language® using BILOG, and thus lead to a reduction in examinee sample sizes. The development and maintenance of item pools to support computer‐based testing (CBT) programs have placed much greater demands on the item pretesting process than was the case with paper‐and‐pencil testing, and this study attempted to show whether, using this procedure, more items than might be expected could be pretested, given a fixed overall examinee volume. Data from three TOEFL pretest item pools used in implementing CBT were used to simulate conditions under which the effects of reduced sample sizes, augmented by statistical collateral information from classical item statistics, could be investigated. Results indicated that classical statistics alone do not provide a sufficient level of collateral information to allow a reduction in pretest sample sizes. Reasons for these results are offered and suggestions for further research are provided.