bluuebunny/crossref_metadata_2025_split huggingface dataset:
USING THE FREE‐RESPONSE SCORING TOOL TO AUTOMATICALLY SCORE THE FORMULATING‐HYPOTHESES ITEM
Authors: Kaplan Randy M. , Bennett Randy Elliot 
Published: August 2014

ABSTRACTLarge‐scale institutional testing, and testing in general, are in a period of rapid change. Among the more obvious dimensions is the growing use of constructed‐response items and of computer‐based testing. This study explores the potential for using a computer‐based scoring procedure for the formulating‐hypotheses item. This item type presents a situation and asks the examinee to generate explanations for it. Each explanation is judged right or wrong and the number of creditable explanations summed to produce an item score. Scores were generated for 30 examinees' responses to each of eight items by a semantic pattern‐matching program and independently by five human raters. On its initial scoring run, the program agreed highly with the raters' mean item scores for some questions and improved its concurrence substantially as modifications to the automatic scoring process were made. By the final run, correlations between the program and the raters on item scores ranged from .89 to .97, and mean human‐machine discrepancies ran from .6 to 1.1 on a 16‐point scale. At the individual‐hypothesis level, the proportion agreement ranged from .80 to .94, which, given the large disproportion of correct responses in the sample, was little better than chance. Also detected was a tendency on the part of the program to erroneously classify wrong responses as correct. We conclude that F‐H items might be more effectively scored by a semiautomatic system that combines machine processing with a small number of human judges, and we present a preliminary configuration for such a process.
BACKGROUND CHARACTERISTICS AND FUTURE PLANS OF HIGH‐SCORING GRE GENERAL TEST EXAMINEES
Authors: Pennock‐Román María 
Published: August 2014

ABSTRACTThe purpose of this study was to describe the background characteristics and future plans of average and above average Graduate Record Examinations (GRE) General Test takers in the various racial/ethnic groups. In this report, the characteristics included percentage of females, undergraduate college major and grades, intended field of graduate study, graduate degree objective, year of receipt of baccalaureate degree, location of high school, family socioeconomic background, citizenship status, and language spoken in the home. The distribution of these characteristics is given in tables for students classified according to level of GRE Verbal (GRE‐V), GRE Quantitative (GRE‐Q), or GRE Analytic (GRE‐A) scores in intervals of 50 points beginning with scores of 500‐549 until 750‐800.It was found that high parental socioeconomic indicators were associated with higher GRE scores, but a substantial proportion (17%) of the high‐scoring students came from homes where fathers had less than a high school education. This figure was particularly large for the Mexican American group (36%). The distribution of parental income showed that, for all groups, there was a sizable percentage who would be unable to afford graduate school without financial aid.Most examinees with very high GRE‐Q scores tended to take the GRE soon after they had graduated from college, but the opposite tended to be true for very high GRE‐V scorers in the non‐Hispanic White group. This pattern is discussed in light of choice of major (humanities versus science) and the availability of financial aid for graduate study in various fields.One surprising finding was that the pattern of choice of major for racial/ethnic groups in the high scoring levels was substantially different from that in the overall GRE population. Holding GRE‐Q level constant, high‐scoring non‐Hispanic Black, Hispanic American, and Asian American students were more likely to major in engineering and computer science than were non‐Hispanic White test takers.
OPTIONAL USE OF CALCULATORS ON A MATHEMATICAL TEST: EFFECT ON ITEM DIFFICULTY AND SCORE EQUATING
Authors: Lawrence Ida M. , Dorans Neil J. 
Published: August 2014

ABSTRACTThis paper describes findings from two studies involving optional use of calculators on SAT mathematical items. The first study looked at the effects of calculator use on estimates of item difficulty. The second study looked at the effect of calculator use on equating results from an anchor test design. Study 1 looked at data on specific items that become inappropriate for a test that permits calculators because the skills measured by the item administered with a calculator are quite different when the item is administered without a calculator. Study 2 showed that, because the use of a calculator sometimes makes items easier, any equating design that utilizes an anchor test design needs to ensure that the anchor test is administered under the same condition, i.e., with a calculator or without a calculator. A solution for making this adjustment with a special equating study is described.
THE EFFECT OF THE PROBABILITY OF CORRECT RESPONSE ON THE VARIABILITY OF MEASURES OF DIFFERENTIAL ITEM FUNCTIONING
Authors: Zwick Rebecca 
Published: August 2014

ABSTRACTThe Mantel‐Haenszel (MH; 1959) approach of Holland and Thayer (1988) is a well‐established method for assessing differential item functioning (DIF). The formula for the variance of the MH DIF statistic is based on work by Phillips and Holland (1987) and Robins, Breslow and Greenland (1986). Recent simulation studies showed that the MH variances tended to be larger when items were administered to “examinees” who were randomly selected from a population than when items were administered adaptively. An analytic perspective shed some light on this puzzling result. Although the general form of the MH variance is complex and does not provide an intuitive understanding of the phenomenon, application of certain Rasch model assumptions yields a simple expression that appears to explain the difference in variances for adaptive versus nonadaptive administration.
AN ALTERNATIVE METHOD FOR SCORING ADAPTIVE TESTS1
Authors: Stocking Martha L. 
Published: August 2014

ABSTRACTModern applications of computerized adaptive testing (CAT) are typically grounded in Item Response Theory (IRT; Lord, 1980). While the IRT foundations of adaptive testing provide a number of approaches to adaptive test scoring that may seem natural and efficient to psychometricians, these approaches may be more demanding for test‐takers, test score users, interested regulatory institutions, and so forth, to comprehend. An alternative method, based on more familiar equated number‐correct scores and identical to that used to score and equate many conventional tests, is explored and compared with one that relies more directly on IRT. The conclusion is reached that scoring adaptive tests using the familiar number‐correct score, accompanied by the necessary equating to adjust for the intentional differences in adaptive test difficulty, is a statistically viable, although slightly less efficient, method of adaptive test scoring. To enhance the prospects for enlightened public debate about adaptive testing, it may be preferable to use this more familiar approach. Public attention would then likely be focussed on issues more central to adaptive testing, namely the adaptive nature of the test.
SETTING STANDARDS ON THE ASSESSOR PROFICIENCY TEST FOR THE PRAXIS III: CLASSROOM PERFORMANCE ASSESSMENT
Authors: Livingston Samuel A. , Sims‐Gunzenhauser Alice 
Published: August 2014

ABSTRACTThe Praxis III Assessor's job is to interview the beginning teacher, observe the teacher in the classroom, score the teacher's performance on nineteen criteria, and summarize the evidence for each score. The Assessor Proficiency Test (APT) consists of performing the assessor's task for a videotaped lesson. It yields two scores: accuracy and documentation. In this study, five judges judged the accuracy and documentation of APT records produced by fifteen assessor trainees. Documentation judgments were made by group consensus; accuracy judgments were individual. Logistic regression analysis showed that both types of judgments were strongly related to the corresponding APT scores, but judges' individual standards for accuracy varied greatly.
A TRANSPORTABILITY STUDY OF THE READING, WRITING, AND MATHEMATICS SKILLS IMPORTANT FOR TEACHERS OF DEAF AND HARD OF HEARING STUDENTS
Authors: Rosenfeld Michael , Mounty Judith L. , Ehringhaus Michael E. 
Published: August 2014

ABSTRACTThe purpose of this study was to determine if the 57 reading, writing, and mathematics skill statements derived from the test specifications for the Praxis I Basic Skills Assessments and the Pre‐Professional Skills Test (PPST) are considered to be important for entry‐level teachers of deaf and hard of hearing students. The skill statements were placed in a survey format, and judgments of importance were made using a 5‐point rating scale. Two thousand surveys were mailed to administrators at schools and programs for deaf and hard of hearing students and to program directors at teacher training institutions throughout the United States. The administrators were asked to distribute the survey to teachers, administrators, and teacher educators at their institution and to give special consideration to the inclusion of deaf and hard of hearing faculty and staff members. In selecting the sample of institutions, every attempt was made to represent the full range of perspectives and practices in the education of deaf and hard of hearing students with respect to language and communication and program type.Usable surveys were returned by 833 teachers, administrators, and teacher educators. A skill statement was judged to be appropriate for entry‐level teachers of deaf and hard of hearing students if it was judged to have a mean importance rating of 3.00 or greater (moderately important) from the total group of respondents as well as from each of the 44 subgroup comparisons that were conducted. All 57 skill statements received mean ratings above 3.00 from the total group as well as from all relevant subgroups of respondents. It may be concluded from the survey results that the skill statements used to build test specifications for the Praxis I Basic Skills Assessments and the PPST are appropriate for entry‐level teachers of deaf and hard of hearing students. That is, the content of these assessments was judged to be important. However, it cannot be overemphasized that this finding does not imply that the items or modes of assessment being used to measure these skills have been judged to be appropriate for deaf and hard of hearing candidates. Additional studies need to be conducted to determine the impact of these measures on deaf and hard of hearing candidates as well as the need for any special accommodations to ensure that candidates with this disability have an adequate opportunity to demonstrate their level of skill on these measures.
AN ASSESSMENT OF SELECTED VALIDITY‐RELATED PROPERTIES OF A SHORTENED VERSION OF THE SECONDARY LEVEL ENGLISH PROFICIENCY TEST AND LOCALLY DEVELOPED WRITING TESTS IN THE LACCD CONTEXT
Authors: Wilson Kenneth M. 
Published: August 2014

ABSTRACTThis is the report of a study that was underaken to obtain direct empirical evidence regarding aspects of the validity and usefulness for ESL placement of (a) a shortened version of the Secondary Level English Proficiency (SLEP) Test, being used for ESL placement by colleges in the Los Angeles Community College District (LACCD), and (b) locally developed and scored writing tests. The LACCD Central Office provided scores on the shortened version of the SLEP and the writing samples, grades in ESL courses, and background data (gender, language, educational status, and so on) for over 10,000 students. This report documents and evaluates• patterns of performance on the shortened SLEP and the writing tests in the general ESL population and in selected demographic subpopulations,• concurrent relationships among scores on the components of the LACCD placement battery,• observed levels of correlation between scores on the shortened SLEP test, the writing test, and the placement composite, on the one hand, and student performance in ESL courses, as indexed by grade earned (a grade on the “A‐F” scale, or a Pass/Fail grade), on the other, by course and by college, and in various subgroups (e.g., gender, educational level, age, language), and• the extent to which observed relationships in placed samples are influenced by non‐validity‐related factors (for example, differential restriction of range on the tests that were used to place students, sample size, and type of grading system).The findings of this collaborative undertaking provide direct empirical evidence that the shortened SLEP and locally developed writing tests are providing valid information regarding related aspects of ESL proficiency in the demographically diverse ESL student population being served by the LACCD; and the findings logically extend available evidence supportive of the validity of the Secondary Level English Proficiency Test for ESL assessment purposes in the LACCD and elsewhere. Based on study findings, the local writing tests and the shortened SLEP appear to be providing an effective basis for placing students, within time constraints that appear to be considered necessary, from an administrative perspective. Research is needed to address questions regarding the extent to which use of the full‐length version of the SLEP would enhance the overall validity of placement. Some pertinent lines of inquiry are suggested.
USING TRILINEAR PLOTS FOR NAEP STATE DATA1
Authors: Wainer Howard 
Published: August 2014

ABSTRACTUnderstanding the distribution of achievement levels of students' performance on NAEP is aided through the use of the trilinear chart. This chart is described and its use illustrated with data from the 1992 state NAEP mathematics assessment. It is shown that one can see readily the trends in performance for different demographic groups for all of the 44 participating jurisdictions simultaneously.
ANALYSIS OF PROPOSED REVISIONS OF THE TEST OF SPOKEN ENGLISH
Authors: Henning Grant , Schedl Mary , Suomi Barbara K. 
Published: August 2014

ABSTRACTThis research was conducted to compare a prototype revised form of the Test of Spoken English (TSE®) with the current version of the same test. The study compared interrater reliability, frequency of rater discrepancy at all score levels, component task adequacy, scoring efficacy, and other concurrent and construct validity evidence, including oral proficiency interview correlations for a subset of the examinee sample. The study employed a representative examinee sample of 342 nonnative speakers of English, purposely sampled from among the two professional domains of prospective university teaching assistants (N = 184) and prospective licensed medical professionals (N = 158).One somewhat unusual component of the study was the attempt to involve persons most at risk in the judgment process. Thus, in addition to employing the usual group of trained raters for the scoring of the current and prototype versions of the test, 16 naive adult raters were purposely selected (eight first‐year university students from four broad academic disciplines and eight nondegreed prospective medical outpatients within four broad age levels) for having limited exposure to foreign languages and cultures. These 16 naive raters (eight females and eight males) provided concurrent judgments of the comprehensibility and communicative effectiveness of a subset of 40 recorded prototype examinations.In general, the comparative evidence gathered appeared to underscore the psychometric quality of the prototype revised TSE and to support conclusions of its adequacy as an instrument used to make judgments of the oral English language proficiency of nonnative speakers in the targeted populations. Some additional suggestions are provided on ways to implement the scoring of the prototype version of the test.
A COGNITIVE ANALYSIS OF SOLUTIONS FOR VERBAL, INFORMAL, AND FORMAL‐DEDUCTIVE REASONING PROBLEMS
Authors: Enright Mary K. , Tucker Carol B. , Katz Irvin R. 
Published: August 2014

ABSTRACTA previous study of new item types for the analytical measure of the GRE General Test found that the items loaded on three of four separable factors that were labeled verbal reasoning, informal reasoning, formal‐deductive reasoning, and quantitative reasoning. The present study examined the issue of how processing differed for these item types in the context of a problem‐space framework. Protocols of examinees solving a small set of problems aloud were collected. These protocols were examined with respect to two phases of the problem‐solving process: problem representation and problem solution. For formal‐deductive items, all the information necessary to solve the problem was provided in the problem statement. The representation of formal‐deductive items involved the use of meaning‐reduced tokens and spatial diagrams. The units involved in the representation of informal reasoning and verbal reasoning item types included meaningful propositions and meaning‐emphasizing paraphrases. Reference to common background knowledge occurred. The analysis of the problem‐solution phase focused on the processes of evaluation, (judgments of the correctness of an option) and justification (statements of an argument or of evidence for why an option was or was not correct). First, the order of these processes was found to differ for formal‐deductive items and other item types. Secondly, item solutions varied in terms of the kinds of justifications that were offered by the examinees for accepting or rejecting options. These results illustrate how the addition of some item types to the GRE analytical measure will expand the variety of reasoning skills assessed. Implications of these results for cognitive models of reasoning are also discussed.
A TASK TYPE FOR MEASURING THE REPRESENTATIONAL COMPONENT OF QUANTITATIVE PROFICIENCY
Authors: Bennett Randy Elliot , Sebrechts Marc M. , Rock Donald A. 
Published: August 2014

ABSTRACTTwo computer‐based categorization tasks were developed and pilot tested. For Study I, the task asked examinees to sort mathematical word problem stems according to prototypes. Results showed that those who sorted well tended to have higher GRE General Test scores and college grades than did examinees who sorted less proficiently. Examinees generally preferred this task to multiple‐choice items like those found on General Test quantitative section and felt the task was a fairer measure of their ability to succeed in graduate school. For study II, the task involved rating the similarity of item pairs. Both mathematics test developers and students participated, with the results analyzed by individual differences multidimensional scaling. Experts produced more scaleable ratings overall and primarily attended to two dimensions. Students used the same two dimensions with the addition of a third. Students who rated like the experts in terms of the dimensions used tended to have higher admissions test scores than those who used other criteria. Finally, examinees preferred multiple‐choice questions to the rating task and felt that the former was a fairer indicator of their scholastic abilities. The major implication of this work is in identifying a new task type for admissions tests, as well as for instructional assessment products that might help lower scoring examinees localize and remediate problem‐solving difficulties.
ASSESSING SOME OF THE PROPERTIES OF LONGER BLOCKS IN THE 1992 NAEP READING ASSESSMENT
Authors: Donoghue John R. , Mazzeo John 
Published: August 2014

ABSTRACTAt Grades 8 and 12, the 1992 NAEP reading assessment contained a small number of 50 minute blocks, in addition to the usual 25 minute blocks. In order to determine whether to incorporate the 50 minute blocks into the operational scaling, this study sought to determine whether longer blocks measure a different construct from that assessed by the 25 minute blocks. Structural equation modeling tested the hypothesis that the structural parameters relating reading ability to demographic variables do not differ across block type. A multiple group analysis, where type of block (25 minute or 50 minute) defined the two groups, was used. The null hypothesis for this comparison was that the two types of blocks measure the same trait but may differ in observed mean and variance.Results of the main analysis did not reject the hypothesis of invariant structural parameters, and so the 50 minute blocks were incorporated in the 1992 NAEP scales. Sensitivity analyses indicated that this conclusion was moderately robust to assumptions made about missing data for items which were not reached, although Grade 8 results were more robust than those for Grade 12. Analyses using other measures of fit yielded the same pattern of results, although the magnitude of the χ2 statistics were affected by the fit measure chosen, particularly the asymptotically distribution free method.Attempts to replicate the main analysis in independent samples yielded similar χ2 values at Grade 8, but Grade 12 yielded χ2 values which were substantially higher for some of the samples. The Grade 12 results raise questions as to the generalizability of the main analysis. Alternatives to the reliance on χ2 measures are discussed for future research.
GE FRST EVALUATION REPORT: HOW WELL DOES A STATISTICALLY‐BASED NATURAL LANGUAGE PROCESSING SYSTEM SCORE NATURAL LANGUAGE CONSTRUCTED‐RESPONSES?
Authors: Burstein Jill C. , Kaplan Randy M. 
Published: August 2014

ABSTRACTThere is a considerable interest at Educational Testing Service (ETS) to include performance‐based, natural language constructed‐response items on standarized tests. Such items can be developed, but the projected time and costs required to have these items scored by human graders would be prohibitive. In order for ETS to include these types of items on standardized tests, automated scoring systems need to be developed and evaluated. Automated scoring systems could decrease the time and costs required for human graders to score these items. This report details the evaluation of a statistically‐based scoring system, the General Electric Free‐Response Scoring Tool (GE FRST). GE FRST was designed to score short‐answer, constructed‐responses of up to 17 words. The report describes how the system performs for responses on three different item types. For the sake of efficiency, it is important to evaluate systems on a number of item types to see if the system's scoring method can generalize to a number of item types.
TOWARD COMPUTER‐BASED PERFORMANCE ASSESSMENT IN MATHEMATICS
Authors: Singley Mark K. , Bennett Randy Elliot 
Published: August 2014

ABSTRACTOne of the main limitations of the current generation of computer‐based tests is its dependency on the multiple‐choice item. Our work is aimed at extending computer‐based testing by bringing limited forms of performance assessment to it in the domain of mathematics. This endeavor involves not only building task types that better reflect valued problem solving, but also creating an integrated set of supports including easy‐to‐use interfaces, tutorials to teach novice computer users how to negotiate those interfaces, tools that help test developers create items quickly, and mechanisms for scoring constructed responses more efficiently.