bluuebunny/crossref_metadata_2025_split huggingface dataset:
MANAGING THE INFLUENCE OF DIF FROM BIG ITEMS: THE 1988 ADVANCED PLACEMENT HISTORY TEST AS AN EXAMPLE
Authors: Wainer Howard , Lukhele Robert 
Published: August 2014

ABSTRACTBuilding tests out of large items brings with it a number of problems. One major problem is that it is often too difficult and too expensive to extensively pretest large items. Thus the sorts of screening for flaws that are pro forma for multiple choice items is not often done for large items. In addition, since there are so few large items on an operational test, not counting an entire item that is found to be flawed in an operational administration may be tantamount to aborting that administration. In this paper we examine the efficacy of the alternative of continuous item weighting. This alternative is illustrated on data from the 1988 administration of the College Board's Advanced Placement History Test.
CORRELATES OF SATISFACTION WITH GRADUATE SCHOOL APPLICANTS' PERFORMANCE ON THE GRE WRITING MEASURE
Authors: Powers Donald E. , Fowles Mary E. 
Published: August 2014

ABSTRACTThe focus of the study was the new GRE writing measure and the proposed GRE scoring guide. The objective was to determine the degree to which the features of student essays upon which scores will be based are the same features that graduate educators use at their institutions to evaluate students' writing. A sample of essays, which had been previously reviewed and judged by graduate deans and faculty, were rescored several times by trained readers. Each rescoring was based on a specific trait in the GRE scoring guide – development of ideas, sentence structure, and so on. The influence of each feature on the judgments made by faculty/deans and on the scores assigned by trained readers was compared. No evidence was uncovered to suggest any differences between graduate deans/faculty and GRE essay readers with respect to the bases on which they judge essay quality.
EFFECTS OF DISCLOSING ESSAY TOPICS FOR A NEW GRE WRITING TEST
Authors: Powers Donald E. , Fowles Mary E. 
Published: August 2014

ABSTRACTTo determine the effects on test performance and test validity of releasing GRE essay topics before an examination, some 300 prospective graduate students each wrote two essays, one on a previously unseen topic and one on a topic that was made available about two weeks before the test. Although for each of the four essay topics studied, performance was slightly better when a topic had been disclosed, analyses did not reveal any statistically significant effect of disclosure on test performance. Nor was there any detectable effect of disclosure on the meaning of test scores, as reflected in the relationship of test performance to several independent indicators of writing skill.Limitations of the study are discussed in terms of its generalizability to a high‐stakes testing situation and the extent to which it addressed both the positive and the negative effects of disclosure.
USING A NEURAL NET TO PREDICT ITEM DIFFICULTY
Authors: Boldt R. F. , Freedle Roy 
Published: August 2014

ABSTRACTSeveral studies that attempted to predict item difficulty using linear weighted composites have been done at ETS and elsewhere. The variables in the composites were coded item characteristics. The validity of these predictions was evaluated using multiple correlations between predicted and actual item difficulties. Results of these studies indicate that the validities of the linear predictions were not zero, but neither were they high. Also, success varied with the particular prediction task attempted, with validities varying from .45 to .77. The figures were promising, but more precision would be helpful for item design and test specification. The present study sought to improve on the validity of the linear predictions by using a “neural net,” a technique that is widely used for nonlinear prediction and is borrowed from artificial intelligence applications. Predictions made using the neural net might be more accurate than those made using linear prediction for two reasons: (a) the latter is a special case of neural net prediction, one in which the variables do not interact, and (b) it is thought that item characteristics do interact to produce item difficulty. Unfortunately, it was found that using the neural net did not lead to substantial improvement in prediction when using the same characteristics as arguments that had proven superior for linear prediction. Perhaps the performance of the neural net did not compare more favorably with that of linear prediction because the variables used had already been chosen to be optimal for linear prediction. For this reason, in an extension of the original project, variables that were optimal for the neural net were sought using a search procedure known as “genetic algorithm.” As used in this study, the genetic algorithm is a constrained search procedure intended to find the best of a fixed number of predictors, that is, the same number used in linear prediction. Applying the genetic algorithm did, indeed, lead to improved prediction. And as expected, most of the variables that were effective in linear prediction did not survive the search. However, it proved to be useful to use the variables that were effective in linear prediction as the starting set for the genetic algorithm, rather than using a randomly selected set. Only two variables were found that were useful for prediction in both the linear system and a neural net; several other variables were found that might be useful in a neural net application but were not effective in a linear application. Substantial capitalization on chance was noted in this study, which can weaken the substantive inferences. Finally, results of this study indicate that the performance of the genetic algorithm could be improved by including chromosomes that contain variables found to be effective in a linear prediction system.
JOB ANALYSIS OF THE KNOWLEDGE IMPORTANT FOR NEWLY LICENSED SOCIAL STUDIES TEACHERS
Authors: Norback Judith Shaul , Bukatko Patricia A. , Wesley Scott , Rosenfeld Michael 
Published: August 2014

ABSTRACTThe purpose of this study was to conduct a job analysis focusing on the knowledge important for newly licensed (certified) Social Studies teachers. The results of the job analysis will be used to develop test specifications for the Subject Assessment in Social Studies of the Praxis Series: Professional Assessments for Beginning Teachers™.A domain consisting of 10 major knowledge areas (United States History, World History, Skills/Themes, Government/Civics/Political Science, Sociology, Anthropology, Psychology, Geography, Economics, and Pedagogy Specific to Social Studies) and 483 knowledge statements was developed by subject‐matter experts.Due to the large size of the content domain, two survey forms were developed. Form 1 consisted of the major areas of U.S. History, World History, and Skills/Themes and contained 249 specific knowledge statements. Form 2 consisted of the areas of World History, Government/Civics/Political Science, Sociology, Anthropology, Psychology, Geography, Economics, and Pedagogy Specific to Social Stuies and contained 234 statements.These survey forms were then administered to a national sample of social studies teachers, administrators, and college faculty whose names were obtained from the membership of the National Council of the Social Studies (NCSS). In addition, a supplementary group of teachers from schools with high levels of minority students (80–100%) and teachers with five or less years of teaching experience were included in order to increase the likelihood that a sufficient number of responses from these groups would be available to reflect their perspectives on the job of a newly licensed social studies teacher. These professionals rated the importance of each knowledge area and statement using a 5‐point rating scale that ranged from 0 (of no importance) to 4 (very important).A cut point of a mean importance rating of 2.50 (the midpoint between moderately important [scale value 2] and important [scale value 3] was established to identify the core of important statements. The results of the mean analysis conducted for teachers and college faculty in the primary sample and for teachers in the supplementary sample for Form 1 showed that 19 of the 249 knowledge statements were rated less then 2.50 by one or more of the three groups. This represents 7.6% of the content domain. The corresponding results for Form 2 yielded 21 (9.0%) statements with mean ratings below the cut point. In the mean analysis for subgroups, 34 (13.7%) statements were rated below 2.50 for Form 1, while in Form 2, 29 (12.4%) were rated below the cut point.The knowledge statements that were verified to be important by the surveyed teachers, college faculty, and the demographic subgroups should be used as the foundation for the development of test specifications.
USING MULTIMEDIA IN LARGE‐SCALE COMPUTER‐BASED TESTING PROGRAMS
Authors: Bennett Randy Elliot , Goodman Madeline , Hessinger James , Ligget John , Marshall Geoff , Kahn Helen , Zack June 
Published: August 2014

ABSTRACTTo make substantial improvements in how and what we measure, future generations of tests will need to tap nontraditional constructs, base test designs on cognitive principles, and increase the diversity of problem types. The advent of computer‐based tests makes some of these improvements more feasible, especially diversifying problem types. One means of diversification is by including dynamic stimuli such as audio, video, and animation. We give examples of how such stimuli might be employed to assess skill in history, physical education, and the sciences. In addition, we pose an initial set of issues for consideration by operational testing programs.
A TREE‐BASED APPROACH TO PROFICIENCY SCALING & DIAGNOSTIC ASSESSMENT
Authors: Sheehan Kathleen M. 
Published: August 2014

ABSTRACTA new procedure for generating instructionally‐relevant diagnostic feedback is proposed. The approach involves first constructing a strong model of student proficiency and then testing whether individual students' observed item response vectors are consistent with that model. Diagnoses are specified in terms of the combinations of skills needed to score at increasingly higher levels on a tests's reported score scale. The approach is applied to the problem of developing diagnostic feedback for the SAT I Verbal Reasoning Test. Using a variation of Wright's (1977) person‐fit statistic, it is shown that the estimated proficiency model accounts for 91% of the “explainable” variation is students' observed item response vectors.
SIMULATING THE USE OF DISCLOSED ITEMS IN COMPUTERIZED ADAPTIVE TESTING1,2
Authors: Stocking Martha L. , Ward William C. , Potenza Maria T. 
Published: August 2014

ABSTRACTRegular use of questions previously made available to the public (disclosed) may provide one way to meet the requirement for large numbers of questions in a continuous testing environment, that is, an environment in which testing is offered at test‐taker convenience throughout the year rather than on a few prespecified test dates. First it must be shown that such use has effects on test scores small enough to be acceptable. In this study simulations are used to explore the use of disclosed items under a worst case scenario which assumes that disclosed items are always answered correctly. Some item pool and test designs were identified in which the use of disclosed items produces effects on test scores that may be viewed as negligible.
VALIDITY AND AUTOMATED SCORING: IT'S NOT ONLY THE SCORING
Authors: Bennett Randy Elliot , Bejar Isaac I. 
Published: August 2014

ABSTRACTEarly work on automated scoring predated the ready availability of mechanisms for inexpensively delivering computer‐based tests and collecting responses. Hence, this work used responses to conventionally delivered tasks that had somehow been translated to machine‐readable form. The necessity of operating in this manner focused attention initially on the empirical characteristics of automated scores. As the availability of computer‐based testing environments grew it became possible to implement entire operational exams and, thus, to think broadly about the implications of automated scoring for validity. In this paper we argue that a comprehensive discussion of validity and automated scoring includes the interplay among construct definition and test and task design; examinee interface; tutorial; test development tools; automated scoring; and reporting—for in the development process these components affect one another. As modern validity theory postulates, the validation argument must, therefore, ideally provide not only empirical evidence of score relationships but also theoretical rationales to support a variety of design decisions. We further argue that the interdependency among computer‐based test components provides a unique opportunity to greatly improve educational and occupational assessment.
A SOFTWARE TOOL FOR RAPIDLY PROTOTYPING NEW FORMS OF COMPUTER‐BASED ASSESSMENTS
Authors: Katz Irvin R. 
Published: August 2014

ABSTRACTThis report describes the Free‐Response Authoring, Delivery, and Scoring System (FRADSS). FRADSS allows test developers, who might have no programming experience, to put their ideas for computer‐based test items directly onto computer. The system was used for the GRE Mathematical Reasoning pre‐pilot and is currently being used for the GRE Engineering New Testing Initiative. The report begins with a description of the “niche” in the test development process that FRADSS is intended to fill, then presents several items authored using FRADSS. After describing the various components of the system (authoring, delivery, and scoring), the report discusses benefits of using FRADSS in the early stages of test development. These benefits were gleaned from interviews with item writers who have used FRADSS in their work.
TOWARD ASSESSMENT OF DESIGN SKILL IN ENGINEERING
Authors: Katz Irvin R. , James Carolyn M. 
Published: August 2014

ABSTRACTDesign is an important part of engineering education, but cannot be assessed easily through current multiple‐choice tests. In this report, we explore the possibility of including design problems in assessments of engineering students. We report a study comparing constructed‐response design problems and constructed‐response versions of typical engineering assessments, labeled “analysis” problems. Through analyses of verbal protocols, we argue that design problems can elicit qualitatively different behaviors compared with analysis problems, and therefore may assess different underlying constructs. While design problems may be a worthwhile addition to assessments of engineering students, such problems present several challenges, including issues of automatic scoring, the amount of information about candidate skill contained in responses, and the potential for construct‐irrelevant variance. These challenges are surmountable, but require careful construction of design assessments to avoid potential pitfalls. This work represents one part of a research program aimed at informing the redesign of the GRE Engineering Subject Test.
EXAMINING THE VALIDITY OF A COMPUTER‐BASED GENERATING‐EXPLANATIONS TEST IN AN OPERATIONAL SETTING
Authors: Bennett Randy Elliot , Rock Donald A. 
Published: August 2014

ABSTRACTGenerating Explanations (GE) is a computer‐delivered item type that presents a situation and asks the examinee to pose as many plausible reasons for it as possible. Previous research suggests that GE measures a divergent thinking ability largely independent of the convergent skills tapped by the GRE General Test. This study was conducted to determine if prior GE validity results generalized to the GRE candidate population, how population groups performed, what effects partial‐credit modeling might have for validity, and what problems were associated with operational administration. Validity results showed that earlier findings were generally supported: GE was found to be reliable but only marginally related to the General Test and to make significant (but small) independent contributions to the explanation of relevant criteria. With respect to population groups, GE produced smaller gender and ethnic group differences than did the General Test and showed the same relations to outside criteria across groups, suggesting it was measuring similar skills in each population. Attempts to model GE responses on a partial‐credit IRT scale succeeded but produced no improvement in relations with external criteria over those obtained by summing raw item scores. Finally, interviews conducted with examinees to detect potential delivery problems suggested that the directions needed to be shortened.
AN INVESTIGATION OF THE VALIDITY OF AN EMPIRICAL BAYES APPROACH TO MANTEL‐HAENSZEL DIF ANALYSIS
Authors: Zwick Rebecca , Thayer Dorothy T. , Lewis Charles 
Published: August 2014

ABSTRACTWe investigated the utility of an empirical Bayes (EB) enhancement to Mantel‐Haenszel (MH) DIF analysis. We assume that the MH statistics are normally distributed and that the prior distribution of underlying DIF parameters is also normal. We use the posterior distribution of DIF parameters to make inferences about the item's true DIF status (the True DIF method) and the posterior predictive distribution to predict the item's future observed status (the Future DIF method). DIF status is expressed in terms of the probabilities associated with each of the five DIF levels defined by the ETS classification system: C‐, B‐, A, B+, and C+. The proposed methods have several possible advantages. First, the use of prior information allows for more stable DIF estimation, especially in small samples‐a feature that should be advantageous in computer‐adaptive testing. The EB approach may convey information about DIF stability in a more useful way. The results represent the state of knowledge about an item's true DIF status as probabilistic, not deterministic. The EB DIF results lend themselves well to graphic display, which may be useful in conveying findings to test developers and DIF committees. Also, results can be projected for future administration conditions that may differ from the present ones. As an alternative method of DIF flagging, we also investigated the use of a loss function to identify potential DIF items; this approach is very general and is not linked to the ETS DIF classification system. A possible advantage of the loss function procedure over the ETS classification system is its greater flexibility.
A BAYESIAN RANDOM EFFECTS MODEL FOR TESTLETS
Authors: Bradlow Eric T. , Wainer Howard , Wang Xiaohui 
Published: August 2014

ABSTRACTStandard item response theory (IRT) models fit to dichotomous examination responses ignore the fact that sets of items (testlets) often come from a single common stimuli (e.g. a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences such as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g an adaptive test). To model examinations that may be a mixture of independent items and testlets, we modified one standard IRT model to include an additional random effect for items nested within the same testlet. We use a Bayesian framework to facilitate posterior inference via a Data Augmented Gibbs Sampler (DAGS, Tanner and Wong 1987). The modified and standard IRT models are both applied to a data set from a disclosed form of the SAT. We also provide simulation results that indicates that the degree of precision bias is a function of the variability of the testlet effects, as well as the testlet design.
INQUIRING ABOUT EXAMINEES' ETHNICITY AND SEX: EFFECTS ON AP® CALCULUS AB EXAMINATION PERFORMANCE
Authors: Stricker Lawrence J. 
Published: August 2014

ABSTRACTSteele and Aronson (1995) found that the performance of African‐American subjects on test items portrayed as a problem‐solving task, in a laboratory experiment, was adversely affected when they were asked about their ethnicity. This outcome was attributed to “stereotype threat”: performance was disrupted by the subjects' concerns about fulfilling the negative stereotype concerning African Americans' intellectual ability. Extending that research, this field experiment evaluated the effects of inquiring about ethnicity and sex on the performance of examinees taking the Advanced Placement (AP) Calculus AB Examination in an actual test administration. With a minor exception, this inquiry had no statistically and practically significant effects on the test performance of African‐American, female, or other subgroups of examinees.
COMPUTER FAMILIARITY AMONG TOEFL EXAMINEES
Authors: Kirsch Irwin , Jamieson Joan , Taylor Carol , Eignor Daniel 
Published: August 2014

ABSTRACTThe growing use of computer‐based testing raises concerns about access and equity. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 may introduce bias by confounding the measurement of English‐language proficiency with computer familiarity. As the first study in a series of research efforts to address this question, a 23‐item questionnaire focusing on examinees' access to, attitude toward, and experience using computers as well as related technologies was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration.The examinee response rate for the survey was over 90%, and the sample of respondents was determined to be highly similar to the annual TOEFL examinee population in terms of gender, native language, test center region, TOEFL test score ranges, and reason for taking the test.Factor analysis procedures (specifically, a principal factors approach with varimax rotation) were used to determine the dimensionality of the survey questionnaire. The first factor from a two‐factor common factor solution consisted of 11 items and had a reliability of .91. This factor was used to classify TOEFL examinees into one of three groups according to their level of familiarity with computers: low, moderate, or high. Overall, some 16% of the TOEFL population was judged to have low computer familiarity, another 34% to have moderate familiarity, and approximately 50% to have high familiarity. In terms of background characteristics, computer familiarity was found to be unrelated to age, but was related to gender, native language, region of the world where the examinee was born, and test‐center region. Computer familiarity was also shown to be related to individuals' TOEFL test scores and their reason for taking the test but unrelated to whether or not they had taken the test previously.Analyses of variance were conducted to investigate the relationships among proficiency as measured by the paper‐and‐pencil TOEFL test, computer familiarity, and selected background characteristics, including gender, age, native language, and reason for taking the test. Effect sizes for each variable as well as for the interactions among the variables are discussed herein. The study results are also considered in light of existing research relating to various aspects of computer familiarity as well as in terms of their implications for professionals in universities and intensive language programs who will need to serve the needs of this diverse set of learners.