bluuebunny/crossref_metadata_2025_split huggingface dataset:
THE PREDICTION OF SAT READING COMPREHENSION ITEM DIFFICULTY FOR EXPOSITORY PROSE PASSAGES
Authors: Freedle Roy , Kostin Irene 
Published: August 2014

ABSTRACTThe primary goal of this project was to examine the predictability of SAT reading item difficulty (equated delta) for main idea items, and collectively, the predictability of three major reading item types: main idea, inference and explicit statement items. A secondary purpose in predicting item difficulty was to contrast the responses of high verbal and low verbal ability examinees. Primary attention was paid to studying 110 main idea reading items and their associated passages. However, additional results are reported for 285 reading items taken from 34 disclosed SAT forms which represented a wider range of reading item types.The percent variance of main idea item difficulty accounted for varied from 46% to 59% depending upon the particular analysis. The predictability of all three reading item types (n = 285) varied from 21% to 29%, depending upon the particular analysis.Details of item predictability were explored by evaluating several hypotheses. Results indicated that (1) multiple‐choice reading items are sensitive to variables similar to those reported in the experimental literature on comprehension, (2) many of these variables provide significant independent predictive information in regression analyses, (3) the placement (early versus middle of text) of relevant main idea information affects item difficulty, and (4) considerable agreement between SAT and GRE reading predictability was found. Additional results contrast the performance of high and low ability groups.
CROSS‐VALIDATION OF A PROPORTIONAL ITEM RESPONSE CURVE MODEL
Authors: Boldt Robert F. 
Published: August 2014

ABSTRACTIn a previous research study, factor analyses of TOEFL section item correlations (phi‐coefficients) yielded a single factor in each section. This result is consistent with the assumption that, within sections, the item response curves are proportional, a possibility not previously suggested or explored in item response theoretical studies. One potential use to the TOEFL program is that this assumption could serve as a basis for simpler equating methods than are now being used. Use of the assumption entails estimating fewer parameters and the computations could be simpler than with current methods. Other anticipated benefits are that it could reduce the chances of error in calibrating items and would permit the use of smaller equating samples, which would in turn allow pretest evaluation of substantially more items for possible operational use.To increase the body of data evaluating the model, a cross‐validation study was undertaken in which PIRC, 3PL, and a modified Rasch model were used to predict item scores of selected examinees on selected items. The study also compared predicted half‐test scores with actual scores for each method. These comparisons were made using varying amounts of data to calibrate the items and calculate scores for the examinees in accordance with different estimation sample sizes. Surprisingly, the accuracy of predictions made by the models was approximately the same, and the estimation sample size appeared to make little difference. Further research in terms of randomization trials on the current data set and self‐equatings similar to those conducted by Hicks (1984) is discussed.
A FRAMEWORK FOR STUDYING DIFFERENCES BETWEEN MULTIPLE‐CHOICE AND FREE‐RESPONSE TEST ITEMS
Authors: Mislevy Robert J. 
Published: August 2014

ABSTRACTThis paper lays out a framework for comparing the qualities and the quantities of information about student competence provided by multiple‐choice and free‐response test items. After discussing the origins of multiple‐choice testing and recent influences for change, the paper outlines an “inference network” approach to test theory, in which students are characterized in terms of levels of understanding of key concepts in a learning area. It then describes how to build inference networks to address questions concerning the information about various criteria conveyed by alternative response formats. The types of questions that can be addressed in this framework include those that can be studied within the framework of standard test theory. Moreover, questions can be asked about generalized kinds of reliability and validity for inferences cast in terms of recent developments in cognitive psychology.
ALTERNATIVE MATHEMATICAL APTITUDE ITEM TYPES: DIF ISSUES
Authors: Schmitt Alicia P. , Crone Carolyn R. 
Published: August 2014

ABSTRACTAlternative mathematical items administered as prototypes at the Spring 1989 Field Trials are evaluated for differential item functioning (DIF) and differential speededness. Results for Algebra Placement (AP) and Student Produced Response (SPR) items are presented and contrasted to results obtained on the two current SAT‐Math items: Regular Math and Quantitative Comparison. Analyses on comparisons between female and comparable male examinees, and between Asian‐American, Black, and Hispanic examinees in comparison to comparable White examinees indicate that both of these alternative items appear to have DIF. Additional DD? analyses comparing the use of an internal versus an external matching criteria for the SPR items show evidence of negative DIF with either criteria. Results using the MH D‐DIF statistic are more extreme than DIF results using the STD P‐DIF index. The metric used to calculate the DIF indices may be accountable for the differences observed. Differential speededness results indicate that the two Math prototypes have slightly higher levels of differential speededness than the SAT‐M.The SPR items pose an interesting problem for DIF. The definition of an appropriate DIF matching criterion for constructed response item types needs more study. Metric differences between methods and their effect on difficult or easy items also needs further exploration. Until these methodological issues are resolved, results of DD? studies on constructed response items should be interpreted with caution.
ON THE MEANINGS OF CONSTRUCTED RESPONSE
Authors: Bennett Randy Elliot 
Published: August 2014

ABSTRACTThis paper explores the multiple meanings of constructed response. The term is conceptualized as a superordinate category subsuming a variety of item formats ranging from minor adaptations of multiple choice to “authentic” assessment tasks. The meaning of these formats is explored through four validity facets: construct validity, relevance/utility, value implications, and social consequences. This exploration leads to a multi‐dimensional structure for conceptualizing constructed‐response formats. From this conception, it is argued that appreciating the term's multiple meanings broadens the intellectual and political debate, identifies gaps in our knowledge base, and suggests a balanced approach to educational assessment.
A KNOWLEDGE BASE FOR TEACHER LICENSURE: A COMPARISON OF STATE REQUIREMENTS, PROFESSIONAL ASSOCIATION RECOMMENDATIONS AND TEACHER, TEACHER EDUCATOR, AND ADMINISTRATOR OPINIONS OF TEACHER PREPARATION ON THE PRINCIPLES OF LEARNING AND TEACHING
Authors: Wesley Scott , Klem Lynn , Reynolds Anne 
Published: August 2014

ABSTRACTA notable aspect of the educational reform movement is the increasing attention placed on procedures and assessments for licensing teachers. Currently, Educational Testing Service (ETS) is engaged in a large‐scale development project related to teacher licensure. This project will develop a new generation of assessments for licensing beginning teachers. At the foundation of these assessments are two research projects. One of these, Project C H A R T, is an effort to identify state requirements for licensure in all subject areas currently being taught in elementary, middle, and secondary schools. This project is also identifying recommended standards established by professional organizations and associations concerning the skills and competencies on which teacher preparation programs should focus. The second project is a series of studies aimed at identifying what teacher candidates and new teachers should know and be able to do. The methodology being used in this research is job analysis. The purpose of these job analyses is to identify, through surveys of practicing professionals, the important knowledge areas and skills for new teachers.With these two projects, the question arises as to whether there is agreement among states, professional associations, and practicing professionals about what new teachers should know and be able to do. This paper addresses this question by comparing the results of Project C H A R T with the results of the job analysis for one area, General Principles of Teaching and Learning. General Principles of Teaching and Learning is a subject assessment that focuses on pedagogical principles that cross subject‐matter and grade boundaries.The results of this comparison indicate that there is considerable agreement among states, professional associations, and the members of the educational community. There are, however, several areas of disagreement between and within these groups. The paper discusses the areas of agreement and disagreement as well as their implications for teacher preparation and regulation and the development of a knowledge base for teacher education.
EVALUATING HYPOTHESES ABOUT DIFFERENTIAL ITEM FUNCTIONING1,2
Authors: Schmitt Alicia P. , Holland Paul W. , Dorans Neil J. 
Published: August 2014

ABSTRACTStudies evaluating hypotheses about sources of differential item functioning (DIF) are classified into two categories: observational studies evaluating operational items and randomized DIF studies evaluating specially constructed items. For observational studies, advice is given for item classification, sample selection, the matching criterion, and the choice of DIF techniques, as well as how to summarize, synthesize and translate DIF data into DIF hypotheses. In randomized DIF studies of specially constructed items, specific hypotheses, often generated from observational studies, are evaluated under rigorous conditions. Advice for these studies focuses on the importance of carefully constructed items to assess DIF hypotheses. In addition, randomized DDF studies are cast within a causal inference framework, which provides a justification for the use of standardization analyses or logistic regression analysis to estimate effect sizes. Two studies that have components spanning the observational and controlled domains are summarized for illustrative purposes. Standardization analyses are used for both studies. Special logistic regression analyses of an item from one of these studies are provided to illustrate a new approach in the assessment of DIF hypotheses using specially constructed items.
MEASUREMENT PROBLEMS
Authors: Wainer Howard 
Published: August 2014

ABSTRACTHistory teaches the continuity of science; the developments of tomorrow have their genesis in the problems of today. Thus any attempt to look forward is well begun with an examination of unsettled questions. Since a clearer idea of where we are going smooths the path into the unknown future, a periodic review of such questions is prudent. The present day, lying near the juncture of the centuries, is well suited for such a review. This paper reports twenty unsolved problems in educational measurement and points toward what seem to be promising avenues of solution.
PROBABILITY‐BASED INFERENCE IN A DOMAIN OF PROPORTIONAL REASONING TASKS
Authors: Béland Anne , Mislevy Robert J. 
Published: August 2014

ABSTRACTEducators and psychologists are increasingly interested in modelling the processes and knowledge structures by which people learn and solve problems. Progress has been made in developing cognitive models in several domains, and in devising observational settings that provide clues about subjects' cognition from this perspective. Less attention has been paid to procedures for inference or decision‐making with such information, given that it provides only imperfect information about cognition—in short, test theory for cognitive assessment. This paper describes probability‐based inference in this context, and illustrates its application with an example concerning proportional reasoning.
GIVING A CHOICE OF TOPICS ON A TEST OF BASIC WRITING SKILLS: DOES IT MAKE ANY DIFFERENCE?
Authors: Powers Donald E. , Fowles Mary E. , Farnum Marisa , Gerritz Kalle 
Published: August 2014

ABSTRACTAs part of the development of the academic skills portion of The Praxis Series: Professional Assessments for Beginning Teachers™ (the successor to the NTE examinations), a sample of undergraduate students each wrote at least two 50‐minute essays. For each essay, the topic was selected by the examinee from a pair of topics. Before writing, all students indicated their preferences for each of 20 possible topics from which their topics would be drawn. Subsequently, comparisons were made between performance on examinees' high and low preferred topics. The relationship of selected variables (e.g., undergraduate grades and admissions test scores) to performance on each kind of topic was also compared.Student preferences for different topics varied considerably across topics, and topics that were most preferred by some examinees were often least preferred by others. Preferences, however, exhibited little if any relationship to essay scores. Finally, scores based on high and low preferred topics had similar patterns of correlations with other variables.
HOW TO EQUATE TESTS WITH LITTLE OR NO DATA
Authors: Mislevy Robert J. , Sheehan Kathleen M. , Wingersky Marilyn 
Published: August 2014

ABSTRACTStandard procedures for equating tests, including those based on item response theory (IRT), require item responses from large numbers of examinees. Such data may not be forthcoming for reasons theoretical, political, or practical. Information about items' operating characteristics may be available from other sources, however, such as content and format specifications, expert opinion, or psychological theories about the skills and strategies required to solve them. This paper shows how, in the IRT framework, collateral information about items can be exploited to augment or even replace examinee responses when linking or equating new tests to established scales. The procedures are illustrated with data from the Pre‐Professional Skills Test (PPST).
ASSESSING THE PROFESSIONAL KNOWLEDGE OF PROSPECTIVE TEACHERS: IDENTIFYING THE QUESTIONS THAT WORK BEST
Authors: Powers Donald E. , Pitcher Barbara 
Published: August 2014

ABSTRACTAn item‐level analysis was undertaken for one national administration of the Professional Knowledge test of the NTE Core Battery. The performances of several relevant subgroups of examinees defined by undergraduate major field, were compared for individual test items. Analyses revealed that a subset of test items was differentially easier for particular subgroups of examinees who had received greater exposure to teacher preparation. A smaller subset of items favored examinees having little if any such exposure. The nature of each kind of item is discussed with respect to performance expectations for various subgroups of examinees.
THE INTERPLAY OF EVIDENCE AND CONSEQUENCES IN THE VALIDATION OF PERFORMANCE ASSESSMENTS
Authors: Messick Samuel 
Published: August 2014

ABSTRACTAuthentic and direct assessments of performances and products are conceptualized in terms of multiple distinctions having implications for validation. These include contrasts between performances and products, between assessment of performance per se and performance assessment of competence or other constructs, between structured and unstructured problems and response modes, between decomposed task skills and complex task performance, and between contextualized and decontextualized knowledge and skill. The concepts of “authenticity” and “directness” of assessment are analyzed as promissory validity claims that they offset, respectively, the two major threats to construct validity, namely, construct underrepresentation and construct‐irrelevant variance. These distinctions are examined in the context of an overarching contrast between task‐driven and construct‐driven performance assessment. With respect to validation, the salient role of both positive and negative consequences is underscored as well as the need for evidence bearing on the various aspects of construct validity (content, substantive, structural, external, generalizability, and consequential).
PROFILES OF PERFORMANCE BY EDUCATIONAL LEVEL ON THE NTE CORE BATTERY
Authors: Powers Donald E. 
Published: August 2014

ABSTRACTSeveral comparisons were made across educational level with respect to performance on each of the three NTE Core Battery tests‐Communication Skills, General Knowledge, and Professional Knowledge. Various controls were implemented in order to discount selected threats to the interpretation of between‐level test score differences as a function of teacher education. Several alternative analyses yielded results that were generally consistent with an interpretation of the Professional Knowledge test as a measure of students' knowledge of the teaching profession. A major limitation of the data on which the results are based was the lack of any specific information about the nature and timing of relevant coursework.
IDENTIFYING DIFFERENTIALLY FUNCTIONING ITEMS IN THE NTE CORE BATTERY
Authors: McPeek W. Miles , Wild Cheryl L. 
Published: August 2014

ABSTRACTThe purpose of this study is to evaluate the use of the Mantel‐Haenszel statistic as a methodology for identifying differentially functioning items on the NTE Programs Core Battery tests. Retrospective analyses of data collected over a three‐year period are reported for Black/White, Hispanic/White, and female/male comparisons. The three tests that comprise the Core Battery are included in the analyses—Communication Skills, General Knowledge, and Professional Knowledge. Data concerning the replicability of the results, the number of items identified as differentially difficult, and the characteristics of the test questions associated with differential difficulty are discussed. Replicability of DIF values from two administrations based on correlational data was modest to high, with greatest replicability found in the largest samples. Although the replicability is modest to high when considering all the items, the results are less consistent when only the items identified as showing differential difficulty are considered. Generally, 6 to 9 percent of the items were identified as being differentially difficult per comparison, with a greater number of items flagged in the female/male analyses than in the Black/White or Hispanic/White analyses. There was little overlap among the items flagged in the three different analyses. Although analyses suggested some content characteristics that may be related to differential item functioning, the findings should be considered exploratory due to the small number of items studied and the post hoc nature of the sample. Also, the findings should not be assumed to apply to populations other than NTE test takers.
AN INITIAL EVALUATION OF THE USE OF BIVARIATE MATCHING IN DIF ANALYSES FOR FORMULA SCORED TESTS12
Authors: Pomplun Mark , Baron Patricia , McHale Fred 
Published: August 2014

ABSTRACTThis study evaluated the use of bivariate matching as a solution to the problem of studying DIF with formula scored tests. This problem specifically involves including in or excluding from a formula scored matching criterion formula scored items to‐be‐studied for a DIF analysis. Using SAT Verbal data with large and small samples both Male‐Female and Black‐White group comparisons were investigated. MH D‐DIF values and DIF category classifications based on bivariate matching were compared with MH D‐DIF values and categories based on rights scored and formula scored matching criteria. When large samples were used, MH D‐DIF values based on the bivariate matching criterion were ordered very similarly to MH D‐DIF values based on the other criteria. The DIF category classifications were almost identical. However, with small samples the MH D‐DIF values based on the bivariate matching criterion displayed only moderate correlations with MH D‐DIF values from the other criteria. In addition, the DIF category classifications based on the bivariate matching criterion showed fewer high DIF items than those based on the rights or formula scored matching criteria. As a secondary result, this study documented the differences between formula and rights scored criteria in DIF analyses of formula scored tests. These results showed that the substitution of a rights scored criterion in a DIF analysis of a formula scored test resulted in MH D‐DIF values that were ordered very similarly as those based on the proper formula scored criterion. The MH D‐DIF values based on the rights score criterion were, however, different in magnitude from those based on the formula scored criterion. The differences were related to item difficulty and were greater for comparisons with larger ability differences.
CHARACTERISTICS ASSOCIATED WITH DIFFERENTIAL ITEM FUNCTIONING ON THE SCHOLASTIC APTITUDE TEST: GENDER AND MAJORITY/MINORITY GROUP COMPARISONS
Authors: Carlton Sydell T. , Harris Abigail M. 
Published: August 2014

ABSTRACTThe purpose of the study was to investigate whether selected test and item characteristics in the SAT are associated with unexpected differential item functioning (DIF) for males and females and for majority and minority group members (i.e., White performance compared with Black, Asian‐American, Hispanic, and American Indian performance). Six forms of the SAT that were administered relatively recently were used. Each of the six forms consists of Verbal sections (containing Analogies, Antonyms, Sentence Completions, and Reading Comprehension), Mathematics sections (containing Regular Mathematics problem‐solving and Quantitative Comparisons), and the Test of Standard Written English (TSWE) (containing Usage and Sentence Correction). Findings from previous studies, test specifications, and suggestions offered by test development experts led to the identification of more than one hundred a priori item coding categories, and each SAT item was coded accordingly. Items were coded with regard to type, content, and format.The Mantel‐Haenszel procedure was used to provide an index of differential item functioning (DIF) for each reference/focal group comparison. Females, Asian‐Americans, Blacks, Hispanics, and American Indians were the focal groups; males and Whites were the reference groups. With the exception of the American Indians, each focal group was represented in sufficient numbers on each test form to lead to meaningful interpretation of data. For each item category, one‐way analyses of variance were computed using as the dependent variable the Mantel‐Haenszel DIF values. Analyses were run separately for each reference/focal group investigated.The study reports on patterns of differential performance across SAT sections as well as on section‐specific differences. Patterns across several ethnic groups and between two gender groups, as well as results specific to each of the groups, are identified. The report addresses the following questions:• Are there unexpected group differences associated with the points tested (e.g., percentages, verb forms)?• What aspects of item content (e.g., subject matter, gender and ethnic references, level of language) are associated with unexpected group differences?• Are there elements of test or item format (e.g., length of stem, formatting or location of the stem or options) that are associated with unexpected group differences?