bluuebunny/crossref_metadata_2025_split huggingface dataset:
EXTENDING THE RULE SPACE MODEL TO A SEMANTICALLY‐RICH DOMAIN: DIAGNOSTIC ASSESSMENT IN ARCHITECTURE
Authors: Katz Irvin R. , Martinez Michael E. , Sheehan Kathleen , Tatsuoka Kikumi K. 
Published: August 2014

ABSTRACTThis paper presents a technique for applying the Rule Space model of cognitive diagnosis (Tatsuoka, 1983) to assessment in a semantically‐rich domain. Responses to 22 architecture test items, developed to assess a range of architectural knowledge, were analyzed using Rule Space. Verbal protocol analyses guided the construction of a model of examinee performance, consisting of processes for constructing an initial representation of an item (labeled understand), forming goals and performing actions based on those goals (solve), and determining whether goals have been attempted and satisfied (check). Item attributes, derived from these processes, formed the basis for diagnosis. Our technique extends Rule Space's applicability by defining attributes in terms of item characteristics and the causal relations between characteristics and the problem‐solving model.Data were collected from 122 architects of various ability levels (students, architecture interns, and professional architects). Rule Space successfully classified approximately 65%, 90%, and 40% of examinees based, respectively, on attributes associated with the understand, solve, and check processes of the problem‐solving model. The findings support the effectiveness of Rule Space in a complex domain and suggest directions for developing new architecture items by using attributes particularly effective at distinguishing among examinees of different ability levels.
TEST THEORY RECONCEIVED
Authors: Mislevy Robert J. 
Published: August 2014

ABSTRACTEducational test theory consists of statistical and methodological tools to support inference about examinees' knowledge, skills, and accomplishments. The evolution of test theory has been shaped by the nature of users' inferences, which, until recently, have been framed almost exclusively in terms of trait and behavioral psychology. Progress in the methodology of test theory enabled users to extend the range of inference, sharpen the logic, and ground their interpretations more solidly within these psychological paradigms. In particular, the focus remained on students' overall tendency to perform in prespecified ways in prespecified domains of tasks; for example, to make correct answers to mixed‐number subtraction problems. Developments in cognitive and developmental psychology broaden the range of desired inferences, especially to conjectures about the nature and acquisition of students' knowledge. Commensurately broader ranges of data‐types and student models are entertained. The same underlying principles of inference that led to standard test theory can be applied to support inference in this broader universe of discourse. Familiar models and methods—sometimes extended, sometimes reinterpreted, sometimes applied to problems wholly different from those for which they were first devised—can play a useful role to this end.
MODELS FOR SCORING MISSING RESPONSES TO MULTIPLE‐CHOICE ITEMS
Authors: Longford Nicholas T. 
Published: August 2014

ABSTRACTThis is a critical evaluation of the rules for coding and scoring of missing responses to multiple‐choice items in educational tests. The focus is on tests in which the test‐takers have little or no motivation; in such tests omitting and not reaching (as classified by the currently adopted operational rules) is quite frequent. Data from the 1991 NAEP Reading Assessment of 17‐year‐olds are used in analyses and illustrative examples. Alternative rules for scoring based on hypothesized behaviour of the test‐takers are proposed.
EVALUATION OF THE MAGNITUDE OF DIFFERENTIAL ITEM FUNCTIONING IN POLYTOMOUS ITEMS
Authors: Zwick Rebecca , Thayer Dorothy T. 
Published: August 2014

ABSTRACTSeveral recent studies have investigated the application of statistical inference procedures to the analysis of differential item functioning (DIF) in test items that are scored on an ordinal scale. Mantel's extension of the Mantel‐Haenszel test is a possible hypothesis‐testing method for this purpose. The development of descriptive statistics for characterizing DIF in polytomous test items has received less attention. A statistic that appears well‐suited as a summary index was proposed by Dorans and Schmitt. In this paper, two possible standard error formulas for this statistic are derived and evaluated, hypothesis testing procedures based directly on this descriptive statistic are outlined, and the results of applications to simulated data are presented.
APPROACHES TO NONIGNORABLE NONRESPONSE WITH APPLICATION TO SELECTION BIAS*
Authors: Allen Nancy L. , Holland Paul W. , Thayer Dorothy T. 
Published: August 2014

ABSTRACTThe main purpose of this paper is to illuminate different approaches to nonignorable nonresponse. Previously literature in educational settings has emphasized bias due to using techniques that are appropriate in the case of ignorable nonresponse when nonignorable nonresponse has actually occurred. The literature in biometrics, econometrics, and statistics contains work that more specifically applicable in this setting. Methods introduced by Pearson, Rubin, Heckman, and Tukey are described and contrasted for two different cases. The two cases include: (1) the simple case where there is one variable of interest and the sampling rule depends on that variable, and (2) the more complicated case where there is one variable of interest and a covariate. In this second case, as in the first, the sampling rule depends on the variable of interest. Finally, the methods, as applied in case 1, are used in an example. The goal of the example is to estimate the total distribution of scores for an essay, when the essay is answered by only some of the test‐takers. Data from an actual test administration where examinees were given an optional essay question were used.
A MISSING DATA APPROACH TO ESTIMATING DISTRIBUTIONS OF SCORES FOR OPTIONAL TEST SECTIONS*
Authors: Allen Nancy L. , Hollanda Paul W. , Thayer Dorothy T. 
Published: August 2014

ABSTRACTMany testing programs include a section of optional questions, in addition to mandatory parts of a test. For instance, some testing programs have examinees select one of several essays or performance tasks as part of the test. These optional parts of a test are not often truly parallel to one another, making direct equating impossible. Groups of examinees selecting each optional test section are not equivalent to one another, so statistical moderation is impossible unless explicit assumptions are made. The purpose of this paper is to provide a general method based on missing‐data methods for nonignorable nonresponse (Little & Rubin, 1987) to estimate distributions of scores on an optional part of a test. The distributions for each selected test part can then be linked to one another to develop comparable scales. The method is compared with other methods used to put optional test sections on comparable scales. If issues about the validity of assumption are resolved, this method could eventually provide a basis for the combination of scores based on optional portions of a test with those based on common sections of a test.This paper is the second in a series of three papers. In the first, “Approaches to Nonignorable Nonresponse with Application to Selection Bias” (Allen, Holland, and Thayer, in press, a), we apply missing‐data methods from settings external to education to this setting. A method from the econometric literature (Heckman, 1976) is contrasted to methods proposed by Little and Rubin (1987) and Tukey (Holland, 1986). In this paper, the perspective of measurement specialists who proposed methods to transform scores for optional questions to comparable scales is contrasted with the approach based on the work of Little and Rubin. In the final paper, “Estimating Scores for an Optional Test Section Using Information from a Common Section” (Allen, Holland, & Thayer, in press, b), the methods of Rubin and Tukey are extended to include common scores for the mandatory part of the test. Data from an example are actually transformed to comparable scales.
COLLEGE MAJOR AND GENDER DIFFERENCES IN THE PREDICTION OF COLLEGE GRADES
Authors: Pennock‐RomÁn MarÍa 
Published: August 2014

ABSTRACTRecent studies have found substantial reductions in gender differences in the prediction of academic achievement in college when variations in grading standards among courses were taken into account. The purpose of this project was to examine gender differences in the prediction of freshman grades after controlling for differential course grading based on college majors. This method involved deriving a variable that measured grading leniency using residual scores from the within‐gender regressions of freshman grades on high school grades and scores on the SAT for the non‐Latino white group. The procedure worked quite well and generalized to other groups not involved in the derivation of the grading‐leniency scale. Nevertheless, there were modest, sometimes statistically significant, gender differences in prediction that remained after this control variable was introduced into the regressions. The largest and smallest differences for females between actual grades and grades predicted from the males' regressions tended to be found in the African American and Asian American groups, respectively. The results imply that the use of information on college majors is a reasonable, practical procedure for controlling for grading leniency.
THE COLLEGE BOARD VOCABULARY STUDY
Authors: Breland Hunter M. , Jones Robert J. , Jenkins Laura , Paynter Marion , Pollack Judith , Fong Y. Fai 
Published: August 2014

ABSTRACTThis study was conducted to provide an up‐to‐date source of word frequency information based on the kinds of reading materials to which high school and first‐year college students are exposed. It began with a comprehensive listing of reading materials from curriculum surveys, state curriculum guides, private school reading lists, research surveys, federal reports, recommended reading lists, and other sources. Materials mentioned most often were sampled or entire documents were obtained when they were available in electronic form. Included in the sample of reading materials were American and British novels, poetry, drama, essays, biographies, autobiographies, current periodicals of various types, historical documents, and text from an encyclopedia.A corpus of 14,360,884 words of running text was assembled. This corpus was analyzed using the most sophisticated lexicographic methods available and the following statistics were generated: the overall frequency of occurrence of each word in the corpus, an index of dispersion for each word over 27 text categories, an estimate of the number of occurrences per one million words of running text for each word that would be expected in a similar but different corpus, and a standard frequency index developed from a logarithmic transformation.This report describes the development of the corpus and the computation of the word frequency indexes. It also compares the corpus with other existing corpora and demonstrates the importance of up‐to‐date word frequency information. The comprehensive listing of reading materials examined and a list of sampled materials are included in the Appendixes.
DELIMITING THE VERBAL DOMAIN
Authors: DeMauro Gerald E. , Merritt Ave , Adams Richard 
Published: August 2014

ABSTRACTA functional analysis was made of the verbal domain as it is defined by the test development process at Educational Testing Service. The thesis of the study is that the verbal domain consists of semantic relations that examinees must interpret. Test developers use a system of rules to sample stimulus materials from among infinite possibilities that elicit an interpretive response. Many of these rules are codified in a body of memoranda and corporate publications, and others are in different stages of formation, beginning in the use by individual test development professionals, and eventually becoming part of generally accepted practice. Because the process of test development is central to the formation of the verbal domain, a job analysis approach was used to examine how rules are employed to present the desired semantic relations to the examinees for their interpretation. This approach explains the process of delimiting the verbal domain through a comprehensive review of the uses of the rule system that defines when the items are functioning properly and when they are not. Test developers from two major divisions of ETS were observed and interviewed. They also rated the importance of each of the rules, suggested rules that they use in their work that were not listed, and delineated the professional activities for which these rules are most important. The study indicates that the rule system for sampling semantic relations is an evolving body, both for the individual test developer and for test development as an institution.
A PRELIMINARY STUDY OF THE EFFECTS OF WITHIN‐GROUP COVARIANCE STRUCTURE ON RECOVERY IN CLUSTER ANALYSIS
Authors: Donoghue John R. 
Published: August 2014

ABSTRACTTwo Monte Carlo studies investigated the effects of within‐group covariance structure on subgroup recovery by several widely used hierarchical clustering methods. Data sets were 100 bivariate observations from two subgroups, generated according to a finite normal mixture model. In Study 1, subgroup size, within‐group correlation, within‐group variance, and distance between subgroup centroids were manipulated. All clustering methods were strongly affected by within‐group correlation; negative correlation yielded much poorer recovery. Smaller effects were found for the interaction of clustering method with within‐group variance.Study 2 separated the effects of direction of correlation from the direction of differences in the subgroup centroids. Subgroup size, within‐group correlation, direction of the vector separating subgroup centroids, and distance between subgroup centroids were manipulated. Superior recovery was associated with within‐group correlation that matched the direction of subgroup separation. Overall, the EML algorithm of SAS yielded best recovery, followed closely by Ward's method, average linkage, and a version of the beta‐flexible algorithm, although several interactions were noted.The results are interpreted according to the weakness of the (squared) Euclidean distance as a measure of (dis)similarity for cluster analysis. Several alternative measures are discussed, and promising alternatives are identified for future investigation.
DEFINING THE CONTENT DOMAIN FOR THE PRAXIS II SUBJECT ASSESSMENT IN FRENCH: A JOB ANALYSIS FOCUSING ON KNOWLEDGE AND ABILITIES
Authors: Tannenbaum Richard J. 
Published: August 2014

ABSTRACTThe purpose of this study was to conduct a job analysis focusing on the knowledge and abilities important for beginning French teachers. The results of the job analysis will be used to define the content domain of the Subject Assessment in French for the Praxis Series: Professional Assessments for Beginning Teachers®.A domain of 212 knowledge statements and ability statements was developed by subject‐matter experts. These statements were then incorporated into a survey that was administered to a large number of French teachers, teacher educators, and state administrators. These professionals rated the importance of the knowledge statements and ability statements using a 5‐point scale.A cut point of a mean of 2.50 was established to differentiate important statements (M ≥ 2.50) and unimportant statements (M < 2.50). The results indicated that a total of 19 knowledge statements and ability statements failed to pass the cut point. It is recommended that these 19 statements not be included in the development of the Subject Assessment in French; emphasis, instead, should be put on the 193 statements that did pass the cut point.
DEFINING THE CONTENT DOMAIN FOR THE PRAXIS II SUBJECT ASSESSMENT IN BUSINESS EDUCATION: A JOB ANALYSIS FOCUSING ON KNOWLEDGE AND SKILLS
Authors: Tannenbaum Richard J. 
Published: August 2014

ABSTRACTThe purpose of this study was to conduct a job analysis of the knowledge and skills important for the competent performance of beginning business education teachers. The results of the job analysis will be used to define the content domain of the Subject Assessment in Business Education for the Praxis Series: Professional Assessments for Beginning Teachers®.A domain of 207 important knowledge statements and skill statements was developed by subject‐matter experts. These statements were then incorporated into a survey that was administered to a large number of business education teachers, administrators, and teacher educators. These business education professionals rated the importance of the knowledge statements and skill statements using a 5‐point scale. The survey component functioned as an independent verification or refutation of the importance of the statements defined by the subject‐matter experts. Statements that were judged to have a mean importance rating equal to or greater than 2.50 (midpoint between moderately important and important) by all groups (teachers, administrators, and teacher educators) and all subgroups (race or ethnicity, sex, geographic region, and teaching experience) of survey respondents were considered eligible for inclusion in the development of the Subject Assessment in Business Education.The results indicated that of the 207 knowledge statements and skill statements, 17 (8%) had a mean rating of less than 2.50 across all groups and subgroups of survey respondents. It is recommended that these 17 statements not be included in the development of the Subject Assessment in Business Education. Primary emphasis, in the test develop process, should be placed on the 190 knowledge or skill statements that did pass the 2.50 cut point.
AN ELECTRONIC INFRASTRUCTURE FOR A FUTURE GENERATION OF TESTS
Authors: Bennett Randy Elliot 
Published: August 2014

ABSTRACTETS is moving rapidly to computerize its tests for admissions to post‐secondary education and occupational licensure/certification. Computerized tests offer important advantages, including immediate score reporting, the convenience of testing when the examinee wishes, and for adaptive tests, equal accuracy throughout the score scale and a shorter test with no loss in measurement precision. There is much more that technology can achieve, however. This paper describes an electronic infrastructure for integrating the best of traditional testing approaches with new technology. This multi‐organizational infrastructure has the potential to help assessment contribute more positively to learning and decision making. It can do this by making it easier to deliver tests that employ performance tasks, include important skills not well‐measured by current examinations, sample behavior frequently over a student's school career, and give instructionally useful feedback to individuals.
EVALUATING A PROTOTYPE ESSAY SCORING PROCEDURE USING OFF‐THE‐SHELF SOFTWARE
Authors: Kaplan Randy M. , Burstein Jill , Trenholm Harriet , Lu Chi , Rock Donald , Kaplan Bruce , Wolff Susanne 
Published: August 2014

ABSTRACTConstructed‐response items, whose responses consist of words, phrases, sentences, paragraphs, and essays are among the most difficult and costly to score. The increased use of constructed‐response items like essays creates a need for tools to partially or fully automatically score these responses. This study explores one approach to analyzing essay‐length natural language constructed‐responses.In this study we develop and evaluate a decision model for scoring essays. The decision model uses off‐the‐shelf software for grammar and style checking of the English language. The first part of this study consisted of an evaluation of several commercial grammar checking programs. From this evaluation we select the best performing grammar checking programs to construct a decision model for scoring the essays. The second part of the study uses data produced from the selected grammar checking program(s) to make a decision about the score for an essay. Through statistical and linguistic methods, we analyze the performance of the decision model in an effort to understand its usefulness and practicality in a production scoring setting.
AN ANALYSIS OF THE DIMENSIONALITY OF TOEFL READING COMPREHENSION ITEMS
Authors: Schedl Mary , Gordon Ann , Carey Patricia A. , Tang K. Linda 
Published: August 2014

ABSTRACTThe issue of exactly what is measured by different types of reading items has been a matter of interest in the field of reading research for many years. Language teaching and testing specialists have raised the question of whether a reading test for foreign students wishing to enter university in the United States should include questions testing abilities beyond linguistic and very general discourse competencies, or indeed whether it is possible to separate these language competencies from other competencies. The purpose of this study was to investigate the dimensionality of the TOEFL® reading test, based on the specifications in use as of April 1991. Of particular interest was whether four item types identified in the test specifications as “reasoning items” could be shown to measure, in addition to general reading ability, any abilities not measured by the other item types in the TOEFL reading test. Two techniques, Stout's procedure and NOHARM analyses, were employed to investigate the hypothesized two‐factor model. In both cases the data failed to fit the model, indicating that TOEFL “reasoning items” cannot be shown to measure a unique construct. However, the follow‐up exploratory analyses indicated that all 10 test forms used in the study violated the assumption of essential unidimensionality, and all of the forms appeared to fit a two‐factor model where the second factor may be related to passage content or position.
THE ROLE OF PROBABILITY‐BASED INFERENCE IN AN INTELLIGENT TUTORING SYSTEM
Authors: Mislevy Robert J. , Gitomer Drew H. 
Published: August 2014

ABSTRACTProbability‐based inference in complex networks of interdependent variables is an active topic in statistical research, spurred by such diverse applications as forecasting, pedigree analysis, troubleshooting, and medical diagnosis. This paper concerns the role of Bayesian inference networks for updating student models in intelligent tutoring systems (ITSs). Basic concepts of the approach are briefly reviewed, but the emphasis is on the considerations that arise when one attempts to operationalize the abstract framework of probability‐based reasoning in a practical ITS context. The discussion revolves around HYDRIVE, an ITS for learning to troubleshoot an aircraft hydraulics system. HYDRIVE supports generalized claims about aspects of student proficiency through probability‐based combination of rule‐based evaluations of specific actions. The paper highlights the interplay among inferential issues, the psychology of learning in the domain, and the instructional approach upon which the ITS is based.
RELIABILITY OF ADVANCED PLACEMENT EXAMINATIONS
Authors: Bridgeman Brent , Morgan Rick , Wang Ming‐mei 
Published: August 2014

ABSTRACTThe various methods for computing the reliability of scores on Advanced Placement examinations are summarized. For the free response portion of the examinations, raters can contribute to score unreliability through both systematic severity errors (in which some raters consistently rate more severely than other raters) and through inconsistency. Inconsistency appears to be a much greater problem then systematic severity errors. Question to question variation (or score reliability) is seen as a greater problem than rater inconsistencies. The impact of increasing or decreasing the number of topics is demonstrated by showing the proportion of students correctly classified as the number of topics changes. Procedures to enhance both rater and score reliability are discussed.