bluuebunny/crossref_metadata_2025_split huggingface dataset:
A PRELIMINARY COMPARISON OF THE EFFECTIVENESS OF CLUSTER ANALYSIS WEIGHTING PROCEDURES FOR WITHIN‐GROUP COVARIANCE STRUCTURE
Authors: Donoghue John R. 
Published: August 2014

ABSTRACTA Monte Carlo study compared the usefulness of six variable weighting methods for cluster analysis. Datasets were 100 bivariate observations from two subgroups, generated according to a finite normal mixture model. Subgroup size, within‐group correlation, within‐group variance, and distance between subgroup centroids were manipulated.Of the clustering methods examined, the flexible average clustering algorithm with β = ‐. 15 or ‐.20 gave the best recovery. Of the remaining methods, Ward's method yielded the best recovery, followed closely by beta‐flexible linkage (β = ‐.50) and SAS's EML algorithm.In the absence of variable weights, negative within‐group correlation resulted in much poorer recovery for all clustering algorithms. The ACE weighting method of Art, Gnanadesikan, and Kettenring provided a net improvement in 17‐24% of the datasets when used with better clustering algorithms. When used with the same clustering alogrithms, De Soete's ultrametric weighting yielded improved recovery 16‐22% of the time. However, although ultrametric weighting was more sensitive than ACE to negative within‐subgroup correlation. Clustering based on principal components was less effective. Therefore, the ACE method is preferred overall.There is still room for improvement, however. Clustering with Mahalanobis distance based on the pooled within‐group covariance matrix indicated that knowing the correct covariance matrix would yield improved recovery (over ACE) approximately 10% of the time.
READER CALIBRATION AND ITS POTENTIAL ROLE IN EQUATING FOR THE TEST OF WRITTEN ENGLISH
Authors: Myford Carol M. , Marr Diana B. , Linacre J. Michael 
Published: August 2014

ABSTRACTWhen judges use a rating scale to rate performances, some judges may rate more severely than others, giving lower ratings. Judges may also differ in the consistency with which they apply rating criteria. In this study, we pilot tested a quality control procedure that provides a means for monitoring and adjusting for differences in reader performance. We employed FACETS, a Rasch‐based rating scale analysis procedure, to calibrate readers within and across two Test of Written English (TWE®) administrations. Our study had four goals: (1) to determine the extent to which individual readers can be considered interchangeable, both within and across TWE administrations; (2) to investigate reader characteristics and their relationships to the volume and quality of ratings; (3) to examine the effectiveness of third readings to adjudicate rating discrepancies; and (4) to make a preliminary determination of the feasibility of using FACETS Reader Severity Measures as a first step toward equating TWE scores across different topics and readers.
FACTORS IN PERFORMANCE ON BRIEF, IMPROMPTU ESSAY EXAMINATIONS
Authors: BRELAND HUNTER M. , BONNER MARILYN W. , KUBOTA MELVIN Y. 
Published: August 2014

ABSTRACTBrief, impromptu essays written for the 1990 administration of the College Board's English Composition Test (ECT) were randomly sampled for four different groups of examinees. These essays were subjected to further holistic ratings beyond those conducted for the ECT administration, and analytical ratings were also obtained. The holistic scores were correlated with the analytical scores to determine which essay characteristics were most closely associated with high holistic scores. The results indicated that overall organization, use of supporting materials, noteworthy ideas, rhetorical strategy, and thesis statement were the strongest correlates. Comparisons with the results of a previous study of 1979 ECT essays, written on a different topic, showed that these same characteristics were also most closely associated with high holistic scores in that study. These characteristics correlated strongly with high holistic scores for all subgroups randomly sampled, and for males and females.Additional factors associated with better holistic scores but studied only for the 1990 topic included following instructions, completion of the task, use of a standard essay format, use of literary references, and proficiency in the English language.Certain types of essay content were also associated with higher scores on the 1990 topic. Essays combining current affairs with literature and history or combining literature and history received slightly higher scores on average than essays based only on current affairs, literature, history, or personal experience. Nevertheless, some essays receiving high scores were based solely on personal experience, did not contain literary references, and did not use a standard essay format. The analyses suggest that some practice with this type of brief, impromptu essay, particularly under strict time constraints, may be useful as preparation for taking such essay examinations.
THE DIMENSIONALITY OF RESPONSES TO SAT ANALOGY ITEMS
Authors: Diones Ruth , Bejar Isaac I. , Chaffin Roger 
Published: August 2014

ABSTRACTThis study continued the research on analogy problem‐solving on psychometric tests pursued by Bejar, Chaffin and Embretson (1991). In specific, characteristics of a semantic taxonomy and a cognitively and empirically motivated intensional/pragmatic (I/P) dichotomy were explored. There were two research questions: (1) Could Bejar et al.'s results be replicated with SAT items? and (2) Would factor analyses support the bidimensional processing structure suggested by the I/P distinction? A specially constructed test of disclosed SAT analogies was administered to a group of 189 undergraduate students. Though factor analyses did not support the expected bidimensionality, a better understanding of both the semantic taxonomy and the I/P dichotomy was achieved. Suggestions for future work were given.
GENDER AND ETHNIC GROUP DIFFERENCES ON THE GMAT ANALYTICAL WRITING ASSESSMENT
Authors: Bridgeman Brent , McHale Frederick 
Published: August 2014

ABSTRACTGender and ethnic group differences on the Analytical Writing Assessment that is part of the Graduate Management Admissions Test were evaluated. Data from the first operational administration in October of 1994 were used. Standardized differences from the White male reference group were computed separately for men and women in four ethnic groups: White, Asian American, African American, and Hispanic/Latino. Within the White, African American, and Hispanic/Latino groups, women received higher scores than men on the Analytical Writing Assessment; in the Asian American group, men received higher writing scores, but the difference was not as great as on the Verbal score. Examinees whose best language was not English scored relatively higher on the Analytical Writing Assessment than on the Verbal measure. Simulations of eligibility for an admissions pool suggested that the addition of the Analytical Writing score would noticeably increase the number of the women in the pool, but would have virtually no impact on ethnic minorities. Rater and score reliability were reasonably consistent across ethnic and gender groups.
THE RELIABILITY OF DOCUMENT‐BASED ESSAY QUESTIONS ON ADVANCED PLACEMENT HISTORY EXAMINATIONS
Authors: Bridgeman Brent , Morgan Rick , Wang Ming‐mei 
Published: August 2014

ABSTRACTIf a student writes two essays, the score reliability can be estimated from the correlation between the essays. However, if the essays are in different modes or require different skills, the reliability may be underestimated from the correlation. In Advanced Placement history examinations, students wrote one standard essay and one essay that required analysis and synthesis of historical documents that were included with the question statement. If these document‐based questions (DBQ) were assessing substantially different skills from the standard essays, then the reliability of the DBQ scores would be underestimated from their correlation with a standard essay score. A sample of 1045 U. S. history students and 891 European history students participated in a special study in which they wrote essays for either 2 DBQ questions and one standard essay question or 3 standard essay questions. The DBQ correlated as highly with a standard essay as with another DBQ, suggesting that the simple correlation of the two types of scores did not underestimate the reliability of the essay scores.
ASSESSING UNUSUAL AGREEMENT BETWEEN THE INCORRECT ANSWERS OF TWO EXAMINEES USING THE K‐INDEX: STATISTICAL THEORY AND EMPIRICAL SUPPORT
Authors: Holland Paul W. 
Published: August 2014

ABSTRACTTest security and other concerns can lead to an interest in the assessment of how unusual it is for the answers of two different examinees to agree as much as they do. At Educational Testing Service, a measure called the K‐index is used to assess ‘unusual agreement’ between the incorrect answers of two examinees on a multiple‐choice test. Here, I describe the K‐index and give the results of an empirical study of some of the assumptions that underlie it and its use. The results of this study show that the K‐index can be expected to give a conservative estimate of the probability of chance agreement in the typical situations for which it is used, and that several important assumptions underlying the K‐index are supported by relevant data. In addition, the results presented here suggest a minor modification of the current (as of 1993) application of K‐index to part of the SAT to better insure that it is a conservative measure of chance agreement.
GENERATIVE RESPONSE MODELING: LEVERAGING THE COMPUTER AS A TEST DELIVERY MEDIUM
Authors: Bejar Isaac I. 
Published: August 2014

ABSTRACTGenerative response modeling is an approach to test development and response modeling that calls for the creation of items in such a way that the parameters of the items on some response model can be anticipated through knowledge of the psychological processes and knowledge required to respond to the item. That is, the computer would not merely retrieve an item from a database, as is the case in adaptive testing, but wouldcompose it, or assist in doing so, according to desired specifications. This approach to assessment has implications for both the economics and validity of computer administered tests. To illustrate the concept, a system for measuring writing skills will be outlined where the examinee is expected to rewrite sentences, rather than just recognize errors in a sentence, using a multiple choice format. The possibility of estimating the psychometric parameters of items based on a psychological analysis of the response process will then be examined and shown to be feasible. Such estimates are less precise than estimates based on large samples of test takers. A Monte Carlo study is presented to investigate the possibility of compensating for that imprecision when estimating ability or proficiency. The paper concludes that a generative approach is feasible, and can be a mechanism for taking advantage of the considerable investment required for computer‐based testing.
DESCRIBING COMPLEX ACADEMIC TASKS FROM SIX GRADUATE DISCIPLINES USING MULTIDIMENSIONAL SCALING AND CLUSTERING
Authors: Enright Mary K. , Oltman Philip K. 
Published: August 2014

ABSTRACTThe problem of defining the nature and variety of academic tasks is growing in importance as more complex assessment tasks are introduced in many educational contexts. In this study, multidimensional scaling and cluster analysis were used to describe and categorize tasks from six graduate disciplines including academic psychology, applied psychology, English literature, journalism, physics, and electrical engineering. A sample of task descriptions was constructed through interviews with graduate students from these disciplines. A rating instrument was designed to describe task goals and to evaluate whether the tasks were well or ill structured with respect to various aspects of problem definition and problem solution. Graduate faculty used the rating instrument to characterize a sample of tasks from their discipline.The scales were found reasonably reliable and were useful in identifying and describing task clusters and how such clusters varied both within and across disciplines. A cluster of short‐term problems that were posed by someone other than the student was found in every field although the other characteristics of this cluster of tasks varied with discipline. For example, the short‐term tasks in engineering and physics were well‐structured, requiring the application of established principles, and having objective standards for judging performance. In contrast, a cluster of short‐term tasks in English literature were very ill‐structured because different conceptual approaches could be relevant, there were alternative methods for accomplishing the tasks, many possible solutions existed, and the student had to define an issue or question to consider. In all disciplines except physics, a cluster of complex tasks emerged that was characterized as having multiple objectives that needed to be satisfied. The cluster of complex tasks that was found in physics was not described clearly by the scales. Problem‐finding was an important task characteristic in the social sciences and humanities but not in the physical sciences. The relevance of multidimensional scaling and clustering to test design is discussed.
AN INVESTIGATION OF PROBLEM SOLVING TASKS IN DIFFERENT GRADUATE DISCIPLINES
Authors: Enright Mary K. , Norback Judith Shaul 
Published: August 2014

ABSTRACTOne way to contextualize reasoning is to create assessment problems that embody critical features of the problems typical of a discipline. Problem characteristics are an important determinant of problem solving and reasoning processes. Critical problem characteristics are likely to vary among disciplines in relation to disciplinary differences in content, structure, and epistemology. However, no systematic or comprehensive description of problem categories and characteristics presently exists. A more abstract description of problems would contribute to the development of assessment problems that better match tasks performed in graduate school. It would also provide a basis for grouping disciplines in terms of critical problem characteristics so that assessment problems appropriate for broad areas rather than specific disciplines can be developed. The goal of this research was to develop a broad overview of problem solving tasks typically encountered in graduate education as well as a conceptual framework for classifying these tasks and their attributes. An overview of problem solving tasks was obtained through interviews with graduate students from six diverse disciplines and the collection of examples of problem solving tasks encountered in different contexts. A preliminary scheme for classifying these tasks was developed and provided a basis for rating scales to identify critical features of the tasks. Graduate faculty were interviewed about the appropriateness of the rating scales for characterizing problem solving tasks within their disciplines and about important student qualities. The implications of this study for developing new forms of assessment are discussed.
EFFECTS OF APPLYING DIFFERENT TIME LIMITS TO A PROPOSED GRE WRITING TEST
Authors: Powers Donald E. , Fowles Mary E. 
Published: August 2014

ABSTRACTTo determine the role of time limits on both test performance and test validity, we asked approximately 300 prospective graduate students to each write two essays — one in a 40‐minute time period and the other in 60 minutes. Analyses revealed that, on average, test performance was significantly better when examinees were given 60 minutes instead of 40. However, there was no interaction between test‐taking style (fast vs. slow) and time limits. That is, examinees who described themselves as slow writers/test takers did not benefit any more (or any less) from generous time limits than did their quicker counterparts. In addition, there was no detectable effect of different time limits on the meaning of essay scores, as suggested by their relationship to several nontest indicators of writing ability.
INTELLIGENT TUTORING AND ASSESSMENT BUILT ON AN UNDERSTANDING OF A TECHNICAL PROBLEM‐SOLVING TASK
Authors: Steinberg Linda S. , Gitomer Drew H. 
Published: August 2014

ABSTRACTIn this paper we describe Hydrive, an operational computer‐based intelligent tutoring system built to help Air Force technicians develop generalizable skills for troubleshooting hydraulics systems. We use Hydrive as an example of how effective training and assessment is developed from a coherent understanding of a target task and how this understanding can be consistently represented in all aspects of a training/assessment instrument. We show how an organizing principle of expert behavior, active path analysis, is used to inform the design and content of the tutor's system, student and instructional models and the design of the tutor's interface. We conclude with information about Hydrive's field trial evaluation and some thoughts on knowledge acquisition and generalizability.
MEASURING SELF‐CONCEPT AND RELATING IT TO ACADEMIC ACHIEVEMENT: STATISTICAL ANALYSES OF THE MARSH SELF‐DESCRIPTION QUESTIONNAIRE
Authors: Wenglinsky Harold 
Published: August 2014

ABSTRACTEducational intervention programs that seek to raise student self‐concept as a lever for raising student achievement are hampered both by limitations in measurement tools and a lack of knowledge about the relationship between self‐concept and student achievement. Advances in self‐concept research indicate that student self‐concept includes many domains corresponding to the various endeavors of adolescents, from relations with peers to physical abilities. The most state‐of‐the art instrument for adolescents, which takes the multiple domains of self‐concept into account, is the Marsh Self‐Description Questionnaire II (SDQ II). Yet the instrument is lengthy, containing 102 questions including some that are regarded by teachers as insensitive to the feelings of their students, creating problems in administering the instrument. Further, little research has been done on the relationship between specific domains of self‐concept and academic achievement; most researchers tend to claim either that all domains of self‐concept affect academic achievement, or that self‐concept is entirely a by‐product of previous student performance, and therefore does not affect it at all. This study addresses these two issues. First, it applies confirmatory factor analysis to a database constructed by the author from an administration of the 102‐item SDQ II to a sample of 1,130 students participating in Outward Bound, USA programs, finding that the instrument can be reduced to thirty items with little loss of reliability. Second, it applies regression analysis to the National Educational Longitudinal Study (NELS), which contains items from the SDQ II, to measure the relationship between each self‐concept domain and achievement in reading and mathematics. This analysis finds that the effect of self‐concept on achievement is domain specific; academic domains of self‐concept affect academic achievement, while non‐academic domains do not. The study concludes with some suggestions for how intervention programs such as Outward Bound, USA can measure self‐concept and use increases in specific types of self‐concept as a lever for raising student achievement.
SURVEY OF STANDARDS FOR FOREIGN STUDENT APPLICANTS
Authors: Boldt R. F. , Courtney R. G. 
Published: August 2014

ABSTRACTThis project consisted of a survey of practices and standards in the use of the Test of English as a Foreign Language (TOEFL®) and attempted to learn the processes by which these standards were developed. By “standards” we mean the information and requirements established by institutions to evaluate applicants' English proficiency. Information, including other measures that were used with the TOEFL test, was obtained toward that end.Several findings emerged. The most common practice for using the TOEFL test in admissions was that students scoring above a specified score were judged adequate from the standpoint of English proficiency, but students who scored below it had to provide other evidence of English proficiency that could justify admission. Only a few institutions used a rigid cut score. The measures most commonly allowed as substitutes for the TOEFL test in admissions and placement decisions were the Scholastic Assessment Test (SAT®) (The College Board, 1995), the American College Testing Assessment (ACT™) (ACT, 1988) score, and the Michigan Test of English Language Proficiency (MTELP) (Buros, 1965). It was found that most changes in standards consisted of increases in the minimum test scores, and that they were initiated in response to problems noted in admitted foreign students' performance. In addition to performance, practices of similar institutions were commonly used to set minimum scores. Participants most often mentioned in revised standard setting were administrators (policy), ESL staff (language trainers), and faculty (teachers of academic content).This survey provides information about the standards for different institutions and how those standards have been set and maintained. Users had the opportunity to tell us about their use of the TOEFL test. The following suggestions emerged:• When score user surveys are conducted, relevant institutional characteristics could be collected and used to categorize the results. Such information would facilitate the process of adjusting qualifying scores.• Data that connects TOEFL scores with speaking and essay scores could be made available. Such data could be useful when essay or speaking measures are used to supplement processing of applicants who achieve marginal scores on TOEFL.• A system of behaviorally anchored language proficiency scales based on incidents that have stimulated changes in cut scores could be developed. Availability of such scales would facilitate cross‐institutional communication about language proficiency levels and could add a new dimension to test validation.
VALIDATION AND EXTENSION OF THE MATHEMATICAL EXPRESSION RESPONSE TYPE: APPLICATIONS OF SCHEMA THEORY TO AUTOMATIC SCORING AND ITEM GENERATION IN MATHEMATICS
Authors: Singley Mark Kevin , Bennett Randy Elliot 
Published: August 2014

ABSTRACTIn this report, we describe the development of general, accurate, cost‐effective, and immediately usable automatic analysis routines that dichotomously score rational expressions of arbitrary complexity. This development has led to the new Mathematical Expression (ME) response type, which may appear operationally in the new GRE Mathematical Reasoning test scheduled for deployment in 1999. We detail work confirming the accuracy of these scoring routines and then describe extensions to the existing capability based on schema theory. We show how schema theory can be applied not only to the scoring of more complex test responses but also to the automatic generation of items and intelligent tutoring.
MODELING THE PRODUCTION FUNCTION: ASSOCIATIONS AMONG SCHOOL DISTRICT EXPENDITURES, SCHOOL RESOURCES AND STUDENT ACHIEVEMENT
Authors: Wenglinsky Harold 
Published: August 2014

ABSTRACTThis study examines the relationships among school district expenditures, school resources and student achievement for a nationally representative sample of fourth graders. The results of prior studies of these relationships, usually taking a production function approach, have been inconclusive, some finding expenditures and resources to be strongly related to student achievement and others finding no such relationship; because of this, some researchers have proposed alternatives to production function research. The current study, however, seeks to salvage the production function approach through addressing its shortcomings. It applies structural equation modeling to a database that is national in scope and distinguishes between different types of expenditure. It finds that some expenditures and resources, such as instructional spending and teacher‐student ratios, are indeed associated with student achievement, while others are not.