bluuebunny/crossref_metadata_2025_split huggingface dataset:
THE DEVELOPMENT, INVESTIGATION, AND EVALUATION OF NEW ITEM TYPES FOR THE GRE ANALYTICAL MEASURE
Authors: Emmerich Walter , Enright Mary K. , Rock Donald A. , Tucker Carol 
Published: August 2014

ABSTRACTThe aim of this research was to identify, develop, and evaluate empirically new reasoning item types that might be used to broaden the analytical measure of the GRE General Test and to strengthen its construct validity. Item types were identified that varied in the aspects of reasoning they measure. Six item types were selected for empirical evaluation, including the two currently used in the GRE analytical measure. All of the experimental item types were developed in a 3‐option multiple‐choice format, and four of them also were developed in a multiple‐yes/no format. Two experimental batteries were assembled, one using the 3‐option format and the other using the multiple‐yes/no format. Two samples of approximately 370 examinees each, all of whom had recently taken the GRE General Test, were administered one or the other experimental battery. Item analyses and analyses of sex differences, criterion‐related validity, and relationships of the experimental item types to the current GRE measures were conducted. All but one of the experimental item types exhibited promise for strengthening the GRE analytical measure, and even the one exception appeared to be a possible item type for the GRE verbal measure. Evidence for interactions between item type and item format suggested that varying the format may result in the assessment of a different aspect of reasoning for some but not all reasoning item types. Different combinations of the experimental item types were evaluated in a series of confirmatory factor analyses, supplemented by correlational analyses and an exploratory factor analysis. Findings indicated that the convergent validity of the GRE analytical measure probably can be strengthened by selectively adding or substituting some of the experimental item types. However, such alterations of the GRE analytical measure probably would not improve the measure's discriminant validity. The study also provided evidence suggesting that the reasoning domain consists of two major subdomains: informal reasoning and formal‐deductive reasoning. This outcome has implications both for understanding the structure of the reasoning domain and for predicting the impact of different combinations of reasoning item types on the construct validity of the GRE analytical measure.
A GENERATIVE APPROACH TO PSYCHOLOGICAL AND EDUCATIONAL MEASUREMENT
Authors: Bejar Isaac I. 
Published: August 2014

ABSTRACTResponse generative modeling (RGM) is an approach to psychological measurement which involves a “grammar” capable of assigning a psychometric description to every item in a universe of items and is also capable of generating all the items in that universe. The purpose of this chapter is to: 1) elaborate on the rationale behind RGM; 2) review its roots and how it relates to current thinking on validity; and 3) assess its feasibility in a wide variety of domains. The chapter concludes with a brief review of possible theoretical approaches to a psychologically sound approach to test construction and modelling.
AN EXTENSION OF CART'S PRUNING ALGORITHM
Authors: Kim Sung‐Ho 
Published: August 2014

ABSTRACTAmong the computer‐based methods used for the construction of trees such as AID, THAID, CART and FACT, the only one that uses an algorithm that first grows a tree and then prunes the tree is CART. The pruning component of CART is analogous in spirit to the backward elimination approach in regression analysis. This idea provides a tool in controling the tree sizes to some extent and thus estimating the prediction error by the tree within a certain range of tree size. In the CART pruning process, Breiman, Friedman, Olshen, and Stone (1984) use a linear combination of the expected loss of the decisions by the tree and the total number of the terminal nodes of the tree. In this paper, CART's pruning is extended by considering a function of all the nodes of the tree in addition to the factors involved in the linear combination. For example, if we consider the cost of observing a variable at each node as is the main concern of this paper, or the structural complexity of the tree, we can see such an extention.
SEX DIFFERENCES IN SAT® PREDICTIONS OF COLLEGE GRADES
Authors: Stricker Lawrence J. , Rock Donald A. , Burton Nancy W. 
Published: August 2014

ABSTRACTThis study examined the role that sex‐related differences in the nature of the grade criterion and in variables associated with academic performance play in the over‐ and underprediction of college grades by the SAT when the test is used alone and in combination with high school grades. An entire freshman class at a large state university was studied. Women's GPA was underpredicted but only by a small amount. Adjusting the GPA for differences in grading standards for individual courses did not affect the underprediction, but controlling for sex differences in individual‐difference variables concerned with academic preparation, studiousness, and attitudes about mathematics reduced or essentially eliminated it.
AN EXPLORATION OF THE STABILITY OF FRESHMAN GPA, 1978–1985
Authors: Pomplun Mark , Burton Nancy , Lewis Charles 
Published: August 2014

This study is one of several meant to explain a decline in the predictive validity of the SAT and high school grades between 1975 and 1988, This particular study asks whether the criterion, freshman grade point average (FGPA), has changed. There may have been changes in the mix of academic competencies that students use in earning freshman grades. Possible changes in general reasoning abilities, technical skills, or subject knowledge were considered. College Board tests that are usually thought of as predictors were used in this study as proxies for the concurrent academic competencies that comprise FGPA. The academic components or factors, defined by various combinations of SAT, and TSWE items and scores, HSGPA, and Achievement Test scores, included verbal and mathematical reasoning ability; mathematics, language, and writing skills; and knowledge of science and history. Confirmatory factor analysis was used to test the consistency between 1978 and 1985 of the relationship of the academic factors to FGPA. The primary results indicate that FGPA has been stable from 1978 to 1985 in relation to the abilities, skills and subject knowledge measured by the SAT, TSWE, Achievement Tests, and HSGPA. No evidence was found to suggest that a change in the meaning of FGPA has contributed to the predictive validity decline.This study was designed as an initial exploratory analysis, to be followed if warranted by a study with better proxies for the academic components of FGPA and with the addition of noncognitive variables such as study habits and interest. However, the results were so stable over a number of different proxies, and over a number of different ways of combining proxies, that the researchers conclude that no follow‐up study is warranted.
BOOLEAN ALGEBRA APPLIED TO DETERMINATION OF UNIVERSAL SET OF KNOWLEDGE STATES
Authors: Tatsuoka Kikumi K. 
Published: August 2014

ABSTRACTincomplete knowledge in a domain. In this study, the foundations of a combinatorial algorithm that will provide the universal set of states of knowledge will be introduced. Each state of knowledge is represented by a list of “can/cannot” cognitive tasks and processes (called cognitively relevant attributes or latent variables) which are usually unobservable. A Boolean descriptive function will be introduced as a mapping between the attribute space spanned by latent attribute variables and the item response space spanned by item score variables. The Boolean descriptive function plays the role of uncovering the unobservable content of a black box. Once all the possible classes are retrieved explicitly and expressed by a set of ideal item response patterns which are described by a “can/cannot” list of latent attributes, the notion of bug distributions and statistical pattern classification techniques will enable us to diagnose students' states of knowledge accurately. Moreover, investigations on algebraic properties of these logically‐derived ideal‐response patterns will provide an insight into the structures of the test and dataset.Diagnosing cognitive errors possessed by examinees can be considered as a pattern classification problem which is designed to classify a sequential input of stimuli into one of several predetermined groups. The sequential inputs in our context are item responses and the predetermined groups are various states of knowledge resulting from misconceptions or different degrees of incomplete knowledge in a domain. In this study, the foundations of a combinatorial algorithm that will provide the universal set of states of knowledge will be introduced. Each state of knowledge is represented by a list of “can/cannot” cognitive tasks and processes (called cognitively relevant attributes or latent variables) which are usually unobservable. A Boolean descriptive function will be introduced as a mapping between the attribute space spanned by latent attribute variables and the item response space spanned by item score variables. The Boolean descriptive function plays the role of uncovering the unobservable content of a black box. Once all the possible classes are retrieved explicitly and expressed by a set of ideal item response patterns which are described by a “can/cannot” list of latent attributes, the notion of bug distributions and statistical pattern classification techniques will enable us to diagnose students' states of knowledge accurately. Moreover, investigations on algebraic properties of these logically‐derived‐ideal‐response patterns will provide an insight into the structures of the test and dataset.
SEX DIFFERENCES IN PERFORMANCE ON THE MATHEMATICS SECTION OF THE SCHOLASTIC APTITUDE TEST: A BIDIRECTIONAL VALIDITY STUDY1
Authors: Wainer Howard , Steinberg Linda S. 
Published: August 2014

ABSTRACTWe examine the average mathematics scores on the SAT for almost 12 thousand men and women within five specific types of first‐year math courses and at each grade level achieved in those courses. We find that after retrospective matching of both performance and course, women score about 35 SAT points lower than do their male counterparts. In prospective regression analyses we find somewhat larger sex effects in the same direction. We conclude that these findings provide evidence of sex differences in this measure of the validity of the SAT‐M.Although the data do not allow any conclusions about cause, we believe that exploratory analyses like these can point the way toward the construction of formal models that can more adequately describe the observed phenomena. Pending the availability of such models, we support recommendations made by Petersen and Novick (1976) regarding efficacious ways to remedy possible inequities.
A CONTINUITY PRINCIPLE FOR CALIBRATION OF SCORES WITHIN MASTERY ASSESSMENT SYSTEMS*
Authors: Tucker Ledyard R 
Published: August 2014

ABSTRACTA continuity principle is suggested for scaling of assessment scores from different levels of a multilevel mastery training program. In such training programs students are self‐paced and work at levels of tasks appropriate for their levels of performance. The problem addressed in this report concerns the scaling of assessment scores at different levels so as to form a single basic scale. The solution proposed here invloves relative scalings of the scores at different levels so that the mean converted score over all students forms a smooth, continuous curve. The smoothness of this mean curve is measured by higher order differences between consecutive scores. Computational procedures to obtain as smooth a curve as possible are presented.
EDUCATIONAL EFFECTIVENESS OF SESAME STREET: A REVIEW OF THE FIRST TWENTY YEARS OF RESEARCH* 1969 – 1989
Authors: Murphy Richard T. 
Published: August 2014

ABSTRACTIn this report, we review Sesame Street research studies that focused on the educational effectiveness of the series and provide a summary and synthesis of the results of the research. By educational effectiveness, we mean effectiveness in those areas related to beginning schooling, to the early years of children's formal education. An important goal of Sesame Street was to help children prepare for their initial school experience by exposing them to a variety of rich and appealing examples. Studies that attempt to provide evidence about the successes and/or failures of Sesame Street in that effort are the focus of this review.
SOME EMPIRICAL GUIDELINES FOR BUILDING TESTLETS1
Authors: Wainer Howard , Kaplan Bruce , Lewis Charles 
Published: August 2014

ABSTRACTA series of computer simulations were run to measure the relationship between testlet validity and the factors of item pool size and testlet length for both adaptive and linearly constructed testlets. We confirmed the generality of earlier empirical findings (Wainer, Lewis, Kaplan & Braswell, 1991) that making a testlet adaptive yields only marginal increases in aggregate validity because of the peakedness of the typical proficiency distribution.
HOW WELL CAN WE EQUATE TEST FORMS THAT ARE CONSTRUCTED BY EXAMINEES?1
Authors: Wainer Howard , Wang Xiang‐Bo , Thissen David 
Published: August 2014

ABSTRACTWhen an exam consists, in whole or in part, of constructed response items, it is a common practice to allow the examinee to choose among a variety of questions. This procedure is usually adopted so that the limited number of items that can be completed in the allotted time does not unfairly affect the examinee. This results in the de facto administration of several different test forms, where the exact structure of any particular form is determined by the examinee. When different forms are administered, a canon of good testing practice requires that those forms be equated to adjust for differences in their difficulty. When the items are chosen by the examinee traditional equating procedures do not strictly apply. In this paper we explore how one might equate within an IRT framework. We illustrate our procedure with data from the College Board's Advanced Placement Test in Chemistry.
AN EXPERIMENT IN THE APPLICATION OF AN AUTOMATED ITEM SELECTION METHOD TO REAL DATA*
Authors: Stocking Martha L. , Swanson Len , Pearlman Mari 
Published: August 2014

ABSTRACTA previously developed method of automatically selecting items for inclusion in a test subject to constraints on item content and statistical properties is applied to real data. Two tests are first assembled by experts in test construction who normally assemble such tests on a routine basis. Using the same pool of items and constraints articulated by test construction experts the same two tests are reassembled automatically. The manual and automatic assemblies are compared by test specialists who were not involved in the original manual assembly. Based on this experiment, barriers to future progress in the improvement of automatic test assembly methods seem not to be in the development of different algorithms, nor in the improvement of computer time and cost. Rather, the focus of future improvements in the process of automatic test assembly lies more in the direction of complete specifications of constraints on item selection and detailed coding of item properties.
INSTABILITY IN A TREE APPROACH TO REGRESSION
Authors: Kim Sung‐Ho 
Published: August 2014

ABSTRACTOne of the major problems that a tree‐approach to data analysis often encounters is instability of tree‐structures. Thus if one wishes to interprete the data structure by the tree‐approach, the instability issue must be dealt with.Examining instability at a node of a tree provides insight into the instability of the whole tree, since the same theory of instability applies to all the nodes. Thus, this paper deals with the instability issue at a single node of a tree.We assume that data are from a regression model, and examine what factors in that model affect the instability. Squared‐error loss is considered as a criterion for tree‐construction (“ls” criterion in CART program). The selection rate of a regressor variable at a node of a tree is used as a measure of instability. The selection rate mainly depends on (i) regression coefficients, (ii) (conditional) variance‐covariance structure of the regressor variables (given a subset of the regressor variables), (iii) the sample size, and (iv) noise in the response variable. We report simulation results that show patterns of instability for several different settings of regression models.
A PRELIMINARY STUDY OF THE NATURE OF COMMUNICATIVE COMPETENCE
Authors: Henning Grant , Cascallar Eduardo 
Published: August 2014

ABSTRACTSince the introduction of the expression “communicative competence” by Hymes in 1972, programs for the teaching and testing of English as a second or foreign language have increasingly been evaluated in terms either of their ability to promote communicative competence or of their sensitivity to a communicative view of language competence and performance. Unfortunately, despite this evident trend, there is no consensus on the nature of the communicative competence construct(s), nor has sufficient empirical support been provided for the various explanatory models proposed.The present study was conducted to survey the theoretical literature related to communicative competence; to identify major variables said to comprise the construct(s); to test for the comparative presence and measurability of such variables in typical native/nonnative speaker university academic communication; to propose a tentative model of communicative competence as‐ a synthesis of these variables; and to examine the relationship of Test of English as a Foreign Language (TOEFL), Test of Spoken English (TSE), and Test of Written English (TWE) scores with the various elements of the tentative model. Accordingly, a schema was devised for the systematic elicitation of academic communicative language performance and this schema was applied in the assessment of the 79 adult, English‐as‐a‐second language students who were chosen for the study. In all, 158 hours of video‐recorded language‐interaction episodes and 12 communicative writing samples were gathered by trained interlocutors, and observations on 40 communicative competence variables (21 in an oral modality, and 19 in a written modality, of which 18 in each modality are reported here) were rated by trained evaluators. Variables were examined for reliability and correlation with subtest and total scores of the TOEFL, TSE, and TWE examinations.Results provide information about the comparative contributions of some theory‐based communicative competence variables to domains of linguistic, discourse, sociolinguistic, and strategic competencies. In turn, these competency domains were investigated for their relation to components of language proficiency as assessed by the TOEFL, TSE, and TWE tests. Twelve oral and twelve written communication tasks were also analyzed and rank ordered for suitability in eliciting communicative language performance.
METHODS USED TO ESTABLISH SCORE COMPARABILITY ON THE ENHANCED ACT ASSESSMENT AND THE SAT®
Authors: Marco Gary L. , Abdel‐fattah A. A. , Baron Patricia A. 
Published: August 2014

ABSTRACTRecently Marco and Abdel‐fattah (1991) reported newly established relationships between scores on the enhanced American College Testing Program (ACT) Assessment and scores on the College Board's Scholastic Aptitude Test (SAT). The current report provides a detailed description of the methodology that was used to develop the “concordance” tables reported in that study.Fourteen large universities provided data on applicants who had taken both the enhanced ACT Assessment and the SAT. The ACT Composite scores and the SAT‐verbal and SAT‐mathematical (SAT‐V + M) scores used in the comparability study came from ACT Assessment test editions administered from October 1989 to June 1990 and SAT editions administered from March 1989 to June 1990. The total sample consisted of 40,492 students. A subsample of 40,051 students who look the enhanced ACT Assessment and the SAT no more than 217 days apart was also used in the comparability study.The equipercentile procedure, a curvilinear procedure, was used to scale the scores. Steps were taken to ensure the accuracy of the conversions by (1) weighting the scores to reduce or eliminate the effect of the time differential between ACT and SAT testings or (2) smoothing the score distributions before scaling. The score conversions derived by applying the weighting procedures provided score adjustments that were in the direction implied by the length of time between testings.
AN INVESTIGATION OF THE APPROPRIATENESS OF THE TOEFL TEST AS A MATCHING VARIABLE TO EQUATE TWE TOPICS
Authors: DeMauro Gerald E. 
Published: August 2014

ABSTRACTThe feasibility of using linear and equipercentile equating methods (Angoff, 1984) to equate forms of the Test of Written English (TWE) by using the TOEFL test as an anchor was explored. These two equating methods assume that either the TOEFL test and TWE test measure the same skills or that the examinee groups across TWE administrations are equivalent in skills. The differences between equated and observed scores (equating residuals) and differences among the mean equated scores for examinee groups were further examined in terms of characteristics of the TWE topics.An evaluation of the assumptions underlying the equating methods suggests that the TOEFL and TWE tests do not measure the same skills and that examinee groups are often dissimilar in skills. Therefore, use of the TOEFL test as an anchor to equate TWE tests does not appear appropriate. An alternative equating model based on expert judgment during pretest evaluation of potential essay prompts is recommended for future investigation.
A COMPARISON OF THE PREDICTIVE VALIDITY OF THE CURRENT SAT AND AN EXPERIMENTAL PROTOTYPE
Authors: Hale Gordon A. , Bridgeman Brent , Lewis Charles , Pollack Judith M. , Wang Ming‐mei 
Published: August 2014

ABSTRACTAs part of a large‐scale project to remodel the Scholastic Aptitude Test (SAT), this study examined the predictive validity of a prototype revised SAT, which incorporated many of the important structural changes planned for the test. This prototype was compared to a form of the current SAT with regard to several validity‐related issues. The results showed that the test revisions (a) tended to increase predictive validity for the verbal score, the mathematical score, and the composite of verbal and mathematical scores; (b) slightly increased the incremental validity of the test over high school rank; and (c) produced a modest reduction in gender‐related prediction differences for the verbal score. Also, inclusion of scores from a new writing test tended to increase validity and to reduce gender‐related prediction differences. This evidence is regarded as preliminary, as the data were collected under experimental conditions with a limited sample of colleges, and the prototype used here was not identical in form to the remodeled SAT as it will be implemented operationally. Tentatively, however, the results were consistent with the goals of the overall project in that, where the present revisions had effects, they tended to be in the desired direction.