bluuebunny/crossref_metadata_2025 huggingface dataset:
APPRAISING THE COGNITIVE FEATURES OF SUBJECT TESTS
Authors: Emmerich Walter
Published: August 2014

The study's aim was to develop a procedure that could be used to appraise the cognitive features of subject (achievement) tests. Cognitive taxonomies and an accompanying coding scheme were developed and applied to the GRE Subject Tests in Psychology and Literature in English. The cognitive profiles of these two tests were found to be strikingly different, indicating that the taxonomies were operating as intended. Ways of improving and extending the procedure are discussed.
A COMPARISON OF MULTIPLE‐CHOICE AND CONSTRUCTED FIGURAL RESPONSE ITEMS
Authors: Martinez Michael E.
Published: August 2014

In contrast to multiple‐choice test questions, figural response items call for constructed responses and rely upon figural material, such as illustrations and graphs, as the response medium. Figural response questions in various science domains were created and administered to a sample of 4th‐, 8th‐, and 12th‐grade students. Item and test statistics from parallel sets of figural response and multiple‐choice questions were compared. Figural response items were generally more difficult, especially for questions that were difficult (p<.5) in their constructed‐response forms. Figural response questions were also slightly more discriminating and reliable than their multiple‐choice counterparts, but had higher omit rates. The paper addresses the relevance of guessing to figural response items and the diagnostic value of the item type. Plans for future research on figural response items are discussed.
THE THREE STAGE DEVELOPMENT OF THE NTE SCHOOL PSYCHOLOGIST SPECIALTY AREA TEST FROM A JOB ANALYSIS PERSPECTIVE
Authors: DeMauro Gerald E.
Published: August 2014

The utilization of survey data and expert judgment in test development is described in the three stages of developing the NTE School Psychologist Specialty Area Test. The first stage is described in the paper entitled “Knowledge Areas Important to School Psychology.” A survey of the membership of the National Association of School Psychologists helped determine knowledge areas and job activities important to the professional functions of an entry‐level school psychologist. Expert committee judgments were then used to link these knowledge areas to job domains. The second stage, described in the paper entitled “Specifying the Content Domain of the NTE School Psychologist Specialty Area Test” reviews how the findings of the first stage were used by the Committee of Examiners to establish content specifications and the representation of component knowledge areas for the test. The findings were also used to establish a job context for each test question. Finally, the “Follow Up: Specifying the Content and Context of NTE School Psychologist Items for Item Writers” documents how the directions of the Committee of Examiners were transmitted to item writers. The three papers, taken together, provide a step by step process of item development based on expert judgment.
SEX DIFFERENCES IN THE PERFORMANCE OF HIGH‐SCORING EXAMINEES ON THE SAT®‐M
Authors: Gallagher Ann M.
Published: August 2014

Performance of high‐scoring males and females on the mathematics section of three forms of the College Board's Scholastic Aptitude Test (SAT‐M) was examined to determine how item content, solution strategy, and speededness differentially affect performance. The mathematical and verbal sections of the SAT were also compared for similarities in the performance patterns of high scorers.

Items on three forms of the SAT‐M were classified on the basis of content and potential solution strategies. Male and female performance in the top 5 percent was examined using the Mantel‐Haenszel Differential Item Functioning (MH‐DIF) procedure. A main effect was found for item‐solution strategy, but not for item content. Males tended to outperform females on items requiring logical estimation or insight even when subjects were matched on years of mathematics course taking and average mathematics grades.

Conventional measures indicated that the test was not differentially speeded. However, omit rates indicated that females omitted a greater proportion of items requiring estimation. Different patterns by gender were found on the mathematical and verbal sections of the test.
EXPERIMENTAL STUDY OF THE EFFECTS OF CALCULATOR USE ON THE ADVANCED PLACEMENT CALCULUS EXAMINATIONS1
Authors: Morgan Rick , Stevens Joe
Published: August 2014

Advanced Placement Calculus examinations were administered to nearly 7,000 students from more than 400 high schools in order to determine the impact of calculator use. Both the Calculus AB and Calculus BC experimental examinations had two sections. Section I items were designed so that a calculator was not needed to arrive at the correct solution. Approximately half of the students were permitted to use calculators on this section. Section II items were developed to require a calculator to arrive at the correct solution. All students were permitted calculator use for these items. Analyses indicated that calculator use resulted in a substantial improvement in performance for 3 of the 40 Section I items. Those with calculators capable of graphics or the taking of derivatives performed better on both sections of the exams than those with scientific calculators. The internal consistency reliability of the Section I items was higher for students not permitted calculator use. A pattern of negative D‐DIF values for the Section II items indicated that these items were differentially more difficult for females than the Section I items. The D‐DIF value of only one Section I item appeared to be substantially impacted by calculator use. Total test scores were found to be higher for those permitted to use calculators, for males in comparison to females, and for those reporting the use of advanced calculators in comparison to those reporting the use of scientific calculators.
DEVELOPMENT OF PROCEDURES FOR RESOLVING IRREGULARITIES IN THE ADMINISTRATION OF THE LISTENING COMPREHENSION SECTION OF THE TOEFL TEST
Authors: Way Walter D. , McKinley Robert L.
Published: August 2014

The purpose of this study was to develop and evaluate two procedures to be used in determining whether examinees in a given test center are affected by a testing irregularity on the Listening Comprehension section of the Test of English as a Foreign Language (TOEFL). One approach employs analysis of covariance (ANCOVA) on Listening Comprehension (Section 1) means using scores on Structure and Written Expression (Section 2) and scores on Reading and Vocabulary (Section 3) as covariates. The second procedure entails a Bayesian approach that uses prior information collected about performance at the center in question. Analyses using these two procedures were carried out using both simulated data and data from actual testing irregularities.

The results of this study support the following conclusions. First, both the ANCOVA and Bayesian procedures appear to provide useful information related to the effects of testing irregularities on Section 1 of the TOEFL test; therefore, both procedures should be incorporated into the operational procedures for resolving testing irregularities. Second, the two procedures will usually agree about the effects of testing irregularities, although differences between the procedures may occur in situations where the Bayesian procedure indicates an effect of one scaled score point and the ANCOVA procedure does not indicate a statistically significant difference between the irregularity and comparison groups at some specified level. Finally, results based on the two procedures should be interpreted with caution, particularly in situations where the two procedures produce discrepant results.
AUTOMATED ITEM SELECTION USING ITEM RESPONSE THEORY*
Authors: Stocking Martha L. , Swanson Len , Pearlman Mari
Published: August 2014

This paper presents a new heuristic approach to interactive test assembly that is called the successive item replacement algorithm. This approach builds on the work of van der Linden (1987a) and van der Linden and Boekkooi‐Timminga (1989) in which methods of mathematical optimization are combined with Item Response Theory (IRT) to construct tests from larger collections or pools of items. This new approach is contrasted with two more formal models as well as other heuristic approaches that appear in the literature. An experiment using quasi‐realistic data is performed that serves to illustrate the differences in approaches for typical practical test construction applications.
AGREEMENT BETWEEN EXPERT‐SYSTEM AND HUMAN RATERS' SCORES ON COMPLEX CONSTRUCTED‐RESPONSE QUANTITATIVE ITEMS
Authors: Sebrechts Marc M. , Bennett Randy Elliot , Rock Donald A.
Published: August 2014

This study evaluated agreement between expert‐system and human scores on 12 constructed‐response algebra word problems taken by Graduate Record Examination General Test examinees. Problems were drawn from three content classes (rate × time, work, and interest) and presented in four constructed‐response formats (open ended, goal specification, equation setup, and faulty solution). Agreement was evaluated for each item separately by comparing the system's scores to the mean scores taken across five content experts. The expert system produced scores for all responses and duplicated the judgments of raters with reasonable accuracy; the median of 12 correlations between the system and human scores was .88, and the largest average discrepancy was 1.2 on a 16‐point scale. No obvious differences in scoring agreement between constructed‐response formats or content classes emerged. Ideas are discussed for using expert scoring systems in large‐scale assessment programs and in interactive diagnostic assessment.
EFFECTS OF GRADUATE COURSEWORK ON THE GRE QUANTITATIVE SCORE FOR RECENT AND NONRECENT COLLEGE GRADUATES
Authors: Hale Gordon A. , Rock Donald A. , Longford Nicholas T.
Published: August 2014

Previous research has shown that the mean quantitative score on the Graduate Record Examinations (GRE) General Test or GRE quantitative score, declines with age of the examinee, while the mean GRE verbal score remains relatively constant. It is assumed that the age‐related decline in quantitative score is due, at least in part, to an increase in time away from formal academic work. A hypothesis that follows from this observation is that taking formal quantitative coursework, such as that provided in graduate school, should cause nonrecent graduates' mean GRE quantitative score to rebound to a level that is closer to that of recent graduates.

To test this hypothesis, recent and nonrecent college graduates who were currently enrolled in graduate programs in the social sciences, for whom preadmission GRE scores were available, and who had some quantitative coursework in those programs, were identified. Recent graduates were those who had graduated one year or less before taking the GRE, and nonrecent graduates were those who had graduated (or last received postgraduate schooling) seven years or more before taking the GRE. All students were administered an abbreviated General Test.

The results failed to support the hypothesis; the difference between 264 recent and 66 nonrecent graduates' quantitative performance on the second (abbreviated) GRE test was just as pronounced, relative to scale, as was the difference on the first (preadmission) GRE test. Possible reasons for these results are discussed.
THE DEVELOPMENT, INVESTIGATION, AND EVALUATION OF NEW ITEM TYPES FOR THE GRE ANALYTICAL MEASURE
Authors: Emmerich Walter , Enright Mary K. , Rock Donald A. , Tucker Carol
Published: August 2014

The aim of this research was to identify, develop, and evaluate empirically new reasoning item types that might be used to broaden the analytical measure of the GRE General Test and to strengthen its construct validity. Item types were identified that varied in the aspects of reasoning they measure. Six item types were selected for empirical evaluation, including the two currently used in the GRE analytical measure. All of the experimental item types were developed in a 3‐option multiple‐choice format, and four of them also were developed in a multiple‐yes/no format. Two experimental batteries were assembled, one using the 3‐option format and the other using the multiple‐yes/no format. Two samples of approximately 370 examinees each, all of whom had recently taken the GRE General Test, were administered one or the other experimental battery. Item analyses and analyses of sex differences, criterion‐related validity, and relationships of the experimental item types to the current GRE measures were conducted. All but one of the experimental item types exhibited promise for strengthening the GRE analytical measure, and even the one exception appeared to be a possible item type for the GRE verbal measure. Evidence for interactions between item type and item format suggested that varying the format may result in the assessment of a different aspect of reasoning for some but not all reasoning item types. Different combinations of the experimental item types were evaluated in a series of confirmatory factor analyses, supplemented by correlational analyses and an exploratory factor analysis. Findings indicated that the convergent validity of the GRE analytical measure probably can be strengthened by selectively adding or substituting some of the experimental item types. However, such alterations of the GRE analytical measure probably would not improve the measure's discriminant validity. The study also provided evidence suggesting that the reasoning domain consists of two major subdomains: informal reasoning and formal‐deductive reasoning. This outcome has implications both for understanding the structure of the reasoning domain and for predicting the impact of different combinations of reasoning item types on the construct validity of the GRE analytical measure.
A GENERATIVE APPROACH TO PSYCHOLOGICAL AND EDUCATIONAL MEASUREMENT
Authors: Bejar Isaac I.
Published: August 2014

Response generative modeling (RGM) is an approach to psychological measurement which involves a “grammar” capable of assigning a psychometric description to every item in a universe of items and is also capable of generating all the items in that universe. The purpose of this chapter is to: 1) elaborate on the rationale behind RGM; 2) review its roots and how it relates to current thinking on validity; and 3) assess its feasibility in a wide variety of domains. The chapter concludes with a brief review of possible theoretical approaches to a psychologically sound approach to test construction and modelling.
AN EXTENSION OF CART'S PRUNING ALGORITHM
Authors: Kim Sung‐Ho
Published: August 2014

Among the computer‐based methods used for the construction of trees such as AID, THAID, CART and FACT, the only one that uses an algorithm that first grows a tree and then prunes the tree is CART. The pruning component of CART is analogous in spirit to the backward elimination approach in regression analysis. This idea provides a tool in controling the tree sizes to some extent and thus estimating the prediction error by the tree within a certain range of tree size. In the CART pruning process, Breiman, Friedman, Olshen, and Stone (1984) use a linear combination of the expected loss of the decisions by the tree and the total number of the terminal nodes of the tree. In this paper, CART's pruning is extended by considering a function of all the nodes of the tree in addition to the factors involved in the linear combination. For example, if we consider the cost of observing a variable at each node as is the main concern of this paper, or the structural complexity of the tree, we can see such an extention.
SEX DIFFERENCES IN SAT® PREDICTIONS OF COLLEGE GRADES
Authors: Stricker Lawrence J. , Rock Donald A. , Burton Nancy W.
Published: August 2014

This study examined the role that sex‐related differences in the nature of the grade criterion and in variables associated with academic performance play in the over‐ and underprediction of college grades by the SAT when the test is used alone and in combination with high school grades. An entire freshman class at a large state university was studied. Women's GPA was underpredicted but only by a small amount. Adjusting the GPA for differences in grading standards for individual courses did not affect the underprediction, but controlling for sex differences in individual‐difference variables concerned with academic preparation, studiousness, and attitudes about mathematics reduced or essentially eliminated it.
AN EXPLORATION OF THE STABILITY OF FRESHMAN GPA, 1978–1985
Authors: Pomplun Mark , Burton Nancy , Lewis Charles
Published: August 2014

This study is one of several meant to explain a decline in the predictive validity of the SAT and high school grades between 1975 and 1988, This particular study asks whether the criterion, freshman grade point average (FGPA), has changed. There may have been changes in the mix of academic competencies that students use in earning freshman grades. Possible changes in general reasoning abilities, technical skills, or subject knowledge were considered. College Board tests that are usually thought of as predictors were used in this study as proxies for the concurrent academic competencies that comprise FGPA. The academic components or factors, defined by various combinations of SAT, and TSWE items and scores, HSGPA, and Achievement Test scores, included verbal and mathematical reasoning ability; mathematics, language, and writing skills; and knowledge of science and history. Confirmatory factor analysis was used to test the consistency between 1978 and 1985 of the relationship of the academic factors to FGPA. The primary results indicate that FGPA has been stable from 1978 to 1985 in relation to the abilities, skills and subject knowledge measured by the SAT, TSWE, Achievement Tests, and HSGPA. No evidence was found to suggest that a change in the meaning of FGPA has contributed to the predictive validity decline.

This study was designed as an initial exploratory analysis, to be followed if warranted by a study with better proxies for the academic components of FGPA and with the addition of noncognitive variables such as study habits and interest. However, the results were so stable over a number of different proxies, and over a number of different ways of combining proxies, that the researchers conclude that no follow‐up study is warranted.
BOOLEAN ALGEBRA APPLIED TO DETERMINATION OF UNIVERSAL SET OF KNOWLEDGE STATES
Authors: Tatsuoka Kikumi K.
Published: August 2014

incomplete knowledge in a domain. In this study, the foundations of a combinatorial algorithm that will provide the universal set of states of knowledge will be introduced. Each state of knowledge is represented by a list of “can/cannot” cognitive tasks and processes (called cognitively relevant attributes or latent variables) which are usually unobservable. A Boolean descriptive function will be introduced as a mapping between the attribute space spanned by latent attribute variables and the item response space spanned by item score variables. The Boolean descriptive function plays the role of uncovering the unobservable content of a black box. Once all the possible classes are retrieved explicitly and expressed by a set of ideal item response patterns which are described by a “can/cannot” list of latent attributes, the notion of bug distributions and statistical pattern classification techniques will enable us to diagnose students' states of knowledge accurately. Moreover, investigations on algebraic properties of these logically‐derived ideal‐response patterns will provide an insight into the structures of the test and dataset.

Diagnosing cognitive errors possessed by examinees can be considered as a pattern classification problem which is designed to classify a sequential input of stimuli into one of several predetermined groups. The sequential inputs in our context are item responses and the predetermined groups are various states of knowledge resulting from misconceptions or different degrees of incomplete knowledge in a domain. In this study, the foundations of a combinatorial algorithm that will provide the universal set of states of knowledge will be introduced. Each state of knowledge is represented by a list of “can/cannot” cognitive tasks and processes (called cognitively relevant attributes or latent variables) which are usually unobservable. A Boolean descriptive function will be introduced as a mapping between the attribute space spanned by latent attribute variables and the item response space spanned by item score variables. The Boolean descriptive function plays the role of uncovering the unobservable content of a black box. Once all the possible classes are retrieved explicitly and expressed by a set of ideal item response patterns which are described by a “can/cannot” list of latent attributes, the notion of bug distributions and statistical pattern classification techniques will enable us to diagnose students' states of knowledge accurately. Moreover, investigations on algebraic properties of these logically‐derived‐ideal‐response patterns will provide an insight into the structures of the test and dataset.
SEX DIFFERENCES IN PERFORMANCE ON THE MATHEMATICS SECTION OF THE SCHOLASTIC APTITUDE TEST: A BIDIRECTIONAL VALIDITY STUDY1
Authors: Wainer Howard , Steinberg Linda S.
Published: August 2014

We examine the average mathematics scores on the SAT for almost 12 thousand men and women within five specific types of first‐year math courses and at each grade level achieved in those courses. We find that after retrospective matching of both performance and course, women score about 35 SAT points lower than do their male counterparts. In prospective regression analyses we find somewhat larger sex effects in the same direction. We conclude that these findings provide evidence of sex differences in this measure of the validity of the SAT‐M.

Although the data do not allow any conclusions about cause, we believe that exploratory analyses like these can point the way toward the construction of formal models that can more adequately describe the observed phenomena. Pending the availability of such models, we support recommendations made by Petersen and Novick (1976) regarding efficacious ways to remedy possible inequities.
A CONTINUITY PRINCIPLE FOR CALIBRATION OF SCORES WITHIN MASTERY ASSESSMENT SYSTEMS*
Authors: Tucker Ledyard R
Published: August 2014

A continuity principle is suggested for scaling of assessment scores from different levels of a multilevel mastery training program. In such training programs students are self‐paced and work at levels of tasks appropriate for their levels of performance. The problem addressed in this report concerns the scaling of assessment scores at different levels so as to form a single basic scale. The solution proposed here invloves relative scalings of the scores at different levels so that the mean converted score over all students forms a smooth, continuous curve. The smoothness of this mean curve is measured by higher order differences between consecutive scores. Computational procedures to obtain as smooth a curve as possible are presented.
EDUCATIONAL EFFECTIVENESS OF SESAME STREET: A REVIEW OF THE FIRST TWENTY YEARS OF RESEARCH* 1969 – 1989
Authors: Murphy Richard T.
Published: August 2014

In this report, we review Sesame Street research studies that focused on the educational effectiveness of the series and provide a summary and synthesis of the results of the research. By educational effectiveness, we mean effectiveness in those areas related to beginning schooling, to the early years of children's formal education. An important goal of Sesame Street was to help children prepare for their initial school experience by exposing them to a variety of rich and appealing examples. Studies that attempt to provide evidence about the successes and/or failures of Sesame Street in that effort are the focus of this review.
SOME EMPIRICAL GUIDELINES FOR BUILDING TESTLETS1
Authors: Wainer Howard , Kaplan Bruce , Lewis Charles
Published: August 2014

A series of computer simulations were run to measure the relationship between testlet validity and the factors of item pool size and testlet length for both adaptive and linearly constructed testlets. We confirmed the generality of earlier empirical findings (Wainer, Lewis, Kaplan & Braswell, 1991) that making a testlet adaptive yields only marginal increases in aggregate validity because of the peakedness of the typical proficiency distribution.
HOW WELL CAN WE EQUATE TEST FORMS THAT ARE CONSTRUCTED BY EXAMINEES?1
Authors: Wainer Howard , Wang Xiang‐Bo , Thissen David
Published: August 2014

When an exam consists, in whole or in part, of constructed response items, it is a common practice to allow the examinee to choose among a variety of questions. This procedure is usually adopted so that the limited number of items that can be completed in the allotted time does not unfairly affect the examinee. This results in the de facto administration of several different test forms, where the exact structure of any particular form is determined by the examinee. When different forms are administered, a canon of good testing practice requires that those forms be equated to adjust for differences in their difficulty. When the items are chosen by the examinee traditional equating procedures do not strictly apply. In this paper we explore how one might equate within an IRT framework. We illustrate our procedure with data from the College Board's Advanced Placement Test in Chemistry.
AN EXPERIMENT IN THE APPLICATION OF AN AUTOMATED ITEM SELECTION METHOD TO REAL DATA*
Authors: Stocking Martha L. , Swanson Len , Pearlman Mari
Published: August 2014

A previously developed method of automatically selecting items for inclusion in a test subject to constraints on item content and statistical properties is applied to real data. Two tests are first assembled by experts in test construction who normally assemble such tests on a routine basis. Using the same pool of items and constraints articulated by test construction experts the same two tests are reassembled automatically. The manual and automatic assemblies are compared by test specialists who were not involved in the original manual assembly. Based on this experiment, barriers to future progress in the improvement of automatic test assembly methods seem not to be in the development of different algorithms, nor in the improvement of computer time and cost. Rather, the focus of future improvements in the process of automatic test assembly lies more in the direction of complete specifications of constraints on item selection and detailed coding of item properties.
INSTABILITY IN A TREE APPROACH TO REGRESSION
Authors: Kim Sung‐Ho
Published: August 2014

One of the major problems that a tree‐approach to data analysis often encounters is instability of tree‐structures. Thus if one wishes to interprete the data structure by the tree‐approach, the instability issue must be dealt with.

Examining instability at a node of a tree provides insight into the instability of the whole tree, since the same theory of instability applies to all the nodes. Thus, this paper deals with the instability issue at a single node of a tree.

We assume that data are from a regression model, and examine what factors in that model affect the instability. Squared‐error loss is considered as a criterion for tree‐construction (“ls” criterion in CART program). The selection rate of a regressor variable at a node of a tree is used as a measure of instability. The selection rate mainly depends on (i) regression coefficients, (ii) (conditional) variance‐covariance structure of the regressor variables (given a subset of the regressor variables), (iii) the sample size, and (iv) noise in the response variable. We report simulation results that show patterns of instability for several different settings of regression models.
A PRELIMINARY STUDY OF THE NATURE OF COMMUNICATIVE COMPETENCE
Authors: Henning Grant , Cascallar Eduardo
Published: August 2014

Since the introduction of the expression “communicative competence” by Hymes in 1972, programs for the teaching and testing of English as a second or foreign language have increasingly been evaluated in terms either of their ability to promote communicative competence or of their sensitivity to a communicative view of language competence and performance. Unfortunately, despite this evident trend, there is no consensus on the nature of the communicative competence construct(s), nor has sufficient empirical support been provided for the various explanatory models proposed.

The present study was conducted to survey the theoretical literature related to communicative competence; to identify major variables said to comprise the construct(s); to test for the comparative presence and measurability of such variables in typical native/nonnative speaker university academic communication; to propose a tentative model of communicative competence as‐ a synthesis of these variables; and to examine the relationship of Test of English as a Foreign Language (TOEFL), Test of Spoken English (TSE), and Test of Written English (TWE) scores with the various elements of the tentative model. Accordingly, a schema was devised for the systematic elicitation of academic communicative language performance and this schema was applied in the assessment of the 79 adult, English‐as‐a‐second language students who were chosen for the study. In all, 158 hours of video‐recorded language‐interaction episodes and 12 communicative writing samples were gathered by trained interlocutors, and observations on 40 communicative competence variables (21 in an oral modality, and 19 in a written modality, of which 18 in each modality are reported here) were rated by trained evaluators. Variables were examined for reliability and correlation with subtest and total scores of the TOEFL, TSE, and TWE examinations.

Results provide information about the comparative contributions of some theory‐based communicative competence variables to domains of linguistic, discourse, sociolinguistic, and strategic competencies. In turn, these competency domains were investigated for their relation to components of language proficiency as assessed by the TOEFL, TSE, and TWE tests. Twelve oral and twelve written communication tasks were also analyzed and rank ordered for suitability in eliciting communicative language performance.
METHODS USED TO ESTABLISH SCORE COMPARABILITY ON THE ENHANCED ACT ASSESSMENT AND THE SAT®
Authors: Marco Gary L. , Abdel‐fattah A. A. , Baron Patricia A.
Published: August 2014

Recently Marco and Abdel‐fattah (1991) reported newly established relationships between scores on the enhanced American College Testing Program (ACT) Assessment and scores on the College Board's Scholastic Aptitude Test (SAT). The current report provides a detailed description of the methodology that was used to develop the “concordance” tables reported in that study.

Fourteen large universities provided data on applicants who had taken both the enhanced ACT Assessment and the SAT. The ACT Composite scores and the SAT‐verbal and SAT‐mathematical (SAT‐V + M) scores used in the comparability study came from ACT Assessment test editions administered from October 1989 to June 1990 and SAT editions administered from March 1989 to June 1990. The total sample consisted of 40,492 students. A subsample of 40,051 students who look the enhanced ACT Assessment and the SAT no more than 217 days apart was also used in the comparability study.

The equipercentile procedure, a curvilinear procedure, was used to scale the scores. Steps were taken to ensure the accuracy of the conversions by (1) weighting the scores to reduce or eliminate the effect of the time differential between ACT and SAT testings or (2) smoothing the score distributions before scaling. The score conversions derived by applying the weighting procedures provided score adjustments that were in the direction implied by the length of time between testings.
AN INVESTIGATION OF THE APPROPRIATENESS OF THE TOEFL TEST AS A MATCHING VARIABLE TO EQUATE TWE TOPICS
Authors: DeMauro Gerald E.
Published: August 2014

The feasibility of using linear and equipercentile equating methods (Angoff, 1984) to equate forms of the Test of Written English (TWE) by using the TOEFL test as an anchor was explored. These two equating methods assume that either the TOEFL test and TWE test measure the same skills or that the examinee groups across TWE administrations are equivalent in skills. The differences between equated and observed scores (equating residuals) and differences among the mean equated scores for examinee groups were further examined in terms of characteristics of the TWE topics.

An evaluation of the assumptions underlying the equating methods suggests that the TOEFL and TWE tests do not measure the same skills and that examinee groups are often dissimilar in skills. Therefore, use of the TOEFL test as an anchor to equate TWE tests does not appear appropriate. An alternative equating model based on expert judgment during pretest evaluation of potential essay prompts is recommended for future investigation.
A COMPARISON OF THE PREDICTIVE VALIDITY OF THE CURRENT SAT AND AN EXPERIMENTAL PROTOTYPE
Authors: Hale Gordon A. , Bridgeman Brent , Lewis Charles , Pollack Judith M. , Wang Ming‐mei
Published: August 2014

As part of a large‐scale project to remodel the Scholastic Aptitude Test (SAT), this study examined the predictive validity of a prototype revised SAT, which incorporated many of the important structural changes planned for the test. This prototype was compared to a form of the current SAT with regard to several validity‐related issues. The results showed that the test revisions (a) tended to increase predictive validity for the verbal score, the mathematical score, and the composite of verbal and mathematical scores; (b) slightly increased the incremental validity of the test over high school rank; and (c) produced a modest reduction in gender‐related prediction differences for the verbal score. Also, inclusion of scores from a new writing test tended to increase validity and to reduce gender‐related prediction differences. This evidence is regarded as preliminary, as the data were collected under experimental conditions with a limited sample of colleges, and the prototype used here was not identical in form to the remodeled SAT as it will be implemented operationally. Tentatively, however, the results were consistent with the goals of the overall project in that, where the present revisions had effects, they tended to be in the desired direction.
ATTITUDES TOWARD BORROWING AND PARTICIPATION IN POSTSECONDARY EDUCATION
Authors: Ekstrom Ruth B.
Published: August 2014

High school seniors who answered that they would choose to borrow, when asked what they would do if college costs were $1,500 more than they, their family, and a scholarship could provide, were significantly more likely to attend college in the next four years than were students who answered that they would choose other options (delaying college entrance, attending a less expensive college, or getting a job). This held true even after controlling for other variables such as educational aspirations, tested achievement, influence from others, and socioeconomic status. This finding supports the contention that students who are reluctant to borrow are less likely to enroll in postsecondary education. Among the students who did enroll in college, willingness to borrow was significantly associated with attendance at a four‐year college rather than a two‐year college and with full‐time rather than part‐time attendance. This may mean that recent changes in financial aid packaging, with fewer grants and more loans, have had an effect not only on college enrollments but also on the type of college that students attend and on the proportion of students enrolling in college on a part‐time basis. The results also suggest that, all else being equal, students who are knowledgeable about financial aid sources may be more likely to enroll in college than are students with less knowledge of these sources.
A REVIEW OF AUTOMATICALLY SCORABLE CONSTRUCTED‐RESPONSE ITEM TYPES FOR LARGE‐SCALE ASSESSMENT
Authors: Martinez Michael E. , Bennett Randy Elliot
Published: August 2014

The use of automated scanning of test sheets, beginning in the 1930s, led to widespread use of the multiple‐choice format in standardized testing. New forms of automated scoring now hold out the possibility of making a wide range of constructed‐response item formats feasible for use on a large‐scale basis. We describe new developments in five domains: mathematical reasoning, algebra problem solving, computer science, architecture, and natural language. For each one, we describe the task as presented to the examinee, the methods used to score the response, and the psychometric properties of the item responses. We then highlight general challenges and issues spanning these technologies. We conclude by offering our views on the ways in which such technologies are likely to shape the future of testing.
FIGURAL‐RESPONSE ASSESSMENT: SYSTEM DEVELOPMENT AND PILOT RESEARCH IN CELL AND MOLECULAR BIOLOGY
Authors: Martinez Michael E. , Jenkins Jeffrey B.
Published: August 2014

The purpose of the GRE figural‐response project was to design a prototype assessment system for delivering and scoring figural‐response items in the domain of cell and molecular biology, and to begin to investigate properties of the item format This report describes progress to date in an effort that is intended to be continuous and lead to program implementation. We first describe features of the delivery system and give sample items; ancillary developments, such as a tutorial, are also noted. Then, findings from a pilot research study are described. The essence of the pilot study was to examine the relationships between two item types (figural‐response and open‐ended verbal questions) and measures of figural and verbal ability. The data hint that whereas verbal items draw from verbal ability, figural‐response items draw from figural and verbal ability. The report concludes with a discussion of possible new directions for research, development, and eventual program use of the item format.
ASSESSING THE CLASSROOM PERFORMANCE OF BEGINNING TEACHERS: EDUCATORS' APPRAISAL OF PROPOSED EVALUATION CRITERIA
Authors: Powers Donald E.
Published: August 2014

Surveys were conducted of two multi‐state samples of educators, who provided ratings of the importance of two preliminary versions of the classroom performance assessment criteria being developed for The Praxis Series: Professional Assessments for Beginning Teachers™. There was considerable agreement about the importance of the criteria across several classifications of educators according to ethnicity, instructional level, years of teaching experience, subject area, and orientation to teaching.
COGNITIVE TASK ANALYSIS, INTERFACE DESIGN, AND TECHNICAL TROUBLESHOOTING
Authors: Steinberg Linda S. , Gitomer ew H.
Published: August 2014

We propose a model of the interface design process that makes use of two interdependent levels of cognitive analysis: 1) the study of the criterion task through an analysis of expert/novice differences and; 2) the evaluation of the working user interface design through the application of a practical interface analysis methodology (GOMS model). We review this dual analysis in the context of HYDRIVE, a video‐disc based intelligent tutoring system designed to facilitate the development of troubleshooting skills for aircraft hydraulics systems. The initial cognitive task analysis enabled the identification of critical troubleshooting skills and troubleshooting procedures. We find, though, mat even with an in‐depth initial cognitive task analysis, the GOMS interface analysis resulted in significant and beneficial design changes.
EXAMINEE BACKGROUND CHARACTERISTICS AND GRE GENERAL TEST PERFORMANCE
Authors: Stricker Lawrence J. , Rock Donald A.
Published: August 2014

The relationships between examinee background characteristics and performance on the GRE General Test were appraised by a structural equation modeling analysis. The examinees' initial characteristics (sex, ethnicity, parental education, geographic region, and age) had modest relationships with their test performance. Of these, parental education had the most consistent and strongest association. Sex also had an appreciable association, but it was limited to the quantitative score. College‐related characteristics (college major and the institution's public vs. private control, Carnegie classification, selectivity, and Ph.D. productivity) and undergraduate grade‐point average (GPA) generally had stronger and more pervasive relationships with test performance than did the examinees' initial characteristics, not only mediating the associations of the examinees' initial characteristics with test performance but also making independent contributions in their own right. The associations were especially strong for school quality (a composite of public vs. private control, selectivity, and Ph.D. productivity), college major, and undergraduate GPA.
PARTICIPATION IN RELIGIOUS ACTIVITIES BY COLLEGE‐BOUND HIGH SCHOOL SENIORS
Authors: Grandy Jerilee
Published: August 2014

Under a grant from the Lilly Endowment, ETS analyzed the 1989–90 Scholastic Aptitude Test (SAT) database to create profiles of students who participate in religious activities or organizations while in high school. Analyses included the comparison of profiles of students participating in religious activities versus those who did not, based on the entire student population. Then two denominational case studies of Presbyterian students and Roman Catholic students sought to answer questions of specific concern to policy makers in those denominations.

On the average, test takers who participated in religious activities differed from those who did not participate in the following ways:

• Their families had more education and a higher income, their SAT scores were higher, their grades in math, science, and writing were higher, more had taken honors courses, and they had higher degree aspirations.

• Students who participated in religious activities also participated more frequently in other activities, especially community or service activities.

• Students who participated in religious activities more often chose humanities, social sciences, health sciences, and education as college majors. They chose business less often than did students who did not participate in religious activities.

Additional information from the case study of Presbyterians was the following:

• If students were not participating in religious activities or organizations by ninth grade, they generally did not join later in high school.

• On the average, students who remained in religious activities or organizations throughout high school earned higher grades and test scores than those who dropped out.

The case study of Roman Catholic test takers yielded the following findings:

• A smaller percentage of Roman Catholic students participated in religious activities than did SAT takers as a whole. Catholic students attending religiously affiliated high schools participated in religious activities at about the same rate as mainline Protestant students attending all types of schools.

• Roman Catholic students attending religiously affiliated schools reported greater participation in religious activities and community activities than did Catholic students in public schools, but the number planning to participate in religious activities in college was not significantly higher for those attending religiously affiliated schools than for those in public schools.

• A comparison among Roman Catholic ethnic groups attending religiously affiliated schools showed Asian Americans reporting the highest levels of participation in religious activities in high school, participation in community service activities in high school, intention to participate in religious activities in college, and intention to attend a religiously affiliated college.
THE MATTER OF STYLE: MANIFESTATIONS OF PERSONALITY IN COGNITION, LEARNING, AND TEACHING
Authors: Messick Samuel
Published: August 2014

The historical roots of cognitive styles are traced in differential psychology, psychoanalytic ego psychology, gestalt and cognitive‐developmental psychology to illuminate the varied theoretical issues that energize (and fragment) style research. Optimal measurement of cognitive styles as information‐processing regularities and as intraindividual contrasts of abilities or other attributes is discussed in terms of both typical performance and contrasted performance on measures of opposing ends of bipolar style dimensions, with special emphasis on the contrasted measurement of field independence versus field dependence. The role of styles as both performance variables and competence variables in learning and teaching is examined, as are various critiques of style research that appear to be excessively polarized in either supporting or undercutting styles as meaningful constructs. This polarization appears to reflect different stances not just with respect to scientific evidence but with respect to ideology. Concluding remarks broach the issue of how styles are organized, not just within an information‐processing framework but within the structure of personality.
THE MATTER OF STYLE: MANIFESTATIONS OF PERSONALITY IN COGNITION, LEARNING, AND TEACHING
Authors: Messick Samuel
Published: August 2014

This project has been an outgrowth of the recently completed report Change in Field of Study from Undergraduate to Graduate School: Creation of a GRE Data Base for Studying Talent Flow by Jerilee Grandy and Nancy Robertson. Both examinee‐level and matrix data files created and analyzed under the original project were rearranged to make them more accessible to researchers. These multiyear files (covering testing years 1977–78 through 1986–87) have also been documented so that they can be easily located and analyzed. Extension of the data base through 1991–92 is underway and will be completed in 1993.
GENERALIZABILITY, VALIDITY, AND EXAMINEE PERCEPTIONS OF A COMPUTER‐DELIVERED FORMULATING‐HYPOTHESES TEST
Authors: Bennett Randy Elliot , Rock Donald A.
Published: August 2014

Formulating‐Hypotheses (F‐H) items present a situation and ask the examinee to generate as many explanations for it as possible. This study examined the generalizability, validity, and examinee perceptions of a computer‐delivered version of the task. Eight F‐H questions were administered to 192 graduate students. Half of the items restricted examinees to 7 words per explanation and half allowed up to 15 words. Generalizability results showed high interrater agreement, with tests of between two and four items scored by one judge achieving coefficients in the .80s. As in studies of paper‐and‐pencil versions, validity analyses found that although F‐H was highly reliable, it was only weakly related to GRE General Test scores, differing from that test primarily in relating more strongly to a measure of ideational fluency. Versions of F‐H based on different response limits tapped somewhat different abilities, with items employing the 15‐word constraint appearing more useful for graduate assessment. These items added to conventional measures in explaining school performance and creative expression. Finally, although the overwhelming majority of examinees found the F‐H interface easy to use, some experienced difficulty, suggesting the possibility that computer familiarity constitutes a source of irrelevant variance in F‐H scores.