bluuebunny/crossref_metadata_2025 huggingface dataset:
CATEGORIZING AND CONCEPTUALIZING STYLES IN YOUNGER AND OLDER ADULTS1
Authors: Kogan Nathan
Published: August 2014

The classification behavior of male and female college students was compared with that of healthy well‐educated older males and females. Object‐sorting and photo‐sorting (male and female faces) procedures were employed. On measures of categorizing style–conceptual differentiation and compartmentalization–older adults were, respectively, lower and higher than college students. On measures of conceptualizing style–categorical‐inferential, analytic‐descriptive, and relational thematic–younger adults exhibited stronger preferences for the first two, older adults for the third. Females manifested higher levels of conceptual differentiation than males on both sorting tasks and lower levels of compartmentalization on object‐sorting. Five content categories for sorting human faces were derived–physical, age, psychological, demographic, and preferential. No systematic age or sex differences for sorting male photos was obtained. For female photos, however, younger adults were more likely to use physical attributes for grouping, males were more likely to group on the basis of age, and females more strongly favored demographically‐based groupings. On the whole, the results failed to confirm other published evidence maintaining that aging is marked by conceptual deficits or a regressed mode of cognitive functioning.
QUICK ESTIMATES OF THE RELATIVE EFFICIENCY OF TWO TESTS AS A FUNCTION OF ABILITY LEVEL1
Authors: Lord Frederic M.
Published: August 2014

When comparing two tests that measure the same trait, an overall comparison is not enough. Separate comparisons should be made at different levels of the trait. A simple, practical, approximate formula is given for doing this. The adequacy of the approximation is illustrated using data comparing seven nationally known sixth‐grade reading tests.
LONG‐TERM PREDICTIVE VALIDITY OF DIVERGENT‐THINKING TESTS: SOME NEGATIVE EVIDENCE1
Authors: Kogan Nathan , Pankove Ethel
Published: August 2014

For two samples of middle‐class Ss, assessments of divergent‐thinking performance and intellective aptitude were obtained at 5th and 10th grades. Nonacademic attainments of these Ss were assessed with a biographical questionnaire in the 10th grade and upon graduation. Stepwise multiple regression analysis indicated that 5th‐grade ideational productivity had no predictive power in respect to nonacademic attainment at graduation, whereas 10th‐grade ideational productivity made a marginally significant contribution to that criterion in one of the school systems. Contrary to expectations, 5th‐ and 10th‐grade intellective‐aptitude measures accounted for modest to substantial amounts of the variation in nonacademic attainments. Assessment of such attainments was found to be fairly stable from 10th grade to graduation. Examination of separate fields of nonacademic attainment indicated diverse relationships with cognitive assessments. Implications for creativity research are considered.
A RELATIVE INTERACTION CRITERION AND ITS USE IN ORTHOGONAL ROTATION
Authors: Balloun Joe L. , Kearns Jack
Published: August 2014

A new criterion for analytic rotation based upon the concept of simple structure is introduced. The rationale for the method stems from a consideration of the way in which the variance of squared factor loadings is partitioned as in an analysis of variance. It is argued that a maximization of the relative amount of squared factor loading variation which is due to the “interaction” of factors and tests should produce results consistent with the simple structure hypothesis. The criterion is applied to orthogonal rotation using a gradient method of maximization.
REGIONAL DIFFERENCES IN GRADING PRACTICES
Authors: Pinchak Barbara Moretti
Published: August 2014

In 1972, 1069 high schools participated in a nationwide survey. Of those schools reporting information about their grading systems, 67.6% indicated that only letter grades were used, 16.3% indicated that only percentage grades were used, 5.4% used a system which combined letter and percentage grades, 2.6% used only numerical grades, 1.7% used only other types of grades (such as pass‐fail), 3.6% used a combination of grade types other than letter and percentage grades, and 2.7% of the schools did not indicate the exact system used. However, on a regional basis, the popularity of particular grade types varied. Also, within regions, there was quite a bit of variance in grade type used from state to state. This survey points out the difficulty in comparing grades from different parts of the country.
METHODS FOR EVALUATING EMPIRICAL BAYES POINT ESTIMATES OF LATENT TRAIT SCORES12
Authors: Kearns Jack , Meredith William
Published: August 2014

Empirical Bayes point estimates of latent trait scores, derived under the assumptions of one of several test theory models, display a certain degree of instability unless the sample size is sufficiently large. A measure of this instability over repeated sampling is the distribution of the overall expected squared error loss which converges, both in probability and in the mean, to the minimum (Bayes) overall expected loss as sample size increases. An asymptotic distribution theory is developed, and the resulting large sample approximation is compared with results obtained from simulated data. Attention is also given to the effects of using a smoothing procedure.
PRACTICAL METHODS FOR REDESIGNING A HOMOGENEOUS TEST, ALSO FOR DESIGNING A MULTILEVEL TEST*
Authors: Lord Frederic M.
Published: August 2014

Some practical methods for redesigning a homogeneous test are described. The effects of various possible changes in the Scholastic Aptitude Test are explored. Methods for designing and evaluating multilevel tests are described and illustrated.
DEFINITIONS, MEASUREMENTS, AND CLASSIFICATIONS OF STIMULI, SITUATIONS, AND ENVIRONMENTS
Authors: Pervin Lawrence A.
Published: August 2014

A review of issues relevant to the definition, measurement, and classification of stimuli, situations, and environments. Problems such as the lack of adequate definitions of concepts, error and bias in measurement procedures, confusion between measurement of a concept and measurement of its behavioral effects, and the lack of agreement among alternative measures are emphasized. It is suggested that concepts be defined in terms of objective characteristics while allowing for the study of the transactional relationship between organism and environment. The work of the ethologists in defining stimuli while studying their relationship to different organismic states and situational contexts is emphasized in this regard. Following Brunswik, it is also suggested that wherever possible there be a representative sampling of variables in natural settings.
A NOTE ON USE OF ANALYSIS OF VARIANCE IN DEVELOPMENTAL RESEARCH
Authors: Hale Gordon A.
Published: August 2014

Standard ANOVA provides a relatively ineffective test of developmental change when several age levels are involved and a roughly monotonic age trend can be assumed. A hypothetical example here serves to illustrate the point. An alternative use of ANOVA is suggested, and the value of the proposed solution is shown in reanalysis of published data.
EVALUATION WITH ARTIFICIAL DATA OF A PROCEDURE FOR ESTIMATING ABILITY AND ITEM CHARACTERISTIC CURVE PARAMETERS1
Authors: Lord Frederic M.
Published: August 2014

The LOGIST computer program was tested out on various sets of artificial data, giving a picture of the accuracy with which different parameters are estimated. A likelihood ratio test for the fit of the model to the data is explored and found not satisfactory.
FIELD DEPENDENCE AND LATERALIZATION OF VERBAL AND CONFIGURATIONAL PROCESSING
Authors: Zoccolotti Pierluigi , Oltman Philip K.
Published: August 2014

With tachistoscopic presentation, relatively field‐independent 18‐ to 30‐year‐old males were found to show a significant right‐visual‐field advantage in reaction time in a letter discrimination task, while relatively field‐dependent subjects did not show a significant hemifield difference. In a second study, relatively field‐independent 18‐ to 30‐year‐old males again showed a significant right‐visual‐field superiority in reaction time to letter discrimination, and in addition showed a significant left‐visual‐field superiority in tachistoscopic face discrimination, while relatively field‐dependent subjects showed no significant hemifield difference on either task. The results are interpreted as indicating a link between Witkin's concept of psychological differentiation and differentiation at the neural level, as manifested by specialization of function of the cerebral hemispheres.
CONSTRUCT VALIDITY OF THE SAT ACROSS POPULATIONS—AN EMPIRICAL CONFIRMATORY STUDY
Authors: Rock Donald A. , Werts Charles E.
Published: August 2014

An analysis of covariance structural factor model was posed that allowed for tests of invariance across populations of (1) similarity of the factor pattern, (2) equivalence of units of measurement, (3) standard errors of measurement of each factor, (A) reliability of each factor and (5) a factor mean structure. The analysis of the SAT‐V and the SAT‐M indicated support for the hypothesis that the respective factors for SAT‐V and M were measuring the same things across populations in the same units with equivalent accuracy as measured by the standard error of measurement.

The factor analysis of item type classifications indicated that little useful information would be obtained if the scores were reported by item type. This is particularly true for the SAT‐M. The verbal item type analysis did suggest the possibility that the analogy scores be reported separately and the remaining item types be combined in a single reading‐vocabulary score.
CONSTRUCTS AND THEIR VICISSITUDES IN EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT1
Authors: Messick Samuel
Published: August 2014

This paper examines some implications for educational and psychological measurement of three existential perspectives on the nature of mediating variables underlying and relating consistencies in test and non‐test behaviors. These three perspectives, respectively, treat mediators as (a) intervening variables operationally tied to real causal entities or source variables, such as personality traits or environmental contingencies; (b) hypothetical constructs that organize and summarize behavioral consistencies but have no reality outside the theoretical system; and, (c) manisfestations of real entities that are understood only in terms of constructs that summarize their empirical properties in relation to other constructs in a theoretical network. All three existential perspectives are applicable to personality traits or situational forces or their interactions, so implications for measurement hold whether for personal or environmental sources of behavioral consistency. In particular, the summary power of constructs in both theory and measurement that led to the predominance of construct validity principles in trait measurement implies that these principles should hold with equal cogency for situational and interactive measurement. Some slippery methodological problems arise in applying these principles, however, when clear distinctions are not articulated between the assessment model for measuring a construct and the substantive theory relating the construct to other constructs, each of which should have its own possibly different assessment model coordinated to its salient structural features.
THE USEFULNESS OF SELECTION TESTS IN COLLEGE ADMISSIONS
Authors: Beaton Albert E. , Barone John L.
Published: August 2014

In many selection situations, some predictor of success, such as a test, is used to discriminate between the applicants to be accepted and those to be rejected. Using an interpretation of Brogden [1946], this paper shows the effect of using a correlated variable on the mean criterion score of the selected group. The distribution of the means of all possible selections is discussed and a simple formula is provided for calculating the probability of achieving as high or higher mean using random selection as opposed to using a correlated variable. It is shown that even fairly small correlations result in selection decisions that are not often exceeded by chance. The Brogden interpretation and significance test is shown to extend simply to the case of several predictors.
SEX DIFFERENCES IN THE ENVIRONMENTS OF GRADUATE STUDENTS AND FACULTY
Authors: Hartnett Rodney T.
Published: August 2014

This paper considers some important psychological aspects of the academic experience for male and female graduate faculty members and students. Drawing on data collected in a recent national study of doctoral program quality, information pertaining to the graduate department's environment for learning, the extent of faculty members' concern for students, graduate student assistantship experiences, and faculty members' satisfactions and views about various departmental practices and policies are examined. Gender differences in both student and faculty member perceptions of their environments were found to be generally slight and to vary by discipline.
CURTAILED SAMPLING FOR CASE‐CONTROL STUDIES
Authors: Herrmann Nira , Szatrowski Ted H.
Published: August 2014

The use of curtailed sampling procedures for individually matched case‐control studies is proposed as an alternative to fixed sample size plans. Curtailed sampling provides a hypothesis testing procedure with the same power function as the corresponding fixed sample size test while guaranteeing that the random sample size is less than or equal to the fixed sample size in all cases. Several tables for the expected savings in sample size from curtailed procedures and an accurate, easy to calculate formula for approximating the expected savings are presented. In view of the substantial savings in sample size available under certain conditions and the accuracy of the approximating formula, it is recommended that curtailed procedures be given serious consideration in the design of matched pair case‐control studies.
AN UPPER ASYMPTOTE FOR THE THREE‐PARAMETER LOGISTIC ITEM‐RESPONSE MODEL*
Authors: Barton Mark A. , Lord Frederic M.
Published: August 2014

An upper‐asymptote parameter was added to the three‐parameter logistic item response model. This four‐parameter model was compared to the three‐parameter model on four data sets. The fourth parameter increased the likelihood in only two of the four sets. Ability estimates for the students were generally unchanged by the introduction of the fourth parameter.
GROUP COMPARISONS FOR BASIC SKILLS MEASURES
Authors: Breland Hunter M. , Griswold Philip A.
Published: August 2014

Correlation, regression, and score interval analyses were conducted for six academic measures as predictors of essay writing and overall performance. Comparisons for all analyses were made for men, women, Asians, blacks, Hispanics, and whites. The correlational comparisons showed few differences across groups, except that correlations tended to be lower for the white sample because of variance restrictions. The regression comparisons agreed with previous studies showing blacks and Hispanics to be generally overpredicted. On essay‐writing performance, men were also overpredicted by conventional basic skills measures. In contrast, women tended to write better essays than would have been predicted by conventional basic skills measures. False negatives, those people who score low on a predictor but who excel on a criterion, occurred least in the black and Hispanic groups.
STUDENT SELF‐SELECTION AND TEST REPETITION
Authors: Alderman Donald L.
Published: August 2014

Student self‐selection in deciding to repeat a test was examined by contrasting the test performance of students taking the College Board's Scholastic Aptitude Test (SAT) as juniors and again as seniors with the test performance of students taking the SAT only once as juniors. Estimates of expected test performance on a common initial administration in the junior year were derived from separate equating sections and background variables. Residuals of observed minus expected test scores revealed statistically significant differences between students who took a single administration of the SAT as juniors and students who took the same initial administration but also repeated the test as seniors; the initial observed scores of students later repeating the test were consistently lower than their expected scores for both the verbal and mathematical sections. These results indicate that self‐selection occurs when students decide to repeat a test and that score changes among these students reflect negative errors of measurement on the initial test administration as well as other factors.
PYRAMID POWER: SEARCHING FOR AN ERROR IN TEST SCORING WITH 830,000 HELPERS
Authors: Wainer Howard
Published: August 2014

A mathematics item was asked on the Preliminary Scholastic Aptitude Test which was keyed incorrectly. This subsequently was discovered and became the subject of national attention. In this paper we examine the data generated by this item by almost 830,000 examinees and discover that detailed statistical analysis with even this enormous sample size would not have yielded clues to the blunder. The epistemological and practical consequences of this are also discussed.
A COMPARISON OF FREE‐RESPONSE AND MULTIPLE‐CHOICE FORMS OF VERBAL APTITUDE TESTS
Authors: Ward William C.
Published: August 2014

Three verbal item types employed in standardized aptitude tests were administered in four formats–conventional multiple‐choice along with three formats requiring the examinee to produce rather than simply to recognize correct answers. For two item types, Sentence Completion and Antonyms, the response format made no difference in the pattern of correlations among the tests. Only for a multiple‐answer open‐ended Analogies test were any systematic differences found; even the interpretation of these is uncertain, since they may result from the speededness of the test rather than from its response requirements. In contrast to several kinds of problem‐solving tasks that have been studied, discrete verbal item types appear to measure essentially the same abilities regardless of the format in which the test is administered.
THE IDENTIFICATION OF BIASED ITEMS
Authors: Sinnott Loraine T.
Published: August 2014

A standard method for exploring item bias is the intergroup comparison of item difficulties. This paper describes a refinement and generalization of this technique. In contrast to prior approaches, the proposed method deletes outlying items from the formulation of a criterion for identifying items as deviant. It also extends the mathematical framework of item difficulty comparisons to allow the simultaneous analysis of any number of groups. As an example, the proposed method is applied to a set of quantitative items selected from a business school admission test.
EMPIRICAL BAYES ESTIMATION OF COEFFICIENTS IN THE GENERAL LINEAR MODEL FROM DATA OF DEFICIENT RANK1
Authors: Braun Henry I. , Jones Douglas H. , Rubin Donald B. , Thayer Dorothy T.
Published: August 2014

Empirical Bayes methods are shown to provide a practical alternative to standard least squares methods in fitting high dimensional models to sparse data. An example concerning prediction bias in educational testing is presented as an illustration.
PREDICTIVE VALIDITY OF THE SAT ON TWO HANDICAPPED GROUPS: THE DEAF AND THE LEARNING DISABLED
Authors: Jones Douglas H. , Ragosta Marjorie
Published: August 2014

Because of the ambiguity of Section 504 of the Rehabilitation Act of 1973, with regard to the need for predictive‐validity studies of admissions tests for handicapped persons, a preliminary study was undertaken. The purpose of the study was to evaluate the predictive validity of the SAT for deaf students at one institution and learning disabled (LD) students at another. SAT scores and measures of high school and college performance were obtained at each institution for both handicapped and nonhandicapped students. Additional descriptive information was obtained for the handicapped groups. The SAT scores of deaf students (when available) were significantly lower than the verbal and mathematical scores of the hearing students and tended to under predict deaf students' college performance. The combination of high school grade point average and SAT scores predicted performance equivalently for deaf and hearing students. For the learning disabled, the SAT verbal score and the high school and college performance measures were all significantly lower than those for the non‐LD students. The SAT validity coefficients for the two groups were not statistically significantly different. The combination of high school rank and SAT scores produced a significant lower validity coefficient for LD students than for non‐LD students. Since these were preliminary studies, no implications are being drawn pending further research.
MATHEMATICS ITEMS WITH HINTS
Authors: Trismen Donald A.
Published: August 2014

A set of 40 mathematics items was developed, intended for high school students who had completed at least one high school course in each of algebra and plane geometry. The items represented relatively unstructured problem situations, and each was supplemented by a sequence of hints which would progressively structure the problem for the examinee. A subset of nine of the items was rewritten, replacing the original hints with multiple‐choice question hints. The rewritten items were administered to a group of 107 examinees, using latent image materials.

This report describes the characteristics of hints, their relationships to the items they accompany, and their effectiveness in helping examinees to answer the item correctly. Its primary aim is to examine the function of hints in the problem‐solving process.

Three major findings were suggested:

The use of hints increases the percentage of correct answers to an item, regardless of whether the hints themselves are answered correctly.

The presentation of an open‐ended item, recast in multiple‐choice format, can, in some instances, act as a hint.

Answering hints correctly is related to answering the item correctly when a specific examinee misconception of the problem is involved.
ESTIMATING THE EFFECTS OF VARIOUS METHODS OF PREPARING FOR THE SAT
Authors: Powers Donald E.
Published: August 2014

A national sample of Scholastic Aptitude Test (SAT) candidates was surveyed to determine the ways in which they had prepared for the examination. Candidates' SAT scores were retrieved from test files, along with selected background information that included rank in high school class and scores on the Preliminary Scholastic Aptitude Test (PSAT), which most students had taken about eight months before they took the SAT.

Multiple regression analyses revealed effects of less than ten points on the 200‐800 SAT score scale and less than two points on the 20–60 score scale for the Test of Standard Written English (TSWE) for each of the methods of test preparation. A significant number of negative effect estimates, coupled with significant differences between the background characteristics of preparing and nonpreparing groups, strongly suggested the influence of selection bias in the estimates. Exploratory analyses of the effects of combinations of methods of preparation showed that various methods may interact in complex ways with other methods and with some student traits. Furthermore, self‐selection may not be uniform across levels of ability and previous achievement, and may also depend on the candidates' selection of other methods of preparation. The implications of the results for the design of further research on the effects of special preparation for tests are discussed.
MULTIVARIATE DISPLAY FROM QUIPUS TO FACES
Authors: Wainer Howard
Published: August 2014

The past decade has seen a substantial growth in methods and schemes for the display of multivariate data. This paper encompasses a sketch of the history of multivariate displays, examines a number of techniques, describes their construction, illustrates their use, and comments on their efficacy.
ON LORD'S PARADOX
Authors: Holland Paul W. , Rubin Donald B.
Published: August 2014

Lord's Paradox is analyzed in terms of a simple mathematical model for causal inference. The resolution of Lord's Paradox from this perspective has two aspects. First, the descriptive, non‐causal conclusions of the two hypothetical statisticians are both correct. They appear contradictory only because they describe quite different aspects of the data. Second, the causal inferences of the statisticians are neither correct nor incorrect since they are based on different assumptions that our mathematical model makes explicit but neither assumption can be tested using the data set that is described in the example. We identify these differing assumptions and show how each may be used to justify the differing causal conclusions of the two statisticians. In addition to analyzing the classic “diet” example which Lord used to introduce his paradox, we also examine three other examples that appear in the three papers where Lord discusses the paradox and related matters.
LONG‐TERM PREDICTIVE AND CONSTRUCT VALIDITY OF TWO TRADITIONAL PREDICTORS OF LAW SCHOOL PERFORMANCE
Authors: Powers Donald E.
Published: August 2014

The relative contribution of two traditional, widely used preadmission measures, undergraduate grade point average (UGPA) and the Law School Admission Test (LSAT), to predicting academic performance in each of the three years of law school was investigated in a heterogeneous sample of 23 law schools. The contribution of each measure was uneven across years, with the LSAT making its greatest contribution in early years and UGPA in later years. Differences between law school years with respect to the nature of instructional techniques and curricular emphases were considered in order to gain a better understanding of the construct validity of each measure as a predictor of performance in law school. The results were consistent with the test sponsor's interpretation of the LSAT as a measure of the ability to confront new situations and new problems such as those encountered in the first year of law school. They were also consistent with an interpretation of UGPA as a measure not only of academic competence but also of other qualities (such as persistence and motivation) that are needed over the long haul for the study of law.

The author wishes to acknowledge the Law School Admission Council for its support of the project on which this paper is based. Thanks go to Franklin R. Evans for helpful suggestions in conducting the research; to Thomas F. Donlon, Kenneth M. Wilson, and three anonymous reviewers for thoughtful reviews of this paper; to Miriam Godshalk for helpful editorial suggestions; and to the admissions staff of the participating schools for providing the data. Special thanks go to Frieda Hardy and Marie Ormond for managing the data collection activities; to Christopher Constantini, Katherine Kornhauser, and Norma Norris for processing and analyzing the data; to Henry Braun for statistical consultation; and to Lorraine Simon for her careful attention to numerous administrative tasks and to the final preparation of this manuscript.
MULTIVARIATE VARIANCE COMPONENT ANALYSIS: AN APPLICATION IN TEST DEVELOPMENT
Authors: Longford Nicholas T.
Published: August 2014

We present a multilevel variance component analysis of data from the pilot year of pretesting of an educational test. The multilevel nature of the data is induced by the clustering of students within colleges, and by having multivariate observations (scores) on students. The presence of a multidimensional trait underlying the scores is formulated as a hypothesis about the full rank of a variance matrix.
THE VALIDITY OF THE GMAT FOR THE PREDICTION OF SUCCESS IN DOCTORAL STUDY IN BUSINESS AND MANAGEMENT
Authors: Zwick Rebecca
Published: August 2014

The focus of this validity study was the examination of the degree to which GMAT scores and undergraduate grade‐point average (UGPA) could predict first‐year average (FYA) and final grade‐point average in doctoral programs in business. It was found that the three preadmissions measures were associated with graduate school grades, but to a lesser degree than in M.B.A programs. In general, the prediction achieved using UGPA alone as a predictor tended to be more accurate than that obtained using GMAT verbal (GMATV) and GMAT quantitative (GMATQ) scores together. Using all three predictors was more effective than using UGPA alone. The most likely explanation for the lower levels of prediction is that doctoral programs tend to be more selective than M.B.A. programs. Within‐school means on GMATV, GMATQ, UGPA, and FYA were higher than those found in M.B.A. validity studies; within‐school standard deviations on FYA tended to be smaller. Among these very select, academically competent students, highly accurate prediction of success in doctoral programs may not be possible.
CURRENT STATE SCIENCE ASSESSMENTS: IS NOTHING BETTER THAN SOMETHING?
Authors: Gong Brian , Lahart Colleen , Courtney Rosalea
Published: August 2014

This study reports a survey of current state science assessments. The study examined the extent to which the assessments supported standards of student performance consistent with three goals of science education reform: 1) focus on or promote development of depth of knowledge rather than breadth; 2) focus on central concepts and skills rather than on lower‐level facts; 3) make connections to technical, personal, and social applications.

Information about each state's science assessment practices was collected, including whether a state science assessment was mandated, the publisher of the assessment, and use of test scores. A sample of state science test instruments were subjected to content and cognitive analysis.

The analysis indicated that for a large majority of states, science assessments were not a driving force in directing the curriculum or instructional practices towards the standards of science education reform. Almost 50% of the states did not have a state science assessment. The state science assessments currently used were not consistent with the science education reform goals examined in this study. The implications of these findings were discussed for assessment and reform of science education at the state level.
AN ALTERNATIVE THREE‐PARAMETER LOGISTIC ITEM RESPONSE MODEL
Authors: Pashley Peter J.
Published: August 2014

Birnbaum's three‐parameter logistic function has become a common basis for item response theory modeling, especially within situations where significant guessing behavior is evident. This model is formed through a linear transformation of the two‐parameter logistic function in order to facilitate a lower asymptote. This paper discusses an alternative three‐parameter logistic model in which the asymptote parameter is a linear component within the logit of the function. This alternative is derived from a more general four‐parameter model based on a transformed hyperbola.
DIFFERENCES AMONG RACIAL AND ETHNIC GROUPS IN MEAN SCORES ON THE GRE AND SAT: LONGITUDINAL COMPARISONS
Authors: Pennock‐Román María
Published: August 2014

If one compares racial/ethnic group differences on verbal and quantitative tests from published data on the Scholastic Aptitude Test (SAT) and Graduate Record Examinations (GRE) General Test, some group differences appear to be relatively larger (in standard deviation units) on the GRE General Test than on the SAT. The objective of this study was to undertake a retrospective, longitudinal analysis of a group of individuals who had taken both tests to see if group differences remain the same or change over time.

After merging records on the GRE General Test with SAT records for the same individuals, the realized sample had 69,494 cases.

In this sample, the pattern of results from longitudinal evidence was very different from that found in cross‐sectional comparisons of published GRE and SAT data for the various groups. The changes in the relative size of mean differences were in the direction of closing the gap between non‐Hispanic (NH) White and minority groups, especially for Asian American and Puerto Rican examinees.

The regressions of GRE General Test scores on SAT scores (i.e., growth) for various racial/ethnic groups were dissimilar from that of the NH White group, but the differences in growth patterns did not widen the distance in means between NH White test takers and minority groups.

There was also evidence that racial/ethnic groups may vary in the degree of self‐selection that occurs for entry into the graduate‐school‐bound pool of GRE test takers. This pattern may explain the relatively larger disparity in group means in the GRE population than in the SAT population that is evident in the cross‐sectional comparisons. However, this explanation needs to be verified with more complete data on pathways to graduate school that include non‐SAT test takers and students who apply to professional schools.
NOTE TAKING AND LISTENING COMPREHENSION ON THE TEST OF ENGLISH AS A FOREIGN LANGUAGE
Authors: Hale Gordon A. , Courtney Rosalea
Published: August 2014

This study examined the effects of taking notes in the portion of the TOEFL listening comprehension section that contains short monologues, or “mini‐talks.” These effects were assessed in experimental testing sessions with students in intensive English language programs and with undergraduate and graduate international students. A multiple‐choice questionnaire surveyed the students' reactions to the opportunity to take notes and their previous note‐taking experiences.

Allowing students to take notes had little effect on their performance, and urging students to take notes significantly impaired their performance. These effects were observed even for students who reported being in the habit of taking many notes or reported having had classroom instruction in note taking. Apparently, then, little benefit is gained by taking notes in the context of the present TOEFL mini‐talks, perhaps because they are designed to assess listening comprehension with minimal demand placed on memory. Responses to the questionnaire aid in understanding the results and provide useful general information about the students' note‐taking experiences and habits.
A STUDY OF THE EFFECTS OF CONTEXTUALIZATION AND FAMILIARIZATION ON RESPONSES TO THE TOEFL VOCABULARY TEST ITEMS
Authors: Henning Grant
Published: August 2014

In an effort to evaluate criticisms of the current TOEFL® vocabulary item format and to ascertain the comparative effectiveness of alternative vocabulary test item formats, the present study investigated the functioning of eight different multiple‐choice formats that differed with regard to the length and inference‐generating quality of the stem, with regard to the nature of the task (matching versus supply), and with regard to the degree of passage embeddedness of item stems or response options. In all, 1,040 vocabulary test items (80 familiarization items and 960 experimental items) were developed and administered in counterbalanced clusters to a total participating sample of 190 adult English‐as‐a‐second‐language students (99 persons with a controlled prior 75‐minute familiarization activity and 91 with no familiarization activity) at two language institutes. These two levels of familiarization were examined to ascertain effects of familiarization on performance with the eight item types. Also, self‐reports of levels of prior exposure to particular item types were correlated with actual performance with those item types.

Results indicated that, for the present sample of persons and items, the current TOEFL vocabulary item format performed comparatively well in terms of appropriateness of difficulty, mean internal consistency reliability, and criterion‐related validity. Such performance came in spite of earlier suggestions that current item stems may be too long, lack inference‐generating information about the meaning of the targeted vocabulary, and place undue reliance on matching rather than supply format.

Among alternative item formats considered, only items embedded in reading passages appeared to outperform current TOEFL vocabulary format in terms of estimates of both reliability and criterion‐related validity. Use of an item format that differed from current TOEFL vocabulary format only in the uniform presence of inference‐generating information in the item stem and in reduction of mean length of stem also slightly outperformed current TOEFL vocabulary item format in terms of the particular criterion validity indicated by correlation with vocabulary total score; however, these correlational differences did not quite attain statistical significance. In general, multiple‐choice items using stems with underscored target vocabulary in context performed better than did either multiple‐choice single‐word‐or‐phrase matching tasks or multiple‐choice supply‐type items.

Participation in the familiarization activity did not relate significantly or differentially to performance with any item type. However, self‐reports of familiarity with particular item types suggested that some types were more familiar than others, and the three most familiar item types, including current TOEFL vocabulary format, exhibited a significant positive correlation between self‐report of familiarity with the item type and successful performance with the item type.
A PROCEDURE FOR ESTIMATING THE CONDITIONAL STANDARD ERRORS OF MEASUREMENT FOR GRE GENERAL AND SUBJECT TESTS
Authors: Cowell William R.
Published: August 2014

A series of computer programs were written for computing the conditional standard errors of measurement (CSEM) for both rights‐scored and formula‐scored tests based on a method suggested in Lord (1984), commonly known as Lord's Method IV or the compound binomial method. These programs estimate the conditional standard errors of measurement for both raw and scaled scores, average results for two or more forms, and compute form‐to‐form difference statistics for pairs of forms.

Conditional standard errors of measurement, averages, and differences have been computed for the verbal, quantitative, and analytical raw and converted scores for eight forms of the GRE General Test and for two forms each of 15 GRE Subject Tests.

The Standards for Educational and Psychological Testing (Committee of AERA, APA, & NCME to Develop Standards, 1985) recommends that test publishers provide estimates of the standard error measurement at a number of widely spaced score levels. The CSEM data produced in this study have been made available to three programs which use GRE scores, along with other criteria, for awarding fellowships. These data also have been made available for use in GRE program publications and in correspondence.
NEGATIVE COEFFICIENTS IN THE GRE VALIDITY STUDY SERVICE
Authors: Longford Nicholas T.
Published: August 2014

Operational procedures for the Graduate Record Examinations Validity Study Service (GREVSS) are reviewed, with the emphasis on the problem of frequent occurrence of negative coefficients in the fitted within‐department regressions obtained by the empirical Bayes method of Braun and Jones (1985). Several alterations of the operational procedures are proposed that would reduce the frequency of negative coefficients, and, if desired, completely eliminate them. It is argued, however, that there are no a priori reasons for assuming thatallthe coefficients are nonnegative. Reports of the fitted within‐department regressions should be based on a single model, that would be found by model exploration. The estimation procedures could be improved by employing more flexible software for modelling between‐department variation.
COHORT DIFFERENCES ASSOCIATED WITH TRENDS IN SAT® SCORE AVERAGES
Authors: Morgan Rick
Published: August 2014

The potential impact of cohort changes with regard to ethnic group, gender, class rank, and first language learned on yearly average Scholastic Aptitude Test (SAT) and Test of Standard Written English (TSWE) scores was studied. Regression and cross‐classification analyses were conducted on data from 1985, and 1987 to 1990. Both analyses adjusted test score means based on common cohorts of students. The analyses indicated that approximately half of the decline in SAT verbal (SAT‐V) scores was associated with cohort change. Average SAT mathematics (SAT‐M) scores could possibly have risen by three points, rather than remaining constant, if the 1987 and 1990 cohorts were the same.
USING A TRAINABLE PATTERN‐DIRECTED COMPUTER PROGRAM TO SCORE NATURAL LANGUAGE ITEM RESPONSES
Authors: Kaplan Randy M.
Published: August 2014

In this study we investigated a computer scoring procedure for constructed‐response items whose responses consist of natural language phrases or sentences. The costs for scoring such items either manually or by a traditional natural language computer processing approach is typically expensive.

The purpose of this study was to investigate the feasibility of an alternate method to the traditional natural language processing approach. The two main goals of the study were to create a prototype scoring program based on an appropriate methodology and to carry out a preliminary comparison of the results of the scoring methodology to the scores of human raters.

Our approach to scoring is based on the assumption that a set of responses can be described as a small language. From this language we can construct a grammar that describes the language. The grammar can then be used as a tool for recognizing sentences in the language.

With respect to the task of scoring, this means creating a grammar from a sample of responses that have been scored correct or incorrect. This grammar is then used by a program to classify responses that were not used to create the grammar. The program automatically classifies a response as correct (or incorrect).

The computer program was used to score sets of responses from three items. Two sets of data consisted of short responses (3 to 5 words). The third set of data consisted of longer responses (12 to 15 words). The scoring program rating agreed perfectly with those of human raters for the first two data sets and achieved a maximum agreement of 90% for the third data set.

We conclude from these preliminary results that a trainable pattern‐directed computer scoring program may be a viable, cost‐effective approach to scoring short natural language responses. Longer responses present a number of problems that require further investigation. We suggest a number of ways that the prototype program can be extended to compensate for these more complex responses.
TOWARD A TEST THEORY FOR ASSESSING STUDENT UNDERSTANDING
Authors: Mislevy Robert J. , Yamamoto Kentaro , Anacker Steven
Published: August 2014

The view of learning that underlies standard test theory is inconsistent with the view rapidly emerging from cognitive and educational psychology. Learners become more competent not simply by learning more facts and skills, but by reconfiguring their knowledge; by “chunking” information to reduce memory loads; and by developing strategies and models that help them discern when and how facts and skills are important. Neither classical test theory nor item response theory (IRT) is designed to inform educational decisions conceived from this perspective. This paper sketches the outlines of a test theory built around models of student understanding, as inspired by the substance and the psychology of the domain of interest. The ideas are illustrated with a simple numerical example based on Siegler's balance beam tasks. Directions in which the approach must be developed to be broadly useful in educational practice are discussed.
EFFECTS OF PROSE COMPLEXITY ON ACHIEVEMENT TEST ITEM DIFFICULTY
Authors: Scheuneman Janice , Gerritz Kalle , Embretson Susan
Published: August 2014

Better understanding of sources of difficulty in test items would improve the test development process by bringing the functioning of items more under the control of the test developer. To help increase this understanding, a study was undertaken to evaluate the effects of various aspects of prose complexity on the difficulty of achievement test items. The items of interest were those that presented a verbal stimulus followed by a question about the stimulus and a standard set of multiple‐choice options. Items were selected for study from two tests with differing demands on an examinee's knowledge base, NTE Communications Skills and GRE Subject Test in Psychology. Standard multiple regression analyses and Embretson's model fitting procedures were used to evaluate the contribution of various complexity factors to the prediction of difficulty. These factors, which included measures of item structure, readability, semantic content, cognitive demand and knowledge demand, were found to be successful in predicting item difficulty for these items. The immediate usefulness of the results for test development practice, however, are limited by the fact that only a single item type was studied and by the time required to develop the complexity measures.
CONSTRUCTED RESPONSE AND DIFFERENTIAL ITEM FUNCTIONING: A PRAGMATIC APPROACH1
Authors: Dorans Neil J. , Schmitt Alicia P.
Published: August 2014

Differential item functioning (DIF) assessment attempts to identify items or item types for which subpopulations of examinees exhibit performance differentials that are not consistent with the performance differentials typically seen for those subpopulations on collections of items that purport to measure a common construct. DIF assessment requires a rule for scoring items and a matching variable on which different subpopulations can be viewed as comparable for purposes of assessing their performance on items. Typically, DIF is operationally defined as a difference in item performance between subpopulations, e.g., Blacks and Whites, that exists after members of the different subpopulations have been matched on some total score. Constructed‐response items move beyond traditional multiple‐choice items, for which DIF methodology is well‐defined, towards item types involving selection or identification, reordering or rearrangement, substitution or correction, completion, construction, and performance or presentation. This paper defines DIF, describes two standard procedures for measuring DIF and indicates how DIF might be assessed for certain constructed‐response item types. The description of DIF assessment presented in this paper is applicable to computer‐delivered constructed‐response items as well as paper and pencil delivered items.
SEX DIFFERENCES IN THE RELATIONSHIP OF ADVANCED PLACEMENT ESSAY AND MULTIPLE‐CHOICE SCORES TO GRADES IN COLLEGE COURSES
Authors: Bridgeman Brent , Lewis Charles
Published: August 2014

Essay and multiple‐choice scores from Advanced Placement (AP) examinations in American History, European History, English Language and Composition, and Biology were matched with freshman grades in a sample of 32 colleges. Multiple‐choice scores from the American History and Biology examinations were superior to essays for predicting overall GPA, but essay scores were essentially equivalent to multiple‐choice scores for predicting grades in history, English, and biology. In history courses, males and females received comparable grades and had nearly equal scores on the AP essays, but the multiple‐choice scores of males were about half of a standard deviation higher than the scores of females. To the extent that the AP history examinations are intended to emulate performance in college history courses, placing greater weight on the essay component of the AP history examinations would reduce sex differences without compromising content or predictive validity.