bluuebunny/crossref_metadata_2025 huggingface dataset:
TEST FAIRNESS: A COMMENT ON FAIRNESS IN STATISTICAL ANALYSIS
Authors: Myers Charles T.
Published: August 2014

An argument is presented to suggest that the analysis of covariance may in some circumstances be an unfair method to use in the study of the question of test fairness. As an alternative, the use of equipercentile methods or equivalent linear methods may be preferred in these circumstances.
FACULTY ACCEPTANCE OF INSTRUCTIONAL TECHNOLOGY: ATTITUDES TOWARD EDUCATIONAL PRACTICES AND COMPUTER‐ASSISTED INSTRUCTION AT COMMUNITY COLLEGES
Authors: Alderman Donald L. , Mahler William A.
Published: August 2014

Teacher resistance is often cited as a primary reason behind schools' reluctance to adopt instructional technology. This study concerns faculty beliefs about educational practices and attitudes toward computer‐assisted instruction at community colleges. The major dimensions of such beliefs and attitudes, as well as faculty comments and responses to specific questions, help in understanding school acceptance of instructional technology. Completed surveys from approximately 300 faculty members at six colleges indicated current opinions about educational practices and provided a baseline of initial attitudes toward CAI.
THE REACTIVE EFFECTS OF NEGOTIATING DETOURS: COGNITIVE STRESS AND THE RESTRUCTURING OF MENTAL MAPS
Authors: Ryan Michael P.
Published: August 2014

A spatial analogue of a dissonance manipulation was used to test the claim that the mere acquisition of incongruent data produces cognitive reorganization. Undergraduates ranked distances among 13 campus landmarks before and after detouring around an imaginary barrier on embedding new landmarks in their mental maps. Scaling procedures revealed that barrier detours generated structural distortions of existing cognitive maps whereas landmark substitutions did not. The detour transformation produced systematic biases in subjects' postmanipulation maps; the substitution transformation was completely nonreactive. It is concluded that mental acts which create momentary distortions of cognitive structure can permanently reshape one's internal representations.
PREDICTION OF COLLEGE ACHIEVEMENT AMONG MEXICAN‐AMERICAN STUDENTS IN CALIFORNIA
Authors: Warren Jonathan R.
Published: August 2014

Equations for the prediction of college grades from test scores and high school grades were compared for Mexican‐Americans and other students at three universities of the California State University and Colleges system. With rare exceptions, regression equations for the prediction of grades for Mexican‐Americans did not differ from those for other students at the freshman, sophomore, or junior level. The prediction of senior year grades did require different equations for Mexican‐Americans, but the direction of the difference was reversed at the two universities involved. The need for separate prediction equations for Mexican‐American applicants to college is not supported. Nevertheless, the modest size of the multiple correlation coefficients, usually in the range from .30 to .50, suggests that considerations other than test scores and prior grades should enter admission decisions. The few comparisons across universities did not indicate a need for different equations for different institutions, but a difference in grade distributions at the two institutions compared suggests that the question be examined again with more substantial data. Men and women differed in their prediction equations, although neither sex was consistently over‐nor underpredicted. Men, however, almost invariably showed higher test scores and lower grades at all levels than did women.
COGNITIVE PERFORMANCE IN THE 12‐WEEK‐OLD INFANT: THE EFFECTS OF BIRTH ORDER, BIRTH SPACING, SEX, AND SOCIAL CLASS
Authors: Lewis Michael , Gallas Howard
Published: August 2014

Twelve‐week‐old infants were given a battery of perceptual‐cognitive tasks. These tasks included the Corman‐Escalona Scales of Object Permanence, the Mental Development Index of the Bayley Scales, and an attention task measuring habituation and dishabituation. The data failed to show any sex or social class differences, although the MDI was affected by birth order with first borns showing superior performance. In general these three measures of perceptual‐cognitive ability were unrelated. These data are used to question the notion of a “g” factor in infancy and therefore the usefulness of a test of infant intelligence.
THE SIGI PREDICTION SYSTEM: PREDICTING COLLEGE GRADES WITH AND WITHOUT TESTS
Authors: Norris Lila
Published: August 2014

This paper gives an example of how to predict college grades in the absence of tests. It describes components of the prediction system which is one section of the computer‐based System of Interactive Guidance and Information (SIGI). Students who use the SIGI prediction system are provided with detailed criterion information and then asked to rate themselves on “achievement factors” and give estimates of their performance. These ratings and estimates, as well as self‐reported high school record and biographical data, are used as predictors in a regression analysis.

Predictive validities for final grades in four courses at Illinois State University were examined. Since students at ISU are required to take ACT, separate analyses were run using SIGI variables, ACT scores, and SIGI plus ACT. Consistent findings were found across the four courses: (1) multiple R's, using two or three SIGI variables, are quite high, falling in the range of .45 to .66,

(2) SIGI variables afford a higher level of prediction than do ACT scores, and

(3) the addition of ACT scores to SIGI variables does not appreciably improve the prediction levels.
VERBAL AND MATHEMATICAL ABILITY OF HIGH SCHOOL JUNIORS IN 1974: A NORMS STUDY OF PSAT/NMSQT
Authors: Jackson Rex , Schrader William B.
Published: August 2014

This study was undertaken to obtain current information on the performance of a national sample of all high school juniors on the Preliminary Scholastic Aptitude Test/National Merit Scholarship Qualifying Test (PSAT/NMSQT). The distributions of scores for such a sample are useful in score interpretation because they make it possible to find a student's relative standing in a suitable reference group. Beginning in 1960, results of similar surveys have been used to prepare national norms for the PSAT and SAT. Because the students taking these tests are self‐selected with respect to ability, national norms based on all students provide a useful supplement to descriptive statistics based on test candidate groups. The steady decline of SAT scores beginning in 1963–64 suggested the need for new data to replace those obtained from a similar study conducted in 1966.

The sample design for this study was similar to that used in the surveys conducted in 1960 and 1966. Provision was made, however, to replace nonparticipating schools in the initial sample with similar schools. Substantial attention was given to evaluating the representativeness of the sample by comparing participating with nonparticipating schools on the basis of geographical region, control, and percentage of minority students and by comparing student data for the norms sample with data from the National Longitudinal Study.

The results of this study were compared with those of the 1960 and 1966 norms studies. These comparisons, although not definitive, should be considered in assessing whether a shift in the ability level of high school students generally can help to account for the SAT score decline.
MEASURING THE APPROPRIATENESS OF MULTIPLE‐CHOICE TEST SCORES1,2
Authors: Levine Michael V. , Rubin Donald B.
Published: August 2014

A student may be so atypical and unlike other students that his aptitude test score fails to be a completely appropriate measure of his relative ability. We consider the problem of using the student's pattern of multiple‐choice aptitude test answers to decide whether his score is an appropriate ability measure. Several indicators of appropriateness are formulated and evaluated with a simulation of the Scholastic Aptitude Test.
FIELD DEPENDENCE AND THE ROLE OF VISUAL FRAMEWORKS IN THE PERCEPTION OF SIZE
Authors: Sigman Eric , Oltman Philip K.
Published: August 2014

The relationship between the field‐dependence dimension and the influence of visual frameworks on the perception of size was examined. Two spatial arrangements were used so that the influence of the frame would lead to an illusion in one instance and a more veridical response in the other. While the relatively field‐dependent subjects showed a greater error in the illusion condition, they showed more veridical size judgments in the condition in which use of the visual framework could contribute to more accurate size matching. Field dependence and field independence are viewed as perceptual strategies, each of which may be adaptive in certain situations.
INHERITANCE OF OCULAR DOMINANCE
Authors: Zoccolotti Pierluigi
Published: August 2014

The research investigated whether or not ocular dominance has an hereditary basis. Results show that families that contain at least one parent with a left eye preference tend to contain at least one child with a left eye preference. This finding is interpreted as an indication that there is an hereditary basis for ocular dominance.
THE USE OF ANALYSIS OF COVARIANCE STRUCTURES FOR ASSESSING THE PSYCHOMETRIC UTILITY OF MULTIPLE CRITERIA IN DIFFERENTIAL PREDICTION STUDIES
Authors: Rock D. A. , Werts C. , Flaugher R.
Published: August 2014

It is assumed that the dependent variables in differential prediction studies, test bias studies, analysis of variance and covariance designs are characterized by certain psychometric properties which are invariant across the subgroups of interest. More specifically, the validity of the results in such analyses depend on the assumption that the dependent variables are measuring the same constructs in the same metrics with equivalent reliabilities across all subgroups. Systematic procedures are outlined for testing these assumptions.
COGNITIVE STYLES IN THE DEVELOPMENT OF MEDICAL CAREERS*
Authors: Goodenough Donald R. , Oltman Philip K. , Friedman Florence , Moore Carol Ann , Witkin Herman A.
Published: August 2014

Recent research has shown that academic choice and achievement may be partly a function of the student's standing on the field‐dependence‐independence cognitive‐style dimension. The results of two longitudinal studies suggest that information about field dependence‐independence may be of value for student guidance in the medical setting. The first, a study of college undergraduates who early expressed an interest in medicine, showed that these cognitive styles play a discernible role in determining who will eventually enter medical school. The second, a study of medical students, showed that field‐dependent and field‐independent students subsequently tend to choose different medical specialties. These results are consistent with cognitive‐style theory, which proposes that field‐independent people will choose vocations that require cognitive restructuring skills, whereas field‐dependent people will choose vocations that require greater social‐interpersonal involvement.
THE USE, ACCEPTANCE, AND IMPACT OF TAKING THE SAT—A TEST FAMILIARIZATION BOOKLET
Authors: Powers Donald E. , Alderman Donald L.
Published: August 2014

The College Board recently published Taking the SAT, a booklet designed to improve candidates' familiarity with the Scholastic Aptitude Test (SAT) and the Test of Standard Written English (TWSE), which is administered with the SAT. Prepublication copies were mailed to a random sample of SAT candidates, and the recipients, after taking the tests, were surveyed for their perceptions of the booklet and its influence on their test‐taking. An equivalent random sample received only About the SAT, a briefer Board publication that Taking the SAT is to replace, and were surveyed for their test‐taking experiences and their opinions of the older test familiarization booklet. Test scores of the students in both groups were subsequently obtained from SAT score files for comparison.

The survey showed that the first group had reacted positively to the new publication and used it extensively. These candidates perceived numerous benefits from its use, some of them greater than the benefits perceived by candidates who had used only the older, briefer publication. The analysis did not, however, uncover any effects of the new booklet on scores for any section of the SAT or TSWE. Nevertheless, many students believed that Taking the SAT had helped them achieve higher scores.

The report documents the strong points of the new booklet and makes recommendations for improving some sections in future editions.
AFFECT JUDGMENT AND FIELD DEPENDENCE
Authors: Rapaczynski Wanda , Welkowitz Joan , Sadd Susan
Published: August 2014

Ten women, 5 low scorers (field‐dependent) and 5 high scorers (field‐independent) on the Group Embedded‐Figures Test, listened to 24, 15‐minute‐long segments of a series of therapy sessions and were then asked: (a) to adopt the role of the patient and rate the warmth felt by the patient toward the therapist, and (b) to adopt the role of therapist and rate the warmth felt by the therapist toward the patient, using a 4‐point rating scale. The accuracy of subjects' ratings was evaluated against the criterion of actual ratings made by patient and therapist after each therapy session. As predicted, field‐dependent subjects were more accurate in judging warmth than field‐independent subjects. In addition, there was a significant interaction between subject's cognitive style and the person being judged. Field‐dependent subjects were more accurate in the role of patient than in the role of therapist; the opposite was true for field‐independent subjects. Suggestions are made as to possible routes used by field‐dependent and field‐independent subjects to judge emotions experienced by others.
THE PERFORMANCE OF NON‐NATIVE SPEAKERS OF ENGLISH ON TOEFL AND VERBAL APTITUDE TESTS
Authors: Angelis Paul J. , Swinton Spencer S. , Cowell William R.
Published: August 2014

This study examined the performance of two groups of non‐native English speakers on the Test of English as a Foreign Language (TOEFL) and an appropriate verbal aptitude test. One group of graduate applicants took both TOEFL and the verbal section of the Aptitude Test of the Graduate Record Examinations (GRE). Another group of undergraduate applicants took TOEFL, the verbal section of the College Board Scholastic Aptitude Test (SAT), and the Test of Standard Written English (TSWE). Data are presented showing how native and non‐native speakers compare on each set of tests. Information is also provided to aid in interpreting test results for non‐native speakers who have taken both types of test. The appendix to the report summarizes item reviews, by specialists in English as a Second Language, which suggest future directions for TOEFL test development.
TEST VALIDITY AND THE ETHICS OF ASSESSMENT
Authors: Messick Samuel
Published: August 2014

Questions of the adequacy of a test as a measure of the characteristic it is interpreted to assess are answerable on scientific grounds by appraising psychometric evidence, especially construct validity. Questions of the appropriateness of test use in proposed applications are answerable on ethical grounds by appraising potential social consequences of the testing. The first set of answers provides an evidential basis for test interpretation and the second set provides a consequential basis for test use. In addition, this paper stresses the importance of construct validity for test use since it provides a rational foundation for predictiveness and relevance, as well as the importance of taking into account the value implications of test interpretations per se. By thus considering both the evidential and consequential bases of both test interpretation and test use, the roles of evidence and social values in the overall validation process are illuminated, and test validity comes to be based on ethical as well as evidential grounds.
FEEDBACK AS AN INCENTIVE FOR RESPONDING TO A MAIL QUESTIONNAIRE
Authors: Powers Donald E. , Alderman Donald L.
Published: August 2014

The effect of offering respondents feedback on questionnaire results was investigated in a national mail survey of college‐bound secondary school students. It was found that offering feedback had a significant positive effect on response rate, but the magnitude of that effect was slightly less than the increase in response rate resulting from a shorter questionnaire and considerably less than a follow‐up contact with nonrespondents.
ISSUES OF EFFECTIVENESS AND EQUITY IN THE COACHING CONTROVERSY: IMPLICATIONS FOR EDUCATIONAL AND TESTING PRACTICE1
Authors: Messick Samuel
Published: August 2014

Multiple sources of controversy over coaching for scholastic aptitude tests are addressed. These include disputes over the meaning of scholastic aptitude, over the meaning of coaching, over the nature of the fundamental research questions, over the adequacy of the empirical evidence for coaching effectiveness, over the possible implications of effective coaching for student performance and for test validity, and over the consequent ethical imperatives for educational and testing practice. Three possible outcomes of coaching having different implications for college performance are discussed — genuine improvements in the abilities measured by the test resulting in commensurate increases in test scores; enhanced test‐taking sophistication (item familiarization, guessing, pacing) resulting in increased test scores that are more accurate assessments of ability; and, heightened test‐taking artifice (stratagems and answer‐selection tricks) resulting in increased test scores that are inaccurately high assessments of ability. The first two if realized would be beneficial from the standpoint of both test validity and student performance, while the third is only incidental with professionally made tests. A framework for gauging the practical utility of coaching effects is presented and several ostensible yardsticks of coaching impact are discounted as largely irrelevant.
THE GRADUATE MANAGEMENT ADMISSION TEST PREDICTION BIAS STUDY
Authors: Braun Henry T. , Jones Douglas H. , Rubin Donald B. , Thayer Dorothy
Published: August 2014

This document presents the results of an investigation into the prediction of first‐year average in graduate management school based on undergraduate grade point average and GMAT scores. It is found that the use of a single prediction equation for both black students and white students tends to produce over predictions of the former's graduate school performance. Substantial effort is devoted to developing a prediction system which takes into account ethnic status. This approach is made practicable by employing the empirical Bayes estimation paradigm, and the new predictions represent a substantial improvement over the ones based on a single equation for both races.
SCORING AND RESPONSE STYLE ON WRITTEN SIMULATION PROBLEMS
Authors: Alwood James E. , Scheuneman Janice Dowd
Published: August 2014

Written Simulation Problems are a form of examination which offers an alternative to multiple choice testing for the evaluation of problem solving ability in fields such as medicine or other health professions. Although these problems have been used for assessment and evaluation for over a decade, relatively little attention has been paid to the psychometric properties of the scores derived from this type of examination. Of particular concern in this study are those scores derived for problem sections in which the examinee controls the number of options to be selected. Eight different procedures for scoring these sections were examined with respect to the impact of different types of error on the score, the susceptibility of the scoring procedure to response tendencies toward selecting many rather than few options, and the effects on the score of different examinees responding to different sections of the problem. Reliability of the scores was also determined. Results showed that the scores were generally adequate measures if the problems were balanced with regard to positive and negative options. With increasing imbalance, however, several of the scores became unsatisfactory, including some of the more commonly recommended. Suggestions for improved design and analysis of the problems were also made.
UNBIASED PREDICTIONS IN SPARSE DATA PROBLEMS
Authors: Braun Henry I. , Jones Douglas H.
Published: August 2014

This paper presents a brief synopsis of some recent work of the authors (Braun and Jones, 1981) in connection with the use of regression methods to predict performance in graduate schools of management on the basis of undergraduate grade point average (UGPA) and the Graduate Management Admission Test (GMAT) score. The central problem is how to provide stable unbiased predictions for identifiable groups of candidates whose members may be sparsely distributed across the institutions under study. It is shown how empirical Bayes methods provide a workable solution even when classical procedures can not be implemented.
MAXIMUM LIKELIHOOD ESTIMATION OF THE JOINT COVARIANCE MATRIX FOR SECTIONS OF TESTS GIVEN TO DISTINCT SAMPLES WITH APPLICATION TO TEST EQUATING*
Authors: Thayer Dorothy T.
Published: August 2014

Consider an old test X consisting of s sections and two new tests Y and Z similar to X consisting of p and q sections respectively. All subjects are given test X plus two variable sections from either test Y or Z. Different pairing of variable sections are given to each subsample of subjects. We present a method to estimate the covariance matrix of the combined test (X1, …, Xs, Y1, …, Yp, Z1, …, Zq) and describe an application of these estimation techniques to linear, observed‐score, test‐equating.
STATISTICAL BIAS IN MAXIMUM LIKELIHOOD ESTIMATORS OF ITEM PARAMETERS
Authors: Lord Frederic M.
Published: August 2014

Formulas are derived for the bias in the maximum likelihood estimators of the item parameters in the logistic item response model when examinee abilities are known. Numerical results are given for a typical verbal test for college admission.
DIFFERENTIAL TRENDS IN LAW GRADES OF MINORITY AND NONMINORITY LAW STUDENTS
Authors: Powers Donald E.
Published: August 2014

Three‐year trends in the law averages awarded to nonminority students and to two principal minority groups—Black students and Chicano students—were studied in two sets of law schools that enrolled more than 900 Black students and more than 300 Chicano students. In 18 of 21 law schools, Black students showed greater improvement than White students when third‐year grades were compared with first‐year grades. In 10 of 21 schools the improvement of Black students was significantly greater, statistically, than that of White students. Chicano students, who generally showed less prior educational disadvantagement than Black students, exhibited differential improvement in six of nine schools, but a statistically significant improvement in only one school.

Various reasons for the differential improvements in grades are considered, including statistical artifacts (ceiling and floor effects, changes in criterion reliability, and unequal units of measurement in the grading of minority and nonminority students) and substantive factors such as course selection patterns, differences in the academic demands of each law school year, and support systems available to disadvantaged students. The implications of the findings are discussed for counseling students, for academic dismissal and probation policies, and for the study of effective institutional support services.
HOW ARE WE DOING?: A REVIEW OF SOCIAL INDICATORS III
Authors: Wainer Howard
Published: August 2014

The aim of the international social indicators movement is to gather and disseminate data on a variety of variables that will track the social trends of our civilization, much as economic indicators track economic trends.Social Indicators IIIis the third in a series of compendia of statistical information selected and organized to depict important aspects of our current social situation and the underlying trends and developments. This paper reviews this volume.
UNDERGRADUATE DEBT AND PARTICIPATION IN GRADUATE EDUCATION: THE RELATIONSHIP BETWEEN EDUCATIONAL DEBT AND GRADUATE SCHOOL ASPIRATIONS, APPLICATIONS, AND ATTENDANCE AMONG STUDENTS WITH A PATTERN OF FULL‐TIME, CONTINUOUS POSTSECONDARY EDUCATION
Authors: Ekstrom Ruth , Goertz Margaret , Pollack Judith , Rock Donald
Published: August 2014

This study explored the relationship between undergraduate educational loan indebtedness and graduate school aspirations, application, enrollment, and persistence, using data from the National Longitudinal Study of 1972 and from the 1980 and 1982 cohorts of High School and Beyond. The analysis was limited to full‐time students who did not interrupt their postsecondary education. The results show that aspirations and application for graduate education were slightly higher for indebted than non‐indebted college seniors in 1976 and 1984 and essentially equal for these two groups in 1986. Slightly more college graduates with educational debt than without enrolled in graduate or professional school in 1976 and 1984. Regression analyses showed little unique contribution of debt level to graduate aspirations, application, enrollment and persistence after controlling for factors such as background and differences in undergraduate educational experience.
FOUNDATIONS OF A NEW TEST THEORY
Authors: Mislevy Robert J.
Published: August 2014

It is only a slight exaggeration to describe the test theory that dominates educational measurement today as the application of twentieth century statistics to nineteenth century psychology. Sophisticated estimation procedures, new techniques for missing‐data problems, and theoretical advances into latent‐variable modeling have appeared—all applied with psychological models that explain problem‐solving ability in terms of a single, continuous variable. This caricature suffices for many practical prediction and selection problems because it expresses patterns in data that are pertinent to the decisions that must be made. It falls short for placement and instruction problems based on students' internal representations of systems, problem‐solving strategies, or reconfigurations of knowledge as they learn. Such applications demand different caricatures of ability—more realistic ones that can express patterns suggested by recent developments in cognitive and educational psychology. The application of modern statistical methods with modern psychological models constitutes the foundation of a new test theory.
VALIDATING THE GRE ANALYTICAL ABILITY MEASURE AGAINST FACULTY RATINGS OF ANALYTICAL REASONING SKILLS
Authors: Enright Mary K. , Powers Donald E.
Published: August 2014

The aim of this study was to develop a criterion of graduate school success as an alternative to first‐year average. More specifically, faculty rating scales of students' analytical abilities were developed as a potential criterion against which to validate both the current Graduate Record Examinations (GRE) analytical measure and future modifications of it.

The rating instrument was based on previous research (Powers & Enright, 1986, 1987), which identified a number of independent dimensions underlying faculty perceptions of the importance of a wide variety of reasoning skills. The instrument included six separate scales for faculty to rate individual students with respect to their skills in analyzing arguments, drawing inferences, defining problems, reasoning inductively, and generating alternatives, as well as their overall analytical style.

The rating scales were completed by faculty members in a sample of 24 graduate departments representing six disciplines. Three important results have implications for the use of faculty ratings as a criterion of success. First, faculty raters were not able to distinguish among students on the six individual scales, which exhibited very high intercorrelations. This suggests that the rating instrument could be simplified for future use.

Secondly, although the ratings and first‐year grades were highly correlated, indicating that both criteria reflect success in graduate school, evidence that ratings and first‐year grades measure somewhat different aspects of success in graduate school was also found. Each of the three GRE General Test measures–verbal, quantitative, and analytical–was more highly correlated, on the average, with ratings than with first‐year averages. Undergraduate grades, on the other hand, correlated better with first‐year grades than with the ratings.

Finally, results were mixed with respect to the validity of faculty ratings of students' analytical abilities. When the three GRE measures were ranked with respect to their predictive effectiveness for each department, the analytical measure was significantly more often the best or second best predictor of faculty ratings than of first‐year average, while the verbal and quantitative measures tended to be the best predictors about equally often for ratings and grades. This suggests that the ratings may be more reflective of analytical ability than of verbal or quantitative ability. However, the verbal measure was, on average, more highly correlated with faculty ratings of students' analytical skills than was the analytical measure. This suggests that faculty ratings of students' analytical skills may have been influenced by students' verbal reasoning skills. This failure to find unequivocal evidence of discriminant validity of the ratings may reflect problems with the ratings, with the way in which faculty rated students, or with the discriminant validity of the analytical measure, A recommendation was made to continue research on the development of these scales.
AUTOMATIC SCORING OF PAPER‐AND‐PENCIL FIGURAL RESPONSES
Authors: Martinez Michael E. , Ferris John J. , Kraft William , Manning Winton H.
Published: August 2014

Large‐scale testing is dominated by the multiple‐choice question format. Widespread use of the format is due, in part, to the ease with which multiple‐choice items can be scored automatically. This paper examines automatic scoring procedures for an alternative item type: figural response. Figural response items call for the completion or modification of figural material, including illustrations, diagrams, and graphs. Twenty‐five science items were written in cooperation with the National Assessment of Educational Progress (NAEP) and printed with an ink that was invisible to scanning equipment. The items were answered with pencils; response sheets were scanned and the resulting data were processed by computer‐based scoring algorithms. The paper describes the technology that lead to successful pilot scoring of seven items for ten subjects. Implications of this technology for the future of large‐scale testing are discussed.
THE EFFECTS ON OBSERVED‐ AND TRUE‐SCORE EQUATING PROCEDURES OF MATCHING ON A FALLIBLE CRITERION: A SIMULATION WITH TEST VARIATION1
Authors: Eignor Daniel R. , Stocking Martha L. , Cook Linda L.
Published: August 2014

Two recent simulation studies were conducted to aid in the diagnosis and interpretation of equating differences found between random and matched (nonrandom) samples for four commonly used equating procedures: Tucker, Levine equally reliable, and Chained equipercentile observed‐score procedures and the 3PL IRT true‐score equating procedure. For these simulations, test forms were equated to themselves, a situation that does not pattern reality. In the current simulation, test variation was added as an additional variable for study. The results of the current simulation confirmed the results of the previous two simulations and support the prediction based on theoretical grounds that observed‐score equating methods, such as Tucker and Chained equipercentile, are more affected by sample variation than are a true‐score method (IRT) or an observed‐score method based on true‐score assumptions (Levine equally reliable). The results further suggest that matching equating samples on the basis of a fallible measure of ability, such as anchor test score, is not advisable for any equating method studied except possibly the Tucker method.
AN INVESTIGATION OF THE USE OF SIMPLIFIED IRT MODELS FOR SCALING AND EQUATING THE TOEFL TEST
Authors: Way Walter D. , Reese Clyde M.
Published: August 2014

The purpose of this study was to explore the use of two alternative item response theory estimation models in the scaling and equating of TOEFL – a modified one‐parameter model (M1PL) and a modified two‐parameter model (M2PL) – and to compare item scaling and test equating results based on these two alternative models with results based on the three‐parameter model (3PL) that is currently being used to scale and equate the TOEFL. The study employed a design in which a typical TOEFL equating was simulated using artificial data. The simulated equatings were compared in terms of correlations between estimated and generating parameters, model‐data fit, and concordance of simulated score conversions with conversions based on the generating parameters.

The results of the study clearly indicated that the 3PL model performed better than the M1PL and M2PL models on the basis of each of the evaluation criteria. There was also evidence that the M2PL model performed better than the M1PL model, particularly in terms of model‐data fit and in the weighted root mean square difference statistics used to evaluate the simulated score conversions. The results of the study also indicated that discrepancies between score conversions based on the M1PL and M2PL model and those based on the 3PL model tended to occur at the lower and upper ends of the score scales. Finally, the results of the study for the 3PL model indicated that while correlations between item parameter estimates and generating parameters tended to be affected by sample size, neither the quality of model‐data fit nor the quality of simulated equatings appeared to be sensitive to the different sample sizes used in the study.
ESSAYS AND MULTIPLE‐CHOICE TESTS AS PREDICTORS OF COLLEGE FRESHMAN GPA
Authors: Bridgeman Brent
Published: August 2014

The incremental validity of a short holistically‐scored expository essay for predicting freshman grade point average was explored in two samples. In one of the samples the essay was administered to incoming freshman at state colleges as part of a basic skills assessment battery. In the second sample the essay was part of an achievement test that is one of the admissions tests used by highly selective colleges. In both samples, the essay added essentially nothing to what could be predicted from high school grade point average, Scholastic Aptitude Test scores, and a multiple‐choice test of writing‐related skills.
DIFFERENCES AMONG RACIAL AND ETHNIC GROUPS IN MEAN SCORES ON THE GRE AND SAT: CROSS‐SECTIONAL COMPARISONS
Authors: Pennock‐Román María
Published: August 2014

The objective here is to highlight the limitations of making longitudinal inferences about racial/ethnic group differences in test scores using cross‐sectional, published data provided by the Scholastic Aptitude Test (SAT) and Graduate Record Examinations (GRE) programs. Published data from the two programs are not comparable for a variety of reasons. For example, they use different definitions of demographic group membership (e.g., citizenship status), and these differences have a large effect on the population statistics. Furthermore, one year of GRE data corresponds to many years of SAT data.

The lack of comparability of the two sets of data is illustrated by calculating mean differences between the non‐Hispanic White group and minority groups on test scores (in standard deviation units) under two conditions. First, standardized group mean differences are derived from statistics published for one year of GRE data and for one year of SAT data. Second, differences are derived after making corresponding groups in the two populations more comparable according to residency, citizenship status, and year of baccalaureate degree. Estimates of the relative distance between groups change under the two conditions.

Even after making the aforementioned corrections, the two data sets are not comparable because of possible unequal distributions of undergraduate major in the two populations and possible variability across racial/ethnic groups in the characteristics of students who choose to apply to graduate school. Thus, longitudinal changes in group differences cannot be studied using cross‐sectional comparisons of the two populations.
AN ANALYSIS OF GRADUATE SCHOOL CAREERS IN THREE UNIVERSITIES: DIFFERENCES IN ATTAINMENT PATTERNS ACROSS ACADEMIC PROGRAMS AND DEMOGRAPHIC GROUPS
Authors: Zwick Rebecca
Published: August 2014

The graduate careers of nearly 5,000 Ph.D.‐seeking students from 11 departments in each of three major universities were investigated, with a special focus on minority students. Minorities and women were found to be underrepresented in graduate school and to have generally lower candidacy and graduation rates than their White and male counterparts. In two of the three schools, foreign students had higher candidacy and graduation rates than did White Americans. Also, in two of the three schools, the percentage of foreign students increased substantially in recent years.

A more general finding was that the candidacy and graduation rates in the eight years following matriculation were higher in quantitatively oriented departments than in the humanities and social sciences. In general, undergraduate grades and Graduate Record Examinations scores had only a minimal association with the attainment of candidacy and graduation. Among these academically select students, nonacademic factors may play a crucial role in determining who ultimately attains the doctoral degree.
RESPONSE LATENCY MEASURES FOR BIOGRAPHICAL INVENTORIES
Authors: Stricker Lawrence J. , Alderton David L.
Published: August 2014

This study assessed the usefulness of response latency data for biographical inventory items in enhancing the inventory's validity. The Armed Services Applicant Profile (ASAP) was computer administered to Navy recruits, and the regular score, latency‐weighted scores, and measures of deviant latencies were obtained. The latency‐weighted scores did not improve the ASAP's validity in predicting six‐month retention, when used instead of or in addition to the regular score, and the deviant latency measures did not function as suppressor or moderator variables to increase the ASAP's validity. But subgroups of items with differing latencies varied systematically in their internal‐consistency reliability (with increased reliability for subgroups with shorter latencies), and a small subgroup of items with moderate latencies was almost as valid as the regular score, suggesting that latency data may be useful in writing and selecting inventory items.
NEW VIEWS OF STUDENT LEARNING: IMPLICATIONS FOR EDUCATIONAL MEASUREMENT
Authors: Masters Geofferey N. , Mislevy Robert J.
Published: August 2014

Recent research in cognitive psychology has drawn attention to the important role that students' personal understandings and representations of subject matter play in the learning process. This paper briefly reviews some of this research, and contrasts the kind of learning that results in an individual's changed conception or view of a phenomenon with the more passive, additive kind of learning assessed by most traditional achievement tests. To be consistent with a view of learning as an active, constructive process, educational tests are required which focus on key concepts in an area of learning, and which take into account the variety of types and levels of understanding that students have of those concepts. In these tests, scoring responses right and wrong is likely to be less appropriate than using students' answers to infer their levels of understanding. This will require not only imaginative new types of test items, but statistical models that permit inferences about students' understandings once their responses have been observed. Psychometric approaches are sketched to construct measures of achievement from such tests.
SEMANTIC AND STRUCTURAL FACTORS AFFECTING THE PERFORMANCE OF MATCHED BLACK AND WHITE EXAMINEES ON ANALOGY ITEMS FROM THE SCHOLASTIC APTITUDE TEST
Authors: Freedle Roy , Kostin Irene
Published: August 2014

Analogy items from the Scholastic Aptitude Test (SAT) were evaluated for differential performance by Black and White examinees. Black and White examinees were first matched for overall SAT‐V scores prior to conducting item analyses. A content and psycholinguistic analysis of 220 disclosed SAT analogy items was performed. Regression analyses indicate that Black examinees consistently perform differentially better than matched White examinees on the hard analogy items! However, for easy items, particularly those that involve “science” content, White examinees appear consistently to perform differentially better than matched Black examinees. In addition, semantic relationships dealing with part/whole relationships in the item stem also contributed negatively to Black examinee percent correct responses. Three variables (item difficulty, science content, part/whole relationship) together account for 30% of the variance between the two ethnic (Black and White) groups. Of these three significant predictors, two are semantic (part/whole and science content) while the third (item difficulty) reflects a non‐semantic factor. Several hypotheses are advanced to explain these findings.
THE PREDICTION OF SAT READING COMPREHENSION ITEM DIFFICULTY FOR EXPOSITORY PROSE PASSAGES
Authors: Freedle Roy , Kostin Irene
Published: August 2014

The primary goal of this project was to examine the predictability of SAT reading item difficulty (equated delta) for main idea items, and collectively, the predictability of three major reading item types: main idea, inference and explicit statement items. A secondary purpose in predicting item difficulty was to contrast the responses of high verbal and low verbal ability examinees. Primary attention was paid to studying 110 main idea reading items and their associated passages. However, additional results are reported for 285 reading items taken from 34 disclosed SAT forms which represented a wider range of reading item types.

The percent variance of main idea item difficulty accounted for varied from 46% to 59% depending upon the particular analysis. The predictability of all three reading item types (n = 285) varied from 21% to 29%, depending upon the particular analysis.

Details of item predictability were explored by evaluating several hypotheses. Results indicated that (1) multiple‐choice reading items are sensitive to variables similar to those reported in the experimental literature on comprehension, (2) many of these variables provide significant independent predictive information in regression analyses, (3) the placement (early versus middle of text) of relevant main idea information affects item difficulty, and (4) considerable agreement between SAT and GRE reading predictability was found. Additional results contrast the performance of high and low ability groups.
CROSS‐VALIDATION OF A PROPORTIONAL ITEM RESPONSE CURVE MODEL
Authors: Boldt Robert F.
Published: August 2014

In a previous research study, factor analyses of TOEFL section item correlations (phi‐coefficients) yielded a single factor in each section. This result is consistent with the assumption that, within sections, the item response curves are proportional, a possibility not previously suggested or explored in item response theoretical studies. One potential use to the TOEFL program is that this assumption could serve as a basis for simpler equating methods than are now being used. Use of the assumption entails estimating fewer parameters and the computations could be simpler than with current methods. Other anticipated benefits are that it could reduce the chances of error in calibrating items and would permit the use of smaller equating samples, which would in turn allow pretest evaluation of substantially more items for possible operational use.

To increase the body of data evaluating the model, a cross‐validation study was undertaken in which PIRC, 3PL, and a modified Rasch model were used to predict item scores of selected examinees on selected items. The study also compared predicted half‐test scores with actual scores for each method. These comparisons were made using varying amounts of data to calibrate the items and calculate scores for the examinees in accordance with different estimation sample sizes. Surprisingly, the accuracy of predictions made by the models was approximately the same, and the estimation sample size appeared to make little difference. Further research in terms of randomization trials on the current data set and self‐equatings similar to those conducted by Hicks (1984) is discussed.
A FRAMEWORK FOR STUDYING DIFFERENCES BETWEEN MULTIPLE‐CHOICE AND FREE‐RESPONSE TEST ITEMS
Authors: Mislevy Robert J.
Published: August 2014

This paper lays out a framework for comparing the qualities and the quantities of information about student competence provided by multiple‐choice and free‐response test items. After discussing the origins of multiple‐choice testing and recent influences for change, the paper outlines an “inference network” approach to test theory, in which students are characterized in terms of levels of understanding of key concepts in a learning area. It then describes how to build inference networks to address questions concerning the information about various criteria conveyed by alternative response formats. The types of questions that can be addressed in this framework include those that can be studied within the framework of standard test theory. Moreover, questions can be asked about generalized kinds of reliability and validity for inferences cast in terms of recent developments in cognitive psychology.
ALTERNATIVE MATHEMATICAL APTITUDE ITEM TYPES: DIF ISSUES
Authors: Schmitt Alicia P. , Crone Carolyn R.
Published: August 2014

Alternative mathematical items administered as prototypes at the Spring 1989 Field Trials are evaluated for differential item functioning (DIF) and differential speededness. Results for Algebra Placement (AP) and Student Produced Response (SPR) items are presented and contrasted to results obtained on the two current SAT‐Math items: Regular Math and Quantitative Comparison. Analyses on comparisons between female and comparable male examinees, and between Asian‐American, Black, and Hispanic examinees in comparison to comparable White examinees indicate that both of these alternative items appear to have DIF. Additional DD? analyses comparing the use of an internal versus an external matching criteria for the SPR items show evidence of negative DIF with either criteria. Results using the MH D‐DIF statistic are more extreme than DIF results using the STD P‐DIF index. The metric used to calculate the DIF indices may be accountable for the differences observed. Differential speededness results indicate that the two Math prototypes have slightly higher levels of differential speededness than the SAT‐M.

The SPR items pose an interesting problem for DIF. The definition of an appropriate DIF matching criterion for constructed response item types needs more study. Metric differences between methods and their effect on difficult or easy items also needs further exploration. Until these methodological issues are resolved, results of DD? studies on constructed response items should be interpreted with caution.
ON THE MEANINGS OF CONSTRUCTED RESPONSE
Authors: Bennett Randy Elliot
Published: August 2014

This paper explores the multiple meanings of constructed response. The term is conceptualized as a superordinate category subsuming a variety of item formats ranging from minor adaptations of multiple choice to “authentic” assessment tasks. The meaning of these formats is explored through four validity facets: construct validity, relevance/utility, value implications, and social consequences. This exploration leads to a multi‐dimensional structure for conceptualizing constructed‐response formats. From this conception, it is argued that appreciating the term's multiple meanings broadens the intellectual and political debate, identifies gaps in our knowledge base, and suggests a balanced approach to educational assessment.