bluuebunny/crossref_metadata_2025_split huggingface dataset:
A COMPARISON OF FREE‐RESPONSE AND MULTIPLE‐CHOICE QUESTIONS IN THE ASSESSMENT OF READING COMPREHENSION
Authors: Ward William C. , Dupree David , Carlson Sybil B. 
Published: September 2014

ABSTRACTUndergraduate students completed a Reading Comprehension test in which each of a number of passages served as the basis for four to eight questions, half framed in multiple‐choice format and half requiring that the examinee produce an appropriate answer. Questions within a passage were balanced according to the kind of information processing they were thought to require: Explicit questions dealt with information given explicitly in the passage; Inference questions required inference based on material presented; and Application and Evaluation questions required that the examinee go beyond what was presented to consider applications of ideas presented or evaluation of the logic or the style of the passage. It was hypothesized that questions of the first type would show no systematic differences associated with the format of the question, while those of the second might, and those of the third would, draw on somewhat different cognitive abilities when examinees were required to produce rather than merely to recognize an appropriate answer.Correlational and factor analyses provided only minimal indications of systematic differences associated with the response format. Estimated true correlations across formats averaged .91, a relation of the same magnitude as that found when multiple‐choice Reading Comprehension items differing in their information processing requirements are contrasted with one another. Weak evidence for a factor loaded by scores based on performance on multiple‐choice but not free‐response items was found, but only one score had a loading as great as .3 on this factor.Relations of test scores with other cognitive measures also failed to show systematic differences associated with format. Regardless of differences in item type and response format, all scores based on the Reading Comprehension test showed fairly substantial relations with tests of Reasoning and Vocabulary and with the ACT English achievement test, moderate relations with the ACT mathematics achievement score, and near‐zero relations with a test of Divergent Thinking.Despite this evidence for similarity in performance across formats, however, there is also evidence of systematic differences for individuals. The subjects by format interaction in the analysis of variance, while not of great strength, was significant at the five percent level of confidence. This effect indicates that some students showed larger differences in performance between the two response formats than did others.It was speculated that two changes in test items might be required to elicit the differences in performance associated with response format that have been observed in some other testing situations. Items may need to be structured so that they draw less on information given in or derivable from the test materials and more from information whose relevance is not specified for the examinee; and types of questions with which examinees are not so well practiced may need to be employed.
EXPLORING THE USE OF IRT EQUATING FOR THE GRE SUBJECT TEST IN MATHEMATICS
Authors: McKinley Robert , Kingston Neal 
Published: September 2014

ABSTRACTA study was conducted to investigate the feasibility of using IRT equating for the GRE Subject Test in Mathematics. Two forms of the test were equated using the three‐parameter logistic (3PL) model, and the results were compared to the results of the Tucker equating procedure currently used operationally, as well as to equipercentile equating. In addition, the two basic assumptions of the 3PL model were investigated to determine whether they were reasonable within the context of the GRE Mathematics Test. These assumptions were that of unidimensionality, which was investigated using nonlinear, full‐information factor analysis, and that the item response function follows a logistic form with three parameters, which was investigated using two different goodness‐of‐fit procedures.The results of the study indicate that the GRE Mathematics Test is reasonably unidimensional, and that the three‐parameter logistic model does yield reasonable fit to the data. It was found that both the IRT and the equipercentile equating procedures yielded somewhat different results than the Tucker method. The results for the IRT procedure and the equipercentile procedure were quite similar.
SYNTACTIC COMPLEXITY AND PSYCHOMETRIC DIFFICULTY: A PRELIMINARY INVESTIGATION
Authors: Bejar Isaac I. , Stabler Edward P. , Camp Roberta 
Published: September 2014

ABSTRACTAn earlier investigation (Bejar, 1983) had argued that experts' judgment of item difficulty could perhaps be usefully supplemented with linguistic information about the sentence from which the item was derived. To investigate that idea, we analyzed items from the earlier study to determine their syntactic structure. Three potential independent variables were studied by themselves and in conjunction with subject‐matter ratings. The analysis suggested that the combination of experts' judgments and syntactic information about the sentence on which the item was based collectively predicted difficulty better than either judgment or syntactic information alone. Moreover, the proportions of variance in item difficulty accounted for by the judgments and syntactic information was 31%.
TEST CONSTRUCTION MANIPULATING SCORE DIFFERENCES BETWEEN BLACK AND WHITE EXAMINEES: PROPERTIES OF THE RESULTING TESTS
Authors: Hackett Rachelle Kisst , Holland Paul W. , Pearlman Mari , Thayer Dorothy T. 
Published: September 2014

ABSTRACTTest forms for two item types were developed that maximized or minimized the differences in average scores between Black and White examinees. The test forms were then administered to random samples of examinees along with conventionally constructed control forms. The results suggest that impact manipulation is possible while maintaining content specifications and average difficulty level. However, the distribution of item difficulties was altered as were the reliability of the experimental sections and the relationships of these sections to other parts of the test.
ONE‐STAGE LOGIST1
Authors: Wingersky Marilyn S. 
Published: September 2014

ABSTRACTApplications of item response theory frequently require estimates of only the item parameters. In this paper, a reparameterization of the logistic model is developed which involves substituting an implicit function of the number‐right true score for ability. The number‐right true score needs only to be computed once since the observed number‐right score is a good estimator of it. This reparameterization avoids estimating each examinee's ability in every stage of the estimation procedure. An estimation method containing this reparameterization was devised and incorporated into a modified version of the LOGIST computer program. The procedure, called One‐Stage, was tried out on artificial and real data. The results on the artificial data were compared to the true values and to the estimates from the operational LOGIST program. The results on the real data were compared to the estimates from the operational LOGIST program.In the comparisons of the parameter estimates from One‐Stage and LOGIST to the true values for the artificial data, One‐Stage did not estimate the item parameters as well as LOGIST. One potential benefit of this procedure, the reduction in computer costs due to not estimating an ability for each examinee in each stage, was not realized when the procedure was applied to real data with a lot of not reached items.
ANALYZING THE OPTION EFFECTS OF DIFFICULT TOEFL ITEMS WITH LOW BISERIALS: METHODS DEVELOPED FOR USE BY TEST ASSEMBLERS
Authors: Hicks Marilyn M. 
Published: September 2014

ABSTRACTSeveral exploratory analyses of the fifths data generated by TOEFL item analyses were developed in order to evaluate the effects of options on the discriminability of difficult items, and to identify difficult items with low, unreliable biserials which have been rejected by Test Development but for which acceptable a‐parameters are probably estimable. Intended for use by test assemblers subsequent to an item analysis, the methods were mainly graphical, but also included the evaluation of a distance measure and other simple statistics.An effective distracter has the property that examinees are attracted to it in inverse order of ability. To the extent that this ordering is violated for certain ability levels, localized option effects occur which can impair item discrimination as well as the fit of the IRT model. The negative impact of these effects on model fit was illustrated, and methods for analyzing them were suggested. If item writers could account for the factors underlying the interaction between ability level and option responses, it might be possible to modify options accordingly, thereby improving the measurement effectiveness of the item. Departing from the usual reliance on a single index, the approaches in these analyses included, among other things, an evaluation of the biplot generated from a correspondence analysis of the matrix of fifths information, and an analysis of the total option response configuration. Many examples of these analyses were provided.A significant limitation of the r‐biserial for very difficult items which restricts the ability of test assemblers to construct tests with effective measurement properties at high score levels was illustrated. The index developed in this study to identify such items is regarded as an interim strategy until a conventional measure of item discrimination which is optimal over the entire scale of difficulty is developed, a current critical need.The implications of introducing other dimensions into the test by items with nonmonotonic response patterns due to option effects was briefly discussed. It is possible that application of the procedures developed in the study might provide a method of excercising control over the dimensionality of the measuring instrument at the practical level of item construction.
AGREEMENT BETWEEN EXPERT SYSTEM AND HUMAN RATINGS OF CONSTRUCTED‐RESPONSES TO COMPUTER SCIENCE PROBLEMS
Authors: Bennett Randy Elliot , Gong Brian , Kershaw Roger C. , Rock Donald A. , Soloway Elliot , Macalalad Alex 
Published: September 2014

ABSTRACTIf computers can be programmed to score complex constructed response items, substantial savings in selected ETS programs might be realized and the development of mastery assessment systems that incorporate “real‐world” tasks might be facilitated. This study investigated the extent of agreement between MicroPROUST, a prototype microcomputer‐based expert scoring system, and human readers for two Advanced Placement Computer Science free‐response items. To assess agreement, a balanced incomplete block design was used with two groups of four readers grading 43 student solutions to the first problem and 45 solutions to the second. Readers assigned numeric grades and diagnostic comments in separate readings. Results showed MicroPROUST to be unable to grade a significant portion of solutions, but to perform impressively on those solutions it could analyze. For one problem, MicroPROUST assigned grades and diagnostic comments similar to those assigned by readers. For the other problem, MicroPROUST's agreement with readers on grades was lower than the agreement of readers among themselves, its grades were higher, and it gave fewer comments, particularly on structure and style. The extent of disagreement on grades, however, was small and much of the disagreement disappeared when papers were rescored discounting style. MicroPROUST's interchangeability with human readers on one problem suggests that there are conditions under which automated scoring of complex constructed‐responses might be implemented by ETS.
THE STANDARDIZATION APPROACH TO ASSESSING DIFFERENTIAL SPEEDEDNESS
Authors: Dorans Neil J. , Schmitt Alicia P. , Bleistein Carole A. 
Published: September 2014

ABSTRACTAn adaptation of the standardization approach to assessing differential item functioning that applies to all item responses, including omits and not reached, is described. Applications of this method to evaluate differential speededness show that there is evidence of differential speededness for Blacks and Hispanics, but not for Asian‐Americans. There may be a dependency between differential speededness and test section location: Differential speededness may be more noticeable when the test section is located at the beginning of the test. Implications of these findings for evaluations of content‐related differential item functioning and on differential test‐taking strategies are described.
DIFFERENTIAL ITEM FUNCTIONING FOR MINORITY EXAMINEES ON THE SAT1,2
Authors: Schmitt Alicia P. , Dorans Neil J. 
Published: September 2014

ABSTRACTThe standardization approach to assessing differential item functioning (DIF), including standardized distractor analysis, is described. The results of studies conducted on Asian‐Americans, Hispanics (Mexican‐Americans and Puerto Ricans), and Blacks on the Scholastic Aptitude Test are described and then synthesized across studies. Where the groups were limited to include only examinees who spoke English as their best language, very few items across forms and ethnic groups exhibited large DIF. Major findings include evidence of differential speededness for Blacks and Hispanics, and when the item content is of special interest, advantages for the relevant ethnic group. In addition homographs tend to disadvantage all three ethnic groups, but the effect of vertical relationships are not as consistent. Although these findings are important in understanding DIF they do not seem to account for all differences. Other variables related to DIF still need to be identified. Furthermore, these findings are seen as tentative until corraborated by studies using controlled data collection designs.
FACTORS AFFECTING THE SAMPLE INVARIANT PROPERTIES OF LINEAR AND CURVILINEAR OBSERVED‐ AND TRUE‐SCORE EQUATING PROCEDURES1,2
Authors: Stocking Martha L. , Eignor Daniel R. , Cook Linda L. 
Published: September 2014

ABSTRACTA sequence of simulations was carried out to aid in the diagnosis and interpretation of equating differences found between random and matched (nonrandom) samples for four commonly used equating procedures: Tucker linear observed‐score equating, Levine equally reliable linear observed‐score equating, Equipercentile curvilinear observed‐score equating, and IRT curvilinear true‐score equating. The results support the prediction based on theoretical grounds that observed‐score equating methods are more affected by sample variation than are true‐score equating methods. These results further suggest that matching equating samples on the basis of fallible measures of ability may not be advisable for any conventional equating method except the Tucker method. In addition, the results support a particular hypothesis about IRT equating, suggesting that the use of matched samples cannot be recommended for this equating method either.
A STUDY OF THE DIFFERENTIAL IMPACT OF CURRICULUM ON APTITUDE TEST SCORES
Authors: Angoff William H. , Johnson Eugene G. 
Published: September 2014

ABSTRACTA sample of 22,923 students who had taken the GRE General Test in the academic years 1983‐84 and 1984–85 and who had also taken the SAT four or five years earlier were found, and classified by undergraduate field of study (four major categories of curriculum) and sex. Several analyses were undertaken to determine the degree of differential impact that sex and field of study might have on GRE‐verbal, GRE‐quantitative, and GRE‐analytical scores, after controlling on SAT‐verbal and SAT‐mathematical scores. It was found, first, that the correlations of SAT‐verbal with GRE‐verbal and SAT‐mathematical with GRE‐quantitative were extremely high, both for the entire sample, and within it, for the eight subgroups defined by field of study and sex. The correlations were .86 in the total sample and ranged from the low to middle .80s in the eight subgroups. The impact of curriculum and sex was found to be low on GRE‐verbal scores, but relatively high for GRE‐quantitative, with students in heavily quantitative fields enjoying an advantage over their peers in less quantitative fields of study. The impact was moderate for GRE‐analytical. Further studies designed to “purify” the fields of study and include only clearly verbal fields and clearly mathematical fields–omitting entirely students in social and biological science–showed small additional impact. An additional study indicated that there was a generally slight effect of the institution attended on GRE‐quantitative scores, after controlling for major field of study and initial ability, although the importance of institution attended was somewhat greater for higher ability students. Although these studies helped a bit to clarify the results, the basic conclusions remained unchanged.In a separate phase of the study an attempt was made by means of Mantel‐Haenszel analyses to identify the kinds of items that were relatively resistant to curricular and sex effects. Although the items differed from one another with respect to impact, they did not fall into identifiable categories that would make it possible to predict which items would be likely to show such impact and which would not.
THE INFLUENCE OF READING PATTERNS ON THE PROFICIENCIES OF YOUNG ADULTS
Authors: Kirsch Irwin S. , Mosenthal Peter B. , Rock Donald A. 
Published: September 2014

ABSTRACTUsing data collected by the National Assessment of Educational Progress (NAEP) through its literacy assessment of young adults, this study addressed the question, Do adult patterns of reading activity relate to demonstrated proficiencies? Five reading patterns were developed that represent various combinations of high and low activity with newspapers, magazines, books, and brief documents. These patterns were found to be related to performance on the three NAEP literacy scales — prose, document, and quantitative — for the total population and for major subgroups as well. Through regression analyses, these patterns were shown to have importance in explaining the literacy performance levels of America's 21‐ to 25‐ year olds. These findings are discussed in the context of recent cognitive research that emphasizes the importance of graded contextual practice in the transition from novice to expert status.
MEASUREMENT SCIENCE AND TRAINING
Authors: Bunderson C. Victor 
Published: September 2014

ABSTRACTThe need for training and retraining is a central element in current discussions about the economy of the United States. This paper was designed to introduce training practitioners to some new concepts about how measurement science can provide a new framework for assessing progress and can add new discipline to the development, implementation, and conduct of training. The paper is a discussion‐focusing chapter in a forthcoming ASTD‐sponsored book in which other chapters are written by training practitioners. The paper demonstrates that measurement science, revitalized by interactive technologies and the disciplines of cognitive science, instructional science, applied artificial intelligence, and studies of organizational change, can be used to address the economic challenge our nation currently faces. Basic concepts of measurement science as applied to training and performance are described, and newer concepts, such as mastery assessment systems and the four generations of computerized measurement, are explored to demonstrate their applicability to training and performance measurement. A hypothetical future training application, making use of the advantages offered by advances in measurement science, is described.
ADJUSTING SCORES ON EXAMINATIONS OFFERING A CHOICE OF ESSAY QUESTIONS
Authors: Livingston Samuel A. 
Published: September 2014

ABSTRACTWhen test‐takers are offered a choice of essay questions, some questions may be harder than others. If the test includes a common portion taken by all test‐takers, an adjustment to the scores is possible. Previously proposed adjustment procedures disregard the test‐makers' efforts to create questions of equal difficulty; these procedures tend to make larger adjustments when the scores to be adjusted are less correlated with scores on the common portion. This paper suggests an adjustment procedure that makes smaller adjustments when the corelation between the scores to be adjusted and the scores on the common portion is weak. The paper includes a derivation of the adjustment formula and a numerical example of the resulting adjustment.
DIFFERENTIAL ITEM FUNCTIONING FOR MALES AND FEMALES ON SAT‐VERBAL READING SUBSCORE ITEMS: FOLLOW‐UP STUDY1,2
Authors: Lawrence Ida M. , Curley W. Edward 
Published: September 2014

ABSTRACTThis study was conducted to attempt to verify findings from an earlier study by Lawrence, Curley, and McHale (1988) in which DIF was examined for females on Reading subscore items in four forms of SAT‐Verbal. Specifically, the focus of this research was on verifying the technical science hypothesis as a contributing factor to DIF for females on science reading comprehension questions, and the true‐science classification as a contributing factor to DIF for females on sentence completion items. Confirmatory evidence for the factors identified in the earlier study is not clear‐cut in this study. For reading comprehension items, the presence of technical science material in a reading passage tends to make the corresponding items more difficult for females. However, some items associated with highly technical passages do not function differently for males and females. With respect to sentence completion items, there was partial support for the hypothesis that true‐science items are more difficult for females. However, the limited number of items studied does not warrant statements about the differential functioning of sentence completion items.