bluuebunny/crossref_metadata_2025_split huggingface dataset:
SIGNIFICANT IMPROVEMENTS TO LOGIST*
Authors: Wingersky Marilyn S. 
Published: August 2014

ABSTRACTStocking (1989) discovered a problem with the LOGIST estimation procedure. The automatic procedure in this program produced different item parameter estimates when the true item discriminations were used as starting values for the iteration procedure than when the default starting value of one for the item discriminations was used. When a straight run to convergence was performed, the different initial starting values converged to the same item parameter estimates.This study investigated several methods for improving the automatic procedure. When these methods failed to give the improvement necessary, several methods were tried to obtain the same results as the run to convergence in a shorter amount of computer time. A method was devised that takes much less time and gives nearly as good, and in some cases better, estimates as estimates obtained from running to convergence. This method involves adding a step to get better initial item parameter estimates for the automatic procedure. In this step, the abilities are grouped very coarsely and the item parameters and the grouped abilities are estimated iteratively, alternating back and forth between items and abilities until the maximum difference between the estimated item characteristic curves is less than some criterion.The new procedure gives item parameter estimates that are closer to the true values than the current 4‐step method does. However, there is a definite nonlinear relationship between the estimated item parameters for the two methods after the parameters have been linearly transformed to the same scale. Consequently, in an ongoing series of calibrations, switching to this procedure from the old procedure will produce a discontinuity in the parameter estimates in the same manner as would be caused by switching from LOGIST to BILOG.The effects of putting a beta prior on c were also investigated. The results were not conclusive. An option to put a beta prior on c was added to the program.This new method has been incorporated into a new version of LOGIST called LOGIST7.
COMPARISON OF EFFICIENCY OF JACKKNIFE AND VARIANCE COMPONENT ESTIMATORS OF STANDARD ERRORS
Authors: Longford Nicholas T. 
Published: August 2014

ABSTRACTLarge scale surveys usually employ a complex sampling design, and as a consequence no standard methods for estimation of the standard errors associated with the estimates of population means are available. Resampling methods, such as jackknife or bootstrap are often used, with reference to their properties of robustness and reduction of bias. We examine a method based on variance component models as an alternative to the jackknife procedure used for calculation of the standard errors for the subpopulation means of proficiency scores in a large scale survey of education in the U.S.A.
PLACEMENT VALIDITY OF A PROTOTYPE SAT WITH AN ESSAY
Authors: Bridgeman Brent , Hale Gordon A. , Lewis Charles , Pollack Judith , Wang Ming‐mei 
Published: August 2014

ABSTRACTGrades in college freshman English composition courses were predicted from high school rank in class, multiple‐choice writing scores, essays, current SAT‐Verbal scores, and scores from a revised version of SAT‐Verbal. Data were obtained from 21 English courses at 17 different colleges with some supplementary data provided by an additional college. In general, a writing composite score consisting of essay and multiple‐choice writing scores appeared to outperform current or revised SAT‐Verbal scores; validity coefficients were as high or higher for the writing composite score, and the underprediction of the grades of women students was reduced. The best predictions were obtained from the combination of high school rank with the writing composite score.
AN EXPLORATORY STUDY OF CHARACTERISTICS RELATED TO IRT PARAMETER INVARIANCE WITH THE TEST OF ENGLISH AS A FOREIGN LANGUAGE
Authors: Way Walter D. , Carey Patricia A. , Golub‐Smith Marna L. 
Published: August 2014

ABSTRACTIRT equating methods have been used successfully with the TOEFL® test for many years, and for the most part the observed properties of items have been consistent with model predictions. However, items that do not appear to hold their IRT pretest estimates do exist. If relationships can be found between features of TOEFL items in pretest calibrations and subsequent lack of model‐data fit when these items are used in final forms, steps to eliminate the use of such items in TOEFL final forms can be taken. The purpose of this study was to provide an exploratory investigation of item features that may contribute to a lack of invariance of TOEFL item parameters.The results of the study indicated the following: (1) subjective and quantitative measures developed for the study provided consistent information related to the model‐data fit of TOEFL test items, (2) for Sections 1 and 2, items that were pretested before 1986 exhibited poorer model‐data fit than items that were pretested after 1986, and (3) for Section 3 reading comprehension, model‐data fit appeared to be related to changes in the relative position of items within the sections from the pretest to the final form administrations.Based on the results of the study, it was recommended that (1) the TOEFL program investigate the feasibility of not using pretest IRT statistics for items pretested before 1986 for Sections 1 and 2 and (2) that guidelines be developed for test developers to use with reading comprehension items to limit the change in relative positions of items in the test from pretest to final form administrations.
WILL THEY THINK LESS OF MY HANDWRITTEN ESSAY IF OTHERS WORD PROCESS THEIRS? EFFECTS ON ESSAY SCORES OF INTERMINGLING HANDWRITTEN AND WORD‐PROCESSED ESSAYS
Authors: Powers Donald E. , Fowles Mary E. , Farnum Marisa , Ramsey Paul 
Published: August 2014

ABSTRACTA study was undertaken to determine the effects on essay scores of intermingling handwritten and word‐processed versions of student essays. A sample of examinees, each of whom had produced both a handwritten and a word‐processed essay, was drawn from a larger sample of students who had participated in a pilot study of item types being considered for the new academic skills assessments of The Praxis Series: Professional Assessments for Beginning Teachers™.Students' original handwritten essays were converted to word‐processed versions, and their original word‐processed essays were converted to handwritten versions. In a preliminary study, handwritten and word‐processed essays were then intermingled and rescored.Analyses revealed higher average scores for essays scored in the handwritten mode than for essays scored as word‐processed, regardless of the mode in which essays were originally produced. Several hypotheses were advanced to explain the discrepancies between scores on handwritten and word‐processed essays. The training of essay readers was subsequently modified on the basis of these hypotheses, and the experiment was repeated using the modified training with a new set of readers.The results of this second study showed an average reduction of about 25% in the discrepancy between essays read in the handwritten and word‐processed modes, compared with the results of the initial study. The effects computed in the second experiment were small by most standards, and predicted to have very little if any impact on certification decisions. Nonetheless, a recommendation is made not only to adopt the modified training but also to monitor this effect throughout the operational scoring sessions.
DEVELOPMENT AND EVALUATION OF COMPUTER‐ADMINISTERED ANALYTICAL QUESTIONS FOR THE GRADUATE RECORD EXAMINATIONS GENERAL TEST
Authors: Bridgeman Brent , Rock Donald A. 
Published: August 2014

ABSTRACTThree new computer‐administered items types for the analytical scale of the Graduate Record Examination General Test were developed and evaluated. One item type was a free‐response version of the current analytical reasoning item type. The second item type was a somewhat constrained free‐response version of the pattern identification (or number series) item type in which the student had to state the rule that generated the series. The third item type used the computer to administer yes/no analysis of explanations questions with a limited branching strategy. The computer tests were administered at four ETS regional offices to a sample of students who had previously taken the GRE General Test. Scores from the regular GRE administration and the special computer administration were matched for a sample of 349 students. A number of test administration design issues were identified, including the need to provide adequate practice exercises, design of an interface comfortable for computer‐literate students, and problems with item‐level timing. The pattern identification items were too difficult (or the practice was inadequate), but the other items appeared to function well. There was no evidence that the open‐ended analytical reasoning items were measuring anything beyond what is measured by the current multiple‐choice version of these items.
AN APPROACH TO CLASSIFYING TEACHERS ACCORDING TO THEIR ORIENTATIONS TO TEACHING
Authors: Powers Donald E. 
Published: August 2014

ABSTRACTA small, representative sample of U.S. teachers was asked to indicate the extent to which their teaching corresponded to each of five orientations to (“models” of) teaching. Respondents were then clustered according to the similarity of their responses. Four categories emerged. A major perspective that distinguished the groups was whether or not they perceived their teaching as corresponding to an “additive model,” i.e., an approach in which content coverage is emphasized. The clusters also differed with respect to subject matter taught and teachers' assessments of the importance of several teaching functions.
DIAGNOSING KNOWLEDGE STATES IN ALGEBRA USING THE RULE SPACE MODEL
Authors: Birenbaum Menucha , Kelly Anthony E. , Tatsuoka Kikumi K. 
Published: August 2014

ABSTRACTThis paper illustrates the use of rule space as a tool to support cognitive analyses of students' mathematical behavior. The rule space approach is explained and is then used to classify students into one of two methods for solving linear algebraic equations in one unknown and to diagnose their knowledge states in this topic. A 32‐item test with open‐ended questions was administered to 231 eighth and ninth graders. The following outcomes of the rule space model were presented: (a) a classification of examinees into knowledge states resulting from the two solution approaches at the group level along with individual examples; (b) tree‐diagrams of the transitional relationships among the states for each strategy. Implications for using the feedback provided by the rule space model in the context of instruction and assessment are discussed.
ADJUSTING COLLEGE GRADE‐POINT AVERAGE FOR VARIATIONS IN GRADING STANDARDS
Authors: Stricker Lawrence J. , Rock Donald A. , Burton Nancy W. , Muraki Eiji , Jirele Thomas J. 
Published: August 2014

ABSTRACTThis study compared the effectiveness of several existing and proposed methods for statistically adjusting college GPAs for course and departmental differences in grading standards, using first‐semester grades from an entire entering class at a large state university. Most of the adjusted GPAs produced by these methods functioned similarly and, despite high correlations with actual GPA, had greater internal‐consistency reliability than actual GPA and were more predictable from SAT scores and high school rank (HSR). Most of the adjusted GPAs also functioned similarly with regard to sex differences in over‐underprediction. The adjusted GPAs and actual GPA exhibited the same small but significant sex differences in over‐underprediction by SAT scores, but the adjusted GPAs displayed smaller differences than actual GPA in over‐underprediction by SAT scores and HSR.
GRAPHICAL IRT‐BASED DIF ANALYSES
Authors: Pashley Peter J. 
Published: August 2014

ABSTRACTThe detection of differential item functioning (DIF) has become an important psychometric research topic in recent years. A number of item response theory (IRT) methods for solving this problem have been suggested. A common approach is to calculate some function of the area between item response curves estimated from the subpopulations of interest. While these methods relay overall item level DIF information, they do not indicate the location and magnitude of DIF along the ability continuum. In order to provide these important details, this paper presents a method for producing simultaneous confidence bands for the difference between item response curves. After these bands have been plotted, the size and regions of DIF are easily identified. Implementation considerations and illustrative examples are also given.
CHOOSING: A TEST1
Authors: Wainer Howard , Thissen David 
Published: August 2014

ABSTRACTIf examinees are permitted to choose to answer a subset of the questions on a test, by knowing only which questions were chosen we can obtain a measure of proficiency that may be as reliable as would have been obtained from the test graded traditionally. This new method of scoring is much less time consuming and expensive for both the examinee and the testing organization. Moreover, because of the decreased response burden we would expect that its use may reduce the nonresponse rate in such low impact educational assessments as the NAEP. We recommend that in assessments that allow the examinee to choose among a set of items, an attempt be made to encourage the examinee to record the item that will be answered before they are actually attempted. In this way we may be able to obtain much of the information that would have been contained in the actual responses, even if the examinee chooses finally not to answer.
INEXPERIENCED AND ANXIOUS COMPUTER USERS: COPING WITH A COMPUTER‐ADMINISTERED TEST OF ACADEMIC SKILLS
Authors: Powers Donald E. , O'Neill Kathleen 
Published: August 2014

ABSTRACTThe objective of this study was to assess the degree to which the mode of administration of the computer‐based Academic Skills Assessments of The Praxis Series: Professional Assessments for Beginning Teachers™ contributes to performance differences among test takers. To make this determination, inexperienced or anxious computer users were recruited to take the assessments. The degree to which test design and test familiarization procedures effectively minimized variation due to comfort and familiarity with computers was examined from three perspectives:
the extent to which the availability of a personal, information‐providing test center supervisor influenced test performances, beyond the help provided by a computerized test familiarization tutorial,
the effect of within‐test practice on later performance on a subsequent section of the test, and
the relationship of computer‐based test performance to attitudes toward computers and experience in using them.
The conclusion was that performance on the tests is not unduly affected by computer administration.
“THIN” VERSUS “THICK” MATCHING IN THE MANTEL‐HAENSZEL PROCEDURE FOR DETECTING DIF
Authors: Donoghue John R. , Allen Nancy L. 
Published: August 2014

ABSTRACTThis Monte Carlo study examined strategies for forming the matching variable for the Mantel‐Haenszel DIF procedure; “thin” matching on total test score was compared to forms of “thick” matching, pooling levels of the matching variable. Data were generated using a three‐parameter item response theory model with common guessing parameter. Number of subjects and test length were manipulated, as were the difficulty, discrimination, and presence/absence of DIF in the studied item. Outcome measures were the transformed log‐odds Δ̂MH, its standard error, and the MH chi‐square statistic.For short tests (5 or 10 items), thin matching yielded very poor results, with a tendency to falsely identify items as possessing DIF against the reference group. The best methods of thick matching yielded outcome measure values closer to the expected value for non‐DIF items, and a larger value than thin matching when the studied item possessed DIF. Intermediate length tests yielded similar results for thin matching and the best methods of thick matching. The method of thick matching that performed best depended upon the measure used to detect DIF. Both difficulty and discrimination of the studied item were found to have a strong effect on the value of ΔMH.
A DIAGNOSTIC CLASSIFICATION MODEL FOR DOCUMENT PROCESSING SKILLS
Authors: Sheehan Kathleen , Tatsuoka Kikumi , Lewis Charles 
Published: August 2014

ABSTRACTThis paper introduces a modification to the Rule Space diagnostic classification procedure which allows for processing of response vectors containing missing data. Rule Space is an approach to diagnostic classification which involves characterizing examinees' performances in terms of an underlying cognitive model of generalized problem‐solving skills. It has two components: (1) a procedure for determining a comprehensive set of knowledge states, where each state is characterized in terms of a unique subset of mastered skills; and (2) a procedure for classifying examinees into one or another of the specified states. The procedure for determining a comprehensive set of knowledge states is based on the Boolean descriptive function given in Tatsuoka (1991). The procedure for classifying examinees involves comparing examinees' scored response vectors to the patterns expected within each of the specified knowledge states (Tatsuoka, 1983, 1985, and 1987). Missing data is expected to be a common problem for this approach because, although the procedure for determining the comprehensive set of knowledge states requires a large pool of items, the procedure for examinee classification can be performed with smaller (less expensive) item subsets. This approach to diagnostic classification is illustrated with data collected in the Survey of Young Adult Literacy, a nationwide survey of literacy skills conducted by the National Assessment of Educational Progress (NAEP) in 1985.
DIRECT ASSESSMENT, DIRECT VALIDATION? AN EXAMPLE FROM THE ASSESSMENT OF WRITING
Authors: Powers Donald E. , Fowles Mary E. , Willard Ann E. 
Published: August 2014

ABSTRACTComplex performance assessments are becoming increasingly popular, at least in part because of the directness with which these assessments are accomplished. As Messick (1992) has argued, however, the apparent directness of such measures does not exempt them from the need to marshal validity evidence in support of their use.An example is given of one kind of evidence – assessment users' satisfaction with the performances generated by examinees – that may contribute to the validation of performance assessments. The example is from the realm of the direct assessment of writing in the context of graduate school admissions.
REVISING SAT®‐VERBAL ITEMS TO ELIMINATE DIFFERENTIAL ITEM FUNCTIONING
Authors: Curley W. Edward , Schmitt Alicia P. 
Published: August 2014

ABSTRACTBased on initial SAT‐Verbal pretest data and/or hypotheses advanced in the research literature, the authors selected 7 sentence completions and 16 analogies with extreme levels of differential item functioning (DIF) and then systematically revised and readministered the items in an attempt to reduce or eliminate DIF. Several diverse conclusions can be drawn from the data. First, because of the apparent success in reducing extreme levels of DIF in SAT‐Verbal items, the authors recommend that such efforts be continued. Second, the particular terminology used in stems and keys (rather than the underlying reasoning skill being measured) seems to be a recurring source of DIF in SAT‐Verbal items. Third, larger sample sizes, particularly for minority focal groups, would help to stabilize the DIF categories used by Educational Testing Service (ETS) test developers. Fourth, because the ETS delta metric is unbounded at the extremes, the use of both the Standardization (p‐metric) and Mantel‐Haenszel (delta‐metric) methodologies is recommended for classifying the level of DIF for very easy and very difficult items. Finally, the paper concludes with a suggestion for further research concerning the possible relationship between DIF and predictive validity.
THE ACCURACY OF AUTOMATIC QUALITATIVE ANALYSES OF CONSTRUCTED‐RESPONSE SOLUTIONS TO ALGEBRA WORD PROBLEMS
Authors: Bennett Randy Elliot , Sebrechts Marc M. 
Published: August 2014

ABSTRACTThis study evaluated expert system diagnoses of examinees' solutions to complex constructed‐response algebra word problems. Problems were presented to three samples, each of which had taken the GRE General Test. One sample took the problems in paper‐and‐pencil form and the other two on computer. Responses were then diagnostically analyzed by an expert system, GIDE, and by four ETS mathematics test developers using a fine‐grained categorization of error types. Results were highly consistent across the samples. Human judges agreed among themselves almost perfectly in describing responses as right or wrong but concurred at much lower levels (37% to 64% agreement) in categorizing the specific bugs they detected in incorrect solutions. The expert system agreed highly with the judges' right/wrong decisions (95% to 97% concurrence) and somewhat less closely (71% to 74%) with the bug categorizations that judges, themselves, agreed on. Seven principal causes of machine‐rater disagreement were detected, most of which could be remedied by making adjustments to GIDE, modifying the test presentation interface to constrain the form of examinee solutions, and working with test developers to specify rules for automatically dealing with special cases. These results suggest that highly accurate diagnostic analysis through knowledge‐based understanding of complex responses may be difficult to achieve at the fine‐grained level used by GIDE. The accuracy of qualitative judgments might be increased by using a smaller set of more general diagnostic categories and by integrating information from other sources, including performance on diverse item types.
FLAWED ITEMS IN COMPUTERIZED ADAPTIVE TESTING1,2
Authors: Potenza Maria T. , Stocking Martha L. 
Published: August 2014

ABSTRACTA (multiple‐choice) test item is identified as flawed if it has no single best answer. In spite of extensive quality control procedures, the administration of flawed items to test‐takers is inevitable. Common strategies for dealing with flawed items in conventional testing, grounded in the principle of fairness to test‐takers, are reexamined in the context of adaptive testing. An additional strategy, available for adaptive testing, of retesting from a pool cleansed of flawed items, is compared to the existing strategies. Retesting was found to be no practical improvement over current strategies.