bluuebunny/crossref_metadata_2025_split huggingface dataset:
A GENERATIVE APPROACH TO THE DEVELOPMENT OF HIDDEN‐FIGURE ITEMS
Authors: Bejar Isaac I. , Yocom Peter 
Published: September 2014

ABSTRACTThis report explores an approach to item development and psychometric modeling which explicitly incorporates knowledge about the mental models used by examinees in the solution of items into a psychometric model that characterize performances on a test, as well as incorporating that knowledge into the item development process. The paper focuses on the hidden figure item type. Although there is an extensive literature on the correlates of performance for this type of item little is known about the mental models that may explain performance on the item. The approach taken in this paper is to search for a complexity dimension that accounts for the difficulty of hidden figures. Although several complexity dimensions can be postulated we chose one inspired by artificial intelligence research on vision. A computer‐based system was developed to analyze as well as generate items based on this framework. To empirically determine the validity of the chosen framework two experiments were conducted. The results suggest that this approach to psychometric modeling is viable. The practical and theoretical implications of the research are discussed.
TESTING HANDICAPPED PEOPLE—THE VALIDITY ISSUE
Authors: Willingham Warren W. 
Published: September 2014

ABSTRACTAdmissions testing programs provide modified tests and test procedures for handicapped individuals in order to reduce the likelihood that a sensory or physical disability may spuriously lower test scores. Regulations under Section 504 of the Rehabilitation Act require that the validity of such tests be demonstrated. The meaning of validity in this instance is examined in consideration of federal legislation, current test theory and practice, and unusual circumstances encountered in the testing of handicapped people. It is concluded that test validity for handicapped subgroups means primarily test fairness and the measure of test fairness is comparability. Five measures of comparability between test scores for handicapped and nonhandicapped examinees are distinguished. For considerations of equity as well as the inherent difficulty in establishing score comparability, it is important to give careful attention also to the comparability of the task presented by the assessment procedure.
KEYLIST ITEMS FOR THE MEASUREMENT OF VERBAL APTITUDE
Authors: Ward William C. , Dworkin Dan , Carlson Sybil B. 
Published: September 2014

ABSTRACTTwo verbal item types employed in standardized aptitude tests were administered in a conventional multiple‐choice format and in the keylist format, in which the examinee is required to think of an answer, look it up in a long ordered list, and write its number. The keylist format provides a machine‐scorable surrogate for a truly free‐response test. Its potential attractions include the increased acceptability of items given in a production rather than a recognition format, resistance to coaching based on “gaming” strategies for eliminating multiple‐choice alternatives, and elimination of the need in item writing to produce plausible distractors for an item.Relations among tests employing different response formats were analyzed and their correlations with other measures of aptitude and achievement were compared. As in several previous studies, these analyses indicated that the format has little or no systematic effect on the construct validity of tests employing item types used in standardized tests of verbal aptitude.One of the purposes of the study was to determine the degree to which experienced test developers could agree on the set of keys that should be supplied for each keylist item. Agreement among reviewers was far from the near‐perfect consensus that would be required for use of this format, perhaps because the two item types investigated, Antonyms and Analogies, represent tests dealing with word meanings taken largely out of context. Many English words can convey multiple shades of meaning and can be constrasted along multiple dimensions. Without the constraint imposed by context, the number of possibly acceptable answers can become unmanageably great, particularly if it is required that all acceptable keys be included in the list and that all that are included must be clearly defensible. Several suggestions were offered of situations in which variations on this format could appropriately be employed.
DEVELOPMENT OF CLOZE‐ELIDE TESTS OF ENGLISH AS A SECOND LANGUAGE
Authors: Manning Winton H. 
Published: September 2014

ABSTRACTThe purpose of this study was to investigate the validity of cloze‐elide tests of English proficiency for students who are similar to the TOEFL candidate population. Cloze‐elide tests used in this research consisted of exercises in which an examinee is required to edit a prose passage by eliminating extraneous words that have been randomly interspersed throughout the original text of the passage. Students enrolled in university‐level intensive English language programs were administered a series of tests including, in addition to cloze‐elide tests, a form of the TOEFL, a multiple‐choice cloze test, a traditional cloze exercise, and an essay that was holistically scored. Students were also rated by their instructors in twelve areas of English proficiency, and the students rated their own language competency through self‐assessments in ten areas. A variety of student background information was also obtained. The design of the study aimed at contributing evidence of the construct validity of cloze‐elide tests. Concurrent validity information is relevant to this question, but the evidence arising from factor analyses of the intercorrelations among these many measures is more pertinent. In summary, the new cloze‐elide measures demonstrated very strong concurrent validity for TOEFL and other more widely used measures of second language proficiency. The factor analyses suggest that cloze‐elide tests are good, indirect measures of English language proficiency, comparing very favorably with more commonly used testing procedures. Multiple regression analyses confirmed the usefulness of cloze‐elide tests, which were generally one of the two best predictors of teacher ratings of students' English proficiency.
COGNITIVE PSYCHOLOGY AND THE SAT: A REVIEW OF SOME IMPLICATIONS
Authors: Bejar Isaac I. , Embretson Susan , Mayer Richard E. 
Published: September 2014

ABSTRACTThat the SAT has not changed fundamentally since its inception without diminishing its pragmatic utility suggests that the SAT is appropriate for its intended purpose. Nevertheless, the social and scientific milieu has certainly changed since the 1920s. This paper reviews the implications of scientific developments accompanying the revival of cognitive psychology. We distinguished early in the paper between a process‐oriented or diagnostic test and an outcomes‐oriented test such as the SAT. Since the SAT does not aim to be a diagnostic test the implications of cognitive theories for test construction that would otherwise be applicable were not emphasized. Nevertheless, an analysis was presented based on quantitative items that illustrates how tests with a diagnostic orientation could be based on cognitive principles. Most of the paper, however, is devoted to an explication of the linkage of cognitive principles with psychometric considerations for an outcomes‐oriented test like the SAT. The essence of that linkage is an accounting of item difficulty, that is, an understanding of the differences in difficulty among items. This understanding, can be viewed as an additional requirement for construct validity, which until recently focused almost exclusively on an accounting of the covariation of test scores in terms of “abilities.” In other words, explaining covariation and item difficulty are now equally important aspects of validation. Although work from this enlarged validational perspective on SAT items is limited, there is work based on similar items, especially analogy and reading comprehension, which suggests that the SAT will fare well when relevant studies are conducted. Despite this positive outlook additional studies need to be conducted based on an enlarged validational framework. Even though the predictive power of the test is not likely to increase significantly because of this research, it is nevertheless essential as a means of realizing the programmatic implications discussed in the paper.
CHARACTERISTICS OF EFFECTIVE COMPUTER IN‐SERVICE PROGRAMS
Authors: Stecher Brian M. , Solorzano Ronald 
Published: September 2014

ABSTRACTThis study sought to describe effective computer in‐service programs and to identify the components that contributed to their effectiveness. An Advisory Committee identified eight school districts from across the country that appeared to be providing outstanding computer training for teachers. Case studies of these eight model districts were conducted in which computer coordinators, trainers and teachers were interviewed and in‐service classes were observed.The data analysis focused on five issues. First, the outcomes of the in‐service classes were examined, including the knowledge and skills acquired by teachers concerning computers, their use of computers in instruction and the impact of computer use on students. This information helped to establish the degree to which the eight districts provided models of good computer in‐service training. Second, the in‐service delivery systems were described and effective instructional practices were identifed. Third, teacher characteristics were examined and relationships between teacher characteristics and outcomes were explored. Organizational context was the fourth issue to be reviewed, and a number of context variables were identified that were related to program effectiveness. Finally, a few unanticipated factors that helped to explain in‐service results were identified.
A CONSUMER'S GUIDE TO LOGIST AND BILOG1
Authors: Mislevy Robert J. , Stocking Martha L. 
Published: September 2014

ABSTRACTSince its release in 1976, Wingersky, Barton, and Lord's (1982) LOGIST has been the most widely used computer program for estimating the parameters of the 3‐parameter logistic (Birnbaum) item response model. An alternative program, Mislevy and Bock's (1983) BILOG, has recently become available. This paper compares the approaches taken by the two programs, and offers some initial guidelines for choosing between the two programs for particular applications.
MODELING ITEM RESPONSES WHEN DIFFERENT SUBJECTS EMPLOY DIFFERENT SOLUTION STRATEGIES1
Authors: Mislevy Robert J. , Verhelst Norman 
Published: September 2014

ABSTRACTA model is presented for item responses when different examinees employ different strategies to arrive at their answers, and when only those answers, not choice of strategy or subtask results, can be observed. Using substantive theory to differentiate the likelihoods of response vectors under a fixed set of solution strategies, we model responses in terms of item parameters associated with each strategy, proportions of the population employing each, and the distributions of examinee parameters within each. Posterior distributions can then be obtained for each examinee, giving the probabilities that they employed each of the strategies and their proficiency under each. The ideas are illustrated with a conceptual example about response strategies for spatial rotation items, and a numerical example resolving a population of examinees into subpopulations of valid responders and random guessers.
METHODS FOR ANALYZING THE ATTAINMENT OF GRADUATE SCHOOL MILESTONES: A CASE STUDY
Authors: Zwick Rebecca , Braun Henry I. 
Published: September 2014

ABSTRACTUsing a data base compiled by the Graduate School of Northwestern University, a longitudinal study of the graduate school careers of students in fourteen programs was conducted. Among the most prominent findings was the increase in the enrollment of foreign students. The patterns of attainment of graduate school milestones, such as Ph.D. candidacy and graduation, were examined for each graduate program and for gender and ethnic groups. There was substantial variation across programs and, to a lesser degree, across demographic groups. Graduation rates for foreign students were higher than those for U.S. citizens. The association between the attainment of milestones and measures of academic potential, such as undergraduate grade‐point average (UGPA) and Graduate Record Examination (GRE) scores was also investigated. The likelihood of attaining candidacy or of completing a doctorate was found to bear little relation to UGPA and GRE scores. This finding is probably a result of the use of UGPA and GRE in the selection of students into graduate programs.
AN EMPIRICAL INVESTIGATION OF ONE VARIABLE SECTION PRE‐EQUATING1
Authors: Wightman Linda F. (Leary) , Wightman Lawrence E. 
Published: September 2014

ABSTRACTSection Pre‐Equating (SPE) is a method used to equate test forms that consist of multiple separately‐timed sections. The unique contribution of SPE is that it does not require examinees to take two complete forms of the test. Instead, all of the old form and one or two sections of the new form are administered to each examinee. Missing data techniques are employed to estimate the necessary equating parameters. When a test has two variable sections, the estimation is fairly straightforward.When a test includes only one variable section, there is no simple way to obtain an estimate of the correlation between pairs of sections from the new test because no one group takes a pair of sections. This study was designed to utilize empirical data to evaluate several methods for obtaining reasonable values for these correlations. The methods for obtaining correlations that are evaluated are borrowing correlations from another population who have taken the same test at a different administration, borrowing correlations from another parallel test taken by the same population, and imputing the unknown partial correlations. This study explored the effect on equating of using each of these methods. The criterion for evaluating the one‐variable‐section test equatings was the equating obtained using a two variable section model.Comparison of the converted scores obtained using the three different methods for estimating correlations under a one‐variable‐section test model were consistent with one another. Results from this study also demonstrated that equating results obtained from a one‐variable‐section test model were very consistent with those obtained from a two‐variable‐section test model.
EVALUATION OF AL MANAAHIL
Authors: Murphy Richard T. 
Published: September 2014

ABSTRACTBetween April 1985 and January 1988, Educational Testing Service conducted an evaluation of AL MANAAHIL (The Sources), a children's television series in reading for Arab children. The series, produced and broadcast in Jordan, contained 65 half‐hour shows in Modern Standard Arabic (MSA) designed for children in Grades One through Four. The shows were intended to expose children to modern vocabulary and examples of accurate uses of MSA in everyday situations.More than 5,000 children in Jordan, Morocco, and Tunisia were tested in the evaluation. Based on the Jordan test data, AL MANAAHIL had a positive impact on children's reading skills. The positive findings were replicated in Government schools, United Nations schools, and Private schools. The findings were similar for boys and girls. They held up across different tests. The positive results were more pronounced on items more clearly related to specific goals of the series than on items less clearly related. The majority of children who participated in the testing program reported that they watched the AL MANAAHIL series and liked it.Responses of teachers, parents, and other adults to an ETS survey about the series were also positive. The majority of teachers (65%) indicated that they liked the series or liked it very much. Most (74%) noted that the series had a positive or very positive impact on the children. When asked about AL MANAAHIL's effect on their teaching, a majority (54%) of teachers indicated that AL MANAAHIL had a positive or very positive effect.In several adult literacy programs, participants were asked to take the AL MANAAHIL tests and respond to questions about how frequently they viewed the series and how much they liked or disliked it. Of 83 adults in the literacy sample, 33 (40% of the sample) reported that they watched the programs on a regular basis. That group of 33 performed better on the AL MANAAHIL test battery than a group of 28 who did not watch the series at all ‐ a result that supports the overall positive findings of the evaluation.In summary, based on the data gathered in 1986 and 1987, AL MANAAHIL had a positive effect on children in this study. It is likely to have a similar effect on children who view the series in the future in Jordan and other Arab countries.
CONSIDERATIONS AND TECHNIQUES FOR THE ANALYSIS OF NAEP DATA
Authors: Johnson Eugene G. 
Published: September 2014

ABSTRACTThe special characteristics of the National Assessment data which affect the validity of conventional techniques of statistical analysis are considered. In contrast to the assumptions underlying standard methods of statistical analysis, the NAEP samples are obtained via a stratified multi‐stage probability sampling design in which clusters of students are selected and in which certain subpopulations are sampled at a higher rate. The resulting samples have different statistical characteristics from those of a simple random sample. Analytic techniques which take the structure of the data into account are discussed and their properties explored. A second feature of the National Assessment data arises from the fact that multiple matrix sampling techniques are used for the assignment of subsets of the pool of exercises to subsamples of students. National Assessment accounts for the effect of the missing data on its scaling models through the use of multiple imputation procedures; creating vectors of “plausible values” for the scale‐scores. The use of plausible values and their effect on analysis are illustrated. The analyses are illustrated with results from the NAEP assessments of mathematics.
THE EFFECTS ON IRT AND CONVENTIONAL ACHIEVEMENT TEST EQUATING RESULTS OF USING EQUATING SAMPLES MATCHED ON ABILITY1,2
Authors: Cook Linda L. , Eignor Daniel R. , Schmitt Alicia P. 
Published: September 2014

ABSTRACTResearch by Cook, Eignor, and Taft (1988) has clearly demonstrated the need for equating certain achievement tests using samples of students who take the new and old forms at comparable points in the school year. In their study, Cook et al. equated two forms of an achievement test in Biology using two different sample combinations, a spring new‐form/fall old‐form combination and a fall new‐form/fall old‐form pairing. Equating results were quite different across the two sample combinations and all equatings, based on a variety of methods, done using the spring new‐form/fall old‐form combination provided unacceptable results. However, results from recently completed research by Lawrence and Dorans (1988) suggests that matching samples of differing ability (such as the spring new‐form and fall old‐form samples in the Cook et al. study) on an available covariate prior to equating may improve equating results.The purpose of the present study was to examine the effects on IRT and conventional equating results of matching the Cook et al. spring new‐form and fall old‐form Biology samples using two different covariate measures: 1) observed scores on an internal common item equating block; and 2) responses to selected questions on the Student Description Questionnaire (SDQ), a questionnaire that candidates respond to when filling out their registration forms for the test. These matched sample equatings were then compared to the unmatched spring new‐form/fall old‐form equating results and to the equatings based on the samples of comparable ability, the fall new‐form/fall old‐form sample combination.Results of this study indicate that: 1) matching on a set of common items provides greater agreement among the results of the various equating procedures studied than if matching does not take place; and 2) for all equating procedures, the results of the matched group (on the common items) equating agreed more closely with the fall new‐form/fall old‐form criterion equatings than did the fall/spring unmatched equating results. An explanation for why the matching on SDQ responses provided unacceptable equating results is presented in the paper. Finally, the results of this study are compared to the results of other recently completed studies that have also dealt with the matching process.
GRADE LEVEL AND PERFORMANCE ON THE CORE BATTERY, 1986–87 NATIONAL ADMINISTRATION1
Authors: DeMauro Gerald E. 
Published: September 2014

ABSTRACTAmong the three NTE Core Battery tests, Professional Knowledge purports to measure skills that are acquired only after academic experience as an education major, while Communication Skills and General Knowledge both purport to measure skills that do not require this specialized experience. The scores of education majors for the 1986–87 national test administrations were analyzed for differential sensitivity of the three tests to educational attainment. Since examinees with different degrees of experience who choose to take the tests may have different levels of skills, test scores were adjusted for grade point average. Results showed that Professional Knowledge was more sensitive to academic experience than were General Knowledge or Communication Skills. This outcome is interpreted as support of the construct validity of the Core Battery.
CLUSTERS AS THE UNIT OF ANALYSIS IN DIFFERENTIAL ITEM FUNCTIONING
Authors: Bennett Randy Elliot , Rock Donald A. , Novatkoski Inge 
Published: September 2014

ABSTRACTThis study developed, applied, and evaluated a theory‐based method of detecting the underlying causes of differential difficulty. The method was intended to improve on traditional approaches that too often produce uninterpretable results. Key elements of the method were the analysis of item clusters and the incorporation of theoretical predictions about cluster performance. The method was applied in two subgroups taking SAT‐M and involved (1) reviewing literature syntheses to identify factors that might cause differential item functioning, (2) forming item categories based on those factors, (3) identifying categories that functioned differentially, (4) assessing the functioning of the items composing deviant categories, and (5) relating item and category functioning. Results were compared to a traditional item‐level analysis. In both subgroups, the cluster and traditional methods agreed on the overall extent of differential functioning (substantial in the first group, virtually absent in the second). Additionally, the pattern of differential functioning detected was interpretable. At the same time, several important limitations were apparent. The method would seem to be applied most productively when a small number of hypotheses can be derived from a reasonably strong research base, overlap among cluster structures can be avoided, and results can be supplemented with experimental studies or protocol analyses.