bluuebunny/crossref_metadata_2025 huggingface dataset:
EVALUATING THE WORK EXPERIENCE OF RESPIRATORY THERAPY PRACTITIONERS FOR THE NBRC
Authors: Nettles Steven S.
Published: September 2014

The purpose of this study was twofold: (1) to develop a standardized self‐report instrument that would allow respiratory therapy practitioners applying for registration by the National Board for Respiratory Care (NBRC) to describe their relevant work experience in a manner useful to the NBRC, and (2) to determine if this information was accurate. With the help of an NBRC advisory committee, ETS staff developed an instrument for objectively quantifying the work experience of respiratory therapy practitioners. The instrument, a Work Experience Inventory, provided data on many job characteristics, including level of responsibility on the job, exposure to different work situations, to different job tasks, and to a variety of respiratory knowledges, skills, and abilities. To determine if the instrument was sensitive to differences between Registered Respiratory Therapist (RRT) and nonRegistered Respiratory Therapist (nonRRT) practitioners, two geographically representative random samples (N = 84 and N = 54, respectively) from the above two populations of practitioners were asked to complete the inventory.

The majority of the participants found the inventory easy to complete and confirmed that the instrument accurately reflected their work experience. Agreement was obtained between the participants' and their employers' ratings on several characteristics used to estimate the level of responsibility present in participants' work experience. Statistical analysis indicated that all sections of the inventory collected unique information. Items in all sections of the instrument distinguished between the registered and nonregistered practitioners.

Because of the encouraging results of this demonstration project, project staff recommend that the NBRC use the instrument in a pilot study with substantially larger samples of registry applicants, to determine definitively the feasibility of integrating the Work Experience Inventory into the NBRC credentialing program.
THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT
Authors: Messick Samuel
Published: September 2014

Since school learning entails not just accretion of knowledge but the structuring and restructuring of knowledge and cognitive skills, the conception and construction of educational achievement measures must be cast in developmental terms. Since student characteristics as well as social and educational experiences influence current performance, the interpretation and implications of educational achievement measures must be relative to intrapersonal and situational contexts. These points imply a strategy of comprehensive assessment in context that focusses on the processes and structures involved in subject‐matter competence as moderated in performance by personal and environmental influences. This article addresses in detail both the nature of developing competence and its measurement in terms of context‐dependent task performance. Construct‐irrelevant task difficulty that might jeopardize the meaning of test scores as well as construct‐irrelevant influences that might jeopardize implications for action are taken into account via the comprehensive measurement of relevant contextual factors. Comprehensive assessment in context thus facilitates valid interpretations of the meaning and implications of ability and achievement scores in particular instances, thereby lightening the interpretive and ethical burdens on test users and enhancing the validity of test use.
TESTING THE LOCAL INDEPENDENCE ASSUMPTION IN ITEM RESPONSE THEORY
Authors: Rosenbaum Paul R.
Published: September 2014

Local independence and monotone increasing item characteristic curves imply nonnegative conditional covariances between all monotone increasing functions of a set of item responses given any function of the remaining item responses. This general result provides a basis for testing the local independence assumption without first specifying a parametric form for the item characteristic curve. The proposed tests are simple, have known asymptotic null distributions, and possess certain optimal properties. In an example, the local independence hypothesis is rejected for all possible forms of the monotone item characteristic curves.
THE ASSESSMENT OF WRITING ABILITY: A REVIEW OF RESEARCH
Authors: Cooper Peter L.
Published: September 2014

The assessment of writing ability has recently recevied much attention from educators, legislators, and measurement experts, especially because the writing of students in all disciplines and at all educational levels seems, on the whole, less proficient than the writing produced by students five or ten years ago. The GRE Research Committee has expressed interest in the psychometric and practical issues that pertain to the assessment of writing ability. This paper presents not a new study but a review of major research in light of GRE Board concerns. Specifically, recent scholarship and information from established programs are used to investigate the nature and limitations of essay and multiple‐choice tests of writing ability, the statistical relationship of performances on these types of tests, the performance of population subgroups on each kind of task, the possible need of different disciplines for different tests of composition skill, and the cost and usefulness of various strategies for evaluating writing ability.

The literature indicates that essay tests are often considered more valid than multiple‐choice tests as measures of writing ability. Certainly they are favored by English teachers. But although essay tests may sample a wider range of composition skills, the variance in essay test scores can reflect such irrelevant factors as speed and fluency under time pressure or even penmanship. Also, essay test scores are typically far less reliable than multiple‐choice test scores. When essay test scores are made more reliable through multiple assessments, or when statistical corrections for unreliability are applied, performance on multiple‐choice and essay measures can correlate very highly. The multiple‐choice measures, though, tend to overpredict the performance of minority candidates on essay tests. It is not certain whether multiple‐choice tests have essentially the same predictive validity for candidates in different academic disciplines, where writing requirements may vary. Still, at all levels of education and ability, there appears to be a close relationship between performance on multiple‐choice and essay tests of writing ability. And yet each type of measure contributes unique information to the overall assessment. The best measures of writing ability have both essay and multiple‐choice sections, but this design can be prohibitively expensive. Cost cutting alternatives such as an unscored or locally scored writing sample may compromise the quality of the essay assessment. For programs considering an essay writing exercise, a discussion of the cost and uses of different scoring methods is included. The holistic method, although having little instructional value, offers the cheapest and best means of rating essays for the rank ordering and selection of candidates.
CONSIDERATIONS FOR DEVELOPING MEASURES OF SPEAKING AND LISTENING
Authors: Powers Donald E.
Published: September 2014

The College Board has identified several basic intellectual competencies thought to be essential for effective work in all fields of college study, among them listening and speaking. An issue that arises in connection with these competencies is the availability of suitable measures to assess students' development in these areas.

This report considers the availability and adequacy of existing measures of speaking and listening, and discusses a number of issues that should be considered in any efforts to develop new measures of these skills.
ARE THE ITEM RESPONSE PATTERNS OF TWO GROUPS OF EXAMINEES CONSISTENT WITH A DIFFERENCE IN THE DISTRIBUTION OF A UNIDIMENSIONAL LATENT VARIABLE?
Authors: Rosenbaum Paul R.
Published: September 2014

If one group of examinees can be described as having generally higher ability to respond correctly to the items on an exam than some other group – where ability is defined by an unidimensional item response model – then the observable distributions of item responses must satisfy certain constraints. Standard statistical methods for discrete data may be used to test the consistency of these constraints with observed data. In an example, the observed pattern of item responses is found to be inconsistent with a difference in the distribution of a unidimensional latent variable.
USE OF EMPIRICAL BAYES METHODS IN THE STUDY OF THE VALIDITY OF ACADEMIC PREDICTORS OF GRADUATE SCHOOL PERFORMANCE
Authors: Braun Henry I. , Jones Douglas H.
Published: September 2014

Graduate education in the United States is characterized by an enormous diversity of disciplines and the predominance of relatively small enrollments in individual departments. In this setting, a validity study based on a single department's data and employing classical statistical methods can only be of limited utility and applicability. In order to participate in the Graduate Record Examinations Validity Study Service, a department must have at least 25 students in its entering class. Only validities for single predictors are provided; estimates of the validity of two or more predictors, used jointly, are considered too unreliable because the corresponding prediction equations often possess implausible characteristic, such as negative coefficients. These constraints were introduced by the Validity Study Service to reduce the chance that the results in the report to a department would be overly influenced by statistical artifacts in the data and hence serve more to mislead than to inform.

In this study we investigated two statistical methods, empirical Bayes and cluster analysis, to determine whether their application to the problems faced by the Validity Study Service could result in useful improvements. Considerable effort was expended in developing and studying a new and more general class of empirical Bayes models that can accommodate the complex structure of the Validity Study Service data base.

The principal methodological conclusions of this study are that, through the use of a new class of empirical Bayes methods, it is possible to obtain, at the departmental level, useful and reliable estimates of the joint validity of several predictors of academic performance and that these methods may be further refined to address the question of differential predictive validity, again at the departmental level. These results have important practical implications for the GRE Validity Study Service.
BAYESIAN ESTIMATORS, ROBUST ESTIMATORS: A COMPARISON AND SOME ASYMPTOTIC RESULTS
Authors: Jones Douglas H.
Published: September 2014

Combinations of Bayes and resistant estimators of latent ability were studied in Wainer and Thissen (1984), who show that they have desirable performances for small to medium samples especially in the presence of contamination. While their study was based on computer simulations, this note provides the first order asymptotic theory, from which one may compute approximations to the standard errors of the estimators. A side issue concerns the “overshrinking phenomenon” which caused Wainer and Thissen to expand the scale of the estimators in order to increase their accuracy and to make them more comparable to the traditional estimators. This note derives a differential equation specifying a transform that may make Bayes/resistant estimators even more favorable.
REPORT OF A STUDY OF THE CONTENT OF THREE GRE ADVANCED TESTS
Authors: DeVore Richard , McPeek Miles
Published: September 2014

This paper reports the findings from content studies of three GRE Advanced Tests: Biology, Literature in English, and Political Science. The studies were designed to indicate the degree to which the content specified for and included in recent test forms for these fields is representative of the content actually taught in the undergraduate curricula in the fields. The studies provided substantively useful information concerning the content representativeness of the tests.

Informed judgments were relied on as the technique for assessing the representativeness of the content. Groups of college and university faculty knowledgeable in curricular practices at the undergraduate level were asked to make judgments about the relationship between curriculum and test content by examining both individual test questions and the content specifications for the tests. By inference, within the error limits of the judgments, an assessment of the relevance of test content to undergraduate curricula was obtained.

Although the studies supply evidence that the content of each of the tests generally is representative of the content of the respective undergraduate curricula, modest differences of opinion were found among the respondents in the Biology and Political Science studies as to the degree of emphasis that should be given to certan subareas of the content.
ESTIMATING THE NUMBER OF EXAMINEES WHO DID NOT REACH THE LAST ITEM OF A SECTION
Authors: Wainer Howard
Published: September 2014

A statistical method is proposed and illustrated for estimating the number of individuals that did not reach the last item on a section. This is compared with the current method and shown superior.
PROGRESS TOWARD STANDARDS AS STANDARDS FOR PROGRESS: A POTENTIAL ROLE FOR THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Authors: Messick Samuel
Published: September 2014

As currently constituted and implemented, the National Assessment of Educational Progress incorporates three key elements of a responsible standard‐setting process — namely, the choice of educational objectives, the description of current group performance ranges and trends, and the identification of educational contexts differentially related to performance. What is still required to move on to the setting of quality educational standards are value judgments as to which ranges of performance are deemed unsatisfactory, adequate, or excellent by different societal groups. This article addresses the type of pluralistic judgmental process needed for standard‐setting when a diversity of values as well as consensus must be taken into account.
AN INVESTIGATION OF THE FEASIBILITY OF APPLYING ITEM RESPONSE THEORY TO EQUATE ACHIEVEMENT TESTS1,2,3
Authors: Cook Linda L. , Eignor Daniel R.
Published: September 2014

The purpose of this study was to examine the feasibility of using item response theory (IRT) procedures to equate different forms of three achievement tests (Biology, American History and Social Studies, and Mathematics Level II) that are administered by the College Board Admissions Testing program and one achievement test (Advanced Biology) that is part of the Graduate Record Examinations Achievement Test battery. All of the tests investigated in this study are typically equated using conventional linear or curvilinear (equipercentile) methods.

Because the investigators had access to data from large scale testing programs that administer many different forms of a test, it was possible to design the study such that a unique criterion for evaluating the equating results, scale drift, was available. Scale drift will occur if the result of equating, say, Form A to Form C through Form B is not the same as that obtained by equating Form A directly to Form C. This study was designed so that for all the Achievement tests investigated, the drift in each scale when IRT equating was used could be compared to the drift occurring when the conventional linear and equipercentile equating methods, used operationally to put new forms on scale, were applied.

The results of the study indicate, using scale drift as the evaluative criterion, that it is feasible to use item response theory to equate the four Achievement tests selected for investigation. The results also indicate that the conventional linear methods typically used to equate the tests perform quite adequately. The question of whether the IRT procedures used in the study are sufficiently robust to violations of the assumption of unidimensionality, or whether Achievement tests, of the type used in this study, give rise to sufficiently unidimensional data, must be resolved before the results of the study can be generalized to other Achievement testing situations.
ON CORRELATIONS, DISTANCES AND ERROR RATES1
Authors: Dorans Neil J.
Published: September 2014

A unified framework for classification and prediction problems is presented. Well known and lesser known relationships among correlations, distances and error rates are established. A new population distance, the shrunken generalized distance, and a new estimator of the actual error rate are introduced.
MINORITY CONTRIBUTIONS TO THE SAT SCORE TURNAROUND: AN EXAMPLE OF SIMPSON'S PARADOX
Authors: Wainer Howard
Published: September 2014

Since 1980 the decline in SAT scores has stopped and the scores have started to creep back up. The scores for White Americans have increased 8 points during this period, for non‐Whites 15 points. It was thus surprising to discover that the overall mean increased only 7 points. This is not an arithmetic error but rather an example of a well‐known statistical phenomenon called Simpson's Paradox. In this note we explain the paradox and describe a method which will avoid it in the future.
BACKGROUND CHARACTERISTICS OF EXAMINEES SHOWING UNUSUAL TEST BEHAVIOR ON THE GRADUATE RECORD EXAMINATIONS
Authors: Oltman Philip K.
Published: September 2014

We ordinarily expect item difficulty to be related to errors on tests; examinees generally tend to make errors on more difficult items and to answer easier items correctly. However, some examinees miss easy items and get more difficult items correct. The extent to which correct and incorrect responses are predicted by the difficulty of test items has been quantified in various ways. One method, originated by Sato (1975) and modified by Harnisch and Linn (1981), was used in the present study of item level data from the Graduate Record Examinations General Test. The modified Sato caution index was found to be of low reliability in these data and to be generally unrelated to a variety of background variables, although ethnic group showed a small but significant relation to the index. In these data the modified caution index showed a curvilinear relation to total test score, with examinees who showed very high or very low scores having higher index values than those in the middle range of test scores. Indexes calculated on the three sections of the test were uncorrelated with each other. Finally, the index did not moderate the relationship between test scores and self‐reported grades, which it should have done if it indeed indicates how well the test measures the intended construct for any individual. The conclusion reached is that the modified caution index adds little information that would be of value in interpreting GRE test scores.
A STUDY OF THE TEMPORAL STABILITY OF IRT ITEM PARAMETER ESTIMATES1,2
Authors: Cook Linda L. , Eignor Daniel R. , Petersen Nancy S.
Published: September 2014

This research report focuses on the stability of item response theory (IRT) item parameter estimates when the items are calibrated on two different samples of examinees who have responded to the items at two different points in time, i.e., the temporal stability of the parameter estimates. The data were collected from regular administrations of the College Board Admissions Testing Program Scholastic Aptitude Test (SAT) and Achievement Tests. The three‐parameter logistic model was used to characterize the relationship between the underlying trait and performance on an item and a variety of methods were used to assess parameter estimate stability.

Two important conclusions were drawn from the study. One, stability of parameter estimates is clearly related to type of test. Parameter estimates obtained from SAT‐verbal and SAT‐mathematical are more likely to exhibit stability over time than those obtained for achievement tests of Biology or American History and Social Studies. Secondly, lack of stability of item parameter estimates appears to be more closely related to differences in ability among calibration samples than to the lapse of time between administrations of the test, which, in the study, ranged from 14 to 52 months. It should be noted, however, that for the particular tests studied, ability differences in calibration samples appeared to be unrelated to time differences between administrations. This may not be typical of other testing situations, where ability differences in the calibration samples can be directly related to length of time between test administrations. This could be brought about, for example, by changes in curricular emphasis.
A BAYESIAN TREATMENT OF LATENT VARIABLES IN SAMPLE SURVEYS
Authors: Mislevy Robert J.
Published: September 2014

Consider the problem of estimating a functionof values of the variables z and y in a finite population, when values of y are known a priori for all units but values of z are not known for any. Data are then obtained from a sample of units; not in the form of values of z, however, but in the form of secondary variables x whose values depend stochastically on z. A framework for Bayesian analysis of T along the lines of Ericson (1969) is presented and illustrated with data from theProfile of American Youth(U.S. Department of Defense 1982).
CALIBRATION OF ESSAY READERS FINAL REPORT
Authors: Braun Henry I.
Published: September 2014

Scoring reliability of essays and other free response questions is of considerable concern. This report describes a statistically designed experiment that was carried out in an operational setting to determine the contributions of different sources of variation to the unreliability of scoring. The experiment made novel use of partially balanced incomplete block designs that facilitated the unbiased estimation of certain main effects without requiring readers to assess the same paper several times. In addition, estimates were obtained of the improvement in reliability that result from removing variability from systematic sources of variation by an appropriate adjustment of the raw scores. This statistical calibration appears to be a cost‐effective approach to enhancing scoring reliability when compared to simply increasing the number of readings per paper. The results of the experiment also provide a framework for examining other, simpler calibration strategies. One such strategy is briefly examined.
A MULTIVARIATE PERSPECTIVE ON CONTINGENCY TABLE ANALYSIS
Authors: Zwick Rebecca , Cramer Elliot M.
Published: September 2014

Psychological research often involves analysis of an I × J contingency table consisting of the responses of J groups of individuals on a criterion variable with I nominal categories. The conventional statistical approach for comparing responses across groups is the Pearson chi‐square test. Alternatively, this analysis can be viewed as a multivariate analysis of variance with binary dependent variables, a canonical correlation analysis with two sets of binary variables, or a form of correspondence analysis. Although these analysis approaches stem from different traditions, they produce equivalent results when applied to an I × J table.
AN INVESTIGATION OF POSSIBLE CAUSES FOR THE INADEQUACY OF IRT PRE‐EQUATING1,2,3
Authors: Eignor Daniel R. , Stocking Martha L.
Published: September 2014

A previous study of pre‐equating the Scholastic Aptitude Test (SAT) using item response theory provided unacceptable equating results for SAT‐mathematical data. The purpose of this study was to investigate two possible explanations for these unacceptable pre‐equating results. Specifically, the calibration process, which made use of the three‐parameter model and LOGIST, and the linking procedure used to place parameter estimates on the same scale were further investigated in a two stage process to see if either was responsible for the poor IRT pre‐equating results found for the SAT‐mathematical data in the previous study.
A GENERATIVE APPROACH TO THE DEVELOPMENT OF HIDDEN‐FIGURE ITEMS
Authors: Bejar Isaac I. , Yocom Peter
Published: September 2014

This report explores an approach to item development and psychometric modeling which explicitly incorporates knowledge about the mental models used by examinees in the solution of items into a psychometric model that characterize performances on a test, as well as incorporating that knowledge into the item development process. The paper focuses on the hidden figure item type. Although there is an extensive literature on the correlates of performance for this type of item little is known about the mental models that may explain performance on the item. The approach taken in this paper is to search for a complexity dimension that accounts for the difficulty of hidden figures. Although several complexity dimensions can be postulated we chose one inspired by artificial intelligence research on vision. A computer‐based system was developed to analyze as well as generate items based on this framework. To empirically determine the validity of the chosen framework two experiments were conducted. The results suggest that this approach to psychometric modeling is viable. The practical and theoretical implications of the research are discussed.
TESTING HANDICAPPED PEOPLE—THE VALIDITY ISSUE
Authors: Willingham Warren W.
Published: September 2014

Admissions testing programs provide modified tests and test procedures for handicapped individuals in order to reduce the likelihood that a sensory or physical disability may spuriously lower test scores. Regulations under Section 504 of the Rehabilitation Act require that the validity of such tests be demonstrated. The meaning of validity in this instance is examined in consideration of federal legislation, current test theory and practice, and unusual circumstances encountered in the testing of handicapped people. It is concluded that test validity for handicapped subgroups means primarily test fairness and the measure of test fairness is comparability. Five measures of comparability between test scores for handicapped and nonhandicapped examinees are distinguished. For considerations of equity as well as the inherent difficulty in establishing score comparability, it is important to give careful attention also to the comparability of the task presented by the assessment procedure.
KEYLIST ITEMS FOR THE MEASUREMENT OF VERBAL APTITUDE
Authors: Ward William C. , Dworkin Dan , Carlson Sybil B.
Published: September 2014

Two verbal item types employed in standardized aptitude tests were administered in a conventional multiple‐choice format and in the keylist format, in which the examinee is required to think of an answer, look it up in a long ordered list, and write its number. The keylist format provides a machine‐scorable surrogate for a truly free‐response test. Its potential attractions include the increased acceptability of items given in a production rather than a recognition format, resistance to coaching based on “gaming” strategies for eliminating multiple‐choice alternatives, and elimination of the need in item writing to produce plausible distractors for an item.

Relations among tests employing different response formats were analyzed and their correlations with other measures of aptitude and achievement were compared. As in several previous studies, these analyses indicated that the format has little or no systematic effect on the construct validity of tests employing item types used in standardized tests of verbal aptitude.

One of the purposes of the study was to determine the degree to which experienced test developers could agree on the set of keys that should be supplied for each keylist item. Agreement among reviewers was far from the near‐perfect consensus that would be required for use of this format, perhaps because the two item types investigated, Antonyms and Analogies, represent tests dealing with word meanings taken largely out of context. Many English words can convey multiple shades of meaning and can be constrasted along multiple dimensions. Without the constraint imposed by context, the number of possibly acceptable answers can become unmanageably great, particularly if it is required that all acceptable keys be included in the list and that all that are included must be clearly defensible. Several suggestions were offered of situations in which variations on this format could appropriately be employed.
DEVELOPMENT OF CLOZE‐ELIDE TESTS OF ENGLISH AS A SECOND LANGUAGE
Authors: Manning Winton H.
Published: September 2014

The purpose of this study was to investigate the validity of cloze‐elide tests of English proficiency for students who are similar to the TOEFL candidate population. Cloze‐elide tests used in this research consisted of exercises in which an examinee is required to edit a prose passage by eliminating extraneous words that have been randomly interspersed throughout the original text of the passage. Students enrolled in university‐level intensive English language programs were administered a series of tests including, in addition to cloze‐elide tests, a form of the TOEFL, a multiple‐choice cloze test, a traditional cloze exercise, and an essay that was holistically scored. Students were also rated by their instructors in twelve areas of English proficiency, and the students rated their own language competency through self‐assessments in ten areas. A variety of student background information was also obtained. The design of the study aimed at contributing evidence of the construct validity of cloze‐elide tests. Concurrent validity information is relevant to this question, but the evidence arising from factor analyses of the intercorrelations among these many measures is more pertinent. In summary, the new cloze‐elide measures demonstrated very strong concurrent validity for TOEFL and other more widely used measures of second language proficiency. The factor analyses suggest that cloze‐elide tests are good, indirect measures of English language proficiency, comparing very favorably with more commonly used testing procedures. Multiple regression analyses confirmed the usefulness of cloze‐elide tests, which were generally one of the two best predictors of teacher ratings of students' English proficiency.
COGNITIVE PSYCHOLOGY AND THE SAT: A REVIEW OF SOME IMPLICATIONS
Authors: Bejar Isaac I. , Embretson Susan , Mayer Richard E.
Published: September 2014

That the SAT has not changed fundamentally since its inception without diminishing its pragmatic utility suggests that the SAT is appropriate for its intended purpose. Nevertheless, the social and scientific milieu has certainly changed since the 1920s. This paper reviews the implications of scientific developments accompanying the revival of cognitive psychology. We distinguished early in the paper between a process‐oriented or diagnostic test and an outcomes‐oriented test such as the SAT. Since the SAT does not aim to be a diagnostic test the implications of cognitive theories for test construction that would otherwise be applicable were not emphasized. Nevertheless, an analysis was presented based on quantitative items that illustrates how tests with a diagnostic orientation could be based on cognitive principles. Most of the paper, however, is devoted to an explication of the linkage of cognitive principles with psychometric considerations for an outcomes‐oriented test like the SAT. The essence of that linkage is an accounting of item difficulty, that is, an understanding of the differences in difficulty among items. This understanding, can be viewed as an additional requirement for construct validity, which until recently focused almost exclusively on an accounting of the covariation of test scores in terms of “abilities.” In other words, explaining covariation and item difficulty are now equally important aspects of validation. Although work from this enlarged validational perspective on SAT items is limited, there is work based on similar items, especially analogy and reading comprehension, which suggests that the SAT will fare well when relevant studies are conducted. Despite this positive outlook additional studies need to be conducted based on an enlarged validational framework. Even though the predictive power of the test is not likely to increase significantly because of this research, it is nevertheless essential as a means of realizing the programmatic implications discussed in the paper.
CHARACTERISTICS OF EFFECTIVE COMPUTER IN‐SERVICE PROGRAMS
Authors: Stecher Brian M. , Solorzano Ronald
Published: September 2014

This study sought to describe effective computer in‐service programs and to identify the components that contributed to their effectiveness. An Advisory Committee identified eight school districts from across the country that appeared to be providing outstanding computer training for teachers. Case studies of these eight model districts were conducted in which computer coordinators, trainers and teachers were interviewed and in‐service classes were observed.

The data analysis focused on five issues. First, the outcomes of the in‐service classes were examined, including the knowledge and skills acquired by teachers concerning computers, their use of computers in instruction and the impact of computer use on students. This information helped to establish the degree to which the eight districts provided models of good computer in‐service training. Second, the in‐service delivery systems were described and effective instructional practices were identifed. Third, teacher characteristics were examined and relationships between teacher characteristics and outcomes were explored. Organizational context was the fourth issue to be reviewed, and a number of context variables were identified that were related to program effectiveness. Finally, a few unanticipated factors that helped to explain in‐service results were identified.
A CONSUMER'S GUIDE TO LOGIST AND BILOG1
Authors: Mislevy Robert J. , Stocking Martha L.
Published: September 2014

Since its release in 1976, Wingersky, Barton, and Lord's (1982) LOGIST has been the most widely used computer program for estimating the parameters of the 3‐parameter logistic (Birnbaum) item response model. An alternative program, Mislevy and Bock's (1983) BILOG, has recently become available. This paper compares the approaches taken by the two programs, and offers some initial guidelines for choosing between the two programs for particular applications.
MODELING ITEM RESPONSES WHEN DIFFERENT SUBJECTS EMPLOY DIFFERENT SOLUTION STRATEGIES1
Authors: Mislevy Robert J. , Verhelst Norman
Published: September 2014

A model is presented for item responses when different examinees employ different strategies to arrive at their answers, and when only those answers, not choice of strategy or subtask results, can be observed. Using substantive theory to differentiate the likelihoods of response vectors under a fixed set of solution strategies, we model responses in terms of item parameters associated with each strategy, proportions of the population employing each, and the distributions of examinee parameters within each. Posterior distributions can then be obtained for each examinee, giving the probabilities that they employed each of the strategies and their proficiency under each. The ideas are illustrated with a conceptual example about response strategies for spatial rotation items, and a numerical example resolving a population of examinees into subpopulations of valid responders and random guessers.
CHECKING THE EQUIVALENCE OF NEARLY IDENTICAL TEST EDITIONS
Authors: Dorans Neil J. , Lawrence Ida M.
Published: September 2014

A procedure for checking the score equivalence of nearly identical editions of a test is described. The procedure employs the standard error of equating and utilizes graphical representation of score conversion deviation from the identity function in standard error units. Two illustrations of the procedure involving SAT data are presented.
METHODS FOR ANALYZING THE ATTAINMENT OF GRADUATE SCHOOL MILESTONES: A CASE STUDY
Authors: Zwick Rebecca , Braun Henry I.
Published: September 2014

Using a data base compiled by the Graduate School of Northwestern University, a longitudinal study of the graduate school careers of students in fourteen programs was conducted. Among the most prominent findings was the increase in the enrollment of foreign students. The patterns of attainment of graduate school milestones, such as Ph.D. candidacy and graduation, were examined for each graduate program and for gender and ethnic groups. There was substantial variation across programs and, to a lesser degree, across demographic groups. Graduation rates for foreign students were higher than those for U.S. citizens. The association between the attainment of milestones and measures of academic potential, such as undergraduate grade‐point average (UGPA) and Graduate Record Examination (GRE) scores was also investigated. The likelihood of attaining candidacy or of completing a doctorate was found to bear little relation to UGPA and GRE scores. This finding is probably a result of the use of UGPA and GRE in the selection of students into graduate programs.
AN EMPIRICAL INVESTIGATION OF ONE VARIABLE SECTION PRE‐EQUATING1
Authors: Wightman Linda F. (Leary) , Wightman Lawrence E.
Published: September 2014

Section Pre‐Equating (SPE) is a method used to equate test forms that consist of multiple separately‐timed sections. The unique contribution of SPE is that it does not require examinees to take two complete forms of the test. Instead, all of the old form and one or two sections of the new form are administered to each examinee. Missing data techniques are employed to estimate the necessary equating parameters. When a test has two variable sections, the estimation is fairly straightforward.

When a test includes only one variable section, there is no simple way to obtain an estimate of the correlation between pairs of sections from the new test because no one group takes a pair of sections. This study was designed to utilize empirical data to evaluate several methods for obtaining reasonable values for these correlations. The methods for obtaining correlations that are evaluated are borrowing correlations from another population who have taken the same test at a different administration, borrowing correlations from another parallel test taken by the same population, and imputing the unknown partial correlations. This study explored the effect on equating of using each of these methods. The criterion for evaluating the one‐variable‐section test equatings was the equating obtained using a two variable section model.

Comparison of the converted scores obtained using the three different methods for estimating correlations under a one‐variable‐section test model were consistent with one another. Results from this study also demonstrated that equating results obtained from a one‐variable‐section test model were very consistent with those obtained from a two‐variable‐section test model.
EVALUATION OF AL MANAAHIL
Authors: Murphy Richard T.
Published: September 2014

Between April 1985 and January 1988, Educational Testing Service conducted an evaluation of AL MANAAHIL (The Sources), a children's television series in reading for Arab children. The series, produced and broadcast in Jordan, contained 65 half‐hour shows in Modern Standard Arabic (MSA) designed for children in Grades One through Four. The shows were intended to expose children to modern vocabulary and examples of accurate uses of MSA in everyday situations.

More than 5,000 children in Jordan, Morocco, and Tunisia were tested in the evaluation. Based on the Jordan test data, AL MANAAHIL had a positive impact on children's reading skills. The positive findings were replicated in Government schools, United Nations schools, and Private schools. The findings were similar for boys and girls. They held up across different tests. The positive results were more pronounced on items more clearly related to specific goals of the series than on items less clearly related. The majority of children who participated in the testing program reported that they watched the AL MANAAHIL series and liked it.

Responses of teachers, parents, and other adults to an ETS survey about the series were also positive. The majority of teachers (65%) indicated that they liked the series or liked it very much. Most (74%) noted that the series had a positive or very positive impact on the children. When asked about AL MANAAHIL's effect on their teaching, a majority (54%) of teachers indicated that AL MANAAHIL had a positive or very positive effect.

In several adult literacy programs, participants were asked to take the AL MANAAHIL tests and respond to questions about how frequently they viewed the series and how much they liked or disliked it. Of 83 adults in the literacy sample, 33 (40% of the sample) reported that they watched the programs on a regular basis. That group of 33 performed better on the AL MANAAHIL test battery than a group of 28 who did not watch the series at all ‐ a result that supports the overall positive findings of the evaluation.

In summary, based on the data gathered in 1986 and 1987, AL MANAAHIL had a positive effect on children in this study. It is likely to have a similar effect on children who view the series in the future in Jordan and other Arab countries.
CONSIDERATIONS AND TECHNIQUES FOR THE ANALYSIS OF NAEP DATA
Authors: Johnson Eugene G.
Published: September 2014

The special characteristics of the National Assessment data which affect the validity of conventional techniques of statistical analysis are considered. In contrast to the assumptions underlying standard methods of statistical analysis, the NAEP samples are obtained via a stratified multi‐stage probability sampling design in which clusters of students are selected and in which certain subpopulations are sampled at a higher rate. The resulting samples have different statistical characteristics from those of a simple random sample. Analytic techniques which take the structure of the data into account are discussed and their properties explored. A second feature of the National Assessment data arises from the fact that multiple matrix sampling techniques are used for the assignment of subsets of the pool of exercises to subsamples of students. National Assessment accounts for the effect of the missing data on its scaling models through the use of multiple imputation procedures; creating vectors of “plausible values” for the scale‐scores. The use of plausible values and their effect on analysis are illustrated. The analyses are illustrated with results from the NAEP assessments of mathematics.
THE EFFECTS ON IRT AND CONVENTIONAL ACHIEVEMENT TEST EQUATING RESULTS OF USING EQUATING SAMPLES MATCHED ON ABILITY1,2
Authors: Cook Linda L. , Eignor Daniel R. , Schmitt Alicia P.
Published: September 2014

Research by Cook, Eignor, and Taft (1988) has clearly demonstrated the need for equating certain achievement tests using samples of students who take the new and old forms at comparable points in the school year. In their study, Cook et al. equated two forms of an achievement test in Biology using two different sample combinations, a spring new‐form/fall old‐form combination and a fall new‐form/fall old‐form pairing. Equating results were quite different across the two sample combinations and all equatings, based on a variety of methods, done using the spring new‐form/fall old‐form combination provided unacceptable results. However, results from recently completed research by Lawrence and Dorans (1988) suggests that matching samples of differing ability (such as the spring new‐form and fall old‐form samples in the Cook et al. study) on an available covariate prior to equating may improve equating results.

The purpose of the present study was to examine the effects on IRT and conventional equating results of matching the Cook et al. spring new‐form and fall old‐form Biology samples using two different covariate measures: 1) observed scores on an internal common item equating block; and 2) responses to selected questions on the Student Description Questionnaire (SDQ), a questionnaire that candidates respond to when filling out their registration forms for the test. These matched sample equatings were then compared to the unmatched spring new‐form/fall old‐form equating results and to the equatings based on the samples of comparable ability, the fall new‐form/fall old‐form sample combination.

Results of this study indicate that: 1) matching on a set of common items provides greater agreement among the results of the various equating procedures studied than if matching does not take place; and 2) for all equating procedures, the results of the matched group (on the common items) equating agreed more closely with the fall new‐form/fall old‐form criterion equatings than did the fall/spring unmatched equating results. An explanation for why the matching on SDQ responses provided unacceptable equating results is presented in the paper. Finally, the results of this study are compared to the results of other recently completed studies that have also dealt with the matching process.
GRADE LEVEL AND PERFORMANCE ON THE CORE BATTERY, 1986–87 NATIONAL ADMINISTRATION1
Authors: DeMauro Gerald E.
Published: September 2014

Among the three NTE Core Battery tests, Professional Knowledge purports to measure skills that are acquired only after academic experience as an education major, while Communication Skills and General Knowledge both purport to measure skills that do not require this specialized experience. The scores of education majors for the 1986–87 national test administrations were analyzed for differential sensitivity of the three tests to educational attainment. Since examinees with different degrees of experience who choose to take the tests may have different levels of skills, test scores were adjusted for grade point average. Results showed that Professional Knowledge was more sensitive to academic experience than were General Knowledge or Communication Skills. This outcome is interpreted as support of the construct validity of the Core Battery.
CLUSTERS AS THE UNIT OF ANALYSIS IN DIFFERENTIAL ITEM FUNCTIONING
Authors: Bennett Randy Elliot , Rock Donald A. , Novatkoski Inge
Published: September 2014

This study developed, applied, and evaluated a theory‐based method of detecting the underlying causes of differential difficulty. The method was intended to improve on traditional approaches that too often produce uninterpretable results. Key elements of the method were the analysis of item clusters and the incorporation of theoretical predictions about cluster performance. The method was applied in two subgroups taking SAT‐M and involved (1) reviewing literature syntheses to identify factors that might cause differential item functioning, (2) forming item categories based on those factors, (3) identifying categories that functioned differentially, (4) assessing the functioning of the items composing deviant categories, and (5) relating item and category functioning. Results were compared to a traditional item‐level analysis. In both subgroups, the cluster and traditional methods agreed on the overall extent of differential functioning (substantial in the first group, virtually absent in the second). Additionally, the pattern of differential functioning detected was interpretable. At the same time, several important limitations were apparent. The method would seem to be applied most productively when a small number of hypotheses can be derived from a reasonably strong research base, overlap among cluster structures can be avoided, and results can be supplemented with experimental studies or protocol analyses.