bluuebunny/crossref_metadata_2025_split huggingface dataset:
EVALUATING THE WORK EXPERIENCE OF RESPIRATORY THERAPY PRACTITIONERS FOR THE NBRC
Authors: Nettles Steven S. 
Published: September 2014

ABSTRACTThe purpose of this study was twofold: (1) to develop a standardized self‐report instrument that would allow respiratory therapy practitioners applying for registration by the National Board for Respiratory Care (NBRC) to describe their relevant work experience in a manner useful to the NBRC, and (2) to determine if this information was accurate. With the help of an NBRC advisory committee, ETS staff developed an instrument for objectively quantifying the work experience of respiratory therapy practitioners. The instrument, a Work Experience Inventory, provided data on many job characteristics, including level of responsibility on the job, exposure to different work situations, to different job tasks, and to a variety of respiratory knowledges, skills, and abilities. To determine if the instrument was sensitive to differences between Registered Respiratory Therapist (RRT) and nonRegistered Respiratory Therapist (nonRRT) practitioners, two geographically representative random samples (N = 84 and N = 54, respectively) from the above two populations of practitioners were asked to complete the inventory.The majority of the participants found the inventory easy to complete and confirmed that the instrument accurately reflected their work experience. Agreement was obtained between the participants' and their employers' ratings on several characteristics used to estimate the level of responsibility present in participants' work experience. Statistical analysis indicated that all sections of the inventory collected unique information. Items in all sections of the instrument distinguished between the registered and nonregistered practitioners.Because of the encouraging results of this demonstration project, project staff recommend that the NBRC use the instrument in a pilot study with substantially larger samples of registry applicants, to determine definitively the feasibility of integrating the Work Experience Inventory into the NBRC credentialing program.
THE PSYCHOLOGY OF EDUCATIONAL MEASUREMENT
Authors: Messick Samuel 
Published: September 2014

ABSTRACTSince school learning entails not just accretion of knowledge but the structuring and restructuring of knowledge and cognitive skills, the conception and construction of educational achievement measures must be cast in developmental terms. Since student characteristics as well as social and educational experiences influence current performance, the interpretation and implications of educational achievement measures must be relative to intrapersonal and situational contexts. These points imply a strategy of comprehensive assessment in context that focusses on the processes and structures involved in subject‐matter competence as moderated in performance by personal and environmental influences. This article addresses in detail both the nature of developing competence and its measurement in terms of context‐dependent task performance. Construct‐irrelevant task difficulty that might jeopardize the meaning of test scores as well as construct‐irrelevant influences that might jeopardize implications for action are taken into account via the comprehensive measurement of relevant contextual factors. Comprehensive assessment in context thus facilitates valid interpretations of the meaning and implications of ability and achievement scores in particular instances, thereby lightening the interpretive and ethical burdens on test users and enhancing the validity of test use.
TESTING THE LOCAL INDEPENDENCE ASSUMPTION IN ITEM RESPONSE THEORY
Authors: Rosenbaum Paul R. 
Published: September 2014

ABSTRACTLocal independence and monotone increasing item characteristic curves imply nonnegative conditional covariances between all monotone increasing functions of a set of item responses given any function of the remaining item responses. This general result provides a basis for testing the local independence assumption without first specifying a parametric form for the item characteristic curve. The proposed tests are simple, have known asymptotic null distributions, and possess certain optimal properties. In an example, the local independence hypothesis is rejected for all possible forms of the monotone item characteristic curves.
THE ASSESSMENT OF WRITING ABILITY: A REVIEW OF RESEARCH
Authors: Cooper Peter L. 
Published: September 2014

ABSTRACTThe assessment of writing ability has recently recevied much attention from educators, legislators, and measurement experts, especially because the writing of students in all disciplines and at all educational levels seems, on the whole, less proficient than the writing produced by students five or ten years ago. The GRE Research Committee has expressed interest in the psychometric and practical issues that pertain to the assessment of writing ability. This paper presents not a new study but a review of major research in light of GRE Board concerns. Specifically, recent scholarship and information from established programs are used to investigate the nature and limitations of essay and multiple‐choice tests of writing ability, the statistical relationship of performances on these types of tests, the performance of population subgroups on each kind of task, the possible need of different disciplines for different tests of composition skill, and the cost and usefulness of various strategies for evaluating writing ability.The literature indicates that essay tests are often considered more valid than multiple‐choice tests as measures of writing ability. Certainly they are favored by English teachers. But although essay tests may sample a wider range of composition skills, the variance in essay test scores can reflect such irrelevant factors as speed and fluency under time pressure or even penmanship. Also, essay test scores are typically far less reliable than multiple‐choice test scores. When essay test scores are made more reliable through multiple assessments, or when statistical corrections for unreliability are applied, performance on multiple‐choice and essay measures can correlate very highly. The multiple‐choice measures, though, tend to overpredict the performance of minority candidates on essay tests. It is not certain whether multiple‐choice tests have essentially the same predictive validity for candidates in different academic disciplines, where writing requirements may vary. Still, at all levels of education and ability, there appears to be a close relationship between performance on multiple‐choice and essay tests of writing ability. And yet each type of measure contributes unique information to the overall assessment. The best measures of writing ability have both essay and multiple‐choice sections, but this design can be prohibitively expensive. Cost cutting alternatives such as an unscored or locally scored writing sample may compromise the quality of the essay assessment. For programs considering an essay writing exercise, a discussion of the cost and uses of different scoring methods is included. The holistic method, although having little instructional value, offers the cheapest and best means of rating essays for the rank ordering and selection of candidates.
CONSIDERATIONS FOR DEVELOPING MEASURES OF SPEAKING AND LISTENING
Authors: Powers Donald E. 
Published: September 2014

ABSTRACTThe College Board has identified several basic intellectual competencies thought to be essential for effective work in all fields of college study, among them listening and speaking. An issue that arises in connection with these competencies is the availability of suitable measures to assess students' development in these areas.This report considers the availability and adequacy of existing measures of speaking and listening, and discusses a number of issues that should be considered in any efforts to develop new measures of these skills.
ARE THE ITEM RESPONSE PATTERNS OF TWO GROUPS OF EXAMINEES CONSISTENT WITH A DIFFERENCE IN THE DISTRIBUTION OF A UNIDIMENSIONAL LATENT VARIABLE?
Authors: Rosenbaum Paul R. 
Published: September 2014

ABSTRACTIf one group of examinees can be described as having generally higher ability to respond correctly to the items on an exam than some other group – where ability is defined by an unidimensional item response model – then the observable distributions of item responses must satisfy certain constraints. Standard statistical methods for discrete data may be used to test the consistency of these constraints with observed data. In an example, the observed pattern of item responses is found to be inconsistent with a difference in the distribution of a unidimensional latent variable.
USE OF EMPIRICAL BAYES METHODS IN THE STUDY OF THE VALIDITY OF ACADEMIC PREDICTORS OF GRADUATE SCHOOL PERFORMANCE
Authors: Braun Henry I. , Jones Douglas H. 
Published: September 2014

ABSTRACTGraduate education in the United States is characterized by an enormous diversity of disciplines and the predominance of relatively small enrollments in individual departments. In this setting, a validity study based on a single department's data and employing classical statistical methods can only be of limited utility and applicability. In order to participate in the Graduate Record Examinations Validity Study Service, a department must have at least 25 students in its entering class. Only validities for single predictors are provided; estimates of the validity of two or more predictors, used jointly, are considered too unreliable because the corresponding prediction equations often possess implausible characteristic, such as negative coefficients. These constraints were introduced by the Validity Study Service to reduce the chance that the results in the report to a department would be overly influenced by statistical artifacts in the data and hence serve more to mislead than to inform.In this study we investigated two statistical methods, empirical Bayes and cluster analysis, to determine whether their application to the problems faced by the Validity Study Service could result in useful improvements. Considerable effort was expended in developing and studying a new and more general class of empirical Bayes models that can accommodate the complex structure of the Validity Study Service data base.The principal methodological conclusions of this study are that, through the use of a new class of empirical Bayes methods, it is possible to obtain, at the departmental level, useful and reliable estimates of the joint validity of several predictors of academic performance and that these methods may be further refined to address the question of differential predictive validity, again at the departmental level. These results have important practical implications for the GRE Validity Study Service.
BAYESIAN ESTIMATORS, ROBUST ESTIMATORS: A COMPARISON AND SOME ASYMPTOTIC RESULTS
Authors: Jones Douglas H. 
Published: September 2014

ABSTRACTCombinations of Bayes and resistant estimators of latent ability were studied in Wainer and Thissen (1984), who show that they have desirable performances for small to medium samples especially in the presence of contamination. While their study was based on computer simulations, this note provides the first order asymptotic theory, from which one may compute approximations to the standard errors of the estimators. A side issue concerns the “overshrinking phenomenon” which caused Wainer and Thissen to expand the scale of the estimators in order to increase their accuracy and to make them more comparable to the traditional estimators. This note derives a differential equation specifying a transform that may make Bayes/resistant estimators even more favorable.
REPORT OF A STUDY OF THE CONTENT OF THREE GRE ADVANCED TESTS
Authors: DeVore Richard , McPeek Miles 
Published: September 2014

ABSTRACTThis paper reports the findings from content studies of three GRE Advanced Tests: Biology, Literature in English, and Political Science. The studies were designed to indicate the degree to which the content specified for and included in recent test forms for these fields is representative of the content actually taught in the undergraduate curricula in the fields. The studies provided substantively useful information concerning the content representativeness of the tests.Informed judgments were relied on as the technique for assessing the representativeness of the content. Groups of college and university faculty knowledgeable in curricular practices at the undergraduate level were asked to make judgments about the relationship between curriculum and test content by examining both individual test questions and the content specifications for the tests. By inference, within the error limits of the judgments, an assessment of the relevance of test content to undergraduate curricula was obtained.Although the studies supply evidence that the content of each of the tests generally is representative of the content of the respective undergraduate curricula, modest differences of opinion were found among the respondents in the Biology and Political Science studies as to the degree of emphasis that should be given to certan subareas of the content.
PROGRESS TOWARD STANDARDS AS STANDARDS FOR PROGRESS: A POTENTIAL ROLE FOR THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Authors: Messick Samuel 
Published: September 2014

ABSTRACTAs currently constituted and implemented, the National Assessment of Educational Progress incorporates three key elements of a responsible standard‐setting process — namely, the choice of educational objectives, the description of current group performance ranges and trends, and the identification of educational contexts differentially related to performance. What is still required to move on to the setting of quality educational standards are value judgments as to which ranges of performance are deemed unsatisfactory, adequate, or excellent by different societal groups. This article addresses the type of pluralistic judgmental process needed for standard‐setting when a diversity of values as well as consensus must be taken into account.
AN INVESTIGATION OF THE FEASIBILITY OF APPLYING ITEM RESPONSE THEORY TO EQUATE ACHIEVEMENT TESTS1,2,3
Authors: Cook Linda L. , Eignor Daniel R. 
Published: September 2014

ABSTRACTThe purpose of this study was to examine the feasibility of using item response theory (IRT) procedures to equate different forms of three achievement tests (Biology, American History and Social Studies, and Mathematics Level II) that are administered by the College Board Admissions Testing program and one achievement test (Advanced Biology) that is part of the Graduate Record Examinations Achievement Test battery. All of the tests investigated in this study are typically equated using conventional linear or curvilinear (equipercentile) methods.Because the investigators had access to data from large scale testing programs that administer many different forms of a test, it was possible to design the study such that a unique criterion for evaluating the equating results, scale drift, was available. Scale drift will occur if the result of equating, say, Form A to Form C through Form B is not the same as that obtained by equating Form A directly to Form C. This study was designed so that for all the Achievement tests investigated, the drift in each scale when IRT equating was used could be compared to the drift occurring when the conventional linear and equipercentile equating methods, used operationally to put new forms on scale, were applied.The results of the study indicate, using scale drift as the evaluative criterion, that it is feasible to use item response theory to equate the four Achievement tests selected for investigation. The results also indicate that the conventional linear methods typically used to equate the tests perform quite adequately. The question of whether the IRT procedures used in the study are sufficiently robust to violations of the assumption of unidimensionality, or whether Achievement tests, of the type used in this study, give rise to sufficiently unidimensional data, must be resolved before the results of the study can be generalized to other Achievement testing situations.
MINORITY CONTRIBUTIONS TO THE SAT SCORE TURNAROUND: AN EXAMPLE OF SIMPSON'S PARADOX
Authors: Wainer Howard 
Published: September 2014

ABSTRACTSince 1980 the decline in SAT scores has stopped and the scores have started to creep back up. The scores for White Americans have increased 8 points during this period, for non‐Whites 15 points. It was thus surprising to discover that the overall mean increased only 7 points. This is not an arithmetic error but rather an example of a well‐known statistical phenomenon called Simpson's Paradox. In this note we explain the paradox and describe a method which will avoid it in the future.
BACKGROUND CHARACTERISTICS OF EXAMINEES SHOWING UNUSUAL TEST BEHAVIOR ON THE GRADUATE RECORD EXAMINATIONS
Authors: Oltman Philip K. 
Published: September 2014

ABSTRACTWe ordinarily expect item difficulty to be related to errors on tests; examinees generally tend to make errors on more difficult items and to answer easier items correctly. However, some examinees miss easy items and get more difficult items correct. The extent to which correct and incorrect responses are predicted by the difficulty of test items has been quantified in various ways. One method, originated by Sato (1975) and modified by Harnisch and Linn (1981), was used in the present study of item level data from the Graduate Record Examinations General Test. The modified Sato caution index was found to be of low reliability in these data and to be generally unrelated to a variety of background variables, although ethnic group showed a small but significant relation to the index. In these data the modified caution index showed a curvilinear relation to total test score, with examinees who showed very high or very low scores having higher index values than those in the middle range of test scores. Indexes calculated on the three sections of the test were uncorrelated with each other. Finally, the index did not moderate the relationship between test scores and self‐reported grades, which it should have done if it indeed indicates how well the test measures the intended construct for any individual. The conclusion reached is that the modified caution index adds little information that would be of value in interpreting GRE test scores.
A STUDY OF THE TEMPORAL STABILITY OF IRT ITEM PARAMETER ESTIMATES1,2
Authors: Cook Linda L. , Eignor Daniel R. , Petersen Nancy S. 
Published: September 2014

ABSTRACTThis research report focuses on the stability of item response theory (IRT) item parameter estimates when the items are calibrated on two different samples of examinees who have responded to the items at two different points in time, i.e., the temporal stability of the parameter estimates. The data were collected from regular administrations of the College Board Admissions Testing Program Scholastic Aptitude Test (SAT) and Achievement Tests. The three‐parameter logistic model was used to characterize the relationship between the underlying trait and performance on an item and a variety of methods were used to assess parameter estimate stability.Two important conclusions were drawn from the study. One, stability of parameter estimates is clearly related to type of test. Parameter estimates obtained from SAT‐verbal and SAT‐mathematical are more likely to exhibit stability over time than those obtained for achievement tests of Biology or American History and Social Studies. Secondly, lack of stability of item parameter estimates appears to be more closely related to differences in ability among calibration samples than to the lapse of time between administrations of the test, which, in the study, ranged from 14 to 52 months. It should be noted, however, that for the particular tests studied, ability differences in calibration samples appeared to be unrelated to time differences between administrations. This may not be typical of other testing situations, where ability differences in the calibration samples can be directly related to length of time between test administrations. This could be brought about, for example, by changes in curricular emphasis.
A BAYESIAN TREATMENT OF LATENT VARIABLES IN SAMPLE SURVEYS
Authors: Mislevy Robert J. 
Published: September 2014

ABSTRACTConsider the problem of estimating a functionof values of the variables z and y in a finite population, when values of y are known a priori for all units but values of z are not known for any. Data are then obtained from a sample of units; not in the form of values of z, however, but in the form of secondary variables x whose values depend stochastically on z. A framework for Bayesian analysis of T along the lines of Ericson (1969) is presented and illustrated with data from theProfile of American Youth(U.S. Department of Defense 1982).
CALIBRATION OF ESSAY READERS FINAL REPORT
Authors: Braun Henry I. 
Published: September 2014

ABSTRACTScoring reliability of essays and other free response questions is of considerable concern. This report describes a statistically designed experiment that was carried out in an operational setting to determine the contributions of different sources of variation to the unreliability of scoring. The experiment made novel use of partially balanced incomplete block designs that facilitated the unbiased estimation of certain main effects without requiring readers to assess the same paper several times. In addition, estimates were obtained of the improvement in reliability that result from removing variability from systematic sources of variation by an appropriate adjustment of the raw scores. This statistical calibration appears to be a cost‐effective approach to enhancing scoring reliability when compared to simply increasing the number of readings per paper. The results of the experiment also provide a framework for examining other, simpler calibration strategies. One such strategy is briefly examined.
A MULTIVARIATE PERSPECTIVE ON CONTINGENCY TABLE ANALYSIS
Authors: Zwick Rebecca , Cramer Elliot M. 
Published: September 2014

ABSTRACTPsychological research often involves analysis of an I × J contingency table consisting of the responses of J groups of individuals on a criterion variable with I nominal categories. The conventional statistical approach for comparing responses across groups is the Pearson chi‐square test. Alternatively, this analysis can be viewed as a multivariate analysis of variance with binary dependent variables, a canonical correlation analysis with two sets of binary variables, or a form of correspondence analysis. Although these analysis approaches stem from different traditions, they produce equivalent results when applied to an I × J table.
AN INVESTIGATION OF POSSIBLE CAUSES FOR THE INADEQUACY OF IRT PRE‐EQUATING1,2,3
Authors: Eignor Daniel R. , Stocking Martha L. 
Published: September 2014

ABSTRACTA previous study of pre‐equating the Scholastic Aptitude Test (SAT) using item response theory provided unacceptable equating results for SAT‐mathematical data. The purpose of this study was to investigate two possible explanations for these unacceptable pre‐equating results. Specifically, the calibration process, which made use of the three‐parameter model and LOGIST, and the linking procedure used to place parameter estimates on the same scale were further investigated in a two stage process to see if either was responsible for the poor IRT pre‐equating results found for the SAT‐mathematical data in the previous study.