bluuebunny/crossref_metadata_2025_split huggingface dataset:
MARGINAL MAXIMUM LIKELIHOOD ESTIMATION FOR A PSYCHOMETRIC MODEL OF DISCONTINUOUS DEVELOPMENT
Authors: Mislevy Robert J. , Wilson Mark 
Published: August 2014

ABSTRACTStandard item response theory (IRT) models posit latent variables to account for regularities in students' performances on test items. They can accommodate learning only if the expected changes in performance are smooth and, in an appropriate metric, uniform over items. Wilson's “Saltus” model extends the ideas of IRT to development that occurs in stages, where expected changes can be discontinuous, show different patterns for different types of items, and even exhibit reversals in probabilities of success on certain tasks. Examples include Piagetian stages of psychological development and Siegler's rule‐based learning. This paper derives marginal maximum likelihood (MML) estimation equations for the structural parameters of the Saltus model and suggests a computing approximation based on the EM algorithm. For individual examinees, Empirical Bayes probabilities of learning‐stage are given, along with proficiency parameter estimates conditional on stage membership. The MML solution is illustrated with simulated data and an example from the domain of mixed number subtraction.
ITEM DIFFICULTY ADJUSTMENT STUDY: GRE VERBAL DISCRETES
Authors: Adams Richard , Carson John , Cureton Kevin 
Published: August 2014

ABSTRACTThe purpose of this study was to determine whether it is both possible and cost‐effective to revise middle‐difficulty GRE discrete items in order to produce items of higher or lower difficulty. It was found that it is significantly easier to increase the difficulty of middle‐difficulty items than to reduce the difficulty of such items, and that the difficulties of antonyms and analogies are much easier to manipulate than those of sentence completions. The evidence also suggests that producing harder analogies and antonyms by revising items in this manner would be a cost‐effective procedure.
THE OPTIONAL ESSAY PROBLEM AND THE HYPOTHESIS OF EQUAL DIFFICULTY
Authors: Allen Nancy L. , Holland Paul W. , Thayer Dorothy T. 
Published: August 2014

ABSTRACTWe examine a special case of examinee choice, the Optional Essay Problem, from the point of view of test equating. The Optional Essay Problem involves equating essay scores when the examinees are required to select an optional essay topic from a list of topics in addition to taking a mandatory test required of all examinees. We derive conditions that must be satisfied if the null hypothesis of ‘equal difficulty’ of the essays is true. (We call this Livingston's Null Hypothesis.) If this hypothesis holds then there is no need to equate the scores on the optional essays. Our conditions take the form of inequalities about unobservable quantities that may be displayed graphically. We illustrate them using a real example from the Advanced Placement Examinations. Then we analyze Livingston's (1988) proposal for adjusting essay scores in the Optional Essay Problem and explain it from the perspective of test equating. We use our explanation to generalize his proposal to two new proposals that are explicit about the assumptions they make concerning the unobserved data. (We argue that every method for adjusting the essay scores in the Optional Essay Problem must make assumptions about unobserved data.) We illustrate the adjustment methods with an example from the Advanced Placement Examinations. Finally, we use the results for adjusting optional essay scores to propose comparable procedures for directly adjusting linear composite scores that include both a mandatory and an optional test score.
THE E‐STEP OF THE MGROUP EM ALGORITHM
Authors: Thomas Neal 
Published: August 2014

ABSTRACTMislevy(1984, 1985) introduced an EM algorithm for estimating the parameters of a latent distribution model that is used extensively by the National Assessment of Educational Progress. Second order asymptotic corrections are derived and applied along with more common first order asymptotic corrections to approximate the expectations required by the E‐step of the EM algorithm. These corrections produce much more accurate approximations than those produced by two different normal approximations. The asymptotic corrections are applicable in high dimensional models. The number of function evaluations required by these methods grows linearly as dimension increases, in contrast to the exponential growth with dimension of product quadrature rules which yield similar accuracy.
SIMULATED EQUATING USING SEVERAL ITEM RESPONSE CURVES
Authors: Boldt R. F. 
Published: August 2014

ABSTRACTPrevious research has shown that a single factor, factor being used in the sense of factor analysis, gave a very good account of item covariances within TOEFL® sections. This result is consistent with the assumption that the product of a person parameter and an item parameter models the probability that the person would pass the item. The assumption forms a simple item response model. A subsequent cross‐validation study using this model supported the efficacy of that assumption by predicting item success as accurately as did the 3‐Parameter Logistic model, and a modified Rasch model. The purpose of the current study was to extend the comparison of models to an equating context. “Equating” is a statistical process that identifies comparable scores from parallel tests administered to different populations. In an operational context, equating serves to facilitate comparison of scores generated on different forms of a test.The present study consisted of simulation trials designed to “equate the test to itself.” That is, equating sample data were generated from administration of identical item sets. It is useful to do this as a test of model validity, because if the same item sets are used to equate, an accurate equating would identify equal scores as comparable. Discrepancies between comparable scores signify error model misfit or random error.Equatings that used procedures based on each model were accomplished under several conditions and the results were compared. The conditions varied by sample size, anchor test difficulty, and the TOEFL section equated. In order to compound the difficulty of the equating task, results were based on equating samples that were mismatched in performance on a correlated measure.Most discrepancies between comparable scores were largest at the extremes. The largest discrepancies between scores identified as comparable occurred for the 3PL and modified Rasch models at the lower extreme scores, and for the simple models at the upper extreme score. For the 1,000‐case sample, most were in fractions of score points. As expected, 3PL equatings exhibited the largest discrepancies for the 100‐case sample. The simple item response model yielded the most discrepancies that were in excess of the standard error of measurement, in part because with that model the maximum discrepancies occurred at the top of the score range, where the standard errors of measurement approach zero. Imposing an upper bound on the probability of correct response in the simple model markedly reduced its errors.TOEFL scores are used for educational decisions. If it is true that most institutions' cut scores occur in the mid‐score ranges, the present study suggests that 3PL should not be used if equating samples are substantially reduced from the present size. The other models are promising for small‐sample equating, with the one‐parameter logistic models being most promising.
THE EFFECT OF SMALL CALIBRATION SAMPLE SIZES ON TOEFL IRT‐BASED EQUATING
Authors: Tang K. Linda , Way Walter D. , Carey Patricia A. 
Published: August 2014

ABSTRACTThe present study compared the performance of LOGIST and BILOG on TOEFL IRT‐based scaling and equating using both real and simulated data and two calibration structures. Applications of IRT for the TOEFL program are based on the three‐parameter logistic (3PL) model.The results of the study show that item parameter estimates obtained from the smaller real data sample sizes were more consistent with the larger sample estimates when based on BILOG than when based on LOGIST. In addition, the root mean squared error statistics suggest that the BILOG estimates for the item parameters and item characteristic curves were closer in magnitude to the “true” parameter values than were the LOGIST estimates.The equating results based on the parameter estimates suggest that the rule of thumb recommendation that pretest sample sizes be at least 1000 for LOGIST should be retained if at all possible.
PROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS
Authors: Mislevy Robert J. 
Published: August 2014

ABSTRACTRecent developments in cognitive psychology suggest models for knowledge and learning that often fall outside the realm of standard test theory. This paper concerns probability‐based inference in terms of such models. An approach utilizing Bayesian inference networks is outlined. Basic ideas of structure and computation in inference networks are discussed, and illustrated with an example from the domain of mixed‐number subtraction.
ASSESSING THE CLASSROOM PERFORMANCE OF BEGINNING TEACHERS: WHAT EVIDENCE DO TEACHERS REGARD AS RELEVANT?
Authors: Morris Lori V. , Powers Donald E. 
Published: August 2014

ABSTRACTOne of the three components of The Praxis Series: Professional Assessments for Beginning Teachers™ is a classroom performance assessment concerning the new teacher's ability to apply basic elements of good teaching in a classroom setting. At the core of this classroom assessment are 19 criteria for which assessors are to identify examples of evidence supporting whether or not a teacher can perform the function(s) implied by each criterion. The purpose of this study was to delineate further the kinds of teacher cognitions, decisions, and behaviors that may appropriately be regarded as evidence for each criterion. The focus was to establish the kinds of “generic” evidence that might cut across contexts, as well as determining what might be specific to particular situations. Survey responses from 131 teachers across subject matters are analyzed and both positive and negative evidence statements are developed for each criterion. Examples of both “generic” evidence and subject matter specific evidence are provided. The data gathered in this study could be useful for training assessors for the classroom assessment component of The Praxis Series: Professional Assessments for Beginning Teachers™.
ESTIMATING SCORES FOR AN OPTIONAL TEST SECTION USING INFORMATION FROM A COMMON SECTION*
Authors: Allen Nancy L. , Holland Paul W. , Thayer Dorothy T. 
Published: August 2014

ABSTRACTSome tests include a section of optional questions in addition to mandatory parts of the test. The purpose of this paper is to provide a general method based on missing‐data methods for nonignorable nonresponse (Little & Rubin, 1987) to estimate distributions of scores on an optional part of the test taking into account information from a mandatory part of the test. The scores for the optional section can then be placed on comparable scales using “equating” or moderation methods.This paper is the third in a series of three papers. In the first “Approaches to Nonignorable Nonresponse with Application to Selection Bias” (Allen, Holland, & Thayer, in press, a), we apply missing‐data methods from settings external to education to this setting. A method from the econometric literature (Heckman, 1976) is contrasted to methods proposed by Little and Rubin (1987) and Tukey (Holland, 1986). In the second paper, “A Missing Data Approach to Estimating Distributions of Scores for Optional Test Sections” (Allen, Holland, & Thayer, in press, b) the perspective of measurement specialists who proposed methods to transform scores for optional questions to comparable scales is contrasted with the approach based on the work of Little and Rubin. In this paper, the methods of Rubin and Tukey are extended to include common scores for the mandatory part of the test. Data from an example are actually transformed to comparable scales.
A COMPARISON OF THE PROPERTIES OF IRT PARAMETER ESTIMATES USING TWO DIFFERENT CALIBRATION DESIGNS
Authors: Wightman Lawrence E. , Champlain Andre F. De 
Published: August 2014

ABSTRACTThe purpose of this study was to compare two different methods of obtaining 3PL IRT pretest item parameter estimates for the Graduate Management Admissions Testing Program. The first method consisted of calibrating pretest and operational items simultaneously in a LOGIST run, that is, a concurrent calibration design. The second approach entailed analyzing the pretest items separately from the operational items holding examinee ability scores constant from a previous operational items run, that is, using a two‐stage calibration design. Results show that the means of the item difficulty (b‐parameter) estimates were very similar, regardless of the method employed. However, the higher b‐parameter values using the two‐stage calibration run method (i.e. holding ability fixed excluding the studied items from the criterion) were slightly overestimated and the lower b‐parameter values were slightly underestimated. The a‐parameters were consistently underestimated using the two‐stage estimation procedure. Finally, the slopes of the item‐ability regressions using a concurrent calibration (including the studied items) are steeper for nearly all of the items. These preliminary results are consistent with those reported in past studies (Stocking & Eignor, 1986) and suggest that non‐operational (pretest) items should be calibrated concurrently with operational items for item banking purposes.
GENDER DIFFERENCES IN HIGH SCHOOL GRADES: AN EXPLORATORY STUDY
Authors: Ekstrom Ruth B. 
Published: August 2014

ABSTRACTThis study used the 1980 High School and Beyond data set to examine the variables associated with the grades that college‐bound high school sophomores received in English, algebra, and geometry courses. Special concerns included determining if gender differences in high school grades could be explained and how teachers' perceptions of students, student characteristics, and High School and Beyond test scores were related to grades. A model of factors that might affect grades was developed; it included students' background characteristics, general attitudes, curriculum, educational aspirations, attitudes toward school and individual academic subjects, school behaviors, and scores on relevant High School and Beyond tests as well as teachers' perceptions of the students. The full models explained 46 percent of the variance in English grades, 42 percent of the variance in algebra grades, and 44 percent of the variance in geometry grades. The full models tended to explain more of the variance in males' than in females' grades, with a more marked difference in English and Algebra 1 than in Algebra 2 or geometry. This suggests that some other variable, not included in the models, may have been affecting grades.Even after controlling for all the variables in the full models, a significant association between gender and course grades remained in English and in Algebra 1; there was no significant relationship between gender and grades in Algebra 2 or in geometry. Teachers' comments were significantly associated with grades for both males and females in English, Algebra 1, and geometry, but they were not significantly associated with males' grades in Algebra 2. Teachers' perceptions of students were significantly associated with gender even after controlling for all the variables in the full models. These findings suggest that teachers may have gender‐related expectations for the students in their courses. The High School and Beyond verbal test scores were significantly associated with grades for both females and males in English and High School and Beyond math test scores were significantly associated with grades in geometry for both males and females. High School and Beyond math test scores were not however consistently associated with grades in algebra.
VARIABLE SCREENING FOR CLUSTER ANALYSIS
Authors: Donoghue John R. 
Published: August 2014

ABSTRACTInclusion of irrelevant variables in a cluster analysis adversely affects subgroup recovery. This paper examines using moment‐based statistics to screen variables; only variables which pass the screening are then used in clustering. Normal mixtures are analytically shown often to possess negative kurtosis. Two related measures,mand coefficient of bimodalityb, are also examined.A Monte Carlo study compared the screening measures to no selection, De Soete's (1988) ultrametric weights, and Fowlkes, Gnanadesikan, and Kettenring's (1988) forward selection procedure. Screening based on kurtosis degraded recovery and is not recommended. In contrast, screening onmor onbimproved recovery over both no selection and forward selection, and screening performed as well as ultrametric weights. Combining screening with ultrametric weights performed extremely well. All methods were found to be somewhat sensitive to other types of error.Screening variables appears a viable alternative to both ultrametric weights and forward selection. The potential advantages and disadvantages of screening are considered.
VALIDITY OF PSYCHOLOGICAL ASSESSMENT: VALIDATION OF INFERENCES FROM PERSONS' RESPONSES AND PERFORMANCES AS SCIENTIFIC INQUIRY INTO SCORE MEANING
Authors: Messick Samuel 
Published: August 2014

ABSTRACTThe traditional conception of validity divides it into three separate and substitutable types – namely, content, criterion, and construct validities. This view is fragmented and incomplete, especially in failing to take into account evidence of the value implications of score meaning as a basis for action and of the social consequences of score use. The new unified concept of validity interrelates these issues as fundamental aspects of a more comprehensive theory of construct validity addressing both score meaning and social values in both test interpretation and test use. That is, unified validity integrates considerations of content, criteria, and consequences into a construct framework for empirically testing rational hypotheses about score meaning and theoretically relevant relationships, including those of both an applied and a scientific nature. Six distinguishable aspects of construct validity are highlighted as a means of addressing central issues implicit in the notion of validity as a unified concept. These are content, substantive, structural, generalizability, external, and consequential aspects of construct validity. In effect, these six aspects function as general validity criteria or standards for all educational and psychological measurement, including performance assessments, which are discussed in some detail because of their increasing emphasis in educational and employment settings.
DEFINING THE CONTENT DOMAIN FOR THE PRAXIS II SUBJECT ASSESSMENT IN EARTH AND SPACE SCIENCE: KNOWLEDGE IMPORTANT FOR BEGINNING TEACHERS
Authors: Tannenbaum Richard J. 
Published: August 2014

ABSTRACTThe purpose of this study was to conduct a job analysis focusing on the knowledge important for beginning Earth and Space Science teachers. The results of the job analysis will be used to define the content domain of the Subject Assessment in Earth and Space Science for the Praxis Series: Professional Assessments for Beginning Teachers®.A domain of 292 knowledge statements and elements (specific descriptors used to clarify the meaning of each, more broadly defined, knowledge statement) was developed by subject‐matter experts. These statements and elements were then incorporated into a survey that was administered to a large number of Earth and Space Science teachers, teacher educators, and state administrators. These professionals rated the importance of each knowledge statement and knowledge element using a 0‐ to 4‐point scale.A cut point of a mean of 2.50 (midpoint between moderately important and important) was established to differentiate important knowledge (M ≥ 2.50) and unimportant knowledge (M < 2.50). The results indicated that a total of 36 knowledge statements and knowledge elements (12% of the total domain) failed to pass the cut point. It is recommended that these 36 statements and elements not be included in the development of the Subject Assessment in Earth and Space Science; emphasis, instead, should be put on the 256 knowledge statements and knowledge elements that did pass the cut point.
STANDARDS‐BASED SCORE INTERPRETATION: ESTABLISHING VALID GROUNDS FOR VALID INFERENCES
Authors: Messick Samuel 
Published: August 2014

ABSTRACTThe construct validity of content standards is addressed in terms of their representative coverage of a construct domain and their alignment with the students' cognitive level of developing expertise in the subject matter. The construct validity of performance standards is addressed in terms of the extent to which they reflect increasing levels of construct complexity as opposed to construct‐irrelevant difficulty. Also critical is the extent to which performance standards characterize the knowledge and skills operative at each level both to accredit specific accomplishment and to serve as goals for further learning. All of this depends on construct‐valid assessment attuned to the content standards and the development of dependable scoring rubrics and measurement scales for representing the performance standards.
FEASIBILITY OF USING THE SAT®IN ACADEMIC GUIDANCE
Authors: STRICKER LAWRENCE J. , ROCK DONALD A. , BURTON NANCY W. 
Published: August 2014

ABSTRACTThis study appraised the validity of SAT scores, in combination with grades in high school courses and the number and difficulty level of these courses, in predicting college grades in various fields of study, with the objective of providing SAT takers with predictions of their academic performance in different fields for guidance purposes. The possible impact of this feedback on the flow of students into specific major fields was also assessed. Data on an entering class at a large state university provided the basis for the study. The SAT and collateral variables were able to predict college grades in different academic areas by taking into account marked variations in grade distributions among fields of study. These predictions could be potentially useful in academic guidance. Students' predicted grades in the various fields and their intended majors were unrelated. Most students received their highest predicted grades in the humanities, although few intended to major in this area. Hence, such feedback would probably stimulate students to take courses or major in the humanities.
DETECTING DIF FOR POLYTOMOUSLY SCORED ITEMS: AN ADAPTATION OF THE SIBTEST PROCEDURE
Authors: Chang Hua‐Hua , Mazzeo John , Roussos Louis 
Published: August 2014

ABSTRACTRecently, Shealy and Stout (1993) proposed a DIF detecting procedure SIBTEST, which is 1) IRT model based, 2) non‐parametric, 3) does not require IRF estimation, 4) provides a test of significance, and 5) estimates the amount of DIF. Current versions of SIBTEST can only be used for dichotomously scored items. However, in this paper an extension to handle polytomous items is developed. This paper presents: (1) a discussion of an appropriate definition of DIF for polytomously scored items, (2) a modified SIBTEST procedure for detecting DIF for polytomous items, and (3) the results of two simulation studies comparing the modified SIBTEST with the Mantel and SMD procedures, one study with data constrained by the Rasch‐like partial credit model (same discrimination across polytomous items), and the other study with data having distinctly discrimations across items. These simulation studies indicate that the methodology ofincluding the studied item in matching subtestfor controling impact induced (group ability differences existing) Type I error tends to yield Type‐I/Type II error inflation rates that are highly unacceptable when theequal discriminationcondition is violated. These simulation studies provide compelling evidence that the modified SIBTEST procedure is much more robust with regard to controlling impact‐induced Type I error rate inflation than the other procedures.