bluuebunny/crossref_metadata_2025_split huggingface dataset:
A STUDY OF FREQUENCY ESTIMATION EQUIPERCENTILE EQUATING WHEN THERE ARE LARGE ABILITY DIFFERENCES
Authors: Guo Hongwen , Oh Hyeonjoo J. 
Published: August 2014

ABSTRACTIn operational equating, frequency estimation (FE) equipercentile equating is often excluded from consideration when the old and new groups have a large ability difference. This convention may, in some instances, cause the exclusion of one competitive equating method from the set of methods under consideration. In this report, we study the possibility of using the FE equating method when the group ability difference is large. Three situations are identified: (a) a situation in which neither the two forms nor the observed conditional distributions are very different so that the FE equating assumptions are likely to hold, and FE equating is recommended; (b) a situation in which forms are not very different, but the observed conditional distributions are different, so that FE equating is not recommended; and (c) a situation in which forms are very different, but the observed conditional distributions are not different, so that FE equating is not recommended. Statistical analysis procedures for comparing distributions are provided. An application of equating to a large‐scale admission test is discussed to illustrate the proposed methodology.
PSEUDO BAYES ESTIMATES FOR TEST SCORE DISTRIBUTIONS AND CHAINED EQUIPERCENTILE EQUATING
Authors: Moses Tim , Oh Hyeonjoo J. 
Published: August 2014

ABSTRACTPseudo Bayes probability estimates are weighted averages of raw and modeled probabilities; these estimates have been studied primarily in nonpsychometric contexts. The purpose of this study was to evaluate pseudo Bayes probability estimates as applied to the estimation of psychometric test score distributions and chained equipercentile equating functions. Population test score distributions were created from actual test data and random samples of varied size were drawn from the populations. Pseudo Bayes estimation was applied to the random samples, using ranges of loglinear models and weights in the pseudo Bayes estimates' weighted averages of the raw and modeled test score probabilities. Equipercentile equating functions based on the pseudo Bayes estimates were also evaluated. Results indicated that the pseudo Bayes estimates have the potential to improve estimation accuracy for test score distributions and chained equipercentile equating functions in situations where loglinear modeling is not ideal and where finding the population loglinear model selection is not likely.
DIFFERENTIAL ITEM FUNCTIONING: ITS CONSEQUENCES
Authors: Lee Yi‐Hsuan , Zhang Jinming 
Published: August 2014

ABSTRACTThis report examines the consequences of differential item functioning (DIF) using simulated data. Its impact on total score, item response theory (IRT) ability estimate, and test reliability was evaluated in various testing scenarios created by manipulating the following four factors: test length, percentage of DIF items per form, sample sizes of different examinee groups, and types of responses. The results indicate that the greatest score difference was observed between the examinee groups on forms with DIF items, and the magnitude was less than 2 points on the 0‐60 total score scale and .15 on the IRT ability scale. The influence on reliability was rather limited.
TEST TAKERS' ATTITUDES ABOUT THE TOEFL IBT™
Authors: Stricker Lawrence J. , Attali Yigal 
Published: August 2014

ABSTRACTThe principal aims of this study, a conceptual replication of an earlier investigation of the TOEFL® computer‐based test, or TOEFL CBT, in Buenos Aires, Cairo, and Frankfurt, were to assess test takers' reported acceptance of the TOEFL Internet‐based test, or TOEFL iBT™, and its associations with possible determinants of this acceptance and with test performance; evaluate differences in the pattern of results for test takers from different countries; and compare the findings with those for the TOEFL CBT. A questionnaire concerning attitudes about the test and other relevant variables was administered by the Internet to TOEFL iBT examinees in 4 diverse countries with large testing volumes: China, Colombia, Egypt, and Germany. Overall attitudes about the TOEFL iBT were moderately positive in most countries, but neutral in Germany; attitudes about the Listening and Writing sections of the test were uniformly favorable in every country; but attitudes about the Speaking section were consistently less favorable in all countries and were unfavorable in Germany and Colombia. Attitudes about the test had similar patterns of relationships in the 4 countries: moderate correlations with attitudes about admissions tests in general, slight and inconsistent correlations with TOEFL performance and computer anxiety, and minimal correlations with other variables. All in all, these results were very similar to those in the earlier investigation, with 1 exception: the previous study found uniformly moderate positive attitudes about the TOEFL CBT in the 3 countries surveyed.
DEVELOPMENT OF A STUDENT HEALTH ASSESSMENT SYSTEM: HEALTH KNOWLEDGE, ATTITUDES, AND BEHAVIORS IN MIDDLE‐SCHOOL STUDENTS
Authors: MacCann Carolyn , Roberts Richard D. 
Published: August 2014

ABSTRACTNewly developed assessments of nutrition and exercise knowledge, attitudes, and behavior were administered to 383 eighth‐graders. Evidence for the validity of assessment scores was evaluated with five findings. First, parent‐ and self‐reported behaviors were similar and congruent for healthy eating and exercising but not for sedentary behaviors or unhealthy eating. Second, the theory of planned behavior structural model was confirmed for exercise and nutrition attitudes: Attitudes predicted exercising (R2=.34), healthy eating (R2=.21), sedentary behaviors (R2=.11), and unhealthy eating (R2=.09). Third, gender differences replicated previous research: Girls have better diets and nutrition attitudes whereas boys exercise more frequently. The assessments distinguished normal‐weight from obese individuals, and showed incremental validity in predicting school grades. The paper concludes with a discussion of the assessments' potential for informing interventions.
CHAINED VERSUS POST‐STRATIFICATION EQUATING IN A LINEAR CONTEXT: AN EVALUATION USING EMPIRICAL DATA
Authors: Puhan Gautam 
Published: August 2014

ABSTRACTThis study used real data to construct testing conditions for comparing results of chained linear, Tucker, and Levine‐observed score equatings. The comparisons were made under conditions where the new‐ and old‐form samples were similar in ability and when they differed in ability. The length of the anchor test was also varied to enable examination of its effect on the three different equating methods. Two tests were used in the study, and the three equating methods were compared to a criterion equating to obtain estimates of random equating error, bias, and root mean squared error (RMSE). Results showed that for most of the conditions studied, chained linear score equating produced fairly good equating results in terms of low bias and RMSE. In some conditions, Levine‐observed score equating also produced low bias and RMSE. Although the Tucker method always produced the lowest random equating error, it produced a larger bias and RMSE than either of the other equating methods. Based on these results, it is recommended that either chained linear or Levine score equating be used when new‐ and old‐form samples differ in ability and/or when the anchor‐to‐total correlation is not very high.
STUDIES OF A LATENT CLASS SIGNAL DETECTION MODEL FOR CONSTRUCTED RESPONSE SCORING II: INCOMPLETE AND HIERARCHICAL DESIGNS
Authors: DeCarlo Lawrence T. 
Published: August 2014

ABSTRACTA basic consideration in large‐scale assessments that use constructed response (CR) items, such as essays, is how to allocate the essays to the raters that score them. Designs that are used in practice are incomplete, in that each essay is scored by only a subset of the raters, and also unbalanced, in that the number of essays scored by each rater differs across the raters. In addition, all of the possible rater pairs may not be used. The present study examines the effects of these factors on parameter recovery and classification accuracy using simulations of a latent class model based on signal detection theory (SDT). Many tests also include more than one CR item, which introduces a nested or hierarchical structure into the design, in that raters are nested within essays (i.e., there are multiple raters per essay) and essays are nested within examinees (i.e., each examinee provides two or more essays). A hierarchical rater model (HRM) has previously been developed to recognize the nested structure. A version of the HRM that incorporates a latent class signal detection model in the first level, referred to as the HRM‐SDT model, is presented. Parameter recovery in the HRM‐SDT model is examined in simulations. The model is applied to data from several ETS tests.
SINGLE‐ VERSUS DOUBLE‐SCORING OF TREND RESPONSES IN TREND SCORE EQUATING WITH CONSTRUCTED‐RESPONSE TESTS
Authors: Tan Xuan , Ricker Kathryn L. , Puhan Gautam 
Published: August 2014

ABSTRACTThis study examines the differences in equating outcomes between two trend score equating designs resulting from two different scoring strategies for trend scoring when operational constructed‐response (CR) items are double‐scored—the single group (SG) design, where each trend CR item is double‐scored, and the nonequivalent groups with anchor test (NEAT) design, where each trend CR item is single‐scored during trend score equating—for varying sample sizes (n =150, 200, 250, 300, 400). Overall results suggest larger equating errors with smaller sample sizes, though errors were small regardless of sample size. The NEAT design performed about as well as the SG design with respect to conditional and summative standard errors of equating, though it did tend to produce larger bias and root mean‐squared differences (RMSDs). When accounting for the total number of trend scores required to do analyses, the NEAT design performed as well or better than the SG design (e.g., when the NEAT n =150 and the SG n = 300). This result might be partially attributable to a larger operational sample size (n = 792) and a good correlation between anchor and total score for the trend sample (r = 0.73). These results suggest that under these testing conditions, the NEAT design performed about as well as the SG design, but further research is required to assess the generalizability of the results.
A SUMMARY OF MODELS AND STANDARDS‐BASED APPLICATIONS FOR GRADE‐TO‐GRADE GROWTH ON STATEWIDE ASSESSMENTS AND IMPLICATIONS FOR STUDENTS WITH DISABILITIES
Authors: Buzick Heather M. , Laitusis Cara Cahalan 
Published: August 2014

ABSTRACTRecently growth‐based approaches to accountability have received considerable attention because they have the potential to reward schools and teachers for improving student performance over time by measuring the progress of students at all levels of the performance spectrum (including those who have not yet reached proficiency on state accountability assessments). While the use of growth in accountability holds promise for students with disabilities, measuring changes over time in their academic performance is complex. This paper summarizes models and approaches that use individual student test scores from multiple years for 3 different purposes: determination of adequate yearly progress under the federal accountability system, research on individual growth trajectories, and evaluation of the contribution of teachers and schools to student learning. Practical challenges in measuring and modeling growth for students with disabilities are then discussed. Finally, 3 areas in need of research on the measurement of growth from large‐scale annual accountability assessments are identified and described: testing accommodations, test difficulty, and understanding the longitudinal characteristics of the population of students with disabilities.
A COMPARISON OF METHODS FOR ESTIMATING CONDITIONAL ITEM SCORE DIFFERENCES IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ASSESSMENTS
Authors: Moses Tim , Miao Jing , Dorans Neil 
Published: August 2014

ABSTRACTThis study compared the accuracies of four differential item functioning (DIF) estimation methods, where each method makes use of only one of the following: raw data, logistic regression, loglinear models, or kernel smoothing. The major focus was on the estimation strategies' potential for estimating score‐level, conditional DIF. A secondary focus was on assessing the accuracy of strategies' overall DIF effect sizes and statistical significance tests. A real data simulation was used to evaluate the estimation strategies with 6 items representing DIF and No DIF situations, and with 4 sample size combinations for the reference and focal group data. Results showed that the logistic regression estimation strategy was the most highly recommended strategy in terms of the bias and variability of its estimates and the power of its statistical significance test. The loglinear models strategy had flexibility advantages, but these advantages only offset the greater variability of its estimates and its reduced statistical power when sample sizes were large. The kernel smoothing estimation strategy was the least accurate of the considered strategies due to estimation problems when the reference and focal groups differed in overall ability.
LINKING MIXED‐FORMAT TESTS USING MULTIPLE‐CHOICE ANCHORS
Authors: Walker Michael E. , Kim Sooyeon 
Published: August 2014

ABSTRACTThis study examined the use of an all multiple‐choice (MC) anchor for linking mixed format tests containing both MC and constructed‐response (CR) items, in a nonequivalent groups design. An MC‐only anchor could effectively link two such test forms if either (a) the MC and CR portions of the test measured the same construct, so that the MC anchor adequately represented the entire test, or (b) the relationship between the MC portion and the total test remained constant across the new and reference linking groups. The study also evaluated whether linking mixed‐format tests through MC‐only anchors would be more effective than a two‐stage strategy in which MC portions were equated through MC anchors and then composite scores were scaled to the MC scores. Anchor linking and two‐stage linking yielded identical (or nearly so) results for both linear and nonlinear chained linking methods. With post‐stratification linking methods the two‐stage strategy resulted in smaller bias. The paper discusses some advantages of both approaches.
UNFAIR TREATMENT VS. CONFIRMATION BIAS? COMMENTS ON SANTELICES AND WILSON
Authors: Dorans Neil J. 
Published: August 2014

ABSTRACTSantelices and Wilson (2010) claimed to have addressed technical criticisms of Freedle (2003) presented in Dorans (2004a) and elsewhere. Santelices and Wilson's abstract claimed that their study confirmed that SAT® verbal items do function differently for African American and White subgroups. In this commentary, I demonstrate that the Santelices and Wilson article provided no evidence to confirm differential item functioning (DIF) and failed to address my technical criticisms of Freedle. Instead, Santelices and Wilson contained several misrepresentations, including substituting “considered serious” for “more unusual” to describe an effect size and claiming to have studied four editions of the SAT when only two were actually studied. Central to its thesis was a hypothesis about DIF/difficulty correlations that was misattributed to Dorans. Santelices and Wilson based their argument for DIF on correlations between highly correlated variations on an index of DIF with item difficulty. While failing to demonstrate either unfairness in the test items or unfairness in the treatment of Freedle, Santelices and Wilson did contain evidence of confirmation bias.
TEACHER LEADERSHIP: AN ASSESSMENT FRAMEWORK FOR AN EMERGING AREA OF PROFESSIONAL PRACTICE
Authors: Jackson Teresa , Burrus Jeremy , Bassett Katherine , Roberts Richard D. 
Published: August 2014

ABSTRACTThis paper examines various definitions and frameworks that have been used to justify the emergence of a new category for the teacher professional: teacher leader. The emergence of this new professional category may lead to greater retention levels, and improved knowledge management and transfer within the teaching profession. Various key dimensions of this profession are examined, allowing us to highlight some key personal skills that would appear requisite for the teacher leader. An agenda for developing and validating assessments of teacher leadership is then proposed. It is argued that these assessments have the potential of legitimizing research within this field, as well as providing the opportunity to better understand what it takes to become a successful professional in this new domain of teaching practice.
EQUATING OF SUBSCORES AND WEIGHTED AVERAGES UNDER THE NEAT DESIGN
Authors: Sinharay Sandip , Haberman Shelby 
Published: August 2014

ABSTRACTRecently, the literature has seen increasing interest in subscores for their potential diagnostic values; for example, one study suggested the report of weighted averages of a subscore and the total score, whereas others showed, for various operational and simulated data sets, that weighted averages, as compared to subscores, lead to more accurate diagnostic information. To report weighted averages, the averages should be comparable across different test forms; that is, the averages should be equated. This report discusses how to equate weighted averages. Results from operational and simulated data sets demonstrate the small error found when equating weighted averages.
STATISTICAL PROCEDURES TO EVALUATE QUALITY OF SCALE ANCHORING
Authors: Haberman Shelby J. , Sinharay Sandip , Lee Yi‐Hsuan 
Published: August 2014

ABSTRACTProviding information to test takers and test score users about the abilities of test takers at different score levels has been a persistent problem in educational and psychological measurement (Carroll, 1993). Scale anchoring (Beaton & Allen, 1992), a technique that describes what students at different points on a score scale know and can do, is a tool to provide such information. Scale anchoring for a test involves substantial amount of work, both by the statistical analysts and test developers involved with the test. In addition, scale anchoring involves considerable use of subjective judgment, so its conclusions may be questionable. This paper describes statistical procedures that can be used to determine if scale anchoring is likely to be successful for a test. If these procedures indicate that scale anchoring is unlikely to be successful, then there is little reason to perform a detailed scale anchoring study. The procedures are applied to several data sets from a teacher licensing test.
ASSESSING THE FALSIFIABILITY OF EXTREME LINKING
Authors: Middleton Kyndra , Dorans Neil J. 
Published: August 2014

ABSTRACTExtreme linkings are performed in settings in which neither equivalent groups nor anchor material is available to link scores on two assessments. Examples of extreme linkages include links between scores on tests administered in different languages or between scores on tests administered across disability groups. The strength of interpretation attached to a linkage depends on the proper design and execution of a sound data collection plan. The current paper uses a real data set to illustrate how to indirectly assess the quality of linking the scores on two assessments that contain neither equivalent groups nor common anchor material.
EVALUATING EMPIRICAL RELATIONSHIPS AMONG PREDICTION, MEASUREMENT, AND SCALING INVARIANCE
Authors: Moses Tim 
Published: August 2014

ABSTRACTThe purpose of this study was to consider the relationships of prediction, measurement, and scaling invariance when these invariances were simultaneously evaluated in psychometric test data. An approach was developed to evaluate prediction, measurement, and scaling invariance based on linear and nonlinear prediction, measurement, and scaling functions. The approach was used to evaluate the relationships among 12 pairs of tests in 6 datasets for gender invariance. The prediction, measurement, and scaling invariance results were found to be similar for most of the test relationships evaluated, in that all 3 invariances were more likely to be established for highly correlated tests than for less highly correlated tests. The invariance results appeared to be well summarized by intercept differences in the linear prediction, measurement, and scaling functions. The implications of the results were discussed with respect to the relationships among prediction, measurement, and scaling invariances described in prior theoretical and empirical research. Suggestions for extending theoretical and empirical invariance research were provided.
AUTOMATED SCORING WITHIN A DEVELOPMENTAL, COGNITIVE MODEL OF WRITING PROFICIENCY
Authors: Deane Paul , Quinlan Thomas , Kostin Irene 
Published: August 2014

ABSTRACTETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.
MEASUREMENT OF NEW ATTRIBUTES FOR CHILE'S ADMISSIONS SYSTEM TO HIGHER EDUCATION
Authors: Santelices María Verónica , Ugarte Juan José , Flotts Paulina , Radovic Darinka , Kyllonen Patrick 
Published: August 2014

ABSTRACTThis paper presents the development and initial validation of new measures of critical thinking and noncognitive attributes that were designed to supplement existing standardized tests used in the admissions system for higher education in Chile. The importance of various facets of this process, including the establishment of technical rigor and political support of instrument development, piloting, and implementation is also emphasized because the results from this pilot study could affect the admissions process for some of the most prestigious and competitive universities in the country. The new instruments were pilot tested on a group of high school seniors (n = 1,568) and first‐year college students (n = 1,443) during 2008. The sample included students from 20 secondary schools and 4 universities. Initial results were promising and showed that the measures were effectively assessing attributes not currently considered in the Chilean higher education admissions process. In addition, performance on the new instruments was found to be less correlated with students' sociodemographic characteristics than were measures currently used for undergraduate admissions.
USE OF E‐RATER® IN SCORING OF THE TOEFL IBT® WRITING TEST
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTAlternative approaches are discussed for use of e‐rater® to score the TOEFL iBT® Writing test. These approaches involve alternate criteria. In the 1st approach, the predicted variable is the expected rater score of the examinee's 2 essays. In the 2nd approach, the predicted variable is the expected rater score of 2 essay responses by the examinee on a parallel form. This 2nd approach is related to prediction of the expected rater score of 2 essay responses on an actual form taken later by an examinee. The relationship of e‐rater scores to scores on other sections of TOEFL® is also considered. These alternative approaches suggest somewhat different procedures for scoring.