bluuebunny/crossref_metadata_2025_split huggingface dataset:
A BAYESIAN HIERARCHICAL MODEL FOR LARGE‐SCALE EDUCATIONAL SURVEYS: AN APPLICATION TO THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Authors: Johnson Matthew S. , Jenkins Frank 
Published: August 2014

ABSTRACTLarge‐scale educational assessments such as the National Assessment of Educational Progress (NAEP) sample examinees to whom an exam will be administered. In most situations the sampling design is not a simple random sample and must be accounted for in the estimating model. After reviewing the current operational estimation procedure for NAEP, this paper describes a Bayesian hierarchical model for the analysis of complex large‐scale assessments. The model clusters students within schools and schools within primary sampling units. The paper discusses an estimation procedure that utilizes a Markov chain Monte Carlo algorithm to approximate the posterior distribution of the model parameters. Results from two Bayesian models, one treating item parameters as known and one treating them as unknown, are compared to results from the current operational method on a simulated data set and on a subset of data from the 1998 NAEP reading assessment. The point estimates from the Bayesian model and the operational method are quite similar in most cases, but there does seem to be systematic differences in measures of uncertainty (e.g., standard errors, confidence intervals).
UNDERSTANDING WHAT SAT REASONING TEST™ SCORES ADD TO HIGH SCHOOL GRADES: A STRAIGHTFORWARD APPROACH
Authors: Bridgeman Brent , Pollack Judy , Burton Nancy 
Published: August 2014

ABSTRACTAlthough a number of large‐scale studies have demonstrated an increase in predictive validity when SAT® scores are added to high school grades, this increment is often described in terms of a seemingly very small increase in explained variance. This may have led to the erroneous belief among test critics and others that students with low SAT scores would perform, on average, almost as well in college as students with high SAT scores. Using a sample of 41 colleges, the current study shows substantial differences in the percent of students who succeed (defined by a 2.5 or 3.5 college grade point average [CGPA] at the end of one year or four years in college) by SAT score level, even when intensity of the high school curriculum and high school grades are taken into account.
PERFORMANCE OF MEN AND WOMEN ON MULTIPLE‐CHOICE AND CONSTRUCTED‐RESPONSE TESTS FOR BEGINNING TEACHERS
Authors: Livingston Samuel A. , Rupp Stacie L. 
Published: August 2014

ABSTRACTSome previous research results imply that women tend to perform better, relative to men, on constructed‐response (CR) tests than on multiple‐choice (MC) tests in the same subjects. An analysis of data from several tests used in the licensing of beginning teachers supported this hypothesis, to varying degrees, in most of the tests investigated. The hypothesis was strongly supported in Praxis™ Principles of Learning and Teaching tests for secondary school teachers and in subject‐knowledge tests for social studies teachers, science teachers, and middle school mathematics teachers. Evidence for the hypothesis was weak in subject‐knowledge tests for middle school English teachers and for secondary school mathematics teachers. Subject‐knowledge tests for secondary school English teachers did not show the hypothesized relationship. The analysis was based on plots showing the cumulative percentages of men and women attaining each possible score on the MC and CR tests.
CONDITIONAL COVARIANCE THEORY AND DETECT FOR POLYTOMOUS ITEMS
Authors: Zhang Jinming 
Published: August 2014

ABSTRACTThis paper extends the theory of conditional covariances to polytomous items. It has been mathematically proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, is positive if the two items are dimensionally homogeneous and negative otherwise. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as NAEP data. As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or by random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade. The results also indicate that the degree of multidimensionality of the NAEP reading data is weak.
THE CORRELATION BETWEEN THE SCORES OF A TEST AND AN ANCHOR TEST
Authors: Sinharay Sandip , Holland Paul 
Published: August 2014

ABSTRACTIt is a widely held belief that an anchor test used in equating should be a miniature version (orminitest) of the tests to be equated; that is, the anchor test should be proportionally representative of the two tests in content and statistical characteristics. This paper examines the scientific foundation of this belief, especially regarding statistical characteristics. In particular, we study, through several simulation studies and a real data example, the correlation of the raw score of the total test with several anchor tests, including those having statistical characteristics that differ from those of the total test. Anchor tests with spread of item difficulties less than that of a total test are shown to consistently have higher correlations than a minitest. Because of the general view that higher correlations of the scores of an anchor test and the total test lead to better equating, our results suggest that requiring an anchor test to mimic the statistical characteristics of the total test may be too restrictive and may not be optimal.
MODEL ANALYSIS AND MODEL CREATION: CAPTURING THE TASK‐MODEL STRUCTURE OF QUANTITATIVE ITEM DOMAINS
Authors: Deane Paul , Graf Edith Aurora , Higgins Derrick , Futagi Yoko , Lawless René 
Published: August 2014

ABSTRACTThis study focuses on the relationship between item modeling and evidence‐centered design (ECD); it considers how an appropriately generalized item modeling software tool can support systematic identification and exploitation of task‐model variables, and then examines the feasibility of this goal, using linear‐equation items as a test case. The first half of the study examines task‐model structures for linear equations and their relevance to item difficulty within ECD. The second half of the study presents prototype software, a Model Creator system for pure math items, designed to partially automate the creation of variant item models reflecting different combinations of task‐model variables. The prototype is applied to linear equations but is designed to generalize over a range of pure mathematical content types.
TENSIONS, TRENDS, TOOLS, AND TECHNOLOGIES: TIME FOR AN EDUCATIONAL SEA CHANGE
Authors: Shute Valerie J. 
Published: August 2014

ABSTRACTThis report outlines three educational approaches: (a) traditional, the currently dominant approach, a largely lecture‐oriented, authoritarian style that makes heavy use of assessments of learning, which are useful for accountability purposes but only marginally useful for guiding day‐to‐day instruction; (b) progressive, a highly student‐centered approach that relies on assessments for learning, which can be very useful in guiding day‐to‐day instruction; and (c) unified, a new, integrated approach that uses the best of both kinds of assessments—for and of learning—and which leverages computer technology, educational measurement, and cognitive science to address factors that undermined earlier attempts to implement the progressive approach. This report examines some of the research, trends, and factors that should be considered, understood, and, in some cases, leveraged, in order to move toward the unified approach. Further, this report presents examples of how ETS projects are moving toward the new approach to harness assessment in the service of learning.
THE IMPACT OF CHANGES IN THE TOEFL EXAMINATION ON TEACHING AND LEARNING IN CENTRAL AND EASTERN EUROPE: PHASE 1, THE BASELINE STUDY
Authors: Wall Dianne , Horák Tania 
Published: August 2014

ABSTRACTThe purpose of this report is to present the findings of the first phase in a longitudinal study of the impact of changes in the TOEFL® test on teaching and learning in test preparation classrooms. Observations were carried out and interviews conducted with teachers, students, and directors of studies at 10 institutions in six countries in Central and Eastern Europe. The findings constitute a baseline against which finding from future phases can be compared. The report includes an analysis of the types of impact originally envisaged by advisers to the new TOEFL and a discussion of factors in the educational context that may influence the types of impact that may appear once the new test is operational.
ADAPTIVE NUMERICAL INTEGRATION FOR ITEM RESPONSE THEORY
Authors: Antal Tamás , Oranje Andreas 
Published: August 2014

ABSTRACTWell‐known numerical integration methods are applied to item response theory (IRT) with special emphasis on the estimation of the latent regression model of NAEP. An argument is made that the Gauss‐Hermite rule enhanced with Cholesky decomposition and normal approximation of the response likelihood is a fast, precise, and reliable alternative for the numerical integration in NAEP and in IRT in general.
THE IMPACT OF SHORT‐TERM USE OF CRITERIONSM ON WRITING SKILLS IN NINTH GRADE
Authors: Rock JoAnn Leah 
Published: August 2014

ABSTRACTIn this study, a hierarchical linear model (HLM) was used within a cluster random assignment design to test the impact of Criterionsm on students' writing skill when Criterion was used as a supplementary instructional tool in ninth grade English classrooms over a 4 week period. The study was conducted in three states: Indiana, Pennsylvania, and New Jersey. The sample consisted of 1,464 ninth graders aggregated across 10 schools, 34 teachers, and 61 classes. Findings indicate that over a short time period, Criterion use improved the mechanical aspects of student essays as measured by the analytic scores given to their essays. Students in ninth grade English classes who used Criterion several times a week over a 1 month period received higher analytic scores on their essays written at the end of the study period than those who were in classes that did not. The effect was small, but statistically significant at the .05 level using a one‐tailed test. These findings suggest a role for automated feedback systems in supporting writing instruction.
CONSTRUCT VALIDITY OF E‐RATER® IN SCORING TOEFL® ESSAYS
Authors: Attali Yigal 
Published: August 2014

ABSTRACTThis study examined the construct validity of the e‐rater® automated essay scoring engine as an alternative to human scoring in the context of TOEFL® essay writing. Analyses were based on a sample of students who repeated the TOEFL within a short time period. Two e‐rater scores were investigated in this study, the first based on optimally predicting the human essay score and the second based on equal weights for the different features of e‐rater.Within a multitrait‐multimethod approach, the correlations and reliabilities of human and e‐rater scores were analyzed together with TOEFL subscores (structured writing, reading, and listening) and with essay length. Possible biases between human and e‐rater scores were examined with respect to differences in performance across countries of origin and differences in difficulty across prompts. Finally, a factor analysis was conducted on the e‐rater features to investigate the interpretability of their internal structure and determine which of the two e‐rater scores reflects this structure more closely.Results showed that the e‐rater score based on optimally predicting the human score measures essentially the same construct as human‐based essay scores with significantly higher reliability and consequently higher correlations with related language scores. The equal‐weights e‐rater score showed the same high reliability but significantly lower correlation with essay length. It is also aligned with the 3‐factor hierarchical (word use, grammar, and discourse) structure that was discovered in the factor analysis. Both e‐rater scores also successfully replicate human score differences between countries and prompts.
DISCLOSURE RISK IN EDUCATIONAL SURVEYS: AN APPLICATION TO THE NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Authors: Oranje Andreas , Freund David , Lin Mei‐jang , Tang Yuxin 
Published: August 2014

ABSTRACTIn this paper, a data perturbation method for minimizing the possibility of disclosure of participants' identities on a survey is described in the context of the National Assessment of Educational Progress (NAEP). The method distinguishes itself from most approaches because of the presence of cognitive tasks. Hence, a data edit should have minimal impact on both relations among demographic variables and relations between demographic and proficiency variables. Furthermore, since only a few students are at risk to be disclosed in a typical sampling setting common to educational surveys, the proposed data perturbation is governed by a nonuniform probabilistic process. The method is applied to data from NAEP and impact is computed using proficiency averages, demographic proportions, statistical inference results, and loglinear models. Results show that the proposed perturbation method has very little impact on NAEP results, even at relatively large editing rates. Some data coarsening results are reported as well. While the univariate results are relatively unaffected from the coarsening, loglinear models from higher order contingency tables are affected. It is recommended to restrict disclosure limitation techniques to perturbation methods in the case of NAEP.
SUPPORTING EFFICIENT, EVIDENCE‐CENTERED ITEM DEVELOPMENT FOR THE GRE VERBAL MEASURE
Authors: Sheehan Kathleen M. , Kostin Irene , Futagi Yoko 
Published: August 2014

ABSTRACTThis paper explores alternative approaches for facilitating efficient, evidence‐centered item development for a new type of verbal reasoning item developed for use on the GRE® General Test. Results obtained in two separate studies are reported. The first study documented the development and validation of a fully automated approach for locating the types of stimulus materials needed to support more efficient generation of the studied item type. The validation results confirmed that the proposed filtering technique can help item writers increase the percentage of acceptable stimulus paragraphs located per unit time interval from the current level of about 10% to nearly 30%. The second study documented the development and validation of a set of task models designed to help item writers generate new items that are optimally constructed to provide high‐quality evidence about targeted skills. The proposed models were validated by considering the percentage of difficulty variance accounted for by the specified item classifications. That amount ranged from slightly more than 30% for items designed to test vocabulary skills to slightly more than 40% for items designed to test additional verbal reasoning skills, such as generating near and far inferences and understanding complex oppositional reasoning.
THE IMPACT OF THE NATIONAL BOARD FOR PROFESSIONAL TEACHING STANDARDS: A REVIEW OF THE RESEARCH
Authors: Gitomer ew H. 
Published: August 2014

ABSTRACTInterest in the impact of the teacher advanced certification system instituted by the National Board for Professional Teaching Standards (NBPTS) has resulted in a quickly expanding body of literature. This review examines five critical issues covered in the literature: the extent to which National Board certified teachers (NBCTs) engage in stronger instructional practice, are associated with stronger learning gains in their classrooms and schools, benefit from participating in the assessment process, influence the teaching quality of their peers, and are present in schools and classrooms with different characteristics. While most comparisons provide evidence that support the incremental effectiveness of NBCTs, the effects are generally modest. However, all findings must be interpreted with caution, given the limitations of research designs, methods, and tools used to explore teacher effects.
THE CORRELATION BETWEEN ITEM PARAMETERS AND ITEM FIT STATISTICS
Authors: Sinharay Sandip , Lu Ying 
Published: August 2014

ABSTRACTDodeen (2004) studied the correlation between the item parameters of the three‐parameter logistic model and two item fit statistics, and found some linear relationships (e.g., a positive correlation between item discrimination parameters and item fit statistics) that have the potential for influencing the work of practitioners who employ item response theory. This paper examines the same type of linear relationships as studied in Dodeen. However, this paper adds to the literature by employing item fit statistics not considered in Dodeen, which have been recently suggested and whose Type I error rates have been demonstrated to be generally close to the nominal level. Detailed simulations show that if one uses certain of the recently suggested item fit statistics, there is no need to worry about any linear relationships between the item parameters and item fit statistics.
INVESTIGATING THE EFFECTIVENESS OF A SYNTHETIC LINKING FUNCTION ON SMALL SAMPLE EQUATING
Authors: Kim Sooyeon , von Davier Alina A. , Haberman Shelby 
Published: August 2014

ABSTRACTThe synthetic function, which is a weighted average of the identity (the trivial linking function for forms that are known to be completely parallel) and a traditional equating method, has been proposed as an alternative for performing linking with very small samples (Kim, von Davier, & Haberman, 2006). The purpose of the present study was to investigate the benefits of the synthetic function using various real data sets gathered from different administrations of tests from a licensure testing program. We investigated the chained linear, Tucker, Levine, and mean equating methods, along with the identity and the synthetic functions with small samples (N = 19 to 70). Neither the identity nor the synthetic functions worked as well as did other linear equating methods, because test forms differed markedly in difficulty. The synthetic function cannot be used as a solution or methodological fix to a problem that is caused by poor data collection design.
THE IMPACT OF ANCHOR TEST LENGTH ON EQUATING RESULTS IN A NONEQUIVALENT GROUPS DESIGN
Authors: Ricker Kathryn L. , von Davier Alina A. 
Published: August 2014

ABSTRACTThis study explored the effects of external anchor test length on final equating results of several equating methods, including equipercentile (frequency estimation), chained equipercentile, kernel equating (KE) poststratification PSE with optimal bandwidths, and KE PSE linear (large bandwidths) when using the nonequivalent groups anchor test (NEAT) design. This study used pseudotests constructed of item responses from a real operational test. The equating methods were evaluated using an equating criterion. Conditional differences between the criterion scores and equated scores, and root mean square error of the differences (RMSE) were used as measures to compare the methods to the criterion equating, which in this study is an equivalent groups (EG) equipercentile equating function. The results indicate that bias tended to increase in the conversions as the anchor test length decreased, but the KE PSE with optimal bandwidths and equipercentile (frequency estimation) methods were less sensitive to this change than the other methods. The KE PSE linear method with large bandwidths performed poorly compared to the criterion across all anchor test lengths.
AN INITIAL FIELD TRIAL OF AN INSTRUMENT FOR MEASURING LEARNING STRATEGIES OF MIDDLE SCHOOL STUDENTS
Authors: Liu Ou Lydia , Jackson Teresa , Ling Guangming 
Published: August 2014

ABSTRACTLearning strategies have been increasingly recognized as a useful tool to promote effective learning. In response to the lack of available learning strategies measures for middle school students, this study designed an instrument for these students, assessing behavioral, cognitive, and metacognitive strategies. This instrument, the Middle School Learning Strategies (MSLS) scale, is examined in terms of factorial structure, reliability, and correlates. Three factors emerge from the analyses: effective strategies, help seeking, and bad habits. The subscales displayed a reasonable reliability, ranging from .70 to .87. Student grades in language arts, social studies, math, and science were collected as criterion variables. As expected, grades in these four subjects correlated positively with both effective strategies and help seeking, yet negatively with bad habits. As a pilot measure, this instrument has demonstrated promising features as a useful tool for students to evaluate and enhance their learning strategies.
CONTINUOUS EXPONENTIAL FAMILIES: AN EQUATING TOOL
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTContinuous exponential families may be employed to find continuous distributions with the same initial moments as the discrete distributions encountered in typical applications of classical equating. These continuous distributions provide distribution functions and quantile functions that may be employed in equating. To illustrate, an application is considered for a randomly equivalent groups design.
APPROACHES TO THE DESIGN OF DIAGNOSTIC ITEM MODELS
Authors: Graf Edith Aurora 
Published: August 2014

ABSTRACTQuantitative item models are item structures that may be expressed in terms of mathematical variables and constraints. An item model may be developed as a computer program from which large numbers of items are automatically generated. Item models can be used to produce large numbers of items for use in traditional, large‐scale assessments. But they have potential for use in other areas as well, including diagnostic assessment. In this report, I first review research on diagnostic assessment and then discuss how approaches to diagnostic assessment can inform the design of diagnostic item models.
IMPOSSIBLE SCORES RESULTING IN ZERO FREQUENCIES IN THE ANCHOR TEST: IMPACT ON SMOOTHING AND EQUATING
Authors: Puhan Gautam , von Davier Alina , Gupta Shaloo 
Published: August 2014

ABSTRACTEquating under the external anchor design is frequently conducted using scaled scores on the anchor test. However, scaled scores often lead to the unique problem of creating zero frequencies in the score distribution because there may not always be a one‐to‐one correspondence between raw and scaled scores. For example, raw scores of 17 and 18 may correspond to scaled scores of 150 and 153, thereby creating zero frequencies for scaled scores of 151 and 152. These gaps in the frequency distribution may adversely impact smoothing and equating. This study examines the effect of these zero frequencies on log‐linear smoothing (Holland & Thayer, 1987) of score distributions and final equating results. Results suggest that although smoothing is significantly affected by the presence of these zero frequencies, as indicated by the likelihood‐ratio chi‐square, Akaike information criterion (Akaike, 1977), and Freeman‐Tukey deviates, the impact on the actual equating results is minimal.