bluuebunny/crossref_metadata_2025_split huggingface dataset:
LOGLINEAR SMOOTHING: AN ALTERNATIVE NUMERICAL APPROACH USING SAS
Authors: Moses Tim , von Davier Alina A. , Casabianca Jodi 
Published: August 2014

ABSTRACTThe purpose of this report is to demonstrate loglinear smoothing using SAS PROC GENMOD. The results from four published examples, which include the smoothing of a) univariate distributions, b) bivariate distributions, c) distributions with teeth, and d) bivariate distributions with structural zeros, are reproduced to show the flexibility of the SAS procedure. Comparisons of graphical displays and likelihood ratio statistics show that the SAS procedure yields results identical to the published results. SAS PROC GENMOD provides an alternative approach to smoothing that is readily available for researchers and graduate students in educational measurement, as well as researchers from other scientific fields.
TOLERABLE VARIATION IN ITEM PARAMETER ESTIMATES FOR LINEAR AND ADAPTIVE COMPUTER‐BASED TESTING
Authors: Rizavi Saba , Way Walter D. , Davey Tim , Herbert Erin 
Published: August 2014

ABSTRACTItem parameter estimates vary for a variety of reasons, including estimation error, characteristics of the examinee samples, and context effects (e.g., item location effects, section location effects, etc.). Although we expect variation based on theory, there is reason to believe that observed variation in item parameter estimates exceeds what theory would predict. This study examined both items that were administered linearly in a fixed order each time that they were used and items that had appeared in different adaptive testing item pools. The study looked at both the magnitude of variation in the item parameter estimates and the impact of this variation in the estimation of test‐taker scores. The results showed that the linearly administered items exhibited remarkably small variation in parameter estimates over repeated calibrations. Similar findings with adaptively administered items in another high stakes testing program were also found when initial adaptively based item parameter estimates were compared with estimates from repeated use. The results of this study also indicated that context effects played a more significant role in adaptive item parameters when the comparisons were made to the parameters that were initially obtained from linear paper‐and‐pencil testing.
AN INVESTIGATION OF THE IMPACT OF COMPOSITION MEDIUM ON THE QUALITY OF TOEFL WRITING SCORES
Authors: Wolfe Edward W. , Manalo Jonathan R. 
Published: August 2014

ABSTRACTThis study examined scores from 133,906 operationally scored Test of English as a Foreign Language™ (TOEFL®) essays to determine whether the choice of composition medium has any impact on score quality for subgroups of test‐takers. Results of analyses demonstrate that (a) scores assigned to word‐processed essays are slightly more reliable than scores assigned to handwritten essays and exhibit higher correlations with TOEFL multiple‐choice subscores; (b) female test‐takers, examinees whose native language is not based on a Roman/Cyrillic alphabet, and examinees with lower English proficiency are more likely to choose the handwriting medium; (c) the probability of choosing handwriting as the composition medium increases with age for Asian examinees, but decreases with age for most European examinees; and (d) examinees with lower TOEFL multiple‐choice scores tend to have higher handwritten than word‐processed essay scores, while examinees with higher TOEFL multiple‐choice scores tend to have similar scores in either medium.
A COMPARISON OF TWO PROCEDURES FOR CONSTRAINED ADAPTIVE TEST CONSTRUCTION
Authors: Robin Frédéric , van der Linden Wim J. , Eignor Daniel R. , Steffen Manfred , Stocking Martha L. 
Published: August 2014

ABSTRACTThe relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high‐stakes on‐demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.
A STUDY OF FATIGUE EFFECTS FROM THE NEW SAT®
Authors: Liu Jinghua , Allspach Jill R. , Feigenbaum Miriam , Oh Hyeon‐Joo , Burton Nancy 
Published: August 2014

ABSTRACTThis study evaluated whether the addition of a writing section to the SAT Reasoning Test™ (referred to as the SAT® in this study) would impact test‐taker performance because of fatigue caused by increased test length. The study also investigated test‐takers' subjective feelings of fatigue. Ninety‐seven test‐takers were randomly assigned to three groups: the first group took a current SAT with no essay; the second group took a pseudo new SAT composed of the current SAT plus an essay, with the essay appearing in the first section of the test; and the third group also took the pseudo new SAT with an essay, but with the essay in the last section. Test‐taker performance on the verbal and math sections and the essay was then evaluated and compared. The results indicated that while the extended testing time for the new SAT may cause test‐takers to feel fatigued, fatigue did not affect test‐taker performance.
DESIGNING ADAPTIVE, DIAGNOSTIC MATH ASSESSMENTS FOR INDIVIDUALS WITH AND WITHOUT VISUAL DISABILITIES
Authors: Shute Valerie J. , Graf Edith Aurora , Hansen Eric G. 
Published: August 2014

ABSTRACTThis report summarizes the design and development of an adaptive e‐learning prototype for middle school mathematics for use with both sighted and visually disabled students. Adaptation refers to the system's ability to adjust itself to suit particular characteristics of the learner. The main parts of the report describe the system's theoretical foundation, architecture, models, and adaptive algorithm. We also review approaches for making assessment systems accessible to students with visual disabilities. Finally, we conclude with a summary of upcoming studies in relation to important research questions concerning micro‐ and macroadaptation. Using a design approach like the one described in this report may set a new precedent for environments that adapt to support student learning based on larger sets of incoming abilities and disabilities than have been considered previously.
JOINT AND CONDITIONAL ESTIMATION FOR IMPLICIT MODELS FOR TESTS WITH POLYTOMOUS ITEM SCORES
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTMultinomial‐response models are available that correspond implicitly to tests in which a total score is computed as the sum of polytomous item scores. For these models, joint and conditional estimation may be considered in much the same way as for the Rasch model for right‐scored tests. As in the Rasch model, joint estimation is only attractive if both the number of items and the number of examinees are large, while conditional estimation can be employed for a large number of examinees whether or not the number of items is large. In neither case is computation difficult given currently available computers. Large‐sample results favor use of conditional estimation, although some use of joint estimation can be contemplated if the number of items is large.
A SAS MACRO FOR LOGLINEAR SMOOTHING: APPLICATIONS AND IMPLICATIONS
Authors: Moses Tim P. , von Davier Alina A. 
Published: August 2014

ABSTRACTThe two purposes of this paper are to provide a SAS IML macro that performs loglinear smoothing and to apply this macro to loglinear smoothing problems that have not been extensively discussed in the test‐equating literature. The SAS macro is demonstrated on univariate, bivariate, and trivariate smoothing problems. The univariate and bivariate examples reproduce published results (von Davier, Holland, & Thayer, 2004). The trivariate example extends the bivariate smoothing example to allow for comparisons of subgroups' univariate and bivariate distributions. The implications are that important questions about distribution differences and subpopulation invariance of equating functions can be considered through comparisons and evaluations of complex loglinear models that are easily fit with this SAS IML macro.
LIMITS ON LOG CROSS‐PRODUCT RATIOS FOR ITEM RESPONSE MODELS
Authors: Haberman Shelby J. , Holland Paul W. , Sinharay Sandip 
Published: August 2014

ABSTRACTBounds are established for log cross‐product ratios (log odds ratios) involving pairs of items for item response models. First, expressions for bounds on log cross‐product ratios are provided for unidimensional item response models in general. Then, explicit bounds are obtained for the Rasch model and the two‐parameter logistic (2PL) model. Results are also illustrated through an example from a study of model‐checking procedures. The bounds obtained can provide a basis for assessment of goodness of fit of these models.
AN ELEMENTARY TEST OF THE NORMAL 2PL MODEL AGAINST THE NORMAL 3PL ALTERNATIVE
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTA simple score test of the normal two‐parameter logistic (2PL) model is presented that examines the potential attraction of the normal three‐parameter logistic (3PL) model for use with a particular item. Application is made to data from a test from the Praxis™ series. Results from this example raise the question whether the normal 3PL model should be used routinely in preference to the normal 2PL model unless evidence exists that a substantial gain in description of data is achieved.
TESTING THE UNTESTABLE ASSUMPTIONS OF THE CHAIN AND POSTSTRATIFICATION EQUATING METHODS FOR THE NEAT DESIGN
Authors: Holland Paul W. , von Davier Alina A. , Sinharay Sandip , Han Ning 
Published: August 2014

ABSTRACTThis paper focuses on the Non‐Equivalent Groups with Anchor Test (NEAT) design for test equating and on two classes of observed—score equating (OSE) methods—chain equating (CE) and poststratification equating (PSE). These two classes of methods reflect two distinctly different ways of using the information provided by the anchor test for computing OSE functions. Each of the two classes includes linear and nonlinear equating methods. In practical situations, it is known that the PSE and CE methods tend to give different results when the two groups of examinees differ in ability. However, given that both methods are justified by making untestable assumptions, it is difficult to conclude which, if either, of the two equating approaches is more correct. This study compares predictions from both the PSE and the CE assumptions that can be tested in a comparable way with the data from a special study. Results indicate that both CE and PSE make very similar predictions but that those of CE are slightly more accurate than those of PSE.
TOWARD AN UNDERSTANDING OF THE ROLE OF SPEECH RECOGNITION IN NONNATIVE SPEECH ASSESSMENT
Authors: Zechner Klaus , Bejar Isaac I. , Hemat Ramin 
Published: August 2014

ABSTRACTThe increasing availability and performance of computer‐based testing has prompted more research on the automatic assessment of language and speaking proficiency. In this investigation, we evaluated the feasibility of using an off‐the‐shelf speech‐recognition system for scoring speaking prompts from the LanguEdge field test of 2002. We first established the level of agreement between two trained scorers. We then adapted a speech engine to the language backgrounds and proficiency ranges of the speakers and developed a classification and regression tree (CART) for each of five prompts based on features computed from the output of the speech recognizer. In a validation on held‐out data, we found that while our features are not sufficiently comprehensive to adequately score these prompts, collectively these features appear to capture reliably some aspects of speaking proficiency.
CONFIDENCE AND COGNITIVE TEST PERFORMANCE
Authors: Stankov Lazar , Lee Jihyun 
Published: August 2014

ABSTRACTThis paper examines the nature of confidence in relation to cognitive abilities, personality traits, and metacognition. Confidence was measured as it was expressed in answers to each test item during the administration of reading and listening sections of the TOEFL® iBT. The confidence scores were correlated with the accuracy scores from the TOEFL iBT, SAT®, high school grade point averages (HS‐GPA), and several measures of personality and metacognition. The results indicate that confidence is a separate psychological trait, somewhere between cognitive ability and personality traits. In addition, our findings suggest that confidence is related to, but separate from, metacognition. We also demonstrated gender and ethnic differences in confidence, with male and African American students showing higher overconfidence bias than females and White or Hispanic students, respectively. Finally, our data show small but statistically significant incremental validity of the confidence scores above and beyond the accuracy scores in predicting numeracy test scores, the total TOEFL iBT scores, and TOEFL iBT subscores on the writing and speaking sections. We found no incremental validity of the confidence scores in predicting the SAT and HS‐GPA. Theoretical and practical implications of these findings are discussed as well.
A NOTE ON GAIN SCORES AND THEIR INTERPRETATION IN DEVELOPMENTAL MODELS DESIGNED TO MEASURE CHANGE IN THE EARLY SCHOOL YEARS
Authors: Rock Donald A. 
Published: August 2014

ABSTRACTThis paper presents a strategy for measuring cognitive gains in reading during the early school years. It is argued that accurate estimates of gain scores and their appropriate interpretation requires the use of adaptive tests with multiple criterion referenced points that mark learning milestones. It is further argued that two different measures of gain are necessary because of the nonequivalence of scale score points. Empirical results are presented in support of the use of adaptive tests with multiple criteria referenced scale points for the measurement of gain.
ESTIMATION OF STANDARD ERROR OF REGRESSION EFFECTS IN LATENT REGRESSION MODELS USING BINDER'S LINEARIZATION
Authors: Li Deping , Oranje Andreas 
Published: August 2014

ABSTRACTTwo versions of a general method for approximating standard error of regression effect estimates within an IRT‐based latent regression model are compared. The general method is based on Binder's (1983) approach, accounting for complex samples and finite populations by Taylor series linearization. In contrast, the current National Assessment of Educational Progress procedure assumes a simple random sample for standard error of regression effects and applies a jackknife estimator to statistics of interest as a way to account for NAEP's complex sample. In this study, the versions of the general method are formally defined and the general method is extended to multiple dimensions. Furthermore, they are applied in an empirical study to the 2004 NAEP long‐term trend data comparing both large, nearly saturated, and small models. Subsequently, the results are compared to the operational‐based imputation method. Results show no impact on the imputation‐based results, limited impact on large models, and reasonable impact on small models. While it is not readily apparent to what this differential impact can be attributed, several explanations are discussed.
SMALL‐SAMPLE DIF ESTIMATION USING LOG‐LINEAR SMOOTHING: A SIBTEST APPLICATION
Authors: Puhan Gautam , Moses Tim P. , Yu Lei , Dorans Neil J. 
Published: August 2014

ABSTRACTThe purpose of the current study was to examine whether log‐linear smoothing of observed score distributions in small samples results in more accurate differential item functioning (DIF) estimates under the simultaneous item bias test (SIBTEST) framework. Data from a teacher certification test were analyzed using White candidates in the reference group and African American candidates in the focal group. Smoothed and raw DIF estimates from 100 replications under seven different sample‐size conditions were compared to a criterion to determine the effect of smoothing on small‐sample DIF estimation. Root‐mean‐squared deviation and bias were used to evaluate the accuracy of DIF detection in the smoothed versus raw data conditions. Results indicate that, for most studied items, smoothing the raw score distributions reduced variability and bias of the DIF estimates especially in the small‐sample‐size conditions. Implications of these results for actual testing programs and future directions for research are discussed.
FOCUS ON FORMATIVE FEEDBACK
Authors: Shute Valerie J. 
Published: August 2014

ABSTRACTThis paper reviews the corpus of research on feedback, with a particular focus on formative feedback—defined as information communicated to the learner that is intended to modify the learner's thinking or behavior for the purpose of improving learning. According to researchers in the area, formative feedback should be multidimensional, nonevaluative, supportive, timely, specific, credible, infrequent, and genuine (e.g., Brophy, 1981; Schwartz & White, 2000). Formative feedback is usually presented as information to a learner in response to some action on the learner's part. It comes in a variety of types (e.g., verification of response accuracy, explanation of the correct answer, hints, worked examples) and can be administered at various times during the learning process (e.g., immediately following an answer, after some period of time has elapsed). Finally, there are a number of variables that have been shown to interact with formative feedback's success at promoting learning (e.g., individual characteristics of the learner and aspects of the task). All of these issues will be discussed in this paper. This review concludes with a set of guidelines for generating formative feedback.
ON THE LATENT REGRESSION MODEL OF ITEM RESPONSE THEORY
Authors: Antal Tamás 
Published: August 2014

ABSTRACTFull account of the latent regression model for the National Assessment of Educational Progress is given. The treatment includes derivation of the EM algorithm, Newton‐Raphson method, and the asymptotic standard errors. The paper also features the use of the adaptive Gauss‐Hermite numerical integration method as a basic tool to evaluate expectations necessary to perform the parameter estimations.
SYMMETRIC NONLINEAR REGRESSION
Authors: Antal Tamás 
Published: August 2014

ABSTRACTAn estimation tool for symmetric univariate nonlinear regression is presented. The method is based on introducing a nontrivial set of affine coordinates for diffeomorphisms of the real line. The main ingredient making the computations possible is the Connes‐Moscovici Hopf algebra of these affine coordinates.
RELIABILITY AND THE NONEQUIVALENT GROUPS WITH ANCHOR TEST DESIGN
Authors: Moses Tim , Kim Sooyeon 
Published: August 2014

ABSTRACTThis study evaluated the impact of unequal reliability on test equating methods in the nonequivalent groups with anchor test (NEAT) design. Classical true score‐based models were compared in terms of their assumptions about how reliability impacts test scores. These models were related to treatment of population ability differences by different NEAT equating methods. A score model was then developed based on the most important features of the reviewed score models and used to study reliability in a simulation study across a total of 45 measurement conditions (= 5 test and anchor reliability combinations × 3 population ability difference conditions × 3 sample sizes). Ten equating methods were considered: chained linear, chained equipercentile with raw and smoothed frequencies, Tucker, frequency estimation equipercentile with raw and smoothed frequencies, Levine observed using Angoff‐estimated and the “correct” reliabilities based on the data generation model used in this study, and Levine true using Angoff‐estimated and correct reliabilities. The results were consistent with what is known about equating functions and their variability. Unequal and/or low reliability inflates equating function variability and alters equating functions when population abilities differ.
AN EXPLORATION OF KERNEL EQUATING USING SAT® DATA: EQUATING TO A SIMILAR POPULATION AND TO A DISTANT POPULATION
Authors: Liu Jinghua , Low Albert C. 
Published: August 2014

ABSTRACTThis study applied kernel equating (KE) in two scenarios: equating to a very similar population and equating to a very different population, referred to as a distant population, using SAT® data. The KE results were compared to the results obtained from analogous classical equating methods in both scenarios. The results indicate that KE results are comparable to the results of other methods. Further, the results show that when the two populations taking the two tests are similar on the anchor score distributions, different equating methods yield the same or very similar results, even though they have different assumptions.
THE INFORMATION A TEST PROVIDES ON AN ABILITY PARAMETER
Authors: Haberman Shelby J. 
Published: August 2014

ABSTRACTIn item‐response theory, if a latent‐structure model has an ability variable, then elementary information theory may be employed to provide a criterion for evaluation of the information the test provides concerning ability. This criterion may be considered even in cases in which the latent‐structure model is not valid, although interpretation of the criterion is more complex in this case. It is also possible to consider reliability results for inferences about the ability variable. Use of sampling to develop required parameter estimates is straightforward. Applications of the criterion in zero‐, one‐, and two‐parameter logistic (0PL, 1PL, and 2PL) examples are provided.
PARAMETER RECOVERY AND SUBPOPULATION PROFICIENCY ESTIMATION IN HIERARCHICAL LATENT REGRESSION MODELS
Authors: Li Deping , Oranje Andreas , Jiang Yanlin 
Published: August 2014

ABSTRACTThe hierarchical latent regression model (HLRM) is a flexible framework for estimating group‐level proficiency while taking into account the complex sample designs often found in large‐scale educational surveys. A complex assessment design in which information is collected at different levels (such as student, school, and district), the model also provides a mechanism for estimating group differences at various levels and for partitioning variance components among those levels. This study examines parameter recovery in the HLRM and compares it to regular latent regression models (LRMs) through simulation for various levels of cluster variation. Results show that regression effect estimates are similar between the HLRM and the LRM, in particular under small cluster variation. Similarly, student posterior mean estimates and marginal maximum likelihood mean estimates for student groups are comparable across the two model approaches. However, substantial differences are found for the residual variance estimates, the standard errors for regression effect estimates and related standard errors for group estimates, and for students posterior variance estimates. As expected, these differences are larger when the variation across clusters is larger, since a substantial portion of variance remains unexplained in LRM.