bluuebunny/crossref_metadata_2025 huggingface dataset:
APPLYING THE ONLINE SCORING NETWORK (OSN) TO ADVANCED PLACEMENT PROGRAM® (AP®) TESTS
Authors: Zhang Yuli (Lilly) , Powers Donald E. , Wright Wendi , Morgan Rick
Published: August 2014

This project explored the feasibility of using the ETS Online Scoring Network (OSN) to score selected Advanced Placement Program® (AP®) tests. In particular, the quality of scores obtained from traditional large‐scale readings was compared with the quality of scores obtained from OSN scoring. (For the former, readers convene in a central location to evaluate student responses in a paper‐based format; for the latter, they work alone from remote locations to evaluate responses that are transmitted via the Internet.) The study also obtained readers' reactions to OSN training and to OSN capabilities for monitoring reader performance online.

The results, based on more than 11,000 test takers who had taken either the AP English Language and Composition exam or the AP Calculus AB exam, revealed little if any difference between scores assigned at traditional central readings and those assigned when readers used OSN. The lack of differences applied to (a) score level and variability, (b) interreader agreement, and (c) passing rates on the two AP exams that were investigated. Study participants did, however, suggest several areas in which improvements could be made to OSN procedures.
EFFECT OF EXTRA TIME ON GRE® QUANTITATIVE AND VERBAL SCORES
Authors: Bridgeman Brent , Cline Frederick , Hessinger James
Published: August 2014

The purpose of this study was to test the assumption that the Graduate Record Examinations (GRE®) General Test is a measure of academic reasoning abilities in which speed of responding plays at most a minor role. In addition to completing the operational GRE General Test, participants each completed a research version of either the GRE verbal or quantitative test within either the standard time limit or within one‐and‐a‐half times the standard time limit. Scores obtained from 15,948 examinees indicate that extra time added about seven points (on the 200‐800 score scale) to examinees' verbal scores and seven points to their quantitative scores. Scores under the different timing conditions were generally comparable across gender and ethnic groups, but quantitative scores were slightly higher for lower‐ability examinees who had more time.
VARIANCE COMPONENT TESTING IN GENERALIZED LINEAR MIXED MODELS
Authors: Sinharay Sandip , Stern Hal S.
Published: August 2014

Generalized linear mixed models (GLMM) are used in situations where a number of characteristics (covariates) affect a nonnormal response variable and the responses are correlated. For example, in a number of biological applications, the responses are correlated due to common genetic or environmental factors. In many applications, the magnitude of the variance components corresponding to one or more of the random effects are of interest, especially the point null hypothesis that one or more of the variance components are zero. This work reviews a number of approaches for estimating the Bayes factor comparing the models with and without the random effects in question. The computations involved with finding Bayes factors for these models pose many challenges, and we discuss how one can overcome them.

We perform a comparative study of the different approaches to compute Bayes factors for GLMMs by applying them to two different data sets. The first example employs a probit regression model with a single variance component to data from a natural selection study on turtles. The second example uses a disease mapping model from epidemiology, a Poisson regression model with two variance components being used to analyze the data. The importance sampling method is found to be the method of choice to compute the Bayes factor of interest for these problems. Chib's method is also found to be efficient.
CALIBRATION OF POLYTOMOUS ITEM FAMILIES USING BAYESIAN HIERARCHICAL MODELING
Authors: Johnson Matthew S. , Sinharay Sandip
Published: August 2014

For complex educational assessments, there is an increasing use of item families, which are groups of related items. However, calibration or scoring for such an assessment requires fitting models that take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families with multiple‐choice items. This paper extends the model to take into account item families with polytomous items and designs a Markov chain Monte Carlo (MCMC) algorithm for the Bayesian estimation of the model parameters. The hierarchical model, which accounts for the dependence structure inherent among the items, implicitly defines the family response function for the score categories. This paper suggests a way to combine the family response functions over the score categories to obtain a family score function, which is a quick graphical summary of the expected score of an individual with a certain ability on an item randomly generated from an item family. This paper also suggests a method for the Bayesian estimation of the family response function and family score function. This work is a significant step towards building a tool to analyze data involving item families and may be very useful practically, for example, in automatic item generation systems that create tests involving item families.
NOTES ON LINEAR EQUATING METHODS FOR THE NON‐EQUIVALENT‐GROUPS DESIGN
Authors: von Davier Alina A.
Published: August 2014

This paper focuses on three observed‐score methods for linear equating in a Non‐Equivalent‐groups Anchor Test (NEAT) design—Tucker, Levine observed‐score, and chain. The methods and their assumptions are described, and theoretical conditions under which the three methods give identical results are presented.
WHAT IS QUANTITATIVE REASONING? DEFINING THE CONSTRUCT FOR ASSESSMENT PURPOSES
Authors: Dwyer Carol Anne , Gallagher Ann , Levin Jutta , Morley Mary E.
Published: August 2014

In order to create fair and valid assessments, it is necessary to be clear about what is to be measured and how the resulting data should be interpreted. For a number of historical and practical reasons described in this paper, adequately detailed statements with both a theoretical and empirical base do not currently exist for the construct of quantitative reasoning for use in assessments. There is also no adequate explanation of the important differences between assessments that measure quantitative reasoning constructs and those that are intended to measure achievement in related mathematical content areas.

The literature in psychology, psychometrics, philosophy, and education, while containing much that is relevant to the construct of quantitative reasoning, unfortunately does not provide materials that can be used in research and development to address such practical issues or to explain the basic nature of quantitative reasoning assessments. This paper briefly discusses the importance and use of constructs and the quantitative reasoning and standards literature. It then presents a statement about the construct of quantitative reasoning for assessment purposes within a construct validity framework that includes both a definition of the construct and threats to valid score interpretation. These threats are based on related but distinguishable constructs and other types of construct‐irrelevant variance in the assessments.
BAYESIAN ITEM FIT ANALYSIS FOR DICHOTOMOUS ITEM RESPONSE THEORY MODELS
Authors: Sinharay Sandip
Published: August 2014

Detecting item fit for common dichotomous item response theory (IRT) models has always been an issue of enormous interest, but there exists no unanimously agreed upon item fit diagnostic for the models. This paper employs theposterior predictive model checkingmethod (Guttman, 1967; Rubin, 1981, 1984), a popular Bayesian model checking tool, to examine item fit for common dichotomous item response theory models. An item fit plot, comparing the observed and predicted proportion correct scores of examinee groups (with groups based on raw scores), promises to be useful in real applications. This paper also suggests how to obtain posterior predictive p‐values for the χ2‐type test statistics of Orlando and Thissen (2000) comparing the observed and predicted proportion correct scores for different raw score groups. A number of simulation studies and real data applications examine the effectiveness of the suggested item fit diagnostics. The suggested p‐values seem to have low Type I error rate, low false alarm rate, and adequate power, indicating that researchers might find the methods more acceptable than the existing item fit measures.
A SIMULATION STUDY TO EXPLORE CONFIGURING THE NEW SAT® CRITICAL READING SECTION WITHOUT ANALOGY ITEMS
Authors: Liu Jinghua , Feigenbaum Miriam , Cook Linda
Published: August 2014

This study explored possible configurations of the new SAT® critical reading section without analogy items. The item pool contained items from SAT verbal (SAT–V) sections of 14 previously administered SAT tests, calibrated using the three‐parameter logistic IRT model. Multiple versions of several prototypes that do not contain analogy items were assembled. Item statistics and test statistics for the simulated forms were compared to the average of 13 forms of the SAT–V. These statistics included: IRT scaled score reliability, scaled score standard error of measurement, conditional scaled score standard error of measurement, r‐biserial, and equated deltas. The results indicated that it is possible to maintain measurement precision for the new SAT critical reading section without analogy items, but it may be necessary to modify the distribution of item difficulty in order to obtain adequate precision at the ends of the score scale.
A COMPARISON OF TWO MODELS FOR COGNITIVE DIAGNOSIS
Authors: Yan Duanli , Almond Russell , Mislevy Robert
Published: August 2014

Diagnostic score reports linking assessment outcomes to instructional interventions are one of the most requested features of assessment products. There is a body of interesting work done in the last 20 years including Tatsuoka's rule space method (Tatsuoka, 1983), Haertal and Wiley's binary skills model (Haertal, 1984; Haertal & Wiley, 1993), and Mislevy, Almond, Yan, and Steinberg's Bayesian inference network (Mislevy, Almond, Yan, & Steinberg, 1999). Recent research has resulted in major breakthroughs for the use of a parametric IRT model for performing skills diagnoses. Hartz, Roussos, and Stout (2002) have developed an identifiable and flexible model called the fusion model.

This paper compares the fusion model and the Bayesian inference network in the design and analysis of an assessment, including theQ‐matrix, and then compares other related models, including rule space, item response theory (IRT), and general multivariate latent class models. The paper also attempts to characterize the kinds of problems for which each type of measurement model is well‐suited. A general Bayesian psychometric framework provides common language, making it easier to appreciate the differences.

In addition, this paper explores some of the strengths and weaknesses of each approach based on a comparative analysis of a cognitive assessment, the mixed number subtraction data set. In this case, both the fusion model and Bayesian network approaches yield similar performance characteristics and also seem to pick up on different characteristics.
ELICITED SPEECH FROM GRAPH ITEMS ON THE TEST OF SPOKEN ENGLISH™
Authors: Katz Irvin R. , Xi Xiaoming , Kim Hyun‐Joo , Cheng Peter C‐H.
Published: August 2014

This research applied a cognitive model to identify item features that lead to irrelevant variance on the Test of Spoken English™ (TSE®). The TSE is an assessment of English oral proficiency and includes an item that elicits a description of a statistical graph. This item type sometimes appears to tap graph‐reading skills—an irrelevant construct; TSE raters report that many examinees perform worse on this item type than they do on the other 11 items in the test. We adapted a cognitive theory of graph comprehension to predict the degree to which TSE graph items tap irrelevant skills such as graph reading. Through analyses of existing TSE data as well as an experiment, we show how the theory provides specific, empirically justified recommendations on the construction of graph items that minimize the influence of extraneous skills.
A UNIFIED APPROACH TO IRT SCALE LINKING AND SCALE TRANSFORMATIONS
Authors: von Davier Matthias , von Davier Alina A.
Published: August 2014

This paper examines item response theory (IRT) scale transformations and IRT scale linking methods used in the Non‐Equivalent Groups with Anchor Test (NEAT) design to equate two tests, X and Y. It proposes a unifying approach to the commonly used IRT linking methods: mean‐mean, mean‐var linking, concurrent calibration, Stocking and Lord and Haebara characteristic curves approaches, and fixed‐item parameters scale linkage. The main idea is to view any linking procedure as a restriction on the item parameter space. Then a rewriting of the log‐likelihood function together with an appropriately implemented maximization procedure of the log‐likelihood function under linear (or nonlinear restrictions) will accomplish the linking. The proposed method is general enough to cover both the dichotomous item response models (the one parameter logistic (1PL) model, 2PL, and 3PL) and the polytomous unidimensional IRT models like the generalized partial credit model.
A GRAPHICAL APPROACH TO ITEM ANALYSIS
Authors: Livingston Samuel A. , Dorans Neil J.
Published: August 2014

This paper describes an approach to item analysis that is based on the estimation of a set of response curves for each item. The response curves show, at a glance, the difficulty and the discriminating power of the item and the popularity of each distractor, at any level of the criterion variable (e.g., total score). The curves are estimated by Gaussian kernel smoothing, a weighted moving average process with a parameter that can be varied at the user's discretion. The response curve for the correct answer can be accompanied by curves indicating a confidence region. The response curves also form the basis for estimating item statistics for any group of examinees for which the distribution of the criterion variable is known.
NATIONAL BOARD FOR PROFESSIONAL TEACHING STANDARDS BIAS‐REDUCTION TRAINING: IMPACT ON ASSESSORS' AWARENESS
Authors: Wylie E. Caroline , Szpara Michelle Y.
Published: August 2014

This study is an in‐depth investigation of the National Board for Professional Teaching Standards (NBPTS) bias‐reduction training, from the perspective of assessors. The research examined how successful the bias training was in guiding assessors to recognize their biases and to identify actions to be used to reduce the impact of bias on their scoring decisions. The study focused on portfolio video entries to allow for a greater exploration of bias, since assessors are provided with a visual and aural depiction of the candidates and their students.
ASSESSING COGNITIVE ACHIEVEMENT GROWTH DURING THE KINDERGARTEN AND FIRST GRADE YEARS
Authors: Rock Donald A. , Pollack Judy M. , Weiss Michael
Published: August 2014

This study attempts to identify different patterns of cognitive growth in kindergarten and first grade associated with selected subpopulations. The results are based upon a nationally representative sample of fall kindergartners who were retested in the spring of their kindergarten year and then again in the fall and spring of their first grade year. Of special interest here is the estimation of the growth rates of subpopulations of children that are often considered to be at risk, educationally and/or economically. This study also investigates whether or not the absence of formal schooling during the summer differentially impacts subpopulations who are “at risk” because they may lack a strong educational support system in the home.
COMPARABILITY OF TOEFL CBT WRITING PROMPTS: RESPONSE MODE ANALYSES
Authors: Breland Hunter , Lee Yong‐Won , Muraki Eiji
Published: August 2014

Eighty‐three Test of English as a Foreign Language™ (TOEFL®) CBT writing prompts that were administered between July 1998 and August 2000 were examined in order to identify differences in scores that could be attributed to the response mode chosen by examinees (handwritten or word processed). Differences were examined statistically using polytomous logistic regression. An English language ability (ELA) variable was developed from the multiple‐choice components of the TOEFL examination and used as a matching variable. Although there was little observed difference in mean writing scores, when examinees were matched on English language ability, small differences were observed in effect sizes consistently favoring the handwritten response mode. The difference favoring the handwritten response mode occurred for all of the writing prompts analyzed; however, the differences for individual writing prompts were small. This difference suggests a general effect for response mode.
LOGLINEAR SMOOTHING: AN ALTERNATIVE NUMERICAL APPROACH USING SAS
Authors: Moses Tim , von Davier Alina A. , Casabianca Jodi
Published: August 2014

The purpose of this report is to demonstrate loglinear smoothing using SAS PROC GENMOD. The results from four published examples, which include the smoothing of a) univariate distributions, b) bivariate distributions, c) distributions with teeth, and d) bivariate distributions with structural zeros, are reproduced to show the flexibility of the SAS procedure. Comparisons of graphical displays and likelihood ratio statistics show that the SAS procedure yields results identical to the published results. SAS PROC GENMOD provides an alternative approach to smoothing that is readily available for researchers and graduate students in educational measurement, as well as researchers from other scientific fields.
TOLERABLE VARIATION IN ITEM PARAMETER ESTIMATES FOR LINEAR AND ADAPTIVE COMPUTER‐BASED TESTING
Authors: Rizavi Saba , Way Walter D. , Davey Tim , Herbert Erin
Published: August 2014

Item parameter estimates vary for a variety of reasons, including estimation error, characteristics of the examinee samples, and context effects (e.g., item location effects, section location effects, etc.). Although we expect variation based on theory, there is reason to believe that observed variation in item parameter estimates exceeds what theory would predict. This study examined both items that were administered linearly in a fixed order each time that they were used and items that had appeared in different adaptive testing item pools. The study looked at both the magnitude of variation in the item parameter estimates and the impact of this variation in the estimation of test‐taker scores. The results showed that the linearly administered items exhibited remarkably small variation in parameter estimates over repeated calibrations. Similar findings with adaptively administered items in another high stakes testing program were also found when initial adaptively based item parameter estimates were compared with estimates from repeated use. The results of this study also indicated that context effects played a more significant role in adaptive item parameters when the comparisons were made to the parameters that were initially obtained from linear paper‐and‐pencil testing.
AN INVESTIGATION OF THE IMPACT OF COMPOSITION MEDIUM ON THE QUALITY OF TOEFL WRITING SCORES
Authors: Wolfe Edward W. , Manalo Jonathan R.
Published: August 2014

This study examined scores from 133,906 operationally scored Test of English as a Foreign Language™ (TOEFL®) essays to determine whether the choice of composition medium has any impact on score quality for subgroups of test‐takers. Results of analyses demonstrate that (a) scores assigned to word‐processed essays are slightly more reliable than scores assigned to handwritten essays and exhibit higher correlations with TOEFL multiple‐choice subscores; (b) female test‐takers, examinees whose native language is not based on a Roman/Cyrillic alphabet, and examinees with lower English proficiency are more likely to choose the handwriting medium; (c) the probability of choosing handwriting as the composition medium increases with age for Asian examinees, but decreases with age for most European examinees; and (d) examinees with lower TOEFL multiple‐choice scores tend to have higher handwritten than word‐processed essay scores, while examinees with higher TOEFL multiple‐choice scores tend to have similar scores in either medium.
A COMPARISON OF TWO PROCEDURES FOR CONSTRAINED ADAPTIVE TEST CONSTRUCTION
Authors: Robin Frédéric , van der Linden Wim J. , Eignor Daniel R. , Steffen Manfred , Stocking Martha L.
Published: August 2014

The relatively new shadow test approach (STA) to computerized adaptive testing (CAT) proposed by Wim van der Linden is a potentially attractive alternative to the weighted deviation algorithm (WDA) implemented at ETS. However, it has not been evaluated under testing conditions representative of current ETS testing programs. Of interest was whether STA would, under typical high‐stakes on‐demand testing situations, produce tests of comparable or better psychometric quality as those produced by the current weighted deviation algorithm. Based on simulated data, we found that the STA performed as well or slightly better than the WDA on two of the three commonly accepted testing objectives: measurement and content. The WDA appeared to perform slightly better than the STA when the issue is security or item exposure control. The paper provides a review of the rationale that led to the specific testing objectives employed, an outline of the test construction steps common to the two procedures investigated, and a description of the specific models and algorithms employed by both procedures. Detailed description of the simulation study conducted and the results obtained from both of the procedures for one of the pools are also provided. Results are summarized and further research needs in particular areas discussed.
A STUDY OF FATIGUE EFFECTS FROM THE NEW SAT®
Authors: Liu Jinghua , Allspach Jill R. , Feigenbaum Miriam , Oh Hyeon‐Joo , Burton Nancy
Published: August 2014

This study evaluated whether the addition of a writing section to the SAT Reasoning Test™ (referred to as the SAT® in this study) would impact test‐taker performance because of fatigue caused by increased test length. The study also investigated test‐takers' subjective feelings of fatigue. Ninety‐seven test‐takers were randomly assigned to three groups: the first group took a current SAT with no essay; the second group took a pseudo new SAT composed of the current SAT plus an essay, with the essay appearing in the first section of the test; and the third group also took the pseudo new SAT with an essay, but with the essay in the last section. Test‐taker performance on the verbal and math sections and the essay was then evaluated and compared. The results indicated that while the extended testing time for the new SAT may cause test‐takers to feel fatigued, fatigue did not affect test‐taker performance.
DESIGNING ADAPTIVE, DIAGNOSTIC MATH ASSESSMENTS FOR INDIVIDUALS WITH AND WITHOUT VISUAL DISABILITIES
Authors: Shute Valerie J. , Graf Edith Aurora , Hansen Eric G.
Published: August 2014

This report summarizes the design and development of an adaptive e‐learning prototype for middle school mathematics for use with both sighted and visually disabled students. Adaptation refers to the system's ability to adjust itself to suit particular characteristics of the learner. The main parts of the report describe the system's theoretical foundation, architecture, models, and adaptive algorithm. We also review approaches for making assessment systems accessible to students with visual disabilities. Finally, we conclude with a summary of upcoming studies in relation to important research questions concerning micro‐ and macroadaptation. Using a design approach like the one described in this report may set a new precedent for environments that adapt to support student learning based on larger sets of incoming abilities and disabilities than have been considered previously.
JOINT AND CONDITIONAL ESTIMATION FOR IMPLICIT MODELS FOR TESTS WITH POLYTOMOUS ITEM SCORES
Authors: Haberman Shelby J.
Published: August 2014

Multinomial‐response models are available that correspond implicitly to tests in which a total score is computed as the sum of polytomous item scores. For these models, joint and conditional estimation may be considered in much the same way as for the Rasch model for right‐scored tests. As in the Rasch model, joint estimation is only attractive if both the number of items and the number of examinees are large, while conditional estimation can be employed for a large number of examinees whether or not the number of items is large. In neither case is computation difficult given currently available computers. Large‐sample results favor use of conditional estimation, although some use of joint estimation can be contemplated if the number of items is large.
A SAS MACRO FOR LOGLINEAR SMOOTHING: APPLICATIONS AND IMPLICATIONS
Authors: Moses Tim P. , von Davier Alina A.
Published: August 2014

The two purposes of this paper are to provide a SAS IML macro that performs loglinear smoothing and to apply this macro to loglinear smoothing problems that have not been extensively discussed in the test‐equating literature. The SAS macro is demonstrated on univariate, bivariate, and trivariate smoothing problems. The univariate and bivariate examples reproduce published results (von Davier, Holland, & Thayer, 2004). The trivariate example extends the bivariate smoothing example to allow for comparisons of subgroups' univariate and bivariate distributions. The implications are that important questions about distribution differences and subpopulation invariance of equating functions can be considered through comparisons and evaluations of complex loglinear models that are easily fit with this SAS IML macro.
LIMITS ON LOG CROSS‐PRODUCT RATIOS FOR ITEM RESPONSE MODELS
Authors: Haberman Shelby J. , Holland Paul W. , Sinharay Sandip
Published: August 2014

Bounds are established for log cross‐product ratios (log odds ratios) involving pairs of items for item response models. First, expressions for bounds on log cross‐product ratios are provided for unidimensional item response models in general. Then, explicit bounds are obtained for the Rasch model and the two‐parameter logistic (2PL) model. Results are also illustrated through an example from a study of model‐checking procedures. The bounds obtained can provide a basis for assessment of goodness of fit of these models.
AN ELEMENTARY TEST OF THE NORMAL 2PL MODEL AGAINST THE NORMAL 3PL ALTERNATIVE
Authors: Haberman Shelby J.
Published: August 2014

A simple score test of the normal two‐parameter logistic (2PL) model is presented that examines the potential attraction of the normal three‐parameter logistic (3PL) model for use with a particular item. Application is made to data from a test from the Praxis™ series. Results from this example raise the question whether the normal 3PL model should be used routinely in preference to the normal 2PL model unless evidence exists that a substantial gain in description of data is achieved.
TESTING THE UNTESTABLE ASSUMPTIONS OF THE CHAIN AND POSTSTRATIFICATION EQUATING METHODS FOR THE NEAT DESIGN
Authors: Holland Paul W. , von Davier Alina A. , Sinharay Sandip , Han Ning
Published: August 2014

This paper focuses on the Non‐Equivalent Groups with Anchor Test (NEAT) design for test equating and on two classes of observed—score equating (OSE) methods—chain equating (CE) and poststratification equating (PSE). These two classes of methods reflect two distinctly different ways of using the information provided by the anchor test for computing OSE functions. Each of the two classes includes linear and nonlinear equating methods. In practical situations, it is known that the PSE and CE methods tend to give different results when the two groups of examinees differ in ability. However, given that both methods are justified by making untestable assumptions, it is difficult to conclude which, if either, of the two equating approaches is more correct. This study compares predictions from both the PSE and the CE assumptions that can be tested in a comparable way with the data from a special study. Results indicate that both CE and PSE make very similar predictions but that those of CE are slightly more accurate than those of PSE.
TOWARD AN UNDERSTANDING OF THE ROLE OF SPEECH RECOGNITION IN NONNATIVE SPEECH ASSESSMENT
Authors: Zechner Klaus , Bejar Isaac I. , Hemat Ramin
Published: August 2014

The increasing availability and performance of computer‐based testing has prompted more research on the automatic assessment of language and speaking proficiency. In this investigation, we evaluated the feasibility of using an off‐the‐shelf speech‐recognition system for scoring speaking prompts from the LanguEdge field test of 2002. We first established the level of agreement between two trained scorers. We then adapted a speech engine to the language backgrounds and proficiency ranges of the speakers and developed a classification and regression tree (CART) for each of five prompts based on features computed from the output of the speech recognizer. In a validation on held‐out data, we found that while our features are not sufficiently comprehensive to adequately score these prompts, collectively these features appear to capture reliably some aspects of speaking proficiency.
CONFIDENCE AND COGNITIVE TEST PERFORMANCE
Authors: Stankov Lazar , Lee Jihyun
Published: August 2014

This paper examines the nature of confidence in relation to cognitive abilities, personality traits, and metacognition. Confidence was measured as it was expressed in answers to each test item during the administration of reading and listening sections of the TOEFL® iBT. The confidence scores were correlated with the accuracy scores from the TOEFL iBT, SAT®, high school grade point averages (HS‐GPA), and several measures of personality and metacognition. The results indicate that confidence is a separate psychological trait, somewhere between cognitive ability and personality traits. In addition, our findings suggest that confidence is related to, but separate from, metacognition. We also demonstrated gender and ethnic differences in confidence, with male and African American students showing higher overconfidence bias than females and White or Hispanic students, respectively. Finally, our data show small but statistically significant incremental validity of the confidence scores above and beyond the accuracy scores in predicting numeracy test scores, the total TOEFL iBT scores, and TOEFL iBT subscores on the writing and speaking sections. We found no incremental validity of the confidence scores in predicting the SAT and HS‐GPA. Theoretical and practical implications of these findings are discussed as well.
A NOTE ON GAIN SCORES AND THEIR INTERPRETATION IN DEVELOPMENTAL MODELS DESIGNED TO MEASURE CHANGE IN THE EARLY SCHOOL YEARS
Authors: Rock Donald A.
Published: August 2014

This paper presents a strategy for measuring cognitive gains in reading during the early school years. It is argued that accurate estimates of gain scores and their appropriate interpretation requires the use of adaptive tests with multiple criterion referenced points that mark learning milestones. It is further argued that two different measures of gain are necessary because of the nonequivalence of scale score points. Empirical results are presented in support of the use of adaptive tests with multiple criteria referenced scale points for the measurement of gain.
ESTIMATION OF STANDARD ERROR OF REGRESSION EFFECTS IN LATENT REGRESSION MODELS USING BINDER'S LINEARIZATION
Authors: Li Deping , Oranje Andreas
Published: August 2014

Two versions of a general method for approximating standard error of regression effect estimates within an IRT‐based latent regression model are compared. The general method is based on Binder's (1983) approach, accounting for complex samples and finite populations by Taylor series linearization. In contrast, the current National Assessment of Educational Progress procedure assumes a simple random sample for standard error of regression effects and applies a jackknife estimator to statistics of interest as a way to account for NAEP's complex sample. In this study, the versions of the general method are formally defined and the general method is extended to multiple dimensions. Furthermore, they are applied in an empirical study to the 2004 NAEP long‐term trend data comparing both large, nearly saturated, and small models. Subsequently, the results are compared to the operational‐based imputation method. Results show no impact on the imputation‐based results, limited impact on large models, and reasonable impact on small models. While it is not readily apparent to what this differential impact can be attributed, several explanations are discussed.
SMALL‐SAMPLE DIF ESTIMATION USING LOG‐LINEAR SMOOTHING: A SIBTEST APPLICATION
Authors: Puhan Gautam , Moses Tim P. , Yu Lei , Dorans Neil J.
Published: August 2014

The purpose of the current study was to examine whether log‐linear smoothing of observed score distributions in small samples results in more accurate differential item functioning (DIF) estimates under the simultaneous item bias test (SIBTEST) framework. Data from a teacher certification test were analyzed using White candidates in the reference group and African American candidates in the focal group. Smoothed and raw DIF estimates from 100 replications under seven different sample‐size conditions were compared to a criterion to determine the effect of smoothing on small‐sample DIF estimation. Root‐mean‐squared deviation and bias were used to evaluate the accuracy of DIF detection in the smoothed versus raw data conditions. Results indicate that, for most studied items, smoothing the raw score distributions reduced variability and bias of the DIF estimates especially in the small‐sample‐size conditions. Implications of these results for actual testing programs and future directions for research are discussed.
FOCUS ON FORMATIVE FEEDBACK
Authors: Shute Valerie J.
Published: August 2014

This paper reviews the corpus of research on feedback, with a particular focus on formative feedback—defined as information communicated to the learner that is intended to modify the learner's thinking or behavior for the purpose of improving learning. According to researchers in the area, formative feedback should be multidimensional, nonevaluative, supportive, timely, specific, credible, infrequent, and genuine (e.g., Brophy, 1981; Schwartz & White, 2000). Formative feedback is usually presented as information to a learner in response to some action on the learner's part. It comes in a variety of types (e.g., verification of response accuracy, explanation of the correct answer, hints, worked examples) and can be administered at various times during the learning process (e.g., immediately following an answer, after some period of time has elapsed). Finally, there are a number of variables that have been shown to interact with formative feedback's success at promoting learning (e.g., individual characteristics of the learner and aspects of the task). All of these issues will be discussed in this paper. This review concludes with a set of guidelines for generating formative feedback.
ON THE LATENT REGRESSION MODEL OF ITEM RESPONSE THEORY
Authors: Antal Tamás
Published: August 2014

Full account of the latent regression model for the National Assessment of Educational Progress is given. The treatment includes derivation of the EM algorithm, Newton‐Raphson method, and the asymptotic standard errors. The paper also features the use of the adaptive Gauss‐Hermite numerical integration method as a basic tool to evaluate expectations necessary to perform the parameter estimations.
SYMMETRIC NONLINEAR REGRESSION
Authors: Antal Tamás
Published: August 2014

An estimation tool for symmetric univariate nonlinear regression is presented. The method is based on introducing a nontrivial set of affine coordinates for diffeomorphisms of the real line. The main ingredient making the computations possible is the Connes‐Moscovici Hopf algebra of these affine coordinates.
RELIABILITY AND THE NONEQUIVALENT GROUPS WITH ANCHOR TEST DESIGN
Authors: Moses Tim , Kim Sooyeon
Published: August 2014

This study evaluated the impact of unequal reliability on test equating methods in the nonequivalent groups with anchor test (NEAT) design. Classical true score‐based models were compared in terms of their assumptions about how reliability impacts test scores. These models were related to treatment of population ability differences by different NEAT equating methods. A score model was then developed based on the most important features of the reviewed score models and used to study reliability in a simulation study across a total of 45 measurement conditions (= 5 test and anchor reliability combinations × 3 population ability difference conditions × 3 sample sizes). Ten equating methods were considered: chained linear, chained equipercentile with raw and smoothed frequencies, Tucker, frequency estimation equipercentile with raw and smoothed frequencies, Levine observed using Angoff‐estimated and the “correct” reliabilities based on the data generation model used in this study, and Levine true using Angoff‐estimated and correct reliabilities. The results were consistent with what is known about equating functions and their variability. Unequal and/or low reliability inflates equating function variability and alters equating functions when population abilities differ.
AN EXPLORATION OF KERNEL EQUATING USING SAT® DATA: EQUATING TO A SIMILAR POPULATION AND TO A DISTANT POPULATION
Authors: Liu Jinghua , Low Albert C.
Published: August 2014

This study applied kernel equating (KE) in two scenarios: equating to a very similar population and equating to a very different population, referred to as a distant population, using SAT® data. The KE results were compared to the results obtained from analogous classical equating methods in both scenarios. The results indicate that KE results are comparable to the results of other methods. Further, the results show that when the two populations taking the two tests are similar on the anchor score distributions, different equating methods yield the same or very similar results, even though they have different assumptions.
THE INFORMATION A TEST PROVIDES ON AN ABILITY PARAMETER
Authors: Haberman Shelby J.
Published: August 2014

In item‐response theory, if a latent‐structure model has an ability variable, then elementary information theory may be employed to provide a criterion for evaluation of the information the test provides concerning ability. This criterion may be considered even in cases in which the latent‐structure model is not valid, although interpretation of the criterion is more complex in this case. It is also possible to consider reliability results for inferences about the ability variable. Use of sampling to develop required parameter estimates is straightforward. Applications of the criterion in zero‐, one‐, and two‐parameter logistic (0PL, 1PL, and 2PL) examples are provided.
PARAMETER RECOVERY AND SUBPOPULATION PROFICIENCY ESTIMATION IN HIERARCHICAL LATENT REGRESSION MODELS
Authors: Li Deping , Oranje Andreas , Jiang Yanlin
Published: August 2014

The hierarchical latent regression model (HLRM) is a flexible framework for estimating group‐level proficiency while taking into account the complex sample designs often found in large‐scale educational surveys. A complex assessment design in which information is collected at different levels (such as student, school, and district), the model also provides a mechanism for estimating group differences at various levels and for partitioning variance components among those levels. This study examines parameter recovery in the HLRM and compares it to regular latent regression models (LRMs) through simulation for various levels of cluster variation. Results show that regression effect estimates are similar between the HLRM and the LRM, in particular under small cluster variation. Similarly, student posterior mean estimates and marginal maximum likelihood mean estimates for student groups are comparable across the two model approaches. However, substantial differences are found for the residual variance estimates, the standard errors for regression effect estimates and related standard errors for group estimates, and for students posterior variance estimates. As expected, these differences are larger when the variation across clusters is larger, since a substantial portion of variance remains unexplained in LRM.
EVALUATION OF AUTOMATED DIAGNOSIS AND INSTRUCTION FOR MATHEMATICS PROBLEM‐SOLVING
Authors: Underwood Jody S.
Published: August 2014

The past several decades have seen numerous approaches toward automated diagnosis and instructional support of students engaged in mathematics problem‐solving. These approaches typically involve detailed analysis of potential solution paths for problems, formal representations of correct and incorrect answers, and support in the form of feedback or explanations to students during the process of solving a problem. The approaches of each of a number of representative systems (ACED, ALEKS, Cognitive Tutors, Andes, and Assistments) will be described through a critical evaluation of how they represent content knowledge, their approaches to diagnosis, and their approaches to instructional support. Finally, recommendations are made for new approaches to automated diagnosis and instructional support of mathematics problem‐solving.
ON MULTIDIMENSIONAL ITEM RESPONSE THEORY: A COORDINATE‐FREE APPROACH
Authors: Antal Tamás
Published: August 2014

A coordinate‐free definition of complex‐structure multidimensional item response theory (MIRT) for dichotomously scored items is presented. The point of view taken emphasizes the possibilities and subtleties of understanding MIRT as a multidimensional extension of the classical unidimensional item response theory models. The main theorem of the paper is that every monotonic MIRT model looks the same; they are all trivial extensions of univariate item response theory.
SCALE DRIFT IN EQUATING ON A TEST THAT EMPLOYS CUT SCORES
Authors: Puhan Gautam
Published: August 2014

The purpose of this study is to determine the extent of scale drift on a test that employs cut scores. It is essential to examine scale drift in a testing program using new forms that are often put on scale through a series of intermediate equatings (known as equating chains). This may cause equating error to accumulate to a point where scale scores are rendered incomparable across two parallel chains or time periods. The study examined whether scale drift occurred for two conditions (i.e., parallel equating chains and a single long chain). Data from three tests that employed cut scores were used in this study. Results indicated that although there was some difference observed between the actual equating conversions derived via different equating chains, the effect of these differences on actual pass/fail status of test takers was not large. However, these results may change if new cut scores are introduced. Also, for tests that do not employ cut scores, stability in most parts of the score scale is important, not just at particular cut score points.
THE STANDARDIZED LETTER OF RECOMMENDATION: IMPLICATIONS FOR SELECTION
Authors: Liu Ou Lydia , Minsky Jennifer , Ling Guangming , Kyllonen Patrick
Published: August 2014

In an effort to standardize academic application procedures, the Standardized Letter of Recommendation (SLR) was developed to capture important cognitive and noncognitive qualities of graduate school candidates. The SLR consists of seven scales (knowledge, analytical skills, communication skills, motivation, self‐ organization, professionalism and maturity, and teamwork) and was applied to an intern‐selection scenario. Both professor ratings (N = 414) during the application process and mentor ratings of the selected students (N = 51) after the internship was completed were collected using the SLR. A multidimensional Rasch investigation suggests that the seven scales of the SLR displayed satisfactory psychometric properties in terms of reliability, model fit, item fit statistics, and discrimination. The two cognitive scales, knowledge and analytical skills, were found to be the best predictors for intern selection. The professor ratings and mentor ratings had moderate to high correlations, with the professor ratings being systematically higher than the mentor ratings. Possible reasons for the rating discrepancies are discussed. Also, implications for how the SLR can be used and improved in other selection situations are suggested.
ON‐THE‐FLY CUSTOMIZATION OF AUTOMATED ESSAY SCORING
Authors: Attali Yigal
Published: August 2014

Because there is no commonly accepted view of what makes for good writing, automated essay scoring (AES) ideally should be able to accommodate different theoretical positions, certainly at the level of state standards but also perhaps among teachers at the classroom level. This paper presents a practical approach and an interactive computer program for judgment‐based customization.

This approach is based on the AES system, e‐rater®. Through this new approach, a user can gain easy accessibility to system components, flexibility in adjusting scoring parameters, and procedures for making scoring adjustments that can be based on only a few benchmark essays. The interactive prototype program that implements this approach allows the user to customize e‐rater and watch the effect on benchmark essay scores as well as on score distributions for a reference testing program of the user's choice. The paper presents results for the use of this approach in customizing e‐rater to the standards of different assessments.
PREDICTING GRADES IN DIFFERENT TYPES OF COLLEGE COURSES
Authors: Bridgeman Brent , Pollack Judith , Burton Nancy
Published: August 2014

The ability of high school grades (high school GPA) and SAT® scores to predict cumulative grades in different types of college courses was evaluated in a sample of 26 colleges. Each college contributed data from three cohorts of entering freshmen, and each cohort was followed for at least four years. Colleges were separated into four levels by average SAT scores. Grade point averages for four categories of courses (English; science, math, and engineering [S/M/E]; social science; and education) were computed, and analyses were run separately for gender within race/ethnicity classifications. Correlations of the combined predictors with course grades over four or more years, corrected for range restriction, ranged from .45 for education courses to 0.64 for S/M/E courses. The SAT increment, that is, the increase in the multiple correlations when SAT scores are added to high school grades, ranged from 0.03 in education courses to 0.08 in S/M/E courses. Because these seemingly small numbers are frequently misinterpreted, an additional analysis showed how the percentage of students succeeding at a high level (cumulative GPA of 3.5 or higher) increases as SAT scores increase for students with similar high school grades. For example, for students with a high school GPA of 3.7 or higher in colleges where the mean combined SAT score is below 1200, only 2 percent of the students at the lowest SAT level (800 or lower combined score) were highly successful in social science courses. At the highest SAT level (1410‐1600), 77 percent were highly successful.