bluuebunny/crossref_metadata_2025 huggingface dataset:
IMPOSSIBLE SCORES RESULTING IN ZERO FREQUENCIES IN THE ANCHOR TEST: IMPACT ON SMOOTHING AND EQUATING
Authors: Puhan Gautam , von Davier Alina , Gupta Shaloo
Published: August 2014

Equating under the external anchor design is frequently conducted using scaled scores on the anchor test. However, scaled scores often lead to the unique problem of creating zero frequencies in the score distribution because there may not always be a one‐to‐one correspondence between raw and scaled scores. For example, raw scores of 17 and 18 may correspond to scaled scores of 150 and 153, thereby creating zero frequencies for scaled scores of 151 and 152. These gaps in the frequency distribution may adversely impact smoothing and equating. This study examines the effect of these zero frequencies on log‐linear smoothing (Holland & Thayer, 1987) of score distributions and final equating results. Results suggest that although smoothing is significantly affected by the presence of these zero frequencies, as indicated by the likelihood‐ratio chi‐square, Akaike information criterion (Akaike, 1977), and Freeman‐Tukey deviates, the impact on the actual equating results is minimal.
EXAMINING AN ALTERNATIVE TO SCORE EQUATING: A RANDOMLY EQUIVALENT FORMS APPROACH
Authors: Liao Chi‐Wen , Livingston Samuel A.
Published: August 2014

Randomly equivalent forms (REF) of tests in listening and reading for nonnative speakers of English were created by stratified random assignment of items to forms, stratifying on item content and predicted difficulty. The study included 50 replications of the procedure for each test. Each replication generated 2 REFs. The equivalence of those 2 forms was evaluated by comparing the raw‐score distributions focusing on the greatest difference in the cumulative distributions. For listening, 10 replications produced cumulative distributions that differed at some point by more than 0.10, and 4 replications produced differences greater than 0.15. For reading, only 3 replications produced differences greater than 0.10. The difference between the results for listening and reading reflects the greater variation, within strata, in the difficulty of the listening items. The REF procedure may become more effective if item difficulty can be predicted more accurately.
ONE APPROACH TO DETECTING THE INVARIANCE OF PROFICIENCY STANDARDS OVER TIME
Authors: Qian Jiahe
Published: August 2014

This study explores the use of a mapping technique to test the invariance of proficiency standards over time for state performance tests. First, the state proficiency standards are mapped onto the National Assessment of Educational Progress (NAEP) scale. Then, rather than looking at whether there is a deviation in proficiency standards directly, the invariance of their NAEP equivalents is tested over time. The basis of the mapping technique is an enhanced method that was originally designed for comparing performance standards for public school students set by different states when the state tests are comparable. This approach can also be used to detect score inflation over time for state tests.
EFFECT OF IMMEDIATE FEEDBACK AND REVISION ON PSYCHOMETRIC PROPERTIES OF OPEN‐ENDED SENTENCE‐COMPLETION ITEMS
Authors: Attali Yigal , Powers Don , Hawthorn John
Published: August 2014

Registered examinees for the GRE® General Test answered open‐ended sentence‐completion items. For half of the items, participants received immediate feedback on the correctness of their answers and up to two opportunities to revise their answers. A significant feedback‐and‐revision effect was found. Participants were able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of the revised scores and their correlation with GRE verbal scores were higher. The possibility of using revision scores as a basis for measuring potential future learning is discussed.
EFFECT OF IMMEDIATE FEEDBACK AND REVISION ON PSYCHOMETRIC PROPERTIES OF OPEN‐ENDED GRE® SUBJECT TEST ITEMS
Authors: Attali Yigal , Powers Don
Published: August 2014

Registered examinees for the GRE® Subject Tests in Biology and Psychology participated in a Web‐based experiment where they answered open‐ended questions that required a short answer of 1‐3 sentences. Responses were automatically scored by natural language processing methods (the c‐rater™ scoring engine) immediately after participants submitted their responses. Based on natural language processing methods (the c‐rater scoring engine), participants received immediate feedback on the correctness of their answers and an opportunity to revise their answers. A significant revision effect was found. Participants were able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of revised scores was higher than initial scores, although the correlations of the initial and revised scores with the GRE Subject Test scores were similar.
EXAMINING THE IMPACT OF AUDIO PRESENTATION ON TESTS OF READING COMPREHENSION
Authors: Laitusis Cara Cahalan , Cook Linda L. , Cline Frederick , King Teresa C. , Sabatini John
Published: August 2014

This study examined the impact of a read‐aloud accommodation on standardized test scores of reading comprehension at Grades 4 and 8. Under a repeated measures design, students with and without reading‐based learning disabilities took both a standard administration and a read‐aloud administration of a reading comprehension test. Results show that the mean score on the audio version was higher than scores on the standard version for both groups of students at both grade levels. Students with reading‐based learning disabilities at both levels benefited differentially more than students with no disability. This finding continues to hold after controlling for reading fluency and ceiling effects at both grades. The results also examined the relationship between test scores and teachers' ratings of reading comprehension to determine which measures are the best predictors of teachers' ratings of reading comprehension by grade and disability classification.
THE INFLUENCE OF STRATEGIES FOR SELECTING LOGLINEAR SMOOTHING MODELS ON EQUATING FUNCTIONS
Authors: Moses Tim , Holland Paul
Published: August 2014

This study addressed 2 issues of using loglinear models for smoothing univariate test score distributions and for enhancing the stability of equipercentile equating functions. One issue was a comparative assessment of several statistical strategies that have been proposed for selecting 1 from several competing model parameterizations. Another issue was an evaluation of the influence of the selection strategies on equating function accuracy. These issues were considered in a simulation study, where the accuracies of 17 selection strategies for loglinear models and their effects on equating function accuracies were assessed across a range of sample sizes, test score distributions, and population equating functions. The results differentiate the selection strategies in terms of their accuracies in selecting correct model parameterizations and define the situations where their use has the most important implications for equating function accuracy.
THE IMPACT OF CHANGES IN THE TOEFL EXAMINATION ON TEACHING AND LEARNING IN CENTRAL AND EASTERN EUROPE: PHASE 2, COPING WITH CHANGE
Authors: Wall Dianne , Horák Tania
Published: August 2014

The aim of this report is to present the findings of the second phase in a longitudinal study of the impact of changes in the TOEFL® test on teaching and learning in test preparation classrooms. The focus of this phase was to monitor six teachers from five countries in Central and Eastern Europe as they received news about changes in the TOEFL and began thinking about how these might affect their teaching in the future. Data were gathered during the period of January to May 2005. The teachers responded to monthly tracking questions and tasks that explored their awareness of the old and new TOEFL tests, the features of their test preparation classes, their reactions to the most innovative parts of the new test, and their thoughts about the type of content and activities they would offer once the new TOEFL was operational in their countries. The report includes an analysis of the teachers' awareness, attitudes, and plans, and a discussion of the types of factors that could affect the shape and intensity of TOEFL washback in years to come.
UNDERSTANDING WHAT THE NUMBERS MEAN: A STRAIGHTFORWARD APPROACH TO GRE PREDICTIVE VALIDITY
Authors: Bridgeman Brent , Burton Nancy , Cline Frederick
Published: August 2014

Descriptions of validity results for the GRE® General Test based solely on correlation coefficients or percentage of the variance accounted for are not merely difficult to interpret, they are likely to be misinterpreted. Predictors that apparently account for a small percentage of the variance may actually be highly important from a practical perspective. This study used 2 existing data sets to demonstrate alternative methods of showing the value of the GRE as an indicator of 1st‐year graduate grades. The combined data sets contained 4,451 students in 6 graduate fields: biology, chemistry, education, English, experimental psychology, and clinical psychology. In one set of analyses, students within a department were divided into quartiles based on GRE scores and the percentage of students in the top and bottom quartiles earning a 4.0 average was noted. Students in the top quartile were 3 to 5 times as likely to earn 4.0 averages compared to students in the bottom quartile. Even after controlling for undergraduate grade point average quartiles, substantial differences related to GRE quartile remained.
MEASURING LEARNING OUTCOMES IN HIGHER EDUCATION USING THE MEASURE OF ACADEMIC PROFICIENCY AND PROGRESS (MAPP)
Authors: Liu Ou Lydia
Published: August 2014

The Secretary of Education's Commission on the Future of Higher Education emphasizes accountability in higher education as one of the key areas of interest. The Voluntary System of Accountability (VSA) was developed to evaluate the effectiveness of general public college education. This study examines how student progress in college, indicated by the performance difference between freshmen and seniors after controlling for admission scores, can be measured using the Measure of Academic Proficiency and Progress (MAPP) test. A total of 6,196 students from 23 institutions were included in this study. Results indicated that MAPP was able to differentiate the performance between freshmen and seniors after controlling for SAT®/ACT scores. The institutions were classified into 10 groups on the basis of the difference in the actual vs. expected MAPP performance. This study provides an example of how MAPP can be used to evaluate value‐added performance in college education. Issues such as student sampling and test‐taking motivation are discussed.
DESIGN PATTERNS FOR IMPROVING ACCESSIBILITY FOR TEST TAKERS WITH DISABILITIES
Authors: Hansen Eric G. , Mislevy Robert J.
Published: August 2014

There is a great need to help test designers determine how to make tests that are accessible to individuals with disabilities. This report takes design patterns, which were developed at SRI for assessment design, and uses them to clarify issues related to accessibility features for individuals with disabilities—such as low‐vision and blindness—taking a test of reading. Design patterns appear useful in clarifying how variable features of a test design need to be matched to disability‐related characteristics of test takers in order to ensure accessibility. Giving consideration to accessibility issues during the development and use of design patterns may help improve the validity and fairness of tests, as well as their accessibility for individuals with disabilities.
INVESTIGATING THE EFFECTIVENESS OF COLLATERAL INFORMATION ON SMALL‐SAMPLE EQUATING
Authors: Kim Sooyeon , Livingston Samuel A. , Lewis Charles
Published: August 2014

This paper describes an empirical evaluation of a Bayesian procedure for equating scores on test forms taken by small numbers of examinees, using collateral information from the equating of other test forms. In this procedure, a separate Bayesian estimate is derived for the equated score at each raw‐score level, making it unnecessary to specify a parametric model for the equating function. Collateral information can come either from other forms of the same test or, possibly, from other tests having a similar structure. Our evaluation consisted of two resampling studies. Each study applied the Bayesian procedure to small samples drawn from large‐sample data collected for an anchor equating. The large‐sample equating function served as the criterion. The results of the two studies were somewhat inconsistent, leading to different conclusions regarding the use of the empirical Bayesian procedure with small samples.
COMPARISON OF SUBSCORES BASED ON CLASSICAL TEST THEORY METHODS
Authors: Puhan Gautam , Sinharay Sandip , Haberman Shelby , Larkin Kevin
Published: August 2014

Will reporting subscores provide any additional information than the total score? Is there a method that can be used to provide more trustworthy subscores than observed subscores? These 2 questions are addressed in this study. To answer the 2nd question, 2 subscore estimation methods (i.e., subscore estimated from the observed total score or subscore estimated using both the observed subscore and observed total score) are compared. Analyses conducted on 8 certification tests indicated that reporting subscores at the examinee level may not be necessary as they do not provide much additional information than the total score. However, at the institutional level (for institution size greater than 30), reporting subscores may not be harmful, although it may be redundant. Finally, results indicated that subscores estimated using both the observed subscore and observed total score were the most trustworthy and may be used if subscores were to be reported.
COGNITIVE MODELS OF WRITING: WRITING PROFICIENCY AS A COMPLEX INTEGRATED SKILL
Authors: Deane Paul , Odendahl Nora , Quinlan Thomas , Fowles Mary , Welsh Cyndi , Bivens‐Tatum Jennifer
Published: August 2014

This paper undertakes a review of the literature on writing cognition, writing instruction, and writing assessment with the goal of developing a framework and competency model for a new approach to writing assessment. The model developed is part of the Cognitively Based Assessments of, for, and as Learning (CBAL) initiative, an ongoing research project at ETS intended to develop a new form of kindergarten through Grade 12 (K–12) assessment that is based on modern cognitive understandings; built around integrated, foundational, constructed‐response tasks that are equally useful for assessment and for instruction; and structured to allow multiple measurements over the course of the school year. The model that emerges from a review of the literature on writing places a strong emphasis on writing as an integrated, socially situated skill that cannot be assessed properly without taking into account the fact that most writing tasks involve management of a complex array of skills over the course of a writing project, including language and literacy skills, document‐creation and document‐management skills, and critical‐thinking skills. As such, the model makes strong connections with emerging conceptions of reading and literacy, suggesting an assessment approach in which writing is viewed as calling upon a broader construct than is usually tested in assessments that focus on relatively simple, on‐demand writing tasks.
STUDIES OF A LATENT‐CLASS SIGNAL‐DETECTION MODEL FOR CONSTRUCTED‐RESPONSE SCORING
Authors: DeCarlo Lawrence T.
Published: August 2014

Rater behavior in essay grading can be viewed as a signal‐detection task, in that raters attempt to discriminate between latent classes of essays, with the latent classes being defined by a scoring rubric. The present report examines basic aspects of an approach to constructed‐response (CR) scoring via a latent‐class signal‐detection model. The model provides a psychological framework for CR scoring and includes rater parameters with a clear cognitive basis. Simulations are used to examine how well rater parameters and latent‐class sizes are recovered as well as the accuracy of classification. The relation of rater parameters to agreement statistics and classification accuracy is examined. The effects of using a balanced, incomplete block design are compared to those for a fully crossed design. The model is applied to several ETS datasets.
EFFECTIVE AND SCALABLE TEACHER PROFESSIONAL DEVELOPMENT: A REPORT OF THE FORMATIVE RESEARCH AND DEVELOPMENT
Authors: Wylie E. Caroline , Lyon Christine J. , Mavronikolas Elia
Published: August 2014

This study is a qualitative analysis of data collected during a yearlong series of teacher learning community meetings and classroom observations. The participants are middle and high school mathematics teachers from 2 school districts. Teachers were introduced to the research behind formative assessment and how to apply that research to their teaching during a 3‐day summer workshop. Teachers then met monthly in small school‐based groups to deepen their understanding of formative assessment and to talk about their own classroom experiences. The analyses focused on how teachers' understanding changed over time, how the teacher learning communities supported the teachers, how learning translated into classroom practice, and the factors that supported or hindered the development of teachers' understanding and practice. Lessons learned during the study itself and from subsequent analyses of the data had a significant impact on the development of ETS's Keeping Learning on Track® program.
AN EVALUATION OF KERNEL EQUATING: PARALLEL EQUATING WITH CLASSICAL METHODS IN THE SAT SUBJECT TESTS™ PROGRAM
Authors: Grant Mary C. , Zhang Lilly (Yu‐Li) , Damiano ichele
Published: August 2014

This study investigated kernel equating methods by comparing these methods to operational equatings for two tests in the SAT Subject Tests™ program. GENASYS (ETS, 2007) was used for all equating methods and scaled score kernel equating results were compared to Tucker, Levine observed score, chained linear, and chained equipercentile equating results. The results of the kernel chained equatings using a large fixed bandwidth were nearly identical to the results from the chained linear equatings whereas the comparisons of the kernel poststratification equatings using a large bandwidth showed differences from both Tucker and Levine observed score equating results, although for most of the score range those differences were small. Similarly, the differences were small for most of the score range in the comparisons of kernel poststratification equatings and kernel chained equatings using the optimal bandwidth with chained equipercentile equatings.
SCORE EQUITY ASSESSMENT: DEVELOPMENT OF A PROTOTYPE ANALYSIS USING SAT® MATHEMATICS TEST DATA ACROSS SEVERAL ADMINISTRATIONS
Authors: Dorans Neil J. , Liu Jinghua
Published: August 2014

The equating process links scores from different editions of the same test. For testing programs that build nearly parallel forms to the same explicit content and statistical specifications and administer forms under the same conditions, the linkings between the forms are expected to be equatings. Score equity assessment (SEA) provides a useful tool to check form equatability. We suggest use of SEA as a quality control tool to evaluate how well a test assembly process works over several administrations. The examination of multiple forms should provide a proper assessment of the fairness of the test assembly process. We illustrate how to include SEA into statistical and psychometric practice with data from several administrations of the SAT®‐Math.
SUPPLEMENTING A TRADITIONAL MATH CURRICULUM WITH AN INQUIRY‐BASED PROGRAM: A PILOT OF MATH OUT OF THE BOX
Authors: Rock JoAnn L. , Courtney Rosalea , Handwerk Philip G.
Published: August 2014

This study examined math achievement of elementary school students when Math Out of the Box (MTB), an inquiry‐based math program, was used to supplement curriculum. The sample consisted of 767 New Jersey students in the third, fourth, and fifth grades, with approximately one third using MTB. Math achievement was measured by an assessment developed by ETS and by New Jersey's standardized test of math proficiency (NJ ASK). On the ETS assessment, a small, statistically significant difference was found in each of the three grades between students who used MTB and those who did not. On NJ ASK, a small, statistically significant difference was found in the third grade only. While these findings are an important step in establishing the efficacy of MTB, selection bias may weaken the causal inferences drawn.
EVALUATION OF THE EFFECTS OF LOGLINEAR SMOOTHING MODELS ON EQUATING FUNCTIONS IN THE PRESENCE OF STRUCTURED DATA IRREGULARITIES
Authors: Liu Jinghua , Moses Tim , Low Albert
Published: August 2014

Presmoothing a score distribution involves finding a balance between a good representation of the original data and smoothness. Smoothing reduces sampling variability, while a good data representation reduces the possibility of bias. When the test score distributions contain systematic irregularities that are due to scoring practice (e.g., formula scoring) rather than random irregularities, this trade‐off becomes complicated. The current study explored whether the systematic irregularities should be preserved or smoothed and the corresponding impacts on score equating. Two properties of the equating function were evaluated: distribution matching and smoothness. The overall results did not exhibit a consistent pattern in terms of the impacts on the equating function caused by smoothing out or preserving the systematic irregularities.
SMALL‐SAMPLE EQUATING WITH PRIOR INFORMATION
Authors: Livingston Samuel A. , Lewis Charles
Published: August 2014

This report proposes an empirical Bayes approach to the problem of equating scores on test forms taken by very small numbers of test takers. The equated score is estimated separately at each score point, making it unnecessary to model either the score distribution or the equating transformation. Prior information comes from equatings of other test forms, with an appropriate adjustment for possible differences in test length.
THE EFFECTS OF DIFFERENT TYPES OF ANCHOR TESTS ON OBSERVED SCORE EQUATING
Authors: Liu Jinghua , Sinharay Sandip , Holland Paul W. , Feigenbaum Miriam , Curley Edward
Published: August 2014

This study explores the use of a different type of anchor, a midi anchor, that has a smaller spread of item difficulties than the tests to be equated, and then contrasts its use with the use of a mini anchor. The impact of different anchors on observed score equating were evaluated and compared with respect to systematic error (bias), random equating error (standard errors of equating), and total equating error (RMSE) using empirical data. The results show that the midi anchor generally produces more accurate equating results than the mini anchor—with a few exceptions. Our findings suggest that a midi anchor test would be preferred to a mini anchor test if equating accuracy at the top and at the bottom of the score scale is not a primary concern.
A SCALE DRIFT STUDY
Authors: Liu Jinghua , Curley Edward , Low Albert
Published: August 2014

This study examines the stability of the SAT® scale from 1994 to 2001. A 1994 form and a 2001 form were readministered in a 2005 SAT administration, and the 1994 form was equated to the 2001 form. The new conversion was compared to the old conversion. Both the verbal and math sections exhibit a similar degree of scale drift, but in opposite directions: the verbal scale has drifted upward, whereas the math scale has drifted downward. We suggest testing programs monitor the score scales periodically by building a testing form schedule that allows a systematic and periodic checking of scale stability.
A STUDY OF FREQUENCY ESTIMATION EQUIPERCENTILE EQUATING WHEN THERE ARE LARGE ABILITY DIFFERENCES
Authors: Guo Hongwen , Oh Hyeonjoo J.
Published: August 2014

In operational equating, frequency estimation (FE) equipercentile equating is often excluded from consideration when the old and new groups have a large ability difference. This convention may, in some instances, cause the exclusion of one competitive equating method from the set of methods under consideration. In this report, we study the possibility of using the FE equating method when the group ability difference is large. Three situations are identified: (a) a situation in which neither the two forms nor the observed conditional distributions are very different so that the FE equating assumptions are likely to hold, and FE equating is recommended; (b) a situation in which forms are not very different, but the observed conditional distributions are different, so that FE equating is not recommended; and (c) a situation in which forms are very different, but the observed conditional distributions are not different, so that FE equating is not recommended. Statistical analysis procedures for comparing distributions are provided. An application of equating to a large‐scale admission test is discussed to illustrate the proposed methodology.
PSEUDO BAYES ESTIMATES FOR TEST SCORE DISTRIBUTIONS AND CHAINED EQUIPERCENTILE EQUATING
Authors: Moses Tim , Oh Hyeonjoo J.
Published: August 2014

Pseudo Bayes probability estimates are weighted averages of raw and modeled probabilities; these estimates have been studied primarily in nonpsychometric contexts. The purpose of this study was to evaluate pseudo Bayes probability estimates as applied to the estimation of psychometric test score distributions and chained equipercentile equating functions. Population test score distributions were created from actual test data and random samples of varied size were drawn from the populations. Pseudo Bayes estimation was applied to the random samples, using ranges of loglinear models and weights in the pseudo Bayes estimates' weighted averages of the raw and modeled test score probabilities. Equipercentile equating functions based on the pseudo Bayes estimates were also evaluated. Results indicated that the pseudo Bayes estimates have the potential to improve estimation accuracy for test score distributions and chained equipercentile equating functions in situations where loglinear modeling is not ideal and where finding the population loglinear model selection is not likely.
DIFFERENTIAL ITEM FUNCTIONING: ITS CONSEQUENCES
Authors: Lee Yi‐Hsuan , Zhang Jinming
Published: August 2014

This report examines the consequences of differential item functioning (DIF) using simulated data. Its impact on total score, item response theory (IRT) ability estimate, and test reliability was evaluated in various testing scenarios created by manipulating the following four factors: test length, percentage of DIF items per form, sample sizes of different examinee groups, and types of responses. The results indicate that the greatest score difference was observed between the examinee groups on forms with DIF items, and the magnitude was less than 2 points on the 0‐60 total score scale and .15 on the IRT ability scale. The influence on reliability was rather limited.
TEST TAKERS' ATTITUDES ABOUT THE TOEFL IBT™
Authors: Stricker Lawrence J. , Attali Yigal
Published: August 2014

The principal aims of this study, a conceptual replication of an earlier investigation of the TOEFL® computer‐based test, or TOEFL CBT, in Buenos Aires, Cairo, and Frankfurt, were to assess test takers' reported acceptance of the TOEFL Internet‐based test, or TOEFL iBT™, and its associations with possible determinants of this acceptance and with test performance; evaluate differences in the pattern of results for test takers from different countries; and compare the findings with those for the TOEFL CBT. A questionnaire concerning attitudes about the test and other relevant variables was administered by the Internet to TOEFL iBT examinees in 4 diverse countries with large testing volumes: China, Colombia, Egypt, and Germany. Overall attitudes about the TOEFL iBT were moderately positive in most countries, but neutral in Germany; attitudes about the Listening and Writing sections of the test were uniformly favorable in every country; but attitudes about the Speaking section were consistently less favorable in all countries and were unfavorable in Germany and Colombia. Attitudes about the test had similar patterns of relationships in the 4 countries: moderate correlations with attitudes about admissions tests in general, slight and inconsistent correlations with TOEFL performance and computer anxiety, and minimal correlations with other variables. All in all, these results were very similar to those in the earlier investigation, with 1 exception: the previous study found uniformly moderate positive attitudes about the TOEFL CBT in the 3 countries surveyed.
DEVELOPMENT OF A STUDENT HEALTH ASSESSMENT SYSTEM: HEALTH KNOWLEDGE, ATTITUDES, AND BEHAVIORS IN MIDDLE‐SCHOOL STUDENTS
Authors: MacCann Carolyn , Roberts Richard D.
Published: August 2014

Newly developed assessments of nutrition and exercise knowledge, attitudes, and behavior were administered to 383 eighth‐graders. Evidence for the validity of assessment scores was evaluated with five findings. First, parent‐ and self‐reported behaviors were similar and congruent for healthy eating and exercising but not for sedentary behaviors or unhealthy eating. Second, the theory of planned behavior structural model was confirmed for exercise and nutrition attitudes: Attitudes predicted exercising (R2=.34), healthy eating (R2=.21), sedentary behaviors (R2=.11), and unhealthy eating (R2=.09). Third, gender differences replicated previous research: Girls have better diets and nutrition attitudes whereas boys exercise more frequently. The assessments distinguished normal‐weight from obese individuals, and showed incremental validity in predicting school grades. The paper concludes with a discussion of the assessments' potential for informing interventions.
CHAINED VERSUS POST‐STRATIFICATION EQUATING IN A LINEAR CONTEXT: AN EVALUATION USING EMPIRICAL DATA
Authors: Puhan Gautam
Published: August 2014

This study used real data to construct testing conditions for comparing results of chained linear, Tucker, and Levine‐observed score equatings. The comparisons were made under conditions where the new‐ and old‐form samples were similar in ability and when they differed in ability. The length of the anchor test was also varied to enable examination of its effect on the three different equating methods. Two tests were used in the study, and the three equating methods were compared to a criterion equating to obtain estimates of random equating error, bias, and root mean squared error (RMSE). Results showed that for most of the conditions studied, chained linear score equating produced fairly good equating results in terms of low bias and RMSE. In some conditions, Levine‐observed score equating also produced low bias and RMSE. Although the Tucker method always produced the lowest random equating error, it produced a larger bias and RMSE than either of the other equating methods. Based on these results, it is recommended that either chained linear or Levine score equating be used when new‐ and old‐form samples differ in ability and/or when the anchor‐to‐total correlation is not very high.
STUDIES OF A LATENT CLASS SIGNAL DETECTION MODEL FOR CONSTRUCTED RESPONSE SCORING II: INCOMPLETE AND HIERARCHICAL DESIGNS
Authors: DeCarlo Lawrence T.
Published: August 2014

A basic consideration in large‐scale assessments that use constructed response (CR) items, such as essays, is how to allocate the essays to the raters that score them. Designs that are used in practice are incomplete, in that each essay is scored by only a subset of the raters, and also unbalanced, in that the number of essays scored by each rater differs across the raters. In addition, all of the possible rater pairs may not be used. The present study examines the effects of these factors on parameter recovery and classification accuracy using simulations of a latent class model based on signal detection theory (SDT). Many tests also include more than one CR item, which introduces a nested or hierarchical structure into the design, in that raters are nested within essays (i.e., there are multiple raters per essay) and essays are nested within examinees (i.e., each examinee provides two or more essays). A hierarchical rater model (HRM) has previously been developed to recognize the nested structure. A version of the HRM that incorporates a latent class signal detection model in the first level, referred to as the HRM‐SDT model, is presented. Parameter recovery in the HRM‐SDT model is examined in simulations. The model is applied to data from several ETS tests.
SINGLE‐ VERSUS DOUBLE‐SCORING OF TREND RESPONSES IN TREND SCORE EQUATING WITH CONSTRUCTED‐RESPONSE TESTS
Authors: Tan Xuan , Ricker Kathryn L. , Puhan Gautam
Published: August 2014

This study examines the differences in equating outcomes between two trend score equating designs resulting from two different scoring strategies for trend scoring when operational constructed‐response (CR) items are double‐scored—the single group (SG) design, where each trend CR item is double‐scored, and the nonequivalent groups with anchor test (NEAT) design, where each trend CR item is single‐scored during trend score equating—for varying sample sizes (n =150, 200, 250, 300, 400). Overall results suggest larger equating errors with smaller sample sizes, though errors were small regardless of sample size. The NEAT design performed about as well as the SG design with respect to conditional and summative standard errors of equating, though it did tend to produce larger bias and root mean‐squared differences (RMSDs). When accounting for the total number of trend scores required to do analyses, the NEAT design performed as well or better than the SG design (e.g., when the NEAT n =150 and the SG n = 300). This result might be partially attributable to a larger operational sample size (n = 792) and a good correlation between anchor and total score for the trend sample (r = 0.73). These results suggest that under these testing conditions, the NEAT design performed about as well as the SG design, but further research is required to assess the generalizability of the results.
A SUMMARY OF MODELS AND STANDARDS‐BASED APPLICATIONS FOR GRADE‐TO‐GRADE GROWTH ON STATEWIDE ASSESSMENTS AND IMPLICATIONS FOR STUDENTS WITH DISABILITIES
Authors: Buzick Heather M. , Laitusis Cara Cahalan
Published: August 2014

Recently growth‐based approaches to accountability have received considerable attention because they have the potential to reward schools and teachers for improving student performance over time by measuring the progress of students at all levels of the performance spectrum (including those who have not yet reached proficiency on state accountability assessments). While the use of growth in accountability holds promise for students with disabilities, measuring changes over time in their academic performance is complex. This paper summarizes models and approaches that use individual student test scores from multiple years for 3 different purposes: determination of adequate yearly progress under the federal accountability system, research on individual growth trajectories, and evaluation of the contribution of teachers and schools to student learning. Practical challenges in measuring and modeling growth for students with disabilities are then discussed. Finally, 3 areas in need of research on the measurement of growth from large‐scale annual accountability assessments are identified and described: testing accommodations, test difficulty, and understanding the longitudinal characteristics of the population of students with disabilities.
A COMPARISON OF METHODS FOR ESTIMATING CONDITIONAL ITEM SCORE DIFFERENCES IN DIFFERENTIAL ITEM FUNCTIONING (DIF) ASSESSMENTS
Authors: Moses Tim , Miao Jing , Dorans Neil
Published: August 2014

This study compared the accuracies of four differential item functioning (DIF) estimation methods, where each method makes use of only one of the following: raw data, logistic regression, loglinear models, or kernel smoothing. The major focus was on the estimation strategies' potential for estimating score‐level, conditional DIF. A secondary focus was on assessing the accuracy of strategies' overall DIF effect sizes and statistical significance tests. A real data simulation was used to evaluate the estimation strategies with 6 items representing DIF and No DIF situations, and with 4 sample size combinations for the reference and focal group data. Results showed that the logistic regression estimation strategy was the most highly recommended strategy in terms of the bias and variability of its estimates and the power of its statistical significance test. The loglinear models strategy had flexibility advantages, but these advantages only offset the greater variability of its estimates and its reduced statistical power when sample sizes were large. The kernel smoothing estimation strategy was the least accurate of the considered strategies due to estimation problems when the reference and focal groups differed in overall ability.
LINKING MIXED‐FORMAT TESTS USING MULTIPLE‐CHOICE ANCHORS
Authors: Walker Michael E. , Kim Sooyeon
Published: August 2014

This study examined the use of an all multiple‐choice (MC) anchor for linking mixed format tests containing both MC and constructed‐response (CR) items, in a nonequivalent groups design. An MC‐only anchor could effectively link two such test forms if either (a) the MC and CR portions of the test measured the same construct, so that the MC anchor adequately represented the entire test, or (b) the relationship between the MC portion and the total test remained constant across the new and reference linking groups. The study also evaluated whether linking mixed‐format tests through MC‐only anchors would be more effective than a two‐stage strategy in which MC portions were equated through MC anchors and then composite scores were scaled to the MC scores. Anchor linking and two‐stage linking yielded identical (or nearly so) results for both linear and nonlinear chained linking methods. With post‐stratification linking methods the two‐stage strategy resulted in smaller bias. The paper discusses some advantages of both approaches.
UNFAIR TREATMENT VS. CONFIRMATION BIAS? COMMENTS ON SANTELICES AND WILSON
Authors: Dorans Neil J.
Published: August 2014

Santelices and Wilson (2010) claimed to have addressed technical criticisms of Freedle (2003) presented in Dorans (2004a) and elsewhere. Santelices and Wilson's abstract claimed that their study confirmed that SAT® verbal items do function differently for African American and White subgroups. In this commentary, I demonstrate that the Santelices and Wilson article provided no evidence to confirm differential item functioning (DIF) and failed to address my technical criticisms of Freedle. Instead, Santelices and Wilson contained several misrepresentations, including substituting “considered serious” for “more unusual” to describe an effect size and claiming to have studied four editions of the SAT when only two were actually studied. Central to its thesis was a hypothesis about DIF/difficulty correlations that was misattributed to Dorans. Santelices and Wilson based their argument for DIF on correlations between highly correlated variations on an index of DIF with item difficulty. While failing to demonstrate either unfairness in the test items or unfairness in the treatment of Freedle, Santelices and Wilson did contain evidence of confirmation bias.
TEACHER LEADERSHIP: AN ASSESSMENT FRAMEWORK FOR AN EMERGING AREA OF PROFESSIONAL PRACTICE
Authors: Jackson Teresa , Burrus Jeremy , Bassett Katherine , Roberts Richard D.
Published: August 2014

This paper examines various definitions and frameworks that have been used to justify the emergence of a new category for the teacher professional: teacher leader. The emergence of this new professional category may lead to greater retention levels, and improved knowledge management and transfer within the teaching profession. Various key dimensions of this profession are examined, allowing us to highlight some key personal skills that would appear requisite for the teacher leader. An agenda for developing and validating assessments of teacher leadership is then proposed. It is argued that these assessments have the potential of legitimizing research within this field, as well as providing the opportunity to better understand what it takes to become a successful professional in this new domain of teaching practice.
EQUATING OF SUBSCORES AND WEIGHTED AVERAGES UNDER THE NEAT DESIGN
Authors: Sinharay Sandip , Haberman Shelby
Published: August 2014

Recently, the literature has seen increasing interest in subscores for their potential diagnostic values; for example, one study suggested the report of weighted averages of a subscore and the total score, whereas others showed, for various operational and simulated data sets, that weighted averages, as compared to subscores, lead to more accurate diagnostic information. To report weighted averages, the averages should be comparable across different test forms; that is, the averages should be equated. This report discusses how to equate weighted averages. Results from operational and simulated data sets demonstrate the small error found when equating weighted averages.
STATISTICAL PROCEDURES TO EVALUATE QUALITY OF SCALE ANCHORING
Authors: Haberman Shelby J. , Sinharay Sandip , Lee Yi‐Hsuan
Published: August 2014

Providing information to test takers and test score users about the abilities of test takers at different score levels has been a persistent problem in educational and psychological measurement (Carroll, 1993). Scale anchoring (Beaton & Allen, 1992), a technique that describes what students at different points on a score scale know and can do, is a tool to provide such information. Scale anchoring for a test involves substantial amount of work, both by the statistical analysts and test developers involved with the test. In addition, scale anchoring involves considerable use of subjective judgment, so its conclusions may be questionable. This paper describes statistical procedures that can be used to determine if scale anchoring is likely to be successful for a test. If these procedures indicate that scale anchoring is unlikely to be successful, then there is little reason to perform a detailed scale anchoring study. The procedures are applied to several data sets from a teacher licensing test.
ASSESSING THE FALSIFIABILITY OF EXTREME LINKING
Authors: Middleton Kyndra , Dorans Neil J.
Published: August 2014

Extreme linkings are performed in settings in which neither equivalent groups nor anchor material is available to link scores on two assessments. Examples of extreme linkages include links between scores on tests administered in different languages or between scores on tests administered across disability groups. The strength of interpretation attached to a linkage depends on the proper design and execution of a sound data collection plan. The current paper uses a real data set to illustrate how to indirectly assess the quality of linking the scores on two assessments that contain neither equivalent groups nor common anchor material.
EVALUATING EMPIRICAL RELATIONSHIPS AMONG PREDICTION, MEASUREMENT, AND SCALING INVARIANCE
Authors: Moses Tim
Published: August 2014

The purpose of this study was to consider the relationships of prediction, measurement, and scaling invariance when these invariances were simultaneously evaluated in psychometric test data. An approach was developed to evaluate prediction, measurement, and scaling invariance based on linear and nonlinear prediction, measurement, and scaling functions. The approach was used to evaluate the relationships among 12 pairs of tests in 6 datasets for gender invariance. The prediction, measurement, and scaling invariance results were found to be similar for most of the test relationships evaluated, in that all 3 invariances were more likely to be established for highly correlated tests than for less highly correlated tests. The invariance results appeared to be well summarized by intercept differences in the linear prediction, measurement, and scaling functions. The implications of the results were discussed with respect to the relationships among prediction, measurement, and scaling invariances described in prior theoretical and empirical research. Suggestions for extending theoretical and empirical invariance research were provided.
AUTOMATED SCORING WITHIN A DEVELOPMENTAL, COGNITIVE MODEL OF WRITING PROFICIENCY
Authors: Deane Paul , Quinlan Thomas , Kostin Irene
Published: August 2014

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.
MEASUREMENT OF NEW ATTRIBUTES FOR CHILE'S ADMISSIONS SYSTEM TO HIGHER EDUCATION
Authors: Santelices María Verónica , Ugarte Juan José , Flotts Paulina , Radovic Darinka , Kyllonen Patrick
Published: August 2014

This paper presents the development and initial validation of new measures of critical thinking and noncognitive attributes that were designed to supplement existing standardized tests used in the admissions system for higher education in Chile. The importance of various facets of this process, including the establishment of technical rigor and political support of instrument development, piloting, and implementation is also emphasized because the results from this pilot study could affect the admissions process for some of the most prestigious and competitive universities in the country. The new instruments were pilot tested on a group of high school seniors (n = 1,568) and first‐year college students (n = 1,443) during 2008. The sample included students from 20 secondary schools and 4 universities. Initial results were promising and showed that the measures were effectively assessing attributes not currently considered in the Chilean higher education admissions process. In addition, performance on the new instruments was found to be less correlated with students' sociodemographic characteristics than were measures currently used for undergraduate admissions.
USE OF E‐RATER® IN SCORING OF THE TOEFL IBT® WRITING TEST
Authors: Haberman Shelby J.
Published: August 2014

Alternative approaches are discussed for use of e‐rater® to score the TOEFL iBT® Writing test. These approaches involve alternate criteria. In the 1st approach, the predicted variable is the expected rater score of the examinee's 2 essays. In the 2nd approach, the predicted variable is the expected rater score of 2 essay responses by the examinee on a parallel form. This 2nd approach is related to prediction of the expected rater score of 2 essay responses on an actual form taken later by an examinee. The relationship of e‐rater scores to scores on other sections of TOEFL® is also considered. These alternative approaches suggest somewhat different procedures for scoring.