bluuebunny/crossref_metadata_2025_split huggingface dataset:
EXAMINING AN ALTERNATIVE TO SCORE EQUATING: A RANDOMLY EQUIVALENT FORMS APPROACH
Authors: Liao Chi‐Wen , Livingston Samuel A. 
Published: August 2014

ABSTRACTRandomly equivalent forms (REF) of tests in listening and reading for nonnative speakers of English were created by stratified random assignment of items to forms, stratifying on item content and predicted difficulty. The study included 50 replications of the procedure for each test. Each replication generated 2 REFs. The equivalence of those 2 forms was evaluated by comparing the raw‐score distributions focusing on the greatest difference in the cumulative distributions. For listening, 10 replications produced cumulative distributions that differed at some point by more than 0.10, and 4 replications produced differences greater than 0.15. For reading, only 3 replications produced differences greater than 0.10. The difference between the results for listening and reading reflects the greater variation, within strata, in the difficulty of the listening items. The REF procedure may become more effective if item difficulty can be predicted more accurately.
ONE APPROACH TO DETECTING THE INVARIANCE OF PROFICIENCY STANDARDS OVER TIME
Authors: Qian Jiahe 
Published: August 2014

ABSTRACTThis study explores the use of a mapping technique to test the invariance of proficiency standards over time for state performance tests. First, the state proficiency standards are mapped onto the National Assessment of Educational Progress (NAEP) scale. Then, rather than looking at whether there is a deviation in proficiency standards directly, the invariance of their NAEP equivalents is tested over time. The basis of the mapping technique is an enhanced method that was originally designed for comparing performance standards for public school students set by different states when the state tests are comparable. This approach can also be used to detect score inflation over time for state tests.
EFFECT OF IMMEDIATE FEEDBACK AND REVISION ON PSYCHOMETRIC PROPERTIES OF OPEN‐ENDED SENTENCE‐COMPLETION ITEMS
Authors: Attali Yigal , Powers Don , Hawthorn John 
Published: August 2014

ABSTRACTRegistered examinees for the GRE® General Test answered open‐ended sentence‐completion items. For half of the items, participants received immediate feedback on the correctness of their answers and up to two opportunities to revise their answers. A significant feedback‐and‐revision effect was found. Participants were able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of the revised scores and their correlation with GRE verbal scores were higher. The possibility of using revision scores as a basis for measuring potential future learning is discussed.
EFFECT OF IMMEDIATE FEEDBACK AND REVISION ON PSYCHOMETRIC PROPERTIES OF OPEN‐ENDED GRE® SUBJECT TEST ITEMS
Authors: Attali Yigal , Powers Don 
Published: August 2014

ABSTRACTRegistered examinees for the GRE® Subject Tests in Biology and Psychology participated in a Web‐based experiment where they answered open‐ended questions that required a short answer of 1‐3 sentences. Responses were automatically scored by natural language processing methods (the c‐rater™ scoring engine) immediately after participants submitted their responses. Based on natural language processing methods (the c‐rater scoring engine), participants received immediate feedback on the correctness of their answers and an opportunity to revise their answers. A significant revision effect was found. Participants were able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of revised scores was higher than initial scores, although the correlations of the initial and revised scores with the GRE Subject Test scores were similar.
EXAMINING THE IMPACT OF AUDIO PRESENTATION ON TESTS OF READING COMPREHENSION
Authors: Laitusis Cara Cahalan , Cook Linda L. , Cline Frederick , King Teresa C. , Sabatini John 
Published: August 2014

ABSTRACTThis study examined the impact of a read‐aloud accommodation on standardized test scores of reading comprehension at Grades 4 and 8. Under a repeated measures design, students with and without reading‐based learning disabilities took both a standard administration and a read‐aloud administration of a reading comprehension test. Results show that the mean score on the audio version was higher than scores on the standard version for both groups of students at both grade levels. Students with reading‐based learning disabilities at both levels benefited differentially more than students with no disability. This finding continues to hold after controlling for reading fluency and ceiling effects at both grades. The results also examined the relationship between test scores and teachers' ratings of reading comprehension to determine which measures are the best predictors of teachers' ratings of reading comprehension by grade and disability classification.
THE INFLUENCE OF STRATEGIES FOR SELECTING LOGLINEAR SMOOTHING MODELS ON EQUATING FUNCTIONS
Authors: Moses Tim , Holland Paul 
Published: August 2014

ABSTRACTThis study addressed 2 issues of using loglinear models for smoothing univariate test score distributions and for enhancing the stability of equipercentile equating functions. One issue was a comparative assessment of several statistical strategies that have been proposed for selecting 1 from several competing model parameterizations. Another issue was an evaluation of the influence of the selection strategies on equating function accuracy. These issues were considered in a simulation study, where the accuracies of 17 selection strategies for loglinear models and their effects on equating function accuracies were assessed across a range of sample sizes, test score distributions, and population equating functions. The results differentiate the selection strategies in terms of their accuracies in selecting correct model parameterizations and define the situations where their use has the most important implications for equating function accuracy.
THE IMPACT OF CHANGES IN THE TOEFL EXAMINATION ON TEACHING AND LEARNING IN CENTRAL AND EASTERN EUROPE: PHASE 2, COPING WITH CHANGE
Authors: Wall Dianne , Horák Tania 
Published: August 2014

ABSTRACTThe aim of this report is to present the findings of the second phase in a longitudinal study of the impact of changes in the TOEFL® test on teaching and learning in test preparation classrooms. The focus of this phase was to monitor six teachers from five countries in Central and Eastern Europe as they received news about changes in the TOEFL and began thinking about how these might affect their teaching in the future. Data were gathered during the period of January to May 2005. The teachers responded to monthly tracking questions and tasks that explored their awareness of the old and new TOEFL tests, the features of their test preparation classes, their reactions to the most innovative parts of the new test, and their thoughts about the type of content and activities they would offer once the new TOEFL was operational in their countries. The report includes an analysis of the teachers' awareness, attitudes, and plans, and a discussion of the types of factors that could affect the shape and intensity of TOEFL washback in years to come.
UNDERSTANDING WHAT THE NUMBERS MEAN: A STRAIGHTFORWARD APPROACH TO GRE PREDICTIVE VALIDITY
Authors: Bridgeman Brent , Burton Nancy , Cline Frederick 
Published: August 2014

ABSTRACTDescriptions of validity results for the GRE® General Test based solely on correlation coefficients or percentage of the variance accounted for are not merely difficult to interpret, they are likely to be misinterpreted. Predictors that apparently account for a small percentage of the variance may actually be highly important from a practical perspective. This study used 2 existing data sets to demonstrate alternative methods of showing the value of the GRE as an indicator of 1st‐year graduate grades. The combined data sets contained 4,451 students in 6 graduate fields: biology, chemistry, education, English, experimental psychology, and clinical psychology. In one set of analyses, students within a department were divided into quartiles based on GRE scores and the percentage of students in the top and bottom quartiles earning a 4.0 average was noted. Students in the top quartile were 3 to 5 times as likely to earn 4.0 averages compared to students in the bottom quartile. Even after controlling for undergraduate grade point average quartiles, substantial differences related to GRE quartile remained.
MEASURING LEARNING OUTCOMES IN HIGHER EDUCATION USING THE MEASURE OF ACADEMIC PROFICIENCY AND PROGRESS (MAPP)
Authors: Liu Ou Lydia 
Published: August 2014

ABSTRACTThe Secretary of Education's Commission on the Future of Higher Education emphasizes accountability in higher education as one of the key areas of interest. The Voluntary System of Accountability (VSA) was developed to evaluate the effectiveness of general public college education. This study examines how student progress in college, indicated by the performance difference between freshmen and seniors after controlling for admission scores, can be measured using the Measure of Academic Proficiency and Progress (MAPP) test. A total of 6,196 students from 23 institutions were included in this study. Results indicated that MAPP was able to differentiate the performance between freshmen and seniors after controlling for SAT®/ACT scores. The institutions were classified into 10 groups on the basis of the difference in the actual vs. expected MAPP performance. This study provides an example of how MAPP can be used to evaluate value‐added performance in college education. Issues such as student sampling and test‐taking motivation are discussed.
DESIGN PATTERNS FOR IMPROVING ACCESSIBILITY FOR TEST TAKERS WITH DISABILITIES
Authors: Hansen Eric G. , Mislevy Robert J. 
Published: August 2014

ABSTRACTThere is a great need to help test designers determine how to make tests that are accessible to individuals with disabilities. This report takes design patterns, which were developed at SRI for assessment design, and uses them to clarify issues related to accessibility features for individuals with disabilities—such as low‐vision and blindness—taking a test of reading. Design patterns appear useful in clarifying how variable features of a test design need to be matched to disability‐related characteristics of test takers in order to ensure accessibility. Giving consideration to accessibility issues during the development and use of design patterns may help improve the validity and fairness of tests, as well as their accessibility for individuals with disabilities.
INVESTIGATING THE EFFECTIVENESS OF COLLATERAL INFORMATION ON SMALL‐SAMPLE EQUATING
Authors: Kim Sooyeon , Livingston Samuel A. , Lewis Charles 
Published: August 2014

ABSTRACTThis paper describes an empirical evaluation of a Bayesian procedure for equating scores on test forms taken by small numbers of examinees, using collateral information from the equating of other test forms. In this procedure, a separate Bayesian estimate is derived for the equated score at each raw‐score level, making it unnecessary to specify a parametric model for the equating function. Collateral information can come either from other forms of the same test or, possibly, from other tests having a similar structure. Our evaluation consisted of two resampling studies. Each study applied the Bayesian procedure to small samples drawn from large‐sample data collected for an anchor equating. The large‐sample equating function served as the criterion. The results of the two studies were somewhat inconsistent, leading to different conclusions regarding the use of the empirical Bayesian procedure with small samples.
COMPARISON OF SUBSCORES BASED ON CLASSICAL TEST THEORY METHODS
Authors: Puhan Gautam , Sinharay Sandip , Haberman Shelby , Larkin Kevin 
Published: August 2014

ABSTRACTWill reporting subscores provide any additional information than the total score? Is there a method that can be used to provide more trustworthy subscores than observed subscores? These 2 questions are addressed in this study. To answer the 2nd question, 2 subscore estimation methods (i.e., subscore estimated from the observed total score or subscore estimated using both the observed subscore and observed total score) are compared. Analyses conducted on 8 certification tests indicated that reporting subscores at the examinee level may not be necessary as they do not provide much additional information than the total score. However, at the institutional level (for institution size greater than 30), reporting subscores may not be harmful, although it may be redundant. Finally, results indicated that subscores estimated using both the observed subscore and observed total score were the most trustworthy and may be used if subscores were to be reported.
COGNITIVE MODELS OF WRITING: WRITING PROFICIENCY AS A COMPLEX INTEGRATED SKILL
Authors: Deane Paul , Odendahl Nora , Quinlan Thomas , Fowles Mary , Welsh Cyndi , Bivens‐Tatum Jennifer 
Published: August 2014

ABSTRACTThis paper undertakes a review of the literature on writing cognition, writing instruction, and writing assessment with the goal of developing a framework and competency model for a new approach to writing assessment. The model developed is part of the Cognitively Based Assessments of, for, and as Learning (CBAL) initiative, an ongoing research project at ETS intended to develop a new form of kindergarten through Grade 12 (K–12) assessment that is based on modern cognitive understandings; built around integrated, foundational, constructed‐response tasks that are equally useful for assessment and for instruction; and structured to allow multiple measurements over the course of the school year. The model that emerges from a review of the literature on writing places a strong emphasis on writing as an integrated, socially situated skill that cannot be assessed properly without taking into account the fact that most writing tasks involve management of a complex array of skills over the course of a writing project, including language and literacy skills, document‐creation and document‐management skills, and critical‐thinking skills. As such, the model makes strong connections with emerging conceptions of reading and literacy, suggesting an assessment approach in which writing is viewed as calling upon a broader construct than is usually tested in assessments that focus on relatively simple, on‐demand writing tasks.
STUDIES OF A LATENT‐CLASS SIGNAL‐DETECTION MODEL FOR CONSTRUCTED‐RESPONSE SCORING
Authors: DeCarlo Lawrence T. 
Published: August 2014

ABSTRACTRater behavior in essay grading can be viewed as a signal‐detection task, in that raters attempt to discriminate between latent classes of essays, with the latent classes being defined by a scoring rubric. The present report examines basic aspects of an approach to constructed‐response (CR) scoring via a latent‐class signal‐detection model. The model provides a psychological framework for CR scoring and includes rater parameters with a clear cognitive basis. Simulations are used to examine how well rater parameters and latent‐class sizes are recovered as well as the accuracy of classification. The relation of rater parameters to agreement statistics and classification accuracy is examined. The effects of using a balanced, incomplete block design are compared to those for a fully crossed design. The model is applied to several ETS datasets.
EFFECTIVE AND SCALABLE TEACHER PROFESSIONAL DEVELOPMENT: A REPORT OF THE FORMATIVE RESEARCH AND DEVELOPMENT
Authors: Wylie E. Caroline , Lyon Christine J. , Mavronikolas Elia 
Published: August 2014

ABSTRACTThis study is a qualitative analysis of data collected during a yearlong series of teacher learning community meetings and classroom observations. The participants are middle and high school mathematics teachers from 2 school districts. Teachers were introduced to the research behind formative assessment and how to apply that research to their teaching during a 3‐day summer workshop. Teachers then met monthly in small school‐based groups to deepen their understanding of formative assessment and to talk about their own classroom experiences. The analyses focused on how teachers' understanding changed over time, how the teacher learning communities supported the teachers, how learning translated into classroom practice, and the factors that supported or hindered the development of teachers' understanding and practice. Lessons learned during the study itself and from subsequent analyses of the data had a significant impact on the development of ETS's Keeping Learning on Track® program.
AN EVALUATION OF KERNEL EQUATING: PARALLEL EQUATING WITH CLASSICAL METHODS IN THE SAT SUBJECT TESTS™ PROGRAM
Authors: Grant Mary C. , Zhang Lilly (Yu‐Li) , Damiano ichele 
Published: August 2014

ABSTRACTThis study investigated kernel equating methods by comparing these methods to operational equatings for two tests in the SAT Subject Tests™ program. GENASYS (ETS, 2007) was used for all equating methods and scaled score kernel equating results were compared to Tucker, Levine observed score, chained linear, and chained equipercentile equating results. The results of the kernel chained equatings using a large fixed bandwidth were nearly identical to the results from the chained linear equatings whereas the comparisons of the kernel poststratification equatings using a large bandwidth showed differences from both Tucker and Levine observed score equating results, although for most of the score range those differences were small. Similarly, the differences were small for most of the score range in the comparisons of kernel poststratification equatings and kernel chained equatings using the optimal bandwidth with chained equipercentile equatings.
SCORE EQUITY ASSESSMENT: DEVELOPMENT OF A PROTOTYPE ANALYSIS USING SAT® MATHEMATICS TEST DATA ACROSS SEVERAL ADMINISTRATIONS
Authors: Dorans Neil J. , Liu Jinghua 
Published: August 2014

ABSTRACTThe equating process links scores from different editions of the same test. For testing programs that build nearly parallel forms to the same explicit content and statistical specifications and administer forms under the same conditions, the linkings between the forms are expected to be equatings. Score equity assessment (SEA) provides a useful tool to check form equatability. We suggest use of SEA as a quality control tool to evaluate how well a test assembly process works over several administrations. The examination of multiple forms should provide a proper assessment of the fairness of the test assembly process. We illustrate how to include SEA into statistical and psychometric practice with data from several administrations of the SAT®‐Math.
SUPPLEMENTING A TRADITIONAL MATH CURRICULUM WITH AN INQUIRY‐BASED PROGRAM: A PILOT OF MATH OUT OF THE BOX
Authors: Rock JoAnn L. , Courtney Rosalea , Handwerk Philip G. 
Published: August 2014

ABSTRACTThis study examined math achievement of elementary school students when Math Out of the Box (MTB), an inquiry‐based math program, was used to supplement curriculum. The sample consisted of 767 New Jersey students in the third, fourth, and fifth grades, with approximately one third using MTB. Math achievement was measured by an assessment developed by ETS and by New Jersey's standardized test of math proficiency (NJ ASK). On the ETS assessment, a small, statistically significant difference was found in each of the three grades between students who used MTB and those who did not. On NJ ASK, a small, statistically significant difference was found in the third grade only. While these findings are an important step in establishing the efficacy of MTB, selection bias may weaken the causal inferences drawn.
EVALUATION OF THE EFFECTS OF LOGLINEAR SMOOTHING MODELS ON EQUATING FUNCTIONS IN THE PRESENCE OF STRUCTURED DATA IRREGULARITIES
Authors: Liu Jinghua , Moses Tim , Low Albert 
Published: August 2014

ABSTRACTPresmoothing a score distribution involves finding a balance between a good representation of the original data and smoothness. Smoothing reduces sampling variability, while a good data representation reduces the possibility of bias. When the test score distributions contain systematic irregularities that are due to scoring practice (e.g., formula scoring) rather than random irregularities, this trade‐off becomes complicated. The current study explored whether the systematic irregularities should be preserved or smoothed and the corresponding impacts on score equating. Two properties of the equating function were evaluated: distribution matching and smoothness. The overall results did not exhibit a consistent pattern in terms of the impacts on the equating function caused by smoothing out or preserving the systematic irregularities.
SMALL‐SAMPLE EQUATING WITH PRIOR INFORMATION
Authors: Livingston Samuel A. , Lewis Charles 
Published: August 2014

ABSTRACTThis report proposes an empirical Bayes approach to the problem of equating scores on test forms taken by very small numbers of test takers. The equated score is estimated separately at each score point, making it unnecessary to model either the score distribution or the equating transformation. Prior information comes from equatings of other test forms, with an appropriate adjustment for possible differences in test length.
THE EFFECTS OF DIFFERENT TYPES OF ANCHOR TESTS ON OBSERVED SCORE EQUATING
Authors: Liu Jinghua , Sinharay Sandip , Holland Paul W. , Feigenbaum Miriam , Curley Edward 
Published: August 2014

ABSTRACTThis study explores the use of a different type of anchor, a midi anchor, that has a smaller spread of item difficulties than the tests to be equated, and then contrasts its use with the use of a mini anchor. The impact of different anchors on observed score equating were evaluated and compared with respect to systematic error (bias), random equating error (standard errors of equating), and total equating error (RMSE) using empirical data. The results show that the midi anchor generally produces more accurate equating results than the mini anchor—with a few exceptions. Our findings suggest that a midi anchor test would be preferred to a mini anchor test if equating accuracy at the top and at the bottom of the score scale is not a primary concern.
A SCALE DRIFT STUDY
Authors: Liu Jinghua , Curley Edward , Low Albert 
Published: August 2014

ABSTRACTThis study examines the stability of the SAT® scale from 1994 to 2001. A 1994 form and a 2001 form were readministered in a 2005 SAT administration, and the 1994 form was equated to the 2001 form. The new conversion was compared to the old conversion. Both the verbal and math sections exhibit a similar degree of scale drift, but in opposite directions: the verbal scale has drifted upward, whereas the math scale has drifted downward. We suggest testing programs monitor the score scales periodically by building a testing form schedule that allows a systematic and periodic checking of scale stability.