ethanporcaro/eric-fulltext-meta huggingface dataset:
The Effectiveness of Feedback for L1-English and L2-Writing Development: A Meta-Analysis. TOEFL iBT™ Research Report. TOEFL iBT-14. ETS RR-11-05
Author: Biber, Douglas
Date: 2011-02-00
Type: Journal Articles
Subjects: English (Second Language),Second Language Instruction,Feedback (Response),Writing Skills,Writing Instruction,Teaching Methods,Qualitative Research,Statistical Analysis,Native Speakers,Surveys,Meta Analysis,Coding,Peer Influence,Teacher Influence,Effect Size,Language Proficiency,Language Tests,Second Language Learning

This research project undertook a review and synthesis of previous research on the effectiveness of feedback for individual writing development. The work plan was divided into two main phases. First, we surveyed all available studies that have investigated the effectiveness of writing feedback, including both quantitative and qualitative research, for students who have learned English as a first language (L1-English), students who have learned English as a second language (L2-English), and students who have learned second languages other than English. The results of this survey are described in a narrative overview of previous research pertaining to the role of feedback in the development of writing proficiency. The survey also identified the major theoretical constructs used in this research domain, providing the basis for subsequent statistical analysis. Second, we built on this survey to carry out a meta-analysis of empirical studies in this research area. The goal of the meta-analysis was to provide a quantitative investigation of the extent and ways in which feedback has been effective, summarizing the findings of previous quantitative studies that have employed suitable statistical measures. Several analytical steps were required for the meta-analysis: developing a coding rubric; analyzing the research design and adequacy of reporting in studies to determine if they were suitable for inclusion; coding each study for all relevant research design factors; computing effect sizes for each study; and analyzing and interpreting the general patterns that hold across this set of studies. The meta-analysis compared the gains in writing development with respect to several different kinds of feedback. Overall, feedback was found to result in gains in writing development. Beyond that, there were several predictable findings (e.g., that written feedback is more effective than oral feedback for writing development) and several other more noteworthy trends (e.g., that peer feedback is more effective than teacher feedback for L2-English students; commenting is more effective than error location; and in general, focus on form and content seems to be more effective than an exclusive focus on form). [A list of studies used for the research synthesis is included.]
Evaluating Educational Programs. ETS R&amp;D Scientific and Policy Contributions Series. ETS SPC-11-01. ETS Research Report No. RR-11-15
Author: Ball, Samuel
Date: 2011-04-00
Type: Journal Articles
Subjects: Program Evaluation,Evaluation Research,Outcome Measures,Program Effectiveness,Field Studies,Data Analysis,Data Interpretation,Educational Objectives

Since its founding in 1947, ETS has conducted a significant and wide-ranging research program that has focused on, among other things, psychometric and statistical methodology; educational evaluation; performance assessment and scoring; large-scale assessment and evaluation; cognitive, developmental, personality, and social psychology; and education policy. This broad-based research program has helped build the science and practice of educational measurement, as well as inform policy debates. In 2010, we began to synthesize these scientific and policy contributions, with the intention to release a series of reports sequentially over the course of 2011 and 2012. This paper, written by Samuel Ball, was originally published in 1979. It is the only reissue in the series but is a fitting inaugural: The report documents the vigorous program of evaluation research conducted at ETS in the 1960s and 1970s, which helped lay the foundation for this fledgling field.
Mapping State Standards to the NAEP Scale. Research Report. ETS RR-08-57
Author: Braun, Henry
Date: 2008-11-00
Type: Journal Articles
Subjects: State Standards,Comparative Analysis,Public Schools,National Competency Tests,Error of Measurement,Sampling,Computation,Weighted Scores,Mathematics Achievement,Reading Achievement,Educational Legislation,Federal Legislation,Grade 4,Grade 8,Mathematics Tests,Reading Tests

This report describes the derivation and evaluation of a method for comparing the performance standards for public school students set by different states. It is based on an approach proposed by McLaughlin and associates, which constituted an innovative attempt to resolve the confusion and concern that occurs when very different proportions of students in various states are declared to have met a standard with the same label. Our method, like McLaughlin's, employs equipercentile methods to map state standards on to a common scale, that associated with the National Assessment of Educational Progress (NAEP). We have also derived error estimates that take into account both NAEP's complex sampling design and measurement errors. The method was applied to two data sets, and the results were qualitatively similar to those obtained by McLaughlin's method. The paper notes the superior statistical properties of the proposed method and presents evidence that supports the viability and general utility of this approach.
Continuous Exponential Families: An Equating Tool. Research Report. ETS RR-08-05
Author: Haberman, Shelby J.
Date: 2008-02-00
Type: Journal Articles
Subjects: Equated Scores,Statistical Distributions,Probability,Computation,Comparative Analysis,Sampling

Continuous exponential families may be employed to find continuous distributions with the same initial moments as the discrete distributions encountered in typical applications of classical equating. These continuous distributions provide distribution functions and quantile functions that may be employed in equating. To illustrate, an application is considered for a randomly equivalent groups design.
Comparing Multiple-Group Multinomial Log-Linear Models for Multidimensional Skill Distributions in the General Diagnostic Model. Research Report. ETS RR-08-35
Author: Xu, Xueli
Date: 2008-06-00
Type: Journal Articles
Subjects: Comparative Analysis,Models,Computation,National Competency Tests,Least Squares Statistics,Grade 4,Grade 8,Reading Tests

The general diagnostic model (GDM) utilizes located latent classes for modeling a multidimensional proficiency variable. In this paper, the GDM is extended by employing a log-linear model for multiple populations that assumes constraints on parameters across multiple groups. This constrained model is compared to log-linear models that assume separate sets of parameters to fit the distribution of latent variables in each group of a multiple-group model. Estimation of these constrained log-linear models using iterative weighted least squares (IWLS) methods is outlined and an application to NAEP data exemplifies the differences between constrained and unconstrained models in the presence of larger numbers of group-specific proficiency distributions. The use of log-linear models for the latent skill space distributions using constraints across populations allows for efficient computations in models that include many proficiency distributions.
Development of Approximations to Population Invariance Indices. Research Report. ETS RR-08-36
Author: Liu, Jinghua
Date: 2008-00-00
Type: Journal Articles
Subjects: Raw Scores,Test Format,Comparative Analysis,Test Construction,Critical Reading,Statistical Analysis,Equated Scores,College Entrance Examinations,Verbal Ability,Gender Differences,Sample Size,Racial Differences

The purpose of this paper is to explore methods to approximate population invariance without conducting multiple linkings for subpopulations. Under the single group or equivalent groups design, no linking needs to be performed for the parallel-linear system linking functions. The unequated raw score information can be used as an approximation. For other linking functions that are nonparallel-linear, linking only needs to be conducted for the total population. The difference of the standardized mean differences between each subpopulation and the total population across the old form and the new form can be used as an approximation of population invariance. Under the nonequivalent groups with anchor test design, conducting separate subpopulation linking and comparing them to the total population linking may still be the best way to estimate population invariance.
The Impact of Changes in the TOEFL® Examination on Teaching and Learning in Central and Eastern Europe: Phase 2, Coping with Change. TOEFL iBT Research Report. TOEFL iBT-05. ETS RR-08-37
Author: Wall, Dianne
Date: 2008-07-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Foreign Countries,Second Language Instruction,Longitudinal Studies,Test Construction,Testing Problems,Teacher Attitudes,Language Teachers,Knowledge Level,Course Content,Teaching Methods,Test Preparation,Grammar,Change,Writing Skills,Reading Skills,Listening Skills,Oral Language,Interpersonal Communication,Futures (of Society),Interviews,Observation,Computer Mediated Communication

The aim of this report is to present the findings of the second phase in a longitudinal study of the impact of changes in the TOEFL® test on teaching and learning in test preparation classrooms. The focus of this phase was to monitor six teachers from five countries in Central and Eastern Europe as they received news about changes in the TOEFL and began thinking about how these might affect their teaching in the future. Data were gathered during the period of January to May 2005. The teachers responded to monthly tracking questions and tasks that explored their awareness of the old and new TOEFL tests, the features of their test preparation classes, their reactions to the most innovative parts of the new test, and their thoughts about the type of content and activities they would offer once the new TOEFL was operational in their countries. The report includes an analysis of the teachers' awareness, attitudes, and plans, and a discussion of the types of factors that could affect the shape and intensity of TOEFL washback in years to come.
Asymptotic Limits of Item Parameters in Joint Maximum-Likelihood Estimation for the Rasch Model. Research Report. ETS RR-08-04
Author: Haberman, Shelby J.
Date: 2008-02-00
Type: Journal Articles
Subjects: Item Response Theory,Maximum Likelihood Statistics,Computation

Techniques are developed for approximation and exact computation of the asymptotic limit of the item parameter estimates obtained by application of joint maximum-likelihood estimation to the Rasch model.
Effect of Immediate Feedback and Revision on Psychometric Properties of Open-Ended Sentence- Completion Items. ETS GRE Board Research Report No. 03-15. ETS RR-08-16
Author: Attali, Yigal
Date: 2008-04-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Sentences,Psychometrics,Revision (Written Composition),Computer Assisted Testing,Statistical Analysis,Correlation,Scores,Validity,Reliability,Reaction Time,Surveys,Feedback (Response)

Registered examinees for the GRE® General Test answered open-ended sentence-completion items. For half of the items, participants received immediate feedback on the correctness of their answers and up to two opportunities to revise their answers. A significant feedback-and-revision effect was found. Participants were able to correct many of their initial incorrect answers, resulting in higher revised scores. In addition, the reliability of the revised scores and their correlation with GRE verbal scores were higher. The possibility of using revision scores as a basis for measuring potential future learning is discussed.
Examining an Alternative to Score Equating: A Randomly Equivalent Forms Approach. Research Report. ETS RR-08-14
Author: Liao, Chi-Wen
Date: 2008-04-00
Type: Journal Articles
Subjects: Equated Scores,Item Analysis,Test Items,Difficulty Level,Prediction,Accuracy,Comparative Analysis,Raw Scores,English (Second Language),Reading Tests,Listening Comprehension Tests,Language Tests,Statistical Analysis,Test Format,Second Language Learning

Randomly equivalent forms (REF) of tests in listening and reading for nonnative speakers of English were created by stratified random assignment of items to forms, stratifying on item content and predicted difficulty. The study included 50 replications of the procedure for each test. Each replication generated 2 REFs. The equivalence of those 2 forms was evaluated by comparing the raw-score distributions focusing on the greatest difference in the cumulative distributions. For listening, 10 replications produced cumulative distributions that differed at some point by more than 0.10, and 4 replications produced differences greater than 0.15. For reading, only 3 replications produced differences greater than 0.10. The difference between the results for listening and reading reflects the greater variation, within strata, in the difficulty of the listening items. The REF procedure may become more effective if item difficulty can be predicted more accurately.
Evidence-Centered Assessment Design for Reasoning about Accommodations for Individuals with Disabilities in NAEP Reading and Mathematics. Research Report. ETS RR-08-38
Author: Hansen, Eric G.
Date: 2008-07-00
Type: Journal Articles
Subjects: National Competency Tests,Disabilities,Reading Instruction,Mathematics Instruction,Measurement,Validity,Access to Education,Evidence,Reading Comprehension,Reading Aloud to Others,Testing Accommodations,Job Skills,Ability,Knowledge Level,Bayesian Statistics,Individualized Education Programs

Accommodations play a key role in enabling individuals with disabilities to participate in the National Assessment of Educational Progress (NAEP) and other large-scale assessments. However, it can be difficult to know how accommodations affect the validity of results, thus making it difficult to determine which accommodations should be allowed. This study describes recent extension of evidence-centered assessment design (ECD) for reasoning about the impact of accommodations and other accessibility features (e.g., universal design features) on the validity of assessment results, using examples from NAEP reading and mathematics. The study found that the ECD-based techniques were useful in analyzing the effects of accommodations and other accessibility features on validity. Such design capabilities may increase assessment designers' capacity to employ accessibility features without undermining validity.
Small-Sample Equating by the Circle-Arc Method. Research Report. ETS RR-08-39
Author: Livingston, Samuel A.
Date: 2008-07-00
Type: Journal Articles
Subjects: Sample Size,Equated Scores,Scores,Models,Mathematical Formulas

This paper suggests two new, related methods for estimating a test-score equating relationship from small samples of test takers. These methods do not require the estimated equating transformation to be linear. Instead, they constrain the estimated equating curve to pass through 2 prespecified end-points and a middle point determined from the data. Some preliminary results indicate that these methods outperform mean equating and other methods used for equating in small samples.
Approaches to the Design of Diagnostic Item Models. Research Report. ETS RR-08-07
Author: Graf, Edith Aurora
Date: 2008-02-00
Type: Journal Articles
Subjects: Test Items,Models,Diagnostic Tests,Statistical Analysis,Automation,Test Construction,Multiple Choice Tests,Test Format,Responses,Test Validity

Quantitative item models are item structures that may be expressed in terms of mathematical variables and constraints. An item model may be developed as a computer program from which large numbers of items are automatically generated. Item models can be used to produce large numbers of items for use in traditional, large-scale assessments. But they have potential for use in other areas as well, including diagnostic assessment. In this report, I first review research on diagnostic assessment and then discuss how approaches to diagnostic assessment can inform the design of diagnostic item models.
Evaluating the Effectiveness of a Full-Population Estimation Method. Research Report. ETS RR-08-18
Author: Braun, Henry
Date: 2008-04-00
Type: Journal Articles
Subjects: Research Methodology,Computation,Disabilities,English Language Learners,National Competency Tests,Scores,Evaluation Methods,Test Bias,Sampling,Grade 4,Reading Achievement,Statistical Analysis,Comorbidity,Geographic Location

At present, although the percentages of students with disabilities (SDs) and/or students who are English language learners (ELL) excluded from a NAEP administration are reported, no statistical adjustment is made for these excluded students in the calculation of NAEP results. However, the exclusion rates for both SD and ELL students vary substantially across jurisdictions at a given administration, and, in some cases, have changed substantially over time within a jurisdiction. Consequently, comparisons of performance based on reported NAEP scores may indeed be biased by differential exclusion and identification practices. Using only NAEP data, this report investigates plausible explanations for the observed heterogeneity among jurisdictions in exclusion rates. It also examines the operating characteristics of a particular class of methods that carry out statistical adjustments to NAEP's reported scores to address the possible bias due to differential exclusion rates. The final results of such adjustments are termed full-population estimates (FPEs). The conclusions are that there is both a strong likelihood of bias and that neither the current NAEP procedure nor the FPE methodologies constitutes an ideal solution. The former because it assumes that all excluded students could not meaningfully participate in NAEP, and the latter because they implicitly assume that all students could obtain a proper NAEP score.
&quot;L&quot;-Bivariate and &quot;L&quot;-Multivariate Association Coefficients. Research Report. ETS RR-08-40
Author: Kong, Nan
Date: 2008-07-00
Type: Journal Articles
Subjects: Correlation,Information Theory,Mathematical Logic,Validity,Multivariate Analysis

Given a system of multiple random variables, a new measure called the &quot;L&quot;-multivariate association coefficient is defined using (conditional) entropy. Unlike traditional correlation measures, the L-multivariate association coefficient measures the multiassociations or multirelations among the multiple variables in the given system; that is, the &quot;L&quot;-multivariate association coefficient measures the degree of the association for the given system. The &quot;L&quot;-multivariate association coefficient for the system of two random variables is also called the &quot;L&quot;-bivariate association coefficient. The association measured by the &quot;L&quot;-multivariate association coefficient is a general type of association, not any specific type of a linear or nonlinear association. Unlike the K-dependence coefficient, which is an asymmetrical measure, the &quot;L&quot;-multivariate association coefficient is a symmetrical measure. A direct application of the &quot;L&quot;-multivariate association coefficient is in variables selection or variables reduction. This paper also explores the relationship between the &quot;L&quot;-multivariate association coefficient and the &quot;K&quot;-dependence coefficient.
A Developmental Writing Scale. Research Report. ETS RR-08-19
Author: Attali, Yigal
Date: 2008-04-00
Type: Journal Articles
Subjects: Grade 4,Grade 6,Grade 8,Grade 10,Grade 12,Descriptive Writing,Persuasive Discourse,Writing Assignments,Writing Skills,Grammar,Language Usage,Vocabulary,Scoring,Automation,Essays,Longitudinal Studies,Student Characteristics,Factor Analysis,Prompting,Questionnaires,Institutional Characteristics

This report describes the development of grade norms for timed-writing performance in two modes of writing: persuasive and descriptive. These norms are based on objective and automatically computed measures of writing quality in grammar, usage, mechanics, style, vocabulary, organization, and development. These measures are also used in the automated essay scoring system &quot;e-rater&quot;® V.2. Norms were developed through a large-scale data collection effort that involved a national sample of 170 schools, more than 500 classes from 4th, 6th, 8th, 10th, and 12th grades and more than 12,000 students. Personal and school background information was also collected. These students wrote (in 30-minute sessions) up to 4 essays (2 in each mode of writing) on topics selected from a pool of 20 topics. The data allowed us to explore a range of questions about the development and nature of writing proficiency. Specifically, this paper provides a description of the trajectory of development in writing performance from 4th grade to 12th grade. The validity of a single developmental writing scale is examined through a human scoring experiment and a longitudinal study. The validity of the single scale is further explored through a factor analysis (exploratory and confirmatory) of the internal structure of writing performance and changes in this structure from 4th grade to 12th grade. The paper also explores important factors affecting performance, including prompt difficulty, writing mode, and student background (gender, ethnicity, and English language background).
Linking English-Language Test Scores onto the Common European Framework of Reference: An Application of Standard-Setting Methodology. TOEFL iBT Research Report TOEFL iBt-06. ETS RR-08-34
Author: Tannenbaum, Richard J.
Date: 2008-06-00
Type: Journal Articles
Subjects: Language Tests,Second Language Learning,English (Second Language),Computer Assisted Testing,Foreign Countries,Language Proficiency,Language Skills,Standard Setting,Scores,Specialists,Cutting Scores,Reading Skills,Writing Skills,Listening Skills,Oral Language,Guidelines

The Common European Framework of Reference (CEFR) describes language proficiency in reading, writing, speaking, and listening on a 6-level scale. In this study, English-language experts from across Europe linked CEFR levels to scores on three tests: the TOEFL® iBT test, the TOEIC® assessment, and the TOEIC &quot;Bridge&quot;™ test. Standard-setting methodology (a modified Angoff approach and a modified examinee paper selection approach) was used to construct the linkages. Linkages were established for TOEFL iBT at levels B1, B2, and C1. Linkages were established for TOEIC at levels A1 through C1, with the exception of Reading at the C1 level. The TOEIC &quot;Bridge&quot; test was linked to its three targeted levels of the CEFR. The report details the methods, procedures, and results of the study.
Outliers in Assessments. Research Report. ETS RR-08-41
Author: Haberman, Shelby J.
Date: 2008-07-00
Type: Journal Articles
Subjects: Educational Assessment,Quality Assurance,Scores,Regression (Statistics),Tests,Correlation,Probability,Statistical Analysis

Outliers in assessments are often treated as a nuisance for data analysis; however, they can also assist in quality assurance. Their frequency can suggest problems with form codes, scanning accuracy, ability of examinees to enter responses as they intend, or exposure of items.