ethanporcaro/eric-fulltext-meta huggingface dataset:
Factor Structure of the TOEFL Internet-Based Test across Subgroups. TOEFL iBT Research Report. TOEFL iBT-07. ETS Research Report. RR-08-66
Author: Stricker, Lawrence J.
Date: 2008-12-00
Type: Journal Articles
Subjects: Factor Structure,English (Second Language),Language Tests,Computer Assisted Testing,Indo European Languages,Language Classification,Language Usage,Scoring Formulas,Test Validity,Test Reliability,Factor Analysis,Learning Experience,Time Factors (Learning),Outcome Measures,Models,Second Language Learning

This study assessed the invariance in the factor structure of the &quot;Test of English as a Foreign Language&quot;™ Internet-based test (TOEFL® iBT) across subgroups of test takers who differed in native language and exposure to the English language. The subgroups were defined by (a) Indo-European and Non-Indo-European language family, (b) Kachru's classification of outer and expanding circles of countries (based on prevalence of English use in educational and business contexts), and (c) years of classroom instruction in the English language. The same factor structure (four first-order factors corresponding to the test sections and a single higher-order factor encompassing these factors) was identified in each subgroup. The results support the present scoring scheme for the TOEFL iBT assessment and suggest that the test functions the same way for diverse subgroups of test takers.
Effective and Scalable Teacher Professional Development: A Report of the Formative Research and Development. Research Report. ETS RR-08-65
Author: Wylie, E. Caroline
Date: 2008-12-00
Type: Journal Articles
Subjects: Qualitative Research,Communities of Practice,Faculty Development,Formative Evaluation,Mathematics Teachers,Secondary School Teachers,Summer Programs,Workshops,Algebra,Middle Schools,High Schools,Observation,Mathematics Instruction,Surveys,Interviews

This study is a qualitative analysis of data collected during a yearlong series of teacher learning community meetings and classroom observations. The participants are middle and high school mathematics teachers from 2 school districts. Teachers were introduced to the research behind formative assessment and how to apply that research to their teaching during a 3-day summer workshop. Teachers then met monthly in small school-based groups to deepen their understanding of formative assessment and to talk about their own classroom experiences. The analyses focused on how teachers' understanding changed over time, how the teacher learning communities supported the teachers, how learning translated into classroom practice, and the factors that supported or hindered the development of teachers' understanding and practice. Lessons learned during the study itself and from subsequent analyses of the data had a significant impact on the development of ETS's &quot;Keeping Learning on Track&quot;® program.
Model-Based Weighting and Comparisons: Research Report. ETS RR-08-17
Author: Qian, Jiahe
Date: 2008-04-00
Type: Journal Articles
Subjects: Surveys,Models,High Schools,School Effectiveness,Probability,Statistical Analysis,Weighted Scores,Longitudinal Studies,Grade 10,Grade 8,Monte Carlo Methods

In survey research, sometimes the formation of groupings, or aggregations of cases on which to make an inference, are of importance. Of particular interest are the situations where the cases aggregated carry useful information that has been transferred from a sample employed in a previous study. For example, a school to be included in the sample of the High School Effectiveness (HSES) study must contain one or more cases transferred from the National Educational Longitudinal Study of 1988 (NELS:88). To calculate the aggregation inclusion probabilities, this study investigated three statistical models and, based on these models, derived the school weights for the HSES study. This study also assessed the effects of weighting by comparing the statistics yielded from different sets of weights: (a) those from an empirical population database and (b) those from data generated from simulation based on the principles of a superpopulation. Both categorical data and continuous variables were analyzed in the comparison.
Studies of a Latent-Class Signal-Detection Model for Constructed-Response Scoring. Research Report. ETS RR-08-63
Author: DeCarlo, Lawrence T.
Date: 2008-12-00
Type: Journal Articles
Subjects: Scoring,Responses,Test Format,Bias,Classification,Models,Accuracy,Comparative Analysis,Simulation,Writing Tests

Rater behavior in essay grading can be viewed as a signal-detection task, in that raters attempt to discriminate between latent classes of essays, with the latent classes being defined by a scoring rubric. The present report examines basic aspects of an approach to constructed-response (CR) scoring via a latent-class signal-detection model. The model provides a psychological framework for CR scoring and includes rater parameters with a clear cognitive basis. Simulations are used to examine how well rater parameters and latent-class sizes are recovered as well as the accuracy of classification. The relation of rater parameters to agreement statistics and classification accuracy is examined. The effects of using a balanced, incomplete block design are compared to those for a fully crossed design. The model is applied to several ETS datasets.
Establishing the Validity of TOEIC Bridge™ Test Scores for Students in Colombia, Chile, and Ecuador. Research Report. ETS RR-08-58
Author: Sinharay, Sandip
Date: 2008-11-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Test Validity,Scores,Comparative Analysis,Self Evaluation (Individuals),Teacher Attitudes,Academic Achievement,Language Skills,Foreign Countries,Questionnaires,Correlation,Statistical Analysis,Reading Tests,Listening Comprehension Tests,Student Attitudes,Factor Analysis,Cross Cultural Studies,Elementary School Students,High School Students,Vocational Education,National Competency Tests

The validity of TOEIC Bridge™ scores as a measure of English language skill was examined from the standpoint of a unified concept of test validity. In this study, more than 6,000 test takers in 3 Latin American countries (Chile, Colombia, and Ecuador) took 1 form of the TOEIC Bridge test, and their scores were compared to additional information about the students (teacher judgments, self-assessments, and performance on other academic achievement measures). The evidence collected was generally quite consistent with the interpretation of TOEIC Bridge scores as indicators of the English language competencies for the students examined.
Notes on a General Framework for Observed Score Equating. Research Report. ETS RR-08-59
Author: Moses, Tim
Date: 2008-11-00
Type: Journal Articles
Subjects: Equated Scores,Error of Measurement,Statistical Analysis,Computation

The purpose of this paper is to extend von Davier, Holland, and Thayer's (2004b) framework of kernel equating so that it can incorporate raw data and traditional equipercentile equating methods. One result of this more general framework is that previous equating methodology research can be viewed more comprehensively. Another result is that the standard error of equated score difference (SEED) has a wider application than originally proposed. The methods described in this paper are empirically evaluated in an accompanying simulation study (Moses &amp; Holland, 2007).
Ensuring the Fairness of GRE Writing Prompts: Assessing Differential Difficulty. Research Report. ETS GRE Board Research Report No. 02-07R. ETS RR-05-11
Author: Broer, Markus
Date: 2005-07-00
Type: Journal Articles
Subjects: Culture Fair Tests,Item Response Theory,Test Items,Cues,Prompting,College Entrance Examinations,Graduate Study,Regression (Statistics),Gender Differences,Whites,African Americans,Asian Americans,English (Second Language),Second Language Learning,Writing Tests,Comparative Analysis,Disadvantaged,Scores,Difficulty Level

Three polytomous DIF detection techniques--the Mantel test, logistic regression, and polySTAND--were used to identify GRE® Analytical Writing prompts (&quot;Issue&quot; and &quot;Argument&quot;) that are differentially difficult for (a) female test takers; (b) African American, Asian, and Hispanic test takers; and (c) test takers whose strongest language skills are in a language other than English. Prompts in the gender comparison showed low differential item functioning (DIF) values. Higher DIF values were found in the White versus African American comparison (for Argument prompts), while a smaller number of prompts showed moderate DIF to the disadvantage of the Hispanic (for Argument prompts), Asian American (for Issue prompts), and English as a second language groups (for Issue prompts). No DIF values were found that were large enough to warrant the removal of prompts from the item pool. Several potential causes of high DIF values for some prompts are discussed with respect to the content characteristics of these prompts.
When Can Subscores Have Value? Research Report. ETS RR-05-08
Author: Haberman, Shelby J.
Date: 2005-05-00
Type: Journal Articles
Subjects: Scores,Test Items,Error of Measurement,Computation,Regression (Statistics),Reliability,Test Theory,College Entrance Examinations

In educational tests, subscores are often generated from a portion of the items in a larger test. Guidelines based on mean-squared error are proposed to indicate whether subscores are worth reporting. Alternatives considered are direct reports of subscores, estimates of subscores based on total score, combined estimates based on subscores and total scores, and residual analysis of subscore. Applications are made to data from two testing programs.
Interpretations of Reliability. Research Report. ETS RR-05-29
Author: Haberman, Shelby J.
Date: 2005-12-00
Type: Journal Articles
Subjects: Probability,Reliability,Intervals,Computation,Correlation,Cutting Scores

Some probabilistic illustrations of the reliability coefficient are provided to assist in interpretation of this measure. All explanations are derived under the assumption that the joint distribution of examinee scores from two parallel tests is well approximated by a bivariate normal distribution.
Bias Correction for the Maximum Likelihood Estimate of Ability. Research Report. ETS RR-05-15
Author: Zhang, Jinming
Date: 2005-09-00
Type: Journal Articles
Subjects: Statistical Bias,Maximum Likelihood Statistics,Computation,Ability,Test Items,Methods,Item Response Theory,Simulation,Comparative Analysis

Lord's bias function and the weighted likelihood estimation method are effective in reducing the bias of the maximum likelihood estimate of an examinee's ability under the assumption that the true item parameters are known. This paper presents simulation studies to determine the effectiveness of these two methods in reducing the bias when the item parameters are unknown. The simulation results show that Lord's bias function and the weighted likelihood estimation method might not be as effective in bias reduction in the 3PL cases when item parameters are unknown as they are when the true item parameters are given. Algorithms and methods for obtaining the global maximum value of a likelihood function or a weighted likelihood function are discussed in this paper.
Theoretical and Empirical Standard Errors for Two Population Invariance Measures in the Linear Equating Case. Research Report. ETS RR-08-24
Author: von Davier, Alina A.
Date: 2008-04-00
Type: Journal Articles
Subjects: Equated Scores,Error of Measurement,Statistical Analysis,Sampling,Licensing Examinations (Professions)

The standard errors of the 2 most widely used population-invariance measures of equating functions, root mean square difference (RMSD) and root expected mean square difference (REMSD), are not derived for common equating methods such as linear equating. Consequently, it is unknown how much noise is contained in these estimates. This paper describes 2 methods for obtaining the standard errors for RMSD and REMSD. The delta method relies on an analytical approximation and provides asymptotic standard errors. The grouped jackknife method is a sampling-based method. Both methods were applied to a real data application. The results showed that there was very little difference between the standard errors found by the 2 methods.
Mapping English Language Proficiency Test Scores onto the Common European Framework. TOEFL® Research Reports. RR-80. ETS RR-05-18
Author: Tannenbaum, Richard J.
Date: 2005-11-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Foreign Countries,Language Proficiency,Guidelines,Reading Skills,Writing Skills,Listening Skills,Oral Language,Scores,Standard Setting,Multiple Choice Tests,Benchmarking,Test Format,Specialists,Raw Scores

The Common European Framework describes language proficiency in reading, writing, speaking, and listening on a six-level scale. The Framework provides a common language with which to discuss students' progress. This report describes a study conducted with two panels of English language experts to map scores from four tests that collectively assess Reading, Writing, Speaking, and Listening on to two levels of the Framework. Panel 1 recommended cut scores for the Test of English as a Foreign Language™ (TOEFL®), the Test of Spoken English™ (TSE®), and the Test of Written English™ (TWE®). Panel 2 recommended cut scores for The Test of English for International Communication™ (TOEIC®). A modification of the Angoff (1971) standard-setting approach was used for multiple-choice questions, and a benchmark method (Faggen, 1994) or examinee paper selection method (Hambleton, Jaeger, Plake, &amp; Mills, 2000) was used for constructed-response questions.
Multilingual Generalization of the ModelCreator Software for Math Item Generation. Research Report. ETS RR-05-02
Author: Higgins, Derrick
Date: 2005-04-00
Type: Journal Articles
Subjects: Multilingualism,Computer Software,Japanese,Spanish,Test Items,Difficulty Level,Natural Language Processing,Educational Assessment,Item Analysis,Second Languages,Mathematics Tests,Word Problems (Mathematics),Verbs,Form Classes (Languages),Nouns,Grammar

This paper reports on the process of modifying the ModelCreator item generation system to produce output in multiple languages. In particular, Japanese and Spanish are now supported in addition to English. The addition of multilingual functionality was considerably facilitated by the general formulation of our natural language generation system, which proceeds in three stages, whereby the first two stages are largely language-independent. This multilingual capability is very promising for the domain of educational assessment because, when combined with &quot;item modeling&quot;, it provides the potential to produce a large pool of automatically constructed items, with difficulty estimates and balanced forms for speakers of different languages.
Psychometric and Cognitive Analysis as a Basis for the Design and Revision of Quantitative Item Models. Research Report. ETS RR-05-25
Author: Graf, Edith Aurora
Date: 2005-12-00
Type: Journal Articles
Subjects: Psychometrics,Accuracy,Item Analysis,High Stakes Tests,Statistical Analysis,Models,Test Items,Difficulty Level,Test Content,Specialists,Mathematics Tests,Item Response Theory,Test Validity,College Entrance Examinations,Graduate Study,Test Construction,Equations (Mathematics)

We describe the item modeling development and evaluation process as applied to a quantitative assessment with high-stakes outcomes. In addition to expediting the item-creation process, a model-based approach may reduce pretesting costs, if the difficulty and discrimination of model-generated items may be predicted to a predefined level of accuracy. The development and evaluation of item models represents a collaborative effort among content specialists, statisticians, and cognitive scientists. A cycle for developing and revising item models that generate items with more predictable statistics is described. We review the goals of item modeling from different perspectives and recommend a method for structuring families of models that span content and generate items with more predictable psychometric parameters. [This report combines two documents, the original versions of which were included in a set of symposium papers presented at the International Association for Education Assessment conference in June 2004.]
Identifiability of Parameters in Item Response Models with Unconstrained Ability Distributions. Research Report. ETS RR-05-24
Author: Haberman, Shelby J.
Date: 2005-12-00
Type: Journal Articles
Subjects: Item Response Theory,Ability,Models,Maximum Likelihood Statistics

If a parametric model for the ability distribution is not assumed, then the customary two-parameter and three-parameter logistic models for item response analysis present identifiability problems not encountered with the Rasch model. These problems impose substantial restrictions on possible models for ability distributions.
Automated Scoring of Spontaneous Speech Using SpeechRater? v1.0. Research Report. ETS RR-08-62
Author: Xi, Xiaoming
Date: 2008-11-00
Type: Journal Articles
Subjects: Speech,Scoring,Scoring Rubrics,Scoring Formulas,English Language Learners,Computer Assisted Testing,Program Validation,Test Validity,Computer System Design,Database Design,Database Management Systems,Second Language Learning,Testing Programs,Speech Tests,Evaluation Methods

This report presents the results of a research and development effort for SpeechRater? Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL®) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short- and long-term goals for iterative improvements to SpeechRater scoring.
Evaluating the Comparability of Paper-and-Pencil and Computerized Versions of a Large-Scale Certification Test. Research Report. ETS RR-05-21
Author: Puhan, Gautam
Date: 2005-10-00
Type: Journal Articles
Subjects: Comparative Analysis,Test Items,Statistical Analysis,Teacher Certification,Test Format,Item Analysis,Reading Skills,Mathematics Skills,Writing Skills,Item Response Theory,Scores,Computer Assisted Testing,Effect Size

The study evaluated the comparability of two versions of a teacher certification test: a paper-and-pencil test (PPT) and computer-based test (CBT). Standardized mean difference (SMD) and differential item functioning (DIF) analyses were used as measures of comparability at the test and item levels, respectively. Results indicated that effect sizes derived from the SMD were small (d &lt; 0.20) and not statistically significant (p &gt; 0.05), suggesting no substantial difference between the two test versions. Moreover, DIF analysis revealed that reading and mathematics items were comparable for both versions. However, five writing items were flagged for DIF. Substantive reviews failed to identify format differences that could explain the performance differences, so the causes of DIF could not be identified.
Tight but Loose: Scaling up Teacher Professional Development in Diverse Contexts. Research Report. ETS RR-08-29
Author: Wylie, E. Caroline, Ed.
Date: 2008-06-00
Type: Journal Articles
Subjects: Faculty Development,Program Development,Program Implementation,Fidelity,Change Strategies,Educational Change,Teacher Role,Educational Assessment,Beginning Teacher Induction,Scaling,School Districts,Urban Schools,Low Achievement,Pilot Projects

This series of papers was originally presented as a symposium at the annual meetings of the American Educational Research Association (AERA) and the National Council on Measurement in Education (NCME) held between April 9, 2007, and April 13, 2007, in Chicago, IL. The authors represent school districts and departments of education across the United States, as well as researchers at Cleveland State University, Educational Testing Service (ETS), the Institute for Education in London, and the University of Wyoming at Laramie. All of the current ETS staff, along with Dylan Wiliam and Marnie Thompson, worked at ETS for several years on an iterative research and development program, out of which grew the Keeping Learning on Track® (KLT) program. These papers represents the thinking about the theory behind the KLT program, describes the range of contexts used to implement the program, and illustrates the inherent tensions between the desire to maintain fidelity to a theory of action and the need to demonstrate flexibility in order to accommodate local situations. Papers 2 through 6 present descriptions of five implementations in chronological order. The papers included here are as follows: (1) Tight but Loose: A Conceptual Framework for Scaling Up School Reforms (Marnie Thompson and Dylan Wiliam); (2) A Teacher-Driven Implementation of Assessment &quot;for&quot; Learning in New Jersey (Laura Goe and Diane Mardy); (3) The &quot;Keeping Learning on Track&quot;® Program in New Teacher Induction (Jeff Maher and Dylan Wiliam); (4) Letting Go of the Reins: Learning About Scalability in the Context of One District-Wide Implementation (Christine J. Lyon, Donna Cleland, and Maureen Gannon); (5) &quot;Keeping Learning on Track&quot;® in an Urban District's Low Performing Schools (E. Caroline Wylie, Marnie Thompson, Christine J. Lyon, and Donna Snodgrass); (6) A State-Sponsored Pilot Project in Selected Schools in Vermont (Cynthia Tocci and Gayle Taylor); (7) Scaling Up Across Diverse Contexts: Lessons Learned From Five Implementations of the &quot;Keeping Learning on Trac&quot;k® Program (Dawn Marie Leusner, Judith Ellsworth, and Laura Goe); and (8) Tight but Loose: Through the Looking Glass (Margaret Heritage and W. James Popham).
Latent-Class Item Response Models. Research Report. ETS RR-05-28
Author: Haberman, Shelby J.
Date: 2005-12-00
Type: Journal Articles
Subjects: Item Response Theory,Computation,Probability,Structural Equation Models,Licensing Examinations (Professions)

Latent-class item response models with small numbers of latent classes are quite competitive in terms of model fit to corresponding item-response models, at least for one- and two-parameter logistic (1PL and 2PL) models. Provided that care is taken in terms of computational procedures and in terms of use of only limited numbers of latent classes, computations are relatively simple in the case of latent classes.
Effects of Preexamination Disclosure of Essay Prompts for the GRE® Analytical Writing Assessment. ETS GRE® Board Research Report No. 01-07R. ETS RR-05-01
Author: Powers, Donald E.
Date: 2005-02-00
Type: Journal Articles
Subjects: Essays,Prompting,Cues,Writing Tests,Test Validity,Test Preparation,College Entrance Examinations,Graduate Study,Memorization,Probability,Statistical Analysis

This study examined how the practice of prepublishing prompts used on the writing section of the Graduate Record Examinations® (GRE®) General Test impacts test-preparation behavior, test performance, test validity, and examinee perceptions of the value of prompt prepublication. Researchers imposed modest experimental control over how participants used the prompts to prepare for an upcoming test. The strategy test-takers reported using most frequently was simply to &quot;think generally about the potential topics.&quot; Slightly fewer than half of study participants wrote sample essays to prepare for the test, and very few (4%) admitted to memorizing essays that could be recalled during testing. Results provided no indication that participants benefited from encountering a prompt for which they had prepared. The vast majority of study participants, however, thought that making the GRE essay prompts available ahead of time is a good testing policy.