ethanporcaro/eric-fulltext-meta huggingface dataset:
Teaching Vocabulary to ESL Students
Author: McBain, Robert
Date: 2011-09-00
Type: Reports - Descriptive
Subjects: Vocabulary Development,English (Second Language),Teaching Methods,Metacognition,Skill Development,Reading Skills,Second Language Instruction

This paper looks at the ways in which vocabulary is taught and in particular what important learning points ESL students should know in order to understand new vocabulary words. It also discusses various ideas of how teachers could teach vocabulary. It highlights the importance of a theory that states there are 3 key stages that students progress through when they are learning new vocabulary. In addition it focuses on how this theory can be applied in the classroom when designing new materials, and on the types of exercises and learning experiences that ESL students could use to develop their meta-cognition and self-regulation skills as a view to long term strengthening of their vocabulary study to enhance reading and in-depth understanding. The article concludes with a suggestion on how vocabulary logs could be designed and organized from simple word associational exercises to comprehension exercises and finally through to generative style exercises.
Sources of Score Scale Inconsistency. Research Report. ETS RR-11-35
Author: Fife, James H.
Date: 2011-08-00
Type: Numerical/Quantitative Data
Subjects: Mathematics Tests,Responses,Test Construction,Construct Validity

Six tasks, selected from assessments administered in 2007 as part of the Cognitively-Based Assessments of, for, and as Learning (CBAL) project, were revised in an effort to remove difficulties with the tasks that were unrelated to the construct being assessed. Because the revised tasks were piloted on a different population from the original tasks, it was not possible to make direct comparisons between the performance of the revised tasks and that of the original tasks, other than to make a qualitative assessment of whether or not the nonconstruct difficulties had, in fact, been removed. But we were able to pilot between 2 and 4 versions of each revised task, and we could compare the performance of our pilot sample on the various versions of each task. For Mix It Up, we prepared 2 nonparallel versions--the first attempted to preserve the construct-related difficulty of the original while removing the nonconstruct-related ambiguities, and the second was intended to be an easier task that measured the same skills and abilities. For Fruit Drink and Paste we created 4 versions, carefully varying different aspects of the language while keeping other aspects constant. For the 2 tasks from Bigfoot, we varied 2 features independently, creating 2 versions of each feature and therefore 4 versions of each task. Finally, for Forest Carbon, we created 4 versions, varying from unscaffolded to carefully scaffolded. Because the revision of each task was its own experiment, the analysis of each task, and our conclusions from that analysis, are described separately. Appended are: (1) Revised Items; and (2) Codes Used in Scoring Tasks and Frequency of Response Data. (Contains 29 figures and 36 tables.)
Recommendations for Conducting Differential Item Functioning (DIF) Analyses for Students with Disabilities Based on Previous DIF Studies. Research Report. ETS RR-11-34
Author: Buzick, Heather
Date: 2011-08-00
Type: Reports - Research
Subjects: Test Bias,Students,Disabilities,Research Design,Evaluation Research

The purpose of this study is to help ensure that strategies for differential item functioning (DIF) detection for students with disabilities are appropriate and lead to meaningful results. We surveyed existing DIF studies for students with disabilities and describe them in terms of study design, statistical approach, sample characteristics, and DIF results. Based on descriptive and graphical summaries of previous DIF studies, we make recommendations for future studies of DIF for students with disabilities. Differential Item Functioning (DIF) Studies Surveyed are appended. (Contains 3 tables, 6 figures and 7 notes.)
Score Comparability for Language Minority Students on the Content Assessments Used by Two States. Research Report. ETS RR-11-27
Author: Young, John W.
Date: 2011-05-00
Type: Reports - Research
Subjects: Language Minorities,Second Language Learning,Grade 8,Minority Group Students,English (Second Language),Test Bias,Reliability,Scores,Educational Testing,Grade 4,Comparative Analysis,Comparative Testing,Mathematics Tests,High Stakes Tests,Test Validity,Educational Assessment,Factor Analysis,Standardized Tests

In this research investigation of score comparability for language minority students (English language learners [ELLs] and former English language learners), we examined 3 indicators of score comparability (reliability, internal test structure, and differential item functioning) for 4th and 8th grade students who took the NCLB-mandated content assessments in English-language arts and mathematics in 2 different U. S. states. Overall, for the 8 assessments we examined, a high degree of score comparability was found for ELLs and former ELLs, when compared with native English speakers. The results from this study showed that although the assessments from the 2 states differed somewhat with respect to the 3 indicators, a high degree of score comparability was found for both states' content assessments. (Contains 13 tables and 2 notes.)
Fit of Item Response Theory Models: A Survey of Data from Several Operational Tests. Research Report. ETS RR-11-29
Author: Sinharay, Sandip
Date: 2011-07-00
Type: Numerical/Quantitative Data
Subjects: Item Response Theory,Goodness of Fit,Statistical Analysis,Language Tests,Mathematics Tests,Reading Tests,Science Tests,Writing Tests,College Entrance Examinations

Standard 3.9 of the "Standards for Educational and Psychological Testing" (American Educational Research Association, American Psychological Association, & National Council for Measurement in Education, 1999) demands evidence of model fit when an item response theory (IRT) model is used to make inferences from a data set. We applied two recently suggested methods for assessing goodness of fit of IRT models--generalized residual analysis (Haberman, 2009) and residual analysis for assessing item fit (Bock & Haberman, 2009)--to several operational data sets. We assessed the practical significance of misfit whenever possible. This report summarizes our findings. Though evidence of misfit of the IRT model was found for all the data sets, the misfit was not always practically significant. (Contains 3 tables, 50 figures and 6 notes.)
Use of e-rater[R] in Scoring of the TOEFL iBT[R] Writing Test. Research Report. ETS RR-11-25
Author: Haberman, Shelby J.
Date: 2011-06-00
Type: Reports - Descriptive
Subjects: Writing Tests,Scoring,Essays,Language Tests,English (Second Language),Computer Assisted Testing,Predictor Variables,Correlation,Test Theory,Test Validity,Test Reliability,Evaluation Criteria,Accuracy,Interrater Reliability

Alternative approaches are discussed for use of e-rater[R] to score the TOEFL iBT[R] Writing test. These approaches involve alternate criteria. In the 1st approach, the predicted variable is the expected rater score of the examinee's 2 essays. In the 2nd approach, the predicted variable is the expected rater score of 2 essay responses by the examinee on a parallel form. This 2nd approach is related to prediction of the expected rater score of 2 essay responses on an actual form taken later by an examinee. The relationship of e-rater scores to scores on other sections of TOEFL[R] is also considered. These alternative approaches suggest somewhat different procedures for scoring. (Contains 5 tables.)
CBAL: Results from Piloting Innovative K-12 Assessments. Research Report. ETS RR-11-23
Author: Bennett, Randy Elliot
Date: 2011-06-00
Type: Reports - Research
Subjects: Educational Assessment,Elementary Secondary Education,Summative Evaluation,Formative Evaluation,Reading Tests,Writing Tests,Mathematics Tests,Innovation,Scoring,Test Bias,Psychometrics,Computer Assisted Testing

CBAL, an acronym for Cognitively Based Assessment of, for, and as Learning, is a research initiative intended to create a model for an innovative K-12 assessment system that provides summative information for policy makers, as well as formative information for classroom instructional purposes. This paper summarizes empirical results from 16 CBAL summative assessment pilots involving almost 10,000 online administrations conducted between 2007 and 2010. The results suggest that, on average, the CBAL initiative was successful in building innovative assessments in reading, writing, and mathematics that worked as intended. However, there was considerable variation in the functioning of test forms, especially in writing and math. That variation might suggest that the knowledge needed to produce high-quality forms of these innovative tests in a replicable and scalable manner is not yet at hand. Further, although the results described offer a significant start, many critical questions must be answered before CBAL assessments (or ones like them) are ready for high-stakes operational use. (Contains 8 tables, 6 figures and 12 notes.)
Smoothing and Equating Methods Applied to Different Types of Test Score Distributions and Evaluated with Respect to Multiple Equating Criteria. Research Report. ETS RR-11-20
Author: Moses, Tim
Date: 2011-04-00
Type: Reports - Research
Subjects: Equated Scores,Data Analysis,Scores,Methods,Error of Measurement,Comparative Analysis

In equating research and practice, equating functions that are smooth are typically assumed to be more accurate than equating functions with irregularities. This assumption presumes that population test score distributions are relatively smooth. In this study, two examples were used to reconsider common beliefs about smoothing and equating. The first example involves a relatively smooth population test score distribution and the second example involves a population test score distribution with systematic irregularities. Various smoothing and equating methods (&quot;presmoothing&quot;, &quot;equipercentile&quot;, &quot;kernel&quot;, and &quot;postsmoothing&quot;) were compared across the two examples with respect to how well the test score distributions were reflected in the equating functions, the smoothness of the equating functions, and the standard errors of equating. The smoothing and equating methods performed more similarly in the first example than in the second example. The results of the second example illustrate that when dealing with systematically irregular test score distributions, smoothing and equating methods can be used in different ways to satisfy different equating criteria. A Measure of Smoothness for Equating Functions is appended. (Contains 6 tables, 6 figures and 1 note.)
Automated Scoring within a Developmental, Cognitive Model of Writing Proficiency. Research Report. ETS RR-11-16
Author: Deane, Paul
Date: 2011-04-00
Type: Numerical/Quantitative Data
Subjects: Automation,Scoring,Educational Assessment,Writing Tests,Scoring Rubrics,Factor Analysis

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning. Appended are: (1) Feature Sources; (2) Additional Tables (e-rater Microfeatures); and (3) Existing and Proposed Scoring Rubrics for Cognitively Based Assessments of, for, and as Learning (CBAL) Writing Strands I and II (as Used in the Pilot and Proposed Revision). (Contains 48 tables, 1 figure and 30 notes.)
Writing Assessment and Cognition. Research Report. ETS RR-11-14
Author: Deane, Paul
Date: 2011-04-00
Type: Reports - Research
Subjects: Writing (Composition),Writing Evaluation,Writing Tests,Cognitive Ability,Social Influences,Writing Instruction,Social Theories,Writing Skills,Reading Skills,Correlation,Literacy Education,Evaluation Methods,Critical Thinking,Thinking Skills,Grade 8,Scoring

This paper presents a socio-cognitive framework for connecting writing pedagogy and writing assessment with modern social and cognitive theories of writing. It focuses on providing a general framework that highlights the connections between writing competency and other literacy skills; identifies key connections between literacy instruction, writing assessment, and activity and genre theories; and presents a specific proposal about how writing assessment can be organized to promote best practices in writing instruction. Reflective Strategies, Genres, and Writing Development are appended. (Contains 8 tables, 7 figures and 2 notes.)
Computer-Adaptive Testing for Students with Disabilities: A Review of the Literature. Research Report. ETS RR-11-32
Author: Stone, Elizabeth
Date: 2011-08-00
Type: Information Analyses
Subjects: Disabilities,Adaptive Testing,Accountability,Computer Assisted Testing,Literature Reviews,Evaluation Methods,Student Evaluation,Special Education,Item Response Theory,Student Needs,Test Items,Difficulty Level,Test Bias,Educational Technology

There has been an increased interest in developing computer-adaptive testing (CAT) and multistage assessments for K-12 accountability assessments. The move to adaptive testing has been met with some resistance by those in the field of special education who express concern about routing of students with divergent profiles (e.g., some students with math-based learning disabilities may have difficulty with basic computation but not high level problem solving) and poor performance on early test questions. This paper consists of a literature review focusing on adaptive testing issues for students with disabilities in the K-12 sector. While it is clear that there are issues that will present obstacles to administering accountability tests adaptively to students with disabilities, this synthesis of research and policy developments with respect to this topic will be useful both for development of research agendas and to inform states that are currently using or are considering moving to CAT.
Sources of Score Scale Inconsistency. Research Report. ETS RR-11-10
Author: Haberman, Shelby J.
Date: 2011-03-00
Type: Reports - Descriptive
Subjects: Scores,Reliability,Equated Scores,Test Construction,Sampling,Error of Measurement,Goodness of Fit

For testing programs that administer multiple forms within a year and across years, score equating is used to ensure that scores can be used interchangeably. In an ideal world, samples sizes are large and representative of populations that hardly change over time, and very reliable alternate test forms are built with nearly identical psychometric properties. Under these conditions, most equating methods produce score conversions close to the identity function. Unfortunately, equating is sometimes performed on small non-representative samples with variable distributions of ability, and administered tests are built to vague specifications. Here, different equating methods produce different results because they are based on different assumptions. In the nearly ideal case, there are smaller deviations from the identity function because great effort is taken to control variation. Even when equating is conducted under these desirable conditions, the random variation in form-to-form equating, when concatenated over time, can produce substantial shifts in score conversions, that is, scale drift. In this paper, we make distinctions among different sources of variation that may contribute to score-scale inconsistency, and identify practices that are likely to contribute to it. (Contains 1 table.)
The Single Group with Nearly Equivalent Tests (SiGNET) Design for Equating Very Small Volume Multiple-Choice Tests. Research Report. ETS RR-11-31
Author: Grant, Mary C.
Date: 2011-07-00
Type: Reports - Descriptive
Subjects: Multiple Choice Tests,Equated Scores,Test Items

The &quot;single group with nearly equivalent tests&quot; (SiGNET) design proposed here was developed to address the problem of equating scores on multiple-choice test forms with very small single-administration samples. In this design, the majority of items in each new test form consist of items from the previous form, and the new items that were administered as unscored items in the previous form. Each form is equated using data from examinees who took the previous form. As the equating is a single-group design, with the 2 forms having a large number of overlapping items, the size of the equating sample can be much smaller than in other designs. (Contains 5 figures.)
Measurement Error in Nonparametric Item Response Curve Estimation. Research Report. ETS RR-11-28
Author: Guo, Hongwen
Date: 2011-06-00
Type: Reports - Research
Subjects: Error of Measurement,Nonparametric Statistics,Item Response Theory,Computation,Comparative Analysis

Nonparametric, or kernel, estimation of item response curve (IRC) is a concern theoretically and operationally. Accuracy of this estimation, often used in item analysis in testing programs, is biased when the observed scores are used as the regressor because the observed scores are contaminated by measurement error. In this study, we investigate the deconvolution kernel estimation of IRC, correcting for the measurement error in the regressor variable. Using item response theory (IRT) simulated data and some real data, we compared the traditional kernel estimation and the deconvolution estimation of IRC. Results show that in capturing important features of the IRC, the traditional kernel estimation is comparable to the deconvolution kernel estimation in item analysis. Measurement Error is appended. (Contains 3 tables and 10 figures.)
The CBAL Reading Assessment: An Approach for Balancing Measurement and Learning Goals. Research Report. ETS RR-11-21
Author: Sheehan, Kathleen M.
Date: 2011-05-00
Type: Reports - Research
Subjects: Educational Assessment,Reading Tests,Formative Evaluation,Summative Evaluation,Middle School Students,Reading Skills,Test Reliability,Construct Validity

&quot;No Child Left Behind&quot; has highlighted the need for new types of assessments that not only provide high-quality evidence about what students know and can do, but also help to move learning forward. This paper describes a linked set of formative and summative reading assessments designed to address the tradeoffs inherent in these two goals. Targeted skills include the full range of competencies underlying proficient reading at the middle-school level, including both lower-order skills such as oral reading fluency and decoding, and higher-order skills such as the ability to integrate and synthesize information from multiple texts. Data collected in pilot administrations of two prototype test forms are presented. Analyses suggest that this new approach yields acceptable measurement properties while simultaneously addressing crucial learning outcomes. This paper was presented June 23, 2009 at the National Conference on Student Assessment, Los Angeles, CA. (Contains 4 figures, 7 tables and 2 notes.)
Use of Continuous Exponential Families to Link Forms via Anchor Tests. Research Report. ETS RR-11-11
Author: Haberman, Shelby J.
Date: 2011-04-00
Type: Reports - Research
Subjects: Equated Scores,Statistical Analysis,Language Tests,Mathematics Tests,Verbal Tests,Writing Tests

Continuous exponential families are applied to linking test forms via an internal anchor. This application combines work on continuous exponential families for single-group designs and work on continuous exponential families for equivalent-group designs. Results are compared to those for kernel and equipercentile equating in the case of chained equating. The conversions produced by all methods are quite similar. (Contains 5 tables and 8 figures.)
Examining American Post-Secondary Education. Research Report. ETS RR-11-22
Author: Liu, Ou Lydia
Date: 2011-05-00
Type: Reports - Research
Subjects: Higher Education,Testing Programs,Accountability,Strategic Planning,Research Projects,Postsecondary Education,Educational Research,College Outcomes Assessment,Enrollment,Educational Attainment,School Holding Power,Objectives,Research Needs,Nonprofit Organizations,Educational Policy,Graduation Rate,Minority Group Students,Dropout Rate,Academic Achievement,Instructional Effectiveness,Achievement Gap,Racial Differences,Gender Differences

The purpose of this report is to identify the most prominent issues in U.S. higher education and to develop strategic research plans to address the issues that are most relevant to ETS's capabilities in measurement and assessment through the ETS's higher education research initiative. In the United States, issues related to higher education such as improved performance and effective accountability have received unprecedented attention from stakeholders at many levels. At the national level, President Obama has set forth an ambitious agenda for American postsecondary education such that by 2020, the United States should once again have the largest concentration of citizens with a postsecondary degree. At the corporate level, ETS, as the world's largest educational research and  testing organization, is ready to move beyond testing program-based research in higher education and has the capability to deal with some of the most thorny issues in higher education. By strategically expanding post-secondary research, ETS will establish itself as a pioneer and thought leader in the field of higher education. The first part of the research report identifies four key issues existing in American higher education: "enrollment and performance", "retention and degree attainment", "student learning and experience", and "learning outcomes and accountability". The second part of the research report develops an ETS research agenda with short-term and long-term plans to address these issues. The agenda specifies short-term and long-term research goals that are specific, attainable, and measurable. Research findings from the studies proposed in this agenda have a potential for advancing understanding of the current situation and future needs of American higher education and also contributing to enhanced student learning at postsecondary institutions. Reaffirming and strengthening American higher education is critical to this country's success in the 21st century. (Contains 17 figures, 2 tables and 2 notes.)