ethanporcaro/eric-fulltext-meta huggingface dataset:
Conceptualizing Accessibility for English Language Proficiency Assessments. Research Report. ETS RR-16-07
Author: Guzman-Orth, Danielle
Date: 2016-00-00
Type: Journal Articles
Subjects: Language Tests,Language Proficiency,English Language Learners,English (Second Language),Testing Accommodations,Elementary Secondary Education,Computer Assisted Testing,Disabilities

This paper is the second in a series from Educational Testing Service (ETS) that conceptualizes next-generation English language proficiency (ELP) assessment systems for K-12 English learners (ELs) in the United States.The first paper articulated a high-level conceptualization of next-generation ELP assessment systems (Hauck, Wolf, &amp; Mislevy, 2016), the third paper addressed issues related to summative ELP assessments that emerged from the presentations and discussions at the English Language Proficiency Assessment Research working meeting (Wolf, Guzman-Orth, &amp; Hauck, 2016), and the fourth paper focused on a key concern within such systems--the initial identification and classification of ELs (Lopez, Pooler, &amp; Linquanti, 2016). The goal of this paper is to address accessibility issues in the context of ELP assessments and to discuss critical considerations to improve the accessibility of ELP assessments for ELs and ELs with disabilities. Although accessibility for ELs and ELs with disabilities who are taking content assessments is also important, a discussion about content assessments is beyond the scope of the paper at this time. In this paper, we discuss challenges and areas of possible directions to pursue for ongoing and future ELP assessment development, policy implications, and research considerations to improve the ELP testing experience for all users. [This paper is a collaborative effort between ETS and the National Center on Educational Outcomes (NCEO). A previous version of this report was published in October 2014.]
Building &quot;e-rater&quot;® Scoring Models Using Machine Learning Methods. Research Report. ETS RR-16-04
Author: Chen, Jing
Date: 2016-02-00
Type: Journal Articles
Subjects: Scoring,Models,Artificial Intelligence,Automation,Multiple Regression Analysis,Correlation,Essay Tests

The &quot;e-rater&quot;® automated scoring engine used at Educational Testing Service (ETS) scores the writing quality of essays. In the current practice, e-rater scores are generated via a multiple linear regression (MLR) model as a linear combination of various features evaluated for each essay and human scores as the outcome variable. This study evaluates alternative scoring models based on several additional machine learning algorithms, including support vector machines (SVM), random forests (RF), and &quot;k&quot;-nearest neighbor regression (k-NN). The results suggest that models based on the SVM algorithm outperform MLR models in predicting human scores. Specifically, SVM-based models yielded the highest agreement between human and e-rater scores. Furthermore, compared with MLR, SVM-based models improved the agreement between human and e-rater scores at the ends of the score scale. In addition, the high correlation between SVM-based e-rater scores with external measures such as examinee's scores on the other parts of the test provided some validity evidence for SVM-based e-rater scores. Future research is encouraged to explore the generalizability of these findings.
Creating a Next-Generation System of K-12 English Learner Language Proficiency Assessments. Research Report. ETS RR-16-06
Author: Hauck, Maurice Cogan
Date: 2016-00-00
Type: Journal Articles
Subjects: English Language Learners,Language Proficiency,Elementary Secondary Education,Student Evaluation,Evaluation Methods,Computer Assisted Testing,Disabilities,Summative Evaluation,Educational Change

This paper is the first in a series from Educational Testing Service (ETS) concerning English language proficiency (ELP) assessments for K-12 English learners (ELs). The goal of this paper, and the series, is to present research-based ideas, principles, and recommendations for consideration by those who are conceptualizing, developing, and implementing ELP assessments for K-12 ELs and by all stakeholders in their education and assessment. We also hope to contribute to the active current discussion in the field on improving the instruction and assessment of EL students as well as contribute to bringing these ideas into practice. This paper articulates a high-level vision for a next-generation assessment system serving K-12 English learners with a brief overview of the current state of the art for K-12 ELP assessments and the context of current reforms. In addition, the paper discusses the role of digital technology in next-generation K-12 ELP assessment systems, noting the distinct conceptual and practical advantages that digital technology offers in assessing ELP as well as principles for evaluating the potential challenges of implementing a digitally based assessment system against these benefits. A proposed theory of action for K-12 ELP assessment systems is also discussed in this paper. The second paper in the series addressed accessibility issues in the context of ELP assessments for ELs and ELs with disabilities (Guzman-Orth, Laitusis, Thurlow, &amp; Christensen, 2016), the third paper addressed issues related to summative ELP assessments that emerged from the presentations and discussions at the English Language Proficiency Assessment Research working meeting (Wolf, Guzman-Orth, &amp; Hauck, 2016), and the fourth paper focused on a key concern within such systems--the initial identification and classification of ELs (Lopez, Pooler, &amp; Linquanti, 2016).
A Study of the Use of the &quot;TOEFL iBT&quot;® Test Speaking and Listening Scores for International Teaching Assistant Screening. &quot;TOEFL iBT&quot;® Research Report. TOEFL iBT-27. ETS Research Report. RR-16-18
Author: Wagner, Elvis
Date: 2016-05-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Test Use,Scores,Teaching Assistants,Screening Tests,Speech Tests,Listening Comprehension Tests,Urban Universities,Correlation,Student Evaluation of Teacher Performance,Predictor Variables

Although the speaking section of the &quot;TOEFL iBT&quot;® test is used by many universities to determine if international teaching assistants (ITAs) have the oral proficiency necessary to be classroom instructors, relatively few studies have investigated the validity of using TOEFL iBT scores for ITA screening. The primary purpose of this study was to determine the effectiveness of using TOEFL iBT Speaking and Listening scores for ITAs in the context of an urban research university. This was accomplished by correlating the TOEFL iBT test scores with the scores obtained by prospective ITAs on local screening tests and student evaluations of ITAs' language and teaching competence and by using the TOEFL iBT test scores as predictors of ITA teaching performance as measured by student evaluations. Given that listening comprehension is an important aspect of instructional language competence, both TOEFL iBT Speaking and TOEFL iBT Listening scores were used as predictors. The development of ITAs' oral proficiency over a semester in an English-speaking environment was also examined by comparing their TOEFL iBT Speaking and Listening scores at the beginning and at the end of their first semester as ITAs. The results indicate that TOEFL iBT Speaking and Listening scores correlate moderately with the local screening exam, but more importantly, the results indicate that TOEFL iBT Listening might be a better predictor of teaching competence. Although TOEFL iBT Speaking scores did not correlate with or predict the various measures of teaching competence, TOEFL iBT Listening scores did. In addition, the study found that ITAs living and teaching in an English-speaking environment did make measurable gains in oral proficiency over the course of a 3-month period. Although these gains were relatively small, and the listening score gains were not statistically significant, this is the first study that has actually tried to measure the improvement in ITAs' oral proficiency while living and studying (and sometimes teaching) on campus.The importance of this study goes beyond the local context. This is the first study to examine the relationship between TOEFL iBT Speaking and Listening scores and instructional performance of ITAs, thus serving as a validation study of the use of the TOEFL iBT test for ITA placement purposes. The study also offers insights into ITAs' development of oral proficiency once they are in an English-speaking environment, which can serve as a basis for future ITA curriculum development.
Next-Generation Summative English Language Proficiency Assessments for English Learners: Priorities for Policy and Research. Research Report. ETS RR-16-08
Author: Wolf, Mikyung Kim
Date: 2016-00-00
Type: Journal Articles
Subjects: English (Second Language),Language Proficiency,English Language Learners,Summative Evaluation,Language Tests,Elementary Secondary Education

This paper is the third in a series concerning English language proficiency (ELP) assessments for K-12 English learners (ELs). The series, produced from Educational Testing Service (ETS), is intended to provide theory- and evidence-based principles and recommendations for improving next-generation ELP assessment systems, policies, and practices and to stimulate discussion on better serving K-12 EL students. The first paper articulated a high-level conceptualization of next-generation ELP assessment systems (Hauck, Wolf, &amp; Mislevy, 2016). The second paper addressed accessibility issues in the context of ELP assessments for ELs and ELs with disabilities (Guzman-Orth, Laitusis, Thurlow, &amp; Christensen, 2016). The fourth paper dealt with one of the major uses of ELP assessments--the initial identification and classification of ELs (Lopez, Pooler, &amp; Linquanti, 2016). The present paper focuses on summative ELP assessments that states use for accountability purposes. Based on the conversation that took place at the K-12 ELP Assessment Working Meeting on summative ELP assessments, we identify critical policy and research issues. We also highlight the challenges associated with measuring ELP in a diverse group of EL students and recommend ways to embrace and support EL students through shared goals and active communication across federal agencies, state agencies, local agencies, researchers, and practitioners.
Key Issues and Opportunities in the Initial Identification and Classification of English Learners. Research Report. ETS RR-16-09
Author: Lopez, Alexis A.
Date: 2016-00-00
Type: Journal Articles
Subjects: English Language Learners,Classification,Student Needs,Needs Assessment,Language Proficiency,Ability Identification,Elementary Secondary Education,Test Construction,Student Evaluation,Evaluation Methods,Cultural Differences,Student Diversity,Disabilities

English learners (ELs) require access to appropriate instructional services that match their strengths and needs to have an equal opportunity to achieve the same academic standards as other students. If ELs are not properly identified and classified, they may be excluded from services that would better help them meet high academic standards, and they may not receive optimal supports to advance both linguistically and academically. Thus, it is critical to accurately and properly identify newly enrolled students who are in need of supplemental instruction and support. The purpose of this report is to identify issues in current initial English-language proficiency assessment practices used to determine initial EL classification and offer recommendations to strengthen the policies, process, and tools used in the identification and classification of ELs.
Exploring the Relationships among Teacher Confidence, Learning, and Test Performance within the English-for-Teaching Course. Research Report. ETS RR-16-24
Author: Gu, Lin
Date: 2016-06-00
Type: Journal Articles
Subjects: English (Second Language),Second Language Instruction,Faculty Development,Teacher Competencies,Language Proficiency,Language Teachers,Self Esteem,Teacher Evaluation,Foreign Countries,Elementary Secondary Education,Teacher Surveys,Online Courses,Statistical Analysis

In responding to the increasingly expanded emphasis on English language teaching (ELT) around the globe, Educational Testing Service (ETS) collaborated with National Geographic Learning to develop the &quot;ELTeach&quot;™ program (http://www.elteach.com), an online professional development program consisting of two courses, English-for-Teaching and Professional Knowledge for ELT, with the goal of improving ELT teachers' classroom English language proficiency. Within the English-for-Teaching course, there are three interrelated components: a precourse planner that asks teachers to rate their confidence in using English to teach English, a learning course that teaches the type of English needed to carry out essential classroom activities in English, and the &quot;TEFT&quot;™ assessment, which is developed based on the same framework as the learning course. Using data collected during the pilot administration of the &quot;ELTeach&quot; program in 2012, this study examines the relationship between self-reported confidence level and engagement with the learning course, and the relationship between self-reported confidence level and performance on the &quot;TEFT&quot; assessment. The findings suggested how users with different confidence levels might make use of the learning materials and how they might score on the &quot;TEFT&quot; assessment.
Setting Language Proficiency Score Requirements for English-as-a-Second-Language Placement Decisions in Secondary Education. Research Report. ETS RR-16-17
Author: Baron, Patricia A.
Date: 2016-06-00
Type: Journal Articles
Subjects: Language Proficiency,Scores,Cutting Scores,Language Tests,Student Placement,Standard Setting,English (Second Language),Second Language Learning,Decision Making,Secondary School Students,Foreign Students,Admission (School),Admission Criteria,Private Schools,Reading Comprehension,Listening Comprehension,Speech Skills,Writing Skills,Feedback (Response),Surveys

The purpose of this study was to collect recommendations for minimum score requirements (cut scores) on the &quot;TOEFL Junior&quot;® English language proficiency test in order to guide decisions on the placement of learners into English as a second language (ESL) support classes. The TOEFL Junior test, intended primarily for students ages 11 and above, measures the academic and social English-language skills representative of English-medium instructional environments in secondary education. The study focused on three ESL placement decisions that affect international students applying for admission to English-medium independent or private secondary schools: (a) deny admission due to insufficient English-language skills, (b) admit conditionally with placement into ESL support, and (c) admit unconditionally (for students who have sufficient language skills at the time of admission to attend mainstream classes without ESL support). A combination of a modified Angoff standard-setting approach and a modified Performance Profile standard-setting approach was followed to identify the TOEFL Junior scores for schools to include in their decision-making process. A total of 16 language experts from private schools in 12 U.S. states served on the standard-setting panel. The results of this study provide policymakers with the minimum cut scores recommended by the panel for each section of the two testing modes: the TOEFL Junior Standard and the TOEFL Junior Comprehensive. Recommendations for the use of these scores and limitations of the results are discussed.
Vertical Articulation of Cut Scores across the Grades: Current Practices and Methodological Implications in the Light of the Next Generation of K-12 Assessments. Research Report. ETS RR-16-29
Author: Kannan, Priya
Date: 2016-08-00
Type: Journal Articles
Subjects: Evaluation Methods,Cutting Scores,Elementary Secondary Education,Student Evaluation,Psychometrics,Surveys,Incidence,Standard Setting,Accountability,Educational Legislation,Federal Legislation,Scoring,Statistical Analysis

Federal accountability requirements after the No Child Left Behind (NCLB) Act of 2001 and the need to report progress for various disaggregated subgroups of students meant that the methods used to set and articulate performance standards across the grades must be revisited. Several solutions that involve either &quot;a priori&quot; deliberations or &quot;post-hoc&quot; adjustments have been offered over the years. In this paper, I provide a methodological review of the alternative cut-score articulation methods, including some novel solutions (e.g., using predictive methods) that have been proposed in the context of the next-generation K-12 assessments. In systematically evaluating these methods, I focus on the psychometric challenges they might present and the practical feasibility of their operational implementation. In addition, results from a survey of several state departments of education help to provide information on the prevalence of these methods across the states. Overall, this review shows that none of the alternative methods is completely free of limitation; yet, each method provides solutions that are appropriate for addressing certain methodological and practical requirements.Therefore, in the context of the next-generation assessments and the need to identify students who are on track to being college and career ready, practitioners are advised to consider a combination of methods and cautioned against overreliance on any single method.
A Review of Evidence Presented in Support of Three Key Claims in the Validity Argument for the &quot;TextEvaluator&quot;® Text Analysis Tool. Research Report. ETS RR-16-12
Author: Sheehan, Kathleen M.
Date: 2016-06-00
Type: Journal Articles
Subjects: Automation,Evaluation Methods,Reading Material Selection,Common Core State Standards,Scores,Scoring,Standardized Tests,High Stakes Tests,Reading Skills,Difficulty Level,Language Usage,Concept Formation,Cognitive Processes,Test Reliability

The &quot;TextEvaluator&quot;® text analysis tool is a fully automated text complexity evaluation tool designed to help teachers and other educators select texts that are consistent with the text complexity guidelines specified in the Common Core State Standards (CCSS). This paper provides an overview of the TextEvaluator measurement approach and summarizes evidence related to three key claims in the TextEvaluator validity argument: (a) TextEvaluator has succeeded in expanding construct coverage beyond the two dimensions of text variation that are traditionally assessed by readability metrics; (b) the TextEvaluator strategy of estimating distinct prediction models for informational, literary, and mixed texts has succeeded in generating text complexity predictions that exhibit little, if any, genre bias; and (c) TextEvaluator scores are highly correlated with text complexity judgments provided by human experts, including judgments generated via the inheritance method and judgments generated via the exemplar method. Implications with respect to the goal of helping teachers and other educators select texts that are closely aligned with the accelerated text complexity exposure trajectory outlined in the CCSS are discussed.
Assessing Digital Information Literacy in Higher Education: A Review of Existing Frameworks and Assessments with Recommendations for Next-Generation Assessment. Research Report. ETS RR-16-32
Author: Sparks, Jesse R.
Date: 2016-12-00
Type: Journal Articles
Subjects: Technological Literacy,Information Literacy,Higher Education,Definitions,Evaluation Methods,Problem Solving,Ethics,Information Technology,Legal Responsibility,Multiple Choice Tests,Performance Based Assessment,Test Reliability,Test Validity,Literature Reviews

Digital information literacy (DIL)--generally defined as the ability to obtain, understand, evaluate, and use information in a variety of digital technology contexts--is a critically important skill deemed necessary for success in higher education as well as in the global networked economy. To determine whether college graduates possess the requisite knowledge and skills in DIL, higher education institutions must be able to administer and use results from valid assessments of DIL. In this paper, we provide a comprehensive review of existing definitions of this construct in major frameworks from higher education and the workforce and propose an operational definition of DIL. Next, we provide a review of existing assessments of information literacy and related constructs, including features of the assessments, construct alignment, and psychometric properties (i.e., reliability and validity evidence). Finally, we discuss challenges and considerations surrounding the design, implementation, and use of next-generation assessments of DIL. We offer this review as a resource for higher education institutions in selecting among existing assessments or in designing their own measures.
Agent-Based Modeling of Collaborative Problem Solving. Research Report. ETS RR-16-27
Author: Bergner, Yoav
Date: 2016-07-00
Type: Journal Articles
Subjects: Problem Solving,Cooperative Learning,Evaluation Methods,Models,Individual Characteristics,Psychometrics,Group Activities,Group Testing,Teamwork,Simulation

Collaborative problem solving (CPS) is a critical competency in a variety of contexts, including the workplace, school, and home. However, only recently have assessment and curriculum reformers begun to focus to a greater extent on the acquisition and development of CPS skill. One of the major challenges in psychometric modeling of CPS is collecting large-scale data on teams and processes. In this study, we explore the use of agent-based modeling (ABM) to model the CPS process, test the sensitivity of outcomes to different population characteristics, and generate simulated data that can provide a novel means by which to refine and develop psychometric models. Methods of adapting trait-based stochastic processes to a specific task are described, and preliminary results are presented.
Evaluating the Advisory Flags and Machine Scoring Difficulty in the &quot;e-rater&quot;® Automated Scoring Engine. Research Report. ETS RR-16-30
Author: Zhang, Mo
Date: 2016-08-00
Type: Journal Articles
Subjects: Essays,Scoring,Automation,Test Scoring Machines,Responses,Probability,Error Patterns

Successful detection of unusual responses is critical for using machine scoring in the assessment context. This study evaluated the utility of approaches to detecting unusual responses in automated essay scoring. Two research questions were pursued. One question concerned the performance of various prescreening advisory flags, and the other related to the degree of machine scoring difficulty and whether the size of the human-machine discrepancy could be predicted. The results suggested that some advisory flags operated more consistently across measures and tasks in detecting responses that the machine was likely to score differently from human raters than did other flags, and relatively little scoring difficulty was found for three of the four tasks examined in this study, with the relationship between machine and human scores being reasonably strong. Limitations and future studies are also discussed.