ethanporcaro/eric-fulltext-meta huggingface dataset:
Reading for Understanding: How Performance Moderators and Scenarios Impact Assessment Design. Research Report. ETS RR-13-31
Author: O'Reilly, Tenaha
Date: 2013-12-00
Type: Journal Articles
Subjects: Reading Comprehension,Reading Tests,Test Construction,Student Characteristics,Performance Factors,Scores,Test Validity,Test Interpretation,Reading Instruction,Test Format,Vignettes,Prior Learning,Metacognition,Reading Strategies,Reading Motivation,Learning Strategies

This paper represents the third installment of the Reading for Understanding (RfU) assessment framework. This paper builds upon the two prior installments (Sabatini &amp; O'Reilly, 2013; Sabatini, O'Reilly, &amp; Deane, 2013) by discussing the role of performance moderators in the test design and how scenario-based assessment can be used as a tool for assessment delivery. Performance moderators are characteristics of students that impact reading performance but are not considered a part of the reading construct. These include (a) background and prior knowledge, (b) metacognitive and self-regulatory strategies and behavior, (c) reading strategies, and (d) student motivation and engagement. In this paper, we argue there is added value in incorporating performance moderators into a reading test design. We characterize added value with respect to the validity of the claims derived from test scores, the interpretation of the test scores, and the relevance to instruction. As a second aim, we present a case for using scenario-based assessments and how they can be used to integrate into the test design both the performance moderators as well as other features that make the assessment more instructionally relevant.
Understanding Students' Experiences of Well-Being in Learning Environments
Author: Stanton, Alisa
Date: 2016-00-00
Type: Journal Articles
Subjects: Student Attitudes,Well Being,Semi Structured Interviews,Focus Groups,Health Promotion,Universities,Learning Experience,Institutional Advancement,Foreign Countries,Qualitative Research,School Holding Power

With the recent release of a new international charter on health promoting universities and institutions of higher education, universities and colleges are increasingly interested in providing learning experiences that enhance and support student well-being. Despite the recognition of learning environments as a potential setting for creating and enhancing well-being, limited research has explored students' own perceptions of well-being in learning environments. This article provides a qualitative exploration of students' lived experiences of well-being in learning environments within a Canadian post-secondary context. A semi-structured focus group and interview protocol was used to explore students' own definitions and experiences of well-being in learning environments. The findings illuminate several pathways through which learning experiences contribute to student well-being, and offer insight into how courses may be designed and delivered in ways that enhance student well-being, learning and engagement. The findings also explore the interconnected nature of well-being, satisfaction and deep learning. The relevance for the design and delivery of higher education learning experiences are discussed, and the significance of the findings for university advancement decisions are considered.
Identifying Speech Acts in E-Mails: Toward Automated Scoring of the &quot;TOEIC&quot;® E-Mail Task. Research Report. ETS RR-12-16
Author: De Felice, Rachele
Date: 2012-09-00
Type: Journal Articles
Subjects: Speech Acts,Electronic Mail,Language Tests,Second Language Learning,Scoring Rubrics,Computational Linguistics,Accuracy,English,Native Speakers,Comparative Analysis,English (Second Language),Error Analysis (Language),Language Usage,Form Classes (Languages)

This study proposes an approach to automatically score the &quot;TOEIC&quot;® Writing e-mail task. We focus on one component of the scoring rubric, which notes whether the test-takers have used particular speech acts such as requests, orders, or commitments. We developed a computational model for automated speech act identification and tested it on a corpus of TOEIC responses, achieving up to 79.28% accuracy. This model represents a positive first step toward the development of a more comprehensive scoring model. We also created a corpus of speech actannotated native English workplace e-mails. Comparisons between these and the TOEIC data allow us to assess whether English learners are approximating native models and whether differences between native and non-native data can have negative consequences in the global workplace.
Investigating 10-Year Trends of Learning Outcomes at Community Colleges. Research Report. ETS RR-13-34
Author: Liu, Ou Lydia
Date: 2013-12-00
Type: Journal Articles
Subjects: Community Colleges,Two Year College Students,Trend Analysis,Reading Achievement,Writing Achievement,Mathematics Achievement,Critical Thinking,Liberal Arts,Thinking Skills,Minority Groups,English (Second Language),Achievement Gap,Ethnic Groups,Racial Differences,Predictor Variables,Comparative Analysis,Gender Differences,Grade Point Average,Institutional Characteristics,White Students,Hispanic American Students,African American Students,Asian American Students,Pacific Islanders,American Indian Students,Alaska Natives,Regression (Statistics)

Community colleges currently enroll about 44% of the undergraduate students in the United States and are rapidly expanding. It is of critical importance to obtain direct evidence of student learning to see if students receive adequate training at community colleges. This study investigated the 10-year trends of community college students' (n = 46,403) performance in reading, writing, mathematics, and critical thinking, as assessed by the ETS[TM] Proficiency Profile (EPP), an assessment of college-level learning outcomes. Results showed that community college students caught up with and significantly outperformed students from liberal arts colleges by the end of the 10-year period and made significant improvement in critical-thinking skills. An increasing gender gap was observed in mathematics at community colleges. Prevalent ethnic minority and English as a second language (ESL) gaps were noted but gaps between ESL and non-ESL students and between Hispanic and White students were decreasing. Additionally, Asian students at community colleges showed an overall decline in performance. Findings from this study provide significant implications for community college leaders, researchers, and policymakers.
Dimensionality Analysis of &quot;CBAL&quot;™ Writing Tests. Research Report. ETS RR-13-10
Author: Fu, Jianbin
Date: 2013-05-00
Type: Journal Articles
Subjects: Cognitive Measurement,Cognitive Ability,Scoring,Grade 4,Grade 8,Writing Tests,Writing Skills,Essays,Persuasive Discourse,Letters (Correspondence),Research Reports,Factor Analysis,Correlation,Accountability

The Cognitively Based Assessment of, for, and as Learning (&quot;CBAL&quot;™) research initiative is aimed at developing an innovative approach to K-12 assessment based on cognitive competency models. Because the choice of scoring and equating approaches depends on test dimensionality, the dimensional structure of CBAL tests must be understood. The purpose of this study was to investigate the dimensionality of 4 CBAL Grade 8 writing tests. Each of the 4 tests focused on one of the following writing genres: persuasive letter/memorandum, research-based report (pamphlet), persuasive essay, and literature essay. Dimensionality was investigated using exploratory and confirmatory factor analyses. The results show multidimensionality and support subscore structures and bifactor task models for all 4 tests.
What Can Repeated Cross-Sectional Studies Tell Us about Student Growth? Research Report. ETS RR-12-17
Author: Almond, Russell G.
Date: 2012-10-00
Type: Journal Articles
Subjects: Educational Research,Case Studies,Research Methodology,Literature Reviews,Longitudinal Studies,National Competency Tests,Growth Models,Grade 4,Grade 8,Scaling

To answer questions about how students' proficiencies are changing over time, educational researchers are looking for data sources that span many years. Clearly, for answering questions about student growth, a longitudinal study--in which a single sample is followed over many years--is preferable to repeated cross-sectional samples--in which a separate sample is taken every year. Repeated cross-sectional studies, such as the National Assessment of Educational Progress (NAEP), however, are often readily available. Repeated cross-sectional studies conflate several sources of variability (differences in the initial status of individuals, individual differences in the growth curves, and individual-by-measurement-occasion differences) in ways that are not easily separated. Although repeated cross-sectional studies can provide information about the growth of the averages, the growth of the averages corresponds to the average of the growth curves only in very restricted circumstances. This paper reviews the literature on modeling growth with an eye to characterizing the limitations of repeated cross-sectional studies and understanding the sensitivity of the results to key decisions (particularly, choices of cut points). In most cases, repeated cross-sectional studies should be used to confirm and contextualize the results of more targeted longitudinal studies.
A Note on Explaining Away and Paradoxical Results in Multidimensional Item Response Theory. Research Report. ETS RR-12-13
Author: van Rijn, Peter W.
Date: 2012-08-00
Type: Journal Articles
Subjects: Item Response Theory,Educational Testing,Bayesian Statistics,Statistical Analysis,Logical Thinking,Philosophy

Hooker and colleagues addressed a paradoxical situation that can arise in the application of multidimensional item response theory (MIRT) models to educational test data. We demonstrate that this MIRT paradox is an instance of the explaining-away phenomenon in Bayesian networks, and we attempt to enhance the understanding of MIRT models by placing the paradox in a broader statistical modeling perspective.
Poststratification Equating Based on True Anchor Scores and Its Relationship to Levine Observed Score Equating. Research Report. ETS RR-13-11
Author: Chen, Haiwen
Date: 2013-05-00
Type: Journal Articles
Subjects: Equated Scores,Test Items,Methods

This paper presents a new equating method for the nonequivalent groups with anchor test design: poststratification equating based on true anchor scores. The linear version of this method is shown to be equivalent, under certain conditions, to Levine observed score equating, in the same way that the linear version of poststratification equating is equivalent to Tucker equating. Some issues related to this result are discussed.
Statistical Report of 2011 &quot;CBAL&quot;™ Multistate Administration of Reading and Writing Tests. Research Report. ETS RR-12-24
Author: Fu, Jianbin
Date: 2012-12-00
Type: Journal Articles
Subjects: Cognitive Ability,Grade 8,Writing Tests,Grade 7,Reading Tests,Correlation,Scoring,Test Items,Error Patterns,Test Bias,Comparative Analysis,Item Analysis,Summative Evaluation

In the Cognitively Based Assessment of, for, and as Learning (&quot;CBAL&quot;™) research initiative, innovative K-12 prototype tests based on cognitive competency models are developed. This report presents the statistical results of the 2 CBAL Grade 8 writing tests and 2 Grade 7 reading tests administered to students in 20 states in spring 2011. Specifically, classical item statistics including rater reliabilities for human-scored items, item &quot;p&quot;+ values, item-total correlations, item response times, item omit and not-reached rates, system error rates, differential item functioning, interscore correlations, and reliabilities of subscores and total scores are reported. In addition, &quot;t&quot;-tests, multiple comparisons, correlations, and mixed models are used to examine the factors influencing test scores, including test form, test order, and various background variables at the student, teacher, and school levels. The results show that these 4 tests performed reasonably well.
The Effects of Audio-Visual Recorded and Audio Recorded Listening Tasks on the Accuracy of Iranian EFL Learners' Oral Production
Author: Drood, Pooya
Date: 2016-00-00
Type: Journal Articles
Subjects: Foreign Countries,Audio Equipment,Audiovisual Instruction,English (Second Language),Second Language Instruction,Oral Language,Language Proficiency,Accuracy,Teaching Methods,Educational Technology,Video Technology,Experimental Groups,Control Groups,Instructional Effectiveness,Comparative Analysis,Adults,Statistical Analysis,Pretests Posttests

The ways in which task in classrooms has developed and proceeded have receive great attention in the field of language teaching and learning in the sense that they draw attention of learners to the competing features such as accuracy, fluency, and complexity. English audiovisual and audio recorded materials have been widely used by teachers and students, and have been the important resources of teaching and self-study. Nowadays, the environment we are living in is abundant with audio visual input and we as teachers ,thus, should be aware of the fact that environment can change students' behavior towards language and language learning. What effects do these materials have on English speaking ability? The objective of this study is to find out whether there is difference in Iranian EFL learners' accuracy in both audiovisual recorded (videos, movies, etc.) and audio recorded tasks. For this purpose, 40 students of intermediate level were chosen and then were randomly assigned into two experimental and control groups each of which was under different listening tasks. (Audio visual- and audio only). Data analysis showed that the group which was trained under AV listening tasks showed different effects on students' accuracy, compared to the other group positioned using audio recorded. Based on the results of this study, it is imperative that teachers consider the types of activities and methods that can have influence over language learners' speaking ability.
Examining Linguistic Characteristics of Paraphrase in Test-Taker Summaries. Research Report. ETS RR-12-18
Author: Burstein, Jill
Date: 2012-10-00
Type: Journal Articles
Subjects: Essay Tests,Scores,Native Speakers,English (Second Language),Writing Processes,Language Tests,Second Language Learning,Language Usage,Differences,Documentation,Interrater Reliability

This annotation study is designed to help us gain an increased understanding of paraphrase strategies used by native and nonnative English speakers and how these strategies might affect test takers' essay scores. Toward that end, this study aims to examine and analyze the paraphrase and the types of linguistic modifications used in paraphrase in test-taker responses and differences that may exist between native and nonnative English speakers. We are also interested in how these factors might influence final essay score. Outcomes discussed in this report can be used to inform the development of new &quot;e-rater&quot;® scoring engine features that capture information related to paraphrase, specifically in nonnative speaker responses to the &quot;TOEFL&quot;® exam integrated writing task.
Mitigating Transitional Challenges of Chinese Students in U.S. Higher Education
Author: Liu, Dongfang
Date: 2016-00-00
Type: Journal Articles
Subjects: Foreign Countries,Higher Education,College Students,Foreign Students,Phenomenology,Student Attitudes,Semi Structured Interviews,Study Abroad,Educational Environment,Transitional Programs,Student Adjustment

The number of Chinese international students enrolled in U.S. higher education has significantly grown over the past two decades. In 2015, Chinese international students accounted for the largest group of international students from any one single country. Previous research acknowledges Chinese students encountering significant difficulties in U.S. education institutions. However, research specifically targeting the Chinese demographic within U.S. higher education institutions has not been extensively explored toward mitigation. This study tracks Chinese international students' transition to the United States, while primarily focusing on student-perspective of their preparation in China, acclimation experience in the U.S., and response to environmental change.
High School Students' Topic Preferences and Oral Development in an English-Only Short-Term Intensive Language Program
Author: Hsieh, Hui-Chen
Date: 2016-00-00
Type: Journal Articles
Subjects: English (Second Language),Second Language Learning,Second Language Instruction,Learning Motivation,High School Students,Speech Communication,Foreign Countries,Educational Policy,Pretests Posttests,Questionnaires,Student Attitudes,Intensive Language Courses,Grade 10,Oral Language,Preferences,Statistical Analysis

Developing the ability to speak English is a daunting task that has long been omitted in a test-driven pedagogy context (Chang, 2011; Li, 2012a, 2012b; Chen &amp; Tsai, 2012; Katchen, 1989, 1995). Since speaking is not tested for school admissions, most students are not motivated to learn it (Chang, 2011; Chen &amp; Tsai, 2012). Now, globalization makes English Lingua Franca; speaking English is definitely bound to be one key capability to connect oneself with the world (Graddol, 2007). Thus, teachers strive to help learners learn English by selecting appropriate and interesting topics to motivate them to learn more effectively (Dörnyei &amp; Csizér, 1998; Spratt, Pulverness &amp; Williams, 2011), especially in speaking. However, with only one internationally published research on Taiwanese college students' topics preference (Chen, 2012) and none on high school students, selecting appropriate topics seems challenging. Consequently, this study intended to investigate the potential topics that motivated learners to practice speaking and their oral performance. The results show that learners preferred topics related to their daily life and their speaking improved in terms of speech unit, clause unit, and words uttered.
Collaborative Problem Solving and the Assessment of Cognitive Skills: Psychometric Considerations. Research Report. ETS RR-13-41
Author: von Davier, Alina A.
Date: 2013-12-00
Type: Journal Articles
Subjects: Cooperative Learning,Problem Solving,Research,Psychometrics,Cognitive Ability,Teamwork,Individual Characteristics,Group Dynamics,Educational Environment,Statistical Analysis,Models,Data Collection,Evaluation Methods

Collaboration is generally recognized as a core competency of today's knowledge economy and has taken a central role in recent theoretical and technological developments in education research. Yet, the methodology for assessing the learning benefits of collaboration continues to rely on educational tests designed for isolated individuals. Thus, what &quot;counts&quot; as evidence of learning does not correspond to current best practices for teaching, and it does not reflect what students are ultimately expected to be able to do with their knowledge. The goals of this paper are to give an overview of the research conducted in several fields of work related to collaboration, propose &quot;a framework for the assessment of cognitive skills (such as science or math) through collaborative problem-solving tasks,&quot; and propose several statistical approaches to model the data collected from collaborative interactions. This research contributes to the knowledge needed to support a new generation of assessments based on collaboration.
Examining Evidence of Reliability, Validity, and Fairness for the &quot;SuccessNavigator&quot;™ Assessment. Research Report. ETS RR-13-12
Author: Markle, Ross
Date: 2013-07-00
Type: Journal Articles
Subjects: Self Evaluation (Individuals),Computer Assisted Testing,Test Reliability,Test Validity,Test Bias,College Bound Students,Psychometrics,Success,Factor Analysis,Test Anxiety

The &quot;SuccessNavigator&quot;™ assessment is an online, 30 minute self-assessment of psychosocial and study skills designed for students entering postsecondary education. In addition to providing feedback in areas such as classroom and study behaviors, commitment to educational goals, management of academic stress, and connection to social resources, it is also designed to predict a range of early academic outcomes. By indicating students' likely success, advisors, faculty, and staff can target their interactions with students to increase their likelihood of success. This report outlines evidence of reliability, validity, and fairness to demonstrate the appropriateness of SuccessNavigator for these purposes.
TOEFL11: A Corpus of Non-Native English. Research Report. ETS RR-13-24
Author: Blanchard, Daniel
Date: 2013-11-00
Type: Journal Articles
Subjects: Language Tests,Second Language Learning,English (Second Language),Writing Tests,Native Language,Identification,Error Patterns,Error Correction,Essay Tests,Computer Assisted Testing,Scoring,Natural Language Processing,Prompting,Scores,Taxonomy,Evaluators,Student Evaluation,Grammar

This report presents work on the development of a new corpus of non-native English writing. It will be useful for the task of native language identification, as well as grammatical error detection and correction, and automatic essay scoring. In this report, the corpus is described in detail.
Developing Analytic Rating Guides for &quot;TOEFL iBT&quot;® Integrated Speaking Tasks. &quot;TOEFL iBT&quot;® Research Report, TOEFL iBT-20. ETS Research Report. RR-13-13
Author: Jamieson, Joan
Date: 2013-07-00
Type: Journal Articles
Subjects: Oral Language,Language Proficiency,Scaling,Scores,Language Usage,Language Tests,Second Language Learning,English (Second Language),Scoring Rubrics,Computer Assisted Testing,Formative Evaluation,Evaluation Criteria,Standards,Statistical Analysis,Evaluators,Novices,Multiple Regression Analysis,Interrater Reliability,Pronunciation,Intonation,Mixed Methods Research,Chinese,Spanish,Native Language,Rating Scales,College Admission

Research and development of a new type of scoring rubric for the integrated speaking tasks of &quot;TOEFL iBT&quot;® are described. These &quot;analytic rating guides&quot; could be helpful if tasks modeled after those in TOEFL iBT were used for formative assessment, a purpose which is different from TOEFL iBT's primary use for admission decisions. Two questions motivated the project: What can be done to make the criteria and standards for good performance clear not only to English language teachers and their students but also to novice raters? How can test-takers be guided in the steps necessary to improve performance? Previous research, quantitative results of linguistic features present in spoken responses, and qualitative themes of raters' judgments were analyzed. Salient features associated with performance were then synthesized to develop three analytic rating guides, each using a series of yes/no questions. The rating guides expanded the current scoring scales for the three dimensions of delivery, language use, and topic development, so that key features of increasingly proficient speaking performance were described in more detail. Suggestions for future validation studies are provided in terms of an iterative process of trials and revisions.