ethanporcaro/eric-fulltext-meta huggingface dataset:
Examining the Calibration Process for Raters of the &quot;GRE&quot;® General Test. ETS GRE® Board Research Report. GRE®-19-01. Research Report Series. ETS RR-19-09
Author: Wendler, Cathy
Date: 2019-12-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Accuracy,Test Reliability,Scoring,Interrater Reliability,Examiners,Quality Control,Essays,Writing Evaluation

One of the challenges in scoring constructed-response (CR) items and tasks is ensuring that rater drift does not occur during or across scoring windows. Rater drift reflects changes in how raters interpret and use established scoring criteria to assign essay scores. Calibration is a process used to help control rater drift and, as such, serves as a type of quality control during CR scoring. Calibration sets are designed to provide sufficient evidence that raters have understood and internalized the rubrics and can score accurately across all score points of the score scale. This study examined the calibration process used to qualify raters to score essays from the &quot;GRE&quot;® Analytical Writing measure. A total of 46 experienced raters participated in the study, and each rater scored up to 630 essays from 1 of 2 essay prompt types. Two research questions were evaluated: &quot;Does calibration influence scoring accuracy?&quot; and &quot;Does reducing the frequency of calibration impact scoring accuracy?&quot; While the distribution of score points represented by the essays used in the study did not necessarily reflect what raters see during operational scoring, results suggest that the influence of calibration on Day 1 remains with raters through at least 3 scoring days. Results further suggest that scoring accuracy may be moderated by prompt type. Nevertheless, study results indicate that daily calibration for GRE prompt types may not be necessary and that reducing the frequency of calibration is unlikely to reduce scoring accuracy.
Justifying the Construct Definition for a New Language Proficiency Assessment: The Redesigned &quot;TOEIC Bridge&quot;® Tests--Framework Paper. Research Report. ETS RR-19-30
Author: Schmidgall, Jonathan
Date: 2019-12-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Language Proficiency,Standards,Scores,Test Construction,Construct Validity,Daily Living Skills,Language Skills,Definitions,Adults,Language Usage,Models,Guidelines,Professional Associations,International Organizations

One of the most critical steps in the test development process is defining the construct, or the knowledge, skills, or abilities, to be assessed. This foundational step provides the basis for initial assumptions about the meaning of test scores and serves as a reference for subsequent validity research. In this paper, we describe the purpose of the redesigned &quot;TOEIC Bridge&quot;® 4 skills assessments and elaborate the theoretical basis of its construct definition. We also describe how an evidence-centered design (ECD) approach was used to develop the redesigned TOEIC Bridge assessments and the first stage of that approach, the domain analysis. The domain analysis begins by elaborating a clearer definition of the context in which language is evaluated by the redesigned TOEIC Bridge assessments, &quot;everyday adult life.&quot; Next, we review research literature and relevant language proficiency standards to highlight the knowledge, skills, and abilities relevant to beginner to low-intermediate English proficiency for everyday adult life. This information is synthesized in the construct definitions for reading, listening, speaking, and writing ability for beginner to low-intermediate levels of general English proficiency in the context of everyday adult life.
Bounded Rationality, Uncertainty, and Complexity as Decision-Making Contexts: A Case of One University in Russia
Author: Drugova, Elena A.
Date: 2019-00-00
Type: Journal Articles
Subjects: Decision Making,Universities,Power Structure,Risk,Case Studies,Graduate Students,Standards,Government Publications,Governance,Political Attitudes,Foreign Countries,Educational Legislation,School Policy,Administrator Attitudes,Stakeholders,College Administration

Decision making in universities cannot be implemented in an entirely rational mode because of the effects of the bounded rationality, uncertainty, and complexity inherent to the social and organisational contexts. An empirical case study related to launching a new postgraduate educational programme as the in-depth investigation of the chain of related decisions in one university in Russia was used to explain the features of the decision-making process. Data was gathered through 1) five in-depth interviews with the key stakeholders; 2) observations of the situation stages and interviews with other stakeholders; 3) analysis of governmental documents and standards; and 4) analyses of university policies and local documents. Case analysis showed the unclear authority of the institutions involved, vague rules, exclusion of the important stakeholders from the decision process, a lack of experience and expertise, and unclear procedures. The case was interpreted through the lenses of the organized anarchy theory; power and authority perspective; risk avoidance perspective; and bounded rationality perspective. As a discussion and conclusion, it was shown that some strategies can reduce the level of uncertainty and increase the quality of the decisions made, such as decision analysis perspective, decomposition perspective, participation perspective, information perspective, and groupthink avoidance perspective. Also, the role of political negotiations, information provision, additional actions and meetings organization, systematization of the complicated issues, and organizational learning were considered. All these support more mature solutions in the university context.
The Key Practice, Building and Sharing Stories and Social Understandings: The Intrinsic Value of Narrative. Research Report. ETS RR-19-31
Author: Deane, Paul
Date: 2019-12-00
Type: Journal Articles
Subjects: Language Arts,Story Telling,Reading,Writing (Composition),Personal Narratives,Autobiographies,Theory of Mind,Models,Language Skills,Interaction,Interpersonal Relationship,Executive Function,Social Development,Perspective Taking,Emotional Response,Personality Traits,Vocabulary Development,Moral Development,Reflection,Common Core State Standards

One of the major goals of the English Language Arts is to teach students to read, understand, and write narratives. This report examines the ways in which the skills that support narrative develop during the school years, outlines a model of narrative as a &quot;key practice&quot; in which the ability to model social situations supports narrative understanding, and feeds into the ability to use stories to reflect about stories and the classes of social situations they represent. Narrative is important precisely because it helps people develop their understanding of the social world and reason about their place in it. Assessments of narrative reading and writing need to take this broader construct into account.
Exploring the Potential of a Video-Mediated Interactive Speaking Assessment. Research Report. ETS RR-19-05
Author: Ockey, Gary J.
Date: 2019-12-00
Type: Journal Articles
Subjects: Computer Assisted Testing,Speech Skills,Interpersonal Competence,Videoconferencing,Oral Language,Technology Integration,Instructional Effectiveness,Interaction,College Students,Student Satisfaction,Foreign Countries,English Language Learners,Usability

The construct of oral ability is multifaceted, but due to technological and practical constraints, the majority of computer-delivered speaking assessments are designed to measure only certain aspects of this ability. Most notably, interactional competence, which we define as the ability to actively structure dialogic speech in real time, is often not assessed. Innovations in technology, namely, computermediated video, make it possible for test takers in different locations to see and speak with others in real time and may make it achievable for computer-based tests to assess more aspects of oral communication, including interactional competence. This report gives the findings from a study that explored to what extent computer-mediated video, namely, Skype, could function in conjunction with a platform designed to present four innovative speaking tasks that could conceivably assess a broad construct of oral ability. The overarching goals of this project were twofold. First, we aimed to determine (a) the degree to which current computer video-mediated technology can be used effectively to deliver assessments remotely and (b) the extent to which participants felt that four specific tasks could assess speaking ability by means of this technology. The speaking tasks included giving short responses to an interlocutor's questions, summarizing a proposal, defending a position in a group discussion, and giving a prepared presentation and responding to questions from other participants. Two data collections were conducted: one with all 72 participants located in the United States and one with all 74 participants located in China. The findings provide preliminary evidence that the stability of computer video-mediated technology varies considerably, with technical disruptions being relatively few in the U.S. trial but very frequent in the China context. Moreover, the findings suggest that participants viewed the tasks as generally representing interactive speaking activities that they encounter in the oral language use domain, affording them the opportunity to demonstrate their oral abilities, and that these tasks can be effectively completed in a computer video-mediated environment when technology cooperates.
Large-Scale Estimates of LGBQ-Heterosexual Disparities in the Presence of Potentially Mischievous Responders: A Preregistered Replication and Comparison of Methods
Author: Cimpian, Joseph R.
Date: 2019-00-00
Type: Journal Articles
Subjects: LGBTQ People,At Risk Students,Responses,High School Students,Data Collection,Validity,Questionnaires,Sexual Identity,Student Behavior,Probability,Student Characteristics,Data Analysis,Adolescents,Health Behavior,Gender Differences,Deception

Although numerous survey-based studies have found that students who identify as lesbian, gay, bisexual, or questioning (LGBQ) have elevated risk for many negative academic, disciplinary, psychological, and health outcomes, the validity of the types of data on which these results rest have come under increased scrutiny. Over the past several years, a variety of data-validity screening techniques have been used in attempts to scrub data sets of &quot;mischievous responders,&quot; youth who systematically provide extreme and untrue responses to outcome items and who tend to falsely report being LGBQ. We conducted a preregistered replication of Cimpian et al. with the 2017 Youth Risk Behavior Survey to (1) estimate new LGBQ-heterosexual disparities on 20 outcomes; (2) test a broader, mechanistic theory relating mischievousness effects with a feature of items (i.e., item response-option extremity); and (3) compare four techniques used to address mischievous responders. Our results are consistent with Cimpian et al.'s findings that potentially mischievous responders inflate LGBQ-heterosexual disparities, do so more among boys than girls, and affect outcomes differentially. For example, we find that removing students suspected of being mischievous responders can cut male LGBQ-heterosexual disparities in half overall and can completely or mostly eliminate disparities in outcomes including fighting at school, driving drunk, and using cocaine, heroin, and ecstasy. Methodologically, we find that some methods are better than others at addressing the issue of data integrity, with boosted regressions coupled with data removal leading to potentially very large decreases in the estimates of LGBQ-heterosexual disparities, but regression adjustment having almost no effect. While the empirical focus of this article is on LGBQ youth, the issues discussed are relevant to research on other minority groups and youth generally, and speak to survey development, methodology, and the robustness and transparency of research.
Reducing Gender Stereotypes in Toys and Play for Smarter, Stronger, and Kinder Kids
Author: Weisgram, Erica S.
Date: 2019-00-00
Type: Journal Articles
Subjects: Sex Stereotypes,Toys,Gender Differences,Play,Child Development,Childrens Television,Environmental Influences,Cognitive Development,Young Children

The author discusses gender differences in children's play and its relation to the programs of &quot;Sesame Street,&quot; which for fifty years has sought to enhance children's physical, cognitive, and social development through playful learning. Gender differences in children's play, she asserts, are vast, and she notes that, consequently, boys and girls may develop different skills and learn different concepts through their play. Given this, &quot;Sesame Street&quot; remains committed to gender equity and opposed to gender stereotypes and seeks to prevent the development of and reduce the endorsement of these stereotypes among young children, especially in their play.
Graduate Admissions Practices: A Targeted Review of the Literature. Research Report. ETS RR-19-33
Author: Michel, Rochelle S.
Date: 2019-12-00
Type: Journal Articles
Subjects: College Admission,Graduate Study,Administrative Organization,Standardized Tests,Scores,College Applicants,Student Diversity,Decision Making,Committees,College Entrance Examinations,Grade Point Average,Interpersonal Competence,Holistic Approach,Personality Traits,Thinking Skills,Admission Criteria

This review of the literature covers 5 major areas within the graduate admissions process: (a) the decentralized nature of graduate admissions; (b) the types of materials that are collected as part of the application process, including standardized test scores; (c) the variety of admissions models that are used to make admissions decisions; (d) training and tools used by graduate admissions committees to support the graduate admissions process; and (e) admitting diverse applicants. The literature review concludes with a discussion of the findings and the areas for future research.
Language-in-Education Policies and Indigenous Language Revitalization Efforts in Canada: Considerations for Non-Dominant Language Education in the Global South
Author: McIvor, Onowa
Date: 2019-00-00
Type: Journal Articles
Subjects: Indigenous Knowledge,Native Language,Foreign Countries,Language Maintenance,Childrens Rights,Language Usage,Language of Instruction,Student Rights,Native Speakers,Canada Natives,Government Role,Adult Education,Native Language Instruction

Indigenous languages are struggling for breath in the Global North. In Canada, Indigenous language medium schools and early childhood programs remain independent and marginalized. Despite government commitments, there is little support for Indigenous language-in-education policy and initiatives. This article describes an inaugural, countrywide, federally-funded, Indigenous-led language revitalization research project, entitled NE?OL?EW__(&quot;one mind-one people&quot;). The project brings together nine Indigenous partners to build a country-wide network and momentum to pressure multi-levels of government to honour agreements enshrining the right of children to learn their Indigenous language. The project is documenting approaches to create new Indigenous language speakers, focusing on adult language learners able to keep the language vibrant and teach their language to children. The article reflects upon how this Northern emphasis on Indigenous language revitalization and country-wide networking initiative is relevant to mother tongue-based education and policy examples in the Global South. The article underscores the need for both community level initiatives (top-down) and government level policy and funding (bottom up) to support child and adult Indigenous language learning.
Pedagogic Principles in Digital Pragmatics Learning Materials: Learner Experiences and Perceptions. Research Report. ETS RR-19-34
Author: Wain, Jennifer
Date: 2019-12-00
Type: Journal Articles
Subjects: English Language Learners,Second Language Instruction,Student Attitudes,Pragmatics,Work Environment,Language Usage,Language Skills,Teaching Methods,Computer Assisted Instruction,English (Second Language),Adults,Educational Technology,Professional Personnel

This study reports on the development and evaluation of a prototype for an interactive, self-access computer application titled Words at Work that was developed by the authors. Words at Work aims to raise adult English language learners' (ELLs') pragmatic awareness in the English-medium workplace. The study focuses in particular on the pedagogic principles of second-language (L2) instruction underlying the design of the computer-based learning environment and seeks to explore the perceived usefulness of computer-assisted language learning (CALL)-mediated L2 pragmatic instruction, responding to this overarching research question: To what extent do learners feel that Words at Work successfully operationalizes the principles of teaching L2 pragmatics as proposed in the research literature? Accordingly, we first review teaching principles for L2 pragmatics instruction and present an overview of existing standalone materials for L2 pragmatics learning and teaching. Then, we outline the development and design features of Words at Work, before presenting the findings of a usability study that was conducted with 19 adult ELLs. We found evidence that users desired immersive learning environments that incorporate authentic audiovisual input, personalized and immediate feedback, and extensive opportunities for interaction and self-reflection. Implications for the design of computer-mediated L2 pragmatic learning tools and tasks will be discussed both for instruction and assessment.
Stakes in Testing: Not a Simple Dichotomy but a Profile of Consequences That Guides Needed Evidence of Measurement Quality. Research Report. ETS RR-19-19
Author: Tannenbaum, Richard J.
Date: 2019-12-00
Type: Journal Articles
Subjects: High Stakes Tests,Testing Programs,Measurement,Evaluation Criteria,Licensing Examinations (Professions),Accountability,Elementary Secondary Education,Occupational Tests,Educational Testing,Test Results

Testing programs are often classified as high or low stakes to indicate how stringently they need to be evaluated. However, in practice, this classification falls short. A high-stakes label is taken to imply that all indicators of measurement quality must meet high standards; whereas a low-stakes label is taken to imply the opposite. This approach can result in inappropriate allocation of resources and inadequate attention to needed evidence. We argue that &quot;stakes&quot; are better thought of as a profile of consequences. We suggest generalizable criteria for evaluating and responding to stakes in testing, with applications to licensure, employment, and K-12 accountability testing.
Teachers Carrying out Their Professional Roles as View of Zarqa Town&apos;s Teachers
Author: Al-Zoubi, Zuhair
Date: 2019-00-00
Type: Journal Articles
Subjects: Foreign Countries,Teacher Role,Professionalism,Teacher Attitudes,Public Schools,Teacher Student Relationship,Teacher Characteristics,Teacher Effectiveness,Elementary School Teachers,Secondary School Teachers

The teacher is the dynamic axis in the educational process. An effective teacher is a source of inspiration and has an effective and positive impact on students learning. Students rely on teachers to make decisions about their studies, work, and living in the future. This study aimed to investigate the degree of how much teachers carry out their professional roles and the affecting factors from the point of view of Zarqa Governorate's teachers. To achieve these goals, researchers created a questionnaire that measures the degree of teachers doing their professional roles and another questionnaire that measures teachers' estimations for the most important affecting factors on carrying out the professional role. The study's sample consisted of (1200) teachers from public schools in Zarqa Governorate's education directorates. This study was conducted during academic year 2018-2019. The study results showed that the percentage of teachers doing their professional role in Zarqa Governorate's education directorates were high, there are no statistically significant differences for the degree of teachers doing their professional roles according to sex, educational qualification, level and years of experience. The researchers recommend to focus on highlighting teacher's role during teaching through the curricula and focusing on field exercise for future teachers.
State-Level Perspectives on Kindergarten Readiness. Policy Information Report and ETS Research Report Series No. RR-19-07
Author: Slutzky, Carly
Date: 2019-00-00
Type: Journal Articles
Subjects: Kindergarten,School Readiness,Standards,State Policy,Child Development,Definitions,Early Childhood Education,Public Agencies,Educational Policy,Age Differences,Skill Development,Knowledge Level,Online Surveys,Administrator Attitudes,Policy Formation

Kindergarten readiness has received significant attention in recent years in response to concerns regarding preparing young children for school and beyond and mitigating persistent achievement gaps. Currently, there is little consensus around what factors drive and define kindergarten readiness. In this study, we sought to expand understanding of perspectives on how kindergarten readiness is defined, the role of early learning standards for defining kindergarten readiness, views about the implications of defining kindergarten readiness, and the utility of state definitions for guiding kindergarten readiness practices. Data sources included surveys, focus groups and interviews with individuals representing state early childhood education agencies, and existing state kindergarten readiness definitions. Collective study findings indicated that there is evidence of persisting variation in perspectives on how to define kindergarten readiness, reflecting continuing tension about the construct's meaning. Although state policy makers tend to view kindergarten readiness through a developmental lens rather than as a ready-or-not statement, they also emphasize the role of child attributes such as age or skills and knowledge outlined in early learning standards. Additionally, participants reported both positive views and apprehension around adopting a common definition to guide efforts to promote readiness. This investigation underscores the need for further consideration of adopting a universal, multifaceted kindergarten readiness definition that calls attention to the role of child and environmental inputs as a starting point for preparing all young children for school.