ethanporcaro/eric-fulltext-meta huggingface dataset:
Evidence of the Generalization and Construct Representation Inferences for the &quot;GRE&quot;® Revised General Test Sentence Equivalence Item Type. ETS GRE® Board Research Report. ETS GRE®-17-02. ETS Research Report. RR-17-05
Author: Bejar, Isaac I.
Date: 2017-03-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Generalization,Inferences,Test Items,Sentences,Evidence,Vocabulary,Test Validity,Scores,Test Construction,Difficulty Level,Psychometrics,Syntax,Semantics,Word Frequency,Item Response Theory,Familiarity,Prediction,Models,Regression (Statistics),Statistical Analysis

The report is the first systematic evaluation of the sentence equivalence item type introduced by the &quot;GRE&quot;® revised General Test. We adopt a validity framework to guide our investigation based on Kane's approach to validation whereby a hierarchy of inferences that should be documented to support score meaning and interpretation is evaluated. We present evidence relevant to the generalization inference as well as evidence of construct representation. We analyzed the pool of sentence equivalence items in three studies. The first and second studies focused on the generalization inference and sought to document the construction principles behind the sentence equivalence items, specifically the nature of the vocabulary tested. The third study focused on construct representation and evaluated the contribution of the stem, the keys, and the distractors to item difficulty. We concluded that the vocabulary tested by the sentence equivalence items is appropriate given the purpose of the GRE, namely, to assist in the selection of graduate students. The difficulty of the items was shown to be, in part, a function of the familiarity of the vocabulary as well as the context in which the vocabulary is tested, which we argue is positive validity evidence.
Examining Students&apos; Ability to Critique Arguments and Exploring the Implications for Assessment and Instruction. Research Report. ETS RR-17-16
Author: Song, Yi
Date: 2017-08-00
Type: Journal Articles
Subjects: Criticism,Persuasive Discourse,Academic Ability,Grade 8,Thinking Skills,Task Analysis,Integrated Activities,Reading Tests,Writing Tests,Vignettes,Scoring Rubrics,Rhetorical Criticism

This study assessed 1,706 eighth-grade students' reasoning abilities through a critique task in an integrated argumentative reading and writing test. Results indicate that the majority of students did not detect fallacious arguments or clearly explain problems in the arguments. They encountered various challenges in critiquing an argument: (a) being off-task, (b) failing to identify fallacious arguments, (c) having difficulty explaining specific problems in reasoning, and (d) not connecting their criticisms with particular parts of the text being critiqued. Task design and instructional implications are discussed.
Expanding Audio Access to Mathematics Expressions by Students with Visual Impairments via MathML. Research Report. ETS RR-17-13
Author: Frankel, Lois
Date: 2017-04-00
Type: Journal Articles
Subjects: Visual Impairments,Research Reports,Audiovisual Aids,Assistive Technology,Accessibility (for Disabled),Mathematics Activities,Mathematical Formulas,Secondary School Mathematics,Algebra,Pilot Projects,Usability,Delivery Systems,Preferences,Self Efficacy,Problem Solving,Feedback (Response),Navigation (Information Systems),Comparative Analysis,Questionnaires,Teaching Methods,Statistical Analysis,Qualitative Research,Secondary School Teachers

This report describes the pilot conducted in the final phase of a project, Expanding Audio Access to Mathematics Expressions by Students With Visual Impairments via MathML, to provide easy-to-use tools for authoring and rendering secondary-school algebra-level math expressions in synthesized speech that is useful for students with blindness or low vision. The pilot evaluated the authoring and speech-rendition tools, including interactive navigation. Teachers participated in the portion of the study that evaluated the authoring tools. Secondary school students with blindness or low vision participated in the portion that evaluated the speech-rendition and navigation tools, comparing those tools to each student's usual method (braille or print) for working with math. The teachers received an interactive authoring tutorial and the students received an interactive navigation tutorial before using the tools studied. Prior to the pilot, feedback studies on authoring and navigation gathered information that was used to fine-tune the tutorials and the functionalities being evaluated. In the pilot we attempted to simulate a likely use-case for each group. In the students' case, the tools for reading and navigating spoken mathematics were compared directly with the methods each student typically used for reading math (braille or print in a size appropriate for the student). Teachers and content providers were instructed to author math expressions that matched those used in the pilot instruments given to the students. The studies showed that the authoring tools were easy for teachers to use as intended and that little observable difference existed between students' success in answering math and math-parsing questions when using the spoken math tools and their success in answering parallel questions posed in braille or print of appropriate size. Please see Appendix E for information on obtaining a version of this report that is fully accessible using the tools described.
Gender and Minority Achievement Gaps in Science in Eighth Grade: Item Analyses of Nationally Representative Data. Research Report. ETS RR-17-36
Author: Qian, Xiaoyu
Date: 2017-07-00
Type: Journal Articles
Subjects: Item Analysis,Gender Differences,Achievement Gap,Grade 8,Minority Group Students,Science Achievement,Research Reports,National Surveys,Item Response Theory,Racial Differences,Statistical Analysis,Science Tests,Spatial Ability,Performance Factors,Test Bias,Student Characteristics,Predictor Variables,Teacher Effectiveness,Teaching Methods,Difficulty Level,Test Items,Bayesian Statistics,Achievement Tests,Mathematics Achievement,Elementary Secondary Education,Foreign Countries,International Assessment,Mathematics Tests,Monte Carlo Methods

In this study, we investigated gender and minority achievement gaps on 8th-grade science items employing a multilevel item response methodology. Both gaps were wider on physics and earth science items than on biology and chemistry items. Larger gender gaps were found on items with specific topics favoring male students than other items, for example, an earth science item requiring visual-spatial ability. Minority students were more likely than White students to score lower on harder constructed-response (CR) items. Some teachers were more likely to reduce minority achievement gaps on easier CR items than other teachers. Implications for instruction in terms of improving visual-spatial awareness, efficacy of female students, and modeling scientific literacy for minority students were discussed.
Network Analysis of Conversation Data for Engineering Professional Skills Assessment. Research Report. ETS RR-17-59
Author: Zhu, Mengxiao
Date: 2017-12-00
Type: Journal Articles
Subjects: Network Analysis,Skill Analysis,Research Reports,Engineering Education,Professional Education,Data Analysis,Group Discussion,Vignettes,Social Networks,Discourse Analysis,Problem Solving,Cooperative Learning,Undergraduate Students,Problem Based Learning,Predictor Variables,Statistical Analysis,Performance Based Assessment

In this paper, we examine the student group discussion processes in a scenario-based assessment of engineering professional skills called Engineering Professional Skills Assessment (EPSA). In the assessment, the students were evaluated through a discussion on a scenario related to an engineering problem with no clear-cut solution. We applied various social network analysis (SNA) techniques and constructed 2 types of networks to model the patterns of the communication process in the student discussion and the connections among engineering professional skills (EPS) during the discussion process. We found that some network statistics were statistically related to team scores (or performance), such as the density of the EPS networks. We also investigated the temporal pattern of the discussion process by comparing the network statistics across 3 different time points. Results showed that the middle and end of the discussion processes differed significantly on 2 network measures: communication density and EPS connectedness. Results of the study have implications for instruction and scoring of such performance tasks. Limitations and next steps are also discussed.
Exploring Online Learning Data Using Fractal Dimensions. Research Report. ETS RR-17-15
Author: Guo, Hongwen
Date: 2017-04-00
Type: Journal Articles
Subjects: Data Collection,Electronic Learning,Intelligent Tutoring Systems,Information Utilization,Item Response Theory,Learning Processes,Accuracy,Geometric Concepts,Correlation,Computation,Mathematical Formulas,Data Analysis

Data collected from online learning and tutoring systems for individual students showed strong autocorrelation or dependence because of content connection, knowledge-based dependency, or persistence of learning behavior. When the response data show little dependence or negative autocorrelations for individual students, it is suspected that students are randomly guessing the answers or that they are inconsistent in learning behavior. In addition, the global and local rates of correct responses may reflect students' proficiency in the learning process. This study shows that the dependence of online data may be characterized by the fractal dimension as a summary statistic locally and globally. The rate of correct responses and the global and local fractal dimensions of individual students' responses may indicate their learning behavior in short and long learning windows. The results may shed light on when individual students are experiencing difficulties in the learning process.
Preschoolers&apos; Executive Function: Importance, Contributors, Research Needs and Assessment Options. Policy Information Report and ETS Research Report Series No. RR-17-22
Author: Ackerman, Debra J.
Date: 2017-00-00
Type: Journal Articles
Subjects: Preschool Children,Executive Function,Research Needs,Research Reports,Early Childhood Education,Cognitive Ability,Relevance (Education),Young Children,Cognitive Development,Performance Factors,Environmental Influences,Child Development,Learning Activities,Intervention,Role of Education,Task Analysis,Meta Analysis

The early education field increasingly is recognizing the key role played by young children's executive function (EF) skills, generally defined as the cognitive abilities that consciously support goal-directed behaviors. To provide the field with an overview of research conducted on this topic over the past 15 years, we review research on the traits and skills that fall under the broader umbrella of preschool EF and the role it plays in young children's developmental and academic outcomes. Also addressed are the child, environmental, activity-related, and curricular factors potentially impacting the development of EF and some EF-related topics for which additional research is needed. Finally, we provide practical and psychometric information regarding six examples of measures that focus on assessing preschoolers' EF skills. This report can serve as a resource for early childhood researchers and practitioners who are interested in understanding EF development during the early years. By highlighting some topics for which additional research is needed and providing information regarding examples of valid and reliable measures to assess EF in children ages 3-5 years, we hope this report also will serve as a springboard for future studies related to preschool EF.
CPS-Rater: Automated Sequential Annotation for Conversations in Collaborative Problem-Solving Activities. Research Report. ETS RR-17-58
Author: Hao, Jiangang
Date: 2017-12-00
Type: Journal Articles
Subjects: Problem Solving,Cooperative Learning,Teamwork,Documentation,Interpersonal Communication,Automation,Discourse Analysis,Computer Simulation,Classification,Coding,Science Instruction

Conversations in collaborative problem-solving activities can be used to probe the collaboration skills of the team members. Annotating the conversations into different collaboration skills by human raters is laborious and time consuming. In this report, we report our work on developing an automated annotation system, CPS-rater, for conversational data from collaborative activities. The linear chain conditional random field method is used to model the sequential dependencies between the turns of the conversations, and the resulting automated annotation system outperforms those systems that do not model the sequential dependency.
Developing a Useful and Integrative STEM Disciplinary Language
Author: Capraro, Robert M.
Date: 2018-00-00
Type: Journal Articles
Subjects: STEM Education,Language Usage,Vocabulary,Secondary School Students,Summer Programs,Educational Technology,Technology Uses in Education,Cooperative Learning,Problem Solving,Pretests Posttests,Qualitative Research,Communication Strategies,Postsecondary Education,Problem Based Learning,Learning Activities,College Preparation,Community Involvement,Informal Education,Mentors,Role Models,Extracurricular Activities,Career Choice,Academic Achievement,Computer Peripherals,Observation,Discourse Analysis,Freehand Drawing,Coding

STEM disciplinary language is a necessary component for STEM success. It can be developed through experiences and attention to the development of STEM activities that are rich in language and can be acquired through practical experiences and systematic practice. Secondary students participated in an informal STEM summer camp where they learned to use Google Sketchup® and a 3-D printer to design their own objects. They interacted with peers and collaborated on issues as they arose, to solve problems. They took the &quot;Aural/Spatial Interactions and Invariant Components of Vocabulary for STEM&quot; instrument pre- and posttest, and the results were analyzed qualitatively. Students progressed through styles of communication that were precipitated by terminological inexactitude, cognitive negotiation and then analogous/metaphoric, clarifying, and technical discourses to build mutual understanding and success.
Continuing a Culture of Evidence: Student-Level Assessment. Research Report. ETS RR-17-10
Author: Markle, Ross
Date: 2017-00-00
Type: Journal Articles
Subjects: Higher Education,Evaluation Methods,Student Evaluation,Scores,Outcomes of Education,Formative Evaluation,Educational Change,Competency Based Education,Models,Academic Achievement,Educational Assessment,College Students,Educational Research,Public Policy,Credentials

From 2006 to 2008, Educational Testing Service (ETS) produced a series of reports titled &quot;A Culture of Evidence&quot;, designed to capture a changing climate in higher education assessment. A decade later, colleges and universities already face a new set of challenges resulting from societal, technological, and various other influences. ETS is now initiating a new series of reports that describe current issues and trends in assessment and how they relate to the needs of higher education institutions. In this report, I discuss an increasing need to understand individual students rather than whole institutions. Previous models of assessment, which focused on comparability and accountability, made inferences at the group level, often comparing one institution to a group of similar institutions. However, emerging trends in the ways students learn, the sources of the learning, and the use of assessment results have made aggregate data alone insufficient. Greater consideration of student-level data could help to foster emerging educational trends as well as guide learning in existing colleges and universities.
A Video-Based Measure of Preservice Teachers&apos; Abilities to Predict Elementary Students&apos; Scientific Reasoning
Author: Akerson, Valarie L.
Date: 2018-00-00
Type: Journal Articles
Subjects: Video Technology,Preservice Teachers,Mixed Methods Research,Prediction,Science Process Skills,Thinking Skills,Logical Thinking,Pedagogical Content Knowledge,Science Instruction,Elementary School Teachers,Field Experience Programs,Scientific Principles,Pretests Posttests,Science Teachers,Grade 4,Mathematics Instruction,Teacher Education Programs,Mathematics Teachers,Interviews,Formative Evaluation

In this mixed methods study, the researchers developed a video-based measure called a &quot;Prediction Assessment&quot; to determine preservice elementary teachers' abilities to predict students' scientific reasoning. The instrument is based on teachers' need to develop pedagogical content knowledge for teaching science. Developing a knowledge base for aiding teachers in their abilities to predict students' scientific reasoning promotes student learning because it enables teachers to understand students' current conceptions and to be able to build lessons to improve upon those ideas. To determine whether preservice teachers are improving in their abilities to predict scientific reasoning it is necessary to have an instrument that can measure their current abilities and then to measure whether those abilities improve through instruction. In the second part of this study the authors used the prediction assessment to determine whether the traditional or new (Iterative Model Building, or IMB) field experience improved preservice elementary teachers' abilities to make reasonable predictions of student scientific reasoning. It was found that though preservice teachers in both the traditional and IMB field experience approaches improved their abilities to make predictions, there was a greater number in the IMB group who made reasonable predictions and based those predictions on student reasoning.
Online Questionnaires Use with Automatic Feedback for e-Innovation in University Students
Author: Remesal, Ana
Date: 2017-00-00
Type: Journal Articles
Subjects: Questionnaires,Feedback (Response),Independent Study,Automation,Computer Uses in Education,Educational Innovation,Preservice Teachers,College Freshmen,Formative Evaluation,Student Attitudes,Statistical Analysis,Use Studies,Foreign Countries,Student Behavior,Preschool Education,Elementary Education

Technological tools have permeated higher education programs. However, their mere introduction does not guarantee instructional quality. This article presents the results of an innovation project aimed at fostering autonomous learning among students at a Pre-School and Primary Teacher degree. For one semester all freshmen students used a system for autonomous learning embedded in the institutional online platform (Moodle), which included automatic formative feedback. The system was part of a complex formative assessment program. We present results of the experience concerning two aspects: the students' actual use of the system, and their final appraisal of it. The quantitative descriptive analysis focuses on the students' perspective to evaluate the adequacy of the instructional decisions. Results indicate that students need certain limits to be able to manage their learning better if we pursue the quality of innovation. These limits refer mainly to the time of accessibility and the limitation of attempts of practice. With respect to time, an appropriate span of time (neither too long nor too short) must be chosen; with respect to the number of attempts, it is expedient to limit rather than promote free endless access.
Role of Visualization in Mathematical Abstraction: The Case of Congruence Concept
Author: Yilmaz, Rezan
Date: 2018-00-00
Type: Journal Articles
Subjects: Geometry,Visualization,Mathematical Logic,Mathematics Instruction,Mathematical Concepts,Case Studies,Preservice Teachers,Mathematics Teachers,Secondary School Teachers,Secondary School Mathematics,Interviews,Content Analysis

Mathematical abstraction is an important process in mathematical thinking. Also, visualization is a strong tool for searching mathematical problems, giving meaning to mathematical concepts and the relationships between them. In this paper, we aim to investigate the role of visualizations in mathematical abstraction through a case study on five pre-service secondary mathematics teachers. To do this, we formulated instructions that are appropriate for abstracting the concept of congruent figures or objects, and then determined the visualizations used by the participants, how they established these and what kind of visual images they had. The data were collected through class observations and semi-structured interviews. Data analysis showed that visualizations were used frequently by the participants and differently while engaged in abstraction. They used isomorphic and homeomorphic visualizations and these were mostly in coordination and internalization processes. The participants also produced different visual images which were commonly concrete and dynamic ones. The visualizations that were used in the abstraction processes significantly aided the understanding of the concept of congruent figures and strengthened the visual images of the participants.
Developing a Machine-Supported Coding System for Constructed-Response Items in PISA. Research Report. ETS RR-17-47
Author: Yamamoto, Kentaro
Date: 2017-12-00
Type: Journal Articles
Subjects: Foreign Countries,Achievement Tests,International Assessment,Secondary School Students,Test Items,Responses,Scoring,Computer Assisted Testing,Test Scoring Machines,Reading Tests,Interrater Reliability,Accuracy,Efficiency

Approximately a third of the Programme for International Student Assessment (PISA) items in the core domains (math, reading, and science) are constructed-response items and require human coding (scoring). This process is time-consuming, expensive, and prone to error as often (a) humans code inconsistently, and (b) coding reliability in international large-scale assessments varies across items and countries. The shift in PISA 2015 from paper- to computer-based assessment digitized all responses and associated coding, thus providing opportunities to introduce technology and analytical methods to improve data processing and analyses in future cycles. In this computer-based setting, this research explains the framework and approach for improving the accuracy and efficiency of the coding process in constructed-response items for future PISA cycles. In the pilot study associated with this research, we demonstrate how the proposed machine-supported coding system can be utilized using sample reading items from the PISA 2015 cycle.
Definitions Matter: Investigating and Comparing Different Operalionalizations of Academic Undermatching
Author: Gansemer-Topf, Ann M.
Date: 2017-00-00
Type: Journal Articles
Subjects: Differences,Definitions,Vocabulary,Comparative Analysis,College Students,College Choice,National Surveys,Longitudinal Studies,Reliability,Student Characteristics,Statistical Analysis,Evaluation Problems

Effective assessment practice requires clearly defining and operationalizing terminology. We illustrate the importance of this practice by focusing on academic &quot;undermatching&quot;--when students enroll in colleges that are less academically selective than those for which they are academically prepared. Undermatching has been viewed as a potential obstacle in the United States' goal of increasing degree attainment but operationalizing undermatching is difficult. Using ELS: 2002, a national dataset from the U.S. Department of Education National Center for Education Statistics (NCES, 2014), we developed eight operationalizations of undermatching by altering three commonly used variables. We then compared the number and demographics of students who were identified as undermatched. Differences in operationalizations resulted in significant differences in undermatching by gender, race, parental education, and socioeconomic status. Results of this study illustrate the importance of the need to operationalize terminology used in assessment carefully and consistently.