ethanporcaro/eric-fulltext-meta huggingface dataset:
Using Vision and Speech Features for Automated Prediction of Performance Metrics in Multimodal Dialogs. Research Report. ETS RR-17-20
Author: Ramanarayanan, Vikram
Date: 2017-12-00
Type: Journal Articles
Subjects: Automation,Prediction,Man Machine Systems,Open Source Technology,Computer Uses in Education,Artificial Intelligence,Performance,Audio Equipment,Video Technology,Predictive Validity,Speech,Qualitative Research,Statistical Analysis

Predicting and analyzing multimodal dialog user experience (UX) metrics, such as overall call experience, caller engagement, and latency, among other metrics, in an ongoing manner is important for evaluating such systems. We investigate automated prediction of multiple such metrics collected from crowdsourced interactions with an open-source, cloud-based multimodal dialog system in the educational domain. We extract features from both the audio and video signals and examine the efficacy of multiple machine learning algorithms in predicting these performance metrics. The best performing audio features consist of multiple low-level audio descriptors--intensity, loudness, cepstra, pitch, and so on--and their functionals, extracted using the OpenSMILE toolkit, while the video features are bags of visual words that use 3D Scale-Invariant Feature Transform descriptors. We find that our proposed methods outperform the majority vote classification baseline in predicting various UX metrics rated by both the user and experts. Our results suggest that such automated prediction of performance metrics can not only inform the qualitative and quantitative analysis of dialogs but also be potentially incorporated into dialog management routines for positively impacting UX and other metrics during the course of the interaction.
Evaluating the Stability of Test Score Means for the &quot;TOEIC&quot;® Speaking and Writing Tests. Research Report. ETS RR-17-50
Author: Qu, Yanxuan
Date: 2017-12-00
Type: Journal Articles
Subjects: Scores,English (Second Language),Language Tests,Second Language Learning,Speech Tests,Writing Tests,Reliability,Statistical Analysis,Regression (Statistics),Foreign Countries

For educational tests, it is critical to maintain consistency of score scales and to understand the sources of variation in score means over time. This practice helps to ensure that interpretations about test takers' abilities are comparable from one administration (or one form) to another. This study examines the consistency of reported scores for the &quot;TOEIC&quot;® Speaking and Writing tests using statistical procedures. Specifically, the stability of the TOEIC Speaking score means from 431 forms administered in a 3-year period was evaluated using harmonic regression, and the stability of TOEIC Writing score means from 66 forms administered in a 3-year period was evaluated using analysis of variance. Results indicated that the fluctuations in the TOEIC Speaking or Writing score means mainly reflect changes in test takers' overall English speaking or writing ability levels instead of score inaccuracies. For both speaking and writing test scores, a large proportion of the variation in score means can be explained by seasonality (the rise or fall of score means associated with specific times of the year) and test takers' demographic information, which have been shown to be related to test-taker ability. As a result, this finding provides evidence for the consistency of the TOEIC Speaking and Writing score scales across forms.
Use of Automated Scoring in Spoken Language Assessments for Test Takers with Speech Impairments. Research Report. ETS RR-17-42
Author: Loukina, Anastassia
Date: 2017-12-00
Type: Journal Articles
Subjects: Automation,Scoring,Language Tests,Speech Tests,Speech Impairments,English (Second Language),Second Language Learning,Adults,Interrater Reliability,Test Scoring Machines,Hearing Impairments

This study is an evaluation of the performance of automated speech scoring for speakers with documented or suspected speech impairments. Given that the use of automated scoring of open-ended spoken responses is relatively nascent and there is little research to date that includes test takers with disabilities, this small exploratory study focuses on one type of scoring technology, automatic speech scoring (the &quot;SpeechRater&quot;? automated scoring engine); one type of assessment, spontaneous spoken English by nonnative adults (six &quot;TOEFL iBT&quot;® test speaking items per test taker); and one category of disability, speech impairments. The results show discrepancies between human and SpeechRater scores for speakers with documented speech or hearing impairments who receive accommodations and for speakers whose responses were deferred to the scoring leader by human raters because the responses exhibited signs of a speech impairment. SpeechRater scores for these studied groups tended to be higher than the human scores. Based on a smaller subsample, the word error rate was higher for these groups relative to the control group, suggesting that the automatic speech recognition system contributed to the discrepancies between SpeechRater and human scores.
Toward the Automated Scoring of Written Arguments: Developing an Innovative Approach for Annotation. Research Report. ETS RR-17-11
Author: Song, Yi
Date: 2017-12-00
Type: Journal Articles
Subjects: Scoring,Automation,Persuasive Discourse,Documentation,Essays,Writing Tests,Classification,Reliability,Identification,College Students

This project focuses on laying the foundations for automated analysis of argumentation schemes, supporting identification and classification of the arguments being made in a text, for the purpose of scoring the quality of written analyses of arguments. We developed annotation protocols for 20 argument prompts from a college-level test under the framework of the theory of argumentation schemes, which defines reasoning patterns in argumentation. The annotation protocols listed critical questions associated with each argumentation scheme, which makes the argument structure in a text explicit and classifiable. Furthermore, we annotated 200 student essays across four selected argument prompts to test if the annotation protocols can be applied reliably by human annotators. Preliminary results indicate that this method of analyzing argument structure is reliable and promising.
Helping Students Select Appropriately Challenging Text: Application to a Test of Second Language Reading Ability. Research Report. ETS RR-17-33
Author: Sheehan, Kathleen M.
Date: 2017-12-00
Type: Journal Articles
Subjects: Reading Ability,Second Language Learning,Difficulty Level,Reading Material Selection,Developmentally Appropriate Practices,Reader Text Relationship,Reading Tests,Reading Comprehension,Educational Strategies,Educational Practices,English (Second Language),Language Tests

A model-based approach for matching language learners to texts of appropriate difficulty is described. Results are communicated to test takers via a targeted reading range expressed on the reporting scale of an automated text complexity measurement tool (ATCMT). Test takers can use this feedback to select reading materials that are well matched to their abilities, that is, materials that are expected to be challenging, yet not so challenging as to cause frustration or reduce motivation. An application to the problem of helping students prepare to take the &quot;TOEFL iBT&quot;® test is presented.
The Consistency of &quot;TOEIC&quot;® Speaking Scores across Ratings and Tasks. Research Report. ETS RR-17-46
Author: Schmidgall, Jonathan E.
Date: 2017-12-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Speech Tests,Scores,Reliability,Scoring,Test Construction,Generalizability Theory

This report briefly reviews the design and scoring procedure for the &quot;TOEIC&quot;® Speaking test and summarizes existing evidence about the consistency of TOEIC Speaking test scores. It then describes several analyses conducted using generalizability theory to provide additional information about the consistency of scores across different aspects of the scoring procedure. The results of these analyses provide more robust information about consistency of TOEIC Speaking scores with respect to important facets of the assessment procedure, such as tasks and ratings. Specifically, the results provide evidence to support claims about the consistency of ratings across different levels of the scoring procedure: tasks, claims, and scale scores.
Implementing a Contributory Scoring Approach for the &quot;GRE&quot;® Analytical Writing Section: A Comprehensive Empirical Investigation. Research Report. ETS RR-17-14
Author: Breyer, F. Jay
Date: 2017-04-00
Type: Journal Articles
Subjects: College Entrance Examinations,Scoring,Essay Tests,Writing Evaluation,Computer Assisted Testing,Computation,Evaluation Methods,Design,Regression (Statistics),Correlation,Weighted Scores,Graduate Students

In this research report, we present an empirical argument for the use of a contributory scoring approach for the 2-essay writing assessment of the analytical writing section of the &quot;GRE&quot;® test in which human and machine scores are combined for score creation at the task and section levels. The approach was designed to replace a currently operational all-human check scoring approach in which machine scores are used solely as quality-control checks to determine when additional human ratings are needed due to unacceptably large score discrepancies. We use data from 6 samples of essays collected from test takers during operational administrations and special validity studies to empirically evaluate 6 different score computation methods. During the presentation of our work, we critically discuss key methodological design decisions and underlying rationales for these decisions. We close the report by discussing how the research methodology is generalizable to other testing programs and use contexts.
Integrating Cognitive Views into Psychometric Models for Reading Comprehension Assessment. Research Report. ETS RR-17-35
Author: Rahman, Taslima
Date: 2017-12-00
Type: Journal Articles
Subjects: Reading Tests,Reading Comprehension,Psychometrics,Test Items,Difficulty Level,Item Response Theory,Prior Learning,Learner Engagement,Grade 8,Middle School Students,Multiple Choice Tests,Familiarity,Statistical Analysis

To demonstrate how methodologies for assessing reading comprehension can grow out of views of the construct suggested in the reading research literature, we constructed tasks and carried out psychometric analyses that were framed in accordance with 2 leading reading models. In estimating item difficulty and subsequently, examinee proficiency, an item response theory (IRT) model called the linear logistic test model was extended to incorporate reader as well as task attributes as covariates. A novel aspect of this modeling was reader effects--interest and prior knowledge-- specific to text passages that the examinees read in the assessment. In the demonstration, the theory-motivated task and reader attributes were found to be significantly related to item difficulty. In particular, examinees' comprehension proficiency estimates positively affected within-person effects concerning the reader's familiarity and interest in a passage. This study suggests that it is both feasible and informative to incorporate variables for various comprehension components into the psychometric analysis.
Cyberaggression among Adolescents: Prevalence and Gender Differences
Author: Álvarez-García, David
Date: 2017-00-00
Type: Journal Articles
Subjects: Foreign Countries,Computer Mediated Communication,Aggression,Adolescents,Incidence,Gender Differences,Secondary School Students,Questionnaires,Victims,Telecommunications,Handheld Devices,Internet,Factor Analysis,Statistical Analysis

The objective of the present work is to analyse the prevalence of cyber-aggression and cyber-victimization among adolescents in Asturias (Spain) and to identify possible gender differences. To this end, 3,175 adolescents aged 12 to 18 years were randomly selected from the student population attending compulsory secondary education in Asturias and assessed. They completed three self-reported tests: an ad hoc questionnaire on sociodemographic data and communication technologies management; the &quot;Cyber-aggression Questionnaire for Adolescents&quot; (CYBA), to assess how frequently adolescents acknowledge having exercised various cyber-aggressive behaviours in the previous three months; and the &quot;Cyber-victimization Questionnaire for Adolescents&quot; (CYVIC), to assess how frequently adolescents acknowledge having been a victim of various types of cyber-aggression in the previous three months. The results obtained show a high variation in prevalence based on the type of cyber-aggression or cyber-victimization analysed. verbal cyber-aggression and online exclusion are more common than impersonation and visual cyber-aggression. There are generally no statistically significant differences between boys and girls. When differences do appear, boys generally tend to be more aggressive than girls, while girls are more likely to be victims. However, these differences are either small or very small. The implications of these results for future research and educational treatment of the problem are discussed.
Insights into Using &quot;TOEIC&quot;® Test Scores to Inform Human Resource Management Decisions. Research Report. ETS RR-17-48
Author: Oliveri, María Elena
Date: 2017-12-00
Type: Journal Articles
Subjects: English (Second Language),Second Language Learning,Language Tests,Scores,Human Resources,Personnel Management,Personnel Selection,Promotion (Occupational),Decision Making,Test Use,Job Training,International Trade,Business,Foreign Countries

This report explores the ways in which human resource (HR) managers use &quot;TOEIC&quot;® scores to inform hiring, promotion, and training decisions in an international workplace. Two data sources were used (a) previously collected test users' testimonials that described managers' use of TOEIC scores to inform HR decisions and (b) test-use examples collected from HR managers and TOEIC representatives specifically for this project. Our analysis of test-use examples provided insight into how companies use TOEIC scores to inform HR decisions related to hiring, promotion, and training of employees in international businesses. To conclude, we provide suggestions for future research such as providing services to test users (e.g., assessment literacy, the development of assessments measuring additional components of workplace English, or algorithms to help analyze the various variables relevant to informing HR decisions) to continue to support meaningful and relevant score-based HR decisions and score interpretations.
Mobile Instant Messaging: Whatsapp and Its Potential to Develop Oral Skills
Author: Andújar-Vaca, Alberto
Date: 2017-00-00
Type: Journal Articles
Subjects: Foreign Countries,Handheld Devices,Computer Oriented Programs,Computer Mediated Communication,Second Language Learning,English (Second Language),Language Skills,Speech Communication,Oral Language,Educational Technology,Electronic Learning,College Students,Experimental Groups,Control Groups,Interaction,Mixed Methods Research,Language Proficiency,Student Improvement,Instructional Effectiveness,Outcomes of Education,Language Tests,Speech Tests,Pretests Posttests,Pronunciation,Grammar,Vocabulary,Comprehension,Correlation

This study investigates the benefits of Mobile Mediated Communication (MMC) to develop oral skills in second-language learners. A total of 80 Spanish students taking a B1 English course at the University of Almería were studied in this research. According to treatment type, subjects were divided in two groups, experimental and control. A &quot;Whatsapp&quot; group was created where 40 of these students participated in a daily interaction during six months. The samples collected in the application as well as a speaking were used to measure the students' degree of oral development and the type and triggers of the language related episodes (LRE) given rise to mobile chat-based oral interaction. This study focuses on such interaction and seeks to measure the students' degree of oral development through a mixed analysis approach. A temporal axis is used to measure the differences between the groups studied. Significant improvements in term of oral proficiency were observed in the experimental group and negotiations were the LRE most common throughout the activity. it is worth mention that Mobile learning offers an environment where learners can ubiquitously negotiate meaning, reflect and evaluate on their own performance through authentic interaction and feedback, constituting a powerful tool for developing second language proficiency.
Digital Leisure and Perceived Family Functioning in Youth of Upper Secondary Education
Author: Valdemoros-San-Emeterio, M-Angeles
Date: 2017-00-00
Type: Journal Articles
Subjects: Leisure Time,Youth,Secondary School Students,Foreign Countries,Family Relationship,Measures (Individuals),Questionnaires,Statistical Analysis,Secondary Education,Likert Scales,Internet,Video Games,Social Networks

The &quot;Network Society&quot; is identified by accelerated changes that occur between real and virtual worlds. The progress of digital devices has generated a new model of leisure that has conditioned family interactions. The aim of this research was to identify the relationship between digital leisure experiences and perceived family functioning in post-compulsory secondary education Spanish students. The sample was composed of 1,764 Spanish young people 15-18 years old; all of them were post-compulsory secondary education students. Students' digital leisure activities were measured by an opening question by which they indicated the three most important leisure activities for them, and family functioning was measured by the answers from the Spanish adaptation for FACES iv questionnaire (Family Adaptability and Cohesion Scale). A descriptive analysis about digital leisure activities of young people was used. The family functioning coefficient of each subject was determined and, finally, the relationship between students' family functioning perceived and students' digital leisure practices assessed by a factorial analysis of variance (ANOVA). Young people give importance to digital leisure activities, highlighting social network participation, playing video games and browsing the internet. Cohesion, flexibility and family functioning are healthier when children do not point to any digital activity into their preferred leisure practices. The results suggest that new research should be conducted to confirm whether this negative association between family functioning and digital leisure is causal or due to other factors.
A Framework for Assessing Developmental Education Programs
Author: Goldwasser, Molly
Date: 2017-00-00
Type: Journal Articles
Subjects: Developmental Programs,Program Evaluation,Evaluation Methods,Best Practices,Models,Program Costs,Course Organization,Student Placement,Educational Practices,Collegiality,College Programs,Literature Reviews

This paper presents a framework for educators, administrators, and researchers to assess distinct facets of developmental education programs. The researchers review the literature on best practices in developmental education with regards to program cost, program structure, and student placement procedures. This paper also identifies seven model institutions in these areas and includes qualitative research from interviewed individuals who work at these institutions. The evaluation tool developed via this research is rooted in both the research-based best practices and qualitative research from administrators at the identified model institutions. The paper concludes with recommendations for implementation of this assessment tool.
Developing Compressed Beginning and Intermediate Algebra Courses
Author: Walker, Sylvia E.
Date: 2017-00-00
Type: Journal Articles
Subjects: Remedial Mathematics,Introductory Courses,Advanced Courses,Algebra,Sequential Approach,College Mathematics,Computer Assisted Instruction,Homework,Community Colleges,Two Year College Students,Student Educational Objectives,Mathematics Tests

The purpose of this project was two-fold. First, it would provide an opportunity for students to complete the developmental math course sequence more quickly, thereby enabling students to proceed to a college-level mathematics course sooner. To accomplish this, the classroom was designed with computer-assisted homework courses that blended beginning and intermediate algebra into two, 8-week piloted courses. Second, by using a combined beginning and intermediate algebra book, the students' repetition of previously learned material was reduced, thereby enabling more class time for the new material in Intermediate Algebra.
Research-Based Integrated Reading and Writing Course Development
Author: Pierce, Calisa A.
Date: 2017-00-00
Type: Journal Articles
Subjects: Evidence Based Practice,Community Colleges,Integrated Curriculum,College Programs,Reading Instruction,Writing Instruction,Practicums,Developmental Programs,Formative Evaluation,Best Practices,Instructional Design,Curriculum Development,Program Effectiveness,Qualitative Research,Statistical Analysis

With the continuing national emphases on acceleration and completion, an integrated reading and writing course (a combined developmental reading and developmental writing course, with all levels compressed into a single course) is one way to move students more quickly and efficiently through the developmental sequence while still maintaining standards of good instruction. Reading, Reasoning, and Writing, the course designed in this 2011-2012 practicum project, drew on recommended instructional practices for developmental education and principles of adult learning theory such as active learning, collaborative learning, mastery learning, contextual learning, and learning community design (Baker, Hope, &amp; Karandjeff, 2009; Boroch et al., 2010; Boylan &amp; Saxon, 2012; Schwartz &amp; Jenkins, 2007; Simpson, Stahl, &amp; Francis, 2004). Today, the course, now named Integrated Reading and Writing, is still an integral part of BridgeValley Community and Technical College's curriculum.
Psychological Security and Self-Efficacy among Syrian Refugee Students inside and outside the Camps
Author: ALharbi, Bassam H. M.
Date: 2017-12-00
Type: Journal Articles
Subjects: Security (Psychology),Self Efficacy,Refugees,Foreign Countries,Secondary School Students,Grade 8,Grade 9,Grade 10,Self Concept Measures,Affective Measures,Place of Residence,Public Schools

The present study aimed to identify the degree of psychological security and self-efficacy among the Syrian refugee students inside and outside the camps. The sample consisted of 600 students from Syrian refugees inside and outside the camps in the second semester of the academic year 2014-2015. Scales for psychological security and self-efficacy were implemented to answer the study questions. Results indicated low degree of psychological security for Syrian refugee students inside the camps but moderate for those outside the camps. The results also showed low degree of self-efficacy among Syrian refugee students inside and outside the camps. The results indicated also statistically significant correlation between psychological security and self-efficacy.