ethanporcaro/eric-fulltext-meta huggingface dataset:
Analyzing Item Generation with Natural Language Processing Tools for the &quot;TOEIC&quot;® Listening Test. Research Report. ETS RR-17-52
Author: Yoon, Su-Youn
Date: 2017-12-00
Type: Journal Articles
Subjects: Natural Language Processing,Language Tests,Item Banks,Pilot Projects,English (Second Language),Second Language Learning,Teaching Methods,Listening Comprehension Tests,Surveys,Interviews,Regression (Statistics)

In this study, we developed assistive tools and resources to support TOEIC® Listening test item generation. There has recently been an increased need for a large pool of items for these tests. This need has, in turn, inspired efforts to increase the efficiency of item generation while maintaining the quality of the created items. We aimed to address this challenge by creating a set of automated tools and resources that support item generation: an automated system that retrieves appropriate real-world videos, a list of vocabulary tagged with established difficulty levels, and a tool that suggests words and phrases that are similar in distribution to a given word (word similarity tool). These tools and resources were designed to help item writers by providing initial ideas, authentic language, and support for adjusting the variety and complexity of vocabulary in their items. To evaluate the impact of these resources on the efficiency of item generation, seven item writers created listening items using our tools. All tools were considered useful, and the word similarity tool in particular was rated the most useful. The tools are currently applied to English item generation for the TOEIC Listening test, but the method is generic and applicable to other languages.
Investigating the Attitude of PCG University Students towards Their Parents and towards Marriage
Author: Görkem, Anil
Date: 2018-00-00
Type: Journal Articles
Subjects: Foreign Countries,College Students,Counselor Training,Parent Child Relationship,Marriage,Student Attitudes,Gender Differences,Correlation,Student Surveys,Statistical Analysis,Mothers,Fathers

The aim of this study is to investigate students in classes 1-2-3-4, studying Pyschological Guidance and Counselling (PGC) at a university in T.R.N.C. (Turkish Republic of Northern Cyprus) in their relationships with their parents and their attitudes towards marriage. The study includes 325 students and was conducted during the spring of 2015-2016. &quot;Personal information forms&quot;, &quot;Inönü Marriage Attitudes Scale, &quot;Attitudes towards Parents Scale I and II&quot; have been used in order to collect research data. The viewpoints of the student PCG group, who work with child and family, have been particularly focused in connection to all of these. The findings revealed that female students' attitudes towards their mothers are better compared to that of male students. However, in terms of gender, no connection was found between attitude towards their father and their attitude towards marriage. As female PCG students' attitude towards their mother increases, so too has their attitude towards their father and their attitudes towards marriage. For the men whose parents are still together (not separated), there is no link between their strong attiudes towards their parents and their attitude towards marriage. It has come to light that female students, whose parents are separated, have a stronger attitude towards their mother and father. No connection was found between the students' parents being married or having separate living arrangements and the students' gender. Male students with parents who live separately have a stronger approach towards their their mother and father.
Preparing Teachers for Diversity: A Literature Review and Implications from Community-Based Teacher Education
Author: Yuan, Huanshu
Date: 2018-00-00
Type: Journal Articles
Subjects: Literature Reviews,Community Education,Teacher Education Programs,Multicultural Education,Knowledge Base for Teaching,Achievement Gap,Demography,Student Diversity,Diversity (Faculty),Preservice Teacher Education,Indigenous Knowledge,Global Approach,Cultural Relevance

This study reviewed current issues in preparing qualified teachers for increasing diverse student populations in the U.S. and in other multicultural and multiethnic countries. Based on the framework of community-based and multicultural teacher education, this literature review paper analyzed issues and problems existed in the current curriculum, content, and practicum in traditional teacher education programs in addressing cultural knowledge and competence of preparing pre-service teachers in a multicultural society. Drawing from implications from community-based teacher education, this paper proposed several suggested strategies to reform traditional teacher education programs to meet the needs from multicultural and multiracial student populations and instructional context.
A Multimodal Dialog System for Language Assessment: Current State and Future Directions. Research Report. ETS RR-17-21
Author: Suendermann-Oeft, David
Date: 2017-12-00
Type: Journal Articles
Subjects: Language Tests,Computer Assisted Testing,Artificial Intelligence,English (Second Language),Speech,Scoring,Language Proficiency,Recognition (Psychology),Computer System Design

We present work in progress on a multimodal dialog system for English language assessment using a modular cloud-based architecture adhering to open industry standards. Among the modules being developed for the system, multiple modules heavily exploit machine learning techniques, including speech recognition, spoken language proficiency rating, speaker recognition, and the scoring of behaviors in multimodal data streams.
An Empirical Investigation of the Potential Impact of Item Misfit on Test Scores. Research Report. ETS RR-17-60
Author: Kim, Sooyeon
Date: 2017-12-00
Type: Journal Articles
Subjects: Scores,Test Items,Test Bias,International Assessment,Geographic Regions,Item Analysis,Computation,Item Banks,Comparative Analysis,Statistical Analysis,Pretesting,Regression (Statistics)

In this study, we examined the potential impact of item misfit on the reported scores of an admission test from the subpopulation invariance perspective. The target population of the test consisted of 3 major subgroups with different geographic regions. We used the logistic regression function to estimate item parameters of the operational items based on the empirical data accumulated over 3 years. A new set of item parameter estimates derived using the data from each subgroup separately was compared to the original (i.e., operational) item parameter estimates to assess the degree of item misfit due to subgroup memberships. Using the new set of item parameter estimates for each subgroup, we also updated the conversion tables, which were derived from the original item parameter estimates, and compared them to their original conversions to determine whether score invariance was achieved at the scaled score level. Score invariance was not absolutely achieved. Even so, the magnitude of reported score differences (systematic error or bias) caused by subgroup dependence was still smaller than the standard error of measurement (random error) of the test. This study suggests a practical remedy for enhancing the level of score invariance of the test.
An Investigation of the &quot;e-rater&quot;® Automated Scoring Engine&apos;s Grammar, Usage, Mechanics, and Style Microfeatures and Their Aggregation Model. Research Report. ETS RR-17-04
Author: Chen, Jing
Date: 2017-12-00
Type: Journal Articles
Subjects: Computer Assisted Testing,Scoring,Automation,Essay Tests,Natural Language Processing,Grammar,Language Usage,Multiple Regression Analysis,Writing Evaluation,Weighted Scores,Prediction,College Entrance Examinations,Language Tests,Language Proficiency,Correlation,Test Validity

Automated essay scoring (AES) generally computes essay scores as a function of macrofeatures derived from a set of microfeatures extracted from the text using natural language processing (NLP). In the &quot;e-rater&quot;® automated scoring engine, developed at &quot;Educational Testing Service&quot; (ETS) for the automated scoring of essays, each of four macrofeatures (&quot;grammar,&quot; &quot;usage,&quot; &quot;mechanics,&quot; and &quot;style&quot; [GUMS]) is computed from a set of microfeatures. Statistical analyses reveal that some of these microfeatures might not explain much of the variance in human scores regardless of the writing tasks. Currently, the microfeatures in the same macrofeature group are equally weighted to produce the macrofeature score. We propose an alternative weighting scheme that gives higher weights to the microfeatures that are more predictive of human scores in each macrofeature group. Our results suggest that even though there is negligible difference between the proposed and the current equal weighting schemes and the current model in terms of the prediction of human scores and the correlation with external measures, our scheme improves the consistency of the resultant macrofeature scores across writing tasks to a considerable extent.
Articulation of Cut Scores in the Context of the Next-Generation Assessments. Research Report. ETS RR-17-34
Author: Kannan, Priya
Date: 2017-12-00
Type: Journal Articles
Subjects: Regression (Statistics),Cutting Scores,Prediction,Accuracy,Correlation,Simulation,Statistical Bias,Grades (Scholastic),Academic Achievement,Accountability,Educational Legislation,Federal Legislation,Elementary Secondary Education,Mathematics Tests,Evaluation Methods,Multivariate Analysis

Logistic regression (LR)-based methods have become increasingly popular for predicting and articulating cut scores. However, the precision of predictive relationships is largely dependent on the underlying correlations between the predictor and the criterion. In two simulation studies, we evaluated the impact of varying the underlying grade-level correlations on the resultant bias in cut scores articulated using the LR method. In Study 1, we compared different articulation methods (LR and equipercentile smoothing), and in Study 2, we evaluated different criteria for linking (e.g., adjacent grade or end of course). The collective results indicate that as correlations became smaller, cut scores articulated using LR-based predictions became increasingly biased when compared to a true value obtained under perfect correlation. The predicted values are significantly biased for lower achievement levels, irrespective of the linking criteria used. Results from these studies suggest that the LR method must be used with caution, particularly when articulating cut scores for multiple achievement levels.
A Statistical Procedure for Testing Unusually Frequent Exactly Matching Responses and Nearly Matching Responses. Research Report. ETS RR-17-23
Author: Haberman, Shelby J.
Date: 2017-12-00
Type: Journal Articles
Subjects: Student Evaluation,Testing,Item Response Theory,Maximum Likelihood Statistics,Testing Problems,Cheating,Evaluation Methods,Probability,Identification,Mathematical Formulas,Statistical Analysis

In investigations of unusual testing behavior, a common question is whether a specific pattern of responses occurs unusually often within a group of examinees. In many current tests, modern communication techniques can permit quite large numbers of examinees to share keys, or common response patterns, to the entire test. To address this issue, statistical methods are provided to identify examinees in a test with answers that exactly match and to assess whether such exact matches are unusual. In addition, methodology is provided to identify examinees with response patterns unusually similar to circulated keys. Application is made to a testing program.
A Tale of Two Models: Sources of Confusion in Achievement Testing. Research Report. ETS RR-17-44
Author: Reckase, Mark D.
Date: 2017-00-00
Type: Journal Articles
Subjects: Models,Achievement Tests,Test Results,Test Construction,Computer Assisted Testing,Item Response Theory,Psychology,Measurement,Educational Testing

A common interpretation of achievement test results is that they provide measures of achievement that are much like other measures we commonly use for height, weight, or the cost of goods. In a limited sense, such interpretations are correct, but some nuances of these interpretations have important implications for the use of achievement test results. This paper will contrast two different theoretical underpinnings for the interpretation of educational assessments results as measurements. One of these theoretical views comes from initial attempts in psychology to measure the amount of a trait that was exhibited by a person. The other theoretical view comes from early work in education to measure how much of a desired curriculum was acquired by students. At times, these views conflict with each other and lead test developers and policy makers to ask for the impossible. After summarizing the two theoretical positions, the areas of conflict will be discussed. Finally, some recommendations will be given for what can be done to clarify the issues and minimize the problems that result from using conflicting theoretical frameworks.
An Information-Correction Method for Testlet-Based Test Analysis: From the Perspectives of Item Response Theory and Generalizability Theory. Research Report. ETS RR-17-27
Author: Li, Feifei
Date: 2017-06-00
Type: Journal Articles
Subjects: Item Response Theory,Generalizability Theory,Tests,Error of Measurement,Test Items,Methods,Test Theory,Test Construction,Test Length,Reading Tests,Grade 3,Statistical Analysis,Factor Analysis

An information-correction method for testlet-based tests is introduced. This method takes advantage of both generalizability theory (GT) and item response theory (IRT). The measurement error for the examinee proficiency parameter is often underestimated when a unidimensional conditional-independence IRT model is specified for a testlet dataset. By using a design effect ratio composed of random variances that can be easily derived from GT analysis, it becomes possible to adjust the underestimated measurement error from the unidimensional IRT models to a more appropriate level. In this paper, it is demonstrated how the information-correction method can be implemented in the context of a testlet design. Also, through the simulation study, it is shown that the underestimated measurement errors from IRT estimates can be adjusted to the appropriate level despite the varying magnitudes of local item dependence (LID), testlet length, balance of testlet length, and number of item parameters in the model. The real data example provides more details about when and how the information-correction method should be used in a test analysis. Estimation by the information-correction method should be adequate for practical work, given the robustness of the variance ratio.
Conceptualizing the Use of Translanguaging in Initial Content Assessments for Newly Arrived Emergent Bilingual Students. Research Report. ETS RR-17-07
Author: Lopez, Alexis A.
Date: 2017-12-00
Type: Journal Articles
Subjects: Bilingual Students,Bilingual Education,Immigrants,Code Switching (Language),Translation,English,Language Proficiency,Language Usage,Test Construction,Access to Education,Computer Assisted Testing,Student Evaluation,Testing

&quot;Translanguaging&quot; refers to the flexible use of the bilingual repertoire. In this report, we provide a theoretical framework to support the use of translanguaging to assess the academic content knowledge of newly arrived emergent bilingual students. In this report, we argue that translanguaging offers newly arrived emergent bilingual students the opportunity to demonstrate what they know and can do even if their English language and literacy skills are not fully developed. We propose two principles for incorporating translanguaging into initial content assessments for late-arriving emergent bilingual students, and we theorize how translanguaging can be used within an assessment context by discussing test design applications and considerations. We also discuss some challenges when incorporating translanguaging within an assessment context.
Continuing a Culture of Evidence: Assessment for Improvement. Research Report. ETS RR-17-08
Author: Russell, Javarro
Date: 2017-12-00
Type: Journal Articles
Subjects: Evidence Based Practice,Evidence,Educational Testing,Educational Improvement,Evaluation Problems,Evaluation Needs,Trend Analysis,Accountability,Barriers,Higher Education,Educational Practices,Research Reports

From 2006 to 2008, Educational Testing Service (ETS) produced a series of reports titled &quot;A Culture of Evidence,&quot; designed to capture a changing climate in higher education assessment. A decade later, colleges and universities already face new and different challenges resulting from societal, technological, and scientific influences. This new series of reports describes current issues and trends in assessment and how they relate to the needs of higher education institutions. In this report, we discuss the shift from accountability-based assessment to improvement-based assessment. When conducted for improvement, assessment is more inclusive of institutional faculty and staff, more informative to colleges and universities, and--it is hoped--focused on enhancing instruction and learning. However, these positive outcomes do not come without challenges. Ultimately, assessment-for-improvement has implications for many of the people, processes, and tools used to measure student learning
An Examination of a Pre-Service Mathematics Teacher&apos;s Mental Constructions of Relationships in a Right Triangle
Author: Koyunkaya, Melike Yigit
Date: 2018-00-00
Type: Journal Articles
Subjects: Mental Computation,Preservice Teacher Education,Preservice Teachers,Geometric Concepts,Secondary School Mathematics,Interviews,Student Teacher Attitudes,Schemata (Cognition),Concept Teaching,Knowledge Level,Mathematics Achievement,Models

Students need to construct strong knowledge of angles as well as relationships between angles and side lengths in a triangle to succeed in geometry. Although many researchers pointed out the importance of angles and angle-related concepts, students and teachers have had limited knowledge of these concepts. This study is a part of a larger study, and examines pre-service secondary mathematics teachers' (PSMTs) mental constructions of relationships between angles and side lengths in a right triangle (RASR). The Action-Process-Object-Schema (APOS) learning theory was used as the theoretical lens and clinical interview methodology was used as the methodology in the study. The study was conducted with four PSMTs, but it focuses on one of PSMTs, Linda, who revealed evidence of schema for RASR. As a result of fine-grained analysis of Linda's responses to the tasks, this article reports the mental constructions enough to develop a schema for RASR. The model describes that schema for 2-line angles, right triangles and relationships between opposite angles and side lengths and process level for some relationships including &quot;Pythagorean Theorem&quot;, &quot;The hypotenuse is always the longest side in a right triangle&quot;, &quot;Special right triangles&quot;, &quot;Complementary Angles&quot;, and &quot;Triangle inequalities&quot; are enough to construct a schema for RASR.
Challenges and Opportunities in Achieving the National Postsecondary Degree Attainment Goals. Policy Information Report and ETS Research Report Series No. RR-17-38
Author: Nettles, Michael T.
Date: 2017-00-00
Type: Journal Articles
Subjects: Postsecondary Education,Educational Attainment,Educational Objectives,Educational Policy,Educational Trends,Trend Analysis,Race,Ethnicity,Minority Group Students,Sex,Associate Degrees,Bachelors Degrees,African American Students,American Indian Students,Alaska Natives,Hispanic American Students,Prediction,Equal Education,Disproportionate Representation,Labor Force,Unemployment,Employment,Demand Occupations,National Standards,Certification,Racial Differences,Gender Differences,Enrollment,Time to Degree,Stopouts

In 2009, at the end of the 12-month Great Recession in the United States, the U.S. government established a college degree attainment goal for 60% of 25- to 34-year-olds to earn an associate's or bachelor's degree by the year 2020. In the same year, Lumina Foundation set a similar goal for 60% of 25- to 64-year-olds to earn a high-quality certificate, associate's degree, or bachelor's degree by the year 2025. Both the U.S. government's and Lumina Foundation's goals intend to place the United States in a leadership position in the global massification of postsecondary education that has evolved over the past few decades and to address the growing labor market demands for postsecondary education and training. This report provides a view of current progress and forecasts of the nation's long-term progress toward achieving the goals. The analyses in this report represent college degree attainment performance of the U.S. population by race/ethnicity and gender and project through 2060. The projections reveal that neither goal is expected to be reached by target dates for the targeted adult populations, but the associate's and bachelor's degree attainment rates of the Asian American population are already beyond the 60% target and the White population, overall, is approaching at a pace to arrive a few years beyond the target year 2020 established by the U.S. government and 2025 set by Lumina Foundation. The projections for the African American, American Indian/Alaska Native, and Hispanic populations are not promising. Unless aggressive actions are taken to address the inequalities in each level and type of degree, and especially bachelor's and higher degrees, not only will the three underserved population groups (African American, American Indian/Alaska Native, and Hispanic) fail to reach the goals in the foreseeable future, but also the progress that they make could be overrepresented by lower status degrees and certificates and in turn lower status occupations. Each of these three underrepresented population groups may require targeted and tailored initiatives to make substantive progress toward a larger share being college and career ready and ultimately persisting toward and attaining college degrees. The report includes data and analyses that could be useful in designing interventions.
Continuing a Culture of Evidence: Expanding Skills in Higher Education. Research Report. ETS RR-17-09
Author: Olivieri, María Elena
Date: 2017-00-00
Type: Journal Articles
Subjects: Evidence Based Practice,Higher Education,Educational Assessment,Thinking Skills,College Graduates,Trend Analysis,Graduation Requirements,Job Skills,Skill Analysis,Evaluation Needs,Educational Researchers,Public Policy,Organizational Culture

From 2006 to 2008, Educational Testing Service (ETS) produced a series of reports titled &quot;A Culture of Evidence,&quot; designed to capture a changing climate in higher education assessment. A decade later, colleges and universities face a new set of challenges resulting from societal, technological, and other influences, leading to a need to augment the skills expected of college graduates. This report is one of a new series of reports by ETS describing current issues and trends in higher education with a focus on assessment. In this report, we discuss shifts in cognitive and noncognitive skills expected of college graduates in response to new educational and occupational demands. In particular, we describe these demands, the factors influencing them, and the impact of these factors on the instruction and assessment of skills needed for success in higher education and the workforce.