ethanporcaro/eric-fulltext-meta huggingface dataset:
Development of the Behaviorally Anchored Rating Scales for the Skills Demonstration and Progression Guide. Research Report. ETS RR-18-24
Author: Klieger, David M.
Date: 2018-12-00
Type: Journal Articles
Subjects: Test Construction,Behavior Rating Scales,Job Performance,Job Skills,Employees,Occupational Information,Personnel Evaluation,Work Ethic,Communication Skills,Resilience (Psychology),Problem Solving,Responsibility,Teamwork,Citizenship,Leadership

In this research, we developed behaviorally anchored rating scales (BARS) to evaluate the job performance of Zone Three jobs requiring middle-level skills and prior education ranging from vocational training to an associate's degree as well as work-related skills or experience. We ultimately identified 7 relevant job performance dimensions mainly based on prior research literature. The dimensions are thought to reflect socioemotional constructs (soft skills, &quot;21st-century skills&quot;) considered vital to success today in Zone Three jobs. Managers of Zone Three employees helped us develop the final behavioral statements to anchor the 6 BARS points by generating approximately 430 critical incidents on which we based the initial behavioral statements that we wrote. Another group of managers confirmed the relevancy of the statements to the Zone Three workers they supervised, retranslated statements back into dimensions to confirm the dimensions to which they belong, and provided ratings of the effectiveness level that each statement represented. We modified statements to fill any gaps in effectiveness level coverage. Then, we asked leading experts in assessment to confirm the adequacy of the final statements, and after they did so, we created the final anchored scales. Both past literature and our own analyses indicate that these BARS generalize across economic sectors and thus potentially can provide substantial value to organizations that wish to assess, in an efficient and cost-effective manner, the performance of middle-skills employees with many different types of possible job descriptions.
Providing a Context for Interpreting Predictions of Job Performance. Research Report. ETS RR-18-38
Author: Dorans, Neil J.
Date: 2018-12-00
Type: Journal Articles
Subjects: Job Performance,Scores,Data Interpretation,Statistical Distributions,Accuracy,Scaling,Prediction,Occupational Tests

A distinction is made between scores as measures of a construct and predictions of a criterion or outcome variable. The interpretation attached to predictions of criteria, such as job performance or college grade point average (GPA), differs from that attached to scores that are measures of a construct, such as reading proficiency or knowledge about a domain (e.g., physics). In contrast to a score whose meaning is tied to the fidelity with which it measures a construct, the meaning of a prediction is tied to the statistical relationship of its scale to that of the criterion. A contrived example that might be seen in an employment setting is used to illustrate potential misinterpretations associated with making the predicted score distribution look like the criterion score distribution it is predicting.
The Development of a Quadratic Functions Learning Progression and Associated Task Shells. Research Report. ETS RR-18-47
Author: Graf, Edith Aurora
Date: 2018-12-00
Type: Journal Articles
Subjects: Learning Processes,Mathematical Concepts,Equations (Mathematics),Mathematics Instruction,Algebra,Problem Solving,Mathematics Skills,Task Analysis,Word Problems (Mathematics)

In this report, we discuss background research on the development of student understanding of quadratic functions and present a provisional learning progression for quadratic functions and quadratic equations. We also describe task shells that are linked to the learning progression. The learning progression and task shells can be used as a starting point to develop tasks that assess student standing with respect to the theory of development. The intention is that this report should find an audience in both researchers and test developers.
A Robust Microservice Architecture for Scaling Automated Scoring Applications. Research Report. ETS RR-18-14
Author: Madnani, Nitin
Date: 2018-12-00
Type: Journal Articles
Subjects: Automation,Scoring,Computer System Design,Open Source Technology,Computer Oriented Programs,Tests

We present a microservice architecture for large-scale automated scoring applications. Our architecture builds on the open-source Apache Storm framework and facilitates the development of robust, scalable automated scoring applications that can easily be extended and customized. We demonstrate our architecture with an application for automated content scoring.
A Simulation Study to Compare Nonequivalent Groups with Anchor Test Equating and Pseudo-Equivalent Group Linking. Research Report. ETS RR-18-08
Author: Lu, Ru
Date: 2018-12-00
Type: Journal Articles
Subjects: Equated Scores,Comparative Analysis,Test Items,Background,Accuracy,Groups,Ability,Differences

In this paper we compare the newly developed pseudo-equivalent groups (PEG) linking method with the linking methods based on the traditional nonequivalent groups with anchor test (NEAT) design and illustrate how to use the PEG methods under imperfect equating conditions. To do this, we proposed a new method that combines the features of PEG linking and NEAT equating (referred as PEGAT) and compared it with NEAT and PEG. PEG mainly uses test takers' background variables to create PEG and then links scores on different forms whereas NEAT equating adjusts group differences in ability through the anchor test scores. The proposed method, PEGAT, uses background variables and anchor scores to adjust group ability differences. Using simulated data, these 3 linking methods were compared in 2 equating scenarios: small and large group difference in ability. The simulation design was based on real data on a test in operation. The test scores and the background variables were assumed to have a multivariate multinomial distribution. A log linear model was used to manipulate and produce simulated data. For PEG and PEGAT linking, 3 different sets of background variables were manipulated to study the impact of correlation strength of background variables to the total scores. Our results showed that NEAT linking outperformed PEG linking when the group ability difference was large, but that NEAT linking could be improved by incorporating the PEG adjustment procedure based on background variables. When the groups were similar in ability, PEG linking produced comparable results to NEAT. This finding justifies the use of PEG linking when a good anchor test is not available as well as the use of PEGAT when a good anchor is available but needs to be strengthened by background variables.
A Review of Subscore Estimation Methods. ETS RR-18-17
Author: Fu, Jianbin
Date: 2018-12-00
Type: Journal Articles
Subjects: Scores,Evaluation Methods,Item Response Theory,Test Items,Testing Problems,Accuracy,Test Reliability

Various subscore estimation methods that use auxiliary information to improve subscore accuracy and stability have been developed. This report provides a review of various subscore estimation methods described in the literature. The methodology of each method is described, then research studies on these subscore estimation methods are summarized. Comments on the methods and suggestions for future areas of research are provided, and preliminary guidelines for using subscore estimation methods in practice are recommended.
Assessing Civic and Intercultural Competency in Higher Education: The ETS &quot;HEIghten®&quot; Approach. Research Report. ETS RR-18-23
Author: Liu, Ou Lydia
Date: 2018-12-00
Type: Journal Articles
Subjects: Cultural Awareness,Interdisciplinary Approach,Global Approach,Transfer of Training,Education Work Relationship,Pilot Projects,Competence,Citizenship Education,Higher Education,Outcomes of Education,Test Items,Test Construction,Critical Thinking,Numeracy,Communication Skills,Written Language,Definitions,Measurement,Self Control,Metacognition,Self Efficacy,Interpersonal Competence,Undergraduate Students,Correlation,Scores

Economic globalization and interdisciplinary advancements have increased the demand for college graduates to possess transferable skills that would allow them to contribute effectively to the modern workforce. In particular, transferable competencies such as civic competency and intercultural competency are critical for colleges to prepare responsible citizens and productive workers. Despite the recognized importance, the choice and quality of assessment for such competencies have been fairly limited due to the challenges in defining such complex, multidimensional constructs and identifying item types that can adequately assess them. In this report, we describe the principles we followed to operationalize definitions for civic competency and intercultural competency and the process we followed to design assessments for these 2 competencies. Findings from a large-scale pilot test are reported. Results showed that these multidimensional constructs can be adequately assessed and that there is room for students to improve in these areas. Implications for higher education institutions on how to promote these critical competencies are discussed.
A Generalizability Theory Study to Examine Sources of Score Variance in Third-Party Evaluations Used in Decision-Making for Graduate School Admissions. ETS GRE® Board Research Report. ETS GRE®-18-03. ETS RR-18-37
Author: McCaffrey, Daniel F.
Date: 2018-12-00
Type: Journal Articles
Subjects: Generalizability Theory,Scores,College Admission,Admission Criteria,Decision Making,Graduate Study,Graduate Students,Error of Measurement,Interrater Reliability,Bias

Scores from noncognitive measures are increasingly valued for their utility in helping to inform postsecondary admissions decisions. However, their use has presented challenges because of faking, response biases, or subjectivity, which standardized third-party evaluations (TPEs) can help minimize. Analysts and researchers using TPEs, however, need to be mindful of the potential for constructirrelevant differences that may arise in TPEs due to differences in evaluators' rating approaches, which introduces measurement error. Research on sources of construct-irrelevant variance in TPEs is scarce. We address this paucity by conducting generalizability theory (G theory) analyses using TPE data that informs postsecondary admissions decisions. We also demonstrate an approach to assess the size of interevaluator variability and conduct a decision study to determine the number of evaluators necessary to achieve the desired generalizability coefficient. We illustrate these approaches using a TPE whereby applicants select their evaluators, leading to a situation where most evaluators solely rate one applicant. We conclude by presenting strategies to improve the design of TPEs to help increase confidence in their use.
Assessing Elementary Teachers&apos; Content Knowledge for Teaching Science for the &quot;ETS®&quot; Educator Series: Pilot Results. Research Report. ETS RR-18-20
Author: Mikeska, Jamie N.
Date: 2018-12-00
Type: Journal Articles
Subjects: Elementary School Teachers,Pedagogical Content Knowledge,Elementary School Science,Preservice Teachers,Beginning Teachers,Science Tests,Test Items,Difficulty Level,Test Reliability,Teacher Attitudes,Knowledge Level,Test Construction,Individual Characteristics

The purpose of this report is to examine the performance of assessment items designed to measure elementary teachers' content knowledge for teaching (CKT) science as part of the ETS® Educator Series. The Elementary Education: CKT Science assessment is 1 component of licensure examination through the PRAXIS® assessments. The Elementary Education: CKT Science assessment is designed to determine whether kindergarten through 6th-grade elementary teacher candidates have the essential content knowledge needed for teaching elementary science as new teachers at the entry level. This report provides information about the development of the Elementary Education: CKT Science framework and the associated assessment items. The main part of the report focuses on the evidence gathered when piloting 104 CKT science assessment items with 417 preservice and novice elementary teachers. This evidence included the following: (a) how these new CKT science assessment items function, including their item difficulties and discrimination, and reliabilities of the pilot forms and of the classification of examinees; (b) how teachers perceive the importance and relevance of these new assessment items; (c) how teachers' performances on these items relate to their background characteristics and professional and academic preparation; and (d) how teachers' performances on the CKT science items compare to their performances on an assessment designed to measure only their science subject matter knowledge.
Anchored Graphical Representations: A Graphical Alternative to Traditional Just Qualified Candidate Descriptors for Licensure Tests. Research Report. ETS RR-18-40
Author: Kannan, Priya
Date: 2018-12-00
Type: Journal Articles
Subjects: Licensing Examinations (Professions),Visual Aids,Standard Setting,Mathematics Teachers,Social Studies,Teacher Certification,Elementary School Teachers,Scores

A well-constructed just qualified candidate (JQC) description is needed to arrive at a reasonable passing score for licensure tests. Traditionally, such descriptions consist of a list of knowledge and skill statements without sufficient context to internalize its intended meaning, allowing the standard-setting panelists to make idiosyncratic interpretations. In a series of studies, we evaluated an alternative JQC description called anchored graphical representation (AGR). AGRs are intended to provide both text and visual organizers by each tested domain to contextualize the meaning of the JQC. In Study 1, 22 mathematics educators participated in a mock standard-setting study and were randomly assigned to conditions in which they used either the AGR-based or the text-based definitions in making standard-setting judgments. In Study 2, 17 social studies educators were randomly assigned to either the AGR-based or text-based condition. While the overall passing score did not noticeably vary between the conditions, consistent with our expectations, results from both studies showed that the domain passing scores for the AGR groups reflected the relative importance of each test domain for the JQC and that the inter-panelist variability was smaller for the AGR groups. Collectively, our results indicate that use of the AGR resulted in standard-setting ratings that were more aligned with the panelists' differential expectations of JQCs' performance for each domain, and the results add to the procedural validity evidence supporting the reasonableness of standard-setting recommendations.
An Investigation of the Predictive Validity of the &quot;TOEFL iBT&quot;® Test at an English-Medium University in Turkey. Research Reports. TOEFL-RR-83. ETS RR-18-43
Author: O'Dwyer, John
Date: 2018-12-00
Type: Journal Articles
Subjects: Predictive Validity,Computer Assisted Testing,Grade Point Average,Language of Instruction,English (Second Language),Second Language Learning,Correlation,Scores,Language Proficiency,Benchmarking,College Faculty,English for Academic Purposes,College Freshmen,Student Evaluation,Foreign Countries,Language Tests,Course Content,Teacher Attitudes,Academic Achievement

This study reports on an investigation of the predictive validity of the TOEFL iBT®test in an English-medium institution (EMI) in a non-target-language context, namely, Turkey. The relationship between TOEFL iBT scores and academic performance was explored in a cohort of 286 undergraduate students, as was the TOEFL iBT's relationship with an institutional English proficiency exam (Certificate of Proficiency in English [COPE]) used as a benchmark for faculty entry. Performance measures included scores on TOEFL iBT and COPE, grade point averages (GPAs) for content and English for academic purposes (EAP) courses over 2 freshman semesters, and freshman language instructor evaluations of students' freshman EAP performance. Correlations between test scores confirmed a moderate to moderately high predictive validity for content course GPAs and English-language course GPAs, respectively, and for the TOEFL iBT, particularly in technical fields. Instructor evaluations of student performance supported the findings, with fewer deficiencies in academic English skills for students with higher scores on TOEFL iBT. The study concludes that the TOEFL iBT's predictive validity is on par with the institution's own proficiency test and represents a solid performance measure for use in EMIs in a non-target-language context.
Support Receipt: Effect on Postsecondary Success of Students with Learning Disabilities
Author: Newman, Lynn A.
Date: 2019-02-00
Type: Journal Articles
Subjects: College Students,Learning Disabilities,Academic Persistence,Graduation Rate,Academic Accommodations (Disabilities),Accessibility (for Disabled),Access to Education,Student Personnel Services,Academic Achievement,Academic Support Services

In contrast to the increase in college enrollment rates of youth with learning disabilities (LD), graduation rates have remained stagnant and low. Using propensity methods, this study examined the effect of disability-specific and universally available support receipt on the college perseverance and completion of students with LD. Based on secondary analysis of National Longitudinal Transition Study 2 (NLTS2), findings indicate that students who received supports--those available to the full student body and/or disability-specific supports--were more likely to persist in, and successfully complete, 2-year or 4-year college. Implications include that transition staff need to ensure students not only are prepared to seek disability supports once on campus, but that equal emphasis should be placed on helping students access supports available to the full student body.
Postsecondary Education-Focused Transition Planning Experiences of English Learners with Disabilities
Author: Trainor, Audrey A.
Date: 2019-02-00
Type: Journal Articles
Subjects: Postsecondary Education,Transitional Programs,Educational Planning,English Language Learners,Bilingual Students,Disabilities,Individualized Education Programs,Expectation,Educational Attainment,High School Graduates

Transition planning is particularly important for dually identified English learners with disabilities, who frequently face additional challenges to postsecondary education success. This study examined postschool expectations, transition planning experiences, and supports of a nationally representative sample of English learners with disabilities, based on secondary analysis of the National Longitudinal Transition Study (NLTS) 2012. Results demonstrated that these students' experiences were similar to other students with disabilities except that, according to parents, the transition component of the individualized education program (IEP) was likely to be developed by school personnel, with little input from students and family members, and necessary information about careers and financial aid was lacking. These findings underscore the intersectional identities and related experiences of this population and the implications for policy, transition education, and school services.
The Logic Underlying a Research-Based College Access Program: Depicting the Theory of Change of the Princeton University Preparatory Program. Research Report. ETS RR-18-07
Author: Millett, Catherine M.
Date: 2018-00-00
Type: Journal Articles
Subjects: Access to Education,College Preparation,High Achievement,Low Income Students,High School Students,Models,Logical Thinking,Theories,Higher Education

The Princeton University Preparatory Program (PUPP) is a high-touch, intensive college access program aimed at providing a multiyear college preparation experience to high-achieving, low-income students in the Princeton, New Jersey, area. This report presents the theory of change and logic model of PUPP, detailing the theoretical basis for the expectation that PUPP inputs, resources, and activities will lead to the expected program outcomes, which include that program participants gain admittance to and graduate from selective colleges and universities providing comprehensive financial aid in the form of grants and scholarships.
The Validity of GRE® General Test Scores for Predicting Academic Performance at U.S. Law Schools. Research Report. ETS RR-18-26
Author: Klieger, David M.
Date: 2018-12-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Test Validity,Scores,Predictor Variables,Academic Achievement,Law Schools,College Applicants,Legal Education (Professions),Predictive Validity,Grades (Scholastic),Test Reliability

Educational Testing Service (ETS), working with 21 U.S. law schools, evaluated the predictive validity of the GRE® General Test using a sample of 1,587 current and graduated law students. Results indicated that the GRE is a strong, generalizably valid predictor of first-year law school grades and that it provides useful information even when undergraduate grade point average already is available to predict those grades. This report also reviews the reliability of the GRE General Test that had been determined in prior research.
Automated Scoring of Nonnative Speech Using the &quot;SpeechRater&quot;? v. 5.0 Engine. Research Report. ETS RR-18-10
Author: Chen, Lei
Date: 2018-12-00
Type: Journal Articles
Subjects: Computer Assisted Testing,Scoring,Test Scoring Machines,Speech Tests,Automation,Pronunciation,Suprasegmentals,Vocabulary,Grammar,Natural Language Processing,Communicative Competence (Languages),Oral Language,Non English Speaking

This research report provides an overview of the R&amp;D efforts at Educational Testing Service related to its capability for automated scoring of nonnative spontaneous speech with the &quot;SpeechRater&quot;? automated scoring service since its initial version was deployed in 2006. While most aspects of this R&amp;D work have been published in various venues in recent years, no comprehensive account of the current state of SpeechRater has been provided since the initial publications following its first operational use in 2006. After a brief review of recent related work by other institutions, we summarize the main features and feature classes that have been developed and introduced into SpeechRater in the past 10 years, including features measuring aspects of pronunciation, prosody, vocabulary, grammar, content, and discourse. Furthermore, new types of filtering models for flagging nonscorable spoken responses are described, as is our new hybrid way of building linear regression scoring models with improved feature selection. Finally, empirical results for SpeechRater 5.0 (operationally deployed in 2016) are provided.