ethanporcaro/eric-fulltext-meta huggingface dataset:
Synthesizing Frameworks of Higher Education Student Learning Outcomes. Research Report. ETS RR-13-22
Author: Markle, Ross
Date: 2013-11-00
Type: Journal Articles
Subjects: Higher Education,Outcomes of Education,Educational Testing,Creativity,Critical Thinking,Teamwork,Communication Skills,Computer Literacy,Information Literacy,Citizenship,Daily Living Skills,Mathematics Skills,Academic Standards,Models,Career Readiness

The public, education, and workforce sectors all have expressed interest regarding the key knowledge, skills, and abilities that enable individuals to be productive members of society. Although past efforts have attempted to create frameworks of student learning outcomes, the results have varied due to different perspectives and goals. Thus, the purpose of this paper was to gather and review relevant higher education frameworks, determine their commonalities and create domains, and identify the assessments that Educational Testing Service (ETS) has developed in each of the domains. After a thorough review of the relevant frameworks, seven key domains were identified: creativity, critical thinking, teamwork, communication, digital and information literacy, citizenship, and life skills. Also discussed were the issues of education versus work contextualization and the assumption of foundational quantitative reasoning and literacy skills informing these seven domains.
Choice of Target Population Weights in Rater Comparability Scoring and Equating. Research Report. ETS RR-13-03
Author: Puhan, Gautam
Date: 2013-03-00
Type: Journal Articles
Subjects: Scoring,Equated Scores,Sampling,Accuracy,Correlation,Methods

The purpose of this study was to demonstrate that the choice of sample weights when defining the target population under poststratification equating can be a critical factor in determining the accuracy of the equating results under a unique equating scenario, known as &quot;rater comparability scoring and equating.&quot; The nature of data collection under &quot;rater comparability scoring&quot; is such that it results in a very high correlation between the anchor and total score in the new form but only a moderate correlation in the reference form. I demonstrated, using data collected under a &quot;rater comparability scoring&quot; situation, that this difference in the anchor-total correlation in the new and reference forms can have a predictable impact on the equating results based on different sample weights (i.e., the equating results are most accurate when the reference form sample is defined as the target population, least accurate when the new form sample is defined as the target population, and somewhere in the middle when the new and reference form samples are equally weighed when defining the target population).
SARA Reading Components Tests, RISE Forms: Technical Adequacy and Test Design, 2nd Edition. Research Report. ETS RR-15-32
Author: Sabatini, John
Date: 2015-12-00
Type: Journal Articles

This technical report describes the conceptual foundation and measurement properties of the Reading Inventory and Scholastic Evaluation (RISE). The RISE is a 6-subtest, Web-administered reading skills components battery. The theoretical and empirical foundations of each subtest in the battery are reviewed, as well as item designs. The results included in this report feature a vertical extension of the RISE to span Grades 5--10, psychometric analysis of parallel forms of each subtest, results of item response theory (IRT) scaling studies for each of the subtests across the entire grade span, and evaluation of differential item functioning (DIF) for gender and race/ethnicity.
Length of Textual Response as a Construct-Irrelevant Response Strategy: The Case of Shell Language. Research Report. ETS RR-13-07
Author: Bejar, Isaac I.
Date: 2013-04-00
Type: Journal Articles
Subjects: Responses,Language Usage,Natural Language Processing,Computational Linguistics,Writing Tests,College Entrance Examinations,Graduate Study,Scoring,Validity,Interrater Reliability

The paper applies a natural language computational tool to study a potential construct-irrelevant response strategy, namely the use of &quot;shell language.&quot; Although the study is motivated by the impending increase in the volume of scoring of students responses from assessments to be developed in response to the Race to the Top initiative, the data for the study were obtained from the &quot;GRE&quot;™ Analytical Writing measure. The functioning of the shell detection computational tool was first evaluated by applying it to a corpus of over 200,000 issue and argument essays and by means of a study to evaluate whether the shell language score agreed with the characterization of shell by two scoring experts. It was concluded that the computational tool worked well. The tool was then used to select essays for rescoring to determine whether the presence of shell language had had an effect on the operational scores they received. We found no evidence that such an effect was present. However, we did find a leniency effect in the operational scores. That is, the essays that were rescored as part of this project received a lower score compared to the operational score. The validity implications of these results are discussed.
Weighting Test Samples in IRT Linking and Equating: Toward an Improved Sampling Design for Complex Equating. Research Report. ETS RR-13-39
Author: Qian, Jiahe
Date: 2013-12-00
Type: Journal Articles
Subjects: Item Response Theory,Test Items,Sampling,True Scores,Weighted Scores,Item Banks,Comparative Analysis,Test Results,Statistical Analysis,Computation,Equations (Mathematics)

Several factors could cause variability in item response theory (IRT) linking and equating procedures, such as the variability across examinee samples and/or test items, seasonality, regional differences, native language diversity, gender, and other demographic variables. Hence, the following question arises: Is it possible to select optimal samples of examinees so that the IRT linking and equating can be more precise at an administration level as well as over a large number of administrations? This is a question of optimal sampling design in linking and equating. To obtain an improved sampling design for invariant linking and equating across testing administrations, we applied weighting techniques to yield a weighted sample distribution that is consistent with the target population distribution. The goal is to obtain a stable StockingLord test characteristic curve (TCC) linking and a true-score equating that is invariant across administrations. To study the weighting effects on linking, we first selected multiple subsamples from a data set. We then compared the linking parameters from subsamples with those from the data and examined whether the linking parameters from the weighted sample yielded smaller mean square errors (MSE) than those from the unweighted subsample. To study the weighting effects on true-score equating, we also compared the distributions of the equated scores. Generally, the findings were that the weighting produced good results
ETS Research on Cognitive, Personality, and Social Psychology: I. Research Report. ETS RR-13-01. ETS R&amp;D Scientific and Policy Contributions Series. ETS SPC-13-01
Author: Stricker, Lawrence J.
Date: 2013-02-00
Type: Journal Articles
Subjects: Cognitive Psychology,Personality Traits,Social Psychology,Educational Research,Cognitive Ability,Response Style (Tests),Emotional Response,Emotional Intelligence,Social Cognition,Prosocial Behavior,Stereotypes,Motivation,Test Anxiety,Testing,Performance Factors,Risk,Creativity,Academic Aspiration,Academic Achievement

This is an account of a portion of the research on cognitive, personality, and social psychology at ETS since the organization's inception. The topics in cognitive psychology are the structure of abilities; in personality psychology, response styles and social and emotional intelligence; and in social psychology, prosocial behavior and stereotype threat. Research on motivation is also covered.
A Criterion to Evaluate the Individual Raw-to-Scale Equating Conversions. Research Report. ETS RR-13-05
Author: Guo, Hongwen
Date: 2013-03-00
Type: Journal Articles
Subjects: Equated Scores,Test Items,Evaluation Criteria,Error of Measurement,Scaling,Raw Scores

In this study we investigated when an equating conversion line is problematic in terms of gaps and clumps. We suggest using the conditional standard error of measurement (CSEM) to measure the scale scores that are inappropriate in the overall raw-to-scale transformation.
Contributions of a Nonprofit Educational Measurement Organization to Education Policy Research. Research Report. ETS RR-13-06. ETS R&amp;D Scientific and Policy Contributions Series. ETS SPC-13-02
Author: Coley, Richard J.
Date: 2013-04-00
Type: Journal Articles
Subjects: Nonprofit Organizations,Educational Policy,Educational Research,Access to Education,Public Policy,Governance,Educational Finance,Gender Differences,Advanced Placement Programs,Graduate Study,Elementary Secondary Education,Adult Literacy,Outcomes of Education

As part of its mission, Educational Testing Service (ETS) has conducted education policy research focused on promoting equal opportunity for all individuals from early childhood through adulthood. This report provides a sampler of this extensive work in three areas chosen to represent major historical foci: (1) Analyzing, evaluating, and informing public policy in educational governance, including school finance; teacher policy; and federal, state, and local education policy; (2) Examining differential access to educational opportunity in three areas of longstanding interest to ETS: the gender gap, advanced placement programs, and graduate education; and (3) Reporting on the educational outcomes of the U.S. population and describing the contexts for these outcomes and for the gaps in outcomes that exist among segments of the population.
Preliminary Reading Literacy Assessment Framework: Foundation and Rationale for Assessment and System Design. Research Report. ETS RR-13-30
Author: Sabatini, John
Date: 2013-12-00
Type: Journal Articles
Subjects: Reading Tests,Literacy,Models,Testing Programs,Developmentally Appropriate Practices,Preschool Education,Elementary Secondary Education,Test Construction

This report describes the foundation and rationale for a framework designed to measure reading literacy. The aim of the effort is to build an assessment system that reflects current theoretical conceptions of reading and is developmentally sensitive across a prekindergarten to 12th grade student range. The assessment framework is intended to document the aims of the assessment program, define the target constructs to be assessed, and describe the assessment designs that are aligned with the aims and constructs. This framework report is preliminary in that we are engaged in an iterative process of writing and revising the framework, based on what we learn from efforts to instantiate the ideas in new assessment designs and the results we garner from piloting novel designs. We also anticipate drafting further sections to address issues such as the scoring models and analytic plans once assessments have been designed and piloted.
The &quot;Language Muse&quot;? System: Linguistically Focused Instructional Authoring. Research Report. ETS RR-12-21
Author: Burstein, Jill
Date: 2012-10-00
Type: Journal Articles
Subjects: English Language Learners,Scaffolding (Teaching Technique),Educational Technology,Computer Oriented Programs,Feedback (Response),Natural Language Processing,Linguistics,Faculty Development,Pilot Projects,Lesson Plans,Elementary School Teachers,Middle School Teachers,Secondary School Teachers,High Schools,Preservice Teachers,Program Effectiveness

In the United States, English learners (EL) often do not have the academic language proficiency, literacy skills, cultural background, and content knowledge necessary to succeed in kindergarten through 12th grade classrooms. This leads to large achievement gaps. Also, classroom texts are often riddled with linguistically unfamiliar elements, including: unfamiliar vocabulary, idioms, complex phrases or sentences, morphologically complex words, and unfamiliar discourse relations. Lack of familiarity with linguistic elements may result in gaps in a learner's comprehension of key content. It is &quot;not&quot; feasible for teachers to develop additional curriculum for the needs of all ELs in a classroom (who often come from culturally and linguistically diverse backgrounds.) However, it &quot;is&quot; feasible for teachers to develop instructional scaffolding (support) that helps ELs and can be used with all students. To develop effective scaffolding, teachers need to be able to reliably identify linguistic features in texts that could interfere with content comprehension. &quot;Language Muse&quot;? is a web-based application designed to support teachers in the identification of linguistic features in texts and in the development of linguistically focused instructional scaffolding. With regard to system itself, we will discuss (a) the system's motivation, (b) the system's linguistic feedback and instructional authoring components, which are driven by natural language processing, and (c) the system's infrastructure for capturing teachers' system use. In addition, we will also discuss preliminary pilot study findings with three teacher professional development programs. These findings suggest that exposure to Language Muse's linguistic feedback can support teachers in the development of lesson plan scaffolds designed to address language learning needs.
The Stability of the Score Scales for the &quot;SAT Reasoning Test&quot;™ from 2005 to 2010. Research Report. ETS RR-12-15
Author: Guo, Hongwen
Date: 2012-09-00
Type: Journal Articles
Subjects: Aptitude Tests,Cognitive Tests,Thinking Skills,Equated Scores,Comparative Analysis,Critical Reading,Mathematics,Scaling,Error of Measurement,Writing (Composition),Item Response Theory,Multiple Choice Tests,Raw Scores,College Entrance Examinations

This study examines the stability of the &quot;SAT Reasoning Test&quot;™ score scales from 2005 to 2010. A 2005 old form (OF) was administered along with a 2010 new form (NF). A new conversion for OF was derived through direct equipercentile equating. A comparison of the newly derived and the original OF conversions showed that Critical Reading and Mathematics score scales have experienced, at most, a moderate upward scale drift (no greater than 5 points on average), and the drift may be explained by an accumulation of random equating errors. The Writing score scale has experienced a significant upward scale drift (11 points on average), which may be caused by sources other than random equating errors.
Investigating the Suitability of Implementing the &quot;e-rater&quot;® Scoring Engine in a Large-Scale English Language Testing Program. Research Report. ETS RR-13-36
Author: Zhang, Mo
Date: 2013-12-00
Type: Journal Articles
Subjects: Computer Assisted Testing,Computer Software,Scoring,Language Tests,Prompting,English (Second Language),Second Language Learning,Evaluators,Essays,Models,Test Items,Scores,Generalization,Writing Tests,Sample Size,Testing Programs,Comparative Analysis,Interrater Reliability,Participant Characteristics,Correlation,Simulation,Item Analysis,Foreign Countries,Regression (Statistics)

In this research, we investigated the suitability of implementing &quot;e-rater&quot;® automated essay scoring in a high-stakes large-scale English language testing program. We examined the effectiveness of generic scoring and 2 variants of prompt-based scoring approaches. Effectiveness was evaluated on a number of dimensions, including agreement between the automated and the human score and relations with criterion variables. Results showed that the sample size was generally not sufficient for prompt-specific scoring. For the generic scoring model, automated scores agreed with human raters as strongly as, or more strongly than, human raters agreed with one another for more than 97% of the prompts. The impact of substituting e-rater for the second human rater made no practically important impact on test takers' scores at both the item and total test score levels. However, neither automated scoring models nor human raters performed invariantly across all prompts or across different test countries/territories. Further investigation indicated homogeneity in the examinee population, possibly nested within test countries/territories as one potential cause of this lack of invariance. Among other limitations, findings may not be generalizable beyond the examinee population investigated in this study.
Identifying the Most Important 21st Century Workforce Competencies: An Analysis of the Occupational Information Network (O*NET). Research Report. ETS RR-13-21
Author: Burrus, Jeremy
Date: 2013-11-00
Type: Journal Articles
Subjects: College Graduates,Job Skills,Occupational Information,Databases,Competence,Job Performance,Taxonomy,Content Analysis,Componential Analysis,Problem Solving,Mechanical Skills,Helping Relationship,Cultural Literacy,Business Skills,Scientific Literacy,Civics,Information Processing,Athletics,Visual Acuity,Intelligence,Communication Skills,Teamwork,Academic Achievement,Innovation,Critical Thinking,Scores,Correlation,Wages

To identify the most important competencies for college graduates to succeed in the 21st century workforce, we conducted an analysis of the Occupational Information Network (O*NET) database. O*NET is a large job analysis operated and maintained by the U.S. Department of Labor. We specifically analyzed ratings of the importance of abilities (52 ratings), work styles (16 ratings), skills (35 ratings), and knowledge (33 ratings) to succeed in one's occupation. First, we conducted descriptive analyses. Next, data were split into 2 sets, according to the theoretical structure proposed by the O*NET content model, and principal component analyses (PCAs) were run on each dataset. The PCAs identified 15 components: problem solving, mechanical skills, service orientation, cultural literacy, business literacy, science literacy, civic literacy, information processing, athleticism, visual acuity, fluid intelligence, communication skills, teamwork, achievement/innovation, and attention to detail/near vision. Components were then ranked in importance using the mean component scores over all occupations. A comparison of this ranking with previous 21st century competencies frameworks suggested that 5 competencies stand out as important for most occupations: problem solving (e.g., complex problem solving), fluid intelligence (e.g., category flexibility), teamwork (e.g., cooperation), achievement/innovation (e.g., persistence), and communication skills (e.g., oral expression). Consistent with this conclusion, a correlation of component scores with wages found that 4 of these 5 competencies were strongly related to wages, with the exception being teamwork.
ETS Contributions to the Quantitative Assessment of Item, Test, and Score Fairness. Research Report. ETS RR-13-27. ETS R&amp;D Scientific and Policy Contributions Series. ETS SPC-13-04
Author: Dorans, Neil J.
Date: 2013-12-00
Type: Journal Articles
Subjects: Test Bias,Statistical Analysis,Test Validity,Scores,Predictive Validity,Test Items,Sex Fairness,Ethnic Groups,Sample Size

Quantitative fairness procedures have been developed and modified by ETS staff over the past several decades. ETS has been a leader in fairness assessment, and its efforts are reviewed in this report. The first section deals with differential prediction and differential validity procedures that examine whether test scores predict a criterion, such as performance in college, across different subgroups in a similar manner. The bulk of this report focuses on item level fairness, or differential item functioning, which is addressed in the various subsections of the second section. In the third section, I consider research pertaining to whether tests built to the same set of specifications produce scores that are related in the same way across different gender and ethnic groups. Limitations with the approaches reviewed here are discussed in the final section.
Repurposing a Business Learning Outcomes Assessment to College Students outside of the United States: Validity and Reliability Evidence. Research Report. ETS RR-13-40
Author: Ling, Guangming
Date: 2013-12-00
Type: Journal Articles
Subjects: College Outcomes Assessment,Business Administration Education,Test Validity,Test Reliability,Achievement Tests,College Seniors,Foreign Countries,Factor Structure,Scores,Statistical Analysis

One concern when repurposing a test to a new population is whether the test is measuring the same construct in a valid and reliable way that is comparable to the intended population. Following the guidelines of the International Test Commission and the ETS Standards for Quality and Fairness, this study was designed to collect evidence in support of repurposing the Major Field Test for the Bachelor's Degree in Business (MFT-B) for use in undergraduate level business programs outside of the United States. The author examined the test's reliability and internal structure based on 930 senior college students from 9 business programs in Asia, Europe, the Middle East, and South America. Each participating institution satisfied 2 requirements for this study: (a) having a bachelor's level business curriculum that is similar to their U.S. peer programs and (b) using English as the instructional language. The analyses of the data showed that the MFT-B has the same factor structure for the non-U.S.-based sample as that for their U.S. peers. In addition, the test scores for the non-U.S.-based sample are highly reliable (0.90) overall, with some variations among the 9 programs. Both types of evidence support the notion that the MFT-B is appropriate when repurposed for use with non-U.S.-based business programs.
Is There Any Interaction between Background Knowledge and Language Proficiency That Affects &quot;TOEFL iBT&quot;® Reading Performance? TOEFL iBT® Research Report. TOEFL iBT-18. ETS Research Report RR-12-22
Author: Hill, Yao Zhang
Date: 2012-10-00
Type: Journal Articles
Subjects: Language Tests,Second Language Learning,English (Second Language),Internet,Prior Learning,Knowledge Level,Language Proficiency,Reading Tests,Correlation,Scores,Test Items,Reading Achievement,Test Bias,Evaluation Methods,Discriminant Analysis,Matched Groups,Majors (Students)

This study investigated the effect of the interaction between test takers' background knowledge and language proficiency on their performance on the &quot;TOEFL iBT&quot;® reading section. Test takers with the target content background knowledge (the focal groups) and those without (the reference groups) were identified for each of the 5 selected passages based on their self-identified academic and cultural backgrounds. The test takers were further classified into high and low proficiency groups based on their TOEFL iBT scores. Differential functioning was investigated at the item, item bundle, and passage levels. The results suggested that background knowledge interacted with language proficiency on certain items, which could be attributed to idiosyncratic passage and item characteristics (i.e., characteristics that were specific to a particular passage or item). Only 1 of the 5 passages investigated showed intermediate differential bundle functioning, favoring the focal group for both the high and low proficiency groups. There was no differential functioning at the passage level. This research sheds new light on our understanding of the effects of background knowledge and its interaction with language proficiency in the context of second language reading comprehension. It also has significant practical implications for test developers in advancing fair assessments.