ethanporcaro/eric-fulltext-meta huggingface dataset:
Use of Longitudinal Regression in Quality Control. Research Report. ETS RR-14-31
Author: Lu, Ying
Date: 2014-12-00
Type: Journal Articles
Subjects: Quality Control,Regression (Statistics),Scoring,Accuracy,Longitudinal Studies,Elementary Secondary Education,Language Tests,Mathematics Tests

This article explores the use of longitudinal regression as a tool for identifying scoring inaccuracies. Student progression patterns, as evaluated through longitudinal regressions, typically are more stable from year to year than are scale score distributions and statistics, which require representative samples to conduct credibility checks. Historical data from a large-scale K-12 testing program were used to evaluate the usefulness of several proposed longitudinal procedures. Results showed that the use of longitudinal regression in quality control was effective in detecting scoring errors, especially when the scoring errors were non-negligible.
Using Writing Process and Product Features to Assess Writing Quality and Explore How Those Features Relate to Other Literacy Tasks. Research Report. ETS RR-14-03
Author: Deane, Paul
Date: 2014-06-00
Type: Journal Articles
Subjects: Writing Processes,Writing Evaluation,Student Evaluation,Writing Skills,Writing Tests,Essay Tests,Computer Assisted Testing,Pilot Projects,Factor Structure,Factor Analysis,Generalization,Writing Ability,Reading,Accuracy,Editing,Grade 8,Middle School Students,Scores,Prompting,Correlation,Regression (Statistics)

This paper explores automated methods for measuring features of student writing and determining their relationship to writing quality and other features of literacy, such as reading rest scores. In particular, it uses the &quot;e-rater&quot;™ automatic essay scoring system to measure &quot;product&quot; features (measurable traits of the final written text) and features extracted from keystroke logs to measure process features (measurable features of the writing process). These techniques are applied to student essays written during large-scale pilot administrations of writing assessments developed for ETS's &quot;CBAL&quot;™ research initiative. The design makes it possible to explore the factor structures of these product and process features and to examine how well they generalize beyond a single test session to predict underlying traits such as writing ability and reading level. Three product factors are identified, connected to fluency, accuracy, and content. The process factors are identified, corresponding to hesitancy behaviors, editing behaviors, and burst span (the extent to which text is produced in long bursts with only short internal pauses). The results suggest that writing process and product features have stable factor structures that generalize to predict writing quality, though there are some genre- or task-specific differences.
Automated Scoring of Speaking Tasks in the Test of English-for-Teaching (&quot;TEFT&quot;™). Research Report. ETS RR-15-31
Author: Zechner, Klaus
Date: 2015-12-00
Type: Journal Articles
Subjects: Scoring,Scoring Formulas,Speech Communication,Task Analysis,Test Scoring Machines,Pilot Projects,Language Tests,Test Items,Database Design,Database Management Systems,Models

This research report presents a summary of research and development efforts devoted to creating scoring models for automatically scoring spoken item responses of a pilot administration of the Test of English-for-Teaching (&quot;TEFT&quot;™) within the &quot;ELTeach&quot;™ framework.The test consists of items for all four language modalities: reading, listening, writing, and speaking. This report only addresses the speaking items, which elicit responses ranging from highly predictable to semipredictable speech from nonnative English teachers or teacher candidates. We describe the components of the system for automated scoring, comprising an automatic speech recognition (ASR) system, a set of filtering models to flag nonscorable responses, linguistic measures relating to the various construct subdimensions, and multiple linear regression scoring models for each item type. Our system is set up to simulate a hybrid system whereby responses flagged as potentially nonscorable by any component of the filtering model are routed to a human rater, and all other responses are scored automatically by our system.
An Item-Driven Adaptive Design for Calibrating Pretest Items. Research Report. ETS RR-14-38
Author: Ali, Usama S.
Date: 2014-12-00
Type: Journal Articles
Subjects: Adaptive Testing,Simulation,Pretests Posttests,Test Items,Comparative Analysis,Test Format,Test Bias,Reaction Time,Item Response Theory,Computer Assisted Testing,Algebra,Mathematics Tests,Statistical Analysis,Evaluation Criteria,Scoring

Adaptive testing is advantageous in that it provides more efficient ability estimates with fewer items than linear testing does. Item-driven adaptive pretesting may also offer similar advantages, and verification of such a hypothesis about item calibration was the main objective of this study. A suitability index (SI) was introduced to adaptively select pretest items, by which an easy-to-implement calibration methodology--adaptive design--can be used. A simulation study was conducted to evaluate the proposed adaptive design as compared to existing methodologies. Results indicate that the adaptive design has many desired features in item calibration, including less bias and more accurate parameter estimates, than the existing methods do. The SI is promising and flexible enough to apply additional constraints on the calibration sample and on the pretest items, for example, constraints on response time. It can also be used to try out individual item modules such as those used in multistage testing. Study limitations and future research are also covered in this report.
A Descriptive Study of Universities' Use of &quot;GRE&quot;® General Test Scores in Awarding Fellowships to First-Year Doctoral Students. ETS GRE® Board Research Report. ETS GRE®-14-04. ETS Research Report. RR-14-27
Author: Rock, JoAnn Leah
Date: 2014-12-00
Type: Journal Articles
Subjects: Scores,Decision Making,College Entrance Examinations,Graduate Study,Universities,Fellowships,Validity,Qualitative Research,Student Recruitment,High Achievement,Academic Achievement,Doctoral Programs,Graduate Students,Test Use,Reputation,Institutional Characteristics,Periodicals,Structured Interviews,Deans,Administrator Attitudes,Classification,Student Research,Student Characteristics

The purpose of this study was to investigate the ways in which universities use the &quot;GRE&quot;® General Test scores to award merit-based fellowships to first-year graduate students in doctoral programs. While GRE use in fellowship award decisions is a common practice, there is very little validity evidence to support its use in this context. This exploratory qualitative study uses an argument-based validity approach as a conceptual framework to document GRE use systematically in award decisions. The findings from this study indicate that universities use merit-based fellowships to recruit &quot;the best and the brightest&quot; graduate students. The GRE is used as part of the nominee selection process in departments and is also used in the final selection process by university-wide committees. The GRE scores are primarily used to infer whether students have the baseline knowledge and skills needed to succeed in graduate school.These academic competencies were one of the characteristics the decision makers identified as required for successful doctoral students.The participants stated that the factors for success in graduate school are nebulous; however, the students with high GRE scores and other noteworthy supporting materials are expected to perform strongly in graduate school and, therefore, are worthy of the investment of a merit-based fellowship.
New Perspectives on the Validity of the &quot;GRE&quot;® General Test for Predicting Graduate School Grades. ETS GRE® Board Research Report. ETS GRE®-14-03. ETS Research Report. RR-14-26
Author: Klieger, David M.
Date: 2014-12-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Prediction,Predictive Validity,Grades (Scholastic),Multivariate Analysis,Effect Size,Correlation,Writing Evaluation,Student Evaluation,Scores,Grade Point Average,Meta Analysis,Generalization

Given the serious consequences of making ill-fated admissions and funding decisions for applicants to graduate and professional school, it is important to rely on sound evidence to optimize such judgments. Previous meta-analytic research has demonstrated the generalizable validity of the &quot;GRE&quot;® General Test for predicting academic achievement. That research does not address predictive validity for specific populations and situations or the predictive validity of the GRE Analytical Writing section introduced in October 2002. Furthermore, much of the past GRE predictive validity research is primarily based on approaches that are correlational and univariate only. Stakeholders familiar with GRE predictive validity mainly in the form of zero-order correlation coefficients might automatically interpret the usefulness of the GRE solely through the prism of Cohen's (1988) guidelines for judging effect sizes and without regard to the larger context. However, by using innovative and multivariate approaches to conceptualize and measure GRE predictive validity within the larger context, our investigation reveals the substantial value of the GRE General Test, including its Analytical Writing section, for predicting graduate school grades.
A Study of the Influence of the New Careers in Nursing Program on the Culture of Participating Schools of Nursing. New Careers in Nursing. Research Report. ETS RR-15-28
Author: Millett, Catherine M.
Date: 2015-12-00
Type: Journal Articles
Subjects: Nursing,Nursing Education,Focus Groups,Financial Support,Acceleration (Education),Program Evaluation,Grants,Scholarships,Case Studies,Interviews,Enrollment,Males,Minority Group Students,Orientation,Mentors,Leadership,Career Choice,Student Diversity,Educational Finance,Program Descriptions,Organizational Culture,Professional Associations,College Faculty,Teacher Attitudes

In 2014, ETS conducted a study investigating how the New Careers in Nursing (NCIN) program may have influenced the culture of participating schools of nursing. Select schools of nursing received grants from the Robert Wood Johnson Foundation to provide scholarships and support services for students in accelerated nursing programs. Case studies involving multiple interviews and focus groups were conducted in 8 NCIN schools of nursing chosen to represent public and private schools from all US regions. Results suggested that, in addition to benefiting accelerated nursing program students through the financial support of scholarships, the NCIN program may have increased the enrollment of male and minority students in accelerated nursing programs. Furthermore, the orientation, mentoring, and leadership supports provided with NCIN resources may have contributed to positive program and career outcomes for participating students, and in many cases the supports will be extended to all nursing students after the NCIN program ends. Lessons learned for the field include the potential for schools of nursing to employ strategies used by NCIN grantees to increase the diversity of their incoming cohorts and the ongoing need for financial supports for accelerated nursing students.
Enhancing the Equating of Item Difficulty Metrics: Estimation of Reference Distribution. Research Report. ETS RR-14-07
Author: Ali, Usama S.
Date: 2014-06-00
Type: Journal Articles
Subjects: Difficulty Level,Test Items,Equated Scores,Causal Models,Item Response Theory,Models,Statistical Distributions,Computation

Two methods are currently in use at Educational Testing Service (ETS) for equating observed item difficulty statistics. The first method involves the linear equating of item statistics in an observed sample to reference statistics on the same items. The second method, or the item response curve (IRC) method, involves the summation of conditional observed item statistics across the reference population total score frequencies. This article introduces a quick and effective method for obtaining the reference distribution for the transition from the linear equating method to the IRC method without recalculating all the item difficulties. More specifically, a mathematical formula is derived to estimate the score distribution of a reference group that maintains the current item difficulty scale. Future research is needed to compare the performance of the two approaches.
Developing Content Knowledge for Teaching Assessments for the Measures of Effective Teaching Study. Research Report. ETS RR-14-33
Author: Phelps, Geoffrey
Date: 2014-00-00
Type: Journal Articles
Subjects: Pedagogical Content Knowledge,Teacher Effectiveness,Teacher Evaluation,Measures (Individuals),Urban Schools,Urban Teaching,Language Arts,English Instruction,Mathematics Instruction,Elementary Secondary Education,Scores,Correlation,Item Response Theory,Statistical Analysis,Reliability,Psychometrics,Observation,Surveys

This report documents the development of assessments of content knowledge for teaching (CKT) as part of the Measures of Effective Teaching (MET) study, funded by the Bill and Melinda Gates Foundation.The MET study was designed to develop a set of measures that together serve as an accurate indicator of teaching effectiveness. The study was implemented during the 2009-2010 and 2010-2011 school years with more than 3,000 teachers in 6 predominately urban school districts. A total of 5 assessments of CKT were developed, piloted, and then administered as part of the MET study. The CKT assessments focused on the content knowledge used in recognizing, understanding, and responding to the content problems that teachers encounter as they teach a subject. In English language arts (ELA), 2 assessments were developed: 1 for teachers of Grades 4-6 and 1 for Grades 7-9. In mathematics, 3 assessments were developed: 1 for teachers of Grades 4-5, 1 for Grades 6-8, and 1 algebra I. A total of 2,080 final assessments were administered to 1,718 teachers in the 6 participating MET study districts. Assessment results for 194 teachers were excluded due to evidence that assessments were either completed together by 2 or more participants or that insufficient time was devoted to represent a good faith effort at answering the assessment questions. The final sample included 1,886 assessments. Assessment scores included both selected-response and constructed-response (CR) questions. We used information from item level statistics, including percent correct and biserial correlations, to systematically remove poorly performing items in order to improve assessment reliabilities. Item level statistics for each assessment are presented. Descriptive statistics and histograms indicate that participants are well distributed over the range of possible score responses. Assessments had moderate to strong levels of reliability, ranging from 0.69 to 0.83.
Demographically Adjusted Groups for Equating Test Scores. Research Report. ETS RR-14-30
Author: Livingston, Samuel A.
Date: 2014-12-00
Type: Journal Articles
Subjects: Demography,Equated Scores,Cluster Grouping,Ability Grouping,Predictor Variables,Statistical Significance,Testing Programs,Raw Scores,Error of Measurement,Statistical Distributions

In this study, I investigated 2 procedures intended to create test-taker groups of equal ability by poststratifying on a composite variable created from demographic information. In one procedure, the stratifying variable was the composite variable that best predicted the test score. In the other procedure, the stratifying variable was the composite that best indicated group membership (i.e., the propensity score). Applied to 2 groups taking the same test at different administrations, the composite that best predicted the test score reduced the ability difference by about two thirds; the composite that best indicated group membership reduced the ability difference by about half. Prescreening the predictor variables did not improve the performance of either procedure.
Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills. Research Report. ETS RR-15-26
Author: Deane, Paul
Date: 2015-12-00
Type: Journal Articles
Subjects: Writing Processes,Feasibility Studies,Research Reports,Writing Achievement,Writing Exercises,Distinctive Features (Language),Language Arts,Grade 6,Grade 7,Grade 8,Grade 9,Correlation,Writing Evaluation,Writing Skills,Keyboarding (Data Entry),Evaluation Methods

In this report, we examine the feasibility of characterizing writing performance using process features derived from a keystroke log. Using data derived from a set of &quot;CBAL&quot;™ writing assessments, we examine the following research questions: (a) How stable are the keystroke timing and process features across testing occasions?; (b) How consistent are the patterns of feature-human correlation across genres and topics?; (c) How accurately can we predict human ratings on writing fundamentals using a combination of the keystroke timing and process features, and what are the contributions of each feature to the reliable variance in the human ratings?; and (d) If we train a predictive model on one prompt, how well do its predictions generalize to the other prompts of the same or different genre? The results of the study indicate that keystroke log features vary considerably in stability across testing occasions and display somewhat different patterns of feature-human correlation across genres and topics. However, using the most stable features, we can obtain moderate to strong prediction of human essay scores, and those models generalize reasonably well across prompts though more strongly within than across writing genres.
Accelerated Nursing Degree Programs: Insights into Teaching and Learning Experiences. New Careers in Nursing. Research Report. ETS RR-15-29
Author: Millett, Catherine M.
Date: 2015-12-00
Type: Journal Articles
Subjects: Nursing Education,Nursing Students,Predictor Variables,Success,Student Motivation,Career Choice,Academic Degrees,Teaching Methods,Teacher Attitudes,College Faculty,Bachelors Degrees,Masters Degrees,Acceleration (Education),Conventional Instruction,Flexible Scheduling,Program Design,Teacher Role,Online Surveys

The Study of Teaching and Learning in Accelerated Nursing Degree Programs explores how nurse educators are adapting their teaching practices for accelerated, second-degree nursing program students. To provide findings on topics including instructional practices and the roles and attitudes of faculty, a web survey was administered to almost 100 staff members from schools of nursing that received grant and scholarship funds through the Robert Wood Johnson Foundation's New Careers in Nursing program for accelerated nursing students. The study revealed that nursing school faculty have positive perceptions of working with accelerated nursing students and that instructional approaches do not differ much between traditional and nursing students. At the same time, the factors described as most predictive of accelerated nursing student success were noncognitive attributes such as motivation and commitment to the nursing profession; prior degrees in the science or health fields were not necessarily seen as predictive of the success of second-degree students working toward an accelerated bachelor's or master's degree in nursing.
Simulate to Understand Models, Not Nature. Research Report. ETS RR-14-16
Author: Dorans, Neil J.
Date: 2014-12-00
Type: Journal Articles
Subjects: Simulation,Equated Scores,Scores,Scientific Methodology,Models,Evaluation Methods,Educational Practices,Test Validity,Test Reliability

Simulations are widely used. Simulations produce numbers that are deductive demonstrations of what a model says will happen.They produce numerical results that are consistent with the premises of the model used to generate the numbers. These simulated numerical results are not empirical data that address aspects of the world that lies outside the model. In contrast, empirical data are central to the scientific method.When a simulation is substituted for the assessment of hypotheses with real data, a false sense of understanding can ensue and with it a biased perspective on the world. To illustrate the limitations of simulation and their proper role, examples are drawn from simulation studies about score equating.
Process Features in Writing: Internal Structure and Incremental Value over Product Features. Research Report. ETS RR-15-27
Author: Zhang, Mo
Date: 2015-12-00
Type: Journal Articles
Subjects: Middle School Students,Student Writing Models,Writing Processes,Factor Structure,Writing Achievement,Factor Analysis,Essay Tests,Value Added Models,Distinctive Features (Language),Regression (Statistics)

In educational measurement contexts, essays have been evaluated and formative feedback has been given based on the end product. In this study, we used a large sample collected from middle school students in the United States to investigate the factor structure of the writing process features gathered from keystroke logs and the association of that latent structure with the quality of the final product (i.e., the essay text). The extent to which those process factors had incremental value over product features was also examined. We extracted 29 process features using the keystroke logging engine developed at Educational Testing Service (ETS). We identified 4 factors that represent the extent of writing fluency, local word-level editing, phrasal/chunk-level editing, and planning and deliberation during writing. We found that 2 of the 4 factors--writing fluency, and planning and deliberation--significantly related to the quality of the final text, whereas the 4 factors altogether accounted for limited variance in human scores. In 1 of the 2 samples studied, the keystroke-logging fluency factor added incrementally, but only marginally, to the prediction of human ratings of text-production skills beyond product features. The limited power of the writing process features for predicting human scores and the lack of clear additional predictive value over product features are not surprising given that the human raters have no knowledge of the writing process leading to the final text and that the product features measure the basic text quality specified in the human scoring rubric. Study limitations and recommendation for future research are also provided.
The Invariance of Latent and Observed Linking Functions in the Presence of Multiple Latent Test-Taker Dimensions. Research Report. ETS RR-14-41
Author: Dorans, Neil J.
Date: 2014-12-00
Type: Journal Articles
Subjects: Scores,Correlation,Equations (Mathematics),Simulation,Measurement,Multidimensional Scaling,Item Response Theory

This study examines linking relationships among latent test scores and how these latent linking relationships relate to observed-score linkings. Equations are used to describe the effects of correlation between underlying latent dimensions and the similarity or dissimilarity of test composition on linking functions among latent test scores. These equations describing relationships among latent test scores are used to model the results obtained from a previous simulation study, which illustrated that if the two tests have parallel structure then the linking relationship between their observed scores is subpopulation invariant regardless of the correlations between the underlying latent dimensions. The equations also model the effect that the degree of correlation between the latent dimensions has on equatability as the structure departs from parallelism.