ethanporcaro/eric-fulltext-meta huggingface dataset:
An Examination of Rater Orientations and Test-Taker Performance on English-for-Academic-Purposes Speaking Tasks. TOEFL® Monograph Series. MS-29. ETS RR-05-05
Author: Brown, Annie
Date: 2005-04-00
Type: Journal Articles
Subjects: Evaluators,English (Second Language),Language Tests,Second Language Learning,English for Academic Purposes,Grammar,Oral Language,Language Proficiency,Specialists,Rating Scales,Evaluation Criteria,Holistic Approach,Discourse Analysis,Verbal Communication,Phonology,Language Fluency,Pronunciation,Task Analysis,Foreign Countries,Statistical Analysis

This report documents two coordinated exploratory studies into the nature of oral English-for-academic-purposes (EAP) proficiency. Study I used verbal-report methodology to examine field experts? rating orientations, and Study II investigated the quality of test-taker discourse on two different Test of English as a Foreign Language? (TOEFL®) task types (independent and integrated) at different levels of proficiency. Study I showed that, with no guidance, domain experts distinguished and described qualitatively different performances using a common set of criteria very similar to those included in draft rating scales developed for the tasks at ETS. Study II provided empirical support for the criteria applied by the judges. The findings indicate that raters take a range of performance features into account within each conceptual category and that holistic ratings are driven by all of the assessment categories rather than, as has been suggested in earlier studies, predominantly by grammar.
The Impact of Changes in the TOEFL Examination on Teaching and Learning in Central and Eastern Europe: Phase 1, The Baseline Study. TOEFL® Monograph Series. MS-34. ETS RR-06-18
Author: Wall, Dianne
Date: 2006-06-00
Type: Journal Articles
Subjects: Foreign Countries,English (Second Language),Language Tests,Second Language Learning,Second Language Instruction,Test Preparation,Observation,Longitudinal Studies,Interviews,Teacher Attitudes,Student Attitudes,Administrator Attitudes,Comparative Analysis,Futures (of Society),Educational Environment,Educational Change,Change,Test Construction,Familiarity,Computer Literacy,Computer Assisted Testing,Language Skills,Teaching Methods,Course Descriptions,Cross Cultural Studies

The purpose of this report is to present the findings of the first phase in a longitudinal study of the impact of changes in the TOEFL® test on teaching and learning in test preparation classrooms. Observations were carried out and interviews conducted with teachers, students, and directors of studies at 10 institutions in six countries in Central and Eastern Europe. The findings constitute a baseline against which finding from future phases can be compared. The report includes an analysis of the types of impact originally envisaged by advisers to the new TOEFL and a discussion of factors in the educational context that may influence the types of impact that may appear once the new test is operational.
Measuring Learning Outcomes in Higher Education Using the Measure of Academic Proficiency and Progress (MAPP). Research Report. ETS RR-08-47
Author: Liu, Ou Lydia
Date: 2008-09-00
Type: Journal Articles
Subjects: Higher Education,Educational Assessment,Academic Achievement,Scores,College Entrance Examinations,Outcomes of Education,Accountability,Value Added Models,College Freshmen,College Seniors,Statistical Analysis,Correlation

The Secretary of Education's Commission on the Future of Higher Education emphasizes accountability in higher education as one of the key areas of interest. The Voluntary System of Accountability (VSA) was developed to evaluate the effectiveness of general public college education. This study examines how student progress in college, indicated by the performance difference between freshmen and seniors after controlling for admission scores, can be measured using the Measure of Academic Proficiency and Progress (MAPP) test. A total of 6,196 students from 23 institutions were included in this study. Results indicated that MAPP was able to differentiate the performance between freshmen and seniors after controlling for SAT®/ACT scores. The institutions were classified into 10 groups on the basis of the difference in the actual vs. expected MAPP performance. This study provides an example of how MAPP can be used to evaluate value-added performance in college education. Issues such as student sampling and test-taking motivation are discussed.
A Didactic Approach to the Use of IRT True-Score Equating. Research Report. ETS RR-05-26
Author: von Davier, Alina A.
Date: 2005-12-00
Type: Journal Articles
Subjects: Item Response Theory,True Scores,Equated Scores,Test Items,Multiple Choice Tests,Responses,Advanced Placement Programs,Mathematics Tests,Calculus,Comparative Analysis,Goodness of Fit

This paper discusses the assumptions required by the item response theory (IRT) true-score equating method (with Stocking &amp; Lord, 1983; scaling approach), which is commonly used in the nonequivalent groups with an anchor data-collection design. More precisely, this paper investigates the assumptions made at each step by the IRT approach to calibrating items and equating tests, and discusses the approaches that one might take for checking whether these assumptions are met for a particular data set. We investigated two types of tests: tests that consist of multiple-choice items only, and tests that consist of both multiple-choice and free-response items. Real data from the AP® Calculus AB exam are used to illustrate the application of the IRT true-score equating method as well as for the comparisons.
Predicting Long-Term Success in Graduate School: A Collaborative Validity Study. Research Report. ETS GRE Board Research Report No. 99-14R. ETS RR-05-03
Author: Burton, Nancy W.
Date: 2005-04-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Predictor Variables,Success,Scores,Undergraduate Study,Teacher Attitudes,College Faculty,College Students,Student Characteristics,Mastery Learning,Productivity,Communication Skills,Cooperation,Intellectual Disciplines,Masters Degrees,Doctoral Degrees,Correlation,Racial Differences,Gender Differences,Biology,Chemistry,Education,English,Psychology

GRE verbal and quantitative scores and undergraduate grade point average were evaluated as predictors of multiple measures of long-term graduate school success. The measures of success were cumulative graduate grade point average and faculty ratings on three student characteristics: mastery of the discipline, professional productivity, and communication skill. Seven graduate institutions and 21 graduate departments in biology, chemistry, education, English, and psychology collaborated in order to identify measures of valued outcomes, develop reports useful to individual departments and graduate schools, and initiate a database for future studies. Results are reported for all departments combined and by discipline and, where sample sizes permitted, for master?s and doctoral degree students, men and women, U.S. citizens and noncitizens, domestic ethnic groups, and test takers who took the GRE computer-based test and those who took the paper-and-pencil version of the test. The results indicate that the combination of GRE scores and undergraduate grade point average strongly predicts cumulative graduate grade point average and faculty ratings. These results hold in each discipline and appear to hold in the small subgroups.
Analytic Scoring of TOEFL® CBT Essays: Scores from Humans and &quot;E-rater&quot;®. TOEFL® Research Reports. RR-81. ETS RR-08-01
Author: Lee, Yong-Won
Date: 2008-01-00
Type: Journal Articles
Subjects: English (Second Language),Language Tests,Second Language Learning,Scoring,Essays,Writing Tests,Computer Assisted Testing,Computer Software,Cues,Prompting,Correlation,Holistic Approach,Test Reliability,Evaluators,Profiles,Multitrait Multimethod Techniques,Statistical Analysis

The main purpose of the study was to investigate the distinctness and reliability of analytic (or multitrait) rating dimensions and their relationships to holistic scores and &quot;e-rater&quot;® essay feature variables in the context of the TOEFL® computer-based test (CBT) writing assessment. Data analyzed in the study were analytic and holistic essay scores provided by human raters and essay feature variable scores computed by &quot;e-rater&quot; (version 2.0) for two TOEFL CBT writing prompts. It was found that (a) all of the six analytic scores were not only correlated among themselves but also correlated with the holistic scores, (b) high correlations obtained among holistic and analytic scores were largely attributable to the impact of essay length on both analytic and holistic scoring, (c) there may be some potential for profile scoring based on analytic scores, and (d) some strong associations were confirmed between several &quot;e-rater&quot; variables and analytic ratings. Implications are discussed for improving the analytic scoring of essays, validating automated scores, and refining &quot;e-rater&quot; essay feature variables.
Reliability of Scaled Scores. Research Report. ETS RR-08-70
Author: Haberman, Shelby J.
Date: 2008-12-00
Type: Journal Articles
Subjects: Reliability,Scores,Item Response Theory,Computation,Error of Measurement

The reliability of a scaled score can be computed by use of item response theory. Estimated reliability can be obtained even if the item response model selected is not valid.
Limits on Log Cross-Product Ratios for Item Response Models. Research Report. ETS RR-06-10
Author: Haberman, Shelby J.
Date: 2006-05-00
Type: Journal Articles
Subjects: Item Response Theory,Models,Goodness of Fit,Item Analysis,Statistical Analysis,Mathematical Formulas

Bounds are established for log cross-product ratios (log odds ratios) involving pairs of items for item response models. First, expressions for bounds on log cross-product ratios are provided for unidimensional item response models in general. Then, explicit bounds are obtained for the Rasch model and the two-parameter logistic (2PL) model. Results are also illustrated through an example from a study of model-checking procedures. The bounds obtained can provide a basis for assessment of goodness of fit of these models.
Model Analysis and Model Creation: Capturing the Task-Model Structure of Quantitative Item Domains. Research Report. ETS RR-06-11
Author: Deane, Paul
Date: 2006-05-00
Type: Journal Articles
Subjects: Test Items,Models,Computer Software,Equations (Mathematics),Difficulty Level,Test Construction,Statistical Analysis,Mathematics Tests,Design,Evidence,Word Problems (Mathematics),Algebra

This study focuses on the relationship between item modeling and evidence-centered design (ECD); it considers how an appropriately generalized item modeling software tool can support systematic identification and exploitation of task-model variables, and then examines the feasibility of this goal, using linear-equation items as a test case. The first half of the study examines task-model structures for linear equations and their relevance to item difficulty within ECD. The second half of the study presents prototype software, a Model Creator system for pure math items, designed to partially automate the creation of variant item models reflecting different combinations of task-model variables. The prototype is applied to linear equations but is designed to generalize over a range of pure mathematical content types.
Subscores for Institutions. Research Report. ETS RR-06-13
Author: Haberman, Shelby J.
Date: 2006-06-00
Type: Journal Articles
Subjects: Scores,Statistical Analysis,Error of Measurement,Reliability,Tests,Prediction,Scoring,Institutions,Computation,Colleges,Multivariate Analysis

Recently, there has been an increasing level of interest in reporting subscores. This paper examines the issue of reporting subscores at an aggregate level, especially at the level of institutions that the examinees belong to. A series of statistical analyses is suggested to determine when subscores at the institutional level have any added value over the total scores. The methods are applied to two operational data sets. For the data under study, the results provide little support in favor of reporting subscores for either examinees or institutions.
Tensions, Trends, Tools, and Technologies: Time for an Educational Sea Change. Research Report. ETS RR-06-16
Author: Shute, Valerie J.
Date: 2006-06-00
Type: Journal Articles
Subjects: Teaching Methods,Lecture Method,Conventional Instruction,Student Centered Learning,Evaluation Methods,Student Evaluation,Computer Uses in Education,Measurement Techniques,Cognitive Science,Formative Evaluation,Summative Evaluation,Mathematics Achievement,Educational Change,Progressive Education,Feedback (Response),Intervention

This report outlines three educational approaches: (a) &quot;traditional,&quot; the currently dominant approach, a largely lecture-oriented, authoritarian style that makes heavy use of assessments &quot;of&quot; learning, which are useful for accountability purposes but only marginally useful for guiding day-to-day instruction; (b) &quot;progressive,&quot; a highly student-centered approach that relies on assessments &quot;for&quot; learning, which can be very useful in guiding day-to-day instruction; and (c) &quot;unified,&quot; a new, integrated approach that uses the best of both kinds of assessments--&quot;for&quot; and &quot;of&quot; learning--and which leverages computer technology, educational measurement, and cognitive science to address factors that undermined earlier attempts to implement the progressive approach. This report examines some of the research, trends, and factors that should be considered, understood, and, in some cases, leveraged, in order to move toward the unified approach. Further, this report presents examples of how ETS projects are moving toward the new approach to harness assessment in the service of learning.
Using Bursa Folk Songs for Voice Training in Departments of Music Education in Turkey
Author: GüL, Gülnihal
Date: 2016-08-23
Type: Journal Articles
Subjects: Foreign Countries,Singing,Music Teachers,Teacher Education Programs,Training Methods,Culturally Relevant Education,Teacher Educators,Teacher Surveys,Teacher Attitudes,Semi Structured Interviews,Music Education,Audio Equipment,Qualitative Research,Folk Culture

The training of a music teacher's voice, via the examples in his own culture can contribute to getting better results in music education. Therefore, it is believed that transition to Contemporary Turkish Music and then to universal music from our traditional music may be much easier. Accordingly, the use of national resources in voice training is seen to be quite important. In this context, Bursa Folk Songs, whose piano accompanied arrangements were prepared for voice training, and also were trained for one semester by the voice training instructors of three different departments of music education that train music teachers, and then those instructors were interviewed to evaluate the contributions of prepared piano accompanied Bursa Folk Songs for voice training to the field. As a result of the survey, it has been determined that music teacher candidates need to have an extensive repertory, full of examples from their own culture, the repertory of piano-accompanied folk songs is not enough in quantity, and piano accompanied folk songs have both technical and musical contributions during the voice training process.
Subpopulation Invariance of Equating Functions. Research Report. ETS RR-06-25
Author: Puhan, Gautam
Date: 2006-08-00
Type: Journal Articles
Subjects: Equated Scores,Licensing Examinations (Professions),Teacher Certification,Ethnicity,Hispanic Americans,Statistical Analysis,Comparative Analysis,Language Teachers,Spanish,Second Languages,Differences

This study examined population invariance of equating functions over subgroups defined by ethnicity on a teacher certification test. Investigating subgroup equating invariance was important because the total group who took this test consists of two subgroups (i.e., Hispanic and non-Hispanic) and the Hispanic group is a distinctively more able group as compared to the non-Hispanic group on the construct being measured. The study used data collected during the 2003 and 2004 administration of a teacher certification test. The chained equipercentile and linear equating methods were used to derive the equating functions. The root expected square difference was used to compare equating functions derived using the total group with equating functions derived using either the Hispanic or non-Hispanic groups. Findings suggested lack of subgroup invariance in equatings for the first test form (Form X). Also, the Hispanic group equating was less invariant as compared to the non-Hispanic group equating. The second form of the test (Form Y) showed more subgroup invariance in equating. This difference may partly be attributed to the fact that Form Y had a much bigger sample size as compared to Form X and the difference in equating functions observed in Form X for the total group and the two subgroups may be due to sampling variability. Implications of these results on actual pass/fail rates are also presented and discussed.
Inside Sourcefinder: Predicting the Acceptability Status of Candidate Reading-Comprehension Source Documents. Research Report. ETS RR-06-24
Author: Sheehan, Kathleen M.
Date: 2006-08-00
Type: Journal Articles
Subjects: Automation,Prediction,Reading Comprehension,Classification,Regression (Statistics),Probability,Natural Language Processing,Test Items,Test Construction,Verbal Tests,College Entrance Examinations,Graduate Study,Statistical Analysis,Models,Maximum Likelihood Statistics

This paper describes the development, implementation, and evaluation of an automated system for predicting the acceptability status of candidate reading-comprehension stimuli extracted from a database of journal and magazine articles. The system uses a combination of classification and regression techniques to predict the probability that a given document will be deemed acceptable for use in completing a specified passage-creation assignment by at least one test developer. The text features that form the basis of the estimated models are automatically extracted by natural language processing techniques. Model performance is evaluated by comparing the proportion of acceptable documents located with the screening capability turned on to the proportion of acceptable documents located with the screening capability turned off. The evaluation suggests that the estimated models have succeeded in capturing useful information about the characteristics of texts that affect test developers' ratings of source acceptability and that they can help test developers find a greater number of high-quality sources in less time.
Reporting Test Outcomes Using Models for Cognitive Diagnosis. Research Report. ETS RR-06-28
Author: von Davier, Matthias
Date: 2006-09-00
Type: Journal Articles
Subjects: Models,Cognitive Measurement,Scores,Factor Analysis,Item Response Theory,Classification

Models for cognitive diagnosis have been developed as an attempt to provide more than a single test score from item response data. Most approaches are based on a hypothesis that relates items to underlying skills. This relation takes the form of a design matrix that specifies for each cognitive item which skills are required to solve the item and which are not. This report outlines one direction that developments of cognitive diagnosis models is taking. It does not claim completeness, but describes a line of models that can be traced back to Tatsuoka's seminal work on the rule space methodology and that finds its current form in models that combine features of confirmatory latent factor analysis, multiple classification latent class models, and multidimensional item response models.
Choice of Anchor Test in Equating. Research Report. ETS RR-06-35
Author: Sinharay, Sandip
Date: 2006-11-00
Type: Journal Articles
Subjects: Test Items,Equated Scores,Evaluation Methods,Difficulty Level,Test Bias,Correlation,Error of Measurement,Simulation,Statistical Analysis

It is a widely held belief that anchor tests should be miniature versions (i.e., minitests), with respect to content and statistical characteristics of the tests being equated. This paper examines the foundations for this belief. It examines the requirement of statistical representativeness of anchor tests that are content representative. The equating performance of several types of anchor tests, including those having statistical characteristics that differ from those of the tests being equated, is examined through several simulation studies and a real data example. Anchor tests with a spread of item difficulties less than that of a total test seem to perform as well as a minitest with respect to equating bias and equating standard error. Hence, the results demonstrate that requiring an anchor test to mimic the statistical characteristics of the total test may be too restrictive and need not be optimal. As a side benefit, this paper also provides a comparison of the equating performance of post-stratification equating and chain equipercentile equating.
Content Analysis of Memory and Memory-Related Research Studies on Children with Hearing Loss
Author: Dogan, Murat
Date: 2016-08-23
Type: Journal Articles
Subjects: Content Analysis,Memory,Hearing Impairments,Causal Models,Language Acquisition,Statistical Analysis,Correlation,Trend Analysis,Verbal Ability,Assistive Technology,Elementary School Students,Secondary School Students,Educational Research,Literacy,Inclusion,Measures (Individuals),Participant Characteristics,Children,Preschool Children,Adolescents,Evaluation Methods

Memory plays a profound role in explaining language development, academic learning, and learning disabilities. Even though there is a large body of research on language development, literacy skills, other academic skills, and intellectual characteristics of children with hearing loss, there is no holistic study on their memory processes. Accordingly, the aim of this study is to examine memory-related research on children with hearing loss (CHL). In order to distill methodological and topical trends of those studies so as to unfold relevant research needs. In this research, content analysis method was used to analyze 25 quantitative-empirical studies on memory of children below 18 years. Majority of the research studies grouped normal-hearing children as the norm group. Participants with hearing loss were quite heterogeneous in nature. Causal-comparative and correlational designs were jointly used most frequently as the research model, and assessment was based on multi-measure paradigms. Noteworthy popular topical trends include memory of children using verbal language and cochlear implants; the relation of memory to language development and literacy skills, temporary memory processes, and memory of children attending inclusive classes. A significant conclusion of the current study is that topical trends filtered from the international literature indicate the research need of our country. Furthermore, results of the analysis revealed that taking memory processes into account, especially during the assessment of hard-of-hearing children may contribute drastically to the holistic nature of assessment.