ethanporcaro/eric-fulltext-meta huggingface dataset:
Applications of Multidimensional Item Response Theory Models with Covariates to Longitudinal Test Data. Research Report. ETS RR-16-21
Author: Fu, Jianbin
Date: 2016-05-00
Type: Journal Articles
Subjects: Item Response Theory,Longitudinal Studies,Test Bias,Goodness of Fit,Mathematics Tests,Algebra,Data Analysis,Models,National Competency Tests,Grade 8,Correlation

The multidimensional item response theory (MIRT) models with covariates proposed by Haberman and implemented in the &quot;mirt&quot; program provide a flexible way to analyze data based on item response theory. In this report, we discuss applications of the MIRT models with covariates to longitudinal test data to measure skill differences at the individual and group levels. In particular, we describe the differential item functioning procedure to identify common items with item drift across test occasions, and model selection and evaluation based on model comparison, fit statistics, and skill estimates. A real dataset on algebra tests is used to demonstrate the applications.
A Protocol for Annotating Parser Differences. Research Report. ETS RR-16-02
Author: Bruno, James V.
Date: 2016-06-00
Type: Journal Articles
Subjects: Documentation,Classification,Differences,Syntax,Automation,Case Studies,English (Second Language),Natural Language Processing,Written Language,Comparative Analysis

We present an annotation scheme for classifying differences in the outputs of syntactic constituency parsers when a gold standard is unavailable or undesired, as in the case of texts written by nonnative speakers of English. We discuss its automated implementation and the results of a case study that uses the scheme to choose a parser best suited to a downstream task that processes nonnative written text.
Evaluation of Different Scoring Rules for a Noncognitive Test in Development. Research Report. ETS RR-16-03
Author: Guo, Hongwen
Date: 2016-06-00
Type: Journal Articles
Subjects: Scoring,Test Reliability,Statistical Analysis,Psychometrics,Comparative Analysis,Test Items,Item Response Theory,Situational Tests,Nonparametric Statistics,Item Analysis

In this report, systematic applications of statistical and psychometric methods are used to develop and evaluate scoring rules in terms of test reliability. Data collected from a situational judgment test are used to facilitate the comparison. For a well-developed item with appropriate keys (i.e., the correct answers), agreement among various item-scoring rules is expected in the item-option characteristic curves. In addition, when models based on item-response theory fit the data, test reliability is greatly improved, particularly if the nominal response model and its estimates are used in scoring.
Who's on First? Gender Differences in Performance on the &quot;SAT&quot;® Test on Critical Reading Items with Sports and Science Content. Research Report. ETS RR-16-26
Author: Chubbuck, Kay
Date: 2016-08-00
Type: Journal Articles
Subjects: College Entrance Examinations,Gender Differences,Critical Reading,Reading Tests,Test Items,Athletics,Sciences,Test Bias,Difficulty Level,Student Interests,Familiarity,Performance,Statistical Analysis,Qualitative Research,Interviews,Pretests Posttests

This study gathered quantitative and qualitative evidence concerning gender differences in performance by using critical reading material on the &quot;SAT&quot;® test with sports and science content. The fundamental research questions guiding the study were: If sports and science are to be included in a skills test, what kinds of material are appropriate for fair and valid assessments? Does item type matter with regard to differential item functioning (DIF)? What factors can provide explanations for the different levels of difficulty, discrimination, and DIF exhibited? Do differences in students' interest in and familiarity with the material affect their performance? From data gathered during the two phases of our research, it appears that most sports and science material of the sort used in this study should be permissible in a skills test. Particularly in the context of a long passage, in which most information needed to answer an item is provided, very little material appears to be too &quot;specialized.&quot; From participants' statements during cognitive interviews, it seems that interest in and/or familiarity with a subject has little impact on performance--despite previous research to the contrary. On the basis of our observations, we recommend the following: (a) use more than one type of statistic to evaluate differential item functioning; (b) consider the amount of context provided with an item rather than just the particular content of an item; and (c) when possible, use data rather than assumptions about gender bias to make decisions about test content that may generate differential item functioning.
High-Performance Psychometrics: The Parallel-E Parallel-M Algorithm for Generalized Latent Variable Models. Research Report. ETS RR-16-34
Author: von Davier, Matthias
Date: 2016-11-00
Type: Journal Articles
Subjects: Psychometrics,Mathematics,Models,Statistical Analysis,Item Response Theory,Classification,Multitrait Multimethod Techniques,Computation

This report presents results on a parallel implementation of the expectation-maximization (EM) algorithm for multidimensional latent variable models. The developments presented here are based on code that parallelizes both the E step and the M step of the parallel-E parallel-M algorithm. Examples presented in this report include item response theory, diagnostic classification models, multitrait-multimethod (MTMM) models, and discrete mixture distribution models. These types of models are frequently applied to the analysis of multidimensional responses of test takers to a set of items, for example, in the context of proficiency testing. The algorithm presented here is based on a direct implementation of massive parallelism using a paradigm that allows the distribution of work among a number of processor cores. Modern desktop computers as well as many laptops are using processors that contain 2-4 cores and potentially twice the number of virtual cores. Many servers use 2, 4, or more multicore #central processing units (CPUs), which brings the number of cores to 8, 12, 32, or even 64 or more. The algorithm presented here scales the time reduction in the most calculation-intense part of the program almost linearly for some problems, which means that a server with 32 physical cores executes the parallel-E step algorithm up to 24 times faster than a single-core computer or the equivalent nonparallel algorithm. The overall gain (including parts of the program that cannot be executed in parallel) can reach a reduction in time by a factor of 6 or more for a 12-core machine. The basic approach is to utilize the architecture of modern CPUs, which often involves the design of processors with multiple cores that can run programs simultaneously. The use of this type of architecture for algorithms that produce posterior moments has straightforward appeal: The calculations conducted for each respondent or each distinct response pattern can be split up into simultaneous calculations.
Changes in the Motivation of Chinese ESL Learners: A Qualitative Investigation
Author: Li, Qi
Date: 2017-00-00
Type: Journal Articles
Subjects: Foreign Countries,Case Studies,Student Motivation,English (Second Language),Second Language Learning,College Students,Qualitative Research,Diaries,Interviews,Place of Residence,Self Concept,Student Attitudes,Attitude Change

This article reports on a case study that investigated changes in the motivation of Chinese ESL learners over a three month period of residence in an English-speaking environment. The participants consisted of 11 Chinese ESL learners, who were tertiary students and taking general or academic English courses in New Zealand. They took part in the present research shortly after their arrival in New Zealand. Data were collected over a period of three months using qualitative data collection instruments such as learner diaries and individual interviews. The results revealed some dynamic changes in the motivation of Chinese ESL learners over a three month period of residence in an English-speaking country. According to the similarities and differences in their motivation, they were categorized into five learner types. In general, the learners belonging to the first three types were able to maintain or increase their overall motivation. The learners in the last two types were not able to maintain their overall motivation: Their motivation decreased over the three months.
Development and Initial Evaluation of the ClearSpeak Style for Automated Speaking of Algebra. Research Report. ETS RR-16-23
Author: Frankel, Lois
Date: 2016-08-00
Type: Journal Articles
Subjects: Algebra,Automation,Secondary School Mathematics,Artificial Speech,Blindness,Visual Impairments,Testing,Assistive Technology,Comparative Analysis,Mathematics Tests,Evaluation,Statistical Analysis,Qualitative Research

The work described in this report is the first phase of a project to provide easy-to-use tools for authoring and rendering secondary-school algebra-level math expressions in synthesized speech that is useful for students with blindness or low vision. This report describes the initial development, software implementation, and evaluation of the ClearSpeak speech style--implemented for initial testing purposes using prerecorded synthetic speech, and implemented later in the project as a collection of predefined rules and in some cases variations (called &quot;preferences&quot; in ClearSpeak)--for automatically generated synthetic speech. In addition to wording, speech styles can specify pausing (or other prosodic cues) within the speech.The ClearSpeak style focuses on speech for secondary school algebra.The evaluation compares a prototype of the ClearSpeak style to two pre-existing speech styles: MathSpeak and SimpleSpeak. The primary parameters evaluated are students' success in drawing conclusions about the content and structure of math expressions and their perceptions regarding the familiarity, helpfulness, and understandability of the expressions as spoken. Please see Appendix E for information on obtaining a version of this report that is fully accessible using the tools described.
An Evaluation of the Usefulness of Prosodic and Lexical Cues for Understanding Synthesized Speech of Mathematics. Research Report. ETS RR-16-33
Author: Frankel, Lois
Date: 2016-10-00
Type: Journal Articles
Subjects: Artificial Speech,Suprasegmentals,Cues,Usability,Secondary School Mathematics,Algebra,Blindness,Visual Impairments,Feedback (Response),Assistive Technology,Secondary School Students,Mathematics Tests,Questionnaires,Statistical Analysis,Qualitative Research

The work described in this report is the second phase of a project to provide easy-to-use tools for authoring and rendering secondary-school algebra-level math expressions in synthesized speech that is useful for students with blindness or low vision. This report describes the development and results of the second feedback study performed for our project, Expanding Audio Access to Mathematics Expressions by Students With Visual Impairments via MathML. That study focused on the use of certain prosodic and lexical elements in the ClearSpeak speech style and served as a basis for further refinements in that style's definition and implementation in the MathPlayer software. The primary parameters evaluated are students' success in drawing conclusions about the content and structure of certain math expressions and their perceptions regarding the helpfulness of the pace and wording of different text-to-speech renditions of the same or similar mathematical expressions. Please see Appendix A for information on obtaining a version of this report that is fully accessible using the tools described.
Linking Composite Scores: Effects of Anchor Test Length and Content Representativeness. Research Report. ETS RR-16-36
Author: Lin, Peng
Date: 2016-12-00
Type: Journal Articles
Subjects: Equated Scores,Test Length,Test Content,Difficulty Level,Correlation

The nonequivalent groups with anchor test (NEAT) design is frequently used in test score equating or linking. One important assumption of the NEAT design is that the anchor test is a miniversion of the 2 tests to be equated/linked. When the content of the 2 tests is different, it is not possible for the anchor test to be adequately representative of both tests. Lin and Dorans conducted a simulation study in 2010 to investigate the effect of content representativeness of the anchor test on linking via different linking methods when the 2 tests are nonparallel in content structure in the unique case where the groups are equivalent. The current study extends the Lin and Dorans study to the case with nonequivalent group data. Specifically, the current study investigates the impact of content representativeness and length of anchor test on linking when the 2 tests are multidimensional and nonparallel in content structure. The NEAT design was employed. The linking results from 3 classic linear equating methods--Levine observed score, Tucker equating, and chained linear--were examined. The results from the study indicated that equating the tests with different structure should be avoided. For equatings with anchor test, additional bias is likely to be introduced by using an inadequate anchor test.
Learning and Development of Second and Foreign Language Pragmatics as a Higher-Order Language Skill: A Brief Overview of Relevant Theories. Research Report. ETS RR-16-35
Author: Timpe-Laughlin, Veronika
Date: 2016-12-00
Type: Journal Articles
Subjects: Second Language Instruction,Language Skills,Instructional Materials,Theories,Language Usage,Second Language Learning,Adult Learning,Cognitive Processes,Language Processing,Pragmatics

The development of effective second and foreign (L2) language learning materials needs to be grounded in two types of theories: (a) a theory of language and language use and (b) a theory of language learning. Both are equally important, insofar as an effective learning environment requires an understanding of the knowledge, skills, and abilities that are being targeted as well as an idea of how learner development progresses. Although the author and colleagues have previously provided a theoretical basis for the construct of L2 pragmatic ability, and thus a theory of language use, the current paper focuses on learner development, particularly on the theoretical orientation towards L2 pragmatic learning. Given this superordinate goal, the paper will briefly review different theoretical paradigms that have been employed in adult L2 pragmatics development research. In the first part of the paper, I will provide an overview of the cognitive, socially oriented, and emergentist perspectives used in research on L2 pragmatic development. Then, based on the different ontological perspectives, I will discuss the affordances and challenges of applying a theoretical perspective to designing L2 pragmatic learning environments, drawing upon the example of the cognitively oriented information processing theory.
Taming Log Files from Game/Simulation-Based Assessments: Data Models and Data Analysis Tools. Research Report. ETS RR-16-10
Author: Hao, Jiangang
Date: 2016-03-00
Type: Journal Articles
Subjects: Evaluation Methods,Games,Computer Assisted Testing,Data Collection,Programming,Databases,Models,Simulation

Extracting information efficiently from game/simulation-based assessment (G/SBA) logs requires two things: a well-structured log file and a set of analysis methods. In this report, we propose a generic data model specified as an extensible markup language (XML) schema for the log files of G/SBAs. We also propose a set of analysis methods for identifying useful information from the log files and implement the methods in a package in the Python programming language, glassPy. We demonstrate the data model and glassPy with logs from a game-based assessment, SimCityEDU.
Pilot Testing the Chinese Version of the ETS® Proficiency Profile Critical Thinking Test. Research Report. ETS RR-16-37
Author: Liu, Ou Lydia
Date: 2016-12-00
Type: Journal Articles
Subjects: Foreign Countries,Higher Education,Critical Thinking,College Students,Test Validity,Scores,Test Reliability,Correlation,College Entrance Examinations,Standardized Tests,Test Items,Test Bias,Student Evaluation,Translation,Psychometrics

Chinese higher education is experiencing rapid development and growth. With tremendous resources invested in higher education, policy makers have requested more direct evidence of student learning. However, assessment tools that can be used to measure college-level learning are scarce in China. To mitigate this situation, we translated the critical thinking test from the ETS® Proficiency Profile (EPP) into Chinese. EPP has been widely used in the United States to assess general college learning outcomes. We pilot tested the EPP--Chinese test with students from a university in China. Results suggest that (a) the test is unidimensional and therefore is sufficient to report a total score from a practical standpoint; (b) the total score reliability is satisfactory; (c) most items showed moderate correlations with the total score, but the translation of one item needs additional revision; (d) the test is correlated with related constructs (e.g., the Chinese college entrance examination and a national English test); and (e) no item showed differential item functioning or was found to be biased toward any subgroup. In summary, the Chinese version of the critical thinking test showed potential as a suitable assessment tool for Chinese college students.
Bootstrapping Development of a Cloud-Based Spoken Dialog System in the Educational Domain from Scratch Using Crowdsourced Data. Research Report. ETS RR-16-16
Author: Ramanarayanan, Vikram
Date: 2016-05-00
Type: Journal Articles
Subjects: Man Machine Systems,Models,Speech,Educational Technology,Qualitative Research,Statistical Analysis,Systems Development

We propose a crowdsourcing-based framework to iteratively and rapidly bootstrap a dialog system from scratch for a new domain. We leverage the open-source modular HALEF dialog system to deploy dialog applications. We illustrate the usefulness of this framework using four different prototype dialog items with applications in the educational domain and present initial results and insights from this endeavor.
An Investigation of the Use and Predictive Validity of Scores from the &quot;GRE&quot;® revised General Test in a Singaporean University. ETS GRE® Board Research Report. ETS GRE®-16-01. ETS Research Report. RR-16-05
Author: Liu, Ou Lydia
Date: 2016-03-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Test Use,Scores,Predictive Validity,Foreign Countries,College Admission,Grade Point Average,Graduate Students,Doctoral Programs,Masters Programs,College Applicants,Online Surveys,College Faculty,Administrators,Regression (Statistics),Correlation,Interviews,Focus Groups,Statistical Analysis

International institutions have been increasingly using the &quot;GRE&quot;® revised General Test to admit students to graduate programs.However, little is known about how scores from the GRE revised General Test are used in the admission process outside of the United States and their validity in predicting graduate students' performance (e.g., their graduate school grade point averages [GGPAs]). As the GRE revised General Test was launched in August 2011, there is a compelling need to investigate its predictive validity, particularly in an international context. A large percentage of examinees who take the GRE revised General Test from outside of the United States are citizens of Asian countries. Consequently, we examined how scores from the GRE revised General Test predict a range of graduate student performance outcomes at a Singaporean institution that represents the highest caliber of academic excellence in Asian countries. We also interviewed key members of the admissions committees to understand how the GRE revised General Test and its individual sections are used in the admission process. Our analyses revealed that scores from the GRE revised General Test predicted GGPA and program standing. In particular, these scores showed incremental value beyond undergraduate GPA (UGPA) for predicting GGPA. Furthermore, among enrolled students, those who submitted scores from the GRE revised General Test in application had significantly higher GGPAs than those who did not. These findings largely apply to both doctoral and master's students.
Opt Out: An Examination of Issues. Research Report. ETS RR-16-13
Author: Bennett, Randy E.
Date: 2016-06-00
Type: Journal Articles
Subjects: Educational Testing,Standardized Tests,Resistance (Psychology),Activism,Public Opinion,Social Attitudes,Educational Attitudes,Teacher Evaluation,Time on Task,Common Core State Standards,Test Construction

Media reports have recently given significant attention to the opt-out movement, an organized effort to refuse to take standardized tests. Although the narrative often told in early press accounts was of a viral grass-roots effort led by parents who object to state-mandated testing, the reality has turned out to be more complicated. Through a synthesis of news accounts, research studies, survey results, and state and federal education department documents, this paper examines the opt-out movement and some of the dynamics that appear to underlie it. Several topics are covered, including the movement's extent, the demographics of those participating in it, how much time students devote to tests, what other factors might be motivating the movement, and the level of public support for testing in general. The paper concludes with suggestions for how the assessment community might respond to the concerns raised by the movement and by the general public.
An Evaluation of the Single-Group Growth Model as an Alternative to Common-Item Equating. Research Report. ETS RR-16-01
Author: Wei, Youhua
Date: 2016-06-00
Type: Journal Articles
Subjects: Equated Scores,Growth Models,Scaling,Computation,Raw Scores,Language Tests,English

As an alternative to common-item equating when common items do not function as expected, the single-group growth model (SGGM) scaling uses common examinees or repeaters to link test scores on different forms. The SGGM scaling assumes that, for repeaters taking adjacent administrations, the conditional distribution of scale scores in later administration, given the scale score in earlier administration, should generalize from previous repeaters to the repeaters taking the current administration. The current repeaters' scale score distribution is estimated from their earlier scale score distribution. The SGGM scaling first uses previous repeaters' data to estimate the conditional distribution of their later scale scores, given their earlier scale scores. Then the repeaters taking both the current and previous administrations are identified, and their scale score distribution on the current form is estimated based on their previous scale score distribution and the estimated conditional distribution. Finally, a single-group equipercentile equating is performed between the current-form repeaters' observed raw score distribution and their estimated scale score distribution to obtain the raw-to-scale score conversion. This study evaluated the SGGM scaling performance using the common-item equating results for a language test as the criterion. The study found that the raw-to-scale conversions based on SGGM scaling differed from those based on common-item equating. However, the SGGM scaling results did not show a systematic bias in either the average or the variability of examinees' scale scores.