ethanporcaro/eric-fulltext-meta huggingface dataset:
The Log-Linear Cognitive Diagnostic Model (LCDM) as a Special Case of The General Diagnostic Model (GDM). Research Report. ETS RR-14-40
Author: von Davier, Matthias
Date: 2014-12-00
Type: Journal Articles
Subjects: Models,Equations (Mathematics),Measurement Techniques,Item Response Theory,Bayesian Statistics,Factor Analysis

Diagnostic models combine multiple binary latent variables in an attempt to produce a latent structure that provides more information about test takers' performance than do unidimensional latent variable models. Recent developments in diagnostic modeling emphasize the possibility that multiple skills may interact in a conjunctive way within the item function, while individual skills still may retain separable additive effects. This extension of either the conjunctive deterministic-input-noisy-and (DINA) model to the generalized version (G-DINA) or the compensatory/additive general diagnostic model (GDM) to the log-linear cognitive diagnostic model (LCDM) is aimed at integrating models with conjunctive skills and those that assume compensatory functioning of multiple skill variables. More recently, a result was proven mathematically that the fully conjunctive DINA model, which combines all required skills in a single binary function, may be recast as a compensatory special case of the GDM. This can be accomplished in more than one form such that the resulting transformed skill-space definitions and design (Q) matrices are different from each other but mathematically equivalent to the DINA model, producing identical model-based response probabilities. In this report, I extend this equivalency result to the LCDM and show that a mathematically equivalent, constrained GDM can be defined that yields identical parameter estimates based on a transformed set of compensatory skills.
Using Multilevel Analysis to Monitor Test Performance across Administrations. Research Report. ETS RR-14-29
Author: Wei, Youhua
Date: 2014-12-00
Type: Journal Articles
Subjects: Testing,Hierarchical Linear Modeling,Scores,Background,Correlation,Prediction,Performance,English (Second Language),Language Tests,Statistical Analysis,Regression (Statistics),Validity,Questionnaires

For a testing program with frequent administrations, it is important to understand and monitor the stability and fluctuation of test performance across administrations. Different methods have been proposed for this purpose. This study explored the potential of using multilevel analysis to understand and monitor examinees' test performance across administrations based on their background information. Based on the data of 330,091 examinees' test scores and their background information collected from 254 administrations of an English-speaking test, the study found: (a) at the individual examinee level, examinees' background had statistically significant relationships with their test performance, and the relationships varied across administrations; however, the prediction of individuals' test scores based on their background variables was not strong, and (b) at the administration level, group composition had strong relationships with administration means; the prediction of administration means based on group composition variables was fairly strong.The results suggest that multilevel analysis has potential application in understanding and monitoring test performance across administrations by identifying statistical relationships between examinees' characteristics and their test performance at both individual and administration levels.
Investigating the Relationship between Test-Taker Background Characteristics and Test Performance in a Heterogeneous English-as-a-Second-Language (ESL) Test Population: A Factor Analytic Approach. Research Report. ETS RR-15-25
Author: Manna, Venessa F.
Date: 2015-12-00
Type: Journal Articles
Subjects: English (Second Language),Factor Analysis,Language Tests,Student Characteristics,Second Language Learning,Reading Skills,Listening Skills,Speech Skills,Writing Skills,Language Proficiency,Effect Size,Questionnaires,Place of Residence,Foreign Countries,Statistical Analysis,Educational Attainment,Achievement

This study examined the heterogeneity in the English-as-a-second-language (ESL) test population by modeling the relationship between test-taker background characteristics and test performance as measured by the &quot;TOEFL iBT&quot;® using a confirmatory factor analysis (CFA) with covariate approach. The background characteristics studied included: (a) main reason for taking the TOEFL iBT test; (b) time spent studying English; (c) time spent attending a school, college, or university in which content classes were taught in English; and (d) lived in a country where English is the main language. The results indicated that at most levels of the background characteristics studied, there were statistically significant differences in the means of the four underlying latent factors (reading, listening, speaking, and writing) representing English-language proficiency (ELP). Overall, the effect size differences on the reading, listening, speaking, and writing latent factors among the levels for each of the background variables studied ranged from small to medium. The results of this study provide empirical evidence of the association and possible influence of test-taker background characteristics on the four underlying latent factors representing ELP and, thus, on test performance.
Personality, Motivation, and College Readiness: A Prospectus for Assessment and Development. Research Report. ETS RR-14-06
Author: Kyllonen, Patrick C.
Date: 2014-06-00
Type: Journal Articles
Subjects: Personality Traits,Student Motivation,College Readiness,Literature Reviews,Correlation,Evaluation Methods,Intervention,Psychological Patterns,Skills,Middle Schools,High Schools,Academic Standards,Remedial Instruction,Self Evaluation (Individuals),Self Control,Self Efficacy,Cognitive Processes,Vocational Aptitude,Vocational Education

This article concerns how noncognitive constructs--personality and motivation--can be assessed and developed to increase students' readiness for college. We propose a general framework to account for personality and motivational differences between students. We review numerous studies showing that personality and motivational factors are related to educational outcomes, from early childhood to adulthood. We discuss various methods for assessing noncognitive factors, ranging from self-assessments to performance tests. We consider data showing that personality and motivation change over time and find that particular interventions have proven successful in changing particular personality facets, leading to increased achievement. In a final section we propose a strategy for implementing a comprehensive psychosocial skills assessment in middle and high school, which would include setting proficiency standards and providing remedial instruction.
Using No-Stakes Educational Testing to Mitigate Summer Learning Loss: A Pilot Study. Research Report. ETS RR-14-21
Author: Zaromb, Franklin
Date: 2014-12-00
Type: Journal Articles
Subjects: Educational Testing,Summer Programs,Grade 3,Grade 4,Grade 6,Grade 5,Grade 7,Grade 8,Community Programs,Student Evaluation,Reading Skills,Mathematics Skills,Comparative Analysis,Scores,Standardized Tests,Pretests Posttests,Program Effectiveness,Achievement Tests,Reading Tests,Mathematics Tests,Drills (Practice),Quasiexperimental Design

This study investigates the benefits of no-stakes educational testing during students' summer vacation as a strategy to mitigate summer learning loss. Fifty-one students in Grades 3-8 from the Every Child Valued (ECV) and Lawrence Community Center (LCC) summer programs in Lawrenceville, NJ, took short, online assessments throughout the summer, covering knowledge and procedural skills in both mathematics and reading that they had previously learned. Students were randomly assigned to mathematics or reading conditions, whereby over the course of 5 weeks, students in the mathematics condition took online assessments in mathematics fluency two or three times per week and students in the reading condition took online assessments in reading components skills (word recognition and decoding, vocabulary,morphological awareness, and reading comprehension) two or three times per week. All students completed tests in both math and reading components at the beginning and end of the summer in order to compare baseline and final performance. Although students did not show significant summer learning loss in either reading components or mathematics fluency, students in the reading condition scored significantly higher, on average, on standardized tests of reading components administered at the end of the study than students in the mathematics condition. Given the small sample size and variations in grade level, academic subjects, computer programs, and testing conditions, it is premature to draw any firm conclusions from these findings.
What Differential Weighting of Subsets of Items Does and Does Not Accomplish: Geometric Explanation. Research Report. ETS RR-14-20
Author: Carlson, James E.
Date: 2014-12-00
Type: Journal Articles
Subjects: Geometric Concepts,Scores,Validity,Reliability,Equations (Mathematics),Test Items,Statistical Analysis,Weighted Scores,Measurement

A little-known theorem, a generalization of Pythagoras's theorem, due to Pappus, is used to present a geometric explanation of various definitions of the contribution of component tests to their composite. I show that an unambiguous definition of the unique contribution of a component to the composite score variance is present if and only if the component scores are uncorrelated. I further show the effect of differentially weighting the composites on the definitions of unique contributions and discuss some of the implications for composite score reliability and validity.
Building and Sharing Knowledge Key Practice: What Do You Know, What Don't You Know, What Did You Learn? Research Report. ETS RR-15-24
Author: O'Reilly, Tenaha
Date: 2015-12-00
Type: Journal Articles
Subjects: Language Arts,English Instruction,Knowledge Level,Cognitive Measurement,Literacy Education,Reading Skills,Writing Skills,Evaluation Methods,Student Evaluation,Vignettes,Cognitive Processes,Cognitive Structures,Goal Orientation,Reading Comprehension

In this paper we provide the rationale and foundation for the building and sharing knowledge key practice for the &quot;CBAL&quot;™ English language arts competency model. Building and sharing knowledge is a foundational literacy activity that enables students to learn and communicate what they read in texts. It is a strategic process that involves the integration of five key components or phases. Before reading, students &quot;activate their relevant background knowledge&quot; to help set learning goals, identify relevant information, and ask guiding questions that set the context for learning. During reading, students &quot;understand the text&quot; by using a host of strategies to construct a coherent mental model of the text content that is consistent with their background knowledge. Students &quot;clarify meanings&quot; of unknown words and concepts as they engage in metacognitive and self-regulated learning. After reading, students &quot;consolidate&quot; what they have read by using a variety of reading strategies that strengthen the representation in long-term memory. Finally, students &quot;convey&quot; what they have read in writing, speaking, or other representational formats to reflect communication goals and the intended audience. Collectively, the building and sharing knowledge key practice is intended to both model skilled performance and help identify component skill weakness. In this paper we outline the major features of the key practice as well as address potential advantages and challenges of the approach.
Stakeholders' Beliefs about the &quot;TOEFL iBT&quot;® Test as a Measure of Academic Language Ability. &quot;TOEFL iBT&quot;® Research Report. TOEFL iBT-22. ETS Research Report. RR-14-42
Author: Malone, Margaret E.
Date: 2014-12-00
Type: Journal Articles
Subjects: Stakeholders,Beliefs,Language Acquisition,Academic Ability,Test Validity,Outcome Measures,Focus Groups,Student Surveys,Language Proficiency,Teacher Characteristics,Student Characteristics,Context Effect,Simulation,Recall (Psychology),Feedback (Response),Task Analysis,English (Second Language),Qualitative Research,Language Tests,Second Language Learning

The &quot;TOEFL iBT&quot;® test presents test takers with tasks meant to simulate the tasks required of students in English-medium universities. Research establishing the validity argument for the test provides evidence for score interpretation and the use of the test for university admissions and placement. Now that the test has been operational for several years, additional evidence is needed to support the validity argument, as well as to identify directions for future research or changes to the test. To address this need, this study examines the extent to which students, instructors, and university administrators understand and agree with the construct of academic language underlying TOEFL iBT tasks.
Aligning &quot;TextEvaluator&quot;® Scores with the Accelerated Text Complexity Guidelines Specified in the Common Core State Standards. Research Report. ETS RR-15-21
Author: Sheehan, Kathleen M.
Date: 2015-12-00
Type: Journal Articles
Subjects: Scores,Common Core State Standards,Computer Software,Computational Linguistics,Text Structure,College Freshmen,Publishing Industry,Test Construction,Guidelines,Readability,Difficulty Level,Textbook Evaluation,Articulation (Education),Reading Skills,Elementary Secondary Education,Higher Education,Classification,Careers,Statistical Analysis,Mathematical Formulas,Language Styles

The &quot;TextEvaluator&quot;® text analysis tool is a fully automated text complexity evaluation tool designed to help teachers, curriculum specialists, textbook publishers, and test developers select texts that are consistent with the text complexity guidelines specified in the Common Core State Standards.This paper documents the procedure used to align the TextEvaluator reporting scale with the Common Core text complexity scale and provides score ranges for use when placing texts into grade bands.Three evaluations of the proposed score ranges are reported: one implemented with respect to the set of 168 exemplar texts provided in Appendix B of the Common Core State Standards, one implemented with respect to a set of 10 career texts, and one implemented with respect to a set of 59 texts selected from textbooks assigned in first-year, credit-bearing college courses. Results suggest that the proposed ranges can help users determine an appropriate grade band placement for any text that has been evaluated by TextEvaluator, including informational, literary, and mixed texts.
Automated Analysis of Text in Graduate School Recommendations. Research Report. ETS RR-15-23
Author: Heilman, Michael
Date: 2015-12-00
Type: Journal Articles
Subjects: Natural Language Processing,College Admission,Admission Criteria,Referral,Documentation,Distinctive Features (Language),Graduate Study,Content Analysis,Models,Student Characteristics,Affective Measures,Predictive Measurement,Statistical Analysis

Graduate school recommendations are an important part of admissions in higher education, and natural language processing may be able to provide objective and consistent analyses of recommendation texts to complement readings by faculty and admissions staff. However, these sorts of high-stakes, personal recommendations are different from the product and service reviews studied in much of the research on sentiment analysis. In this report, we develop an approach for analyzing recommendations and evaluate the approach on four tasks: (a) identifying which sentences are actually about the student, (b) measuring specificity, (c) measuring sentiment, and (d) predicting recommender ratings. We find substantial agreement with human annotations and analyze the effects of different types of features.
Location Indices for Ordinal Polytomous Items Based on Item Response Theory. Research Report. ETS RR-15-20
Author: Ali, Usama S.
Date: 2015-12-00
Type: Journal Articles
Subjects: Item Response Theory,Test Items,Computer Assisted Testing,Adaptive Testing

Polytomous items are typically described by multiple category-related parameters; situations, however, arise in which a single index is needed to describe an item's location along a latent trait continuum. Situations in which a single index would be needed include item selection in computerized adaptive testing or test assembly. Therefore single location indices for ordinal polytomous items are proposed and studied. The proposed location indices (LIs) for polytomous items are mathematically derived based on the item category response functions (ICRFs) and item response function (IRF) for polytomous items. The ICRF approach resulted in three indices: LI[subscript mean], LI[subscript trimmed mean], and LI[subscript median], and the IRF approach resulted in one proposed index, LI[subscript IRF]. An empirical example of real items is presented to help comprehension of the new location indices. Possible testing applications where the proposed item location indices are useful are discussed.
Is There Really a Labor Market Advantage to Being Bilingual in the U.S.? Policy Information Report and Research Report. ETS RR-15-07
Author: Gándara, Patricia
Date: 2015-12-00
Type: Journal Articles
Subjects: Labor Market,Bilingualism,Immigrants,Hispanic Americans,Language Proficiency,Salaries,Spanish Speaking,Language Minorities,United States History,Income,Monolingualism,Gender Differences,Wages,Multivariate Analysis,Regression (Statistics),Language Usage,Ethnicity,Surveys,Foreign Countries

Although it is commonly thought that people who are bilingual have an advantage in the labor market, studies on this topic have not borne out this perception.The literature, in fact, has found an earnings penalty is associated with bilingualism--people who are bilingual often make less than people who are monolingual in similar jobs. This report reviews those studies and introduces a new set of studies that found different outcomes for bilingual people in terms of education and earnings. In this report I examine why the prior and present studies differ so greatly and what this means for education policy.
Expanding the &quot;CBAL&quot;™ Mathematics Assessments to Elementary Grades: The Development of a Competency Model and a Rational Number Learning Progression. Research Report. ETS RR-14-08
Author: Arieli-Attali, Meirav
Date: 2014-06-00
Type: Journal Articles
Subjects: Mathematics Tests,Elementary School Students,Elementary School Mathematics,Numbers,Competence,Models,Grade 3,Grade 4,Grade 5,Common Core State Standards,Skill Development,Mathematics Skills,Semi Structured Interviews,Elementary School Teachers

Prior work on the &quot;CBAL&quot;™ mathematics competency model resulted in an initial competency model for middle school grades with several learning progressions (LPs) that elaborate central ideas in the competency model and provide a basis for connecting summative and formative assessment. In the current project, we created a competency model for Grades 3-5 that is based on both the middle school competency model and the Common Core State Standards (CCSS). We also developed an LP for rational numbers based on an extensive literature review, consultations with members of the CBAL mathematics team and other related research staff at Educational Testing Service, input from an advisory panel of external experts in mathematics education and cognitive psychology, and the use of small-scale cognitive interviews with students and teachers. Elementary mathematical understanding, specifically that of rational numbers, is viewed as fundamental and critical to developing future knowledge and skill in middle and high school mathematics and therefore essential for success in the 21st century world. The competency model and the rational number LP serve as the conceptual basis for developing and connecting summative and formative assessment as well as professional support materials for Grades 3-5. We report here on the development process of these models and future implications for task development.
Key Practices in the English Language Arts (ELA): Linking Learning Theory, Assessment, and Instruction. Research Report. ETS RR-15-17
Author: Deane, Paul
Date: 2015-12-00
Type: Journal Articles
Subjects: Language Arts,Learning Theories,Best Practices,English Instruction,Language Tests,Literacy,Vignettes,Alignment (Education),Definitions,Early Reading,Silent Reading,Reading Aloud to Others,Educational Practices

This paper presents a framework intended to link the following assessment development concepts into a systematic framework: evidence-centered design (ECD), scenario-based assessment (SBA), and assessment of, for, and as learning. The context within which we develop this framework is the English language arts (ELA) for K-12 students, though the framework could easily be applied to cover reading, writing, and critical thinking skills from pre-K through college. Central to the framework is the concept of a &quot;key practice,&quot; drawn from constructivist learning theory, which emphasizes the purposeful social context within which skills are recruited and organized to carry out complex literacy tasks. We argue that key practices provide a key link between existing &quot;CBAL&quot;™ ELA learning progressions (defined as part of a student model for literacy skills) and the structure of well-designed SBAs. This structure enables us to design assessments that model a key practice, supporting the systematic creation of task sequences that can be used to support both instruction and assessment.
Evaluating the &quot;TOEFL Junior&quot;® Standard Test as a Measure of Progress for Young English Language Learners. Research Report. ETS RR-15-22
Author: Gu, Lin
Date: 2015-12-00
Type: Journal Articles
Subjects: Language Tests,Second Language Learning,English (Second Language),Standardized Tests,Language Proficiency,Scores,Hierarchical Linear Modeling,Statistical Significance,Pretests Posttests,Adolescents,Longitudinal Studies,Data Collection,Student Improvement,Intervals,Achievement Gains

Standardized tests are often designed to provide only a snapshot of test takers' knowledge, skills, or abilities at a single point in time. Sometimes, however, they are expected to serve more demanding functions, one of them is assessing change in knowledge, skills, or ability over time because of learning effects.The latter is the case for the newly developed &quot;TOEFL Junior&quot;®Standard test, which measures improvement in young learners' proficiency in English as a foreign language. In this study,we used nonexperimental-repeated measures data from approximately 4,600 students from multiple countries to examine the extent to which observed patterns in within-individual changes in test scores were consistent with changes in underlying language proficiency because of learning. Because most students were actively participating in English language learning programs, the time interval between test administrations, which varied among students, served as a proxy for the extent of English language learning opportunities. We used hierarchical linear models to model growth in test performance as a function of the time interval between test administrations and found a positive, statistically significant relationship; that is, test takers with longer intervals between retesting exhibited greater gains than did test takers who retested at shorter intervals.The estimated relationship for the total score corresponded to between 0.16 and 0.24 test standard deviations of growth per year, depending on model specification.The findings are robust to sensitivity analyses that explore potential biasing factors.Overall, the findings are consistent with the hypothesis that the &quot;TOEFL Junior&quot; Standard test is capable of reflecting change in English language proficiency over time.