ethanporcaro/eric-fulltext-meta huggingface dataset:
How Do Raters from India Perform in Scoring the TOEFL iBT[TM] Speaking Section and What Kind of Training Helps? TOEFL iBT[TM] Research Report. RR-09-31
Author: Xi, Xiaoming
Date: 2009-08-00
Type: Reports - Evaluative
Subjects: Foreign Countries,English (Second Language),Internet,Language Tests,Scoring,Interrater Reliability,Bias,Multilingualism,Speech Communication,Performance Based Assessment,Job Training,Feedback (Response)

This study investigated the scoring of the Test of English as a Foreign Language[TM] Internet-based Test (TOEFL iBT[TM]) Speaking section by bilingual or multilingual speakers of English and 1 or more Indian languages. We explored the extent to which raters from India, after being trained and certified, were able to score the Speaking section for TOEFL iBT examinees with mixed first language (L1) backgrounds, especially those speaking an Indian language, accurately and consistently. The effectiveness of a special training package designed for scoring Indian examinees was examined as well. A total of 26 trained and certified raters from India were randomly divided into 2 groups and participated in 2 on-site scoring sessions in Mumbai. In the first session, both groups received regular training for scoring the TOEFL iBT Speaking section, which was largely similar to that received by raters in North America. In the second scoring session, 1 group continued to receive the regular training while the second group was trained using a special training package. Rater feedback surveys were also given to the raters. It was found that with training similar to that which operational U.S.-based raters receive, the raters from India performed as well as the operational raters in scoring both Indian and non-Indian examinees. In addition, the special training helped the raters score Indian examinees more consistently, leading to increased score reliability estimates. It also boosted raters' levels of confidence in scoring Indian examinees. Appendices include: (1) Distribution Statistics of the Scoring Samples in Scoring Session 1; (2) Distribution Statistics of the Scoring Samples in Scoring Session 2; and (3) Distributions of the Agreements Between Scores of ETS Raters and Raters From India in Scoring Session 1 With Regular Training. (Contains 13 tables and 4 notes.)
Does Content Knowledge Affect TOEFL iBT[TM] Reading Performance? A Confirmatory Approach to Differential Item Functioning. TOEFL iBT Research Report. RR-09-29
Author: Liu, Ou Lydia
Date: 2009-07-00
Type: Reports - Evaluative
Subjects: English (Second Language),Language Tests,Internet,Computer Assisted Testing,Reading Tests,Test Construction,Test Bias,Universities,Educational Objectives,Reader Text Relationship,Physical Sciences,Familiarity,Performance Based Assessment,Performance Factors

The TOEFL iBT[TM] has increased the length of the reading passages in the reading section compared to the passages on the TOEFL[R] computer-based test (CBT) to better approximate academic reading in North American universities, resulting in a reduced number of passages in the reading test. A concern arising from this change is whether the decrease in topic variety increases the likelihood that an examinee's familiarity with the particular content of a given passage will influence the examinee's reading performance. This study investigated differential item functioning and differential bundle functioning for six TOEFL iBT reading passages, three involving physical science and three involving cultural topics. The majority of items displayed little or no differential item functioning (DIF). When all of the items in a passage were examined, none of the passages showed differential functioning at the passage level. Hypotheses are provided for the DIF occurrences. Implications for fairness issues in test development are also discussed. Appendices include: (1) E-Mail to the Test Takers; and (2) Background Survey for TOEFL iBT. (Contains 3 tables.)
The Speaking Section of the TOEFL iBT[TM] (SSTiBT): Test-Takers' Reported Strategic Behaviors. TOEFL iBT[TM] Research Report. RR-09-30
Author: Swain, Merrill
Date: 2009-09-00
Type: Reports - Research
Subjects: English (Second Language),Language Tests,Internet,Speech Skills,Communication Strategies,Native Speakers,Chinese,Test Preparation,Scores,Metacognition,Speech Communication,Reading Strategies,Program Effectiveness,Statistics

This study responds to the Test of English as a Foreign Language[TM] (TOEFL[R]) research agenda concerning the need to understand the processes and knowledge that test-takers utilize. Specifically, it investigates the strategic behaviors test-takers reported using when taking the Speaking section of the TOEFL iBT[TM] (SSTiBT). It also investigates how the reported strategic behaviors differed across integrated and independent tasks in the SSTiBT, as well as the relationship between test-takers' reported strategic behaviors and their performance on the tasks as determined by their test scores. The participating students were 14 graduate and 16 undergraduate engineering students whose first language was Chinese. The results indicate that test-takers reported using 49 separate strategies when completing the SSTiBT tasks. Of the five strategy categories, the metacognitive, communication, and cognitive strategies were proportionally the most frequently reported. The interrelationships among these three categories were negative. Undergraduates reported using significantly more communication strategies, whereas graduates reported using significantly more cognitive and affective strategies. No statistically significant differences were found in reported strategy use across proficiency levels. The integrated tasks were more alike with respect to reported strategy use than were the independent and integrated tasks. Furthermore, the integrated tasks elicited a wider variety of reported strategy use than the independent tasks. Overall, we found no relationship between the total number of reported strategic behaviors and total test score on the SSTiBT. We conclude that strategy use is integral to performing SSTiBT tasks and should therefore be considered as part of the construct of communicative performance. However, the relationship between strategy use and test performance is varied and is due to complex interactions among test-taker characteristics, tasks, and contexts. Appendices include: (1) A List of Strategic Behaviors; (2) Analysis of Test Scores; (3) Pretest Proficiency Screening; (4) Individual Profile Questionnaire; (5) Stimulated Recall Instructions; (6) Coding Scheme; (7) Results of Normality Tests; (8) Descriptive Statistics for Individual Strategies by Test-Taker Study Level; (9) Descriptive Statistics for Individual Strategies by Task Group; (10) Descriptive Statistics for Individual Strategies by Task; and (11) Excerpts Illustrating Impact of Stimulated Recalls on Test-Takers' Behavior. (Contains 19 tables, 4 figures, and 16 notes.)
Toward a Definition of Verbal Reasoning in Higher Education. Research Report. ETS RR-09-33
Author: Burton, Nancy W.
Date: 2009-11-00
Type: Reports - Evaluative
Subjects: Higher Education,Cognitive Psychology,Developmental Psychology,Critical Reading,Reading Research,Linguistics,Educational Psychology,Thinking Skills,Performance Based Assessment,Verbal Communication,Familiarity

This paper briefly summarizes the literatures of reading and reasoning in the last quarter century, focusing mainly on the disciplines of cognitive science, cognitive developmental psychology, linguistics, and educational psychology. These literatures were synthesized to create a framework for defining verbal reasoning in higher education. Eight general cognitive and meta-cognitive operations were identified (including, for example, evaluating discourse, seeking and solving problems, and monitoring one's comprehension). Several dimensions underlying these operations on which individual skills may vary were identified (such as breadth of understanding, precision of understanding, or familiarity and facility). Finally, these ideal descriptions of verbal reasoning are applied to the assessment of verbal reasoning for selection in higher education. Problems in measurement and unanticipated consequences of measurement are discussed. (Contains 2 tables.)
Strategies for Testing Slope Differences. Research Report. ETS RR-09-32
Author: Moses, Tim
Date: 2009-10-00
Type: Reports - Research
Subjects: Evaluation Methods,Maximum Likelihood Statistics,Regression (Statistics),Robustness (Statistics),Least Squares Statistics,Statistical Analysis,Differences,Comparative Analysis

The robustness and power of 9 strategies for testing the differences in groups' regression slopes were assessed under nonnormality and residual variance heterogeneity. For the conditions considered, the most robust strategies were the trimmed and Winsorized slope estimates used with the James second-order test, the Theil-Sen slope estimates used with James, and the Theil-Sen estimates used with percentile bootstrapping. The use of Theil-Sen slope estimates produced more powerful tests than the use of trimmed and Winsorized slopes. Appendices include: (1) The Minimum Maximum Likelihood Type (MM) Regression Algorithm; and (2) The Critical Value of the James Second-Order Test Statistics. (Contains 11 tables and 4 figures.)
Defining Mathematics Competency in the Service of Cognitively Based Assessment for Grades 6 through 8. Research Report. ETS RR-09-42
Author: Graf, Edith Aurora
Date: 2009-12-00
Type: Reports - Evaluative
Subjects: Mathematics Education,Middle Schools,Cognitive Psychology,Grade 6,Grade 8,Competency Based Education,Evaluation Methods,Cognitive Measurement,Task Analysis,Course Content,Problem Solving,Models,Algebra,Geometry,Teaching Methods

This report makes recommendations for the development of middle-school assessment in mathematics, based on a synthesis of scientific findings in cognitive psychology and mathematics education. The focus is on background research, rather than test specifications or example tasks. Readers interested in early development and pilot efforts associated with the Cognitively Based Assessment "of", "for", and "as" Learning (CBAL) project in mathematics (for which this review helped provide a theoretical foundation) should consult Graf, Harris, Marquez, Fife, and Redman (2009). The organization of the report is motivated by the evidence-centered design (ECD) approach to assessment developed by Mislevy and colleagues (e.g., see Mislevy, Steinberg, & Almond, 2003). The first section consists of a broad literature review that characterizes mathematical competency with respect to both content and process. Subsequent sections discuss: how to model mathematical competency at the middle school level, the kinds of evidence that reflect the level of student competency and support future learning, and how to design tasks that elicit the target evidence. (Contains 2 figures and 2 notes.)
Cognitively Based Assessment of, for, and as Learning: A Framework for Assessing Reading Competency. Research Report. ETS RR-09-26
Author: O'Reilly, Tenaha
Date: 2009-06-00
Type: Reports - Research
Subjects: Reading Skills,Reading Comprehension,Speech,State Standards,Language Skills,Competence,Oral Language,Prerequisites,Models,Evaluation Methods,Tests,Critical Thinking,Accountability

This paper presents the rationale and research base for a reading competency model designed to guide the development of cognitively based assessment of reading comprehension. The model was developed from a detailed review of the cognitive research on reading and learning and a review of state standards for language arts. A survey of the literature revealed three key areas of reading competency: prerequisite reading skill, model building skill, and applied comprehension skill. Prerequisite reading skill is the ability to read text accurately and fluently. Model building skill is the ability to construct meaning from either decoded text or spoken language. Applied comprehension skill is the ability to use and apply the information contained in text for some particular purpose. The framework is discussed in terms of 7 key principles that have implications for the design of a modern assessment of reading. The skills branch of the Cognitively Based Assessment "of", "for  and "as" Learning (CBAL) reading competency model is appended. (Contains 1 figure and 6 notes.)
Small-Sample Equating with Prior Information. Research Report. ETS RR-09-25
Author: Livingston, Samuel A.
Date: 2009-06-00
Type: Reports - Research
Subjects: Test Length,Equated Scores,Bayesian Statistics,Sample Size,Computation,Raw Scores,True Scores,Equations (Mathematics)

This report proposes an empirical Bayes approach to the problem of equating scores on test   forms taken by very small numbers of test takers. The equated score is estimated separately at each score point, making it unnecessary to model either the score distribution or the equating transformation. Prior information comes from equatings of other test forms, with an appropriate adjustment for possible differences in test length. Derivation of estimates for y[subscript prior] and var(y [subscript prior]) is appended. (Contains 4 notes.)
Using the General Diagnostic Model to Measure Learning and Change in a Longitudinal Large-Scale Assessment. Research Report. ETS RR-09-28
Author: von Davier, Matthias
Date: 2009-07-00
Type: Reports - Research
Subjects: Models,Item Response Theory,Longitudinal Studies,Measurement,Comparative Analysis,Learning,Change,Statistical Distributions,Goodness of Fit,Data Analysis,Mathematics Tests,Foreign Countries

A general diagnostic model was used to specify and compare two multidimensional item-response-theory (MIRT) models for longitudinal data: (a) a model that handles repeated measurements as multiple, correlated variables over time (Andersen, 1985) and (b) a model that assumes one common variable over time and additional orthogonal variables that quantify the change (Embretson, 1991). Using MIRT-model ability distributions that we allowed to vary across subpopulations defined by type of school, we also compared (a) a model with a single two-dimensional ability distribution to (b) extensions of the Andersen and Embretson approaches, assuming multiple populations. In addition, we specified a hierarchical-mixture distribution variant of the (Andersen and Embretson) MIRT models and compared it to all four of the above alternatives. These four types of models are growth-mixture models that allow for variation of the mixing proportions across clusters in a hierarchically organized sample. To illustrate the models presented in this paper, we applied the models to the PISA-I-Plus data for assessing learning and change across multiple subpopulations. The results indicate that (a) the Embretson-type model with multiple-group assumptions fits the data better than the other models investigated, and (b) the higher performing group shows larger improvement at Time Point 2 than the lower performing group. (Contains 8 tables and 1 figure.)
Three Multidimensional Models for Testlet-Based Tests: Formal Relations and an Empirical Comparison. Research Report. ETS RR-09-37
Author: Rijmen, Frank
Date: 2009-12-00
Type: Reports - Evaluative
Subjects: Item Response Theory,Models,Graphs,Comparative Analysis,Factor Analysis,Tests,English,Evaluation Methods,Correlation

Three multidimensional item response theory (IRT) models for testlet-based tests are described. In the bifactor model (Gibbons & Hedeker, 1992), each item measures a general dimension in addition to a testlet-specific dimension. The testlet model (Bradlow, Wainer, & Wang, 1999) is a bifactor model in which the loadings on the specific dimensions are restricted to be proportional to the loadings on the general dimension within each testlet (Li, Bolt, & Fu, 2006). In the second-order model, the items only load on the testlet-specific factors. The correlations between the testlet-specific factors are modeled through a second-order factor. The second-order model is formally equivalent to the testlet model. The models were applied to a testlet-based international English assessment test. (Contains 6 figures and 1 table.)
Construction of Chained True Score Equipercentile Equatings under the Kernel Equating (KE) Framework and Their Relationship to Levine True Score Equating. Research Report. ETS RR-09-24
Author: Chen, Haiwen
Date: 2009-06-00
Type: Reports - Research
Subjects: True Scores,Equated Scores,Test Theory,Methods

In this paper, we develop a new chained equipercentile equating procedure for the nonequivalent groups with anchor test (NEAT) design under the assumptions of the classical test theory model. This new equating is named chained true score equipercentile equating. We also apply the kernel equating framework to this equating design, resulting in a family of chained true score equipercentile equating functions, which include the Levine true score equating model as a special case. (Contains 2 figures and 1 note.)
The Effects of Different Types of Anchor Tests on Observed Score Equating. Research Report. ETS RR-09-41
Author: Liu, Jinghua
Date: 2009-12-00
Type: Reports - Research
Subjects: Equated Scores,Test Items,Difficulty Level,Error of Measurement,Bias,Raw Scores,Statistical Analysis,College Entrance Examinations

This study explores the use of a different type of anchor, a &quot;midi anchor&quot;, that has a smaller spread of item difficulties than the tests to be equated, and then contrasts its use with the use of a &quot;mini anchor&quot;. The impact of different anchors on observed score equating were evaluated and compared with respect to systematic error (bias), random equating error (standard errors of equating), and total equating error (RMSE) using empirical data. The results show that the midi anchor generally produces more accurate equating results than the mini anchor--with a few exceptions. Our findings suggest that a midi anchor test would be preferred to a mini anchor test if equating accuracy at the top and at the bottom of the score scale is not a primary concern. (Contains 11 tables, 24 figures, and 1 note.)
Before They Read: Teaching Language and Literacy Development through Conversations, Interactive Read-Alouds, and Listening Games
Author: Miller, Cathy Puett
Date: 2010-00-00
Type: Books
Subjects: Preschool Teachers,Oral Language,Language Skills,Emergent Literacy,Interpersonal Communication,Listening Skills,Interaction,Reading Aloud to Others,Games,Play,Picture Books,Phonemic Awareness,Kindergarten,Reading Readiness,Teacher Student Relationship

Preschool and kindergarten educators know that strong oral language skills must be in place before children can learn to read. In "Before They Read: Teaching Language and Literacy Development through Conversations, Interactive Read-Alouds, and Listening Games," Cathy Puett Miller helps educators teach those early literacy skills with engaging games and activities that are based on her three big ideas for early literacy development: great conversations, good listening skills, and interactive read-alouds. Developed from Miller's successful work with families and early childhood educators around the country, "Before They Read" makes it easy to help every child move through the stages of literacy development at their own pace. Early childhood educators learn how to: take advantage of the learn-through-play style of the preschool and kindergarten child; play simple and effective games and activities that build core early literacy skills; and engage a child in the experience of reading a picture book to target essential concepts. An essential guide for childcare professionals and preschool and kindergarten teachers, "Before They Read" supports educators from the first word games throughout the journey to reading from playing with sounds through advanced phonemic awareness skills. The following are appended: (2) Stages of Phonological Awareness; (2) Shallow-to-Deep Progression; (3) A-Hunting We Will Go Rhyming Words--More Rhymes for Activity #8; (4) Great Rhyming Reads from (Activity #10); (5) Tongue Twisters for Every Letter of the Alphabet (Activity #16); (6) Read-aloud Books with Suggested Targets for Vocabulary and Concepts; (7) List of Rhyming Word "Families" (Activity #10). Lists of professional references and additional resources are provided. [This book was published by Maupin House.]
U.S. Public Libraries and the Use of Web Technologies. A Closer Look
Author: Lietzau, Zeth
Date: 2009-10-00
Type: Reports - Research
Subjects: Public Libraries,Technology Uses in Education,Internet,Web Sites,Library Research,Social Networks,Creativity,Information Technology,Computer Mediated Communication,Sharing Behavior,Institutional Characteristics,Adoption (Ideas),Innovation,Library Services

The use of interactive web technologies on public library web sites in the United States has been a topic of much discussion in recent years, and a shift in certain types of user services is underway. Terms like &quot;Web 2.0&quot; and even &quot;Library 2.0&quot; have become common in library literature and at conferences as those on the early edges of this shift share their successes and failures with the broader library community. The Library Research Service (LRS) was interested in adding to this conversation and, in late 2007, designed the study U.S. Public Libraries and the Use of Web Technologies. In spring 2008, LRS staff visited the web sites of nearly 600 public libraries in the United States, searching for the existence of certain web technologies, including technologies defined as &quot;Web 2.0.&quot; The web sites that were examined were drawn from a stratified sample of public libraries in the United States, with approximately equal numbers of libraries included from each of five different population ranges. Additionally, staff visited the web sites of all public libraries in Colorado. Two main sets of results were derived from the study: (1) how many public libraries in the United States (and Colorado) were implementing certain technologies on their web sites, and (2) what were the characteristics of the libraries that could be identified as &quot;early adopters&quot; of web technologies. A third set of conclusions -- does the early adoption of such technologies increase the libraries success in traditional service areas -- awaits the release of more current, available national public library data. Library 2.0 survey is appended. (Contains 6 tables, 20 charts, and 4 footnotes.) [This report was produced by the Library Research Service (LRS), a unit of the Colorado State Library, Colorado Department of Education Education, that is operated in partnership with the Library and Information Science Program, Morgridge College of Education,  University of Denver.]