ethanporcaro/eric-fulltext-meta huggingface dataset:
Locating the Structural Zeros for Internal Anchor Tests: Including the Case of Rounded Formula Scores. Research Report. ETS RR-05-22
Author: Holland, Paul W.
Date: 2005-11-00
Type: Journal Articles
Subjects: Test Items,Equated Scores,Probability,Statistical Analysis

There are test-equating situations in which it may be appropriate to fit a loglinear or other type of probability model to the joint distribution of a total score on a test and a score on part of that test. For anchor test designs, this situation arises for internal anchor tests, which are embedded within the total test. Similarly, a part-whole relationship arises between two scores when a few test items are dropped from a test and a single group design is used to equate the scores of the full test to the part that remains after those items are deleted. In these part-whole situations, the resulting bivariate frequency distribution will exhibit &quot;structural zeros&quot; due to the fact that some scores on the total test are impossible for specific values on the partial test. Without knowing where the structural zeros are, it is impossible to distinguish them from zero frequencies that are simply due to size of the sample (i.e., sampling zeros). When probability models are estimated for these joint distributions, it is usual to require the models to assign positive probability to the sampling zeros but to avoid assigning positive probability to the structural zeros. To do this, it is important to be able to locate where the structural zeros are in the bivariate distribution. When the scores on the tests are consecutive integers, it is easy to determine the location of the structural zeros. This report gives a solution to the problem of locating the structural zeros that arise for a class of bivariate distributions that includes both number-right scores and formula scores that have been rounded to integer values. The result for rounded formula scores is a simple alteration of the case where all the scores are consecutive integers.
An Evaluation of Statistical Strategies for Making Equating Function Selections. Research Report. ETS RR-08-60
Author: Moses, Tim
Date: 2008-11-00
Type: Journal Articles
Subjects: Equated Scores,Statistical Analysis,Statistical Significance,Regression (Statistics),Differences,Accuracy

Nine statistical strategies for selecting equating functions in an equivalent groups design were evaluated. The strategies of interest were likelihood ratio chi-square tests, regression tests, Kolmogorov-Smirnov tests, and significance tests for equated score differences. The most accurate strategies in the study were the likelihood ratio tests and the significance tests for equated score differences.
Dependability of New ESL Writing Test Scores: Evaluating Prototype Tasks and Alternative Rating Schemes. TOEFL® Monograph Series. MS-31. ETS RR-05-14
Author: Lee, Yong-Won
Date: 2005-08-00
Type: Journal Articles
Subjects: Language Tests,English (Second Language),Second Language Learning,Writing Tests,Reliability,Scores,Generalizability Theory,Scoring,Multivariate Analysis,Foreign Countries

Possible integrated and independent tasks were pilot tested for the writing section of a new generation of TOEFL® (Test of English as a Foreign Language™) examination. This study examines the impact of various rating designs as well as the impact of the number of tasks and raters on the reliability of writing scores based on integrated and independent tasks from the perspective of generalizability theory (G-theory). Both univariate and multivariate G-theory analyses were conducted. It was found that (a) in terms of maximizing the score reliability, it would be more efficient to increase the number of tasks rather than the number of ratings per essay; (b) two particular single-rating designs having different tasks for the same examinee rated by different raters [p × (R:T), R:(p × T)] achieved relatively higher score reliabilities than other single-rating designs; and (c) a somewhat larger gain in composite score reliability was achieved when the number of listening-writing tasks was larger than the number of reading-writing tasks.
Comparisons among Designs for Equating Constructed-Response Tests. Research Report. ETS RR-08-53
Author: Kim, Sooyeon
Date: 2008-10-00
Type: Journal Articles
Subjects: Equated Scores,Comparative Analysis,Test Format,Responses,Test Items,Multiple Choice Tests,Difficulty Level,Design

This study examined variations of a nonequivalent groups equating design used with constructed-response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large-scale exam, the study investigated the use of anchor CR item rescoring in the context of classical equating methods. Four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple-choice (MC) anchor test, and (d) an equivalent groups design incorporating CR items rescored (no anchor test). The use of CR items without rescoring or the use of an external MC anchor resulted in much larger bias than the other two designs. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.
Development and Validity Evidence Supporting a Teamwork and Collaboration Assessment for High School Students. Research Report. ETS RR-08-50
Author: Zhuang, Xiaohua
Date: 2008-10-00
Type: Journal Articles
Subjects: Teamwork,Cooperative Learning,Evaluation Methods,Student Evaluation,High School Students,Self Evaluation (Individuals),Value Judgment,Teacher Attitudes,Factor Analysis,Multivariate Analysis,Test Validity,Test Reliability,Academic Achievement,Personality Traits,Student Characteristics,Grades (Scholastic)

Various policy papers and research studies assert that teamwork is one of the most important skills for students to learn if they are to become meaningful contributors to the 21st century workforce. However, outside of organizational psychology and adult populations, few reliable assessments of this construct exist, with suitable validity evidence scant or nonexistent. To redress this imbalance, teamwork assessments for high school students were developed using multiple methods: self-report ratings, situational judgment testing, and teacher reports. Exploratory factor, confirmatory factor, and latent class analyses were used to determine the structure of the scales. Measures showed reasonable reliability and promising validity evidence, relating to each other and to academic achievement, while remaining relatively independent from personality. The advantages and disadvantages of each methodology and the potential applications for identification and intervention, selection, and evaluation of training programs are discussed. This report also serves as an archival document for the teamwork and collaboration assessments that have been developed at ETS for high school students.
An Examination of the Techno-Pedagogical Education Competencies (TPACK) of Pre-Service Elementary School and Preschool Teachers
Author: Özdemir, Muhammet
Date: 2016-10-00
Type: Journal Articles
Subjects: Technological Literacy,Pedagogical Content Knowledge,Elementary School Teachers,Preschool Teachers,Competence,Knowledge Level,State Universities,Foreign Countries,Preservice Teacher Education,Outcomes of Education,Technology Education,Technology Integration,Practicums,Statistical Analysis,Educational Attainment,Scores,Likert Scales

The purpose of this study is to determine the technological pedagogical content knowledge of pre-service elementary school and preschool teachers. The fundamental problem of the study consists of the investigation of teachers' TPACK by their year of study and fields. This is a descriptive study. It was conducted using the survey model because it aimed to determine TPACK of the 995 pre-service elementary school and preschool teachers who participated in the study. They were in the departments of elementary school teaching and preschool teaching at three state universities in the spring term of the 2014-2015 academic year. The study found the means of junior (third year) and senior (fourth year) pre-service teachers in the departments of elementary school teaching and preschool teaching to be high. Their means were also high in the sub-dimensions of TPACK competence. The TPACK means of the pre-service elementary school and preschool teachers were found to be high. The TPACK means of senior pre-service elementary school and preschool teachers were determined to differ from those of junior pre-service preschool teachers. It can be stated that this difference resulted from technology and material development courses taken by the pre-service teachers in different semesters. Their pre-service education had positive effects on their TPACK. Providing pre-service technology education and practice as part of relevant courses will not be adequate. Along with the education, pre-service elementary school and preschool teachers should be provided with opportunities to use technology. These opportunities should go beyond the basis of courses that they take. It should be supplemented with practice in teaching practicum courses and should be included in practice evaluation forms as a criterion. The subdimensions of TPACK can be examined, and TPACK competency can be assessed in specific teaching programs in further studies.
Exploring Education Culture by Employing Hofstede?s Cultural Dimensions to Evaluate the Effectiveness of the Current ERP Training Approach in Thailand
Author: Chayakonvikom, Monta
Date: 2016-10-00
Type: Journal Articles
Subjects: Foreign Countries,Cognitive Style,Teaching Methods,Asians,Western Civilization,Cultural Differences,Qualitative Research,Information Systems,Cultural Background,Power Structure,Interviews,Employee Attitudes,Individualism,Collectivism,Users (Information),Guides,Time Management,Participant Characteristics,Instructional Effectiveness,Asian Culture

ERP training is a critical success factor in ERP implementation. The current ERP training was largely ineffective and caused user resistance and ERP implementation failure. The objective of this paper is to investigate whether the current ERP training approach can accommodate the cultural learning behaviors of end-users. Hofstede's cultural dimensions are employed to explain end-user dissatisfaction toward the current ERP training approach. The research is founded on the idea that different people learn in different ways, and cultural diversity may also influence how they learn and are taught. The current ERP training approach was design based on a Western learning culture and applied without concern for different learning cultures, which may be problematic. To achieve the objective, a qualitative method was adopted in this research. In-depth interviews were conducted with seventy-two end-users from twenty-two companies that had adopted an ERP system and had experienced ERP training in a Thai context during the implementation phase. The findings support the assumption by showing that the current ERP training approach did not accommodate Thai cultural learning behaviors in many areas. The content in this paper is a part of a comprehensive research to propose an ERP training framework to support the variety of end-user learning styles and different cultural learning behaviors.
Applying Content Similarity Metrics to Corpus Data: Differences between Native and Non-Native Speaker Responses to a TOEFL® Integrated Writing Prompt. Research Report. ETS RR-08-51
Author: Deane, Paul
Date: 2008-10-00
Type: Journal Articles
Subjects: Test Content,Computational Linguistics,Native Speakers,Writing Tests,English (Second Language),Second Language Learning,Prompting,Cues,Comparative Analysis,Language Tests,Grammar,Language Usage,Phrase Structure,Scores,Correlation,Multiple Regression Analysis,Statistical Analysis

For many purposes, it is useful to collect a corpus of texts all produced to the same stimulus, whether to measure performance (as on a test) or to test hypotheses about population differences. This paper examines several methods for measuring similarities in phrasing and content and demonstrates that these methods can be used to identify population differences between native and non-native speakers of English in a writing task.
Educational Assessment Using Intelligent Systems. Research Report. ETS RR-08-68
Author: Shute, Valerie J.
Date: 2008-12-00
Type: Journal Articles
Subjects: Educational Assessment,Intelligent Tutoring Systems,Artificial Intelligence,Summative Evaluation,Formative Evaluation,Evidence Based Practice,Validity,Reliability,Models,Measurement,Definitions,Student Evaluation

Recent advances in educational assessment, cognitive science, and artificial intelligence have made it possible to integrate valid assessment and instruction in the form of modern computer-based intelligent systems. These intelligent systems leverage assessment information that is gathered from various sources (e.g., summative and formative). This paper analyzes the role of educational assessment in intelligent systems, summarizes the characteristics of successfully deployed intelligent systems, and describes an evidence-based approach to incorporating valid and reliable assessments into enhanced intelligent systems.
Learning Styles: A Review of Validity and Usefulness
Author: Li, Yulong
Date: 2016-10-00
Type: Journal Articles
Subjects: Validity,Cognitive Style,Teaching Methods,Visual Perception,Auditory Perception,Kinesthetic Perception

The debate about learning styles has been on going for nearly half a century, many researchers have categorised them into style families. For some reasons, VAK is the most often heard, however, it is only one of the many categories. Current study gives a synthetic introduction of the existing strands of learning style categorizations, and critically discusses the limitations and chances of learning styles in educational studies.
One Approach to Detecting the Invariance of Proficiency Standards over Time. Research Report. ETS RR-08-15
Author: Qian, Jiahe
Date: 2008-04-00
Type: Journal Articles
Subjects: National Competency Tests,State Standards,Scores,Achievement Tests,Statistical Analysis,Educational Legislation,Federal Legislation,Reading Tests,Mathematics Tests,Grade 4,Elementary School Students,Visual Aids,Weighted Scores

This study explores the use of a mapping technique to test the invariance of proficiency standards over time for state performance tests. First, the state proficiency standards are mapped onto the National Assessment of Educational Progress (NAEP) scale. Then, rather than looking at whether there is a deviation in proficiency standards directly, the invariance of their NAEP equivalents is tested over time. The basis of the mapping technique is an enhanced method that was originally designed for comparing performance standards for public school students set by different states when the state tests are comparable. This approach can also be used to detect score inflation over time for state tests.
Investigation of Primary School Teachers' Conflict Resolution Skills in Terms of Different Variable
Author: Bayraktar, Hatice Vatansever
Date: 2016-09-00
Type: Journal Articles
Subjects: Elementary School Teachers,Conflict Resolution,Statistical Analysis,Test Results,Conflict,Problem Solving,Differences,Age Differences,Teacher Persistence,Foreign Countries,Likert Scales

In this study, it is aimed to determine the level of conflict resolution skills of primary school teachers and whether they vary by different variables. The study was organised in accordance with the scanning model. The universe of the study consists of primary school teachers working at 14 primary schools, two from each of the seven geographical regions. The sample group consists of 191 primary school teachers that were randomly selected from schools that make up the universe. The &quot;Conflict Resolution Skills&quot; scale that was developed by Rahim in order to determine conflict resolution skills and adapted into Turkish by Gümüseli (2001) was used in this study. As a result of this study, it was found out that the conflict resolution skills of primary school teachers are at the medium level. It is seen that primary school teachers generally prefer the Integrating Approach as a general way of conflict resolution. Upon examining the relations of teachers' conflict resolution skills with their demographic properties, a significant difference was found between the independent variables of age and duration of service and compromising approach. It was concluded that teachers between the ages of 34 and 39 tend to use the compromising approach more than teachers in other age groups, while teachers with a period of service of 21 and more tend to use the compromising approach more than teachers working at other durations of service. Conflict resolution skills of primary school teachers do not vary significantly by the variables of marital status, gender, educational status, geographical region and the province they work.
Monitoring and Fostering Learning through Games and Embedded Assessments. Research Report. ETS RR-08-69
Author: Shute, Valerie J.
Date: 2008-12-00
Type: Journal Articles
Subjects: Educational Games,Formative Evaluation,Instructional Design,Evidence Based Practice,Student Evaluation,Psychometrics,Case Studies,Computer Games,Computer Simulation,Persistence,Problem Solving,Attention Control,Reading Skills,Listening Skills,Creativity,Creative Thinking,Models,Bayesian Statistics,Competence,Performance Based Assessment

To reveal what is being learned during the gaming experience, this report proposes an approach for embedding assessments in immersive games, drawing on recent advances in assessment design. Key to this approach are formative assessment to guide instructional experiences and evidence-centered design to systematically analyze the assessment argument (including the claims about the learner and the evidence that supports or fails to support those claims). Elements of this approach that have been applied in a nongame setting are shown and ideas are discussed for applying the approach to an existing immersive game setting. Finally, the report offers suggestions for extending and applying this approach for existing games and the design of new ones.
Comparison of Subscores Based on Classical Test Theory Methods. Research Report. ETS RR-08-54
Author: Puhan, Gautam
Date: 2008-10-00
Type: Journal Articles
Subjects: Comparative Analysis,Scores,Tests,Certification,Institutional Characteristics,Item Response Theory,Statistical Analysis,Multiple Choice Tests,Graphs,Error of Measurement,Prediction

Will reporting subscores provide any additional information than the total score? Is there a method that can be used to provide more trustworthy subscores than observed subscores? These 2 questions are addressed in this study. To answer the 2nd question, 2 subscore estimation methods (i.e., subscore estimated from the observed total score or subscore estimated using both the observed subscore and observed total score) are compared. Analyses conducted on 8 certification tests indicated that reporting subscores at the examinee level may not be necessary as they do not provide much additional information than the total score. However, at the institutional level (for institution size greater than 30), reporting subscores may not be harmful, although it may be redundant. Finally, results indicated that subscores estimated using both the observed subscore and observed total score were the most trustworthy and may be used if subscores were to be reported.
Analysis of Discourse Features and Verification of Scoring Levels for Independent and Integrated Prototype Written Tasks for the New TOEFL®. TOEFL® Monograph Series. MS-30. ETS RM-05-13
Author: Cumming, Alister
Date: 2006-03-00
Type: Journal Articles
Subjects: Discourse Analysis,Essays,Scores,Language Proficiency,Syntax,Multivariate Analysis,English (Second Language),Language Tests,Second Language Learning,Writing Tests,Factor Structure,Scoring,Test Length,Foreign Countries,Statistical Analysis

We assessed whether and how the discourse written for prototype integrated tasks (involving writing in response to print or audio source texts) field tested for the new TOEFL® differs from the discourse written for independent essays (i.e., the TOEFL essay). We selected 216 compositions written for 6 tasks by 36 examinees in a field test--representing Score Levels 3, 4, and 5 on the TOEFL essay--then coded the texts for lexical and syntactic complexity, grammatical accuracy, argument structure, orientations to evidence, and verbatim uses of source text. Analyses with nonparametric MANOVAs, following a 3-by-3 (task type by English proficiency level) within-subjects factorial design, showed that the discourse produced for the integrated writing tasks differed significantly at the lexical, syntactic, rhetorical, and pragmatic levels from the discourse produced in the independent essay on most of these variables. In certain analyses, these differences were also obtained across the 3 ESL proficiency levels.
Factor Structure of the TOEFL® Internet-Based Test (iBT): Exploration in a Field Trial Sample. TOEFL iBT Research Report. TOEFL iBT-04. ETS RR-08-09
Author: Sawaki, Yasuyo
Date: 2008-03-00
Type: Journal Articles
Subjects: Factor Structure,Computer Assisted Testing,Multitrait Multimethod Techniques,Scores,Test Construction,English (Second Language),Language Tests,Second Language Learning,Correlation,Reading Skills,Writing Skills,Listening Skills,Oral Language,Test Items,Factor Analysis,Item Analysis

The present study investigated the factor structure of a field trial sample of the Test of English as a Foreign Language™ Internet-based test (TOEFL® iBT). An item-level confirmatory factor analysis (CFA) was conducted for a polychoric correlation matrix of items on a test form completed by 2,720 participants in the 2003-2004 TOEFL iBT Field Study. CFA-based multitrait-multimethod (MTMM) analyses for the Reading and Listening sections showed that the language abilities assessed in each section were essentially unidimensional, while the factor structure of the entire test was best represented by a higher-order factor model with a general factor (English as a second language/English as a foreign language ability) and four group factors for reading, listening, speaking, and writing. The integrated Speaking and Writing tasks, which require language processing in multiple modalities, well defined the target modalities (speaking and writing). These results broadly support the current reporting of four scores corresponding to the modalities and a total score, as well as the test design where the integrated tasks contribute only to the scores for the target modalities.
Factor Structure of the LanguEdge™ Test across Language Groups. TOEFL® Monograph Series. MS-32. ETS RR-05-12
Author: Stricker, Lawrence J.
Date: 2005-09-00
Type: Journal Articles
Subjects: Factor Structure,Language Tests,Factor Analysis,Semitic Languages,Chinese,Spanish,Native Language,Error of Measurement,Test Items,Listening Skills,Reading Skills,Writing Skills,Oral Language,Courseware,English (Second Language),Second Language Learning,Second Language Instruction,Computer Assisted Testing,Markov Processes,Monte Carlo Methods,Goodness of Fit,Statistical Analysis

This study assessed the factor structure of the LanguEdge™ test and the invariance of its factors across language groups. Confirmatory factor analyses of individual tasks and subsets of items in the four sections of the test, Listening, Reading, Speaking, and Writing, was carried out for Arabic-, Chinese-, and Spanish-speaking test takers. Two factors were identified, Speaking and a fusion of the other sections of the test. The number of factors, the factor loadings, and the factors' error variances were invariant in the three samples, although the correlations between the factors differed. The failure to find separate factors for each section of the LanguEdge test necessarily raises questions about the test's functioning that need to be resolved.