ethanporcaro/eric-fulltext-meta huggingface dataset:
Calculator Use on the &quot;GRE&quot;® Revised General Test Quantitative Reasoning Measure. ETS GRE® Board Research Report. ETS GRE®-14-02. ETS Research Report. RR-14-25
Author: Attali, Yigal
Date: 2014-12-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Calculators,Test Items,Item Analysis,Difficulty Level,High Achievement,Mathematics Skills,Test Construction,Prediction,Accuracy,Correlation,Computer Assisted Testing,Cognitive Ability,Gender Differences,Racial Differences,Statistical Analysis

Previous research on calculator use in standardized assessments of quantitative ability focused on the effect of calculator availability on item difficulty and on whether test developers can predict these effects. With the introduction of an on-screen calculator on the Quantitative Reasoning measure of the &quot;GRE&quot;® revised General Test, it is possible to explore calculator use in more detail. This study investigates calculator usage for the GRE examinee population across examinee and item characteristics. Results suggest that calculator usage in the GRE is very common (75% of responses used a calculator) but is less prevalent for low and high ability examinees and for male, Black, and Hispanic examinees. Items associated with lower levels of calculator usage tended to be more difficult, required less time to answer, and did not present a real-world application. Quantitative comparison items were also associated with less calculator usage, but other item type and content distinctions showed small differences in usage. Most items exhibited a positive relation between calculator usage and response accuracy (after controlling for examinee ability), that is, responses that used the calculator were more likely to be correct. Items that did not show an association between calculator use and accuracy tended to be easier and had less overall usage of the calculator. [The report presents the findings of a research project funded by and carried out under the auspices of the Graduate Record Examinations Board.]
Creating Vocabulary Item Types That Measure Students' Depth of Semantic Knowledge. Research Report. ETS RR-14-02
Author: Deane, Paul
Date: 2014-06-00
Type: Journal Articles
Subjects: Vocabulary,Test Items,Language Tests,Semantics,Knowledge Level,Language Usage,Language Patterns,Difficulty Level,Test Construction,Association (Psychology),Grade 7,Grade 8,Item Analysis,Item Response Theory,Test Validity,Natural Language Processing,Middle School Students,Multiple Choice Tests

We expect that word knowledge accumulates gradually. This article draws on earlier approaches to assessing depth, but focuses on one dimension: richness of semantic knowledge. We present results from a study in which three distinct item types were developed at three levels of depth: knowledge of common usage patterns, knowledge of broad topical associations, and knowledge of specific conceptual relationships. We attempted to avoid common sources of variance across items (such as attractive distracters) and hypothesized that the item types that required greater depth of semantic knowledge would tend to show greater difficulty and discrimination after other sources of variance were accounted for. Our results, while still exploratory, support the conclusion that the item types measure different aspects of lexical knowledge, consistent with the hypothesis of increasing semantic depth.
The 3 x 2 Achievement Goal Model in Predicting Online Student Test Anxiety and Help-Seeking
Author: Yang, Yan
Date: 2016-00-00
Type: Journal Articles
Subjects: Prediction,Online Courses,Test Anxiety,Help Seeking,Student Attitudes,Achievement,Distance Education,Regression (Statistics),Goal Orientation,Self Efficacy,Student Characteristics,Age Differences,Gender Differences,Questionnaires,Likert Scales,College Students,Learning Strategies

This study investigates the utility of the new 3 × 2 achievement goal model in predicting online student test anxiety and help-seeking. Achievement goals refer to students' general aims for participating in learning and the standard by which they judge their achievement (Pintrich, 2000). According to Elliot and his colleagues (2011), there are six types of achievement goals based on three dimensions of competence (self-, task-, and other-) and two dimensions of valence (approach and avoidance). The sample included 209 students enrolled in distance education classes who volunteered for the study. Separate hierarchical regression was employed to examine the predictive power of achievement goals in online student test anxiety and help-seeking beyond self-efficacy and demographic differences. Achievement goals predicted online help-seeking differently from traditional classes. Students who endorsed other-avoidance or self-approach goals reported more help-seeking, while those with other-approach or self-avoidance goals reported less help-seeking. The study results and practical implications for online instruction and course design are discussed. [Note: The volume number shown on the PDF is incorrect. The correct volume number is v31.]
A Design Framework for the ELTeach Program Assessments. Research Report. ETS RR-14-36
Author: Young, John W.
Date: 2014-12-00
Type: Journal Articles
Subjects: English (Second Language),Second Language Learning,Second Language Instruction,Faculty Development,Standards,Instructional Materials,Language Tests,Scores,Test Construction,Program Evaluation,Guidelines,Language Teachers,Pedagogical Content Knowledge,Online Courses,Teacher Certification,Program Descriptions,Scoring

ELTeach is an online professional development program consisting of two courses, English-for-Teaching and Professional Knowledge for English Language Teaching (ELT). Each course includes a coordinated assessment leading to a score report and certificate for individual teachers. Developed with reference to international and national teaching standards, and drawing on the resources of various national English-language curricula and teaching materials, the program is designed to ensure that teachers have the functional classroom English language and the professional knowledge to support the implementation of the English-language curricula they are expected to teach. In this document, we outline the design and development process followed in creating the two ELTeach assessments--the Test of English-for-Teaching (&quot;TEFT&quot;™) and the Test of Professional Knowledge (&quot;TPK&quot;™). The paper is organized into the following sections: (a) the purposes and intended uses for test scores/certificates, (b) the target populations, (c) the construct definitions, (d) the program frameworks and domain descriptions, (e) the test designs, (f) the scoring systems, and (g) directions for research in support of the program.
The Short Term Effectiveness of an Outdoor Environmental Education on Environmental Awareness and Sensitivity of In-Service Teachers
Author: Okur-Berberoglu, Emel
Date: 2015-01-00
Type: Journal Articles
Subjects: Program Effectiveness,Outdoor Education,Environmental Education,Faculty Development,Pretests Posttests,Consciousness Raising,Biodiversity,Statistical Analysis,Questionnaires,Likert Scales,Factor Analysis,Foreign Countries

Outdoor education is mostly mentioned in terms of environmental education. The aim of this research is to determine the short term effectiveness of an outdoor environmental education program on biodiversity awareness, environmental awareness and sensitivity to natural environment. The data is collected from an outdoor environmental education project which is financed by TUBITAK and its name is &quot;Nature-based Outdoor Environmental Education in Canakkale and Suburbs, 2008.&quot; There are 27 in-service teachers in the project and the project is only for 10 days. An environmental awareness and sensitivity scale is developed for the study. The research design is pre-posttest design. The data is analysed by Wilcoxon signed rank test because of the data is non-parametric. It is found that the nature-based outdoor education program is effective to improve environmental awareness and sensitivity to natural environment; on the other hand it is not successful to improve biodiversity awareness. [Note: The publication year shown on the PDF (2014) is incorrect. The correct publication year is 2015.]
Estimating Item Difficulty with Comparative Judgments. Research Report. ETS RR-14-39
Author: Attali, Yigal
Date: 2014-12-00
Type: Journal Articles
Subjects: Test Items,Difficulty Level,Comparative Analysis,College Entrance Examinations,Mathematics Tests,Scaling,Interrater Reliability,Simulation,Correlation

Previous investigations of the ability of content experts and test developers to estimate item difficulty have, for themost part, produced disappointing results. These investigations were based on a noncomparative method of independently rating the difficulty of items. In this article, we argue that, by eliciting comparative judgments of difficulty, judges can more accurately estimate item difficulties. In this study, judges from different backgrounds rank ordered the difficulty of &quot;SAT&quot;® mathematics items in sets of 7 items. Results showed that judges are reasonably successful in rank ordering several items in terms of difficulty, with little variability across judges and content areas. Simulations of a possible implementation of comparative judgments for difficulty estimation show that it is possible to achieve high correlations between true and estimated difficulties with relatively few comparisons. Implications of these results for the test development process are discussed.
Measuring Motivation in Low-Stakes Assessments. Research Report. ETS RR-15-19
Author: Finn, Bridgid
Date: 2015-12-00
Type: Journal Articles
Subjects: Research Reports,Student Motivation,Self Disclosure (Individuals),Construct Validity,Accountability,Outcome Measures,Relevance (Education),Individual Differences,Predictor Variables,Testing Problems,Self Concept Measures

There is a growing concern that when scores from low-stakes assessments are reported without considering student motivation as a construct of interest, biased conclusions about how much students know will result. Low motivation is a problem particularly relevant to low-stakes testing scenarios, which may be low stakes for the test taker but have considerable consequences for teachers, school districts, or educational and governmental institutions. The current review addresses the impact of motivation on assessment scores and research that have identified methods for minimizing the error introduced by unmotivated test takers. A comprehensive review of the test conditions that have been shown to influence motivation is provided. In addition, the review identifies a number of circumstances under which motivation can be enhanced or diminished. The benefits and limitations of the various measurement techniques that have been used to mitigate the negative impact of low-motivation test takers on score interpretation are discussed.
Statistical Methods for Assessments in Simulations and Serious Games. Research Report. ETS RR-14-12
Author: Fu, Jianbin
Date: 2014-12-00
Type: Journal Articles
Subjects: Simulation,Evaluation Methods,Games,Data Collection,Data Analysis,Problem Solving,Computer Assisted Testing,Probability,Bayesian Statistics,Scores,Scaling,Multivariate Analysis,Markov Processes

Simulation or game-based assessments produce outcome data and process data. In this article, some statistical models that can potentially be used to analyze data from simulation or game-based assessments are introduced. Specifically, cognitive diagnostic models that can be used to estimate latent skills from outcome data so as to scale these assessments are presented under the framework of Bayesian networks; 5 prospective data mining methods that can be employed to discover problem-solving strategies from process data are described. Some studies in the literature that apply some of these methods to analyze simulation or game-based assessments are presented as application examples. Recommendations are provided for selecting appropriate scaling and data mining methods for these assessments; future directions of research are proposed.
Considerations for Providing Test Translation Accommodations to English Language Learners on Common Core Standards-Based Assessments. Research Report. ETS RR-14-05
Author: Turkan, Sultan
Date: 2014-06-00
Type: Journal Articles
Subjects: English Language Learners,Testing Accommodations,Translation,Media Adaptation,Interviews,Common Core State Standards,Educational Assessment,State Policy,Standardized Tests,Test Validity,State Surveys

In this article, we review translation, adaptation policies, and practices in providing test accommodation for English language learners (ELLs) in the United States. We collected documents and conducted interviews with officials in the 12 states that provide translation accommodations to ELLs on content assessments. We then summarized challenges to ensuring fair and valid accommodations to ELLs and provided recommendations to address the challenges involved in translating a content test while preserving its validity.
A Generalization of Pythagoras's Theorem and Application to Explanations of Variance Contributions in Linear Models. Research Report. ETS RR-14-18
Author: Carlson, James E.
Date: 2014-12-00
Type: Journal Articles
Subjects: Least Squares Statistics,Geometry,Geometric Concepts,Statistical Analysis,Regression (Statistics),Mathematical Models,Mathematical Formulas

Many aspects of the geometry of linear statistical models and least squares estimation are well known. Discussions of the geometry may be found in many sources. Some aspects of the geometry relating to the partitioning of variation that can be explained using a little-known theorem of Pappus and have not been discussed previously are the topic of this report. I discuss, using the theorem, how geometric explanation helps us understand issues relating to contributions of independent variables to explanation of variance in a dependent variable. A particular concern that the theorem helps explain involves nonorthogonal linear models including correlated regressors and analysis of variance.
Estimating Conditional Distributions of Scores on an Alternate Form of a Test. Research Report. ETS RR-15-18
Author: Livingston, Samuel A.
Date: 2015-12-00
Type: Journal Articles
Subjects: Scores,Statistical Distributions,Research Reports,Equated Scores,Statistical Analysis,Test Reliability,Alternative Assessment,Accuracy,Computation

Quantitative information about test score reliability can be presented in terms of the distribution of equated scores on an alternate form of the test for test takers with a given score on the form taken. In this paper, we describe a procedure for estimating that distribution, for any specified score on the test form taken, by estimating the joint distribution of equated scores on two forms of the test. The procedure requires as input the score distribution and the reliability coefficient, estimated for the test-taker population. In tryouts, the procedure performed well for score distributions like those commonly encountered in large-scale testing situations.
Effect of Item Response Theory (IRT) Model Selection on Testlet-Based Test Equating. Research Report. ETS RR-14-19
Author: Cao, Yi
Date: 2014-12-00
Type: Journal Articles
Subjects: Item Response Theory,Equated Scores,Test Items,Simulation,Comparative Analysis,Multiple Choice Tests,Models

The local item independence assumption underlying traditional item response theory (IRT) models is often not met for tests composed of testlets. There are 3 major approaches to addressing this issue: (a) ignore the violation and use a dichotomous IRT model (e.g., the 2-parameter logistic [2PL] model), (b) combine the interdependent items to form a polytomous item and apply a polytomous IRT model (e.g., the graded response model [GRM]), and (c) apply a model that explicitly takes into account the dependence at the item level (e.g., the testlet response theory [TRT] model). In this study, a simulation was conducted to compare the performance of these 3 approaches on number-correct score equating when degrees of testlet effect were manipulated. The traditional equipercentile method was used as an evaluation baseline. The results show that the 2PL and the TRT approaches produce comparable results that more closely agree with the results of the equipercentile method than the GRM does. And the number-correct equating using the 2PL is robust to the violation of local item independence.
Assessing Critical Thinking in Higher Education: Current State and Directions for Next-Generation Assessment. Research Report. ETS RR-14-10
Author: Liu, Ou Lydia
Date: 2014-06-00
Type: Journal Articles
Subjects: Critical Thinking,Higher Education,Undergraduate Students,Definitions,Psychometrics,Student Evaluation,Evaluation Methods,Design,Educational Assessment,Cognitive Tests,Thinking Skills,Essay Tests,Validity,Outcomes of Education

Critical thinking is one of the most important skills deemed necessary for college graduates to become effective contributors in the global workforce. The first part of this article provides a comprehensive review of its definitions by major frameworks in higher education and the workforce, existing assessments and their psychometric qualities, and challenges surrounding the design, implementation, and use of critical thinking assessment. In the second part, we offer an operational definition that is aligned with the dimensions of critical thinking identified from the reviewed frameworks and discuss the key assessment considerations when designing a next-generation critical thinking assessment. This article has important implications for institutions that are currently using, planning to adopt, or designing an assessment of critical thinking.
The Key Practice, Discuss and Debate Ideas: Conceptual Framework, Literature Review, and Provisional Learning Progressions for Argumentation. Research Report. ETS RR-15-33
Author: Deane, Paul
Date: 2015-12-00
Type: Journal Articles
Subjects: Debate,Best Practices,Literature Reviews,Discourse Modes,Persuasive Discourse,Rhetorical Invention,Skill Analysis,Learning Activities,Communication Skills,Communication Strategies

In this paper, we provide a comprehensive literature review on the development of key argumentation skills to lay a foundation for a framework of the key practice, discuss and debate ideas, which is centrally involved in the expectations for academic reading and writing. Specifically, the framework includes 5 phases of core activities and related sets of argumentation skills, and for each set of skills, a provisional learning progression is designed to identify qualitative shifts in the development of critical argumentation skills informed by the developmental literature. These learning may have the potential to support teachers' instructional decisions that effectively scaffold their students to the next level.
Strengthening Statistics Graduate Programs with Statistical Collaboration--The Case of Hawassa University, Ethiopia
Author: Goshu, Ayele Taye
Date: 2016-00-00
Type: Journal Articles
Subjects: Foreign Countries,Graduate Study,Statistics,Cooperation,Masters Degrees,Doctoral Degrees,Scientific Research,Researchers,College Faculty,Graduate Students

This paper describes the experiences gained from the established statistical collaboration center at Hawassa University as part of LISA 2020 network. The center has got similar setup as LISA at Virginia Tech. Statisticians are trained on how to become more effective scientific collaborators with researchers. The services are being delivered since May 2015. The University has a well established and strong academic graduate programs of statistics. The master programs are: Applied Statistics, and Mathematical and Statistical Modelling launched in 2008 and 2010, respectively. They are research based studies. They have produced over one hundred sixty graduates to-date, with current enrollment of over fifty students. The doctoral program started in 2013, with enrollment of ten students. The graduate students are the main role players as statistical collaborators at the center. The collaborators and clients have revealed positive feedback about the services. It is observed that the collaboration scheme seem to have an impact on research quality of the non-statistician researchers. The role of statisticians will be important in the scientific researches of the University so as to meet the societal needs. The collaboration practice has a potential to enhance the statistics education and research at the department itself. The center needs to be strengthened and expanded for greater services and sharing experiences with other similar higher institutions.
SARA Reading Components Tests, RISE Forms: Technical Adequacy and Test Design, 2nd Edition. Research Report. ETS RR-15-32
Author: Sabatini, John
Date: 2015-12-00
Type: Journal Articles
Subjects: Reading Tests,Reading Skills,Test Construction,Research Reports,Item Response Theory,Test Bias,Reading Comprehension,Grade 5,Grade 6,Grade 7,Grade 8,Grade 9,Grade 10,Testing,Word Recognition,Decoding (Reading),Vocabulary,Morphology (Languages),Sentence Structure,Correlation,Psychometrics

This technical report describes the conceptual foundation and measurement properties of the Reading Inventory and Scholastic Evaluation (RISE). The RISE is a 6-subtest, Web-administered reading skills components battery. The theoretical and empirical foundations of each subtest in the battery are reviewed, as well as item designs. The results included in this report feature a vertical extension of the RISE to span Grades 5-10, psychometric analysis of parallel forms of each subtest, results of item response theory (IRT) scaling studies for each of the subtests across the entire grade span, and evaluation of differential item functioning (DIF) for gender and race/ethnicity.
Improving Content Assessment for English Language Learners: Studies of the Linguistic Modification of Test Items. Research Report. ETS RR-14-23
Author: Young, John W.
Date: 2014-12-00
Type: Journal Articles
Subjects: English Language Learners,Test Items,Mathematics Tests,Science Tests,Elementary School Students,Testing Accommodations,State Standards,Test Construction,Item Analysis,Language Usage,Achievement Gap,Statistical Analysis,Student Attitudes,Cognitive Processes,Spanish Speaking,Language Tests,Language Proficiency,Interviews,Qualitative Research

This article describes two research studies conducted on the linguistic modification of test items from K-12 content assessments. In the first study, 120 linguistically modified test items in mathematics and science taken by fourth and sixth graders were found to have a wide range of outcomes for English language learners (ELLs) and non-ELLs, with regard to performance on the original and modified versions of the same items. The original items were disclosed items from two states and were used in their standards-based assessments.The modified versions of these items were developed by a team of Educational Testing Service (ETS) assessment developers and researchers. The ETS research team identified several kinds of modifications that appeared promising and applied the modifications systematically. Ideally, modifying items leads to improved performance by ELLs while having little to no impact on the performance of non-ELLs. However, both groups of students performed better on some items, about the same on some other items, and worse on yet some other items. Cognitive interviews were conducted as a follow-up study to investigate which features of the linguistically modified items produced the observed outcomes for ELLs and non-ELLs. The results from this study added little clarity with regards to which of the linguistic modifications were effective and how they improved the understanding and performance of ELLs on these content items.