ethanporcaro/eric-fulltext-meta huggingface dataset:
An Initial Field Trial of an Instrument for Measuring Learning Strategies of Middle School Students. Research Report. ETS RR-08-03
Author: Liu, Ou Lydia
Date: 2008-02-00
Type: Journal Articles
Subjects: Learning Strategies,Middle School Students,Measurement Techniques,Metacognition,Factor Analysis,Pilot Projects,Teacher Effectiveness,Questionnaires,Statistical Analysis,Correlation,Grade 6,Grade 7,Grade 8,Likert Scales

Learning strategies have been increasingly recognized as a useful tool to promote effective learning. In response to the lack of available learning strategies measures for middle school students, this study designed an instrument for these students, assessing behavioral, cognitive, and metacognitive strategies. This instrument, the Middle School Learning Strategies (MSLS) scale, is examined in terms of factorial structure, reliability, and correlates. Three factors emerge from the analyses: &quot;effective strategies,&quot; &quot;help seeking,&quot; and &quot;bad habits.&quot; The subscales displayed a reasonable reliability, ranging from 0.70 to 0.87. Student grades in language arts, social studies, math, and science were collected as criterion variables. As expected, grades in these four subjects correlated positively with both effective strategies and help seeking, yet negatively with bad habits. As a pilot measure, this instrument has demonstrated promising features as a useful tool for students to evaluate and enhance their learning strategies.
Automated Scoring of Short-Answer Open-Ended GRE® Subject Test Items. ETS GRE® Board Research Report No. 04-02. ETS RR-08-20
Author: Attali, Yigal
Date: 2008-04-00
Type: Tests/Questionnaires
Subjects: College Entrance Examinations,Graduate Study,Scoring,Test Construction,Test Items,Biology,Psychology,Automation,Test Scoring Machines,Feedback (Response),Scores,Questionnaires,Attitude Measures,Student Attitudes

This report describes the development, administration, and scoring of open-ended variants of GRE® Subject Test items in biology and psychology. These questions were administered in a Web-based experiment to registered examinees of the respective Subject Tests. The questions required a short answer of 1-3 sentences, and responses were automatically scored by natural language processing methods, using the &quot;c-rater&quot;™ scoring engine, immediately after participants submitted their responses. Participants received immediate feedback on the correctness of their answers, and an opportunity to revise their answers. Subsequent human scoring of the responses allowed an evaluation of the quality of automated scoring. This report focuses on the success of the automated scoring process. A separate report describes the feedback and revision results.
Impossible Scores Resulting in Zero Frequencies in the Anchor Test: Impact on Smoothing and Equating. Research Report. ETS RR-08-10
Author: Puhan, Gautam
Date: 2008-03-00
Type: Reports - Research
Subjects: Equated Scores,Test Items,Raw Scores,Statistical Analysis,Cutting Scores,Multiple Choice Tests,Responses,Error of Measurement

Equating under the external anchor design is frequently conducted using scaled scores on the anchor test. However, scaled scores often lead to the unique problem of creating zero frequencies in the score distribution because there may not always be a one-to-one correspondence between raw and scaled scores. For example, raw scores of 17 and 18 may correspond to scaled scores of 150 and 153, thereby creating zero frequencies for scaled scores of 151 and 152. These gaps in the frequency distribution may adversely impact smoothing and equating. This study examines the effect of these zero frequencies on log-linear smoothing (Holland &amp; Thayer, 1987) of score distributions and final equating results. Results suggest that although smoothing is significantly affected by the presence of these zero frequencies, as indicated by the likelihood-ratio chisquare, Akaike information criterion (Akaike, 1977), and Freeman-Tukey deviates, the impact on the actual equating results is minimal.
Comparing Alternative Kernels for the Kernel Method of Test Equating: Gaussian, Logistic, and Uniform Kernels. Research Report. ETS RR-08-12
Author: Lee, Yi-Hsuan
Date: 2008-03-00
Type: Journal Articles
Subjects: Equated Scores,Comparative Analysis,Methods,Accuracy,Statistical Analysis,Efficiency

The kernel equating method (von Davier, Holland, &amp; Thayer, 2004) is based on a flexible family of equipercentile-like equating functions that use a Gaussian kernel to continuize the discrete score distributions. While the classical equipercentile, or percentile-rank, equating method carries out the continuization step by linear interpolation, in principle the kernel equating methods could use various kernel smoothings to replace the discrete score distributions. This paper expands the work of von Davier et al. (2004) in investigating alternative kernels for equating practice. To examine the influence of different kernel functions on the equating results, this paper focuses on two types of kernel functions: the logistic kernel and the continuous uniform distribution (known to be the same as the linear interpolation). The Gaussian kernel is used for reference. By employing an equivalent-groups design, the results of the study indicate that the tail properties of kernel functions have great impact on the continuized score distributions. However, the equated scores based on different kernel functions do not vary much, except for extreme scores. The results presented in this paper not only support the previous findings on the efficiency and accuracy of the existing continuization methods, but also enrich the information on observed-score equating models.
Robustness of Value-Added Analysis of School Effectiveness. Research Report. ETS RR-08-22
Author: Braun, Henry
Date: 2008-04-00
Type: Journal Articles
Subjects: Value Added Models,School Effectiveness,Robustness (Statistics),Computation,Item Response Theory,Comparative Analysis,Growth Models,Longitudinal Studies,Surveys,Elementary School Students,Primary Education

This paper reports on a study conducted to investigate the consistency of the results between 2 approaches to estimating school effectiveness through value-added modeling. Estimates of school effects from the layered model employing item response theory (IRT) scaled data are compared to estimates derived from a discrete growth model based on the analysis of transitions along an ordinal developmental scale. The data were extracted from the longitudinal records maintained in the Early Childhood Longitudinal Study-Kindergarten Cohort (ECLS-K) archive for students remaining in the same school from the beginning of kindergarten through the end of Grade 3. The results of different comparisons indicated that the estimates from the 2 approaches are moderately consistent.
A Study of Confidence and Accuracy Using the Rasch Modeling Procedures. Research Report. ETS RR-08-42
Author: Paek, Insu
Date: 2008-07-00
Type: Journal Articles
Subjects: Self Esteem,Accuracy,Item Response Theory,English,Language Tests,Language Proficiency,Test Items,Difficulty Level,Test Wiseness,Psychometrics,Equations (Mathematics),Reading,Listening

This study investigated the relationship between students' actual performance (accuracy) and their subjective judgments of accuracy (confidence) on selected English language proficiency tests. The unidimensional and multidimensional IRT Rasch approaches were used to model the discrepancy between confidence and accuracy at the item and test level and to assess disattenuated strength of association between accuracy and confidence. The analysis results indicate a pattern of overconfidence bias (i.e., overestimation of success rate), which was related to item difficulty. In addition, the strength of association between accuracy and confidence dimension was relatively high: The confidence dimension explained 45% and 52% of the variability in the accuracy dimension for the two tests employed in this study.
Investigating the Effectiveness of Collateral Information on Small-Sample Equating. Research Report. ETS RR-08-52
Author: Kim, Sooyeon
Date: 2008-10-00
Type: Journal Articles
Subjects: Equated Scores,Statistical Analysis,Sample Size,Bayesian Statistics,Computation,Raw Scores,Error of Measurement,Accuracy,Licensing Examinations (Professions),Teacher Certification

This paper describes an empirical evaluation of a Bayesian procedure for equating scores on test forms taken by small numbers of examinees, using collateral information from the equating of other test forms. In this procedure, a separate Bayesian estimate is derived for the equated score at each raw-score level, making it unnecessary to specify a parametric model for the equating function. Collateral information can come either from other forms of the same test or, possibly, from other tests having a similar structure. Our evaluation consisted of two resampling studies. Each study applied the Bayesian procedure to small samples drawn from large-sample data collected for an anchor equating. The large-sample equating function served as the criterion. The results of the two studies were somewhat inconsistent, leading to different conclusions regarding the use of the empirical Bayesian procedure with small samples.
The Assessment of Information Literacy: A Case Study. Research Report. ETS RR-08-33
Author: Katz, Irvin R.
Date: 2008-06-00
Type: Journal Articles
Subjects: Information Literacy,Information Skills,Skill Analysis,Case Studies,Undergraduate Students,Outcome Measures,Basic Skills,Portfolio Assessment,Performance Based Assessment,Writing Tests,Interrater Reliability,Scoring,Cohort Analysis,Correlation,Predictor Variables,Construct Validity,Achievement Gains

This study presents an investigation of information literacy as defined by the ETS iSkills™ assessment and by the New Jersey Institute of Technology (NJIT) Information Literacy Scale (ILS). As two related but distinct measures, both iSkills and the ILS were used with undergraduate students at NJIT during the spring 2006 semester. Undergraduate students (n = 331), first through senior years, took the iSkills and submitted portfolios to be judged by the ILS. First-year students took the Core iSkills assessment, which was designed to provide administrators and faculty with an understanding of the information and communication technology (ICT) literacy of a student doing entry-level coursework (n = 155). Upper classmen took the more difficult Advanced iSkills assessment, appropriate for rising juniors (n = 176). Across all class levels, iSkills scores varied as expected. First-year basic skills writing students performed at lower levels than first-year students enrolled in traditional composition and cultural history courses; seniors performed at higher levels than sophomores and juniors. Because the NJIT ILS scores were designed to be curriculum sensitive, portfolio scores did not similarly follow grade levels. Analyses revealed weak correlations between portfolio and Core iSkills scores and moderate correlations between portfolio and Advanced iSkills scores. As two associated yet distinct systems of inquiry designed to explore undergraduate student performance, the ETS iSkills assessment and the NJIT ILS--taken both individually and together--yield important information regarding student performance.
Sample-Size Requirements for Automated Essay Scoring. Research Report. ETS RR-08-32
Author: Haberman, Shelby J.
Date: 2008-06-00
Type: Journal Articles
Subjects: Sample Size,Scoring,Essays,Automation,Test Scoring Machines,Regression (Statistics),Prompting

Sample-size requirements were considered for automated essay scoring in cases in which the automated essay score estimates the score provided by a human rater. Analysis considered both cases in which an essay prompt is examined in isolation and those in which a family of essay prompts is studied. In typical cases in which content analysis is not employed and in which the only object is to score individual essays to provide feedback to the examinee, it appears that several hundred essays are sufficient. For application of one model to a family of essays, fewer than 100 essays per prompt may often be adequate. The cumulative logit model was explored as a possible replacement of the linear regression model usually employed in automated essay scoring; the cumulative logit model performed somewhat better than did the linear regression model.
Fitting the Structured General Diagnostic Model to NAEP Data. Research Report. ETS RR-08-27
Author: Xu, Xueli
Date: 2008-05-00
Type: Journal Articles
Subjects: National Competency Tests,Models,Data Analysis,Reading Tests,Educational Assessment,Reading Achievement,Test Items,Item Response Theory,Statistical Analysis,Probability,Inferences,Cognitive Ability

Xu and von Davier (2006) demonstrated the feasibility of using the general diagnostic model (GDM) to analyze National Assessment of Educational Progress (NAEP) proficiency data. Their work showed that the GDM analysis not only led to conclusions for gender and race groups similar to those published in the NAEP Report Card, but also allowed flexibility in estimating multidimensional skills simultaneously. However, Xu and von Davier noticed that estimating the latent skill distributions will be much more challenging with this model when there is a large number of subgroups to estimate. To make the GDM more applicable to NAEP data analysis, which requires a fairly large subgroups analysis, this study developed a log-linear model to reduce the number of parameters in the latent skill distribution without sacrificing the accuracy of inferences. This paper describes such a model and applies the model in the analysis of NAEP reading assessments for 2003 and 2005. The comparisons between using this model and the unstructured model were made through the use of various results, such as the differences between item parameter estimates and the differences between estimated latent class distributions. The results in general show that using the log-linear model is efficient.
Linking with Continuous Exponential Families: Single-Group Designs. Research Report. ETS RR-08-61
Author: Haberman, Shelby J.
Date: 2008-11-00
Type: Journal Articles
Subjects: Equated Scores,Probability,Statistical Distributions,Models,Comparative Analysis,Computation,Reliability,Mathematics,Multivariate Analysis

Continuous exponential families are applied to linking forms via a single-group design. In this application, a distribution from the continuous bivariate exponential family is used that has selected moments that match those of the bivariate distribution of scores on the forms to be linked. The selected continuous bivariate distribution then yields continuous univariate marginal distributions for the two forms. These marginal distributions then provide distribution functions and quantile functions that may be employed in equating. Normal approximations are obtained for the sample distributions of the conversion functions.
Response to Assessment Feedback: The Effects of Grades, Praise, and Source of Information. Research Report. ETS RR-08-30
Author: Lipnevich, Anastasiya A.
Date: 2008-06-00
Type: Journal Articles
Subjects: Feedback (Response),Information Sources,College Students,Essays,Writing (Composition),Predictor Variables,Grades (Scholastic),Recognition (Achievement),Student Improvement,Essay Tests,Scores,Evaluation Problems,Performance Based Assessment,Computer Assisted Testing,Student Motivation,Self Efficacy,Accuracy,Usability,Instructional Effectiveness,Questionnaires,Likert Scales,Multivariate Analysis,Statistical Analysis

This experiment involved college students (N = 464) working on an authentic learning task (writing an essay) under 3 conditions: no feedback, detailed feedback (perceived by participants to be provided by the course instructor), and detailed feedback (perceived by participants to be computer generated). Additionally, conditions were crossed with 2 factors of grade (receiving grade or not) and praise (receiving praise or not). Detailed feedback specific to individual work was found to be strongly related to student improvement in essay scores, with the influence of grades and praise more complex. Overall, detailed, descriptive feedback was found to be most effective when given alone, unaccompanied by grades or praise. The results have implications for theory and practice of assessment.
Effects of Calculator Availability on GRE Quantitative Questions. ETS GRE Board Research Report No. 03-09. ETS RR-08-31
Author: Bridgeman, Brent
Date: 2008-06-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Calculators,Test Items,Effect Size,Difficulty Level,Computation,Logical Thinking,Thinking Skills,Scores,Regression (Statistics)

In order to estimate the likely effects on item difficulty when a calculator becomes available on the quantitative section of the Graduate Record Examinations® (GRE®-Q), 168 items (in six 28-item forms) were administered either with or without access to an on-screen four-function calculator. The forms were administered as a special research section at the end of operational tests, with student volunteers randomly assigned to the calculator or no-calculator groups. Usable data were obtained from 13,159 participants. Test development specialists were asked to rate which items they thought would become easier with a calculator. In general, the specialists were successful in identifying the items with relatively large calculator effects, though even these effects were quite small. An increase of only about four points in the percent correct should suffice for the items identified as likely to show calculator effects with no adjustment needed for the majority of the items. Introduction of a calculator should have little or no effect on gender and ethnic differences.
Estimating Multidimensional Item Response Models with Mixed Structure. Research Report. ETS RR-05-04
Author: Zhang, Jinming
Date: 2005-04-00
Type: Journal Articles
Subjects: Item Response Theory,Computation,Mathematics,Simulation,Test Items,Correlation

This study derived an expectation-maximization (EM) algorithm for estimating the parameters of multidimensional item response models. A genetic algorithm (GA) was developed to be used in the maximization step in each EM cycle. The focus of the EM-GA algorithm developed in this paper was on multidimensional items with &quot;mixed structure.&quot; Simulated item response data were generated and then estimated by a computer program based on the EM-GA algorithm. The simulation results demonstrate that the EM-GA algorithm is a very promising approach in estimating multidimensional item response model parameters.
A Review of Recent Developments in Differential Item Functioning. Research Report. ETS RR-08-43
Author: Mapuranga, Raymond
Date: 2008-08-00
Type: Journal Articles
Subjects: Test Bias,Educational Development,Evaluation Methods,Statistical Analysis,Evaluation Criteria,Classification,Nonparametric Statistics,Causal Models,Item Response Theory,Test Theory,Data Interpretation,Error of Measurement,Design Requirements,Cost Effectiveness,Educational Practices

In many practical settings, essentially the same differential item functioning (DIF) procedures have been in use since the late 1980s. Since then, examinee populations have become more heterogeneous, and tests have included more polytomously scored items. This paper summarizes and classifies new DIF methods and procedures that have appeared since the early 1990s and assesses their appropriateness for practical use. Widely used DIF methods are evaluated alongside these new methods for completeness, clarity, and comparability.
Cognitive Models of Writing: Writing Proficiency as a Complex Integrated Skill
Author: Deane, Paul
Date: 2008-10-00
Type: Journal Articles
Subjects: Writing Skills,Writing Instruction,Schemata (Cognition),Writing Evaluation,Models,Kindergarten,Elementary Secondary Education,Test Construction,Test Format,Writing Tests,Critical Thinking,Literacy,Writing Processes,Audiences,Teaching Methods,Scaffolding (Teaching Technique),Multivariate Analysis,Expository Writing,Prior Learning,Persuasive Discourse,Cognitive Processes

This paper undertakes a review of the literature on writing cognition, writing instruction, and writing assessment with the goal of developing a framework and competency model for a new approach to writing assessment. The model developed is part of the Cognitively Based Assessments of, for, and as Learning (CBAL) initiative, an ongoing research project at ETS intended to develop a new form of kindergarten through Grade 12 (K-12) assessment that is based on modern cognitive understandings; built around integrated, foundational, constructed-response tasks that are equally useful for assessment and for instruction; and structured to allow multiple measurements over the course of the school year. The model that emerges from a review of the literature on writing places a strong emphasis on writing as an integrated, socially situated skill that cannot be assessed properly without taking into account the fact that most writing tasks involve management of a complex array of skills over the course of a writing project, including language and literacy skills, document-creation and document-management skills, and critical-thinking skills. As such, the model makes strong connections with emerging conceptions of reading and literacy, suggesting an assessment approach in which writing is viewed as calling upon a broader construct than is usually tested in assessments that focus on relatively simple, on-demand writing tasks.
DIF Detection with Small Samples: Applying Smoothing Techniques to Frequency Distributions in the Mantel-Haenszel Procedure. Research Report. ETS RR-08-44
Author: Yu, Lei
Date: 2008-08-00
Type: Journal Articles
Subjects: Test Bias,Item Response Theory,Sample Size,Evaluation Criteria,Error Patterns,Comparative Analysis,Statistical Analysis,College Entrance Examinations,White Students,Sampling,Asian Americans

All differential item functioning (DIF) methods require at least a moderate sample size for effective DIF detection. Samples that are less than 200 pose a challenge for DIF analysis. Smoothing can improve upon the estimation of the population distribution by preserving major features of an observed frequency distribution while eliminating the noise brought about by irregular data points. This study applied smoothing techniques to frequency distributions and investigated the impact of smoothed data on the Mantel-Haenszel (MH) DIF detection in small samples. Eight sample-size combinations were randomly drawn from a real data set to make the study realistic and were replicated 80 times to produce stable results. The population DIF results were used as the criteria to evaluate sample estimates using root-mean square difference (RMSD), bias analysis, and Type II error rate. Loglinear smoothing was found to provide slight to moderate improvements in MH DIF estimation with small samples.