ethanporcaro/eric-fulltext-meta huggingface dataset:
A Bayesian Hierarchical Model for Large-Scale Educational Surveys: An Application to the National Assessment of Educational Progress. Research Report. ETS RR-04-38
Author: Johnson, Matthew S.
Date: 2005-01-00
Type: Journal Articles
Subjects: Bayesian Statistics,Hierarchical Linear Modeling,National Competency Tests,Sampling,Computation,Markov Processes,Monte Carlo Methods,Comparative Analysis,Item Response Theory,Reading Tests,Grade 8

Large-scale educational assessments such as the National Assessment of Educational Progress (NAEP) sample examinees to whom an exam will be administered. In most situations the sampling design is not a simple random sample and must be accounted for in the estimating model. After reviewing the current operational estimation procedure for NAEP, this paper describes a Bayesian hierarchical model for the analysis of complex large-scale assessments. The model clusters students within schools and schools within primary sampling units. The paper discusses an estimation procedure that utilizes a Markov chain Monte Carlo algorithm to approximate the posterior distribution of the model parameters. Results from two Bayesian models, one treating item parameters as known and one treating them as unknown, are compared to results from the current operational method on a simulated data set and on a subset of data from the 1998 NAEP reading assessment. The point estimates from the Bayesian model and the operational method are quite similar in most cases, but there does seem to be systematic differences in measures of uncertainty (e.g., standard errors, confidence intervals). [Note: Figure 1 is missing from the ERIC version of this report.]
Multiple Personality Domains in Relation to Occupational Choice and Performance among Established Teachers. Research Report. ETS RR-04-41
Author: Emmerich, Walter
Date: 2004-10-00
Type: Journal Articles
Subjects: Personality,Career Choice,Job Performance,Interests,Teacher Effectiveness,Specialization,Attribution Theory,Personality Measures,Construct Validity,Teaching Experience,Science Teachers,English Teachers,Mathematics Teachers,Art Teachers,Special Education Teachers,Vocational Education Teachers,Secondary School Teachers,Elementary School Teachers

Does personality impact differently on occupational choice and occupational performance? In a study of established teachers, interests, the Five-Factor Model, and attributions were examined in relation to occupational choice and performance. Attributions were assessed using a new instrument designed for teachers. Choice of teaching specialty was found to be most strongly associated with interest in the arts and sciences, openness to experience, and internal attributions in response to a positive classroom event. On the other hand, quality of teaching assessed by the National Board of Professional Teaching Standards (NBPTS) was associated (negatively) with business interests and with a tendency to externalize blame in response to a problematic teaching situation. Study findings suggested the hypothesis that broad traits located relatively high in a personality hierarchy influence trait matching in selecting an occupation, whereas more specific motivational/dynamic factors located lower in the personality hierarchy influence performance within the chosen niche. The report includes supplemental evidence on the construct validity of the personality measures for assessing teachers.
Automated Essay Scoring with e-rater® v.2.0. Research Report. ETS RR-04-45
Author: Attali, Yigal
Date: 2005-11-00
Type: Journal Articles
Subjects: Essay Tests,Automation,Scoring,Comparative Analysis,Validity,Reliability,Business Administration Education,Business Schools,College Entrance Examinations,Graduate Study,Language Tests,English (Second Language),Second Language Learning,Writing Tests,Grade 6,Grade 7,Grade 8,Grade 9,Grade 10,Grade 11,Grade 12

The e-rater® system has been used by ETS for automated essay scoring since 1999. This paper describes a new version of e-rater (v.2.0) that differs from the previous one (v.1.3) with regard to the feature set and model building approach. The paper describes the new version, compares the new and previous versions in terms of performance, and presents evidence on the validity and reliability of scores produced by the new version.
Loglinear Smoothing: An Alternative Numerical Approach Using SAS. Research Report. ETS RR-04-27
Author: Moses, Tim
Date: 2004-07-00
Type: Journal Articles
Subjects: Statistical Analysis,Statistical Distributions,Comparative Analysis,Graphs,Goodness of Fit,Models

The purpose of this report is to demonstrate loglinear smoothing using SAS PROC GENMOD. The results from four published examples, which include the smoothing of a) univariate distributions, b) bivariate distributions, c) distributions with teeth, and d) bivariate distributions with structural zeros, are reproduced to show the flexibility of the SAS procedure. Comparisons of graphical displays and likelihood ratio statistics show that the SAS procedure yields results identical to the published results. SAS PROC GENMOD provides an alternative approach to smoothing that is readily available for researchers and graduate students in educational measurement, as well as researchers from other scientific fields.
Examining Freedle's Claims about Bias and His Proposed Solution: Dated Data, Inappropriate Measurement, and Incorrect and Unfair Scoring. Research Report. ETS RR-04-26
Author: Dorans, Neil J.
Date: 2004-07-00
Type: Journal Articles
Subjects: Test Bias,Statistical Bias,Psychometrics,College Entrance Examinations,Scoring,Difficulty Level,Test Reliability,Scores,Correlation,Test Items

In the Spring 2003 issue of &quot;Harvard Educational Review,&quot; Roy Freedle stated that the SAT® is both culturally and statistically biased, and he proposed a solution to ameliorate this bias. His claims, which garnered national attention, were based on serious errors in his analysis. We begin our analyses by assessing the psychometric properties of Freedle's recommended hard half-test that he thinks should form the basis for the supplemental SAT score he proposes to report. Next we demonstrate his justification for a score based on this half-test is based on a flawed analysis. The numbers in his critical Table 2 do not represent what he claims they do. We then demonstrate what occurs when current data are used with both the correct scoring and the incorrect scoring proposed by Freedle. When the table is constructed correctly using current data, the effects that Freedle reported are reduced substantially in magnitude to the point where they do not warrant any of the corrective actions he proposes. We conclude our analysis of Freedle's claims with DIF analyses of our own that are inconsistent with his DIF analyses of old SAT test editions he used as his initial motivation for correcting the SAT scoring procedure.
Improved Reliability Estimates for Small Samples Using Empirical Bayes Techniques. Research Report. ETS RR-09-46
Author: Oh, Hyeonjoo J.
Date: 2009-12-00
Type: Journal Articles
Subjects: Standardized Tests,Test Reliability,Sample Size,Bayesian Statistics,Accuracy,Error of Measurement,Statistical Distributions,Aptitude Tests,High Schools,College Entrance Examinations,High School Students,Scores,Equations (Mathematics)

Issues of equity and fairness across subgroups of the population (e.g., gender or ethnicity) must be seriously considered in any standardized testing program. For this reason, many testing programs require some means for assessing test characteristics, such as reliability, for subgroups of the population. However, often only small sample sizes are available for the subgroups of interest. Traditionally used reliability estimates (e.g., Cronbach's alpha) can have low precision for small samples. This study investigated whether an empirical Bayes (EB) technique could produce more precise reliability estimates than traditional methods in the presence of small samples. Several Bayesian estimates were compared to estimates obtained by other methods (e.g., the traditionally and currently used Cronbach's alpha coefficient), in terms of both bias and variance. A secondary purpose of this study was to compare the various EB approaches across different sample sizes. This paper also discusses EB estimates of standard error of measurement (SEM), their accuracy and precision, and how they compare with SEM estimates derived from the alpha.
A Study of Frequency Estimation Equipercentile Equating When There Are Large Ability Differences. Research Report. ETS RR-09-45
Author: Guo, Hongwen
Date: 2009-12-00
Type: Journal Articles
Subjects: Equated Scores,Computation,Statistical Analysis,Test Items,Difficulty Level,Probability,Ability,Validity,Mathematical Logic

In operational equating, frequency estimation (FE) equipercentile equating is often excluded from consideration when the old and new groups have a large ability difference. This convention may, in some instances, cause the exclusion of one competitive equating method from the set of methods under consideration. In this report, we study the possibility of using the FE equating method when the group ability difference is large. Three situations are identified: (a) a situation in which neither the two forms nor the observed conditional distributions are very different so that the FE equating assumptions are likely to hold, and FE equating is recommended; (b) a situation in which forms are not very different, but the observed conditional distributions are different, so that FE equating is not recommended; and (c) a situation in which forms are very different, but the observed conditional distributions are not different, so that FE equating is not recommended. Statistical analysis procedures for comparing distributions are provided. An application of equating to a large-scale admission test is discussed to illustrate the proposed methodology.
Stereotype Threat, the Test-Center Environment, and Performance on the GRE General Test. GRE Board Report. GRE 01-03R. ETS Research Report. RR-04-37
Author: Walters, Alyssa M.
Date: 2004-12-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Education Service Centers,Testing Problems,Stereotypes,Scores,African American Students,Hispanic American Students,Asian American Students,Females,Environmental Influences,Supervision,Ethnicity,Statistical Analysis,Correlation

The study investigated the applicability of previous experimental research on stereotype threat to operational Graduate Record Examinations® (GRE®) General Test testing centers. The goal was to document any relationships between features of the testing environment that might cue stereotype threat as well as any impact on GRE test scores among African American, Hispanic, Asian American, and female test-takers. Among such features were the gender and ethnicity of test proctors and more general factors, such as the size, activity level, and social atmosphere of test centers. Our analyses revealed several relationships among environmental factors and several variations in test performance for all groups. However, we found no direct support for stereotype threat and, in fact, found some effects for proctor ethnicity that ran counter to a stereotype-threat explanation.
Jackknifing Techniques for Evaluation of Equating Accuracy. Research Report. ETS RR-09-39
Author: Haberman, Shelby J.
Date: 2009-12-00
Type: Journal Articles
Subjects: Equated Scores,Accuracy,Sampling,Statistical Analysis,Item Response Theory,True Scores

Grouped jackknifing may be used to evaluate the stability of equating procedures with respect to sampling error and with respect to changes in anchor selection. Properties of grouped jackknifing are reviewed for simple-random and stratified sampling, and its use is described for comparisons of anchor sets. Application is made to examples of item response theory (IRT) true-score equating in which two-parameter logistic and general partial credit models are employed.
Using Score Equity Assessment to Evaluate the Equatability of the Hardest Half of a Test to the Total Test. Research Report. ETS RR-04-43
Author: Dorans, Neil J.
Date: 2004-10-00
Type: Journal Articles
Subjects: Scores,Scoring,Equated Scores,College Entrance Examinations,Test Bias,Prediction,Statistical Analysis,White Students,African American Students,Regression (Statistics)

In an article published in the spring 2003 issue of &quot;Harvard Educational Review&quot;, Roy Freedle stated that the SAT® is both culturally and statistically biased. Freedle proposed a solution to this bias, which involved using a half-test made up of the most difficult items culled from complete SAT examination. His claims, which garnered national attention, were based on serious errors in his analysis. In Dorans and Zeller (2004), we demonstrated that the effects Freedle reported are reduced substantially when the data are analyzed correctly. Here, we describe a sound way of assessing whether the current SAT scoring procedure is fair, and we examine what happens when we subject the Freedle's hard half-test to a score-equity assessment.
Tolerable Variation in Item Parameter Estimates for Linear and Adaptive Computer-Based Testing. Research Report. ETS RR-04-28
Author: Rizavi, Saba
Date: 2004-07-00
Type: Journal Articles
Subjects: Test Items,Computer Assisted Testing,Computation,Adaptive Testing,Item Banks,High Stakes Tests,Scores,Context Effect,Case Studies

Item parameter estimates vary for a variety of reasons, including estimation error, characteristics of the examinee samples, and context effects (e.g., item location effects, section location effects, etc.). Although we expect variation based on theory, there is reason to believe that observed variation in item parameter estimates exceeds what theory would predict. This study examined both items that were administered linearly in a fixed order each time that they were used and items that had appeared in different adaptive testing item pools. The study looked at both the magnitude of variation in the item parameter estimates and the impact of this variation in the estimation of test-taker scores. The results showed that the linearly administered items exhibited remarkably small variation in parameter estimates over repeated calibrations. Similar findings with adaptively administered items in another high stakes testing program were also found when initial adaptively based item parameter estimates were compared with estimates from repeated use. The results of this study also indicated that context effects played a more significant role in adaptive item parameters when the comparisons were made to the parameters that were initially obtained from linear paper-and-pencil testing.
Conditional Covariance Theory and DETECT for Polytomous Items. Research Report. ETS RR-04-50
Author: Zhang, Jinming
Date: 2004-12-00
Type: Journal Articles
Subjects: Test Items,Test Theory,Correlation,National Competency Tests,Reading Tests,Grade 4,Grade 8,Scoring,Statistical Analysis,Simulation

This paper extends the theory of conditional covariances to polytomous items. It has been mathematically proven that under some mild conditions, commonly assumed in the analysis of response data, the conditional covariance of two items, dichotomously or polytomously scored, is positive if the two items are dimensionally homogeneous and negative otherwise. The theory provides a theoretical foundation for dimensionality assessment procedures based on conditional covariances or correlations, such as DETECT and DIMTEST, so that the performance of these procedures is theoretically justified when applied to response data with polytomous items. Various estimators of conditional covariances are constructed, and special attention is paid to the case of complex sampling data, such as NAEP data. As such, the new version of DETECT can be applied to response data sets not only with polytomous items but also with missing values, either by design or by random. DETECT is then applied to analyze the dimensional structure of the 2002 NAEP reading samples of grades 4 and 8. The DETECT results show that the substantive test structure based on the purposes for reading is consistent with the statistical dimensional structure for either grade. The results also indicate that the degree of multidimensionality of the NAEP reading data is weak. [This report is an extension and revision of the paper presented at the annual meeting of American Educational Research Association, Chicago, Illinois, April 1997. The research was supported by Educational Testing Service and the National Assessment of Educational Progress.]
Evaluating Subpopulation Invariance of Linking Functions to Determine the Anchor Composition for a Mixed-Format Test. Research Report. ETS RR-09-36
Author: Kim, Sooyeon
Date: 2009-12-00
Type: Journal Articles
Subjects: Test Format,Equated Scores,Test Items,Multiple Choice Tests,Responses,Gender Differences,Comparative Analysis,Cutting Scores

We examined the appropriateness of the anchor composition in a mixed-format test, which includes both multiple-choice (MC) and constructed-response (CR) items, using subpopulation invariance indices. We derived linking functions in the nonequivalent groups with anchor test (NEAT) design using two types of anchor sets: (a) MC only and (b) a mix of MC and CR. In each anchor condition, we derived the linking functions separately for males and females and then compared those subpopulation functions to the total group function. In the MC-only condition, the difference between the subpopulation functions and the total group function was not trivial in a score region that included cut scores, leading to inconsistent pass/fail decisions for low-performing examinees in particular. Overall, the mixed anchor was a better choice than the MC-only anchor to achieve subpopulation invariance between males and females. The research reinforces subpopulation invariance indices as a means of determining the adequacy of the anchor.
Methods of Linking with Small Samples in a Common-Item Design: An Empirical Comparison. Research Report. ETS RR-09-38
Author: Kim, Sooyeon
Date: 2009-12-00
Type: Journal Articles
Subjects: Sampling,Sample Size,Accuracy,Test Items,Equated Scores,Test Construction,Error of Measurement,Comparative Analysis,Ability,Scoring

A series of resampling studies was conducted to compare the accuracy of equating in a common item design using four different methods: chained equipercentile equating of smoothed distributions, chained linear equating, chained mean equating, and the circle-arc method. Four operational test forms, each containing more than 100 items, were used for the equating, with new-form samples of 100, 50, 25, and 10 examinees and reference-form samples three times as large. Accuracy was assessed in terms of the root-mean-squared difference (over 1,000 replications) of the sample equatings from the criterion equating. Overall, chained mean equating produced the most accurate results for low scores, but the circle-arc method produced the most accurate results, particularly in the upper half of the score distribution.
Three Statistical Testing Procedures in Logistic Regression: Their Performance in Differential Item Functioning (DIF) Investigation. Research Report. ETS RR-09-35
Author: Paek, Insu
Date: 2009-12-00
Type: Journal Articles
Subjects: Test Bias,Statistical Analysis,Regression (Statistics),Maximum Likelihood Statistics,Comparative Analysis,Simulation,Differences,Sample Size

Three statistical testing procedures well-known in the maximum likelihood approach are the Wald, likelihood ratio (LR), and score tests. Although well-known, the application of these three testing procedures in the logistic regression method to investigate differential item function (DIF) has not been rigorously made yet. Employing a variety of simulation conditions, this research (a) assessed the three tests' performance for DIF detection and (b) compared DIF detection in different DIF testing modes (targeted vs. general DIF testing). Simulation results showed small differences between the three tests and different testing modes. However, targeted DIF testing consistently performed better than general DIF testing; the three tests differed more in performance in general DIF testing and nonuniform DIF conditions than in targeted DIF testing and uniform DIF conditions; and the LR and score tests consistently performed better than the Wald test.
Study of the Impact of the California Formative Assessment and Support System for Teachers, Report 1. Beginning Teachers' Engagement with BTSA/CFASST. Research Report. ETS RR-04-30
Author: Thompson, Marnie
Date: 2004-08-00
Type: Journal Articles
Subjects: Beginning Teachers,Program Effectiveness,Formative Evaluation,Beginning Teacher Induction,Teacher Influence,Academic Achievement,Evaluation Research,Questionnaires,Surveys,Elementary School Teachers,Prediction,Grade 3,Grade 4,Grade 5,Interviews,Teacher Effectiveness,Multiple Regression Analysis,Statistical Analysis

This report is the first of four that stem from a study of The Impact of Approved Induction Programs on Student Learning (IAIPSL), conducted by Educational Testing Service (ETS) and funded by the California Commission on Teacher Credentialing (CCTC). The IAIPSL study began in July 2002 and continued through April 2004. The purpose of the study is to investigate the implementation and impact of the California Formative Assessment and Support System for Teachers (CFASST), within the context of the Beginning Teacher Support and Assessment program (BTSA)1. The study addresses three research questions: (1) What is the impact of BTSA/CFASST on the teaching effectiveness of beginning teachers who participate in the program? (2) What is the impact of BTSA/CFASST on the learning of the students of beginning teachers who participate in the program? (3) What are the features of successful BTSA/CFASST programs that make them more effective in impacting beginning teachers' growth as teachers? In addition, the IAIPSL study provides the grist for a &quot;meta-evaluation&quot; in which the study is examined for insights into the processes and validity of conducting evaluation research on statewide induction programs such as BTSA/CFASST. This report focuses on what has been learned about how BTSA/CFASST is being implemented and experienced at the level of the beginning teacher. The report presents and analyzes findings from two main data sources: a written questionnaire and follow-up telephone interviews with BTSA/CFASST &quot;graduates.&quot; Other reports in the series present the methods and findings of other phases of data collection and analysis that comprise the larger study
Comparison of the Effects of Discrete Anchor Items and Assage-Based Anchor Items on Observed-Score Equating Results. Research Report. ETS RR-09-44
Author: Zu, Jiyun
Date: 2009-12-00
Type: Journal Articles
Subjects: Comparative Analysis,Equated Scores,Test Items,Simulation,Bias,Error of Measurement

Equating of tests composed of both discrete and passage-based items using the nonequivalent groups with anchor test (NEAT) design is popular in practice. This study investigated the impact of discrete anchor items and passage-based anchor items on observed score equating via simulation. Results suggested that an anchor with a larger proportion of passage-based items and/or a larger degree of local dependence among passage-based items produces larger equating errors, especially when group ability differences are not minimal. Our findings challenge the common belief that an anchor should be a miniature version of the tests to be equated.