ethanporcaro/eric-fulltext-meta huggingface dataset:
Examining the Impact of Audio Presentation on Tests of Reading Comprehension. Research Report. ETS RR-08-23
Author: Laitusis, Cara Cahalan
Date: 2008-04-00
Type: Journal Articles
Subjects: Standardized Tests,Oral Reading,Reading Comprehension,Grade 4,Grade 8,Audio Equipment,Auditory Stimuli,Reading Difficulties,Learning Disabilities,Scores,Reading Fluency,Reading Tests,Achievement Tests,Statistical Analysis,Surveys,Regression (Statistics)

This study examined the impact of a read-aloud accommodation on standardized test scores of reading comprehension at Grades 4 and 8. Under a repeated measures design, students with and without reading-based learning disabilities took both a standard administration and a read-aloud administration of a reading comprehension test. Results show that the mean score on the audio version was higher than scores on the standard version for both groups of students at both grade levels. Students with reading-based learning disabilities at both levels benefited differentially more than students with no disability. This finding continues to hold after controlling for reading fluency and ceiling effects at both grades. The results also examined the relationship between test scores and teachers' ratings of reading comprehension to determine which measures are the best predictors of teachers' ratings of reading comprehension by grade and disability classification.
Extension of the NAEP BGROUP Program to Higher Dimensions. Research Report. ETS RR-05-27
Author: Sinharay, Sandip
Date: 2005-12-00
Type: Journal Articles
Subjects: National Competency Tests,Regression (Statistics),Predictor Variables,Student Characteristics,Teacher Characteristics,Institutional Characteristics,Models,Computation,Maximum Likelihood Statistics,Measurement Techniques,Grade 8,Grade 12

The reporting methods used in large scale assessments such as the National Assessment of Educational Progress (NAEP) rely on a &quot;latent regression model.&quot; The first component of the model consists of a &quot;p&quot;-scale IRT measurement model that defines the response probabilities on a set of cognitive items in &quot;p&quot; scales depending on a &quot;p&quot;-dimensional latent trait variable ? = (?[subscript 1], . . . ?[subscript &quot;p&quot;]). In the second component, the conditional distribution of this latent trait variable ? is modeled by a multivariate, multiple linear regression on a set of predictor variables, which are usually based on student, school and teacher variables in assessments such as NAEP. To fit the latent regression model using the maximum (marginal) likelihood estimation technique, multivariate integrals have to be evaluated. In the computer program MGROUP used by ETS for fitting the latent regression model to data from NAEP and other programs, the integration is currently done either by numerical quadrature for problems up to two dimensions or by an approximation of the integral. CGROUP, the current operational version of the MGROUP program used in NAEP and other assessments since 1993, is based on Laplace approximation, which may not provide fully satisfactory results, especially if the number of items per scale is small (see, e.g., Thomas, 1993a, or von Davier &amp; Sinharay, 2004). There is scope for improvement in the technique used. This paper extends the NAEP BGROUP program to higher dimensions. Two real data analyses, one with a medium-sized data set and another with a large data set, show that the extension promises to be useful for fitting the NAEP model.
An Alternative Data Collection Design for Equating with Very Small Samples. Research Report. ETS RR-08-11
Author: Puhan, Gautam
Date: 2008-03-00
Type: Journal Articles
Subjects: Data Collection,Equated Scores,Item Sampling,Sample Size,Test Format,Test Construction,Error of Measurement,Statistical Bias

A single group (SG) equating design with nearly equivalent test forms (SiGNET) design was developed by Grant (2006) to equate small volume tests. The basis of this design is that examinees take two largely overlapping test forms within a single administration. The scored items for the operational form are divided into mini-tests called testlets. An additional testlet is created but not scored for the first form. If the scored testlets are Testlets 1-6 and the unscored testlet is Testlet 7, then the first form is composed of Testlets 1-6, the second form is composed of Testlets 2-7, and Testlets 2-6 are common to both test forms. They are administered as a single &quot;administered form&quot;, and when a sufficient number of examinees have taken the administered form for an SG equating, the second form (Testlets 2-7) is equated to the first form (Testlets 1-6) using SG equating. As evident, there are at least two merits of the SiGNET design over the nonequivalent groups with anchor test (NEAT) design. First, it facilitates the use of an SG equating design, which has the least random equating error, and second, it allows for the accumulation of sufficient data to equate the second form. Since the examinees scores are based on only the first form (i.e., the operational form), the two forms can be administered until sufficient data are collected to equate the second form. This study compared equatings under the SiGNET and NEAT designs and found reduced bias and error for the SiGNET design in very small sample size situations (e.g., N = 10 or 15). Implications for practice using the SiGNET design are also discussed.
Comparison of Multidimensional Item Response Models: Multivariate Normal Ability Distributions versus Multivariate Polytomous Ability Distributions. Research Report. ETS RR-08-45
Author: Haberman, Shelby J.
Date: 2008-08-00
Type: Journal Articles
Subjects: Item Response Theory,Comparative Analysis,Ability,Models,Goodness of Fit,Multivariate Analysis,Mathematics

Multidimensional item response models can be based on multivariate normal ability distributions or on multivariate polytomous ability distributions. For the case of simple structure in which each item corresponds to a unique dimension of the ability vector, some applications of the two-parameter logistic model to empirical data are employed to illustrate how, at least for the example under study, comparable results can be achieved with either approach. Comparability involves quality of model fit as well as similarity in terms of parameter estimates and computational time required. In both cases, numerical work can be performed quite efficiently. In the case of the multivariate normal ability distribution, multivariate adaptive Gauss-Hermite quadrature can be employed to greatly reduce computational labor. In the case of a polytomous ability distribution, use of log-linear models permits efficient computations.
Comparing Different Approaches of Bias Correction for Ability Estimation in IRT Models. Research Report. ETS RR-08-13
Author: Lee, Yi-Hsuan
Date: 2008-03-00
Type: Journal Articles
Subjects: Item Response Theory,Comparative Analysis,Computation,Ability,Test Items,Test Length,Error of Measurement,Statistical Bias,Maximum Likelihood Statistics,Methods

The method of maximum-likelihood is typically applied to item response theory (IRT) models when the ability parameter is estimated while conditioning on the true item parameters. In practice, the item parameters are unknown and need to be estimated first from a calibration sample. Lewis (1985) and Zhang and Lu (2007) proposed the expected response functions (ERFs) and the corrected weighted-likelihood estimator (CWLE), respectively, to take into account the uncertainty regarding item parameters for purposes of ability estimation. In this paper, we investigate the performance of ERFs and of the CWLE in different situations, such as various test lengths and levels of measurement error in item parameter estimation. Our empirical results indicate that ERFs can cause the bias in ability estimation to fall within [-0.2, 0.2] for all conditions, whereas the CWLE can effectively reduce the bias in ability estimation provided that it has a good foundation to start from.
The Influence of Strategies for Selecting Loglinear Smoothing Models on Equating Functions. Research Report. ETS RR-08-25
Author: Moses, Tim
Date: 2008-05-00
Type: Journal Articles
Subjects: Equated Scores,Selection,Models,Statistical Analysis,Accuracy,Simulation,Sample Size

This study addressed 2 issues of using loglinear models for smoothing univariate test score distributions and for enhancing the stability of equipercentile equating functions. One issue was a comparative assessment of several statistical strategies that have been proposed for selecting 1 from several competing model parameterizations. Another issue was an evaluation of the influence of the selection strategies on equating function accuracy. These issues were considered in a simulation study, where the accuracies of 17 selection strategies for loglinear models and their effects on equating function accuracies were assessed across a range of sample sizes, test score distributions, and population equating functions. The results differentiate the selection strategies in terms of their accuracies in selecting correct model parameterizations and define the situations where their use has the most important implications for equating function accuracy.
Consistency of SAT® I: Reasoning Test Score Conversions. Research Report. ETS RR-08-67
Author: Haberman, Shelby J.
Date: 2008-12-00
Type: Journal Articles
Subjects: College Entrance Examinations,Thinking Skills,Logical Thinking,Scaling,Scores,Trend Analysis,Mathematics Achievement,Verbal Ability,Raw Scores,Comparative Analysis,High School Students,Equated Scores,Statistical Analysis

This study uses historical data to explore the consistency of SAT® I: Reasoning Test score conversions and to examine trends in scaled score means. During the period from April 1995 to December 2003, both Verbal (V) and Math (M) means display substantial seasonality, and a slight increasing trend for both is observed. SAT Math means increase more than SAT Verbal means. Several statistical indices indicate that, during the period under study, raw-to-scale conversions are very stable, although conversions for extreme raw score points are less stable than are other conversions.
Analysis of Data from an Admissions Test with Item Models. Research Report. ETS RR-05-06
Author: Sinharay, Sandip
Date: 2005-04-00
Type: Journal Articles
Subjects: Item Analysis,Test Items,Scoring,Psychometrics,Costs,Models,Diagnostic Tests,Bayesian Statistics,College Entrance Examinations,Graduate Study,Difficulty Level,High Stakes Tests,Comparative Analysis,Mathematical Formulas

&quot;Item models&quot; (LaDuca, Staples, Templeton, &amp; Holzman, 1986) are classes from which it is possible to generate/produce items that are equivalent/isomorphic to other items from the same model (e.g., Bejar, 1996; Bejar, 2002). They have the potential to produce large number of high-quality items at reduced cost. This paper introduces data from the first known application of items automatically generated from item models in a large-scale assessment and deals with several research questions associated with the data. We begin by reviewing calibration techniques for the analysis of data involving item models; one method assumes that the items are isomorphic, while the other treats items generated from the same item model as distinct, but related. A major question for these type of data is whether these items are isomorphic, that is, if they behave the same psychometrically. This paper describes a number of rough diagnostic measures and a rigorous statistical diagnostic to assess the extent of isomorphicity in the items generated from an item model. Finally, this paper discusses the issue of scoring, an area that needs more research, with data involving item models.
Equating of Mixed-Format Tests in Large-Scale Assessments. Research Report. ETS RR-08-26
Author: Kim, Sooyeon
Date: 2008-05-00
Type: Journal Articles
Subjects: Equated Scores,Test Format,Multiple Choice Tests,Responses,Test Items,Measurement

This study examined variations of the nonequivalent-groups equating design for mixed-format tests--tests containing both multiple-choice (MC) and constructed-response (CR) items--to determine which design was most effective in producing equivalent scores across the two tests to be equated. Four linking designs were examined: (a) an anchor with only MC items; (b) a mixed-format anchor containing both MC and CR items; (c) a mixed-format anchor incorporating CR item rescoring; and (d) a hybrid combining single-group and equivalent-groups designs, thereby avoiding the need for an anchor test. Designs using MC items alone or those using a mixed anchor without CR item rescoring resulted in much larger bias than the other two design approaches. The hybrid design yielded the smallest root mean squared error value.
Linking for the General Diagnostic Model. Research Report. ETS RR-08-08
Author: Xu, Xueli
Date: 2008-02-00
Type: Journal Articles
Subjects: National Competency Tests,Probability,Reading Achievement,Test Items,Reading Skills,Test Construction,Cognitive Ability,Models,Equations (Mathematics),Item Response Theory,Grade 4,Reading Tests,Evaluation Research,Comparative Analysis

Three strategies for linking two consecutive assessments are investigated and compared by analyzing reading data for the National Assessment of Educational Progress (NAEP) using the general diagnostic model. These strategies are compared in terms of marginal and joint expectations of skills, joint probabilities of skill patterns, and item parameter estimates. The results indicate that fixing item parameter values at their previously calibrated values is sufficient to establish a comparable scale for the subsequent year.
The Effectiveness of Enhancing Test Security by Using Multiple Item Pools. Research Report. ETS RR-05-19
Author: Zhang, Jinming
Date: 2005-09-00
Type: Journal Articles
Subjects: Comparative Analysis,Test Items,Item Banks,Computer Assisted Testing,Probability,Cheating

This paper compares the use of multiple pools versus a single pool with respect to test security against large-scale item sharing among some examinees in a computer-based test, under the assumption that a randomized item selection method is used. It characterizes the conditions under which employing multiple pools is better than using a single whole pool in terms of minimizing the expected number of compromised items encountered by an examinee. The results obtained in this paper provide guidelines for constructing multiple pools optimally and evaluating the risks associated with some test designs that employ item reuse schemes. Finally, an adjusted-pool approach is proposed to achieve better test security than either the single-pool approach or the multiple-pool approach.
Design Patterns for Improving Accessibility for Test Takers with Disabilities. Research Report. ETS RR-08-49
Author: Hansen, Eric G.
Date: 2008-09-00
Type: Journal Articles
Subjects: Disabilities,Adaptive Testing,Testing Accommodations,Test Construction,Accessibility (for Disabled),Knowledge Level

There is a great need to help test designers determine how to make tests that are accessible to individuals with disabilities. This report takes &quot;design patterns&quot;, which were developed at SRI for assessment design, and uses them to clarify issues related to accessibility features for individuals with disabilities--such as low-vision and blindness--taking a test of reading. Design patterns appear useful in clarifying how variable features of a test design need to be matched to disability-related characteristics of test takers in order to ensure accessibility. Giving consideration to accessibility issues during the development and use of design patterns may help improve the validity and fairness of tests, as well as their accessibility for individuals with disabilities.
The Redesigned TOEIC® (Listening and Reading) Test: Relations to Test-Taker Perceptions of Proficiency in English. Research Report. ETS RR-08-56
Author: Powers, Donald E.
Date: 2008-11-00
Type: Journal Articles
Subjects: Test Construction,English (Second Language),Test Validity,Language Tests,Second Language Learning,Scores,Listening Comprehension Tests,Reading Tests,Foreign Countries,Correlation,Language Skills,Language Proficiency,Self Evaluation (Individuals),Asians

To facilitate the interpretation of test scores from the redesigned TOEIC® (listening and reading) test as a measure of English language proficiency, we administered a self-assessment inventory to TOEIC examinees in Japan and Korea that gathered perceptions of their ability to perform a variety of everyday English language tasks. TOEIC scores related relatively strongly to test-taker self-reports for both reading and listening tasks. The results were, with few exceptions, extraordinarily consistent, with examinees at each higher TOEIC score level being more likely to report that they could successfully accomplish each of the everyday language tasks in English. The pattern of correlations also showed modest discriminant validity of the listening and reading components of the redesigned TOEIC, suggesting that both sections contribute to the measurement of English language skills.
Understanding What the Numbers Mean: A Straightforward Approach to GRE Predictive Validity. ETS GRE Board Research Report No. 04-03. ETS RR-08-46
Author: Bridgeman, Brent
Date: 2008-09-00
Type: Journal Articles
Subjects: College Entrance Examinations,Graduate Study,Test Validity,Grades (Scholastic),Graduate Students,Scores,Predictive Validity,Intellectual Disciplines,Biology,Chemistry,English,Education,Experimental Psychology,Clinical Psychology,Undergraduate Study,Grade Point Average

Descriptions of validity results for the GRE® General Test based solely on correlation coefficients or percentage of the variance accounted for are not merely difficult to interpret, they are likely to be misinterpreted. Predictors that apparently account for a small percentage of the variance may actually be highly important from a practical perspective. This study used 2 existing data sets to demonstrate alternative methods of showing the value of the GRE as an indicator of 1st-year graduate grades. The combined data sets contained 4,451 students in 6 graduate fields: biology, chemistry, education, English, experimental psychology, and clinical psychology. In one set of analyses, students within a department were divided into quartiles based on GRE scores and the percentage of students in the top and bottom quartiles earning a 4.0 average was noted. Students in the top quartile were 3 to 5 times as likely to earn 4.0 averages compared to students in the bottom quartile. Even after controlling for undergraduate grade point average quartiles, substantial differences related to GRE quartile remained.
Investigating the Criterion-Related Validity of the TOEFL® Speaking Scores for ITA Screening and Setting Standards for ITAS. TOEFL iBT Research Report. TOEFL iBT-03. ETS RR-08-02
Author: Xi, Xiaoming
Date: 2008-01-00
Type: Journal Articles
Subjects: Test Validity,Criterion Referenced Tests,English (Second Language),Language Tests,Second Language Learning,Scores,Standard Setting,Oral Language,Correlation,Test Use,Computer Assisted Testing,Teaching Assistants,Foreign Students,College Admission,Classification,Observation,Regression (Statistics),Predictor Variables,Cutting Scores,Higher Education,Screening Tests

Although the primary use of the speaking section of the Test of English as a Foreign Language™ Internet-based test (TOEFL® iBT Speaking test) is to inform admissions decisions at English medium universities, it may also be useful as an initial screening measure for international teaching assistants (ITAs). This study provides criterion-related validity evidence for the use of TOEFL iBT Speaking for ITA screening and evaluates the effectiveness of using the scores for teaching assistantship (TA) assignment classification. Four universities participated in this study. Local ITA-screening tests or instructor recommendations were used as the criterion measures. Relationships between the TOEFL Speaking test and the local ITA tests were explored through observed and disattenuated correlations. These relationships were moderately strong, supporting the use of the TOEFL Speaking test for ITA screening. However, the strengths of the relationship between the TOEFL Speaking test and the local ITA tests were found to be somewhat different across universities depending on the extent to which the local test engaged and evaluated nonlanguage abilities. Implications of these findings are discussed. Binary and ordinal logistic regressions were used to investigate how effective TOEFL Speaking scores were in separating students into distinct TA assignment categories. At all four universities, TOEFL Speaking scores were significant predictors of students' TA assignments and were fairly accurate in classifying students for TA assignments. ROC curves were used to determine TOEFL Speaking cut scores for TA assignments at each university that would minimize false positives (i.e., true nonpasses classified as passes). The results have considerable potential value in providing guidance on using the TOEFL iBT Speaking scores for ITA screening.
The Fusion Model for Skills Diagnosis: Blending Theory with Practicality. Research Report. ETS RR-08-71
Author: Hartz, Sarah
Date: 2008-10-00
Type: Journal Articles
Subjects: Skill Development,Educational Diagnosis,Theory Practice Relationship,Standardized Tests,Learning Processes,Models,Mastery Learning,Item Response Theory,Computation,Evaluation Methods,Accuracy,Goodness of Fit,Simulation,Robustness (Statistics),Formative Evaluation,Markov Processes,Monte Carlo Methods,Q Methodology,Matrices,Bayesian Statistics

This paper presents the development of the fusion model skills diagnosis system (fusion model system), which can help integrate standardized testing into the learning process with both skills-level examinee parameters for modeling examinee skill mastery and skills-level item parameters, giving information about the diagnostic power of the test. The development of the fusion model system involves advancements in modeling, parameter estimation, model-fitting methods, and model-fit evaluation procedures, which are described in detail in the paper. To document the accuracy of the estimation procedure and the effectiveness of the model-fitting and model-fit evaluation procedures, this paper also presents a series of simulation studies. Special attention is given to evaluating the robustness of the fusion model system to violations of various modeling assumptions. The results demonstrate that the fusion model system is a promising tool for skills diagnosis that merits further research and development.
Subscores and Validity. Research Report. ETS RR-08-64
Author: Haberman, Shelby J.
Date: 2008-12-00
Type: Journal Articles
Subjects: Scores,Validity,Educational Testing,Correlation,Criterion Referenced Tests,Prediction,Reliability,Measurement,Test Theory,Equations (Mathematics),True Scores,Regression (Statistics),Scoring Formulas,Aptitude Tests,Verbal Tests,Mathematics Tests

In educational testing, subscores may be provided based on a portion of the items from a larger test. One consideration in evaluation of such subscores is their ability to predict a criterion score. Two limitations on prediction exist. The first, which is well known, is that the coefficient of determination for linear prediction of the criterion score by the subscore cannot exceed the reliability coefficient of the subscore. The second limitation is on incremental validity. The coefficient of determination for linear prediction of the criterion score by both the total score and the subscore is at least as great as the coefficient of determination for linear prediction of the criterion score by only the total score. Incremental validity may be measured by the difference between these two coefficients of determination. This difference is no greater than the reliability of the residual from linear prediction of the subscore by the total score.