bluuebunny/crossref_metadata_2025_split huggingface dataset:
EVALUATION OF AUTOMATED DIAGNOSIS AND INSTRUCTION FOR MATHEMATICS PROBLEM‐SOLVING
Authors: Underwood Jody S. 
Published: August 2014

ABSTRACTThe past several decades have seen numerous approaches toward automated diagnosis and instructional support of students engaged in mathematics problem‐solving. These approaches typically involve detailed analysis of potential solution paths for problems, formal representations of correct and incorrect answers, and support in the form of feedback or explanations to students during the process of solving a problem. The approaches of each of a number of representative systems (ACED, ALEKS, Cognitive Tutors, Andes, and Assistments) will be described through a critical evaluation of how they represent content knowledge, their approaches to diagnosis, and their approaches to instructional support. Finally, recommendations are made for new approaches to automated diagnosis and instructional support of mathematics problem‐solving.
ON MULTIDIMENSIONAL ITEM RESPONSE THEORY: A COORDINATE‐FREE APPROACH
Authors: Antal Tamás 
Published: August 2014

ABSTRACTA coordinate‐free definition of complex‐structure multidimensional item response theory (MIRT) for dichotomously scored items is presented. The point of view taken emphasizes the possibilities and subtleties of understanding MIRT as a multidimensional extension of the classical unidimensional item response theory models. The main theorem of the paper is that every monotonic MIRT model looks the same; they are all trivial extensions of univariate item response theory.
SCALE DRIFT IN EQUATING ON A TEST THAT EMPLOYS CUT SCORES
Authors: Puhan Gautam 
Published: August 2014

ABSTRACTThe purpose of this study is to determine the extent of scale drift on a test that employs cut scores. It is essential to examine scale drift in a testing program using new forms that are often put on scale through a series of intermediate equatings (known as equating chains). This may cause equating error to accumulate to a point where scale scores are rendered incomparable across two parallel chains or time periods. The study examined whether scale drift occurred for two conditions (i.e., parallel equating chains and a single long chain). Data from three tests that employed cut scores were used in this study. Results indicated that although there was some difference observed between the actual equating conversions derived via different equating chains, the effect of these differences on actual pass/fail status of test takers was not large. However, these results may change if new cut scores are introduced. Also, for tests that do not employ cut scores, stability in most parts of the score scale is important, not just at particular cut score points.
THE STANDARDIZED LETTER OF RECOMMENDATION: IMPLICATIONS FOR SELECTION
Authors: Liu Ou Lydia , Minsky Jennifer , Ling Guangming , Kyllonen Patrick 
Published: August 2014

ABSTRACTIn an effort to standardize academic application procedures, the Standardized Letter of Recommendation (SLR) was developed to capture important cognitive and noncognitive qualities of graduate school candidates. The SLR consists of seven scales (knowledge, analytical skills, communication skills, motivation, self‐ organization, professionalism and maturity, and teamwork) and was applied to an intern‐selection scenario. Both professor ratings (N = 414) during the application process and mentor ratings of the selected students (N = 51) after the internship was completed were collected using the SLR. A multidimensional Rasch investigation suggests that the seven scales of the SLR displayed satisfactory psychometric properties in terms of reliability, model fit, item fit statistics, and discrimination. The two cognitive scales, knowledge and analytical skills, were found to be the best predictors for intern selection. The professor ratings and mentor ratings had moderate to high correlations, with the professor ratings being systematically higher than the mentor ratings. Possible reasons for the rating discrepancies are discussed. Also, implications for how the SLR can be used and improved in other selection situations are suggested.
ON‐THE‐FLY CUSTOMIZATION OF AUTOMATED ESSAY SCORING
Authors: Attali Yigal 
Published: August 2014

ABSTRACTBecause there is no commonly accepted view of what makes for good writing, automated essay scoring (AES) ideally should be able to accommodate different theoretical positions, certainly at the level of state standards but also perhaps among teachers at the classroom level. This paper presents a practical approach and an interactive computer program for judgment‐based customization.This approach is based on the AES system, e‐rater®. Through this new approach, a user can gain easy accessibility to system components, flexibility in adjusting scoring parameters, and procedures for making scoring adjustments that can be based on only a few benchmark essays. The interactive prototype program that implements this approach allows the user to customize e‐rater and watch the effect on benchmark essay scores as well as on score distributions for a reference testing program of the user's choice. The paper presents results for the use of this approach in customizing e‐rater to the standards of different assessments.
PREDICTING GRADES IN DIFFERENT TYPES OF COLLEGE COURSES
Authors: Bridgeman Brent , Pollack Judith , Burton Nancy 
Published: August 2014

ABSTRACTThe ability of high school grades (high school GPA) and SAT® scores to predict cumulative grades in different types of college courses was evaluated in a sample of 26 colleges. Each college contributed data from three cohorts of entering freshmen, and each cohort was followed for at least four years. Colleges were separated into four levels by average SAT scores. Grade point averages for four categories of courses (English; science, math, and engineering [S/M/E]; social science; and education) were computed, and analyses were run separately for gender within race/ethnicity classifications. Correlations of the combined predictors with course grades over four or more years, corrected for range restriction, ranged from .45 for education courses to 0.64 for S/M/E courses. The SAT increment, that is, the increase in the multiple correlations when SAT scores are added to high school grades, ranged from 0.03 in education courses to 0.08 in S/M/E courses. Because these seemingly small numbers are frequently misinterpreted, an additional analysis showed how the percentage of students succeeding at a high level (cumulative GPA of 3.5 or higher) increases as SAT scores increase for students with similar high school grades. For example, for students with a high school GPA of 3.7 or higher in colleges where the mean combined SAT score is below 1200, only 2 percent of the students at the lowest SAT level (800 or lower combined score) were highly successful in social science courses. At the highest SAT level (1410‐1600), 77 percent were highly successful.
FACTOR STRUCTURE OF THE TOEFL® INTERNET‐BASED TEST (IBT): EXPLORATION IN A FIELD TRIAL SAMPLE
Authors: Sawaki Yasuyo , Stricker Lawrence , Oranje Andreas 
Published: August 2014

ABSTRACTThe present study investigated the factor structure of a field trial sample of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT). An item‐level confirmatory factor analysis (CFA) was conducted for a polychoric correlation matrix of items on a test form completed by 2,720 participants in the 2003–2004 TOEFL iBT Field Study. CFA‐based multitrait‐multimethod (MTMM) analyses for the Reading and Listening sections showed that the language abilities assessed in each section were essentially unidimensional, while the factor structure of the entire test was best represented by a higher‐order factor model with a general factor (English as a second language/English as a foreign language ability) and four group factors for reading, listening, speaking, and writing. The integrated Speaking and Writing tasks, which require language processing in multiple modalities, well defined the target modalities (speaking and writing). These results broadly support the current reporting of four scores corresponding to the modalities and a total score, as well as the test design where the integrated tasks contribute only to the scores for the target modalities.
COMPARING DIFFERENT APPROACHES OF BIAS CORRECTION FOR ABILITY ESTIMATION IN IRT MODELS
Authors: Lee Yi‐Hsuan , Zhang Jinming 
Published: August 2014

ABSTRACTThe method of maximum‐likelihood is typically applied to item response theory (IRT) models when the ability parameter is estimated while conditioning on the true item parameters. In practice, the item parameters are unknown and need to be estimated first from a calibration sample. Lewis (1985) and Zhang and Lu (2007) proposed the expected response functions (ERFs) and the corrected weighted‐likelihood estimator (CWLE), respectively, to take into account the uncertainty regarding item parameters for purposes of ability estimation. In this paper, we investigate the performance of ERFs and of the CWLE in different situations, such as various test lengths and levels of measurement error in item parameter estimation. Our empirical results indicate that ERFs can cause the bias in ability estimation to fall within [‐0.2, 0.2] for all conditions, whereas the CWLE can effectively reduce the bias in ability estimation provided that it has a good foundation to start from.
THEORETICAL AND EMPIRICAL STANDARD ERRORS FOR TWO POPULATION INVARIANCE MEASURES IN THE LINEAR EQUATING CASE
Authors: von Davier Alina A. , Manalo Jonathan R. , Rijmen Frank 
Published: August 2014

ABSTRACTThe standard errors of the 2 most widely used population‐invariance measures of equating functions, root mean square difference (RMSD) and root expected mean square difference (REMSD), are not derived for common equating methods such as linear equating. Consequently, it is unknown how much noise is contained in these estimates. This paper describes 2 methods for obtaining the standard errors for RMSD and REMSD. The delta method relies on an analytical approximation and provides asymptotic standard errors. The grouped jackknife method is a sampling‐based method. Both methods were applied to a real data application. The results showed that there was very little difference between the standard errors found by the 2 methods.
EQUATING OF MIXED‐FORMAT TESTS IN LARGE‐SCALE ASSESSMENTS
Authors: Kim Sooyeon , Walker Michael E. , McHale Frederick 
Published: August 2014

ABSTRACTThis study examined variations of the nonequivalent‐groups equating design for mixed‐format tests—tests containing both multiple‐choice (MC) and constructed‐response (CR) items—to determine which design was most effective in producing equivalent scores across the two tests to be equated. Four linking designs were examined: (a) an anchor with only MC items; (b) a mixed‐format anchor containing both MC and CR items; (c) a mixed‐format anchor incorporating CR item rescoring; and (d) a hybrid combining single‐group and equivalent‐groups designs, thereby avoiding the need for an anchor test. Designs using MC items alone or those using a mixed anchor without CR item rescoring resulted in much larger bias than the other two design approaches. The hybrid design yielded the smallest root mean squared error value.
FITTING THE STRUCTURED GENERAL DIAGNOSTIC MODEL TO NAEP DATA
Authors: Xu Xueli , von Davier Matthias 
Published: August 2014

ABSTRACTXu and von Davier (2006) demonstrated the feasibility of using the general diagnostic model (GDM) to analyze National Assessment of Educational Progress (NAEP) proficiency data. Their work showed that the GDM analysis not only led to conclusions for gender and race groups similar to those published in the NAEP Report Card, but also allowed flexibility in estimating multidimensional skills simultaneously. However, Xu and von Davier noticed that estimating the latent skill distributions will be much more challenging with this model when there is a large number of subgroups to estimate. To make the GDM more applicable to NAEP data analysis, which requires a fairly large subgroups analysis, this study developed a log‐linear model to reduce the number of parameters in the latent skill distribution without sacrificing the accuracy of inferences. This paper describes such a model and applies the model in the analysis of NAEP reading assessments for 2003 and 2005. The comparisons between using this model and the unstructured model were made through the use of various results, such as the differences between item parameter estimates and the differences between estimated latent class distributions. The results in general show that using the log‐linear model is efficient.
STEREOTYPE THREAT SPILLOVER AND SAT® SCORES
Authors: Walker Michael E. , Bridgeman Brent 
Published: August 2014

ABSTRACTA recent study by Beilock, Rydell, and McConnell (2007) suggested that stereotype threat experienced in one domain (e.g., math) triggered by knowledge of a negative stereotype about a social group in that particular domain can spill over into subsequent tasks in totally unrelated domains (e.g., reading). The authors suggested that these findings might have implications for how the ordering of sections on standardized tests such as the SAT® or GRE could affect examinee performance. To test the authors' assertions, this study used data from a recent SAT administration in which either a reading, a math, or a writing section preceded a reading section. Performance on the subsequent reading section of members of a stereotype threatened group (i.e., women) who took the math section first was compared to performance of those who took the reading or writing section first. Results were inconsistent with the stereotype threat spillover hypothesis and serve to justify the warning of Cullen, Hardison, and Sackett (2004) to exercise caution in generalizing lab findings on stereotype threat to operational testing situations.
EVIDENCE‐CENTERED ASSESSMENT DESIGN FOR REASONING ABOUT ACCOMMODATIONS FOR INDIVIDUALS WITH DISABILITIES IN NAEP READING AND MATHEMATICS
Authors: Hansen Eric G. , Mislevy Robert J. , Steinberg Linda S. 
Published: August 2014

ABSTRACTAccommodations play a key role in enabling individuals with disabilities to participate in the National Assessment of Educational Progress (NAEP) and other large‐scale assessments. However, it can be difficult to know how accommodations affect the validity of results, thus making it difficult to determine which accommodations should be allowed. This study describes recent extension of evidence‐centered assessment design (ECD) for reasoning about the impact of accommodations and other accessibility features (e.g., universal design features) on the validity of assessment results, using examples from NAEP reading and mathematics. The study found that the ECD‐based techniques were useful in analyzing the effects of accommodations and other accessibility features on validity. Such design capabilities may increase assessment designers' capacity to employ accessibility features without undermining validity.
SMALL‐SAMPLE EQUATING BY THE CIRCLE‐ARC METHOD
Authors: Livingston Samuel A. , Kim Sooyeon 
Published: August 2014

ABSTRACTThis paper suggests two new, related methods for estimating a test‐score equating relationship from small samples of test takers. These methods do not require the estimated equating transformation to be linear. Instead, they constrain the estimated equating curve to pass through 2 prespecified end‐points and a middle point determined from the data. Some preliminary results indicate that these methods outperform mean equating and other methods used for equating in small samples.
L‐BIVARIATE AND L‐MULTIVARIATE ASSOCIATION COEFFICIENTS
Authors: Kong Nan , Lewis Charles 
Published: August 2014

ABSTRACTGiven a system of multiple random variables, a new measure called theL‐multivariate association coefficient is defined using (conditional) entropy. Unlike traditional correlation measures, theL‐multivariate association coefficient measures the multiassociations or multirelations among the multiple variables in the given system; that is, theL‐multivariate association coefficient measures the degree of the association for the given system. TheL‐multivariate association coefficient for the system of two random variables is also called theL‐bivariate association coefficient. The association measured by theL‐multivariate association coefficient is a general type of association, not any specific type of a linear or nonlinear association. Unlike theK‐dependence coefficient, which is an asymmetrical measure, theL‐multivariate association coefficient is a symmetrical measure. A direct application of theL‐multivariate association coefficient is in variables selection or variables reduction. This paper also explores the relationship between theL‐multivariate association coefficient and theK‐dependence coefficient.
A STUDY OF CONFIDENCE AND ACCURACY USING THE RASCH MODELING PROCEDURES
Authors: Paek Insu , Lee Jihyun , Stankov Lazar , Wilson Mark 
Published: August 2014

ABSTRACTThis study investigated the relationship between students' actual performance (accuracy) and their subjective judgments of accuracy (confidence) on selected English language proficiency tests. The unidimensional and multidimensional IRT Rasch approaches were used to model the discrepancy between confidence and accuracy at the item and test level and to assess disattenuated strength of association between accuracy and confidence. The analysis results indicate a pattern of overconfidence bias (i.e., overestimation of success rate), which was related to item difficulty. In addition, the strength of association between accuracy and confidence dimension was relatively high: The confidence dimension explained 45% and 52% of the variability in the accuracy dimension for the two tests employed in this study.
A REVIEW OF RECENT DEVELOPMENTS IN DIFFERENTIAL ITEM FUNCTIONING
Authors: Mapuranga Raymond , Dorans Neil J. , Middleton Kyndra 
Published: August 2014

ABSTRACTIn many practical settings, essentially the same differential item functioning (DIF) procedures have been in use since the late 1980s. Since then, examinee populations have become more heterogeneous, and tests have included more polytomously scored items. This paper summarizes and classifies new DIF methods and procedures that have appeared since the early 1990s and assesses their appropriateness for practical use. Widely used DIF methods are evaluated alongside these new methods for completeness, clarity, and comparability.
DEVELOPMENT AND VALIDITY EVIDENCE SUPPORTING A TEAMWORK AND COLLABORATION ASSESSMENT FOR HIGH SCHOOL STUDENTS
Authors: Zhuang Xiaohua , MacCann Carolyn , Wang Lijuan , Liu Lydia , Roberts Richard D. 
Published: August 2014

ABSTRACTVarious policy papers and research studies assert that teamwork is one of the most important skills for students to learn if they are to become meaningful contributors to the 21st century workforce. However, outside of organizational psychology and adult populations, few reliable assessments of this construct exist, with suitable validity evidence scant or nonexistent. To redress this imbalance, teamwork assessments for high school students were developed using multiple methods: self‐report ratings, situational judgment testing, and teacher reports. Exploratory factor, confirmatory factor, and latent class analyses were used to determine the structure of the scales. Measures showed reasonable reliability and promising validity evidence, relating to each other and to academic achievement, while remaining relatively independent from personality. The advantages and disadvantages of each methodology and the potential applications for identification and intervention, selection, and evaluation of training programs are discussed. This report also serves as an archival document for the teamwork and collaboration assessments that have been developed at ETS for high school students.
THE REDESIGNED TOEIC® (LISTENING AND READING) TEST: RELATIONS TO TEST‐TAKER PERCEPTIONS OF PROFICIENCY IN ENGLISH
Authors: Powers Donald E. , Kim Hae‐Jin , Weng Vincent Z. 
Published: August 2014

ABSTRACTTo facilitate the interpretation of test scores from the redesigned TOEIC® (listening and reading) test as a measure of English language proficiency, we administered a self‐assessment inventory to TOEIC examinees in Japan and Korea that gathered perceptions of their ability to perform a variety of everyday English language tasks. TOEIC scores related relatively strongly to test‐taker self‐reports for both reading and listening tasks. The results were, with few exceptions, extraordinarily consistent, with examinees at each higher TOEIC score level being more likely to report that they could successfully accomplish each of the everyday language tasks in English. The pattern of correlations also showed modest discriminant validity of the listening and reading components of the redesigned TOEIC, suggesting that both sections contribute to the measurement of English language skills.
NOTES ON A GENERAL FRAMEWORK FOR OBSERVED SCORE EQUATING
Authors: Moses Tim , Holland Paul 
Published: August 2014

ABSTRACTThe purpose of this paper is to extend von Davier, Holland, and Thayer's (2004b) framework of kernel equating so that it can incorporate raw data and traditional equipercentile equating methods. One result of this more general framework is that previous equating methodology research can be viewed more comprehensively. Another result is that the standard error of equated score difference (SEED) has a wider application than originally proposed. The methods described in this paper are empirically evaluated in an accompanying simulation study (Moses & Holland, 2007).
FACTOR STRUCTURE OF THE TOEFL INTERNET‐BASED TEST ACROSS SUBGROUPS
Authors: Stricker Lawrence J. , Rock Donald A. 
Published: August 2014

ABSTRACTThis study assessed the invariance in the factor structure of the Test of English as a Foreign Language™ Internet‐based test (TOEFL® iBT) across subgroups of test takers who differed in native language and exposure to the English language. The subgroups were defined by (a) Indo‐European and Non‐Indo‐European language family, (b) Kachru's classification of outer and expanding circles of countries (based on prevalence of English use in educational and business contexts), and (c) years of classroom instruction in the English language. The same factor structure (four first‐order factors corresponding to the test sections and a single higher‐order factor encompassing these factors) was identified in each subgroup. The results support the present scoring scheme for the TOEFL iBT assessment and suggest that the test functions the same way for diverse subgroups of test takers.
CONSISTENCY OF SAT® I: REASONING TEST SCORE CONVERSIONS
Authors: Haberman Shelby J. , Guo Hongwen , Liu Jinghua , Dorans Neil J. 
Published: August 2014

ABSTRACTThis study uses historical data to explore the consistency of SAT® I: Reasoning Test score conversions and to examine trends in scaled score means. During the period from April 1995 to December 2003, both Verbal (V) and Math (M) means display substantial seasonality, and a slight increasing trend for both is observed. SAT Math means increase more than SAT Verbal means. Several statistical indices indicate that, during the period under study, raw‐to‐scale conversions are very stable, although conversions for extreme raw score points are less stable than are other conversions.