bluuebunny/crossref_metadata_2025_split huggingface dataset:
EVIDENCE RELATED TO THE VALIDITY OF THE ENGLISH LANGUAGE PROFICIENCY TEST
Authors: Bridgeman Brent , Harvey Anne 
Published: August 2014

ABSTRACTThe validity of the English Language Proficiency Test was evaluated by considering six aspects of construct validity identified by Messick: content, substantive, structural, generalizability, external, and consequential. Analyses evaluated both the internal structure of the test and the relationship of the scores to external criteria. In one set of analyses, high school and college language teachers were asked to evaluate their students using the same five proficiency categories that are used in the test, and the teacher ratings were then compared to the test scores. Correlations were averaged over 15 classes in the college sample and over 32 classes in the high school sample. In the college sample, the averaged correlations, corrected for range restriction, were .50 and .57 for reading and listening respectively. In the high school sample the corresponding correlations were .68 and .71. For both the reading and listening proficiency scales, teachers tended to rate their students about one category higher than they were rated by the test.
SEX AND ETHNIC GROUP DIFFERENCES ON ACCOMPLISHMENTS MEASURES
Authors: Stricker Lawrence J. , Rock A. , Bennett Randy Elliot 
Published: August 2014

ABSTRACTThis study investigated sex and ethnic group differences on measures of accomplishments–notable attainments that have been publicly recognized–for GRE test takers. Six accomplishments scales (Academic, Leadership, Practical Language, Aesthetic Expression, Science, and Mechanical) were studied. With the exception of the Mechanical scale, men and women did not differ in their level of performance on the scales, and ethnic groups did not differ on any scale. Furthermore, the factor structure of the scales did not differ in most respects for men and women. Concerns about sex and ethnic group differences on accomplishments measures are probably unwarranted. (Key words: accomplishments, sex differences, ethnic group differences, factor analysis)
TEST ANXIETY AND TEST PERFORMANCE: COMPARING PAPER‐BASED AND COMPUTER‐ADAPTIVE VERSIONS OF THE GRE GENERAL TEST
Authors: Powers Donald E. 
Published: August 2014

ABSTRACTDespite some assumptions to the contrary, there is reason to believe that the introduction of computer‐adaptive testing may actually help to alleviate test anxiety and diminish the relationship between test anxiety and test performance. This study provided a test of this hypothesis. Results are based on a sample of GRE General Test takers who took the computer‐adaptive version of the test, and another sample of GRE examinees who took the paper‐based version of the test. After taking the test, all examinees completed both a test anxiety inventory and an inventory concerning attitudes toward computers. Relationships were examined between performance on each of the three GRE General Test measures and reports of test anxiety (both worry and emotionality) and computer attitudes (both anxiety and confidence). For both the test anxiety and the computer attitudes scales, the relationship to GRE scores was similar for the computer‐adaptive and paper‐based GRE General Test. Thus, there was no support for the study's major hypothesis. Several ancillary findings, however, do have implications for large‐scale testing programs, especially those moving to computer‐based testing.
TRENDS AND PROFILES: STATISTICS ABOUT GRE® GENERAL TEST EXAMINEES BY GENDER, AGE, AND ETHNICITY SECOND EDITION
Authors: Grandy Jerilee 
Published: August 2014

ABSTRACTThis second edition of Trends and Profiles presents statistics from the files of the Graduate Record Examinations (GRE) from 1982 through 1996. In question‐and‐answer format with many tables and graphs, the report addresses frequently asked questions about U.S. citizens who take the GRE. It covers demographics, undergraduate studies, GRE test preparation, test scores, and graduate school plans. Statistical profiles for 1996 and trends from 1982 to 1996 are presented by gender, ethnic group, and age group.
ENGLISH PROFICIENCY AND DIFFERENCES AMONG RACIAL AND ETHNIC GROUPS IN MEAN SAT® AND GRE® SCORES: A LONGITUDINAL ANALYSIS
Authors: Pennock‐Román Maria 
Published: August 2014

ABSTRACTAlthough limited proficiency in the language of a test is known to depress aptitude test scores, the changes that occur as proficiency rises over time have been less well studied. The objective here is to contrast longitudinal changes in test performance for persons who indicated that English was (EBL) or was not their best language (ENBL). Analyses were based on a sample of U.S. citizens and permanent residents (N=65,987 EBL and N=1,592 ENBL); each individual had taken both the Scholastic Assessment Test (SAT)1 and the Graduate Record Examinations (GRE) General Test some years later. The distance in verbal mean scores between the EBL non‐Hispanic White group and ENBL groups (such as, non‐Hispanic White, Asian‐American, Black, Puerto Rican, other Hispanic) grew closer by 0.21 to 0.48 standard deviation units from the taking of the SAT to the taking of the GRE. These findings have implications for the interpretation of test scores in longitudinal studies of linguistic minorities and for selective admissions of students with limited proficiency in English.
VALIDATING A TEST DESIGNED TO ASSESS ESL PROFICIENCY AT LOWER DEVELOPMENTAL LEVELS
Authors: Wilson Kenneth M. 
Published: August 2014

ABSTRACTThis exploratory study examined relationships between a test designed to assess English‐language listening comprehension and reading skills in samples of nonnative‐English speakers at lower levels of developed proficiency in English as a second or foreign language (ESL or EFL). The test—called ESL‐EZY—was developed by using items similar to but easier, on the average, than those being used in an existing ESL proficiency test designed for intermediate‐ or higher‐level ESL users/learners. This paper reports evidence regarding the relationship between ESL‐EZY scores and teachers' ratings of oral English proficiency in samples—assessed in diverse settings in the United States and Japan—selected to include subgroups that tend to differ relatively widely in average level of developed English proficiency. Scores on ESL‐EZY were found to correlate relatively strongly with teachers' ratings within the respective subgroups; correlations were especially strong in relatively less proficient subgroups, suggesting that for younger ESL‐students and other ESL‐learners with limited developed functional proficiency in English as a foreign or second language, a test embodying properties similar to those represented by ESL‐EZY might provide useful supplementary assessment information.
COMPARING THE INCOMPARABLE: AN ESSAY ON THE IMPORTANCE OF BIG ASSUMPTIONS AND SCANT EVIDENCE
Authors: Wainer Howard 
Published: August 2014

ABSTRACTComparisons between groups usually involve making statistical adjustments so that the comparisons are, statistically speaking, ceteris paribus. Usually included within those adjustments are some untestable assumptions. These untestable assumptions are often determined by the goals of the testing program; they are sometimes justified on the basis of theory, sometimes on the basis of political expediency, and sometimes on just a hope and a prayer. If we are vigilant, bits of evidence may emerge from the data that can corroborate or refute our assumptions. In this essay I describe several examples, drawn from mental testing, that illustrate these issues, and suggest a plausible path of investigation that may provide a better estimate of the practical limitations of accuracy for any proposed solution to this vexing equity issue.
USER'S GUIDE FOR SCORIGHT (VERSION 1.2): A COMPUTER PROGRAM FOR SCORING TESTS BUILT OF TESTLETS
Authors: Wang Xiaohui , Bradlow Eric T. , Wainer Howard 
Published: August 2014

ABSTRACTSCORIGHT is a very general computer program for scoring tests. It models tests that are made up of dichotomously or polytomously rated items, or any kind of combination of the two through the use of a generalized IRT formulation. The items can be presented independently, or grouped into clumps of allied items (testlets), or in any combination of the two. When there are testlets the program assesses the degree of local dependence and adjusts the estimates accordingly. The estimation is accomplished within a fully Bayesian framework using Markov chain Monte Carlo procedures. This methodology yields the complete posterior distributions of the parameters, which allows the easy calculation of many important characteristics of the scores that are not available with other methods.
USING A NEW STATISTICAL MODEL FOR TESTLETS TO SCORE TOEFL
Authors: Wainer Howard , Wang Xiaohui 
Published: August 2014

ABSTRACTStandard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimuli (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, and Du, 2000). This parameter, γ, characterizes the amount of local dependence in a testlet.We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters [discrimination (a), difficulty (b), and guessing (c)] with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over‐estimated when conditional independence was incorrectly assumed.
GRE® AUTOMATED‐EDITING‐TASK PHASE II REPORT: ITEM ANALYSES, REVISIONS, VALIDITY STUDY, AND TAXONOMY DEVELOPMENT
Authors: Breland Hunter , Kukich Karen , Hemat Lisa 
Published: August 2014

ABSTRACTTwo automated editing tasks developed in a Phase I study were subjected to item analyses, revised, and then used in a computer‐based test administration at a local college. The data collected in the administration were compared with questionnaire data obtained from students to examine the construct validity of the tasks. In a second approach to construct validation, a taxonomy of writing skills was developed and compared to the skills assessed by the editing tasks. Data analyses indicate that total editing score correlates more strongly with self‐reported English grades than with self‐reported mathematics grades, and that total editing score correlates positively with student self‐assessments of their writing skill, recent grades on writing assignments, and college grade point average. A review of the task elements against the taxonomy indicates that the editing tasks assess important writing skills not assessed by free‐response essays. For this and other reasons, it was concluded that automated editing tasks would serve as a useful complement to free‐response writing assessments.
EFFECTS OF THE PRESENCE AND ABSENCE OF VISUALS ON PERFORMANCE ON TOEFL® CBT LISTENING‐COMPREHENSION STIMULI
Authors: Ginther April 
Published: August 2014

ABSTRACTThis study was conducted in order to begin to understand the effects of providing different types of visuals – such as pictures, diagrams, and still photos of the speakers – with items on the computer‐based Test of English as a Foreign Language (TOEFL®). A nested crossover design (subjects nested in proficiency, level, and form) was used to examine the effects of language proficiency (high or low), still photos (present or absent), and type of stimuli (dialogues/short conversations, academic discussions, mini‐talks with context visuals, mini‐talks with content visuals) on performance on TOEFL multiple‐choice items. Three two‐way interactions were significant: proficiency by type of stimuli, type of stimuli by visual condition, and type of stimuli by time. The weakest of these interactions, type of stimuli by visual condition, was the most interesting; it indicated that the presence of visuals results in facilitation of performance when the visuals bear information that complement the audio portion of the stimulus. The interaction effect between type of stimulus and time suggests that the trend toward the use of longer stimuli is not without consequences, and these consequences may extend to testing situations. Responses to a series of questionnaires indicate that the majority of the subjects preferred the stimuli that were accompanied by visuals.
THE INTERNATIONAL ADULT LITERACY SURVEY (IALS): UNDERSTANDING WHAT WAS MEASURED
Authors: Kirsch Irwin S. 
Published: August 2014

ABSTRACTThis paper offers a framework that has been used for both developing the tasks used to measure literacy and for understanding the meaning of what has been reported with respect to the comparative literacy proficiencies of adults in participating countries. The framework consists of six parts that represent a logical sequence of steps, from needing to define and represent a particular domain of interest, to identifying and operationalizing characteristics used to construct items, to providing an empirical basis for interpreting results. The various parts of the framework are seen as important in that they help to provide a deeper understanding of the construct of literacy and the various processes associated with it. A processing model is proposed and variables associated with performance on the literacy tasks are identified and verified through regression analyses. These variables are shown to account for between 79% and 89% of the variance in task difficulty. Collectively, these process variables provide a means for moving away from interpreting performance on large‐scale surveys in terms of discrete tasks or a single number, toward identifying levels of performance that have generalizability across pools of tasks and toward what Messick (1989) has called a higher level of measurement.
CAUSATION AND RACE
Authors: Holland Paul W. 
Published: August 2014

ABSTRACTRace is often viewed as a causal variable and “RACE effects” found from regression analyses are sometimes given causal interpretations. I argue that this is a mistaken way to proceed. RACE is not a causal variable in a very important sense of the word, and yet it does have a significant role in causal studies. The key role that RACE can play is to help our understanding of the effects of causes or interventions through the statistical “interaction” of RACE with causal variables, rather than as the main effect of RACE. These ideas are briefly illustrated using data from a study of tests constructed to manipulate the distribution of scores of Black and White test takers.
KNOWLEDGE AND SKILL INDICATORS IMPORTANT FOR BEGINNING TEACHERS: A SURVEY OF PENNSYLVANIA TEACHERS AND TEACHER EDUCATORS
Authors: Wylie E. Caroline , Tannenbaum Richard J. 
Published: August 2014

ABSTRACTThe study investigated the job relevance of 70 knowledge and/or skill indicators across five content areas (Literature, Fine Arts, Mathematics, Social Studies, and Science) for the development of a Pennsylvania initial teacher licensure assessment. A random sample of 1,700 teachers and 300 teacher educators was selected from Pennsylvania. Respondents judged each indicator on a scale ranging from 1 (of no importance) to 5 (very high level of importance). Means (and standard deviations) were computed across all survey respondents, as well as for subgroups of teacher respondents. Using a mean of 3.50 as the criterion for job relevance, 24 of the 70 (34%) of the knowledge and/or skill indicators were judged to be important (job‐relevant) enough for consideration in the development of the licensure assessment. These 24 knowledge and/or skill indicators are recommended as the primary content of the test.
ASSESSING CONVERGENCE OF THE MARKOV CHAIN MONTE CARLO ALGORITHMS: A REVIEW
Authors: Sinharay Sandip 
Published: August 2014

ABSTRACTMarkov chain Monte Carlo (MCMC) algorithms are in wide use for fitting complicated statistical models in psychometrics in situations where the traditional estimation techniques are very difficult to apply. One of the stumbling blocks in using an MCMC algorithm is determining the convergence of the algorithm. Because the convergence is not that of a scalar quantity to a point, but that of a distribution to another distribution, the issue remains an enigma to many users of MCMC, especially to those without a sound knowledge of mathematical statistics. This article is an attempt to provide psychometricians using the MCMC algorithms a better understanding of the concept of convergence of the algorithms and an improved knowledge about the diagnostics tools to assess convergence of the MCMC algorithms.
ANALYSIS OF NAEP COMBINED NATIONAL AND STATE SAMPLES
Authors: Qian Jiahe , Kaplan Bruce , Weng Vincent 
Published: August 2014

ABSTRACTThis National Assessment of Educational Progress (NAEP) study analyzes the efficacy of combined national and state NAEP samples. The main goal of combining NAEP national and state samples is to provide an integrated way to implement NAEP assessments and to reduce the burden of state assessments by administrating one assessment. The study is based on combined samples of the 1998 and 2000 NAEP assessments.
WRITING DIFFERENCES IN TEACHER PERFORMANCE ASSESSMENTS: USE OF AFRICAN AMERICAN LANGUAGE
Authors: Wylie E. Caroline , Szpara Michelle Y. 
Published: August 2014

ABSTRACTThis study investigated potential writing differences between African American and European American candidates in a National Board for Professional Teaching Standards (NBPTS) portfolio entry from the Middle Childhood/Generalist Certificate. The use of features other than Edited American English was examined as a possible source of construct‐irrelevant variance in assessors' scoring judgments. Thirty‐two written commentaries, 18 from African American candidates and 14 from European American candidates, were coded for grammatical, lexical, and discourse features. The coding frame identified features of African American Language (AAL) and Speech Code Errors (SCE). Instances of AAL were fewer than instances of SCE and clustered according to potential users of AAL.
POPULATION INVARIANCE OF SCORE LINKING: THEORY AND APPLICATIONS TO ADVANCED PLACEMENT PROGRAM® EXAMINATIONS
Authors: Dorans Neil J. , von Davier Alina A. , Dorans Neil J. , Holland Paul W. , Tateneni Krishna , Thayer Dorothy T. , Yang Wen‐Ling , Green Bert F. , Kolen Michael L. 
Published: August 2014

ABSTRACTTest equating methods are intended to produce interchangeable scores and should not be strongly influenced by the group of examinees on which they are computed. Advanced Placement Program® (AP®) exams, which include both multiple‐choice (MC) items and constructed‐response (CR) items, provide an environment for studying the effects of subpopulations on the estimation of equating functions. The exams are equated to past forms using an internal anchor‐test design in which the linking items are restricted to MC items due to the immediate disclosure of the CR sections of the tests (precluding their reuse as linking items). This collection of closely related papers uses the variety of cases provided by AP exams to assess the effects of subpopulations on various important aspects of test equating. Earlier versions of these papers were prepared for a symposium at the 2002 Annual Meetings of the National Council on Measurement in Education.
RECONSIDERING THE IMPACT OF HIGH‐STAKES TESTING
Authors: Braun Henry 
Published: August 2014

ABSTRACTOver the last 15 years, many states have implemented high‐stakes tests as part of an effort to strengthen accountability for schools, teachers, and students. Predictably, there has been vigorous disagreement regarding the contributions of such policies to increasing test scores and, more importantly, to improving student learning. A recent study by Amrein and Berliner (2002a) has received a great deal of media attention. Employing various databases covering the period 1990–2000, the authors conclude that there is no evidence that states that implemented high‐stakes tests demonstrated improved student achievement on various external measures such as performance on the SAT®, ACT, AP®, or NAEP. In a subsequent study in which they conducted a more extensive analysis of state policies (Amrein & Berliner, 2002b), they reach a similar conclusion. However, both their methodology and their findings have been challenged by a number of authors.In this paper, we undertake an extended reanalysis of one component of Amrein and Berliner (2002a). We focus on the performance of states, over the period 1992 to 2000, on the NAEP mathematics assessments for grades 4 and 8. In particular, we compare the performance of the high‐stakes testing states, as designated by Amrein and Berliner, with the performance of the remaining states (conditioning, of course, on a state's participation in the relevant NAEP assessments). For each grade, when we examine the relative gains of states over the period, we find that the comparisons strongly favor the high‐stakes testing states. Moreover, the results cannot be accounted for by differences between the two groups of states with respect to changes in percent of students excluded from NAEP over the same period. On the other hand, when we follow a particular cohort (grade 4, 1992 to grade 8, 1996 or grade 4, 1996 to grade 8, 2000), we find the comparisons slightly favor the low‐stakes testing states, although the discrepancy can be partially accounted for by changes in the sets of states contributing to each comparison. In addition, we conduct a number of ancillary analyses to establish the robustness of our results, while acknowledging the tentative nature of any conclusions drawn from highly aggregated, observational data.
A UNIFIED APPROACH TO LINEAR EQUATING FOR THE NON‐EQUIVALENT GROUPS DESIGN
Authors: von Davier Alina A. , Kong Nan 
Published: August 2014

ABSTRACTThis paper describes a new, unified framework for linear equating in a Non‐Equivalent‐groups Anchor Test (NEAT) design. We focus on three methods for linear equating in the NEAT design—Tucker, Levine observed‐score, and chain—and develop a common parameterization that allows us to show that each particular equating method is a special case of the linear equating function in the NEAT design. We use a new concept, the Method Function, to distinguish among the linear equating functions, in general, and among the three equating methods, in particular. This approach leads to a general formula for the standard error of equating for all equating functions in the NEAT design. We also present a new tool, the standard error of equating difference, to investigate if the observed difference in the equating functions is statistically significant.