bluuebunny/crossref_metadata_2025 huggingface dataset:
DEVELOPMENT OF A SCALE FOR ASSESSING THE LEVEL OF COMPUTER FAMILIARITY OF TOEFL EXAMINEES
Authors: Eignor Daniel , Taylor Carol , Kirsch Irwin , Jamieson Joan
Published: August 2014

The increasing use of computer‐based testing raises concerns about equity and bias. Specifically, many in the field of language testing are concerned that the introduction of a computer‐based TOEFL test in 1998 may introduce bias by confounding the measurement of English‐language proficiency with level of computer familiarity. As the first step in a series of research efforts to address this question, a 23‐item questionnaire focusing on individuals' access to, attitude toward, and experience with using computers as well as related technology was developed and administered to TOEFL test takers in April and May of 1996 as part of a TOEFL test administration. Results of this survey administration are documented in a paper by Kirsch, Jamieson, Taylor, and Eignor (1998).

This paper has two purposes. The first is to explain decisions as to which questionnaire items would subsequently be used together as a score to separate TOEFL examinees into levels of computer familiarity. The second purpose is to provide details on the procedures used to assess the underlying factor structure of the complete questionnaire. Factor analysis procedures (specifically, a principal factors approach with both varimax orthogonal and promax oblique rotations) were used for both purposes.

Eleven items, which loaded heavily on the first factor of a two‐factor common factor solution with promax oblique rotation, were selected to create a computer familiarity score. This score was then used to classify TOEFL examinees into one of three familiarity groups: low, moderate, and high. A four‐factor common factor solution with promax oblique rotation was seen as providing the best accounting of the underlying factor structure of the complete questionnaire.
DEVELOPMENT, SELECTION AND VALIDATION OF A SET OF COGNITIVE AND LINGUISTIC ATTRIBUTES FOR THE SAT I VERBAL: SENTENCE COMPLETION SECTION1
Authors: Buck Gary , VanEssen Tom , Tatsuoka Kikumi , Kostin Irene , Lutz Donna , Phelps Matthew
Published: August 2014

The purpose of this study is to use the rule space procedure to develop a general set of cognitive and linguistic attributes that account for performance on the Sentence Completion Section of the SAT I Verbal. The overall research strategy was to develop a set of cognitive and linguistic attributes on one form of the Sentence Completion Section, the experimental form, and then to examine the generalizability of these by applying the same set of attributes to a different form of the test, the generalizability form. Results produced a set 20 cognitive and linguistic attributes, along with 3 interaction attributes, that were able to classify 97% of test‐takers into their latent knowledge states on the experimental form; the same set of attributes were able to classify 98% of test takers into their latent knowledge states on the generalizability form. The analyses provide an indication of the important determinants of success on the SAT Sentence Completion task, and provide evidence that these are relatively stable across forms.
EFFECTS OF EXAMINEE CHOICE ON A TEST OF DIVERGENT THINKING
Authors: Powers Donald E. , Bennett Randy Elliot
Published: August 2014

Under some circumstances, allowing examinees to choose which items to respond to may increase test validity. In this study, we explored how choice affected examinee performance and related characteristics for a test of the ability to generate hypotheses about a given situation. Four forms of this experimental test were randomly spiraled within a large sample of students taking the computer‐based GRE General Test, with two of the forms allowing examinee choice. Results suggested that items were, in fact, differentially attractive to gender, ethnic, undergraduate major, and ability groups, indicating that, in some form, choice was operating. Further, performance was significantly higher when a given item was chosen than when it was required. Effects of choice on subgroup performance were generally inconclusive. Finally, there was some indication that internal consistency was improved by the choice option.
PREPARING FOR THE SAT® I: REASONING TEST—AN UPDATE
Authors: Powers Donald E.
Published: August 2014

To document the extent of special test preparation for the SAT® I: Reasoning Test, we surveyed a stratified random sample of some 6,700 students who registered to take the SAT I in 1995–96. A smaller companion survey sought information about special preparation programs from a stratified random sample of secondary schools whose students take the SAT I. The objectives were to:

determine the availability, and incidence of use, of a variety of programs and resources designed to prepare students to take the SAT I;

describe some of the salient features of these resources; and

estimate the amount of time (and money) that students spend on preparing for the test.

Though the surveys differed slightly from similar surveys conducted in 1986–87, they were designed generally to enable comparison with the results of the earlier surveys.

The student survey found that prospective SAT I takers participate, to varying degrees, in a variety of preparation activities. Taking the PSAT/NMSQT for practice and using the test familiarization materials provided by the College Board are the most frequently used strategies. Other commercially available books and texts used in regular courses are also consulted relatively frequently. Engagement is much less frequent with such resources as test‐preparation software, special programs given either at school or outside school, or private tutoring. When particular programs or resources were available, cost was cited less often than some other factor as a reason for not using the resource. This was true for each of several resources, including coaching courses. About 12 percent of all students in the survey said they had attended preparation or coaching sessions outside school, where this minority of students paid, on average, about $400.

On average, students currently spend a total of approximately 11 hours preparing for the SAT I, about the same amount (10 hours) that students reported in the 1986–87 survey. Currently, about 10 percent of all students report spending 54 hours or more preparing for the test (about the same as the 58 hours reported in 1986–87). Now, approximately 3 percent of test takers say they do not devote any time at all to SAT preparation (compared with 9 percent in 1986–87).

The results of the school survey revealed that a slight majority (52 percent) of all secondary schools now offer programs to prepare students for the SAT I, about the same proportion (49 percent) as in 1986–87.
TEST TAKERS' JUDGMENTS ABOUT GRE WRITING TEST PROMPTS
Authors: Powers Donald E. , Fowles Mary E.
Published: August 2014

This study gathered the judgments of various GRE test takers about a total of 78 essay prompts that were being considered for possible use in a GRE writing test. The objective was to determine the kinds of prompts and topics on which examinees feel they can write strong essays, as well as those that they perceive as more difficult. The study identified several features that underlie examinee assessments of essay prompts.

Some of the study participants also wrote essays on a small subset of the prompts, and their opinions of the prompts were compared to the scores that GRE readers assigned to their essays. This comparison revealed only a weak and inconsistent relationship between writers' judgments about the prompts and their performance on those prompts.

The report discusses the implications of the findings for the development of the GRE writing test and for advising GRE examinees on how to approach the test.
SOME PATHS TOWARD MAKING PRAXIS SCORES MORE USEFUL
Authors: Wainer Howard , Sheehan Kathleen , Wang Xiaohui
Published: August 2014

In this study we describe an analytic method for aiding in the generation of subscales that characterize the deep structure of tests. In addition we also derive a procedure for estimating scores for these scales that are much more statistically stable than subscores computed solely from the items that are contained on that scale. These scores achieve their stability through augmentation with information from other related information on the test. These methods were used to complement each other on a data set obtained from a Praxis administration. We found that the deep structure of the test yielded ten subscales, and that because the test was essentially unidimensional ten subscores could be computed; all with very high reliability. This result was contrasted with the calculation of six traditional subscales based on surface features of the items. These subscales also yielded augmented subscores of high reliability.
CONSEQUENCES OF TEST INTERPRETATION AND USE: THE FUSION OF VALIDITY AND VALUES IN PSYCHOLOGICAL ASSESSMENT
Authors: Messick Samuel
Published: August 2014

This paper addresses the role of social values in educational and psychological measurement, with special attention to the consequences of testing as validity evidence. Test use depends for its legitamicy on adherence to established principles of reliability, validity, and fairness. These principles are organized within an overarching conception of construct validity. Desired reliability depends on the meaning of the construct being measured while test fairness depends on comparable construct validity across groups or settings. In the name of fairness, value judgements need to be made throughout the testing process, as they do in the name of validity throughout the validation process.
DETERMINANTS OF PERSISTENCE OF GRADUATE STUDENTS IN PH.D. PROGRAMS
Authors: Attiyeh Gregory M.
Published: August 2014

This study utilizes longitudinal data on individual students enrolled in doctoral programs from 1989 to 1993 to identify patterns and determinants of their persistence. For each of five disciplines (biochemistry, economics, English, mathematics, and mechanical engineering), a survival model is specified to explain student persistence through the first four years of graduate study. Variables hypothesized to explain persistence include student aptitude and achievement, financial support, and demographic characteristics, as well as characteristics of the programs in which students were enrolled. The statistical estimates reveal significant differences across disciplines, but several general results are apparent: a student with given characteristics is more likely to persist at more selective institutions; other things being equal, a student is more likely to persist if he or she has greater financial support, higher GRE Verbal or Quantitative scores, and a master's degree. Student citizenship, gender, ethnicity, and age are not consistently related to persistence across disciplines.
EVIDENCE RELATED TO THE VALIDITY OF THE ENGLISH LANGUAGE PROFICIENCY TEST
Authors: Bridgeman Brent , Harvey Anne
Published: August 2014

The validity of the English Language Proficiency Test was evaluated by considering six aspects of construct validity identified by Messick: content, substantive, structural, generalizability, external, and consequential. Analyses evaluated both the internal structure of the test and the relationship of the scores to external criteria. In one set of analyses, high school and college language teachers were asked to evaluate their students using the same five proficiency categories that are used in the test, and the teacher ratings were then compared to the test scores. Correlations were averaged over 15 classes in the college sample and over 32 classes in the high school sample. In the college sample, the averaged correlations, corrected for range restriction, were .50 and .57 for reading and listening respectively. In the high school sample the corresponding correlations were .68 and .71. For both the reading and listening proficiency scales, teachers tended to rate their students about one category higher than they were rated by the test.
SEX AND ETHNIC GROUP DIFFERENCES ON ACCOMPLISHMENTS MEASURES
Authors: Stricker Lawrence J. , Rock A. , Bennett Randy Elliot
Published: August 2014

This study investigated sex and ethnic group differences on measures of accomplishments–notable attainments that have been publicly recognized–for GRE test takers. Six accomplishments scales (Academic, Leadership, Practical Language, Aesthetic Expression, Science, and Mechanical) were studied. With the exception of the Mechanical scale, men and women did not differ in their level of performance on the scales, and ethnic groups did not differ on any scale. Furthermore, the factor structure of the scales did not differ in most respects for men and women. Concerns about sex and ethnic group differences on accomplishments measures are probably unwarranted. (Key words: accomplishments, sex differences, ethnic group differences, factor analysis)
TEST ANXIETY AND TEST PERFORMANCE: COMPARING PAPER‐BASED AND COMPUTER‐ADAPTIVE VERSIONS OF THE GRE GENERAL TEST
Authors: Powers Donald E.
Published: August 2014

Despite some assumptions to the contrary, there is reason to believe that the introduction of computer‐adaptive testing may actually help to alleviate test anxiety and diminish the relationship between test anxiety and test performance. This study provided a test of this hypothesis. Results are based on a sample of GRE General Test takers who took the computer‐adaptive version of the test, and another sample of GRE examinees who took the paper‐based version of the test. After taking the test, all examinees completed both a test anxiety inventory and an inventory concerning attitudes toward computers. Relationships were examined between performance on each of the three GRE General Test measures and reports of test anxiety (both worry and emotionality) and computer attitudes (both anxiety and confidence). For both the test anxiety and the computer attitudes scales, the relationship to GRE scores was similar for the computer‐adaptive and paper‐based GRE General Test. Thus, there was no support for the study's major hypothesis. Several ancillary findings, however, do have implications for large‐scale testing programs, especially those moving to computer‐based testing.
TRENDS AND PROFILES: STATISTICS ABOUT GRE® GENERAL TEST EXAMINEES BY GENDER, AGE, AND ETHNICITY SECOND EDITION
Authors: Grandy Jerilee
Published: August 2014

This second edition of Trends and Profiles presents statistics from the files of the Graduate Record Examinations (GRE) from 1982 through 1996. In question‐and‐answer format with many tables and graphs, the report addresses frequently asked questions about U.S. citizens who take the GRE. It covers demographics, undergraduate studies, GRE test preparation, test scores, and graduate school plans. Statistical profiles for 1996 and trends from 1982 to 1996 are presented by gender, ethnic group, and age group.
ENGLISH PROFICIENCY AND DIFFERENCES AMONG RACIAL AND ETHNIC GROUPS IN MEAN SAT® AND GRE® SCORES: A LONGITUDINAL ANALYSIS
Authors: Pennock‐Román Maria
Published: August 2014

Although limited proficiency in the language of a test is known to depress aptitude test scores, the changes that occur as proficiency rises over time have been less well studied. The objective here is to contrast longitudinal changes in test performance for persons who indicated that English was (EBL) or was not their best language (ENBL). Analyses were based on a sample of U.S. citizens and permanent residents (N=65,987 EBL and N=1,592 ENBL); each individual had taken both the Scholastic Assessment Test (SAT)1 and the Graduate Record Examinations (GRE) General Test some years later. The distance in verbal mean scores between the EBL non‐Hispanic White group and ENBL groups (such as, non‐Hispanic White, Asian‐American, Black, Puerto Rican, other Hispanic) grew closer by 0.21 to 0.48 standard deviation units from the taking of the SAT to the taking of the GRE. These findings have implications for the interpretation of test scores in longitudinal studies of linguistic minorities and for selective admissions of students with limited proficiency in English.
VALIDATING A TEST DESIGNED TO ASSESS ESL PROFICIENCY AT LOWER DEVELOPMENTAL LEVELS
Authors: Wilson Kenneth M.
Published: August 2014

This exploratory study examined relationships between a test designed to assess English‐language listening comprehension and reading skills in samples of nonnative‐English speakers at lower levels of developed proficiency in English as a second or foreign language (ESL or EFL). The test—called ESL‐EZY—was developed by using items similar to but easier, on the average, than those being used in an existing ESL proficiency test designed for intermediate‐ or higher‐level ESL users/learners. This paper reports evidence regarding the relationship between ESL‐EZY scores and teachers' ratings of oral English proficiency in samples—assessed in diverse settings in the United States and Japan—selected to include subgroups that tend to differ relatively widely in average level of developed English proficiency. Scores on ESL‐EZY were found to correlate relatively strongly with teachers' ratings within the respective subgroups; correlations were especially strong in relatively less proficient subgroups, suggesting that for younger ESL‐students and other ESL‐learners with limited developed functional proficiency in English as a foreign or second language, a test embodying properties similar to those represented by ESL‐EZY might provide useful supplementary assessment information.
COMPARING THE INCOMPARABLE: AN ESSAY ON THE IMPORTANCE OF BIG ASSUMPTIONS AND SCANT EVIDENCE
Authors: Wainer Howard
Published: August 2014

Comparisons between groups usually involve making statistical adjustments so that the comparisons are, statistically speaking, ceteris paribus. Usually included within those adjustments are some untestable assumptions. These untestable assumptions are often determined by the goals of the testing program; they are sometimes justified on the basis of theory, sometimes on the basis of political expediency, and sometimes on just a hope and a prayer. If we are vigilant, bits of evidence may emerge from the data that can corroborate or refute our assumptions. In this essay I describe several examples, drawn from mental testing, that illustrate these issues, and suggest a plausible path of investigation that may provide a better estimate of the practical limitations of accuracy for any proposed solution to this vexing equity issue.
USER'S GUIDE FOR SCORIGHT (VERSION 1.2): A COMPUTER PROGRAM FOR SCORING TESTS BUILT OF TESTLETS
Authors: Wang Xiaohui , Bradlow Eric T. , Wainer Howard
Published: August 2014

SCORIGHT is a very general computer program for scoring tests. It models tests that are made up of dichotomously or polytomously rated items, or any kind of combination of the two through the use of a generalized IRT formulation. The items can be presented independently, or grouped into clumps of allied items (testlets), or in any combination of the two. When there are testlets the program assesses the degree of local dependence and adjusts the estimates accordingly. The estimation is accomplished within a fully Bayesian framework using Markov chain Monte Carlo procedures. This methodology yields the complete posterior distributions of the parameters, which allows the easy calculation of many important characteristics of the scores that are not available with other methods.
USING A NEW STATISTICAL MODEL FOR TESTLETS TO SCORE TOEFL
Authors: Wainer Howard , Wang Xiaohui
Published: August 2014

Standard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimuli (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ending an examination in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, and Du, 2000). This parameter, γ, characterizes the amount of local dependence in a testlet.

We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters [discrimination (a), difficulty (b), and guessing (c)] with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over‐estimated when conditional independence was incorrectly assumed.
GRE® AUTOMATED‐EDITING‐TASK PHASE II REPORT: ITEM ANALYSES, REVISIONS, VALIDITY STUDY, AND TAXONOMY DEVELOPMENT
Authors: Breland Hunter , Kukich Karen , Hemat Lisa
Published: August 2014

Two automated editing tasks developed in a Phase I study were subjected to item analyses, revised, and then used in a computer‐based test administration at a local college. The data collected in the administration were compared with questionnaire data obtained from students to examine the construct validity of the tasks. In a second approach to construct validation, a taxonomy of writing skills was developed and compared to the skills assessed by the editing tasks. Data analyses indicate that total editing score correlates more strongly with self‐reported English grades than with self‐reported mathematics grades, and that total editing score correlates positively with student self‐assessments of their writing skill, recent grades on writing assignments, and college grade point average. A review of the task elements against the taxonomy indicates that the editing tasks assess important writing skills not assessed by free‐response essays. For this and other reasons, it was concluded that automated editing tasks would serve as a useful complement to free‐response writing assessments.
EFFECTS OF THE PRESENCE AND ABSENCE OF VISUALS ON PERFORMANCE ON TOEFL® CBT LISTENING‐COMPREHENSION STIMULI
Authors: Ginther April
Published: August 2014

This study was conducted in order to begin to understand the effects of providing different types of visuals – such as pictures, diagrams, and still photos of the speakers – with items on the computer‐based Test of English as a Foreign Language (TOEFL®). A nested crossover design (subjects nested in proficiency, level, and form) was used to examine the effects of language proficiency (high or low), still photos (present or absent), and type of stimuli (dialogues/short conversations, academic discussions, mini‐talks with context visuals, mini‐talks with content visuals) on performance on TOEFL multiple‐choice items. Three two‐way interactions were significant: proficiency by type of stimuli, type of stimuli by visual condition, and type of stimuli by time. The weakest of these interactions, type of stimuli by visual condition, was the most interesting; it indicated that the presence of visuals results in facilitation of performance when the visuals bear information that complement the audio portion of the stimulus. The interaction effect between type of stimulus and time suggests that the trend toward the use of longer stimuli is not without consequences, and these consequences may extend to testing situations. Responses to a series of questionnaires indicate that the majority of the subjects preferred the stimuli that were accompanied by visuals.
THE INTERNATIONAL ADULT LITERACY SURVEY (IALS): UNDERSTANDING WHAT WAS MEASURED
Authors: Kirsch Irwin S.
Published: August 2014

This paper offers a framework that has been used for both developing the tasks used to measure literacy and for understanding the meaning of what has been reported with respect to the comparative literacy proficiencies of adults in participating countries. The framework consists of six parts that represent a logical sequence of steps, from needing to define and represent a particular domain of interest, to identifying and operationalizing characteristics used to construct items, to providing an empirical basis for interpreting results. The various parts of the framework are seen as important in that they help to provide a deeper understanding of the construct of literacy and the various processes associated with it. A processing model is proposed and variables associated with performance on the literacy tasks are identified and verified through regression analyses. These variables are shown to account for between 79% and 89% of the variance in task difficulty. Collectively, these process variables provide a means for moving away from interpreting performance on large‐scale surveys in terms of discrete tasks or a single number, toward identifying levels of performance that have generalizability across pools of tasks and toward what Messick (1989) has called a higher level of measurement.
CAUSATION AND RACE
Authors: Holland Paul W.
Published: August 2014

Race is often viewed as a causal variable and “RACE effects” found from regression analyses are sometimes given causal interpretations. I argue that this is a mistaken way to proceed. RACE is not a causal variable in a very important sense of the word, and yet it does have a significant role in causal studies. The key role that RACE can play is to help our understanding of the effects of causes or interventions through the statistical “interaction” of RACE with causal variables, rather than as the main effect of RACE. These ideas are briefly illustrated using data from a study of tests constructed to manipulate the distribution of scores of Black and White test takers.
KNOWLEDGE AND SKILL INDICATORS IMPORTANT FOR BEGINNING TEACHERS: A SURVEY OF PENNSYLVANIA TEACHERS AND TEACHER EDUCATORS
Authors: Wylie E. Caroline , Tannenbaum Richard J.
Published: August 2014

The study investigated the job relevance of 70 knowledge and/or skill indicators across five content areas (Literature, Fine Arts, Mathematics, Social Studies, and Science) for the development of a Pennsylvania initial teacher licensure assessment. A random sample of 1,700 teachers and 300 teacher educators was selected from Pennsylvania. Respondents judged each indicator on a scale ranging from 1 (of no importance) to 5 (very high level of importance). Means (and standard deviations) were computed across all survey respondents, as well as for subgroups of teacher respondents. Using a mean of 3.50 as the criterion for job relevance, 24 of the 70 (34%) of the knowledge and/or skill indicators were judged to be important (job‐relevant) enough for consideration in the development of the licensure assessment. These 24 knowledge and/or skill indicators are recommended as the primary content of the test.
ASSESSING CONVERGENCE OF THE MARKOV CHAIN MONTE CARLO ALGORITHMS: A REVIEW
Authors: Sinharay Sandip
Published: August 2014

Markov chain Monte Carlo (MCMC) algorithms are in wide use for fitting complicated statistical models in psychometrics in situations where the traditional estimation techniques are very difficult to apply. One of the stumbling blocks in using an MCMC algorithm is determining the convergence of the algorithm. Because the convergence is not that of a scalar quantity to a point, but that of a distribution to another distribution, the issue remains an enigma to many users of MCMC, especially to those without a sound knowledge of mathematical statistics. This article is an attempt to provide psychometricians using the MCMC algorithms a better understanding of the concept of convergence of the algorithms and an improved knowledge about the diagnostics tools to assess convergence of the MCMC algorithms.
ANALYSIS OF NAEP COMBINED NATIONAL AND STATE SAMPLES
Authors: Qian Jiahe , Kaplan Bruce , Weng Vincent
Published: August 2014

This National Assessment of Educational Progress (NAEP) study analyzes the efficacy of combined national and state NAEP samples. The main goal of combining NAEP national and state samples is to provide an integrated way to implement NAEP assessments and to reduce the burden of state assessments by administrating one assessment. The study is based on combined samples of the 1998 and 2000 NAEP assessments.
WRITING DIFFERENCES IN TEACHER PERFORMANCE ASSESSMENTS: USE OF AFRICAN AMERICAN LANGUAGE
Authors: Wylie E. Caroline , Szpara Michelle Y.
Published: August 2014

This study investigated potential writing differences between African American and European American candidates in a National Board for Professional Teaching Standards (NBPTS) portfolio entry from the Middle Childhood/Generalist Certificate. The use of features other than Edited American English was examined as a possible source of construct‐irrelevant variance in assessors' scoring judgments. Thirty‐two written commentaries, 18 from African American candidates and 14 from European American candidates, were coded for grammatical, lexical, and discourse features. The coding frame identified features of African American Language (AAL) and Speech Code Errors (SCE). Instances of AAL were fewer than instances of SCE and clustered according to potential users of AAL.
POPULATION INVARIANCE OF SCORE LINKING: THEORY AND APPLICATIONS TO ADVANCED PLACEMENT PROGRAM® EXAMINATIONS
Authors: Dorans Neil J. , von Davier Alina A. , Dorans Neil J. , Holland Paul W. , Tateneni Krishna , Thayer Dorothy T. , Yang Wen‐Ling , Green Bert F. , Kolen Michael L.
Published: August 2014

Test equating methods are intended to produce interchangeable scores and should not be strongly influenced by the group of examinees on which they are computed. Advanced Placement Program® (AP®) exams, which include both multiple‐choice (MC) items and constructed‐response (CR) items, provide an environment for studying the effects of subpopulations on the estimation of equating functions. The exams are equated to past forms using an internal anchor‐test design in which the linking items are restricted to MC items due to the immediate disclosure of the CR sections of the tests (precluding their reuse as linking items). This collection of closely related papers uses the variety of cases provided by AP exams to assess the effects of subpopulations on various important aspects of test equating. Earlier versions of these papers were prepared for a symposium at the 2002 Annual Meetings of the National Council on Measurement in Education.
RECONSIDERING THE IMPACT OF HIGH‐STAKES TESTING
Authors: Braun Henry
Published: August 2014

Over the last 15 years, many states have implemented high‐stakes tests as part of an effort to strengthen accountability for schools, teachers, and students. Predictably, there has been vigorous disagreement regarding the contributions of such policies to increasing test scores and, more importantly, to improving student learning. A recent study by Amrein and Berliner (2002a) has received a great deal of media attention. Employing various databases covering the period 1990–2000, the authors conclude that there is no evidence that states that implemented high‐stakes tests demonstrated improved student achievement on various external measures such as performance on the SAT®, ACT, AP®, or NAEP. In a subsequent study in which they conducted a more extensive analysis of state policies (Amrein & Berliner, 2002b), they reach a similar conclusion. However, both their methodology and their findings have been challenged by a number of authors.

In this paper, we undertake an extended reanalysis of one component of Amrein and Berliner (2002a). We focus on the performance of states, over the period 1992 to 2000, on the NAEP mathematics assessments for grades 4 and 8. In particular, we compare the performance of the high‐stakes testing states, as designated by Amrein and Berliner, with the performance of the remaining states (conditioning, of course, on a state's participation in the relevant NAEP assessments). For each grade, when we examine the relative gains of states over the period, we find that the comparisons strongly favor the high‐stakes testing states. Moreover, the results cannot be accounted for by differences between the two groups of states with respect to changes in percent of students excluded from NAEP over the same period. On the other hand, when we follow a particular cohort (grade 4, 1992 to grade 8, 1996 or grade 4, 1996 to grade 8, 2000), we find the comparisons slightly favor the low‐stakes testing states, although the discrepancy can be partially accounted for by changes in the sets of states contributing to each comparison. In addition, we conduct a number of ancillary analyses to establish the robustness of our results, while acknowledging the tentative nature of any conclusions drawn from highly aggregated, observational data.
A UNIFIED APPROACH TO LINEAR EQUATING FOR THE NON‐EQUIVALENT GROUPS DESIGN
Authors: von Davier Alina A. , Kong Nan
Published: August 2014

This paper describes a new, unified framework for linear equating in a Non‐Equivalent‐groups Anchor Test (NEAT) design. We focus on three methods for linear equating in the NEAT design—Tucker, Levine observed‐score, and chain—and develop a common parameterization that allows us to show that each particular equating method is a special case of the linear equating function in the NEAT design. We use a new concept, the Method Function, to distinguish among the linear equating functions, in general, and among the three equating methods, in particular. This approach leads to a general formula for the standard error of equating for all equating functions in the NEAT design. We also present a new tool, the standard error of equating difference, to investigate if the observed difference in the equating functions is statistically significant.
AUTOMATED TOOLS FOR SUBJECT MATTER EXPERT EVALUATION OF AUTOMATED SCORING
Authors: Williamson David M. , Bejar Isaac I. , Sax Anne
Published: August 2014

As automated scoring of complex constructed‐response examinations reaches operational status, the process of evaluating the quality of resultant scores, particularly in contrast to scores of expert human graders, becomes as complex as the data itself. Using a vignette from the Architectural Registration Examination (ARE), this paper explores the potential utility of classification and regression trees (CART) and Kohonen self‐organizing maps (SOM) as tools to facilitate subject matter expert (SME) examination of the fine‐grained (feature level) quality of automated scores for complex data, with implications for the validity of the resultant scores. The paper explores both supervised and unsupervised learning techniques, the former being represented by CART (Breiman, Friedman, Olshen, & Stone, 1984) and the latter by SOM (Kohonen, 1989). Three applications comprise this investigation, the first of which suggests that CART can facilitate efficient and economical identification of specific elements of complex solutions that contribute to automated and human score discrepancies. The second application builds on the first by exploring CART's value for efficiently and accurately automating case selection for human intervention to ensure score validity. The final application explores the potential for SOM to reduce the need for SMEs in evaluating automated scoring. While both supervised and unsupervised methodologies examined were found to be promising tools for facilitating SME roles in maintaining and improving the quality of automated scoring, such applications remain unproven and further studies are necessary to establish the reliability of these techniques.
TOEIC®/LPI RELATIONSHIPS IN ACADEMIC AND EMPLOYMENT CONTEXTS IN THAILAND
Authors: Wilson Kenneth M. , Nagara Supalak Komarakul Na , Woodhead Robert
Published: August 2014

The Test of English for International Communication (TOEIC®) is an ETS‐developed measure of proficiency in English as a second language (ESL) or foreign language (EFL), designed to assess aspects of English language listening comprehension (LC) and reading comprehension (RC). In general samples of native speakers of Japanese, Korean, French, Spanish, and Arabic, respectively, scores on the TOEIC have been found to be strongly related to performance on the Language Proficiency Interview (LPI) procedure, a widely recognized direct measure of speaking proficiency. In this exploratory study, TOEIC/LPI relationships in subgroups (e.g., by gender, educational level, organizational setting at time of assessment) were assessed within a general national/linguistic sample—native speakers of Thai who are involved in operational assessments under the aegis of the TOEIC representative agency in Thailand, the Center for Professional Assessment. Observed relationships were relatively strong and positive in the total sample and within each of the subgroups studied. Observed coefficients were noticeably larger in subgroups of females than in corresponding subgroups of males and for a sample of university students than for examinees tested in places of employment. Means of observed LPI rating for the respective subgroups were consistent with expectations based on TOEIC scores. Further research is needed to assess the extent to which the subgroup findings may tend to be generalizable in the Center (Thailand) context or elsewhere.
STATISTICAL AND MEASUREMENT PROPERTIES OF FEATURES USED IN ESSAY ASSESSMENT
Authors: Haberman Shelby J.
Published: August 2014

Statistical and measurement properties are examined for features used in essay assessment to determine the generalizability of the features across populations, prompts, and individuals. Data are employed from TOEFL® and GMAT® examinations and from writing for CriterionSM.
COMPARABILITY OF TOEFL CBT WRITING PROMPTS FOR DIFFERENT NATIVE LANGUAGE GROUPS
Authors: Lee Yong‐Won , Breland Hunter , Muraki Eiji
Published: August 2014

This study has investigated the comparability of computer‐based testing (CBT) writing prompts in the Test of English as a Foreign Language™ (TOEFL®) for examinees of different native language backgrounds. A total of 81 writing prompts introduced from July 1998 through August 2000 were examined using a three‐step logistic regression procedure for ordinal items. An English language ability (ELA) variable was created by summing the standardized TOEFL Reading, Listening, and Structure scale scores. This ELA variable was used to match examinees of East Asian (Chinese, Japanese, and Korean) and European (German, French, and Spanish) language groups. Although about one third of the 81 prompts were initially flagged because of statistically significant group effects, the effect sizes were too small for any of those flagged prompts to be classified as having an important group effect.
EXAMINING FREEDLE'S CLAIMS ABOUT BIAS AND HIS PROPOSED SOLUTION: DATED DATA, INAPPROPRIATE MEASUREMENT, AND INCORRECT AND UNFAIR SCORING
Authors: Dorans Neil J. , Zeller Karin
Published: August 2014

In the Spring 2003 issue of Harvard Educational Review, Roy Freedle stated that the SAT® is both culturally and statistically biased, and he proposed a solution to ameliorate this bias. His claims, which garnered national attention, were based on serious errors in his analysis. We begin our analyses by assessing the psychometric properties of Freedle's recommended hard half‐test that he thinks should form the basis for the supplemental SAT score he proposes to report. Next we demonstrate his justification for a score based on this half‐test is based on a flawed analysis. The numbers in his critical Table 2 do not represent what he claims they do. We then demonstrate what occurs when current data are used with both the correct scoring and the incorrect scoring proposed by Freedle. When the table is constructed correctly using current data, the effects that Freedle reported are reduced substantially in magnitude to the point where they do not warrant any of the corrective actions he proposes. We conclude our analysis of Freedle's claims with DIF analyses of our own that are inconsistent with his DIF analyses of old SAT test editions he used as his initial motivation for correcting the SAT scoring procedure.
THE BEST LINEAR PREDICTOR FOR TRUE SCORE FROM A DIRECT ESTIMATE AND SEVERAL DERIVED ESTIMATES
Authors: Haberman Shelby J. , Qian Jiahe
Published: August 2014

Statistical prediction problems often involve both a direct estimate of a true score and covariates of this true score. Given the criterion of mean squared error, this study determines the best linear predictor of the true score given the direct estimate and the covariates. Results yield an extension of Kelley's formula for estimation of the true score to cases in which covariates are present. The best linear predictor is a weighted average of the direct estimate and of the linear regression of the direct estimate onto the covariates. The weights depends on the reliability of the direct estimate and on the multiple correlation of the true score with the covariates. One application of the best linear predictor is to approximate the human true score from the observed holistic score of an essay and from essay features derived from a computer analysis.
POPULATION INVARIANCE AND LINEAR EQUATING FOR THE NON‐EQUIVALENT GROUPS DESIGN
Authors: von Davier Alina A. , Han Ning
Published: August 2014

This study investigates the population sensitivity of the commonly used linear equating methods in the Non‐Equivalent‐groups with an Anchor Test (NEAT) design: the Tucker, the Levine observed‐score, and the chain linear methods. For a detailed analysis of the subject, we apply three distinctive approaches to a real data set from a NEAT design: a) the RMSD index for the NEAT design of von Davier, Holland, and Thayer (2004); b) the parallel‐linking system of Dorans and Holland (2000), and c) the pseudo‐NEAT design approach of von Davier, Holland, and Thayer (2003).

The data set is used to illustrate the derivations on male and female subpopulations. Comparisons of the results obtained using linear and equipercentile equating are also presented. For this particular data set the Levine function seems to vary less across subpopulations than the other methods; the Tucker method seems to vary the most.
USER'S GUIDE FOR SCORIGHT (VERSION 3.0): A COMPUTER PROGRAM FOR SCORING TESTS BUILT OF TESTLETS INCLUDING A MODULE FOR COVARIATE ANALYSIS
Authors: Wang Xiaohui , Bradlow Eric T. , Wainer Howard
Published: August 2014

SCORIGHT is a very general computer program for scoring tests. It models tests that are made up of dichotomously or polytomously rated items or any kind of combination of the two through the use of a generalized item response theory (IRT) formulation. The items can be presented independently or grouped into clumps of allied items (testlets) or in any combination of the two. When there are testlets, the program assesses the degree of local dependence and adjusts the estimates accordingly. The estimation is accomplished within a fully Bayesian framework using Markov chain Monte Carlo procedures, which allows the easy calculation of many important characteristics of the scores that are not available with other methods. The current version of SCORIGHT, version 3.0, includes a new module that allows the user to include covariates in the analysis.
COGNITIVE DIAGNOSIS FOR NAEP PROFICIENCY DATA
Authors: Xu Xueli , von Davier Matthias
Published: August 2014

More than a dozen statistical models have been developed for the purpose of cognitive diagnosis. These models are supposed to extract a much finer level of information from item responses than traditional unidimensional item response models. In this paper, a general diagnostic model (GDM) was used to analyze a set of simulated sparse data and real data from National Assessment of Educational Progress (NAEP) assessments. The purpose of this study was to find out (a) whether the parameters can be recovered for a sparse data matrix in the framework of the GDM and (b) how to estimate group characteristics for large survey data, such as NAEP, in the framework of the GDM. The results of the simulation study show that GDM parameters can be recovered satisfactorily. The GDM under single group and multiple group assumptions were employed to fit NAEP assessment data. The results under these two assumptions and between the GDM and the statistics derived using the operational NAEP model were compared. The findings indicate that it is possible to conduct cognitive diagnosis for NAEP proficiency data.
ADAPTIVE TECHNOLOGIES
Authors: Shute Valerie J. , Zapata‐Rivera Diego
Published: August 2014

This paper describes research and development efforts related to adaptive technologies, which can be combined with other technologies and processes to form an adaptive system. The goal of an adaptive system, in the context of this paper, is to create an instructionally sound and flexible environment that supports learning for students with a range of abilities, disabilities, interests, backgrounds, and other characteristics. After defining key terms and establishing a rationale for adaptation, we present a general framework to organize adaptive technologies. We then describe experts' thoughts on what to adapt and how to adapt and conclude with a summary of key challenges and potential futures of adaptive technologies.
AN ENTROPY‐BASED MEASURE OF DEPENDENCE BETWEEN TWO GROUPS OF RANDOM VARIABLES
Authors: Kong Nan
Published: August 2014

In multivariate statistics, the linear relationship among random variables has been fully explored in the past. This paper looks into the dependence of one group of random variables on another group of random variables using (conditional) entropy. A new measure, called the K‐dependence coefficient or dependence coefficient, is defined using (conditional) entropy. This paper shows that the K‐dependence coefficient is a measure of the degree of dependence of one group of random variables on another group of random variables. The dependence measured by the K‐dependence coefficient includes both linear and nonlinear dependencies between two groups of random variables. Therefore, the K‐dependence coefficient measures the total degree of dependence and not just the linear component of the dependence between the two groups of random variables. Furthermore, the concept of the K‐dependence coefficient is extended by defining the partial K‐dependence coefficient and the semipartial K‐dependence coefficient. Properties of partial and semipartial K‐dependence coefficients are also explored.
IMPACT OF GROUP WORK AND EXTENDED ESSAY WRITING ON ONLINE ADVANCED PLACEMENT PROGRAM® STUDENT PERFORMANCE
Authors: Handwerk Phil
Published: August 2014

Online high schools are growing significantly in number, popularity, and function. However, little empirical data has been published about the effectiveness of these institutions. This research examined the frequency of group work and extended essay writing among online Advanced Placement Program® (AP®) students, and how these tasks may have impacted their AP exam performance. Hierarchical, set‐wise regression models found little to no impact of group work or extended essay writing on multiple‐choice, free response, or composite scores across 9 AP exams administered from 2003–2005. These models included student sex, race, and year in school as independent variables; covariates were the number of previous AP exams taken by the student and PSAT/NMSQT® scores. Implications for online providers and for future research of online high schools are discussed.