bluuebunny/crossref_metadata_2025 huggingface dataset:
GENERATIVE RESPONSE MODELING: LEVERAGING THE COMPUTER AS A TEST DELIVERY MEDIUM
Authors: Bejar Isaac I.
Published: August 2014

Generative response modeling is an approach to test development and response modeling that calls for the creation of items in such a way that the parameters of the items on some response model can be anticipated through knowledge of the psychological processes and knowledge required to respond to the item. That is, the computer would not merely retrieve an item from a database, as is the case in adaptive testing, but wouldcompose it, or assist in doing so, according to desired specifications. This approach to assessment has implications for both the economics and validity of computer administered tests. To illustrate the concept, a system for measuring writing skills will be outlined where the examinee is expected to rewrite sentences, rather than just recognize errors in a sentence, using a multiple choice format. The possibility of estimating the psychometric parameters of items based on a psychological analysis of the response process will then be examined and shown to be feasible. Such estimates are less precise than estimates based on large samples of test takers. A Monte Carlo study is presented to investigate the possibility of compensating for that imprecision when estimating ability or proficiency. The paper concludes that a generative approach is feasible, and can be a mechanism for taking advantage of the considerable investment required for computer‐based testing.
DESCRIBING COMPLEX ACADEMIC TASKS FROM SIX GRADUATE DISCIPLINES USING MULTIDIMENSIONAL SCALING AND CLUSTERING
Authors: Enright Mary K. , Oltman Philip K.
Published: August 2014

The problem of defining the nature and variety of academic tasks is growing in importance as more complex assessment tasks are introduced in many educational contexts. In this study, multidimensional scaling and cluster analysis were used to describe and categorize tasks from six graduate disciplines including academic psychology, applied psychology, English literature, journalism, physics, and electrical engineering. A sample of task descriptions was constructed through interviews with graduate students from these disciplines. A rating instrument was designed to describe task goals and to evaluate whether the tasks were well or ill structured with respect to various aspects of problem definition and problem solution. Graduate faculty used the rating instrument to characterize a sample of tasks from their discipline.

The scales were found reasonably reliable and were useful in identifying and describing task clusters and how such clusters varied both within and across disciplines. A cluster of short‐term problems that were posed by someone other than the student was found in every field although the other characteristics of this cluster of tasks varied with discipline. For example, the short‐term tasks in engineering and physics were well‐structured, requiring the application of established principles, and having objective standards for judging performance. In contrast, a cluster of short‐term tasks in English literature were very ill‐structured because different conceptual approaches could be relevant, there were alternative methods for accomplishing the tasks, many possible solutions existed, and the student had to define an issue or question to consider. In all disciplines except physics, a cluster of complex tasks emerged that was characterized as having multiple objectives that needed to be satisfied. The cluster of complex tasks that was found in physics was not described clearly by the scales. Problem‐finding was an important task characteristic in the social sciences and humanities but not in the physical sciences. The relevance of multidimensional scaling and clustering to test design is discussed.
AN INVESTIGATION OF PROBLEM SOLVING TASKS IN DIFFERENT GRADUATE DISCIPLINES
Authors: Enright Mary K. , Norback Judith Shaul
Published: August 2014

One way to contextualize reasoning is to create assessment problems that embody critical features of the problems typical of a discipline. Problem characteristics are an important determinant of problem solving and reasoning processes. Critical problem characteristics are likely to vary among disciplines in relation to disciplinary differences in content, structure, and epistemology. However, no systematic or comprehensive description of problem categories and characteristics presently exists. A more abstract description of problems would contribute to the development of assessment problems that better match tasks performed in graduate school. It would also provide a basis for grouping disciplines in terms of critical problem characteristics so that assessment problems appropriate for broad areas rather than specific disciplines can be developed. The goal of this research was to develop a broad overview of problem solving tasks typically encountered in graduate education as well as a conceptual framework for classifying these tasks and their attributes. An overview of problem solving tasks was obtained through interviews with graduate students from six diverse disciplines and the collection of examples of problem solving tasks encountered in different contexts. A preliminary scheme for classifying these tasks was developed and provided a basis for rating scales to identify critical features of the tasks. Graduate faculty were interviewed about the appropriateness of the rating scales for characterizing problem solving tasks within their disciplines and about important student qualities. The implications of this study for developing new forms of assessment are discussed.
EFFECTS OF APPLYING DIFFERENT TIME LIMITS TO A PROPOSED GRE WRITING TEST
Authors: Powers Donald E. , Fowles Mary E.
Published: August 2014

To determine the role of time limits on both test performance and test validity, we asked approximately 300 prospective graduate students to each write two essays — one in a 40‐minute time period and the other in 60 minutes. Analyses revealed that, on average, test performance was significantly better when examinees were given 60 minutes instead of 40. However, there was no interaction between test‐taking style (fast vs. slow) and time limits. That is, examinees who described themselves as slow writers/test takers did not benefit any more (or any less) from generous time limits than did their quicker counterparts. In addition, there was no detectable effect of different time limits on the meaning of essay scores, as suggested by their relationship to several nontest indicators of writing ability.
INTELLIGENT TUTORING AND ASSESSMENT BUILT ON AN UNDERSTANDING OF A TECHNICAL PROBLEM‐SOLVING TASK
Authors: Steinberg Linda S. , Gitomer Drew H.
Published: August 2014

In this paper we describe Hydrive, an operational computer‐based intelligent tutoring system built to help Air Force technicians develop generalizable skills for troubleshooting hydraulics systems. We use Hydrive as an example of how effective training and assessment is developed from a coherent understanding of a target task and how this understanding can be consistently represented in all aspects of a training/assessment instrument. We show how an organizing principle of expert behavior, active path analysis, is used to inform the design and content of the tutor's system, student and instructional models and the design of the tutor's interface. We conclude with information about Hydrive's field trial evaluation and some thoughts on knowledge acquisition and generalizability.
MEASURING SELF‐CONCEPT AND RELATING IT TO ACADEMIC ACHIEVEMENT: STATISTICAL ANALYSES OF THE MARSH SELF‐DESCRIPTION QUESTIONNAIRE
Authors: Wenglinsky Harold
Published: August 2014

Educational intervention programs that seek to raise student self‐concept as a lever for raising student achievement are hampered both by limitations in measurement tools and a lack of knowledge about the relationship between self‐concept and student achievement. Advances in self‐concept research indicate that student self‐concept includes many domains corresponding to the various endeavors of adolescents, from relations with peers to physical abilities. The most state‐of‐the art instrument for adolescents, which takes the multiple domains of self‐concept into account, is the Marsh Self‐Description Questionnaire II (SDQ II). Yet the instrument is lengthy, containing 102 questions including some that are regarded by teachers as insensitive to the feelings of their students, creating problems in administering the instrument. Further, little research has been done on the relationship between specific domains of self‐concept and academic achievement; most researchers tend to claim either that all domains of self‐concept affect academic achievement, or that self‐concept is entirely a by‐product of previous student performance, and therefore does not affect it at all. This study addresses these two issues. First, it applies confirmatory factor analysis to a database constructed by the author from an administration of the 102‐item SDQ II to a sample of 1,130 students participating in Outward Bound, USA programs, finding that the instrument can be reduced to thirty items with little loss of reliability. Second, it applies regression analysis to the National Educational Longitudinal Study (NELS), which contains items from the SDQ II, to measure the relationship between each self‐concept domain and achievement in reading and mathematics. This analysis finds that the effect of self‐concept on achievement is domain specific; academic domains of self‐concept affect academic achievement, while non‐academic domains do not. The study concludes with some suggestions for how intervention programs such as Outward Bound, USA can measure self‐concept and use increases in specific types of self‐concept as a lever for raising student achievement.
SURVEY OF STANDARDS FOR FOREIGN STUDENT APPLICANTS
Authors: Boldt R. F. , Courtney R. G.
Published: August 2014

This project consisted of a survey of practices and standards in the use of the Test of English as a Foreign Language (TOEFL®) and attempted to learn the processes by which these standards were developed. By “standards” we mean the information and requirements established by institutions to evaluate applicants' English proficiency. Information, including other measures that were used with the TOEFL test, was obtained toward that end.

Several findings emerged. The most common practice for using the TOEFL test in admissions was that students scoring above a specified score were judged adequate from the standpoint of English proficiency, but students who scored below it had to provide other evidence of English proficiency that could justify admission. Only a few institutions used a rigid cut score. The measures most commonly allowed as substitutes for the TOEFL test in admissions and placement decisions were the Scholastic Assessment Test (SAT®) (The College Board, 1995), the American College Testing Assessment (ACT™) (ACT, 1988) score, and the Michigan Test of English Language Proficiency (MTELP) (Buros, 1965). It was found that most changes in standards consisted of increases in the minimum test scores, and that they were initiated in response to problems noted in admitted foreign students' performance. In addition to performance, practices of similar institutions were commonly used to set minimum scores. Participants most often mentioned in revised standard setting were administrators (policy), ESL staff (language trainers), and faculty (teachers of academic content).

This survey provides information about the standards for different institutions and how those standards have been set and maintained. Users had the opportunity to tell us about their use of the TOEFL test. The following suggestions emerged:

• When score user surveys are conducted, relevant institutional characteristics could be collected and used to categorize the results. Such information would facilitate the process of adjusting qualifying scores.

• Data that connects TOEFL scores with speaking and essay scores could be made available. Such data could be useful when essay or speaking measures are used to supplement processing of applicants who achieve marginal scores on TOEFL.

• A system of behaviorally anchored language proficiency scales based on incidents that have stimulated changes in cut scores could be developed. Availability of such scales would facilitate cross‐institutional communication about language proficiency levels and could add a new dimension to test validation.
VALIDATION AND EXTENSION OF THE MATHEMATICAL EXPRESSION RESPONSE TYPE: APPLICATIONS OF SCHEMA THEORY TO AUTOMATIC SCORING AND ITEM GENERATION IN MATHEMATICS
Authors: Singley Mark Kevin , Bennett Randy Elliot
Published: August 2014

In this report, we describe the development of general, accurate, cost‐effective, and immediately usable automatic analysis routines that dichotomously score rational expressions of arbitrary complexity. This development has led to the new Mathematical Expression (ME) response type, which may appear operationally in the new GRE Mathematical Reasoning test scheduled for deployment in 1999. We detail work confirming the accuracy of these scoring routines and then describe extensions to the existing capability based on schema theory. We show how schema theory can be applied not only to the scoring of more complex test responses but also to the automatic generation of items and intelligent tutoring.
MODELING THE PRODUCTION FUNCTION: ASSOCIATIONS AMONG SCHOOL DISTRICT EXPENDITURES, SCHOOL RESOURCES AND STUDENT ACHIEVEMENT
Authors: Wenglinsky Harold
Published: August 2014

This study examines the relationships among school district expenditures, school resources and student achievement for a nationally representative sample of fourth graders. The results of prior studies of these relationships, usually taking a production function approach, have been inconclusive, some finding expenditures and resources to be strongly related to student achievement and others finding no such relationship; because of this, some researchers have proposed alternatives to production function research. The current study, however, seeks to salvage the production function approach through addressing its shortcomings. It applies structural equation modeling to a database that is national in scope and distinguishes between different types of expenditure. It finds that some expenditures and resources, such as instructional spending and teacher‐student ratios, are indeed associated with student achievement, while others are not.
EVALUATION CRITERIA OF TEACHING PRACTICES: A STUDY OF JOB‐RELATEDNESS AND TRAINING NEEDS ASSESSMENT
Authors: Tannenbaum Richard , Rosenfeld Michael
Published: August 2014

In 1996, Danielson proposed 22 criteria that formed a framework for teaching. The framework, according to Danielson, may be used for supervision and evaluation of teaching, teacher reflection and self‐assessment, peer coaching, and mentoring. The framework represents a modification of and extension of the 19 criteria that form the Praxis III: Classroom Performance Assessments. The purpose of this study was to determine the appropriateness of the 22 criteria for use with beginning teachers as well as experienced teachers. Stratified random samples of 3,000 teachers and 1,000 principals were mailed the 22 criteria and asked to make two judgments for each criterion. They were asked to judge the importance of each criterion for the effective classroom performance of all teachers and to judge the importance of each criterion for inclusion in a teacher training and development program. Judgments were made on five‐point rating scales. A total of 1,589 surveys (40%) was returned (1,133 teachers and 383 principals indicated their employment status, 73 respondents did not). The results indicated that, overall, the teachers and principals believed that the criteria were important to a teacher's effective classroom performance and were important for inclusion in a training and development program for teachers. Analyses by experience level confirmed the overall findings. The results of other subgroup analyses are included in the body of the report.
HOW IS RELIABILITY RELATED TO THE QUALITY OF TEST SCORES? WHAT IS THE EFFECT OF LOCAL DEPENDENCE ON RELIABILITY?
Authors: Wainer Howard , Thissen David
Published: August 2014

What are the practical implications of small decreases in reliability coefficients? How does increased item local dependence decrease reliability? How does the new format of more “authentic” reading tests affect reliability?
INQUIRING ABOUT EXAMINEES' ETHNICITY AND SEX: EFFECTS ON COMPUTERIZED PLACEMENT TESTS™ PERFORMANCE
Authors: Stricker Lawrence J. , Ward William C.
Published: August 2014

Laboratory experiments by Steele and Aronson (1995) found that African‐American subjects' performance on difficult verbal items, described as a verbal problem‐solving task, was adversely affected when they were asked about their ethnicity just before working on the items. These results were attributed to stereotype threat: asking about ethnicity primes African‐American subjects' concerns about fulfilling the negative ethnic stereotype about their intellectual ability, thereby disrupting test performance. The present field experiment assessed the effects of asking community college students taking the Computerized Placement Tests™ (CPTs™), in an actual operational setting, about their ethnicity and sex. This inquiry had no statistically and practically significant effects on how well the examinees did on the tests or how long they worked on the tests.
STYLE IN THE ORGANIZATION AND DEFENSE OF COGNITION
Authors: Messick Samuel
Published: August 2014

Cognitive styles bear on the organization and control of attention, thought, feelings, and motives in cognitive processing. Highlighted here are two attentional cognitive styles of sharp‐focus versus broad‐focus scanning and serial scanning for signal detection versus parallel‐process scanning that apprehends incidental information. Defensive styles bear on the organization and control of intrusive affects in cognition; they represent consistent modes of accommodating anxiety and conflict so as to maintain reasonably adaptive cognitive functioning. Because the four major defensive styles associated with obsessive‐compulsive, paranoid, hysterical, and impulsive neuroses are distinguished by differential modes of attentional scanning, the two attentional cognitive styles offer a heuristic means of organizing these defensive styles in behavioral expression.
RELATIONAL DATABASES IN ASSESSMENT: AN APPLICATION TO ONLINE SCORING
Authors: Whalen Sean J. , Bejar Isaac I.
Published: August 2014

The Web has made it possible for interested participants to assemble themselves into virtual educational communities. In this article we discuss the design of an online scoring system that might be used in computer‐based educational contexts. The focus is on the design of the software that supports high psychometric and fairness standards. We first describe the system from the point of view of different participants and then discuss the design of the underlying database, which is at the heart of the system. We close with conclusions regarding the advantages of sound database design and online grading.
CHILD MIGRATION FROM PUERTO RICO TO PUBLIC AND PRIVATE SCHOOLS IN THE UNITED STATES: SAMPLING A DIFFICULT‐TO‐REACH POPULATION
Authors: Laosa Luis M.
Published: August 2014

A study is reported, illustrating the methodological and practical challenges of sampling a difficult‐to‐reach population and of studying a population defined by criteria other than the standard demographic categories. In order to help inform the development of a sampling plan and a research design for a larger scientific investigation of preadolescents who migrate to the United States from Puerto Rico, a study of 286 public and private schools in the United States (New Jersey) was conducted, ascertaining students' migratory movements. Examined are the distribution of arrivals (i.e., transfers‐in from Puerto Rico) among districts and among schools in particular geographic areas, the stability of in‐migration patterns and numbers over time, and the post‐migration mobility. Although focused on this population, the study addresses issues likely to be encountered generally by researchers who study “nonstandard” or difficult‐to‐reach populations. Considered are both cross‐sectional and longitudinal designs.
PSYCHOMETRIC AND COGNITIVE FUNCTIONING OF AN UNDERDETERMINED COMPUTER‐BASED RESPONSE TYPE FOR QUANTITATIVE REASONING
Authors: Bennett Randy Elliot , Morley Mary , Quardt Dennis , Singley Mark K. , Rock Donald A. , Katz Irvin R. , Nhouyvanisvong Adisack
Published: August 2014

This study evaluated the psychometric and cognitive functioning of a new computer‐delivered response type for measuring quantitative reasoning skill. This open‐ended, automatically scorable response type, called “Generating Examples,” presents underdetermined problems that can have many right answers. Two GE tests were randomly spiraled among a group of paid volunteers. The tests differed in the manipulation of specific item features hypothesized to affect difficulty. Both within‐group correlational and between‐group experimental analyses relating to internal consistency reliability, relations with external criteria, features that contribute to item difficulty, adverse impact, and examinee perceptions were performed. Results showed that GE scores were reasonably reliable, but only moderately related to the GRE quantitative section, suggesting the two tests might be tapping somewhat different skills. In the difficulty analyses, two of three item features manipulated had the predicted effect; these features were asking examinees to supply more than one correct answer and to identify whether an item was solvable. Our impact analyses detected no significant gender differences independent of those associated with the General Test. Finally, examinees were evenly divided as to whether they thought GE items provided a fairer indicator of their ability than multiple‐choice items, but as in past studies, they overwhelmingly preferred to take more conventional questions.
THE GENERALIZED GRADED UNFOLDING MODEL: A GENERAL PARAMETRIC ITEM RESPONSE MODEL FOR UNFOLDING GRADED RESPONSES
Authors: Roberts James S. , Donoghue John R. , Laughlin James E.
Published: August 2014

Psychologists have long used binary or graded disagree‐agree responses to measure attitudes. Such data have traditionally been analyzed with cumulative models, but several researchers have recently argued that unfolding models are generally more appropriate. There have been several parametric item response models proposed to unfold disagree‐agree responses. Some of these models allow only for binary responses whereas others permit both binary and graded responses. A new item response model, referred to as the Generalized Graded Unfolding Model (GGUM), is developed in this paper. The GGUM allows for either binary or graded responses and generalizes previous item response models for unfolding in two useful ways. First, it implements a discrimination parameter that varies across items, and thus, items are allowed to discriminate among respondents in different ways. Second, the GGUM allows for distinctively different use of response categories across items. It does this by implementing response category threshold parameters that vary across items. A marginal maximum likelihood algorithm is implemented to estimate GGUM item parameters, whereas person parameters are derived from an expected a posteriori technique. Recovery simulations suggest that accurate item parameter estimates can be obtained with approximately 750 subjects. Additionally, accurate person estimates are derived with approximately 20 6‐category items. The applicability of the GGUM to common attitude testing situations is illustrated with real data on student attitudes toward abortion. Index terms: attitude measurement, unfolding model, item response theory, graded unfolding model, generalized graded unfolding model, Thurstone scale, Likert scale.
RELATIONSHIPS AMONG MULTIPLE‐CHOICE REASONING ITEMS AND A CONSTRUCTED‐RESPONSE GENERATING‐EXPLANATIONS TASK
Authors: Enright Mary K. , Rock Donald A. , Bennett Randy Elliot
Published: August 2014

The potential benefits of computer‐based testing include the ability to present a wider variety of item formats and to tailor tests for specific purposes. In this study we examined the relationships among revised reasoning items that had more varied formats than the traditional items, a constructed‐response generating‐explanations task, and the current GRE General Test. In addition we examined the relationship of these measures to other indicators of achievement. A computer‐based test of the revised reasoning items and the generating‐explanations task was administered to a sample of examinees who previously had taken a paper‐and‐pencil version of the GRE General Test. The revised reasoning item types had acceptable psychometric characteristics and were correlated highly with the current logical reasoning items as well as with items on the GRE verbal measure. The generating‐explanations task was only marginally related to the GRE measures and to the revised reasoning items and was more strongly related to ideational fluency measures. Factor analytic results suggest that if the revised reasoning items were added to the analytical measure, the correlation between the verbal measure and the analytical measure would increase. A better fitting factor model resulted when some of the current and the revised reasoning items were included in the verbal measure. The relationship of the generating‐explanations task and the analytical measure to other indicators of achievement varied for type of achievement and for broad major undergraduate fields of study. The implications of these results for understanding the nature of the skills assessed by different tasks and formats, and for long‐term modifications in the GRE General Test, are discussed.
DESIGN AND EVALUATION OF A COMPUTER‐BASED TOEFL TUTORIAL
Authors: Jamieson Joan , Taylor Carol , Kirsch Irwin , Eignor Daniel
Published: August 2014

In order to train examinees whose native language is not English to take a computerized Test of English as a Foreign Language (TOEFL®), a special set of tutorials was designed and developed. The TOEFL tutorials were field tested as part of a computer familiarity study in 1996. This report describes the development of the tutorials. Also, the experiences of the 1,169 individuals who participated in the computer familiarity study are characterized in terms of timing and performance data, as well as self‐reported attitudes. These analyses took into account computer familiarity and English ability, which both proved to be important in explaining some differences in time to complete the tutorials and perception of the tutorials' usefulness. Most examinees were successful in completing the practice items in the tutorials and thought that the tutorials were helpful. Some changes were subsequently made before operational implementation of the computerized TOEFL test in order to reduce the time needed to complete the TOEFL tutorials.
PRECOLLEGIATE CURRICULA, UNIVERSITY CHARACTERISTICS, AND FIELD PERSISTENCE AMONG SCIENCE MAJORS
Authors: Pennock‐Román María
Published: August 2014

Few studies have attempted to examine simultaneously how student and university characteristics are associated with the persistence of women and men in the sciences. The present study took advantage of a unique data set containing the records of individual students on the SAT (1979‐1982) and their corresponding records on the GRE (1985‐1990), including the questionnaires that accompany each test. Information on the students' undergraduate institutions from the College Board Annual Survey of Colleges was then added to these records. The data set included 48,492 non‐Latino White students with English as their best language who indicated an interest in computer, engineering, mathematical or physical (CEMP) science fields in one or more responses to four questions about their intended major field of study. Two transition points were studied: from intended college major as high school seniors to actual undergraduate major (Stage I), and from actual undergraduate major to intended graduate school major (Stage II). Persistence was defined as staying within the broad category of CEMP sciences.

The descriptive findings showed net losses from the CEMP pool of students at both transition points and greater field switching for women — both into CEMP sciences and out of CEMP sciences. For men at Stage I, 35% of those with a CEMP field as a first choice for major switched to a non‐CEMP field, but new recruits into CEMP undergraduate majors reduced the net loss to only 12%. For women at Stage I, 50% of those with a CEMP field as a first choice for major switched to a non‐CEMP field, but new recruits into CEMP undergraduate majors reduced the net loss to only 8%. For men at Stage II, 19% of CEMP undergraduate majors switched to a non‐CEMP field for graduate study, but recruits into CEMP for intended graduate work reduced the net loss to 15%. For women at Stage II, 31 % of CEMP undergraduate majors switched to a non‐CEMP field for graduate study, but recruits into CEMP fields for intended graduate work reduced the net loss to 24%.

The predictive findings for both transitions, using logistic regression with random coefficients, showed that student and university characteristics (entered jointly) were associated with field persistence for the majority of fields. Significant gender differences in field persistence remained for most, but not all, fields after controlling for student and university characteristics. The higher the percentage of women enrolled in the institution as a whole, the lower the field persistence for both men and women in the majority of fields at Stage I after controlling for other variables.
VALIDITY OF GLOBAL SELF‐RATINGS OF ESL SPEAKING PROFICIENCY BASED ON AN FSI/ILR‐REFERENCED SCALE
Authors: Wilson Kenneth M.
Published: August 2014

This study explored the validity and potential utility for self‐assessment of speaking proficiency in English as a second language (ESL), of a self‐rating scale intended to nominally parallel–in terms of number of points or levels and corresponding descriptions of language‐use behavior–a scale used by members of the Interagency Language Roundtable (ILR) to rate performance in Language Proficiency Interviews (LPI). The study sample was composed primarily of educated, adult ESL users/learners, most of whom were native‐speakers of either French or German, who provided self‐assessments just prior to taking the Test of English for International Communication (TOEIC), in employment‐related ESL training contexts in Switzerland. Correlations between self‐ratings and TOEIC scores centered around .7, and the mean ILR‐referenced self‐rating was very close to that for “predicted ILR‐scaled LPI rating”, where prediction was based on a previously developed equation for predicting LPI rating from TOEIC score. Lines of inquiry warranting further research are suggested.
MONITORING SOURCES OF VARIABILITY WITHIN THE TEST OF SPOKEN ENGLISH ASSESSMENT SYSTEM
Authors: Myford Carol M. , Wolfe Edward W.
Published: August 2014

The purposes of this study were to examine four sources of variability within the Test of Spoken English (TSE®) assessment system, to quantify ranges of variability for each source, to determine the extent to which these sources affect examinee performance, and to highlight aspects of the assessment system that might suggest a need for change. Data obtained from the February and April 1997 TSE scoring sessions were analyzed using Facets (Linacre, 1999a).

The analysis showed that, for each of the two TSE administrations, the test usefully separated examinees into eight statistically distinct proficiency levels. The examinee proficiency measures were found to be trustworthy in terms of their precision and stability. It is important to note, though, that the standard error of measurement varies across the score distribution, particularly in the tails of the distribution.

The items on the TSE appear to work together; ratings on one item correspond well to ratings on the other items. Yet, none of the items seem to function in a redundant fashion. Ratings on individual items within the test can be meaningfully combined; there is little evidence of psychometric multidimensionality in the two data sets. Consequently, it is appropriate to generate a single summary measure to capture the essence of examinee performance across the 12 items. However, the items differ little in terms of difficulty, thus limiting the instrument's ability to discriminate among levels of proficiency.

The TSE rating scale functions as a five‐point scale, and the scale categories are clearly distinguishable. The scale maintains a similar though not identical category structure across all 12 items. Raters differ somewhat in the levels of severity they exercise when they rate examinee performances. The vast majority used the scale in a consistent fashion, though. If examinees' scores were adjusted for differences in rater severity, the scores of two‐thirds of the examinees in these administrations would have differed from their raw score averages by 0.5 to 3.6 raw score points. Such differences can have important consequences for examinees whose scores lie in critical decision‐making regions of the score distribution.
VARIATIONS IN MEAN RESPONSE TIMES FOR QUESTIONS ON THE COMPUTER‐ADAPTIVE GRE® GENERAL TEST: IMPLICATIONS FOR FAIR ASSESSMENT
Authors: Bridgeman Brent , Cline Frederick
Published: August 2014

In a computer adaptive test (CAT), different examinees receive different sets of questions. Questions at the same overall difficulty level and meeting roughly the same content specifications could vary substantially in the amount of time needed to answer them. If the CAT is speeded (that is, if substantial numbers of students either do not finish or must guess randomly at the end to finish), individuals who happened to get a disproportionate number of questions that took a long time to answer could be disadvantaged. The purpose of this study was to determine whether–in computer‐adaptive testing situations–the administration of a disproportionate number of questions with long, expected response times unfairly disadvantages examinees. Data from 5,957 examinees who took the computer‐delivered Graduate Record Examination (GRE®) quantitative measure and 14,745 examinees who took the computer‐delivered GRE analytical measure were used to investigate variation in response time in light of other factors, such as mean test score and the position of the question on the test. For both measures, substantial variation in response times was found, even for items with the same specifications and same difficulty level. But despite these differences, there was no indication that the scores of students who were administered items with long expected response times were disadvantaged.
STRENGTHENING THE TIES THAT BIND: IMPROVING THE LINKING NETWORK IN SPARSELY CONNECTED RATING DESIGNS
Authors: Myford Carol M. , Wolfe Edward W.
Published: August 2014

The purpose of this study was to evaluate the effectiveness of a strategy for linking raters when there are large numbers of raters involved in a scoring session and the overlap among raters is minimal. In sparsely connected rating designs, the number of examinees any given pair of raters has scored in common is very limited. Connections between raters may be weak and tentative at best. The linking strategy we employed involved having all raters in a Test of Spoken English (TSE®) scoring session rate a small set of six benchmark audiotapes, in addition to those examinee tapes that each rater scored as part of his or her normal workload. Using output from Facets analyses of the rating data, we looked at the effects of embedding blocks of ratings from various smaller sets of these benchmark tapes on key indicators of rating quality. We found that all of our benchmark sets were effective for establishing at least the minimal connectivity needed in the rating design in order to allow placement of all raters and all examinees on a single scale. When benchmark sets were used, the highest scoring benchmarks (i.e., those examinees that scored 50s and 60s across the items) produced the highest quality linking (i.e., the most stable linking). The least consistent benchmark sets (i.e., those that were somewhat harder to rate because an examinee's performance varied across items) tended to provide fairly stable links. The most consistent benchmarks (i.e., those that were somewhat easier to rate because an examinee's performance was similar across items) and middle scoring benchmarks (i.e., those from examinees who scored 30s and 40s across the items) tended to provide less stable linking. Low scoring benchmark sets provided the least stable linking. When a single benchmark tape was used, the highest scoring single tape provided higher quality linking than either the least consistent or most consistent benchmark tape.
DEVELOPMENT OF CONSTRUCTED‐RESPONSE ENGINEERING ITEMS USING A RAPID PROTOTYPING TOOL
Authors: Katz Irvin R.
Published: August 2014

This report is a case study of the process of creating experimental items for a new Graduate Record Examination (GRE®) Engineering Test. Using a prototyping tool called the Free‐Response Authoring and Delivery System (FRADS), a single test developer generated 55 computer‐based test items that require either complex constructed responses or multiple‐choice answers. Many of the items underwent significant evolution from their beginning as specifications written by engineering professors. Using FRADS, the test developer crafted new versions of items by exploring alternative response modes (constructed‐response vs. multiple‐choice, numeric input vs. icon movement, or other alternative tool choices) in addition to creating computer‐based versions of the initially specified items. The creation of the constructed‐response items is particularly noteworthy, because without the prototyping software, this project would likely have required programming staff. Instead, one test developer, working alone, was able to generate, evaluate, and modify these complex constructed‐response items. Although presented in the context of a specific exam, the case study is relevant to other testing programs as a model of how new item types can be quickly and inexpensively explored using rapid prototyping tools.
VALIDITY OF GRE® GENERAL TEST SCORES FOR ADMISSION TO COLLEGES OF VETERINARY MEDICINE
Authors: Powers Donald E.
Published: August 2014

This paper documents a case study of the validity of the Graduate Record Examinations (GRE®) General Test in a comprehensive sample of veterinary medical colleges. Extensive and complete data available from these schools allowed us to control several artifacts that are typical of “one‐shot” validation studies and thus enabled a relatively definitive assessment of the test's validity. Overall undergraduate grade point average (GPA), undergraduate GPA in the last 45 hours of courses, and GRE verbal, quantitative, and analytical scores were examined individually and in combination for their ability to predict success in veterinary school. For each of 16 veterinary medical colleges, statistical methods were applied to correct for the effects of range restriction in the predictors and unreliability of the criterion, and the results were summarized across all schools. When corrections were made for both range restriction and unreliability, the resulting validity coefficients (median multiple correlations) were .53 for three GRE scores, when used together; .56 for overall undergraduate GPA; and .72 for GRE scores and overall undergraduate GPA considered together. Adding GRE scores to undergraduate GPA increased the amount of variance explained, on average, by about 18% – a proportion that can be regarded as being “medium” to “large.”
PREPARING FOR THE PRE‐PROFESSIONAL SKILLS TESTS
Authors: Stricker Lawrence J. , Wilder Gita Z.
Published: August 2014

This study investigated the extent and nature of preparation for the Pre‐Professional Skills Tests (PPST®), the reasons for preparing or not preparing, and differences in these results for White and minority‐group test takers and for middle‐class and working‐class test takers. Recent PPST test takers were surveyed. Preparation for the PPST was limited and mainly involved activities that were free or inexpensive, such as taking a sample test. The reported reasons for not preparing and the empirical correlates of measures of preparation were primarily attitudinal. Ethnic‐group and social‐class differences in the extent and nature of test preparation were minimal, but there were some differences in reported reasons and correlates of preparation, primarily less awareness of test preparation resources by White and middle‐class test takers and few correlates of test preparation for Black test takers.
REFINING A SYSTEM FOR COMPUTERIZED ADAPTIVE TESTING POOL CREATION1
Authors: Way Walter D. , Swanson Len , Steffen Manfred , Stocking Martha L.
Published: August 2014

The purpose of this paper is to describe the development and features of a revised computerized adaptive testing (CAT) item pool creation system. This system refines and upgrades the system that is currently in use for the ongoing creation of CAT pools for several large‐scale, high‐stakes testing programs administered by Educational Testing Service (ETS®). Using data from a large‐scale CAT program administered by ETS, an illustrative application of the software is provided to compare pools created using simultaneous partitioning and sequential partitioning. The results of the comparison indicated that simultaneous partitioning was preferable to sequential partitioning in the conditions studied. The paper concludes by identifying and discussing several issues that remain to be investigated in utilizing the revised pool creation system with ETS CAT programs.
A METHOD FOR BUILDING A REALISTIC MODEL OF TEST TAKER BEHAVIOR FOR COMPUTERIZED ADAPTIVE TESTING
Authors: Stocking Martha L. , Steffen Manfred , Eignor Daniel R.
Published: August 2014

Statistical models of test taker behavior underlie many modern psychometric methodologies. An advantage of this approach is that it supports studies where simulated test taker responses are generated according to the particular statistical model in order to explore model properties. To the extent that such models fail to capture important aspects of real test takers' responses, the predictive utility of such simulation studies may be compromised. In this paper a methodology is explored that builds simulated test taker responses using information distilled from the responses of real test takers. This approach proved effective in capturing some aspects of test taker behavior that simpler models failed to incorporate.
ON THE PAST AND FUTURE OF NULL HYPOTHESIS SIGNIFICANCE TESTING1
Authors: Robinson Daniel H. , Wainer Howard
Published: August 2014

Criticisms of null hypothesis significance testing (NHST) have appeared recently in wildlife research journals (Anderson, Burnham, & Thompson, 2000; Anderson, Link, Johnson, & Burnham, 2001; Cherry, 1998; Guthery, Lusk, & Peterson, 2001; Johnson, 1999). In this essay we discuss these criticisms with regard to both current usage of NHST and plausible future use. We suggest that the historical usage of such procedures was not unreasonable and hence that current users might spend time profitably reading some of Fisher's applied work. However, we also believe that modifications to NHST and to the interpretations of its outcomes might better suit the needs of modern science. Our primary conclusion is that NHST is most often useful as an adjunct to other results (e.g., effect sizes) rather than as a stand‐alone result. We cite some examples, however, where NHST can be profitably used alone. Last, we find considerable experimental support for a less slavish attitude toward the precise value of the probability yielded from such procedures.
AN APPLICATION OF A BAYESIAN HIERARCHICAL MODEL FOR ITEM FAMILY CALIBRATION
Authors: Sinharay Sandip , Johnson Matthew S. , Williamson David M.
Published: August 2014

Item families, which are groups of items related to each other in some way, are increasingly used in complex educational assessments. For example, in automatic item generation (AIG) systems, a test may consist of multiple items generated from a number of item models. Item calibration or scoring for such an assessment requires fitting models that can take into account the dependence structure inherent among the items that belong to the same item family. Glas and van der Linden (2001) suggest a Bayesian hierarchical model to analyze data involving item families. We fit that hierarchical model using the Markov chain Monte Carlo (MCMC) algorithm. Formulating the MCMC algorithm provides little additional difficulty (compared to fitting a simple item response theory model) even with the additional complexity in the form of hierarchy in the model. We show that the model can take into account the dependence structure inherent among the items and hence is an improvement over the models currently used in similar situations. We introduce the notion of the family expected response function (FERF) as a way to summarize the probability of a correct response to an item randomly generated from an item family and suggest a way to estimate the FERFs. Our work is a step towards creating a tool that can save significant amount of resources for tests with item families, because calibrating only the item families might be enough rather than calibrating each item in the families separately.
IMPROVING THE STATISTICAL ASPECTS OF E‐RATER®: EXPLORING ALTERNATIVE FEATURE REDUCTION AND COMBINATION RULES
Authors: Feng Xin , Dorans Neil J. , Patsula Liane N. , Kaplan Bruce
Published: August 2014

This study explores alternative ways of reducing the number of variables/features and additional ways of combining information across features to produce more stable and accurate e‐rater® scores. Following an explanation of the statistical aspects of the process is a description of alternatives to the process. Our explorations resulted in certain conclusions and directions for future research. We have examined enough e‐rater data to conclude that stepwise regression seems to be effective as a feature reduction procedure. However, this may be attributed to the consistently strong relationship with essay score that is observed for the content vector analysis (CVA) variables and the two variables used to approximate word length (number of auxiliary verbs and the ratio of the number of auxiliary verbs to the number of words). To yield better validation results, we also suggest that the hold‐out method for evaluating validity should replace the current two‐stage approach of first developing a model in a quasi‐uniform training sample and then validating these results in a target cross‐validation sample. More research is needed in several areas. First, explicit modeling of the part of essay scores that is unrelated to word length is warranted. The POM (Proportional Odds Model) approach should be investigated in greater depth. Also needed is a statistical justification for using essay scores to score CVA variables. Algorithmic approaches to prediction/classification problem, such as boosting, may prove fruitful. Further investigation of quantile regression and ridge regression should be conducted.