bluuebunny/crossref_metadata_2025_split huggingface dataset:
EVALUATION CRITERIA OF TEACHING PRACTICES: A STUDY OF JOB‐RELATEDNESS AND TRAINING NEEDS ASSESSMENT
Authors: Tannenbaum Richard , Rosenfeld Michael 
Published: August 2014

ABSTRACTIn 1996, Danielson proposed 22 criteria that formed a framework for teaching. The framework, according to Danielson, may be used for supervision and evaluation of teaching, teacher reflection and self‐assessment, peer coaching, and mentoring. The framework represents a modification of and extension of the 19 criteria that form the Praxis III: Classroom Performance Assessments. The purpose of this study was to determine the appropriateness of the 22 criteria for use with beginning teachers as well as experienced teachers. Stratified random samples of 3,000 teachers and 1,000 principals were mailed the 22 criteria and asked to make two judgments for each criterion. They were asked to judge the importance of each criterion for the effective classroom performance of all teachers and to judge the importance of each criterion for inclusion in a teacher training and development program. Judgments were made on five‐point rating scales. A total of 1,589 surveys (40%) was returned (1,133 teachers and 383 principals indicated their employment status, 73 respondents did not). The results indicated that, overall, the teachers and principals believed that the criteria were important to a teacher's effective classroom performance and were important for inclusion in a training and development program for teachers. Analyses by experience level confirmed the overall findings. The results of other subgroup analyses are included in the body of the report.
INQUIRING ABOUT EXAMINEES' ETHNICITY AND SEX: EFFECTS ON COMPUTERIZED PLACEMENT TESTS™ PERFORMANCE
Authors: Stricker Lawrence J. , Ward William C. 
Published: August 2014

ABSTRACTLaboratory experiments by Steele and Aronson (1995) found that African‐American subjects' performance on difficult verbal items, described as a verbal problem‐solving task, was adversely affected when they were asked about their ethnicity just before working on the items. These results were attributed to stereotype threat: asking about ethnicity primes African‐American subjects' concerns about fulfilling the negative ethnic stereotype about their intellectual ability, thereby disrupting test performance. The present field experiment assessed the effects of asking community college students taking the Computerized Placement Tests™ (CPTs™), in an actual operational setting, about their ethnicity and sex. This inquiry had no statistically and practically significant effects on how well the examinees did on the tests or how long they worked on the tests.
STYLE IN THE ORGANIZATION AND DEFENSE OF COGNITION
Authors: Messick Samuel 
Published: August 2014

ABSTRACTCognitive styles bear on the organization and control of attention, thought, feelings, and motives in cognitive processing. Highlighted here are two attentional cognitive styles of sharp‐focus versus broad‐focus scanning and serial scanning for signal detection versus parallel‐process scanning that apprehends incidental information. Defensive styles bear on the organization and control of intrusive affects in cognition; they represent consistent modes of accommodating anxiety and conflict so as to maintain reasonably adaptive cognitive functioning. Because the four major defensive styles associated with obsessive‐compulsive, paranoid, hysterical, and impulsive neuroses are distinguished by differential modes of attentional scanning, the two attentional cognitive styles offer a heuristic means of organizing these defensive styles in behavioral expression.
RELATIONAL DATABASES IN ASSESSMENT: AN APPLICATION TO ONLINE SCORING
Authors: Whalen Sean J. , Bejar Isaac I. 
Published: August 2014

ABSTRACTThe Web has made it possible for interested participants to assemble themselves into virtual educational communities. In this article we discuss the design of an online scoring system that might be used in computer‐based educational contexts. The focus is on the design of the software that supports high psychometric and fairness standards. We first describe the system from the point of view of different participants and then discuss the design of the underlying database, which is at the heart of the system. We close with conclusions regarding the advantages of sound database design and online grading.
CHILD MIGRATION FROM PUERTO RICO TO PUBLIC AND PRIVATE SCHOOLS IN THE UNITED STATES: SAMPLING A DIFFICULT‐TO‐REACH POPULATION
Authors: Laosa Luis M. 
Published: August 2014

ABSTRACTA study is reported, illustrating the methodological and practical challenges of sampling a difficult‐to‐reach population and of studying a population defined by criteria other than the standard demographic categories. In order to help inform the development of a sampling plan and a research design for a larger scientific investigation of preadolescents who migrate to the United States from Puerto Rico, a study of 286 public and private schools in the United States (New Jersey) was conducted, ascertaining students' migratory movements. Examined are the distribution of arrivals (i.e., transfers‐in from Puerto Rico) among districts and among schools in particular geographic areas, the stability of in‐migration patterns and numbers over time, and the post‐migration mobility. Although focused on this population, the study addresses issues likely to be encountered generally by researchers who study “nonstandard” or difficult‐to‐reach populations. Considered are both cross‐sectional and longitudinal designs.
PSYCHOMETRIC AND COGNITIVE FUNCTIONING OF AN UNDERDETERMINED COMPUTER‐BASED RESPONSE TYPE FOR QUANTITATIVE REASONING
Authors: Bennett Randy Elliot , Morley Mary , Quardt Dennis , Singley Mark K. , Rock Donald A. , Katz Irvin R. , Nhouyvanisvong Adisack 
Published: August 2014

ABSTRACTThis study evaluated the psychometric and cognitive functioning of a new computer‐delivered response type for measuring quantitative reasoning skill. This open‐ended, automatically scorable response type, called “Generating Examples,” presents underdetermined problems that can have many right answers. Two GE tests were randomly spiraled among a group of paid volunteers. The tests differed in the manipulation of specific item features hypothesized to affect difficulty. Both within‐group correlational and between‐group experimental analyses relating to internal consistency reliability, relations with external criteria, features that contribute to item difficulty, adverse impact, and examinee perceptions were performed. Results showed that GE scores were reasonably reliable, but only moderately related to the GRE quantitative section, suggesting the two tests might be tapping somewhat different skills. In the difficulty analyses, two of three item features manipulated had the predicted effect; these features were asking examinees to supply more than one correct answer and to identify whether an item was solvable. Our impact analyses detected no significant gender differences independent of those associated with the General Test. Finally, examinees were evenly divided as to whether they thought GE items provided a fairer indicator of their ability than multiple‐choice items, but as in past studies, they overwhelmingly preferred to take more conventional questions.
THE GENERALIZED GRADED UNFOLDING MODEL: A GENERAL PARAMETRIC ITEM RESPONSE MODEL FOR UNFOLDING GRADED RESPONSES
Authors: Roberts James S. , Donoghue John R. , Laughlin James E. 
Published: August 2014

ABSTRACTPsychologists have long used binary or graded disagree‐agree responses to measure attitudes. Such data have traditionally been analyzed with cumulative models, but several researchers have recently argued that unfolding models are generally more appropriate. There have been several parametric item response models proposed to unfold disagree‐agree responses. Some of these models allow only for binary responses whereas others permit both binary and graded responses. A new item response model, referred to as the Generalized Graded Unfolding Model (GGUM), is developed in this paper. The GGUM allows for either binary or graded responses and generalizes previous item response models for unfolding in two useful ways. First, it implements a discrimination parameter that varies across items, and thus, items are allowed to discriminate among respondents in different ways. Second, the GGUM allows for distinctively different use of response categories across items. It does this by implementing response category threshold parameters that vary across items. A marginal maximum likelihood algorithm is implemented to estimate GGUM item parameters, whereas person parameters are derived from an expected a posteriori technique. Recovery simulations suggest that accurate item parameter estimates can be obtained with approximately 750 subjects. Additionally, accurate person estimates are derived with approximately 20 6‐category items. The applicability of the GGUM to common attitude testing situations is illustrated with real data on student attitudes toward abortion. Index terms: attitude measurement, unfolding model, item response theory, graded unfolding model, generalized graded unfolding model, Thurstone scale, Likert scale.
RELATIONSHIPS AMONG MULTIPLE‐CHOICE REASONING ITEMS AND A CONSTRUCTED‐RESPONSE GENERATING‐EXPLANATIONS TASK
Authors: Enright Mary K. , Rock Donald A. , Bennett Randy Elliot 
Published: August 2014

ABSTRACTThe potential benefits of computer‐based testing include the ability to present a wider variety of item formats and to tailor tests for specific purposes. In this study we examined the relationships among revised reasoning items that had more varied formats than the traditional items, a constructed‐response generating‐explanations task, and the current GRE General Test. In addition we examined the relationship of these measures to other indicators of achievement. A computer‐based test of the revised reasoning items and the generating‐explanations task was administered to a sample of examinees who previously had taken a paper‐and‐pencil version of the GRE General Test. The revised reasoning item types had acceptable psychometric characteristics and were correlated highly with the current logical reasoning items as well as with items on the GRE verbal measure. The generating‐explanations task was only marginally related to the GRE measures and to the revised reasoning items and was more strongly related to ideational fluency measures. Factor analytic results suggest that if the revised reasoning items were added to the analytical measure, the correlation between the verbal measure and the analytical measure would increase. A better fitting factor model resulted when some of the current and the revised reasoning items were included in the verbal measure. The relationship of the generating‐explanations task and the analytical measure to other indicators of achievement varied for type of achievement and for broad major undergraduate fields of study. The implications of these results for understanding the nature of the skills assessed by different tasks and formats, and for long‐term modifications in the GRE General Test, are discussed.
DESIGN AND EVALUATION OF A COMPUTER‐BASED TOEFL TUTORIAL
Authors: Jamieson Joan , Taylor Carol , Kirsch Irwin , Eignor Daniel 
Published: August 2014

ABSTRACTIn order to train examinees whose native language is not English to take a computerized Test of English as a Foreign Language (TOEFL®), a special set of tutorials was designed and developed. The TOEFL tutorials were field tested as part of a computer familiarity study in 1996. This report describes the development of the tutorials. Also, the experiences of the 1,169 individuals who participated in the computer familiarity study are characterized in terms of timing and performance data, as well as self‐reported attitudes. These analyses took into account computer familiarity and English ability, which both proved to be important in explaining some differences in time to complete the tutorials and perception of the tutorials' usefulness. Most examinees were successful in completing the practice items in the tutorials and thought that the tutorials were helpful. Some changes were subsequently made before operational implementation of the computerized TOEFL test in order to reduce the time needed to complete the TOEFL tutorials.
PRECOLLEGIATE CURRICULA, UNIVERSITY CHARACTERISTICS, AND FIELD PERSISTENCE AMONG SCIENCE MAJORS
Authors: Pennock‐Román María 
Published: August 2014

ABSTRACTFew studies have attempted to examine simultaneously how student and university characteristics are associated with the persistence of women and men in the sciences. The present study took advantage of a unique data set containing the records of individual students on the SAT (1979‐1982) and their corresponding records on the GRE (1985‐1990), including the questionnaires that accompany each test. Information on the students' undergraduate institutions from the College Board Annual Survey of Colleges was then added to these records. The data set included 48,492 non‐Latino White students with English as their best language who indicated an interest in computer, engineering, mathematical or physical (CEMP) science fields in one or more responses to four questions about their intended major field of study. Two transition points were studied: from intended college major as high school seniors to actual undergraduate major (Stage I), and from actual undergraduate major to intended graduate school major (Stage II). Persistence was defined as staying within the broad category of CEMP sciences.The descriptive findings showed net losses from the CEMP pool of students at both transition points and greater field switching for women — both into CEMP sciences and out of CEMP sciences. For men at Stage I, 35% of those with a CEMP field as a first choice for major switched to a non‐CEMP field, but new recruits into CEMP undergraduate majors reduced the net loss to only 12%. For women at Stage I, 50% of those with a CEMP field as a first choice for major switched to a non‐CEMP field, but new recruits into CEMP undergraduate majors reduced the net loss to only 8%. For men at Stage II, 19% of CEMP undergraduate majors switched to a non‐CEMP field for graduate study, but recruits into CEMP for intended graduate work reduced the net loss to 15%. For women at Stage II, 31 % of CEMP undergraduate majors switched to a non‐CEMP field for graduate study, but recruits into CEMP fields for intended graduate work reduced the net loss to 24%.The predictive findings for both transitions, using logistic regression with random coefficients, showed that student and university characteristics (entered jointly) were associated with field persistence for the majority of fields. Significant gender differences in field persistence remained for most, but not all, fields after controlling for student and university characteristics. The higher the percentage of women enrolled in the institution as a whole, the lower the field persistence for both men and women in the majority of fields at Stage I after controlling for other variables.
VALIDITY OF GLOBAL SELF‐RATINGS OF ESL SPEAKING PROFICIENCY BASED ON AN FSI/ILR‐REFERENCED SCALE
Authors: Wilson Kenneth M. 
Published: August 2014

ABSTRACTThis study explored the validity and potential utility for self‐assessment of speaking proficiency in English as a second language (ESL), of a self‐rating scale intended to nominally parallel–in terms of number of points or levels and corresponding descriptions of language‐use behavior–a scale used by members of the Interagency Language Roundtable (ILR) to rate performance in Language Proficiency Interviews (LPI). The study sample was composed primarily of educated, adult ESL users/learners, most of whom were native‐speakers of either French or German, who provided self‐assessments just prior to taking the Test of English for International Communication (TOEIC), in employment‐related ESL training contexts in Switzerland. Correlations between self‐ratings and TOEIC scores centered around .7, and the mean ILR‐referenced self‐rating was very close to that for “predicted ILR‐scaled LPI rating”, where prediction was based on a previously developed equation for predicting LPI rating from TOEIC score. Lines of inquiry warranting further research are suggested.
MONITORING SOURCES OF VARIABILITY WITHIN THE TEST OF SPOKEN ENGLISH ASSESSMENT SYSTEM
Authors: Myford Carol M. , Wolfe Edward W. 
Published: August 2014

ABSTRACTThe purposes of this study were to examine four sources of variability within the Test of Spoken English (TSE®) assessment system, to quantify ranges of variability for each source, to determine the extent to which these sources affect examinee performance, and to highlight aspects of the assessment system that might suggest a need for change. Data obtained from the February and April 1997 TSE scoring sessions were analyzed using Facets (Linacre, 1999a).The analysis showed that, for each of the two TSE administrations, the test usefully separated examinees into eight statistically distinct proficiency levels. The examinee proficiency measures were found to be trustworthy in terms of their precision and stability. It is important to note, though, that the standard error of measurement varies across the score distribution, particularly in the tails of the distribution.The items on the TSE appear to work together; ratings on one item correspond well to ratings on the other items. Yet, none of the items seem to function in a redundant fashion. Ratings on individual items within the test can be meaningfully combined; there is little evidence of psychometric multidimensionality in the two data sets. Consequently, it is appropriate to generate a single summary measure to capture the essence of examinee performance across the 12 items. However, the items differ little in terms of difficulty, thus limiting the instrument's ability to discriminate among levels of proficiency.The TSE rating scale functions as a five‐point scale, and the scale categories are clearly distinguishable. The scale maintains a similar though not identical category structure across all 12 items. Raters differ somewhat in the levels of severity they exercise when they rate examinee performances. The vast majority used the scale in a consistent fashion, though. If examinees' scores were adjusted for differences in rater severity, the scores of two‐thirds of the examinees in these administrations would have differed from their raw score averages by 0.5 to 3.6 raw score points. Such differences can have important consequences for examinees whose scores lie in critical decision‐making regions of the score distribution.
VARIATIONS IN MEAN RESPONSE TIMES FOR QUESTIONS ON THE COMPUTER‐ADAPTIVE GRE® GENERAL TEST: IMPLICATIONS FOR FAIR ASSESSMENT
Authors: Bridgeman Brent , Cline Frederick 
Published: August 2014

ABSTRACTIn a computer adaptive test (CAT), different examinees receive different sets of questions. Questions at the same overall difficulty level and meeting roughly the same content specifications could vary substantially in the amount of time needed to answer them. If the CAT is speeded (that is, if substantial numbers of students either do not finish or must guess randomly at the end to finish), individuals who happened to get a disproportionate number of questions that took a long time to answer could be disadvantaged. The purpose of this study was to determine whether–in computer‐adaptive testing situations–the administration of a disproportionate number of questions with long, expected response times unfairly disadvantages examinees. Data from 5,957 examinees who took the computer‐delivered Graduate Record Examination (GRE®) quantitative measure and 14,745 examinees who took the computer‐delivered GRE analytical measure were used to investigate variation in response time in light of other factors, such as mean test score and the position of the question on the test. For both measures, substantial variation in response times was found, even for items with the same specifications and same difficulty level. But despite these differences, there was no indication that the scores of students who were administered items with long expected response times were disadvantaged.
STRENGTHENING THE TIES THAT BIND: IMPROVING THE LINKING NETWORK IN SPARSELY CONNECTED RATING DESIGNS
Authors: Myford Carol M. , Wolfe Edward W. 
Published: August 2014

ABSTRACTThe purpose of this study was to evaluate the effectiveness of a strategy for linking raters when there are large numbers of raters involved in a scoring session and the overlap among raters is minimal. In sparsely connected rating designs, the number of examinees any given pair of raters has scored in common is very limited. Connections between raters may be weak and tentative at best. The linking strategy we employed involved having all raters in a Test of Spoken English (TSE®) scoring session rate a small set of six benchmark audiotapes, in addition to those examinee tapes that each rater scored as part of his or her normal workload. Using output from Facets analyses of the rating data, we looked at the effects of embedding blocks of ratings from various smaller sets of these benchmark tapes on key indicators of rating quality. We found that all of our benchmark sets were effective for establishing at least the minimal connectivity needed in the rating design in order to allow placement of all raters and all examinees on a single scale. When benchmark sets were used, the highest scoring benchmarks (i.e., those examinees that scored 50s and 60s across the items) produced the highest quality linking (i.e., the most stable linking). The least consistent benchmark sets (i.e., those that were somewhat harder to rate because an examinee's performance varied across items) tended to provide fairly stable links. The most consistent benchmarks (i.e., those that were somewhat easier to rate because an examinee's performance was similar across items) and middle scoring benchmarks (i.e., those from examinees who scored 30s and 40s across the items) tended to provide less stable linking. Low scoring benchmark sets provided the least stable linking. When a single benchmark tape was used, the highest scoring single tape provided higher quality linking than either the least consistent or most consistent benchmark tape.
DEVELOPMENT OF CONSTRUCTED‐RESPONSE ENGINEERING ITEMS USING A RAPID PROTOTYPING TOOL
Authors: Katz Irvin R. 
Published: August 2014

ABSTRACTThis report is a case study of the process of creating experimental items for a new Graduate Record Examination (GRE®) Engineering Test. Using a prototyping tool called the Free‐Response Authoring and Delivery System (FRADS), a single test developer generated 55 computer‐based test items that require either complex constructed responses or multiple‐choice answers. Many of the items underwent significant evolution from their beginning as specifications written by engineering professors. Using FRADS, the test developer crafted new versions of items by exploring alternative response modes (constructed‐response vs. multiple‐choice, numeric input vs. icon movement, or other alternative tool choices) in addition to creating computer‐based versions of the initially specified items. The creation of the constructed‐response items is particularly noteworthy, because without the prototyping software, this project would likely have required programming staff. Instead, one test developer, working alone, was able to generate, evaluate, and modify these complex constructed‐response items. Although presented in the context of a specific exam, the case study is relevant to other testing programs as a model of how new item types can be quickly and inexpensively explored using rapid prototyping tools.