bluuebunny/crossref_metadata_2025 huggingface dataset:
A KNOWLEDGE BASE FOR TEACHER LICENSURE: A COMPARISON OF STATE REQUIREMENTS, PROFESSIONAL ASSOCIATION RECOMMENDATIONS AND TEACHER, TEACHER EDUCATOR, AND ADMINISTRATOR OPINIONS OF TEACHER PREPARATION ON THE PRINCIPLES OF LEARNING AND TEACHING
Authors: Wesley Scott , Klem Lynn , Reynolds Anne
Published: August 2014

A notable aspect of the educational reform movement is the increasing attention placed on procedures and assessments for licensing teachers. Currently, Educational Testing Service (ETS) is engaged in a large‐scale development project related to teacher licensure. This project will develop a new generation of assessments for licensing beginning teachers. At the foundation of these assessments are two research projects. One of these, Project C H A R T, is an effort to identify state requirements for licensure in all subject areas currently being taught in elementary, middle, and secondary schools. This project is also identifying recommended standards established by professional organizations and associations concerning the skills and competencies on which teacher preparation programs should focus. The second project is a series of studies aimed at identifying what teacher candidates and new teachers should know and be able to do. The methodology being used in this research is job analysis. The purpose of these job analyses is to identify, through surveys of practicing professionals, the important knowledge areas and skills for new teachers.

With these two projects, the question arises as to whether there is agreement among states, professional associations, and practicing professionals about what new teachers should know and be able to do. This paper addresses this question by comparing the results of Project C H A R T with the results of the job analysis for one area, General Principles of Teaching and Learning. General Principles of Teaching and Learning is a subject assessment that focuses on pedagogical principles that cross subject‐matter and grade boundaries.

The results of this comparison indicate that there is considerable agreement among states, professional associations, and the members of the educational community. There are, however, several areas of disagreement between and within these groups. The paper discusses the areas of agreement and disagreement as well as their implications for teacher preparation and regulation and the development of a knowledge base for teacher education.
EVALUATING HYPOTHESES ABOUT DIFFERENTIAL ITEM FUNCTIONING1,2
Authors: Schmitt Alicia P. , Holland Paul W. , Dorans Neil J.
Published: August 2014

Studies evaluating hypotheses about sources of differential item functioning (DIF) are classified into two categories: observational studies evaluating operational items and randomized DIF studies evaluating specially constructed items. For observational studies, advice is given for item classification, sample selection, the matching criterion, and the choice of DIF techniques, as well as how to summarize, synthesize and translate DIF data into DIF hypotheses. In randomized DIF studies of specially constructed items, specific hypotheses, often generated from observational studies, are evaluated under rigorous conditions. Advice for these studies focuses on the importance of carefully constructed items to assess DIF hypotheses. In addition, randomized DDF studies are cast within a causal inference framework, which provides a justification for the use of standardization analyses or logistic regression analysis to estimate effect sizes. Two studies that have components spanning the observational and controlled domains are summarized for illustrative purposes. Standardization analyses are used for both studies. Special logistic regression analyses of an item from one of these studies are provided to illustrate a new approach in the assessment of DIF hypotheses using specially constructed items.
MEASUREMENT PROBLEMS
Authors: Wainer Howard
Published: August 2014

History teaches the continuity of science; the developments of tomorrow have their genesis in the problems of today. Thus any attempt to look forward is well begun with an examination of unsettled questions. Since a clearer idea of where we are going smooths the path into the unknown future, a periodic review of such questions is prudent. The present day, lying near the juncture of the centuries, is well suited for such a review. This paper reports twenty unsolved problems in educational measurement and points toward what seem to be promising avenues of solution.
PROBABILITY‐BASED INFERENCE IN A DOMAIN OF PROPORTIONAL REASONING TASKS
Authors: Béland Anne , Mislevy Robert J.
Published: August 2014

Educators and psychologists are increasingly interested in modelling the processes and knowledge structures by which people learn and solve problems. Progress has been made in developing cognitive models in several domains, and in devising observational settings that provide clues about subjects' cognition from this perspective. Less attention has been paid to procedures for inference or decision‐making with such information, given that it provides only imperfect information about cognition—in short, test theory for cognitive assessment. This paper describes probability‐based inference in this context, and illustrates its application with an example concerning proportional reasoning.
GIVING A CHOICE OF TOPICS ON A TEST OF BASIC WRITING SKILLS: DOES IT MAKE ANY DIFFERENCE?
Authors: Powers Donald E. , Fowles Mary E. , Farnum Marisa , Gerritz Kalle
Published: August 2014

As part of the development of the academic skills portion of The Praxis Series: Professional Assessments for Beginning Teachers™ (the successor to the NTE examinations), a sample of undergraduate students each wrote at least two 50‐minute essays. For each essay, the topic was selected by the examinee from a pair of topics. Before writing, all students indicated their preferences for each of 20 possible topics from which their topics would be drawn. Subsequently, comparisons were made between performance on examinees' high and low preferred topics. The relationship of selected variables (e.g., undergraduate grades and admissions test scores) to performance on each kind of topic was also compared.

Student preferences for different topics varied considerably across topics, and topics that were most preferred by some examinees were often least preferred by others. Preferences, however, exhibited little if any relationship to essay scores. Finally, scores based on high and low preferred topics had similar patterns of correlations with other variables.
HOW TO EQUATE TESTS WITH LITTLE OR NO DATA
Authors: Mislevy Robert J. , Sheehan Kathleen M. , Wingersky Marilyn
Published: August 2014

Standard procedures for equating tests, including those based on item response theory (IRT), require item responses from large numbers of examinees. Such data may not be forthcoming for reasons theoretical, political, or practical. Information about items' operating characteristics may be available from other sources, however, such as content and format specifications, expert opinion, or psychological theories about the skills and strategies required to solve them. This paper shows how, in the IRT framework, collateral information about items can be exploited to augment or even replace examinee responses when linking or equating new tests to established scales. The procedures are illustrated with data from the Pre‐Professional Skills Test (PPST).
ASSESSING THE PROFESSIONAL KNOWLEDGE OF PROSPECTIVE TEACHERS: IDENTIFYING THE QUESTIONS THAT WORK BEST
Authors: Powers Donald E. , Pitcher Barbara
Published: August 2014

An item‐level analysis was undertaken for one national administration of the Professional Knowledge test of the NTE Core Battery. The performances of several relevant subgroups of examinees defined by undergraduate major field, were compared for individual test items. Analyses revealed that a subset of test items was differentially easier for particular subgroups of examinees who had received greater exposure to teacher preparation. A smaller subset of items favored examinees having little if any such exposure. The nature of each kind of item is discussed with respect to performance expectations for various subgroups of examinees.
THE INTERPLAY OF EVIDENCE AND CONSEQUENCES IN THE VALIDATION OF PERFORMANCE ASSESSMENTS
Authors: Messick Samuel
Published: August 2014

Authentic and direct assessments of performances and products are conceptualized in terms of multiple distinctions having implications for validation. These include contrasts between performances and products, between assessment of performance per se and performance assessment of competence or other constructs, between structured and unstructured problems and response modes, between decomposed task skills and complex task performance, and between contextualized and decontextualized knowledge and skill. The concepts of “authenticity” and “directness” of assessment are analyzed as promissory validity claims that they offset, respectively, the two major threats to construct validity, namely, construct underrepresentation and construct‐irrelevant variance. These distinctions are examined in the context of an overarching contrast between task‐driven and construct‐driven performance assessment. With respect to validation, the salient role of both positive and negative consequences is underscored as well as the need for evidence bearing on the various aspects of construct validity (content, substantive, structural, external, generalizability, and consequential).
PROFILES OF PERFORMANCE BY EDUCATIONAL LEVEL ON THE NTE CORE BATTERY
Authors: Powers Donald E.
Published: August 2014

Several comparisons were made across educational level with respect to performance on each of the three NTE Core Battery tests‐Communication Skills, General Knowledge, and Professional Knowledge. Various controls were implemented in order to discount selected threats to the interpretation of between‐level test score differences as a function of teacher education. Several alternative analyses yielded results that were generally consistent with an interpretation of the Professional Knowledge test as a measure of students' knowledge of the teaching profession. A major limitation of the data on which the results are based was the lack of any specific information about the nature and timing of relevant coursework.
IDENTIFYING DIFFERENTIALLY FUNCTIONING ITEMS IN THE NTE CORE BATTERY
Authors: McPeek W. Miles , Wild Cheryl L.
Published: August 2014

The purpose of this study is to evaluate the use of the Mantel‐Haenszel statistic as a methodology for identifying differentially functioning items on the NTE Programs Core Battery tests. Retrospective analyses of data collected over a three‐year period are reported for Black/White, Hispanic/White, and female/male comparisons. The three tests that comprise the Core Battery are included in the analyses—Communication Skills, General Knowledge, and Professional Knowledge. Data concerning the replicability of the results, the number of items identified as differentially difficult, and the characteristics of the test questions associated with differential difficulty are discussed. Replicability of DIF values from two administrations based on correlational data was modest to high, with greatest replicability found in the largest samples. Although the replicability is modest to high when considering all the items, the results are less consistent when only the items identified as showing differential difficulty are considered. Generally, 6 to 9 percent of the items were identified as being differentially difficult per comparison, with a greater number of items flagged in the female/male analyses than in the Black/White or Hispanic/White analyses. There was little overlap among the items flagged in the three different analyses. Although analyses suggested some content characteristics that may be related to differential item functioning, the findings should be considered exploratory due to the small number of items studied and the post hoc nature of the sample. Also, the findings should not be assumed to apply to populations other than NTE test takers.
AN INITIAL EVALUATION OF THE USE OF BIVARIATE MATCHING IN DIF ANALYSES FOR FORMULA SCORED TESTS12
Authors: Pomplun Mark , Baron Patricia , McHale Fred
Published: August 2014

This study evaluated the use of bivariate matching as a solution to the problem of studying DIF with formula scored tests. This problem specifically involves including in or excluding from a formula scored matching criterion formula scored items to‐be‐studied for a DIF analysis. Using SAT Verbal data with large and small samples both Male‐Female and Black‐White group comparisons were investigated. MH D‐DIF values and DIF category classifications based on bivariate matching were compared with MH D‐DIF values and categories based on rights scored and formula scored matching criteria. When large samples were used, MH D‐DIF values based on the bivariate matching criterion were ordered very similarly to MH D‐DIF values based on the other criteria. The DIF category classifications were almost identical. However, with small samples the MH D‐DIF values based on the bivariate matching criterion displayed only moderate correlations with MH D‐DIF values from the other criteria. In addition, the DIF category classifications based on the bivariate matching criterion showed fewer high DIF items than those based on the rights or formula scored matching criteria. As a secondary result, this study documented the differences between formula and rights scored criteria in DIF analyses of formula scored tests. These results showed that the substitution of a rights scored criterion in a DIF analysis of a formula scored test resulted in MH D‐DIF values that were ordered very similarly as those based on the proper formula scored criterion. The MH D‐DIF values based on the rights score criterion were, however, different in magnitude from those based on the formula scored criterion. The differences were related to item difficulty and were greater for comparisons with larger ability differences.
CHARACTERISTICS ASSOCIATED WITH DIFFERENTIAL ITEM FUNCTIONING ON THE SCHOLASTIC APTITUDE TEST: GENDER AND MAJORITY/MINORITY GROUP COMPARISONS
Authors: Carlton Sydell T. , Harris Abigail M.
Published: August 2014

The purpose of the study was to investigate whether selected test and item characteristics in the SAT are associated with unexpected differential item functioning (DIF) for males and females and for majority and minority group members (i.e., White performance compared with Black, Asian‐American, Hispanic, and American Indian performance). Six forms of the SAT that were administered relatively recently were used. Each of the six forms consists of Verbal sections (containing Analogies, Antonyms, Sentence Completions, and Reading Comprehension), Mathematics sections (containing Regular Mathematics problem‐solving and Quantitative Comparisons), and the Test of Standard Written English (TSWE) (containing Usage and Sentence Correction). Findings from previous studies, test specifications, and suggestions offered by test development experts led to the identification of more than one hundred a priori item coding categories, and each SAT item was coded accordingly. Items were coded with regard to type, content, and format.

The Mantel‐Haenszel procedure was used to provide an index of differential item functioning (DIF) for each reference/focal group comparison. Females, Asian‐Americans, Blacks, Hispanics, and American Indians were the focal groups; males and Whites were the reference groups. With the exception of the American Indians, each focal group was represented in sufficient numbers on each test form to lead to meaningful interpretation of data. For each item category, one‐way analyses of variance were computed using as the dependent variable the Mantel‐Haenszel DIF values. Analyses were run separately for each reference/focal group investigated.

The study reports on patterns of differential performance across SAT sections as well as on section‐specific differences. Patterns across several ethnic groups and between two gender groups, as well as results specific to each of the groups, are identified. The report addresses the following questions:

• Are there unexpected group differences associated with the points tested (e.g., percentages, verb forms)?

• What aspects of item content (e.g., subject matter, gender and ethnic references, level of language) are associated with unexpected group differences?

• Are there elements of test or item format (e.g., length of stem, formatting or location of the stem or options) that are associated with unexpected group differences?
MARGINAL MAXIMUM LIKELIHOOD ESTIMATION FOR A PSYCHOMETRIC MODEL OF DISCONTINUOUS DEVELOPMENT
Authors: Mislevy Robert J. , Wilson Mark
Published: August 2014

Standard item response theory (IRT) models posit latent variables to account for regularities in students' performances on test items. They can accommodate learning only if the expected changes in performance are smooth and, in an appropriate metric, uniform over items. Wilson's “Saltus” model extends the ideas of IRT to development that occurs in stages, where expected changes can be discontinuous, show different patterns for different types of items, and even exhibit reversals in probabilities of success on certain tasks. Examples include Piagetian stages of psychological development and Siegler's rule‐based learning. This paper derives marginal maximum likelihood (MML) estimation equations for the structural parameters of the Saltus model and suggests a computing approximation based on the EM algorithm. For individual examinees, Empirical Bayes probabilities of learning‐stage are given, along with proficiency parameter estimates conditional on stage membership. The MML solution is illustrated with simulated data and an example from the domain of mixed number subtraction.
ITEM DIFFICULTY ADJUSTMENT STUDY: GRE VERBAL DISCRETES
Authors: Adams Richard , Carson John , Cureton Kevin
Published: August 2014

The purpose of this study was to determine whether it is both possible and cost‐effective to revise middle‐difficulty GRE discrete items in order to produce items of higher or lower difficulty. It was found that it is significantly easier to increase the difficulty of middle‐difficulty items than to reduce the difficulty of such items, and that the difficulties of antonyms and analogies are much easier to manipulate than those of sentence completions. The evidence also suggests that producing harder analogies and antonyms by revising items in this manner would be a cost‐effective procedure.
COACHING FOR THE SAT: A SUMMARY OF THE SUMMARIES AND AN UPDATE
Authors: Powers Donald E.
Published: August 2014

Why is it important to investigate the effectiveness of coaching for a test such as the SAT? How can you evaluate the effectiveness of coaching? What are some common misconceptions? What do we know and what is still unclear about the effects of coaching? What can we tell students and their families about the results of coaching studies?
THE OPTIONAL ESSAY PROBLEM AND THE HYPOTHESIS OF EQUAL DIFFICULTY
Authors: Allen Nancy L. , Holland Paul W. , Thayer Dorothy T.
Published: August 2014

We examine a special case of examinee choice, the Optional Essay Problem, from the point of view of test equating. The Optional Essay Problem involves equating essay scores when the examinees are required to select an optional essay topic from a list of topics in addition to taking a mandatory test required of all examinees. We derive conditions that must be satisfied if the null hypothesis of ‘equal difficulty’ of the essays is true. (We call this Livingston's Null Hypothesis.) If this hypothesis holds then there is no need to equate the scores on the optional essays. Our conditions take the form of inequalities about unobservable quantities that may be displayed graphically. We illustrate them using a real example from the Advanced Placement Examinations. Then we analyze Livingston's (1988) proposal for adjusting essay scores in the Optional Essay Problem and explain it from the perspective of test equating. We use our explanation to generalize his proposal to two new proposals that are explicit about the assumptions they make concerning the unobserved data. (We argue that every method for adjusting the essay scores in the Optional Essay Problem must make assumptions about unobserved data.) We illustrate the adjustment methods with an example from the Advanced Placement Examinations. Finally, we use the results for adjusting optional essay scores to propose comparable procedures for directly adjusting linear composite scores that include both a mandatory and an optional test score.
FOUNDATIONS OF VALIDITY: MEANING AND CONSEQUENCES IN PSYCHOLOGICAL ASSESSMENT
Authors: Messick Samuel
Published: August 2014

The traditional conception of validity divides it into three separate and substitutable types — namely, content, criterion, and construct validities. This view is fragmented and incomplete, especially in failing to take into account evidence of the value implications of score meaning as a basis for action and of the social consequences of test use. The new unified concept of validity interrelates these issues as fundamental aspects of a more comprehensive theory of construct validity. That is, unified validity integrates considerations of content, criteria, and consequences into a construct framework for empirically testing rational hypotheses about score meaning and theoretically relevant relationships, including those of both an applied and a scientific nature. The essence of unified validity is that the appropriateness, meaningfulness, and usefulness of score‐based inferences are inseparable and that the unifying force behind this integration is the trustworthiness of empirically grounded score interpretation. Thus, construct validity conceived comprehensively becomes the whole of assessment validity for both scientific and applied purposes. In turn, assessment validation becomes scientific inquiry into score meaning and the consequences of score use.
THE E‐STEP OF THE MGROUP EM ALGORITHM
Authors: Thomas Neal
Published: August 2014

Mislevy(1984, 1985) introduced an EM algorithm for estimating the parameters of a latent distribution model that is used extensively by the National Assessment of Educational Progress. Second order asymptotic corrections are derived and applied along with more common first order asymptotic corrections to approximate the expectations required by the E‐step of the EM algorithm. These corrections produce much more accurate approximations than those produced by two different normal approximations. The asymptotic corrections are applicable in high dimensional models. The number of function evaluations required by these methods grows linearly as dimension increases, in contrast to the exponential growth with dimension of product quadrature rules which yield similar accuracy.
SIMULATED EQUATING USING SEVERAL ITEM RESPONSE CURVES
Authors: Boldt R. F.
Published: August 2014

Previous research has shown that a single factor, factor being used in the sense of factor analysis, gave a very good account of item covariances within TOEFL® sections. This result is consistent with the assumption that the product of a person parameter and an item parameter models the probability that the person would pass the item. The assumption forms a simple item response model. A subsequent cross‐validation study using this model supported the efficacy of that assumption by predicting item success as accurately as did the 3‐Parameter Logistic model, and a modified Rasch model. The purpose of the current study was to extend the comparison of models to an equating context. “Equating” is a statistical process that identifies comparable scores from parallel tests administered to different populations. In an operational context, equating serves to facilitate comparison of scores generated on different forms of a test.

The present study consisted of simulation trials designed to “equate the test to itself.” That is, equating sample data were generated from administration of identical item sets. It is useful to do this as a test of model validity, because if the same item sets are used to equate, an accurate equating would identify equal scores as comparable. Discrepancies between comparable scores signify error model misfit or random error.

Equatings that used procedures based on each model were accomplished under several conditions and the results were compared. The conditions varied by sample size, anchor test difficulty, and the TOEFL section equated. In order to compound the difficulty of the equating task, results were based on equating samples that were mismatched in performance on a correlated measure.

Most discrepancies between comparable scores were largest at the extremes. The largest discrepancies between scores identified as comparable occurred for the 3PL and modified Rasch models at the lower extreme scores, and for the simple models at the upper extreme score. For the 1,000‐case sample, most were in fractions of score points. As expected, 3PL equatings exhibited the largest discrepancies for the 100‐case sample. The simple item response model yielded the most discrepancies that were in excess of the standard error of measurement, in part because with that model the maximum discrepancies occurred at the top of the score range, where the standard errors of measurement approach zero. Imposing an upper bound on the probability of correct response in the simple model markedly reduced its errors.

TOEFL scores are used for educational decisions. If it is true that most institutions' cut scores occur in the mid‐score ranges, the present study suggests that 3PL should not be used if equating samples are substantially reduced from the present size. The other models are promising for small‐sample equating, with the one‐parameter logistic models being most promising.
THE EFFECT OF SMALL CALIBRATION SAMPLE SIZES ON TOEFL IRT‐BASED EQUATING
Authors: Tang K. Linda , Way Walter D. , Carey Patricia A.
Published: August 2014

The present study compared the performance of LOGIST and BILOG on TOEFL IRT‐based scaling and equating using both real and simulated data and two calibration structures. Applications of IRT for the TOEFL program are based on the three‐parameter logistic (3PL) model.

The results of the study show that item parameter estimates obtained from the smaller real data sample sizes were more consistent with the larger sample estimates when based on BILOG than when based on LOGIST. In addition, the root mean squared error statistics suggest that the BILOG estimates for the item parameters and item characteristic curves were closer in magnitude to the “true” parameter values than were the LOGIST estimates.

The equating results based on the parameter estimates suggest that the rule of thumb recommendation that pretest sample sizes be at least 1000 for LOGIST should be retained if at all possible.
PROBABILITY‐BASED INFERENCE IN COGNITIVE DIAGNOSIS
Authors: Mislevy Robert J.
Published: August 2014

Recent developments in cognitive psychology suggest models for knowledge and learning that often fall outside the realm of standard test theory. This paper concerns probability‐based inference in terms of such models. An approach utilizing Bayesian inference networks is outlined. Basic ideas of structure and computation in inference networks are discussed, and illustrated with an example from the domain of mixed‐number subtraction.
ASSESSING THE CLASSROOM PERFORMANCE OF BEGINNING TEACHERS: WHAT EVIDENCE DO TEACHERS REGARD AS RELEVANT?
Authors: Morris Lori V. , Powers Donald E.
Published: August 2014

One of the three components of The Praxis Series: Professional Assessments for Beginning Teachers™ is a classroom performance assessment concerning the new teacher's ability to apply basic elements of good teaching in a classroom setting. At the core of this classroom assessment are 19 criteria for which assessors are to identify examples of evidence supporting whether or not a teacher can perform the function(s) implied by each criterion. The purpose of this study was to delineate further the kinds of teacher cognitions, decisions, and behaviors that may appropriately be regarded as evidence for each criterion. The focus was to establish the kinds of “generic” evidence that might cut across contexts, as well as determining what might be specific to particular situations. Survey responses from 131 teachers across subject matters are analyzed and both positive and negative evidence statements are developed for each criterion. Examples of both “generic” evidence and subject matter specific evidence are provided. The data gathered in this study could be useful for training assessors for the classroom assessment component of The Praxis Series: Professional Assessments for Beginning Teachers™.
ESTIMATING SCORES FOR AN OPTIONAL TEST SECTION USING INFORMATION FROM A COMMON SECTION*
Authors: Allen Nancy L. , Holland Paul W. , Thayer Dorothy T.
Published: August 2014

Some tests include a section of optional questions in addition to mandatory parts of the test. The purpose of this paper is to provide a general method based on missing‐data methods for nonignorable nonresponse (Little & Rubin, 1987) to estimate distributions of scores on an optional part of the test taking into account information from a mandatory part of the test. The scores for the optional section can then be placed on comparable scales using “equating” or moderation methods.

This paper is the third in a series of three papers. In the first “Approaches to Nonignorable Nonresponse with Application to Selection Bias” (Allen, Holland, & Thayer, in press, a), we apply missing‐data methods from settings external to education to this setting. A method from the econometric literature (Heckman, 1976) is contrasted to methods proposed by Little and Rubin (1987) and Tukey (Holland, 1986). In the second paper, “A Missing Data Approach to Estimating Distributions of Scores for Optional Test Sections” (Allen, Holland, & Thayer, in press, b) the perspective of measurement specialists who proposed methods to transform scores for optional questions to comparable scales is contrasted with the approach based on the work of Little and Rubin. In this paper, the methods of Rubin and Tukey are extended to include common scores for the mandatory part of the test. Data from an example are actually transformed to comparable scales.
A COMPARISON OF THE PROPERTIES OF IRT PARAMETER ESTIMATES USING TWO DIFFERENT CALIBRATION DESIGNS
Authors: Wightman Lawrence E. , Champlain Andre F. De
Published: August 2014

The purpose of this study was to compare two different methods of obtaining 3PL IRT pretest item parameter estimates for the Graduate Management Admissions Testing Program. The first method consisted of calibrating pretest and operational items simultaneously in a LOGIST run, that is, a concurrent calibration design. The second approach entailed analyzing the pretest items separately from the operational items holding examinee ability scores constant from a previous operational items run, that is, using a two‐stage calibration design. Results show that the means of the item difficulty (b‐parameter) estimates were very similar, regardless of the method employed. However, the higher b‐parameter values using the two‐stage calibration run method (i.e. holding ability fixed excluding the studied items from the criterion) were slightly overestimated and the lower b‐parameter values were slightly underestimated. The a‐parameters were consistently underestimated using the two‐stage estimation procedure. Finally, the slopes of the item‐ability regressions using a concurrent calibration (including the studied items) are steeper for nearly all of the items. These preliminary results are consistent with those reported in past studies (Stocking & Eignor, 1986) and suggest that non‐operational (pretest) items should be calibrated concurrently with operational items for item banking purposes.
GENDER DIFFERENCES IN HIGH SCHOOL GRADES: AN EXPLORATORY STUDY
Authors: Ekstrom Ruth B.
Published: August 2014

This study used the 1980 High School and Beyond data set to examine the variables associated with the grades that college‐bound high school sophomores received in English, algebra, and geometry courses. Special concerns included determining if gender differences in high school grades could be explained and how teachers' perceptions of students, student characteristics, and High School and Beyond test scores were related to grades. A model of factors that might affect grades was developed; it included students' background characteristics, general attitudes, curriculum, educational aspirations, attitudes toward school and individual academic subjects, school behaviors, and scores on relevant High School and Beyond tests as well as teachers' perceptions of the students. The full models explained 46 percent of the variance in English grades, 42 percent of the variance in algebra grades, and 44 percent of the variance in geometry grades. The full models tended to explain more of the variance in males' than in females' grades, with a more marked difference in English and Algebra 1 than in Algebra 2 or geometry. This suggests that some other variable, not included in the models, may have been affecting grades.

Even after controlling for all the variables in the full models, a significant association between gender and course grades remained in English and in Algebra 1; there was no significant relationship between gender and grades in Algebra 2 or in geometry. Teachers' comments were significantly associated with grades for both males and females in English, Algebra 1, and geometry, but they were not significantly associated with males' grades in Algebra 2. Teachers' perceptions of students were significantly associated with gender even after controlling for all the variables in the full models. These findings suggest that teachers may have gender‐related expectations for the students in their courses. The High School and Beyond verbal test scores were significantly associated with grades for both females and males in English and High School and Beyond math test scores were significantly associated with grades in geometry for both males and females. High School and Beyond math test scores were not however consistently associated with grades in algebra.
VARIABLE SCREENING FOR CLUSTER ANALYSIS
Authors: Donoghue John R.
Published: August 2014

Inclusion of irrelevant variables in a cluster analysis adversely affects subgroup recovery. This paper examines using moment‐based statistics to screen variables; only variables which pass the screening are then used in clustering. Normal mixtures are analytically shown often to possess negative kurtosis. Two related measures,mand coefficient of bimodalityb, are also examined.

A Monte Carlo study compared the screening measures to no selection, De Soete's (1988) ultrametric weights, and Fowlkes, Gnanadesikan, and Kettenring's (1988) forward selection procedure. Screening based on kurtosis degraded recovery and is not recommended. In contrast, screening onmor onbimproved recovery over both no selection and forward selection, and screening performed as well as ultrametric weights. Combining screening with ultrametric weights performed extremely well. All methods were found to be somewhat sensitive to other types of error.

Screening variables appears a viable alternative to both ultrametric weights and forward selection. The potential advantages and disadvantages of screening are considered.
VALIDITY OF PSYCHOLOGICAL ASSESSMENT: VALIDATION OF INFERENCES FROM PERSONS' RESPONSES AND PERFORMANCES AS SCIENTIFIC INQUIRY INTO SCORE MEANING
Authors: Messick Samuel
Published: August 2014

The traditional conception of validity divides it into three separate and substitutable types – namely, content, criterion, and construct validities. This view is fragmented and incomplete, especially in failing to take into account evidence of the value implications of score meaning as a basis for action and of the social consequences of score use. The new unified concept of validity interrelates these issues as fundamental aspects of a more comprehensive theory of construct validity addressing both score meaning and social values in both test interpretation and test use. That is, unified validity integrates considerations of content, criteria, and consequences into a construct framework for empirically testing rational hypotheses about score meaning and theoretically relevant relationships, including those of both an applied and a scientific nature. Six distinguishable aspects of construct validity are highlighted as a means of addressing central issues implicit in the notion of validity as a unified concept. These are content, substantive, structural, generalizability, external, and consequential aspects of construct validity. In effect, these six aspects function as general validity criteria or standards for all educational and psychological measurement, including performance assessments, which are discussed in some detail because of their increasing emphasis in educational and employment settings.
DEFINING THE CONTENT DOMAIN FOR THE PRAXIS II SUBJECT ASSESSMENT IN EARTH AND SPACE SCIENCE: KNOWLEDGE IMPORTANT FOR BEGINNING TEACHERS
Authors: Tannenbaum Richard J.
Published: August 2014

The purpose of this study was to conduct a job analysis focusing on the knowledge important for beginning Earth and Space Science teachers. The results of the job analysis will be used to define the content domain of the Subject Assessment in Earth and Space Science for the Praxis Series: Professional Assessments for Beginning Teachers®.

A domain of 292 knowledge statements and elements (specific descriptors used to clarify the meaning of each, more broadly defined, knowledge statement) was developed by subject‐matter experts. These statements and elements were then incorporated into a survey that was administered to a large number of Earth and Space Science teachers, teacher educators, and state administrators. These professionals rated the importance of each knowledge statement and knowledge element using a 0‐ to 4‐point scale.

A cut point of a mean of 2.50 (midpoint between moderately important and important) was established to differentiate important knowledge (M ≥ 2.50) and unimportant knowledge (M < 2.50). The results indicated that a total of 36 knowledge statements and knowledge elements (12% of the total domain) failed to pass the cut point. It is recommended that these 36 statements and elements not be included in the development of the Subject Assessment in Earth and Space Science; emphasis, instead, should be put on the 256 knowledge statements and knowledge elements that did pass the cut point.
STANDARDS‐BASED SCORE INTERPRETATION: ESTABLISHING VALID GROUNDS FOR VALID INFERENCES
Authors: Messick Samuel
Published: August 2014

The construct validity of content standards is addressed in terms of their representative coverage of a construct domain and their alignment with the students' cognitive level of developing expertise in the subject matter. The construct validity of performance standards is addressed in terms of the extent to which they reflect increasing levels of construct complexity as opposed to construct‐irrelevant difficulty. Also critical is the extent to which performance standards characterize the knowledge and skills operative at each level both to accredit specific accomplishment and to serve as goals for further learning. All of this depends on construct‐valid assessment attuned to the content standards and the development of dependable scoring rubrics and measurement scales for representing the performance standards.
FEASIBILITY OF USING THE SAT®IN ACADEMIC GUIDANCE
Authors: STRICKER LAWRENCE J. , ROCK DONALD A. , BURTON NANCY W.
Published: August 2014

This study appraised the validity of SAT scores, in combination with grades in high school courses and the number and difficulty level of these courses, in predicting college grades in various fields of study, with the objective of providing SAT takers with predictions of their academic performance in different fields for guidance purposes. The possible impact of this feedback on the flow of students into specific major fields was also assessed. Data on an entering class at a large state university provided the basis for the study. The SAT and collateral variables were able to predict college grades in different academic areas by taking into account marked variations in grade distributions among fields of study. These predictions could be potentially useful in academic guidance. Students' predicted grades in the various fields and their intended majors were unrelated. Most students received their highest predicted grades in the humanities, although few intended to major in this area. Hence, such feedback would probably stimulate students to take courses or major in the humanities.
DETECTING DIF FOR POLYTOMOUSLY SCORED ITEMS: AN ADAPTATION OF THE SIBTEST PROCEDURE
Authors: Chang Hua‐Hua , Mazzeo John , Roussos Louis
Published: August 2014

Recently, Shealy and Stout (1993) proposed a DIF detecting procedure SIBTEST, which is 1) IRT model based, 2) non‐parametric, 3) does not require IRF estimation, 4) provides a test of significance, and 5) estimates the amount of DIF. Current versions of SIBTEST can only be used for dichotomously scored items. However, in this paper an extension to handle polytomous items is developed. This paper presents: (1) a discussion of an appropriate definition of DIF for polytomously scored items, (2) a modified SIBTEST procedure for detecting DIF for polytomous items, and (3) the results of two simulation studies comparing the modified SIBTEST with the Mantel and SMD procedures, one study with data constrained by the Rasch‐like partial credit model (same discrimination across polytomous items), and the other study with data having distinctly discrimations across items. These simulation studies indicate that the methodology ofincluding the studied item in matching subtestfor controling impact induced (group ability differences existing) Type I error tends to yield Type‐I/Type II error inflation rates that are highly unacceptable when theequal discriminationcondition is violated. These simulation studies provide compelling evidence that the modified SIBTEST procedure is much more robust with regard to controlling impact‐induced Type I error rate inflation than the other procedures.
AN ALGORITHM FOR CONVERTING CONTOURS TO ELEVATION GRIDS
Authors: Reid‐Green Keith S.
Published: August 2014

Some of the test questions for the National Council of Architectural Registration Boards (NCARB) deal with the site, including drainage, regrading, and the like. A site is described by its shape and by contours shown on the plan view. Existing structures may be shown on the site. Some questions are most easily scored by examining contours, such as determining if, in regrading the site, the student changed a contour, causing it to pass through an existing structure. However, water flow questions are best scored from a grid of elevations. This document describes a method for creating an elevation grid from a set of contour lines.
EVIDENCE AND INFERENCE IN EDUCATIONAL ASSESSMENT1,2
Authors: Mislevy Robert J.
Published: August 2014

Educational assessment concerns inference about students' knowledge, skills, and accomplishments. Because data are never so comprehensive and unequivocal as to ensure certitude, test theory evolved in part to address questions of weight, coverage, and import of data. The resulting concepts and techniques can be viewed as applications of more general principles for inference in the presence of uncertainty. Issues of evidence and inference in educational assessment are discussed from this perspective.
DEPICTING ERROR
Authors: Wainer Howard
Published: August 2014

Communicating data without some measure of their precision can lead to misinterpretation and incorrect inferences. Several conventions for displaying error along with the data they modify are described and illustrated. Alternatives are offered that seem to provide improvements in the effective communication of error as well as increasing their ease, and hence their likelihood, of use. These alternatives are illustrated principally with data from the National Assessment of Educational Progress.
PERFORMANCE BY GENDER ON AN UNCONVENTIONAL VERBAL REASONING TASK: ANSWERING READING COMPREHENSION QUESTIONS WITHOUT THE PASSAGES
Authors: POWERS DONALD E.
Published: August 2014

Data were reanalyzed from a previously reported study of the passage dependence of reading comprehension questions being developed for the revised SAT. The objective was to uncover any gender differences in approaches to and performance on a task requiring examinees to answer reading comprehension questions without reading the passages on which the questions were based. Verbally able males and females were compared with respect to their reported use of a variety of test‐taking strategies involving reasoning, personal knowledge, and guessing. A few relatively small (and often inconsistent) differences were detected between male and female test takers. However, far more similarities than differences were noted with respect to both test performance and test‐taking behavior. If these results are generalizable to more typical tests of verbal ability, they would seem to suggest that males and females employ quite similar approaches to standardized test taking.
ESTIMATING RELIABILITY FOR TESTS COMPOSED OF ITEM SETS
Authors: Lawrence Ida M.
Published: August 2014

To what extent, if any, are estimates of reliability for a multiple choice test affected by the presence of large item sets where each set shares common reading material? The purpose of this research was to assess the effect of local item dependence on estimates of reliability for verbal portions of the old and new SAT, where the new test contains larger item sets associated with reading passages. Estimates based on a single administration of the test (estimates based on internal consistency and estimates based on covariances among parts) were compared to estimates based on two administrations of the test. When adjusted for a fixed standard deviation, estimates based on covariances among parts tended to be similar to estimates based on parallel forms. Both types of estimates were lower than the internal consistency estimates.
THE INTRODUCTION AND COMPARABILITY OF THE COMPUTER ADAPTIVE GRE GENERAL TEST
Authors: Schaeffer Gary A. , Steffen Manfred , Golub‐Smith Marna L. , Mills Craig N. , Durso Robin
Published: August 2014

This report summarizes the results from two studies. The first study assessed the comparability of scores derived from linear computer‐based (CBT) and computer adaptive (CAT) versions of the three GRE General Test measures. The verbal and quantitative CATs were found to produce scores that were comparable to their CBT counterparts. However, the analytical CAT produced scores that were judged not to be comparable to the analytical CBT scores. As a result, a second study was performed to further examine the analytical measure to ascertain the extent of the lack of comparability and to obtain statistics that would permit adjustments to restore comparability.

Results of the additional study of the analytical measure indicated that the differences in analytical CAT and CBT scores due to the testing paradigm were large enough to require an adjustment in scores. Therefore, in order to enhance the comparability of analytical CAT and CBT scores, the analytical CAT was equated to the analytical CBT. This equating provided new analytical CAT conversions that resulted in comparable analytical CAT and CBT scores.