bluuebunny/crossref_metadata_2025 huggingface dataset:
A Didactic Explanation of Item Bias, Item Impact, and Item Validity From a Multidimensional Perspective
Authors: Ackerman Terry A.
Published: September 2005

Many researchers have suggested that the main cause of item bias is the misspecification of the latent ability space, where items that measure multiple abilities are scored as though they are measuring a single ability. If two different groups of examinees have different underlying multidimensional ability distributions and the test items are capable of discriminating among levels of abilities on these multiple dimensions, then any unidimensional scoring scheme has the potential to produce item bias. It is the purpose of this article to provide the testing practitioner with insight about the difference between item bias and item impact and how they relate to item validity. These concepts will be explained from a multidimensional item response theory (MIRT) perspective. Two detection procedures, the Mantel‐Haenszel (as modified by Holland and Thayer, 1988) and Shealy and Stout's Simultaneous Item Bias (SIB; 1991) strategies, will be used to illustrate how practitioners can detect item bias.
Overview of the Scaling Methodology Used in the National Assessment
Authors: Beaton Albert E. , Johnson Eugene G.
Published: September 2005

The National Assessment of Educational Progress (NAEP) uses item response theory (IRT)–based scaling methods to summarize the information in complex data sets. Scale scores are presented as tools for illuminating patterns in the data and for exploiting regularities across patterns of responses to tasks requiring similar skills. In this way, the dominant features of the data are captured. Discussed are the necessity of global scores or more detailed subscores, the creation of developmental scales spanning different age levels, and the use of scale anchoring as a way of interpreting the scales.
A Comparison of Score Level Estimates of the Standard Error of Measurement
Authors: Quails‐Payne Audrey L.
Published: September 2005

Numerous methods have been proposed and investigated for estimating · the standard error of measurement (SEM) at specific score levels. Consensus on the preferred method has not been obtained, in part because there is no standard criterion. The criterion procedure in previous investigations has been a single test occasion procedure. This study compares six estimation techniques. Two criteria were calculated by using test results obtained from a test‐retest or parallel forms design. The relationship between estimated score level standard errors and the score scale was similar for the six procedures. These relationships were also congruent to findings from previous investigations. Similarity between estimates and criteria varied over methods and criteria. For test‐retest conditions, the estimation techniques are interchangeable. The user's selection could be based on personal preference. However, for parallel forms conditions, the procedures resulted in estimates that were meaningfully different. The preferred estimation technique would be Feldt's method (cited in Gupta, 1965; Feldt, 1984).
The Standardization Approach to Assessing Comprehensive Differential Item Functioning
Authors: Dorans Neil J. , Schmitt Alicia P. , Bleistein Carole A.
Published: September 2005

The standardization approach to comprehensive differential item functioning (Cdif) is described and contrasted with the log‐linear approach to differential distractor functioning explicated by Green, Crone, and Folk (1989) and with the IRT‐based approach to differential alternative functioning explicated by Thissen, Steinberg, and Wainer (1992). This descriptive approach is used routinely as an adjunct to Mantel‐Haenszel differential item functioning (DIP) detection (Dorans & Holland, 1992) in many operational testing programs at the Educational Testing Service. Data from an edition of the SAT are used to illustrate how the standardization approach to Cdif could be used to uncover differential speededness. Speculations about the sources of differential speededness for Black examinees and Hispanic examinees are offered, and some implications of the existence of differential speededness for DIP detection are mentioned.
To Guess or Not to Guess: A Decision‐Theoretic View of Formula Scoring
Authors: Budescu David , Bar‐Hillel Maya
Published: September 2005

Multiple‐choice tests are often scored by formulas under which the respondent's expected score for an item is the same whether he or she omits it or guesses at random. Typically, these formulas are accompanied by instructions that discourage guessing. In this article, we look at test taking from the normative and descriptive perspectives of judgment and decision theory. We show that for a rational test taker, whose goal is the maximization of expected score, answering is either superior or equivalent to omitting–a fact which follows from the scoring formula. For test takers who are not fully rational, or have goals other than the maximization of expected score, it is very hard to give adequate formula scoring instructions, and even the recommendation to answer under partial knowledge is problematic (though generally beneficial). Our analysis derives from a critical look at standard assumptions about the epistemic states, response strategies, and strategic motivations of test takers. In conclusion, we endorse the number‐right scoring rule, which discourages omissions and is robust against variability in respondent motivations, limitations in judgments of uncertainty, and item vagaries.
An Empirical Study on the Nature of Trick Test Questions
Authors: Roberts Dennis M.
Published: September 2005

This study attempted to better define trick questions and see if students could differentiate between trick and not–trick questions. Phase 1 elicited definitions of trick questions so as to identify essential characteristics. Seven components were found. Phase 2 obtained ratings to see which components of trick questions were considered to be most crucial. The intention of the item constructor and the fact that the questions had multiple correct answers received highest ratings from students. Phase 3 presented a collection of statistics items, some of which were labeled on an a priori basis as being trick or not–trick. The analysis indicated that examinees were able to statistically differentiate between trick and not–trick items, but the difference compared to chance was small. Not–trick items were more successfully sorted than trick items, and trick items that were classified as intentional were sorted about as well as nonintentional items. Evidence seems to suggest that the concept of trickiness is not as clear as some test construction textbook authors suggest.
Logistic Discriminant Function Analysis for DIF Identification of Polytomously Scored Items
Authors: Miller Timothy R. , Spray Judith A.
Published: September 2005

The purpose of this article is to present logistic discriminant function analysis as a means of differential item functioning (DIF) identification of items that are polytomously scored. The procedure is presented with examples of a DIF analysis using items from a 27‐item mathematics test which includes six open‐ended response items scored polytomously. The results show that the logistic discriminant function procedure is ideally suited for DIF identification on nondichotomously scored test items. It is simpler and more practical than polytomous extensions of the logistic regression DIF procedure and appears to fee more powerful than a generalized Mantel‐Haenszelprocedure.
Influence of Item Parameter Estimation Errors in Test Development
Authors: Hambleton Ronald K. , Jones Russell W. , Rogers H. Jane
Published: September 2005

Item response models are finding increasing use in achievement and aptitude test development. Item response theory (IRT) test development involves the selection of test items based on a consideration of their item information functions. But a problem arises because item information functions are determined by their item parameter estimates, which contain error. When the “best” items are selected on the basis of their statistical characteristics, there is a tendency to capitalize on chance due to errors in the item parameter estimates. The resulting test, therefore, falls short of the test that was desired or expected. The purposes of this article are (a) to highlight the problem of item parameter estimation errors in the test development process, (b) to demonstrate the seriousness of the problem with several simulated data sets, and (c) to offer a conservative solution for addressing the problem in IRT‐based test development.
Stability of Individual Differences in Multiwave Panel Studies: Comparison of Simplex Models and One‐Factor Models
Authors: Marsh Herbert W.
Published: September 2005

The purpose of this investigation is to evaluate structural equation models (SEMs) for measures of the same construct collected on multiple occasions (one‐variable, multiwave panel studies). Simplex models hypothesize that a measure at any one wave is substantially influenced by the measure at the 0immediately preceding wave; correlations between the same construct measured on different occasions are predicted to decline systematically as the number of intervening occasions increases. Alternatively, a one‐factor model posits that a person's score at any one time is a function of some underlying “true” score and a random disturbance that is idiosyncratic to the time; no temporal ordering of correlations is assumed. Both the simplex and one‐factor models can befit when there is only a single indicator of each construct at each wave (e.g., scale scores), but there are serious limitations to such models. Stronger models are possible when the same set of multiple indicators (e.g., the items that make up the scale) is measured at each wave. In Study 1, based on students' evaluations of teaching effectiveness collected over an 8‐year period, one‐factor models fit the data well, whereas simplex models did not. In Study 2, based on personality variables collected over a 4‐year period during adolescence, one‐factor models again provided an excellent fit to the data, whereas the simplex model did marginally poorer. The results challenge an overreliance on simplex models and demonstrate that a one‐factor model is a potentially useful alternative that should be considered in multiwave studies.
Assessing Dimensionality of a Set of Item Responses‐Comparison of Different Approaches
Authors: Nandakumar Ratna
Published: September 2005

This study compares the performance of three methodologies for assessing unidi‐mensionality: DIMTEST, Holland and Rosenbaum's approach, and nonlinear factor analysis. Each method is examined and compared with other methods on simulated and real data sets. Seven data sets, all with 2,000 examinees, were generated: three unidimensional and four two‐dimensional data sets. Two levels of correlation between abilities were considered:ρ=3 andρ=. 7. Eight different real data sets were used: Four of them were expected to be unidimensional, and the other four were expected to be two‐dimensional. Findings suggest that all three methods correctly confirmed unidimensionality but differed in their ability to detect lack of unidimensionality. DIMTEST showed excellent power in detecting lack of unidimensionality; Holland and Rosenbaum's and nonlinear factor analysis approaches showed good power, provided the correlation between abilities was low.
Human Self‐Assessment in Multiple‐Choice Testing
Authors: Hassmén Peter , Hunt Darwin P.
Published: September 2005

Research indicates that the multiple‐choice format in itself often seems to favor males over females. The present study utilizes a method that enables test takers to assess the correctness of their answers. Applying this self‐assessment method may not only make the multiple‐choice tests less biased but also provide a more comprehensive measurement of usable knowledge‐that is, the kind of knowledge about which a person is sufficiently sure so that he or she will use the knowledge to make decisions and take actions. The performance of male and female undergraduates on a conventional multiple‐choice test was compared with their performance on a multiple‐choice self‐assessment test. Results show that the difference between test scores of males and those of females was reduced when subjects were allowed to make self‐assessments. This may be explained in terms of the alleged difference in cognitive style between the genders.
Reliability and Information Functions for Percentile Ranks
Authors: May Kim , Nicewander W. Alan
Published: September 2005

Reliabilities and information functions for percentile ranks and number‐right scores were compared in the context of item response theory. The basic results were: (a) The percentile rank is always less informative and reliable than the number‐right score; and (b)for easy or difficult tests composed of highly discriminating items, the percentile rank often yields unacceptably low reliability and information relative to the number‐right score. These results suggest that standardized scores that are linear transformations of the number‐right score (e.g., z scores) are much more reliable and informative indicators of the relative standing of a test score than are percentile ranks. The findings reported here demonstrate that there exist situations in which the percent of items known by examinees can be accurately estimated, but that the percent of persons falling below a given score cannot.
Generalizability, Validity, and Examinee Perceptions of a Computer‐Delivered Formulating‐Hypotheses Test
Authors: Bennett Randy Elliot , Rock Donald A.
Published: September 2005

The Formulating‐Hypotheses (F‐H) item presents a situation and asks examinees to generate as many explanations for it as possible. This study examined the generalizability, validity, and examinee perceptions of a computer‐delivered version of the task. Eight F‐H questions were administered to 192 graduate students. Half of the items restricted examinees to 7 words per explanation, and half allowed up to 15 words. Generalizability results showed high interrater agreement, with tests of between 2 and 4 items scored by one judge achieving coefficients in the .80s. Construct validity analyses found that F‐H was only marginally related to the GRE General Test, and more strongly related than the General Test to a measure of ideational fluency. Different response limits tapped somewhat different abilities, with the 15‐word constraint appearing more useful for graduate assessment. These items added significantly to conventional measures in explaining school performance and creative expression.
Equating Error and Statistical Bias in Small Sample Linear Equating
Authors: Parshall Cynthia G. , Houghton Pansy Du Bose , Kromrey Jeffrey D.
Published: September 2005

A resampling study was conducted to compare the statistical bias and standard errors of nonequivalent‐groups linear test equating in small samples of examinees. Sample sizes of 15, 25, 50, and 100 were examined. One thousand samples of each size were drawn with replacement from each of 5 archival data files from teacher subject area tests. For each test, data files from 2 parallel forms were used. Results suggest trivial levels of equating bias even with small samples, but substantial increases in standard errors as sample size decreases. Results were interpreted in terms of applications to testing situations in which small numbers of examinees are available.
The Effects of Dimensionality on Equating the Law School Admission Test
Authors: Camilli Gregory , Wang Ming‐mei , Fesq Jacqueline
Published: September 2005

Using factor analysis, we conducted an assessment of multidimensionality for 6 forms of the Law School Admission Test (LSAT) and found 2 subgroups of items or factors for each of the 6 forms. The main conclusion of the factor analysis component of this study was that the LSAT appears to measure 2 different reasoning abilities: inductive and deductive. The technique of N. J. Dorans & N. M. Kingston (1985) was used to examine the effect of dimensionality on equating. We began by calibrating (with item response theory [IRT] methods) all items on a form to obtain Set I of estimated IRT item parameters. Next, the test was divided into 2 homogeneous subgroups of items, each having been determined to represent a different ability (i.e., inductive or deductive reasoning). The items within these subgroups were then recalibrated separately to obtain item parameter estimates, and then combined into Set II. The estimated item parameters and true‐score equating tables for Sets I and II corresponded closely.
Robustness of the School‐Level IRT Model
Authors: Tate Richard L.
Published: September 2005

The robustness of the school‐level item response theoretic (IRT) model to violations of distributional assumptions was studied in a computer simulation. Estimated precision of “expected a posteriori” (EAP) estimates of the mean school ability from BILOG 3 was compared with actual precision, varying school size, intraclass correlation, school ability, number of forms comprising the test, and item parameters. Under conditions where the school‐level precision might be possibly acceptable for real school comparisons, the EAP estimates of school ability were robust over a wide range of violations and conditions, with the estimated precision being either consistent with the actual precision or somewhat conservative. Some lack of robustness was found, however, under conditions where the precision was inherently poor and the test would presumably not be used for serious school comparisons.
Estimating the Consistency and Accuracy of Classifications Based on Test Scores
Authors: Livingston Samuel A. , Lewis Charles
Published: September 2005

This article presents a method for estimating the accuracy and consistency of classifications based on test scores. The scores can be produced by any scoring method, including a weighted composite. The estimates use data from a single form. The reliability of the score is used to estimate effective test length in terms of discrete items. The true‐score distribution is estimated by fitting a 4‐parameter beta model. The conditional distribution of scores on an alternate form, given the true score, is estimated from a binomial distribution based on the estimated effective test length. Agreement between classifications on alternate forms is estimated by assuming conditional independence, given the true score. Evaluation of the method showed estimates to be within 1 percentage point of the actual values in most cases. Estimates of decision accuracy and decision consistency statistics were only slightly affected by changes in specified minimum and maximum possible scores.
Scoring a Performance‐Based Assessment by Modeling the Judgments of Experts
Authors: Clauser Brian E. , Subhiyah Raja G. , Nungester Ronald J. , Ripkey Douglas R. , Clyman Stephen G. , McKinley Danette
Published: September 2005

Performance assessments typically require expert judges to individually rate each performance. This results in a limitation in the use of such assessments because the rating process may be extremely time consuming. This article describes a scoring algorithm that is based on expert judgments but requires the rating of only a sample of performances. A regression‐based policy capturing procedure was implemented to model the judgment policies of experts. The data set was a seven‐case performance assessment of physician patient management skills. The assessment used a computer‐based simulation of the patient care environment. The results showed a substantial improvement in correspondence between scores produced using the algorithm and actual ratings, when compared to raw scores. Scores based on the algorithm were also shown to be superior to raw scores and equal to expert ratings for making pass/fail decisions which agreed with those made by an independent committee of experts
Logistic Regression and Its Use in Detecting Differential Item Functioning in Polytomous Items
Authors: French Ann W. , Miller Timothy R.
Published: September 2005

A computer simulation study was conducted to determine the feasibility of using logistic regression procedures to detect differential item functioning (DIF) in polytomous items. One item in a simulated test of 25 items contained DIF; parameters' for that item were varied to create three conditions of nonuniform DIF and one of uniform DIF. Item scores were generated using a generalized partial credit model, and the data were recoded into multiple dichotomies in order to use logistic regression procedures. Results indicate that logistic regression is powerful in detecting most forms of DIF; however, it required large amounts of data manipulation, and interpretation of the results was sometimes difficult. Some logistic regression procedures may be useful in the post hoc analysis of DlF for polytomous items.
Empirical Validation of DIMTEST on Nonnormal Ability Distributions
Authors: Nandakumar Ratna , Yu Feng
Published: September 2005

DIMTEST is a nonparametric statistical test procedure for assessing unidimensionality of binary item response data. The development of Stout's statistic, T, used in the DIMTEST procedure, does not require the assumption of a particular parametric form for the ability distributions or the item response functions. The purpose of the present study was to empirically investigate the performance of the statistic T with respect to different shapes of ability distributions. Several nonnormal distributions, both symmetric and nonsymmetric, were considered for this purpose. Other factors varied in the study were test length, sample size, and the level of correlation between abilities. The results of Type I error and power studies showed that the test statistic T exhibited consistently similar performance for all different shapes of ability distributions investigated in the study, which confirmed the nonparametric nature of the statistic T.
The Contribution of a Response‐Production Component to a Free‐Response Synonym Task
Authors: Janssen Rianne , De Boeck Paul
Published: September 2005

A cognitive approach to the study of format differences is illustrated using synonym tasks. By means of a multiple regression analysis with latent variables, it is shown that both a response‐production component and an evaluation component are involved in answering a free‐response synonym task. Given the results of Janssen, De Boeck, and Vander Steene (1996), the format differences between the multiple‐choice evaluation task and the free‐response synonym task can be explained in terms of the kinds of verbal abilities measured. The evaluation task is a pure measure of verbal comprehension, while the free‐response synonym task is affected by verbal comprehension and verbal fluency, as well. The design used to study format differences controls both for content effects and for the effects of repeating item stems across formats.
Computerized Adaptive and Fixed‐Item Testing of Music Listening Skill: A Comparison of Efficiency, Precision, and Concurrent Validity
Authors: Vispoel Walter P. , Wang Tianyou , Bleiler Timothy
Published: September 2005

We evaluated the efficiency, precision, and concurrent validity of results obtained from adaptive and fired‐item music listening tests in three studies: (a) a computer simulation study in which each of 2,200 simulees completed a computerized adaptive tonal memory test, a computerized fired‐item tonal memory test constructed from items in the adaptive test pool and two standardized group‐administered tonal memory tests; (b) a live testing study in which each of 204 examinees took the computerized adaptive test and the standardized tests; and (c) a live testing study in which randomly equivalent groups took either the computerized adaptive test (n = 86) or the computerized fired‐item test (n = 86). The adaptive music test required 50% to 93% fewer items to match the reliability and concurrent validity of the fired‐item tests, and it yielded higher levels of reliability and concurrent validity than the fired‐item tests when test length was held constant. These findings suggest that computerized adaptive tests, which typically have been limited to visually produced items, may also be well suited for measuring skills that require aurally produced items.
Development of Automated Scoring Algorithms for Complex Performance Assessments: A Comparison of Two Approaches
Authors: Clauser Brian E. , Margolis Melissa J. , Clyman Stephen G. , Ross Linette P.
Published: September 2005

Performance assessments are typically scored by having experts rate individual performances. The cost associated with using expert raters may represent a serious limitation in many large‐scale testing programs. The use of raters may also introduce an additional source of error into the assessment. These limitations have motivated development of automated scoring systems for performance assessments. Preliminary research has shown these systems to have application across a variety of tasks ranging from simple mathematics to architectural problem solving. This study extends research on automated scoring by comparing alternative automated systems for scoring a computer simulation test of physicians'patient management skills; one system uses regression‐derived weights for components of the performance, the other uses complex rules to map performances into score levels. The procedures are evaluated by comparing the resulting scores to expert ratings of the same performances.
IRT Estimation of Domain Scores
Authors: Bock R. Darrell , Thissen David , Zimowski Michele F.
Published: September 2005

In classical test theory, a test is regarded as a sample of items from a domain defined by generating rules or by content, process, and format specifications, l f the items are a random sample of the domain, then the percent‐correct score on the test estimates the domain score, that is, the expected percent correct for all items in the domain. When the domain is represented by a large set of calibrated items, as in item banking applications, item response theory (IRT) provides an alternative estimator of the domain score by transformation of the IRT scale score on the test. This estimator has the advantage of not requiring the test items to be a random sample of the domain, and of having a simple standard error. We present here resampling results in real data demonstrating for uni‐ and multidimensional models that the IRT estimator is also a more accurate predictor of the domain score than is the classical percent‐correct score. These results have implications for reporting outcomes of educational qualification testing and assessment.
Comparison of Item Response Theory and Thurstone Methods of Vertical Scaling
Authors: Yen Wendy M. , Burket George R.
Published: September 2005

Vertical achievement scales, which range from the lower elementary grades to high school, are used pervasively in educational assessment. Using simulated data modeled after real tests, the present article examines two procedures available for vertical scaling: a Thurstone method and three‐parameter item response theory. Neither procedure produced artifactual scale shrinkage; both procedures produced modest scale expansion for one simulated condition.
Grade Equivalent and IRT Representations of Growth
Authors: Schulz E. Matthew , Nicewander W. Alan
Published: September 2005

It has long been a part of psychometric lore that the variance of children's scores on cognitive tests increases with age. This increasing‐variance phenomenon was first observed on Binet's intelligence measures in the early 1900s. An important detail in this matter is the fact that developmental scales based on age or grade have served as the medium for demonstrating the increasing‐variance phenomenon. Recently, developmental scales based on item response theory (IRT) have shown constant or decreasing variance of measures of achievement with increasing age. This discrepancy is o f practical and theoretical importance. Conclusions about the effects of variables on growth in achievement will depend on the metric chosen. In this study, growth in the mean of a latent educational achievement variable is assumed to be a negatively accelerated function o f grade; within‐grade variance is assumed to be constant across grade, and observed test scores are assumed to follow an IRT model. Under these assumptions, the variance of grade equivalent scores increases markedly. Perspective on this phenomenon is gained by examining longitudinal trends in centimeter and age equivalent measures of height.
Properties of Ability Estimation Methods in Computerized Adaptive Testing
Authors: Wang Tianyou , Vispoel Walter P.
Published: September 2005

Simulations of computerized adaptive tests (CATs) were used to evaluate results yielded by four commonly used ability estimation methods: maximum likelihood estimation (MLE) and three Bayesian approaches—Owen's method, expected a posteriori (EAP), and maximum a posteriori. In line with the theoretical nature of the ability estimates and previous empirical research, the results showed clear distinctions between MLE and the Bayesian methods, with MLE yielding lower bias, higher standard errors, higher root mean square errors, lower fidelity, and lower administrative efficiency. Standard errors for MLE based on test information underestimated actual standard errors, whereas standard errors for the Bayesian methods based on posterior distribution standard deviations accurately estimated actual standard errors. Among the Bayesian methods, Owen's provided the worst overall results, and EAP provided the best. Using a variable starting rule in which examinees were initially classified into three broad/ability groups greatly reduced the bias for the Bayesian methods, but had little effect on the results for MLE. On the basis of these results, guidelines are offered for selecting appropriate CAT ability estimation methods in different decision contexts.
Calibration and Scoring of Tests With Multiple‐Choice and Constructed‐Response Item Types
Authors: Ercikan Kadriye , Sehwarz Richard D. , Julian Marc W. , Burket George R. , Weber Melba M. , Link Valerie
Published: September 2005

This article discusses and demonstrates combining scores from multiple‐choice (MC) and constructed‐response (CR) items to create a common scale using item response theory methodology. Two specific issues addressed are (a) whether MC and CR items can be calibrated together and (b) whether simultaneous calibration of the two item types leads to loss of information. Procedures are discussed and empirical results are provided using a set of tests in the areas of reading, language, mathematics, and science in three grades.
Improving Measurement for Graduate Admissions
Authors: Enright Mary K. , Rock Donald A. , Bennett Randy Elliot
Published: September 2005

In this study we examined alternative item types and section configurations for improving the discriminant and convergent validity of the GRE General Test. A computer‐based test of reasoning items and a generating‐explanations measure was administered to a sample of 388 examinees who previously had taken the General Test. Confirmatory factor analyses indicated that three dimensions of reasoning—verbal, analytical, and quantitative—and a fourth dimension of verbal fluency based on the generating‐explanations task could be distinguished. Notably, generating explanations was as distinct from new variations of reasoning items as it was from verbal and quantitative reasoning. In the full sample, this differentiation was evident in relation to such external criteria as undergraduate grade point average (UGPA), self‐reported accomplishments, and a measure of ideational fluency, with generating explanations relating uniquely to aesthetic and linguistic accomplishments and to ideational fluency. For the subset of participants with undergraduate majors in the humanities and social sciences, generating explanations added to the relationship with UGPA over that contributed by the General Test.
An Empirical Bayes Approach to Mantel‐Haenszel DIF Analysis
Authors: Zwick Rebecca , Thayer Dorothy T. , Lewis Charles
Published: September 2005

We developed an empirical Bayes (EB) enhancement to Mantel‐Haenszel (MH) DIF analysis in which we assume that the MH statistics are normally distributed and that the prior distribution of underlying DIF parameters is also normal. We use the posterior distribution of DIF parameters to make inferences about the item's true DIF status and the posterior predictive distribution to predict the item's future observed status. DIF status is expressed in terms of the probabilities associated with each of the five DIF levels defined by the ETS classification system: C–, B–, A, B+, and C+. The EB methods yield more stable DIF estimates than do conventional methods, especially in small samples, which is advantageous in computer‐adaptive testing. The EB approach may also convey information about DIF stability in a more useful way by representing the state of knowledge about an item's DIF status as probabilistic.
Estimating Average Domain Scores
Authors: Pommerich Mary , Nicewander W. Alan , Hanson Bradley A.
Published: September 2005

A simulation study was performed to determine whether a group's average percent correct in a content domain could be accurately estimated for groups taking a single test form and not the entire domain of items. Six Item Response Theory based domain score estimation methods were evaluated, under conditions of few items per content area perform taken, small domains, and small group sizes. The methods used item responses to a single form taken to estimate examinee or group ability; domain scores were then computed using the ability estimates and domain item characteristics. The IRT‐based domain score estimates typically showed greater accuracy and greater consistency across forms taken than observed performance on the form taken. For the smallest group size and least number of items taken, the accuracy of most IRT‐based estimates was questionable; however, a procedure that operates on an estimated distribution of group ability showed promise under most conditions.
Multiple Objective Test Assembly Problems
Authors: Veldkamp Bernard P.
Published: September 2005

Mathematical programming techniques for optimal test assembly are discussed. Most methods optimize a single objective: for instance, the amount of information in a test, subject to a number of constraints. However, some test assembly problems have multiple objectives. A recent example in the literature is the problem of assembling test that measure multiple traits, where the amount of information in the test about each different trait has to be maximized. The present paper proposes methods appropriate for solving multiple objective test assembly problems. An overview of multiple objective optimization methods is given. The impact of the method on the optimality of the solution is shown and the appropriateness of the methods is discussed. The methods are illustrated using an empirical example of a test assembly problem for a two‐dimensional mathematics item pool.
An Investigation of the Power of the Likelihood Ratio Goodness‐of‐Fit Statistic in Detecting Differential Item Functioning
Authors: Ankenmann Robert D. , Witt Elizabeth A. , Dunbar Stephen B.
Published: September 2005

The purpose of this study was to investigate the power and Type I error rate of the likelihood ratio goodness‐of‐fit (LR) statistic in detecting differential item functioning (DIF) under Samejima's (1969, 1972) graded response model. A multiple‐replication Monte Carlo study was utilized in which DIF was modeled in simulated data sets which were then calibrated with MULTILOG (Thissen, 1991) using hierarchically nested item response models. In addition, the power and Type I error rate of the Mantel (1963) approach for detecting DIF in ordered response categories were investigated using the same simulated data, for comparative purposes. The power of both the Mantel and LR procedures was affected by sample size, as expected. The LR procedure lacked the power to consistently detect DIF when it existed in reference/focal groups with sample sizes as small as 500/500. The Mantel procedure maintained control of its Type I error rate and was more powerful than the LR procedure when the comparison group ability distributions were identical and there was a constant DIF pattern. On the other hand, the Mantel procedure lost control of its Type I error rate, whereas the LR procedure did not, when the comparison groups differed in mean ability; and the LR procedure demonstrated a profound power advantage over the Mantel procedure under conditions of balanced DIF in which the comparison group ability distributions were identical. The choice and subsequent use of any procedure requires a thorough understanding of the power and Type I error rates of the procedure under varying conditions of DIF pattern, comparison group ability distributions.–or as a surrogate, observed score distributions–and item characteristics.
Using a New Statistical Model for Testlets to Score TOEFL
Authors: Wainer Howard , Wang Xiaohui
Published: September 2005

Standard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimulus (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ended examinations in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, & Du, 2000). This parameter, γ characterizes the amount of local dependence in a testlet.

We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters (discrimination (a), difficulty (b) and guessing (c)) with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over‐estimated when conditional independence was incorrectly assumed.
Multidimensional Linking: Four Practical Approaches
Authors: Oshima T. C. , Davey T. C. , Lee K.
Published: September 2005

This paper introduces and evaluates four practical multidimensional linking procedures that are based on the theoretical framework recently proposed by Davey, Oshima, and Lee (1996): (a) the Direct method, (b) the Equated Function method, (c) the Test Characteristic Function (TCF) method, and (d) the Item Characteristic Function (ICF) method. The evaluation was conducted using simulated data. As anticipated, the competing procedures yielded different linking parameter estimates. The TCF and ICF methods were found to be more stable and recovered the true linking parameters better than the other two methods. Furthermore, all procedures were found to be acceptable under almost any of the minimization criteria and offered dramatic improvement over not linking at all. It is recommended that the choice of a linking procedure should depend on the purpose of linking.
Toward Intelligent Fault Detection in Turbine Blades: Variational Vibration Models of Damaged Pinned-Pinned Beams
Authors: McAdams Daniel A. , Tumer Irem Y.
Published: September 2005

Inaccuracies in the modeling assumptions about the distributional characteristics of the monitored signatures have been shown to cause frequent false positives in vehicle monitoring systems for high-risk aerospace applications. To enable the development of robust fault detection methods, this work explores the deterministic as well as variational characteristics of failure signatures. Specifically, we explore the combined impact of crack damage and manufacturing variation on the vibrational characteristics of turbine blades modeled as pinned-pinned beams. The changes in the transverse vibration and associated eigenfrequencies of the beams are considered. Specifically, a complete variational beam vibration model is developed and presented that allows variations in geometry and material properties to be considered, with and without crack damage. To simplify variational simulation, separation of variables is used for fast simulations. This formulation is presented in detail. To establish a baseline of the effect of geometric variations on the system vibrational response, a complete numerical example is presented that includes damaged beams of ideal geometry and damaged beams with geometric variation. It is shown that changes in fault detection monitoring signals caused by geometric variation are small with those caused by damage and impending failure. Also, when combined, the impact of geometric variation and damage appear to be independent.
Prediction of Railway-Induced Ground Vibrations in Tunnels
Authors: Lai Carlo G. , Callerio Alberto , Faccioli Ezio , Morelli Vittorio , Romani Pietro
Published: September 2005

The authors of this paper present the results of a study concerned with the assessment of the vibrational impact induced by the passage of commuter trains running in a tunnel placed underground the city of Rome. Since the railway line is not yet operational, it was not possible to make a direct measurement of the ground vibrations induced by the railway traffic and the only way to make predictions was by means of numerical simulations. The numerical model developed for the analyses was calibrated using the results of a vibration measurement campaign purposely performed at the site using as a vibration source a sinusoidal vibration exciter operating in a frequency-controlled mode. The problem of modeling the vibrational impact induced by the passage of a train moving in a tunnel is rather complex because it requires the solution of a boundary value problem of three-dimensional elastodynamics in a generally heterogeneous, nonsimply connected continuum with a moving source. The subject is further complicated by the difficulties of modeling the source mechanism, which constitutes itself a challenge even in the case of railway lines running at the surface. At last, the assessment of the vibrational impact at a receiver placed inside a building (e.g., a human individual or a sensitive instrument) requires an evaluation of the role played by the structure in modifying the computed free-field ground motion. So far, few attempts have been made to model the whole vibration chain (from the source to the receiver) of railway-induced ground vibrations, with results that have been only moderately successful. The numerical simulations performed in this study were made by using a simplified numerical model aimed to capture the essence of the physical phenomena involved in the above vibration chain including the influence of the structural response as well as the dependence of the predicted vibration spectra on the train speed.
Durability studies of micro/nanoelectromechanical systems materials, coatings and lubricants at high sliding velocities (up to 10mm∕s) using a modified atomic force microscope
Authors: Tambe Nikhil S. , Bhushan Bharat
Published: September 2005

Most micro/nanoelectromechanical (MEMS/NEMS) devices and components operate at very high sliding velocities (of the order of tens of mm/s to few m/s). Micro/nanoscale tribology and mechanics of these devices is crucial for evaluating reliability and failure issues. Atomic force microscopy (AFM) studies to investigate potential materials/coatings for these devices have been rendered inadequate due to inherent limitations on the highest sliding velocities achievable with commercial AFMs. We have developed a technique to study nanotribological properties at high sliding velocities (up to 10mm∕s) by modifying the commercial AFM setup with a customized closed loop piezo stage for mounting samples. Durability of materials, silicon, poly(methylmethacrylate) (PMMA) and poly(dimethlysiloxane) (PDMS), diamond-like carbon (DLC) coating and lubricants such as self-assembled monolayer of hexadecanethiol (HDT) and perfluropolyethers Z-15 and Z-DOL used in MEMS/NEMS applications, is studied at various normal loads and sliding velocities. Wear mechanisms involved at high sliding velocities are discussed. The primary wear mechanisms are deformation of the contacting asperities due to impacts in the case of silicon; phase transformation from amorphous to low shear strength graphite for DLC; localized melting due to high frictional energy dissipation for PMMA and PDMS; and displacement or removal of lubricant molecules for HDT, Z-15, and Z-DOL.
Nanotribological characterization of fluoropolymer thin films for biomedical micro/nanoelectromechanical system applications
Authors: Lee Kang Kug , Bhushan Bharat , Hansford Derek
Published: September 2005

A vapor phase deposition system was designed to coat uniform, conformal and ultrathin coatings of fluoropolymer and fluorosilane thin films inside silicon nanochannels. Surface modifications using vapor phase deposition become increasingly important for biomedical micro/nanoelectromechanical system (BioMEMS/NEMS) applications and have advantages over liquid phase deposition since the vapor can permeate more efficiently into silicon nanochannels. In this article, we explore the use of vapor phase deposition to deposit ultrathin films of two fluoropolymers and a fluorosilane, identifying deposition parameters to optimize the process. The films are desirable to control the hydrophobicity of the surface and reduce or prevent undesired protein adsorption or cell interactions, which may cause detrimental effects to the performance of most BioMEMS/NEMS devices. The films were characterized by means of a contact angle analyzer for hydrophobicity and an ellipsometer for film thickness. Atomic force microscopy was used extensively to collect surface images, adhesive and frictional properties of these films, all of which play a very important role in characterizing uniform, conformal and ultrathin films on the surface.
Nanotribological effects of hair care products and environment on human hair using atomic force microscopy
Authors: LaTorre Carmen , Bhushan Bharat
Published: September 2005

Tribological properties are useful in the study of human hair and other biological materials. Major sources of investigation for conditioner treated hair includes localization of conditioner, mechanisms related to changes in surface roughness, friction, and adhesion on the nanoscale due to conditioner agents, and how the products change the microstructure of the cuticle. The paper presents nanotribological studies investigating surface roughness, friction, and adhesion using atomic force/friction force microscopy (AFM/FFM). Test samples include virgin and chemically damaged hair, both with and without commercial conditioner treatment, as well as chemically damaged hair with experimental conditioner treatments. Friction force mapping provides insight into the localized change in friction caused by the application of hair care materials. Adhesive force maps to study adhesion on the cuticle surface provide information about localization and distribution of conditioner as well. A discussion is presented on these properties of hair as a function of relative humidity, temperature, durability, and conditioning treatments.