bluuebunny/crossref_metadata_2025 huggingface dataset:
The Relative Appropriateness of Eight Measurement Models for Analyzing Scores from Tests Composed of Testlets
Authors: Lee Guemin , Dunbar Stephen B. , Frisbie David A.
Published: July 2003

It has been shown that fundamental assumptions associated with conventional one-factor measurement models are frequently violated in analyses of scores from a test composed of testlets. Eight different measurement models were conceptualized for this kind of situation, and the goodness of fit of each model was examined. Conventional essentially tauequivalent and congeneric models present worse model fit to data and overestimate the reliability when testlets are involved. The one-factor congeneric model with correlated error specifications seems to be the best measurement model for a test composed of testlets if dichotomously scored items are used as the unit of analysis. However, in estimating score reliability for tests composed of testlets, the one-factor essentially tauequivalent model with correlated error specifications also provides good estimates. Measurement models using passage (testlet) scores would be alternatives for analyzing scores from tests composed of testlets when passage (testlet) scores are used as the unit of analysis.
The Self-Description Questionnaire II and Gifted Students: Another Look at Plucker, Taylor, Callahan, and Tomchin’s (1997) “Mirror, Mirror on the Wall”
Authors: Marsh Herbert W. , Plucker Jonathan A. , Stocking Vicki B.
Published: July 2003

Plucker, Taylor, Callahan, and Tomchin concluded that confirmatory factor analyses (CFA) of responses to the Self-Description Questionnaire (SDQ) II provided only limited support for construct validity with gifted students because of the modest fit of their CFA models. They acknowledged, however, that potential complications (e.g., their CFA models, treatment of missing data, and highly nonnormal data) meant that their conclusions may be premature. Here, the authors analyzed data from their original study ( N = 376) and new data from another gifted-student program ( N = 837), exploring alternative approaches to missing data and data normalization. The a priori model based on the design of the SDQII resulted in a good fit to the data for the total group, and the factor solution was invariant across the two samples of gifted students. The results strongly support the construct validity of responses to the SDQII by gifted adolescents.
Detecting Dissimulation in Personality Test Scores: A Comparison between Person-Fit Indices and Detection Scales
Authors: Ferrando Pere J. , Chico Eliseo
Published: July 2003

The present study examined whether an internal procedure for assessing the scalability of the response patterns, based on item response theory (IRT), can detect deliberate dissimulation (faking good) in the Extraversion, Neuroticism, and Psychoticism scale scores of the Eysenck Personality Questionnaire Revised. The procedure is compared to the traditional approaches, which use the Lie and the Social Desirability (SD) scales. A data set was analyzed in which participants were either administered the measures in standard conditions or given special instructions to fake good. The results showed that the IRT-based measures were not powerful enough to detect dissimulation, whereas the Lie and SD scales performed much better.
A Comparative Analysis of Contemporary Achievement Goal Orientation Measures
Authors: Jagacinski Carolyn M. , Duda Joan L.
Published: July 2003

As research on achievement goals has increased, so has the number of different measures of goal orientations. In this article, the authors examined three popular measures of task and ego achievement goal orientations in terms of factorial and construct validity, internal consistency reliability, and distributional characteristics. College students completed the Ability and Task Orientation scales from the Patterns of Adaptive Learning Survey (PALS), Task and Ego Motivational Orientation Scales, and the General Learning and Performance Goal Orientations Scales. The first two sets of scales assess goal orientations for schoolwork, whereas the third is more general. To assess construct validity, the scales were correlated with the Jackson Achievement Orientation Scale, beliefs concerning the role of effort and ability in success, and interest/enjoyment of and boredom with schoolwork. The PALS scales generally faired the best in terms of distributional characteristics, factorial validity, and construct validity.
Development of the Domain Identification Measure: A Tool for Investigating Stereotype Threat Effects
Authors: Smith Jessi L. , White Paul H.
Published: July 2003

The present study examined the psychometric properties of an individual difference measure of identification within the mathematics and English domains (which may be substituted for any domain of interest to the researcher). Factor analytic results substantiated the presence of English and Mathematics subscales, which yielded scores that were internally consistent and stable over time. Predicted math test performance differences and gender differences were found as a function of identification, providing support for the reliability and validity of scores on the scale. Use of the scale may aid in the understanding of test performance differences at both the global and individual level, especially for individuals susceptible to stereotype threat.
Treatments of Effect Sizes and Statistical Significance Tests in Textbooks
Authors: Capraro Robert M. , Capraro Mary Margaret
Published: July 2003

The dialog surrounding effect sizes and statistical significance tests often places the two ideas into separate camps amid controversy. In light of recommendations by the Task Force on Statistical Inference and the fifth edition of the American Psychological Association Publication Manual calling for the reporting of effect sizes, a review of treatments of effect sizes in textbooks may be quite timely. This study reviews textbooks published since 1995 and as regards treatments of effect sizes and statistical significance tests. Of The textbooks examined, every textbook ( n = 89) included the topic of statistical significance testing (2,248 pages), whereas only a little more than two thirds of the textbooks ( n = 60) included information on effect sizes (789 pages).
Reliability: Arguments for Multiple Perspectives and Potential Problems with Generalization across Studies
Authors: Dimitrov Dimiter M.
Published: July 2003

The present article addresses reliability issues in light of recent studies and debates focused on psychometrics versus datametrics terminology and reliability generalization (RG) introduced by Vacha-Haase. The purpose here was not to moderate arguments presented in these debates but to discuss multiple perspectives on score reliability and how they may affect research practice, editorial policies, and RG across studies. Issues of classical error variance and reliability are discussed across models of classical test theory, generalizability theory, and item response theory. Potential problems with RG across studies are discussed in relation to different types of reliability, different test forms, different number of items, misspecifications, and confounding independent variables in a single RG analysis.
Psychometric Properties of the Multidimensional-Multiattributional Causality Scale
Authors: Hamilton Richard J. , Akhter Selina
Published: July 2003

This study investigated the construct validity of the dimensions of Multidimensional-Multiattributional Causality Scale (MMCS) based on Winner’s attribution model in both achievement and affiliation goal domains, using confirmatory factor analysis. Participants were 172 students enrolled in B.Ed and in Diploma Education courses at the University of Auckland. The results indicated that in the affiliation domain, all but stability models exhibited adequate fit, whereas in the achievement domain, only externality, stability, and controllability models showed good fit. This provides partial evidence that the MMCS can be used as a goal-specific measure rather than a general measure.
A Meta-Analytic Review of the MaCandrew Alcoholism Scale
Authors: Cooper-Hakim Amy , Viswesvaran Chockalingam
Published: July 2003

Using a meta-analytic approach, the authors examined the predictive validity of scores on the MacAndrew Alcoholism Scale. Across 161 studies involving a total sample of 31,320 individuals, the proportion correct classification was .70. When a cut score of 24 was used across 63 studies involving a total sample of 11,113 individuals, the proportion correct classification was .72. The proportion correct classifications for the different cut scores were cumulated separately, and the functional graph suggested that the best discrimination (.76) occurred at a cut score of 27. It is argued that given the potential personality basis for these predictions, use of continuous measures rather than cut scores is preferable.
Factorial and Criterion Validity of Scores of a Measure of Belonging in Youth Development Programs
Authors: Anderson-Butcher Dawn , Conroy David E.
Published: July 2003

Many youth development programs, including the Boys & Girls Clubs ofAmerica, feature belonging as a central piece in their theories ofchange. From a psychometric perspective, little is known about measures ofbelonging. This research examined the factorial validity, internal consistency, and predictive validity of scores from one measure of belonging to an after-school youth development program. Confirmatory factor analysis yielded a five-item measure from a calibration analysis that demonstrated “tight” cross validity in a cross-validation sample as well as factorial invariance between females and males. Internal consistency estimates for this 5-item scale exceeded .90 in both samples. Belonging scores were positively related to actual program attendance over a 6-month period, self-reported attendance in the last week, and protective factors found in communities. Belonging scores were moderately and negatively related to community-based risk factors.
Using the Imaginary Audience Scale as a Measure of Social Anxiety in Young Adults
Authors: Kelly Kristine M. , Jones Warren H. , Adams Jeffrey M.
Published: July 2003

Two studies explored the utility and meaning of the imaginary audience phenomenon among college students. Study 1 examined the psychometric properties of the Imaginary Audience Scale used for college student samples. The second study correlated imaginary audience scores with a global measure of personality and also compared the accuracy of selfmeasures, social anxiety measures, and a measure of formal reasoning to predict imaginary audience scores. Results indicated that Imaginary Audience Scale scores met conventional standards of validity and reliability in these samples of postadolescent individuals. As expected, imaginary audience behavior was related to measures of social anxiety, the self, and personality. Furthermore, imaginary audience scores were more strongly related to these measures than to an index of formal reasoning. These results suggest that imaginary audience experiences that persist into late adolescence and early adulthood may have more to do with social anxiety than with cognitive development.
Trait Ratings for Automated Essay Grading
Authors: Shermis Mark D. , Koch Chantal Mees , Page Ellis B. , Keith Timothy Z. , Harrington Susanmarie
Published: July 2003

This study employed an automated grader to evaluate essays, both holistically and with the rating of traits (content, organization, style, mechanics, and creativity) for Webbased student essays serving as placement tests at a large Midwestern university. The authors report the results of two combined experiments, based on random selection from 1,193 essays. In the first experiment, the essays of 807 students were used to create statistical predictions for the essay-grading software. In the second experiment, the ratings from a separate, random sample of 386 essays were used to compare the ratings of six human judges against those generated by the computer. The interjudge correlation of the human raters alone was r = .71. But the interrater reliability of all six judges in combination with computer scoring reached .83. The essay-grading software was an efficient means for evaluating the essays, with a capacity for grading approximately six documents every second. Other potential feedback measures for use in writing courses are also discussed.
Developing an Interpretation of Item Parameters for Personality Items: Content Correlates of Parameter Estimates
Authors: Zickar Michael J. , Ury Karen L.
Published: July 2003

The research in this article attempted to relate content features of personality items to item parameter estimates from Muraki’s partial credit model. Goldberg’s Adjective Checklist was administered to 329 participants to calibrate item parameters. These parameters were then related to item ratings of social desirability, item subtlety, frequency of misunderstanding, and word usage frequency indexes. Subtle items tended to have lower discrimination parameter estimates. There was also an interaction between subtlety and social desirability such that items low in subtlety and high in social desirability had lower location parameters. Implications for future psychometric research in the personality domain are discussed.
Two-Stage Equating in Differential Item Functioning Detection under the Graded Response Model with the Raju Area Measures and the Lord Statistic
Authors: Hidalgo-Montesinos Maria Dolores , Lopez-Pina Jose Antonio
Published: July 2003

The differential item functioning (DIF) detection in polytomous response items is an area of study of recent interest. Cohen and colleagues proposed a polytomous extension of the Lord statistic and the Raju exact area measures, when the items fit the graded response model. This study examined the effects of test purification in detecting DIF by means of polytomous extensions of the Raju area measures and the Lord statistic. The factors manipulated were percentage of DIF items in the test (5%, 10%, and 20%), amount of DIF (0.2, 0.4, and 0.8), sample size (250, 500, and 1,000) and test purification (noniterative versus two-stage). The results of this study suggest that the use of the two-stage equating procedure with Z(SA) and χ2-LORD reduces the percentage of false positives and improves the detection of DIF.
Construct Validation of a Revised Strain-Free Negative Affectivity Scale
Authors: Fortunato Vincent J. , Goldblatt Andrew M.
Published: July 2003

Based on research demonstrating that many extant measures of negative affectivity (NA) are contaminated with strain content, Fortunato and Stone-Romero developed the Strain-Free Negative Affectivity Scale (SFNA). However, because of several psychometric concerns involving scores on the original scale, a revision of the SFNA was undertaken. The SFNA-Revised (R) was administered to two samples of participants, along with instruments purportedly measuring neuroticism, negative emotionality, trait negative affect, affect intensity, affective disposition, extroversion, and positive affectivity. Estimates of reliability and factor analytic findings indicate that scores on the SFNA-R exhibited better psychometric properties than scores on the original scale. In addition, evidence for the construct validity of scores on the SFNA-R is presented. Based on the above, the authors conclude that the SFNA-R is a worthy alternative to extant measures of NA.
Development and Validation of the Factors Influencing Pursuit of Higher Education Questionnaire
Authors: Harris Sandra M. , Halpin Glennelle
Published: July 2003

This article addresses the development and validation of the Factors Influencing Pursuit of Higher Education (FIPHE) Questionnaire, a 92-item self-report measure that investigates factors that influence individuals to pursue higher education. Reliability estimates for the nine FIPHE scale scores ranged from .66 to .90. An exploratory factor analysis was conducted on data from a sample ( N = 509) of college students enrolled in two south-eastern universities. The data were subjected to principal components analysis with varimax rotation. Although the researchers hypothesized 10 scales for the questionnaire, the results revealed that a nine-factor solution produced the most interpretable factor patterns. The nine-factor solution accounted for 43% of the common variance. Limitations of the research and implications for future research are discussed.
A Theoretical and Empirical Analysis of the Measurement of Collective Efficacy: The Development of a Short Form
Authors: Goddard Roger
Published: July 2003

The present study reports on the development of a 12-item Likert-type measure of collective efficacy in schools. Designed to assess the extent to which a faculty believes in its conjoint capability to positively influence student learning, the scale is based on a social cognitive model that posits perceptions of collective efficacy develop from the cognitive processing of group members. Faculty perceptions of group competence and the level of difficulty inherent in the educational task faced by the school are tapped by the scale. The 12-item scale is more theoretically pure than an earlier 21-item scale to which the 12-item scale is compared. The internal consistency of scores on the 12-item scale is tested with Cronbach’s alpha, and a test of predictive validity using multilevel modeling is reported.
Psychometric Properties of Shortened Versions of the Automatic Thoughts Questionnaire
Authors: Netemeyer Richard G. , Williamson Donald A. , Burton Scot , Biswas Dipayan , Jindal Supriya , Landreth Stacy , Mills Gregory , Primeaux Sonya
Published: July 2003

Measures of depression are increasingly being used as outcomes or predictors by organizational and consumer psychologists. One such measure is the Automatic Thoughts Questionnaire (ATQ). However, questions about the 30-item ATQ’s factor structure and its length for use in survey research remain. The authors offer 15-and 8-item shortened versions of the ATQ. Two samples ( n = 434 and n = 419) were used to derive the reduced versions. A single factor was found to underlie both reduced versions, with scores on this factor yielding strong estimates of internal consistency and nomological validity. Two more cross-validation samples ( n = 163 and n = 91) also showed support for the 15-and 8-item versions. Overall, results suggest that these reduced-item versions of the ATQ are useful alternatives to measuring cognitions associated with depression.
Psychometric Properties of Scores on the Preliminary Amharic Version of the State-Trait Anxiety Inventory in Ethiopia
Authors: Mulatu Mesfin Samuel
Published: July 2003

Psychometric properties of scores on the Amharic version of the State-Trait Anxiety Inventory (STAI-A) were explored with 551 students in Ethiopia. Results show strong correlations ( r[62] ≥ .89) between the corresponding scores on State Anxiety (SA) and Trait Anxiety (TA) scales of the STAI-A and the original STAI. High levels of internal consistency reliabilities (αs ≥ .85) and moderately high median item-remainder correlations ( rs ≥ .47) were found for scores on the two scales of STAI-A. As predicted, SA scores were higher on a hypothetical exam condition than on a normal condition. SA scores were also more strongly correlated with life event stress scores than were TA scores. TA scores, in turn, were more strongly related to global psychopathology scores than were SA scores. These findings suggest that scores on STAI-A have acceptable reliability and validity and that STAI-A can be used for clinical and research purposes in Ethiopia.
Developing a Measurement Model for Coping Research in Early Adolescence
Authors: Cunningham Everarda Gertruda
Published: July 2003

The purpose of this study was the development of a measurement model for use in coping research in early adolescence. The results from a sample of 359 fifth-and sixth-grade children provided support for measures of children’s coping. Two higher order coping factors, productive and nonproductive coping, were found by using domain-representative parcels of coping strategies in confirmatory factor analyses. Parceling of random items found support for the unidimensional constructs of coping efficacy and family support. A four-factor measurement model incorporating coping efficacy, perceived family support, and the two higher order factors of productive and nonproductive coping behaviors was established. Implications of the finding of relative independence of the productive and nonproductive coping factors are discussed in terms of children’s coping research.
Factor Structure of the Wechsler Intelligence Scale for Children–Third Edition among Gifted Students
Authors: Watkins Marley W. , Greenawalt Chris G. , Marcell Catherine M.
Published: July 2003

Factor analysis was applied to the Wechsler Intelligence Scale for Children–Third Edition (WISC-III) scores of 505 gifted students to evaluate the construct validity of the WISC-III with this population. Multiple criteria were used to determine the number of factors to retain for principal axis extraction. A two-factor solution that roughly mirrored the verbal comprehension and perceptual organization factors of the WISC-III normative sample appeared to be most supportable. Arithmetic, Picture Arrangement, and Coding subtests failed to contribute to this solution. These results are consistent with the hypothesis that subtests that emphasize speed of responding are not valid for gifted children and suggest that an alternative WISC-III composite score, the General Ability Index, may be a better summary of ability for gifted students.
Confidence Intervals for Effect Sizes in Analysis of Variance
Authors: Bird Kevin D.
Published: July 2003

Although confidence interval procedures for analysis of variance (ANOVA) have been available for some time, they are not well known and are often difficult to implement with statistical packages. This article discusses procedures for constructing individual and simultaneous confidence intervals on contrasts on parameters of a number of fixed-effects ANOVA models, including multivariate analysis of variance (MANOVA) models for the analysis of repeated measures data. Examples show how these procedures can be implemented with accessible software. Confidence interval inference on parameters of random-effects models is also discussed.
A History of Effect Size Indices
Authors: Huberty Carl J
Published: July 2003

Depending on how one interprets what an effect size index is, it may be claimed that its history started around 1940, or about 100 years prior to that. An attempt is made in this article to trace histories of a variety of effect size indices. Effect size bases discussed pertain to (a) relationship, (b) group differences, and (c) group overlap. Multivariable as well as univariate indices are considered in reviewing the histories.
Correction for Bias in Estimating Effect Sizes
Authors: Roberts J. Kyle , Henson Robin K.
Published: July 2003

Some authors debate whether effect sizes should be reported (a) for all null hypothesis tests, even non–statistically significant ones, or (b) only after a finding is first determined to be statistically significant. The decision to report and interpret small effects may partially depend on the amount of bias in the effect size measure used. Based on the recognitions that variance-accounted-for effect statistics are positively biased and that standardized difference effect sizes such as Cohen’s d can be converted into r2 metrics and vice versa, the authors considered that d also may be biased. The authors therefore explored the amount of bias in Cohen’s d across a series of simulated study conditions. Results from their simulations indicated relatively no bias (close to zero) in Cohen’s d across all study conditions.
Confirmatory Factor Analysis of an Instrument Designed to Measure Affective and Cognitive Arousal
Authors: Williams Thomas O. , Eaves Ronald C. , Cox Cynthia
Published: July 2003

The Visual Similes Test II is a research instrument designed to measure the construct of arousal. The test consists of two forms: an affective form and a cognitive form. Previous research has indicated that a strong negative relationship exists between affective arousal and behavior problems; likewise, cognitive arousal has a strong positive relationship with measured intelligence. This investigation examined the rationale behind the development of the Visual Similes Test II. In addition, the construct validity of scores produced by the Visual Similes Test II was examined with a sample of 216 children ranging in age from 10 to 12 years through confirmatory factor analysis using the AMOS program. The results of the analysis supported previous research findings and the hypothesis that the Visual Similes Test II measures the concepts of affective arousal and cognitive arousal.
Validity and Reliability of Scores on the Reduced Emotional Intensity Scale
Authors: Geuens Maggie , de Pelsmacker Patrick
Published: July 2003

The purpose of this study was to investigate the underlying factor structure of the 30-item Emotional Intensity Scale (EIS) and to construct a reduced EIS (EIS-R). The psychometric characteristics of the scales were examined from three different samples: 204 employees of the University of Antwerp, a subset of 106 of the first sample who cooperated in a retest, and 510 men and women representative of the Belgian population. The original EIS does not seem to possess an adequate factor structure. Confirmatory factor analysis on a reduced scale of 17 items indicates that two factors underlie the EIS-R: a positive and a negative emotions factor. Furthermore, scores on the EIS-R were shown to have adequate reliability and validity.
The Development of the Self-Regulation of Withholding Negative Emotions Questionnaire
Authors: Kim Youngmee , Deci Edward L. , Zuckerman Miron
Published: July 2003

Based on the Self-Determination Theory, a questionnaire was developed to measure individual differences in the Self-Regulation of Withholding Negative Emotions (SRWNE). Measurement reliability and validity concerning the scale were examined in three studies. Results in Study 1 demonstrated the distinctiveness of the SRWNE from emotional regulation measures, suggesting that the SRWNE may be appropriate to measure styles of self-regulation and to clarify the negative affect-health relation. In Study 2, test-retest reliability of scores on the SRWNE subscales was examined as was validity of the SRWNE with respect to coping strategies and health. The SRWNE was related to self-reports of health and may be relevant for predicting how people cope with stress. Study 3 compared a Korean sample with the U.S. sample in Study 2 and suggested construct comparability of the SRWNE across cultures and genders.
The Score Equivalence of Paper-and-Pencil and Computerized Versions of a Speeded Test of Reading Comprehension
Authors: Pomplun Mark , Frey Sharon , Becker Douglas F.
Published: July 2003

This study investigated the equivalence of scores from computerized and paper-and-pencil versions of a reading placement test. Concerns about score equivalence on the computerized versions were warranted because of the speeded nature of the paper-and-pencil version and differences in text delivery and response modes. The results indicated that both forms of the computerized versions produced higher vocabulary scores than the paper-and-pencil format and one form also had higher comprehension and total scores on the computerized version. These difficulty differences, especially for the vocabulary scores, appeared related to the differences in response speed associated with use of a mouse to record responses in contrast to a pencil and answer sheet. Scale scores for the computerized versions had similar predictive power for course placement as paper-and-pencil scores. However, because these results were based on students from only seven institutions, additional studies are needed to investigate the comparability of these versions.
Multitrait-Multimethod Analyses of Scores on a Chinese Version of the Dimensions of Self-Concept Scale
Authors: Huang Chiungjung , Michael William B.
Published: July 2003

The purpose of this study was to examine the construct validity of scores on a Chinese version of an academic self-concept scale as measured by self-, peer, and parent ratings. The multitrait-multimethod approach, general confirmatory factor analysis models, and correlated uniqueness models were utilized. Overall, the results from three analyses supported the convergent validity as well as discriminant validity of scores and the presence of a method effect. However, the evidence for convergent validity, discriminant validity, and method effect associated with scores was somewhat inconsistent. Alternative analysis procedures are recommended to provide safeguards to construct validation.
Kolb’s Learning Style Instrument: Sensitive to Gender
Authors: Brew Christine R.
Published: July 2003

Kolb’s experiential learning theory, developed in the 1970s, continues to be influential in the educational and management fields. To complement his theory, Kolb developed a learning style preference instrument (LSI) that has been continually hampered by poor psychometric performance. Whether these problems are due to poorly differentiated theory and/or instrument error has been the focus of debate. The principal finding reported in the present study is that the LSI, as modified by Kolb in 1985, is gender sensitive for an Australian cohort of 1st-year university students. For females, the results of a factor analysis were consistent with Kolb’s theory. For males, the results were consistent with previous studies that found the instrument yielded scores with poor construct validity.
Reliability Generalization: Moving toward Improved Understanding and Use of Score Reliability
Authors: Vacha-Haase Tammi , Henson Robin K. , Caruso John C.
Published: July 2003

Reliability generalization (RG) is a measurement meta-analytic method used to explore the variability in score reliability estimates and to characterize the possible sources of this variance. This article briefly summarizes some RG considerations. Included is a description of how reliability confidence intervals might be portrayed graphically. The article includes tabulations across various RG studies, including how frequently authors (a) report score reliabilities for their own data, (b) conduct reliability induction, or (c) do not even mention reliability.
A Reliability Generalization Study of the Marlowe-Crowne Social Desirability Scale
Authors: Beretvas S. Natasha , Meyers Jason L. , Leite Walter L.
Published: July 2003

A reliability generalization (RG) study was conducted for the Marlowe-Crowne Social Desirability Scale (MCSDS). The MCSDS is the most commonly used tool designed to assess social desirability bias (SDB). Several short forms, consisting of items from the original 33-item version, are in use by researchers investigating the potential for SDB in responses to other scales. These forms have been used to measure a wide array of populations. Using a mixed-effects model analysis, the predicted score reliability for male adolescents was.53 and the reliability for men’sresponseswaslower than that for women’s. Suggestions are made concerning the necessity for further psychometric evaluations of the MCSDS.
A Reliability Generalization Study of Select Measures of Adult Attachment Style
Authors: Reese Robert J. , Kieffer Kevin M. , Briggs Barbara K.
Published: July 2003

The concept of attachment has been discussed and debated in the psychological literature for quite some time. More recently, the concept of attachment has been extended to adults, and several measures that purport to measure attachment style have been developed. The purpose of the present study was to provide a reliability generalization study of five of the most prominent adult attachment style measures. Reliability generalization, a relatively new meta-analytic reliability procedure, was used to (a) identify the typical reliability of scores across instruments and studies and (b) examine sources of measurement error across instruments and studies. Results from this investigation of 154 previously published research studies indicated that the average score reliability across studies varied considerably across instruments and subscales. Implications for the use of self-report measures of adult attachment style are offered.
Reliability Generalization of Working Alliance Inventory Scale Scores
Authors: Hanson William E. , Curry Kyle T. , Bandalos Deborah L.
Published: July 2003

Reliability generalization (RG) was used to study five versions of the Working Alliance Inventory (WAI), including scores from 12 different scales. Sixty-seven internal consistency estimates, six interrater reliability estimates, and four study characteristics were analyzed. In general, reliability estimates of WAI scale scores appear to be robust. Mean reliability estimates ranged, in this sample of studies, from .79 to .97, with a modal estimate of .92. Variability in reliability estimates was, based on simple bivariate correlations, associated with client and therapist sample size for WAI total scores (observer version). Implications for measuring alliance using the WAI and conducting future RG studies on psychotherapy process measures are discussed.
Reliability Generalization of the Life Satisfaction Index
Authors: Wallace K. A. , Wheeler A. J.
Published: July 2003

The purpose of the present study was to examine score reliability for a measure of life satisfaction (Life Satisfaction Index [LSI]). This reliability generalization comprised a search of 157 journal articles, which resulted in the inclusion of a total of 34 samples. Results revealed an average reliability of .79 ( SD = .10, median = .79). Bivariate correlational analyses revealed no relationships between score reliability and various sample characteristics, including sample size, number of items, mean age, standard deviation of age, proportion female, mean LSI score, and standard deviation of LSI scores. No significant differences in score reliability were found by language of administration or sample type. These analyses provide evidence for adequate reliability of LSI scores across a variety of sample characteristics; however, they must be interpreted with caution, given the small sample size. In addition, this study documents the poor reporting of psychometric properties in the LSI literature.
An Examination of the Reliability of Scores from Zuckerman’s Sensation Seeking Scales, Form V
Authors: Deditius-Island Heide K. , Caruso John C.
Published: July 2003

A reliability generalization (RG) was conducted on Zuckerman’s Sensation Seeking Scale, Form V (SSS-V). Two hundred and forty-four empirical articles on the SSS-V were reviewed spanning a 20-year period but fewer than 9% provided useable reliability estimates. One hundred and thirteen reliability coefficients were collected from 21 published studies for the five scales and subscales of SSS-V. The reliability of scores was marginal for four of the five scales and low for the Boredom Susceptibility scale (median = .61). Mean age of the respondents had a statistically significantly relationship with score reliability, with older respondents’scores having higher reliability for three of the five scales. Unfortunately, the infrequent reporting of basic descriptive information hindered further examination of factors that may have been related to score reliability. Although certain descriptive information (e.g., score reliability estimates, score SDs) may not bethefocus of substantivestudies, theaccumulation of knowledgeover time requires more complete data reporting in all research, even nonmeasurement studies.
Stability of the Reliability of LibQual+™ Scores a Reliability Generalization Meta-Analysis Study
Authors: Thompson Bruce , Cook Colleen
Published: July 2003

The present study reports a reliability generalization (RG) meta-analysis of subscale and total scale scores on the Web-administered LibQUAL+™ protocol. Data were provided by 18,161 participantsfrom 43 universitiesin the United Statesand Canada. Results indicate that score reliabilities were remarkably invariant across campuses and different user groups.
Psychological Acculturation: Development of A New Measure for Puerto Ricans on the U.S. Mainland
Authors: Tropp Linda R. , Erkut Sumru , Coll Cynthia García , Alarcón Odette , García Heidie A. Vázquez
Published: July 2003

Most instruments designed to measure acculturation have relied on specific cultural behaviors and preferences as primary indicators of acculturation. In contrast, feelings of belonging and emotional attachment to cultural communities have not been widely used. The Psychological Acculturation Scale (PAS) was developed to assess acculturation from a phenomenological perspective, with items pertaining to the individual’s sense of psychological attachment to and belonging within the Anglo-American and Latino/Hispanic cultures. Responses from samples of bilingual individuals and Puerto Rican adolescents and adults are used to establish a high degree of measurement equivalence across the Spanish and English versions of the scale along with high levels of internal consistency and construct validity. The usefulness of the PAS and the importance of studying acculturation from a phenomenological perspective are discussed.
Meta-Analyses of Fakability Estimates: Implications for Personality Measurement
Authors: Viswesvaran Chockalingam , Ones Deniz S.
Published: July 2003

The authors examined whether individuals can fake their responses to a personality inventory if instructed to do so. Between-subjects and within-subject designs were metaanalyzed separately. Across 51 studies, fakability did not vary by personality dimension; all the Big Five factors were equally fakable. Faking produced the largest distortions in social desirability scales. Instructions to fake good produced lower effect sizes compared with instructions to fake bad. Comparing meta-analytic results from within-subjects and between-subjects designs, we conclude, based on statistical and methodological considerations, that within-subjects designs produce more accurate estimates. Between-subjects designs may distort estimates due to Subject × Treatment interactions and low statistical power.
Validity Issues in the Likert and Thurstone Approaches to Attitude Measurement
Authors: Roberts James S. , Laughlin James E. , Wedell Douglas H.
Published: July 2003

This article highlights the theoretical differences between the Likert and Thurstone approaches to attitude measurement and demonstrates how such differences can lead to discrepant attitude estimates for individuals with the most extreme opinions. Both simulated data and real data on attitude toward abortion are used to demonstrate this discrepancy. The results suggest that attitude researchers should, at the very least, devote more attention to the empirical response characteristics of items on a Likert attitude questionnaire. At most, these results suggest that other methods, such as the Thurstone technique or one of its recently developed item response theory counterparts, should be used to derive attitude estimates from disagree-agree responses.
An Empirical Comparison of Three-and Four-Choice Items and Tests: Susceptibility to Testwiseness and Internal Consistency Reliability
Authors: Rogers W. Todd , Harley Dwight
Published: July 2003

Theoretical and test simulation work reveals that under the knowledge-or-randomguessing assumption, three-option item tests are at least as good as four-option item tests in terms of item discrimination and internal consistency. Of concern, however, is the finding that multiple-choice items may be susceptible to testwiseness, thereby contradicting the random-guessing assumption. Both item-level and test-level characteristics were examined for items included in a high stakes school-leaving mathematics examination. As expected, the influence of testwiseness is lessened when three-option items are used instead of four-option items. Differences and nondifferences between the psychometric characteristics of the three-option and four-option test forms tend to agree with the findings of earlier studies: Tests consisting of three-option items are at least equivalent to tests composed of four options in terms of internal consistency score reliability, difficulty is inversely related to the number of options, and the findings for item discrimination are not conclusive.
Defining and Measuring Employees’ Identification with Their Work Groups
Authors: Riordan Christine M. , Weatherly Elizabeth W.
Published: July 2003

The objective of this study was to develop a conceptually and methodologically sound measure of employee identification with the work group. A three-phase analysis approach was used. First, a content analysis was conducted with subject matter experts (SMEs) in the field of organizational behavior and psychology. Second, an exploratory factor analysis of the factor structure was conducted using a sample of employees from a credit union ( N = 140). Finally, confirmatory analyses using LISREL 8 were conducted with a sample of employees derived from four insurance organizations ( N = 309). The exploratory and confirmatory factor analyses supported the factor structure of the identification measure and the scale scores showed acceptable levels of internal consistency in both samples ([.alpha] = .78; [.alpha] = .79, respectively). We also demonstrated that the construct of work group identification is distinct from but related to both work group cohesiveness and work group communication.
Further Factorial Validity Assessment of Scores on the Quick Discrimination Index (Qdi)
Authors: Utsey Shawn O. , Ponterotto Joseph G.
Published: July 2003

This study examined the factorial validity of scores on the Quick Discrimination Index (QDI) across three geographically dispersed samples. A multisample LISREL analysis, incorporating Bagozzi and Heatherton’s recommended aggregate procedure, indicated that the QDI three-factor oblique model was relatively stable across three geographically divergent samples. Coefficient alphas across factors and samples were satisfactory. The Lix Readability Index for the QDI items fell at the ninth-grade level.
The Development of a University Version of the Charles F. Kettering Climate Scale
Authors: Johnson William L. , Johnson Annabel M. , Kranch Douglas A. , Zimmerman Kurt J.
Published: July 2003

This study focused on the development and multivariate analysis of a university version of the Charles F. Kettering Climate Scale, a popular measure that is widely used to gather data for climate assessment and educational planning. A total of 707 university students at a campus in the southwestern United States completed the scale. A second-order analysis shows areas of generalization across the primary components.
The Development and Validation of Scores on a Measure of Teachers’ Attitudes toward Teaching Physical Activity and Fitness
Authors: Kulinna Pamela Hodges , Silverman Stephen
Published: July 2003

The purpose of this study was to develop an instrument to examine teachers’ attitudes toward teaching physical activity and fitness. The study was conducted in multiple phases, including a preliminary study, a content validity study, and a reliability and validity study. Participants for the content validity study were 28 experts in sport pedagogy. They were sent items from the attitude instrument and asked to organize them into the instruments’ four domain areas. Participants for the reliability and validity study were 253 teachers. Statistical results indicate that an instrument that produces reliable and valid scores was developed to measure teachers’attitudes toward teaching physical activity and fitness. Investigating teachers’ attitudes in this area will lead to a better understanding of the role that physical activity and fitness currently play in physical education and may help to define their role in the future.
Minsize2: a Computer Program for Determining Effect Size and Minimum Sample Size for Statistical Significance for Univariate, Multivariate, and Nonparametric Tests
Authors: Morse David T.
Published: July 2003

The American Psychological Association’s editorial style urges authors to provide effect size estimates. Several journals, including Educational and Psychological Measurement, have adopted author guidelines that call for determining the minimum sample size necessary for a given result to have been declared statistically significant. MINSIZE2, a computer program that permits the user to determine both effect size and the minimum sample size needed for the results of a given analysis to be statistically significant, is described. Program applications for statistical significance tests for univariate, multivariate, and nonparametric statistics are provided. Explanation of the program’s operation is given, and examples are furnished.
Testing the Equality of Independent Alpha Coefficients Adjusted for Test Length
Authors: Alsawalmeh Yousef M. , Feldt Leonard S.
Published: July 2003

In comparing the merits of different item types or test formats, test developers are often called on to make a rigorous comparison of the internal consistency reliabilities of the scores from two competing tests. Such comparisons should be made for tests of equal length in terms of time demands and speededness. If the experimental instruments used in the field trials were not equal in length, estimates of the reliabilities of scores on equal-length tests can be obtained by applying the Spearman-Brown formula. In this article, the authors develop a statistical test for the hypothesis that [.alpha]1[.minute] = [.alpha]2[.minute], where [.alpha]i[.minute] is the extrapolated value of Cronbach’s alpha reliability for Test i. The test is shown to exercise tight control of Type I error.
Setting the Value of K in Promax: A Monte Carlo Study
Authors: Tataryn Douglas J. , Wood James M. , Gorsuch Richard L.
Published: July 2003

A Monte Carlo study involving 10,080 factor analyses examined the optimal value of k for promax factor rotations. The value of k was varied from 2 to 10 using three versions of promax. Error and bias of the sample factor pattern were found to be lower when k [.lessequal] 5 than when k > 5 but changed only slightly as k varied between 2 and 5. The best value of k was 2, 3, or 4, depending on the version of promax. It is concluded that in factor analytic studies using promax, the value of k may be appropriately set at 2, 3, or 4.
Statistically Differentiating Between Interaction and Nonlinearity in Multiple Regression Analysis: A Monte Carlo Investigation of a Recommended Strategy
Authors: Kromrey Jeffrey D. , Foster-Johnson Lynn
Published: July 2003

The procedure recommended by Lubinski and Humphreys for differentiating between moderated and nonlinear regression models is shown to evidence statistical problems that are characteristic of stepwise procedures. A Monte Carlo study was conducted in which random samples were generated from populations with known nonlinear or non-additive forms. Factors manipulated in the study included the correlation between regressors, regressor reliability, population R2, effect size of the nonlinear or nonadditive components, and sample size. Type I error was not adequately controlled under any condition examined and statistical power was poor in many conditions. Results were interpreted in terms of researchers’ need to differentiate between exploratory (stepwise regression) and confirmatory (theory testing) aspects of inquiry.
Multiple Approaches to Analyzing Count Data in Studies of Individual Differences: The Propensity for Type I Errors, Illustrated with the Case of Absenteeism Prediction
Authors: Sturman Michael C.
Published: July 2003

The present study compares eight models for analyzing count data: ordinary least squares (OLS), OLS with a transformed dependent variable, Tobit, Poisson, overdispersed Poisson, negative binomial, ordinal logistic, and ordinal probit regressions. Simulation reveals the extent that each model produces false positives. Results suggest that, despite methodological expectations, OLS regression does not produce more false positives than expected by chance. The Tobit and Poisson models yield too many false positives. The negative binomial models produce fewer than expected false positives.