tenure-track position at a major university.
If our p is > .05 it means ruin, despair.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
ard our advisor’s suddenly thinking of a new control condition that should be run.
That attitude really must go. God loves the .06 nearly as much as the .05.
Indeed, I
have it on good authority that she views the strength of evidence for or against the
null as a fairly continuous function of the magnitude ofp. As a matter of fact, two .06
results are much stronger evidence against the null than one .05 result; and 10 p’s of
.10 are stronger evidence against the null than 5 p’s of .05.
The overthrow of the omnibus
test.
It
is common
to find specific questions
addressed by F tests with df >
1 in the numerator or by X
2 tests with df >
1.
For
example, suppose the specific question is whether increased incentive level improves
the productivity of work groups. We employ four levels of incentive so that our
omnibus F test would have 3 df in the numerator or our omnibus X
2 would be on at
least 3 df. Common as these tests are, they reflect poorly on our teaching of data
analytic procedures. The diffuse hypothesis tested by these omnibus tests usually
tells us nothing of importance about our research question. The rule of thumb
is
unambiguous: Whenever we have tested a fixed effect with df >
1 for X
2 or for the
numerator of F, we have tested a question in which we are almost surely not
interested.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
The situation is even worse when there are several dependent variables as well
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
as multiple o'/ for the independent variable. The paradigm case here is canonical
correlation and special cases are MANOVA, MANCOVA, Multiple discriminant
function, multiple path analysis, and complex multiple partial correlation. While all
of these procedures have useful exploratory data analytic applications they are
commonly used to test null hypotheses which are scientifically almost always of
doubtiul value. The effect size estimates they yield (e.g., the canonical correlation)
are also almost always of doubtful value.
This
is not the place to go into detail, but one approach to the problem of
analyzing canonical data structures
is to reduce the set of dependent variables to
some smaller number of composite variables using the principal-components-
follov. ed-by- unit-weighting approach. Each composite can then be analyzed serially.
Meta-analytic questions are basically contrast questions. F tests with df> 1 in
the numerator or X
2 s with df > 1 are useless in meta-analytic work. That leads to
an additional scientific benefit:
The increased recognition ofcontrast analysis. Meta-analytic questions require
precise formulation of questions and contrasts are procedures for obtaining answers
to such questions, often
in an analysis of variance or table analysis context.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Although most textbooks of statistics describe the
logic and the machinery of
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
contrast analyses, one still sees contrasts employed all too rarely. That is a real pity
given the precision of thought and theory they encourage and (especially relevant to
these times of publication pressure) given the boost in power conferred with the
resulting increase in .05 asterisks (Rosenthal & Rosnow, 1985).
A probable increase
in the accurate understanding of interaction effects.
Probably the universally most misinterpreted empirical results in psychology are
the results of interaction effects. A recent survey of 191 research articles involving
interactions found only two articles that showed the authors interpreting inter-
actions in an unequivocally correct manner (i.e., by examining the residuals that
define the interaction) (Rosnow & Rosenthal, 1989). The rest of the articles simply
compared means of conditions with other means,
a procedure that does not
investigate interaction effects but rather the sum of main effects and interaction
effects.
Most standard textbooks of statistics
for psychologists provide accurate
mathematical definitions of interaction effects but then interpret not the residuals
that define those interactions but the means of cells that are the sums of all main
effects and all interactions.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
In addition, users of SPSS, SAS, BMDP, and virtually all other data-analytic
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
software are poorly served in the matter of interactions since virtually no programs
provide convenient tabular output giving the residuals defining interaction.
The
only exception to that of which I am aware is a little-known package called Data-
Text developed by Arthur Couch and David Armor for which William Cochran and
Donald Rubin provided the statistical consultation.
Since many meta-analytic questions are by nature questions of interaction (for
example, that opposite sex dyads will conduct standard transactions more slowly
than will same sex dyads), we can be hopeful that increased use of meta-analytic
procedures will bring with
it increased sophistication about the meaning of
interaction.
Meta-analytic procedures are applicable beyond meta -analyses.
Many of the
techniques of contrast analyses among effect sizes, for example, can be used within a
single study (Rosenthal & Rosnow, 1985).
Computing a single effect size from
correlated dependent variables, or comparing treatment effects on two or more
dependent variables serve as illustrations (Rosenthal & Rubin, 1986).
The decrease in the splendid detachment of the full professor.
Meta-analytic
work requires careful reading of research and moderate data analytic skills. We
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
cannot send an undergraduate research assistant to the library with a stack of 5 X 8
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
cards to bring us back ’’the results.” With narrative reviews that seems often to have
been done. With meta-analysis the reviewer must get involved with the actual data
and that is all to the good.
Conclusion
I hope that this paper has provided some comfort to the afflicted in showing
that many of the findings of our discipline are neither as small nor as unimportant
from a practical point of view as we may have feared. Perhaps I hope, too, that there
may have been some affliction of the comfortable in showing that in our views of
replication and of the cumulation of the wisdom of our field there
is much yet
remaining to be done.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
References
Approved For Release 2000/08/1 0
: CIA-RDP96-00792R0001 001 30006-7
Barnes, D. M. (1986). Promising results halt trial of anti-AIDS drug. Science, 234,
15-16.
Canadian Multicentre Transplant Study Group. (1983). A randomized clinical trial
of cyclosporine in cadaveric renal transplantation. New England Journal of
Medicine, 309, 809-815.
Centers for Disease Control Vietnam Experience Study. (1988). Health status of
Vietnam veterans:
1.
Psychosocial characteristics.
Journal of the American
Medical Association, 259, 2701-2707.
Cohen, J. (1962). the statistical power of abnormal-social psychological research: A
review. Journal ofAbnormal and Social Psychology, 65, 145-153.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences, 2nd ed.,
Hillsdale, NJ: Lawrence Erlbaum Associates.
Collins, H.M. (1985). Changing Order: Replication and Induction in Scientific
Practice. Beverly Hills, CA: Sage.
Cooper, H. M., & Rosenthal, R. (1980). Statistical versus traditional procedures for
summarizing research findings. Psychological Bulletin, 87, 442-449.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Elashoff, J. D. (1978). Box scores are for baseball. The Behavioral and Brain
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Sciences, 3, 392.
Fiske,D.W. (1978). The several kinds of generalization. The Behavioral and Brain
Sciences, 3, 393-394.
Glass, G. V. (1976). Primary, secondary, and meta-analysis of research.
Educational Researcher, 5, 3-8.
Glass, G. V. (1978). In defense of generalization. The Behavioral and Brain
Sciences, 3, 394-395.
Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-Analysis in Social Research.
Beverly Hills, CA: Sage.
Hedges, L. V. (1982). Estimation of effect size from a series of independent
experiments. Psychological Bulletin, 92, 490-499.
Hedges, L. V. (1987). How hard is hard science, how soft is soft science? American
Psychologist, 42, 443-455.
Jung, J. (1978). Self-negating functions of self-fulfilling prophecies. The Behavioral
and Brain Sciences, 3, 397-398.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Lamb, W. K., & Whitla, D. K. (1983). Meta-Analysis and the Integration ofResearch
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Findings: A Trend Analysis and Bibliography Prior to 1983.
Unpublished
manuscript, Harvard University, Cambridge.
Mayo, R„ J. (1978). Statistical considerations in analyzing the results of a collection
of experiments. The Behavioral and Brain Sciences, 3, 400-401.
Nelson, N., Rosenthal, R., & Rosnow, R. L. (1986). Interpretation of significance
levels and effect sizes by psychological researchers. American Psychologist, 41
,
1299-1301.
Pool,R. (1988). Similar experiments, dissimilar results. Science, 242, 192-193.
Rimland, B. (1979). Death knell for psychotherapy? American Psychologist, 34,
192.
Rosenthal, R. (1966). Experimenter Effects in Behavioral Research. New York:
Appleton-Century-Crofts.
Rosenthal, R. (1969). Interpersonal expectations. In R. Rosenthal and R. L. Rosnow
(Eds.), Artifact in Behavioral Research (pp. 181-277). New York:
Academic
Press.
Rosenthal, R. (1979a). The "file drawer problem” and tolerance for null results.
Psychological Bulletin, 86, 638-641.
Approved For Release 2000/08/10
: CIA-RDP9&-00792R0001 001 30006-7
Rosenthal, R. (1979b). Replications and their relative utilities. Replications in
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Social Psychology, 1(1), 15-23.
Rosenthal, R. (1984). Meta-Analytic Procedures for Social Research. Beverly Hills,
CA: Sage.
Rosenthal, R. (1986). Nonsignificant relationships as scientific evidence.
Behavioral and Brain Sciences, 9, 479-481.
Rosenthal, R., & Gaito, J. (1963). The interpretation of levels of significance by
psychological researchers. Journal ofPsychology, 55, 33-38.
Rosenthal, R., & Gaito, J. (1964). Further evidence for the cliff effect in the
interpretation of levels of significance. Psychological Reports, 15, 570.
Rosenthal, R., & Rosnow, R. L. (1984). Essentials ofBehavioral Research: Methods
and Data Analysis. New York: McGraw-Hill.
Rosenthal, R., & Rosnow, R. U (1985). Contrast Analysis: Focused Comparisons in
the Analysis of Variance. New York: Cambridge University Press.
Rosenthal, R., & Rosnow, R. L. (in press). Essentials ofBehavioral Research:
Methods and Data Analysis , 2nd ed.. New York: McGraw-Hill.
Rosenthal, R., & Rubin, D. B. (1978). Interpersonal expectancy effects: The first 345
studies. The Behavioral and Brain Sciences, 3, 377-386.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Rosenthal, R., & Rubin, D. B. (1979). Comparing significance levels ofindependent
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
studies. Psychological Bulletin, 86. 1165-1168.
Rosenthal, R., & Rubin, D. B. (1982a). Comparing effect sizes of independent
studies. Psychological Bulletin, 92, 500-504.
Rosenthal, R., & Rubin, D. B. ( 1982b). A simple, general purpose display of
magnitude of experimental effect. Journal ofEducational Psychology, 74, 166-
169.
Rosenthal, R., & Rubin, D. B. (1985). Statistical analysis: Summarizing evidence
versus establishing facts. Psychological Bulletin, 97, 527-529.
Rosenthal, R., & Rubin, D. B. (1986). Meta-analytic procedures for combining
studies with multiple effect sizes. Psychological Bulletin, 99, 400-406.
Rosenthal, R., & Rubin, D. B. (1988). Comment: Assumptions and procedures in the
file drawer problem. Statistical Science, 3, 120-125.
Rosenthal, R., & Rubin, D. B. (in press). Effect size estimation for one-sample
multiple-choice-type data: Design, analysis, and meta-analysis. Psychological
Bulletin.
Rosnow, R. L., & Rosenthal, R. (1989). Definition and interpretation of interaction
effects. Psychological Bulletin, 105, 143-146.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Sedlmeier, P., & Gigerenzer, G. (in press). Do studies of statistical power have an
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
effect on the power of studies? Psychological Bulletin.
Skinner, B. F. (1983, August). On the value of research. APA Monitor, p. 39.
Smith, M. L., & Glass, G. V (1977). Meta-analysis of psychotherapy outcome
studies. American Psychologist, 32
,
752-760.
Snedecor, G. W., & Cochran, W. G. (1980). Statistical Methods (7th ed.). Ames: Iowa
State University Press.
Steering Committee of the Physicians Health Study Research Group. (1988).
Preliminary report:
Findings from the aspirin component of the ongoing
physicians’ health study. The New England Journal ofMedicine, 318, 262-264.
Tukey, J. W. (1977). Exploratory Data Analysis. Reading, MA: Addison-Wesley.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Author Notes
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 3000V
This paper was presented as an EPA Distinguished Lecture at
t
V
\
Meeting of the Eastern Psychological Association, Boston, April 2, 19/
tion of this paper was supported in part by the National Science Fount
the author was a Fellow at the Center for Advanced Study in the Behavioral
Sciences.
I am grateful for financial support provided by the John D. and Catherine
T. MacArthur Foundation, and for improvements suggested by Lynn Gale, Deanna
Knickerbocker, Harold Luft, and Lincoln Moses.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Apprcatadfor Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Effects of Aspirin on Heart Attacks Among 22,000 Physicians
Heart Attack
No Heart
Attack
Total
I. Raw Counts
Aspirin
104
10,933
11,037
Placebo
189
10,845
11,034
Total
293
21,778
22,071
II. Percentages
Aspirin
0.94
99.06
100
Placebo
1.71
98.29
100
Total
1.33
HI. Binomial Effect Size Display
98.67
100
Aspirin
48.3
51.7
100
Placebo
51.7
48.3
100
Total
100
100
200
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Other Examples of Binomial Effect Size Displays
I. Vietnam Service and Alcohol Problems (r = .07)
Problem
No Problem
Total
Vietnam Veteran
53.5
46.5
100
Non-Vietnam Veteran
46.5
53.5
100
Total
100
100
200
II. AZT in the Treatment ofAIDS (r = .23)
Death
Survival
Total
AZT
38.5
61.5
100
Placebo
61.5
38.5
100
Total
100
100
200
Eli.
Benefits ofPsychotherapy (r = .32)
Greater
Less Benefit
Benefit
Total
Psychotherapy
100
Control
100
Total
100
100
200
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Tahfcj4>roved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Common Model of Successful Replication: Judgment is Dichotomous and Based on
Significance Testing
p < .05 a
Second
Study
p > .05
First Study
p > .05*
p < .05
A
B
Failure to
Successful
Replicate
Replication
C
D
Failure to
Failure to
Establish Effect
Replicate
*By convention .05 but could be any other given level, e.g, .01.
aIn the same tail as the results of the first study.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Table 4
IAippraAied BE«mRsfa^asa£^6/flfiA10an6M&RE^dd4d6tf92R0001 001 30006-7
Investigator
I. Smith
II. Jones
Combined
Treatment Mean
.38
.36
.376
Control Mean
.26
.24
.256
Difference
.12
.12
.120
t
2.21
1.06
2.45
df
two-toil p
.03
.30
.02
effect size da
.50
.50
.50
effect size rb
.24
.24
.24
standard normal Z
2.17c
1.03 c
2.40
95% Confidence intervals
From:
.01
-.12
.02
Mean differences
To:
.23
.36
.22
From:
.02
-.23
.04
Effect size r’s
To:
.44
.62
.42
a Obtained from 2itVdjf.
b Obtained from Vt2 /(i2 +ti/)
.
c These significance levels differ at Z = .81, p — .42 from
(ZrZ2)/V2.
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
Table 5
C(AWJ»l»ftCldF®hJ
:lilS#^)gfl0pWia41(fti6CIA-RDP96-OO792ROOO1 001 30006-7
Replication Sets
Study 1
A
Study 2
B
Study 1
Study 2
C
Study 1
Study 2
N
p (two-tail)
.05
.05
.01
.18
.000001
.33
Zip)
1.96
1.96
2.58
1.34
4.89
-0.97
r
.20
.50
.26
.26
.72
-.18
Zir)
.20
.55
.27
.27
.90
-.18
Cohen’s q (Zr
Z r
)
rl~
r2
.35
.00
1.08
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7