datamatters24/research-document-archive huggingface dataset:
significance of heterogeneity of these Fisher Z’s is desired, the three references above Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 all provide the appropriate formula for computing the Xx? test of heterogeneity as does Hedges (1982), Meta-analytic metrics. As the number of replications for a given research question grows, a full assessment of the success of the replicational effort requires the application of meta-analytic procedures. An informative but slightly unwieldy summary of the meta-analysis might be the stem-and-leaf display of the effect sizes found in the meta-analysis (Tukey, 1977). A more compact summary of the effect sizes might be Tukey’s (1977) box plot, which gives the highest and lowest obtained effect sizes along with those found at the 25th, 50th, and 75th percentiles. For single index values of the consistency of the effect sizes, one could employ (a) the range of effect sizes found between the 75th (Q,) and 25th (Q,) per-entile, (b) some standard fraction of that range (e.g., half or three-quarters), (c) S, the standard deviation of the effect sizes, or (d) SE, the standard error of the effect sizes. Asa slightly more complex index of the stability, replicability, or clarity of the average effect size found in the set of replicates, one could employ the mean effect size divided either by its standard error (S/V k where k is the total number of Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
nee ane” mettle of uARPRAYed Fas Retease 2000108/10..,ClAcRDP9B:N0Z92R0004001 300067): research? Was -’ .90, or .80, or .70, or .60, so that the corresponding r’s would ‘.ave been .95, .89, .84, or .772 No. Well, was r? 50, .40, .30, or even .20, so that the corresponding r’s would have been .71, .63, .55, or .45? No. Actually, what r was, was .0011, with a corresponding r of .034. Insert Table 1 about here Table 1 shows the results of the aspirin study in terms of raw couats, per- centages, and as a Binomial Effect Size Display (BESD). This display is a way of showing the practical importance of any effect indexed by a correlation coefficient. The correlation is shown to be the simple difference in outcome rates between the experimental and the control groups in this standard table which always adds up to column totals of 100 and row totals of 100 (Rosenthal & Rubin, 1982b). This type of result seen in the physicians’ aspirin study is not at all unusual in biomedical research. Some years earlier, on October 29, 1981, the National Heart, Lung, and Blood Institute discontinued its placebo-controlled study of propranolol because results were so favorable to the treatment that it would be unethical to Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
replicates), or simply by S. The latter index of mean effect size divided by its Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 standard deviation (S) is the reciprocal of the coefficient of variation or a kind of coefficient of robustness. The coefficient of robustness of replication. Although the standard error of the mean effect size along with confidence intervals placed around the mean effect size are of great value (Rosenthal & Rubin, 1978), it will sometimes be useful to employ a robustness coefficient that does not increase simply as a function of the increasing number of replications. Thus, if we want to compare two research areas for their robustness, adjusting for the difference in number of replications in each research area, we may prefer the robustness coefficient defined as the reciprocal of the coefficient of variation. The utility of this coefficient is based on two ideas--first, that replication success, clarity, or robustness depends on the homogeneity of the obtained effect size; and second, that it depends also on the unambiguity or clarity of the directionality of the result. Thus, a set of replications grows in robustness as the variance of the effect sizes decreases and as the distance of the mean effect size from zero increases. Incidentally, the mean may be weighted, unweighted, or trimmed (Tukey, 1977). Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
~~ -—s continue Withholding the life-saving drug from the control patients. And What Was the nlagnitude of this ettect?- Once again the eect size r was-04, and the leading digits of the r° were .00! As behavioral researchers we are not used to thinking of r's of .04 as reflecting effect sizes of practical importance. But when we think of an r of .04 as reflecting a 4% decrease in heart attacks, the interpretation given rin a Binomial Effect Size Display, the r does not appear to be quite so small; especially if we can count ourselves among the 4 per 100 who manage to survive. Insert Table 2 about here Additional Results Table 2 gives three further examples of Binomial Effect Size Displays. In a recent study of 4,462 Army veterans of the Vietnam War era (1965-1971), the correlation between having served in Vietnam (rather than elsewhere) and having suffered from alcohol abuse or dependence was .07 (Centers for Disease Control, 1988). The top display of Table 2 shows that the difference between the problem rates of 53.5 and 46.5 per 100 is equal to the correlation coefficient of .07. The center display of Table 2 shows the results of a study of the effects of AZT on the survival of 282 patients suffering from AIDS or AIDs-related complex (ARC) Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
AZ Regprevefl. Fat) Releas d 20NA1084406 GIL RDInag pORABREAetaonsdOdethe clinical trial on the ethical grounds that it would be improper to continue to give placebo to the control group patients. | As a footnote to this display let me add the result of a small informal poll I took a few weeks ago of some physicians spending the year at the Center for Advanced Study in the Behavioral Sciences. I asked them to tell me of some medical break- through that was of very great practical importance. Their consensus was that the breakthrough was the effect of cyclosporine in increasing the probability that the body would not reject an organ transplant and that the recipient patient would not die. A multi-center randomized experiment was published in 1983 (Canadian Multicentre Transplant Study Group, 1983). The results of this breakthrough experiment were less dramatic than the results of the AZT study. For the dependent variable of organ rejection the effect size r was .19 (r? = .036); for the dependent variable of patient survival the effect size r was .15 (r?=.022). The bottom display of Table 2 shows the results of a famous meta-analysis of psychotherapy outcome studies reported by Smith and Glass (1977). An eminent critic (Rimland, 1979) believed that the results of their analysis sounded the “death Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 size was an r of .32 accounting for “only 10% of the variance.” Examination of the bottom display of Tablé-2 shows that it is not very realistic to label as “modest indeed” an effect size equivalent to increasing a success rate from 34% to 66% (for example, reducing a death rate or a failure rate from 66% to 34%). Indeed, as we have seen, the dramatic effects of AZT were substantially smaller (r = .23), and the “breakthrough” effects of cyclosporine were smaller still (r = .19). Telling How Well We're Doing The Binomie! Effect Size Display is a useful way to display the practical magni- tude of an effect size regardless of whether the dependent variable is dichotomous or continuous (Rosenthal & Rubin, 1982b). An especially useful feature of the display is how easily we can go from the display to an r (just take the difference between the success rates of the experimerital versus the control group) and how easily we can go from an effect size r to the display (just compute the treatment success rate as .50 plus one-half of rand the control success rate as .50 minus one-half ofr). One effect of the standard use of a display procedure such as the Binomial Effect Size Display to index the practical value of our research results would be to give us more useful and more realistic assessments of how well we are really doing as Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
researchers in the social and behavioral sciences. Employment of the Binomial Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 Effect Size Display has, in fact, shown that we are doing considerably better in our “softer, wilder” sciences than we may have thought we were doing. So far, our conversation has been intended to comfort the afflicted. In what follows the intent is a bit more to afflict the comfortable. We begin with the topic of replication. | The Meaning of Successful Replication There is a long tradition in psychology of our urging one another to replicate each other's research. Indeed, there seems to be something nearly scriptural about it--I quote: “If a scholar’s work be deemed unreplicable then shall ye gladly cast that scholar out.” (That’s from either Referees I or Editors J, I believe.) Now, while we have been very good at calling for replications we have not been too good at deciding when a‘replication has been successful. The issue we now address is: When shall a study be deemed successfully replicated? Successful replication is ordinarily taken to mean that a null hypothesis that has been rejected at time 1 is rejected again, and with the same direction of outcome, on the basis of a new study at time 2. The basic model of this usage can be seen in Table 3. The results of the first study are described dichotomously as p < .05 or p > Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
Approved For Release JBERTHSPES® TARCRENSSE-00792R000100130006-7 .05 (or some other critical level, e.g., .01). Each of these two possible outcomes is further dichotomized as to the results of the second study as p<.05 or p>.05. Thus, cells A and D of Table 3 are examples of failure to replicate because one study was significant and the other was not. Let us examine more closely a specific example of such a “failure to replicate.” Pseudo-Failures to Replicate The saga of Smith and Jones. Smith has published the results of an experiment in which a certain treatment procedure was predicted to improve performance. She reported results significant at p<.05 in the predicted direction. Jones publishes a rebuttal to Smith claiming a failure to replicate. Insert Table 4 about here Table 4 shows the results of these two experiments in greater detail. Smith’s results were more significant than Jones's, to be sure, but the studies were in perfect agreement as to their estimated sizes of effect as defined either by Cohen's d [(Mean, - Mean ,)/ 0] or by r, the correlation between group membership and performance Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
score (Cohen, 1977; 1988; Rosenthal, 1984). Notonlydidthe effect sizesofthe two = Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 studies agree, but even the significance levels of .03 and .30 did not differ very significanisy: (Z 9, -Z 55)/ V2 = (2.17-1.03)/V2 = Z = 81, p = .42; for details on the comparison of significance levels and effect sizes see Rosenthal and Rubin (1979; 1982a) or asummary in Rosenthal (1984). Table 4 shows very clearly that Jones was very much in error when he claimed that his study failed to replicate that of Smith. Such errors are made very frequently in most areas of psychology and the other behavioral and social sciences. The final column of Table 4 shows that the combined result of both experiments is associated with a more significant ¢t and with a smaller confidence interval (for the difference between the means and for the effect size r) than is either of the individual studies. On: the odds against replicating significant results. A related error often found in the behavioral and social sciences is the implicit assumption that if an effect is “real,” we should therefore expect it to be found significant again upon replication. Nothing could be further from the truth. Suppose there isin nature a real effect with a magnitude out there in the world of d = .50 (i.e., (Mean, - Mean,] /o = .50 9 units), or, equivalently, r = .24 (a difference in success rate of 62% versus 38%). Then suppose an investigator studies Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
WaT UT Ue SUIeTtS OF SU, BIVilig tie researcuier da lever ul statistical powePA BEd FA RELRASR FART PLR a UT REAR AP TeAR MOPAR TERA IEE¢ 30 years (Cohen, 1962; S.ilmeier & Gigerenzer, in press). Even though a d of .50 or an r of .24 is a very important effect (as we saw earlier in this paper), there is only one chance in four that both the original investigator and a replicator will get results significant at the .05 level. If there were two replications of the original study there would be only one chance in eight that all three studies would be significant, even though we know the effect in nature is very real and very important. If five studies investigated this phenomenon, there is only 4 50:50 chance that three or more of them would find significant results. In short, given the levels of statistical power at which we normally operate, we have no right to expect the proportion of significant results that we typically do expect, even ifin nature there is avery real and very important effect. Pseudo-Successful Replications Returning now to Table 3, we focus attention on cell B, the cell of “successful replication.” Suppose that two investigators both rejected the null hypothesis at p<.05 with both results in the same direction. Suppose further, however, that in one study the effect size r was .90 while in the other study the effect size r was only .10, Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
significantly smaller than the r of .90 (Rosenthal & Rubin, 1982a). In this case our Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 interpretation is more complex. We have indeed had a successful replication of the rejection of the null hypothesis but we have not come even close to a successful replication of the effect size. “Successful Replication” of Type 2 Error Cell C of Table 3 represents the situation in which both studies failed to reject the null hypothesis. Under those conditions investigators might conclude that there was no relationship between the variables investigated. Such a conclusion could be very much in error, the more so as the power of the two studies was low (Cohen, 1977; 1988; Rosenthal, 1986). If power levels of the two studies (assuming medium effect sizes in the population) were very high, say .90 or .95, then two failures to obtain a significant relationship would :rovide evidence that the effect investigated was not likely to be a very large effect. If power calculations had been made assuming a very small effect size, two failures to reject the null while not providing strong evidence for the null would at least suggest that the size of the effect in the population was probably quite modest. If sample sizes of the two studies failing to reject the null were small so that power to detect all but the largest effects was low, very little could be concluded from Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
two failures to reject except that the effect sizes were unlikely to be enormous. For Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 example, two investigators with N’s of 20 and 40, respectively, find results not significant at p<.05. The effect sizes phi (i.e., ; for dichotomous variables) were .29 and .20, respectively, and both p’s were approximately .20. The combined p of these two results, however, is .035 [(Z, +Z,) / V2 = Z|, and the mean effect size in the mid- .20’s is not trivial, as we saw earlier in this paper. Contrasting Views of Replication The traditional, not very useful view of replication modeled in Table 3 has two primary characteristics: (1) It focuses on significance level as the relevant summary statistic of a study, and (2) It makes its evaluation of wheter replication has been successful in a dichotomous fashion. ‘For example, replications are successful if both or neither p<.05 (or .01, etc.), and they are unsuccessful if one p<.05 (or .01, etc.) and the other p>.05 (or .01, ete.).. Psychologists’ reliance on a dichotomous decision procedure accompanied by an untenable discontinuity of credibility in results varying in p levels has been well documented (Nelson, Rosenthal, & Rosnow, 1986; Rosenthal & Gaito, 1963, 1964). Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
The newer, more useful views of replication success have two primary Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 characteristics: 1. A focus on effect size as the more important summary statistic of a study with only a relatively minor interest in the statistical significance level, and 2. An evaluation of whether replication has been successful made in a con- tinuous fashion. For example, two studies are not said to be successful or unsuccess- ful replicates of each other, but rather the degree of failure to replicate is specified. Insert Table 5 about here Table 5 shows three sets of replications. Replication set A shows two results both rejecting the null but with a difference in effect sizes of .30 in units of r or .35 in units of Fisher’s Z transformation of r (Cohen, 1977, 1988; Rosenthal & Rosnow, 1984; Snedecor & Cochran, 1980), That difference, in units of r or Fisher's Z is the degree of failure to replicate. The fact that both studies were able to reject the null and at exactly the same p level is simply a function of sample size. Replication set B shows two studies with different p values, one significant at <.05, the other not significant. However, the two effect size estimates are in excellent agreement. We would say, accordingly, that replication set B shows more successful replication than Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
EC tts ee ee oe eee Approved For Release 2000gi40 RGJArfRaPY6-00792R0001001 30663 Harvard University My talk today is designed in part both to comfort the afflicted and to afflict the comfortable. The afflicted are those of us who work in the softer, wilder areas of our field--the areas where the results seem ephemeral and unreplicable, and where the rs seem always to be approaching zero as a limit. These softer, wilder areas include those of social, personality, clinical, developmental, educational, organizational, and health psychology. They also include parts of psvchobiology and cognitive psy- chology. These softer, wilder areas, however, may not include too much of psychophysics. My message to those of us toiling in these muddy vineyards will be that we are doing better that we might have thought. My message to those of us in any areas in which we feel we have-pretty well nailed things down will be that we haven't, and that we could be doing a whole lot better. How Large Must an Effect Be, To Be Im ortant? There is a bit of good news-bad news abroad in the land. The good news is that more sophisticated editors, referees, and researchers are becoming aware that reporting the results of a significance test is not a sufficiently en:ighteningprocedure Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
: * Approved'Fdr Reléase 2000/08/10 :*CIA-RDP96to0792Rb002003f0005-7 * * * * * * * UNCLASSIFIED k kK kk kK Rk kK kK kK Kk kK Rk RK RK RK KR KK Rk KR RK KR KR KR KR KR KR kK KR KK SG1J SEQUENCE NR: CIF90005966  USER-[— ID: 43070-92APR28/10.52.06/A01 PFN INFORMATION: PART 029 RELATES: . MOZHUANG LIU IS A MEMBER OF HEBEI PROVINCIAL ACAD OF MEDICAL SCIENCES. XIUREN YAO IS A MEMBER OF CHINESE ACAD OF GEOLOGICAL SCIENCES. PERSON: LIU, MOZHUANG PROTITLE: SENIOR DOCTOR PERS CTY: PEOPLES REPUBLIC OF CHINA (CH) PERSON: YAO, XIUREN PROTITLE: RESEARCH FELLOW PERS CTY: PEOPLES REPUBLIC OF CHINA (CH) FACILITY: HEBEI PROVINCIAL ACAD OF MEDICAL SCIENCES FACL CTY: PEOPLES REPUBLIC OF CHINA (CH) FACILITY: CHINESE ACAD OF GEOLOGICAL SCIENCES FACL CTY: PEOPLES REPUBLIC OF CHINA (CH) PFN INFORMATION: PART 030 RELATES: SHUNLIU ZONG IS A MEMBER OF GENERAL LOGISTICS DEPARTMENT. SHUNLIU ZONG IS ASSOCIATED WITH PLA. PERSON: ZONG, SHUNLIU PROTITLE: DEPUTY DIRECTOR PERS CTY: PEOPLES REPUBLIC OF CHINA (CH) PERSON: ZONG, SHUNLIU PROTITLE: DEPUTY DIRECTOR PERS CTY: PEOPLES REPUBLIC OF CHINA (CH) FACILITY: GENERAL LOGISTICS DEPARTMENT FACL CTY: PEOPLES REPUBLIC OF CHINA (CH) FACLDATA: FACILITY NAME EQUATE AVAILABLE FACILITY: PLA FACL CTY: PEOPLES REPUBLIC OF CHINA (CH) PFN INFORMATION: PART 031 FACILITY: ARMY AVIATION FACL CTY: PEOPLES REPUBLIC OF CHINA (CH) FACILITY: NAVAL AIR FORCE kok Kk Rk wk Kk Kk Rk RK Kk kK Rk Kk OR KR Rk KR KR KR KR Rk kK Rk kk KK RK RK KK UNCLASSIFIED PAGE 66 kok RK Kk KR Rk RK Kk Rk RK KR KR KR KR KR RR KR Rk KR KR KR KR Kk RK RK kK RK Approved For Release 2000/08/10 : CIA-RDP96-00792R000200310005-7
does replication set A.
Replication set C shows two studies differing markedly in
Approved For Release 2000/08/10
: CIA-RDP96-00792R0001 001 30006-7
both level of significance and magnitude (and direction) of effect size. Replication set
C, then, is a not very subtle example of a clear failure to replicate.
It should be noted that the values of Table 5 were chosen so that the combined
probability of the two studies of sets A, B, and C would
all be identical
to one
another; (Z
t + Z2 ) / V2 = Z of 2. 77, p =
.0023, one-tailed.
Some Metrics of the Success of Replication
Once we adopt a view of the success of replication as a function of similarity of
effect sizes obtained, we can become more precise in our assessments of the success of
replication.
Insert Figure 1 about here
The replication diagonal. Figure 1 shows the "replication plane” generated by
crossing the results of the first study conducted (expressed in units of the effect size r)
by the results of the second study conducted. All perfect replications, those in which
the effect sizes are identical in the two studies, fall on a diagonal rising from the
lower left corner (-1.00, -1.00) to the upper right corner ( + 1.00, +1.00). The results
of replication set B from Table 5 are shown to fall exactly on the diagonal of perfect
Approved For Release 2000/08/10
; CIA-RDP96-00792R0001 001 30006-7
Table 5 Coppnésedof th Relsase 20 091N8110,;5 CIA-RDP96-00792R000100130006-7 Replication Sets A B C Study1 Study2 Studyl Study2 Studyl1 Study2 ON 96 15 98 27 12 32 p (two-tail) 05 05 01 18 .000001 33 Z(p) 196 1.96 2.58 1.34 4.89 ~0.97 r 20 50 26 26 72 -.18 Zr) .20 155 27 27 90 -.18 Cohen’s q(Z,, _Z;,) 35 00 1.08 Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
Table 6 SeveAPREvesbF Vn Relenge(20 CPMRI NS; fe. AAR DR On00 J A2RANB 120d SAQQEG.00 Degree of Variability Replicate Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 .02 10 .20 .40 .60 80 1.00 .O1 .05 10 20 30 .40 50 00 00 00 00 00 00 00 -.01 -.05 -10  -=.20 -.30 —.40 -.90 -.02 -.10 -.20 -.40 -.60 ~.80 -1.00 S .016 O79 .158 | 316 474 .632 791 Range 04 .20 .40 .80 1.20 1.60 2.00 Equal Steps of 01 05 10 .20 30 40 50 Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
Table 7 Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7 Replication Robustness Coefficients for Four Levels of Mean Effect Size (Zr) and Six Degrees of Vari-.oility of Effect Size (S) -Mean Effect Size (Zr) S 10 30 50 .70 Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
oy Table 8 | ProbAeRERy ee 25 RS OB AAS IPR Glas ROME NQOY PRO AR WO AA OARET G0 Studies of Table 4 StudyI: Smith Probability of Not Probability of Rejecting False Null Rejecting False Null (Type Il Error Rate = (Power = .60) x 40) Study II: Jones Probability of Rejecting False Null (Power = .18) 18 Probability of Not Rejecting False Nuil (Type 0 Error Rate 82 = .82) Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7
e- Figure I Approved For Release,?00 08110 CIARDP96-00792R000100130006-7 ication 1.00 -80 60 -40 20 RESULTS 00 OF SECOND -.20 STUDY (r) -.40 -.60 -.80 -10 -10 -80 -60 -.40 -.20 00 .20 .40 60 .80~ 1.00 Approved For Release 2000/08/10 : CIA-RDP96-00792R000100130006-7