(288 | The Journal of Parapsychology ‘TABLE 2 oO SIGNIFICANCE LEVEL AND EFFECT S1zE FOR TRIMMED SAMPLE : . oO z ES —- fan 0.38 0.012 & 1.45 0.065 @rer 95% confidence estimate 0.23 0.005 QI Combined z = 6.02, p = 1.1 x 107° S WES) = 2.90, 247 df, p = .002 é ao icant outcomes at the 5% level. The mean (investigator) ES is 90 (SD = .05). © For the trimmed sample, the difference in ES across investiga- ss Is not significant: Kruskal-Wallis one-way ANOVA by ranks, 56) = 59.34, p = .355. If investigators contributing more than ee studies are eliminated, leaving 37 investigators, the combined zas still 5.00 (6 = 3.0 x 107’) and the mean ES is 0.022 (SD = £56). Figure 2 shows the mean effect size by investigator. O Thus, elimination of the outliers does not substantially affect the ggrclusions drawn from our analysis of the database as a whole. Erere clearly is a nonchance effect. In the remainder of this report, @ use the trimmed sample to examine covariations in effect size @ad a variety of methodological and other study features. 0 oO oO ° STUDY QUALITY =) © Because target stimuli in precognition experiments are selected oBly after the subjects’ responses have been registered, precognition gydies are usually not vulnerable to sensory leakage problems. Gther potential threats to validity must, however, be considered. Pe problem of variations in research quality remains a source of Geptroversy in meta-analysis. Some meta-analysts advocate eliminat- u@ low quality studies whereas others recommend empirically ac- ‘Bring the impact of variations in quality on study outcome. Rosen- tl (1984) points out that the practice of discarding studies is eaent to assigning them weights of zero, and he recommends weighting study z scores in relation to ratings of research Study Quality Criteria Ideally, the assessment of study quality should be performed by knowledgeable specialists who are blind to the study outcomes. In A Meta-Analysis of Forced-Choice Precognition Experiments 289 _ ! re) = 2,0 = Oo) 9? = 0 O 6 5 = 3° O oO a oO @ b q oO fe) o a a) oS one oO = P fe) N a 9 Oo N S = ° © fe) S 3 a 10 2 4 o - O19 co a Mo S = ge ° 19 9 co 1 > fe) ' ou [e] 8 Q fe) $, < ' O -0.20 -015 -0.10 -0.05 0.00 0.05 0.10 0.15 0.20 S) : © Mean effect size r=) oO Figure 2. Mean effect size by investigator for trimmed sample. N = 57 ife vestigators. N , ® practice, this is usually not feasible, particularly when, as in the pres ent case, large numbers of studies are involved. For our analysis ® study quality, statistical and methodological variables are defined and coded in terms of procedural descriptions (or their absence) ff the research reports. This approach: was used in an earlier metas analysis of psi ganzfeld research (Honorton, 1985), and it led oa study quality ratings that were generally in agreement, 7(26) = 768 p = 107°, with independent “flaw” ratings by an outside critic (Hp man, 1985). ss One point is given (or withheld) for each of the following eight criteria:  < Specification of sample size. Does the investigator preplan the num- ber of trials to be included in the study or is the study vulnerable to the possibility of optional stopping? Credit is given to reports that explicitly specify the sample size. Studies involving group testing, in which it is not feasible to specify the sample size precisely, are also
290 ° The Journal of Parapsychology given credit. No credit is given to studies in which the sample size is either not preplanned or not addressed in the experimental re- 9 port. © ~Preplanned analysis. Is the method of statistical analysis, including S the outcome (dependent variable) measure, preplanned? Credit is “= given to studies explicitly specifying the form of analysis and the © outcome measure. No credit is given to those not explicitly stating q the form of the analysis or those in which the analysis is clearly post QS hoe. © — Randomization method. Credit is given for use of random number o tables, random number generators, and mechanical shuiflers. No Nd credit is given for failure to randomize (i.e., use of “quasi-random © naturalistic events”) or for informal methods such as hand-shuffling, 9° die-casting, and drawing lots. © Conirols. Credit is given to studies reporting randomness control O checks, such as random number generator (RNG) control series and O¢ <mpirical cross-check controls. , Recording. One point is allotted for automated recording of tar- gets and responses, and another for duplicate recording. Checking. One point is allotted for automated checking of matches between target and response, and another for duplicate checking of hits. 6 D CIA Study Quality Analysis Each study received a quality weight between 0 and 8 (mean = 3.3, SD = 1.8). We find no significant relationship between study quality and £S: 7(246) = .081, p = .202, two-tailed. ‘This tendency for study outcomes to correlate positively with study quality has the consequence that the quality-weighted z score of 6.26 1s slightly larger than the unweighted z of 6.02. ‘Table 3 shows the correlations between effect size and each of the eight individual quality meas- ures.’ The mean effect sizes by quality level are displayed graphi- cally in Figure 3. pproved For Release 2000/08/08 * The correlation between ES and study quality is also nonsignificant for the un- trimmed sample of 309 studies: r(307) = —.060, p = .289. The quality-weighted z score is 7.38: p = 2.32 x 10°™. However, three of the individual quality measures are significantly related to performance. Controls and duplicate checking correlate significantly positively with ES, and randomization correlates significantly negatively with ES. These correlations appear to be due to a few studies with z scores that are extreme outliers (z > 7). When the 10 studies with z > 7 are eliminated, the signifi- cant correlations between quality and £S disappear. A A Meta-Analysis of Forced-Choice Precognition Experiments 291 TABLE 3 CORRELATIONS BETWEEN Errecr SIzk AND QUALITY MEASURES ° Quality measure r(246) oO fo] Sample size specified in advance — .100 f=) Preplanned analysis — .001 co Randomization — .Oll S Controls 058 P< Automated recording 169 N : . fo] Duplicate recording 047 oO Automated checking .136 [aa Duplicate checking .078 4 MM . fo] Quality Extremes =) o Is there a tendency for extremely weak studies to show largerp effects than exceptionally “good” studies? Analysis on the extremes QO of the quality ratings indicates that this is not the case. . This analysis, based on the untrimmed sample of 309 studies, S uses studies with quality ratings outside the interquartile range of O the rating distribution (median = 4, Q; = 2, Qs = 5). There are 56 bo “low-quality” studies (ratings of 0-1) and 35 “high-quality” studies © (ratings of 6~8). The high-quality studies have effect sizes that are 00 not significantly lower than the low-quality studies; the ES meansS R 0 oO are 0.017 (SD = 0.063) and 0.037 (SD = 0.137), for the low- and© high-quality studies, respectively: 482) = —.92, p = .358, two-& tailed. oo Quality Variation in Publication Sources ro) ® a Precognition ES is not significantly related to source of publica- tion: Kruskal-Wallis one-way ANOVA, x*(4). = 0.78, p = .942. re} However, the sources of publication differ significantly in studyle quality: Kruskal-Wallis one-way ANOVA, x*(4) = 17.19, p = .002.3 This is due largely to the lower quality of studies published m the > Journal of the Society for Psychical Research and in Research in Parapsy- 2 chology. 2 Study Quality in Relation to Year of Publication Precognition effect size has remained constant over a half-cen- tury of research, even though the methodological quality of the re-
292 ‘ The Journal of Parapsychology CIA-RDP96-06789RG02200610001-0 -0.10 — 0.05 0.00 0.05 0.10 Mean effect size 0/08/08 Fighe 3. Precognition effect size in relation to study quality, with 95% con- fidange limits. N = 248 studies. seath has improved significantly during this period. The correla- tiorgpbetween ES and year of publication is —.071: 1(307) = — 1.25, p =®213, two-tailed. Study quality and year of publication are, how- ev positively and significantly correlated: 1(246) = .282, p = 2 x 106 two-tailed. UGritics of parapsychology have long believed that evidence for patgbsychological effects disappears as the methodological rigor in- cremes. ‘The precognition database does not support this belief. , a a « ng REAL-TIME” ALTERNATIVES TO PRECOGNITION Investigators have long been aware of the possibility that precog- nition effects could be modeled without assuming either ume rever- sal or backward causality. For example, outcomes from studies with A Meta-Analysis of Forced-Choice Precognition Experunents 2938 targets based on indeterminate random number generators (RNGs) could be due to a causal influence on the RNG—a psychokinetic (PK) effect—rather than information acquisition concerning its fu- ture state. In experiments with targets based on prepared tables of random numbers, the possibility exists that the experimenter or other randomizer may be the actual psi source, unconsciously using “real-time” ESP combined with PK to choose an entry point in the random .number sequence that will significantly match the “sub- ject’s” responses. While the latter possibility may seem far-fetched, it cannot be logically eliminated if one accepts the existing evidence for contemporaneous ESP and PK, and it has been argued that it is less far-fetched than the alternative of “true” precognition. Morris (1982) discusses models of experimental precognition based on “real-time” psi alternatives and methods for testing “true” precognition. In general terms, these methods constrain the selec- tion of the target sequence so as to eliminate nonprecognitive psi intervention. In the most common procedure, attributed to Mangan (1955), dice are thrown to generate a set of numbers that are math- ematically manipulated to obtain an entry point in the random num- ber table. This procedure is sufficiently complex “as to be appar- ently beyond the capacities of the human brain, thus ruling out PK because the ‘PKer’ would not know what to do even via ESP” (Mor- ris, 1982, p. 329). ‘Two features of precognition study target determination proce- dures were coded to assess “real-time” psi alternatives to precogni- | tion: method of determining random number table entry point and use of Mangan’s method. : Methods of eliminating “real-time” psi alternatives have not been used in studies with random number generators and have only been used in a small number of studies involving randomization by hand- shuffling. These analyses are therefore restricted to studies using random number tables (NV = 138). - Method of Determining RNT Entry Point The reports describe six different methods of obtaining entry. points in random number tables. If the study outcomes were due to subjects’ precognitive functioning rather than to alternative psi modes on the part of the experimenter or the experimenter’s as- sistants, there should be no difference in mean effect size across the various methods used to determine the entry point. Indeed, our analysis indicates that the study effect sizes do not vary systemati- : CIA-RDP96-00789R002200610001-0 Approved For Release 2000/08/08