Approved For Release 2000/08/08 : CIA-RDP96-00789R002200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 13 STUDY QUALITY Study Quality Criteria Since target stimuli in precognition experiments are selected only after the subjects’ responses have been registered, precognition studies are usually not vulnerable to sensory leakage problems. Other potential threats to validity that must be, however, considered. For our analysis of study quality, statistical and methodological variables are defined and coded in terms of procedural descriptions (or their absence) in the research reports. One point is given (or withheld) for each of the following eight criteria: Specification of Sample Size. Does the investigator preplan the number of trials to be included in the study or is the study vulnerable to the possibility of optional stopping? Credit is given to reports that explicitly specify the sample size. Studies involving group testing, in which it is not feasible to specify the sample size precisely, are also given credit. No credit is given to studies in which the sample size is either not preplanned or not addressed in the experimental report. Preplanned Analysis. Is the method of statistical analysis, including the outcome (dependent variable) measure, preplanned? Credit is given to studies explicitly specifying the form of analysis and the outcome measure. No credit is given to those not explicitly stating the form of the analysis or those in which the analysis is clearly post hoc. Randomization Method. Credit is given for use of random number tables, random number generators, and mechanical shufflers, but not for hand shuffling, die casting, or drawing lots. Controls. Credit is given to studies reporting randomness control checks, such as random number generator (RNG) control series and empirical cross-check controls. Recording. One point is allotted for automated recording of targets and responses and another for duplicate recording. Checking. One point is allotted for automated checking of matches between target and response and another for duplicate checking of hits. Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789R002200410001-2
i | Bette anal by anaes i ‘earactenee, Pine ead te cnmaanemetiad i Approved For Release 2000/08/08 : CIA-RDP96-00789R002200410001-2 Meta-analysis of Forced-Choice Precognition Experiments 14 Study Quality Analysis Each study received a quality weight between 0 and 8 (mean = 3.3, sd = 1.8). We find no relationship between study quality and effect size for either the directional (r24g = .029, p = .646, 2-tailed) or predicted (7246 = .006, p = .919, 2-tailed) effect sizes. Nor are any of the eight individual quality measures significantly related to effect size (Table 5). TABLE 5: Point-biserial Correlations Quality Measure ESair ESprea Sample size specified in advance -.146 -017 Preplanned analysis -.042 -.002 Randomization -.085 -.051 Controls 036 .004 Automated recording .139 -.016 Duplicate recording .054 074 Automated checking 105 -.023 Duplicate checking 015 035 The mean effect sizes by quality level are displayed graphically in Figure 2 (directional outcomes) and Figure 3 (predicted outcomes). FIGURE 2: Directional Outcomes in relation to Study Quality (95% Confidence Limits.) 0.100 0.075 0.0890 0.025 0.000 —0.025 —0.050 MEAN DIRECTIONAL EFFECT SIZE —0.076 2) G4) G3) 2) (7) 3) ~—-0.100 STUDY QUALITY Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789R002200410001-2
Approved Fer Ratiics geen wey el -RDP9 ion apersate | 1000- -2 ta-ana recognition 15 FIGURE 3: Predicted Outcomes in relation to Study Quality (95% Confidence Limits.) 6.100 0.075 0.060 0.000 —0.025 —~0.060 MEAN PREDICTED EFFECT SIZE —0.075 —0.100 STUDY QUALITY Quality Extremes Is there a tendency for extremely weak studies to show larger effects than exceptionally “good” studies? The grouped data presented in Figures 2 and 3 suggest that this is not the case and analysis on the extremes of the quality ratings indicates that the methodologically superior studies actually have somewhat larger mean effect sizes than studies with weaker methodology. This analysis uses studies with quality ratings outside the interquartile range of the rating distribution (median = 3, Q1 = 2, Q3 = 4). There are 46 studies at each extreme (“low quality” = ratings of 0-1, “high quality” = ratings of 5-8). The high quality studies have Jarger effect sizes than the low quality studies in both the directional and predicted analyses. For the directional analysis, the effect size means are 0.034 (sd = 0.061) and 0.016 (sd = 0.091), for the high and low quality studies respectively (¢ = -1.09, 90 df, p = .278, 2-tailed). For the predicted analysis, the effect size means are 0.038 (sd = 0.059) and 0.023 (sd = 0.089), for the high and low quality studies respectively (t = -0.90, 90 df, p = .368, 2-tailed). Quality Variation in Publication Sources Study quality does vary significantly across the five publication sources. Although neither significance level nor effect size are significantly related to source of publication, the five journals do vary significantly in quality Psychophysical Research Laboratories Approved For Release 2000/08/08 : CIA-RDP96-00789R002200410001-2