Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Meta-analysis of Forced-Choice Precognition Experiments
5% level and 30 investigators (52.6%) have significant predicted outcomes.
The mean (investigator) directional effect size is 0.023 (sd = .052), and the
meanpredicted effect size is 0.028 (sd = .047). Both results remain above 0
on lower-bound 99.9% confidence estimates.
Thus, elimination of the outliers does not substantially affect the
conclusions drawn from our analysis of the database as a whole.
There
clearly is a nonchance effect.
In the remainder of this report, we use the
adjusted sample to examine covariations in magnitude of effect and a variety
of methodological and other study features.
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Meta-analysis of Forced-Choice Precognition Experiments
STUDY QUALITY
Study Quality Criteria
Since target stimuli in precognition experiments are selected only after
the subjects’ responses have been registered, precognition studies are
usually not vulnerable to sensory leakage problems. Other potential threats
to validity that must be, however, considered.
For our analysis of study
quality, statistical and methodological variables are defined and coded in
terms of procedural descriptions (or their absence) in the research reports.
One point is given (or withheld) for each of the following eight criteria:
Specification of Sample Size. Does the investigator preplan the number
of trials to be included in the study or is the study vulnerable to the possibility
of optional stopping?
Credit is given to reports that explicitly specify the
sample size.
Studies involving group testing, in which
it is not feasible to
specify the sample size precisely, are also given credit. No credit is given to
studies in which the sample size is either not preplanned or not addressed
in the experimental report
Preplanned Analysis.
Is the method of statistical analysis, including the
outcome (dependent variable) measure, preplanned?
Credit is given to
studies explicitly specifying the form of analysis and the outcome measure.
No credit is given to those not explicitly stating the form of the analysis or
those in which the analysis is clearly post hoc.
Randomization Method. Credit is given for use ofrandom number tables,
random number generators, and mechanical shufflers, but not for hand
shuffling, die casting, or drawing lots.
Controls. Credit is given to studies reporting randomness control checks,
such as random number generator (RNG) control series and empirical
cross-check controls.
Recording. One point is allotted for automated recording of targets and
responses and another for duplicate recording.
Checking.
One point
is allotted for automated checking of matches
between target and response and another for duplicate checking of hits.
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Meta-analysis of Forced-Choice Precognition Experiments
Study Quality Analysis
Each study received a quality weight between 0 and 8 (mean =
3.3,
sd -
1.8). We find no relationship between study quality and effect size for
either the directional
(rz46 =
.029, p =
.646, 2-tailed) or predicted
(/246 = .006, p =
.919, 2-tailed) effect sizes.
Nor are any of the eight
individual quality measures significantly related to effect size (Table 5).
TABLE 5: Point-biserial Correlations
Quality Measure
ES&xt
-ESpred
Sample size specified in advance
-.146
-.017
Preplanned analysis
-.042
-.002
Randomization
-.085
-.051
Controls
.036
.004
Automated recording
.139
-.016
Duplicate recording
.054
.074
Automated checking
.105
-.023
Duplicate checking
.015
.035
The mean effect sizes by quality level are displayed graphically in Figure
2 (directional outcomes) and Figure 3 (predicted outcomes).
FIGURE 2: Directional Outcomes in relation to Study Quality
(95% Confidence Limits.)
STUDY QUALITY
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2
Approved Fpr Release 2000/08/08
: CIA-RDP96-00789RQ02200410001-2
Meta-analysis or Forced-Choice Precognition Experiments
FIGURE 3: Predicted Outcomes in relation to Study Quality
(95% Confidence Limits.)
STUDY QUALITY
Quality Extremes
Is there a tendency for extremely weak studies to show larger effects than
exceptionally “good” studies? The grouped data presented in Figures 2 and
3 suggest that this is not the case and analysis on the extremes of the quality
ratings indicates that the methodologically superior studies actually have
somewhat larger mean effect sizes than studies with weaker methodology.
This analysis uses studies with quality ratings outside the interquartile
range of the rating distribution (median = 3, Q\ = 2, Qs - 4). There are
46 studies at each extreme (“low quality” = ratings of 0-1, “high quality” =
ratings of 5-8). The high quality studies have larger effect sizes than the low
quality studies in both the directional and predicted
analyses. For the
directional analysis, the effect size means are 0.034 (sd — 0.061) and 0.016
(sd = 0.091), for the high and low quality studies respectively (/ = -1.09, 90
df, p = .278, 2-tailed). For the predicted analysis, the effect size means are
0.038 (sd = 0.059) and 0.023 (sd =
0.089), for the high and low quality
studies respectively (t = -0.90, 90 df,p = .368, 2-tailed).
Quality Variation In Publication Sources
Study quality does vary significantly across the five publication sources.
Although neither significance level nor effect size are significantly related
to source of publication, the five journals do vary significantly in quality
Psychophysical Research Laboratories
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200410001-2